Search Results: "tach"

24 September 2014

Junichi Uekawa: Sending GCM send message from nodejs.

Sending GCM send message from nodejs. I wanted to send a message from emacs, but it seemed to be relatively difficult to send HTTPS POST request from emacs, so I just decided to use handy nodejs. The attached 'data' is attached to intent as extra, so it can be extracted by intent.getExtras.getString("message") in IntentService#onHandleIntent()

22 September 2014

Gunnar Wolf: One month later: How is the set of Debian keyrings faring?

OK, it's almost one month since we (the keyring-maintainers) gave our talk at DebConf14; how are we faring regarding key transitions since then? You can compare the numbers (the graphs, really) to those in our DC14 presentation. Since the presentation, we have had two keyring pushes: First of all, the Non-uploading keyring is all fine: As it was quite recently created, and as it is much smaller than our other keyrings, it has no weak (1024 bit) keys. It briefly had one in 2010-2011, but it's long been replaced. Second, the Maintainers keyring: In late July we had 222 maintainers (170 with >=2048 bit keys, 52 with weak keys). By the end of August we had 221: 172 and 49 respectively, and by September 18 we had 221: 175 and 46. As for the Uploading developers, in late July we had 1002 uploading developers (481 with >=2048 bit keys, 521 with weak keys). By the end of August we had 1002: 512 and 490 respectively, and by September 18 we had 999: 531 and 468. Please note that these numbers do not say directly that six DMs or that 50 uploading DDs moved to stronger keys, as you'd have to factor in new people being added, keys migrating between different keyrings (mostly DM DD), and people retiring from the project; you can get the detailed information looking at the public copy of our Git repository, particularly of its changelog. And where does that put us? Of course, I'm very happy to see that the lines in our largest keyring have already crossed. We now have more people with >=2048 bit keys. And there was a lot of work to do this processing done! But that still means... That in order not to lock a large proportion of Debian Developers and Maintainers out of the project, we have a real lot of work to do. We would like to keep the replacement slope high (because, remember, in January 1st we will remove all small keys from the keyring). And yes, we are willing to do the work. But we need you to push us for it: We need you to get a new key created, to gather enough (two!) DD signatures in it, and to request a key replacement via RT. So, by all means: Do keep us busy!
AttachmentSize
Debian Developers (uploading)266.66 KB
Debian Developers (non-uploading)204.17 KB
Debian Maintainers296.73 KB

18 September 2014

Dariusz Dwornikowski: RFS health in Debian

I am working on a small project to create WNPP like statistics for open RFS bugs. I think this could improve a little bit effectiveness of sponsoring new packages by giving insight into bugs that are on their way to being starved (i.e. not ever sponsored, or rotting in a queue). The script attached in this post is written in Python and uses Debbugs SOAP interface to get currently open RFS bugs and calculates their dust and age. The dust factor is calculated as an absolute value of a difference between bugs's age and log_modified. Later I would like to create fully blown stats for an RFS queue, taking into account the whole history (i.e. 2012-1-1 until now), and check its health, calculate MTTGS (mean time to get sponsored). The list looks more or less like this:
Age  Dust Number  Title
37   0    757966  RFS: lutris/0.3.5-1 [ITP]
1    0    762015  RFS: s3fs-fuse/1.78-1 [ITP #601789] -- FUSE-based file system backed by Amazon S3
81   0    753110  RFS: mrrescue/1.02c-1 [ITP]
456  0    712787  RFS: distkeys/1.0-1 [ITP] -- distribute SSH keys
120  1    748878  RFS: mwc/1.7.2-1 [ITP] -- Powerful website-tracking tool
1    1    762012  RFS: fadecut/0.1.4-1
3    1    761687  RFS: abraca/0.8.0+dfsg-1 -- Simple and powerful graphical client for XMMS2
35   2    758163  RFS: kcm-ufw/0.4.3-1 ITP
3    2    761636  RFS: raceintospace/1.1+dfsg1-1 [ITP]
....
....
The script rfs_health.py can be found below, it uses SOAPpy (only python <3 unfortunately).
#!/usr/bin/python
import SOAPpy
import time
from datetime import date, timedelta, datetime
url = 'http://bugs.debian.org/cgi-bin/soap.cgi'
namespace = 'Debbugs/SOAP'
server = SOAPpy.SOAPProxy(url, namespace)
class RFS(object):
    def __init__(self, obj):
        self._obj = obj
        self._last_modified = date.fromtimestamp(obj.log_modified)
        self._date = date.fromtimestamp(obj.date)
        if self._obj.pending != 'done':
            self._pending = "pending"
            self._dust = abs(date.today() - self._last_modified).days
        else:
            self._pending = "done"
            self._dust = abs(self._date - self._last_modified).days
        today = date.today()
        self._age = abs(today - self._date).days
    @property
    def status(self):
        return self._pending
    @property
    def date(self):
        return self._date
    @property
    def last_modified(self):
        return self._last_modified
    @property
    def subject(self):
        return self._obj.subject
    @property
    def bug_number(self):
        return self._obj.bug_num
    @property
    def age(self):
        return self._age
    @property
    def dust(self):
        return self._dust
    def __str__(self):
        return "  subject:   age:  dust: ".format(self._obj.bug_num, self._obj.subject, self._age, self._dust)
if __name__ == "__main__":
    bugi = server.get_bugs("package", "sponsorship-requests", "status", "open")
    buglist = [RFS(b.value) for b in server.get_status(bugi).item]
    buglist_sorted_by_dust = sorted(buglist, key=lambda x: x.dust, reverse=False)
    print("Age  Dust Number  Title")
    for i in buglist_sorted_by_dust:
        print(" :<4   :<4   :<7   ".format(i.age, i.dust, i.bug_number, i.subject))

Jaldhar Vyas: Scotland: Vote NO

        _  __<;
      </_/ _/__   
     /> >  7   )  
     ~;</7    /   
     /> /   _*<---- Perth    
     ~ </7  7~\_  
        </7     \ 
         /_ _ _   
If you don't, the UK will have to rename itself the K. And that's just silly. Also vote yes on whether Alex Trebek should keep his mustache.

12 September 2014

Dariusz Dwornikowski: forwarding messages with attachments in mutt

This is a pain for every mutt user. I do not know why this solution is so hard to find. Just add these two lines to your .muttrc.
set mime_forward
set mime_forward_rest=yes
This will forward an email with all the attachments, no scripts needed, no fancy tagging or reediting.

11 September 2014

Steve Kemp: A small email utility and other updates.

Last night I was looking for an image I knew a model had mailed me a few months ago, as we were talking about rescheduling a shoot at the weekend. I couldn't find it, even with my awesome mail client and filing system. With some free time I figured I could write a little utility to dump all attachments from email folders, and find it that way. It did cross my mind that there is the simple mail-utility for dumping headers, etc, called formail, which is distributed alongside procmail, but it doesn't handle attachments .. I was tempted to write a general purpose script to dump attachments, email header values, etc, etc but given the lack of time I merely solved my own problem. I suspect there is room for a "mail utilities" package, similar to Joey's "moreutils" and my "sysadmin utils". However I note that there is a GNU Mailutils which does things differently than I'd expect - i.e. it contains a POP3 server. Still if you want to dump attachments from emails, have GMIME installed, and want to filter by attachment-name, or MIME-type, you might look at my trivial attachment-dump program. Related to that I spent some time last night updating my photography site, so the animals & pets section has updated images at least. During the course of that I found a bug in my static-site generator, templer which stopped it from automatically populating image height/widths when called in a glob:
Title: Pets &amp; Animals
Images: file_glob( "*.jpg" )
---
This is the page body, it now has access to a variable called 'images'
which is a HTML::Template loop-structure containing name/height/width/etc
for each image in the current directory.
That should now be resolved, and life should once again be good.

4 September 2014

Steve Kemp: systemd, a brave new world

After spending a while fighting with upstart, at work, I decided that systemd couldn't be any worse and yesterday morning upgraded one of my servers to run it. I have two classes of servers: I thought it would be a fair test to upgrade one of each systems, to see how it worked. The Debian wiki has instructions for installing Systemd, and both systems came up just fine. Although I realize I should replace my current runit jobs with systemd units I didn't want to do that. So I wrote a systemd .service file to launch runit against /etc/service, as expected, and that was fine. Docker was a special case. I wrote a docker.service + docker.socket file to launch the deamon, but when I wrote a graphite.service file to start a docker instance it kept on restarting, or failing to stop. In short I couldn't use systemd to manage running a docker guest, but that was probably user-error. For the moment the docker-host has a shell script in root's home directory to launch the guest:
#!/bin/sh
#
# Run Graphite in a detached state.
#
/usr/bin/docker run -d -t -i -p 8080:80 -p 2003:2003 skxskx/graphite
Without getting into politics (ha), systemd installation seemed simple, resulted in a faster boot, and didn't cause me horrific problems. Yet. ObRandom: Not sure how systemd is controlling prosody, for example. If I run the status command I can see it is using the legacy system:
root@chat ~ # systemctl status prosody.service 
prosody.service - LSB: Prosody XMPP Server
      Loaded: loaded (/etc/init.d/prosody)
      Active: active (running) since Wed, 03 Sep 2014 07:59:44 +0100; 18h ago
      CGroup: name=systemd:/system/prosody.service
            942 lua5.1 /usr/bin/prosody
I've installed systemd and systemd-sysv, so I thought /etc/init.d was obsolete. I guess it is making pretend-services for things it doesn't know about (because obviously not all packages contain /lib/systemd/system entries), but I'm unsure how that works.

23 August 2014

Antti-Juhani Kaijanaho: A milestone toward a doctorate

Yesterday I received my official diploma for the degree of Licentiate of Philosophy. The degree lies between a Master s degree and a doctorate, and is not required; it consists of the coursework required for a doctorate, and a Licentiate Thesis, in which the student demonstrates good conversance with the field of research and the capability of independently and critically applying scientific research methods (official translation of the Government decree on university degrees 794/2004, Section 23 Paragraph 2). The title and abstract of my Licentiate Thesis follow:
Kaijanaho, Antti-Juhani
The extent of empirical evidence that could inform evidence-based design of programming languages. A systematic mapping study.
Jyv skyl : University of Jyv skyl , 2014, 243 p.
(Jyv skyl Licentiate Theses in Computing,
ISSN 1795-9713; 18)
ISBN 978-951-39-5790-2 (nid.)
ISBN 978-951-39-5791-9 (PDF)
Finnish summary Background: Programming language design is not usually informed by empirical studies. In other fields similar problems have inspired an evidence-based paradigm of practice. Central to it are secondary studies summarizing and consolidating the research literature. Aims: This systematic mapping study looks for empirical research that could inform evidence-based design of programming languages. Method: Manual and keyword-based searches were performed, as was a single round of snowballing. There were 2056 potentially relevant publications, of which 180 were selected for inclusion, because they reported empirical evidence on the efficacy of potential design decisions and were published on or before 2012. A thematic synthesis was created. Results: Included studies span four decades, but activity has been sparse until the last five years or so. The form of conditional statements and loops, as well as the choice between static and dynamic typing have all been studied empirically for efficacy in at least five studies each. Error proneness, programming comprehension, and human effort are the most common forms of efficacy studied. Experimenting with programmer participants is the most popular method. Conclusions: There clearly are language design decisions for which empirical evidence regarding efficacy exists; they may be of some use to language designers, and several of them may be ripe for systematic reviewing. There is concern that the lack of interest generated by studies in this topic area until the recent surge of activity may indicate serious issues in their research approach. Keywords: programming languages, programming language design, evidence-based paradigm, efficacy, research methods, systematic mapping study, thematic synthesis
A Licentiate Thesis is assessed by two examiners, usually drawn from outside of the home university; they write (either jointly or separately) a substantiated statement about the thesis, in which they suggest a grade. The final grade is almost always the one suggested by the examiners. I was very fortunate to have such prominent scientists as Dr. Stefan Hanenberg and Prof. Stein Krogdahl as the examiners of my thesis. They recommended, and I received, the grade very good (4 on a scale of 1 5). The thesis has been accepted for publication published in our faculty s licentiate thesis series and will in due course appear has appeared in our university s electronic database (along with a very small number of printed copies). In the mean time, if anyone wants an electronic preprint, send me email at antti-juhani.kaijanaho@jyu.fi.
Figure 1 of the thesis: an overview of the mapping processFigure 1 of the thesis: an overview of the mapping process
As you can imagine, the last couple of months in the spring were very stressful for me, as I pressed on to submit this thesis. After submission, it took me nearly two months to recover (which certain people who emailed me on Planet Haskell business during that period certainly noticed). It represents the fruit of almost four years of work (way more than normally is taken to complete a Licentiate Thesis, but never mind that), as I designed this study in Fall 2010.
Figure 8 of the thesis: Core studies per publication yearFigure 8 of the thesis: Core studies per publication year
Recently, I have been writing in my blog a series of posts in which I have been trying to clear my head about certain foundational issues that irritated me during the writing of the thesis. The thesis contains some of that, but that part of it is not very strong, as my examiners put it, for various reasons. The posts have been a deliberately non-academic attempt to shape the thoughts into words, to see what they look like fixed into a tangible form. (If you go read them, be warned: many of them are deliberately provocative, and many of them are intended as tentative in fact if not in phrasing; the series also is very incomplete at this time.) I closed my previous post, the latest post in that series, as follows:
In fact, the whole of 20th Century philosophy of science is a big pile of failed attempts to explain science; not one explanation is fully satisfactory. [...] Most scientists enjoy not pondering it, for it s a bit like being a cartoon character: so long as you don t look down, you can walk on air.
I wrote my Master s Thesis (PDF) in 2002. It was about the formal method called B ; but I took a lot of time and pages to examine the history and content of formal logic. My supervisor was, understandably, exasperated, but I did receive the highest possible grade for it (which I never have fully accepted I deserved). The main reason for that digression: I looked down, and I just had to go poke the bridge I was standing on to make sure I was not, in fact, walking on air. In the many years since, I ve taken a lot of time to study foundations, first of mathematics, and more recently of science. It is one reason it took me about eight years to come up with a doable doctoral project (and I am still amazed that my department kept employing me; but I suppose they like my teaching, as do I). The other reason was, it took me that long to realize how to study the design of programming languages without going where everyone has gone before. Debian people, if any are still reading, may find it interesting that I found significant use for the dctrl-tools toolset I have been writing for Debian for about fifteen years: I stored my data collection as a big pile of dctrl-format files. I ended up making some changes to the existing tools (I should upload the new version soon, I suppose), and I wrote another toolset (unfortunately one that is not general purpose, like the dctrl-tools are) in the process. For the Haskell people, I mainly have an apology for not attending to Planet Haskell duties in the summer; but I am back in business now. I also note, somewhat to my regret, that I found very few studies dealing with Haskell. I just checked; I mention Haskell several times in the background chapter, but it is not mentioned in the results chapter (because there were not studies worthy of special notice). I am already working on extending this work into a doctoral thesis. I expect, and hope, to complete that one faster.

20 August 2014

Gunnar Wolf: Bigger than the cloud

Summer is cool in Mexico City. It is cool because, unlike Spring, this is our rainy season And rains are very predictable. Almost every day we wake up with a gorgeous, clean, blue sky. Cool, nice temperature, around 15 C. The sun slowly evaporates the rain throughout the morning; when I go out for lunch, the sky is no longer so blue, giving way to a seemingly dirty white/grayish tint. No, it's not our world-famous pollution: It's just yesterday's rain. Rain starts falling usually between 4 and 7 PM. Sometimes it starts as a light rain, sometimes it starts with all of its thunder, all of its might. But anyway, almost every night, there is a moment of awe, of not believing how much rain we are getting today. It slowly fades away during the late night. And when I wake up, early next morning, everything is wet and still smells fresh. Yes, I love our summer, even though it makes shy away from my much enjoyed cycling to work and school. And I love taking some minutes off work, look through the window of my office (located ~70m over the level of our mostly flat city) and watching how different parts of the city have sun or rain; learning to estimate the distance to the clouds, adding it to the direction and guessing which of my friends have which weather. But I didn't realize our city had so clearly defined micro-climates... (would they really be *micro*-climates?) In fact, it even goes against my knowledge of Mexico City's logic I always thought Coyoac n, towards the South of the city, got more rain than the Center and North because we are near the mountains, and the dominant air currents go Southwards, "clumping" the clouds by us. But no, or at least, not this year. Regina (still in the far South Far because she's too far away from me and I'm too egocentric; she returns home after DebConf) often asks me about the weather, as our friends working nearer the center of the city. According to the photos they post on their $social_media_of_the_day accounts, rains are really heavier there. Today I heard on the radio accounts of yesterday's chaos after the rain. This evening, at ESIME-Culhuac n, I saw one of the reported fallen trees (of course, I am not sure if it's from yesterday's rain). And the media pushes galleries of images of a city covered in hail... While in Copilco we only had a regular rain, I'd even say a mild one. This city is bigger than any cloud you can throw at it.
AttachmentSize
IMG_20140819_155052.jpg929.69 KB

18 August 2014

Julien Danjou: OpenStack Ceilometer and the Gnocchi experiment

A little more than 2 years ago, the Ceilometer project was launched inside the OpenStack ecosystem. Its main objective was to measure OpenStack cloud platforms in order to provide data and mechanisms for functionalities such as billing, alarming or capacity planning. In this article, I would like to relate what I've been doing with other Ceilometer developers in the last 5 months. I've lowered my involvement in Ceilometer itself directly to concentrate on solving one of its biggest issue at the source, and I think it's largely time to take a break and talk about it. Ceilometer early design For the last years, Ceilometer didn't change in its core architecture. Without diving too much in all its parts, one of the early design decision was to build the metering around a data structure we called samples. A sample is generated each time Ceilometer measures something. It is composed of a few fields, such as the the resource id that is metered, the user and project id owning that resources, the meter name, the measured value, a timestamp and a few free-form metadata. Each time Ceilometer measures something, one of its components (an agent, a pollster ) constructs and emits a sample headed for the storage component that we call the collector. This collector is responsible for storing the samples into a database. The Ceilometer collector uses a pluggable storage system, meaning that you can pick any database system you prefer. Our original implementation has been based on MongoDB from the beginning, but we then added a SQL driver, and people contributed things such as HBase or DB2 support. The REST API exposed by Ceilometer allows to execute various reading requests on this data store. It can returns you the list of resources that have been measured for a particular project, or compute some statistics on metrics. Allowing such a large panel of possibilities and having such a flexible data structure allows to do a lot of different things with Ceilometer, as you can almost query the data in any mean you want. The scalability issue We soon started to encounter scalability issues in many of the read requests made via the REST API. A lot of the requests requires the data storage to do full scans of all the stored samples. Indeed, the fact that the API allows you to filter on any fields and also on the free-form metadata (meaning non indexed key/values tuples) has a terrible cost in terms of performance (as pointed before, the metadata are attached to each sample generated by Ceilometer and is stored as is). That basically means that the sample data structure is stored in most drivers in just one table or collection, in order to be able to scan them at once, and there's no good "perfect" sharding solution, making data storage scalability painful. It turns out that the Ceilometer REST API is unable to handle most of the requests in a timely manner as most operations are O(n) where n is the number of samples recorded (see big O notation if you're unfamiliar with it). That number of samples can grow very rapidly in an environment of thousands of metered nodes and with a data retention of several weeks. There is a few optimizations to make things smoother in general cases fortunately, but as soon as you run specific queries, the API gets barely usable. During this last year, as the Ceilometer PTL, I discovered these issues first hand since a lot of people were feeding me back with this kind of testimony. We engaged several blueprints to improve the situation, but it was soon clear to me that this was not going to be enough anyway.
Thinking outside the box Unfortunately, the PTL job doesn't leave him enough time to work on the actual code nor to play with anything new. I was coping with most of the project bureaucracy and I wasn't able to work on any good solution to tackle the issue at its root. Still, I had a few ideas that I wanted to try and as soon as I stepped down from the PTL role, I stopped working on Ceilometer itself to try something new and to think a bit outside the box. When one takes a look at what have been brought recently in Ceilometer, they can see the idea that Ceilometer actually needs to handle 2 types of data: events and metrics. Events are data generated when something happens: an instance start, a volume is attached, or an HTTP request is sent to an REST API server. These are events that Ceilometer needs to collect and store. Most OpenStack components are able to send such events using the notification system built into oslo.messaging. Metrics is what Ceilometer needs to store but that is not necessarily tied to an event. Think about an instance CPU usage, a router network bandwidth usage, the number of images that Glance is storing for you, etc These are not events, since nothing is happening. These are facts, states we need to meter. Computing statistics for billing or capacity planning requires both of these data sources, but they should be distinct. Based on that assumption, and the fact that Ceilometer was getting support for storing events, I started to focus on getting the metric part right. I had been a system administrator for a decade before jumping into OpenStack development, so I know a thing or two on how monitoring is done in this area, and what kind of technology operators rely on. I also know that there's still no silver bullet this made it a good challenge. The first thing that came to my mind was to use some kind of time-series database, and export its access via a REST API as we do in all OpenStack services. This should cover the metric storage pretty well. Cooking Gnocchi
A cloud of gnocchis!
At the end of April 2014, this led met to start a new project code-named Gnocchi. For the record, the name was picked after confusing so many times the OpenStack Marconi project, reading OpenStack Macaroni instead. At least one OpenStack project should have a "pasta" name, right? The point of having a new project and not send patches on Ceilometer, was that first I had no clue if it was going to make something that would be any better, and second, being able to iterate more rapidly without being strongly coupled with the release process. The first prototype started around the following idea: what you want is to meter things. That means storing a list of tuples of (timestamp, value) for it. I've named these things "entities", as no assumption are made on what they are. An entity can represent the temperature in a room or the CPU usage of an instance. The service shouldn't care and should be agnostic in this regard. One feature that we discussed for several OpenStack summits in the Ceilometer sessions, was the idea of doing aggregation. Meaning, aggregating samples over a period of time to only store a smaller amount of them. These are things that time-series format such as the RRDtool have been doing for a long time on the fly, and I decided it was a good trail to follow. I assumed that this was going to be a requirement when storing metrics into Gnocchi. The user would need to provide what kind of archiving it would need: 1 second precision over a day, 1 hour precision over a year, or even both. The first driver written to achieve that and store those metrics inside Gnocchi was based on whisper. Whisper is the file format used to store metrics for the Graphite project. For the actual storage, the driver uses Swift, which has the advantages to be part of OpenStack and scalable. Storing metrics for each entities in a different whisper file and putting them in Swift turned out to have a fantastic algorithm complexity: it was O(1). Indeed, the complexity needed to store and retrieve metrics doesn't depends on the number of metrics you have nor on the number of things you are metering. Which is already a huge win compared to the current Ceilometer collector design. However, it turned out that whisper has a few limitations that I was unable to circumvent in any manner. I needed to patch it to remove a lot of its assumption about manipulating file, or that everything is relative to now (time.time()). I've started to hack on that in my own fork, but then everything broke. The whisper project code base is, well, not the state of the art, and have 0 unit test. I was starring at a huge effort to transform whisper into the time-series format I wanted, without being sure I wasn't going to break everything (remember, no test coverage). I decided to take a break and look into alternatives, and stumbled upon Pandas, a data manipulation and statistics library for Python. Turns out that Pandas support time-series natively, and that it could do a lot of the smart computation needed in Gnocchi. I built a new file format leveraging Pandas for computing the time-series and named it carbonara (a wink to both the Carbon project and pasta, how clever!). The code is quite small (a third of whisper's, 200 SLOC vs 600 SLOC), does not have many of the whisper limitations and it has test coverage. These Carbonara files are then, in the same fashion, stored into Swift containers. Anyway, Gnocchi storage driver system is designed in the same spirit that the rest of OpenStack and Ceilometer storage driver system. It's a plug-in system with an API, so anyone can write their own driver. Eoghan Glynn has already started to write a InfluxDB driver, working closely with the upstream developer of that database. Dina Belova started to write an OpenTSDB driver. This helps to make sure the API is designed directly in the right way. Handling resources Measuring individual entities is great and needed, but you also need to link them with resources. When measuring the temperature and the number of a people in a room, it is useful to link these 2 separate entities to a resource, in that case the room, and give a name to these relations, so one is able to identify what attribute of the resource is actually measured. It is also important to provide the possibility to store attributes on these resources, such as their owners, the time they started and ended their existence, etc.
Relationship of entities and resources
Once this list of resource is collected, the next step is to list and filter them, based on any criteria. One might want to retrieve the list of resources created last week or the list of instances hosted on a particular node right now. Resources also need to be specialized. Some resources have attributes that must be stored in order for filtering to be useful. Think about an instance name or a router network. All of these requirements led to to the design of what's called the indexer. The indexer is responsible for indexing entities, resources, and link them together. The initial implementation is based on SQLAlchemy and should be pretty efficient. It's easy enough to index the most requested attributes (columns), and they are also correctly typed. We plan to establish a model for all known OpenStack resources (instances, volumes, networks, ) to store and index them into the Gnocchi indexer in order to request them in an efficient way from one place. The generic resource class can be used to handle generic resources that are not tied to OpenStack. It'd be up to the users to store extra attributes. Dropping the free form metadata we used to have in Ceilometer makes sure that querying the indexer is going to be efficient and scalable.
The indexer classes and their relations
REST API All of this is exported via a REST API that was partially designed and documented in the Gnocchi specification in the Ceilometer repository; though the spec is not up-to-date yet. We plan to auto-generate the documentation from the code as we are currently doing in Ceilometer. The REST API is pretty easy to use, and you can use it to manipulate entities and resources, and request the information back.
Macroscopic view of the Gnocchi architecture
Roadmap & Ceilometer integration All of this plan has been exposed and discussed with the Ceilometer team during the last OpenStack summit in Atlanta in May 2014, for the Juno release. I led a session about this entire concept, and convinced the team that using Gnocchi for our metric storage would be a good approach to solve the Ceilometer collector scalability issue. It was decided to conduct this project experiment in parallel of the current Ceilometer collector for the time being, and see where that would lead the project to. Early benchmarks Some engineers from Mirantis did a few benchmarks around Ceilometer and also against an early version of Gnocchi, and Dina Belova presented them to us during the mid-cycle sprint we organized in Paris in early July. The following graph sums up pretty well the current Ceilometer performance issue. The more you feed it with metrics, the more slow it becomes.
For Gnocchi, while the numbers themselves are not fantastic, what is interesting is that all the graphs below show that the performances are stable without correlation with the number of resources, entities or measures. This proves that, indeed, most of the code is built around a complexity of O(1), and not O(n) anymore.
Next steps
Cl ment drawing the logo
While the Juno cycle is being wrapped-up for most projects, including Ceilometer, Gnocchi development is still ongoing. Fortunately, the composite architecture of Ceilometer allows a lot of its features to be replaced by some other code dynamically. That, for example, enables Gnocchi to provides a Ceilometer dispatcher plugin for its collector, without having to ship the actual code in Ceilometer itself. That should help the development of Gnocchi to not be slowed down by the release process for now. The Ceilometer team aims to provide Gnocchi as a sort of technology preview with the Juno release, allowing it to be deployed along and plugged with Ceilometer. We'll discuss how to integrate it in the project in a more permanent and strong manner probably during the OpenStack Summit for Kilo that will take place next November in Paris.

16 August 2014

Matthias Klumpp: AppStream/DEP-11 Debian progress

There hasn t been a progress-report on DEP-11 for some time, but that doesn t mean there was no work going on on it. DEP-11 is Debian s implementation of AppStream, as well as an effort to enhance the metadata available about software in Debian. While initially, AppStream was only about applications, DEP-11 was designed with a larger scope, to collect data about libraries, binaries and things like Python modules. Now, since AppStream 0.6, DEP-11 and AppStream have essentially the same scope, with the difference of DEP-11 metadata being described in YAML, while official AppStream data is XML. That was due to a request by our ftpmasters team, which doesn t like XML (which is also not used anywhere in Debian, as opposed to YAML). But this doesn t mean that people will have to deal with the YAML file format: The libappstream library will just take DEP-11 data as another data source for it s Xapian database, allowing anything using libappstream to access that data just like the XML stuff. Richards libappstream-glib will also receive support for the DEP-11 format soon, filling it s in-memory data cache and enabling the use of GNOME-Software on Debian. So, what has been done so far? The past months, my Google Summer of Code student. Abhishek Bhattacharjee, was working hard to integrate DEP-11 support into dak, the Debian Archive Kit, which maintains the whole Debian archive. The result will be an additional metadata table in our internal Postgres database, storing detailed information about the software available in a Debian package, as well as Components-<arch>.yml.gz files in the Debian repositories. Dak will also produce an application icon-cache and a screenshots repository. During the time of the SoC, Abhishek focused mainly on the applications part of things, and less on the other components (like extracting data about Python modules or libraries) these things can easily be implemented later. The remaining steps will be to polish the code and make it merge-ready for Debian s dak (as soon as it has received enough testing, we will likely give it a try on the Tanglu Debian derivative). Following that, Apt will be extended to fetch the DEP-11 data on-demand on systems where it is useful (which is currently mostly desktop-systems) if you want to save a little bit of space, you will be able to disable downloading this extra metadata in Apt. From there, libappstream will take the data for it s Xapian db. This will lead to the removal of the much-hated (from ftpmasters and maintainers side) app-install-data package, which has not been updated for two years and only contains a small fraction of the metadata provided by DEP-11. What Debian will ultimately gain from this effort is support for software-centers like GNOME-Software, and improved support for tools like Apper and Muon in displaying applications. Long-term, with more metadata being available, It would be cool to add support for it to specialized package managers , like Python s pip, npm or gem, to make them fetch information about available distribution software and install that instead of their own copies from 3rd-party repositories, if possible. This should ultimatively lead to less code duplication on distributions and will likely result in fewer security issues, since the officially maintained and integrated distribution packages can easily be used, if possible. This is no attempt to make tools like pip obsolete, but an attempt to have the different tools installing software on your machine communicate better, instead of creating parallel worlds in terms of software management. Another nice sideeffect of more metadata will be options to search for tools handling mimetypes in the software repos (in case you can t open a file), smart software centers installing missing firmware, and automatic suggestions for developers which software they need to install in order to build a specific software package. Also, the data allows us to match software across distributions, for that, I will have some news soon (not sure how soon though, as I am currently in thesis-writing-mode, and therefore have not that much spare time). Since the goal is to have these features available on all distributions supporting AppStream, it will take longer to realize but we are on a good way. So, if you want some more information about my student s awesome work, you can read his blogpost about it. He will also be at Debconf 14 (Portland). (I can t make it this time, but I surely won t miss the next Debconf) Sadly, I only see a very small chance to have the basic DEP-11 stuff land in-time for Jessie (lots of review work needs to be done, and some more code needs to be written), but we will definitively have it in Jessie+1. A small example on how this data will look like can be found here a larger, actual file is available here. Any questions and feedback are highly appreciated.

15 August 2014

Steve Kemp: A tale of two products

This is a random post inspired by recent purchases. Some things we buy are practical, others are a little arbitrary. I tend to avoid buying things for the sake of it, and have explicitly started decluttering our house over the past few years. That said sometimes things just seem sufficiently "cool" that they get bought without too much thought. This entry is about two things. A couple of years ago my bathroom was ripped apart and refitted. Gone was the old and nasty room, and in its place was a glorious space. There was only one downside to the new bathroom - you turn on the light and the fan comes on too. When your wife works funny shifts at the hospital you can find that the (quiet) fan sounds very loud in the middle of the night and wakes you up.. So I figured we could buy a couple of LED lights and scatter them around the place - when it is dark the movement sensors turn on the lights. These things are amazing. We have one sat on a shelf, one velcroed to the bottom of the sink, and one on the floor, just hidden underneath the toilet. Due to the shiny-white walls of the room they're all you need in the dark. By contrast my second purchase was a mistake - The Logitech Harmony 650 Universal Remote Control should be great. It clearly has the features I want - Able to power: The problem is solely due to the horrific software. You program the device via an application/website which works only under Windows. I had to resort to installing Windows in a virtual machine to make it run:
# Get the Bus/ID for the USB device
bus=$(lsusb  grep -i Harmony   awk ' print $2 '   tr -d 0)
id=$(lsusb  grep -i Harmony   awk ' print $4 '   tr -d 0:)
# pass to kvm
kvm -localtime ..  -usb -device usb-host,hostbus=$bus,hostaddr=$id ..
That allows the device to be passed through to windows, though you'll later have to jump onto the Qemu console to re-add the device as the software disconnects and reconnects it at random times, and the bus changes. Sigh. I guess I can pretend it works, and has cut down on the number of remotes sat on our table, but .. The overwhelmingly negative setup and configuration process has really soured me on it. There is a linux application which will take a configuration file and squirt it onto the device, when attached via a USB cable. This software, which I found during research prior to buying it, is useful but not as much as I'd expected. Why? Well the software lets you upload the config file, but to get a config file you must fully complete the setup on Windows. It is impossible to configure/use this device solely using GNU/Linux. (Apparently there is MacOS software too, I don't use macs. *shrugs*) In conclusion - Motion-activated LED lights, more useful than expected, but Harmony causes Discord.

7 August 2014

Lars Wirzenius: On ticketing systems

I don't really like any of the ticketing systems I've ever needed to use, whether they've been used as bug tracking systems, user support issue management systems, or something else. Some are not too bad. I currently rely most on debbugs and ikiwiki. debbugs is the Debian bug tracking system. See https://www.debian.org/Bugs/ for an entry point. It's mostly mail based, with a read-only web interface. You report a bug by sending an email to submission address, and (preferably) include a few magic "pseudo-headers" at the top of your message body ot identify the package and version. There's tools to make this easier, but mostly it's just about sending an e-mail. All replies are via e-mails as well. Effectively, each bug becomes is own little dedicated mailing list. This is important. A ticket, whether it is a bug report or a support request, is all about the discussion. "Hey I have this problem..." followed by "Have you tried..." and so forth. Anything that makes that discussion easier and faster to have is better. It is my very strong opinion, and long experience, that the best way to have such a discussion is over e-mail. A lot of modern ticketing systems are web based. They might have an e-mail mode, perhaps read-only, but that's mostly an afterthought. It's a thing bolted onto the side of the system because people like me whinge otherwise. I like e-mail for this for several reasons. For these reasons, I strongly prefer ticketing systems in which e-mails are the primary form of discussions, and e-mail is a first class citizen. I don't mind if there's other ways to participate in the discussion, but if I have to use something else than e-mail, I tend not to be happy. I use ikiwiki to provide a distributed, shared notebook on bugs. It's a bit cumbersome, and doesn't work well for discussions. I think we can improve on the way debbugs works, however. I've been thinking about ticketing systems for Obnam (my backup program), since it gaining enough users that it's getting hard to keep track of discussions with just an e-mail client. Here's what I want: I will eventually have this. I'm not saying I'm working on this, since I don't have enough free time to do that, but there's a git repository, and some code, and it imports e-mails automatically now. Some day there may even be a web interface. (This has been a teaser.)

3 August 2014

Holger Levsen: 20140803-torbrowser-launcher

About torbrowser-launcher in all current Debian distros plus some thoughts and scripts for running it more securely So, torbrowser-launcher 0.1.2-1 is now in sid (only that version has the script examples discussed below), and 0.1.1-2(~bpo70+1) are in jessie and wheezy-backports. Originally Jacob Appelbaum packaged torbrowser-launcher, then Ulrike Uhlig stepped in and fixed some major bugs, I sponsored her uploads and somehow the idea emerged to team maintain the package, so pkg-anonymity-tools was founded. So far it's only used for having a mailing list which is used for the Maintainer: field of the torbrowser-launcher package. But we invite all maintainers of anonymity related packages to join the team! Currently there ain't even a Debian teams wiki page about it (it would be great if YOU could fix that!), so that will probably be the next thing that will happen. As for version control we intend to use the collab-maint project on alioth. So joining the team is not done by joining the alioth project (technically you can do this, but it's rather pointless), but rather by putting the pkg-anonymity-tools mailing list into the Maintainer: field of your package (and you and other people into the Uploaders: field) and subscribing to that very mailing list. Once more packages are maintained that way we'll need to see whether we'll need more mailing lists (eg one specific for commit notifications) or if we rely on client side filtering only or what else should be done. The example scripts (available in /usr/share/doc/torbrowser-launcher/examples in the package from sid or in git) show how to run torbrowser-launcher, confined with AppArmor, in Xephyr (a virtual Xserver running on another Xserver) as another user. This, using AppArmor and Xephyr, shall have two effects: Does that really help? Feedback welcome. Full quote of /usr/share/doc/torbrowser-launcher/examples/README:
torbrowser-launcher launcher scripts
====================================
These scripts are intended to run torbrowser-launcher (and thus torbrowser) as
another user in an Xephyr window server running inside your normal Xorg
session.
They assume the following packages are installed:
- torbrowser-launcher
- apparmor
- xserver-xephyr, awesome
- sudo, slay, psmisc
AppArmor should be enabled, but doesn't have to. I followed the HowTo from
https://wiki.debian.org/AppArmor, which can be summed up as just adding one
parameter to the kernel to enable it, followed by a reboot.
Using Apparmor has the advantage that the browser process cannot most of the
filesystem, eg saving downloads only works in ~/.torbrowser/tbb/x86_64/tor-browser_en-US/Desktop/
On wheezy, I'm using backports for torbrowser-launcher and apparmor.
The scripts assume they have been copied to /usr/local/bin/ and that there is
a user called "foo" (for running the actuall torbrowser(-launcher) process,
and that the current user has sudo rights for the following commands:
- sudo -i -u foo /usr/local/bin/tbb-l-wrapper
- sudo slay foo
There are two scripts, tbb-in-xephyr and tbb-l-wrapper. Only tbb-in-xephyr is
to be called directly and will result in torbrowser running in Xephyr.
Known problems:
---------------
- dbus is not started, so some input methods won't work. (Personally I don't
  want/need dbus though, so I'm awaiting a solution to
  https://trac.torproject.org/projects/tor/ticket/10014)
- not everybody likes awesome as the window manager being used ;)
Ideas, questions and ToDo:
--------------------------
- maybe all of this functionality could be integrated into.
  torbrowser-launcher itself, just writing this in shell was so easy.
- or for the time being, merge these two scripts into one, doing both,
  depending on how its called. Also make them run from everywhere.
- run this in an unprivileged LXC container, which is also apparmor confined.
- (when) does this double confinement make sense?
- use a more sensible named default user (instead of foo).
- there should really be an option, so torbrowser-launcher doesn't detach
  itself, so that this "while;ps fax grep" hack can go away.
- ship an usable sudoers.d example too.
- support for more users / instances
Feedback welcome, especially accompanied by patches! 

30 July 2014

Konstantinos Margaritis: SIMD book, update 3! Addition/Subtraction mostly finished

Finally. Apologies for the delay, but it's been a busy month. This time I will hold true to my word and upload a PDF for people to see (attached to this page). So, what's new? Here is a list of things done: * Finished ALL addition/subtraction related instructions for all engines and major derivatives (SSE*/AVX, VMX/VSX, NEON/armv8 NEON). With diagrams (these were the reasons it has taken so long).
* Reorganized the structure (split the book into Parts I/II, the instruction index will be in Part II, Part I will carry the design analysis of each SIMD engine.
* Added an TOC/index.
* So far, with just Addition/Subtraction Chapter and the rest empty sections, it has reached 175 pages (B5, again I'm not fixed on the size, it might actually change)! My estimate is that the whole book will surpass 800 pages with everything included. TODO:

19 July 2014

Jo Shields: Transition tracker

Friday was my last day at Collabora, the awesome Open Source consultancy in Cambridge. I d been there more than three years, and it was time for a change. As luck would have it, that change came in the form of a job offer 3 months ago from my long-time friend in Open Source, Miguel de Icaza. Monday morning, I fly out to Xamarin s main office in Boston, for just over a week of induction and face time with my new co workers, as I take on the title of Release Engineer. My job is to make sure Mono on Linux is a first-class citizen, rather than the best-effort it s been since Xamarin was formed from the ashes of the Attachmate/Novell deal. I m thrilled to work full-time on what I do already as community work including making Mono great on Debian/Ubuntu and hope to form new links with the packer communities in other major distributions. And I m delighted that Xamarin has chosen to put its money where its mouth is and fund continued Open Source development surrounding Mono. If you re in the Boston area next week or the week after, ping me via the usual methods! IMG_20140719_203043

5 July 2014

Mario Lang: I love my MacBookAir with Debian

In short: I love my MacBook Air. It is the best (laptop) hardware I ever owned. I have seen hardware which was much more flaky in the past. I can set the display backlight to zero via software, which saves me a lot of battery life and also offers a bit of anti-spy-acroos-my-shoulder support. WLAN and bluetooth work nicely. And I just love the form-factor and the touch-feeling of the hardware. I even had the bag I use to carry my braille display modified so that the Air just fits in. I can't say how it behaves with X11. Given how flaky accessibility with graphical desktops on Linux is, I have still not made the switch. My MacBookAir is my perfect mobile terminal, I LOVE it. I am sort of surprised about the recent rant of Paul about MacBook Hardware. It is rather funny that we perceive the same technology so radically different. And after reading the second part of his rant I am wondering if I am no longer allowed to consider myself part of the "hardcore F/OSS world", because I don't consider Apple as evil as apparently most others. Why? Well, first of all, I actually like the hardware. Secondly, you have to show me a vendor first that builds usable accessibility into their products, and I mean all their products, without any extra price-tag attached. Once the others start to consider people with disabilities, we can talk about apple-bashing again. But until then, sorry, you don't see the picture as I do. Apple was the first big company on the market to take accessibility seriously. And they are still unbeaten, at least when it comes to bells and whistles included. I can unbox and configure any Apple product sold currently completely without assisstance. With some products, you just need to know a signle keypress (tripple-press the home button for touch devices and Cmd+F5 for Mac OS/X), and with others, during initial bootup, a speech synthesizer even tells you how to enable accessibility in case you need it. And after that is enabled, I can perform the setup of the device completely on my own. I don't need help from anyone else. And after the setup is complete, I can use 95% of the functionality provided by the operating system. And I am blind, part of a very small margin group so to speak. In Debian circles, I have even heard the sentiment that we supposedly have to accept that small margin groups are ignored sometimes. Well, as long as we think that way, as long as we strictly think economically, we will never be able to go there, fully. And we will never be the universal operating system, actually. Sorry to say that, but I think there is some truth to it. So, who is evil? Scratch your own itch doesn't always work to cover everything. How do we motivate contributors to work on things they don't personally need (yet)? How can we ensure that complicated but seldomly used features stay stable and do not fall to dust just because some upstream decides to rewrite an essential subcomponent of the dependency tree? I don't know. All I know is that these issues need to be solved in an universal operating system.

2 July 2014

Mike Hommey: Firefox and Gtk+ 3

Folks from Collabora and Red Hat have been working on making Firefox on Gtk+ 3 a thing. See Emilio s blog post for some recent update. But getting Firefox to build and run locally is unfortunately not the whole story. I ve been working on getting Gtk+ 3 Firefox builds going on Mozilla build infrastructure, and I m proud to announce today that those builds are now going through Mozilla continuous integration on a project branch: Elm, and receive the same automated testing as mozilla-central. And when I said getting Firefox to build and run was unfortunately not the whole story, I meant it: if you click on the Elm link above, you ll notice that there s a lot of orange, when it should be all green. So, yes, Firefox on Gtk+ 3 is a thing, and it now has continuous integration. But there s still a whole bunch of things to fix. So if you re interested in making those builds work better, you can hop in, there are many things you can do:

29 June 2014

Craig Small: Sneak peek of top graphs

Jim has been busy as part of the procps-ng team that looks after top. Basically all the changes you find in top from around 2.7 or so are by him. Not satisfied enough with fixing top, making it faster and showing more fields, he has given us CPU and memory graphs. He also thinks I don t have enough colours (or as he would put it colors) on my top output so I ve posted what the new top looks like for me so you can see the graphs and he can see my colours.
top, with colours

top, with colours

I think it is both colourful and useful addition. The colours have been available for a while now and the graphs will appear in the next upstream release of procps-ng.

22 June 2014

Gunnar Wolf: Yo tampoco / #yotampoco / neither do I A bit of local action

Only a very short summary in English: I am Mexican. I am Jewish. I am almost completely disconnected from the local Jewish communities. And understanding the local Jewish communities is hard. There is a very interesting and brave campaign, recently started, called Neither do I The Mexican Jewish gay activist group Guimel, started off with this video (with English subtitles, if you are interested in following along). But how did I learn about this very bold initiative? By getting a hateful spam, inviting people to join a hate campaign. Right, the hate mail is not calling to violence, but it is based on premises as stupid as everybody's right not to include (to begin with). So, the least I can do about this is to share both said hate mail and publicly denounce my shock on reading this nonsense nowadays. And, also in Spanish (I know many people following me don't understand it Sorry, it would just take too long, and after all, it's mainly for local "consumption"), this is the reply I sent to them (and to the other recipients). Sorry in advance to the Spanish speakers for my exabrupts :) This was written "as is", without much prior thought, and quite angry about what I had just read.
Buf... Expresiones como esta, que hacen evidente la cerraz n de tanta gente en las comunidades jud as, justifican claramente por qu tantos nos hemos ido abriendo, integr ndonos a la sociedad circundante. Me da verg enza ser asociado como jud o a comunidades donde se defienden estos puntos de vista; si bien yo no soy de la comunidad Monte Sinai (como resultar obvio por mi nombre), para la sociedad en su conjunto s soy un jud o (as , a secas: Jud o). Mucho nos hemos quejado colectivamente a lo largo de los siglos acerca de la discriminaci n, de que seamos considerado la "basura" del mundo, los "apestados". Y es precisamente esa actitud, ante todo y en todo momento, lo que me ha llevado a alejarme del juda smo. Soy un ser humano m s, con una historia personal nica, con una historia cultural compartida pero tambi n nica, con algunas elecciones y algunas caracter sticas que me son inherentes nicas. Algo de eso no le gusta a alguien? Por motivos racionales o irracionales? No lo puedo ni pienso evitar. Pero el que una comunidad completa sea llamada a ignorar, humillar, alienar a sus hijos, por una estupidez as de anacr nica (en el caso que lo presenta el video al que hacen referencia, y a la gente que valientemente ha conformado Guimel) me parece que simplemente va m s all de toda estupidez. Piensen bien antes de adherirse a esta campa a de odio. Substituyan la sexualidad por cualquier otra raz n en la que somos minor a. No les da asco vivir con un vendedor de telas? Con un jud o apestoso? Y s , aqu ya estoy yo tambi n dejando ir a esos demonios... que hay que mantener bajo control. Porque somos hombres y mujeres de nuestro siglo, y porque el mundo y las nuevas generaciones merecen nuestro mejor esfuerzo para ir colectivamente destruyendo la estupidez de nuestros ancestros. Seamos m s racionales. Dejemos el odio, dejemos los prejuicios. Aceptemos a los diferentes, porque todos queremos que nos acepten. Porque todos somos diferentes.
AttachmentSize
tambien.txt8.55 KB

Next.

Previous.