Sending GCM send message from nodejs.
I wanted to send a message from emacs, but it seemed to be
relatively difficult to send HTTPS POST request from emacs,
so I just decided to use handy nodejs.
The attached 'data' is attached to intent as extra, so it can be extracted by
intent.getExtras.getString("message") in IntentService#onHandleIntent()
OK, it's almost one month since we (the keyring-maintainers) gave our talk at DebConf14; how are we faring regarding key transitions since then? You can compare the numbers (the graphs, really) to those in our DC14 presentation.
Since the presentation, we have had two keyring pushes:
First of all, the Non-uploading keyring is all fine: As it was quite recently created, and as it is much smaller than our other keyrings, it has no weak (1024 bit) keys. It briefly had one in 2010-2011, but it's long been replaced.
Second, the Maintainers keyring: In late July we had 222 maintainers (170 with >=2048 bit keys, 52 with weak keys). By the end of August we had 221: 172 and 49 respectively, and by September 18 we had 221: 175 and 46.
As for the Uploading developers, in late July we had 1002 uploading developers (481 with >=2048 bit keys, 521 with weak keys). By the end of August we had 1002: 512 and 490 respectively, and by September 18 we had 999: 531 and 468.
Please note that these numbers do not say directly that six DMs or that 50 uploading DDs moved to stronger keys, as you'd have to factor in new people being added, keys migrating between different keyrings (mostly DM DD), and people retiring from the project; you can get the detailed information looking at the public copy of our Git repository, particularly of its changelog.
And where does that put us?
Of course, I'm very happy to see that the lines in our largest keyring have already crossed. We now have more people with >=2048 bit keys. And there was a lot of work to do this processing done! But that still means... That in order not to lock a large proportion of Debian Developers and Maintainers out of the project, we have a real lot of work to do. We would like to keep the replacement slope high (because, remember, in January 1st we will remove all small keys from the keyring).
And yes, we are willing to do the work. But we need you to push us for it: We need you to get a new key created, to gather enough (two!) DD signatures in it, and to request a key replacement via RT.
So, by all means: Do keep us busy!
I am working on a small project to create WNPP like statistics for open RFS bugs. I think this could
improve a little bit effectiveness of sponsoring new packages by giving insight into bugs that are on
their way to being starved (i.e. not ever sponsored, or rotting in a queue).
The script attached in this post is written in Python and uses Debbugs SOAP
interface to get currently open RFS bugs and
calculates their dust and age.
The dust factor is calculated as an absolute value of a difference between bugs's age and
log_modified.
Later I would like to create fully blown stats for an RFS queue, taking into account
the whole history (i.e. 2012-1-1 until now), and check its health, calculate MTTGS (mean time to get sponsored).
The list looks more or less like this:
Last night I was looking for an image I knew a model had mailed me a few months ago, as we were talking about rescheduling a shoot at the weekend. I couldn't find it, even with my awesome mail client and filing system.
With some free time I figured I could write a little utility to dump all attachments from email folders, and find it that way.
It did cross my mind that there is the simple mail-utility for dumping headers, etc, called formail, which is distributed alongside procmail, but it doesn't handle attachments ..
I was tempted to write a general purpose script to dump attachments, email header values, etc, etc but given the lack of time I merely solved my own problem.
I suspect there is room for a "mail utilities" package, similar to Joey's "moreutils" and my "sysadmin utils". However I note that there is a GNU Mailutils which does things differently than I'd expect - i.e. it contains a POP3 server.
Still if you want to dump attachments from emails, have GMIME installed, and want to filter by attachment-name, or MIME-type, you might look at my trivial attachment-dump program.
Related to that I spent some time last night updating my photography site, so the animals & pets section has updated images at least.
During the course of that I found a bug in my static-site generator, templer which stopped it from automatically populating image height/widths when called in a glob:
Title: Pets & Animals
Images: file_glob( "*.jpg" )
---
This is the page body, it now has access to a variable called 'images'
which is a HTML::Template loop-structure containing name/height/width/etc
for each image in the current directory.
That should now be resolved, and life should once again be good.
After spending a while fighting with upstart, at work, I decided that systemd couldn't be any worse and yesterday morning upgraded one of my servers to run it.
I have two classes of servers:
Those that run standard daemons, with nothing special.
Those that run different services under runit
For example docker guests, node.js applications, and similar.
I thought it would be a fair test to upgrade one of each systems, to see how it worked.
The Debian wiki has instructions for installing Systemd, and both systems came up just fine.
Although I realize I should replace my current runit jobs with systemd units I didn't want to do that. So I wrote a systemd .service file to launch runit against /etc/service, as expected, and that was fine.
Docker was a special case. I wrote a docker.service + docker.socket file to launch the deamon, but when I wrote a graphite.service file to start a docker instance it kept on restarting, or failing to stop.
In short I couldn't use systemd to manage running a docker guest, but that was probably user-error. For the moment the docker-host has a shell script in root's home directory to launch the guest:
#!/bin/sh
#
# Run Graphite in a detached state.
#
/usr/bin/docker run -d -t -i -p 8080:80 -p 2003:2003 skxskx/graphite
Without getting into politics (ha), systemd installation seemed simple, resulted in a faster boot, and didn't cause me horrific problems. Yet.
ObRandom: Not sure how systemd is controlling prosody, for example.
If I run the status command I can see it is using the legacy system:
root@chat ~ # systemctl status prosody.service
prosody.service - LSB: Prosody XMPP Server
Loaded: loaded (/etc/init.d/prosody)
Active: active (running) since Wed, 03 Sep 2014 07:59:44 +0100; 18h ago
CGroup: name=systemd:/system/prosody.service
942 lua5.1 /usr/bin/prosody
I've installed systemd and systemd-sysv, so I thought /etc/init.d was obsolete. I guess it is making pretend-services for things it doesn't know about (because obviously not all packages contain /lib/systemd/system entries), but I'm unsure how that works.
Yesterday I received my official diploma for the degree of Licentiate of Philosophy. The degree lies between a Master s degree and a doctorate, and is not required; it consists of the coursework required for a doctorate, and a Licentiate Thesis, in which the student demonstrates good conversance with the field of research and the capability of independently and critically applying scientific research methods (official translation of the Government decree on university degrees 794/2004, Section 23 Paragraph 2).
The title and abstract of my Licentiate Thesis follow:
Kaijanaho, Antti-Juhani
The extent of empirical evidence that could inform evidence-based design of programming languages. A systematic mapping study.
Jyv skyl : University of Jyv skyl , 2014, 243 p.
(Jyv skyl Licentiate Theses in Computing,
ISSN 1795-9713; 18)
ISBN 978-951-39-5790-2 (nid.)
ISBN 978-951-39-5791-9 (PDF)
Finnish summary
Background: Programming language design is not usually informed by empirical studies. In other fields similar problems have inspired an evidence-based paradigm of practice. Central to it are secondary studies summarizing and consolidating the research literature. Aims: This systematic mapping study looks for empirical research that could inform evidence-based design of programming languages. Method: Manual and keyword-based searches were performed, as was a single round of snowballing. There were 2056 potentially relevant publications, of which 180 were selected for inclusion, because they reported empirical evidence on the efficacy of potential design decisions and were published on or before 2012. A thematic synthesis was created. Results: Included studies span four decades, but activity has been sparse until the last five years or so. The form of conditional statements and loops, as well as the choice between static and dynamic typing have all been studied empirically for efficacy in at least five studies each. Error proneness, programming comprehension, and human effort are the most common forms of efficacy studied. Experimenting with programmer participants is the most popular method. Conclusions: There clearly are language design decisions for which empirical evidence regarding efficacy exists; they may be of some use to language designers, and several of them may be ripe for systematic reviewing. There is concern that the lack of interest generated by studies in this topic area until the recent surge of activity may indicate serious issues in their research approach.
Keywords: programming languages, programming language design, evidence-based paradigm, efficacy, research methods, systematic mapping study, thematic synthesis
A Licentiate Thesis is assessed by two examiners, usually drawn from outside of the home university; they write (either jointly or separately) a substantiated statement about the thesis, in which they suggest a grade. The final grade is almost always the one suggested by the examiners. I was very fortunate to have such prominent scientists as Dr. Stefan Hanenberg and Prof. Stein Krogdahl as the examiners of my thesis. They recommended, and I received, the grade very good (4 on a scale of 1 5).
The thesis has been accepted for publication published in our faculty s licentiate thesis series and will in due course appearhas appeared in our university s electronic database (along with a very small number of printed copies). In the mean time, if anyone wants an electronic preprint, send me email at antti-juhani.kaijanaho@jyu.fi.Figure 1 of the thesis: an overview of the mapping process
As you can imagine, the last couple of months in the spring were very stressful for me, as I pressed on to submit this thesis. After submission, it took me nearly two months to recover (which certain people who emailed me on Planet Haskell business during that period certainly noticed). It represents the fruit of almost four years of work (way more than normally is taken to complete a Licentiate Thesis, but never mind that), as I designed this study in Fall 2010.
Figure 8 of the thesis: Core studies per publication year
Recently, I have been writing in my blog a series of posts in which I have been trying to clear my head about certain foundational issues that irritated me during the writing of the thesis. The thesis contains some of that, but that part of it is not very strong, as my examiners put it, for various reasons. The posts have been a deliberately non-academic attempt to shape the thoughts into words, to see what they look like fixed into a tangible form. (If you go read them, be warned: many of them are deliberately provocative, and many of them are intended as tentative in fact if not in phrasing; the series also is very incomplete at this time.)
I closed my previous post, the latest post in that series, as follows:
In fact, the whole of 20th Century philosophy of science is a big pile of failed attempts to explain science; not one explanation is fully satisfactory. [...] Most scientists enjoy not pondering it, for it s a bit like being a cartoon character: so long as you don t look down, you can walk on air.
I wrote my Master s Thesis (PDF) in 2002. It was about the formal method called B ; but I took a lot of time and pages to examine the history and content of formal logic. My supervisor was, understandably, exasperated, but I did receive the highest possible grade for it (which I never have fully accepted I deserved). The main reason for that digression: I looked down, and I just had to go poke the bridge I was standing on to make sure I was not, in fact, walking on air. In the many years since, I ve taken a lot of time to study foundations, first of mathematics, and more recently of science. It is one reason it took me about eight years to come up with a doable doctoral project (and I am still amazed that my department kept employing me; but I suppose they like my teaching, as do I). The other reason was, it took me that long to realize how to study the design of programming languages without going where everyone has gone before.
Debian people, if any are still reading, may find it interesting that I found significant use for the dctrl-tools toolset I have been writing for Debian for about fifteen years: I stored my data collection as a big pile of dctrl-format files. I ended up making some changes to the existing tools (I should upload the new version soon, I suppose), and I wrote another toolset (unfortunately one that is not general purpose, like the dctrl-tools are) in the process.
For the Haskell people, I mainly have an apology for not attending to Planet Haskell duties in the summer; but I am back in business now. I also note, somewhat to my regret, that I found very few studies dealing with Haskell. I just checked; I mention Haskell several times in the background chapter, but it is not mentioned in the results chapter (because there were not studies worthy of special notice).
I am already working on extending this work into a doctoral thesis. I expect, and hope, to complete that one faster.
Summer is cool in Mexico City.
It is cool because, unlike Spring, this is our rainy season And rains are very predictable. Almost every day we wake up with a gorgeous, clean, blue sky.
Cool, nice temperature, around 15 C. The sun slowly evaporates the rain throughout the morning; when I go out for lunch, the sky is no longer so blue, giving way to a seemingly dirty white/grayish tint. No, it's not our world-famous pollution: It's just yesterday's rain.
Rain starts falling usually between 4 and 7 PM. Sometimes it starts as a light rain, sometimes it starts with all of its thunder, all of its might. But anyway, almost every night, there is a moment of awe, of not believing how much rain we are getting today.
It slowly fades away during the late night. And when I wake up, early next morning, everything is wet and still smells fresh.
Yes, I love our summer, even though it makes shy away from my much enjoyed cycling to work and school. And I love taking some minutes off work, look through the window of my office (located ~70m over the level of our mostly flat city) and watching how different parts of the city have sun or rain; learning to estimate the distance to the clouds, adding it to the direction and guessing which of my friends have which weather.
But I didn't realize our city had so clearly defined micro-climates... (would they really be *micro*-climates?) In fact, it even goes against my knowledge of Mexico City's logic I always thought Coyoac n, towards the South of the city, got more rain than the Center and North because we are near the mountains, and the dominant air currents go Southwards, "clumping" the clouds by us.
But no, or at least, not this year. Regina (still in the far South Far because she's too far away from me and I'm too egocentric; she returns home after DebConf) often asks me about the weather, as our friends working nearer the center of the city. According to the photos they post on their $social_media_of_the_day accounts, rains are really heavier there.
Today I heard on the radio accounts of yesterday's chaos after the rain. This evening, at ESIME-Culhuac n, I saw one of the reported fallen trees (of course, I am not sure if it's from yesterday's rain). And the media pushes galleries of images of a city covered in hail... While in Copilco we only had a regular rain, I'd even say a mild one.
This city is bigger than any cloud you can throw at it.
A little more than 2 years ago, the
Ceilometer project was launched inside
the OpenStack ecosystem. Its main objective was to measure OpenStack cloud
platforms in order to provide data and mechanisms for functionalities such
as billing, alarming or capacity planning.
In this article, I would like to relate what I've been doing with other
Ceilometer developers in the last 5 months. I've lowered my involvement in
Ceilometer itself directly to concentrate on solving one of its biggest
issue at the source, and I think it's largely time to take a break and talk
about it.
Ceilometer early design
For the last years, Ceilometer didn't change in its core architecture.
Without diving too much in all its parts, one of the early design decision
was to build the metering around a data structure we called samples. A
sample is generated each time Ceilometer measures something. It is composed
of a few fields, such as the the resource id that is metered, the user and
project id owning that resources, the meter name, the measured value, a
timestamp and a few free-form metadata. Each time Ceilometer measures
something, one of its components (an agent, a pollster ) constructs and
emits a sample headed for the storage component that we call the
collector.
This collector is responsible for storing the samples into a database. The
Ceilometer collector uses a pluggable storage system, meaning that you can
pick any database system you prefer. Our original implementation has been
based on MongoDB from the beginning, but we then added a SQL driver, and
people contributed things such as HBase or DB2 support.
The REST API exposed by Ceilometer allows to execute various reading
requests on this data store. It can returns you the list of resources that
have been measured for a particular project, or compute some statistics on
metrics. Allowing such a large panel of possibilities and having such a
flexible data structure allows to do a lot of different things with
Ceilometer, as you can almost query the data in any mean you want.
The scalability issue
We soon started to encounter scalability issues in many of the read requests
made via the REST API. A lot of the requests requires the data storage to do
full scans of all the stored samples. Indeed, the fact that the API allows
you to filter on any fields and also on the free-form metadata (meaning non
indexed key/values tuples) has a terrible cost in terms of performance (as
pointed before, the metadata are attached to each sample generated by
Ceilometer and is stored as is). That basically means that the sample data
structure is stored in most drivers in just one table or collection, in
order to be able to scan them at once, and there's no good "perfect"
sharding solution, making data storage scalability painful.
It turns out that the Ceilometer REST API is unable to handle most of the
requests in a timely manner as most operations are O(n) where n is the
number of samples recorded (see
big O notation if you're
unfamiliar with it). That number of samples can grow very rapidly in an
environment of thousands of metered nodes and with a data retention of
several weeks. There is a few optimizations to make things smoother in
general cases fortunately, but as soon as you run specific queries, the API
gets barely usable.
During this last year, as the Ceilometer PTL, I discovered these issues
first hand since a lot of people were feeding me back with this kind of
testimony. We engaged several blueprints to improve the situation, but it
was soon clear to me that this was not going to be enough anyway.
Thinking outside the box
Unfortunately, the PTL job doesn't leave him enough time to work on the
actual code nor to play with anything new. I was coping with most of the
project bureaucracy and I wasn't able to work on any good solution to tackle
the issue at its root. Still, I had a few ideas that I wanted to try and as
soon as I stepped down from the PTL role, I stopped working on Ceilometer
itself to try something new and to think a bit outside the box.
When one takes a look at what have been brought recently in Ceilometer, they
can see the idea that Ceilometer actually needs to handle 2 types of data:
events and metrics.
Events are data generated when something happens: an instance start, a
volume is attached, or an HTTP request is sent to an REST API server. These
are events that Ceilometer needs to collect and store. Most OpenStack
components are able to send such events using the notification system built
into oslo.messaging.
Metrics is what Ceilometer needs to store but that is not necessarily tied
to an event. Think about an instance CPU usage, a router network bandwidth
usage, the number of images that Glance is storing for you, etc These are
not events, since nothing is happening. These are facts, states we need to
meter.
Computing statistics for billing or capacity planning requires both of these
data sources, but they should be distinct. Based on that assumption, and the
fact that Ceilometer was getting support for storing events, I started to
focus on getting the metric part right.
I had been a system administrator for a decade before jumping into OpenStack
development, so I know a thing or two on how monitoring is done in this
area, and what kind of technology operators rely on. I also know that
there's still no silver bullet this made it a good challenge.
The first thing that came to my mind was to use some kind of time-series
database, and export its access via a REST API as we do in all OpenStack
services. This should cover the metric storage pretty well.
Cooking Gnocchi
A cloud of gnocchis!
At the end of April 2014, this led met to start a new project code-named
Gnocchi. For the record, the name was picked after confusing so many times
the OpenStack Marconi project, reading OpenStack Macaroni instead. At least
one OpenStack project should have a "pasta" name, right?
The point of having a new project and not send patches on Ceilometer, was
that first I had no clue if it was going to make something that would be any
better, and second, being able to iterate more rapidly without being
strongly coupled with the release process.
The first prototype started around the following idea: what you want is to
meter things. That means storing a list of tuples of (timestamp, value) for
it. I've named these things "entities", as no assumption are made on what
they are. An entity can represent the temperature in a room or the CPU usage
of an instance. The service shouldn't care and should be agnostic in this
regard.
One feature that we discussed for several OpenStack summits in the
Ceilometer sessions, was the idea of doing aggregation. Meaning, aggregating
samples over a period of time to only store a smaller amount of them. These
are things that time-series format such as the
RRDtool have been doing for a long time on
the fly, and I decided it was a good trail to follow.
I assumed that this was going to be a requirement when storing metrics into
Gnocchi. The user would need to provide what kind of archiving it would
need: 1 second precision over a day, 1 hour precision over a year, or even
both.
The first driver written to achieve that and store those metrics inside
Gnocchi was based on whisper. Whisper
is the file format used to store metrics for the
Graphite project. For the actual storage,
the driver uses Swift, which has the advantages to be part of OpenStack and
scalable.
Storing metrics for each entities in a different whisper file and putting
them in Swift turned out to have a fantastic algorithm complexity: it was
O(1). Indeed, the complexity needed to store and retrieve metrics doesn't
depends on the number of metrics you have nor on the number of things you
are metering. Which is already a huge win compared to the current Ceilometer
collector design.
However, it turned out that whisper has a few limitations that I was
unable to circumvent in any manner. I needed to patch it to remove a lot of
its assumption about manipulating file, or that everything is relative to
now (time.time()). I've started to hack on that in my own fork, but then
everything broke. The whisper project code base is, well, not the state of
the art, and have 0 unit test. I was starring at a huge effort to transform
whisper into the time-series format I wanted, without being sure I wasn't
going to break everything (remember, no test coverage).
I decided to take a break and look into alternatives, and stumbled upon
Pandas, a data manipulation and statistics
library for Python. Turns out that Pandas support time-series natively, and
that it could do a lot of the smart computation needed in Gnocchi. I built a
new file format leveraging Pandas for computing the time-series and named it
carbonara (a wink to both the
Carbon project and pasta, how
clever!). The code is quite small (a third of whisper's, 200 SLOC vs 600
SLOC), does not have many of the whisper limitations and it has test
coverage. These Carbonara files are then, in the same fashion, stored into
Swift containers.
Anyway, Gnocchi storage driver system is designed in the same spirit that
the rest of OpenStack and Ceilometer storage driver system. It's a plug-in
system with an API, so anyone can write their own driver. Eoghan Glynn has
already started to write a InfluxDB driver, working
closely with the upstream developer of that database. Dina Belova started to
write an OpenTSDB driver. This helps to make sure
the API is designed directly in the right way.
Handling resources
Measuring individual entities is great and needed, but you also need to link
them with resources. When measuring the temperature and the number of a
people in a room, it is useful to link these 2 separate entities to a
resource, in that case the room, and give a name to these relations, so one
is able to identify what attribute of the resource is actually measured. It
is also important to provide the possibility to store attributes on these
resources, such as their owners, the time they started and ended their
existence, etc.
Relationship of entities and resources
Once this list of resource is collected, the next step is to list and filter
them, based on any criteria. One might want to retrieve the list of
resources created last week or the list of instances hosted on a particular
node right now.
Resources also need to be specialized. Some resources have attributes that
must be stored in order for filtering to be useful. Think about an instance
name or a router network.
All of these requirements led to to the design of what's called the
indexer. The indexer is responsible for indexing entities, resources, and
link them together. The initial implementation is based on
SQLAlchemy and should be pretty efficient. It's
easy enough to index the most requested attributes (columns), and they are
also correctly typed.
We plan to establish a model for all known OpenStack resources (instances,
volumes, networks, ) to store and index them into the Gnocchi indexer in
order to request them in an efficient way from one place. The generic
resource class can be used to handle generic resources that are not tied to
OpenStack. It'd be up to the users to store extra attributes.
Dropping the free form metadata we used to have in Ceilometer makes sure
that querying the indexer is going to be efficient and scalable.
The indexer classes and their relations
REST API
All of this is exported via a REST API that was partially designed and
documented in the
Gnocchi specification in the Ceilometer repository;
though the spec is not up-to-date yet. We plan to auto-generate the
documentation from the code as we are currently doing in Ceilometer.
The REST API is pretty easy to use, and you can use it to manipulate
entities and resources, and request the information back.
Macroscopic view of the Gnocchi architecture
Roadmap & Ceilometer integration
All of this plan has been exposed and discussed with the Ceilometer team
during the last
OpenStack summit in Atlanta
in May 2014, for the Juno release. I led a session about this entire
concept, and convinced the team that using Gnocchi for our metric storage
would be a good approach to solve the Ceilometer collector scalability
issue.
It was decided to conduct this project experiment in parallel of the current
Ceilometer collector for the time being, and see where that would lead the
project to.
Early benchmarks
Some engineers from Mirantis did a few benchmarks around Ceilometer and also
against an early version of Gnocchi, and Dina Belova presented them to us
during the mid-cycle sprint we organized in Paris in early July.
The following graph sums up pretty well the current Ceilometer performance
issue. The more you feed it with metrics, the more slow it becomes.
For Gnocchi, while the numbers themselves are not fantastic, what is
interesting is that all the graphs below show that the performances are
stable without correlation with the number of resources, entities or
measures. This proves that, indeed, most of the code is built around a
complexity of O(1), and not O(n) anymore.
Next steps
Cl ment drawing the logo
While the Juno cycle is being wrapped-up for most projects, including
Ceilometer, Gnocchi development is still ongoing. Fortunately, the composite
architecture of Ceilometer allows a lot of its features to be replaced by
some other code dynamically. That, for example, enables Gnocchi to provides
a Ceilometer dispatcher plugin for its collector, without having to ship the
actual code in Ceilometer itself. That should help the development of
Gnocchi to not be slowed down by the release process for now.
The Ceilometer team aims to provide Gnocchi as a sort of technology preview
with the Juno release, allowing it to be deployed along and plugged with
Ceilometer. We'll discuss how to integrate it in the project in a more
permanent and strong manner probably during the
OpenStack Summit for Kilo
that will take place next November in Paris.
There hasn t been a progress-report on DEP-11 for some time, but that doesn t mean there was no work going on on it.
DEP-11 is Debian s implementation of AppStream, as well as an effort to enhance the metadata available about software in Debian. While initially, AppStream was only about applications, DEP-11 was designed with a larger scope, to collect data about libraries, binaries and things like Python modules. Now, since AppStream 0.6, DEP-11 and AppStream have essentially the same scope, with the difference of DEP-11 metadata being described in YAML, while official AppStream data is XML. That was due to a request by our ftpmasters team, which doesn t like XML (which is also not used anywhere in Debian, as opposed to YAML). But this doesn t mean that people will have to deal with the YAML file format: The libappstream library will just take DEP-11 data as another data source for it s Xapian database, allowing anything using libappstream to access that data just like the XML stuff. Richards libappstream-glib will also receive support for the DEP-11 format soon, filling it s in-memory data cache and enabling the use of GNOME-Software on Debian.
So, what has been done so far? The past months, my Google Summer of Code student. Abhishek Bhattacharjee, was working hard to integrate DEP-11 support into dak, the Debian Archive Kit, which maintains the whole Debian archive. The result will be an additional metadata table in our internal Postgres database, storing detailed information about the software available in a Debian package, as well as Components-<arch>.yml.gz files in the Debian repositories. Dak will also produce an application icon-cache and a screenshots repository. During the time of the SoC, Abhishek focused mainly on the applications part of things, and less on the other components (like extracting data about Python modules or libraries) these things can easily be implemented later.
The remaining steps will be to polish the code and make it merge-ready for Debian s dak (as soon as it has received enough testing, we will likely give it a try on the Tanglu Debian derivative). Following that, Apt will be extended to fetch the DEP-11 data on-demand on systems where it is useful (which is currently mostly desktop-systems) if you want to save a little bit of space, you will be able to disable downloading this extra metadata in Apt. From there, libappstream will take the data for it s Xapian db. This will lead to the removal of the much-hated (from ftpmasters and maintainers side) app-install-data package, which has not been updated for two years and only contains a small fraction of the metadata provided by DEP-11.
What Debian will ultimately gain from this effort is support for software-centers like GNOME-Software, and improved support for tools like Apper and Muon in displaying applications. Long-term, with more metadata being available, It would be cool to add support for it to specialized package managers , like Python s pip, npm or gem, to make them fetch information about available distribution software and install that instead of their own copies from 3rd-party repositories, if possible. This should ultimatively lead to less code duplication on distributions and will likely result in fewer security issues, since the officially maintained and integrated distribution packages can easily be used, if possible. This is no attempt to make tools like pip obsolete, but an attempt to have the different tools installing software on your machine communicate better, instead of creating parallel worlds in terms of software management. Another nice sideeffect of more metadata will be options to search for tools handling mimetypes in the software repos (in case you can t open a file), smart software centers installing missing firmware, and automatic suggestions for developers which software they need to install in order to build a specific software package. Also, the data allows us to match software across distributions, for that, I will have some news soon (not sure how soon though, as I am currently in thesis-writing-mode, and therefore have not that much spare time). Since the goal is to have these features available on all distributions supporting AppStream, it will take longer to realize but we are on a good way.
So, if you want some more information about my student s awesome work, you can read his blogpost about it. He will also be at Debconf 14 (Portland). (I can t make it this time, but I surely won t miss the next Debconf)
Sadly, I only see a very small chance to have the basic DEP-11 stuff land in-time for Jessie (lots of review work needs to be done, and some more code needs to be written), but we will definitively have it in Jessie+1.
A small example on how this data will look like can be found here a larger, actual file is available here. Any questions and feedback are highly appreciated.
This is a random post inspired by recent purchases. Some things we buy are practical, others are a little arbitrary.
I tend to avoid buying things for the sake of it, and have explicitly started decluttering our house over the past few years. That said sometimes things just seem sufficiently "cool" that they get bought without too much thought.
This entry is about two things.
A couple of years ago my bathroom was ripped apart and refitted. Gone was the old and nasty room, and in its place was a glorious space. There was only one downside to the new bathroom - you turn on the light and the fan comes on too.
When your wife works funny shifts at the hospital you can find that the (quiet) fan sounds very loud in the middle of the night and wakes you up..
So I figured we could buy a couple of LED lights and scatter them around the place - when it is dark the movement sensors turn on the lights.
These things are amazing. We have one sat on a shelf, one velcroed to the bottom of the sink, and one on the floor, just hidden underneath the toilet.
Due to the shiny-white walls of the room they're all you need in the dark.
By contrast my second purchase was a mistake - The Logitech Harmony 650 Universal Remote Control should be great. It clearly has the features I want - Able to power:
Our TV.
Our Sky-box.
OUr DVD player.
The problem is solely due to the horrific software. You program the device via an application/website which works only under Windows.
I had to resort to installing Windows in a virtual machine to make it run:
# Get the Bus/ID for the USB device
bus=$(lsusb grep -i Harmony awk ' print $2 ' tr -d 0)
id=$(lsusb grep -i Harmony awk ' print $4 ' tr -d 0:)
# pass to kvm
kvm -localtime .. -usb -device usb-host,hostbus=$bus,hostaddr=$id ..
That allows the device to be passed through to windows, though you'll later have to jump onto the Qemu console to re-add the device as the software disconnects and reconnects it at random times, and the bus changes. Sigh.
I guess I can pretend it works, and has cut down on the number of remotes sat on our table, but .. The overwhelmingly negative setup and configuration process has really soured me on it.
There is a linux application which will take a configuration file and squirt it onto the device, when attached via a USB cable. This software, which I found during research prior to buying it, is useful but not as much as I'd expected. Why? Well the software lets you upload the config file, but to get a config file you must fully complete the setup on Windows. It is impossible to configure/use this device solely using GNU/Linux.
(Apparently there is MacOS software too, I don't use macs. *shrugs*)
In conclusion - Motion-activated LED lights, more useful than expected, but Harmony causes Discord.
I don't really like any of the ticketing systems I've ever needed to
use, whether they've been used as bug tracking systems, user support
issue management systems, or something else. Some are not too bad. I
currently rely most on debbugs and ikiwiki.
debbugs is the Debian bug tracking system. See
https://www.debian.org/Bugs/ for an entry point. It's mostly mail
based, with a read-only web interface. You report a bug by sending an
email to submission address, and (preferably) include a few magic
"pseudo-headers" at the top of your message body ot identify the
package and version. There's tools to make this easier, but mostly
it's just about sending an e-mail. All replies are via e-mails as
well. Effectively, each bug becomes is own little dedicated mailing
list.
This is important. A ticket, whether it is a bug report or a support
request, is all about the discussion. "Hey I have this problem..."
followed by "Have you tried..." and so forth. Anything that makes that
discussion easier and faster to have is better.
It is my very strong opinion, and long experience, that the best way
to have such a discussion is over e-mail. A lot of modern ticketing
systems are web based. They might have an e-mail mode, perhaps
read-only, but that's mostly an afterthought. It's a thing bolted onto
the side of the system because people like me whinge otherwise.
I like e-mail for this for several reasons.
E-mail is push, not pull. I don't need to go look at a web page to
be notified that something's happened.
E-mail requires no extra usernames and passwords to manage. I don't
need to create a new account every time I encounter a new ticketing
system instance.
E-mail makes it very easy to respond. I can just reply to a message.
I don't need to go to a web site, log in, and find a reply button.
I already have archives of my e-mail, so referring to old messages
(or finding them) is easy and quick. (Mutt, offlineimap, and notmuch
is my particular set of choices. But I'm not locked to them, and you
can use whatever you like, too.)
E-mail is a very rich format. Discussions are inherently threaded,
and various character sets, languages, attachments, and other such
things just work.
For these reasons, I strongly prefer ticketing systems in which
e-mails are the primary form of discussions, and e-mail is a first
class citizen. I don't mind if there's other ways to participate in
the discussion, but if I have to use something else than e-mail, I
tend not to be happy.
I use ikiwiki to provide a distributed, shared notebook on bugs. It's
a bit cumbersome, and doesn't work well for discussions.
I think we can improve on the way debbugs works, however. I've been
thinking about ticketing systems for Obnam (my backup program), since
it gaining enough users that it's getting hard to keep track of
discussions with just an e-mail client.
Here's what I want:
Obnam users do not need to care about there being a ticketing
system. They report a problem by e-mailing the support mailing list,
and they keep the list in cc when conducting the discussion. This
is very similar to debbugs, with the distinction that there's no
ticket numbers that must be kept in the replies.
The support staff (that's me, but hopefully others as well) have
access to the ticketing system, which automatically sorts incoming
messages into tickets. Tickets have sufficient metadata that it's
possible to track which ones have been dealt with, or still need
work, and perhaps other things. Each ticket contain a Maildir with
all the e-mails belonging to that ticket.
The ticketing system is distributed. I need to be able to work on
tickets offline, and to synchronise instances between different
computers. Just like git. It's not enough to have an offline mode
(e.g., queuing e-mails on my laptop for sending to debbugs when I'm
back online).
There is a reasonably powerful search engine that can quickly find
the relevant tickets, and messages, based on various criteria.
I will eventually have this. I'm not saying I'm working on this, since
I don't have enough free time to do that, but there's a git
repository, and some code, and it imports e-mails automatically now.
Some day there may even be a web interface.
(This has been a teaser.)
About torbrowser-launcher in all current Debian distros plus some thoughts and scripts for running it more securely
So, torbrowser-launcher 0.1.2-1 is now in sid (only that version has the script examples discussed below), and 0.1.1-2(~bpo70+1) are in jessie and wheezy-backports.
Originally Jacob Appelbaum packaged torbrowser-launcher, then Ulrike Uhlig stepped in and fixed some major bugs, I sponsored her uploads and somehow the idea emerged to team maintain the package, so pkg-anonymity-tools was founded. So far it's only used for having a mailing list which is used for the Maintainer: field of the torbrowser-launcher package. But we invite all maintainers of anonymity related packages to join the team! Currently there ain't even a Debian teams wiki page about it (it would be great if YOU could fix that!), so that will probably be the next thing that will happen. As for version control we intend to use the collab-maint project on alioth. So joining the team is not done by joining the alioth project (technically you can do this, but it's rather pointless), but rather by putting the pkg-anonymity-tools mailing list into the Maintainer: field of your package (and you and other people into the Uploaders: field) and subscribing to that very mailing list. Once more packages are maintained that way we'll need to see whether we'll need more mailing lists (eg one specific for commit notifications) or if we rely on client side filtering only or what else should be done.
The example scripts (available in /usr/share/doc/torbrowser-launcher/examples in the package from sid or in git) show how to run torbrowser-launcher, confined with AppArmor, in Xephyr (a virtual Xserver running on another Xserver) as another user. This, using AppArmor and Xephyr, shall have two effects:
the browser process (and it's subprocesses) can - thanks to AppArmor confinement - only access a tiny part of the filesystem
the real Xserver is not exposed to the browser application, so hopefully that application cannot exploit bugs to grab keyboard input from other applications.
Does that really help? Feedback welcome.
Full quote of /usr/share/doc/torbrowser-launcher/examples/README:
torbrowser-launcher launcher scripts
====================================
These scripts are intended to run torbrowser-launcher (and thus torbrowser) as
another user in an Xephyr window server running inside your normal Xorg
session.
They assume the following packages are installed:
- torbrowser-launcher
- apparmor
- xserver-xephyr, awesome
- sudo, slay, psmisc
AppArmor should be enabled, but doesn't have to. I followed the HowTo from
https://wiki.debian.org/AppArmor, which can be summed up as just adding one
parameter to the kernel to enable it, followed by a reboot.
Using Apparmor has the advantage that the browser process cannot most of the
filesystem, eg saving downloads only works in ~/.torbrowser/tbb/x86_64/tor-browser_en-US/Desktop/
On wheezy, I'm using backports for torbrowser-launcher and apparmor.
The scripts assume they have been copied to /usr/local/bin/ and that there is
a user called "foo" (for running the actuall torbrowser(-launcher) process,
and that the current user has sudo rights for the following commands:
- sudo -i -u foo /usr/local/bin/tbb-l-wrapper
- sudo slay foo
There are two scripts, tbb-in-xephyr and tbb-l-wrapper. Only tbb-in-xephyr is
to be called directly and will result in torbrowser running in Xephyr.
Known problems:
---------------
- dbus is not started, so some input methods won't work. (Personally I don't
want/need dbus though, so I'm awaiting a solution to
https://trac.torproject.org/projects/tor/ticket/10014)
- not everybody likes awesome as the window manager being used ;)
Ideas, questions and ToDo:
--------------------------
- maybe all of this functionality could be integrated into.
torbrowser-launcher itself, just writing this in shell was so easy.
- or for the time being, merge these two scripts into one, doing both,
depending on how its called. Also make them run from everywhere.
- run this in an unprivileged LXC container, which is also apparmor confined.
- (when) does this double confinement make sense?
- use a more sensible named default user (instead of foo).
- there should really be an option, so torbrowser-launcher doesn't detach
itself, so that this "while;ps fax grep" hack can go away.
- ship an usable sudoers.d example too.
- support for more users / instances
Feedback welcome, especially accompanied by patches!
Finally. Apologies for the delay, but it's been a busy month. This time I will hold true to my word and upload a PDF for people to see (attached to this page).
So, what's new? Here is a list of things done:
* Finished ALL addition/subtraction related instructions for all engines and major derivatives (SSE*/AVX, VMX/VSX, NEON/armv8 NEON). With diagrams (these were the reasons it has taken so long).
* Reorganized the structure (split the book into Parts I/II, the instruction index will be in Part II, Part I will carry the design analysis of each SIMD engine.
* Added an TOC/index.
* So far, with just Addition/Subtraction Chapter and the rest empty sections, it has reached 175 pages (B5, again I'm not fixed on the size, it might actually change)! My estimate is that the whole book will surpass 800 pages with everything included.
TODO:
Friday was my last day at Collabora, the awesome Open Source consultancy in Cambridge. I d been there more than three years, and it was time for a change.
As luck would have it, that change came in the form of a job offer 3 months ago from my long-time friend in Open Source, Miguel de Icaza. Monday morning, I fly out to Xamarin s main office in Boston, for just over a week of induction and face time with my new co workers, as I take on the title of Release Engineer.
My job is to make sure Mono on Linux is a first-class citizen, rather than the best-effort it s been since Xamarin was formed from the ashes of the Attachmate/Novell deal. I m thrilled to work full-time on what I do already as community work including making Mono great on Debian/Ubuntu and hope to form new links with the packer communities in other major distributions. And I m delighted that Xamarin has chosen to put its money where its mouth is and fund continued Open Source development surrounding Mono.
If you re in the Boston area next week or the week after, ping me via the usual methods!
In short: I love my MacBook Air. It is the best (laptop) hardware
I ever owned. I have seen hardware which was much more flaky
in the past. I can set the display backlight to zero via software,
which saves me a lot of battery life and also offers a bit
of anti-spy-acroos-my-shoulder support. WLAN and bluetooth work nicely.
And I just love the form-factor and the touch-feeling of the hardware.
I even had the bag I use to carry my braille display
modified so that the Air just fits in.
I can't say how it behaves with X11. Given how flaky accessibility
with graphical desktops on Linux is, I have still not made
the switch. My MacBookAir is my perfect mobile terminal, I LOVE it.
I am sort of surprised about the recent rant of Paul about MacBook Hardware.
It is rather funny that we perceive the same technology so radically different.
And after reading the second part of his rant I am wondering
if I am no longer allowed to consider myself part of the
"hardcore F/OSS world", because I don't consider Apple as evil
as apparently most others. Why? Well, first of all, I actually like
the hardware. Secondly, you have to show me a vendor
first that builds usable accessibility into their products, and I mean
all their products, without any extra price-tag attached. Once the others
start to consider people with disabilities, we can talk about apple-bashing
again. But until then, sorry, you don't see the picture as I do.
Apple was the first big company on the market to take accessibility seriously.
And they are still unbeaten, at least when it comes to bells and whistles included.
I can unbox and configure any Apple product sold currently completely
without assisstance. With some products, you just need to know
a signle keypress (tripple-press the home button for touch devices and Cmd+F5 for
Mac OS/X), and with others, during initial bootup, a speech synthesizer even tells
you how to enable accessibility in case you need it.
And after that is enabled, I can perform the setup of the device
completely on my own. I don't need help from anyone else. And after the setup
is complete, I can use 95% of the functionality provided by the operating system.
And I am blind, part of a very small margin group so to speak.
In Debian circles, I have even heard the sentiment that we supposedly have to
accept that small margin groups are ignored sometimes. Well, as long
as we think that way, as long as we strictly think economically, we will never
be able to go there, fully. And we will never be the universal operating system,
actually. Sorry to say that, but I think there is some truth to it.
So, who is evil? Scratch your own itch doesn't always work to cover everything.
How do we motivate contributors to work on things they don't personally
need (yet)? How can we ensure that complicated but seldomly used
features stay stable and do not fall to dust just because some upstream
decides to rewrite an essential subcomponent of the dependency tree?
I don't know. All I know is that these issues need to be solved in
an universal operating system.
Folks from Collabora and Red Hat have been working on making Firefox on Gtk+ 3 a thing. See Emilio s blog post for some recent update. But getting Firefox to build and run locally is unfortunately not the whole story.
I ve been working on getting Gtk+ 3 Firefox builds going on Mozilla build infrastructure, and I m proud to announce today that those builds are now going through Mozilla continuous integration on a project branch: Elm, and receive the same automated testing as mozilla-central.
And when I said getting Firefox to build and run was unfortunately not the whole story, I meant it: if you click on the Elm link above, you ll notice that there s a lot of orange, when it should be all green.
So, yes, Firefox on Gtk+ 3 is a thing, and it now has continuous integration. But there s still a whole bunch of things to fix. So if you re interested in making those builds work better, you can hop in, there are many things you can do:
check the Gtk+ 3 tracking bug and its dependencies for a list of known issues or improvements to be made.
download one of the builds from the elm branch, test it, and file bugs if you find some that aren t currently tracked. There aren t nightlies, but you can get the latest builds for 32-bits and 64-bits systems.
and if you have level 1 commit access, you can test patches on the Try server, provided you pull from the elm branch or apply this patch on top of the tree you push there.
Jim has been busy as part of the procps-ng team that looks after top. Basically all the changes you find in top from around 2.7 or so are by him. Not satisfied enough with fixing top, making it faster and showing more fields, he has given us CPU and memory graphs.
He also thinks I don t have enough colours (or as he would put it colors) on my top output so I ve posted what the new top looks like for me so you can see the graphs and he can see my colours.
top, with colours
I think it is both colourful and useful addition. The colours have been available for a while now and the graphs will appear in the next upstream release of procps-ng.
Only a very short summary in English: I am Mexican. I am Jewish. I am almost completely disconnected from the local Jewish communities. And understanding the local Jewish communities is hard. There is a very interesting and brave campaign, recently started, called Neither do I The Mexican Jewish gay activist group Guimel, started off with this video (with English subtitles, if you are interested in following along). But how did I learn about this very bold initiative? By getting a hateful spam, inviting people to join a hate campaign. Right, the hate mail is not calling to violence, but it is based on premises as stupid as everybody's right not to include (to begin with).
So, the least I can do about this is to share both said hate mail and publicly denounce my shock on reading this nonsense nowadays. And, also in Spanish (I know many people following me don't understand it Sorry, it would just take too long, and after all, it's mainly for local "consumption"), this is the reply I sent to them (and to the other recipients). Sorry in advance to the Spanish speakers for my exabrupts :) This was written "as is", without much prior thought, and quite angry about what I had just read.
Buf... Expresiones como esta, que hacen evidente la cerraz n de tanta gente en las comunidades jud as, justifican claramente por qu tantos nos hemos ido abriendo, integr ndonos a la sociedad circundante. Me da verg enza ser asociado como jud o a comunidades donde se defienden estos puntos de vista; si bien yo no soy de la comunidad Monte Sinai (como resultar obvio por mi nombre), para la sociedad en su conjunto s soy un jud o (as , a secas: Jud o).
Mucho nos hemos quejado colectivamente a lo largo de los siglos acerca de la discriminaci n, de que seamos considerado la "basura" del mundo, los "apestados". Y es precisamente esa actitud, ante todo y en todo momento, lo que me ha llevado a alejarme del juda smo. Soy un ser humano m s, con una historia personal nica, con una historia cultural compartida pero tambi n nica, con algunas elecciones y algunas caracter sticas que me son inherentes nicas.
Algo de eso no le gusta a alguien? Por motivos racionales o irracionales? No lo puedo ni pienso evitar. Pero el que una comunidad completa sea llamada a ignorar, humillar, alienar a sus hijos, por una estupidez as de anacr nica (en el caso que lo presenta el video al que hacen referencia, y a la gente que valientemente ha conformado Guimel) me parece que simplemente va m s all de toda estupidez.
Piensen bien antes de adherirse a esta campa a de odio. Substituyan la sexualidad por cualquier otra raz n en la que somos minor a. No les da asco vivir con un vendedor de telas? Con un jud o apestoso?
Y s , aqu ya estoy yo tambi n dejando ir a esos demonios... que hay que mantener bajo control. Porque somos hombres y mujeres de nuestro siglo, y porque el mundo y las nuevas generaciones merecen nuestro mejor esfuerzo para ir colectivamente destruyendo la estupidez de nuestros ancestros.
Seamos m s racionales. Dejemos el odio, dejemos los prejuicios. Aceptemos a los diferentes, porque todos queremos que nos acepten. Porque todos somos diferentes.