I have received two MIPS Creator CI20 boards, thanks to Imagination Technologies. It s a small MIPS32 development board: As you can see it comes in a nice packaging with a world-compatible power adapter. It uses a Ingenic JZ4780 SoC with a dual core MIPS32 CPU running at 1.2GHz with a PowerVR SGX540 GPU. The board is fitted with 1GB of RAM, 8GB of NOR flash, HDMI output, USB 2.0 ports, Ethernet + Wi-Fi + BlueTooth, SD card slot, IR receiver, expansion headers and more. The schematics are available. The Linux kernel and the U-Boot bootloader sources are also available. Powering this board with a USB keyboard, a USB mouse and a HDMI display, it boots off the internal flash on a Debian Wheezy up to the XFCE environment. Besides the kernel, the Wi-Fi + Bluetooth firmware, and very few configuration changes, it runs a vanilla Debian. Unfortunately I haven t found time to play more with it yet, but it looks already quite promising. The board has not been formally announced yet, so I do not know when it will become available, nor the price, but if you are interested I ll bring it to DebConf14. Don t hesitate to ask me if you want to look at it or play with it.

We ve been busy setting up a Virtual Machine (VM) image to be used by participants of a MOOC that s opening in early september on Relational Databases at Telecom SudParis. We ve chosen to use Vagrant and VirtualBox which are used to build, distribute and run the box, providing scriptability (reproducibility) and making it portable on most operating systems. The VM itself contains a Debian (jessie) minimal system which runs (in the background) PostgreSQL, Apache + mod_php, phpPgAdmin, and a few applications of our own to play with example databases already populated in PostgreSQL.
As the MOOC s language will be french, we expect the box to be used mostly on machines with azerty keyboards. This and other context elements led us to add some customizations (locale, APT mirror) in provisioning scripts run during the box creation. At the moment, we generate 2 variants of the box, one for 32 bits kernel (i686) and one for 64 bits kernel (amd64) which (once compressed) represent betw. 300 and 350 Mb. The resulting boxes are uploaded to a self-hosting site, and distributed through vagrantcloud. Once the VM are created in VirtualBox, the typical VMDK drives file is around 1.3Gb. We use our own Debian base boxes containing a minimal Debian jessie/testing, instead of relying on someone else s, and recreate them using (the development branch version of) bootsrap-vz. This ensure we can put more trust in the content as it s a native Debian package installation without MITM intervention. The VM are meant to be run headless for the moment, keeping their size to the minimum, even though we also provide a script to install and configure a desktop environment based on XFCE4. The applications are either used through vagrant ssh, for instance for SQL command-line in psql, or in the Web browser, for our own Web based SQL exerciser, or phpPgAdmin (see a demo screencast (in french, w/ english subtitles)), which can then be used even off-line by the participants, which also means this requires no servers availability for our IT staff.
The MOOC includes a section on PHP + SQL programming, whose exercises can be performed using a shared sub-folder of /vagrant/ which allows editing on the host with the favourite native editor/IDE, while running PHP inside the VM s Apache + mod_php. The sources of our environment are available as free software, if you re interested to replicate a similar environment for another project. As we re still polishing the environment before the MOOC opening (on september 10th), I m not mentioning the box URLs but they shouldn t be too hard to find if you re investigating (refering to the fusionforge project s web site). We don t know yet how suitable this environment will be for learning SQL and database design and programming, and if Vagrant will bring more difficulties than benefits. Still we hope that the participants will find this practical, allowing them to work on the lab / exercises whenever and wherever they chose, removing the pain of installing and configuring a RDBMS on their machines, or the need to be connected to a cloud or to our overloaded servers. Of course, one limitation will be the requirements on the host machines, that will need to be reasonably modern, in order to run a virtualized Linux system. Another is access to high bandwidth for downloading the boxes, but this is kind of a requirement already for downloading/watching the videos of the MOOC classes Big thanks go to our intern St phane Germain, who joined us this summer to work on this virtualized environment.

Debian LTS - impressions and thoughts from my first month involvement About LTS - we want feedback and more companies supporting it financially Squeeze LTS, and even more Debian LTS, is a pretty young project, started in May 2014, so it's still a bit unclear where exactly we'll be going One purpose of this post is to spread some information about the initiative and invite you to tell us what you think about it or what your needs are. LTS stands for "Long Term Support" and the goal of the project is to extend the security support for Squeeze (aka the current oldstable distribution) by two years. If it weren't for Squeeze LTS, the security support for it would have been stopped in May 2014 (=one year after the release of the current stable distribution), which for many is a too short timespan after it's release in February 2011. It's an experiment, we hope that there will be a similar Wheezy LTS initiative in future, but the future is unwritten and things will change based on our experiences and your needs. If you have feedback on the direction LTS should take (or anything else LTS related), please comment on the lts mailing list. For immediate feedback there is also the #debian-lts IRC channel. Another quite pragmatic way to express your needs is to read more about how to financially contribute to LTS and then doing exactly that - and unsurprisingly we are prioritizing the updates based on the needs expressed from our paying customers. My LTS work in July 2014 So, "somehow" I started working for money on Debian LTS in July, which means there were 10h I got paid, and probably another 10h where I did LTS related work unpaid. I used those to release four updates for squeeze-lts (linux-2.6, file, munin and reportbug) fixing 22 CVEs in total. The unpaid work was mostly spent on unsuccessfully working on security updates and on adding support for LTS to the security team tracker, which I improved but couldn't fully address and which I haven't properly shared / committed yet... but at least I have a local instance of the tracker now, which - for LTS - is more useful than the .debian.org one. Hopefully during DebConf14 we'll manage to fix the tracker for good. Without success I've looked at libtasn1-3 (where the first fixes applied easily but then the code had changed too much from what was in squeeze compared to the available patches that I gave up) and libxstream-java (which is at version 1.3, while patches exist for upstream branches 1.4 and 2.x, but those need newer java to build and maybe if I'll spend two more hours I'll can get it build and then I'll have to find a useful test case, which looked quite difficult on a brief look.. so maybe I give up on libxstream-java too.... OTOH, if you use it and can do some testing, please do tell me. Working on all these updates turned out to be more team work than expected and a lot of work involving code (which I did expect), and often code which I'd normally not look at... similarily with tools: one has to deal with tools one doesnt like, eg I had to install cdbs... And as I usually like challenges, this has actually been a lot of fun! Though it's also pretty cool to use common best practices, easy and understandable workflows. I love README.Source (or better yet, when it's not needed). And while git is of course really really great, it's still very desirable if your package builds twice (=many times) in a row, without resetting it via git. Some more observations The first 16 updates (until July 19th) didn't have a DLA ID, until I suggested to introduce them and insisted until we agreed. So now we agreed to put the DLA ID in the subject of the announcement mails and there is also some tool support for generating the templates/mails, but enforcing proper subjects is not done, as silent bounces are useless (and non silent ones can be abused too easily). I'm not really happy about this, but who is happy with the way email works today? And I agree, it's not the end of the world if an LTS announcement is done without a proper ID, but it looks unprofessional and could be avoided, but we have more important work to do. But one day we should automate this better. Another detail I'm not entirely happy is the policy/current decision that "almost everything is fine to upload if deemed sensible by the uploader" (which is everyone in the Debian upload keyring(s)). In discussions before actually having the archive some people suggested the desire to upload new upstream versions too (eg newer kernels, iceweasel or other software to keep running a squeeze desktop in the modern world were discussed). I sincerely hope for most intrusive new upstream versions squeeze-(sloppy-)backports is used instead, and squeeze-lts rather not. Luckily so far all uploads were (IMHO) sensible and so, right now, I will just say that I hope it will stay this way. And it's true, one also has to install these upgrades in the first place. But people do that blindly all the time... So by design/desire currently there is no gatekeeping mechanism whatsover (no NEW, no proposed updates), except that only some selected "few" can upload. What is uploaded (and signed correctly), gets pushed to archive, buildds and the mirrors, and hopefully maybe someone will send an announcement. So far this has worked remarkedly well - and it's also the way the Debian Security team works, as I'm told. Looking at this from a process quality / automatisation perspective, all this manual and errorprone labour seems very strange to me. And then there is another thing: as already mentioned, the people working paid hours for this, are prioritizing their work based on customer requests. So we did two updates (python-scipy and file), which are not fixed in wheezy yet. I think this is unfortunate and while I could probably prepare the wheezy DSA for file, I don't really want to join the Security Team... or maybe I want/should join the Security Team and be a seldomly active member (eg fixing file in wheezy now....) A note related to this: out of those 37 uploads done until today, 16 were done by those two people being paid, while the other 21 uploads were done by 10 volunteers (or at least not paid by Debian LTS). It will be interesting to see how this statistics evolves over time. Last, but not least, there is also this can of worms (aka: the discussion) about paying people to work on Debian... I do agree it's work we didnt find volunteers for and I also see how the (financial side of the) setup is done outside of Debian (and well too, btw!), but... then we also use Debian ressources like buildds, the archive itself and official mailing lists. Also I'm currently biased in this discussion, as I directly (and happily) profit from it. I'm mentioning this here, because I believe it's important we discuss this and come to both good and practical conclusions. FWIW, we have also discussed this on the list, feel free to search the archives for it. To bring this post to an end: for those attending DebConf14 (directly or thanks to some ninjas), there will be an event about LTS in Portland, though I'm not sure yet what I will have to talk about what hasn't been already covered here But this probably means that will be a good opportunity for you to do lots of talking instead! I'm curious what you will have to say! Thanks for reading this far. I think I can promise that my next LTS report will be shorter

Today I did something interesting with the Debian packaging for propellor, which seems like it could be a useful technique for other Debian packages as well. Propellor is configured by a directory, which is maintained as a local git repository. In propellor's case, it's ~/.propellor/. This contains a lot of haskell files, in fact the entire source code of propellor! That's really unusual, but I think this can be generalized to any package whose configuration is maintained in its own git repository on the user's system. For now on, I'll refer to this as the config repo. The config repo is set up the first time a user runs propellor. But, until now, I didn't provide an easy way to update the config repo when the propellor package was updated. Nothing would break, but the old version would be used until the user updated it themselves somehow (probably by pulling from a git repository over the network, bypassing apt's signature validation). So, what I wanted was a way to update the config repo, merging in any changes from the new version of the Debian package, while preserving the user's local modifications. Ideally, the user could just run git merge upstream/master, where the upstream repo was included in the Debian package. But, that can't work! The Debian package can't reasonably include the full git repository of propellor with all its history. So, any git repository included in the Debian binary package would need to be a synthetic one, that only contains probably one commit that is not connected to anything else. Which means that if the config repo was cloned from that repo in version 1.0, then when version 1.1 came around, git would see no common parent when merging 1.1 into the config repo, and the merge would fail horribly. To solve this, let's assume that the config repo's master branch has a parent commit that can be identified, somehow, as coming from a past version of the Debian package. It doesn't matter which version, although the last one merged with will be best. (The easy way to do this is to set refs/heads/upstream/master to point to it when creating the config repo.) Once we have that parent commit, we have three things:

1. The current content of the config repo.
2. The content from some old version of the Debian package.
3. The new content of the Debian package.

Now git can be used to merge #3 onto #2, with -Xtheirs, so the result is a git commit with parents of #3 and #2, and content of #3. (This can be done using a temporary clone of the config repo to avoid touching its contents.) Such a git commit can be merged into the config repo, without any conflicts other than those the user might have caused with their own edits. So, propellor will tell the user when updates are available, and they can simply run git merge upstream/master to get them. The resulting history looks like this:

* Merge remote-tracking branch 'upstream/master'
 \  
  * merging upstream version
   \  
    * upstream version
*   user change
 /  
* upstream version

So, generalizing this, if a package has a lot of config files, and creates a git repository containing them when the user uses it (or automatically when it's installed), this method can be used to provide an easily mergable branch that tracks the files as distributed with the package. It would perhaps not be hard to get from here to a full git-backed version of ucf. Note that the Debian binary package doesn't have to ship a git repisitory, it can just as easily ship the current version of the config files somewhere in /usr, and check them into a new empty repository as part of the generation of the upstream/master branch.

A little more than 2 years ago, the Ceilometer project was launched inside the OpenStack ecosystem. Its main objective was to measure OpenStack cloud platforms in order to provide data and mechanisms for functionalities such as billing, alarming or capacity planning. In this article, I would like to relate what I've been doing with other Ceilometer developers in the last 5 months. I've lowered my involvement in Ceilometer itself directly to concentrate on solving one of its biggest issue at the source, and I think it's largely time to take a break and talk about it. Ceilometer early design For the last years, Ceilometer didn't change in its core architecture. Without diving too much in all its parts, one of the early design decision was to build the metering around a data structure we called samples. A sample is generated each time Ceilometer measures something. It is composed of a few fields, such as the the resource id that is metered, the user and project id owning that resources, the meter name, the measured value, a timestamp and a few free-form metadata. Each time Ceilometer measures something, one of its components (an agent, a pollster ) constructs and emits a sample headed for the storage component that we call the collector. This collector is responsible for storing the samples into a database. The Ceilometer collector uses a pluggable storage system, meaning that you can pick any database system you prefer. Our original implementation has been based on MongoDB from the beginning, but we then added a SQL driver, and people contributed things such as HBase or DB2 support. The REST API exposed by Ceilometer allows to execute various reading requests on this data store. It can returns you the list of resources that have been measured for a particular project, or compute some statistics on metrics. Allowing such a large panel of possibilities and having such a flexible data structure allows to do a lot of different things with Ceilometer, as you can almost query the data in any mean you want. The scalability issue We soon started to encounter scalability issues in many of the read requests made via the REST API. A lot of the requests requires the data storage to do full scans of all the stored samples. Indeed, the fact that the API allows you to filter on any fields and also on the free-form metadata (meaning non indexed key/values tuples) has a terrible cost in terms of performance (as pointed before, the metadata are attached to each sample generated by Ceilometer and is stored as is). That basically means that the sample data structure is stored in most drivers in just one table or collection, in order to be able to scan them at once, and there's no good "perfect" sharding solution, making data storage scalability painful. It turns out that the Ceilometer REST API is unable to handle most of the requests in a timely manner as most operations are O(n) where n is the number of samples recorded (see big O notation if you're unfamiliar with it). That number of samples can grow very rapidly in an environment of thousands of metered nodes and with a data retention of several weeks. There is a few optimizations to make things smoother in general cases fortunately, but as soon as you run specific queries, the API gets barely usable. During this last year, as the Ceilometer PTL, I discovered these issues first hand since a lot of people were feeding me back with this kind of testimony. We engaged several blueprints to improve the situation, but it was soon clear to me that this was not going to be enough anyway.

Thinking outside the box Unfortunately, the PTL job doesn't leave him enough time to work on the actual code nor to play with anything new. I was coping with most of the project bureaucracy and I wasn't able to work on any good solution to tackle the issue at its root. Still, I had a few ideas that I wanted to try and as soon as I stepped down from the PTL role, I stopped working on Ceilometer itself to try something new and to think a bit outside the box. When one takes a look at what have been brought recently in Ceilometer, they can see the idea that Ceilometer actually needs to handle 2 types of data: events and metrics. Events are data generated when something happens: an instance start, a volume is attached, or an HTTP request is sent to an REST API server. These are events that Ceilometer needs to collect and store. Most OpenStack components are able to send such events using the notification system built into oslo.messaging. Metrics is what Ceilometer needs to store but that is not necessarily tied to an event. Think about an instance CPU usage, a router network bandwidth usage, the number of images that Glance is storing for you, etc These are not events, since nothing is happening. These are facts, states we need to meter. Computing statistics for billing or capacity planning requires both of these data sources, but they should be distinct. Based on that assumption, and the fact that Ceilometer was getting support for storing events, I started to focus on getting the metric part right. I had been a system administrator for a decade before jumping into OpenStack development, so I know a thing or two on how monitoring is done in this area, and what kind of technology operators rely on. I also know that there's still no silver bullet this made it a good challenge. The first thing that came to my mind was to use some kind of time-series database, and export its access via a REST API as we do in all OpenStack services. This should cover the metric storage pretty well. Cooking Gnocchi

A cloud of gnocchis!

At the end of April 2014, this led met to start a new project code-named Gnocchi. For the record, the name was picked after confusing so many times the OpenStack Marconi project, reading OpenStack Macaroni instead. At least one OpenStack project should have a "pasta" name, right? The point of having a new project and not send patches on Ceilometer, was that first I had no clue if it was going to make something that would be any better, and second, being able to iterate more rapidly without being strongly coupled with the release process. The first prototype started around the following idea: what you want is to meter things. That means storing a list of tuples of (timestamp, value) for it. I've named these things "entities", as no assumption are made on what they are. An entity can represent the temperature in a room or the CPU usage of an instance. The service shouldn't care and should be agnostic in this regard. One feature that we discussed for several OpenStack summits in the Ceilometer sessions, was the idea of doing aggregation. Meaning, aggregating samples over a period of time to only store a smaller amount of them. These are things that time-series format such as the RRDtool have been doing for a long time on the fly, and I decided it was a good trail to follow. I assumed that this was going to be a requirement when storing metrics into Gnocchi. The user would need to provide what kind of archiving it would need: 1 second precision over a day, 1 hour precision over a year, or even both. The first driver written to achieve that and store those metrics inside Gnocchi was based on whisper. Whisper is the file format used to store metrics for the Graphite project. For the actual storage, the driver uses Swift, which has the advantages to be part of OpenStack and scalable. Storing metrics for each entities in a different whisper file and putting them in Swift turned out to have a fantastic algorithm complexity: it was O(1). Indeed, the complexity needed to store and retrieve metrics doesn't depends on the number of metrics you have nor on the number of things you are metering. Which is already a huge win compared to the current Ceilometer collector design. However, it turned out that whisper has a few limitations that I was unable to circumvent in any manner. I needed to patch it to remove a lot of its assumption about manipulating file, or that everything is relative to now (time.time()). I've started to hack on that in my own fork, but then everything broke. The whisper project code base is, well, not the state of the art, and have 0 unit test. I was starring at a huge effort to transform whisper into the time-series format I wanted, without being sure I wasn't going to break everything (remember, no test coverage). I decided to take a break and look into alternatives, and stumbled upon Pandas, a data manipulation and statistics library for Python. Turns out that Pandas support time-series natively, and that it could do a lot of the smart computation needed in Gnocchi. I built a new file format leveraging Pandas for computing the time-series and named it carbonara (a wink to both the Carbon project and pasta, how clever!). The code is quite small (a third of whisper's, 200 SLOC vs 600 SLOC), does not have many of the whisper limitations and it has test coverage. These Carbonara files are then, in the same fashion, stored into Swift containers. Anyway, Gnocchi storage driver system is designed in the same spirit that the rest of OpenStack and Ceilometer storage driver system. It's a plug-in system with an API, so anyone can write their own driver. Eoghan Glynn has already started to write a InfluxDB driver, working closely with the upstream developer of that database. Dina Belova started to write an OpenTSDB driver. This helps to make sure the API is designed directly in the right way. Handling resources Measuring individual entities is great and needed, but you also need to link them with resources. When measuring the temperature and the number of a people in a room, it is useful to link these 2 separate entities to a resource, in that case the room, and give a name to these relations, so one is able to identify what attribute of the resource is actually measured. It is also important to provide the possibility to store attributes on these resources, such as their owners, the time they started and ended their existence, etc.

Relationship of entities and resources

Once this list of resource is collected, the next step is to list and filter them, based on any criteria. One might want to retrieve the list of resources created last week or the list of instances hosted on a particular node right now. Resources also need to be specialized. Some resources have attributes that must be stored in order for filtering to be useful. Think about an instance name or a router network. All of these requirements led to to the design of what's called the indexer. The indexer is responsible for indexing entities, resources, and link them together. The initial implementation is based on SQLAlchemy and should be pretty efficient. It's easy enough to index the most requested attributes (columns), and they are also correctly typed. We plan to establish a model for all known OpenStack resources (instances, volumes, networks, ) to store and index them into the Gnocchi indexer in order to request them in an efficient way from one place. The generic resource class can be used to handle generic resources that are not tied to OpenStack. It'd be up to the users to store extra attributes. Dropping the free form metadata we used to have in Ceilometer makes sure that querying the indexer is going to be efficient and scalable.

The indexer classes and their relations

REST API All of this is exported via a REST API that was partially designed and documented in the Gnocchi specification in the Ceilometer repository; though the spec is not up-to-date yet. We plan to auto-generate the documentation from the code as we are currently doing in Ceilometer. The REST API is pretty easy to use, and you can use it to manipulate entities and resources, and request the information back.

Macroscopic view of the Gnocchi architecture

Roadmap & Ceilometer integration All of this plan has been exposed and discussed with the Ceilometer team during the last OpenStack summit in Atlanta in May 2014, for the Juno release. I led a session about this entire concept, and convinced the team that using Gnocchi for our metric storage would be a good approach to solve the Ceilometer collector scalability issue. It was decided to conduct this project experiment in parallel of the current Ceilometer collector for the time being, and see where that would lead the project to. Early benchmarks Some engineers from Mirantis did a few benchmarks around Ceilometer and also against an early version of Gnocchi, and Dina Belova presented them to us during the mid-cycle sprint we organized in Paris in early July. The following graph sums up pretty well the current Ceilometer performance issue. The more you feed it with metrics, the more slow it becomes.

For Gnocchi, while the numbers themselves are not fantastic, what is interesting is that all the graphs below show that the performances are stable without correlation with the number of resources, entities or measures. This proves that, indeed, most of the code is built around a complexity of O(1), and not O(n) anymore. Next steps

Cl ment drawing the logo

While the Juno cycle is being wrapped-up for most projects, including Ceilometer, Gnocchi development is still ongoing. Fortunately, the composite architecture of Ceilometer allows a lot of its features to be replaced by some other code dynamically. That, for example, enables Gnocchi to provides a Ceilometer dispatcher plugin for its collector, without having to ship the actual code in Ceilometer itself. That should help the development of Gnocchi to not be slowed down by the release process for now. The Ceilometer team aims to provide Gnocchi as a sort of technology preview with the Juno release, allowing it to be deployed along and plugged with Ceilometer. We'll discuss how to integrate it in the project in a more permanent and strong manner probably during the OpenStack Summit for Kilo that will take place next November in Paris.

Update 2014-08-20: apcid needs tweaking. See update section below. Somehow both my regular work laptop and home entertainment computers (both running Debian Jessie) were switched to systemd without me noticing. Judging from by dpkg.log it may have happened quite a while ago. I'm pretty sure that's a compliment to the backwards compatibility efforts made by the systemd developers and a criticism of me (I should be paying closer attention to what's happening on my own machines!). In any event, I've started trying to pay more attention now - particularly learning how to take advantage of this new software. I'll try to keep this blog updated as I learn. For now, I have made a few changes and discoveries. First - I have a convenient bash wrapper I use that both starts my OpenVPN client to a samba server and then mounts the drive. I only connect when I need to and rarely do I properly disconnect (the OpenVPN connection does not start automatically). So, I've written the script to carefully check if my openvpn connection is present and either restart or start depending on the results. I had something like this:

if ps -eFH   egrep [o]penvpn; then
  sudo /etc/init.d/openvpn restart
else
  sudo /etc/init.d/openvpn start
fi

One of the advertised advantages of systemd is the ability to more accurately detect if a service is running. So, first I changed this to:

if systemctl -q is-active openvpn.service; then
  sudo systemctl restart openvpn.service
else
  sudo systemctl start openvpn.service
fi

However, after reviewing the man page I realized I can shorten if further to simply:

  sudo systemctl restart openvpn.service

According to the man page, restart means:

Restart one or more units specified on the command line. If the units are not
running yet, they will be started.

After discovering this meaning for "restart" in systemd, I tested and realized that "restart" works the same way for openvpn using the old sysv style init system. Oh well. At least there's a man page and a stronger guarantee that it will work with all services, not just the ones that happen to respect that convention in their init.d scripts. The next step was to disable openvpn on start up. I confess, I never bothered to really learn update-rc.d. Everytime I read the manpage I ended up throwing up my hands and renaming symlinks by hand. In the case of openvpn I had previously edited /etc/default/openvpn to indicate that "none" of the virtual private networks should be started. Now, I've returned that file to the default configuration and instead I ran:

systemctl disable openvpn.service

UPDATES 2014-08-20: I've recently noticed strange behavior when I wake my laptop. Seems to sometimes go right back to sleep. After doing some digging I traced the problem to some customizations I have made to my laptop's acpid behavior combined with systemd taking over some apci events. Up to now, I have created my own /etc/acpi files so I have full control over the acpi events. In particular, I don't want my laptop to suspend when I close the lid. I only want it to suspend when I press the suspend key. And, when it does suspend, I want it to run my own personal suspend script so I can do things like lock the screen and restart tint2. I've found that systemd launches it's own acpi event monitoring that ignores my /etc/acpid rules (the systemd "unit" that monitors events is called acpid.socket which exists in addition to acpid.service). The systemd reaction to events is controlled by the systemd-logind.service which has a configuration file: /etc/systemd/logind.conf. By default, systemd-logind.service will put my laptop to sleep when the lid is closed and when the suspend button is pushed. systemd seems to get the signal first, putting the laptop to sleep. After I wake it up, acpid gets the signal - so it goes right back to sleep. Reading man logind.conf helps. I was able to restore my desired behavior by adding these lines to /etc/systemd/logind.conf:

HandleSuspendKey=ignore
HandleLidSwitch=ignore

Then: sudo systemctl restart systemd-logind.service.

Yet another update from my internship at Mozilla, as part of the OPW. An online triage workshop One of the most interesting thing I've done during the last weeks has been to held an online Bug Triage Workshop on the #testday channel at irc.mozilla.org.
That was a first time for me: I had been a moderator for a series of training sessions on IRC organized by Debian Women, but never a "speaker".
The experience turned out to be a good one: creating the material for the workshop had me basically summarize (not too much, I'm way too verbose!) all what I've learned in this past months about triaging in Mozilla, and speaking of it on IRC was a sort of challenge to my usual shyness. And I was so very lucky that a participant was able to reproduce the bug I picked as example, thus confirming it! How cool is that? ;) The workshop was about the very basics of triaging for Firefox, and we mostly focused on a simplified lifecycle of bugs, a guided tour of bugzilla (including the quicksearch and the advanced one, the list view, the individual bug view) and an explanation of the workflow of the triager. I still have my notes, and I plan to upload them to the wiki, sooner or later. I'm pretty satisfied of the outcome: the only regret is that the promoting wasn't enough, so we have few participants.
Will try to promote it better next time! :) about:crashes Another thing that had me quite busy in the last weeks was to learn more about crashes and stability in general.
If you are unfortunate enough to experience a crash with Firefox, you're probably familiar with the Mozilla Crash Reporter dialog box asking you to submit the crash report. But how does it works? From the client-side, Mozilla uses Breakpad as set of libraries for crash reporting. The Mozilla specific implementation adds to that a crash-reporting UI, a server to collect and process crash reported data (and particularly to convert raw dumps into readable stack traces) and a web interface, Socorro to view and parse crash reports. Curious about your crashes? The about:crashes page will show you a list of the submitted and unsubmitted crash reports. (And by the way, try to type about:about in the location bar, to find all the super-secret about pages!) For the submitted ones clicking on the CrashID will take you to the crash report on crash-stats, the website where the reports are stored and analyzed. The individual crash report page on crash-stats is awesome: it shows you the reported bug numbers if any bug summaries match the crash signature, as well as many other information. If crash-stats does not show a bug number, you really should file one! The CrashKill team works on these reports tracking the general stability of the various channels, triaging the top crashes, ensuring that the crash bugs have enough information and are reproducible and actionable by the devs.
The crash-stats site is a mine of information: take a look at the Top Crashes for Firefox 34.0a1.
If you click on a individual crash, you will see lots of details about it: just on the first tab ("Signature Summary") you can find a breakdown of the crashes by OS, by graphic vendors or chips or even by uptime range.
A very useful one is the number of crashes per install, so that you know how widespread is the crashing for that particular signature. You can also check the comments the users have submitted with the crash report, on the "Comments" tab. One and Done tasks review Last week I helped the awesome group of One and Done developers, doing some reviewing of the tasks pages. One and Done is a brilliant idea to help people contribute to the QA Mozilla teams.
It's a website proposing the user a series of tasks of different difficulty and on different topics to contribute to Mozilla. Each task is self-contained and can last few minutes or be a bit more challenging. The team has worked hard on developing it and they have definitely done an awesome job! :) I'm not a coding person, so I just know that they're using Django for it, but if you are interested in all the dirty details take a look at the project repository. My job has been only to check all the existent tasks and verify that the description and instruction are correct, that the task is properly tagged and so on. My impression is that this an awesome tool, well written and well thought with a lot of potential for helping people in their first steps into Mozilla. Something that other projects should definitely imitate (cough Debian cough). What's next? Next week I'll be back on working on bugs. I kind of love bugs, I have to admit it. And not squashing them: not being a coder make me less of a violent person toward digital insects. Herding them is enough for me. I'm feeling extremely non-violent toward bugs. I'll try to help Liz with the Test Plan for Firefox 34, on the triaging/verifying bugs part.
I'll also try to triage/reproduce some accessibility bugs (thanks Mario for the suggestion!).

Hi Everybody, So I already explained that we decided to go in a new direction and patch ucf to allow automatic configuration file merging with any custom command. Today I wanted to explain how to use ucf s new three-way-merge-command functionality in
conjunction with Elektra in order to ultilize Elektra s powerful tools in order to allow automatic three-way merges of your package s configuration during upgrades in a way that is more reliable than a diff3 merge. This guide assumes that you are fimilar with ucf already and are just trying to impliment the --three-way-merge-command option using Elektra. The addition of the --three-way-merge-command option was a part of my Google Summer of Code Project. This option takes the form:

--three-way-merge-command command [New File] [Destination]

Where command is the command you would like to use for the merge. New File and Destination are the same as always. We added a new script to Elektra called elektra-merge for use with this new option in ucf. This script acts as a liaison between ucf and Elektra, allowing a regular ucf command to run a kdb merge even though ucf commands only pass New File and Destination whereas kdb merge requires ourpath, theirpath, basepath, and resultpath. Since ucf already performs a three-way merge, it keeps track of all the necessary files to do so, even though it only takes in New File and Destination. In order to use elektra-merge, the current configuration file must be mounted to KDB to serve as ours in the merge. The script automatically mounts theirs, base, and result using the kdb remount command in order to use the same backend as ours (since all versions of the same file should use the same backend anyway) and this way users don t need to worry about specifying the backend for each version of the file. Then the script attempts a merge on the newly mounted KeySets. Once this is finished, either on success or not, the script finishes by unmouting all but our copy of the file to cleanup KDB. Then, if the merge was successful ucf will replace ours with the result providing the package with an automatically merged configuration which will also be updated in KDB itself. Additionally, we added two other scripts, elektra-mount and elektra-umount which act as simple wrappers for kdb mount and kdb umount. That work identically but are more script friendly The full command to use elektra-merge to perform a three-way merge on a file managed by ucf is:

ucf --three-way --threeway-merge-command elektra-merge [New File] [Destination]

Thats it! As described above, elektra-merge is smart enough to run the whole merge off of the information from that command and utilizes the new kdb remount command to do so. Integrating elektra-merge into a package that already uses ucf is very easy! In postinst you should have a line similar to:

ucf [New File] [Destination]

or perhaps:

ucf --three-way [New File] [Destination]
All you must do is in postinst, when run with the configure option you must mount the config file to Elektra:

kdb elektra-mount [New File] [Mounting Destination] [Backend]
Next, you must update the line containing ucf with the options --three-way and --threeway-merge-command like so:

ucf --three-way --threeway-merge-command elektra-merge [New File] [Destination]
Then, in your postrm script, during a purge, you must umount the config file before deleting it:

kdb lektra-umount [Name]
That s it! With those small changes you can use Elektra to perform automatic three-way merges on any files that your package uses ucf to handle! I just wanted to show a quick example, below is a diff representing the changes we made to the samba-common package in order to allow automatic configuration merging for smb.conf using Elektra. We chose this package because it already
uses ucf to handle smb.conf but it frequently requires users to manually merge changes across versions. Here is the patch showing what we changed:

diff samba_orig/samba-3.6.6/debian/samba-common.postinst samba/samba-3.6.6/debian/samba-common.postinst
92c92,93
< ucf --three-way --debconf-ok "$NEWFILE" "$CONFIG"
---
> kdb elektra-mount "$CONFIG" system/samba/smb ini
> ucf --three-way --threeway-merge-command elektra-merge --debconf-ok "$NEWFILE" "$CONFIG"
Only in samba/samba-3.6.6/debian/: samba-common.postinst~
diff samba_orig/samba-3.6.6/debian/samba-common.postrm samba/samba-3.6.6/debian/samba-common.postrm
4a5
> kdb elektra-umount system/samba/smb
As you can see, all we had to do was add the line to mount smb.conf during install, update the ucf command to include the new --threeway-merge-command option, and unmount system/samba/smb during a purge. It really is that easy! Sincerely,
Ian S. Donnelly

There hasn t been a progress-report on DEP-11 for some time, but that doesn t mean there was no work going on on it. DEP-11 is Debian s implementation of AppStream, as well as an effort to enhance the metadata available about software in Debian. While initially, AppStream was only about applications, DEP-11 was designed with a larger scope, to collect data about libraries, binaries and things like Python modules. Now, since AppStream 0.6, DEP-11 and AppStream have essentially the same scope, with the difference of DEP-11 metadata being described in YAML, while official AppStream data is XML. That was due to a request by our ftpmasters team, which doesn t like XML (which is also not used anywhere in Debian, as opposed to YAML). But this doesn t mean that people will have to deal with the YAML file format: The libappstream library will just take DEP-11 data as another data source for it s Xapian database, allowing anything using libappstream to access that data just like the XML stuff. Richards libappstream-glib will also receive support for the DEP-11 format soon, filling it s in-memory data cache and enabling the use of GNOME-Software on Debian. So, what has been done so far? The past months, my Google Summer of Code student. Abhishek Bhattacharjee, was working hard to integrate DEP-11 support into dak, the Debian Archive Kit, which maintains the whole Debian archive. The result will be an additional metadata table in our internal Postgres database, storing detailed information about the software available in a Debian package, as well as Components-<arch>.yml.gz files in the Debian repositories. Dak will also produce an application icon-cache and a screenshots repository. During the time of the SoC, Abhishek focused mainly on the applications part of things, and less on the other components (like extracting data about Python modules or libraries) these things can easily be implemented later. The remaining steps will be to polish the code and make it merge-ready for Debian s dak (as soon as it has received enough testing, we will likely give it a try on the Tanglu Debian derivative). Following that, Apt will be extended to fetch the DEP-11 data on-demand on systems where it is useful (which is currently mostly desktop-systems) if you want to save a little bit of space, you will be able to disable downloading this extra metadata in Apt. From there, libappstream will take the data for it s Xapian db. This will lead to the removal of the much-hated (from ftpmasters and maintainers side) app-install-data package, which has not been updated for two years and only contains a small fraction of the metadata provided by DEP-11. What Debian will ultimately gain from this effort is support for software-centers like GNOME-Software, and improved support for tools like Apper and Muon in displaying applications. Long-term, with more metadata being available, It would be cool to add support for it to specialized package managers , like Python s pip, npm or gem, to make them fetch information about available distribution software and install that instead of their own copies from 3rd-party repositories, if possible. This should ultimatively lead to less code duplication on distributions and will likely result in fewer security issues, since the officially maintained and integrated distribution packages can easily be used, if possible. This is no attempt to make tools like pip obsolete, but an attempt to have the different tools installing software on your machine communicate better, instead of creating parallel worlds in terms of software management. Another nice sideeffect of more metadata will be options to search for tools handling mimetypes in the software repos (in case you can t open a file), smart software centers installing missing firmware, and automatic suggestions for developers which software they need to install in order to build a specific software package. Also, the data allows us to match software across distributions, for that, I will have some news soon (not sure how soon though, as I am currently in thesis-writing-mode, and therefore have not that much spare time). Since the goal is to have these features available on all distributions supporting AppStream, it will take longer to realize but we are on a good way. So, if you want some more information about my student s awesome work, you can read his blogpost about it. He will also be at Debconf 14 (Portland). (I can t make it this time, but I surely won t miss the next Debconf) Sadly, I only see a very small chance to have the basic DEP-11 stuff land in-time for Jessie (lots of review work needs to be done, and some more code needs to be written), but we will definitively have it in Jessie+1. A small example on how this data will look like can be found here a larger, actual file is available here. Any questions and feedback are highly appreciated.

Last time I changed my desktop computer I bought a CPU from the Intel Haswell family, the one available on the market at that time. I carefully selected the CPU to make sure it supports as many instructions extensions as possible in this family (Intel likes segmentation, even high-end CPUs like the Core i7-4770k do not support all possible instructions). I ended-up choosing the Core i7-4771 as it supports the Transactional Synchronization Extensions (Intel TSX) instructions, which provide transactional memory support. Support for it has been recently added in the GNU libc, and has been activated in Debian. By choosing this CPU, I wanted to be sure that I can debug this support in case of bug report, like for example in bug#751147. Recently some computing websites started to mention that the TSX instructions have bugs on Xeon E3 v3 family (and likely on Core i7-4771 as they share the same silicon and stepping), quoting this Intel document. Indeed one can read on page 49:

HSW136. Software Using Intel TSX May Result in Unpredictable System Behavior

Problem: Under a complex set of internal timing conditions and system events, software using the Intel TSX (Transactional Synchronization Extensions) instructions may result in unpredictable system behavior.
Implication: This erratum may result in unpredictable system behavior.
Workaround: It is possible for the BIOS to contain a workaround for this erratum.

And later on page 51:

Due to Erratum HSw136, TSX instructions are disabled and are only supported for software development. See your Intel representative for details.

The same websites tell that Intel is going to disable the TSX instructions via a microcode update. I hope it won t be the case and that they are going to be able to find a microcode fix. Otherwise it would mean I will have to upgrade my desktop computer earlier than expected. It s a bit expensive to upgrade it every year and that s a the reason why I skipped the Ivy Bridge generation which didn t bring a lot from the instructions point of view. Alternatively I can also skip microcode and BIOS updates, in the hope I won t need another fix from them at some point.

This is a random post inspired by recent purchases. Some things we buy are practical, others are a little arbitrary. I tend to avoid buying things for the sake of it, and have explicitly started decluttering our house over the past few years. That said sometimes things just seem sufficiently "cool" that they get bought without too much thought. This entry is about two things. A couple of years ago my bathroom was ripped apart and refitted. Gone was the old and nasty room, and in its place was a glorious space. There was only one downside to the new bathroom - you turn on the light and the fan comes on too. When your wife works funny shifts at the hospital you can find that the (quiet) fan sounds very loud in the middle of the night and wakes you up.. So I figured we could buy a couple of LED lights and scatter them around the place - when it is dark the movement sensors turn on the lights. These things are amazing. We have one sat on a shelf, one velcroed to the bottom of the sink, and one on the floor, just hidden underneath the toilet. Due to the shiny-white walls of the room they're all you need in the dark. By contrast my second purchase was a mistake - The Logitech Harmony 650 Universal Remote Control should be great. It clearly has the features I want - Able to power:

• Our TV.
• Our Sky-box.
• OUr DVD player.

The problem is solely due to the horrific software. You program the device via an application/website which works only under Windows. I had to resort to installing Windows in a virtual machine to make it run:

# Get the Bus/ID for the USB device
bus=$(lsusb  grep -i Harmony   awk ' print $2 '   tr -d 0)
id=$(lsusb  grep -i Harmony   awk ' print $4 '   tr -d 0:)
# pass to kvm
kvm -localtime ..  -usb -device usb-host,hostbus=$bus,hostaddr=$id ..

That allows the device to be passed through to windows, though you'll later have to jump onto the Qemu console to re-add the device as the software disconnects and reconnects it at random times, and the bus changes. Sigh. I guess I can pretend it works, and has cut down on the number of remotes sat on our table, but .. The overwhelmingly negative setup and configuration process has really soured me on it. There is a linux application which will take a configuration file and squirt it onto the device, when attached via a USB cable. This software, which I found during research prior to buying it, is useful but not as much as I'd expected. Why? Well the software lets you upload the config file, but to get a config file you must fully complete the setup on Windows. It is impossible to configure/use this device solely using GNU/Linux. (Apparently there is MacOS software too, I don't use macs. *shrugs*) In conclusion - Motion-activated LED lights, more useful than expected, but Harmony causes Discord.

This is the last week of Google Summer of Code 2014 - all good things must come to an end. To wrap things up, I ve merged all my work on JSCommunicator into a new version with all the added features. You can now demo the new and improved (or at least so I hope) JSCommunicator on rtc.debian.org! JSCommunicator 2.0 has an assortment of new add-ons, the most important new features are the Instant Messaging component and the internationalization support. The UI has been reorganized but we are currently not using a skin for color scheme - will be posting about that in a bit. The idea is to have a more neutral look that can be easily customized and integrated with other web apps. A chat session is automatically opened when you begin a call with someone - unless you already started a chat session with said someone. Sound alerts for new incoming messages are optional in the config file, visual alerts occur when an inactive chat tab receives a new message. Future work includes multiple user chat sessions and adapting the layout to a large amount of chat tabs. Currently it only handles 6. (Should I allow more? Who chats with more than 6 people at once? 14 year old me would, but now I just can t handle that. Anyway, I welcome advice on how to go about this. Should we do infinite tabs or if not, what s the cut-off?) About internationalization, I m uber proud to say we currently run in 6 languages! The 6 are English (default), Spanish, French, Portuguese, Hebrew and German. But one thing I must mention is that since I added new stuff to JSCommunicator, some of the new stuff doesn t have a translation. I took care of the Portuguese translation and Yehuda Korotkin quickly turned in the Hebrew translation, but we are still missing an update for Spanish, French and German. If you can contribute, please do. There are about 10 new labels to translate, you can fix the issue here. Or if you re short on time, shoot me an email with the translation for what s on the right side of the = : welcome = Welcome, call = Call chat = Chat enter_contact = Enter contact type_to_chat = type to chat start_chat = start chat me = me logout = Logout no_contact = Please enter a contact. remember_me = Remember me I ll merge it myself but I ll be sure to add you to the authors list.

Hi Everybody, A few days ago I posted an entry on this blog called dpkg Woes where I explained that due to a lack of response, we were abandoning our plan to patch dpkg for my Google Summer of Code project, and I explained that we had a new solution. Well today I would like to tell you about that solution. Instead of patching dpkg, which would take a long time and seemed like it would never make it upstream, we have added some new features to ucf which will allow my Google Summer of Code project to be realized. If you don t know, ucf, which stands for Update Configuration File, is a popular Debian package whose goal is to preserve user changes to config files. It is meant to act as an alternative to considering a configuration file a conffile on systems that use dpkg. Instead, package maintainers can use ucf to handle these files in a conffile-like way. Where conffiles must work on all systems, because they are shipped with the package, configuration files that use ucf can be handled by maintainer scripts and can vary between systems. ucf exists as a script that allows conffile-like handling of non-conffile configuration files and allows much more flexibility than dpkg s conffile system. In fact, ucf even includes an option to perform a three-way merge on files it manages, it currently only uses diff3 for the task though. As you can see, ucf has a goal that while different than ours, seems naturally compatible to our goal of automatic conffile merging. Obviously, since ucf is a different tool than dpkg we had to re-think how we were going to integrate with ucf. Luckily, integration with ucf proved to be much more simple than integration with dpkg. All we had to do to integrate with ucf was to add a generic hook to attempt a three way merge using any tool created for the task such as Elektra and kdb merge. Felix submitted a pull request with the exact code almost a week ago and we have talked with Manoj Srivastava, the developer for ucf, and he seemed to really like the idea. The only changes we made are to add an option for a three-way merge command, and if one is present, the merge is attempted using the specified command. It s all pretty simple really. Since we decided to include a generic hook for a three-way merge command instead of an Elektra-specific one (which would be less open and would create a dependency on Elektra), we also had to add functionality to Elektra to work with this hook. We ended up writing a new script, called elektra-merge which is now included in our repository. All this script does is act as a liaison between the ucf --three-way-merge-command option and Elektra itself. The script automatically mounts the correct files for theirs and base and dest using the new remount command. Since the only parameters that are passed to the ucf merge command are the paths of ours, theirs, base and result, we were missing vital information on how to mount these files. Our solution was to create the remount command which mirrors the backend configuration of an existing mountpoint to create a new mountpoint using a new file. So if ours is mounted to system/ours using ini, kdb remount /etc/theirs system/theirs system/ours will mount /etc/theirs to system/theirs using the same backend as ours. Since theirs, base, and result should all have the same backend as ours, we can use remount to mount these files even if all we know is their paths. Now, package maintainers can edit their scripts to utilize this new feature. If they want, package maintainers can specify a command to use to merge files using ucf during package upgrades. I will soon be posting a tutorial about how to integrate this feature into a package and how to use Elektra in your scripts in order to allow for automatic three-way merges during package upgrade. I will post a link to the tutorial here once it is published. Sincerely,
Ian S. Donnelly

Hi everybody, Today I wanted to go over what I think is a very useful command in the kdb tool, kdb import. As you know, the kdb tool allows users to interact with the Elektra Key Database (KDB) via the command line. Today I would like to explain the import function of kdb. The command to use kdb import is:

kdb import [options] destination [format]
In this command, destination is where the imported Keys should be stored below. For instance, kdb import system/imported would store all the keys below system/imported. This command takes Keys from stdin to store them into KDB. Typically, this command is used with a pipe to read in the Keys from a file. The format argument you see above can be a very powerful option to use with kdb import. The format argument allows a user to specify which plug-in is used to import the Keys into the Key Database. The user can specify any storage plug-in to serve as the format for the Keys to be imported. For instance, if a user wanted to import a /etc/hosts file into KDB without mounting it, they could use the command cat /etc/hosts kdb import system/hosts hosts. This command would essentially copy the current hosts file into KDB, like mounting it. Unlike mounting it, changes to the Keys would not be reflected in the hosts file and vise versa. If no format is specified, the format dump will be used instead. The dump format is the standard way of expressing Keys and all their relevant information. This format is intended to be used only within Elektra. The dump format is a good means of backing up Keys from the Key Database for use with Elektra later such as reimporting them later. As of this writing, dump is the only way to fully preserve all parts of the KeySet. It is very important to note that the dump does not rename keys by design. If a user exports a KeySet using dump using a command such as kdb export system/backup > backup.ecf, they can only import that keyset back into system/backup using a command like cat backup.ecf kdb import system/backup. The kdb import command only takes one special option:

-s --strategy

which is used to specify a strategy to use if Keys already exist in the specified destination.
The current list of strategies are:

preserve any keys already in the destination will not be overwritten
overwrite any keys already in the destination will be overwritten if a new key has the same name
cut all keys already in the destination will be removed, then new keys will be imported

If no strategy is specified, the command defaults to the preserve strategy as to not be destructive to any previous keys. An example of using the kdb import is as follows:

cat backup.ecf kdb import system/restore

This command would import all keys stored in the file backup.ecf into the Key Database under system/restore. In this example, backup.ecf was exported from the KeySet using the dump format by using the command:
kdb export system/backup > backup.ecf backup.ecf contains all the information about the keys below system/backup:

$cat backup.ecf
kdbOpen 1
ksNew 3
keyNew 19 0
system/backup/key1
keyMeta 7 1
binary
keyEnd
keyNew 19 0
system/backup/key2
keyMeta 7 1
binary
keyEnd
keyNew 19 0
system/backup/key3
keyMeta 7 1
binary
keyEnd
ksEnd
Before the import command, system/backup does not exists and no keys are contained there.
After the import command, running the command kdb ls system/backup prints:

system/backup/key1
system/backup/key2
system/backup/key3
As you can see, the kdb import command is a very useful tool included as part of Elektra. I also wrote a tutorial on the kdb export command. Please go read that as well because those two commands go hand in hand and allow some very powerful usage of Elektra. Sincerely,
Ian S. Donnelly

Hi everybody, I recently posted a tutorial on the kdb import command. Well I also wanted to go over it s sibling function, kdb export. These two commands work very similarly, but there are some differences. First of all, the command to use kdb export is:

kdb export [options] source [format]
In this command, source is the root key of which Keys should be exported. For instance, kdb export system/export would export all the keys below system/export. Additionally, this command exports keys under the system/elektra directory by default. It does this so that information about the keys stored under this directory will be included if the Keys are later imported into an Elektra Key Database. This command export Keys to stdout to store them into the Elektra Key Database. Typically, the export command is used with redirection to write the Keys to a file. As we discussed already, the format argument can be a very powerful option to use with kdb export. Just like with kdb import the format argument allows a user to specify which plug-in is used to export the Keys from the Key Database. The user can specify any storage plug-in to serve as the format for the exported Keys. For instance, if a user mounted their hosts file to system/hosts using kdb mount /etc/hosts system/hosts hosts they would be able to export these Keys using the hosts format by using the command kdb export system/hosts hosts > hosts.ecf. This command would essentially create a backup of their current /etc/hosts file in a valid format for /etc/hosts. If no format is specified, the format dump will be used instead. The dump format is the standard way of expressing Keys and all their relevant information. This format is intended to be used only within Elektra. The dump format is a good means of backing up Keys from the Key Database for use with Elektra later such as reimporting them later. As of this writing, dump is the only way to fully preserve all parts of the KeySet. The kdb export command takes one special option, but it s different than the one for kdb import, that option is:

-E --without-elektra which omits the system/elektra directory of keys
An example of using the kdb export is as follows:

kdb export system/backup > backup.ecf
This command would export all keys stored under system/backup, along with relevant Keys in system/elektra, into a file called backup.ecf. As you can see, the kdb export command is a very useful tool just like it s sibling, kdb import. If you haven t yet, please go read the tutorial I wrote for kdb import because these two commands are best used together and can enable some really great features of Elektra. Sincerely,
Ian S. Donnelly

Preface During Google Summer of Code 2014, I got to work on the Debian WebRTC portal under the mentorship of Daniel Pocock. Now I had every intention of meticulously documenting my progress at each step of development in this blog, but I was a bit late in getting the blog up and running. I ll now be publishing a series of posts to recap all I ve done during GSoC. Better late than never. Intro JSCommunicator is a SIP communication tool developed in HTML and JavaScript. The code was designed to make integrating JSCommunicator with a website or web app as simple as possible. It s quite easy, really. However, I do think a more detailed explanation on how to set things up and a guide to the JSCommunicator architecture could be of use, particularly for those wanting to modify the code in any way. Setup To begin, please fork the JSCommunicator repo. If you are new to git, feel free to follow the steps in section Setup and Clone in this post. If you read the README file (which you always should), you ll see that JSCommunicator needs a SIP proxy that supports SIP over Websockets transport. Some options are:

• repro from reSIProcate
• Kamailio

I didn t find a tutorial for Kamailio setup, I did find one for repro setup. And bonus, here you have a great tutorial on how to setup and configure your SIP proxy AND your TURN server. In your project s root directory, you ll see a file named config-sample.js. Make a copy of that file named config.js. The config-sample.js file has comments that are very instructive. In sum, the only thing you have to modify is the turn_servers property and the websocket property. In my case, debrtc.org was the domain registered for testing my project, so my config file has: Note that unlike the sample, I didn t set a username and password so SIP credentials will be used. Now fill in the websocket property here we use sip5060.net. I m pretty sure you can set the connection_recovery properties to whatever you like. Everything else is optional. If you set the user property, specifically display_name and uri, that will be used to fill in the Login form and takes preference over any Remember me data. If you also set sip_auth_password, JSCommunicator will automatically log in. All the other properties are for other optional functionalities and are well explained. You ll need some third-party javascript that is not included in the JSCommunicator git repo. Namely jQuery version 1.4 or higher and ArbiterJS version 1.0. Download jQuery here and ArbiterJS here and place the .js files in your project s root directory. Do make sure that you are including the correct filename in your html file. For example, in phone-dev.shtml, a file named jquery.js is included. The file you downloaded will likely have version numbers in it s file name. So rename the downloaded file or change the content of src in your includes. This is uber trivial, but I ve made the mistake several times. You ll also need JsSIP.js which can be downloaded here. Same naming care must be taken as is the case for jQuery and ArbiterJS. The recently added Instant Messaging component and some of the new features need jQuery-UI - so far version 1.11.* is known to work. From the downloaded .zip all you need is the jquery-ui-...js file and the jquery-ui-...css file, also to be placed in the project s root directory. If you ll be using the internationalization support you ll also need jquery.i18n.properties.js. To try out JSCommunicator, deploy the website locally by copying your project directory to the apache document root directory (provided you are using apache, which is a good idea.). You ll likely have to restart your server before this works. Now the demo .shtml pages only have a header with all the necessary includes, then a Server Side Include for the body of the page, with all the JSCommunicator html. The html content is in the file jscommunicator.inc. You can enable SSI on apache, OR you can simply copy and paste the content of jscommunicator.inc into phone-dev.shmtl. Start up apache, open a browser and navigate to localhost/jscommunicator/phone-dev.shmtl and you should see: Actually, pending updates to JSCommunicator, you should see a brand new UI! But all the core stuff will be the same. Architecture Disclaimer: I m explaining my view of the JSCommunicator architecture, which currently may not be 100% correct. But so far it s been enough for me to make my modifications and additions to the code, so it could be of use. One I get a JSCommunicator Founding Father s stamp of approval, I ll be sure to confirm the accuracy. Now to explain how the JSCommunicator code interacts, the use of each code item is described, ignoring all the html and css which will vary according to how you choose to use JSCommunicator. I m also not going to explain jQuery which is a dependency but not specific to WebRTC. The core JSCommunicator code is the following:

• config.js
• jssip-helper.js
• parseuri.js
• webrtc-check.js
• JSCommUI.js
• JSCommManager.js
• make-release
• init.js
• Arbiter.js
• JsSIP.js

Each of these files will be presented in what I hope is an intuitive order.

• config.js - As expected, this file contains your custom configuration specifications, i.e. the servers being used to direct traffic, authentication credentials, and a series of properties to enable/disable optional functionalities in JSCommunicator.

The next three files could be considered utils :

• jssip-helper.js - This will load the data from config.js necessary to run JSCommunicator, such as configurations relating to servers, websockets, connection specifications (timeout, recovery intervals), and user credentials. Properties for optional features are ignored of course.
• parseuri.js - Contains a function to segregate data from a URI.
• webrtc-check.js - Verifies if browser is WebRTC compatible by checking if it s possible to enable camera and microphone access.

These two are where the magic happens:

• JSCommUI.js - Responsible for the UI component, controling what becomes visible to the user, audio effects, client-side error handling, and gathering the data that will be fed to JSCommManager.js.
• JSCommManager.js - Initializes the SIP User Agent to manage the session including (but not limited to) beginning/terminating the session, sending/receiving SIP messages and calls, and signaling the state of the session and other important notifications such as incoming calls, messages received, etc.

Now for some extras:

• make-release - Combines the main .js files into one big file. Gets jssip-helper.js, parseuri.js, webrtc-check.js, JSCommUI.js and JSCommManager.js and spits out a new file JSComm.js with all that content. Now you understand why phone-dev.shtml included each of the 5 files mentioned above whereas phone.shmtl includes only JSComm.js which didn t exist until you ran make-release. That confused me a bit.
• init.js - On page load, calls JSCommManager.js init function, which eventually calls JSCommUI.js init function. In other words, starts up the JSCommunicator app. I guess it s only used to show how to start up the app. This could be done directly on the page you re embedding JSCommunicator from. So maybe not entirely needed.

Third party code:

• Arbiter.js - Javascript implmentation of the publish/subscribe patten, written by Matt Kruse. In JSCommunicator it s used in JSCommManager.js to publish signals to describe direct the app s behavior. For example, JSCommManager will publish a message indicating that the user received a call from a certain URI. In event-demo.js we subscribe to this kind of message and when said message is received, an action can be performed such as adding to the app s call history. Very cool.
• JsSIP.js - Implements the SIP WebSocket transport in Javascript. This ensures the transportantion of data is done in adherence to the WebSocket protocol. In JSCommManager.js we initialize a SIP User Agent based in the implementation in JsSIP.js. The User Agent will translate all of the JSCommunicator actions into SIP WebSocket format. For example, when sending an IM, the JSCommunicator app will collect the essential data such as say the origin URI, destination URI and an IM message, while the User Agent is in charge of formating the data so that it can be transported in a SIP message unit. A SIP message contains a lot more information than just the sender, receiver and message. Of course, a lot of the info in a SIP message is irrelevant to the user and in JSCommUI.js we filter through all that data and only display what the user needs to see.

Here s a diagram of sorts to help you visualize how the code interacts: In sum, 1 - JSCommUI.js handles what is displayed in the UI and feeds data to JSCommManager.js; 2 - JSCommManager.js actually does stuff, feeding data to be displayed to JSCommUI.js; 3 - JSCommManager.js calls functions from the three utils , parseuri.js, webrtc-check.js and jssip-helpher.js which organizes the data from config.js; 4 - JSCommManager.js initializes a SIP User Agent based on the implementation in Arbiter.js. When making any changes to JSCommunicator, you will likely only be working with JSCommUI.js and JSCommManager.js.

Preface During Google Summer of Code 2014, I got to work on the Debian WebRTC portal under the mentorship of Daniel Pocock. Now I had every intention of meticulously documenting my progress at each step of development in this blog, but I was a bit late in getting the blog up and running. I ll now be publishing a series of posts to recap all I ve done during GSoC. Better late than never. Intro JSCommunicator is a SIP communication tool developed in HTML and JavaScript. The code was designed to make integrating JSCommunicator with a website or web app as simple as possible. It s quite easy, really. However, I do think a more detailed explanation on how to set things up and a guide to the JSCommunicator architecture could be of use, particularly for those wanting to modify the code in any way. Setup To begin, please fork the JSCommunicator repo. If you are new to git, feel free to follow the steps in section Setup and Clone in this post. If you read the README file (which you always should), you ll see that JSCommunicator needs a SIP proxy that supports SIP over Websockets transport. Some options are:

• repro from reSIProcate
• Kamailio

I didn t find a tutorial for Kamailio setup, I did find one for repro setup. And bonus, here you have a great tutorial on how to setup and configure your SIP proxy AND your TURN server. In your project s root directory, you ll see a file named config-sample.js. Make a copy of that file named config.js. The config-sample.js file has comments that are very instructive. In sum, the only thing you have to modify is the turn_servers property and the websocket property. In my case, debrtc.org was the domain registered for testing my project, so my config file has: Note that unlike the sample, I didn t set a username and password so SIP credentials will be used. Now fill in the websocket property here we use sip5060.net. I m pretty sure you can set the connection_recovery properties to whatever you like. Everything else is optional. If you set the user property, specifically display_name and uri, that will be used to fill in the Login form and takes preference over any Remember me data. If you also set sip_auth_password, JSCommunicator will automatically log in. All the other properties are for other optional functionalities and are well explained. You ll need some third-party javascript that is not included in the JSCommunicator git repo. Namely jQuery version 1.4 or higher and ArbiterJS version 1.0. Download jQuery here and ArbiterJS here and place the .js files in your project s root directory. Do make sure that you are including the correct filename in your html file. For example, in phone-dev.shtml, a file named jquery.js is included. The file you downloaded will likely have version numbers in it s file name. So rename the downloaded file or change the content of src in your includes. This is uber trivial, but I ve made the mistake several times. You ll also need JsSIP.js which can be downloaded here. Same naming care must be taken as is the case for jQuery and ArbiterJS. The recently added Instant Messaging component and some of the new features need jQuery-UI - so far version 1.11.* is known to work. From the downloaded .zip all you need is the jquery-ui-...js file and the jquery-ui-...css file, also to be placed in the project s root directory. If you ll be using the internationalization support you ll also need jquery.i18n.properties.js. To try out JSCommunicator, deploy the website locally by copying your project directory to the apache document root directory (provided you are using apache, which is a good idea.). You ll likely have to restart your server before this works. Now the demo .shtml pages only have a header with all the necessary includes, then a Server Side Include for the body of the page, with all the JSCommunicator html. The html content is in the file jscommunicator.inc. You can enable SSI on apache, OR you can simply copy and paste the content of jscommunicator.inc into phone-dev.shmtl. Start up apache, open a browser and navigate to localhost/jscommunicator/phone-dev.shmtl and you should see: Actually, pending updates to JSCommunicator, you should see a brand new UI! But all the core stuff will be the same. Architecture Disclaimer: I m explaining my view of the JSCommunicator architecture, which currently may not be 100% correct. But so far it s been enough for me to make my modifications and additions to the code, so it could be of use. One I get a JSCommunicator Founding Father s stamp of approval, I ll be sure to confirm the accuracy. Now to explain how the JSCommunicator code interacts, the use of each code item is described, ignoring all the html and css which will vary according to how you choose to use JSCommunicator. I m also not going to explain jQuery which is a dependency but not specific to WebRTC. The core JSCommunicator code is the following:

• config.js
• jssip-helper.js
• parseuri.js
• webrtc-check.js
• JSCommUI.js
• JSCommManager.js
• make-release
• init.js
• Arbiter.js
• JsSIP.js

Each of these files will be presented in what I hope is an intuitive order.

• config.js - As expected, this file contains your custom configuration specifications, i.e. the servers being used to direct traffic, authentication credentials, and a series of properties to enable/disable optional functionalities in JSCommunicator.

The next three files could be considered utils :

• jssip-helper.js - This will load the data from config.js necessary to run JSCommunicator, such as configurations relating to servers, websockets, connection specifications (timeout, recovery intervals), and user credentials. Properties for optional features are ignored of course.
• parseuri.js - Contains a function to segregate data from a URI.
• webrtc-check.js - Verifies if browser is WebRTC compatible by checking if it s possible to enable camera and microphone access.

These two are where the magic happens:

• JSCommUI.js - Responsible for the UI component, controling what becomes visible to the user, audio effects, client-side error handling, and gathering the data that will be fed to JSCommManager.js.
• JSCommManager.js - Initializes the SIP User Agent to manage the session including (but not limited to) beginning/terminating the session, sending/receiving SIP messages and calls, and signaling the state of the session and other important notifications such as incoming calls, messages received, etc.

Now for some extras:

• make-release - Combines the main .js files into one big file. Gets jssip-helper.js, parseuri.js, webrtc-check.js, JSCommUI.js and JSCommManager.js and spits out a new file JSComm.js with all that content. Now you understand why phone-dev.shtml included each of the 5 files mentioned above whereas phone.shmtl includes only JSComm.js which didn t exist until you ran make-release. That confused me a bit.
• init.js - On page load, calls JSCommManager.js init function, which eventually calls JSCommUI.js init function. In other words, starts up the JSCommunicator app. I guess it s only used to show how to start up the app. This could be done directly on the page you re embedding JSCommunicator from. So maybe not entirely needed.

Third party code:

• Arbiter.js - Javascript implmentation of the publish/subscribe patten, written by Matt Kruse. In JSCommunicator it s used in JSCommManager.js to publish signals to describe direct the app s behavior. For example, JSCommManager will publish a message indicating that the user received a call from a certain URI. In event-demo.js we subscribe to this kind of message and when said message is received, an action can be performed such as adding to the app s call history. Very cool.
• JsSIP.js - Implements the SIP WebSocket transport in Javascript. This ensures the transportantion of data is done in adherence to the WebSocket protocol. In JSCommManager.js we initialize a SIP User Agent based in the implementation in JsSIP.js. The User Agent will translate all of the JSCommunicator actions into SIP WebSocket format. For example, when sending an IM, the JSCommunicator app will collect the essential data such as say the origin URI, destination URI and an IM message, while the User Agent is in charge of formating the data so that it can be transported in a SIP message unit. A SIP message contains a lot more information than just the sender, receiver and message. Of course, a lot of the info in a SIP message is irrelevant to the user and in JSCommUI.js we filter through all that data and only display what the user needs to see.

Here s a diagram of sorts to help you visualize how the code interacts: In sum, 1 - JSCommUI.js handles what is displayed in the UI and feeds data to JSCommManager.js; 2 - JSCommManager.js actually does stuff, feeding data to be displayed to JSCommUI.js; 3 - JSCommManager.js calls functions from the three utils , parseuri.js, webrtc-check.js and jssip-helpher.js which organizes the data from config.js; 4 - JSCommManager.js initializes a SIP User Agent based on the implementation in Arbiter.js. When making any changes to JSCommunicator, you will likely only be working with JSCommUI.js and JSCommManager.js.