Search Results: "cas"

19 August 2014

Julien Danjou: Tracking OpenStack contributions in GitHub

I've switched my Git repositories to GitHub recently, and started to watch my contributions statistics, which were very low considering I spend my days hacking on open source software, especially OpenStack.
OpenStack hosts its Git repositories on its own infrastructure at git.openstack.org, but also mirrors them on GitHub. Logically, I was expecting GitHub to track my commits there too, as I'm using the same email address everywhere. It turns out that it was not the case, and the help page about that on GitHub describes the rule in place to compute statistics. Indeed, according to GitHub, I had no relations to the OpenStack repositories, as I never forked them nor opened a pull request on them (OpenStack uses Gerrit). Starring a repository is enough to build a relationship between a user and a repository, so this is was the only thing needed to inform GitHub that I have contributed to those repositories. Considering OpenStack has hundreds of repositories, I decided to star them all by using a small Python script using pygithub.
And voil , my statistics are now including all my contributions to OpenStack!

18 August 2014

Jelmer Vernooij: Using Propellor for configuration management

For a while, I've been wanting to set up configuration management for my home network. With half a dozen servers, a VPS and a workstation it is not big, but large enough to make it annoying to manually log into each machine for network-wide changes. Most of the servers I have are low-end ARM machines, each responsible for a couple of tasks. Most of my machines run Debian or something derived from Debian. Oh, and I'm a member of the declarative school of configuration management.
Propellor Propellor caught my eye earlier this year. Unlike some other configuration management tools, it doesn't come with its own custom language but it is written in Haskell, which I am already familiar with. It's also fairly simple, declarative, and seems to do most of the handful of things that I need. Propellor is essentially a Haskell application that you customize for your site. It works very similar to e.g. xmonad, where you write a bit of Haskell code for configuration which uses the upstream library code. When you run the application it takes your code and builds a binary from your code and the upstream libraries. Each host on which Propellor is used keeps a clone of the site-local Propellor git repository in /usr/local/propellor. Every time propellor runs (either because of a manual "spin", or from a cronjob it can set up for you), it fetches updates from the main site-local git repository, compiles the Haskell application and runs it.
Setup Propellor was surprisingly easy to set up. Running propellor creates a clone of the upstream repository under ~/.propellor with a README file and some example configuration. I copied config-simple.hs to config.hs, updated it to reflect one of my hosts and within a few minutes I had a basic working propellor setup. You can use ./propellor <host> to trigger a run on a remote host. At the moment I have propellor working for some basic things - having certain Debian packages installed, a specific network configuration, mail setup, basic Kerberos configuration and certain SSH options set. This took surprisingly little time to set up, and it's been great being able to take full advantage of Haskell. Propellor comes with convenience functions for dealing with some commonly used packages, such as Apt, SSH and Postfix. For a lot of the other packages, you'll have to roll your own for now. I've written some extra code to make Propellor deal with Kerberos keytabs and Dovecot, which I hope to submit upstream. I don't have a lot of experience with other Free Software configuration management tools such as Puppet and Chef, but for my use case Propellor works very well. The main disadvantage of propellor for me so far is that it needs to build itself on each machine it runs on. This is fine for my workstation and high-end servers, but it is somewhat more problematic on e.g. my Raspberry Pi's. Compilation takes a while, and the Haskell compiler and libraries it needs amount to 500Mb worth of disk space on the tiny root partition. In order to work with Propellor, some Haskell knowledge is required. The Haskell in the configuration file is reasonably easy to understand if you keep it simple, but once the compiler spits out error messages then I suspect you'll have a hard time without any Haskell knowledge. Propellor relies on having a central repository with the configuration that it can pull from as root. Unlike Joey, I am wary of publishing the configuration of my home network and I don't have a highly available local git server setup.

Julien Danjou: OpenStack Ceilometer and the Gnocchi experiment

A little more than 2 years ago, the Ceilometer project was launched inside the OpenStack ecosystem. Its main objective was to measure OpenStack cloud platforms in order to provide data and mechanisms for functionalities such as billing, alarming or capacity planning. In this article, I would like to relate what I've been doing with other Ceilometer developers in the last 5 months. I've lowered my involvement in Ceilometer itself directly to concentrate on solving one of its biggest issue at the source, and I think it's largely time to take a break and talk about it. Ceilometer early design For the last years, Ceilometer didn't change in its core architecture. Without diving too much in all its parts, one of the early design decision was to build the metering around a data structure we called samples. A sample is generated each time Ceilometer measures something. It is composed of a few fields, such as the the resource id that is metered, the user and project id owning that resources, the meter name, the measured value, a timestamp and a few free-form metadata. Each time Ceilometer measures something, one of its components (an agent, a pollster ) constructs and emits a sample headed for the storage component that we call the collector. This collector is responsible for storing the samples into a database. The Ceilometer collector uses a pluggable storage system, meaning that you can pick any database system you prefer. Our original implementation has been based on MongoDB from the beginning, but we then added a SQL driver, and people contributed things such as HBase or DB2 support. The REST API exposed by Ceilometer allows to execute various reading requests on this data store. It can returns you the list of resources that have been measured for a particular project, or compute some statistics on metrics. Allowing such a large panel of possibilities and having such a flexible data structure allows to do a lot of different things with Ceilometer, as you can almost query the data in any mean you want. The scalability issue We soon started to encounter scalability issues in many of the read requests made via the REST API. A lot of the requests requires the data storage to do full scans of all the stored samples. Indeed, the fact that the API allows you to filter on any fields and also on the free-form metadata (meaning non indexed key/values tuples) has a terrible cost in terms of performance (as pointed before, the metadata are attached to each sample generated by Ceilometer and is stored as is). That basically means that the sample data structure is stored in most drivers in just one table or collection, in order to be able to scan them at once, and there's no good "perfect" sharding solution, making data storage scalability painful. It turns out that the Ceilometer REST API is unable to handle most of the requests in a timely manner as most operations are O(n) where n is the number of samples recorded (see big O notation if you're unfamiliar with it). That number of samples can grow very rapidly in an environment of thousands of metered nodes and with a data retention of several weeks. There is a few optimizations to make things smoother in general cases fortunately, but as soon as you run specific queries, the API gets barely usable. During this last year, as the Ceilometer PTL, I discovered these issues first hand since a lot of people were feeding me back with this kind of testimony. We engaged several blueprints to improve the situation, but it was soon clear to me that this was not going to be enough anyway.
Thinking outside the box Unfortunately, the PTL job doesn't leave him enough time to work on the actual code nor to play with anything new. I was coping with most of the project bureaucracy and I wasn't able to work on any good solution to tackle the issue at its root. Still, I had a few ideas that I wanted to try and as soon as I stepped down from the PTL role, I stopped working on Ceilometer itself to try something new and to think a bit outside the box. When one takes a look at what have been brought recently in Ceilometer, they can see the idea that Ceilometer actually needs to handle 2 types of data: events and metrics. Events are data generated when something happens: an instance start, a volume is attached, or an HTTP request is sent to an REST API server. These are events that Ceilometer needs to collect and store. Most OpenStack components are able to send such events using the notification system built into oslo.messaging. Metrics is what Ceilometer needs to store but that is not necessarily tied to an event. Think about an instance CPU usage, a router network bandwidth usage, the number of images that Glance is storing for you, etc These are not events, since nothing is happening. These are facts, states we need to meter. Computing statistics for billing or capacity planning requires both of these data sources, but they should be distinct. Based on that assumption, and the fact that Ceilometer was getting support for storing events, I started to focus on getting the metric part right. I had been a system administrator for a decade before jumping into OpenStack development, so I know a thing or two on how monitoring is done in this area, and what kind of technology operators rely on. I also know that there's still no silver bullet this made it a good challenge. The first thing that came to my mind was to use some kind of time-series database, and export its access via a REST API as we do in all OpenStack services. This should cover the metric storage pretty well. Cooking Gnocchi
A cloud of gnocchis!
At the end of April 2014, this led met to start a new project code-named Gnocchi. For the record, the name was picked after confusing so many times the OpenStack Marconi project, reading OpenStack Macaroni instead. At least one OpenStack project should have a "pasta" name, right? The point of having a new project and not send patches on Ceilometer, was that first I had no clue if it was going to make something that would be any better, and second, being able to iterate more rapidly without being strongly coupled with the release process. The first prototype started around the following idea: what you want is to meter things. That means storing a list of tuples of (timestamp, value) for it. I've named these things "entities", as no assumption are made on what they are. An entity can represent the temperature in a room or the CPU usage of an instance. The service shouldn't care and should be agnostic in this regard. One feature that we discussed for several OpenStack summits in the Ceilometer sessions, was the idea of doing aggregation. Meaning, aggregating samples over a period of time to only store a smaller amount of them. These are things that time-series format such as the RRDtool have been doing for a long time on the fly, and I decided it was a good trail to follow. I assumed that this was going to be a requirement when storing metrics into Gnocchi. The user would need to provide what kind of archiving it would need: 1 second precision over a day, 1 hour precision over a year, or even both. The first driver written to achieve that and store those metrics inside Gnocchi was based on whisper. Whisper is the file format used to store metrics for the Graphite project. For the actual storage, the driver uses Swift, which has the advantages to be part of OpenStack and scalable. Storing metrics for each entities in a different whisper file and putting them in Swift turned out to have a fantastic algorithm complexity: it was O(1). Indeed, the complexity needed to store and retrieve metrics doesn't depends on the number of metrics you have nor on the number of things you are metering. Which is already a huge win compared to the current Ceilometer collector design. However, it turned out that whisper has a few limitations that I was unable to circumvent in any manner. I needed to patch it to remove a lot of its assumption about manipulating file, or that everything is relative to now (time.time()). I've started to hack on that in my own fork, but then everything broke. The whisper project code base is, well, not the state of the art, and have 0 unit test. I was starring at a huge effort to transform whisper into the time-series format I wanted, without being sure I wasn't going to break everything (remember, no test coverage). I decided to take a break and look into alternatives, and stumbled upon Pandas, a data manipulation and statistics library for Python. Turns out that Pandas support time-series natively, and that it could do a lot of the smart computation needed in Gnocchi. I built a new file format leveraging Pandas for computing the time-series and named it carbonara (a wink to both the Carbon project and pasta, how clever!). The code is quite small (a third of whisper's, 200 SLOC vs 600 SLOC), does not have many of the whisper limitations and it has test coverage. These Carbonara files are then, in the same fashion, stored into Swift containers. Anyway, Gnocchi storage driver system is designed in the same spirit that the rest of OpenStack and Ceilometer storage driver system. It's a plug-in system with an API, so anyone can write their own driver. Eoghan Glynn has already started to write a InfluxDB driver, working closely with the upstream developer of that database. Dina Belova started to write an OpenTSDB driver. This helps to make sure the API is designed directly in the right way. Handling resources Measuring individual entities is great and needed, but you also need to link them with resources. When measuring the temperature and the number of a people in a room, it is useful to link these 2 separate entities to a resource, in that case the room, and give a name to these relations, so one is able to identify what attribute of the resource is actually measured. It is also important to provide the possibility to store attributes on these resources, such as their owners, the time they started and ended their existence, etc.
Relationship of entities and resources
Once this list of resource is collected, the next step is to list and filter them, based on any criteria. One might want to retrieve the list of resources created last week or the list of instances hosted on a particular node right now. Resources also need to be specialized. Some resources have attributes that must be stored in order for filtering to be useful. Think about an instance name or a router network. All of these requirements led to to the design of what's called the indexer. The indexer is responsible for indexing entities, resources, and link them together. The initial implementation is based on SQLAlchemy and should be pretty efficient. It's easy enough to index the most requested attributes (columns), and they are also correctly typed. We plan to establish a model for all known OpenStack resources (instances, volumes, networks, ) to store and index them into the Gnocchi indexer in order to request them in an efficient way from one place. The generic resource class can be used to handle generic resources that are not tied to OpenStack. It'd be up to the users to store extra attributes. Dropping the free form metadata we used to have in Ceilometer makes sure that querying the indexer is going to be efficient and scalable.
The indexer classes and their relations
REST API All of this is exported via a REST API that was partially designed and documented in the Gnocchi specification in the Ceilometer repository; though the spec is not up-to-date yet. We plan to auto-generate the documentation from the code as we are currently doing in Ceilometer. The REST API is pretty easy to use, and you can use it to manipulate entities and resources, and request the information back.
Macroscopic view of the Gnocchi architecture
Roadmap & Ceilometer integration All of this plan has been exposed and discussed with the Ceilometer team during the last OpenStack summit in Atlanta in May 2014, for the Juno release. I led a session about this entire concept, and convinced the team that using Gnocchi for our metric storage would be a good approach to solve the Ceilometer collector scalability issue. It was decided to conduct this project experiment in parallel of the current Ceilometer collector for the time being, and see where that would lead the project to. Early benchmarks Some engineers from Mirantis did a few benchmarks around Ceilometer and also against an early version of Gnocchi, and Dina Belova presented them to us during the mid-cycle sprint we organized in Paris in early July. The following graph sums up pretty well the current Ceilometer performance issue. The more you feed it with metrics, the more slow it becomes.
For Gnocchi, while the numbers themselves are not fantastic, what is interesting is that all the graphs below show that the performances are stable without correlation with the number of resources, entities or measures. This proves that, indeed, most of the code is built around a complexity of O(1), and not O(n) anymore.
Next steps
Cl ment drawing the logo
While the Juno cycle is being wrapped-up for most projects, including Ceilometer, Gnocchi development is still ongoing. Fortunately, the composite architecture of Ceilometer allows a lot of its features to be replaced by some other code dynamically. That, for example, enables Gnocchi to provides a Ceilometer dispatcher plugin for its collector, without having to ship the actual code in Ceilometer itself. That should help the development of Gnocchi to not be slowed down by the release process for now. The Ceilometer team aims to provide Gnocchi as a sort of technology preview with the Juno release, allowing it to be deployed along and plugged with Ceilometer. We'll discuss how to integrate it in the project in a more permanent and strong manner probably during the OpenStack Summit for Kilo that will take place next November in Paris.

17 August 2014

Jamie McClelland: Getting to know systemd

Update 2014-08-20: apcid needs tweaking. See update section below. Somehow both my regular work laptop and home entertainment computers (both running Debian Jessie) were switched to systemd without me noticing. Judging from by dpkg.log it may have happened quite a while ago. I'm pretty sure that's a compliment to the backwards compatibility efforts made by the systemd developers and a criticism of me (I should be paying closer attention to what's happening on my own machines!). In any event, I've started trying to pay more attention now - particularly learning how to take advantage of this new software. I'll try to keep this blog updated as I learn. For now, I have made a few changes and discoveries. First - I have a convenient bash wrapper I use that both starts my OpenVPN client to a samba server and then mounts the drive. I only connect when I need to and rarely do I properly disconnect (the OpenVPN connection does not start automatically). So, I've written the script to carefully check if my openvpn connection is present and either restart or start depending on the results. I had something like this: 
if ps -eFH   egrep [o]penvpn; then
  sudo /etc/init.d/openvpn restart
else
  sudo /etc/init.d/openvpn start
fi
One of the advertised advantages of systemd is the ability to more accurately detect if a service is running. So, first I changed this to: 
if systemctl -q is-active openvpn.service; then
  sudo systemctl restart openvpn.service
else
  sudo systemctl start openvpn.service
fi
However, after reviewing the man page I realized I can shorten if further to simply: 
  sudo systemctl restart openvpn.service
According to the man page, restart means: 
Restart one or more units specified on the command line. If the units are not
running yet, they will be started.
After discovering this meaning for "restart" in systemd, I tested and realized that "restart" works the same way for openvpn using the old sysv style init system. Oh well. At least there's a man page and a stronger guarantee that it will work with all services, not just the ones that happen to respect that convention in their init.d scripts. The next step was to disable openvpn on start up. I confess, I never bothered to really learn update-rc.d. Everytime I read the manpage I ended up throwing up my hands and renaming symlinks by hand. In the case of openvpn I had previously edited /etc/default/openvpn to indicate that "none" of the virtual private networks should be started. Now, I've returned that file to the default configuration and instead I ran: 
systemctl disable openvpn.service
UPDATES 2014-08-20: I've recently noticed strange behavior when I wake my laptop. Seems to sometimes go right back to sleep. After doing some digging I traced the problem to some customizations I have made to my laptop's acpid behavior combined with systemd taking over some apci events. Up to now, I have created my own /etc/acpi files so I have full control over the acpi events. In particular, I don't want my laptop to suspend when I close the lid. I only want it to suspend when I press the suspend key. And, when it does suspend, I want it to run my own personal suspend script so I can do things like lock the screen and restart tint2. I've found that systemd launches it's own acpi event monitoring that ignores my /etc/acpid rules (the systemd "unit" that monitors events is called acpid.socket which exists in addition to acpid.service). The systemd reaction to events is controlled by the systemd-logind.service which has a configuration file: /etc/systemd/logind.conf. By default, systemd-logind.service will put my laptop to sleep when the lid is closed and when the suspend button is pushed. systemd seems to get the signal first, putting the laptop to sleep. After I wake it up, acpid gets the signal - so it goes right back to sleep. Reading man logind.conf helps. I was able to restore my desired behavior by adding these lines to /etc/systemd/logind.conf: 
HandleSuspendKey=ignore
HandleLidSwitch=ignore
Then: sudo systemctl restart systemd-logind.service.

16 August 2014

Matthias Klumpp: AppStream/DEP-11 Debian progress

There hasn t been a progress-report on DEP-11 for some time, but that doesn t mean there was no work going on on it. DEP-11 is Debian s implementation of AppStream, as well as an effort to enhance the metadata available about software in Debian. While initially, AppStream was only about applications, DEP-11 was designed with a larger scope, to collect data about libraries, binaries and things like Python modules. Now, since AppStream 0.6, DEP-11 and AppStream have essentially the same scope, with the difference of DEP-11 metadata being described in YAML, while official AppStream data is XML. That was due to a request by our ftpmasters team, which doesn t like XML (which is also not used anywhere in Debian, as opposed to YAML). But this doesn t mean that people will have to deal with the YAML file format: The libappstream library will just take DEP-11 data as another data source for it s Xapian database, allowing anything using libappstream to access that data just like the XML stuff. Richards libappstream-glib will also receive support for the DEP-11 format soon, filling it s in-memory data cache and enabling the use of GNOME-Software on Debian. So, what has been done so far? The past months, my Google Summer of Code student. Abhishek Bhattacharjee, was working hard to integrate DEP-11 support into dak, the Debian Archive Kit, which maintains the whole Debian archive. The result will be an additional metadata table in our internal Postgres database, storing detailed information about the software available in a Debian package, as well as Components-<arch>.yml.gz files in the Debian repositories. Dak will also produce an application icon-cache and a screenshots repository. During the time of the SoC, Abhishek focused mainly on the applications part of things, and less on the other components (like extracting data about Python modules or libraries) these things can easily be implemented later. The remaining steps will be to polish the code and make it merge-ready for Debian s dak (as soon as it has received enough testing, we will likely give it a try on the Tanglu Debian derivative). Following that, Apt will be extended to fetch the DEP-11 data on-demand on systems where it is useful (which is currently mostly desktop-systems) if you want to save a little bit of space, you will be able to disable downloading this extra metadata in Apt. From there, libappstream will take the data for it s Xapian db. This will lead to the removal of the much-hated (from ftpmasters and maintainers side) app-install-data package, which has not been updated for two years and only contains a small fraction of the metadata provided by DEP-11. What Debian will ultimately gain from this effort is support for software-centers like GNOME-Software, and improved support for tools like Apper and Muon in displaying applications. Long-term, with more metadata being available, It would be cool to add support for it to specialized package managers , like Python s pip, npm or gem, to make them fetch information about available distribution software and install that instead of their own copies from 3rd-party repositories, if possible. This should ultimatively lead to less code duplication on distributions and will likely result in fewer security issues, since the officially maintained and integrated distribution packages can easily be used, if possible. This is no attempt to make tools like pip obsolete, but an attempt to have the different tools installing software on your machine communicate better, instead of creating parallel worlds in terms of software management. Another nice sideeffect of more metadata will be options to search for tools handling mimetypes in the software repos (in case you can t open a file), smart software centers installing missing firmware, and automatic suggestions for developers which software they need to install in order to build a specific software package. Also, the data allows us to match software across distributions, for that, I will have some news soon (not sure how soon though, as I am currently in thesis-writing-mode, and therefore have not that much spare time). Since the goal is to have these features available on all distributions supporting AppStream, it will take longer to realize but we are on a good way. So, if you want some more information about my student s awesome work, you can read his blogpost about it. He will also be at Debconf 14 (Portland). (I can t make it this time, but I surely won t miss the next Debconf) Sadly, I only see a very small chance to have the basic DEP-11 stuff land in-time for Jessie (lots of review work needs to be done, and some more code needs to be written), but we will definitively have it in Jessie+1. A small example on how this data will look like can be found here a larger, actual file is available here. Any questions and feedback are highly appreciated.

15 August 2014

Aurelien Jarno: Intel about to disable TSX instructions?

Last time I changed my desktop computer I bought a CPU from the Intel Haswell family, the one available on the market at that time. I carefully selected the CPU to make sure it supports as many instructions extensions as possible in this family (Intel likes segmentation, even high-end CPUs like the Core i7-4770k do not support all possible instructions). I ended-up choosing the Core i7-4771 as it supports the Transactional Synchronization Extensions (Intel TSX) instructions, which provide transactional memory support. Support for it has been recently added in the GNU libc, and has been activated in Debian. By choosing this CPU, I wanted to be sure that I can debug this support in case of bug report, like for example in bug#751147. Recently some computing websites started to mention that the TSX instructions have bugs on Xeon E3 v3 family (and likely on Core i7-4771 as they share the same silicon and stepping), quoting this Intel document. Indeed one can read on page 49:
HSW136. Software Using Intel TSX May Result in Unpredictable System Behavior

Problem: Under a complex set of internal timing conditions and system events, software using the Intel TSX (Transactional Synchronization Extensions) instructions may result in unpredictable system behavior.
Implication: This erratum may result in unpredictable system behavior.
Workaround: It is possible for the BIOS to contain a workaround for this erratum.
And later on page 51:
Due to Erratum HSw136, TSX instructions are disabled and are only supported for software development. See your Intel representative for details.
The same websites tell that Intel is going to disable the TSX instructions via a microcode update. I hope it won t be the case and that they are going to be able to find a microcode fix. Otherwise it would mean I will have to upgrade my desktop computer earlier than expected. It s a bit expensive to upgrade it every year and that s a the reason why I skipped the Ivy Bridge generation which didn t bring a lot from the instructions point of view. Alternatively I can also skip microcode and BIOS updates, in the hope I won t need another fix from them at some point.

14 August 2014

Gregor Herrmann: RC bugs 2014/13 - 2014/33

perl 5.20 got uploaded to debian unstable a few minutes ago; be prepared for some glitches when upgrading sid machines/chroots in the next days, while all 557 reverse dependencies are rebuilt via binNMUs. how does this relate to this blog post's title? it does, since during the last weeks I was mostly trying to help with the preparation of this transition. & we managed to fix quite a few bugs while they were not bumped to serious yet, otherwise the list below would be a bit longer :) anyway, here are the the RC bugs I've worked on in the last 20 or so weeks: p.s.: & now, go & enjoy the new perl 5.20 features :)

Daniel Pocock: Bug tracker or trouble ticket system?

One of the issues that comes up from time to time in many organizations and projects (both community and commercial ventures) is the question of how to manage bug reports, feature requests and support requests. There are a number of open source solutions and proprietary solutions too. I've never seen a proprietary solution that offers any significant benefit over the free and open solutions, so this blog only looks at those that are free and open. Support request or bug? One common point of contention is the distinction between support requests and bugs. Users do not always know the difference. Some systems, like the Github issue tracker, gather all the requests together in a single list. Calling them "Issues" invites people to submit just about anything, such as "I forgot my password". At the other extreme, some organisations are so keen to keep support requests away from their developers that they operate two systems and a designated support team copies genuine bugs from the customer-facing trouble-ticket/CRM system to the bug tracker. This reduces the amount of spam that hits the development team but there is overhead in running multiple systems and having staff doing cut and paste. Will people use it? Another common problem is that a full bug report template is overkill for some issues. If a user is asking for help with some trivial task and if the tool asks them to answer twenty questions about their system, application version, submit log files and other requirements then they won't use it at all and may just revert to sending emails or making phone calls. Ideally, it should be possible to demand such details only when necessary. For example, if a support engineer routes a request to a queue for developers, then the system may guide the support engineer to make sure the ticket includes attributes that a ticket in the developers' queue should have. Beyond Perl Some of the most well known systems in this space are Bugzilla, Request Tracker and OTRS. All of these solutions are developed in Perl. These days, Python, JavaScript/Node.JS and Java have taken more market share and Perl is chosen less frequently for new projects. Perl skills are declining and younger developers have usually encountered Python as their main scripting language at university. My personal perspective is that this hinders the ability of Perl projects to attract new blood or leverage the benefits of new Python modules that don't exist in Perl at all. Bugzilla has fallen out of the Debian and Ubuntu distributions after squeeze due to its complexity. In contrast, Fedora carries the Bugzilla packages and also uses it as their main bug tracker. Evaluation I recently started having a look at the range of options in the Wikipedia list of bug tracking systems. Some of the trends that appear:
  • Many appear to be bug tracking systems rather than issue tracking / general-purpose support systems. How well do they accept non-development issues and keep them from spamming the developers while still providing a useful features for the subset of users who are doing development?
  • A number of them try to bundle other technologies, like wiki or FAQ systems: but how well do they work with existing wikis? This trend towards monolithic products is slightly dangerous. In my own view, a wiki embedded in some other product may not be as well supported as one of the leading purpose-built wikis.
  • Some of them also appear to offer various levels of project management. For development tasks, it is just about essential for dependencies and a roadmap to be tightly integrated with the bug/feature tracker but does it make the system more cumbersome for people dealing with support requests? Many support requests, like "I've lost my password", don't really have any relationship with project management or a project roadmap.
  • Not all appear to handle incoming requests by email. Bug tracking systems can be purely web/form-based, but email is useful for helpdesk systems.
Questions This leaves me with some of the following questions:
  • Which of these systems can be used as a general purpose help-desk / CRM / trouble-ticket system while also being a full bug and project management tool for developers?
  • For those systems that don't work well for both use cases, which combinations of trouble-ticket system + bug manager are most effective, preferably with some automated integration?
  • Which are more extendable with modern programming practices, such as Python scripting and using Git?
  • Which are more future proof, with choice of database backend, easy upgrades, packages in official distributions like Debian, Ubuntu and Fedora, scalability, IPv6 support?
  • Which of them are suitable for the public internet and which are only considered suitable for private access?

11 August 2014

Cyril Brulebois: Mark a mail as read across maildirs

Problem Discussions are sometimes started by mailing a few different mailing lists so that all relevant parties have a chance to be aware of a new topic. It s all nice when people can agree on a single venue to send their replies to, but that doesn t happen every time. Case in point, I m getting 5 copies of a bunch of mails, through the following debian-* lists: accessibility, boot, cd, devel, project. Needless to say: Reading, or marking a given mail as read once per maildir rapidly becomes a burden. Solution I know some people use a duplicate killer at procmail time (hello gregor) but I d rather keep all mails in their relevant maildirs. So here s mark-read-everywhere.pl which seems to do the job just fine for my particular setup: all maildirs below ~/mails/* with the usual cur, new, tmp subdirectories. Basically, given a mail piped from mutt, compute a hash on various headers, look at all new mails (new subdirectories), and mark the matching ones as read (move to the nearby cur subdirectories, and change suffix from , to ,S). Mutt key binding (where X is short for cross post): 
macro index X "<pipe-message>~/bin/mark-as-read-everywhere.pl<enter>"
This isn t pretty or bulletproof but it already started saving time! Now to wonder: was it worth the time to automate that?

Juliana Louback: JSCommunicator - Setup and Architecture

Preface During Google Summer of Code 2014, I got to work on the Debian WebRTC portal under the mentorship of Daniel Pocock. Now I had every intention of meticulously documenting my progress at each step of development in this blog, but I was a bit late in getting the blog up and running. I ll now be publishing a series of posts to recap all I ve done during GSoC. Better late than never. Intro JSCommunicator is a SIP communication tool developed in HTML and JavaScript. The code was designed to make integrating JSCommunicator with a website or web app as simple as possible. It s quite easy, really. However, I do think a more detailed explanation on how to set things up and a guide to the JSCommunicator architecture could be of use, particularly for those wanting to modify the code in any way. Setup To begin, please fork the JSCommunicator repo. If you are new to git, feel free to follow the steps in section Setup and Clone in this post. If you read the README file (which you always should), you ll see that JSCommunicator needs a SIP proxy that supports SIP over Websockets transport. Some options are: I didn t find a tutorial for Kamailio setup, I did find one for repro setup. And bonus, here you have a great tutorial on how to setup and configure your SIP proxy AND your TURN server. In your project s root directory, you ll see a file named config-sample.js. Make a copy of that file named config.js. The config-sample.js file has comments that are very instructive. In sum, the only thing you have to modify is the turn_servers property and the websocket property. In my case, debrtc.org was the domain registered for testing my project, so my config file has: 
turn_servers: [
	  server:"turn:debrtc.org"       
],
Note that unlike the sample, I didn t set a username and password so SIP credentials will be used. Now fill in the websocket property here we use sip5060.net. 
websocket:  
    servers: 'wss://ws.sip5060.net',
    connection_recovery_min_interval: 2,
    connection_recovery_max_interval: 30,
   ,
I m pretty sure you can set the connection_recovery properties to whatever you like. Everything else is optional. If you set the user property, specifically display_name and uri, that will be used to fill in the Login form and takes preference over any Remember me data. If you also set sip_auth_password, JSCommunicator will automatically log in. All the other properties are for other optional functionalities and are well explained. You ll need some third-party javascript that is not included in the JSCommunicator git repo. Namely jQuery version 1.4 or higher and ArbiterJS version 1.0. Download jQuery here and ArbiterJS here and place the .js files in your project s root directory. Do make sure that you are including the correct filename in your html file. For example, in phone-dev.shtml, a file named jquery.js is included. The file you downloaded will likely have version numbers in it s file name. So rename the downloaded file or change the content of src in your includes. This is uber trivial, but I ve made the mistake several times. You ll also need JsSIP.js which can be downloaded here. Same naming care must be taken as is the case for jQuery and ArbiterJS. The recently added Instant Messaging component and some of the new features need jQuery-UI - so far version 1.11.* is known to work. From the downloaded .zip all you need is the jquery-ui-...js file and the jquery-ui-...css file, also to be placed in the project s root directory. If you ll be using the internationalization support you ll also need jquery.i18n.properties.js. To try out JSCommunicator, deploy the website locally by copying your project directory to the apache document root directory (provided you are using apache, which is a good idea.). You ll likely have to restart your server before this works. Now the demo .shtml pages only have a header with all the necessary includes, then a Server Side Include for the body of the page, with all the JSCommunicator html. The html content is in the file jscommunicator.inc. You can enable SSI on apache, OR you can simply copy and paste the content of jscommunicator.inc into phone-dev.shmtl. Start up apache, open a browser and navigate to localhost/jscommunicator/phone-dev.shmtl and you should see: jscommunicatorRaw Actually, pending updates to JSCommunicator, you should see a brand new UI! But all the core stuff will be the same. Architecture Disclaimer: I m explaining my view of the JSCommunicator architecture, which currently may not be 100% correct. But so far it s been enough for me to make my modifications and additions to the code, so it could be of use. One I get a JSCommunicator Founding Father s stamp of approval, I ll be sure to confirm the accuracy. Now to explain how the JSCommunicator code interacts, the use of each code item is described, ignoring all the html and css which will vary according to how you choose to use JSCommunicator. I m also not going to explain jQuery which is a dependency but not specific to WebRTC. The core JSCommunicator code is the following: Each of these files will be presented in what I hope is an intuitive order. The next three files could be considered utils : These two are where the magic happens: Now for some extras: Third party code: Here s a diagram of sorts to help you visualize how the code interacts: jscommArch In sum, 1 - JSCommUI.js handles what is displayed in the UI and feeds data to JSCommManager.js; 2 - JSCommManager.js actually does stuff, feeding data to be displayed to JSCommUI.js; 3 - JSCommManager.js calls functions from the three utils , parseuri.js, webrtc-check.js and jssip-helpher.js which organizes the data from config.js; 4 - JSCommManager.js initializes a SIP User Agent based on the implementation in Arbiter.js. When making any changes to JSCommunicator, you will likely only be working with JSCommUI.js and JSCommManager.js.

Juliana Louback: JSCommunicator - Setup and Architecture

Preface During Google Summer of Code 2014, I got to work on the Debian WebRTC portal under the mentorship of Daniel Pocock. Now I had every intention of meticulously documenting my progress at each step of development in this blog, but I was a bit late in getting the blog up and running. I ll now be publishing a series of posts to recap all I ve done during GSoC. Better late than never. Intro JSCommunicator is a SIP communication tool developed in HTML and JavaScript. The code was designed to make integrating JSCommunicator with a website or web app as simple as possible. It s quite easy, really. However, I do think a more detailed explanation on how to set things up and a guide to the JSCommunicator architecture could be of use, particularly for those wanting to modify the code in any way. Setup To begin, please fork the JSCommunicator repo. If you are new to git, feel free to follow the steps in section Setup and Clone in this post. If you read the README file (which you always should), you ll see that JSCommunicator needs a SIP proxy that supports SIP over Websockets transport. Some options are: I didn t find a tutorial for Kamailio setup, I did find one for repro setup. And bonus, here you have a great tutorial on how to setup and configure your SIP proxy AND your TURN server. In your project s root directory, you ll see a file named config-sample.js. Make a copy of that file named config.js. The config-sample.js file has comments that are very instructive. In sum, the only thing you have to modify is the turn_servers property and the websocket property. In my case, debrtc.org was the domain registered for testing my project, so my config file has: 
turn_servers: [
	  server:"turn:debrtc.org"       
],
Note that unlike the sample, I didn t set a username and password so SIP credentials will be used. Now fill in the websocket property here we use sip5060.net. 
websocket:  
    servers: 'wss://ws.sip5060.net',
    connection_recovery_min_interval: 2,
    connection_recovery_max_interval: 30,
   ,
I m pretty sure you can set the connection_recovery properties to whatever you like. Everything else is optional. If you set the user property, specifically display_name and uri, that will be used to fill in the Login form and takes preference over any Remember me data. If you also set sip_auth_password, JSCommunicator will automatically log in. All the other properties are for other optional functionalities and are well explained. You ll need some third-party javascript that is not included in the JSCommunicator git repo. Namely jQuery version 1.4 or higher and ArbiterJS version 1.0. Download jQuery here and ArbiterJS here and place the .js files in your project s root directory. Do make sure that you are including the correct filename in your html file. For example, in phone-dev.shtml, a file named jquery.js is included. The file you downloaded will likely have version numbers in it s file name. So rename the downloaded file or change the content of src in your includes. This is uber trivial, but I ve made the mistake several times. You ll also need JsSIP.js which can be downloaded here. Same naming care must be taken as is the case for jQuery and ArbiterJS. The recently added Instant Messaging component and some of the new features need jQuery-UI - so far version 1.11.* is known to work. From the downloaded .zip all you need is the jquery-ui-...js file and the jquery-ui-...css file, also to be placed in the project s root directory. If you ll be using the internationalization support you ll also need jquery.i18n.properties.js. To try out JSCommunicator, deploy the website locally by copying your project directory to the apache document root directory (provided you are using apache, which is a good idea.). You ll likely have to restart your server before this works. Now the demo .shtml pages only have a header with all the necessary includes, then a Server Side Include for the body of the page, with all the JSCommunicator html. The html content is in the file jscommunicator.inc. You can enable SSI on apache, OR you can simply copy and paste the content of jscommunicator.inc into phone-dev.shmtl. Start up apache, open a browser and navigate to localhost/jscommunicator/phone-dev.shmtl and you should see: jscommunicatorRaw Actually, pending updates to JSCommunicator, you should see a brand new UI! But all the core stuff will be the same. Architecture Disclaimer: I m explaining my view of the JSCommunicator architecture, which currently may not be 100% correct. But so far it s been enough for me to make my modifications and additions to the code, so it could be of use. One I get a JSCommunicator Founding Father s stamp of approval, I ll be sure to confirm the accuracy. Now to explain how the JSCommunicator code interacts, the use of each code item is described, ignoring all the html and css which will vary according to how you choose to use JSCommunicator. I m also not going to explain jQuery which is a dependency but not specific to WebRTC. The core JSCommunicator code is the following: Each of these files will be presented in what I hope is an intuitive order. The next three files could be considered utils : These two are where the magic happens: Now for some extras: Third party code: Here s a diagram of sorts to help you visualize how the code interacts: jscommArch In sum, 1 - JSCommUI.js handles what is displayed in the UI and feeds data to JSCommManager.js; 2 - JSCommManager.js actually does stuff, feeding data to be displayed to JSCommUI.js; 3 - JSCommManager.js calls functions from the three utils , parseuri.js, webrtc-check.js and jssip-helpher.js which organizes the data from config.js; 4 - JSCommManager.js initializes a SIP User Agent based on the implementation in Arbiter.js. When making any changes to JSCommunicator, you will likely only be working with JSCommUI.js and JSCommManager.js.

10 August 2014

Cyril Brulebois: Why is my package blocked?

A bit of history: A while ago udeb-producing packages were getting frozen on a regular fashion, when a d-i release was about to be cut. While I wasn t looking at the time, I can easily understand the reasons behind that: d-i is built upon many components, it takes some time to make sure it s basically in shape for a release, and it s very annoying when a regression sneaks in right before the installation images get built. I took over d-i release maintenance in May 2012 and only a few uploads happened before the wheezy freeze. I was only discovering the job at the time, and I basically released whatever was in testing then. The freeze began right after that (end of June), so I started double checking things affecting d-i (in addition to or instead of the review performed by other release team members), and unblocking packages when changes seemed safe, or once they were tested. A few uploads happened after the wheezy release and there s already a Jessie Alpha 1 release. I was about to release Jessie Beta 1 after some fair bits of testing, a debian-installer upload, and the only remaining bits were: building installation images (hello Steve), and of course communication (mail announce and website update). Unfortunately a new upstream release reached testing in the meanwhile, breaking the installer in several ways. I ll give details below, of course not because I want to point finger at the maintainer, but to illustrate the ramifications that a single package s migrating to testing can induce. Since I d like to avoid such experience in the future, I ll probably reintroduce the old method and freeze all udeb-producing packages during next d-i releases. So you know why it might happen. Your next question might be: What to do when your package is getting caught in that net? . In that case, please get in touch with both debian-release@ and debian-boot@ asking for an unblock. I ll then review your particular package, and either let it migrate to testing, or delay it until after the release. Update: official announcement.

Ian Donnelly: How-To: kdb merge

Hi Everybody, As you may know, part of my Google Summer of Code project involved the creation of merge tools for Elektra. The one I am going to focus on today is kdb merge. The kdb tool allows users to access and perform functions on the Elektra Key Database from the command line. We added a new command to this very useful tool, the merge command. This command allows a user to perform a three-way merge of KeySets from the kdb tool.
The command to use this tool is:
kdb merge [options] ourpath theirpath basepath resultpath

The standard naming scheme for a three-way merge consists of ours, theirs, and base: This works very similarly for KeySets, especially ones that consist of mounted conffiles. For mounted conffiles: If the user is just trying to accomplish is a three-way merge using any two arbitrary keysets that share a base,
it doesn t matter which ones are defined as ours or theirs as long as they use the correct base KeySet.
In kdb merge, ourpath, theirpath, and basepath work just like ours, theirs, and base except each one represents the
root of a KeySet. Resultpath is pretty self-explanatory, it is just where you want the result of the merge to be saved under.
It s worth noting, resultpath should be empty before attempting a merge, otherwise there can be unintended consquences. As for the options, there are a few basic options:

-i --interactive which attempts the merge in an interactive way
-t --test which tests the proposed merge and informs you about possible conflicts
-f --force which overwrites any Keys in resultpath
 Additionally there is an option to specify a merge strategy, which is very important. The other option, strategy is:
s --strategy which is used to specify a strategy to use in case of a conflict

The current list of strategies are:
preserve the merge will fail if a conflict is detected
ours the merge will use our version during a conflict
theirs the merge will use their version during a conflict
base the merge will use the base version during a conflict
 If no strategy is specified, the merge will default to the preserve strategy as to not risk making the wrong decision. If any of the other strategies are specified, when a conflict is detected, merge will use the Key specified by the strategy (ours, theirs, or base) for the resulting Key. An example of using the kdb merge command:
kdb merge -s ours system/hosts/ours system/hosts/theirs system/hosts/base system/hosts/result
 For more in depth examples, please see the full tutorial on our GitHub Documentation. -Ian S. Donnelly

8 August 2014

Ian Donnelly: The Line Plug-In

Hi Everybody, As you may have noticed I wrote a new plug-in for Elektra called line . I used it for a lot of examples in my tutorial, How-To: Write a Plug-In. The line plug-in is a very simple storage plug-in for Elektra. This plug-in stores files into the Elektra Key Database creating a new Key for each line and setting the string value of the Key to the string value of the line of that file. So if we have a file called hello.txt :

Hello
World!

And we mount it to kdb like so: kdb mount ~/line.txt system/hello_line line. The output of kdb ls system/hello_line the output would be:

system/hello_line/#1
system/hello_line/#2

With the getString values of #1 and #2 being Hello and World! respectively. If this seems like a very simple plug-in, that s because it is. Obviously, this plug-in isn t a great showcase for the robustness of Elektra, any data structure could store a file line by line relatively easy, so why did we add a line plug-in at all? The answer is that we included a line plug-in to allow any line-based file to use functions of Elektra, particularly the new Merge function. My Google Summer of Code project is to allow for automatic three-way merge of Debian conffiles during package upgrades as opposed to the current prompt and manual merging a user must do if a conffile is edited. Using Elektra and the new mergecode we can mount a conffile with the best plug-in for it, (the ini plug-in for Samba s smb.conf for instance) and that allows for a very powerful merging ability with a lot more success than a simple diff merge. However, there are a lot of conffiles that don t use any particular standard (such as ini, xml, or JSON) to store data. That is where the line plug-in comes in. We can still mount these files using the line plug-in and attempt a merge. Of course it is much more likely to have conflicts, and this type of merge is still susceptible to many of the same flaws as regular file merges (such as not being able to detect when a line has been moved), but it simple cases, the merge may succeed which would reduce the overall number of times a user would be prompted during an upgrade. Basically, I wrote a line plug-in for Elektra as a fallback for conffile merges when we can t mount the conffile in any more meaningful way. While merges using KeySets that were mounted using line are more likely to fail than other, more specialized plug-ins, there are cases that these merges will succeed and the user will not have to deal with a confusing prompt. The whole point of my Google Summer of Code project is to make upgrading packages and dealing with conffiles much smoother and easier than it is now by including a three-way mere and this line plug-in will help with this goal. Sincerely,
Ian S. Donnelly

Sune Vuorela: Fun and joy with .bat files

Occasionally, one gets in touch with kind of foreign technologies and needs to get stuff working anyways. Recently, I had to do some various hacking with and around .bat files. Bat files are a kind of script files for Microsoft Windows. Calling external commands Imagine need to call some other command, let s say git diff. So from a cmd thing, you would write
git diff
similar to writing shell scripts on unixes. But there is a catch. If the thing you want to call is another bat-script, just calling it ensures it replaces the current script and never returns. So you need
call git diff
if the command you want to run is a bat file and you want to return to your script. Calling an external helper next to your script
If you for some reason needs to call some external helper placed next to your script, there is a helpful thing to do that as well. Imagine your helper is called helper.bat
call %~dp0helper.bat
is the very self-explanatory way of doing that. Stopping execution of your script If you somehow encounter some condition in your script that requires you to stop your script, there is a command exit handy. It even takes a argument for what error code there is.
exit 2
stops your script with return code 2. But it also have the nice added feature that if you do it in a script you run by hand in a terminal, it also exits the terminal. Luckily there is also a fix for that:
exit /b 2
and it doesn t exit your interactive terminal, and it sets the %ERRORLEVEL% variable to the exit code. Fortunately, the fun doesn t stop here. If the script is run non-interactively, exit /b doesn t set the exit code for for example perl s system() call. You need to use exit without /b for that. So now you need two scripts. one for interactive use that calls exit /b and a similar one using exit for use by other apps/scripts. Or, we can combine some of our knowledge and add a extra layer of indirection. and then success. Oh. and on a unrelated note. Windows can t schedule tasks for users that aren t logged in and don t have a password set. The response Access Denied is the only clue given.

Ian Donnelly: How-To: Write a Plug-In (Part 3, Coding)

Hi Everybody! Hope you have been enjoying my tutorial on writing plug-ins so far. In Part 1 we covered the basic overview of a plug-in. Part 2 covered a plug-in s contract and the best way to write one. Now, for Part 3 we are going to cover the meat of a plug-in, the actual coding. As you should know from reading Part 1, there are five main functions used for plug-ins, elektraPluginOpen, elektraPluginGet, elektraPluginSet, ELEKTRA_PLUGIN_EXPORT(Plugin), where Plugin should be replaced with the name of your plug-in. We are going to start this tutorial by focusing on the elektraPluginGet because it usually is the most critical function. As we discussed before, elektraPluginGet is the function responsible for turning information from a file into a usable KeySet. This function usually differs pretty greatly between each plug-in. This function should be of type int, it returns 0 on success or another number on an error. The function will take in a Key, usually called parentKey which contains a string containing the path to the file that is mounted. For instance, if you run the command kdb mount /etc/linetest system/linetest line then keyString(parentKey) should be equal to /etc/linetest . At this point, you generally want to open the file so you can begin saving it into keys. Here is the trickier part to explain. Basically, at this point you will want to iterate through the file and create keys and store string values inside of them according to what your plug-in is supposed to do. I will give a few examples of different plug-ins to better explain. My line plug-in was written to read files into a KeySet line by line using the newline character as a delimiter and naming the keys by their line number such as (#1, #2, .. #_22) for a file with 22 lines. So once I open the file given by parentKey, every time a I read a line I create a new key, let s call it new_key using dupKey(parentKey). Then I set new_keys s name to lineNN (where NN is the line number) using keyAddBaseName and store the string value of the line into the key using keySetString. Once the key is initialized, I append it to the KeySet that was passed into the elektraPluginGet function, let s call it returned for now, using ksAppendKey(return, new_key). Now the KeySet will contain new_key with the name lineNN properly saved where it should be according to the kdb mount command (in this case, system/linetest/lineNN), and a string value equal to the contents of that line in the file. MY plug-in repeats these steps as long as it hasn t reached end of file, thus saving the whole file into a KeySet line by line. The simpleini plug-in works similarly, but it parses for ini files instead of just line-by-line. At their most simple level, ini files are in the format of name=value with each pair taking one line. So for this plug-in, it makes a lot of sense to name each Key in the KeySet by the string to the left of the = sign and store the value into each key as a string. For instance, the name of the key would be name and keyGetString(name) would return value .
As you may have noticed, simpleini and line plug-ins work very similarly. However, they just parse the files differently. The simpleini plug-in parses the file in a way that is more natural to ini file (setting the key s name to the left side of the equals sign and the value to the right side of the equals sign.) The elektraPluginGet function is the heart of a storage plug-in, its what allows Elektra to store configurations in it s database. This function isn t just run when a file is first mounted, but whenever a file gets updated, this function is run to update the Elektra Key Database to match. We also gave a brief overview of elektraPluginSet function. This function is basically the opposite of elektraPluginGet. Where elektraPluginGet reads information from a file into the Elektra Key Database, elektraPluginSet writes information from the database back into the mounted file. First have a look at the signature of elektraLineSet: elektraLineSet(Plugin *handle ELEKTRA_UNUSED, KeySet *toWrite, Key *parentKey) Lets start with the most important parameters, the KeySet and the parentKey. The KeySet supplied is the KeySet that is going to be persisted in the file. In our case it would contain the Keys representing the lines. The parentKey is the topmost Key of the KeySet at serves several purposes. First, it contains the filename of the destination file as its value. Second, errors and warnings can be emitted via the parentKey. We will discuss error handling in more detail later. The Plugin handle can be used to persist state information in a threadsafe way with elektraPluginSetData. As our plugin is not stateful and therefore does not use the handle, it is marked as unused in order to supress compiler warnings. Basically the implementation of elektraLineSet can be described with the following pseudocode:

open the file
if (error)

ELEKTRA_SET_ERROR(74, parentKey, keyString(parentKey));

for each key

write the key value together with a newline

close the file
 The fullblown code can be found at https://github.com/ElektraInitiative/libelektra/blob/master/src/plugins/line/line.c As you can see, all elektraLineSet does is open a file, take each Key from the KeySet (remember they are named #1, #2 #_22) in order, and write each key as it s own line in the file. Since we don t care about the name of the Key in this case (other than for order), we just write the value of keyString for each Key as a new line in the file. That s it. Now, each time the mounted KeySet is modified, elektraPluginSet will be called and the mounted file will be updated. We haven t discussed ELEKTRA_SET_ERROR yet. Because Elektra is a library, printing errors to stderr wouldn t be a good idea. Instead, errors and warnings can be appended to a key in the form of metadata. This is what ELEKTRA_SET_ERROR does. Because the parentKey always exists even if a critical error occurres, we append the error to the parentKey. The first parameter is an id specifying the general error that occurred. A listing of existing errors together with a short description and a categorization can be found at https://github.com/ElektraInitiative/libelektra/blob/master/src/liberror/specification. The third parameter can be used to provide additional information about the error. In our case we simply supply the filename of the file that caused the error. The kdb tools will interprete this error and print it in a pretty way. Notice that this can be used in any plugin function where the parentKey is available. The elektraPluginOpen and elektraPluginClose functions are not commonly used for storage plug-ins, but they can be useful and are worth reviewing. elektraPluginOpen function runs before elektraPluginGet and is useful to do initialization if necessary for the plug-in. On the other hand elektraPluginClose is run after other functions of the plug-in and can be useful for freeing up resources. The last function, one that is always needed in a plug-in, is ELEKTRA_PLUGIN_EXPORT. This functions is responsible for letting Elektra know that the plug-in exists and which methods it implements. The code from my line function is a good example and pretty self-explanatory:

Plugin *ELEKTRA_PLUGIN_EXPORT(line)

return elektraPluginExport("line",
ELEKTRA_PLUGIN_GET, &elektraLineGet,
ELEKTRA_PLUGIN_SET, &elektraLineSet,
ELEKTRA_PLUGIN_END);

 There you have it! This is the last part of my tutorial on writing a storage plug-in for Elektra. Hopefully you now have a good understanding of how Elektra plug-ins work and you ll be able to add some great functionality into Elektra through development of new plug-ins. I hope you enjoyed this tutorial and if you have any questions just leave a comment! Happy coding!
Ian S. Donnelly

7 August 2014

Jordi Mallach: A pile of reasons why GNOME should be Debian jessie s default desktop environment

GNOME has, for some reason or another, always been the default desktop environment in Debian since the installer is able to install a full desktop environment by default. Release after release, Debian has been shipping different versions of GNOME, first based on the venerable 1.2/1.4 series, then moving to the time-based GNOME 2.x series, and finally to the newly designed 3.4 series for the last stable release, Debian 7 wheezy . During the final stages of wheezy s development, it was pointed out that the first install CD image would not longer hold all of the required packages to install a full GNOME desktop environment. There was lots of discussion surrounding this bug or fact, and there were two major reactions to it. The Debian GNOME team rebuilt some key packages so they would be compressed using xz instead of gzip, saving the few megabytes that were needed to squeeze everything in the first CD. In parallel, the tasksel maintainer decided switching to Xfce as default desktop was another obvious fix. This change, unannounced and two days before the freeze, was very contested and spurred the usual massive debian-devel threads. In the end, and after a few default desktop flip flops, it was agreed that GNOME would remain as the default for the already frozen wheezy release, and this issue would be revisited later on during jessie s development. And indeed, some months ago, Xfce was again reinstated as Debian s default desktop for jessie as announced: 
Change default desktop to xfce.
This will be re-evaluated before jessie is frozen. The evaluation will
start around the point of DebConf (August 2014). If at that point gnome
looks like a better choice, it ll go back as the default.
Some criteria for that choice will include:
* Popcon numbers for gnome on jessie. If gnome installations continue to
  rise fast enough despite xfce being the default (compared with, say
  kde installations), then we ll know that users prefer gnome.
  Currently we have no data about how many users would choose gnome when
  it s not the default. Part of the reason for switching to xfce now
  is to get such data.
* The state of accessability support, particularly for the blind.
* How well the UI works for both new and existing users. Gnome 3
  seems to be adding back many gnome 2 features that existing users
  expect, as well as making some available via addons. If it feels
  comfortable to gnome 2 (and xfce) users, that would go a long way
  toward switching back to it as the default. Meanwhile, Gnome 3 is also
  breaking new ground in its interface; if the interface seems more
  welcoming to new users, or works better on mobile devices, etc, that
  would again point toward switching back.
* Whatever size constraints exist for CD or other images at the time.
--
Hello to all the tech journalists out there. This is pretty boring.
Why don t you write a story about monads instead?
Joey Hess in dfca406eb694e0ac00ea04b12fc912237e01c9b5.
Suffice to say that the Debian GNOME team participants have never been thrilled about how the whole issue is being handled, and we ve been wondering if we should be doing anything about it, or just move along and enjoy the smaller amount of bug reports against GNOME packages that this change would bring us, if it finally made it through to the final release. During our real life meet-ups in FOSDEM and the systemd+GNOME sprint in Antwerp, most members of the team did feel Debian would not be delivering a graphical environment with the polish we think our users deserve, and decided we at least should try to convince the rest of the Debian project and our users that Debian will be best suited by shipping GNOME 3.12 by default. Power users, of course, can and know how to get around this default and install KDE, Xfce, Cinnamon, MATE or whatever other choice they have. For the average user, though, we think we should be shipping GNOME by default, and tasksel should revert the above commit again. Some of our reasons are: In short, we think defaulting to GNOME is the best option for the Debian release, and in contrast, shipping Xfce as the default desktop could mean delivering a desktop experience that has some incomplete or rough edges, and not on par with Debian quality standards for a stable release. We believe tasksel should again revert the change and be uploaded as soon as possible, in order to get people testing images with GNOME the sooner the better, with the freeze only two months away. We would also like that in the future, changes of this nature will not be announced in a git commit log, but widely discussed in debian-project and the other usual development/decision channels, like the change of init system happened recently. We will, whichever the final decision is, continue to package GNOME with great care to ensure our users get the best possible desktop experience Debian can offer.

Niels Thykier: Recent improvements to Britney2

As mentioned by Rapha l Hertzog, I have been spending some time on improving Britney2. Just the other day I submitted a second branch for review that I expect to merge early next week. I also got another set patches coming up soon. Currently, none of them are really user visible, so unless you are hosting your own version of Britney, these patches are probably not all that interesting to you. The highlights:
  1. Reduce the need for backtracking by finding semantically equivalent packages.
  2. Avoid needing to set up a backtrack point in some cases.
    • This has the side-effect of eliminating some O(e^n) runtime cases.
  3. Optimise installability testing of packages affected by a hinted migration.
    • This has the side-effect of avoiding some O(e^n) runtime cases when the post-hint state does not trigger said behaviour.
    • There is a follow-up patch for this one coming in the third series to fix a possible bug for a corner-case (causing a valid hint to be incorrectly rejected when it removed an uninstallable package).
  4. Reduce the number of affected packages to test when migrating items by using knowledge about semantically equivalent packages.
    • In some cases, Britney can now do free migrations when all binaries being updated replace semantically equivalent packages.
  5. (Merge pending) Avoid many redundant calls to sort_actions() , which exhibits at least O(n^2) runtime in some cases.
    • For the dataset Rapha l submitted, this patch shaves off over 30 minutes runtime. In the particular case, each call to sort_actions takes 3+ minutes and it was called at least 10 times, where it was not needed.
    • That said, sort_actions have a vastly lower runtime in the runs for Debian (and presumably also Ubuntu, since no one complained from their side so far).
The results so far: After the first patch series was merged, the Kali dataset (from Rapha l) could be processed in only ~2 hours. With the second patch series merged, the dataset will drop by another 30-50 minutes (most of which are thanks to the change mentioned in highlight #5). The third patch series currently do not have any mention-worthy performance related changes. It will probably be limited to bug fixes and some refactoring. Reflections: The 3 first highlights only affects the new installability tester meaning that the Britney2 instances at Ubuntu and Tanglu should be mostly unaffected by the O(n^2) runtime. Although those cases will probably just fail with several AIEEE s. :) The 5th highlight should equally interesting to all Britney2 instances though. For me, the most interesting part is that we have never observed the O(n^2) behaviour in a daily sid -> testing run. The dataset from Rapha l was basically a stable -> testing/sid run, which is a case I do not think we have ever done before. Despite our current updates, there is still room for improvements on that particular use case. In particular, I was a bit disheartened at how poorly our auto hinter(s) performed on this dataset. Combined they only assisted with the migration of something like 28 items . For comparison, the main run migrated ~7100 items and 9220 items were unable to migrate. Furthermore, the Original auto hinter spend the better part of 15 minutes computing hints at least it results in 10 items migrating. Links to the patches:

6 August 2014

Julian Andres Klode: Configuring an OpenWRT Router as a repeater for a FRITZ!Box with working Multicast

Since some time, those crappy Fritz!Box devices do not support WDS anymore, but rather a proprietary solution created by AVM. Now what happens if you have devices in another room that need/want wired access (like TVs, Playstations) or if you want to extend the range of your network? Buying another Fritz!Box is not very cost efficient What I did was to buy a cheap TP-Link TL-WR841N (can be bought for 18 euros) and installed OpenWRT on it. Here s how I configured it to act as a WiFi bridge. Basic overview: You configure OpenWRT into station mode (that is, as a WiFi client) and use relayd to relay between the WiFi network and your local network. You also need igmpproxy to proxy multicast packages between those networks, other UPnP stuff won t work. I did this on the recent Barrier Braker RC2. It should work on older versions as well, but I cannot promise it (I did not get igmpproxy to work in Attitude Adjustment, but that was probably my fault). Note: I don t know if it works with IPv6, I only use IPv4. You might want to re-start (or start) services after the steps, or reboot the router afterwards. Configuring WiFi connection to the FRITZ!Box Add to: /etc/config/network 
config interface 'wwan'
	option proto 'dhcp'
(you can use any other name you want instead of wwan, and a static IP. This will be your uplink to the Fritz!Box) Replace wifi-iface in: /etc/config/wireless: 
config wifi-iface
	option device 'radio0'
	option mode 'sta'
	option ssid 'FRITZ!Box 7270'
	option encryption 'psk2'
	option key 'PASSWORD'
	option network 'wwan'
(adjust values as needed for your network) Setting up the pseudo-bridge First, add wwan to the list of networks in the lan zone in the firewall. Then add a forward rule for the lan network (not sure if needed). Afterwards, configure a new stabridge network and disable the built-in DHCP server. Diff for /etc/config/firewall 
@@ -10,2 +10,3 @@ config zone
 	list network 'lan'
+	list network 'wwan'
 	option input 'ACCEPT'
@@ -28,2 +29,7 @@ config forwarding
 
+# Not sure if actually needed
+config forwarding
+	option src 'lan'
+	option dest 'lan'
+
 config rule
Add to /etc/config/network 
config interface 'stabridge'
	option proto 'relay'
	option network 'lan wwan'
	option ipaddr '192.168.178.26'
(Replace 192.168.178.26 with the IP address your OpenWRT router was given by the Fritz!Box on wlan0) Also make sure to ignore dhcp on the lan interface, as the DHCP server of the FRITZ!Box will be used: Diff for /etc/config/dhcp 
@@ -24,2 +24,3 @@ config dhcp 'lan'
        option ra 'server'
+       option ignore '1'
Proxying multicast packages For proxying multicast packages, we need to install igmpproxy and configure it: Add to: /etc/config/firewall 
# Configuration for igmpproxy
config rule
	option src      lan
	option proto    igmp
	option target   ACCEPT
config rule
	option src      lan
	option proto    udp
	option dest     lan
	option target   ACCEPT
(OpenWRT wiki gives a different 2nd rule now, but this is the one I currently use) Replace /etc/config/igmpproxy with: 
config igmpproxy
	option quickleave 1
config phyint
	option network wwan
	option direction upstream
	list altnet 192.168.178.0/24
config phyint
	option network lan
	option direction downstream
	list altnet 192.168.178.0/24
(Assuming Fritz!Box uses the 192.168.178.0/24 network) Don t forget to enable the igmpproxy script: 
# /etc/init.d/igmpproxy enable
Optional: Repeat the WiFi signal If you want to repeat your WiFi signal, all you need to do is add a second wifi-iface to your /etc/config/wireless. 
config wifi-iface
	option device 'radio0'
	option mode 'ap'
	option network 'lan'
	option encryption 'psk2+tkip+ccmp'
	option key 'PASSWORD'
	option ssid 'YourForwardingSSID'
Known Issues If I was connected via WiFi to the OpenWRT AP and switch to the FRITZ!Box AP, I cannot connect to the OpenWRT router for some time. The igmpproxy tool writes to the log about changing routes. Future Work I ll try to get the FRITZ!Box replaced by something that runs OpenWRT as well, and then use OpenWRT s WDS support for repeating; because the FRITZ!Box 7270v2 is largely crap loading a page in its web frontend takes 5 (idle) 20 seconds (1 download), and it s WiFi speed is limited to about 20 Mbit/s in WiFi-n mode (2.4 GHz (or 5 GHz, does not matter), 40 MHz channel). It seems the 7270 has a really slow CPU.
Filed under: OpenWRT

5 August 2014

Francesca Ciceri: Just Rockin' and Rollin'!

[Warning: quite a bit of pics in this post] [Edit: changed the post title, while I love the music, the actual lyrics of "Shake Rattle and Roll" made me facepalm. Ronnie Dawson's song is better :)] Last weekend I've been in Senigallia for the 15th edition of Summer Jamboree.
It was my first time there, and it was epic. Really.
If you are into roots music and early rock'n'roll and/or into vintage 40s and 50s clothes, go there.
You won't regret it! (You have time until August 10th, hurry up!) If you follow my identi.ca account (whooo! shameless plug!), you may know that I love music in general and Blues, Jazz and Rockabilly in particular.
If you read my blog, you may know that I make clothes - particularly reproductions of 50s and retro clothes.
So, it's not much of a surprise that going to the Summer Jamboree has been a mindblowing experience to me.
What surprised me it's that I've felt the very same wonder of my first Debconf: the amazing feeling that you are not alone, there are other people like you out there, who love the same things you love, who are silly about the same little details (yes, I equally despise historically innacurate pin up shoes and non free software), who dance - metaphorically and not - at your same beat.
Same wonder I felt when I first read some authors - Orwell and David Foster Wallace, just to mention a couple - or when I first delved in anarchist thinkers.
By nature I'm not much of a social person, and I tend to live and love alone. But that sense of being part of something, to find like-minded people always blows me away. I'm not much of a blog writer, so I won't probably be able to give you a good impression of the awesomness of it.
But hey, watch me trying. The Vintage Market I spent most of the morning travelling by train to reach Senigallia (and met the most beautiful French girl ever in the process, who sketched me in her notebook because, hey!, I was already in full Rockabilly gear).
The hotel was pretty close to the station, and to the part of the city where the festival was taking place, so I spent a couple of hours sleeping, then started the adventure.
The festival takes place mostly near the Rocca Roveresca, a beautiful fifteenth century castle, and on its gardens, but the all the other venues are in walking distance.
All around the Rocca there is a market with vintage clothes, records, shoes, retro jewelry. A special mention for two fantastic dressmakers: Laura of Bloody Edith Atelier from Rome and Debora of The Black Pinafore from Sarzana. I bought just a piece from each of them, but I was able to do that only with a huge amount of self restraint. Guitars! Tattoos! Yes, I may have spent a bit drooling on the Gibson Cherry Red, and I tried (without amp, though) that beautiful orange Gretsch Electromatic. guitars! And Greg Gregory of the Travel Ink Tattoo Studio from UK was there, with his shiny Airstream. The airstream of Travel Ink Tattoo I also spent a while among the records in the Bear Family Records booth. They are a Germany based independent record label specialised in reissues of country and 50s rock'n'roll. Couldn't resist, and I bought a beautiful Sun Records' tshirt. Just Rockin' and Rollin'. Aka: dance time After that, it was time to dance. I missed the dance camp of the afternoon, but the DJ sets were fantastic, all 40s and 50s stuff, and I fell in love with Lindy Hop and Boogie Woogie, and well, obviously, Jive. I could have spent hours watching the people dancing, and clumsily trying the most basic moves myself. people dancing more dancers People And the people, did I mention the people?
They were cosplaying the 40s and 50s so wonderfully I couldn't help but take some photos (and find a new fetish of mine: men in 40s clothes. Sexy as hell). For instance, Angelo Di Liberto, artistic director of the festival with the beautiful burlesque artist Grace Hall. Angelo Di Liberto and Grace Hall Or the amazingly dressed German couple I met in via Carducci. A beautifully dressed couple And this couple too, was pretty cool. And another very in-character couple The Prettiest Smile award goes to these lovely ladies! Smiling lovely ladies Cars Who knows me, can tell that I don't love cars.
They stink, they are noisy, they are big.
But these ones where shiny and looked beautiful. Oldtimer cars Also, the black Cadillac had the terrible effect on me of putting "Santa Claus is Back in Town" in my head (or, more precisely, Elvis tomcatting his way through the song, singing "Got no sleigh with reindeer / No sack on my back / You're gonna see me comin' in a big black Cadillac"). the big black cadillac cadillac detail Music! Sadly, I missed Stray Cat's Slim Jim Phantom but I was just in time for Ben E. King.
It was lovely: backed by the house band (The Good Fellas), he sang a lot of old Drifters hits, from On Broadway to Save the Last Dance for Me to - obviously - the great Stand By Me. Then a bit of hillbilly country, with Shorty Tom and the Longshots, a French combo consisting of a double bass, a rhythm guitar and a steel guitar. Shorty Tom and the Longshots And, well, more dancing: the dj sets on the three stages went on until 3 am. Day 2 The next morning I took advantage of the early opening of Rocca Roveresca to visit it. The Rocca itself is beautiful and very well maintained, and hosts various exhibitions.
"Marilyn In White" shows the incredible photos taken by George Barris on the set of "The Seven Year Itch" as well as some taken in 1962. Beautiful, really, especially the series on the beach. photos from the exhibition But the ones moving me were the pics from "Buddy Holly, The Day The Music Dies": a collection of photos taken by Bill Francis during the (sadly brief) career of Buddy Holly from the very beginnings to his death. After that, it was time to come back to year 2014, but really I felt like I've walked for a while in another decade and planet. And the cool thing is that I could enjoy the great 40s and 50s music and dances (and clothes!) without the horrible stereotypes and cultural norms of the time period. A total win. :) So, ehm, that's it. I'm a bit sad to be back, and to cheer myself up I'm already planning to attend Wanda Jackson gig in Aarburg (CH) next month.
And take Lindy Hop and Boogie lessons, obviously.

Next.

Previous.