Packaging LAVA for Debian unstable with notes on other distributions I ve been building packages for LAVA on Debian unstable for several months now and I ve been running LAVA jobs on the laptop and on devices in my home lab and on an ARMv7 arndale too. Current LAVA installations use lava-deployment-tool which has only supported Ubuntu 12.04 LTS Precise Pangolin. There has been a desire in LAVA to move away from a virtual environment, to put configuration files in FHS compliant paths, to use standard distribution packages for dependencies and so to make LAVA available on more platforms than just precise. Packaging opens the door to installing LAVA on Debian, Ubuntu, Fedora and any other recent distribution. Despite LAVA currently being reliant on 12.04 Precise, some of the python dependencies of LAVA have been able to move forward using the virtual environment provided by builtout and pypi. This means that LAVA, as packaged, requires a newer base OS suite than precise for Ubuntu, the minimal base is Saucy Salamander 13.10 and for Debian it would be Jessie (testing) although there is currently a transition ongoing in Debian which means that uwsgi is not in testing and Debian unstable would be needed instead. The work to migrate configuration snippets out of deployment-tool and to ensure that the tarball built using setuptools contains all of the necessary files for the package has already been done. The packaging itself is clean and most of the work is done upstream. There is, as ever, more to do but the packages work smoothly for single install LAVA servers where the dispatcher is on the same machine as the django web frontend. The packages have also migrated to Django1.6, something which is proving difficult with the deployment-tool as it has not kept pace with the changes outside the virtual environment, even if other parts of LAVA have. LAVA will be switching to packages for installation instead of deployment-tool and this will mean changes to how LAVA works outside the Cambridge lab. When the time comes to swich to packaging, the plan is to update deployment-tool so that it no longer updates /srv/lava/ but instead migrates the instance to packages. Main changes

1. Configuration files move into /etc/
   • Device configuration files /etc/lava-dispatcher/devices/
   • Instance configuration files/etc/lava-server/
2. Log files move into /var/log/
   • Adding logrotate support no more multi-Gb log files in /srv/lava/
3. Commitment to keeping the upstream code up to date with dependencies
4. Support for migrating existing instances, using South.
5. Packaging helpers
   • add devices over SSH instead of via a combination of web frontend and SSH.
   • Developer builds with easily identifiable version strings, built as packages direct from your git tree.
6. New frontend
   • Although django1.6 does not change the design of the web frontend at all, LAVA will take the opportunity to apply a bootstrap frontend which has greater support for browsers on a variety of devices, including mobile. This also helps identify a packaged LAVA from a deployment LAVA.
7. Documentation and regular updates

The Plan LAVA has made regular releases based on a monthly cycle and these will be provided as source tarballs at http://www.linaro.org/downloads/ for distributions to download. The
official monthly release and any intervening updates will be made available for distributions to use for their own packaging. Additionally, Debian packages will be regularly built for use within LAVA and these will be available for those who choose to migrate from Ubuntu Precise to Debian Jessie. LAVA will assist maintainers who want to package LAVA for their distributions and we welcome patches from such maintainers. This can include changes to the developer build support script to automate the process of supporting development outside LAVA. Initially, LAVA will migrate to packaging internally, to prove the process and to smooth out the migration. Other LAVA instances are welcome to follow this migration or wait until the problems have been ironed out. The Issues

• Remote workers this is work to be completed during the migration to packaging within LAVA as well as pending work upstream on the internals of the connection between a remote worker and the master scheduler. Expect some churn in this area whilst the code is being finalised. A lava-worker package is being prepared which borrows enough code from lava-server to run the lava-scheduler-daemon on the remote worker, until such time as the remote worker communications are refactored.
• Integration there are plans to integrate some of the modules which LAVA uses which are not commonly packaged: linaro-dashboard-bundle and linaro-django-xmlrpc
• OpenID the certificates which underpin OpenID use with Launchpad have been removed in Debian and LAVA is currently investigating alternatives.
• json-schema-validator currently has a broken test suite, so this will need to be patched by LAVA to allow the package to build. The code may be replaced with a different validator if the issue persists.
• namespace handling the current package install is unnecessarily noisy with complaints about the lava namespace. This will need a fix but does not affect current operation.
• Unit tests LAVA is working hard to add a much larger coverage of internal unit tests. These use a temporary database which is not generally available during the build of a distribution package. LAVA is already running continuous integration tests to ensure that these tests continue to pass and the packages will gain documentation on how to run these tests after installation.

Build Dependencies For Debian unstable, the list of packages which must be installed on your Debian system to be able to build packages from the lava-server and lava-dispatcher source code trees are:

debhelper (>= 8.0.0) python   python-all   python-dev   python-all-dev 
python-sphinx (>= 1.0.7+dfsg)   python3-sphinx python-mocker 
python-setuptools python-versiontools

(python-versiontools may disappear before the packages are finalised) In addition, to be able to install lava-server, these packages need to be built from tarballs released by Linaro (the list may shorten as changes upstream are applied)

linaro-django-pagination, 
python-django-restricted-resource (>= 0.2.7), 
lava-tool (>= 0.2), lava-utils-interface (>= 1.0), 
linaro-django-xmlrpc (>= 0.4),
python-versiontools (>= 1.8),
python-longerusername,
linaro-dashboard-bundle (>= 1.10.2), 
lava-dispatcher (>= 0.33.3)
lava-coordinator, lava-server-doc

The list for lava-dispatcher is currently:

python-json-schema-validator, lava-tool (>= 0.4), 
lava-utils-interface, linaro-dashboard-bundle (>= 1.10.2), 

The packages available from my experimental repository are using a new packaging branch of lava-server and lava-dispatcher where we are also migrating the CSS to Bootstrap CSS. Installing LAVA on Debian unstable
$ sudo apt-get install emdebian-archive-keyring
Add the link to my experimental repository (amd64, i386, armhf & arm64) to your apt sources, e.g. by creating a file /etc/apt/sources.list.d/lava.list containing:

deb http://people.linaro.org/~neil.williams/lava sid main
Update with the new key

$ sudo apt-get update
It is always best to install postgresql first:

$ sudo apt-get install postgresql
There are then three options for the packages to install (Please be careful with remote worker setup, it is not suitable for important installations at this time.):

1. Single instance, server and dispatcher with recommended tools
   apt-get install lava
   Installs linaro-image-tools and guestfs tools.
2. Single instance, server and dispatcher
   apt-get install lava-server
   Installs enough to run LAVA on a single machine, running jobs on boards on the same LAN.
3. Experimental remote worker support
   apt-get install lava-worker
   Needs a normal lava-server installation to act as the master scheduler but is aimed at supporting a dispatcher and boards which are remote from that master.

The packages do not assume that your apache2.4 setup is identical to that used in other LAVA installations, so the LAVA apache config is installed to /etc/apache2/sites-available/ but is not enabled by default. If you choose to use the packaged apache config, you can simply run:

$ sudo a2ensite lava-server
$ sudo apache2ctl restart
(If this is a fresh apache install, use a2dissite to disable to default configuration before restarting.) Information on creating a superuser, adding devices and administering your LAVA install is provided in the README.Debian file in lava-server:

$ zless /usr/share/doc/lava-server/README.Debian.gz
Provisos and limitations Please be aware that packaged LAVA is still a work-in-progress but do let us know if there are problems. Now is the time to iron out install bugs and other issues as well as to prepare LAVA packages for other distributions. It will be a little while before the packages are ready for upload to Debian I ve got to arrange the download location and upload the dependencies first and some of that work will wait until more work has gone in upstream to consolidate some of the current dependencies.

krb5-strength is the password strength checking code that we use at Stanford for our primary Kerberos realm. We've had quite a lot of difficulty deciding exactly what password strength checking we want to do and how we want to handle password history. The good part is that this is to the advantage of everyone else, since now more flexible password strength checking code is available. The major change in this release is the addition of a password history implementation for Heimdal. Implemented as an external password quality check program, it can stack with other password quality check programs, such as the one included in this distribution. Previous passwords are hashed with PBKDF2 with SHA-2. Note that this has somewhat extensive Perl module dependencies, since it was originally written as a separate project. Also in this release is yet another password dictionary type: SQLite. This is probably a bit slower than straight lookups in CDB, and it's definitely less space-efficient since it stores each word both forward and reversed, but it can reject any password that is within edit distance one of a dictionary word. (Edit distance one means that the word can be formed from the password by adding, removing, or changing a single character.) The cdbmake-wordlist utility has been renamed to krb5-strength-wordlist and can now generate the SQLite dictionary as well as the CDB dictionary. Finally, another configuration option has been added: minimum_different. If set, passwords must contain at least this many different characters. This can be used to reject passwords that are long strings of the same character or short repeating patterns, which are otherwise difficult to detect with a straight dictionary-driven approach. You can get the latest release from the krb5-strength distribution page.

For years, I've used a gmail account to subscribe to public mailing lists because it was convenient, it had excellent spam filtering and since the lists were already available on google-indexed public archive I felt that there was no real privacy issue.
I already have a local copy of every message in the account, in case I lose access to it for some random reason, and my real contacts already know and use my other email address(es), so I'm not really worried about continuing to use gmail for this kind of traffic.

When I started contributing to Debian I also used that email address, because I was already subscribed to a few mailing lists under it, and it was going to end up being used on the BTS and other spam-attracting public places like that, but I was less happy with the choice.

Lately, however, gmail's spam filters have started to behave oddly, with lots of false positives including a number of messages from @debian.org addresses, and this basically removes the main reason why I was using gmail in the first place, so I'm seriously considering dropping it also for publicly indicized communications.

My first step, right before leaving for Barcelona was to create a couple other email addresses, one for mailing lists and the other for debian-related work, and add them to my gpg/OpenPGP key, so that any new signature on them from the conf would also include the new UIDs.

If you've received a badly signed message (thanks heirloom mailix for mangling stuff :) ) followed by an hopefully correctly signed one where I was asking you to sign new UIDs, and you were wondering, this is the reason.

Now that the new UIDs have been signed by a few DDs I'm going to start using the new address in Debian: I believe I have changed it everywhere I needed to, except for the maintainer/uploader field for my packages, where it will change on their next uploads.

As for mailing lists, I still don't know how to proceed: I could proceed in a systematic way, but I'm lazy and I will probably just change a few main subscriptions, use the new address for new subscriptions and then migrate the rest a few at a time, unless some other event happens and I'll have to hurriedly migrate everything or risk losing important messages. Even in the latter case, I don't need any information that is only stored on google's servers, so I won't will panic just a bit :)

Now, where did I put that documentation about xmpp servers?

As usual, I look at the application stats for phpMyAdmin just after student application period of Google Summer of Code is over. First of all we got more proposals than in last years, this time there is way more students from India and discussions on mentors lists shows this is quite similar for other projects. Maybe it's just different timing which works better for students there, but there might be different reasons as well. There is also quite low number of spam or bogus proposals. Same as in past years, people leave the submission to the last moment, even though we encourage them to submit early so that they can adjust the application based on our feedback. Anyway we're just working on evaluation and will finalize it in upcoming days. Of course you will know the results from Google on April 21st.

Filed under: English phpMyAdmin 0 comments Flattr this!

I will try to report my Debian activities here. I am trying to retake them and as I'm usually busy at work, I have limited amount of time to do so, they stand small for now. A new mailing list, debian-astro, has been created. This was requested to create a space to allow a discussion space around astronomy for Debian, both for professionals and amateurs, to potentially look into a blend, and in general to talk about packaging and development of astronomy-related tools into Debian. Thanks to those interested on making this happen. Go subscribe today!

I have just released version 1.7.2 of Obnam, my backup program. While I was releasing 1.7.1, I found a new problem, so I fixed that. Due to sillinesses in my CI/release system, I had to bump the version number to 1.7.2. Version 1.7.2, released 2014-03-22 Bug fixes:

• Fix another bug in the FUSE plugin's file reading code, found during the release process of 1.7.2.

Version 1.7.1, released 2014-03-22

• The dump-repo command now outputs JSON instead of YAML. The dependency on PyYAML is no longer.

Bug fixes:

• Nemo Inis found a bug in the FUSE plugin (obnam mount), where Obnam would return the wrong data when the program reading the file didn't read the whole file from the beginning in one read(2) system call.
• The test suite now skips tests that require use of extended attributes in the user namespace. This should allow the test suite to be run on more build servers run by various distributions.

Core Rendering with Glamor I ve hacked up the intel driver to bypass all of the UXA paths when Glamor is enabled so I m currently running an X server that uses only Glamor for all rendering. There are still too many fall backs, and performance for some operations is not what I d like, but it s entirely usable. It supports DRI3, so I even have GL applications running. Core Rendering Status I ve continued to focus on getting the core X protocol rendering operations complete and correct; those remain a critical part of many X applications and are a poor match for GL. At this point, I ve got accelerated versions of the basic spans functions, filled rectangles, text and copies. GL and Scrolling OpenGL has been on a many-year vendetta against one of the most common 2D accelerated operations copying data within the same object, even when that operation overlaps itself. This used to be the most performance-critical operation in X; it was used for scrolling your terminal windows and when moving windows around on the screen. Reviewing the OpenGL 3.x spec, Eric and I both read the glCopyPixels specification as clearly requiring correct semantics for overlapping copy operations it says that the operation must be equivalent to reading the pixels and then writing the pixels. My CopyArea acceleration thus uses this path for the self-copy case. However, the ARB decided that having a well defined blt operation was too nice to the users, so the current 4.4 specification adds explicit language to assert that this is not well defined anymore (even in the face of the existing language which is pretty darn unambiguous). I suspect we ll end up creating an extension that offers what we need here; applications are unlikely to stop scrolling stuff around, and GPUs (at least Intel) will continue to do what we want. This is the kind of thing that makes GL maddening for 2D graphics the GPU does what we want, and the GL doesn t let us get at it. For implementations not capable of supporting the required semantic, someone will presumably need to write code that creates a temporary copy of the data. PBOs for fall backs For operations which Glamor can t manage, we need to fall back to using a software solution. Direct-to-hardware acceleration architectures do this by simply mapping the underlying GPU object to the CPU. GL doesn t provide this access, and it s probably a good thing as such access needs to be carefully synchronized with GPU access, and attempting to access tiled GPU objects with the CPU require either piles of CPU code to de-tile accesses (ala wfb), or special hardware detilers (like the Intel GTT). However, GL does provide a fairly nice abstraction called pixel buffer objects (PBOs) which work to speed access to GPU data from the CPU. The fallback code allocates a PBO for each relevant X drawable, asks GL to copy pixels in, and then calls fb, with the drawable now referencing the temporary buffer. On the way back out, any potentially modified pixels are copied back through GL and the PBOs are freed. This turns out to be dramatically faster than malloc ing temporary buffers as it allows the GL to allocate memory that it likes, and for it to manage the data upload and buffer destruction asynchronously. Because X pixmaps can contain many X windows (the root pixmap being the most obvious example), they are often significantly larger than the actual rendering target area. As an optimization, the code only copies data from the relevant area of the pixmap, saving considerable time as a result. There s even an interface which further restricts that to a subset of the target drawable which the Composite function uses. Using Scissoring for Clipping The GL scissor operation provides a single clipping rectangle. X provides a list of rectangles to clip to. There are two obvious ways to perform clipping here either perform all clipping in software, or hand each X clipping rectangle in turn to GL and re-execute the entire rendering operation for each rectangle. You d think that the former plan would be the obvious choice; clearly re-executing the entire rendering operation potentially many times is going to take a lot of time in the GPU. However, the reality is that most X drawing occurs under a single clipping rectangle. Accelerating this common case by using the hardware clipper provides enough benefit that we definitely want to use it when it works. We could duplicate all of the rendering paths and perform CPU-based clipping when the number of rectangles was above some threshold, but the additional code complexity isn t obviously worth the effort, given how infrequently it will be used. So I haven t bothered. Most operations look like this:

Allocate VBO space for data
Fill VBO with X primitives
loop over clip rects  
    glScissor()
    glDrawArrays()
 

This obviously out-sources as much of the problem as possible to the GL library, reducing the CPU time spent in glamor to a minimum. A Peek at Some Code With all of these changes in place, drawing something like a list of rectangles becomes a fairly simple piece of code: First, make sure the program we want to use is available and can be used with our GC configuration:

prog = glamor_use_program_fill(pixmap, gc,
                               &glamor_priv->poly_fill_rect_program,
                               &glamor_facet_polyfillrect);
if (!prog)
    goto bail_ctx;

Next, allocate the VBO space and copy all of the X data into it. Note that the data transfer is simply memcpy here that s because we break the X objects apart in the vertex shader using instancing, avoiding the CPU overhead of computing four corner coordinates.

/* Set up the vertex buffers for the points */
v = glamor_get_vbo_space(drawable->pScreen, nrect * (4 * sizeof (GLshort)), &vbo_offset);
glEnableVertexAttribArray(GLAMOR_VERTEX_POS);
glVertexAttribDivisor(GLAMOR_VERTEX_POS, 1);
glVertexAttribPointer(GLAMOR_VERTEX_POS, 4, GL_SHORT, GL_FALSE,
                      4 * sizeof (GLshort), vbo_offset);
memcpy(v, prect, nrect * sizeof (xRectangle));
glamor_put_vbo_space(screen);

Finally, loop over the pixmap tile fragments, and then over the clip list, selecting the drawing target and painting the rectangles:

glEnable(GL_SCISSOR_TEST);
glamor_pixmap_loop(pixmap_priv, box_x, box_y)  
    int nbox = RegionNumRects(gc->pCompositeClip);
    BoxPtr box = RegionRects(gc->pCompositeClip);
    glamor_set_destination_drawable(drawable, box_x, box_y, TRUE, FALSE, prog->matrix_uniform, &off_x, &off_y);
    while (nbox--)  
        glScissor(box->x1 + off_x,
                  box->y1 + off_y,
                  box->x2 - box->x1,
                  box->y2 - box->y1);
        box++;
        glDrawArraysInstanced(GL_TRIANGLE_STRIP, 0, 4, nrect);
     
 

GL texture size limits X pixmaps use 16 bit dimensions for width and height, allowing them to be up to 65536 x 65536 pixels. Because the X coordinate space is signed, only a quarter of this space is actually useful, which makes the useful size of X pixmaps only 32767 x 32767. This is still larger than most GL implementations offer as a maximum texture size though, and while it would be nice to just say we don t allow pixmaps larger than GL textures , the reality is that many applications expect to be able to allocate such pixmaps today, largely to hold the ever increasing size of digital photographs. Glamor has always supported large X pixmaps; it does this by splitting them up into tiles, each of which is no larger than the largest texture supported by the driver. What I ve added to Glamor is some simple macros that walk over the array of tiles, making it easy for the rendering code to support large pixmaps without needing any special case code. Glamor also had some simple testing support you can compile the code to ignore the system-provided maximum texture size and supply your own value. This code had gone stale, and couldn t work as there were parts of the code for which tiling support just doesn t make sense, like the glyph cache, or the X scanout buffer. I fixed things so that you could leave those special cases as solitary large tiles while breaking up all other pixmaps into tiles no larger than 32 pixels square. I hope to remove the single-tile case and leave the code supporting only the multiple-tile case; we have to have the latter code, and so having the single-tile code around simply increases our code size for not obvious benefit. Getting accelerated copies between tiled pixmaps added a new coordinate system to the mix and took a few hours of fussing until it was working. Rebasing Many (many) Times I m sure most of us remember the days before git; changes were often monolithic, and the notion of changing how the changes were made for the sake of clarity never occurred to anyone. It used to be that the final code was the only interesting artifact; how you got there didn t matter to anyone. Things are different today; I probably spend a third of my development time communicating how the code should change with other developers by changing the sequence of patches that are to be applied. In the case of Glamor, I ve now got a set of 28 patches. The first few are fixes outside of the glamor tree that make the rest of the server work better. Then there are a few general glamor infrastructure additions. After that, each core operation is replaced, one a at a time. Finally, a bit of stale code is removed. By sequencing things in a logical fashion, I hope to make review of the code easier, which should mean that people will spend less time trying to figure out what I did and be able to spend more time figuring out if what I did is reasonable and correct. Supporting Older Versions of GL All of the new code uses vertex instancing to move coordinate computation from the CPU to the GPU. I m also pulling textures apart using integer operations. Right now, we should correctly fall back to software for older hardware, but it would probably be nicer to just fall back to simpler GL instead. Unless everyone decides to go buy hardware with new enough GL driver support, someone is going to need to write simplified code paths for glamor. If you ve got such hardware, and are interested in making it work well, please take this as an opportunity to help yourself and others. Near-term Glamor Goals I m pretty sure we ll have the code in good enough shape to merge before the window closes for X server 1.16. Eric is in charge of the glamor tree, so it s up to him when stuff is pulled in. He and Markus Wick have also been generating code and reviewing stuff, but we could always use additional testing and review to make the code as good as possible before the merge window closes. Markus has expressed an interest in working on Glamor as a part of the X.org summer of code this year; there s clearly plenty of work to do here, Eric and I haven t touched the render acceleration stuff at all, and that code could definitely use some updating to use more modern GL features. If that works as well as the core rendering code changes, then we can look forward to a Glamor which offers GPU-limited performance for classic X applications, without requiring GPU-specific drivers for every generation of every chip.

Christine's accidential blog spam on planet.d.o just now gave me the chance to re-read the comments in her post. The state from back then is still the current state on up-to-date Debian unstable:

• Microphone button does nothing.
• USB 3.0 docking station's display does not work thanks to the content mafiaa with an admittedly small petition trying to fix this. Obviously it is going to be ignored.
• No one using Debian seems to care about the fingerprint reader.
• UMTS/WWAN modem works fine on Windows, but Linux loses connection to the USB device all the time. As a result, UMTS does not work

The last item has the most impact on me. The need to tether when you have a dedicated SIM card, built-in modem, and good antennas in your laptop is... infuriating. Especially as it's working as intended on Windows. As an added benefit, even though I saved the PIN in network manager, it asks me for the PIN every time I log in and every time after hibernating. For a device which I can't use in the first place. Argh!

Installing fresh hot Debian 7.0 on a shiny new ThinkPad X1 Carbon laptop turns out to be easy as cake. You just need to make sure to grab the wifi firmware from unstable instead of the all-in-one firmware tarballs, which contain a version that is missing a couple required files.

wget http://cdimage.debian.org/debian-cd/7.0.0/multi-arch/iso-cd/debian-7.0.0-amd64-i386-netinst.iso
dd if=debian-7.0.0-amd64-i386-netinst.iso of=/dev/sdb

(Make sure /dev/sdb is really the usb stick you want to overwrite with the installer!)

wget http://ftp.us.debian.org/debian/pool/non-free/f/firmware-nonfree/firmware-iwlwifi_0.38_all.deb

And put that on a second usb stick for the installer to load the firmware off of. As far as I can tell, everything works. (Did not mess around with the fingerprint reader, don't care.)

I just donated $500 to OpenHatch. Here's why you should donate too:

1. Diversity in open source matters. We can't keep making the software the world runs on without involving people of all sorts, from all backgrounds.
2. OpenHatch is run by community members who I've known for years and trust. They care about data-driven effectiveness and are always getting better at what they do.
3. A rising tide floats all boats. More contributors == more awesome.
4. If you donate before December 24th, your donation makes twice the difference.

Diversity and education initiatives are the reason I'm a part of the free and open source software community today. (Thanks, Debian Women.) You don't have to donate $500 to make a difference. $5, $10, $25 from a hundred people all adds up. Please join me in supporting OpenHatch today.

My blog is included now in Planet Debian, so this post is to say hello :) And what can be better to explain who I am, that saying that I am a Debian Contributor? What does Debian Contributor mean? There are lots of tasks that people accomplish to make Debian better, and Debian Contributor is a new hat to credit any participation in the Debian Project, of any kind (even if the contributor is not a full-member of Debian). The idea was started and promoted by Enrico Zini. In less than one year, a website with a list of Debian Contributors, data source schemes and much more pieces have been designed. The site is working, and evolving, according to a roadmap. There are some working data sources already, some other proof of concept , pending to adopt or refine. There are much more kind of contributions that still have no data source created. Everybody can help on that! An example: my contributions I m listed now as wiki editor (at least, I edit once a month when doing my spam fighting tasks), website committer (I upload translations to the Spanish part of the website), and publicity (I ve sent some news to Debian Project News from time to time). Having your name in a list may seem a small thing, but happiness is made of small things!! The website committers data source is one of the proof of concepts and it s not being updated recently. I hope that somebody in the www team takes care of that oh wait, I m part of the www team I m part of the Spam Fighting Force too, but the data source tracks the spam reports reviewers, and I don t do that, I only report spam. That s why I m not there (yet). I m also a translator, member of the Spanish team. Translators are already credited in the translation coordination pages (for example this one), but we want to include them in the main list of Debian Contributors too. That means modifying the bot that tracks the mailing lists. I have joined the i18n team and I hope I can understand how it works and try to propose the required changes. If in the meanwhile, other person comes with the solution, the better! I have given a talk (and two lightning talks) in the last MiniDebConf Barcelona 2014. Most of the speakers in Debian events are available via the Debconf website, for example: http://penta.debconf.org/dc13_schedule/speakers.en.html I m sure a data source can be created somehow! More examples of data sources As I said, these are only examples (it s not about crediting my contributions, but everybody s). So if you explore the already managed data sources, you can find the Debian Sysadmin and Security teams, package maintainers, package uploaders to FTP sites, committers to several source code repositories of Debian-related software, people helping mentoring other people in Debian, people helping in the New Member process, people subtitling talks But we need more!! Please have a look at DcSiteDevel where you can learn how to create a data source and the tools for extracting information in an easy way. If each Debian Team cares of their own data source (which allows to credit their members) it is a sustainable way and also improves those Team s visibility (and new contributions, probably!). Identity management The last updates to Debian Contributors project are about identity management. Right now the site only shows data of contributors having a @debian.org account or an account in Alioth (the Debian software forge, used for many subprojects). It also uses the email field to merge contributions made in areas where you login with other kind of user (e.g. the wiki). This is a great step towards crediting people that are not full Debian members (yet), if they want their names to appear in the Debian Contributors list (they can opt-out too, or show/hide each contribution separately). So, what next? There is a Roadmap and a TODO list. But nobody as individual can know all the roles and actual work in the Debian project. It s better that each team cares about how to send their contribution info to the site. There are some mining tools available to extract for example names of committers in source code repositories, or senders in mailing lists. Not much code is required. Please think about it and join the wave of making Debian diversity skills more visible!
Filed under: Interesting sites, My experiences and opinion, Uncategorized Tagged: Communities, Contributing to libre software, data mining, Debian, Developer motivations, English, libre software, Project Management, statistics, translations

I know, I know, I should resist saying this. But every time I see it, I wonder why it happens, and I should just get this off my chest. The difference between "its" and "it's" is something that many people, even native english speakers, seem to miss. Yet, it's so extremely simple that I, a non-native english speaker, have been baffled about that common mistake for as long as I can remember. The apostrophe (') in any sentence usually means that something at the location of that apostrophe is gone out to lunch. In this particular case, it means that the " " and the "i" in the phrase "it is" were hungry. So rather than "it is", we contract that to "it's", and allow the space and the i to enjoy their meal while the apostrophe keeps their seats warm. Practically, what that means is that every time you want to write "it's", you should consider whether you can replace it with "it is" without making the sentence sound like junk. If you can't, you probably meant to write "its" rather than "it's". For instance, consider the following sentence: "It's not possible to repair this car within the budget that its owner wants to pay" It's perfectly possible to say "it is not possible" here, so we need to have the apostrophe keep a seat warm for the space and the i. It makes no sense to say "it is owner", unless you're trying to speak a much deformed form of english, so that makes it a possessive pronoun (similar to "hers", "his", "theirs", etc) and you shouldn't use an apostrophe. Speaking of food, it's time for lunch now.

To provide geolocation-aware answers with BIND, a common solution is to use a patch adding GeoIP support. A client can be directed to the closest (and hopefully fastest) web server: However, an end user does not usually talk directly to authoritative servers. They proxy the query to a third-party recursor server which will query the authoritative server on their behalf. The recursor also caches the answer to be able to serve it directly to other clients. On most cases, we can still rely on the recursor GeoIP location to forward the client to the closest web server because it is located in the client s ISP network, as shown on the following schema:

1. Juan is located in China and wants to know the IP address of www.example.com. She queries her ISP resolver.
2. The resolver asks the authoritative server for the answer.
3. Because the IP address of the resolver is located in China, the authoritative server decides to answer with the IP address of the web server located in Japan which is the closest one.
4. Juan can now enjoy short round-trips with the web server.

However, this is not the case when using a public recursor as provided by Google or OpenDNS. In this case, the IP address of the end client and the source IP address of the recursor may not share the same locality. For example, in the following schema, the authoritative server now thinks it is in relation with an European customer and answers with the IP address of the web server located in Europe: Moreover, caching makes the problem worse. To solve this problem, a new EDNS extension to expose the client subnet has been proposed. When using this extension, the recursor will provide the client subnet to the authoritative server for it to build an optimized reply. The subnet is vague enough to respect client s privacy but precise enough to be able to locate it. A patched version of dig allows one to make queries with this new extension: In the above example, a client located in France gets a reply with two IP addresses located in France. If we now are an US client, we will get IP addresses located in the US: The recursor is expected to cache the two different answers and only serve them if the client matches the appropriate subnet (the one confirmed in the answer from the authoritative server). With this new extension, the authoritative server knows that Juan is located in China and answers with the appropriate IP address: Not many authoritative servers support this extension (PowerDNS and gdnsd, as far as I know). At Dailymotion, we have built a patch for BIND. It only works when BIND is configured as an authoritative server and it doesn t expose no configuration knobs. Feel free to use it (at your own risk). Once installed, you need to register yourself to OpenDNS and to Google to receive queries with the extension enabled.

I was pretty exhausted after yesterday's sustained activities. By the time I'd blogged, done some photo management, gone to yoga class, made dinner and called my girlfriend, it was late, and I'd gotten to that tired point where I couldn't be bothered getting off the couch, so I ended up having a self-inflicted late night. This morning I tried to go for a run, but I died at the 2km mark, half way up Hawthorne Road, so I walked for the remainder of my 5km circuit. I'm going to say it was a combination of tiredness, insufficient hydration, the heat, and my head space. I got home from my "run", and had a shower and cleaned up the kitchen a bit and decided I'd rather get out of the house than work on my taxes, so I headed over to Moorooka to talk to the folks at Northfield about a new car radio. I've been wanting to get a better radio for the car for a while. The factory one has no line in, so I can't play anything off my phone through the car speakers. I've been wanting one that can do Bluetooth and play USB media. That way I can stick all of Zoe's CDs onto a USB stick and give her (and me) more variety than the current 3 track CD from dentist that she's favouring, and with Bluetooth I can listen to Pandora without needing a cable. I got to Northfield, and was pleased that they could do something within the budget I was looking at, and bought the DEH-X6650BT, which they were able to install for me on the spot. They said it'd take an hour and half, so I went for a walk down the road to Moorooka's main street to kill some time. I ended up having lunch at a little Ethiopian cafe where I swear the lady didn't understand English. Moorooka's quite nice and ethnic. After lunch I walked back, with the intention of reading the manual while I waited, and was pleasantly surprised to find they'd already finished the installation. When the guy was walking me through how to use it, I was really pleasantly surprised to discover that it also included Bluetooth hands free for my phone, with a little microphone glued to the top of my windscreen. That really made my day. I don't have to faff around with my little Bluetooth earpiece in the car any more. Then I discovered it had built-in Pandora support. So I'm really happy with this purchase, and really glad I didn't have to make a second trip to get it installed. I scouted around for a small USB thumb drive on the way home, and copied all Zoe's music off my laptop when I got home. By this stage, it was too late to bike to Kindergarten, and I wasn't really feeling like it anyway, because it was too hot, and I wanted to show Zoe the new radio anyway. It was a good call, because Zoe was fast asleep, and didn't take very kindly to being woken up. It took about 20 minutes to get her calm enough to get her back to the car. We got home, and we chilled out on the couch and watched a bunch of Youtube videos on the TV from when she was in the US, and then I showed her some Winter Olympics figure skating and speed skating, and then she watched a bit of DVD while I cleaned up the living room. Sarah picked up Zoe, and then I hosted our Body Corporate's annual general meeting.

When I installed Debian on my MacBookAir4,1 in 2012, there was no way to do it without manual intervention yet. I followed a tutorial on how to boot GRUB from the EFI Boot option in the MacBook EFI Menu. I did not want to fiddle with rEFIt, and I didn't want to boot GRUB by default. When I want to boot Linux, I press the Option key during power up and select EFI Boot from the Apple boot menu. Unfortunately, I neglected to collect notes about how I did it manually. However, Linux 3.2 is getting a bit old, so I finally wanted to replace my manual boot configuration with something handled by the package system. In case you don't have /boot/efi in /etc/fstab yet, you need to mount /dev/sda1 on /boot/efi for the following to work. The documentation I found on the net suggested to just reinstall grub-efi-amd64 and everything should work. That is not quite true. When I do Nothing changes in /boot/efi. I sort of expected that /boot/efi/EFI/debian would be created, and the EFI image should be placed in there. However, that did not happen. Why is that? It turns out that when I installed grub-efi-amd64 manually in 2012, I created /boot/efi/EFI/boot/bootx64.efi which is the EFI fallback location, and apparently exactly what I want on this MacBook which does not support multiple boot options. Matthew Garrett posted an interesting article called Booting with EFI which sheds light on this issue, go and read it. Looking at /var/lib/dpkg/info/grub-efi-amd64.postinst revealed that /boot/efi/EFI/debian needs to be created manually first. If this directory does not exist, grub-efi-amd64 basically does nothing on reinstall. Running grub-install will actually create a new EFI image. However, it is being created in the wrong place for this machine. does the trick. Now /boot/efi/EFI/debian/grubx64.efi gets created. However, since I don't want to make GRUB the default, there is yet another manual step to do: Now I can select EFI Boot after pressing the Option key during startup. GRUB is loaded and Linux 3.13 gets booted. Strike! Looking more closely reveals that there is actually a way to tell grub-install that it should install to the fallback location directly. The --removable option does that. For the faint of heart, what does grub-install actually do on an EFI system? It does not directly write to the disk, therefore it does not need a device specified. It looks for files in /boot/efi and assumes the EFI partition is mounted there. So for my use case, the correct way to upgrade to a current GRUB EFI image should have been: Meanwhile I've been made aware of Bug#708430. I guess it would be nice to have an option in /etc/default/grub which would indicate that installation to the fallback location is desired. While this is a rather ugly hack to work around a stupid limitation, it is still what I'd like on this MacBook. At least since I don't have a triplle boot situation. Fallback location works fine with just two OSes coexisting.

On three Saturdays in April and May, I will be helping run three day-long project-based workshops at the University of Washington in Seattle. The workshops are for anyone interested in learning how to use programming and data science tools to ask and answer questions about online communities like Wikipedia, Twitter, free and open source software, and civic media. The workshops are for people with no previous programming experience and the goal is to bring together researchers as well as participants and leaders in online communities. The workshops will all be free of charge and open to the public given availability of space. Our goal is that, after the three workshops, participants will be able to use data to produce numbers, hypothesis tests, tables, and graphical visualizations to answer questions like:

• Are new contributors to an article in Wikipedia sticking around longer or contributing more than people who joined last year?
• Who are the most active or influential users of a particular Twitter hashtag?
• Are people who participated in a Wikipedia outreach event staying involved? How do they compare to people that joined the project outside of the event?

If you are interested in participating, fill out our registration form here. The deadline to register is Wednesday March 26th. We will let participants know if we have room for them by Saturday March 29th. Space is limited and will depend on how many mentors we can recruit for the sessions. If you already have experience with Python, please consider helping out at the sessions as a mentor. Being a mentor will involve working with participants and talking them through the challenges they encounter in programming. No special preparation is required. If you re interested, send me an email.

I m currently using BTRFS on most systems that I can access easily. It s not nearly reliable enough that I want to install it on a server in another country or an embedded device that s only accessible via 3G, but for systems where I can access the console it s not doing too badly. Balancing and Space Allocation # btrfs filesystem df /
Data, single: total=103.97GiB, used=85.91GiB
System, DUP: total=32.00MiB, used=20.00KiB
Metadata, DUP: total=1.78GiB, used=1.31GiB
# df -h /

Filesystem	Size	Used	Avail	Use%	Mounted on
/dev/disk/by-uuid/ac696117-473c-4945-a71e-917e09c6503c	108G	89G	19G	84%	/

Currently there are still situations where it can run out of space and deadlock on freeing space. The above shows the output of the btrfs df command and the regular df command, I have about 106G of disk space used by data and metadata in BTRFS while df shows that the entire filesystem (IE the block device) is 108G. So if I use another 2G of data or metadata then the system is at risk of deadlocking. To avoid that happening I have to run btrfs balance start / to start a balance which defragments the space use and frees some blocks. Currently there is a bug in BTRFS (present in all Debian/Unstable kernels) which prevents a balance operation from completing when systemd is used in a default configuration (there s something about the way systemd accesses it s journal files that triggers a BTRFS bug). This is really inconvenient, particularly given that there s probably a strong correlation between people who use experimental filesystems and people who use experimental init programs. When you get to the stage of the filesystem being deadlocked you can sometimes recover by removing snapshots and sometimes by adding a new device to the filesystem (even a USB flash drive will do). But I once had a filesystem get into a state where there wasn t enough space to balance, add a device, or remove a snapshot so I had to do a backup/format/restore. Quota Groups Last time I asked the developers (a few weeks ago) they told me that quota groups aren t ready to use. They also said that they know about enough bugs that there s no benefit in testing that feature. Even people who want to report bugs in BTRFS shouldn t use quotas. Kernel Panics with Kmail I ve had three systems develop filesystem corruption on files related to Kmail (the email program from KDE). I suspect that Kmail is triggering a bug in BTRFS. On all three systems the filesystem developed corruption that persisted across a reboot. One of the three systems was fixed by deleting the file for the Outbox, the others are waiting for kernel 3.14 which is supposed to fix the bug that causes kernel panics when accessing the corrupted files in question. I don t know whether kernel 3.14 will fix the bug that caused the corruption in the first place. Conclusion As I don t use quotas BTRFS is working well for me on systems that have plenty of storage space and don t run Kmail. There are some systems running systemd where I plan to upgrade the kernel before all the filesystem space is allocated. One of my systems is currently running SysVinit so I can balance the filesystem. Apart from these issues BTRFS is working reasonably well for me. I haven t yet had it s filesystem checksums correct corrupted data from disk in any situation other than tests (I have had ZFS correct such an error, so hardware I use does benefit from this). I have restored data from BTRFS snapshots on many occasions, so that feature has been a major benefit for me. When I had a system with faulty RAM the internal checks in BTRFS alerted me to the problem and I didn t lose any data, the filesystem became read-only and I was able to copy everything off even though it was too corrupted for writes.

As I posted some weeks ago, I have been playing with my CuBox-i4Pro, a gorgeous little ARM machine by SolidRun, built around an iMX6 system-on-a-chip. My first stabs at using it resulted in my previous post on how to get a base, almost-clean Debian distribution to run (Almost? Yes, the kernel requires some patches not yet accepted upstream, so I'm still running with a patched 3.0.35-8 kernel). After writing this step by step instructions, I followed them and built images ready to dd to a SD card and start running (available at my people.debian.org space. Now, what to do with this little machine? My version is by no means a limited box: 4 ARM cores, 2GB RAM make a quite decent box. In my case, this little machine will most likely be a home storage server with little innovation. However, the little guy is a power server, at only 3W consumption. I wanted to test its capabilities to do some number crunching and aid some of my friends The obvious candidate is building a Blender render farm. Right, the machines might be quite underpowered, but they are cheap (and look gorgeous!), so at least it's worth playing a bit! Just as a data point, running on an old hard disk (and not on my very slow SD card), the little machine was able to compile the Blender sources into a Debian package in 89m13.537s, that is, 5353 seconds. According to the Debian build logs (yes, for a different version, I tried with the version in Wheezy and in a clean Wheezy system), the time it took to build on some other architectures' build daemons was 1886s on i386, 1098s on PowerPC, 2003s on AMD64, 11513s on MIPS and 27721 on ARMHF. That means, my little machine is quite slower than desktop systems, but not unbearably so. But sadly, I have hit a wall, and have been unable to do any further progress. Blender segfaults at startup under the Debian armhf architecture. I have submitted bug report #739194 about this, but have got no replies to it yet. I did get the great help from my friends in the OFTC #debian-arm channel, but they could only help up to a given point. It seems the problem lies in the Python interpreter in armhf, not in Blender itself... But I cannot get much further either. I'm sending this as a blog post to try to get more eyeballs on my problem How selfish, right? :-) So, slightly going over the bug report, blender just dies at startup: After being told that strace is of little help when debugging this kind of issues, I went via gdb. A full backtrace pointed to what feels like the right error point: I'm not pasting here the full bug history (go to the bug report for the full information!), but it does point me to this being a problem in Python-land: It points to something not found at line 59 of Python/errors.c. And what I understand from that line is that some kind of unknown exception is thrown, and the Python interpreter does not now what to do with it. The check done at line 59 is the if (exception != NULL ** ....: So... Dear lazyweb: Any pointers on where to go on from here? kthxbye.

We had a pretty busy day today. Sarah dropped Zoe off at around 8:30am. I reminded Zoe that she'd forgotten to give Sarah her crown that she'd made at Playgroup the previous day, so she gave that to her. After Sarah left, I let Zoe watch a little bit of TV while I finished getting myself organised. We drove to Brazilian Jiu Jitsu class today, because I wanted to go directly to the Planetarium afterwards. Zoe's BFF Megan came to BJJ class today for a trial class. Zoe was very excited. Megan picked everything up really quickly, and the girls had a good time. She didn't want to wear her uniform today though. I managed to get out of her later that she felt it made her sweaty. I told her it was probably all the running around that made her sweaty, not the uniform. We went out for a coffee with Megan, her Dad and her little sister afterwards. We found a little take away shop around the corner. Not quite the place that did babyccinos unfortunately, so Megan's Dad very cleverly got a chocolate milk and split it three ways with the girls and called it a "milkshake". I've realised that for Science Friday, I've pretty much been doing solely chemistry so far, so I wanted to throw in a bit of astronomy this week. The Sir Thomas Brisbane Planetarium had just recently reopened after getting a spanky new projector because 1978 called and wanted theirs back. Unfortunately, they weren't showing Perfect Little Planet today, but the slightly more specific Black Holes: Journey into the Unknown. I wanted Zoe to see something in the auditorium, so I thought we'd give it a shot anyway. I think we got to the Planetarium at around 11am, and the next show was at 1:30pm, so we had a bit of a look around the foyer for a bit, which Zoe quite enjoyed. They had a life-size model of Neil Armstrong's spacesuit, and a scale model of a Saturn V rocket. Zoe quite enjoyed both of them. I'll have to dig up some Youtube footage of a Saturn V launch. We then headed over to the cafe in the adjacent Botantic Gardens, and had a reasonable lunch. After lunch we wandered around the Botanic Gardens to kill some time, so you could say that today's Science Friday also included a goodly amount of botany as well. We went into the Tropical Dome and looked at the fish in the lily pond, and toured the fern garden, before heading back. We timed it pretty well, and got back to the Planetarium with about 10 minutes to kill. I'd prepped Zoe for the presentation, and told her that she needed to keep quiet, and if we had to leave we wouldn't be able to go back in again, and she could whisper to me if she needed to. I have to say, she was so well behaved. She sat through the whole 45 minute presentation and the projected tour of the night sky. I guess the presentation was sufficiently visually interesting to hold her attention for the duration. There was certainly a lot of "wow" out of her. I was so proud of how she went that I had to go and buy her an ice cream as a reward. After that, we went for a drive around Mount Coot-tha. I pointed out the TV towers that we can see from our balcony, and then she fell asleep on the way home. She didn't wake up particularly happy from her late, short nap, when we got home, but once she got herself sorted out, we baked the cookies that we'd made yesterday. They came out very nicely indeed. Once they were all baked, we dropped off a dozen to Megan's place, and then came home to have dinner. Bedtime went pretty well. I hope she has a good night's sleep, as we have a big day tomorrow.

As Jan has previously announced, the Debian project is maintaining a booth at Chemnitzer Linux-Tage 2014, which is also organized by him. This year we will have merchandising at the booth, which is provided by Alexander Wirt and of course a demo system with Debian wheezy BabelBox as the past years. I'll drop it tomorrow, as I have a conflicting appointment on Saterday, maybe I can attend later on Sunday. In case you have spare time at the weekend ahead, it may be worth to spend it with great lectures and meet nice people over there.