Documentation! That is what the next version of Lintian will have. In 2.5.1, Lintian
had support for adding options in the lintianrc. Unfortunately, the
user had to guess the syntax, which makes the feature less useful.
With the next release, the manpage and the example config file[1]
will contain documentation and examples. The User Manual[2] also saw some improvements. Particularly, it will
mention that lintian can process a .changes file and that it does
cross-package checks. Of course, it also documents the format of the
new Vendor Profiles. Though even with these improvements, the current
documentation still leaves much to be wanted. The last unpack script (unpack-srcpkg-l1) was finally removed and
was replaced by a smarter version of index collection. This means
that Lintian can now schedule index in parallel with a few other
collections (namely unpacked, which is the current bottleneck). The speed improvement is probably not noticeable, but it was also a
good time to upgrade our index collection to properly handle
multi-tarball source packages. Previously Lintian only extracted the
index from the base tarball. The fields collection also saw some attention. Previously it
extracted fields from the control file (in the .deb, etc.) into a
directory. Each field became a file and the value of the field would
be written in the field. Today, the collection basically looks like
this:

[...]
if (-d 'fields') 
    system('rm', '-rf', 'fields') == 0
        or die 'Could not remove old fields directory';
 
[...]

It has been deprecated the minute Lintian::Collect API saw a little
upgrade to fetch these fields itself. This change should remove some
unnecessary file I/O in a normal run[3]. If you have a friend with his/her own little home-grown Lintian check
(or collection)[4], they will have to update it to A) not depend on
fields, B) use the Lintian::Collect API for accessing fields and C)
depend on index for source packages if it uses the index information.
For the latter, Lintian::Collect has grown support for source package
index. Lintian has also seen a lot of decrufting in the 2.5.2 development
cycle. The new test suite now features skeletons ( standard base
environments ) and template support for all package test suites[5]. This means it takes a lot less of writing (or copy/waste) to generate
a simple specialized test in the low-level test suites. As a part
of cleaning up the test-suites, I got to run:

$ git rm t/ source,debs /*/copyright

without breaking a single test. I have to admit that commit 8584996
is among my favorite ones in the 2.5.2 development cycle. Finally, if you are one of the few that have ever used the
--packages-file option, then I am pleased to inform you that it will
be deprecated in 2.5.2 and replaced by --packages-from-file. The
old option had some specialized format, whereas the new option parses
the file (or stdin) as one package per line . The latter being far
more "find -name '*.deb' "-friendly. In case you are wondering why --packages-file is deprecated, here is
the packages-file parser from lintian 2.5.1:

    while (my $line = <$pkgin>)  
	chomp($line);
	my (undef, undef, undef, $file) = split(/\s+/, $line, 4);
	$pool->add_file($file);
     

- [1] Installed as /etc/lintianrc [2] Installed as /usr/share/doc/lintian/lintian. html/,txt.gz [3] Namely, 2 opens + 2 closes + 1 read and one write for each field
in every package (the fields were cached, once they were read). Not
to mention a directory listing and having to remove the files again
later. [4] Of course, you do not have such a thing Because we all know
that no one (except this not too close friend of yours) messes with
Internal APIs of other projects [5] The exception being that the changes suite does not have a
standard base environment, since it would be completely unused.

Wondering what is going on with the port of Debian to the Minix 3 microkernal? Here's an update. dpkg Ported Well, the big news as I mentioned in my last post is that dpkg has been successfully ported. I haven't sent the patches to the maintainers just yet as they need review and clean up but I will do that ASAP. The architecture name is defined as minix_i386. (There is no k because I am using the Minix libc as well as the kernel.) I am currently targetting squeeze. A Maze of Twisty Dependencies All Different Armed with my new dpkg, I have begun building packages which has been an interesting experience. The Debian packaging systems greatest asset dependencies is also its most frustrating during initial bootstrapping. Take debhelper for instance; nowadays most packages depend on it and I personally wouldn't think of creating a .deb without it but it build-depends e.g. po4a which build-depends on...debhelper. Also some programs which are assumed to be present on linux such as ldconfig are simply not available on Minix. So currently theres a lot of dpkg --force-depends and even --force-all going on. Of course this is only a temporary problem until I get a full set of base packages built. The equivs package is coming in handy too. In lieu of working .debs, I have been using Minix native packages to fill in the gaps. They use NetBSD pkgsrc which is a new experience for me. I can't say I'm that impressed but they work and, being uptodate, are a good source of patches. Works in Progress The two big packages I need to work on next are perl 5.10.1 and gcc 4.4.5. I haven't touched the latter yet. Perl is failing in its second-stage bootstrap for reasons I haven't gotten to the bottom of just yet. I suspect it has more to do with deficiencies in my build environment than a problem in perl itself. libc Deficiencies As I mentioned last time, the situation has improved tremendously since Minix switched over to using the NetBSD libc but I am still running into problems with unimplemented APIs. Take getpwent(3) and getpwuid(3) for instance. Being able to map between user ids and names is kind of important you know? Even worse instead of being completely absent some of these exist as stubs. So they compile and link without warning but always return ENOFILE and so on. I was surprised to find that there is no standardized API for getting information on mounted filesystems. (Which e.g. df(1) needs.) Linux uses getmnt(3) similar but incompatible with System V. The BSDs use getmntinfo so probably this is what Minix will implement. GNU programs are good about portability but a lot of Free software is unfortunately Linux-centric. Hopefully the Debian FreeBSD port already has patches for these cases. There are several annoyances like this but I suppose I shouldn't complain too much. I am using an unreleased version of Minix which is under heavy development and I must say the upstream developers have been good about responding to my concerns. Looking forward: fakeroot, grub2, ext2 fakeroot is a very handy development tool that frees a developer from having to create .debs as root. Alas it won't work under Minix as it depends on LD_PRELOAD and Minix doesn't even support shared libraries. fakeroot-ng sounds a bit more promising as it uses the ptrace(2) system call which is implemented in the Minix kernel though I don't know how well. Minix is now multiboot compliant so it is possible to use grub2 instead of the Minix boot loader. Definitely something I'm going to look at soon. Minix now supports ext2 (though not ext3 or ext4) in addition to its native file system. For compatability this should become the default in the Debian port. While the kernel support is part of the standard Minix system, the userland support (e2fsprogs) is not for some reason. So I'll have to see about porting that. Can I Help the Porting Effort? Sorry not yet. Though if you are determined I'll try and help you get set up. Hopefully in a month or less I will be able to distribute a pre-alpha version so I suggest waiting for that. I am definitely going to be looking for outside help.

NOTE: This post uses MathML. You will need a modern browser to view things correctly. I just had a fairly annoying experience with pocketronics and have left negative feedback for the first time. Basically they expected the customer (my mother in this case), to contact them when Australia Post failed to deliver. And, rather than sending out another item as they had to anyway they wanted to wait until Australia Post reported back. Which takes 10 working days. So for about ~AUD$30 they decided it was worth throwing away their reputation. Obviously a company with a different view of the life time value of a customer and the earning of goodwill than most. With eBay generally almost never leave 'positive' feedback because, frankly, that should be the default case. Anyway. But some feedback about eBay's feedback system. Limiting to 80 characters is pointless. I could have put in a bit.ly link to a twitter status as my feedback for pocketronics. Or a blog post (like this), linking to their site (as above). If I want to write an article, then let me. And just as I can respond as many times as like via other media, I ought to be able to do the same via the internal ones. The score calculation is weighted towards positive feedback. As explained scores are: $\frac{\mathrm{positive}}{\mathrm{positive}+\mathrm{negative}}$ Which for pocketronics comes to: $\mathrm{99.7}=\frac{\mathrm{5071}}{\mathrm{5071}+\mathrm{16}}\mathrm{100}$ If the calculcation instead took into account neutral feedback too, then things change - not by much, but enough. To wit: $\frac{\mathrm{positive}}{\mathrm{positive}+\mathrm{negative}+\mathrm{neutral}}$ And it becomes: $\mathrm{99.0}=\frac{\mathrm{5071}}{\mathrm{5071}+\mathrm{16}+\mathrm{36}}\mathrm{100}$ I think that also by default feedback to be neutral by default, it would reduce the gaming of feedback. To have a positive score you would need to be exceptional. Not just run of the mill. Which is what the current system encourages.

When I think about Margarita, I always remember her as a friendly and welcoming person. Like most of the Debian Women members by the way. But she likes to spread some love and organized a Debian Appreciation Day for example. I think I met her in real life for the first time at Debconf 6 in Oaxtepec (Mexico). She deeply cares about Debian in general. She has proven it multiple times with her DPL candidacy and by giving talks like Making Debian rule again. One last thing, Debconf11 is just over and you will see that Debconf4 has had a big influence on Marga. My advice is simple: next time there s a Debconf on your continent, make sure to take a few days off and come to meet us! It really gives another picture of the Debian community. Now let s proceed with the interview. Raphael: Who are you? Margarita: I m Margarita Manterola, a Software Developer from Argentina. I work developing software in Python in a Debian-friendly company during the day, and teach programming at a local university during the evenings. I m married to Maximiliano Curia who is also a Debian Developer, most of our Free Software work has been done together. I only maintain a handful of packages in Debian, I m more interested in fixing bugs than in packaging new software. I ve also been a part of the organizing team of many of the previous Debian Conferences. One of the biggest commitments and the biggest success of my participation in Debian was being part of the organizing team of DebConf8, in Argentina. Raphael: How did you start contributing to Debian? Margarita: I started using Debian around 2000. Soon after we had learned the grips of general GNU/Linux usage, Maxy and I started giving an introductory course at our local university, and became quite involved with the local LUG. At some point in 2002/2003 I became a Debian Bug Reporter : most of my friends would report bugs to me, and I would then write them in the proper form to the BTS. I would also be very attentive about reporting any bugs that I might encounter myself trying to create good bug reports. The turning point in my participation in Debian was DebConf4 in Porto Alegre, Brazil. Being so close to Argentina meant that we felt specially invited to be there, and Maxy and I decided to go to DebConf for our honeymoon. We didn t really know much about DebConf dynamics, but we were really eager to learn more about Debian and become more involved. What happened was that meeting with DDs from all over the world transformed our lives, we became part of the Debian family and wanted to be more and more involved. Soon after that we both started maintaining packages and not long after that, applied to become Developers. The Debian Women project also meant a lot to me. I felt encouraged all along the way, encouraged to learn, to ask questions and to lose the fear of making mistakes. I became a Debian Developer on November 2005. Since then, Debian has always been one of the most important things I do in my life. Raphael There was a Debian Women BoF during debconf. What are the plans for Debian Women in the upcoming months? Margarita: I was not there in person, but thanks to the awesome work of the video team, and of Christian Perrier s typing efforts when something failed, I was able to experience much of what was discussed. One of the many points that came up during the BOF is that many people Want to help but don t know where to start or how to go about it. It s a challenge for the Debian Women project to find a way to allow these people to become involved in Debian through Mini projects or something like that. Another of the subjects that was brought up was the Debian Women mentoring project, which has been going on for quite a while now, but lacks enough publicity. So, we need to reach more people about it, and maybe also improve it with some templates, similar to the New Maintainer templates, so that mentees that don t know where to start have some sort of general path to follow. Raphael: You created very useful diagrams documenting how package maintainer scripts are invoked by dpkg. How did you do it and was that a useful experience? Margarita: I did those diagrams to be able to answer one of the questions in the NM templates, regarding the order of the maintainer script execution. Answering the question in text was basically copying and pasting the part of the Debian Policy that explained it, which wasn t really too clear for me, so I decided to go and make a diagram of it, so that I could really understand it. I did it by the best of all debugging techniques: adding prints to each of the maintainer scripts, and testing them in all the different orders that I could think of. It was a useful experience at the time, because I learned a lot of how maintainers scripts work. I didn t expect the diagrams to become so famous, though, I only did them to answer one NM question, that I assumed most other people had already answered before Raphael: You participated in a DPL election. This is a big commitment to make. What were your motivations? Margarita: As I said, I was part of the organizing team of DebConf8, in Argentina. Which was quite a success, a lot of people enjoyed it and praised the good work that had been done by the local team. During said DebConf8, I had a dream (it was almost a nightmare, actually): I woke up and just like that, I was the DPL. I spoke to some people about this dream and to my complete surprise many said that I should actually do it. After giving that possibility a year and a half of thoughts, during the 2010 campaign I was talked into participating myself as a candidate, and it was a very interesting experience. However, I m very glad that Zack got elected and not me, I think he makes a much better DPL that I would have made. Raphael: What s the biggest problem of Debian? Margarita: I think the main problem that we have is our communication, both inside the project and outside the project. Most of us are very technical people, our skills lay in the technical part of Debian (preparing packages, fixing bugs, writing software, administering systems) not in the social part. And thus, we lack a general empathy that is quite needed when interacting with people from all over the world. Raphael: Do you have wishes for Debian Wheezy? Margarita: Not particularly. I do want it to be a great release with good quality, stable software. I would also like to keep making Debian more and more universal with each release, making it more user friendly, more accessible, and more robust than any other previous release. Raphael: Is there someone in Debian that you admire for their contributions? Margarita: I admire a lot of people in Debian. There s a lot of people that contribute a lot of time to Debian, amounts of time that I can t begin to understand how they can afford. I admire Stefano Zacchiroli, our current project leader. And Steve McIntyre, the project leader before him. Also Bdale Garbee, who s also been a DPL in the past. Making this list I realize that Debian has been blessed by quite a number of great leaders in the past. I admire Holger Levsen, for his contributions to the DebConf video team, that have made it possible year after year for the whole project to participate in DebConf remotely. I admire Steve Langasek and Andreas Barth (etch is still my favourite release). I admire Christian Perrier for his work on internationalization. I admire Joerg Jaspert for the incredible amounts of time that he puts into Debian. And actually, I could go on admiring people all night long. I admire so many people that this interview could become a very boring list of names. I guess it s better to leave it at saying that Debian is lucky to have quite a lot of excellent hackers around.

---

Thank you to Marga for the time spent answering my questions. I hope you enjoyed reading her answers as I did. Subscribe to my newsletter to get my monthly summary of the Debian/Ubuntu news and to not miss further interviews. You can also follow along on Identi.ca, Twitter and Facebook.

14 comments Liked this article? Click here. My blog is Flattr-enabled.

The construction, deployment, and use of software involves a number of roles. I do not have an exhaustive list, but I have a couple in my list that seem to be surprising to some developers. In the traditional Unix (mainframe) model, there are three roles:

• developers actually create the software; they include architects, coders, documenters, testers, maintenance programmers, etc; their deliverable is a release tarball
• sysadmins install the software on a host, and install new versions when appropriate; often, the sysadmin is expected to carefully read README files, and construct or tweak configuration files accoriding to the specific needs of their users; their deliverable is some files in /usr/local and /etc, plus perhaps an announcement e-mail to an internal mailing list
• users use the software; they have no deliverable in this context

In my personal reality distortion bubble, this model is badly broken, and has been since the middle of the 1990s. Here's an updated model:

• users control their own machines, and are not experts on the software, or on using computers at all, and requiring them to be would be to limit their freedom in important ways; the big change is that users now have much more control of their computing environment than before
• sysadmins mainly work for companies or other organizations; they need not be experts on what their users do, and in fact often aren't, but they know how to handle systems that are by nature too complicated or hard for the layman to handle, because of performance, reliability, security, or other such needs
• distro developers create a system consisting of thousands or tens of thousands of components (mostly provided by upstream developers), and integrate and adjust things so that they work together; they also monitor upstream projects and other news sources for updates, and provide those
• upstream developers actually create the software; they include architects, coders, documenters, testers, maintenance programmers, etc;
  their deliverable is a release tarball (no change from earlier)

A common misconception is that distro developers merely compile software and shove the binaries into glorified tarballs. That is, of course, part of the job, but it's also the part that most distro's tools completely automate. There's a lot more to do, though. For example, just keeping track of known security problems among thirty thousand packages, and making or finding fixes for them, and testing the fixes, and releasing updates for them, takes up quite a bit of effort. Another thing distro developers have to spend effort on is dealing with the multitude of build systems upstream projects use. The freedom to experiment with different solutions is one of the cornerstones of free software development. However, when it comes to build systems, many of them are not serious experiments, and instead are quick hacks thrown together to imperfectly solve problems that sometimes should not exist at all. This highlights one role prominently still missing: build engineer. Their task is to solve the problem of taking the upstream source code, dealing with build-time configuration issues, actually building the software, and then installing it in whatever location the person installing the software wants it installed. Further, their task is to provide the software to do this in a sensible manner. Build engineers already exist, of course. For example, the developers of the autotools suite are such. Every upstream project also has someone who sets up autotools for their project, or some other build tools, or writes them from scratch for that project. However, what's lacking, perhaps, is recognition that build engineering is an important part of software development. Mostly, build systems are set up (or, worse, written) by upstream developers to fit their narrow view of how things should work, a view anchored in the old, broken model of roles. This attitude matters, because for the most part, build systems do not seem to be written to the same quality standards as the rest of the upstream software. For example, when an upstream developer writes a Makefile, and does not consider distro development, they'll often make it hard to differentiate between install-time location and run-time location of various components of the software. Thus, it might be hard for the Debian packager to get the software installed into debian/tmp rather than /usr/local, because the build system hardcodes debian/tmp/usr/lib/foo/plugins as the location of the plugins, since that's where the software gets installed, even if, in the Debian scenario, it's not where it gets run from. Luckily for much software these problems have been solved or avoided. Indeed, as crufty as autotools is, it's one of the things that has kept upstreams sane about their build systems, by providing such good tools that it's more work to write a new, bad one than to use autotools. We currently have roving bands of translators, who move from project to project and translate them into various natural languages. I would like to see a future where there are people dedicated to build engineering, who go from project to project and set up or fix their build problems. A good build system, once set up, should require a minimum of work to keep working. I'd also like to see much improvement in the build software. As good as autotools has been to free software, I'm sure there's a lot of room for improvement. For example, autotools requires setting up a bunch of boilerplate files in each project, which seems like something that could be avoided by relying on specific conventions. However, those improvements are a separate topic. I might go into them some day, if I decide to start developing build tools myself, but until then, I'll let those who do that decide how they want to do it. PS. There are lots more missing roles in my model of free software development, of course. For example, security engineers (find and fix problems, but also provide ways to prevent problems, or limit their scope), internal politicians (keep the project's human wheels from squeaking too much), diplomats (keep different projects from clashing too much), release engineers, etc. Someone should make an exhaustive list.

FreedomBox activities at Debconf11 I spent the last two weeks of July 2011 in Banja Luka. The occasion was the annual Debian developer's conference, Debconf11 and preceding work week known as Debcamp. This was my tenth successive year attending Debconf, and I had a very productive and pleasant time! The facilities were good, the local team was friendly, enthusiastic, and very helpful, and in addition to giving three talks and hosting a couple panel discussions, I managed to put a burst of energy into work on FreedomBox. Several other developers working on FreedomBox were also present, and a good number of Sheeva and Dream plugs were evident in the hacklabs sporting new FreedomBox stickers. Working together in the same place for several days, we made good progress on several projects, and also had some great discussions about what we want to do going forward. image building tools For some time, I've been working towards a light-weight tool set to build FreedomBox software images. Shortly before Debconf started, I chose the name 'freedom-maker' for this tool and shared a link to a readable copy of my git repository with other developers I expected to work with in Banja Luka. With input from Bert Agaz and Jonas Smedegaard during Debcamp, freedom-maker went from almost useful to actually useful. It still deserves work to be more useful to others, but I have now pushed a copy of the git repository to git.debian.org so that we can take advantage of the tools supported there to enable others to more easily contribute to the code. Very soon, Bert plans to add support to freedom-maker for using Lars Wirzenius' vmdebootstrap to build x86 images suitable for testing in a virtualized environment. At the same time, we plan to refactor the existing code slightly to enable lists of desired packages for the various image flavors we expect to produce independently of the configuration for each specific image building tool. Jonas continued in parallel to work on his alternate packaging toolset boxer. It offers some potentially interesting features for the future, and we may eventually merge some or all of it into freedom-maker, but for now it remains a separate utility. uAP user space tools Several weeks ago, we received from Marvell the source code to two user space programs that are necessary for configuring and monitoring the binary firmware provided for the uAP wireless chip used in the DreamPlug. Early during my stay in Banja Luka, I packaged these for Debian as uaputl and uapevent, and I am pleased to note that they were quickly accepted into the archive and are now present in Debian mirrors. u-boot Another bit of code received very shortly before Debconf started was the source for the version of u-boot shipped by Globalscale in the DreamPlug units we're working with. During Debcamp, Clint Adams passed a copy of this source to Jason Cooper, who was already trying to add support for the DreamPlug to upstream u-boot, but had stalled due to a lack of information. Jason has now merged his own work with the sources we got from the manufacturer, and is making good progress towards merging DreamPlug support into upstream u-boot. Once that happens, we should be able to flash our Sheeva and Dream plug devices with a u-boot image built from the source in the Debian u-boot package, in the process enabling things that matter to us like the ability to boot from an ext2 partition, and hopefully the ability to execute command scripts from that partition instead of having to hard-code kernel filenames in flash. This will allow us to support the ongoing effort in Debian to move away from the need for kernel symlinks. DreamPlug kernels With respect to kernels, another work stream at Debconf primarily involving H ctor Or n and Nick Bane was to analyze the current state of the patches from Marvell and Globalscale used to support the DreamPlug against both upstream and current Debian kernel sources. To my surprise and our collective pleasure, the remaining patch set required against current upstream kernels is much smaller than we previously believed! There are still several patches critical to us that are not merged upstream, but the work remaining to be able to build images for our devices from mainline and Debian kernel source trees now seems like something we might be able to complete before Debian's next stable release. One of our discoveries during the u-boot and kernel work during Debconf was that Globalscale did not obtain a new machine id for the DreamPlug, but instead re-used the one for the GuruPlug series, despite there being some differences in the hardware that require at least one additional driver. After much discussion, we plan to continue using the existing machine id instead of requesting another, particularly because the ARM kernel community has apparently stopped issuing new ids for the moment. We will add a new kernel config option for the DreamPlug, however, and are likely to build distinct Sheeva and Dream kernel packages that do not require initrd for use in FreedomBox images, even if doing so is not strictly necessary. This will allow us to optimize both the in-memory footprint and boot times for our devices. software configuration Another area of investigation in Banja Luka was technology for package configuration. Mirsal Ennaime performed various tests using debconf and Config::Model, with some results reflected in this commit relating to configuring the bitcoind daemon in the bitcoin package for Debian. identity and trust management While we did not actually do any FreedomBox specific work on the trust management layer we know is necessary, after several rounds of conversation, I am now more convinced than ever that the right path forward is to base our trust relationships on OpenPGP keys using GnuPG and Monkeysphere as starting software elements. Our thinking to date is captured on an Identity Management page in the wiki. communication services Another thing that became fairly clear to me during discussions at Debconf is that in the near term, planning to build communication services around XMPP is the approach most likely to give good results. Investigating the software choices available to build an interesting XMPP infrastructure is now a high priority for me. Jonas has done some work towards configuring and integrating ejabberd or Prosody, I've started studying yate as a possible call manager and VoIP server choice with XMPP/jingle support, and we await with great interest a release from the Buddycloud developers to evaluate as a possible basis for deploying social network services. Some of these software choices will lead us to use Apache as our web services base technology because of the need for features that only it supports well among daemons that are Free Software. Jonas completed packaging GNU Sip Witch for Debian, and it is now available in the mirror network. Tzafrir Cohen and Jonas did some initial testing on its use. documentation A number of new wiki pages were written (or at least started) in order to sum up ideas, design various aspects of FreedomBox, and reflect discussions that happened during DebConf11. A lot of work is needed to complete these pages though, as well as others to capture more of the current state of the project. press coverage Finally, while in Banja Luka I got some great press coverage for FreedomBox! On Sunday the 24th, I was interviewed by the main television network serving the Republika Srpska. This led to a couple of minutes of coverage near the top of the national news program that night, immediately following the lead story about the President and several ministers appearing at Debian Day that morning to help open the conference. This interview was later re-used in another TV program that summarized Debconf11. On the morning of Thursday the 28th, I was part of a small group that spent more than an hour meeting with the Minister of Science and Technology in his office, and the relationship between Debian and our work on FreedomBox was one of the items of discussion in that meeting and the associated press conference. I'm told this resulted in more press coverage, but if true I have not seen it yet. summary On Friday afternoon the 29th, I gave a talk in the main Debconf program containing a FreedomBox Progress Report . In it, I talked about the structure of the FreedomBox Foundation, progress the foundation has made, and the work that was still then underway in Banja Luka. It was streamed live over the internet, and replays are available online. The reaction from Debian developers present was very positive, which was good to learn since by that time my energy level was quite low after the nearly two weeks of intense technical and social interaction that is Debconf! All in all, we got lots of work done on FreedomBox in Banja Luka, enough that I think at least the next few steps along the road towards an eventual "1.0" release of a reference implementation are now much clearer than they were two weeks ago!

Todays addition to the tips series has a funny history. A friend of mine was stuck in an envrionment, where she couldn't install Perl or CLive but PHP with some PECL modules was installed. She needed to download a few videos for a presentation, but didn't know how to fetch the videos from YouTube. So I was asked to help. ;-) After some pondering of the problem, I came up with the following script:

<?php
/*
 * Copyright (C) 2011  Kai Wasserb ch <kai@dev.carbon-project.org>
 *
 * Small little script to help out in the absence of Perl/Clive but with
 * PHP present.
 *
 * License: GPLv3 <http://www.gnu.org/licenses/gpl-3.0.txt>
 *
 */
if(php_sapi_name() !== 'cli')
  die("This script needs to be run on the CLI!\n");
if($argc !== 2 )
  die("No URL given on CLI or too much parameters!\n");
// Script expects a URL like http://www.youtube.com/watch?v=[ID]
if(filter_var($argv[1], FILTER_VALIDATE_URL, FILTER_FLAG_QUERY_REQUIRED) === false)
  die("Parameter given as URL does not seem to be an URL or contain a query!\n");
if(!$ch = curl_init($argv[1]))
  die("Couldn't init cURL handle for  ".$argv[1]." \n");
curl_setopt($ch, CURLOPT_HEADER,          false );
curl_setopt($ch, CURLOPT_RETURNTRANSFER,  true  );
$htmlsource = curl_exec($ch);
$error = curl_error($ch);
curl_close($ch);
if($htmlsource === false)
  die('cURL returned an error: '.$error."\n");
unset($error);
if(!preg_match('/url_encoded_fmt_stream_map=([a-zA-Z0-9%\.,\-_]+)&/',
               $htmlsource, $matches))
  die("No matches found for url_encoded_fmt_stream_map!\n");
$final = NULL;
foreach(explode(',', urldecode($matches[1])) as $decoded_url_map)  
  parse_str($decoded_url_map, $tmp);
  if($tmp['quality'] === 'hd1080')  
    $final = $tmp;
    $final['url']   = urldecode($tmp['url']);
    // $final['type']  = urldecode($tmp['type']); // Not used
    break;  // Not really needed, but should save us some
            // iterations, since 1080p seems to come always first.
   
 
if(!is_array($final)   is_null($final))
  die("No 1080p version of the video was found!\n");
if(!preg_match('/<meta property="og:title" content="(.*)"/', $htmlsource, $title))
  die("Couldn't determine title!\n");
$ftitle = preg_replace('/[^\w\d]*/', '', $title[1]);
printf("Found hd1080-quality video for  %s .\n", $argv[1]);
printf("Downloading to %s.mp4 ...", $ftitle);
if(!$ch=curl_init($final['url']))
  die("\nCouldn't create cURL handle for  ".$final['url']." \n");
if(!$fh = fopen(getcwd().DIRECTORY_SEPARATOR.$ftitle.'.mp4', 'wb'))  
  curl_close($ch);
  die("\nCouldn't create  ".$ftitle.".mp4 .\n");
 
curl_setopt($ch, CURLOPT_HEADER,         false );
curl_setopt($ch, CURLOPT_FILE,           $fh   );
curl_setopt($ch, CURLOPT_BINARYTRANSFER, true  );
if(!curl_exec($ch))  
  $error = curl_error($ch);
  fclose($fh);
  curl_close($ch);
  die("\nCouldn't download video. cURL error: ".$error."\n");
 
curl_close($ch);
fclose($fh);
printf(" done.\n");
?>

You can download it from my website or you can just copy the code given above into a file. However you get the code, the execution will look like: php -f /path/to/script.php http://www.youtube.com/watch?v=[ID]. Now, before I get a lot of e-mails telling me, what could be improved on the script or its other limitations (like only being able to download 1080p videos), please remember that it was hacked together for a simple use case with no real intent on my part to use it for anything else. It only later occurred to me, that others might be in a similar pinch sometime. Thus I'm releasing the above code "as is" under the GNU GPLv3. Therefore you're free to adapt it to your needs. Please also note, that I'm not recommending using the above script instead of CLive. If you can install CLive, do so and don't bother with the above.

This is my monthly summary of my Debian related activities. If you re among the people who made a donation to support my work (170 , thanks everybody!), then you can learn how I spent your money. Otherwise it s just an interesting status update on my various projects. This month passed by very quickly since I attended both the Libre Software Meeting / RMLL and the DebConf. Libre Software Meeting / RMLL I attended only 3 days out of the 6 but that was a deliberate choice since I was also attending DebConf for a full week later in the month. During those 3 days I helped with the Debian booth that was already well taken care of by Fr d ric Perrenot and Arnaud Gambonnet. Unfortunately we did not have any goodies to sell. We (as in Debian France) should do better in this regard next time. One of the talks I attended presented EnVenteLibre. This website started as an online shop for two French associations (Ubuntu-fr, Framasoft). They externalize all the logistic to a company and only have to care about ordering goodies and delivering to the warehouse of the logistic company. They can also take some goodies from the warehouse and ship them for a conference, etc. We discussed a bit to see how Debian France could join, they are even ready to study what can be done to operate at the international level (that would be interesting for Debian with all the local associations that we have throughout the world). Back to the LSM, while I had 3 good days in Strasbourg, it seems to mee that the event is slowly fading out it s far from being an international event and the number of talks doesn t make for a better quality. BTW, do you remember that Debconf 0 and Debconf 1 were associated to this event while it was in Bordeaux? dpkg-source improvements During my time in Strasbourg (and in particular the travel to go there and back!) I implemented some changes to 3.0 (quilt) source format. It will now fail to build the source package if there are upstream changes that are not properly recorded in a quilt patch:

dpkg-source: info: local changes detected, the modified files are:
 2ping-1.1/README
dpkg-source: info: you can integrate the local changes with dpkg-source --commit
dpkg-source: error: aborting due to unexpected upstream changes, see /tmp/2ping_1.1-1.diff.cki8YB

As the error message hints, there s a new --commit command supported by dpkg-source that will generate the required quilt patch to fix this. In the processe you will have to submit a name and edit the patch header (pre-formatted with DEP3 compatible fields). You can get back the old behavior with the --auto-commit option. Build flags changes Ever since we adopted the Ubuntu changes to let dpkg-buildpackage set some build related environment variables (see #465282), many Debian people expressed their concerns with this approach both because it broke some packages and because those variables are not set if you execute debian/rules directly. In the end, the change was not quickly reverted and we fixed the package that this change broke. Despite this we later decided that the correct approach to inject build flags would be a new interface: dpkg-buildflags. Before changing dpkg-buildpackage to no longer set the compilation flags, I wanted to ensure dpkg-buildflags had some decent coverage in the archive (to avoid breaking too many packages again). My criteria was that CDBS and dh (of debhelper) should be using it. With the recent debhelper change (see #544844) this has been reached so I changed dpkg-buildpackage accordingly. Makefile snippets provided by dpkg At the same time, I also wanted an easy way for maintainers not using dh or CDBS to be able to fix their package easily and go back to injecting the compilation flags in the environment but doing it from the rules files. Starting with the next version of dpkg, this will be possible with something like this:

DPKG_EXPORT_BUILDFLAGS = 1
include /usr/share/dpkg/default.mk

Without DPKG_EXPORT_BUILDFLAGS the variables are not exported in the environment and have no effect unless you use them somewhere. More than build flags, this will also provide a bunch of other variables that can be useful in a rules files: all the variables provided by dpkg-architecture, vendor related variables/macro and some basic package information (mainly version related). dpkg-buildflags improvements Given the renewed importance that dpkg-buildflags will take now that dpkg-buildpackage no longer sets the corresponding environment variables, I thought that I could give it some love by fixing all the open issues and implementing some suggestions I got. I also had a chat with a few members of the technical committee to discuss how hardening build flags could be enabled in Debian and this also resulted in a few ideas of improvements. In the end, here are the main changes implemented:

• new prepend directive to inject flags at the start (see commit);
• new strip directive to strip flags from the result returned by dpkg-buildflags (see commit);
• new environment variables DEB_flag_MAINT_directive that can be set by the maintainer to adjust what dpkg-buildflags will return (see commit);
• new --export=configure command to inject build flags on the ./configure command line (see commit);
• new --dump command that is the default (see #603435).

Will all those changes, the complete set of compilation flags can be returned by dpkg-buildflags (before it would only return the default flags and it was expected that the Debian packaging would add whatever else is required afterwards). Now the maintainer just has to use the new environment variables to ensure the returned values correspond to what the package needs. DebConf: rolling and hardening build flags I spent a full week in DebConf (from Sunday 24th to Sunday 31th) and as usual, it s been a pleasure to meet again all my Debian friends. It s always difficult to find a good balance between attending talks, working in the hacklab and socializing but I m pretty happy with the result. I did not have any goal when I arrived, except managing the Rolling Bof (slides and video here) but all the discussions during talks always lead to a growing TODO list. This year was no exception. The technical committee BoF resulted in some discussions of some of the pending issues, in particular one that interests me: how to enable hardening build flags in Debian (see #552688). We scheduled another discussion on the topic for Tuesday and the outcome is that dpkg-buildflags is the proper interface to inject hardening build flags provided that it offers a mean to drop unwanted flags and a practical way to inject them in the ./configure command line. Given this I got to work and implemented those new features and worked with Kees Cook to prepare a patch that enables the hardening build flags by default. It s not ready to be merged but it s working already (see my last update in the bug log). A few words about the Rolling BoF too. The room was pretty crowded: as usual the topic generates lots of interest. My goal with the BoF was very limited, I wanted to weigh the importance of the various opinions expressed in the last gigantic discussion on debian-devel. It turns out a vast majority of attendants believe that testing is already usable. But when you ask them if we must advertise it more, answers are relatively mixed. When asked if we can sustain lots of testing/rolling users, few people feel qualified to reply but those that do tend to say yes. More dpkg work Lots of small things done:

• I did again some bug triaging on Launchpad. But Brian Murray did a lot of it and the result is impressive, we re down to 154 bugs (from more than 300 a month ago!).
• I updated my multiarch branch multiple times. I was hoping to meet Guillem during DebConf to make some progress on this front but alas he did not attend. I have been asked a status update multiple times during my time in DebConf.
• I fixed a regression in update-alternatives (#633627), a test-suite failure when run as root (#634961), a segfault in findbreakcycle. There have been a bunch of minor improvements too (#634510, #633539, #608260, #632937).

Package Tracking System and DEHS Christoph Berg recently wrote a replacement for DEHS because the latter was not really reliable and not under control of the QA team. This is a centralized system that uses the watch files to detect new upstream versions of the software available in Debian. I updated the Package Tracking System to use this new tool instead of DEHS. The new thing works well but we re still lacking the mail notifications that DEHS used to send out. If someone wants to contribute it, that would be great! Misc packaging work I did some preliminary work to update the WordPress package to the latest upstream version (3.2). I still have to test the resulting package, replacing upstream shipped copies of javascript/PHP libraries is always a risk and unfortunately all of them had some changes in the integration process. I also updated nautilus-dropbox to version 0.6.8 released upstream. I also uploaded the previous version (that was in testing at that time) to squeeze-backports. So there s now an official package in all the Debian distributions (Squeeze, Wheezy, Sid and Experimental)! Thanks See you next month for a new summary of my activities.

No comment Liked this article? Click here. My blog is Flattr-enabled.

DebConf 11 -- Banja Luka, Bosnia and Herzegovina I'm writing this on my way back from my fifth DebConf, waiting in the Frankfurt Airport for my flight to Munich then another wait for a flight back to Los Angeles. DebConf is the annual convention of Debian Developers and other interested in how Debian Linux (or Debian BSD) works, and want to help make it better. Debian is an international volunteer project, and Debconf is held each year in a different city, normally not on the same Continent two years in a row. Only one so far has been held in the United States, last year Debconf 10 was in New York City. (Mexico and Canada have also hosted Debconfs.) DebConf is a week long conference, with one day designated as "Debian Day" with more introductory level talks, frequently in the local language, and local people are invited to learn about Debian. The rest of DebConf is held in English. This year had over 350 attendees from a couple of dozen countries. Unlike most computer conferences, DebConf is free to attend. (Professional and Corporate memberships are available for those who would like to financially support DebConf.) If you apply early enough, you can request help paying for food, lodging, and travel as well, but such support is limited and you have to justify it. This is all make possible by our generous sponsors, who include some major corporations who use Debian. This year DebConf was held in Banja Luka, Bosnia and Herzegovina. The Bosnian bid won partially based on the generous support of the Bosnian government, not only financial and letting us use one of their buildings for the conference, but also help with getting visas and other such aid. (Visas are not required from the US or most of Europe, but they are from some other countries.) I arrived on Saturday about 24 hours after I left home, starting with too little sleep. I actually managed to eat dinner before passing out in my hotel bed. The hotel room I had in Hotel Vidovic (accent on the c) was nicer than any I have stayed at in Europe (although I always chose by price), and the conference price was less than half what a similar room goes for at a US hotel. There was a nice breakfast buffet included in the hotel cost. (Hot omelets and sausages as well as the usual cold cuts, cheese, fruit, juice, coffee, tea, cereal, yogurt, etc.) The in-room ethernet connection worked all the time except one morning, although it was not particularly fast. The hotel is relatively new, the Lift (elevator) had a date of 2005 on it and I suspect that is when the building was constructed. Sunday was Debian Day, and I went to a couple of sessions before deciding to take a nap in the afternoon. My nap wound up being from 3pm to after 11pm, sleeping through dinner. I was awake for an hour or so checking my email, and then slept till 4am. That was the morning I made it to breakfast before 7am. This was the first time I ate breakfast at normal breakfast hours for a full week for many years. After breakfast Monday, I spent several hours at the front desk helping give out badges, conference bags, and t-shirts. Lunches and dinners were in the Hotel Bosna, across the street from venue. For lunches we were served soup, salad, a plate of food, and desert. For dinners there was a buffet. The food served at Bosna was fairly good, but repetitive from day to day. (I've had much worse hotel food in the US for much more money.) Tuesday was more or less a repeat of Monday, with me opening the front desk by myself and not nearly as many new arrivals. The sessions varied of course, and I played "Debconf Experimental 5-card Mao". (Mao is a card game, popular in Cambridge and at DebConf.) Wednesday was the Day Trip. One day at DebConf is dedicated to socializing while seeing some of the local culture of the hosting city/country. This year there was an option of rafting (with some white water) or visiting a historical monastery and hiking to a waterfall with several grain mills powered by diverted water. This is the first time I had seen one actually grinding grain. (It's mainly a historical tourist attraction, but I don't know what happens to the flower made.) We then met at the rafting location for a BBQ lunch. Apparently there was some miscommunicaiton and the rafting place was not prepared for 70 or 80 people who wished to go rafting, so some did not get to raft until after lunch, and I'm not sure all that wanted to did. Dinner was back at Bosna. Thursday was another normal conference day, with the new wrinkle of distributing food tickets since Bosna was not willing/able to follow our instructions on who should be allowed to eat at Debconf expense. Those who wished to dine with the other Debconf attendees could purchase the tickets from the front desk, although this was not well publicised and few people took us up on it. The special "Conference Dinner" was Thursday night. This was held at a restaurant on the 14th floor of a building a few blocks away, and was a fancier buffet. There didn't seem to be much difference between the first and second course, and they ran out of desert before many people got any. This one was not as memorable as the one in Mexico, but that one was for the wrong reasons, including an unplanned indoor waterfall. After the dinner I played Mao again, this time till 2am. Friday was the only day I did not arrive at the venue in time for the first session (10am). Saturday I opened the front desk by myself again. In the afternoon was the "Debbugs Skills Exchange" that was requested by several people, so Collin Watson, Ian Jackson and I gave some information about it and Don Armstrong (who has done most of the recent coding on debbugs) participated via voip. (Debbugs is software for the Debian Bug Tracking System, often called the BTS.) Collin and Ian are emeritus members of the BTS team, and not active in it, while I am handling the spam filtering and haven't done much with the rest of the BTS. This was in the small "Meeting Room" (rather than the Auditorium or the round room) so we don't have to worry about video archives of it being available. (All the sessions is the two main rooms were streamed live on the internet, and will be edited and archived on http://meetings-archive.debian.net/pub/debian-meetings/2011/debconf11/.) We created a mini intro to the BTS document at that session that should be cleaned up and published. Less than two hours after that talk, the conference was over and teardown was started. As usual, the teardown and cleanup of what we had set up over several days was done is several hours, with many people helping. Sunday was leaving day. I was one of the people on the 11am charter bus to Zagreb DebConf arranged. The two such charter busses were an excellent idea, since the normal busses and trains would not have been able to manage that many extra passengers. This actually took longer than the planned 4 hours due to the long lines at the border crossing on Sunday afternoon. Both Bosnia and Croatia check passports both coming and leaving, so it is checked on both sides of the river that is the border between the countries. The check is pretty cursory unless you are from a country that needs a visa. I then had a plane flight to Frankfurt, and am currently in the middle of my 11 hour layover here. After that I fly to Munich early in the morning, have an 8 hour layover, and fly to Los Angeles. This schedule was in order to get flights at something resembling reasonable cost when I did it. From what I saw of Banja Luka, it's a nice place to visit, although most of the people do not speak English. Vegetarians may have a hard time finding meals, and sometimes they try to serve fish to them. (Assuming vegetarian means "no meat".) Transportation to Banja Luka is a bit hard, there are only six or so flights per week to the local airport, and one of the airlines is not on the international ticket sites. Most DebConf attendees flew to Zagreb then took busses, trains, vans, or taxis to Banja Luka (a 2.5-5 hour ride depending on border lines, road construction, etc.). The weather was rainy off and on throughout Debconf, but the day of the Day trip the rain held off until most of us were back and we had a lovely day. Temperatures were warm but not overly hot, and I never needed my jacket. There are many large alarm clocks in Banja Luka, (such as the one pictured here, across the street from the venue) but there doesn't seem to be a way to change the time setting or put them on snooze.

It s time to change locations for my server. I ve been with CoreNetworks for almost 5 years. They provide a good value, with fully dedicated servers a lot cheaper than most places, and good support to boot targeting people that can handle root on their own box. I m switching, though, to Hetzner Online (of Germany), primarily because I am needing more than 1GB RAM, and they can provide a 2GB box for less than I pay now for 1GB. I asked for feedback on Google+, and got positive feedback. Today, I migrated from one machine to another, copying a dozen or two GB of data, and only had about 15 minutes downtime. I was able to do this without console access to the source machine, though console access wasn t really required for either. This will work with any basic Linux install on the destination, or it could be prepared directly from a rescue environment. Before I explain how I did it, I thought it was interesting to think of the different places that the machine that was known as complete.org has lived over the years. Here s a map (click for detail): Here s the general process. First, well in advance of the move, crank the TTL on the domains way down. This way, old IPs won t be cached for very long once the system moves. Next, start rsyncing data from the old machine to the new. Do not yet shut down daemons on the old. Shut down as much as you can on the new. You ll want to focus on static data, such as /home. /usr is a good candidate as well. /var if you are selective databases may be a good candidate, or may not. The idea is to seed the destination with data so that when we do the real rsync, most of the data will be there; it will have to tidy things up after daemons are stopped, and update some things, but the bulk of the work should be done. Next, start preparing some exclude lists for the final rsync, which will copy an entire machine to the other. You will likely want to exclude files such as /etc/fstab, /etc/network/interfaces, /etc/resolv.conf, /etc/hosts, /etc/default/grub, /sys/*, /proc/*, /dev/*, and the like. I used rsync -v -P -a -H -A -X -S numeric-ids delete-after. Now, start editing config files for the new IP, but do it on the old server (these will be synced over to the new one). Start with the bind config files, and touch anything else that needs it maybe Apache configs, whatever. Next, get ready to do the final sync. In an ideal environment, we d just shut both machines down to single-user mode, but that s not going to quite cut it here. Use ps and shut down all daemons except sshd and udev on the destination. Yes, including even syslog. This is to prevent anything actively accessing the disk during the rsync, and also to prevent any issues with clients accessing server daemons that aren t ready yet. Now, on the source machine, reload bind. This will start answering DNS queries with the new machine s IP, and should also propagate the changes to your secondary DNS. Next, kill off all the daemons on the source machine, except for ssh and bind. You might want to set /etc/nologin to prevent regular users from logging in on ssh, if you have them. Now rsync things over to the new machine. Do any final tweaking over there (merging in /etc/hosts maybe, dealing with the udev rules.d persistent net thing, etc.). Reboot the new server and you should be up and running.

FireGPG is an extension that provides GPG encryption and decryption options for Firefox. It was discontinued a year ago, and didn t support Firefox 4.0 due to changes it introduced. I was happy to see that the source is still available ,and also maintained enough to have a easy option to build the extension manually. For Firefox 5.0 just download the code and use the build.sh script to build an xpi file, then install it as usual. The Firefox 5.0 support got version number 0.8.5. I m thinking about repackaging it to Debian, as it was removed from Debian testing/unstable (version 0.8 is available for squeeze). Thanks bit for the maintenance work and Aaron C. de Bruyn for making sure the single repository holds these changes.
Filed under: Debian GNU/Linux, Free software applications, Mozilla

Boy, the last three months have been very eventful, developmentally. First off, about 2 weeks after she turned 1, Zoe started walking in earnest. Within about 2 weeks of that there was no stopping her. It's funny to think she's only been walking for under 3 months and yet she's very stable on her feet. Secondly, she developed a mild egg allergy. Sarah was giving her some scrambled eggs (one egg's worth) for lunch one day, and she mixed in some cream cheese, and Zoe absolutely scoffed it all down, so she scrambled another egg and she scoffed that down too. She then broke out in some hives around her mouth and Sarah noticed that she was coughing a bit and a little wheezy when she put her down for her afternoon nap. Upon subsequent consultation with her paediatrician and a blood test, the allergy was confirmed and we now have to avoid eggs and Zoe was prescribed an Epi Pen Jr. We went and saw an allergist as a followup, and he did some skin tests that showed eggs were the only thing she's allergic to. He said her allergy was very mild in the grand scheme of things, and we're going to do another skin test when she's 2 to see if she's grown out of it. Eggs in baked goods are generally okay, as the offending protein is denatured at high temperatures. She's more allergic to the egg whites (1.56, which is in the "moderate" range) than the egg yolks (0.55, which is in the "low" range). Egg allergies are apparently something that one does eventually out grow, so that's a good thing. We'd also been keeping her off gluten and dairy because of a strong correlation between them and rather disgusting bowel motions. We've successfully reintroduced gluten with no adverse effects, and we've just started her back on cow's milk. We of course had our epic trip to the Grand Canyon, which Zoe handled absolutely marvelously. That was closely followed by a weekend down in LA, which was a bit more of a disaster as Sarah and I both came down with a 24 hour stomach bug that Zoe had had during the week. There's nothing quite like languishing in a non-childproofed hotel room, feeling like death, with a full of beans toddler busy running around jamming her fingers in drawers. Fortunately we were able to use a babysitter for a few hours respite, but it did rather completely mess up our plans for the weekend. Zoe did handle the drive to and from LA very well though. Zoe's been doing Saturday morning swim classes at the YMCA, which I've taken her to. I don't think she's enjoyed them as much as the indoor ones she did at the Sutton Swim School, where she went with Sarah previously, and I definitely don't feel like the classes are as structured. Unfortunately we just finished an 8 week session, and the next one filled up before we got a chance to book, so we're hoping to get back into something at the Sutton Swim School once an opening comes up. I might do something ad hoc with her on Saturdays at the Rinconada Pool in Palo Alto in the meantime. Zoe has also started going to regular daycare on Thursdays, so Sarah gets a day to focus on her University study. It's a home daycare that isn't too far away, and our friend's daughter goes there too, so there's someone she knows. The first few times Zoe got pretty upset when she was dropped off, but she barely makes a whimper now when Sarah drops her off. She's also usually down to one nap a day now. She has four teeth, top and bottom, with one molar definitely through and another three starting to make an appearance. I just bought an iBert safe-T-seat for my bike, because I've been itching to get out on the bike with Zoe on the weekends when I'm looking after her so Sarah can get some more study done. The weather is currently nice, and I've just learned about a bike trail that cuts across the bay in front of Moffett Field, so I'm keen to check that out. Apparently there's geese out there. I did my first decent ride with her last Sunday to the Mountain View Farmer's Markets and back. Zoe seemed to be okay with it. The helmet we got her drew a lot of comments from people at the markets. She's very much in the "point and grunt" stage at the moment. What's really cool is she'll nod or shake her head in response to yes/no questions. It's so cool when you can ask her a question and she solemnly nods her head in response. Often in the morning Sarah takes her to the park around the back of our home, and so now Zoe's gotten into that habit, so she'll scratch around at the front door and they head over there. The other day, Sarah opened the front door and she took off in the direction of the park, so Sarah just followed her to see where she went. Sure enough, she made her way over to the park. Tonight, Zoe fetched me my shoes and then dragged me over by the hand to the front door to demand that I take her out for a walk. She's also really getting into books now. She actively seeks out books, and has a bit of a flick through them, or brings one over and climbs into our laps, or backs up and sits in our laps if we're sitting on the floor, to have us read them to her. She's now strong enough to hold her bottle herself at bedtime, so I sit with her in my lap in the glider and read a story to her while she has her bedtime bottle. It's indescribably wonderful. She seems to understand quite a large array of words. We can go "Where's your <insert object here> and she'll look around the living room and generally find the item we're talking about. She also seems to understand questions about breakfast, lunch, going to the park, going to the bathroom, having a bath, etc as she seems to respond appropriately. In the speech department, I haven't really counted the words, but she's not saying a huge number of words. This morning, apparently she yelled "MAMA! MAMA! PARK!". "Park" being a new word for her. We bought an annual membership to the Children's Discovery Museum in San Jose, and we've probably taken her there half a dozen times or so since getting the membership. She particularly loves the water area. We finally got around to getting our back yard landscaped and made generally fit for a small child to run around in. Sarah's planted a butterfly garden in one corner, so hopefully once that's established we'll have some fun things for Zoe to watch from the window. I'm looking forward to planting some bulbs in the autumn. I expect the next 3 months will be more of the same. We've got some more overseas travel coming up, but a lot of relatives will be around, so hopefully we'll get a bit of downtime to ourselves.

Due to the midterm evaluation, this report will cover a span of 4 weeks. As a result, it will be quite a bit longer than my previous reports. One of the first things I did after my last report was create a FAQ for the project. At first, the FAQ was a plain text file living only in the git repository. Then, it moved to being maintained on the Debian wiki with a cronjob pulling it into my local repository for me to commit and push. Finally, the cronjob moved to wagner. The FAQ is not actually committed to the git repository, but it is available in the dextools instance running on wagner or on the wiki. There is also now a basic README file that is part of the git repository that simply points people to the FAQ on the wiki (so people who branch the repository can easily find the documentation). For those of you who have been reading my other reports and/or testing the dashboard, you have probably noticed the problems I was having getting the text boxes sized properly. After spending countless hours researching and testing different solutions, we finally decided to give up on this for the time being. Therefore, all of the size attributes have been removed from the text inputs, and the background color has been reset to its default value. This means that we will no longer experience the issue of very wide text boxes and a horizontally stretched table. However, it also means that long strings will get cut-off when displayed in a text input. Since my initial mockup design at the start of the summer, I have had the rows of the table zebra-striped. Based on some feedback from Paul Wise (pabs), I changed this so that the row colors are based on the status of the task. Open tasks are red, closed tasks are green, and in progress tasks are yellow. This makes it much easier to quickly see the progress of the project and find tasks you are interested in. My original dashboard consisted of one main dex.html file. This file used some javascript to pull in each project s task list. We felt that a much more common use case would involve a user only interested in one particular project. As a result, they would not be likely to use the select boxes to change projects. This ultimately led to me creating standalone dex.html dashboards for each project. These per-project dashboards were identical to the original dex.html, except they did not include the select boxes at the top for changing projects. Eventually, I decided to completely get rid of the original dex.html dashboard. That file now consists of a simple list of links to all of the projects sorted by distribution. Another advantage of this change is that it is much easier to link to a specific project. For example, you might now link to /projects/dex-ubuntu-ancient-patches/dex.html (you can get the URL by simply copying it from the address bar). Another advantage is that you can now link directly to a specific project s graphs (just append #graph). Another feature that I am still working on is per-project documentation. From the start, we wanted to make it easy for new contributors to get involved in DEX. One way we can do this is by having clear documentation about what the project is and how to deal with tasks. I decided that the easiest approach would be to have this documentation live on the wiki. This allows users to use the familiar wiki markup language to format their text, and it gives us all of the other benefits a wiki provides. Documentation will live at /DEX/<distribution>/ <project>. The dashboard will then take care of pulling in this documentation and displaying it at the top of the proper page. For a while, I was thinking that it would be necessary to filter the HTML produced by the wiki to prevent malicious code from ending up on the dashboard. However, after playing around for a bit, I think that it should be pretty safe to just use the HTML as-is (if someone spams the wiki, we can always revert the change). I am currently finishing up some of the code to handle this, and it should be live sometime this week (a partially working version is live on wagner). There are also two new checkboxes at the top of each dashboard. These allow users to add completed tasks and/or tasks with completed bugs. This is useful for projects that are close to being complete, as it allows users to quickly view the few remaining tasks. This feature was requested by several people, and I am glad to finally be able to implement it for them. There is also a second graph on each page. This graph is a bar graph that shows the number of tasks each person has completed. It gets the people from the assignee column for the project, and it only counts tasks with a status of Complete . The graph is updated whenever the getbugs.py script runs to update the task listing (currently every 10 minutes). The bars are sorted from tallest to shortest. The idea behind this graph is to recognize the people working on a project. The form processing script has not worked on wagner. This was mainly due to a permission issue. Thanks to a hint from Rapha l Hertzog, I was able to use setfacl to fix this. The form can now be submitted, and it should successfully save all changes made via the web form. This involved some updates to support the per-project dex.html files. The form processing script will also ignore any status that is not complete , in progress , or incomplete , which prevents people from messing up the dashboard by submitting invalid information. Finally, the script will now try to use the HTTP_REFERER header to send the user back to the correct page. If this header is not set, the user will be sent back to the main dex.html index page. Sorting the table has been partially broken since I started including text inputs. I finally got this issue resolved, and I should now be able to sort by text inputs, select boxes, or anything else I might end up including in the table. You can test this by clicking on any header in the table. I will probably add a tiny image to make it more clear which header is being used to sort the table. By default, it is sorted based on the status and then the task name. The status column now has a select box for setting the status. Since the row coloring is done via javascript, the minute you change the status in the select box, the row color will change to reflect the new status. Keep in mind, for the time being, any change you make via the web form will be lost if you do not hit the Submit button at the top of the form. That covers most of the dashboard changes that have taken place since my last report. However, there were also some other changes that are worth noting. Matt Zimmerman, my original mentor, settled down from his traveling and resumed serving as my mentor for the project. I would like to thank Stefano Zacchiroli for all of his help this summer. I look forward to working with him more in the future. I have also started using the issue tracker feature on alioth for dextools. Right now, it is mainly being used to help me keep track of bugs and features that I need to deal with. I am hoping that as dextools starts to be used by more people that they will also help report issues on the tracker. This issue tracker is a good source of information for people interested in knowing what I will be working on in the future. I will briefly talk about some of the more important features that I will be working on. The first issue is one that I have already touched upon in this report. I want to sort out the per-project documentation. This is rather important, as it will most likely be utilized by all projects using the dashboard. It is also essential if we are to get new contributors involved in DEX projects. Second, I want to setup some method for backing up the projects/ directory on wagner. This directory is not maintained in the git repository, and it contains all of the information submitted via the web form and generated by the scripts. My current plan is to have this directory maintained in its own VCS. Every time getbugs.py runs or the form is submitted, a new commit will be made. This will allow us to easily revert any changes (spam) made to the dashboard. Since we do not foresee the dashboard being the target of much abuse, any user will be able to modify data via the web form. I also want to add the ability for users to create new projects and task via the web. I envision this as a separate form where they enter the distribution name, project name, a list of BTS bug numbers, and a list of any additional tasks. The script will then take care of applying the necessary usertag to the bugs and/or creating a tasks file in the projects directory for the non-bts tasks. Doing this will make starting a new project trivial, and will eliminate the need for VCS access to create projects that do not utilize BTS bugs. Finally, I want to eliminate the need for the user to hit the Submit button to save their changes. Any change the user makes should be submitted and saved instantly using AJAX. This will also help eliminate some confusion that was caused by the row color instantly changing when the status is changed. Some users thought that the color change meant that the change was saved when it really was not. Matt asked me the other day what features I think need to get added before the dashboard is really ready for public use. While the features and issues I mentioned above will help make the dashboard easier to use, the dashboard is still perfectly usable without them. There are only a few more weeks left in the Google Summer of Code 2011. By that time, all of these features and bugs should be sorted out, and we should be able to properly announce the dashboard and make it available for use on all DEX projects. That does not mean that the dashboard will be done . Instead, it will probably result in many new bugs and feature requests being submitted. However, at that point, I will make a very strong effort to keep the dashboard stable and to prevent data loss. I am looking forward to the finally finishing up this project and seeing it used by the DEX team. As always, if you are interested in helping out with DEX or dextools, my code is available in a git repository on alioth. There is also the new issue tracker where you can report bugs and request new features. Finally, you can join the debian-derivatives mailing list, join #debian-derivatives on OFTC, or contact me directly via email or on IRC (nhandler).</project></distribution>

As Redis writes to disk asychronously, it is not sufficient to monitor whether it is still listening to TCP/IP connections - you must check whether it is managing to persist the in-memory data to disk. However, as it is not currently possible to deterministically monitor the status of these background saves, we are limited to checking that the number of seconds since the last save has not exceeded some specific number. This is possible by using last_save_time which is available via the INFO command: Checking for this has already saved me twice: once when my vm.overcommit_memory = 1 change was accidentally rolled back (see "Background saving is failing with a fork() error" in the Redis FAQ), and again when the machine simply did not have enough disk space. To monitor this with Nagios: The following invokation will trigger a warning when we haven't saved in an hour and become critical if we haven't saved within 24 hours. (If we cannot connect to the server we are immediately critical, so we avoid an extra check just for this.) This setup works well, although we are now monitoring a non-deterministic heuristic. To remedy this, Redis could expose background save failures in the INFO output (last_bgsave_status=fail, perhaps?). However, the treatment of previous contributions puts me off sending further patches.

I recently got myself a FONIC account for mobile Internet. This (German) prepaid-provider offers a "daily flatrate" for 2.50 per day. After the 10th day of usage (i.e., 25 ) you don't pay any more. This means, even if you need mobile Internet access 31 days a month, you only pay for 10 days. After 500MB/day or 5GB/month you're throttled down to GPRS speed (but you can still connect, and you don't pay more). The FONIC account comes with the "FONIC Surf-Stick", a Huawei E1750 HSPA USB modem (apparently it supports GPRS, EDGE, UMTS, HSDPA (up to 7.2Mbit/s), HSUPA (up to 5.76 Mbit/s), and a SIM card. In order to use the device on Linux you need two packages, usb_modeswitch and wvdial:

  $ apt-get install usb-modeswitch wvdial

Recent versions of usb_modeswitch (and matching udev entries) already support the Huawei E1750 out of the box, a few seconds after attaching the device it's automatically switched into modem mode. After this has been done you should have three new serial devices, usually /dev/ttyUSB0, /dev/ttyUSB1, and /dev/ttyUSB2. You'll need /dev/ttyUSB0 for talking to the device using AT commands. The lsusb output should look like this (see here for full lsusb -vvv):

  $ lsusb
  Bus 001 Device 045: ID 12d1:1436 Huawei Technologies Co., Ltd. 

(before usb_modeswitch was run, the USB IDs were 12d1:1446) The required settings for connecting are documented at fonic.de, specifically the APN (pinternet.interkom.de). A username and/or password is not required. You need to provide your FONIC PIN though. Dialing is done using the *99# number and using the ATDT command. I'm using the following wvdial config file:

  $ cat /etc/wvdial.conf
  [Dialer Defaults]
  Modem = /dev/ttyUSB0
  Baud = 460800
  [Dialer pin]
  Init1 = AT+CPIN=1234
  [Dialer fonic]
  Phone = *99#
  Username = foo
  Password = foo
  Stupid Mode = 1
  Dial Command = ATDT
  Init2 = ATZ
  Init3 = AT+CGDCONT=1,"IP","pinternet.interkom.de"

For mobile Internet access you would do the following:

1. Attach the device via USB, wait a few seconds to let usb_modeswitch do its magic.
2. Run wvdial pin and wait a few seconds (until the prompt returns):

     $ wvdial pin
     --> WvDial: Internet dialer version 1.61
     --> Initializing modem.
     --> Sending: AT+CPIN=1234
     AT+CPIN=1234
     OK
     --> Modem initialized.
     --> Configuration does not specify a valid phone number.
     --> Configuration does not specify a valid login name.
     --> Configuration does not specify a valid password.
   

3. Run wvdial fonic and wait until the "CONNECT" message appears and you get DNS addresses:

     $ wvdial fonic
     --> WvDial: Internet dialer version 1.61
     --> Initializing modem.
     --> Sending: ATZ
     ATZ
     OK
     --> Sending: ATZ
     ATZ
     OK
     --> Sending: AT+CGDCONT=1,"IP","pinternet.interkom.de"
     AT+CGDCONT=1,"IP","pinternet.interkom.de"
     OK
     --> Modem initialized.
     --> Sending: ATDT*99#
     --> Waiting for carrier.
     ATDT*99#
     CONNECT
     --> Carrier detected.  Starting PPP immediately.
     --> Starting pppd at Mon Aug  1 xx:xx:xx 2011
     --> Pid of pppd: 18672
     --> Using interface ppp0
     --> local  IP address xxx.xxx.xxx.xxx
     --> remote IP address yyy.yyy.yyy.yyy
     --> primary   DNS address 193.189.244.225
     --> secondary DNS address 193.189.244.206
   

If everything worked fine you should now have connected successfully. There are other alternatives for achieving the same result, including umtsmon (Qt3 in the last release from 2009, looks a bit unmaintained), kppp, the GNOME NetworkManager, and others, but wvdial worked OK for me. For more details about the Huawei E1750 device (e.g. lsusb -vvv and more photos), see my wiki page at http://randomprojects.org/wiki/FONIC_Surf-Stick_Huawei_E1750 Update 2011-08-03: My measured download speed for a Debian ISO (over HTTP via wget, at night, roughly 22:00 o'clock) is 350-470 KB/s in case anyone is interested. During this download the blue LED on the stick was enabled, which denotes a UMTS connection (green == GPRS/EDGE, turquoise == HSDPA).