So this week I bought a cheap Android tablet, specifically to allow me to read books upon the move and listen to music at home. Currently my desktop PC has a bunch of music on it, and I listen to it locally only. In my living room I have an iPOD in a dock which has identical contents. In my bedroom? No music. This was something I wished to change. The music on my desktop is played via mpd, and I use the sonata client to play it. I figured I could use this for my tablet, as MPD is all about remote control and similar stuff. Unfortunately things didn't prove so easy. I found several android applications that would let me connect to my MPD server and control it, but despite several of them claiming to support streaming I couldn't make music appear upon my local tablet. (I rebuilt MPD with the lame encoder available, and used that too, to no avail). Still if you have music in a central location and you wish to control it then the setup is trivial. (Though there is one caveat with MPD and streaming, my understanding is that you can only stream what you're playing. You cannot configure MPD to stream music and not also play it locally..) So although it was neat to be able to control the music on my desktop host it wasn't what I wanted. Instead I had to install mediatomb to my desktop to serve the media, and use the 2player application to browse and play it. Once I'd configured mediatomb all my music was available to my tablet. Result. Unfortunately 2player didn't play my movies, for that purposes I needed to use bubbleupnp. But that was a trivial install too. So? End result. I have a toy tablet for < 100 which will stream my music to the bedroom. ObQuote: "How about this: I work for you; in exchange, you teach me how to clean. " - L on

This is my monthly summary of my Debian related activities. If you re among the people who made a donation to support my work (72.65 , thanks everybody!), then you can learn how I spent your money. Otherwise it s just an interesting status update on my various projects. This month has been a short one since I have been away for 2 weeks of vacation. Dpkg My dpkg work encompasses a bunch of small tasks:

• I uploaded dpkg 1.16.7 with an important bugfix for a regression.
• I initiated a new round of discussions about how we were going to solve the problem that source packages with Multi-Arch: same binary packages can t be individually bin-nmued.
• Following that discussion, I opened a bunch of bugs to plan/discuss the transition of changelog/copyright files within the package metadata (#681289 on debian-policy, #681293 on apt-listchanges, #681295 on www.debian.org for packages.debian.org).
• I also filed #681292 on sbuild to get it to use dpkg s new syntax for bin-nmus. It will allow us to do binary-only rebuilds with arbitrary versions (instead of only +b1 suffixes). Ubuntu could use this for their +rebuild1, we could use this to build backports which do not require source changes (and share the common source package instead of duplicating it). It can also be useful if we ever get to the situation where transitions are prepared in external repositories and where we want bin-nmus in those repositories to have unique versions (even though the same package might be bin-nmued in multiple repositories in case of concurrent transitions).
• I filed an unblock request for dpkg once it was almost 10 days old.
• I did reconsider the bug #316521 where dpkg looses track of some shared directories with manually created files and proposed an updated patch. No words from Guillem on the patch yet. Fixing this would help to fix a bunch of piuparts issues.
• And just before my vacation, I filed many bugs against dpkg itself, effectively moving some of the items that accumulated in my TODO in a public place where others can help (I ll be happy to mentor anyone who wants to tackle one of these):
  • #681443: dpkg-source commit should be able to merge changes in an existing patch
  • #681470: dpkg-shlibdeps: should also scan Build-Depends-Arch for minimal versions
  • #681474: Dpkg::Vendor: should support /etc/os-release and /etc/os-release.d/*
  • #681477: dpkg-vendor: implement select-closest command
  • #681480: base-files: Provide HOME_URL, SUPPORT_URL and BUG_REPORT_URL in /etc/os-release
  • #681489: base-files: Add /etc/os-release.d/debian and make it easy to provide supplementary /etc/os-release.d/* files
• In #595112 we discussed the specifics of a new dpkg-mainstscript-helper feature to move a conffile from one package to another.

Packaging I updated nautilus-dropbox to version 1.4.0 and python-django-registration to version 0.8. Both have been uploaded to unstable and I initially wanted to request an unblock for the latter, but it turns out it has gained reverse dependencies and version 0.8 introduces API changes so it s not an option at this point of the freeze. QA work I investigated and fixed #678356 where it had been reported that the PTS static news were no longer working as expected. At the start of the month, I also unblocked the mostly-unknown but important mole service it was out of date of several weeks and several people were annoyed that the information about new upstream versions was no longer up-to-date. Vacation Almost no Debian work during my vacation but the lack of Wifi nearby made me look for solutions to connect my computer through my Nokia N900 3G/GPRS connection. I discovered the Mobile Hotspot application (homepage) and it worked like a charm (although it required Maemo s non-default devel repository to be able to install the alternative kernel for power users ). The Debian Handbook Michal iha proposed us to host a Weblate instance to help translate the book with a web interface. He kindly agreed to implement some improvements to better suit my requirements. Those have been completed and the weblate instance is now live at debian.weblate.org. There s no requirement to use Weblate for translations teams but for those that do, it sure makes it easier to recruit volunteers who have no prior knowledge of Git and PO files. If you want to help, please checkout this page first though, you should not start using Weblate without getting in touch with the respective translations teams. Apart from translations, I also had the pleasure to merge some patches from Philipp Kern who improved the section covering IPv6 and a few other parts. We can make the book even better if more people share their expertise in the part of the book where they know better than me and Roland. Thanks See you next month for a new summary of my activities.

No comment Liked this article? Click here. My blog is Flattr-enabled.

More and more people seem to be interested in Weblate and it is becoming less and less comfortable to handle all this communication privately in my mailbox. That's why I've decided to open up mailing list for Weblate. It is now available at weblate@lists.cihar.com, you can subscribe at https://lists.cihar.com/cgi-bin/mailman/listinfo/weblate (request to add this list to Gmane is pending). I hope this will attract more interested people and open up wider discussion about some features. PS: Now there is also #weblate on freenode if you are interested in chatting about Weblate.

Filed under: English Weblate 0 comments Flattr this!

Version 0.2.0 of the recently introduced RcppCNPy package for reading/writing NumPy data in R arrived on CRAN earlier today. The main change are the added ability to also write gzip-ed npy files, to suppress an automatic transposition as well as the correction of one memory leak on data read. The NEWS file entry is below:

Changes in version 0.2.0 (2012-07-30)

• Support for writing of gzip-ed npy files has been added.
• A new option dotranspose has been added to npyLoad() to support data sets that do not need to be transposed to be used in R.
• A memory leak in reading files has been corrected.

CRANberries also provides a diffstat report for 0.2.0 relative to 0.1.0. As always, feedback is welcome and the rcpp-devel mailing list off the R-Forge page for Rcpp is the best place to start a discussion.

A non-profit organisation I support has just bought a Dell PowerEdge T410 server to be mainly used as a file server. We need a reasonable amount of space and really good reliability features because the system may have periods without being actively monitored, it also has to be relatively cheap. Dell servers are quite cheap, but disks are not cheap at all when Dell sells them. I get the impression that disks and RAM are major profit centers for Dell and that the profit margins on the basic servers are quite small. So naturally we decided to buy some SATA disks from a local store, one advantage of this is that Dell sells nothing bigger than 2TB while 3TB disks are available cheaply everywhere else. So we bought 4 cheap 3TB (2.7TiB) SATA disks, connected them to the server, and found that only 2TiB was accessible. The Dell Tech Center says that some of the RAID controllers don t support anything larger than 2TiB [1]. Obviously we have one of the older models. There s lots of SATA sockets on the motherboard that could be used, however there is one problem. The above picture is the side view of the T410, it was taken with a Samsung Galaxy S so the quality is a little poor (click for the original picture). The server is quite neat, not many cables for a system with 6 disks, not the 12 separate cables you would get in a typical white-box system. The above picture shows the disk enclosure. You can see that each disk has a single connector for power and data, also the disks aren t separate, multiple disks have the same power wires and the data cables are paired. Above you can see the SAS controller, it has two large connectors that can each handle the data cables for 4 disks, nothing like the standard cables. It s easy to buy SATA data cables and connect them, but there are no spare power cables in the box. The connector that supplies power to all the disks appears to be something proprietary to Dell which goes straight to the double-connectors on each disk that supply power and data. This setup makes cabling very neat but also provides no good option for cabling regular disks. I m sure I could make my own cables and if I hunted around the net enough I could probably buy some matching power cables, but it would be a hassle and the result wouldn t be neat. So the question was then whether to go to more effort, expense, and possibly risk the warranty to get the full 3TB or to just use the SAS controller and get 2TiB (2.2TB) per disk. One factor we considered is the fact that the higher sector numbers typically give much slower access times due to being on shorter tracks (see my ZCAV results page for test results from past disks [2]). We decided that 2.2TB (2TiB) out of 3TB (2.7TiB) was adequate capacity and that losing some of the slow parts of the disk wasn t a big deal. I ve now setup a RAID-Z2 array on the disks, and ZFS reports 3.78TiB available capacity, which isn t a lot considering that we have 4*3TB disks in the array. But the old server had only 200G of storage, so it s a good improvement in capacity, performance, and RAID-Z2 should beat the hell out of RAID-6 for reliability.

• [1] http://en.community.dell.com/techcenter/extras/w/wiki/2837.aspx
• [2] http://www.coker.com.au/bonnie++/zcav/results.html

Related posts:

1. Dell PowerEdge T105 Today I received a Dell PowerEDGE T105 for use by...
2. ZFS vs BTRFS on Cheap Dell Servers I previously wrote about my first experiences with BTRFS [1]....
3. How I Partition Disks Having had a number of hard drives fail over the...

I tried to send this text to the docbook-apps mailing list at lists.oasis-open.org, but it only accept messages from subscribers and rejected my post, and I completely lack the bandwidth required to subscribe to another mailing list, so instead I try to post my message here and hope my blog readers can help me out. I am quite new to docbook processing, and am climbing a steep learning curve at the moment. To give you some background, I am working on a Norwegian translation of the book Free Culture by Lawrence Lessig, and I use docbook to handle the process. The files to build the book are available from github. The book got around 400 pages with parts, images, footnotes, tables, index entries etc, which has proven to be a challenge for the free software docbook processors. My build platform is Debian GNU/Linux Squeeze. I want to build PDF, EPUB and HTML version of the book, and have tried different tool chains to do the conversion from docbook to these formats. I am currently focusing on the PDF version, and have a few problems.

• Using dblatex, the <part> handling is not the way I want to, as </part> do not really end the <part>. (See BTS report #683166), the xetex backend (needed to process UTF-8) give incorrect hyphens in index references spanning several pages (See BTS report #682901), and I am unable to get the norwegian template texts (See BTS report #682936).
• Using straight xmlto fail with some latex error (See BTS report #683163).
• Using xmlto with the fop backend fail to handle images (do not show up in the PDF), fail to handle a long footnote (overlap footnote and text body, see BTS report #683197), and fail to create a correct index (some lack page ref, and the page refs listed are not right).
• Using xmlto with the dblatex backend behave like dblatex.
• Using docbook-xls with xsltproc + fop have the same footnote and index problems the xmlto + fop processing.

So I wonder, what would be the best way to create the PDF version of this book? Are some of the bugs found above solved in new or experimental versions of some docbook tool chain? What about HTML and EPUB versions?

What an astonishing thing a book is. It s a flat object made from a tree with flexible parts on which are imprinted lots of funny dark squiggles. But one glance at it and you re inside the mind of another person, maybe somebody dead for thousands of years. Across the millennia, an author is speaking clearly and silently inside your head, directly to you. Writing is perhaps the greatest of human inventions, binding together people who never knew each other, citizens of distant epochs. Books break the shackles of time. A book is proof that humans are capable of working magic.

Carl Sagan, The Persistence of Memory

I have had the pleasure of attending GUADEC in full this year. TL;DR: A lot of great presentations + lots of hall conversations about QA stuff + the obligatory be er,ach = . Last year I just went to the hackfest, and I never made it to any previous one, so GUADEC 2012 was a kind of first-time experience for me. It was great to put some faces and personal impressions to a lot of the people I have worked with for many years, as well as catching up with others that I did meet before. I discussed various hardware/software testing stuff with Colin Walters, Matthias Clasen, Lennart Poettering, Bertrand Lorentz, and others, so I have a better idea how to proceed with automated testing in plumbing and GNOME now. It was also really nice to meet my fellow PyGObject co-maintainer Paolo Borelli, as well as seeing Simon Schampier and Ignacio Casal Quinteiro again. No need to replicate the whole schedule here (see for yourself on the interwebs), so I just want to point out some personal highlights in the presentations:

• Jacob Appelbaum s keynote about Tor brought up some surprising facts about how the project has outgrown its past performance problems and how useful it was during e. g. the Arab revolution
.
• Philip Whitnall s presented Bendy Bus, a tool to mock D-Bus services for both unit and fuzz testing. He successfully used it to find and replicate bugs in Evolution (by mocking evolution-data-server) as well as libfolks (by mocking the telepathy daemons). It should work just as well to mock system services like upower or NetworkManager to test the UI bits that use it. This is a topic which has been on my wishlist for a long time already, so I m happy that there is already an existing solution out there. We might have to add some small features to it, but it s by and large what I had in mind, and in the discussion afterwards Philip said he d appreciate patches against it.
• Christophe Fergeau showed how to easily do Windows builds and installers from GNOME tarballs with MinGW-w64, without having to actually touch/use Windows (using cross-building and running tests etc. under wine). I found it surprising how easy that actually is, and it should not be hard to integrate that in a jhbuild-like setup, so that it does not keep breaking every time.
• Colin Walters gave an introduction to OSTree, a project to build bootable images from kernel/plumbing/desktop upstream git heads on a daily basis. This is mostly to avoid the long delays that we otherwise have with doing upstream releases, packaging them, and getting them into a form that can safely be tested by users. In an afterwards discussion we threw some ideas around how we can integrate existing and future tests into this (something in spirit like our autopkgtest). This will be the area where I ll put most focus on in the next time.
• Adam Dingle of yorba fame shared his thoughts about how we can crowdfunding of Free Software Projects work in practice, comparing efforts like codefoundry and kickstarter. Of course he does not have a solution for this yet, but he raised some interesting concerns and it spun off lots of good discussions over lunchtime.
• Last but not least, the sport event on Saturday evening was awesome! In hindsight I was happy to not have signed up for soccer, as people like Bastian or Jordi played this really seriously. I participated in the Basketball competition instead, which was the right mix of fun and competition without seriously trying to hurt each other.

There were a lot of other good ones, some of them technical and some amusing and enlightening, such as Frederico s review of the history of GNOME. On Monday I prepared and participated in a PyGObject hackfest, but I ll blog about this separately. I want to thank all presenters for the excellent program, as well as the tireless GUADEC organizer team for making everything so smooth and working flawlessly! Great Job, and see you again in Strasbourg or Brno!

What is good in "my" job in Debian are opportunities to discover new interesting packages. While surveying the completion of debconf translations in unstable, I thus noticed a new package named "hoteldruid", that has a few questions and interaction with users. After my usual mumble because French and a few other languages were *finally* virtually 100% in unstable for a few days...... I went on my usual task in such cases: propose a review of debconf templates and package description to my fellow debian-l10n-english co-workers (/me bends to Justin B. Rye, our tireless, picky, efficient and very clever Master Reviewer for over 5 years now). Then I discovered what HotelDruid is about: this is a piece of PHP-based software meant to manage....an hotel or bed and breakfast...or any kind of such facility. Real end-user software. Really useful software. For real people...:-) Not the gazillionth obscure development language, or yet another encryption library, or yet another mysterious virtualization thing used by 10 people in the world (even if they host thousands of machines). These are the free software pieces I like the most. Probably Marco Maria Francesco De Santis (the upstream author and Debian package maintainer) somewhere in, I guess, Italy is running a small B&B (or maybe his parents, or his wife/cousin/whatever) *and* is a free software addict. And he wanted to run his business with free software. Same for this French genealogist who once wanted to display his data over the web (and make a real use of that obscure Ocaml language.....yes, pun itended to my friends, here). Of the person who wrote LedgerSMB to manage his business. Or those who use free software to manage hospitals (hi Andreas) or schools (hi DebianEdu folks...and special hi to Petter). Real software for real people. Of course, developed with obscure geeky things used only by those weird geeks who like to sometimes travel half a planet to just gather together and develop the best free operating system ever. Guess what? I like this! And guess what? I proposed the HotelDruid author to check whether we could imrove....translation, of course..:-)

As storage capacities increase the probability of data corruption increases as does the amount of time required for a fsck on a traditional filesystem. Also the capacity of disks is increasing a lot faster than the contiguous IO speed which means that the RAID rebuild time is increasing, for example my first hard disk was 70M and had a transfer rate of 500K/s which meant that the entire contents could be read in a mere 140 seconds! The last time I did a test on a more recent disk a 1TB SATA disk gave contiguous transfer rates ranging from 112MB/s to 52MB/s which meant that reading the entire contents took 3 hours and 10 minutes, and that problem is worse with newer bigger disks. The long rebuild times make greater redundancy more desirable. BTRFS vs ZFS Both BTRFS and ZFS checksum all data to cover the case where a disk returns corrupt data, they don t need a fsck program, and the combination of checksums and built-in RAID means that they should have less risk of data loss due to a second failure during rebuild. ZFS supports RAID-Z which is essentially a RAID-5 with checksums on all blocks to handle the case of corrupt data as well as RAID-Z2 which is a similar equivalent to RAID-6. RAID-Z is quite important if you don t want to have half your disk space taken up by redundancy or if you want to have your data survive the loss or more than one disk, so until BTRFS has an equivalent feature ZFS offers significant benefits. Also BTRFS is still rather new which is a concern for software that is critical to data integrity. I am about to install a system to be a file server and Xen server which probably isn t going to be upgraded a lot over the next few years. It will have 4 disks so ZFS with RAID-Z offers a significant benefit over BTRFS for capacity and RAID-Z2 offers a significant benefit for redundancy. As it won t be upgraded a lot I ll start with Debian/Wheezy even though it isn t released yet because the system will be in use without much change well after Squeeze security updates end. ZFS on Wheezy Getting ZFS to basically work isn t particularly hard, the ZFSonLinux.org site has the code and reasonable instructions for doing it [1]. The zfsonlinux code doesn t compile out of the box on Wheezy although it works well on Squeeze. I found it easier to get a the latest Ubuntu working with ZFS and then I rebuilt the Ubuntu packages for Debian/Wheezy and they worked. This wasn t particularly difficult but it s a pity that the zfsonlinux site didn t support recent kernels. Root on ZFS The complication with root on ZFS is that the ZFS FAQ recommends using whole disks for best performance so you can avoid alignment problems on 4K sector disks (which is an issue for any disk large enough that you want to use it with ZFS) [2]. This means you have to either use /boot on ZFS (which seems a little too experimental for me) or have a separate boot device. Currently I have one server running with 4*3TB disks in a RAID-Z array and a single smaller disk for the root filesystem. Having a fifth disk attached by duct-tape to a system that is only designed for four disks isn t ideal, but when you have an OS image that is backed up (and not so important) and a data store that s business critical (but not needed every day) then a failure on the root device can be fixed the next day without serious problems. But I want to fix this and avoid creating more systems like it. There is some good documentation on using Ubuntu with root on ZFS [3]. I considered using Ubuntu LTS for the server in question, but as I prefer Debian and I can recompile Ubuntu packages for Debian it seems that Debian is the best choice for me. I compiled those packages for Wheezy, did the install and DKMS build, and got ZFS basically working without much effort. The problem then became getting ZFS to work for the root filesystem. The Ubuntu packages didn t work with the Debian initramfs for some reason and modules failed to load. This wasn t necessarily a show-stopper as I can modify such things myself, but it s another painful thing to manage and another way that the system can potentially break on upgrade. The next issue is the unusual way that ZFS mounts filesystems. Instead of having block devices to mount and entries in /etc/fstab the ZFS system does things for you. So if you want a ZFS volume to be mounted as root you configure the mountpoint via the zfs set mountpoint command. This of course means that it doesn t get mounted if you boot with a different root filesystem and adds some needless pain to the process. When I encountered this I decided that root on ZFS isn t a good option. So for this new server I ll install it with an Ext4 filesystem on a RAID-1 device for root and /boot and use ZFS for everything else. Correct Alignment After setting up the system with a 4 disk RAID-1 (or mirror for the pedants who insist that true RAID-1 has only two disks) for root and boot I then created partitions for ZFS. According to fdisk output the partitions /dev/sda2, /dev/sdb2 etc had their first sector address as a multiple of 2048 which I presume addresses the alignment requirement for a disk that has 4K sectors. Installing ZFS deb http://www.coker.com.au wheezy zfs I created the above APT repository (only AMD64) for ZFS packages based on Darik Horn s Ubuntu packages (thanks for the good work Darik). Installing zfs-dkms, spl-dkms, and zfsutils gave a working ZFS system. I could probably have used Darik s binary packages but I think it s best to rebuild Ubuntu packages to use on Debian. The server in question hasn t gone live in production yet (it turns out that we don t have agreement on what the server will do). But so far it seems to be working OK.

• [1] http://zfsonlinux.org/
• [2] http://zfsonlinux.org/faq.html
• [3] https://github.com/dajhorn/pkg-zfs/wiki/HOWTO-install-Ubuntu-to-a-Native-ZFS-Root-Filesystem

Related posts:

1. Discovering OS Bugs and Using Snapshots I m running Debian/Unstable on an EeePC 701, I ve got an...
2. Starting with BTRFS Based on my investigation of RAID reliability [1] I have...
3. ZFS vs BTRFS on Cheap Dell Servers I previously wrote about my first experiences with BTRFS [1]....

Last night I played with an Apple Macbook Pro with the new Retina Display (Wikipedia link). Wikipedia cites some controversy about whether the display actually has higher resolution than the human eye can perceive. When wearing glasses my vision is considerably better than average (I have average vision without glasses) and while kneeling in front of the Macbook I couldn t easily distinguish pixels. So Apple s marketing claims seem technically correct to me. When I tested the Macbook Pro I found that the quality of the text display was very high, even now the 1680*1050 display on my Thinkpad T61 looks completely crap when compared to the 2880*1800 display on the Macbook. The Macbook was really great for text and for a JPEG that was installed on the system for testing. But unfortunately pictures on web sites didn t look particularly good. Pictures on my blog looked quite poor and pictures returned by a Google search for art didn t look that great either. I wonder if Safari (the Apple web browser) isn t properly optimised for the display or if there is something that we should do when preparing pictures for web sites to make them look better on Safari. The retina display has a 71% greater DPI which means 2.93* the total number of pixels of my Thinkpad. The overall quality of the experience for me (apart from web pictures) seems more like a factor of 2.93 when compared with my Thinkpad than a factor of 1.71. This has to be one of the most desirable products I ve seen from a company that s opposed to freedom for it s users. I m not about to buy one though, $2,300 is a lot of money for a system that can t be upgraded, repaired, or recycled, and doesn t even have an Ethernet port. I m sure that if I bought one I would discover that it some of the hardware features don t work properly with Linux. The new Apple design trend of making it impossible to repair anything works reasonably well for phones and tablets which are cheap enough that they are hardly worth repairing when they have been used for a while. Lots of people can afford to spend about $600 on something that may be discarded after a year or two, but very few people can afford to spend more than $2,000 on such a disposable product. Why is Apple the only company producing systems with such displays? If someone produced regular PCs that have the expected features (including an Ethernet port) with such a display at a lower price then I m sure that there would be a great demand. Related posts:

1. Modern Laptops Suck One of the reasons why I m moving from a laptop...
2. A First Digital Camera I ve just been asked for advice on buying a digital...
3. RPC and SE Linux One ongoing problem with TCP networking is the combination of...

The New York Times has an interesting article about hacker hostels [1]. I had an idea for similar things after watching a Japanese movie about Tokiwa-s a shared apartment for Manga artists which among others inspired the creator of Astro Boy [2]. The TED blog has an interesting interview with William Noel about open access to art and historical data [3]. Most of his talk concerns an Archimedes codex which has been recovered and published on the Internet. He advocates publishing all manner of art and historical data under a Creative Commons license. The education system is often criticised for trying too hard to make children feel successful and not teaching them skills needed to be successful, it seems that the US military fails the same way in it s war games [4]. Webroot.com published an interesting article last year about the first BIOS rootkit in the wild [5]. I really wish that they would design motherboards with a switch to enable BIOS writing which would default to off . I recently did a poll at a LUG meeting and found that only half the audience had updated the BIOS on most systems they owned, if the most technical people generally don t need a dangerous feature then it should probably be disabled by default. Matthew Wright wrote an interesting article about the costs of upgrading the electricity grid in Australia vs the costs of upgrading air-conditioners [6]. It seems that it would be a lot cheaper for the government to buy everyone a new air-conditioner than to upgrade the grid. Owl City has a post of 10 Myths About Introverts [7]. That could probably be titled 10 Myths About Aspies and still be correct. Susan Cain gave the most popular talk of TED 2012 about The Power of Introverts , here is an interesting interview about the talk and Introversion [8].

• [1] http://tinyurl.com/d8h3pxs
• [2] http://en.wikipedia.org/wiki/Tokiwa-s
• [3] http://blog.ted.com/2012/05/29/the-wide-open-future-of-the-art-museum-qa-with-william-noel/
• [4] http://www.overcomingbias.com/2012/05/war-games-are-fake.html
• [5] http://blog.webroot.com/2011/09/13/mebromi-the-first-bios-rootkit-in-the-wild/
• [6] http://beyondzeroemissions.org/media/newswire/short-circuiting-electric-price-shock-120607
• [7] http://owlcityblog.com/2011/06/27/10-myths-about-introverts/
• [8] http://blog.ted.com/2012/06/15/its-ok-to-eat-alone-qa-with-susan-cain/

Related posts:

1. Links April 2012 Karen Tse gave an interesting TED talk about how to...
2. Links July 2011 The Reid Report has an article about the marriage pledge...
3. Links March 2012 Washington s Blog has an informative summary of recent articles about...

A copy of this post is sent to soc-coordination@lists.alioth.debian.org as well as to debian-bootstrap@lists.mister-muffin.de.

Diary

July 2

• playing around with syntactic dependency graphs and how to use them to flatten dependencies

July 4

• make work with dose 3.0.2
• add linux-amd64 to source architectures
• remove printing in build_compile_rounds
• catch Not_found exception and print warning
• use the whole installation set in crosseverything.ml instead of flattened dependencies
• detect infinite loop and quit in crosseverything.ml
• use globbing in _tags file
• use wildcards and patsubst in makefile

July 5

• throw a warning if there exist binary packages without source packages
• add string_of_list and string_of_pkglist and adapt print_pkg_list and print_pkg_list_full to use them
• fix and extend flatten_deps - now also tested with Debian Sid

July 6

• do not exclude the crosscompiled packages from being compiled in crosseverything.ml
• clean up basebuildsystem.ml, remove old code, use BootstrapCommon code
• clean up basenocycles.ml, remove unused code and commented out code
• add option to print statistics about the generated dependency graph
• implement most_needed_fast_wrong as well as most_needed_slow_correct and make both available through the menu

July 7

• allow to investigate all scc, not only the full graph and the scc containing the investigated package
• handle Not_found in src_list_from_bin_list with warning message
• handle the event of the whole archive actually being buildable
• replace raise Failure with failwith
• handle incorrectly typed package names
• add first version of reduced_dist.ml to create a self-contained mini distribution out of a big one

July 8

• add script to quickly check for binary packages without source package
• make Debian Sid default in makefile
• add *.d.byte files to .gitignore
• README is helpful now
• more pattern matching and recursiveness everywhere

July 9

• fix termination condition of reduced_dist.ml
• have precise as default ubuntu distribution
• do not allow to investigate an already installable package

July 10

• milestone: show all cycles in a graph
• add copyright info (LGPL3+)

July 11

• advice to use dose tools in README

July 16

• write apt_pkg based python filter script replacing grep-dctrl

July 17

• use Depsolver.listcheck more often
• add dist_graph.ml
• refactor dependency graph code into its own module

July 18

• improve package selection for reduced_dist.ml
• improve performance of cycle enumeration code

July 20

• implement buildprofile support into dose3

July 22

• let dist_graph.ml use commandline arguments

July 23

• allow dose3 to generate source package lists without Build- Depends Conflicts -Indep

July 29

• implement crosscompile support into dose3

Results

Readme There is not yet a writeup on how everything works and how all the pieces of the code work together but the current README file provides a short introduction on how to use the tools.

• build and runtime dependencies
• compile instructions
• execution examples for each program
• step by step guide how to analyze the dependency situation
• explanation of general commandline options

A detailed writeup about the inner workings of everything will be part of a final documentation stage.

License All my code is now released under the terms of the LGPL either version 3, or (at your option) any later version. A special linking exception is made to the license which can be read at the top of the provided COPYING file. The exception is necessary because Ocaml links statically, which means that without that exception, the conditions of distribution would basically equal GPL3+.

reduced_dist.ml Especially the Debian archive is huge and one might want to work on a reduced selection of packages first. Having a smaller selection of the archive would be significantly faster and would also not add thousands of packages that are not important for an extended base system. I call a reduced distribution a set of source packages A and a set of binary packages B which fulfill the following three properties:

• all source packages A must be buildable with only binary packages B being available
• all binary packages B except for architecture:all packages must be buildable from source packages A

The set of binary packages B and source packages A can be retrieved using the reduced_dist program. It allows to either build the most minimal reduced distribution or one that includes a certain package selection. To filter out the package control stanzas for a reduced distribution from a full distribution, I originally used a call to grep-dctrl but later replaced that by a custom python script called filter-packages.py. This script uses python-apt to filter Packages and Sources files for a certain package selection.

dist_graph.ml It soon became obvious that there were not many independent dependency cycle situation but just one big scc that would contain 96% of the packages that are involved in build dependency cycles. Therefor it made sense to write a program that does not iteratively build the dependency graph starting from a single package, but which builds a dependency graph for a whole archive.

Cycles I can now enumerate all cycles in the dependency graph. I covered the theoretical part in another blog post and wrote an email about the achievement to the list. Both resources contain more links to the respective sourcecode. The dependency graph generated for Debian Sid has 39486 vertices. It has only one central scc with 1027 vertices and only eight other scc with 2 to 7 vertices. All the other source and binary packages in the dependency graph for the archive are degenerate components of length one. Obtaining the attached result took 4 hours on my machine (Core i5 @ 2.53GHz). 1.5 h of that were needed to build the dependency graph, the other 2.5 hours were needed to run johnson's algorithm on the result. Memory consumption of the program was at about 700 MB. It is to my joy that apparently the runtime of the cycle finding algorithm for a whole Debian Sid repository as well as the memory requirements are within orders of magnitude that are justifiable when being run on off-the-shelf hardware. It must also be noted that nothing is optimized for performance yet. A list of all cycles in Debian Sid up to length 4 can be retrieved from this email. This cycle analysis assumes that only essential packages, build-essential and dependencies and debhelper are available. Debhelper is not an essential or build-essential package but 79% of the archive build-depends on it. The most interesting cycles are probably those of length 2 that need packages that they build themselves. Noticeable examples for these situations are vala, python, mlton, fpc, sbcl and ghc. Languages seem love to need themselves to be built.

Buildprofiles There is a long discussion of how to encode staged build dependency information in source packages. While the initial idea was to use Build-Depends-StageN fields, this solution would duplicate large parts of the Build-Depends field, which leads to bitrot as well as it is inflexible to possible other build "profiles". To remedy the situation it was proposed to use field names like Build-Depends[stage1 embedded] but this would also duplicate information and would break with the rfc822 format of package description files. A document maintained by Guillem Jover gives even more ideas and details. Internally, Patrick and me decided for another idea of Guillem Jover to annotate staged build dependencies. The format reads like:

Build-Depends: huge (>= 1.0) [i386 arm] <!embedded !bootstrap>, tiny

So each build profile would follow a dependency in <> "brackets" an have a similar format as architecture options. Patrick has a patch for dpkg that implements this functionality while I patched dose3.

Dropping Build-Depends-Indep and Build-Conflicts-Indep When representing the dependencies of a source package, dose3 concatenates its Build-Depends and Build-Depends-Indep dependency information. So up to now, a source package could only be compiled, if it manages to compile all of its binary packages including architecture:all packages. But when bootstrapping a new architecture, it should be sufficient to only build the architecture dependent packages and therefor to only build the build-arch target in debian/rules and not the build-indep target. Only considering the Build-Depends field and dismissing the Build-Depends-Indep field, reduced the main scc from 1027 vertices to 979 vertices. The amount of cycles up to length four reduced from 276 to 206. Especially the cycles containing gtk-doc-tools, doxygen, debiandoc-sgml and texlive-latex-base got much less. Patrick managed to add a Build-Depends-Indep field to four packages so far which reduced the scc further by 14 vertices down to 965 vertices. So besides staged build dependencies and cross building there is now a third method that can be applied to break dependency cycles: add Build-Depends-Indep information to them or update existing information. I submitted a list of packages that have a binary-indep and/or a build-indep target in their debian/rules to the list. I also submitted a patch for dose3 to be able to specify to ignore Build-Depends-Indep and Build-Conflicts-Indep information.

Dose3 crossbuilding So far I only looked at dependency situations in the native case. While the native case contains a huge scc of about 1000 packages, the dependency situation will be much nicer when cross building. But dose3 was so far not able to simulate cross building of source packages. I wrote a patch that implements this functionality and will allow me to write programs that help analyze the cross-situation as well.

Debconf Presentation Wookey was giving a talk at debconf12 for which I was supplying him with slides. The slides in their final version can be downloaded here

Future Patrick maintains a list of "weak" build dependencies. Those are dependencies that are very likely to be droppable in either a staged build or using Build-Depends-Indep. I must make use of this list to make it easier to find packages that can easily be removed of their dependencies. I will have to implement support for resolving the main scc using staged build dependencies. Since it is unlikely that Patrick will be fast enough in supplying me with modified packages, I will need to create myself a database of dummy packages. Another open task is to allow to analyze the crossbuilding dependency situation. What I'm currently more or less waiting on is the inclusion of my patches into dose3 as well as a decision on the buildprofile format. More people need to discuss about it until it can be included into tools as well as policy. Every maintainer of a package can help making bootstrapping easier by making sure that as many dependencies as possible are part of the Build-Depends-Indep field.

This screencast shows what I built. Scroll down for my Yesod braindump. <video controls="controls" src="http://joeyh.name/screencasts/git-annex-webapp.ogg"></video> video link I've been astonished how quickly this went together. This is my first time using any sort of web framework, and I used a still unusual one, Yesod. It's my first time using Bootstrap too. It's also the first time I've done any AJAX programming! Bootstrap was something I'd heard of and seen some suspiciously similar looking sites built with, but it was really a pleasant surprise. Being able to make things that work and look good on the web without struggling with CSS is such a nice change. For the first time, it makes the web feel like a UI toolkit to me. As I've learned Yesod, I kept seeing problems the web framework solves that I either solved in an ad-hoc basis in Ikiwiki, or didn't realize were problems. In the former group are things like having a consistent way to store a URL to redirect to after an action (such as logging in) finishes, and generating urls programmatically in ways that avoid broken links. In the latter group are things like exploiting types to guarantee mistakes can't be made. And also the abstraction of widgets, which combine html, css, and javascript, in a way that can be composed with other widgets. Overall, I'm really enjoying Yesod, and it's making me productive in new ways. I also see a lot of potential in Yesod to improve from where it is.

• Probably the biggest improvement would be generating javascript from haskell code, so the type safety extends to the javascript side. There have been many tries at doing this, the first one I'm really tempted to use is Fay. Rather than build a heavy haskell runtime on top of javascript, it essentially uses javascript as its runtime, and offloads type checking to GHC. This lets it generate javascript code that is short and often comprehensible enough to give insights into what the original haskell code does. I'm betting this will be integrated into Yesod eventually. They have an active wiki page about it.
• There's a WAI library for building local webapps with Yesod, but it was not suitable for my needs (for one thing, it lacks security; for another it kills the haskell program when the web page is closed); so I built my own webapp library. A problem with my current pace of development is that I'm building lots of reusable libraries, but I don't have the time to stabalize them and make them generally available. That one goes in the pile of 2k+ lines of such code.
• Yesod needs a version of the Hamlet markup that can be edited by people who only understand html. That means it should allow closing tags, and tabs, and not have nasty whitespace gotchas. I think this would be easy to build, starting from Hamlet. It could be called "Hecate".. I don't have time right now.
• The compile time error messages are often beyond atrocious. Seriously, I'm tempted to write a filter to filter out common patterns where there's one line about a syntax error in a Hamlet file sandwitched in between 150 lines of type error gobbly-gook and generated code.
• Some really nice things could be done integrating Yesod with Bootstrap. Like the ability to build entire sites by just composing together Bootstrap components, with no need to write a single line of html or css. I'm very tempted to try to build this library.

    webpage = bootstrap Dynamic $ do
        setTitle "FooCorp"
        login <- getLogin
        navbar [FixedTop] $ do
            brand "FooCorp"
            link AboutR
            link BlogR
            nav [PullRight] $
                link . maybe LoginR ProfileR login
        div [ContainerFluid] $ content login
        where
            content Nothing = heroUnit $ do
                para $ "We're the FooCorp for you."
                button "Register Today" [Primary, Large] SignUpR
                carousel
                    [ amazingFeatures
                    , aboutFooCorp
                    , pricing
                    ]
            content (Just user)  = do
                para $ "Welcome back " ++ name user ++ "!"
                showProfile user

during the last weeks I was quite busy with other things (like DebCamp, DebConf, & vacations), so this is a report covering 4 weeks. at least I managed to catch up during the last days a bit

• #639565 liboauth-dev: "liboauth-dev: please make loose libcurl dependency"
  send a patch to the BTS
• #642165 src:libmime-lite-html-perl: "libmime-lite-html-perl: FTBFS: test failures"
  update test reference files, upload to DELAYED/2
• #656505 taxbird: "taxbird: build-dependency on libgeier-dev needs to be versioned"
  propose a different patch in the BTS
• #662606 php-horde-core: "php-horde-core: FTBFS: "This package contains role "horde" and requires package "horde/Role" to be used""
  add missing build dependency, upload to DELAYED/2
• #664724 rrdcached: "rrdcached: fails to install, postinst, invoke-rc.d rrdcached start, start-stop-daemon, segfault"
  (re-)create directories needed by daemon in init script, upload to DELAYED/2
• #665018 src:python-fixtures: "python-fixtures: FTBFS: AttributeError: 'BrokenFixture' object has no attribute 'items'"
  add patch from Emanuele Aina, upload to DELAYED/2
• #666231 src:byzanz: "byzanz: FTBFS with gtk 3.4"
  sponsor NMU from Etienne Millon, upload to DELAYED/2
• #678108 httpcomponents-core: "httpcomponents-core: FTBFS twice in a row - broken clean rule"
  call mh_clean as suggested by Niels Thykier, upload to DELAYED/2
• #678859 libmozjs185-dev: "libmozjs185-dev depends on libffi-dev"
  add missing dependency as proposed by Nicolas Boulenguez, upload to DELAYED/2
• #679669 src:underscore: "underscore: build-dependency on uglifyjs not satisfiable in wheezy"
  propose a patch in the BTS, maintainer prefers alternative ways of fixing it
• #680423 trac-wikiprint: "trac-wikiprint: depends on nonexisting python-pil if rebuilt"
  propose a patch in the BTS
• #680424 trac-odtexport: "trac-odtexport: depends on nonexisting python-pil if rebuilt"
  propose a patch in the BTS, then fixed & closed by maintainer
• #681146 bitlbee-common: "bitlbee-common: fails to install, purge, and install again"
  propose a patch in the BTS, then upload to DELAYED/2 with maintainer's permission
• #681567 opennebula-common: "opennebula-common: fails to install, purge, and install again"
  propose a patch in the BTS, then upload to DELAYED/2 with maintainer's permission

I've been in A Coru a for this year's GUADEC since Tuesday night, and it rocked. I did a late registration after my first week at Collabora, which is sponsoring my stay here.

I came one day early to participate, as Debian's representative, at the yearly GNOME Advisory Board meeting, for the first time. It was a positive experience, which helped me get a grasp of the big picture of what the GNOME Foundation does. I also had the pleasure of visiting Igalia's awesome offices in the city, and puting faces to many names during the meeting. I presented an overview of Debian's relation to GNOME, how our packaging team works and what are our goals and biggest problems as a GNOME downstream. We stirred some good debate as some other Advisory Board members share part of our problems. I should be posting a summary of what happened there for debian-project@ldo as soon as I have the time to scribble it. I've met with GNOME Hispano people I hadn't seen since 2004 or 2006 in the best case, and catched up with many of them. I've also met many GPUL members I had know for over a decade via IRC, but never had met in person, and it was about time. And of course, I've got to known a good number of my new workmates at Collabora, and had fun with them around the conference, the beach and the numerous post-conference events. Last, but not least, I ended up participating in the GNOME Olympics, substituting Rodrigo in Team B Core Dumped , along with Stefano, John, Bastien, Chema and Adam. WE WON, not thanks to me, but the statistics shine: I've won all FreeFA World Cups I've played :P so here's a PROtip: if you want to win next year, be sure to be my team mate, and more importantly, be sure Adam is not your rival. :) Unfortunately, I'm only attending the core days so tonight I'll be flying back to Madrid on my way home in Val ncia. See you next year! A Coru a is a city that has impressed me quite a bit, and I'm looking forward to coming back for some more standard vacation at some point. :)

Brought to you by the studio that brought you buzz, rex, bo, hamm, slink, potato, woody, sarge, etch, lenny, squeeze and wheezy.

Brought to you by the studio that brought you buzz, rex, bo, hamm, slink, potato, woody, sarge, etch, lenny, squeeze and wheezy.

Earlier this year, in March, I switched to the bluetile tiling window, and it has been a great success. So I was interested in seeing Vincent Bernats post about switching to Awesome, I myself intend to do that "eventually", now I've successfully dipped my toes into tiling-land, via bluetiles simple config and gnome-friendly setup. One thing that puts me off is the length of my sessions:

skx@precious:~/hg/blog$ ps -ef   grep skx
skx       2237     1  0 Mar12 ?        00:00:01 /usr/bin/gnome-keyring-daemon ...

As you can see I've been logged into my desktop session for four months. I lock the screen when I wander away, but generally login once when the computer boots and never again for half a year or so. FWIW:

skx@precious:~/hg/blog$ uptime
 23:01:19 up 138 days,  4:44,  4 users,  load average: 0.02, 0.06, 0.02

ObQuote: "I'm 30 years old. I'm almost a grown man. " - Rocketman

I have happily used FVWM as my window manager for more than 10 years. However, I recently got tired of manually arranging windows and using the mouse so much. A window manager is one of the handful pieces of software getting in your way at every moment which explains why there are so many of them and why we might put so much time in it. I decided to try a tiling window manager. While i3 seemed pretty hot and powerful (watch the screencast!), I really wanted something configurable and extensible with some language. So far, the common choices are:

• awesome, written in C, configurable and extensible in Lua,
• StumpWM, written, configurable and extensible in Common Lisp,
• xmonad, written, configurable and extensible in Haskell.

I chose awesome, despite the fact that StumpWM vote for Lisp seemed a better fit (but it is more minimalist). I hope there is some parallel universe where I enjoy StumpWM. Visually, here is what I got so far:

Awesome configuration Without a configuration file, awesome does nothing. It does not come with any hard-coded behavior: everything needs to be configured through its Lua configuration file. Of course, a default one is provided but you can also start from scratch. If you like to control your window manager, this is somewhat wonderful. awesome is well documented. The wiki provides a FAQ, a good introduction and the API reference is concise enough to be read from the top to the bottom. Knowing Lua is not mandatory since it is quite easy to dive into such a language. I have posted my configuration on GitHub. It should not be used as is but some snippets may be worth to be stolen and adapted into your own configuration. The following sections put light on some notable points.

Keybindings Ten years ago was the epoch of scavanger hunts to recover IBM Model M keyboards from waste containers. They were great to type on and they did not feature the infamous Windows keys. Nowadays, this is harder to get such a keyboard. All my keyboards now have Windows keys. This is a major change with respect to configure a window manager: the left Windows key is mapped to Mod4 and is usually unused by most applications and can therefore be dedicated to the window manager. The main problem with the ability to define many keybindings is to remember the less frequently used one. I have monkey-patched awful.key module to be able to attach a documentation string to a keybinding. I have documented the whole process on the awesome wiki.

Quake console A Quake console is a drop-down terminal which can be toggled with some key. I was heavily relying on it in FVWM. I think this is still a useful addition to any awesome configuration. There are several possible solutions documented in the awesome wiki. I have added my own¹ which works great for me.

XRandR XRandR is an extension which allows to dynamically reconfigure outputs: you plug an external screen to your laptop and you issue some command to enable it:

$ xrandr --output VGA-1 --auto --left-of LVDS-1

awesome detects the change and will restart automatically. Laptops usually come with a special key to enable/disable an external screen. Nowadays, this key does nothing unless configured appropriately. Out of the box, it is mapped to XF86Display symbol. I have associated this key to a function that will cycle through possible configurations depending on the plugged screens. For example, if I plug an external screen to my laptop, I can cycle through the following configurations:

• only the internal screen,
• only the external screen,
• internal screen on the left, external screen on the right,
• external screen on the left, internal screen on the right,
• no change.

The proposed configuration is displayed using naughty, the notification system integrated in awesome.

Widgets I was previously using Conky to display various system-related information, like free space, CPU usage and network usage. awesome comes with widgets that can fit the same use. I am relying on vicious, a contributed widget manager, to manage most of them. It allows one to attach a function whose task is to fetch values to be displayed. This is quite powerful. Here is an example with a volume widget:

local volwidget = widget(  type = "textbox"  )
vicious.register(volwidget, vicious.widgets.volume,
         '<span font="Terminus 8">$2 $1%</span>',
        2, "Master")
volwidget:buttons(awful.util.table.join(
             awful.button(   , 1, volume.mixer),
             awful.button(   , 3, volume.toggle),
             awful.button(   , 4, volume.increase),
             awful.button(   , 5, volume.decrease)))

You can also use a function to format the text as you wish. For example, you can display a value in red if it is too low. Have a look at my battery widget for an example.

Miscellaneous While I was working on my awesome configuration, I also changed some other desktop-related bits.

Keyboard configuration I happen to setup all my keyboards to use the QWERTY layout. I use a compose key to input special characters like . I have also recently use Caps Lock as a Control key. All this is perfectly supported since ages by X11 I am also mapping the Pause key to XF86ScreenSaver key symbol which will in turn be bound to a function that will trigger xautolock to lock the screen. Thanks to a great article about extending the X keyboard map with xkb, I discovered that X was able to switch from one layout to another using groups². I finally opted for this simple configuration:

$ setxkbmap us,fr '' compose:rwin ctrl:nocaps grp:rctrl_rshift_toggle
$ xmodmap -e 'keysym Pause = XF86ScreenSaver'

I switch from us to fr by pressing both left Control and left Shift keys.

Getting rid of most GNOME stuff Less than one year ago, to take a step forward to the future, I started to heavily rely on some GNOME components like GNOME Display Manager, GNOME Power Manager, the screen saver, gnome-session, gnome-settings-daemon and others. I had numerous problems when I tried to setup everything without pulling the whole GNOME stack. At each GNOME update, something was broken: the screensaver didn t start automatically anymore until a full session restart or some keybindings were randomly hijacked by gnome-settings-daemon. Therefore, I have decided to get rid of most of those components. I have replaced GNOME Power Manager with system-level tools like sleepd and the PM utilities. I replaced the GNOME screensaver with i3lock and xautolock. GDM has been replaced by SLiM which now features ConsoleKit support³. I use ~/.gtkrc-2.0 and ~/.config/gtk-3.0/settings.ini to configure GTK+. The future will wait.

Terminal color scheme I am using rxvt-unicode as my terminal with a black background (and some light transparency). The default color scheme is suboptimal on the readability front. Sharing terminal color schemes seems a popular activity. I finally opted for the derp color scheme which brings a major improvement over the default configuration. I have also switched to Xft for font rendering using DejaVu Sans Mono as my default font (instead of fixed) with the following configuration in ~/.Xresources:

Xft.antialias: true
Xft.hinting: true
Xft.hintstyle: hintlight
Xft.rgba: rgb
URxvt.font: xft:DejaVu Sans Mono-8
URxvt.letterSpace: -1

The result is less crisp but seems a bit more readable. I may switch back in the future.

Next steps My reliance to the mouse has been greatly reduced. However, I still need it for casual browsing. I am looking at luakit a WebKit-based browser extensible with Lua for this purpose.

---

1. The console gets its own unique name. This allows awesome to reliably detect when it is spawned, even on restart. It is how the Quake console works in the mod of FVWM I was using.
2. However, the layout is global, not per-window. If you are interested by a per-window layout, take a look at kbdd.
3. Nowadays, you cannot really survive without ConsoleKit. Many PolicyKit policies do not rely on groups any more to grant access to your devices.