When packaging software for Debian, there exist two important assumptions:

1. Compilation is done natively
2. Potentially all of Debian is available at compile time

Both assumptions make the life of a package maintainer much easier and they do not create any problem unless you are one of the unlucky few who want to run Debian on an architecture that it does not yet exist for. You will then have to use other distributions like OpenEmbedded or Gentoo which you compiled (or retrieved otherwise) for that new architecture to hack a core of Debian source packages until they build a minimal Debian system that you can chroot into and continue natively building the rest of it. But even if you manage to get that far you will continue to be plagued by cyclic build and runtime dependencies. So you start to hack source packages so that they drop some dependencies and you can break enough cycles to advance step by step. The Debian ports page lists 24 ports of Debian, so despite its unpleasant nature, porting it is something that is not done seldom. The process as laid out above has a number of drawbacks:

• The process is mostly manual and reinvented every time it is done.
• You need another distribution for the bootstrapping process. Debian itself should be sufficient.
• Its complexity and manual nature prevents architectures with little workforce behind them from catching up to the main archive.
• It also avoids that Debian exists in CPU optimized sub-arch builds.

If Debian would provide a set of core packages that are cross-compilable and which suffice for a minimal foreign build system, and if it would also have enough source packages that provide a reduced build dependency set so that all dependency cycles can be broken, building Debian for a yet unknown architecture could be mostly automated. The benefits would be:

• Putting Debian on a foreign architecture would (in the best case) boil down to making the code cross-compile for and native-compile on that architecture.
• Debian would not need any other distribution to be ported to a different architecture. This would make Debian even more "universal".
• Lagging architectures can be more easily updated or rebooted than when they were initially created.
• Debian optimized for specific CPUs (Raspberry Pie, OpenMoko...) would be more attractive.

With three of this year's GSoC projects, this dream seems to come into reach. There is the "Multiarch Cross-Toolchains" project by Thibaut Girka and mentored by Hector Oron and Marcin Juszkiewicz. Cross-compiling toolchains need packages from the foreign architecture to be installed alongside the native libraries. Cross-compiler packages have been available through the emdebian repositories but always were more of a hack. With multiarch, it is now possible to install packages from multiple architectures at once, so that cross-compilation toolchains can be realized in a proper manner and therefor can also enter the main archives. Besides creating multiarch enabled toolchains, he will also be responsible for making them build on the Debian builld system as cross-architecture dependencies are not yet supported. There is also the "Bootstrappable Debian" project by Patrick "P. J." McDermott and mentored by Wookey and Jonathan Austin. He will make a small set of source packages multiarch cross-compilable (using cross-compilers provided by Thibaut Girka) and add a Build-Depends-StageN header to critical packages so that they can be built with reduced build dependencies for breaking dependency cycles. He will also patch tools as necessary to recognize the new control header. And then there is my project: "Port bootstrap build-ordering tool" (Application). It is mentored by Wookey and Pietro Abate. In contrast to the other two, my output will be more on the meta-level as I will not modify any actual Debian package or patch Debian tools with more functionality. Instead the goal of this project is threefold:

1. find the minimal set of source packages that have to be cross compiled
2. help the user to find packages that are good candidates for breaking build dependency cycles through added staged build dependencies or by making them cross-compilable
3. develop a tool that takes the information about packages that can be cross compiled or have staged build dependencies to output an ordering with which packages must be built to go from nothing to a full archive

More on that project in my follow-up post.

Users of Debian sid will have noticed: the final (and interesting) bits of GNOME 3.4 have landed and if all looks as good as it does now, they should migrate to wheezy in about a week. 3.2 3.4 hasn't been as complicated as the previous horrible transition, but still had some complications due to Cogl/Clutter incompatibilities. Other than that, our major problem has been manpower, but this isn't new for many other Debian teams. We've also seen new incarnations of Linux-only technology is now mandatory which makes our lives a bit more miserable due to kfreebsd-* and hurd-i386, but for now we've still been able to dodge it. It seems wheezy+1 will be fun in that regard though, and we might need to take drastic approaches. If all goes well and the current lot (GNOME Shell, Control Center, Settings daemon, Mutter...) transitions without additional problems, we should be wrapping up our transitions for wheezy with Evolution and friends (currently sitting in experimental), and hopefully GDM 3.4. As we get many questions regarding the status of GDM in Debian, let's add a short note on this. Packaging GDM, at least in its current upstream form, is not a matter of unpacking a new tarball and editing debian/changelog. When Joss works on a new major version, the amount of tweaking to break away from stuff that works on other distros but is not so simple in Debian is outstanding (see, for example, the current unfinished work for GDM 3.2 in our SVN repo). In our case, to handle our GDM defaults, we even need changes to the underlying configuration system, dconf. This evidently takes some effort to do, and unfortunately our GDM expert has had little time for Debian lately, but we're confident we'll end up with a GDM in wheezy that is on par with Debian standards. We are, as always, reachable at #debian-gnome in the OFTC IRC network. Have fun!

In case this went unnoticed, Iceweasel ESR has been available in squeeze-backports for a few weeks, now. I highly recommend anyone using Iceweasel on the Debian stable release to upgrade to that version, at the very least. Even newer versions are available through the pkg-mozilla archive.

Last year we invited David to work with us for a few days to add a generic interface to apt to call external solvers. After a few iterations, this patch finally landed in master and recently (about 3 months ago), in debian unstable.

  [ David Kalnischkies ]
  * [ABI-Break] Implement EDSP in libapt-pkg so that all front-ends which
    use the internal resolver can now be used also with external
    ones as the usage is hidden in between the old API
  * provide two edsp solvers in apt-utils:
    - 'dump' to quickly output a complete scenario and
    - 'apt' to use the internal as an external resolver

Today the new version of apt-cudf was upload in unstable and with it the latest bug fixes that makes it ready for daily use. I've used it quite a lot myself to upgrade my machine and it seems working pretty well so far... The most important difference with the old version is the support for multi-arch enabled machines. This marks an important milestone in our efforts to integrate external solvers, built using different technologies, directly into apt. From a user prospective, this means that (s)he will have the possibility to check if there exists a better (best ?) solution to an installation problem then what proposed by the internal apt solver. Moreover, even if apt-get gives very satisfactory answers, there are occasions where it fails miserably, leaving the user wondering how to unravel the complex web of dependencies to accomplish his task. Available cudf solvers in debian are at the moment : aspcud, mccs and packup. From an architectural point of view this is accomplished by abstracting the installation problem via a simple textual protocol (EDSP) and using an external tool to do the heavy duty translation. Since all solvers now available in debian are not meant to be debian-specific, using them involve a two step translation. The EDSP protocol specification is for the moment "hidden" in the apt documentation. I hope to find a better place for it soon : it would be cool if other package managers as smart or cupt could add an implementation on EDSP in their code so to automatically benefit of this technology. To make it happen, Apt first creates an EDSP file that is then handled to apt-cudf that takes care of the translation to cudf and back into EDSP that is then read back by apt. Apt-cudf is the bridge between edsp and the external solvers and takes care of doing the book keeping and to select the right optimization criteria. Roberto recently wrote a very nice article explaining how to use apt-get with an external solver. In a nutshell, if you want to try this out you just need to install apt-cudf, one external solver like aspcud from the university of Pasdam and then call apt-get using the --solver option (that is not yet documented #67442 ). For example : This will install gnome while using the a optimization criteria that tries to minimizing the changes on the system. Various other optimization criteria for all apt-get default actions can be specified in the apt-cudf configuration file /etc/apt-cudf.conf I hope the new release of apt-cudf make it into testing before the freeze. Time to test !!!

This is my monthly summary of my Debian related activities. If you re among the people who made a donation to support my work (338.26 , thanks everybody!), then you can learn how I spent your money. Otherwise it s just an interesting status update on my various projects. Dpkg Like last month, I did almost nothing concerning dpkg. This will probably change in June now that the book is out The only thing worth noting is that I have helped Carey Underwood who was trying to diagnose why btrfs was performing so badly when unpacking Debian packages (compared to ext4). Apparently this already resulted in some btrfs improvements. But not as much as what could be hoped. The sync_file_range() calls that dpkg are doing only force the writeback of the underlying data and not of the meta-data. So the numerous fsync() that follow still create many journal transactions that would be better handled as one big transaction. As a proof of this, replacing the fsync() with a sync() brings the performance on par with ext4. (Beware this is my own recollection of the discussion, while it should be close to the truth, it s probably not 100% accurate when speaking of the brtfs behaviour) Packaging I uploaded new versions of smarty-gettext and smarty-validate because they were uninstallable after the removal of smarty. The whole history of smarty in Debian/Ubuntu has been a big FAIL since the start. Once upon a time, there was a smarty package and some plugins. Everything was great except that the files were installed in a way that differs from the upstream recommendations. So Ubuntu changed the path in their version of the package and did not check whether it broke anything else (and it did break all the plugins). Despite the brokenness of the plugins, this divergence survived for years. So several packages that were using Smarty were modified to use dpkg-vendor to use the correct path depending on whether it was built on Debian or Ubuntu. In 2010, Smarty 3.0 has been released and instead of upgrading the smarty package to this version, one of the smarty co-maintainers introduced a smarty3 package that used yet another path (despite the fact that smarty 3 had a mode to be compatible with smarty 2).
At some point, I informed him that he had to handle the migration of users of smarty to smarty3 he acknowledged and then lost interest in smarty ( I m no longer using it ) and did nothing. After some more bitrot, smarty has been forcefully orphaned in August 2011 by a member of the security team. And in March this year, it has been removed from unstable despite the fact that it still had reverse dependencies (usually removals only happen when they impact no other packages, I don t know why this wasn t the case here). At least the brokenness attracted some attention to the situation and Mike Gabriel contacted me about it. I offered him to take over the various packages since they all needed a real maintainer and he accepted. I sponsored his uploads of all smarty related packages (bringing in the latest upstream versions at the same time). In the end, the situation is looking better now, except that there s no migration path from users who rely on smarty in Squeeze. They will discover that they need smarty3 in Wheezy and that the various paths have to be adjusted. It s probably acceptable since the new upstream versions are no longer backwards compatible with smarty 2 The Debian Administrator s Handbook At the start of the month, I was busy preparing the release of the book. I introduced the publican-debian package to unstable, it s a Publican brand (aka a set of CSS and XSL stylesheets to tailor the output of Publican) using the Debian colors and using the Debian logo. This brand is used by the book. I also created the debian-handbook package and setup the public Git repository on alioth.debian.org. I was ready or so I thought. A few hours after the announce, the website became unusable because the numerous visitors were exhausting the maximum number of client connections. And I could not increase the limit due to Apache s memory usage (with PHP and WordPress). We quickly off-loaded most of the static files traffic to another machine and we setup bittorrent. The problem was solved for the short term. Thousands of persons downloaded the ebook and to this date, 135 copies of the paperback have been sold. Then I took a one-week vacation. Even though I had no Internet at the place I was, I wandered in the street to find a Freewifi wifi network (customers of the Free ISP can use those freely) to stay on top of incoming email. We quickly received some bug reports and I dealt with the easy ones (typos and the like) on the fly. When I came back at home, I manually placed 54 lulu orders for the people who opted for the paperback as reward during the fundraising campaign. A bit tedious but it had to be done (if only Lulu supported a way to batch many orders at once ). I also wanted a long term solution to avoid the use of an external host to serve static files (should a new traffic spike arrive ). So I installed nginx as a front-end. It serves static files directly, as well as WordPress pages which have been cached by wp-super-cache. Apache is still here listening on a local port and responding to the remaining queries forwarded by nginx. Once I ll migrate to wheezy, I might completely ditch apache in favor of php5-fpm to handle the PHP pages. Last but not least, I wanted to bootstrap the various translations that people offered to contribute. I wrote some documentation for interested translators and blogged about it. It s shaping up nicely check it out if you re interested to help! Thanks See you next month for a new summary of my activities.

No comment Liked this article? Click here. My blog is Flattr-enabled.

In the past days, I looked at various memory allocators in a quest to improve performance in my multi-threaded test cases of a reference counting language runtime / object allocator (a fun project). It turns out the glibc s memory allocator is relatively slow, especially if you do loops that create one element and destroy one element at the same time (for example, map() on a list you no longer use after you passed it to map). To fix this problem, I considered various options. The first option was to add a thread-local cache around malloc(). So whenever we want to free() an object, we place it on a thread-local list instead, and if a malloc() request for an object of that size comes in, we reuse the old object. This fixes the problem with the lists, but experiments shown another problem with the glibc allocator: Allocating many objects without releasing others (let s say appending an element to a functional list). I started testing with tcmalloc instead, and noticed that it was several times faster (reducing run time from 6.75 seconds to 1.33 seconds (5 times faster)). As I do not want to depend on a C++ code base, I decided to write a simple allocator that allocates blocks of memory via mmap(), splits those into objects, and puts the objects on a local free list. That allocator performed faster than tcmalloc(), but was also just a simple test case, and not really useable, due to potential fragmentation problems (and because the code was statically linked in, causing thread-local storage to be considerably faster, as it does not need to call a function on every TLS access, but can rather go through a segment register). I then discovered jemalloc, and tried testing with this. It turned out that jemalloc was even faster than tcmalloc in all test cases, and also required about the same amount of memory as tcmalloc (and slightly more than the custom test code, as that combined reference counting with memory allocator metadata and thus had less meta data overhead). Using jemalloc, the list building test performs in 0.9X seconds now (instead of tcmalloc s 1.33 or glibc s 6.75 seconds), requires 0 major page faults (tcmalloc has 2), and uses 5361472 kb memory (tcmalloc uses 5290240, glibc requires 7866608). Given that jemalloc is written in C, I can only recommend it to everyone in need of a scalable memory allocator. Depending on your workload, it might require less memory than tcmalloc (at least that s the case at Facebook) and is most likely faster than tcmalloc. It also provides tcmalloc-compatible profiling facilities (as Facebook needed them). Furthermore, jemalloc is also the memory allocator used by FreeBSD, and is used by Mozilla projects, and on several Facebook projects, and should thus be relatively stable and useable. All tests were run with 4 threads, on a dual-core laptop with hyper-threading, using (e)glibc 2.13, tcmalloc 2.0, and jemalloc 3.0. The tests may not be representative of real world performance, and do not account for fragmentation. Should I try replacing Python s custom caching of malloc() calls with simply calling jemalloc, and run a few Python benchmarks? Anyone interested in that?
Filed under: General

Traffic jams suck; but they do give you a chance to think and blog. Truckers hams are telling me that four trucks are totalled and need to be removed by the THW. As the trucker next to me just broke out his gas grill and several people are joining him in a spontaneous BBQ, I don't expect to be moving any time soon. Kicking things along This is coming along nicely RAID 4 Unless you are forced to support legacy systems or playing RAIDemon and gotta try them all, there is no valid use case for RAID 4. So doing so is a patently bad idea. RAID 5 is always the better option as you are not relying on a single disk which contains the checksums, but use striping to distribute them. RAID 4 has a chance of 1/n that a failed disk will force all remaining disks to read all respective sectors and then the controller/CPU to XOR everything. There is a reason why RAID 3 was replaced with RAID 4; just as there's a reason why RAID 4 was replaced by RAID 5. Don't use RAID 4. Seriously. Tom Tom vs OSM In case you have been following the news in the GPS circles, you will have noticed that TomTom pretty gave up on innovating and resigned themselves to dying a slow death. Well that and to start a nice amount of FUD. While they have been rebuked pretty thoroughly already, I wanted to add my own POV. There is a kernel of truth in what TomTom said inasmuch they state that pedestrian paths are better than car paths. Of course, while they got the relative quality right, they are wrong about the absolute quality of the data. They claim pedestrian is OK and motorized is bad. In my experience, maps for motorized traffic are good while pedestrian tracks are mind-boggingly great. These days, most people (and probably all who read this) will be at least somewhat familiar with the quality of OSM in the random cities they visit over the year. The common verdict is usually "very nice". But that's not where OSM shines. It's in the lesser-known places where you don't expect too many tech-minded people to come along where it starts to really impress me. While I found a few stubs on a dead-end street in Akureyri which were not mapped, every foot trail I ever walked along in Iceland has been mapped. ski-doo tracks on Svalbard? Check. Backyard in Novosibirsk? Check, as well. And we are talking "packed earth track in between garages opening up into a 'playground' where a man and a pregnant women are passing a bottle back and forth while smoking and watching over their two kids who play next to a middle-aged man who is lying on the ground wasted, using a rock as a pillow all the while an extremely blingy Mercedes with two muscle-packed thugs is cruising by, all in the time it took us to walk through". Mapped. Horse tracks and hiking trails in Mongolia? Yep. Hutongs in Beijing including alleys so narrow you have to walk single file? Of course. Richard Fairhurst is right that the plural of anecdote is not data, but OSM is good. TomTom is afraid for a reason. As an aside, I find the picture used in TomTom's newsletter extremely funny; I am sure they didn't intend this, but hey. Windows 100-300 MiB for any random driver installation package is par for the course. Be it GFX, audio, or network card. If you plug in a different USB mouse, Windows 7 will take several seconds while "setting up" a standard USB HID device. And hitting ctrl-alt-del in the login screen or while logged in does not give you a quick way to shut down or reboot the machine. As I am breaking out the Windows installation every month to install updates and every six months to actually use this does not affect me a lot. Still, it's amazing that so many people are putting up with this. UEFI It's nice to see that Matthew Garrett is still going strong. I am not sure if this is the best way, but it seems to be a workable solution while in between a rock and a hard place. Not sure about the requirements of GPLv3 with regards to the keys Fedora will use to sign GRUB 2 with. From what I remember without rereading the license, as long as Fedora is not selling a system on which you can not disable secure boot they should be in the clear. IANAL, IIRC, etc. Wrapping up And I have not moved a single meter, other than making way to a police motorbike, in over an hour. Pity I don't have anything to BBQ with me; smells delicious over there.

I ve expanded a bit on my idea (that is, the sbuilder wrapper to help aid with managing more then one sbuild chroot), and it got out of hand very quickly. It s gone from one small shell script to a set of 8, and growing. Quickly. Until I spin these off into their own repo, they ll be in my dot-bin Before getting too far, please relize this is only how I m using sbuild. There are many other (more) sane configurations that you can (and should) use. This was not made to be useful for anyone other then me :) I ve added the ability to use unionfs-fuse, set up a way to login interactively and a script to create new chroots without having them register as a global schroot. Firstly, forgive me for my sins. This is all a huge hack, but it s use isn t the worst thing in the world. As of the time of this writing, I now have a gcc and clang chroot for each of stable, testing and unstable. Now the big question how do you use this. Rather, this is more of a how do I use sbuilder :) Firstly, I ve created a file for each schroot set that I m using. Here s my /etc/schroot/chroot.d/sid.

[sid]
type=directory
description=Debian sid/amd64 autobuilder
directory=/var/sbuild/sid
groups=root,sbuild
root-groups=root,sbuild
script-config=sbuild/config

It s very similar to the one that sbuilder-createchroot creates. If you re just looking for aufs / unionfs support, keep in mind schroot has some great support. check man 5 schroot.conf. Next, create a dist.d directory in /var/sbuild such as /var/sbuild/sid.d. In the future, this might get automagically created by the sb-create script. For now, it just expects that exists. The arguments for all the sb* commands is always in the order of variant, dist. variant defaults to generic , which I ve set as an alias to the generic chroot, and the dist defaults to sid . At some point, I ll change the default to be the last UNRELEASED entry in the changelog. So, to create a new chroot to become your clang chroot, you may choose to invoke it as sb-create clang, or more explicitly sb-create clang sid. To create a new chroot to become your stable / gcc chroot, you may create it with sb-create gcc stable. After you have a chroot, you can start to use the sb or sbo scripts. sbo is just a wrapper around the sb script that sets an envvar to tell the sb-prehook and sb-posthook to set up the unionfs overlay. The unionfs overlay routine replaces the dist symbolic link (such as /var/sbuild/testing) with a folder, and mounts the real chroot (such as /var/sbuild/testing.d/testing-gcc) under the unionfs divert point (which is something like /var/sbuild/testing.overlay.d/testing-gcc). This lets us make sure the chroot is never affected by a routine build, even if it s designed to trash a chroot (unless, of course, it uses any number of ways to get outside the chroot and trash the pristine directory this is to prevent mistakes, not malicious scripts). By the way, for those interested, there s a small issue with what I m describing it s very subtle. Basically, there s a few assumptions made by how schroot operates, and handles sessions . Since it ll keep stuff mounted if the session is still active, it makes purging all this a real pain. The magic behind the scripts is in the sb-endsessions routine which will properly end all the schroot sessions for that dist, and since we have our own high-level locking system, we shouldn t have a situation where this kills off another build. The code (which might have issues) looks like:

SESSIONS= schroot --all-sessions -l   grep "^session:$DIST-" 
for session in $SESSIONS; do
    echo "W: Ending session $session"
    schroot -e -c $session
done

By backing of schroot s mounts, we can do our mounting / removing without worrying about klobbering schroot. Now, there s also the case of wanting to interactively log into a chroot. For that, there s sb-login and sb-maint, both take the usual variant / dist arguments. sb-login logs in as the current user, and sb-maint logs in as the root user. Eventually, I ll be adding some scripts to upgrade all the build chroots for a dist at once.

Hello!

I'm Sorina and this summer I will be working on Debian Installer network setup during Google Summer of Code. I am a second year bachelor's student at Politehnica University of Bucharest, I am studying Computer Science. I very much enjoy what I am doing even if sometimes I feel that the amount of stuff I want/have to learn/do/work on is infinite and I don't really know where to start.

I am at the beginning of the project, I believe now I am accustomed to the way things work and I am waiting forward to finishing the exam period so that I could really start working.

The hardest part for me in what I've done so far is testing what I am doing. It took me a while to figure out at least a part of how the installer functions and to solve the problems that occurred while building. Luckily, there is documentation and internet and even more important, I had help and support from my mentors.

So far I have only written two patches for netcfg, for bugs #610752 and #636211. For the first one I had a version for a while now, but I was unable to test it (unfortunately, it's likely that you will hear this a lot in the future). The idea was to make the link detection timeout preseedable and for this I have added two new templates, asking the user to provide a value for that matter.

The second one involved a missing "Go Back" button. This seemed to be the result of not letting debconf know that the program is capable of handling the user pressing a back button. That could be solved using deb_capb.

As I said before, testing was my main problem. Eventually, I built the image, wrote it to an USB and rebooted my laptop. This served its purpose, since I had access to anything I needed such as the wireless interface and physical access to the network cable. Running it through a virtual machine did not help much in this case, because I was unable to provide those.

Another thing I considered to be hard was blog posting, but now that I have just finished my first post it seems that I overreacted.

That's all for now, have a nice day!

I like to think that archive rebuilds play an important role in Debian Quality Assurance and Release Management efforts. By trying to rebuild every Debian package from source, one can identify packages that do not build anymore due to changes in other packages (compilers, interpreters, libraries, ). It is also a good way to stress-test all packages that are involved in building other packages. Since 2007, I had been running Debian archive rebuilds on the Grid 5000 testbed, a research infrastructure for performing experiments on distributed systems HPC/Grid/Cloud/P2P. I filed more than 6000 release-critical bugs in the process. Late last year, Amazon kindly offered us a grant to allow us to run such QA tests on Amazon Web Services. With S bastien Badia, we ported the rebuild infrastructure to AWS (scripts), and several rebuilds have already been carried out on AWS. On the technical level, 50 to 100 EC2 spot instances are started, and then controlled from a master instance using SSH. On build instances, a classic sbuild setup is used. Logs are retrieved to the master node after rebuilds, and build instances are simply shut down when there are no more tasks to process. Several tasks are processed simultaneously on each instance, and when they fail, they are retried again with no other concurrent build on the same instance, to eliminate random failures caused by load or timing issues. All the scripts are designed to support other kind of QA tests, not just rebuilds. Moving to Amazon Web Services will facilitate sharing the human workload of doing those tests. It is now possible for developers interested in custom tests to do them themselves (hint hint).

i dislike having to run Windows as much as the next free software developer, but like many sysadmins, i am occasionally asked to maintain some legacy systems. A nice way to keep these systems available (while not having to physically maintain them) is to put them in a virtual sandbox using a tool like kvm. While kvm makes it relatively straightforward to install WinXP from a CD (as long as you have the proper licensing key), it is more challenging to transition a pre-existing hardware windows XP installation into a virtual instance, due to Windows only wanting to boot to ide chipsets that it remembers being installed to. In particular, booting a disk image pulled from a soon-to-be-discarded physical disk can produce a Blue Screen of Death (BSOD) with the message:

Stop error code 0x0000007B

or

(INACCESSABLE_BOOT_DEVICE)

This seems like it's roughly the equivalent (in a standard debian GNU/Linux environment) of specifying MODULES=dep in /etc/initramfs-tools/initramfs.conf, and then trying to swap out all the hardware. At first blush, Microsoft's knowledge base suggests doing an in-place upgrade or full repartition and reinstall, which are both fairly drastic measures -- you might as well just start from scratch, which is exactly what you don't want to have to do for a nursed-along legacy system which no one who originally set it up is even with the organization any more. Fortunately, a bit more digging in the Knowledge Base turned up an unsupported set of steps that appears to be the equivalent of setting MODULES=most (at least for the IDE chipsets). Running this on the old hardware before imaging the disk worked for me, though i did need to re-validate Windows XP after the reboot by typing in the long magic code again. i guess they're keying it to the hardware, which clearly changed in this instance. Such silliness to spend time working around, really, when i'd rather be spending my time working on free software. :/

The new version of Lintian (2.5.8) can pretty much be summed up as:

Its like 2.5.7, only with less false positives and no FTBFS.

Especially people annoyed by the hardening flags will hopefully find that 2.5.8 greatly reduces the number of false positives. I believe this is best demonstrated with an example:

$ lintian --print-version && lintian -q -C binaries amarok_2.5.0-1_i386.deb   wc -l 
2.5.7
94
[...]
$ lintian --print-version && lintian -q -C binaries amarok_2.5.0-1_i386.deb   wc -l
2.5.8
4

However, nothing comes for free. We dropped hardening-no-stackprotector from the default profile (and demoted it to an I tag). For hardening-no-fortify-functions we made a false positive -> false negative trade-off by ignoring binaries if their only unprotected function is memcpy. For more information, please refer to #673112. hardening-no-stackprotector is still available and can be used via the debian/extra-hardening profile (or via the tags argument). For the 2.5.7 behaviour of hardening-no-fortify-functions, you have to use hardening-check directly. But 2.5.7 had other changes besides the myriad of false-positives:

• Around 19 redundant tags were removed.

A consequence of this is that we no longer warn if you use things like dpkg assert-working-epoch or your postinst creates a usr/doc transition symlink .

• The last of the Lintian s (perl) modules have been put under the Lintian name space.

With all the Lintian modules under the Lintian namespace, we can install them in the standard perl @INC path. Admittedly, I am not certain we are ready to commit to the current API in these modules, which is one of the reasons why they are still installed in /usr/share/lintian. But in a couple of releases things may look differently.

• Lazy loading of data files

Lintain 2.5.7 ships over 50 data files of various kinds (usually white- or blacklists of some kind). Before 2.5.7, all data files with a few exceptions would be loaded eagerly (i.e. as soon as the check was run for the first time). A few data files had been special cased with manual laziness . In 2.5.7, Lintian::Data was updated to lazily load the data. So depending on the packages being checked, Lintian may now load fewer data files.

• Create proper data files for tables embedded within checks.

While this is not something exclusive to 2.5.7, we have separated quite a few checks from their data tables by now. My personal favorites are the table of known interpreters and (from 2.5.8) the table of known versioned interpreters.

• Support for Vendor specific data files.

I am certainly biased here. But this is probably the most awesome feature in Lintian 2.5.7. It is now possible for vendors to extend or simply shadow the core Lintian data files. Allow me to demonstrate how this can be used:

$ lintian --tags build-depends-on-obsolete-package a2ps_4.14-1.1.dsc 
E: a2ps source: build-depends-on-obsolete-package build-depends: dpatch
$ cat ~/.lintian/profiles/local/main.profile 
Profile: local/main
Extends: debian/main
$ cat  ~/.lintian/vendors/local/main/data/fields/obsolete-packages 
@include-parent
@delete dpatch
bison
$ lintian --profile local/main --tags build-depends-on-obsolete-package a2ps_4.14-1.1.dsc
E: a2ps source: build-depends-on-obsolete-package build-depends: bison
$

It is a toy example, but I believe it is a good demonstration of the feature. The full documentation of Vendor specific data files can be found in the Lintian User manual (in lintian/2.5.7 or newer). It will also be on lintian.debian.org when we find time to update lintian there.

Android support in Gammu is one of frequently requested features. I don't have neither time to motivation to do that, so unless somebody else will step up, it won't be done. Recently, I got offer from MyPhoneExplorer author to share it's applet for communicating with Android phones. It basically provides protocol similar to Sony-Ericsson phones Gammu already supports, just over TCP/IP. So all what would have to be done is glue layer for using TCP/IP and possibly some minor adjustments to Sony-Ericsson code. Both of these should be quite easy tasks and I'm willing to guide you through this process. In case anybody would be interested in this challenge, just let me know :-).

Filed under: English Gammu 1 comments Flattr this!

As we are getting closer to the World IPv6 Day, I thought that I'd see if I can upgrade my WRT54GL's OpenWRT (White Russian) installation to accept Router Adverts from the ISP to acquire an IPv6 address (rather than via a 4-to-6 tunnel broker, which seems to be the most documented case), and to propagate its IPv6 address space to my local network via my WRT45GL's radvd. Unfortunately, most of the instructions out there concern newer OpenWRT releases to which I cannot upgrade to because their software base takes up significantly more space than White Russian does. Still, I figure that someone in the community must have already succeeded at configuring their WRT54GL for this and might be able to help. Anyone? :)

NetSNMP comes with its own event loop based on the select() system call. While you can build your program around it, you may prefer to use an existing event loop, either a custom one or something like libevent, libev or Twisted.

Own custom event loop Let s start with the easiest case: you have written your own event loop. This means you make a call to select(), poll(), epoll(), kqueue() or something similar. All those functions take a set of file descriptors and wait for any of them to become available in a specified time frame. Here is a typical use of select() (adapted from Quagga)

readfd = m->readfd;
writefd = m->writefd;
exceptfd = m->exceptfd;
timer_wait = thread_timer_wait (&m->timer);
num = select (FD_SETSIZE,
              &readfd, &writefd, &exceptfd,
              timer_wait);
if (num < 0)
   
    if (errno == EINTR)
      continue; /* signal received - process it */
    zlog_warn ("select() error: %s",
               safe_strerror (errno));
    return NULL;
   
if (num == 0)
   
    /* Timeout handling */
    thread_timer_process (&m->timer);
   
if (num > 0)
   
    thread_process_fd (&readfd);
    thread_process_fd (&writefd);
   

thread_process_fd (fds) function iterates on each file descriptor fd and executes the appropriate action if FD_ISSET (fds, fd) is true. Integrating NetSNMP into such an event loop is easy: NetSNMP provides the snmp_select_info() function which alters a set of file descriptors to insert its own. Here is the new code:

#if defined HAVE_SNMP
struct timeval snmp_timer_wait;
int snmpblock = 0;
int fdsetsize;
#endif
/* ... */
#if defined HAVE_SNMP
fdsetsize = FD_SETSIZE;
snmpblock = 1;
if (timer_wait)
   
    snmpblock = 0;
    memcpy(&snmp_timer_wait, timer_wait,
           sizeof(struct timeval));
   
snmp_select_info(&fdsetsize, &readfd,
                 &snmp_timer_wait, &snmpblock);
if (snmpblock == 0)
  timer_wait = &snmp_timer_wait;
#endif
num = select (FD_SETSIZE,
              &readfd, &writefd, &exceptfd,
              timer_wait);
if (num < 0)   /* ... */  
#if defined HAVE_SNMP
if (num > 0)
  snmp_read(&readfd);
else if (num == 0)
   
    snmp_timeout();
    run_alarms();
   
netsnmp_check_outstanding_agent_requests();
#endif
/* ... */

snmp_select_info() may modify the provided set of file descriptors (and therefore, its size). It may also modify the provided timer in case it needs to schedule an action before the original timeout. snmpblock is a tricky variable. While select() can be called with a timeout set to NULL, this is not the case of snmp_select_info(): you have to pass a valid pointer. snmpblock is set to 0 if the provided timeout must be considered or to 1 otherwise. snmp_select_info() will set snmpblock to 0 if and only if it alters the timeout. Here are a two examples of integration. Both are for a subagent but the code for a manager is the same:

• Quagga
• Keepalived

Third-party event loop With a third party event loop, you don t have access to the select() system call anymore. Therefore, you cannot alter it with snmp_select_info(). Instead, at each iteration of the loop, a list of SNMP related file descriptors is kept up-to-date with the help of snmp_select_info(): new ones are added and old ones are removed.

libevent Let s assume that snmp_fds is the list of current SNMP related events¹. A function levent_snmp_update() calls snmp_select_info() to update this list. Here is a partial implementation (error handling removed and some declarations omitted, look at the complete version from lldpd):

static void
levent_snmp_update()
 
    int maxfd = 0;
    int block = 1;
    FD_ZERO(&fdset);
    snmp_select_info(&maxfd, &fdset, &timeout, &block);
    /* We need to untrack any event whose FD is not in  fdset 
       anymore */
    for (snmpfd = TAILQ_FIRST(snmp_fds);
         snmpfd;
         snmpfd = snmpfd_next)  
        snmpfd_next = TAILQ_NEXT(snmpfd, next);
        if (event_get_fd(snmpfd->ev) >= maxfd  
            (!FD_ISSET(event_get_fd(snmpfd->ev), &fdset)))  
            event_free(snmpfd->ev);
            TAILQ_REMOVE(snmp_fds, snmpfd, next);
            free(snmpfd);
          else
            FD_CLR(event_get_fd(snmpfd->ev), &fdset);
     
    /* Invariant: FD in  fdset  are not in list of FD */
    for (int fd = 0; fd < maxfd; fd++)
        if (FD_ISSET(fd, &fdset))  
            snmpfd->ev = event_new(base, fd,
                EV_READ   EV_PERSIST,
                levent_snmp_read,
                NULL);
            event_add(snmpfd->ev, NULL);
            TAILQ_INSERT_TAIL(snmp_fds, snmpfd, next);
         
    /* If needed, handle timeout */
    evtimer_add(snmp_timeout, block?NULL:&timeout);
 

Then, replace the main loop (usually, a call to event_base_dispatch()) with this:

do  
    if (event_base_got_break(base)  
        event_base_got_exit(base))
        break;
    netsnmp_check_outstanding_agent_requests();
    levent_snmp_update();
  while (event_base_loop(base, EVLOOP_ONCE) == 0);

Here is how are defined the two callbacks:

static void
levent_snmp_read(evutil_socket_t fd, short what, void *arg)
 
    FD_ZERO(&fdset);
    FD_SET(fd, &fdset);
    snmp_read(&fdset);
    levent_snmp_update();
 
static void
levent_snmp_timeout(evutil_socket_t fd, short what, void *arg)
 
    snmp_timeout();
    run_alarms();
    levent_snmp_update();
 

Twisted reactor As a second example, here is how to integrate NetSNMP into Twisted reactor. Twisted is an event-driven network programming framework written in Python. It does not come with support for SNMP². Because of the language mismatch, the integration of NetSNMP in Twisted needs to be done using a C extension or the ctypes module. Twisted event loop is called a reactor. Like for libevent, events need to be registered. It is possible to register file descriptor-like objects using a class implementing a handful of methods. Here is the implementation of such a class (adapted from PyNetSNMP, a subproject of Zenoss using the ctypes module):

class SnmpReader:
    "Respond to input events"
    implements(IReadDescriptor)
    def __init__(self, fd):
        self.fd = fd
    def doRead(self):
        netsnmp.snmp_read(self.fd)
    def fileno(self):
        return self.fd

Like for libevent, we have a function to update the list of SNMP related events (stored as a mapping between file descriptors and SnmpReader instances):

class Timer(object):
    callLater = None
timer = Timer()
fdMap =  
def updateReactor():
    "Add/remove event handlers for SNMP file descriptors and timers"
    fds, t = netsnmp.snmp_select_info()
    for fd in fds:
        if fd not in fdMap:
            reader = SnmpReader(fd)
            fdMap[fd] = reader
            reactor.addReader(reader)
    current = Set(fdMap.keys())
    need = Set(fds)
    doomed = current - need
    for d in doomed:
        reactor.removeReader(fdMap[d])
        del fdMap[d]
    if timer.callLater:
        timer.callLater.cancel()
        timer.callLater = None
    if t is not None:
        timer.callLater = reactor.callLater(t, checkTimeouts)

Contrary to libevent, we cannot alter the main loop to call updateReactor() at each iteration. Therefore, it must be called after each SNMP related function. For another example, have a look at my equivalent implementation as a C extension.

Miscellaneous

Lack of asynchronicity Integrating NetSNMP into an event-based program is not risk-free. I have written a fairly comprehensive article on the lack of asynchronicity in NetSNMP AgentX protocol implementation. I urge you to read it if you want to integrate a SNMP subagent into an existing program. On the manager side, you will get similar drawbacks when using SNMPv3. To retrieve or manipulate management information using SNMPv3, it is necessary to know the identifier of the remote SNMP protocol engine. This identifier can be configured directly on each manager but it is usually discovered by querying SNMP-FRAMEWORK-MIB:: snmpEngineID, as described in RFC 5343. Unfortunately, this discovery is done synchronously by NetSNMP. There is a bug report for this issue but it seems difficult to fix. I have not been able to come up with a proper patch but I have described a workaround around snmp_sess_async_send(). There is no such problem with SNMPv2.

Limitation of file descriptor sets snmp_select_info() and snmp_read() uses the fd_set type to handle a set of file descriptors. This type should be manipulated with FD_CLR(), FD_ISSET(), FD_SET() and FD_ZERO(). Those may be defined as functions but they usually are macros. Moreover, no file descriptor greater than FD_SETSIZE (which is usually set to 1024) can be handled with the fd_set type. This means that if you have a file descriptor greater than 1024, you won t be able to use snmp_select_info() with it. Starting from NetSNMP 5.5, you can use snmp_select_info2() and snmp_read2() instead of snmp_select_info() and snmp_read(). They use the netsnmp_large_fd_set. If you want to keep compatibility with NetSNMP 5.4, here is another twist. The fd_set type is usually a fixed-size array of long integers. FD_CLR(), FD_ISSET() and FD_SET() are size-independant. The size only matters for FD_ZERO() which is not used by either snmp_select_info() or snmp_read(). You can therefore allocate a larger fd_set. A common way is to compile your program with -D__FD_SETSIZE=4096. You can still use FD_ZERO() yourself. However, this is not portable (GNU C Library only). Another way is to allocate your own array of long integers and cast it to fd_set. You ll have to redefine FD_ZERO():

typedef struct  
   long int fds_bits[4096/sizeof(long int)];
  my_fdset;
#undef FD_ZERO
#define FD_ZERO(fdset) memset((fdset), 0, sizeof(my_fdset))
 
   /* ... */
   my_fdset fdset;
   FD_ZERO(&fdset);
   /* ... */
   rc = snmp_select_info(&n, (fd_set *)&fdset, &timeout, &block);
   /* ... */
 

Threads NetSNMP provides two API:

• the traditional session API which is not thread-safe and
• the single session API.

In the above examples, I have used the traditional session API. If you are using threads, you need to ensure that all SNMP operations are done in a single thread. Otherwise, you need to adapt the examples to use the single session API. snmp_select_info() is part of the traditional session API. You should replace it with snmp_sess_select_info() and keep a list of SNMP sessions. This API is not available for the agent side of NetSNMP.

---

1. The event list is global but some code could be added to bind a different list for each base in case you use different bases. This is not needed in the case of lldpd which uses only one event base.
2. TwistedSNMP is an SNMP protocol implementation for Twisted based on PySNMP but is not maintained anymore. PySNMP comes with examples on how to integrate with Twisted.

Hi! I have been selected to work on multiarch cross-toolchains as part of the Google summer of Code. Although coding officially started last week, I have been busy with exams and other uni stuff at the time. Anyway, let's talk about the project itself, that is, multiarch cross-toolchains. What is multiarch? In order to understand what the project is about, one need to know what multiarch is. I'll quickly describe it, and I invite anyone interested to read the spec (really, it's interesting, and not that long). Basically, multiarch lets you install binary packages for multiple architectures on one system, and let package managers resolve dependencies as one would expect. This works by identifying four sets of packages:

• Packages that don't need to be aware of multi-arch. Such packages can only be installed for one architecture at a time, and can only be used to resolve dependencies of packages having the same architecture. Those packages are Multi-Arch: none or multiarch-unaware packages.
• Packages that can be installed for multiple architectures at a time, and can only be used to resolve dependencies of packages having the same architecture. Those packages are Multi-Arch: same packages. They typically are libs.
• Packages that can only be installed for one architecture at a time but can satisfy dependencies of any package regardless of its architecture. Those packages are Multi-Arch: foreign packages. They typically are tools working with architecture-independent interfaces. (for instance, gpg, zenity or tar).
• Packages that can only be installed for one architecture at a time but can satisfy any dependency annotated with :any , regardless of the architecture. Those packages are Multi-Arch: allowed packages. They are packages providing both arch-dependent and arch-independent interfaces. They are typically things like python, which can be used to interpret arch-indep files, but providing arch-dependent interfaces for C extensions.

Anyway, multiarch is an amazing thing that may be of use in various cases, such as running i386-only software on an amd64 system, running armel software on an armhf system, running i386 software on an armhf system using qemu-user-static and binfmt, cross-grading (switching, for instance, from i386 to amd64 on a running system), and, in our case, easing cross-compilation. State of cross-toolchains in Debian There aren't really cross-toolchains available in Debian (although some can be built from source packages), but they are available in emdebian's repositories. However, those packages predate multiarch, and don't make use of it. Instead, they depend on special packages converted using dpkg-cross. Such packages are merely arch: all versions of foreign packages, with files moved to arch-qualified paths to prevent conflicts with normal packages. Such packages thus have to be generated from real packages, either manually or using some sort of automation. In addition, this additional step brings several issues, like keeping in sync converted packages with real packages from the archive. With multiarch, such packages are rendered useless, and may in fact bring additional problems, since converted and multiarch packages may share the same files, and thus may not be co-installable. Switching from dpkg-cross to cross-arch deps Cross-toolchains packages currently depend on a few packages converted using dpkg-cross (ex: libc6-$ARCH-cross and libc6-dev-$ARCH-cross for some packages). They need to depend on the real package with the right arch (ex: libc6:$ARCH, libc6-dev:$ARCH). The obvious way to do so is by annotating the dependency with a specific arch qualifier (ex: Depends: libc6:armhf ). This is, however, not covered by the multiarch spec. Note that some lib*-$ARCH-cross packages will remain (for instance, libgcc1-armhf-cross). Indeed, those packages are different from the packages mentioned above in that they are of the architecture of the build system and actually part of the toolchain itself.
The above isn't actually true. Indeed, those packages are runtime library cross-built during gcc's build. Likewise, cross-toolchains may build-depend on foreign packages (ex: libc6-dev). Thus, specific arch qualifiers must be handled in the Build-Depends source package field too. Changing dpkg and apt to support that should be fairly straightforward. In fact, I already have a few patches, but I need to refresh them, maybe clean them up a little, and push them a little harder to get them accepted!

In 2003, a German teacher showed up on the Debian Edu and Skolelinux mailing list with interesting problems and reports proving he setting up Linux for a (for us at the time) lot of pupils. His name was Ralf Gesellensetter, and he has been an important tester and contributor since then, helping to make sure the Debian Edu Squeeze release became as good as it is.. Who are you, and how do you spend your days? I am a teacher from Germany, and my subjects are Geography, Mathematics, and Computer Science ("Informatik"). During the past 12 years (since 2000), I have been working for a comprehensive (and soon, also inclusive) school leading to all kind of general levels, such as O- or A-level ("Abitur"). For quite as long, I've been taking care of our computer network. Now, in my early 40s, I enjoy the privilege of spending a lot of my spare time together with my wife, our son (3 years) and our daughter (4 months). How did you get in contact with the Skolelinux/Debian Edu project? We had tried different Linux based school servers, when members of my local Linux User Group (LUG OWL) detected Skolelinux. I remember very well, being part of a party celebrating the Linux New Media Award ("Best Newcomer Distribution", also nominated: Ubuntu) that was given to Skolelinux at Linux World Exposition in Frankfurt, 2005 (IIRC). Few months later, I had the chance to join a developer meeting in Ulsrud (Oslo) and to hand out the award to Knut Yrvin and others. For more than 7 years, Skolelinux is part of our schools infrastructure, namely our main server (tjener), one LTSP (today without thin clients), and approximately 50 work stations. Most of these have the option to boot a locally installed Skolelinux image. As a consequence, I joined quite a few events dealing with free software or Linux, and met many Debian (Edu) developers. All of them seemed quite nice and competent to me, one more reason to stick to Skolelinux. What do you see as the advantages of Skolelinux/Debian Edu? Debian driven, you are given all the advantages of a community project including well maintained updates. Once, you are familiar with the network layout, you can easily roll out an entire educational computer infrastructure, from just one installation media. As only free software (FOSS) is used, that supports even elderly hardware, up-sizing your IT equipment is only limited by space (i.e. available labs). Especially if you run a LTSP thin client server, your administration costs tend towards zero. What do you see as the disadvantages of Skolelinux/Debian Edu? While Debian's stability has loads of advantages for servers, this might be different in some cases for clients: Schools with unlimited budget might buy new hardware with components that are not yet supported by Debian stable, or wish to use more recent versions of office packages or desktop environments. These schools have the option to run Debian testing or other distributions - if they have the capacity to do so. Another issue is that Debian release cycles include a wide range of changes; therefor a high percentage of human power seems to be absorbed by just keeping the features of Skolelinux within the new setting of the version to come. During this process, the cogs of Debian Edu are getting more and more professional, i.e. harder to understand for novices. Which free software do you use daily? LibreOffice, Wikipedia, Openstreetmap, Iceweasel (Mozilla Firefox), KMail, Gimp, Inkscape - and of course the Linux Kernel (not only on PC, Laptop, Mobile, but also our SAT receiver) Which strategy do you believe is the right one to use to get schools to use free software?

1. Support computer science as regular subject in schools to make people really "own" their hardware, to make them understand the difference between proprietary software products, and free software developing.
2. Make budget baskets corresponding: In Germany's public schools there are more or less fixed budgets for IT equipment (including licenses), so schools won't benefit from any savings here. This privilege is left to private schools which have consequently a large share among German Skolelinux schools.
3. Get free software in the seminars where would-be teachers are trained. In many cases, teachers' software customs are respected by decision makers rather than the expertise of any IT experts.
4. Don't limit ourself to free software run natively. Everybody uses free software or free licenses (for instance Wikipedia), and this general concept should get expanded to free educational content to be shared world wide (school books e.g.).
5. Make clear where ever you can that the market share of free (libre) office suites is much above 20 p.c. today, and that you pupils don't need to know the "ribbon menu" in order to get employed.
6. Talk about the difference between freeware and free software.
7. Spread free software, or even collections of portable free apps for USB pen drives. Endorse students to get a legal copy of Libreoffice rather than accepting them to use illegal serials. And keep sending documents in ODF formats.

I just come across a blog post from Glyn Moody reporting the claimed cost from Microsoft on requiring ODF to be used by the UK government. I just sent him an email to let him know that his assumption are most likely wrong. Sharing it here in case some of my blog readers have seem the same numbers float around in the UK.

Hi. I just noted your http://blogs.computerworlduk.com/open-enterprise/2012/04/does-microsoft-office-lock-in-cost-the-uk-government-500-million/index.htm comment:

"They're all in Danish, not unreasonably, but even with the help of Google Translate I can't find any figures about the savings of "moving to a flexible two standard" as claimed by the Microsoft email. But I assume it is backed up somewhere, so let's take it, and the 500 million figure for the UK, on trust."

I can tell you that the Danish reports are inflated. I believe it is the same reports that were used in the Norwegian debate around 2007, and Gisle Hannemyr (a well known IT commentator in Norway) had a look at the content. In short, the reason it is claimed that using ODF will be so costly, is based on the assumption that this mean every existing document need to be converted from one of the MS Office formats to ODF, transferred to the receiver, and converted back from ODF to one of the MS Office formats, and that the conversion will cost 10 minutes of work time for both the sender and the receiver. In reality the sender would have a tool capable of saving to ODF, and the receiver would have a tool capable of reading it, and the time spent would at most be a few seconds for saving and loading, not 20 minutes of wasted effort. Microsoft claimed all these costs were saved by allowing people to transfer the original files from MS Office instead of spending 10 minutes converting to ODF. :) See http://hannemyr.com/no/ms12_vl02.php and http://hannemyr.com/no/ms12.php for background information. Norwegian only, sorry. :)

Email spam is a plague, but sometimes there is at least on category of spammers it is easy to have fun with: naive manual spammers. People that try to do more or less legal business, using Internet in an illegitimate way, for instance (this is a real example, including the spammer's address: spamming bots, please use it ;-) !):

From: Emma Perroni <emmaperroni@gmail.com>
Subject: Hello
Hello,
Do you have a Facebook page? If you do, I can offer you to add fans by
hunders/thouthands in a single day using an original (and confidential)
technique. We can also add subscribers to Twitter, Youtube or Google+1.
Prices: 10  for 100 fans, -20% for more than 1000 fans ordered.
Regards

Reporting Well, in that case it may or may not be a scam, but even if it is not a criminal activity such as selling fake drugs, it is not a very honest business. Anyway, that lady is using her Google account to send her crap, so neutralizing her is quite easy:

1. checking that it really comes from Google Mail, using DKIM or SPF;
2. writing to Google, reporting that spam (and, in that specific case, the abuse they are offering to commit on Google +1);
3. optional, do not do that with scammers: replying to the spammer explaining that this is spamming, that you reported it and that she can expect to loose her Google account and everything on it.

That should work with any good email provider, that is, any provider that listens to complaints at <abuse@> and takes action on them. For what I have seen, Google and Yahoo! are, but fossils such as traditional national telephonists are not. If the message headers allow you to identify it, you can also report the abuse to the spammer's network operator if it is a good one (this is rarely the case). Consequences With a bit of luck, that can seriously damage the business of the spammer, and if he was really careless (e.g. storing important documents on the same Google account with no local backup), cause serious trouble in his personal stuff. If you replied to the spammer, you can get fun answers such as (these are taken from real examples I got):

• Ha ha ha, why not calling the FBI? That's it, laugh while you can, do you think your provider does not care? Too bad, he does, so you may change of mood when you see the effects loosing one's online account.
• We are doing legal business, you will stop your harrassment or we will file a complain. Yes, please do so if you dare, but may I remember you that you are the one at fault here? (in fact, such a reaction is a very good sign, since it shows that the spammer is actually afraid, probably for his business)
• Why did you do that? I promise, I shall not do it again, please Too late, lamer, you should have thought of that before sending your crap

Conclusion This is too complicated to be done for each spam of course, but it is worth doing for spam messages that are not catch by one's antispam. And the rules for email marketing are well known: only write to people that explicitly allowed you to do so, full stop, so there is no reason to show any mercy for a spammer.

Fedora 17 will be shipping next week. It's got a bunch of new features, none of which I contributed to in the slightest. What I did work on was improving our support for installation on x86 Apple hardware. There's still a few shortcomings in this so it's not an announced or supported feature, but it's sufficient progress that it's worth writing about.

There's a few ways that Apple platforms differs from normal PC hardware. The first is that Apples are very much intended to be EFI first and BIOS second, while that's still not really true for commodity PC hardware. Apple launched Boot Camp shortly after they started shipping x86 Macs (and shortly after I'd got Ubuntu running on one for the first time[1]) but the focus of Boot Camp has always been to be good enough to boot Windows and not much else[2]. The other is the integration with the boot environment. You can boot Linux easily enough by setting the standard EFI boot variables, but if you ever boot back into OS X it's easy to trigger deletion of those. There's then no straightforward way of resetting them, and you're left having to recover your installation.

The other difficult bit has been actually starting the installer at all. If you're happy to use BIOS emulation than this can be done without too much misery, but you're then left with dealing with Apple's custom synchronised GPT/MBR. More modern Macs will boot via the standard EFI mechanisms, but there's a range of problems that can be caused that way. Fedora 17 is the first Linux distribution to ship install media that will EFI boot a Mac when either written to optical media or a USB stick[3]. Put the install media in your system and then either select it in the OS X Startup Disk preferences menu and reboot, or reboot and hold down the alt key. In the latter case there'll be a nice Fedora logo. Click on it and it'll boot.

The biggest difference between Apple hardware and anyone else is that we'd normally put the bootloader in a FAT partition that's shared with any other installed operating systems. That does actually work on Apples, but then you run into the earlier point I mentioned. The Apple startup picker that you get by holding down the alt key doesn't pay any attention to the standard EFI boot variables, so it won't show a FAT partition merely because it's got an EFI bootloader on it. The same is true of the OS X Startup Disk preferences. The best way to get a drive to appear in the menu is to make it look like an OS X drive.

This was problematic in a few ways, due to the requirement for an OS X drive to be HFS+. The first was that we didn't have any support for that in our installer, so work had to be done there. The second was that the state of HFS+ was woefully poor in Linux[4]. Linux will refuse to write to an HFS+ filesystem if it hasn't been cleanly unmounted or fscked, and since we can't rely on everyone always shutting their machine down cleanly that means a working fsck.hfsplus. In theory we were shipping one of those. In practice, it reliably segfaulted on 64-bit systems[5]. So I had to package the latest version of Apple's code, and doing that involved dealing with the fact that Apple now use blocks almost as much as Grub 2 uses nested functions, and that meant using Clang and that meant dealing with a bug where Clang segfaulted if it had been built with gcc 4.7[6], but finally once all of those yaks had been shaven we could trust our filesystem.

Of course, more problems existed. You can't just drop a bootloader onto an HFS+ partition and expect it to work. You need to write its inode value into the superblock. Trivial, except that if you do this from userspace then the kernel kindly overwrites your update when you unmount the disk and it updates the superblock. That required a mechanism for updating that data via the kernel, which meant adding an ioctl. This would have been fine, except the OS X Startup Disk preference looks for a bootloader in a location other than where Fedora installs it. Changing our bootloader install path wasn't a great option, so the easiest thing to do was just to link them together. Except HFS+ doesn't really support hardlinks. When you hardlink something the original file gets moved into a magic directory in the filesystem root and the links are special files that refer to it. And it turns out that it's not actually the inode that the firmware wants when finding the bootloader, it's the catalogue ID, and these aren't the same when hardlinks are involved. So, another kernel patch.

All that was left for the boot partition then was adding an icon and a drive label. We already had code for generating the icon, and the drive label wasn't that difficult. Hurrah!

Ok, so we can construct a filesystem that (a) appears in the bootpicker, and (b) appears in the OS X Startup Disk preferences. We can even put a bootloader on it. This led to the next problem. The bootloader started without any trouble. But the bootloader couldn't read any files off the boot partition, including its configuration. This should have been obvious with hindsight - the problem was simply that grub legacy[7] doesn't support HFS+. Writing an HFS+ driver seemed like an excessive amount of work, especially since the firmware already supported reading HFS+, so instead I wrote a simple grub filesystem driver that uses the UEFI interfaces to read files.

A working bootloader! One more problem there. grub looks in its startup directory to find its configuration file. We have two "copies" of grub hardlinked to each other, each looking in a different directory for configuration. Hardlinking the config files together worked fine, except that most tools that update config files have this irritating habit of writing to a temporary file and moving that over the original, thus breaking the hardlink. Easily solved by using a symlink instead, except that the UEFI filesystem semantics don't really support symlinks because UEFI only really envisaged using FAT. If you use Apple's UEFI HFS+ driver to open a symlink, you get a short file containing the target path of the symlink. Rather less than ideal, and a rather gross hack to cope with it.

All set now, other than a bug in some older Mac firmware where read would return BUFFER_TOO_SMALL without returning the real buffersize and the odd way that Apple treat CDs. Oh, and a bug in Apple's Broadcom wifi driver that could overwrite the kernel, and case-sensitivity, something that obviously isn't something you often hit on UEFI when all you usually see is FAT.

Kernel-side, we had a couple of things. Obviously there the bugs I'd mentioned in the past, but those had already been dealt with. New issues included the fact that we were frequently picking the wrong framebuffer address, especially on machines with multiple GPUs. Fixed now, along with another issue where we'd sometimes get confused by models with different GPU configurations. And at that point, most of the implementation work was done.

Of course, there was also the work of integrating this into the tools that we use to generate our install media, and I'd like to thank Will Woods, Brian Lane and Dennis Gilmore for the work they put into that, along with anyone I've forgotten. Tom Calloway handled getting the device label graphics generated right before deadline. A cast of thousands helped with testing.

So why did I mention shortcomings? First, this only works on machines with 64-bit firmware. Early 64-bit Apples still had 32-bit firmware. In theory we can support that case - in practice it's a moderate amount of writing, including some mildly awkward kernel code. But anything later than mid-2007 should have 64-bit firmware, so it's not a massive concern. Second, we seem to have fairly significant trouble with Radeon hardware on a lot of Macs. We fixed one bug there, but something's still going wrong in setup and you'll frequently end up with a black screen. And thirdly, I'm sure there's some other machines that still don't work for (as yet) undetermined reasons.

With luck we'll get these things dealt with by Fedora 18. But even with these flaws, Fedora 17 will work fine on a lot of Apple hardware, and testing it is as simple as downloading and booting a live image. Bugs go in the usual place.

[1] Note my charming belief that the Ubuntu installer would fully support this hardware via EFI in the near future. 6 years ago. It still doesn't.
[2] This can be fairly easily seen from little things like the name "Windows" being hardcoded into every UI that sees a Boot Camp drive.
[3] I cover this in more depth here
[4] To be fair, it mostly still is - there's plenty of work to be done there.
[5] Approximately nobody had noticed this, which is all kinds of unreassuring.
[6] Thanks to Kalev Lember for dealing with that
[7] Still the EFI bootloader in Fedora 17. We've already swapped Fedora 18 over to grub 2.

comments