One practice that seems relatively common is for an organisation to have two main mailing lists, one for serious discussions that are expected to be relatively topical and another for anything that s not overly offensive. Humans are inherently incapable of avoiding social chatter when doing serious work. The people who don t want certain social interactions with their colleagues can find it annoying to have both social and serious discussions on the same list. While the people who want social discussions get annoyed when people ask them to keep discussions on topic. Organisations that I have been involved with have had mailing lists such as foo-chat and foo-talk for social discussions that involve the same people as the main list named foo , as well as having list names such as memo-list for random discussions that are separate from a large collection of lists which demand on-topic messages. The Debian project has some similar issues with the debian-private mailing list which is virtually required reading for Debian Developers. One complication that Debian has is that the debian-private list has certain privacy requirements (messages will be declassified after 3 years unless the author requests that they remain secret forever) which make it more difficult to migrate a discussion. You can t just migrate a discussion from a private list to a public list without leaking some information. So it seems to me that the best solution might be to have a list named debian-private-chat which has the same secrecy requirements but which is not required reading. As debates about what discussions are suitable for debian-private have been going on for more than 3 years I don t think there s any reason not to publish the fact that such discussions take place. Also it seems that every organisation of moderate scale that has a similar use of email and for which members socialise with each other could benefit from a similar mailing list structure. Note that I use a broad definition of the word socialise there s a lot of people who will never meet in person and a lot of the discussions are vaguely related to the main topic. I wonder whether it might be considered to be a best practice to automatically create a chat list at the same time as creating a serious discussion list.

This last weeks, I've worked on implementing the Desktop Notification Specification into Emacs. It allows sending desktop notification in a very simple way.

(notifications-notify
    :title "You've got mail!"
    :body "There's 34 mails unread"
    :app-icon "~/.emacs.d/icons/mail.png"
    :urgency 'low)

It supports the protocol signals (NotificationClosed and ActionInvoked) and the two main methods (Notify and CloseNotification). The methods specification are implemented entirely (hints, replaces, actions, icon, etc). The signals are supported via callbacks function provided on the notification creation. It have been merged into Emacs trunk today.

2010-06-09  Julien Danjou  <julien@danjou.info>
        * net/notifications.el: New file.

This also allowed me to discover, raise and fix a bug in the D-Bus binding of Emacs, which will be probably fixed in trunk soon.

Early this morning I sent Rcpp version 0.8.1 off to CRAN and Debian. In the meantime, Romain has already provided a very nice blog post about it. There are a few fairly visible new things in this release. As we want to focus the next few minor releases on completing the documentation, we started by adding a total of four (!!) new vignettes:

• Rcpp-package showing how to write your own package using Rcpp,
• Rcpp-FAQ addressing several frequently asked questions,
• Rcpp-modules discussing how to expose C++ functions and modules with ease using an idea borrowed from Boost::Python, and
• Rcpp-extending detailing the steps needed to extend Rcpp with user-provided or third-party classes,

The most interesting new feature is what we call Rcpp modules and is modeled after Boost::Python. This makes it pretty easy to expose C++ functions and classes to R -- without having to write glue code. This is pretty new and may change a tad over the coming releases, but it is also quite exciting. Other changes concern more improvements for use of inline which should now allow packages like our RcppArmadillo to be used with it, and some bug fixes. The full NEWS entry for this release follows below:

0.8.1   2010-06-08
    o   This release adds Rcpp modules. An Rcpp module is a collection of
        internal (C++) functions and classes that are exposed to R. This
        functionality has been inspired by Boost.Python.
        
        Modules are created internally using the RCPP_MODULE macro and
        retrieved in the R side with the Module function. This is a preview 
        release of the module functionality, which will keep improving until
        the Rcpp 0.9.0 release. 
        The new vignette "Rcpp-modules" documents the current feature set of
        Rcpp modules.
        
    o   The new vignette "Rcpp-package" details the steps involved in making a
        package that uses Rcpp.
    o   The new vignette "Rcpp-FAQ" collects a number of frequently asked
        questions and answers about Rcpp.
    o   The new vignette "Rcpp-extending" documents how to extend Rcpp
        with user defined types or types from third party libraries. Based on
        our experience with RcppArmadillo
        
    o   Rcpp.package.skeleton has been improved to generate a package using 
        an Rcpp module, controlled by the "module" argument
    o   Evaluating a call inside an environment did not work properly
        
    o   cppfunction has been withdrawn since the introduction of the more
        flexible cxxfunction in the inline package (0.3.5). Rcpp no longer
        depends on inline since many uses of Rcpp do not require inline at
        all. We still use inline for unit tests but this is now handled
        locally in the unit tests loader runTests.R. 
        Users of the now-withdrawn function cppfunction can redefine it as:
        
           cppfunction <- function(...) cxxfunction( ..., plugin = "Rcpp" )
    o   Support for std::complex was incomplete and has been enhanced.
    o   The methods XPtr<t>::getTag and XPtr<t>::getProtected are deprecated, 
        and will be removed in Rcpp 0.8.2. The methods tag() and prot() should
        be used instead. tag() and prot() support both LHS and RHS use. 
    o   END_RCPP now returns the R Nil values; new macro VOID_END_RCPP
        replicates prior behabiour
        

As always, even fuller details are in Rcpp Changelog page and the Rcpp page which also leads to the downloads, the browseable doxygen docs and zip files of doxygen output for the standard formats. A local directory has source and documentation too. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page

In the past few days, I ve been playing with icegrind, a little valgrind plugin that Taras Glek wrote to investigate how Firefox startup spends its time in libxul.so reading/initialization, mostly. I invite you to check out his various posts on the subject, it s an interesting read. After some fiddling with the original icegrind patch in the bug linked above, I got some better understanding on the binary initialization phase (more on that in another post), as we could call it, and hit some problems due to problems in icegrind and elflog. The latest version of the icegrind patch as of writing solved my main problem half way (though it needs further modifications to work properly, see below). What icegrind actually does is to track memory accesses in mmap()ed memory. When the program being profiled mmap()s a file, icegrind looks for a corresponding sections file next to it (e.g. libxul.so.sections for libxul.so ). The sections file contains offsets, sizes, and sections names, that will be used by icegrind to report the accesses. Whenever the profiled program makes memory accesses (whether it is for instructions or data) within the mmap()ed range, icegrind will add the corresponding section name in a log file (e.g. libxul.so.log for libxul.so ). Please note it will only do so for the first access in the given section. To generate section information for ELF files, Taras also hacked an elflog program that scans an ELF binary (program or library) and will output sections for symbols present in its symtab. This means the binary needs not to be stripped, though the full fledged debugging symbols are not needed. It outputs the sections names in a form that would be reusable in a linker script after a build with the -ffunction-sections and -fdata-sections compiler options, but that s mostly irrelevant when what you are after is only to see what is going on, not to reorder the sections at link time. There are glitches in the way elflog works that makes the section names for .bss symbols wrong (they will start with .symtab, .shstrtab, .comment, etc. instead of .bss). Also, technically, the .bss section is not mapped from the file. Note icegrind has rough edges, is still experimental, and isn t very user friendly, as in unix permissions, because its input and output files need to be next to the file being mmap()ed, and when that is a library in /usr/lib, well, you need write access there ; or you need to copy the library in a different directory and adjust LD_LIBRARY_PATH. Anyways, it s already useful as it currently is. If you want to play with it yourself, here are steps that worked well for me:

• svn co svn://svn.valgrind.org/valgrind/trunk valgrind -r 11100
  When I first tried icegrind, valgrind trunk was broken ; revision 11100 is known to work properly, at least for icegrind.
• cd valgrind
• wget -O - -q https://bug549749.bugzilla.mozilla.org/attachment.cgi?id=449748 patch -p0
  You need to open icegrind/ice_main.c, go on the last line of the track_mmap function and replace mmaps with dest_mmaps.
• mkdir icegrind/tests ; touch icegrind/tests/Makefile.am
  The valgrind build system is a bit lunatic.
• libtoolize ; aclocal ; automake --add-missing ; autoreconf
• ./configure; make
• wget http://hg.mozilla.org/users/tglek_mozilla.com/startup/raw-file/6453ad2a7906/elflog.cpp
• g++ -o elflog elflog.cpp -lelf
  You will need the libelf-dev package.
• make install ; install -m 755 elflog /usr/local/bin
  as root, to install in /usr/local.

Once you re done with this setup, you are ready to start playing:

• elflog --contents libxul.so > libxul.so.sections
  Be aware that if libxul.so is a symbolic link pointing to a file in another directory, as it happens in the dist/bin directory in mozilla builds, the sections file needs to be in that other directory.
• LD_LIBRARY_PATH=. valgrind --tool=icegrind ./firefox-bin -P my-profile -no-remote about:blank

Obviously, you don t have to play around with Firefox. You can try with other libraries, or even programs. You ll see in a subsequent post, however, that to get more interesting information, the elflog output is unfortunately not enough.

DevOps I first heard about DevOps from Lindsay Holmwood at linux.conf.au 2010. Since then, I ve been following the movement with interest. It seems to be about cross-functional involvement in software teams, specifically between software development and system administration (or operations). In many organizations, especially SaaS shops, these two groups are placed in opposition to each other: developers are driven to deliver new features to users, while system administrators are held accountable for the operation of the service. In the best case, they maintain a healthy balance by pushing in opposite directions, but more typically, they resent each other for getting in the way, as a result of this dichotomy:

	Development	Operations
is responsible for	creating products	offering services
is measured on	delivery of new features	high reliability
optimizes by	increasing velocity	controlling change
and so is perceived as	reckless and irresponsible	obstructing progress

Of course, both functions are essential to a viable service, and so DevOps aims to replace this opposition with cooperation. By removing this friction from the organization, we hope to improve efficiency, lower costs, and generally get more work done.
So, DevOps promotes the formation of cross-functional teams, where individuals still take on specialist development or operations roles, but work together toward the common goal of delivering a great experience to users. By working as teammates, rather than passing work over the wall , they can both contribute to development, deployment and maintenance according to their skills and expertise. The team becomes a devops team, and is responsible for the entire product life cycle. Particular tasks may be handled by specialists, but when there s a problem, it s the team s problem. Some take it a step further, and feel that what s needed is to combine the two disciplines, so that individuals contribute in both ways. Rather than thinking of themselves as developers or sysadmins , these folks consider themselves devops . They work to become proficient in both roles, and to synthesize new ways of working by drawing on both types of skills and experience. A common crossover activity is the development of sophisticated tools for automating deployment, monitoring, capacity management and failure resolution. DevOps meets Cloud Like DevOps, cloud is not a specific technology or method, but a reorganization of the model (as I ve written previously). It s about breaking down the problem in a different way, splitting and merging its parts, and creating a new representation which doesn t correspond piece-for-piece to the old one. DevOps drives cloud because it offers a richer toolkit for the way they work: fast, flexible, efficient. Tools like Amazon EC2 and Google App Engine solve the right sorts of problems. Cloud also drives DevOps because it calls into question the traditional way of organizing software teams. A development/operations division just doesn t fit cloud as well as a DevOps model. Deployment is a classic duty of system administrators. In many organizations, only the IT department can implement changes in the production environment. Reaping the benefits of an IaaS environment requires deploying through an API, and therefore deployment requires development. While it is already common practice for system administrators to develop tools for automating deployment, and tools like Puppet and Chef are gaining momentum, IaaS makes this a necessity, and raises the bar in terms of sophistication. Doing this well requires skills and knowledge from both sides of the fence between development and operations, and can accelerate development as well as promote stability in production.
This is exemplified by infrastructure service providers like Amazon Web Services, where customers pay by the hour for black box access to computing resources. How those resources are provisioned and maintained is entirely Amazon s problem, while its customers must decide how to deploy and manage their applications within Amazon s IaaS framework. In this scenario, some operations work has been explicitly outsourced to Amazon, but IaaS is not a substitute for system administration. Deployment, monitoring, failure recovery, performance management, OS maintenance, system configuration, and more are still needed. A development team which is lacking the experience or capacity for this type of work cannot simply switch to an IaaS model and expect these needs to be taken care of by their service provider.
With platform service providers, the boundaries are different. Developers, if they build their application on the appropriate platform, can effectively outsource (mostly) the management of the entire production environment to their service provider. The operating system is abstracted away, and its maintenance can be someone else s problem. For applications which can be built with the available facilities, this will be a very attractive option for many organizations. The customers of these services may be traditional developers, who have no need for operations expertise. PaaS providers, though, will require deep expertise in both disciplines in order to build and improve their platform and services, and will likely benefit from a DevOps approach. Technical architecture draws on both development and operations expertise, because design goals like performance and robustness are affected by all layers of the stack, from hardware, power and cooling all the way up to application code. DevOps itself promotes greater collaboration on architecture, by involving experts in both disciplines, but cloud is a great catalyst because cloud architecture can be described in code. Rather than talking to each other about their respective parts of the system, they can work together on the whole system at once. Developers, sysadmins and hybrids can all contribute to a unified source tree, containing both application code and a description of the production environment: how many virtual servers to deploy, their specifications, which components run on which servers, how they are configured, and so on. In this way, system and network architecture can evolve in lockstep with application architecture. Cloudy promises such as dynamic scaling and fault tolerance call for a DevOps approach in order to be realized in a real-world scenario. These systems involve dynamically manipulating production infrastructure in response to changing conditions, and the application must adapt to these changes. Whether this takes the form of an active, intelligent response or a passive crash-only approach, development and operational considerations need to be aligned. So what? DevOps and cloud will continue to reinforce each other and gain momentum. Both individuals and organizations will need to adapt in order to take advantage of the opportunities provided by these new models. Because they re complementary, it makes sense to adopt them together, so those with expertise in both will be at an advantage.

I just read an interesting article about the dispute between Microsoft and Apple about types of PC [1]. Steve Jobs predicted a switch from desktop PCs to portable devices, while Steve Ballmer of Microsoft claimed that the iPad is just a new PC. Defining a PC I think that the defining characteristic of the IBM Compatible PC was it s open architecture. Right from the start the PC could have it s hardware expanded by adding new circuit boards into slots on the motherboard (similar to other PC systems of that era such as the Apple 2 and the S-100 bus). The deal with IBM included Intel sharing all it s CPU designs with other manufacturers such as NEC and AMD from the 8086 until the mid-90 s. AMD specialised in chips that were close copies of Intel chips at low prices and higher clock rates while NEC added new instructions. Compaq started the PC clone market as well as the laptop market, and system software for the IBM compatible PCs was primarily available from IBM and Microsoft in the early days, along with less popular variants such as CP/M86, Novell Netware and others. In the late 80 s there was OS/2 as an alternate OS and Windows as one of several optional GUI environments to run on top of MS-DOS or PC-DOS. In the mid 90 s PCs were used for running protected mode OSs such as Linux and Windows/NT. Now if we look at a system such as a Netbook then it clearly misses some of the defining characteristics of the desktop PC. I can t upgrade a Netbook in any meaningful way changing a storage device or adding more RAM does not compare to adding an ISA/MCA/EISA/VL-Bus/PCI/PCIe expansion card. With my EeePC 701 I don t even have an option of replacing the storage as it is soldered to the motherboard! A laptop allows me to add a PCMCIA or PC-Card device to expand it, but with a maximum of two cards and a high price this isn t a great option. What is Best for Home Users? For a while now my parents have been using 3G net access for their home Internet use [2]. So it seems that a laptop provides greater benefits for their use now than it previously did when they used Cable and ADSL net access. My parents have been considering getting a new monitor (1920*1080 resolution monitors are getting insanely cheap nowadays) and driving such a monitor effectively might require a more capable PC. I recently bought myself a nice refurbished Thinkpad for $796 [3], it seems likely that I could find a refurbished Thinkpad at auction which is a little older and slower for a lower price, even buying an old T41p would be a reasonable option. This would give my parents not only the option of using the Internet when on holidays, but also in a different part of their house when they are at home. The Apple iPad would probably be quite a reasonable Internet platform for my parents if it wasn t for the fact that it uses DRM. While it s not a great platform for writing, my parents probably don t do enough that it would be a huge problem for them. So I might look for a less restrictive tablet platform for my parents. At the moment the best resolution for a tablet seems to be 1024*768, but I expect that some tablets (maybe with a hybrid tablet/laptop design like the Always Innovating Smartbook [4]) with a higher resolution will be released soon. I hope that the iPad and other closed devices don t get any serious market share, but it seems likely that OSs such as Android which are only slightly more open will have a significant market share. Ultra-Mobile Design vs PCs Design One significant problem with ultra-mobile devices is that they make significant engineering trade-offs to get the small size. For a desktop system there are lots of ways of doing things inefficiently, running the AMD64 or i386 architecture which is wasteful of energy and having lots of unused space inside the box in case you decide to upgrade it. But for a laptop there are few opportunities for being inefficient, and for a tablet or smart phone everything has to be optimised. When the optimisation of a device starts by choosing a CPU that s unlike most other systems (note that there is a significant range of ARM CPUs that are not fully compatible with each other) it makes it very difficult to produce free software to run it. I can salvage a desktop PC from a rubbish bin and run Linux on it (and I ve done that many times), but I wouldn t even bother trying to run Linux on an old mobile phone. It seems that in the near future my parents (and many other people with similar needs) will be best suited by having a limited device such as a tablet that stores all data on the Internet and not having anything that greatly resembles a PC. In many ways it would be easier for me to support my parents by storing their data in the cloud and then automatically backing it up to removable SATA disks than with my current situation of supporting a fully capable PC and backing it up to a USB device whenever I visit them. I m also considering what to do for some relatives who are about to go on a holiday in Europe, they want to be able to send email etc. It might not be possible just yet, but it seems like an ideal way of doing this would be to provide them with something like an iPad that they can use with a local 3G SIM for the country that they stay in and they could then upload all their best photos to some server that I can backup and send email to everyone they know. An iPad isn t good for this now as you don t want to go on holidays in another country while carrying something that is really desirable to thieves. Ultra Mobile Devices are Killing PCs It seems to me that Google Android and the Apple iPad/iPhone OS are taking over significant parts of the PC market. The people who are doing traditional PC things are increasingly using Laptops and Netbooks, and the number of people who get the freedom that a PC user did in the 80 s and 90 s is decreasing rapidly. I predict that by 2012 the majority of Linux systems will be running Google Android on hardware that doesn t easily allow upgrading to more open software. At the moment probably the majority of Linux systems are wireless routers and other embedded devices that people don t generally think about. But when iPad type devices running a locked-down Linux installation start replacing Ubuntu and Fedora desktop systems people will take notice. I don t think that the death of the PC platform as we know it will kill Linux, but it certainly won t do us any good. If there were smarter people at Microsoft then they would be trying to work with the Linux community on developing innovative new ways of using desktop PCs. Of all the attempts that Microsoft has made to leave the PC platform the only success has been the X-Box which is apparently doing well. Tablet devices such as the iPad could work really well in a corporate environment (where MS makes most of it s money). On many occasions I ve been in a meeting and we had to adjourn due to someone needing to go to their desk to look something up. If everyone had an iPad type device at their desk that used a wired network when it was available and encrypted wireless otherwise then for a meeting everyone could take their tablet without it s keyboard and be able to consult all the usual sources of data without any interruption. Could a high-resolution version of the iPad kill MS-Windows in the corporate environment?

• [1] http://www.theage.com.au/digital-life/tablets/the-ipad-is-just-a-pc-ballmer-to-jobs-20100604-xioj.html
• [2] http://etbe.coker.com.au/2010/02/03/3g-broadband-for-home-use/
• [3] http://etbe.coker.com.au/2010/02/15/new-thinkpad-lenovo-web-site-sucks/
• [4] http://etbe.coker.com.au/2010/01/11/always-innovating/

Yesterday afternoon, it was nice outside. Terah, Jacob, Oliver, and I went outside. I had my camera to take photos while Jacob and Oliver mostly played. We did try to pose them a bit, but the overall plan was to just take some photos of them doing their own thing. So I wound up taking 824 photos. Hurray for digital cameras. We narrowed that down to a few of our favorites, which you can view as a slideshow. I ll highlight few of those below.
Jacob walking down the drive way, gazing at all the beans coming up off in the distance
Hmm, Oliver, let me see too
Running is fun!
Peek-a-boo
Goerzen s Law of Child Photography: If you attempt to get them to pose, a minimum of one boy will have a weird expression at any given time.
Also, chances are good that a cat will sneak in and also do something.
My imagined conversation for this one: Here Oliver, this is how you use a piece of bark. Quiet, I m trying to eat this grass.
Oliver decides that grass stuck on his teeth isn t fun
No explanation for this one.
Classic brothers. Jacob is using a piece of bark to mess around in some dirt, while Oliver quietly learns how to do it. He never said it, but I can easily imagine him saying, Here Oliver, here s what you do with bark: you dig up dirt like this! It s funny!
BONZAI!!!
Guess I better watch where I point that camera
Here kitty get ready for some more petting *thwack* Strangely the cat seemed to love it. Our other cat avoids Oliver.
Jacob deriving a great deal of joy from telling us that his shirt says big SPIDER . He came up with that joke himself and thought it was hilarious.
This is one of a few photos of Jacob wearing a suit that I once wore.
It s hard to stand still when the hammock has a great train whistle on it.
I have no idea what he s planning here. But really, I think I should be afraid.
It s a fridge door. Yay?

Apropos Matthew's post... I picked up a cheap netbook for Jo a couple of months ago, an Acer Aspire One 751. She liked the look and feel of the machine, and it was much smaller and more carryable than the massive old Dell "slab" she'd been using previously. Got a very good deal on it too, as it was the last one in stock at the local Comet store. All went well until we got it home and I started playing with it. That was when I found the painful truth: it was based on the Poulsbo, using Intel's crippled GMA500 graphics chipset. Up until then, I'd been very happy with buying Intel kit: for the last few years it's been a sure sign that the hardware would work well with Debian and Linux. Now they've come out with a crappy product that has destroyed that goodwill. I spent many hours working through various options to try and get this new netbook up and running properly using Debian, but to no avail - the various binary blobs wanted newer versions of code than was in Lenny and (by then) older stuff than was available in Squeeze. In the end I gave up and installed an outdated version of Ubuntu on the machine just so Jo could have it working for her when we went on holiday together. Since then, she's had to get used to the machine locking up once or twice a day while she's using it (something I can only assume is caused by the shitty drivers). She's happy enough in that it's much better than her old Vista-infested Dell machine, but it's really not a great experience for her compared to the Thinkpads she sees others using reliably and flawlessly at home. At some point I guess we'll simply ditch the machine altogether and get something less nasty. Thanks for the good stuff you used to make, Intel, but you've lost your way. I'll be damned careful next time to avoid this trap.

Since some time the good people at NlNet have been sponsoring Collabora to add multi-party audio/video calling to XMPP and do an implementation of this in Telepathy. This project is otherwise known as Muji. This work has resulted in:

• an experimental XMPP extension called Muji
• a new and more modern telepathy API for doing calls (currently still in draft), which allows multi-party calls as well as adding the possibility to support call forking (ringing multiple remote devices, call goes to the one that's picked up first).
• An implementation of both of these in telepathy-gabble
• A small telepathy client for testing and demo purposes.

To proof it actually works, a nice screenshot of the demo client in a 4-way call: The screenshot is featuring myself in Collabora's UK office (top-left), Robert McQueen from a hotelroom in Taipei (top-right), Will Thompson and Magical trevor also from the Collabora UK office (bottom-left) and Mike Ruprecht from somewhere in Missouri. So we were quite spread out over our little planet. Both audio and video quality was quite good, even though the network in robs hotel was a bit flaky from time to time. In case you want to try it out have a look at the MujiDemoClient page.

If Debian is to migrate to upstart on Linux, I expect some init.d scripts to migrate (some of) their operations to upstart job while keeping the init.d for hurd and kfreebsd. The packages with such needs will need a way to get their init.d scripts to behave differently when used with sysvinit and with upstart. Because of this, I had a look at the environment variables set when a init.d script is running under upstart, and when it is not. With upstart, I notice these environment variables are set when a script is started from rcS.d/ (ignoring some irrelevant ones like COLUMNS):

DEFAULT_RUNLEVEL=2
previous=N
PREVLEVEL=
RUNLEVEL=
runlevel=S
UPSTART_EVENTS=startup
UPSTART_INSTANCE=
UPSTART_JOB=rc-sysinit

With sysvinit, these environment variables are set for the same script.

INIT_VERSION=sysvinit-2.88
previous=N
PREVLEVEL=N
RUNLEVEL=S
runlevel=S

The RUNLEVEL and PREVLEVEL environment variables passed on from sysvinit are not set by upstart. Not sure if it is intentional or not to not be compatible with sysvinit in this regard. For scripts needing to behave differently when upstart is used, looking for the UPSTART_JOB environment variable seem to be a good choice.

Today, I launched a proposal for a string freeze in Debian Installer. A string freeze is a formal agreement among developers to avoid changing localizable material without concertation. This is a commitment towards translators to guarantee them that, while they're doing an effort to complete their work, this effort won't be broken by a developer adding of modifying translatable material. This can be seen as a first step towards the preparation of a release of D-I. Indeed, the only release we did for Squeeze (D-I alpha1 release) had no formal translation update round. We currently have 45 languages that are considered to be "in good shape" (even if not all complete) and very likely to remain activated in Squeeze. However, 20 (TWENTY) languages are in danger of being deactivated because of low activity. To avoid this, I will once again take my superhero suit, add some leather to it, take a whip and start nagging translators, doing noise and, in short, annoying the entire world (including blogging too often), in order to shake people enough so that we reduce this number to zero. Be prepared. Interesting links:

• Localization statistics for D-I and related packages (the most important part is "level 1"): http://d-i.debian.org/l10n-stats/;
• Documentation for Debian Installer internationalization and localization: http://d-i.alioth.debian.org/doc/i18n/;
• Information about supported languages: http://d-i.alioth.debian.org/doc/i18n/languages.html.

I was trying the last week to import data from one database into a Drupal database. Basically this worked fairly well. The needed fields from the old database are easy to get by simple SELECT statements. Because I don't know anything about PHP, I doing the export/import with a Python script directly into the database. After all this is about articles, comments and tags, that need to be imported into a Drupal database.

The articles will be stored into tables node_revisions, node and node_counter. The tags into tables term_data and term_node. Comments will go into table comments. Here are the INSERTs:

Articles:

"insert into node_revisions (uid, body, teaser, log, timestamp, format, nid, title) values ('3', '%s', '%s', '', '%s', '1', '%s', '%s')" % (body, body, timestamp, i, title)

"insert into node (vid, type, uid, status, created, changed, comment, promote, moderate, sticky, language, tnid, translate, title) values ('%s', 'blog', '3', '1', '%s', '%s', '2', '1', '0', '0', '', '0', '0', '%s')" % (vid, timestamp, timestamp, title)

"insert into node_counter (nid,totalcount,daycount,timestamp) values ('%s', '100', '0','%s')" % (nid, timestamp)

Tags:

"insert into term_data (vid, name) values ('3', '%s')" % tag

"insert into term_node (nid, vid, tid) values ('%s', '%s', '%s')" % (nid, vid, tagtid)

Comments:

"insert into comments (pid, nid, uid, subject, comment, hostname, timestamp, status, format, thread, name, mail, homepage) values ('%s', '%s', '0', '%s', '%s', '%s', '%s', '0', '1', '01', '%s', '%s', '%s')" % (parentid, nid, ctitle, cbody, cip, ctimestamp, cauthor, cemail, curl)

sql="insert into term_node (nid, vid, tid) values ('%s', '%s', '%s')" % (nid, vid, tid)

The import is successful as there are comments and tags attached to each article or node. But there are still some problems. When accessing Recent Posts there are no entries listed. Yet, Drupal still thinks there are so many articles that it lists more than 30 pages:

Something similar happens to the tags. The tags are actually assigned to and are displayed as well with the nodes. But when accessign taxonomys vocabular and trying to list the terms, the resulting page is empty:

If my understanding of the Drupal6 database (on PostgreSQL this is) is correct, everything should be fine and properly stored into the database. But apparently there's something missing. But what? As already mentioned I have no clue about PHP, so using node_api or any other API calls by Drupal/PHP is no option for me. Except when the calls can be placed via Pythons system() call.

If anyone has some tips or hints for me, I would appreciate when you write a comment.

When there's interest I can provide the entire Python import script. I'm going to make it public anyway, when the last problems are solved.

I was trying the last week to import data from one database into a Drupal database. Basically this worked fairly well. The needed fields from the old database are easy to get by simple SELECT statements. Because I don't know anything about PHP, I doing the export/import with a Python script directly into the database. After all this is about articles, comments and tags, that need to be imported into a Drupal database.

The articles will be stored into tables node_revisions, node and node_counter. The tags into tables term_data and term_node. Comments will go into table comments. Here are the INSERTs:

Articles:

"insert into node_revisions (uid, body, teaser, log, timestamp, format, nid, title) values ('3', '%s', '%s', '', '%s', '1', '%s', '%s')" % (body, body, timestamp, i, title)

"insert into node (vid, type, uid, status, created, changed, comment, promote, moderate, sticky, language, tnid, translate, title) values ('%s', 'blog', '3', '1', '%s', '%s', '2', '1', '0', '0', '', '0', '0', '%s')" % (vid, timestamp, timestamp, title)

"insert into node_counter (nid,totalcount,daycount,timestamp) values ('%s', '100', '0','%s')" % (nid, timestamp)

Tags:

"insert into term_data (vid, name) values ('3', '%s')" % tag

"insert into term_node (nid, vid, tid) values ('%s', '%s', '%s')" % (nid, vid, tagtid)

Comments:

"insert into comments (pid, nid, uid, subject, comment, hostname, timestamp, status, format, thread, name, mail, homepage) values ('%s', '%s', '0', '%s', '%s', '%s', '%s', '0', '1', '01', '%s', '%s', '%s')" % (parentid, nid, ctitle, cbody, cip, ctimestamp, cauthor, cemail, curl)

sql="insert into term_node (nid, vid, tid) values ('%s', '%s', '%s')" % (nid, vid, tid)

The import is successful as there are comments and tags attached to each article or node. But there are still some problems. When accessing Recent Posts there are no entries listed. Yet, Drupal still thinks there are so many articles that it lists more than 30 pages:

Something similar happens to the tags. The tags are actually assigned to and are displayed as well with the nodes. But when accessign taxonomys vocabular and trying to list the terms, the resulting page is empty:

If my understanding of the Drupal6 database (on PostgreSQL this is) is correct, everything should be fine and properly stored into the database. But apparently there's something missing. But what? As already mentioned I have no clue about PHP, so using node_api or any other API calls by Drupal/PHP is no option for me. Except when the calls can be placed via Pythons system() call.

If anyone has some tips or hints for me, I would appreciate when you write a comment.

When there's interest I can provide the entire Python import script. I'm going to make it public anyway, when the last problems are solved.

UPDATE:

Killes on IRC hinted me that node_comment_statistics and term_hierarchy needs to be filled as well. Thanks for the hint!

I m currently running google-chrome-beta version 5.0.375.55-r47796 on Debian/Unstable. It s the fastest web browser I ve used in recent times it s the first time that I ve run a browser that feels faster than my recollection of running IBM WebExplorer for OS/2 on a 486-66 system! It has a good feature set, and it s the only browser I ve used that in a typical configuration will make proper use of the screen space by not having a permanent status bar at the bottom of the sceen and by having tabs in the title-bar. But it s not perfect, here is a list of some bugs: Above are three partial screen captures of Chrome, the first is when maximised and the second is when the window isn t maximised. Notice the extra vertical space above the tab in the title bar in the second picture. The third picture shows the right side of the titlebar and you can see a space below the three buttons where you can drag the window around no matter how many tabs you open that space below the three buttons is reserved. If the Chrome developers had removed the extra vertical space in the titlebar and reserved slightly more horizontal space then you would be able to drag the window around. While an anonymous commentator made a good point that the extra vertical space can be used to drag the window around when the maximum number of tabs are open, it seems that there are other ways of achieving that goal without wasting ~18 vertical pixels. Doing so would be a lot less ugly than what they did with finding text in the page. When I visit a web site that uses cookies from an Incognito Window (which means that cookies etc aren t stored) there is no option to say allow all cookies . This is really annoying when you get to a web site such as the IBM one which stores 5 cookies when you first load the page and then at least one new cookie write for every page you visit. Given that cookie data will be discarded as soon as the window is closed it seems like a good idea to have an option to allow all cookies for Incognito Windows even if all cookies aren t allowed for regular windows. Blocking all cookies would be OK too, anything but having to click on Block or Allow multiple times for each page load. The J and K keys don t work in a view of Venus version 0~bzr95-2+lenny1 (the latest version in Debian/Lenny). I once had a situation where I entered a . at the end of a domain name (which is quite legal there is always an implied dot) and Chrome then wouldn t take note of my request to accept all cookies from the domain. I haven t been able to reproduce that bug, but I have noticed that it stores the settings for whether cookies should be stored separately for domains that end in . , so www.cnn.com. is different from www.cnn.com . Iceweasel seems to just quietly strip the trailing dot. Of course this is better than Konqueror which won t even load a URL with a dot at the end. Chrome can be relied on to restore all windows rapidly after a crash, unlike Iceweasel which restores them at it s normal load speed (slow) and Konqueror which doesn t tend to restore windows. This is good as it does seem to crash regularly. In a response to my post about Chrome and SE Linux [1] Ben Hutchings pointed out that the --no-sandbox option to chrome disables the creation of a PID namespace and therefore makes debugging a lot easier, if I get a lot of spare time I ll try and track down some of the Chrome SEGVs. The JavaScript compiler is either buggy or it s not buggy in situations where people expect IE bugs. When using the Dell Australia web site I can t always order all options. When trying to order a Dell R300 1RU server with hot-plug disks in a hardware RAID array it seems impossible to get all the necessary jumpers which is a precondition to completing the order fortunately I only wanted to blog about how cheap Dell servers are so I don t actually need to complete an order. Dell s web site is also difficult in Iceweasel on occasion, so it s obviously more demanding than most sites. It might be a good test site for people who work on browsers as it s both demanding and important. When I select a URL to be opened in a new window (or when JavaScript does this) then the new tab is opened with about:blank listed as the URL. If the URL is for a PDF file (or something else that is to be downloaded) then the URL entry field is never updated to give the real URL. I believe that this is wrong, either the new tab shouldn t be opened or it should have the correct URL on display there is no benefit with a tab open to show nothing but about:blank in the URL entry field. Also if a URL takes some time to load then it may keep about:blank in the URL entry field for some time. This means that if you use the middle mouse button to rapidly open a few new tabs you won t be able to see what is to be loaded in each one. Sometimes I have several tabs loading and I m happy to close some unimportant ones if they are slow but some are worth waiting for. Overall that s not too bad. I can use Dell s site in Iceweasel, so the only critical bug is the cookies issue in Incognito Windows which makes the Incognito feature almost unusable for some sites.

• [1] http://etbe.coker.com.au/2010/05/19/google-chrome-and-se-linux/

Today, I have finally permanently deleted my Facebook account. I have followed Matt Zimmerman s procedure, and it seems everything went well.
Now, I just need to not login to Facebook over the next 14 days, which shouldn t be a problem at all (my last connection was something like 2 or 3 months ago!). My aim is now to make more and more people use identi.ca as their primary social network. I will also follow the Diaspora project. My identi.ca profile : http://identi.ca/julienvalroff

My project. I m publishing my report earlier this week, because there was a lot to talk about. This week I started to make changes in the code going in the direction of what I want to do. I haven t started a final implementation, but I m studying about how what I want to do will affect the rest of the Garbage Collection (GC). I noticed the code in rts/sm/Sweep.c was simple and similar to what I m planning to do, so I started changing how it works. Sweep in the Glasgow Haskell Compiler (GHC) is done by a bitmap, which contains a bit for each word in a memory block, and is set to 1 when there s an object starting in the mapped area and 0 otherwise. When there s a block with no objects starting at it, that is, all bits of the bitmap are set to 0, the block is freed.

        if (resid == 0)
         
            freed++;
            gen->n_old_blocks--;
            if (prev == NULL)  
                gen->old_blocks = next;
              else  
                prev->link = next;
             
            freeGroup(bd);
         

The bits are analyzed in a group of BITS_IN(W_), where BITS_IN(W_) is the number of bits in a word.

        for (i = 0; i < BLOCK_SIZE_W / BITS_IN(W_); i++)
         
            if (bd->u.bitmap[i] != 0) resid++;
         

If more than of the groups are completely set to 0, the block is considered fragmented.

            if (resid < (BLOCK_SIZE_W * 3) / (BITS_IN(W_) * 4))  
                fragd++;
                bd->flags  = BF_FRAGMENTED;
             

Immix, the GC algorithm I plan to implement in GHC, divides the blocks of memory in lines. My initial plan was to identify free lines. I decided to consider a the size of a line fixed in BITS_IN(W_) words, because it will map to a word in the bitmap, and the code was already analyzing in groups of BITS_IN(W_) words. This was very easy with the current code.

            if (bd->u.bitmap[i] != 0) resid++;
            else printf("DEBUG: line_found(%p)\n", bd->start + BITS_IN(W_) * i);

This worked, and showed some free lines. I m sure there are other ways of logging in GHC, but printf was the simplest way I could thought of. I measured the occurrence of free lines using the bernouilli program from the NoFib benchmark suite, calling it with 500 +RTS -w, to make it uses sweep. In 782 calls to GarbageCollect(), sweep() was called 171 times, for 41704 blocks to be swept and found 230461 free lines. This gives us about 5.5 free lines per block, from the 8 lines in each block (on 64 bits systems). The problem is that the bitmap is marked only in the start of the objects allocated, so even in a line that all bits are marked with 0 we can t assume that it s completely free, because there may be an object that starts in the previous line that is using the space of the line. Checking only for the previous line doesn t work either, because a big object can span several lines. What we can do here is a variation of conservative marking, as proposed in the Immix paper, checking only the previous line and working only with objects smaller than a line. To make sure I was working only with objects smaller than a line, I had to mark the blocks that contains medium objects and avoid them when seeking free lines. The block flags are defined in includes/rts/storage/Block.h, so I included another flag in this file, BF_MEDIUM.

/* Block contains objects evacuated during this GC */
#define BF_EVACUATED 1
/* Block is a large object */
#define BF_LARGE     2
/* Block is pinned */
#define BF_PINNED    4
/* Block is to be marked, not copied */
#define BF_MARKED    8
/* Block is free, and on the free list  (TODO: is this used?) */
#define BF_FREE      16
/* Block is executable */
#define BF_EXEC	     32
/* Block contains only a small amount of live data */
#define BF_FRAGMENTED 64
/* we know about this block (for finding leaks) */
#define BF_KNOWN     128
/* Block contains objects larger than a line */
#define BF_MEDIUM    256

The GHC GC is generational, there is, objects are allocated in a generation and, after a time, the ones that are still being used are moved the next generation. This idea assumes the death probability of younger objects is higher, so few objects are moved to the next generation. Sweep and Immix work only in the last generation so, to mark blocks with medium objects we have to check the size of the objects that are moved to the next generation. This is done in the copy_tag function of rts/sm/Evac.c. I inserted a code that checks for the object size and marks the block when it s bigger than BITS_IN(W_).

STATIC_INLINE GNUC_ATTR_HOT void
copy_tag(StgClosure **p, const StgInfoTable *info,
         StgClosure *src, nat size, generation *gen, StgWord tag)
 
    StgPtr to, from;
    nat i;
    to = alloc_for_copy(size,gen);
    if(size > 8)  
        Bdescr(to)->flags  = BF_MEDIUM;
     

So I updated the code in rts/sm/Sweep.c to only inspect for free lines in blocks without BF_MEDIUM mark.

            if (bd->u.bitmap[i] != 0) resid++;
            else if(!(bd->flags & BF_MEDIUM))  
                printf("DEBUG: line_found(%p)\n", bd->start + BITS_IN(W_) * i);
             

This also worked. Now, in the 32012 blocks there were 189015 free lines, found in the same number of GCs, making about 5.9 free lines per block. We considered only blocks with small objects, but we didn t ignore the first line of each group of free lines. This can be achieved by checking if the previous line was also free.

            if (bd->u.bitmap[i] != 0) resid++;
            else if(!(bd->flags & BF_MEDIUM) && i > 0 && bd->u.bitmap[i] == 0)  
                printf("DEBUG: line_found(%p)\n", bd->start + BITS_IN(W_) * i);
             

Now, from the 32239 blocks, 165547 free lines were found, giving 5.1 free lines per block. But there are more things to improve. If the whole block is free, we want to free it, instead of marking it s lines as free. So it s better to mark the lines after we know that the blocks are not completely free. So I left the code that checks the bitmap as it was, and included a line check only for blocks that are not completely free. At this point, I also associated the fragmentation test with blocks with medium objects, because in blocks of small objects we plan to allocate in free lines, so fragmentation is not a (big) issue.

            if (resid < (BLOCK_SIZE_W * 3) / (BITS_IN(W_) * 4) &&
                (bd->flags & BF_MEDIUM))  
                fragd++;
                printf("DEBUG: BF_FRAGMENTED\n");
                bd->flags  = BF_FRAGMENTED;
             
            else if(!(bd->flags & BF_MEDIUM))  
                for(i = 1; i < BLOCK_SIZE_W / BITS_IN(W_); i++)
                 
                    if(bd->u.bitmap[i] == 0 && bd->u.bitmap[i - 1] == 0)  
                        printf("DEBUG: line_found(%p)\n", bd->start + BITS_IN(W_) * i);
                     
                 
             

The total ammount of blocks increased dramatically: the blocks that become fragmented and were not called again in sweep made a huge difference. From the 345143 blocks, 1633268 free lines were found, or about 4.7 free lines per block. 9434 blocks were free, so, from the remaining blocks, we have about 4.9 free lines per block. Something we ll need then is a way to access these lines latter. The simplest way I thought to achieve it is constructing a list of lines, in each the first word of each free line is a pointer to the next free line, and the first word of the last free line is 0. It s useful to keep reporting the lines to stdout, so that we can then follow the list and check if we went to the same lines.

                    if(bd->u.bitmap[i] == 0 && bd->u.bitmap[i - 1] == 0)  
                        StgPtr start = bd->start + BITS_IN(W_) * i;
                        printf("DEBUG: line_found(%p)\n", start);
                        if(line_first == NULL)  
                            line_first = start;
                         
                        if(line_last != NULL)  
                            *line_last = (StgWord) start;
                         
                        line_last = start;
                        *line_last = 0;
                     
                 
             
         
     
    for(line_last = line_first; line_last; line_last = (StgPtr) *line_last)  
        fprintf(stderr, "DEBUG: line_found(%p)\n", line_last);
     

I printed the inclusion of the lines on the list to stdout, and the walking on the list in stderr, so that it d be easy to diff. There was no difference between the lists. There re another improvements that can be made, like using a list of groups of free lines, but I think it ll be better to think about this after studying how the allocation in the free lines will be done. That s where I m going to now.

---

There are some minor things I learned, and thought they worth blogging. The current GHC uses three strategies for collecting the last generation: copying, mark-compact and mark-sweep. Copying is the default until the memory reaches 30% of the maximum heap size; after that, mark-compact is used. Sweep can be chosen by a Real Time System (RTS) flag, -w. To use mark-compact always, the flag is -c. I ve been submitting small patches to the cvs-ghc mailling list, mostly about outdated comments. Most of them were accepted, except for one which contained a lot of commentary, and that indeed was not completely correct. I corrected it and resend to the list, but the message is waiting for approval because the message header matched a filter rule. I believe this is because I replied the message generated from darcs. There s a very useful ghc option, specially for testing the compiler, because in this case you need to rebuild the source, even when there re no changes in it. It s -fforce-recomp, and it makes only sense when used with --make.

Various people observed in a long thread on debian-devel that the grub2 package was in a bit of a mess in terms of its release-critical bug count, and Jordi and Stefano both got in touch with me directly to gently point out that I probably ought to be doing something about it as one of the co-maintainers. Actually, I don't think grub2 was in quite as bad a state as its 18 RC bugs suggested. Of course every boot loader failure is critical to the person affected by it, not to mention that GRUB 2 offers more complex functionality than any other boot loader (e.g. LVM and RAID), and so it tends to accumulate RC bugs at rather a high rate. That said, we'd been neglecting its bug list for some time; Robert and Felix have both been taking some time off, Jordi mostly only cared about PowerPC and can't do that any more due to hardware failure, and I hadn't been able to pick up the slack. Most of my projects at work for the next while involve GRUB in one way or another, so I decided it was a perfectly reasonable use of work time to do something about this; I was going to need fully up-to-date snapshots anyway, and practically all the Debian grub2 bugs affect Ubuntu too. Thus, with the exception of some other little things like releasing the first Maverick alpha, I've spent pretty much the last week and a half solidly trying to get the grub2 package back into shape, with four uploads so far. The RC issues that remain are:

• upgrade-from-grub-legacy problems (#547944, #550477):
  I think this has just been traditionally undertested. I'm setting up a KVM image now with GRUB Legacy which I can snapshot just before and after running upgrade-from-grub-legacy, and I should be able to unpick the bugs this way.
• LVM snapshots break GRUB's LVM module (#574863):
  Sean has been working on this and seems to be nearly there. Yay.
• RAID metadata version 1.x not supported (#492897):
  This became rather more of an issue recently since mdadm switched its default from the old 0.90 format which GRUB understood. Felix put together a branch implementing the hard parts of this a while back, and I've been trying to finish it off. The hard bit is dealing with device naming, especially as the new-format and rather more useful names under /dev/md/ don't show up during d-i after creating RAID volumes; I think this is because we always create them as /dev/md0 etc. It's looking tractable, though.
• Another odd problem probing RAID (#548648):
  Not sure about this one, and I'll need to work with Josip on it as soon as I get a chance.
• Stable device naming #554790) and consequential problems due to grub-install not being properly run (#557425 and many other sub-RC bugs):
  Ubuntu's been carrying a patch to rearrange device presentation in the postinst, which Robert OKed in principle ages ago and so I've been intending to merge it for a while, but there are a few known problems with it that I need to fix first. One known unfixable problem is that it will have to ask some people which devices they want GRUB to be installed on, even if they'd answered that question before: this will be one-time, and it's because it recorded the answer using unstable device names and so has in some sense forgotten. Simple cases (e.g. single-disk) can be handled without needing to ask again, though.
• Alignment errors on SPARC (#560823):
  I have no idea what's going on here, I'm afraid. I'll try to trace it, but may have to downgrade it at some point since after all we don't install GRUB by default on SPARC yet.
• Fonts not shown in gfxmenu (#564844):
  Apparently fixed upstream, but I couldn't find the responsible commit so I want to make sure I can get gfxmenu working before closing this.
• Sensitivity to out-of-date device.map files (#575076 and other sub-RC bugs):
  We're trying to get rid of device.map in general. It was fine in the 1990s but it's hopeless now. Unfortunately there are still a small number of problems with running entirely without one, and one of my patches to help is controversial upstream, so we probably won't get to that for squeeze. In the meantime we'll probably just need some extra sanity-checking and robustness in the event that there's an incorrect or out-of-date device.map lying around, which we may just be able to do in the maintainer scripts or something if necessary.
• Seriously weird failures to load initramfs (#582342):
  If anyone can produce a reproduction recipe for this, that would really help me out. There are too many reports to discount as user error, but I haven't seen this myself yet.
• Build failure on sparc (unfiled):
  We've been discussing this upstream, but for the time being I'm just going to stop building grub-emu on sparc as a workaround.

If we can fix that lot, or even just the ones that are reasonably well-understood, I think we'll be in reasonable shape. I'd also like to make grub-mkconfig a bit more robust in the event that the root filesystem isn't one that GRUB understands (#561855, #562672), and I'd quite like to write some more documentation. On the upside, progress has been good. We have multiple terminal support thanks to a new upstream snapshot (#506707), update-grub runs much faster (#508834, #574088), we have DM-RAID support with a following wind (#579919), the new scheme with symlinks under /dev/mapper/ works (#550704), we have basic support for btrfs / as long as you have something GRUB understands properly on /boot (#540786), we have full info documentation covering all the user-adjustable settings in /etc/default/grub, and a host of other smaller fixes. I'm hoping we can keep this up. If you'd like to help, contact me, especially if there's something particular that isn't being handled that you think you could work on. GRUB 2 is actually quite a pleasant codebase to work on once you get used to its layout; it's certainly much easier to fix bugs in than GRUB Legacy ever was, as far as I'm concerned. Thanks to tools like grub-probe and grub-fstest, it's very often possible to fix problems without needing to reboot for anything other than a final sanity check (although KVM certainly helps), and you can often debug very substantial bits of the boot loader - the bits that actually go wrong - using standard tools such as strace and gdb. Upstream is helpful and I've been able to get many of the problems above fixed directly there. If you have a sound knowledge of C and a decent level of understanding of the environment a boot loader needs to operate in - or for that matter specialist knowledge of interesting device types - then you should be able to find something to do.

Probably it's just a question of me not paying enough attention to the news... So: pointers welcome. There are quie a few shiny new filesystems for local storage, like btrfs or HAMMER (and nilfs, Tux3, ext4, Reiser4...) It seems the distributed storage side is covered as well, with ceph being merged recently. There are other systems (like, for example, Lustre), but they haven't appeared much in the news channels I tend to read. What I'd be curious is if any of these support hierarchical storage architectures like pushing out rarely used data to tape libraries. (But this is just idle speculation, I don't need this anywhere.) But what I would really need is a replacement for NFS (v3): a classical client-server filesystem. I'm not sure NFSv4 is the right solution (where I'd use it, I currently can't because we rely too much on POSIX ACLs there, making the transition to NFSv4 quite a chore.) I think POHMELFS might be a solution in the long term, or CRFS, but I'm not sure how much progress there is on these; apt-cache search is silent, not a good sign. There's Samba 4 I guess I'll have to look at it, since it's supposed to be much cleaner and nicer to use than its predecessors and might be a good solution even if no Microsoft systems are involved. Is this what small environments (a fileserver and 100 clients or so) will want to use? Comments are very welcome.

Joey Hess wrote on Debian-devel about the problem of init scripts not doing adequate checks before using the data from a PID file under /var/run to determine which process to kill [1]. Unfortunately that still doesn t quite solve the problem, there is still the issue of a race condition causing a process to die while you are doing the checks and then be replaced by another process. Below I have included the source code to a little program that will repeatedly fork() until it finds a particular PID and then have it s child call sleep(). So you can run a command such as kill -9 1234 ; ./a.out 1234 and then have this program take over the PID 1234. From testing with this it seems that when you have a shell with it s current working directory as /proc/1234 then once process 1234 is killed the current directory is empty, ls -l returns 0 entries. This isn t surprising, it s the standard Unix behavior when the working directory is removed. So if a program (or even a shell script) changes directory to /proc/1234 it can then verify all attributes of the process (it s CWD, it s root directory, the executable used to run it, it s UID, GID, supplemental groups, it s SE Linux context, and lots of other things all atomically. The only possibility for confusion is that a process might execute a SETUID or SETGID program or a program that has a label which triggers a SE Linux domain transition. It might also change some attributes without executing a new process, for example by using the setuid(), setgid(), setgroups(), or other similar system calls. For the purposes of killing a process I don t think that the possibility of it changing it s own attributes or executing a new program are serious problems, if you want to kill a process then you probably want to kill it after it has called setuid() etc. It seems to me that it would be useful to have a file named /proc/PID/signal (or something similar) to which you could write a number and then have the kernel send the signal in question to that process. So the commands kill -9 1234 and echo 9 > /proc/1234/signal would give the same result. But you could run the command cd /proc/1234 and then some time later you could run echo 9 > signal and know that you would kill the original process 1234 if it was still running and not some other process that replaced it. What do you think? Is this worthy of adding a new feature to the proc filesystem? The Source Run the following program with a single parameter which is the PID that you want it to take. It will keep looping through the PID space until it gets the one you specify, if the one you specify isn t available (EG you give it 1 as a parameter) then it will run forever. It will take some time to get a result on a slower system. On my P3-900MHz test system it took up to 72 seconds to get a result. #include <stdio.h>
#include <unistd.h>
#include <stdlib.h>
#include <sys/types.h>
#include <sys/wait.h> int main(int argc, char **argv)

if(argc != 2)

printf("Specify the desired PID on the command-line\n");
return 1;

pid_t wanted = (pid_t)atoi(argv[1]);
int rc;
int status;
while((rc = fork()) != wanted && rc != 0)

if(rc == -1)

printf("fork error\n");
return 1;

if(wait(&status) == -1)
printf("wait error\n");

if(rc == 0)

if(getpid() == wanted)

printf("Got pid %d, sleeping\n", wanted);
sleep(200);

return 0;

wait(&status);
return 0;

• [1] http://lists.debian.org/debian-devel/2010/05/msg00067.html

RMS will be visiting Kosovo today to speak at the national library. There is no entrance fee, however the number of seats is limited to 250, so please register by writing to register@flossk.org. Less illustriously, MJR will be visiting Plymouth to run a social media surgery as part of Doing business the co-operative way on Friday 25 June (tickets 25 or 75 email now). I ve a few good ideas for what to cover if people are interested but don t have specific questions (and I ll post a few of them here later), but what tip would you give to a new social media user?