I've been meaning to attend the ACCU Conference for some years and finally decided to pay my own way for a couple of days this year. It was an opportunity to hear world experts on C++ and other programming topics and to a smaller extent to meet and socialise with other professional programmers. I attended the following sessions: Keynote: Improving Collaboration in Open Source Projects, Mark Shuttleworth This was a somewhat surprising choice of speaker and subject since I think most attendees work on proprietary software, but I think any sensible developer in the proprietary world will try to build on free libraries under non-copyleft licences, where appropriate. Mark talked about governance and leadership in "open source" projects (actually praising Debian for having a clear constitution) and about the need for good communication between projects, for example about release schedules. He seems to have accepted that Launchpad cannot be a single hub, and called for standard formats for, for example, exchanging information about related bugs between bug trackers. But it doesn't sound like he's ready to take the lead on that. Choose your Poison: Exceptions or Error Codes, Andrei Alexandrescu Andrei looked at the strengths and weaknesses of various error reporting mechanisms in C++ and introduced a technique for making error handling more flexible. Normally a function is designed either to indicate errors through its return value or to throw an exception on error. If a function returns an error indicator, it's easy for callers to ignore it. If it throws an exception on error, then a caller that provokes errors more often than the called function's author anticipated pays a high cost in processing time for those errors. Andrei's solution involves a class template Likely<typename T> whose instances hold either a return value of type T or an exception. A function that can fail has a specialisation of this template as its return type. The caller can check for failure if it knows how to handle it, or it can ignore it - in which case the conversion to T or the destructor will throw the exception. This is clever but doesn't seem to be entirely satisfactory. The destructor which throws does first check that no exception is already being processed, avoiding collision of the two exceptions and consequent abrupt termination of the program, but this can result in errors being unexpectedly ignored. Introduction to Component-Level Testing, John Lakos A marathon 3-hour talk on designing and testing software, particularly C++ programs (the subject of his well-known book) as a set of well-defined components. Lakos first defined these components as being "physical" i.e. sets of source files - normally a header file defining an interface, an implementation file, and a test driver file. They may also include metadata for the build system. He talked about the value of reusable components (as if this wasn't obvious!) and the risk of introducing circular dependencies - which essentially turn many components into one and reduce reusability - if we fail to keep track of dependencies. Software built out of components can be viewed as a hierarchy or stack defined by dependencies. In a large system, components with similar dependencies and dependents can be grouped together into packages or package groups to simplify a view of these dependencies. He explained that the test driver's dependencies count just as much as the implementation's, because we cannot reasonably test a component until we have tested its dependencies. Test drivers should also be written to avoid depending on environmental settings and configuration, so far as possible. Finally, and perhaps most importantly, he spent a long time talking about how to construct test cases and test data. I might try to summarise this in a later entry if anyone's interested. The Appliance of Science: Things in Computer Science that every practitioner should know, Andrei Alexandrescu Andrei began by complaining that OpenOffice Impress was even worse than PowerPoint, but being able to use Linux (specifically Ubuntu) more than made up for the pain. He briefly presented four key ideas that he believes programmers should understand and be ready to make use of:

• Dynamic programming. The Wikipedia article summarises this as "a method of solving problems exhibiting the properties of overlapping subproblems and optimal substructure (described below) that takes much less time than naive methods". As a simple example, Andrei presented a Fibonacci number generator. But since no-one really needs Fibonacci numbers, he also showed its application to calculation of the Levenshtein distance between words which is important in spell-checking and fuzzy string matching. A naive approach to either of these problems is so inefficient as to be useless.
• Garbage collection and its connection to type-safety. Any object model can only provide 2 of the following 3 features without losing type-safety: mutable objects, aliasing (multiple pointers/references to the same objects), and explicit reuse of memory (via free, delete or similar). C and C++ have all three of these. Garbage collection eliminates the need for the third. However it does require more virtual memory and/or more processing time (there is a trade-off between these). (He quoted numbers from Hertz and Berger, Quantifying the Performance of Garbage Collection vs. Explicit Memory Management, 2005. However, these numbers show the cost of GC in Java, specifically Jikes, not as an addition to C or C++.)
• Machine learning. Sometimes we don't know how to solve a problem, but we can have the program gather feedback that will allow it to learn how to do it better. This depends on the "smoothness assumption": small changes in inputs have a small effect on the result, rather than making sudden changes. Examples: image recognition, deciding whether changes to web page layout are helpful, disambiguating search terms.
• Transactional memory is the most promising approach to achieving scalability across massively parallel systems. Most hardware performance improvements over the next 10 years will come from greater parallelism, not higher clock speeds.

Writing for ACCU (BoF) I think I've been convinced to write an article about standardisation of multithreading in C++. C++ Threads, Lawrence Crowl Crowl gave an overview (similar to this) of the extensive work that's been done on standardisation of multithreading in the C++ language and library. This is one of the most important features in C++0x due to the increased parallelism mentioned above. The current semantics of multithreaded programs are not standardised, resulting in subtle but potentially serious differences between current C++ implementations that support multiple threads, and in the need for assembly-language code in programs that use lockless synchronisation. Much of this is also relevant to C and is being considered by the C standard committee. C++0x Initialisation: Lists and a Uniform Syntax, Bjarne Stroustrup Stroustrup recapped the 4 forms of syntax for initialising objects in C++, some inherited from C and some necessarily invented for C++, none of which are entirely generic. He noted the lack of a simple syntax to initialise the elements of containers, which violates the general principle that class types should have the same status as built-in types in C++. He then introduced a new initialiser syntax that he and others have defined for C++0x, which is a brace-enclosed comma-separated list of values, similar to that used for arrays and C-style structures (aggregates). Container types can define a constructor with a parameter of type const std::initializer_list<T> & and when constructed with this new form of initialiser they will receive this wrapper for a constant array of element values. For types that do not, the values in the list are treated as arguments to some other constructor, if one matches. Otherwise, for compatibility, this may be treated as aggregate initialisation. C++ Standard Library report, Alisdair Meredith There was far too much information here for me to even begin to summarise. Suffice to say there are lots of good things coming bu the library committee will have its work cut out to get them all properly specified in time for the Committe Draft coming late this year.

---

I also met old friends and new, put faces to names from newsgroups and books, and got too little sleep.

I'm busy reading this, which outlines a concern that lots of Europeans have, which is how to comply with EU data protection laws and the US court system at the same time. SWIFT appear to be dealing with the problem, which is good. Ars then gives an example of a similar American massive surveillance system, Echelon, and how it feels the US has abused the system to catch Airbus bribing people. Now, I'm naive, but I feel that bribing people is wrong. And while two wrongs don't make a right, I'm not happy with the idea that Europe's main complaint about Echelon is that it was used to catch Europeans breaking their laws. This would be similar to UK drivers complaining about speed cameras by complaining that they catch speeders, rather than them causing accidents or being used as a way to make money. So could ars use a better example of abuse in the future, as I'm sure there must be one? Sorry for moaning, I'll return you to your normal diet of interesting stuff from DDs

Perhaps the most accessible documentation on what PIC code is and how an ELF dynamic linker works is John Levine s Linkers and Loaders (and it has amazing sketches, too!). The Gentoo documentation also has an Introduction to Position Independent Code. I d like to give a few hints on how to fix the shlib-with-non-pic-code lintian error caused by inline assembly on the i386 and amd64 platforms, as well as build errors that may occur due to inline assembly being used. I m not going to cover the trivial all objects were not built using gcc s -fPIC flag problem. It usually requires a fix to the build system, not to the code. gcc can t find a register (i386) PIC on i386 uses a register to store the GOT (global offset table) address. This register is usually %ebx, making it unavailable for use by inline assembly (and also restricting the compiler's register usage when compiling C or C++ code). So the following perfectly valid code will not build with the -fPIC flag:

 void cpuid(uint32_t op, uint32_t reg[4])
  
    asm volatile("cpuid"
                 : "=a"(reg[0]), "=b"(reg[1]), "=c"(reg[2]), "=d"(reg[3])
                 : "a"(op)
                 : "cc");
  

Using -fPIC, gcc will say something around the lines of error: can't find a register in class BREG while reloading asm . Several things need to be done to fix this:

• use a register other than %ebx
• save %ebx if it risks being clobbered by the assembly code, and don't tell gcc about %ebx at all (it doesn't need to know anyway)
• if we saved %ebx by pushing it on the stack, make sure the inline assembly code takes the new stack offset into account

And here is the PIC-compliant version:

 void cpuid(uint32_t op, uint32_t reg[4])
  
    asm volatile("pushl %%ebx      \n\t" /* save %ebx */
                 "cpuid            \n\t"
                 "movl %%ebx, %1   \n\t" /* save what cpuid just put in %ebx */
                 "popl %%ebx       \n\t" /* restore the old %ebx */
                 : "=a"(reg[0]), "=r"(reg[1]), "=c"(reg[2]), "=d"(reg[3])
                 : "a"(op)
                 : "cc");
  

using variables from assembly code (i386) Directly using variables from inline assembly always creates non-PIC code even if -fPIC is being used. There are at least three strategies to consider when trying to fix this. Here is an example of PIC-incompatible code:

 extern uint32_t sym1, sym2, sym3;
 
 void store_in_symbols(uint32_t x)
  
     asm volatile("movl %0, sym1   \n\t"
                  /* ... */
                  "movl %0, sym2   \n\t"
                  /* ... */
                  "movl %0, sym3   \n\t"
                  : : "r"(x));
  

The first strategy is to pass the variable through the usual gcc inline assembly syntax. This is not always possible because there might be a shortage of registers, but here is what it looks like:

 extern uint32_t sym1, sym2, sym3;
 
 void store_in_symbols(uint32_t x)
  
     asm volatile("movl %3, %0   \n\t"
                  /* ... */
                  "movl %3, %1   \n\t"
                  /* ... */
                  "movl %3, %2   \n\t"
                  : "=r"(sym1), "=r"(sym2), "=r"(sym3) : "r"(x));
  

If there are too many variables and not enough registers, one can use the second strategy: put all required variables (or addresses) in a table, thus requiring only one extra register:

 extern uint32_t sym1, sym2, sym3;
 
 void store_in_symbols(uint32_t x)
  
     uint32_t tab[3];
     asm volatile("movl %1, (%0)   \n\t"
                  /* ... */
                  "movl %1, 4(%0)  \n\t"
                  /* ... */
                  "movl %1, 8(%0)  \n\t"
                  : : "r"(tab), "r"(x));
     sym1 = tab[0];
     sym2 = tab[1];
     sym3 = tab[2];
  

If the second method happens to be unsuitable because variable types differ too much, there is a third, more complicated strategy: retrieve the GOT address using some linker-specific magic, then use @GOT addressing to access variables:

 extern uint32_t sym1, sym2, sym3;
 
 void store_in_symbols(uint32_t x)
  
     uint32_t got;
     asm volatile("call get_got                               \n\t"
                  "get_got:                                   \n\t"
                  "popl %0                                    \n\t"
                  "addl $_GLOBAL_OFFSET_TABLE_ - get_got, %0  \n\t"
                  : "=r"(got));
     /* ... */
     asm volatile("pushl %%ebx                \n\t" /* save ebx */
                  "movl sym1@GOT(%0), %%ebx   \n\t" /* retrieve sym1 address */
                  "movl %1,( %%ebx)           \n\t"
                  /* ... */
                  "movl sym2@GOT(%0), %%ebx   \n\t" /* retrieve sym2 address */
                  "movl %1, (%%ebx)           \n\t"
                  /* ... */
                  "movl sym3@GOT(%0), %%ebx   \n\t" /* retrieve sym3 address */
                  "movl %1, (%%ebx)           \n\t"
                  "popl %%ebx                 \n\t" /* restore ebx */
                  : : "r"(got), "r"(x));
  

It is of course possible to merge the two above asm statements into one. However the performance implications of the GOT address retrieval should not be overlooked, especially if the assembly code takes place inside a critical loop, which means the loop might have to be unrolled and the GOT address retrieval done outside the loop. Note also that _GLOBAL_OFFSET_TABLE_ is ELF-specific. On BSD and a.out systems you should use __GLOBAL_OFFSET_TABLE_. I am not aware of a way to do the same on Darwin / Mac OS X, which is a shame because it is the only widespread i386 platform where shared objects cannot be non-PIC. gcc silently generates non-PIC code (amd64) The amd64 architecture has the same problem as i386 when trying to access variables from assembly code. However, the solution is a lot easier: the GOT address can always be retrieved from the special %rip register. This is called rip-relative addressing:

 extern uint32_t sym1, sym2, sym3;
 
 void store_in_symbols(uint32_t x)
  
     asm volatile("pushl %%rax                       \n\t" /* save rax */
                  "movq sym1@GOTPCREL(%rip), %%rax   \n\t" /* retrieve sym1 address */
                  "movl %0,(%%rax)                   \n\t"
                  /* ... */
                  "movq sym2@GOTPCREL(%rip), %%rax   \n\t" /* retrieve sym2 address */
                  "movl %0,(%%rax)                   \n\t"
                  /* ... */
                  "movq sym3@GOTPCREL(%rip), %%rax   \n\t" /* retrieve sym3 address */
                  "movl %0,(%%rax)                   \n\t"
                  "popl %%rax                        \n\t" /* restore rax */
                  : : "r"(x));
  

amd64 also has more registers than i386, which makes the first discussed technique less prone to register shortage. Why bother? Good question. Currently many shared libraries simply disable entire chunks of inline assembly on the amd64 architecture. Whether these routines bring any performance benefit compared to the C alternative is uncertain, but without being able to build the code there is no way to tell. Even fewer developers care about i386, because this architecture can handle non-PIC shared objects quite fine (but then they are no longer really shared). I am in favour of making all shared objects PIC-friendly (ie. not necessarily build them as PIC if there is a valid reason not to, but at least don t make it too difficult to build PIC versions) because that makes it easier to port the code to Darwin. I also believe the memory gain and especially the cache gain when using real shared libraries is underestimated.

Gunnar Wolf commented on my recent series of init script posts. Thanks for the feedback.The thing is that I've been doing a lot of Debian stuff these days (such as our GSoC coordination, and lots of stuff you saw in this blog). But I shouldn't... instead I should be trying to finally get my diploma thesis done.I had my exams last july. If I had been focused, I could have been finished with my degree last christmas. Now it will likely by this fall. :-( All that is remaining is my final thesis. It should be written in 6 months, but I still havn't submitted my topic yet. So if you are wondering what took me 8 months already: I investigated two topics, but they didn't work out. Then I started researching for my 'current' topic, and I have a vision for it, but I still havn't managed to convince my professor (which of course is largely due to me not being able to convey this idea, and that the topic is in information management and representation, where it's probaly often hard to find a solid theoretical part. I have examples I'd like to do, and I hope they are actually convincing, but I have no idea how to 'prove' something here; it's too much about social aspects and human perception I guess, and I'm not really qualified or capable to do solid research in that field.)(And as you can probably imagine, I'm pretty fed up with studying now. A year ago, I'd have considered to go for a PhD afterwards, but now I just want to get out, and work on my projects.)Anyway, I got distracted. Back to the init topic.I already posted this link: Init Hackfest, as well as a pointer to the initscripts-ng mailing list. I'd really like to get this topic on it's way for lenny. So please, join this effort. While I'd like to push it, I shouldn't; on contrary, I should suspend my Debian efforts.As for your (Gunnar) suggestion on how to handle multiple inits: what's wrong with just including multiple init scripts in one package? They are not particularly large and compress well. A single init script package might already be bigger than all the init scripts.The problem is more on how to write these init scripts; by the time we're done with this project, we might actually have hit 1000 init scripts. Yes; I didn't just make up this number: packages.debian.org has 890 hits for /etc/init.d; the init-like scripts in /etc/dbus-1/event.d not included. So if just want to add support to 5 alternate init systems, this means we would need to write 5000 new init scripts. Ouch. So we maybe should convert them to 1000 meta-init-scripts instead, and generate the 6000 real init scripts from them.Please join the discussion, and lets get this project going, despite having to face such scaringly high numbers...

Today's post (previous parts) tries to show what benefits other init systems can offer.I'm picking 'minit' as an example; not because it's the ultimate solution or anything, but because I happen to know it quite well. I used to be the package maintainer for some time, and I've also used it on an embedded system (where it was a perfect match). However, it's not very distribution-friendly; it's great when you can invest some time into configuring it (such as when building an embedded system) but not when it should ideally work automatically out of the box. The Debian package is still there, but it's orphaned. Still it's well written, clean code; there shouldn't be any bigger bugs in there.The first thing to notice is that minit is tiny. It's statically linked, but only about 6.5 kb. Regular init is 86 kb plus some libraries (despite actually not doing much during init!). This makes it very well-suited for embedding.But now for the benefits of minit over regular init:1. Service monitoring. This is the biggest point: minit will keep an eye on the services and restart them when needed. (Another thing very useful in embedded situations, btw.)Minit doesn't use the start/stop paradigm of sysv/shell-style-init. Instead it has a "run" script for every service. If a service is flagged as 'respawn', it will automatically be restarted when the run script quits; therefore the applications should not background themselves. This offers some benefits, for example the init process will be notified by the kernel of a termination, and can immediately respawn the service; in contrast to other monitoring systems suchas monit it doesn't rely on 'busy polling'.2. Dependency handling. Minit handles startup dependencies. If a service lists another service as a dependency, it will bring that service up first.3. Parallel startup. Unless an ordering is given via dependencies, minit will start the services in parallel.4. Control apps. In minit, you don't start a service by running some random shell script. Instead you call "msvc -u servicename", which will signal minit to bring up the service (and it's dependencies). This would (there currently is no policy for minit) be especially useful in SELinux, but it also allows to e.g. give a user the permission of running "sudo msvc -u" without introducing a huge security hole, since this cannot be used to execute arbitrary scripts. "msvc" can also be used to check if a service is up. (Another benefit for embedded systems, since that removes the need for pid files, which then need a ram (writeable) filesystem)I'll also give an example for a minit service:A service is represented by a directory in /etc/minit which can contain various files. I'm going to look at getty/1 example; which will bring up a console login on the first terminal. There are three files:

• a symlink named 'run' to /sbin/getty, which is the daemon we want to bring up
• a file named 'params' containing the parameters for the getty application, one per line
• an empty file named 'respawn', since we want the login screen to reappear

If the service would need it, there could also be a file called "depends" listing service dependencies.This allows minit to start the getty with no overhead. There is no shell script being executed or anything (note: sysvinit also does this for gettys, but it's not being used much for other services).Of course minit isn't perfect. I've already said that it isn't particularly distribution friendly. This is because it uses a lot of magic files (e.g. an empty file named 'respawn' to flag a service as needing to be respawned), and it's configuration files aren't easy to handle automatically. The often frowned-upon 'symlink farms' of sysv-shell-style init are much better here.Another thing that is missing from minit is an automatic rollback of dependencies. If we told minit to bring up service 'a', which lists dependencies 'b' and 'c', minit will bring these three up. If we tell it to shutdown 'a', it will however not shutdown 'b' or 'c'. And unfortunately, even if we shutdown 'b', it won't shutdown 'a'. If you want a more complex dependency handling (or things such as runlevels), you need to do this externally. (An example is my 'enitdir' app, it adds runlevel support to minit, and allows switching of runlevels by replacing a symlink; Services are added and removed from runlevels by adding a symlink to the service into the runlevels directory. By using dnotify, this also takes virtually no CPU time.)Minit also lacks a differentiation between a 'started' and a 'running' service. It has support for 'sync' services, such as e.g. mounting a filesystem. It will wait for this operation to complete before it starts services depending on it. However, there are services that will continue to run, while not being instantly available when the exec() was called. Sometimes it would be nice if a service could go into a 'started' state, then do some cleanup work, and then when actually running the real service signal minit that it's now 'running'.As a last malus, minit won't print progress messages to the console. A minit boot is pretty much silent, directing it's output to some logging processes. There are good reasons for this (since services are started in parallel!), but it makes debugging harder, and users might be wondering what their system is actually doing (though some users prefer to see a colored 'busy' bar)...I guess this concludes this series of blog posts. Maybe someone can follow up with some details on upstart (which seems to be the most promising init replacement)? I hope to have given an impression on where there is room for improvement in our current init to one or the other. Now we just need some people to actually work on a better support for other inits in Debian...

I have just read the Presentation Zen blog post about PowerPoint.

One of the interesting suggestions was that it's not effective to present the same information twice, so you don't have notes covering what you say. Having a diagram that gives the same information is effective though because it gives a different way of analyzing the data. I looked at a couple of sets of slides that I have written and noticed that the ratio of text slides to diagram slides was 6:1 and 3:1 in favor of text, and that wasn't counting the first and last slides that have the title of the talk and a set of URLs respectively.

So it seems that I need more and better diagrams. I'll include most of the diagrams I use in my current SE Linux talks in this post with some ideas on how to improve them. I would appreciate any suggestions that may be offered (either through blog comments or email).

The above diagram shows how the SE Linux identity limits the roles that may be selected, and how the role limits the domains that may be entered. Therefore the identity controls what the user may do and in this example the identity "root" means that the user has little access to the machine (a Play Machine configuration). I think that the above is reasonably effective and have been using it for a few years. I have considered a more complex diagram with the "staff_r" role included as well and possibly including the way that "newrole" can be used to change between roles. So I could have the above as slide #1 about identities and roles with a more detailed diagram following to replace a page of text about role transition.

The above diagram shows the domain transitions used in a typical system boot and login process. It includes the names of the types and a summary of the relevant policy rules used to implement the transitions. I also have another diagram that I have used which is the same but without the file types and policy. In the past I have never used both in the one talk - just used one of the two and had text to describe the information content of the other. To make greater use of diagrams I could start with the simple diagram and then have the following slide have all the detail.

The above diagram simply displays the MCS security model with ellipses representing processes and rectangles representing files.

The above diagram shows a simplified version of the MMCS policy. With MMCS each process has a range with the low level representing the minimum category set of files to which it is permitted to write and the high level representing the files that it may read and write. So to write to a file with the "HR" category the process must have a low level that's no higher than "HR" and a high level that is equal or greater than "HR". The full set of combinations of two categories with low and high levels means 10 different levels of access for processes which makes for a complex diagram. I need something other than plain text for this but the above diagram is overly complex and a full set is even more so. Maybe a table with process contexts on one axis, file contexts on another and access granted being one of "R", "RW" or nothing?

I also have a MLS diagram in the same manner, but I now think it's too awful to put on my blog. Any suggestions on how to effectively design a diagram for MLS? For those of you who don't know how MLS works the basic concept is that every process has an "Effective Clearance" (AKA low level) which determines what it can write, it can't write to anything below that because it might have read data from a file at it's own level and it can't read from a level higher than it's own level. MLS also uses a high level for ranged processes and filesystem objects (but that's when it gets really complex).

This last one is what I consider my most effective diagram. It shows the benefits of SE Linux in confining daemons in a clear and effective manner. Any suggestions for improvement (apart from fixing the varying text size which is due to a bug in Dia) would be appreciated.

The above diagrams are all on my SE Linux talks page, along with the Dia files that were used to create them. They may be used freely for non-commercial purposes.

If anyone has some SE Linux diagrams that they would like to share then please let me know, either through a blog comment, email, or a blog post syndicated on Planet SE Linux.

"Beta" is short for "BETrAying". It means the web site isn't honest to you about the benefits for you, what they are doing with the data or what they are telling you. Examples:

• Download services: 'all download slots are busy, please wait 15 seconds' - not true, but they just would like you to look at the ads for 15 seconds
• Social networking: Myspace claims "foo is in your extended network". Even when you are the Google robot or using an anonymizer. This is a hard-coded string, not a calculated connection.
• Most Web 2.0 services: we have no idea how to continue providing this service for free once our venture capital is used up.

So spelled out it would be like:

Sorry, we're not telling you the truth about our service and the benefits, and that's why we're calling the service beta; once we've found out how to do it right and still earn some money with it, we'll remove the beta sign.

"Beta" is short for "BETrAying". It means the web site isn't honest to you about the benefits for you, what they are doing with the data or what they are telling you.Examples:

• Download services: 'all download slots are busy, please wait 15 seconds' - not true, but they just would like you to look at the ads for 15 seconds
• Social networking: Myspace claims "foo is in your extended network". Even when you are the Google robot or using an anonymizer. This is a hard-coded string, not a calculated connection.
• Most Web 2.0 services: we have no idea how to continue providing this service for free once our venture capital is used up.

So spelled out it would be like:

Sorry, we're not telling you the truth about our service and the benefits, and that's why we're calling the service beta; once we've found out how to do it right and still earn some money with it, we'll remove the beta sign.

Yesterday, I wrote about our init actually being /bin/sh, since all sysvinit does it start a single shell script. Today, I'm going a bit into how services are brought up with sysv-style init. I won't be writing much about runlevels, since most users will not make use of multiple runlevels anyway. So what does a typical init script in /etc/init.d do? If you want to find out yourself, consider looking at /etc/init.d/skeleton, which is a template for init scripts. A modern init script should (as required by the LSB) start with a formatted comment with metadata such as dependencies and description. This is already trying to address some shortcomings of sysv-style init, but no actual code. The first thing most init scripts do is to check if their application is actually still installed. Init scripts are sometimes created by users, or are not automatically uninstalled since being treated as 'configuration files'; so it occurs often that the init script is still around even when the service was uninstalled. So they test if they're installed and executable. Next they probably read a configuration file, usually named /etc/default/packagename, with user settings (people used to do such changes in the init script itself, but for upgrading it proved to be a lot better to put all settings the user might want to modify there). The third step is to do the requested user action, which usually is one of 'start', 'stop', 'reload' and 'restart'. Some init scripts also have extra actions such as 'status'; not all init scripts support 'reload'. Doing such an action usually means to print out a status message and launching the service in the background (usually using the 'start-stop-daemon' daemon helper, which offers functionality to e.g. record the process PID in a file or forcing applications into the background that don't do so on their own), or locating the service (usually via a PID file, again this is commonly done using start-stop-daemon) and reloading or stopping it. Upon success, they again print a status message. An init script could be as simple as:

#!/bin/sh
if [ "$1" = "start" ]; then
  /usr/local/sbin/mycheapdaemon &
fi

There are many good reasons not to do it that way, but it will work. Some characteristics of sysv-style init scripts:

• When starting or stopping services manually, init is not involved. Or notified. (In particular this means init will still try to stop the service on shutdown, and will not try to stop services that aren't on it's list of services to stop.) - there is no system-wide state tracking.
• if a service quits, nobody notices
• init scripts do a best-effort to return a meaningful status if the startup of the service was successful. But often this is not possible (e.g. when a daemon backgrounds unconditionally before checking if it has a working configuration)
• init scripts are often run by the root user, and can sometimes even be executed successfully by non-root-users. This is particularly annoying with SELinux in 'strict' mode, that tries to differentiate between applications started by the system administrator and system services. For full protection, you need to use run_init /etc/init.d/script start, which will transition into the system user role, then start the service.
• many services aren't very cooperative and need the help of start-stop-daemon for writing pid files and/or backgrounding
• Since services are started/stopped by the admin occasionally (e.g. during upgrades), and init scripts are supposed to print a status message (these "Starting service foobar: foobard [done]" messages), they will have file descriptors open to the admins console. They will also inherit environment variables from the admin. When a daemon fails to handle them appropriately, odd things can happen. This ranges from services dying when the admin logs out (and they quite on a SIGPIPE signal triggered by the file descriptor being closed; this is why 'nohup' in the coreutils package exists. An example is IBMs tivoli backup software) to e.g. tomcat behaving differently when it has a $DISPLAY variable (i.e. started with a graphical output available).

Keep your eyes open for the next part of this series, where I'll show how some init systems improve on these issues.

Yesterday, I wrote about our init actually being /bin/sh, since all sysvinit does it start a single shell script.Today, I'm going a bit into how services are brought up with sysv-style init. I won't be writing much about runlevels, since most users will not make use of multiple runlevels anyway.So what does a typical init script in /etc/init.d do?If you want to find out yourself, consider looking at /etc/init.d/skeleton, which is a template for init scripts.A modern init script should (as required by the LSB) start with a formatted comment with metadata such as dependencies and description. This is already trying to address some shortcomings of sysv-style init, but no actual code.The first thing most init scripts do is to check if their application is actually still installed. Init scripts are sometimes created by users, or are not automatically uninstalled since being treated as 'configuration files'; so it occurs often that the init script is still around even when the service was uninstalled. So they test if they're installed and executable.Next they probably read a configuration file, usually named /etc/default/packagename, with user settings (people used to do such changes in the init script itself, but for upgrading it proved to be a lot better to put all settings the user might want to modify there).The third step is to do the requested user action, which usually is one of 'start', 'stop', 'reload' and 'restart'. Some init scripts also have extra actions such as 'status'; not all init scripts support 'reload'. Doing such an action usually means to print out a status message and launching the service in the background (usually using the 'start-stop-daemon' daemon helper, which offers functionality to e.g. record the process PID in a file or forcing applications into the background that don't do so on their own), or locating the service (usually via a PID file, again this is commonly done using start-stop-daemon) and reloading or stopping it. Upon success, they again print a status message.An init script could be as simple as:

#!/bin/sh
if [ "$1" = "start" ]; then
  /usr/local/sbin/mycheapdaemon &
fi

There are many good reasons not to do it that way, but it will work.Some characteristics of sysv-style init scripts:

• When starting or stopping services manually, init is not involved. Or notified. (In particular this means init will still try to stop the service on shutdown, and will not try to stop services that aren't on it's list of services to stop.) - there is no system-wide state tracking.
• if a service quits, nobody notices
• init scripts do a best-effort to return a meaningful status if the startup of the service was successful. But often this is not possible (e.g. when a daemon backgrounds unconditionally before checking if it has a working configuration)
• init scripts are often run by the root user, and can sometimes even be executed successfully by non-root-users. This is particularly annoying with SELinux in 'strict' mode, that tries to differentiate between applications started by the system administrator and system services. For full protection, you need to use run_init /etc/init.d/script start, which will transition into the system user role, then start the service.
• many services aren't very cooperative and need the help of start-stop-daemon for writing pid files and/or backgrounding
• Since services are started/stopped by the admin occasionally (e.g. during upgrades), and init scripts are supposed to print a status message (these "Starting service foobar: foobard [done]" messages), they will have file descriptors open to the admins console. They will also inherit environment variables from the admin. When a daemon fails to handle them appropriately, odd things can happen. This ranges from services dying when the admin logs out (and they quite on a SIGPIPE signal triggered by the file descriptor being closed; this is why 'nohup' in the coreutils package exists. An example is IBMs tivoli backup software) to e.g. tomcat behaving differently when it has a $DISPLAY variable (i.e. started with a graphical output available).

Keep your eyes open for the next part of this series, where I'll show how some init systems improve on these issues.[Update: next part]

The cardinal reason for the switch is collaboration: git scales. During my bzr usage time of the last year i observed the following shortcomings of bzr:

Speed - This improved a bit lately but not enough for my q&d usage.

Documentation - Beside the obvious "svn-style" commands the usage of the other commands isn't obvious and the man pages lack examples. git is not perfect in that regard, but you quickly find typical usage pattern on wiki tutorials or on many google hits.

Pager - On big diff output bzr doesn't pipe that to pager^Wless. I consider that as a big usability bug.

Branches - I recently learned that a bzr repository corresponds to a git repo with just one branch. This seems strangely complicated. I really love to have different branches ready to checkout in my pwd . This short-coming blocks cooperation as i'm not confident enough to fetch random changes from bug reporters in my own branch.

Minor Nitpicks:
* Push command - Defaults on the non-obvious old sftp protocol. Worked with rsync on some very ancient version.
* Repo format - Changes on every major upgrade.
* Patchlevel - Probably for svn compat reason the chosen default patchlevel is -p0 not -p1.

Latest initramfs-tools repo is available at:
git clone git://git.debian.org/git/kernel/initramfs-tools.git Thanks to tailor it is now possible to move from different version control systems. For git as target repository it is recommended to use latest upstream either directly via darcs or at least install the unstable version. The etch version throws strange python backtraces.

If you use grep-dctrl, sort-dctrl or tbl-dctrl, I’d like to see your use cases! I’m building a system test automaton for dctrl-tools, and I’d like to include as much real-world examples as possible so that any regressions get found before uploading. You can send example command lines (and preferably also a test file, unless it’s one of the standard Debian files like Packages) to me by email at <ajk@debian.org>, as a wishlist bug report against dctrl-tools or by a Darcs patch against the experimental repository (see instructions).

Running a Hospital is a blog actually written (not ghostwritten) by the CEO of a large Boston-area hospital, the Beth Israel Deaconess Medical Center. The blog covers hospital/health-care issues as well as some unrelated topics. The emergence of this sort of open and frank discourse from leaders (particularly in medicine) is a real sea change. Just a few years ago, this sort of relatively unedited discussion would have been shut down by the lawyers. It’s an interesting variation on the “medium is the message” concept. Of course, everything this CEO (and others) are writing doesn’t require the web or weblogs. The same content could easily have been provided in a printed newsletter (setting aside the minor cost issue). Weblogs have established new norms, however, that enable openness even in particularly liability-sensitive areas like medicine. For example, see this entry about fatal infections resulting from central lines. On a lighter note, should the hospital use cage-free eggs? (I said yes.) Technorati Tags: Weblogs, Health Care

Coq: Happy to See it as a Geek Toy Daniel, I'm really happy you defined Coq a geek toy. Usually such pieces of software are more seen as academic toys, seeing it promoted as a geeky software is a joy to me. I'm a developer of Matita, a concurrent system of Coq (yet less mature a of today). I'm happy to give some more info about Coq (and hence about Matita, which is based on the same logical foundation of Coq). Coq is not really a programming language (even if the CoqArt book says so, to attract the developer community), it is mainly a proof assistant; i.e. a software in which you give definitions, state theorems and prove them. The added value of the system wrt plain pen and paper is that it grants that you won't be able to produce incorrect proofs (of course you are still free to prove useless theorems ). It is more expressive than predicate calculus, technically it is based on the Calculus of (Co)Inductive Constructions, but the most important differences are that you can give inductive definitions of predicates (or define recursive data structures) and the system will let you play with them freely automatically constructing recursors on these data types. You can even define infinite data structures (using co-induction) mimicking lazy types available in languages as Haskell. Regarding the synergy of Coq with OCaml: yes, you can prove in Coq the correctness of OCaml program, but actually not more than you can do with programs written in other programming languages (see for example Why, Krakatoa, and Caduceus for some practical tools to prove properties about Java and C programs). Yet the synergy with OCaml is even more than that: once you proved properties like the existence of, say, a prime number greater than a given prime number, you can ask Coq to generate for you (using a feature called extraction) an OCaml program which consumes as input a prime number and returns the next one. More generally you can extract programs from (suitable) proofs. Some info on the Debian side: thanks to smimou Coq has been in Debian for a long time now and Why/Krakatoa/Caduceus are coming soon. Happy hacking!

Don't use the browser name for capability detection. I'd like to emphasize that. For example Google Groups won't let me upload an image to my profile, because I'm not running Firefox or Internet Exploder. Well, I tried both Epiphany (which has a very clean and fast UI, unlike Firefox which is totally cluttered) and Iceweasel (which is Firefox, but with the trademark replaced). Both are using Gecko, Gecko/20070324 for Epiphany and Gecko/20070310 for Iceweasel. So I'm very sure they have the same capabilites as Firefox 2 when it comes to web sites. If you want to test for Firefox's capabilities, use the Gecko version number.. Thank you. [Update: it was suggested I point people to GeckoIsGecko.org, which has links on how to properly detect the Gecko engine, instead of relying on the browser name.] [Update: Mike Hommey pointed out that the 'Gecko/date' string is mostly meaningless, and largely is the build date; it doesn't contain tree information. Instead you should be using the "rv: 1.8.0.11" part of the User-Agent. This is also what the getGeckoRv function on the howto linked from GeckoIsGecko.org does. Oh, and Firefox 2 is not gecko 2, but IIRC uses gecko 1.8.x, just like my Epiphany.]

You probably have read it alread Gaim will be renamed to Pidgin. Most likely because of AOL being unhappy with them using AIM in their name (although they support a dozen other messengers as well; getting rid of AIM in the name was also 'wanted' by the Gaim developers I guess). However, such software renames eventually cause some problems. For example, will the API also be renamed? Will libgaim become libpidgin? Will the DBus interfaces change their name? A dozen of applications might be affected by that, including the popular AdiumX messenger for OSX. (One of the prettiest I've seen so far, btw.) Other examples of such renames (and the problems caused by that) is the Mozilla chaos. Mozilla doens't allow to distribute modified versions of Firefox or Thunderbird under these names, so Debian has to replace the names (though most of our patches at some point are adopted by upstream, or they'd be okay with them, Debian also has the requirement that our users must be able to do their own modifications!). This is why etch just shipped with "Iceweasel" and "Icedove" instead. However some web sites don't test for the browsers "Gecko" version, and for example Google Groups, when trying to set the profile image, tells me I need to run Firefox or Internet Explorer. Though I'm pretty sure the function would just work fine with both Epiphany (my primary browser) and Iceweasel. The transition from "cdrecord" to "wodim" was a bit smoother. On the one hand, many application were already expecting functionality not present in the "official" cdrecord (and yes, Joerg, I know that your opinion might differ on that point - I don't care, nor does anybody else I know). On the other hand, it mostly affects programs included in the distribution, so they can be easily handled. There aren't random websites testing for it either, and custom scripts written by users can probably handled via a wrapper (this is much more difficult for libraries I guess, or the DBus API of Gaim. Fortunately for Gaim, these interfaces aren't used much, so it can be solved with a simple search and replace and recompilation in affected applications). I think when more distributions switch from Firefox to Iceweasel because of realizing that it might be important to some of their users and recommend Epiphany because of it's smoother user interface, we might also see less websites doing incorrect browser capability detection.

lkcl: Ratings and reports are explained in the Advogato FAQ, by the way. One thing I can't tell whether you realized or not: The report at /rating/report/lkcl does not show what other people think of your diary. It shows what you (and the people you certified) think of other users' diaries. The lumpiness of the histogram is because most users don't use the full range of ratings. And since confidence scores fall off exponentially by network distance, your confidence-weighted histogram will be skewed toward ratings given out by yourself and people you directly certified. For example, I tend to rate most diaries between 4 and 9, so I expect my report to be skewed toward those values.

Today - as many of you have already noticed - is/was a big day for Debian:

• Sam Hocevar won the DPL election


• Debian (old)stable 3.1 released a new version


• Debian Etch 4.0 was released

Well, I could add another topic:

• Debian is no longer Debian for me because not all of my archs are supported anymore

In a more detailed view:
Sam Hocevar won the DPL election
Congrats to Sam! I can agree with most of the points he made in his DPL platform, e.g. a sexier website, sexier distribution, remaing the universal OS, etc.
sexier website: Really, the current design of Debians website is awful. It's really something back from the 90s. Bad look, bad design and stuff you're looking for is difficult to find, because it's badly structured and stuff is spread all over the place. I think, it would be good to have some decent CMS and some good designers/graphic artists - a good package maintainier doesn't make a good webdesigner.
sexier distribution: Many people are using Ubuntu because they're desktop users. The current release cycles are way too long and drive people away from Debian. It's a different matter, of course, for servers or managed systems like Terminal Servers or such. Those are fine with longer release cycles. Splitting up the release in some core and sub releases is a quite common proposal. Yes, this split is difficult, but I think it's the way to go.
remaing the universal OS: As of the release of Etch, Debian became the no-more-universal OS - just because m68k wasn't released for obscure reasons. I'm a great m68k user and now I'm left behind by Debian with the problem that my pet arch is not supported anymore. Running testing/unstable is not an option for m68k IMHO. Because the m68k porters didn't come up with an alternative release plan so far, I think it's the best to shutdown my 3 m68k machines - all of them are m68k buildds - within the next weeks or months. They're producing (well, some little) heat, some disturbing noise and consume a lot of power of course. Currently I accept those drawbacks in order to help the m68k port by donating CPU power running some buildds for about 7 years now. I've spent much more money on this purpose - not only for paying the electricity bills - for phone calls to get crashed remote machines rebooted, sending replacement parts for other buildds or even driving by car to bring machines or other parts to other persons (like crest.debian.org to Michael Schmitz or accel boards to the Netherlands for porting stuff. Anyway, I will happily reactivate the machines when it's up to make a new release for the m68k architecture again! :)

Some other point Sam mentioned in his platform is this:

There are very useful services around, such as http://www.buildd.net/, http://bts.turmzimmer.net/ or http://bjorn.haxx.se/debian/. I cannot understand why they are not Debian services.

I already had a conversation with Sam about this in which I expressed my general intention to develop at least Buildd.Net beyond its current state - it's quite frustrating for me to see Buildd.Net being stuck in development just because I've no more time and skill to develop Buildd.Net further. Sadly, my call for help wasn't as successful as I wanted. Only two persons wanted to help: faw and joey - and both are quite busy with other stuff.
Apparently, Buildd.Net will most likely suffer from shutting down my m68ks due to the missing m68k Etch release as well, because it was always a frontend to my m68k buildds in my eyes. All other stuff on Buildd.Net is just a nice addition to that. And when I don't operate m68k buildds on a daily basis, it's obvious that I won't invest much more time into the development of Buildd.Net. This is already true for the past year where I haven't found the time to fix some serious problems in the backend scripts and database of Buildd.Net, which basically breaks store_packages2.py (source and therefor ptracker.cgi(example) as well. I even haven't found the time to setup trac and svn again after the move to the new server during last autumn.
Erm, anyway.. back to the topic: I wish Sam the best for his period - but I don't think that he can make a big change. I doubt in general that any DPL is able to push big changes in Debian. One reason for this is the short period of just one year. Another reason is that many DDs won't let big changes to be made.

For a few months I have been spending a lot of time using a Mac running OS/X for 40 hours a week. Recently a discussion started at a client site as to whether Macs or PCs should be used for future desktop machines.

The Apple hardware is very slick, everything fits together nicely and works well. For example my Apple monitor connects to my Mac via a single cable that supplies power, USB, and the video signal. My USB keyboard and mouse connect to my monitor. So my Mac has only three cables connected to it, power, Ethernet, and the monitor cable. The same thing on a PC would require an additional USB cable going to the PC and an additional power cable going to the monitor.

That is just one trivial example of how Macs are slick.

The advantage of a PC is that installing Linux is much easier and better supported (the vast majority of Linux users have PCs). The benefits of slick hardware are greatly diminished if it's only partly supported in terms of drivers and/or getting answers to technical questions on the net.

Another issue is that of software compatibility. When doing Linux work having a desktop machine that runs X is a significant benefit - even if all your work is text based. The Apple X server that I have installed never worked properly and it's a hack like the various Windows X servers. To make things worse the current versions of OpenOffice for Mac use X and therefore don't work for me - I've been told that the next version of OpenOffice will fix this.

Then there's the issue of price. For a Mac you would want to have a machine with an Intel CPU which means a recent and expensive machine. For a PC the benefits of a 64bit machine over an old P3 or P4 are very small. For all the work I do a refurbished IBM or Compaq P3 machine would provide all the performance I require, have better software compatability than a Mac, and cost only $120 for a machine.

Also I imagine that some common and cheap PC expansions (such as KVM switches) would cost a lot more for a Mac.

You can run Linux on a Mac, but what's the point? You lose some of the hardware features so the Mac becomes an expensive PC that's not entirely compatible. I'd rather have a P3 PC on my desktop than the latest Mac.

But I wouldn't rule out recommending a Mac to other people. People who have no-one to help them run Linux are best advised to use a Mac. Of course Windows isn't viable due to security problems.

Add to my Netflix Wishlist (that is, the list of features I wish Netflix had, not the list of movies I wish I had): links from IMDB entries to the respective Netflix page (or, better yet, “add to Netflix queue” directly from IMDB). Links back to IMDB from Netflix would also be nice. This seems like something that would be relatively easy to do with a Mozilla plugin. In general, Netflix would be a prime candidate for a developer/hobbyist ecology similar to that which has grown up around Amazon’s API. The only examples I could find were this Perl module and this Moveable-Type plugin. More development along these lines could be a key advantage for Netflix to preserve its now incumbent position in the market. (Does anyone have any local video stores left? Ours closed down about a year ago.) Update: James points to an impressive list of Greasemonkey/Netflix scripts, two of which seem to do the job: Netflix links in IMDB, IMDB links in Netflix. Another featuer I’ve been looking for: Netflix Ratings Granulizer (allows partial stars in ratings). Thanks, Lazyweb! (Note that these sorts of screen-scraping hacks inevitably break when the content provider modifies its presentation; an open API would obviously be preferable).