Search Results: "ag"

27 October 2006

Simon Richter: Teardown

Scott writes about "teardown", an effort to not run daemons' init scripts' "stop" command when the system is shutting down anyway and all the script would be doing is to send a SIGTERM. I generally like that, not because I'm a speed freak (in fact, I use the shutdown procedure really seldom), but because it forces people to write clean applications that can deal with failures in other applications better. When I'm no longer guaranteed that the database connection is still around when I'm sent a SIGTERM, then I have to design my database so that I can easily recover when my daemon is started up again.

Enrico Zini: dspam-tweaks

Tweaking my dspam setup I was recently disappointed with my dspam setup: while it was catching lots of spam and it would never misclassify ham, I would still get like 50 spams a day in my inbox. In a moment of enlightenment, I realised that it could be because I'm subscribed to loads of mailing lists many of which I read only every so often. This means that I don't feed dspam those spam messages that manage to go through those lists, and so dspam gets to figure out that they are innocent, and therefore ends up with too broad an idea of innocence. The solution was heroically simple: plug dspam only after sorting mailing lists in their own folders, and only on the remaining personal mail. Not so simple was the implementation, since I had dspam integrated with postfix and it would necessarily process all the mail I get. This is what I did:
  1. I removed dspam hooks from postfix
  2. I hooked dspam into procmail:
    :0fw
      /usr/bin/dspam --user=enrico --stdout --deliver=innocent,spam
    :0
    * ^X-DSPAM-Result: spam
    .spam/
    
  3. I figured out that dspam run from postfix in that way would not make use of the server, not even if I use --client. I thus disabled the dspam server in /etc/default/dspam. This is a shame, but luckily I don't have many users and the server load won't suffer much.
  4. I fixed the privileges under /var/spool/dspam so that the users can read their dspam data but not the data of other users. Here's what I conjured:
    drwxrwxr-x dspam  dspam /var/spool/dspam/
    drwxrwxr-x dspam  dspam /var/spool/dspam/data
    drwxrwxr-x dspam  dspam /var/spool/dspam/data/local
    drwxrwx--- enrico dspam /var/spool/dspam/data/local/enrico
    drwxrwx--- enrico dspam /var/spool/dspam/data/local/enrico/enrico.sig
    -rw-rw---- enrico dspam /var/spool/dspam/data/local/enrico/enrico.*
    drwxrwx--- pippo  dspam /var/spool/dspam/data/local/pippo [...]
    drwxrwx--- pluto  dspam /var/spool/dspam/data/local/pluto [...]
    drwxrwxr-x dspam  dspam /var/spool/dspam/opt-in
    drwxrwxr-x dspam  dspam /var/spool/dspam/opt-in/local
    -rw-rw-r-- dspam  dspam /var/spool/dspam/opt-in/local/enrico.dspam
    -rw-rw-r-- dspam  dspam /var/spool/dspam/opt-in/local/pippo.dspam
    -rw-rw-r-- dspam  dspam /var/spool/dspam/opt-in/local/pluto.dspam
    -rw-rw---- dspam  dspam /var/spool/dspam/system.log
    
    the opt-in files are actually empty files and dspam seems to only stat() them, so they are probably ok even if they're not readable by users. This setup of permissions should allow dspam to run as the various users while fully preserving their privacy towards each others.
  5. In /etc/dspam/dspam.conf I commented out the Delivery options, since I let procmail take care of the delivery. I'm also not using ParseToHeaders anymore. The rest of the dspam configuration did not change.
  6. Instead of ParseToHeaders, I set up postfix aliases properly running dspam as the dspam user. The trick is to create an alias file owned by the dspam user and the dspam group, and tell postfix to use it:
    alias_maps = hash:/etc/aliases, hash:/etc/postfix/aliases-dspam
    alias_database = hash:/etc/aliases, hash:/etc/postfix/aliases-dspam
    
    /etc/postfix/aliases-dspam in turn contains:
    spam-enrico:" /usr/bin/dspam --user enrico --class=spam --source=error"
    notspam-enrico:" /usr/bin/dspam --user enrico --class=innocent --source=error"
    spam-pippo:" /usr/bin/dspam --user pippo --class=spam --source=error"
    notspam-pippo:" /usr/bin/dspam --user pippo --class=innocent --source=error"
    spam-pluto:" /usr/bin/dspam --user pluto --class=spam --source=error"
    notspam-pluto:" /usr/bin/dspam --user pluto --class=innocent --source=error"
    
    since I took care of leaving everything in /var/spool/dspam writable by the dspam group, this works perfectly. It also allows me to bounce a mail to the report address instead of forwarding it. With ParseToHeaders I had to forward because the bounced messages would not have the report address in their headers, and so dspam could not identify them. By the way, here's the mutt macro I use to report spam:
    macro index,pager <F12> "bspam-enrico@xxx.xx\n\ns=.spam\n" 'Report as spam'
    
This is it. It took a while as dspam isn't always straightforward to set up, but now I've finally stopped having to remove 20 spam messages from my inbox for every 1 legitimate mail I receive.

Josselin Mouette: Why the etch release will be late

It is currently rumored that, despite great efforts from the release managers and the dunc-tank board, the etch release will not make it on time. Some say this is because of Developers being upset by some random and obscure experiment. Some others think this is because there are too many RC bugs and that we won't be able to fix them on time. Some might say this is because of a surrealist conflict with the Mozilla corporation. Guys. You're all wrong. The main reason for the delay will be the same as that of the woody release. Developers will be too busy testing the brand new Frozen-bubble 2.0.0 release extensively. Instead of hacking, they will spend their nights fighting with each other in up to five players LAN and internet games, enjoying the shining new graphics. Debian unstable packages are available in incoming and tonight on your favorite mirror.

Simon Richter: Wacky ideas #9: rerun maintainer scripts for changes in related packages

Currently one thing that is really annoying for everyone is that all services need to be restarted when we upgrade libc, especially when NSS has changed. It is the duty of the libc postinst to do this, so it needs to know about those services. So the idea is to allow a package to specify "call my postinst again when any of there packages is changed". Since I expect most packages providing a daemon to essentially list all library packages depended on, there might be an incentive to combine this with an earlier idea, splitting the Depends field into ABI-dependent and -independent, and just have a hint field "rerun postinst on reverse dependency upgrade"

MJ Ray: The Dunc-Tank

First, my position: I'm in favour of paying people to work on debian, but I think the one-person job markets, lack of openness and trading on debian's goodwill makes the current configuration of Dunc-Tank unacceptable. This should not be a surprise: I questioned the one-person-time contracts right near the start, but - like so many questions - it went unanswered. Secondly, I think this is wrong for the project. Damage was forseeable and maybe could have been mitigated, but Debian Project Leader Anthony Towns chose to raise the stakes, taking the funding outside the project, leaving no control for the developers except a recall vote, as far as I can see. The DPL wasn't recalled, but is still partly to blame for the damage, no matter whether more than half of DDs support him or not. Finally, I'm unconvinced by Wouter Verhelst's arguments that USD 6000 is a fair price. The one-person market is nothing like a free market and that particular rationale seems wrong because:
  1. VAT is a Tax on Value Added, so the payment of 21% or whatever would be reduced by the VAT paid on any supplies, which reduces the effective total rate. Also, small businesses are VAT-exempt in some states. Finally, Stephen R. Langasek seems to be in Oregon and Oregon has no state sales tax [Oregon Department of Revenue] - does that mean no VAT either? So, still 6000 left;
  2. Suburban office space in Oregon looks like it costs USD 13.50/sf/yr fully-serviced (source), so a month for one person (175sf) would be about USD 200. Even downtown Portland is only about USD 300. (Where did 500 come from?) So, 5700 left;
  3. Utilities: it's GBP 40 for phone+internet here and the rest are included in fully-serviced, as far as I know, so let's say USD 100, even though USA is reputedly cheaper? 5600 left;
  4. Accountant fees are pretty much constant if you have a working business, but USD 3600 for a year? I'd expect closer to USD2000 (so about 200 per month) for a one-person business. SPI seems to pay USD1000 for bookkeeping, but I can't find a direct comparable - anyone know? 5400 left;
  5. Social and other insurances tend to be related to income, working out around to a third or so in the UK IIRC, so USD2000? What's the current US rate? 3400 left;
  6. Double-taxation seems like something a good accountant should help to avoid - not heard of it being a significant problem often; 3400 left.
So, that leaves USD 3400, less some % for VAT maybe. Not well-paid IMO, but much more than living cost for many DDs. If any of the above figures are too high or low, please tell me and I'll update this. Of course, the actual payment rate is minor to me, compared to the problem of dunc-tank being designed as a market for lemons and being driven through by brinkmanship.

Enrico Zini: gpg-select-subkey

Telling gpg not to use the key in the card So, I created the subkeys for the OpenPGP card, and it works. Now I'd like to upload some Debian packages, but the uploads fail because my new subkeys aren't yet known to the Debian keyring. I tried to push my subkeys to keyring.debian.org, but uploading afterwards still was rejected. Maybe it takes some time for propagation, maybe there's some other procedure to follow, I don't know. I didn't manage to figure out what is the procedure for getting a new subkey in the Debian keyring. I wish to replace this paragraph with proper details if I'll ever find out. Now, failing to use the subkeys, I had to convince gpg to use my good old main key. The quick and dirty way was to make a backup of the keyring, delete the subkeys, sign and upload. Seconds after hours of searching terminated in the above crude hack, as it normally happens, someone (Holger in this case) suggested the correct way to do it: use --default-key and append an exclamation mark at the end of the key ID. This was in the gpg manpage, but nowhere near the documentation of --default-key:
Note that you can append an exclamation mark (!) to key IDs or fingerprints.
This flag tells GnuPG to use the specified primary or secondary key and not
to try and calculate which primary or secondary key to use.
So, now I'm happy:
$ gpg --sign  --default-key '797ebfab!'
You need a passphrase to unlock the secret key for
user: [...]
$ gpg --sign
gpg: signatures created so far: xx
Please enter the PIN
[sigs done: xx]

Jonathan McDowell: Another year, Another Expo

I spent the past couple of days at LinuxWorld UK, mostly on the Debian stand. I'm not sure I'd go as a punter these days; I don't think a lot changes year on year. However I do think it's good for us to turn up and show our faces so people can come and talk to us and realise we're wonderful people. Plus it's always good to see people I haven't seen in ages. We sold a lot of T-Shirts (even the red one with yellow writing that looks like something Phil would come up with after too much vodka [I bought one of these myself]). We didn't sell a lot of DVDs, but that's been happening since broadband became more pervasive. There were a reasonable number of people who just stopped to chat (shouting "Use Debian, it's great!" at people as they walked by was surprisingly effective). Lots of questions about how we felt about Ubuntu. Not really many about why we hadn't released yet which is a change! I also found a new toy I think I want; the Nokia E70. Steve has one of these and it looks pretty sweet; 3G+WLAN, a keyboard and a screen that, while not that big, manages a pretty good PuTTY session in 80x43 or so. And all of this in something not that much larger than a normal phone. That's appealing. I'd been sort of lusting after an HTC TyTN, but reports of poor reliability have put me off. And Simon has put me off the 9300i by saying it's quite slow as a phone. Finally, courtesy of Trexy, I present a new, work safe, goat sex: Goat Sex And now I will spend the next week sleeping I think.

Jacobo Tarr&#237;o Barreiro: Dunt thank

Julien: ask and ye shall receive . Anthony Towns on a dunk-tank I took the liberty of preempting Jaldhar’s GIMPing of a piece of headwear on the photo (no, contrary to popular belief, it’s not a wasps’ nest) ;-) Original found in this LWN article.

Erich Schubert: Google SoC and etch

Despite having blogged a lot already today (no, I didn't break planet, these are all new blog entries), I have one more thing to share with you. This image is from the Google Summer of Code mentor summit. Well, strictly speaking it's from the hotel where most of the attendees were staying. This draw was called the "executive toy box". I somewhat enjoyed that there was an "etch" around even before I arrived at Google. ;-) Etch-a-sketch

Lucas Nussbaum: Debian, Ubuntu, Mozilla Firefox , wearing hats, and Epiphany

With my Ubuntu hat, I strongly regret that no Ubuntu developer (to my knowledge) stepped forward to point out the inaccuracies in Chris Beard’s (Mozilla Vice President of Products) Q&A on Mozilla Firefox in Ubuntu. Letting people bash Debian when Ubuntu gets so much stuff from Debian isn’t nice at all. With my Debian hat, I really like how Mike Hommey chose to discuss facts, with references, in his reply to Chris Beard’s post. When it is possible, going back to the facts is really the best thing to do to avoid the usual FUD. It’s not really a surprise that this intelligent behaviour came from the Debian side. I have been a very happy user of Epiphany since I made the switch from Firefox 3 months ago. I would recommend it to anyone still using Firefox who is tired of its slow startup times, and its lack of integration in the Gnome desktop.

Scott James Remnant: Having Left Debian

It’s been over a year now since my last proper upload to Debian, and nine months since I announced my intent to put aside working on Debian for a while. With Matthew Garrett’s resignation from Debian, several people have compared it to my own “resignation” from Debian. It has got my thinking about whether I currently intend to ever end my “Sabbatical” and return as an active Debian Developer. I think that the end of my love-affair with Debian started at Debconf last year where several developers treated those of us who also worked on Ubuntu quite rudely. Someone was attacked for wearing an Ubuntu t-shirt at the conference, while someone else was applauded for wearing a “Fuck Ubuntu” t-shirt. That’s where I realised that maybe I didn’t have as much in common with these people as I thought I did. I still don’t understand why Debian singles Ubuntu out for this kind of treatment, we’re still the only derviative distribution that makes all of our patches to Debian available, yet Ubuntu is claimed to not do anything at all. Another example is that Ubuntu is being asked to change the Maintainer field of every package, something no other derivative is being asked to do or has ever been asked to do. Martin Krafft has had some interesting things to say about this strange relationship in the past. If that was the start of my falling out with Debian, I think that Debian considering removing documentation and firmware from the distribution, especially the documentation, was another point I started wondering whether I shared anything in common with the project anymore. Call me strange, but I think that one of the fundamental purposes of a Linux distribution is to be useful to its users. If nobody can use the distribution because it doesn’t support their hardware, and even if it did, all the documentation has been stripped out; I started to wonder what it’s aims are. It became increasingly apparent that the only users Debian was considering a priority were its own developer. And the third thing is simply a matter of Fun. Fun for me, at the moment at least, has been to build a system that fits together extremely nicely with each component doing the right thing. For Ubuntu this has meant being a driving force for it being Linux 2.6 only, with reliance on udev for hardware, etc. Upstart is just a continuation of these goals, getting a system which all “just works”, even if it means throwing out a few things people were previously fond of. All of these things would have been impossible to do in Debian itself. Getting upstart installable required changes to twelve different packages, including sysvinit itself; at a worst case, this would have required the agreement of twelve different maintainers in Debian. It’s often exhausting just persuading one of the reason for the change, persuading a dozen would have been a herculian effort. Perhaps Matthew is right, what Debian lacks is a single leadership. There’s nobody in Debian I could have gone to for approval over the changes I wanted to make, whose decision developers cannot overrule. Ubuntu has such people (the Technical Board and sabdfl), which gives the project an obvious direction instead of a couple of hundred people all pulling in different directions. In a way, I don’t feel that I’ve left Debian. I feel like I was happily going along, only to realise the mob had gone in a different direction and with no easy way of rejoining the group.

Scott James Remnant: Upstart can now replace sysvinit

Today I reached another milestone in the development of upstart, the packages in universe can now replace the existing sysvinit package. Before trying this, make sure your installation is up to date as we’ve had to split out some parts of sysvinit into a new sysvutils package. If you’re up to date, and want to try it out, install the upstart and upstart-compat-sysv packages from universe. Note that the first reboot after you’ve installed the packages (from sysvinit to upstart) will be a little tricky … use reboot -f. If your system boots and shuts down normally, everything’s working just fine. Note that both will be somewhat more quiet than you’re used to, unless you have usplash running. Throughout the rest of this entry, I’ll try to answer some of the questions and comments that I’ve received since the last post. Events As I talked about previously, upstart is an event-based init daemon. Events are the primary force for having your services and tasks started and stopped at the appropriate time and in the appropriate order. So what are events and where do they come from? (Note that this part is under development, so may change in later releases). Events are just simple strings that may be sent by any process when something it is tracking the state of changes. They have no state or longevity, and if, when queued, they do not cause any job state changes, then they have no effect unless they are sent again. Jobs can list which events cause them to be started if they are not already running and which events cause them to be stopped if they are running. Multiple start and stop events may be listed, in which case the first to occur changes the job until the next one occurs. upstart itself generates the following system events: The shutdown tool included in the package also causes one of the following events to be sent once the “shutdown” event has been handled: Jobs also generate events whenever they change state, this is the primary source of events for ordering: And as mentioned, any other process on the system may send events through the control socket or just by using initctl trigger EVENT. For now this is just the event string, however it’s intended that the event may include other details including environment variables and even file descriptors. Typical example To clarify how it all hangs together, here’s an example (using fictional names) of how the tasks and events can be arranged to provide race-free mounting of filesystems. By breaking this job into these small tasks, we can see how the pieces fit together. Because everything is now done on events, there are no race conditions; we know that any filesystem listed in /etc/fstab will be checked and mounted. The only reason they wouldn’t be is if there’s an error of some kind, and that means you have larger problems anyway and the system administrator would have a shell to fix it. Of course, the moment they finish checking the filesystem and mount it, the boot process would carry on. There’s no reason that any of these events need to be generated by the upstart daemon itself, it can receive them from any other daemon on the system such as udev, acpid, etc. This keeps the focus of the init daemon narrow. A large part of the future development will be working out exactly what kinds of events we want init itself to generate, what kinds we want to come from elsewhere, and what the contents of an event can be. Getting Involved If you want to get involved with trying to nudge the direction of upstart development, you can join the upstart-devel mailing list at http://lists.netsplit.com/ Or if you just want to grab the source code, tarballs are published at http://people.ubuntu.com/\scott/software/upstart/ and the bzr archive is at http://bazaar.launchpad.net/\keybuk/upstart/main

Scott James Remnant: Upstart in Universe

upstart is a replacement for the init daemon, the process spawned by the kernel that is responsible for starting, supervising and stopping all other processes on the system. The existing daemon is based on the one found in UNIX System V, and is thus known as sysvinit. It separates jobs into different “run levels” and can either run a job when particular run levels are entered (e.g. /etc/init.d/rc 2) or continually during a particular run level (e.g. /sbin/getty). The /etc/init.d/rc script is also based on the System V one (and is in the sysv-rc package), it simply executes the stop then start scripts found in /etc/rcN.d (where N is the run level) in numerical order. Why change it? Running a fixed set of scripts, one after the other, in a particular order has served us reasonably well until now. However as Linux has got better and better at dealing with modern computing (arguably Linux’s removable device support is better than Windows’ now) this approach has begun to have problems. The old approach works as long as you can guarantee when in the boot sequence things are available, so you can place your init script after that point and know that it will work. Typical ordering requirements are: This worked ten years ago, why doesn’t it work now? The simple answer is that our computer has become far more flexible: We’ve been able to hack the existing system to make much of this possible, however the result is chock-full of race conditions and bugs. It was time to design a new system that can cope with all of these things without any problems. What we needed was an init system that could dynamically order the start up sequence based on the configuration and hardware found as it went along. Design of upstart upstart is an event-based init daemon; events generated by the system cause jobs to be started and running jobs to be stopped. Events can include things such as: In fact, any process on the system may send events to the init daemon over its control socket (subject to security restrictions, of course) so there is no limit. Each job has a life-cycle which is shown in the graph below: The two states shown in red (“waiting” and “running”) are rest states, normally we expect the job to remain in these states until an event comes in, at which point we need to take actual to get the job into the next state. The other states are temporary states; these allow a job to run shell script to prepare for the job itself to be run (“starting”) and clean up afterwards (“stopping”). For services that should be respawned if they terminate before an event that stops them is received, they may run shell script before the process is started again (“respawning”). Jobs leave a state because the process associated with them terminates (or gets killed) and move to the next appropriate state, following the green arrow if the job is to be started or the red arrow if it is to be stopped. When a script returns a non-zero exit status, or is killed, the job will always be stoped. When the main process terminates and the job should not be respawned, the job will also always be stopped. As already covered, events generated by the init daemon or received from other processes cause jobs to be started or stopped; also manual requests to start or stop a job may be received. The communication between the init daemon and other processes is bi-directional, so the status of jobs may be queries and even changes of state to all jobs be received. How does it differ from launchd? launchd is the replacement init system used in MacOS X developed as an “Open Source” project by Apple. For much of its life so far, the licence has actually been entirely non-free and thus it has only become recently interesting with the licence change. Much of the goal of both systems appears initially to be the same; they both start jobs based on system events, however the launchd system severly limits the events to only the following: Therefore it does not actually allow us to directly solve the problems we currently have; we couldn’t mount filesystems once the “filesystem checked” event has been recived, we couldn’t check filesystems when the block device is added and we certainly couldn’t start daemons once the complete filesystem (as described by /etc/fstab) is available and writable. The launchd model expects the job to “sit and wait” if it is unable to start, rather than provide a mechanism for the job to only be started when it doesn’t need to wait. Jobs that need /usr to be mounted would need to spin in a loop waiting for /usr to be available before continuing (or use a file in a tmpfs to indicate it’s available, and use that modification as the event). This is not especially surprising given that Apple have a high degree of control over both their hardware and the actual underlying operating system; they don’t need to deal with the wide array of different configurations that we have in the Linux world. Had the licence been sufficiently free at the point we began development of our own system, we would probably have extended launchd rather than implement our own. At the point Apple changed the licence, our own system was already more suitable for our purposes. How does it differ from initng? Initng by Jimmy Wennlund is another replacement init daemon intended to replace the sysvinit system used by Linux. It is a dependency-based system, where upstart is an event-based system. The notion of a dependency-based system is interesting to talk about at this point. Jobs declare dependencies on other jobs that need to happen before the job itself can be started. Starting the job causes its dependencies to be started first, and their dependencies, and so on. When jobs are stopped, if running jobs have no dependencies, they themselves can be stopped. It’s a neat solution to the problem of ordering a fixed boot sequence and the problem of keeping the number of running processes to a minimum needed. However this means that you need to have goals in mind when you boot the system, you need to have decided that you want gdm to be started in order for it, and its dependencies, to be started. Initng uses run levels to ensure this happens, where a run level is a list of goal jobs that should be running in that run level. It’s also not clear how the dependencies interact with the different types of job, a dependency on Apache would need the daemon to be running where a dependency on “checkroot” would need the script to have finished running. Upstart handles this by using different events (“apache running” vs. “checkroot stopping”). Again while interesting, Initng does not solve the problems that we wanted to solve. It can reorder a fixed set of jobs, but cannot dynamically determine the set of jobs needed for that particular boot. A typical example would be that if the only dependency on the job that configures networking is the mount network filesystems job, then should that job fail or notbe a goal (e.g. because there are no network filesystems to be mounted) the result is that network devices themselves will not be configured. You could make everything a goal, and just use the dependencies to determine the order, however this is less efficient than just ordering the existing sysv-rc scripts (which can be done at install time). Another example is that often you simply don’t know whether something is a dependency or not without reading other configuration, for example the mount network filesystems may be a dependency of everything under /usr or may just be a dependency of anything allowing the user to login if it just mounts /home. The difference in model can be summed up as “initng starts with a list of goals and works out how to get there, upstart starts with nothing and finds out where it gets to.” How does it differ from Solaris SMF? SMF is another approach to replacing init developed by Sun for the Solaris operating system. Like initng it’s a dependency-based system, so see above for the differences between those systems and upstart. SMF’s main focus is serive management; making sure that once services are running, they stay running, and allowing the system administrator to query and modify the states of jobs on the system. Upstart provides the same set of functionality in this regard, services are respawned when they fail and system administrators can at any time query the state of running services and adjust the state to their liking. Will it replace cron, inetd, etc? The goal of upstart is to replace those daemons, so that there is only one place (/etc/event.d) where system administrators need to configure when and how jobs should be run. In fact, the goal is that upstart should also replace the “run event scripts” functionality of any daemon on the system. Daemons such as acpid, apmd and Network Manager would send events to init instead of running scripts themselves with their own perculiar configuration and semantics. A system administrator who only wanted a particular daemon to be run while the computer was on AC power would simply need to edit /etc/event.d/daemon and change “on startup” to “on ac power”. What about compatibility? There’s a lot of systems administrators out there who have learned how Linux works already and will not want to learn again immediately, there’s also a large number of books that cover the existing software and won’t cover upstart for at least a couple of years. For this reason, compatibility is very important. upstart will continue to run the existing scripts for the forseeable future so that packages will not need to be updated until the author wants. Compatibility command-line tools that behave like their existing equivalents will also be implemented, a system administrator would never need to know that crontab -e is actually changing upstart jobs. Does it use D-BUS?
“To D-BUS people, every problem seems like a D-BUS problem.” —Erik Troan
The UNIX philosophy is that something should do just one job, and do it very well. upstart’s one job is starting, supervising and stopping other jobs; D-BUS’s one job is passing messages between other jobs. D-BUS does provide a mechanism for services to be activated when the first message is sent to them, thereby starting other jobs. Some people have taken this idea and extended it to suggest that all a replacement init system need do is register jobs with D-BUS and turn booting into a simple matter of message parsing. This seems wrong to me, D-BUS would need to be extended to supervise these services, provide means for them to be restarted and stopped; as well as deal with being process #1 which means cleaning up after children whose parent’s have died, etc. It seems far simpler to arrange for D-BUS to send an event to init when it needs a service to be started, and focus on being a very good message passing system. The IPC mechanism used by upstart is not currently D-BUS because of various problems, however it’s always been expected that even if init itself doesn’t communicate with D-BUS directly, there would be a D-BUS proxy that would ensure messages about all init jobs and events would be given to D-BUS and D-BUS clients could send messages to init to query and change the state of jobs. What is the implementation plan? Because this is process #1 we are changing, we want to make sure that we get it right. Therefore instead of releasing a fully-featured daemon and configuration to the world, we’re developing it in the following stages:
  1. Principal development; at the end of this stage the daemon has been implemented and can manage jobs as described.
  2. Replacement of /sbin/init while running the existing sysv-rc scripts. This is the shake-down test of the daemon, can it perform the same job as the existing sysvinit daemon without any regressions?
  3. /etc/rcS.d scripts replaced by upstart jobs. These consitute the majority of tasks for booting the system into at least single-user mode, and contain many of the current ordering problems and race conditions. If the daemon solves the problems here, it will be a success.
  4. Other daemon’s scripts replaced by upstart jobs on a package-by-package basis; this will be an ongoing effort during which upstart will continue running the existing sysv-rc scripts as well as its own jobs. During this time the event system may be tweaked to ensure it truly solves the problems we need.
  5. Replcement of cron, atd, anacron and inetd. This will happen alongside the above and result in a single place to configure system jobs.
  6. Modification of other daemons and processes to send events to init instead of trying to run things themselves.
The current plan is that we will be at least part of the way into stage #3 by the time edgy is released, with that release shipping with upstart as the init daemon and the most critical rcS scripts being run by it to correct the major problems For edgy+1 we hope to have completed stage #5 and be at least part of the way into the implementation of stage #6. From the start of development of edgy+2, no new packages will be accepted unless they provide upstart jobs instead of init scripts and init scripts will be considered deprecated. What state is it in now? The init daemon has been written and is able to manage jobs as described above, receiving events on the control socket to start and stop them. This has now been uploaded to the Ubuntu universe component in the upstart package for testing before it becomes the init daemon. We welcome any experienced users who want to help test this; install the package and follow the instructions in /usr/share/doc/upstart/README.Debian to add a boot option that will use upstart instead of init. If your system boots and shut downs normally (other than a slightly more verbose boot without usplash running) then it is working correctly. Other types of events will be added as required during development and testing. Currently only a basic client tool (initctl) has been written, compatibility tools such as shutdown will be written over the next week or two before it replaces our sysvinit package.

Scott James Remnant: Parallel Peer Programming

Here at Canonical Towers we have several staff who worship at the altar of Extreme Programming, and as such many of the methodologies and rituals prescribed by that religion find their way into our day-to-day working practices. A few of these came together in an interesting way a few weeks ago, and it was suggested by a cow-orker that I blog about it so he could give the URL to people. The first ritual is that of the sprint, I don’t think this is orthodox XP but rather something inspired from it that we picked up from the Python community. With all of the Canonical developers scattered across the globe mostly working from their own homes, it’s become a useful tool; particularly for the Launchpad team. The basic idea for those that haven’t heard of it is simple, and perhaps obvious; you get a selection of the team together in one place, sit them around the same table with particular goals to complete. In effect it’s highly compressed facetime and high bandwidth interaction to nail those tasks that need it, before you head off again and work at a more sedate pace. The second is directly from XP doctrine and is that peculiar observance known as pair programming, something that is often coupled with sprints. For anybody who hasn’t encountered this before, it’s something I’ve always found rather odd. You get two programmers together, both itching to code, and you take one of them’s laptop away; and you don’t just force them to fight over the keyboard either, the laptop-less soul isn’t allowed to touch it. The theory here is that it frees the deprived individual of the hassle of coding and allows them to direct the programmer in ways that may not be immediately obbious, or alternately think about the next bit of work that needs doing. Another interesting side-effect is it can be an interesting way to learn code you’ve not worked with before, if you’re the one doing the coding and you’re being guided what to do by a sage who already knows it. I have a funny story here, so I’m going to digress from the main stream for a moment to tell it. A few months ago I was working in the London office, Mark had been up all night coding and had then given me responsibility to get his changes landed onto the Launchpad mainline. I’m not that hot on quite a bit of Launchpad, and fade to utterly clueless the nearer the code gets to the web application itself; and when trying to merge the two there were conflicts which I needed Mark’s help to resolve. Now as anybody who’s met Mark knows, he’s a pretty good example of a Type-A personality, yet he suggested that we do a spot of pair programming to get the code in and I’d be the one with the laptop as it’d help me get to grips with the affected parts. This hilighted something I’ve always seen as a problem with pair programming, dealing with it when the a person with the keyboard has a totally different way of working to you. In particular, I’m an emacs user, whereas Mark is a die-hard vim user; but also right down to the working directory I would work from, how I test results, etc. When you’re the one without the laptop, this can be quite infuriating, as you know exactly how to solve a problem and the idiot with the keyboard is dithering about doing things you don’t understand just to get there. If you’re a Type-A personality, you generally snatch the laptop away at this point and do it yourself; or at least you would, if you could drive the strange editor the other person uses. I swear Mark was sulking as he gave the laptop back and let me do it. Anyway, there is some relevance to that and we’ll come to it in a moment. The third and final methodology is the concept of unit tests and more specifically test driven development. For anyone who hasn’t come across this before, I highly recommend it; I was suspicious too when Robert Collins took it upon himself to teach me the true way, but now I’m sold. Simply the idea is that if you have any code that doesn’t have another piece of code in a test suite that checks that it’s working correctly, that code is broken. More particularly unit testing involves checking just one feature or requirement at a time, and stacking them together to test all of the code paths. I’ve actually found that this improves the APIs I design, as I write the code in small blocks and functions to make them easier to test. Test driven development takes it even further, you write the unit tests first, before you write the code they’re supposed to test; usually one or two at a time. Obviously these tests will fail at first, it’s then your job to write as little code as possible to make them pass. If your code isn’t right, you add a unit test that will fail, and modify the code to make it pass. This really comes into its own when you’re fixing bugs later, once you’ve identified the bug you write a test case or two that cause the problem; these will of course fail. You can then modify the code to make them pass, and at the same time be sure you’ve not broken other functionality because of the existing test suite. I’ve also found it really useful for particularly complicated or tricky pieces of code, especially those intricate algorithms that do particularly heavy lifting. So onto the event itself, this was a sprint in our London office to indoctrinate Gustavo Niemeyer with the various projects he’d be working on. The goal for this sprint was decided to start the conversion of HCT to Bazaar-NG, and to do this Gustavo and I would pair program. Now my last experience of pair programming had been that story with Mark (see, it had some relevance) and I knew I’d be just as bad if I wasn’t allowed to have the keyboard. I was also acutely aware that if Gustavo was just sitting by and watching, he wouldn’t get much benefit from it either, so he needed to be actively involved in the coding so he could learn the things he’d be working on. I came to what I thought was a pretty neat, and obvious, solution to these problems. We set up my laptop with the code we’d be working on, and on my display were two side-by-side terminals both running screen sessions. Gustavo then set up two same-sized terminals on his, ssh’d into my laptop from both of them and joined the screen sessions. In the left-hand one he ran vi, and in the right-hand one I ran emacs. Thus we both had keyboards, and both had editors, yet could see each other working and even steal the keyboard without danger of violence. We didn’t just sit side-by-side and code on different things though, that’s not pair programming and is just ordinary programming with a bit of a voyeuristic twist. What we did was: in my terminal I started writing the test cases for the code we needed, it was Gustavo’s job to write the code that would make them pass. We actually added a third terminal in which we could run the test cases themselves; so I could run them when I’d added something that would fail, and Gustavo could run them when he thought he’d made them pass again. This turned out to be a rather fun way to work, and at one point I was almost able to try and convince myself that the code was writing itself to pass the test cases I was writing. It also got me thinking that it’d be really neat to use genetic algorithms to breed code to pass test cases.

Scott James Remnant: Apologies to Planet Readers

Apologies to anyone reading my blog on a Planet for the sudden dump of old posts from my blog. In fact, as it’s already been pointed out, I should apologise twice as much since I wrote Planet itself and am doubly-responsible for the flood. In order to satisfy the suggestions from the like of Jono Bacon and Matthew Revell and actually blog more about the things that I do, I decided it was time to try some new blog software. Until now I’ve been using PyBlosxom, and while it’s popular with the geek crowd, I’ve never really got on with it. The principal problem is that it’s a pain to actually add blog entries to it, or to manage those that you’ve added previously. I’m slightly ashamed to say that I wanted something with a web interface. Thom May pointed me at Typo which seems pretty neat so far, and it doesn’t require PHP which can only be a good thing! So now when I don’t post, I won’t be able to blame the tools. Damn. Maybe this wasn’t such a good idea, after all?

Scott James Remnant: What I want in edgy+1

Now that edgy has been frozen for the beta release in a week’s time, and the next developer summit, where the plans for the next release will be discussed, has been announced, I think it’s the perfect time to start thinking about the kinds of features we want to see in that release. At our recent sprint in Wiesbaden, Mark reminded us that when Ubuntu first released we were leading the field in many of the components we installed by default. Many users came to us because we provided Linux 2.6, hotplug/udev and Project Utopia by default; along with the latest GNOME release and development framework. Now everybody does that, butwe led the way. With the dapper release, we produced something that could be supported for a long time. With edgy we’ve scratched our itches and fixed things under the hood that have been causing us problems. edgy+1 seems a good time to integrate the hottest new technologies and lead the way again. So what do I think is exciting, and what do I want in edgy+1? telepathy, farsight and galago Anybody who has used the 2006 software release for the Nokia 770 Internet Tablet will have seen these in action already. They’re, in my opinion, some of the most exciting projects currently being worked on; and ripe for integration into Ubuntu. Telepathy is a communication framework built on DBus that allows any application to establish communication with users all around the world, using any protocol from IRC to MSN to VoIP. It uses Farsight to provide the codecs and protocol support on top of the (GStreamer)[http://gstreamer.freedesktop.org/] media framework. And Galago glues it all together; providing presence information about contacts and allowing users to specify their own status. Galago Presence Applet This means that you’ll be able to contact your contacts. Your calendar alarm tells you that it’s a colleague’s birthday, you’d be able to click their name and see a selection of different ways that you can contact that person right now. From e-mail, opening an MSN or Jabber window or even establishing a VoIP or SIP call to them. The most exciting thing is that because every application on the desktop has easy access to this, people are coming up with fantastic ideas for using it! zeroconf By this, I mean zero configuration networking. Something the OLPC project appears to be going to do very well indeed. Where no existing network infrastructure exists, your computer should still be able to communicate with other nearby computers. It should use ad-hoc wireless networks, link-local IP addresses and server-free DNS resolution with libnss-mdns. Play games with your friends on the train without touching a button. Chat with other people in a lecture hall or at a conference where the existing network is unavailable. Hassle-free home and office networking without any complicated set up. Networking that “just works”. avahi Whether your computer is on an infrastructure or ad-hoc network, it should be able to locate resources on the network and share its own to other people. Avahi allows applications to, via DBus, do this by using a mulicast form of DNS; compatible with Apple’s Bonjour protocol. The immediately obvious benefits of this are that one can locate such things as file servers without any complicated directory; more fun things can be done too, such as sharing your music from RhythmBox with other users on the network. Combine it with telepathy, etc. and you can establish communication with anyone else, with a wide range of different protocols. The immediate problem here is that we currently have a “no open ports” policy, which if we’re going to give the users the opportunity to establish ad-hoc networks is almost certainly a very good thing. We already have exceptions for that policy for DNS and DHCP, it may be that Avahi (mDNS) joins that list. Another option is a “mobile-phone like” interface for enabling and disabling discovery and sharing. Bluetooth I don’t think we should stop discovery and sharing at 802.* networking either. Linux actually has a pretty decent bluetooth stack, it’s something that we should start taking advantage of. Your computer should be able to discover paired or known bluetooth devices when in range without intervention and communicate with them. If I attempt to establish a network connection, and my phone is nearby, that should be offered as a method for getting on the Internet. If possible, it should also be offered through telepathy as a method for contacting people in my address book! Synchronisation These technologies together give us a really big possibility, automatic synchronisation of user’s information. When I bring my laptop home, it should automatically update my desktop with any changes I’ve made to my address book or calendar, and any shared files, etc. Likewise it should be automatically be updated with any changes made on the network, to a company address book on a server, for example. And why stop there? Don’t use share your music, synchronise it so that your laptop, desktop and iPod have the same selection available at all times. Hassle-free backups is an obvious win here. But we’re stopping again. Why is my mobile phone, and PDA excluded from this? Via bluetooth, my address book and calendar should be automatically updated from these and these should be automatically updated themselves. If I add a new number to my phone’s memory, it should appear in my computer address book; and my computer should offer me the ability to contact that person (using VoIP, for example).

Erich Schubert: mDNS is nice

I'm setting up my new laptop, and for obvious reasons I have to copy over some data from the old to the new laptop. The wireless network here at my apartment-sharing is crappy; and it lacks a local DNS server offered by newer DSL routers. So I would have to find out the current IPs of my laptops to copy data around (note that the router also doesn't give you IP persistence apparently, the laptops have new IPs each reconnect...) mDNS to a rescue. Multicast DNS (Bonjour / Zeroconf / Avahi) comes in handy here. It allows me to locate my laptops by just using "hostname.local" (using the libnss-mdns package) or using the avahi zeroconf browser (package: avahi-discover).

Erich Schubert: eBay message userunfriendlyness

eBay (at least ebay.de) doesn't allow quotation marks or stars in messages sent via their UI. That totally sucks. I agree with not allowing HTML tags, to avoid any potential issues with broken webmails and cross site scripting, whatever. But not allowing quotation marks in messages sucks. I use them all the time, basically whenever I quote something of the artice description (that's why they are called quotation marks I guess). Why can't they just fix their code to allow quotation marks?

Julien Danjou: Knud can't

Or Dunc-tank in sorted order. I never wrote about this until now. But today, when I woke up, I had this strange feeling saying "jd, you gotta do something about DT today.". Actually, that was just my f**** alarm clock. And here I am, and I'm gonna talk to you about the dunc-tank experiment. First it's important to know what a real dunk tank is, because personally I've got no idea until I looked up on Google. So, you see? A dunk tank is a funny game finally. I must admit, if Anthony Towns was seated above the dunk, I would have vote in favor of DT!
Maybe I'm trying to give you some idea for the next DebConf, who knows More seriously, after the co-signed mail that Joerg Jaspert just sent, I wanted to say something more.
Some weeks ago, when our DPL (amen) asked us our opinion about that idea, I said "yeah, go ahead". Of course, that was my opinion at this time, and it changed a little. I still don't have objections about people getting paid, because I'm not jealous.
What does that mean? This means that if Debian was only composed of me, Anthony, and two release managers, I wouldn't have any problem. But that's not the case here, I discovered that there is a lots of people in Debian. Maybe should I tell DT people about that, don't you think? Reality is obviously different, and DT has been launched without a real consensus. We lost valuable people, and I dislike this project for that. We lost time. And we lost money, because time is money (don't beat me). That's an organised foutage de gueule [1] . But please, remember this is an experiment, so kids: don't do this at home.
Notes [1] Feel free to translate yourself

Erich Schubert: Turbogears performance

I've chosen Turbogears for a recent web project. Up to now, I was pretty happy with this choice, as it allowed me to made changes to the application rather easily, I happen to like KID templating, and it all was rather painless. However, I'm stuck at two place right now. One is a rather expensive database query I have not yet added - I know that it's going to be expensive, and I'd need some way of caching the results easily. I havn't found a "turbogearish" way of caching these interim results yet. Since that brought me onto the performance thing - when writing an app you don't really know yet how it will be implemented, you best postpone performance considerations a bit, so you don't optimize too early - I did some measurements. And was pretty disappointed by the results. The testing virtual machine (sorry, I don't have hardware specs for the actual machine) is rather slow. But still, static apache can serve some 450 requests per second. Turbogears will serve static files at ~22 r/s and the main page of the webapp at just 4.5 r/s (measured with siege, when it's not crashing). I tried switching from sqlite to mysql in Turbogears, but unfortunately that just makes the application crash altogether at some point (no SQL queries succeed anymore: 'Page handler: "Commands out of sync; you can't run this command now"') Notice that this was by no means a fair benchmark of Turbogears (I'm not a turbogears expert, and the little documentation I've found so far didn't help me at all in optimizing it's performance). I was just evaluating where I'm at, and now I'm about to decide where to go next. So can anyone point me to some docs on how to get turbogears up to an acceptable speed (where acceptable would probably be >20 r/s on this hardware)? Otherwise I guess I'll start a rewrite now, maybe using Java JSF and Facelets. They seem to be a good choice if you want XML-based templating. Don't even suggest PHP to me, it's crap. And I won't be trying Ruby on Rails next, either. I don't need rapid prototyping anymore, but can go for the high-performance implementation now, even if it will be harder to read, write and modify. I have now a better grasp for what I need and how the architecture of the app should be like.

Next.

Previous.