upstart is a replacement for the init daemon, the process spawned by the kernel that is responsible for starting, supervising and stopping all other processes on the system. The existing daemon is based on the one found in UNIX System V, and is thus known as sysvinit. It separates jobs into different “run levels” and can either run a job when particular run levels are entered (e.g. /etc/init.d/rc 2) or continually during a particular run level (e.g. /sbin/getty). The /etc/init.d/rc script is also based on the System V one (and is in the sysv-rc package), it simply executes the stop then start scripts found in /etc/rcN.d (where N is the run level) in numerical order. Why change it? Running a fixed set of scripts, one after the other, in a particular order has served us reasonably well until now. However as Linux has got better and better at dealing with modern computing (arguably Linux’s removable device support is better than Windows’ now) this approach has begun to have problems. The old approach works as long as you can guarantee when in the boot sequence things are available, so you can place your init script after that point and know that it will work. Typical ordering requirements are:

• Hard drive devices must have been discovered, initialised and partitions detected before we try and mount from /etc/fstab.

• Network devices must have been discovered and initialised before we try and start networking.

This worked ten years ago, why doesn’t it work now? The simple answer is that our computer has become far more flexible:

• Drives can be plugged in and removed at any point, e.g. USB drives.

• Storage buses allow more than a fixed number of drives, so they must be scanned for; this operation frequently does not block.

• To reduce power consumption, the drive may not actually be spun up until the bus scan so will not appear for an even longer time.

• Network devices can be plugged in and removed at any point.

• Firmware may need to be loaded after the device has been detected, but before it is usable by the system.

• Mounting a partition in /etc/fstab may require tools in /usr which is a network filesystem that cannot be mounted until after networking has been brought up.

We’ve been able to hack the existing system to make much of this possible, however the result is chock-full of race conditions and bugs. It was time to design a new system that can cope with all of these things without any problems. What we needed was an init system that could dynamically order the start up sequence based on the configuration and hardware found as it went along. Design of upstart upstart is an event-based init daemon; events generated by the system cause jobs to be started and running jobs to be stopped. Events can include things such as:

• the system has started,

• the root filesystem is now writable,

• a block device has been added to the system,

• a filesystem has been mounted,

• at a certain time or repeated time period,

• another job has begun running or has finished,

• a file on the disk has been modified,

• there are files in a queue directory,

• a network device has been detected,

• the default route has been added or removed.

In fact, any process on the system may send events to the init daemon over its control socket (subject to security restrictions, of course) so there is no limit. Each job has a life-cycle which is shown in the graph below: The two states shown in red (“waiting” and “running”) are rest states, normally we expect the job to remain in these states until an event comes in, at which point we need to take actual to get the job into the next state. The other states are temporary states; these allow a job to run shell script to prepare for the job itself to be run (“starting”) and clean up afterwards (“stopping”). For services that should be respawned if they terminate before an event that stops them is received, they may run shell script before the process is started again (“respawning”). Jobs leave a state because the process associated with them terminates (or gets killed) and move to the next appropriate state, following the green arrow if the job is to be started or the red arrow if it is to be stopped. When a script returns a non-zero exit status, or is killed, the job will always be stoped. When the main process terminates and the job should not be respawned, the job will also always be stopped. As already covered, events generated by the init daemon or received from other processes cause jobs to be started or stopped; also manual requests to start or stop a job may be received. The communication between the init daemon and other processes is bi-directional, so the status of jobs may be queries and even changes of state to all jobs be received. How does it differ from launchd? launchd is the replacement init system used in MacOS X developed as an “Open Source” project by Apple. For much of its life so far, the licence has actually been entirely non-free and thus it has only become recently interesting with the licence change. Much of the goal of both systems appears initially to be the same; they both start jobs based on system events, however the launchd system severly limits the events to only the following:

• system startup,

• file modified or placed in queue directory,

• particular time (cron replacement),

• connection on a particular port (inetd replacement).

Therefore it does not actually allow us to directly solve the problems we currently have; we couldn’t mount filesystems once the “filesystem checked” event has been recived, we couldn’t check filesystems when the block device is added and we certainly couldn’t start daemons once the complete filesystem (as described by /etc/fstab) is available and writable. The launchd model expects the job to “sit and wait” if it is unable to start, rather than provide a mechanism for the job to only be started when it doesn’t need to wait. Jobs that need /usr to be mounted would need to spin in a loop waiting for /usr to be available before continuing (or use a file in a tmpfs to indicate it’s available, and use that modification as the event). This is not especially surprising given that Apple have a high degree of control over both their hardware and the actual underlying operating system; they don’t need to deal with the wide array of different configurations that we have in the Linux world. Had the licence been sufficiently free at the point we began development of our own system, we would probably have extended launchd rather than implement our own. At the point Apple changed the licence, our own system was already more suitable for our purposes. How does it differ from initng? Initng by Jimmy Wennlund is another replacement init daemon intended to replace the sysvinit system used by Linux. It is a dependency-based system, where upstart is an event-based system. The notion of a dependency-based system is interesting to talk about at this point. Jobs declare dependencies on other jobs that need to happen before the job itself can be started. Starting the job causes its dependencies to be started first, and their dependencies, and so on. When jobs are stopped, if running jobs have no dependencies, they themselves can be stopped. It’s a neat solution to the problem of ordering a fixed boot sequence and the problem of keeping the number of running processes to a minimum needed. However this means that you need to have goals in mind when you boot the system, you need to have decided that you want gdm to be started in order for it, and its dependencies, to be started. Initng uses run levels to ensure this happens, where a run level is a list of goal jobs that should be running in that run level. It’s also not clear how the dependencies interact with the different types of job, a dependency on Apache would need the daemon to be running where a dependency on “checkroot” would need the script to have finished running. Upstart handles this by using different events (“apache running” vs. “checkroot stopping”). Again while interesting, Initng does not solve the problems that we wanted to solve. It can reorder a fixed set of jobs, but cannot dynamically determine the set of jobs needed for that particular boot. A typical example would be that if the only dependency on the job that configures networking is the mount network filesystems job, then should that job fail or notbe a goal (e.g. because there are no network filesystems to be mounted) the result is that network devices themselves will not be configured. You could make everything a goal, and just use the dependencies to determine the order, however this is less efficient than just ordering the existing sysv-rc scripts (which can be done at install time). Another example is that often you simply don’t know whether something is a dependency or not without reading other configuration, for example the mount network filesystems may be a dependency of everything under /usr or may just be a dependency of anything allowing the user to login if it just mounts /home. The difference in model can be summed up as “initng starts with a list of goals and works out how to get there, upstart starts with nothing and finds out where it gets to.” How does it differ from Solaris SMF? SMF is another approach to replacing init developed by Sun for the Solaris operating system. Like initng it’s a dependency-based system, so see above for the differences between those systems and upstart. SMF’s main focus is serive management; making sure that once services are running, they stay running, and allowing the system administrator to query and modify the states of jobs on the system. Upstart provides the same set of functionality in this regard, services are respawned when they fail and system administrators can at any time query the state of running services and adjust the state to their liking. Will it replace cron, inetd, etc? The goal of upstart is to replace those daemons, so that there is only one place (/etc/event.d) where system administrators need to configure when and how jobs should be run. In fact, the goal is that upstart should also replace the “run event scripts” functionality of any daemon on the system. Daemons such as acpid, apmd and Network Manager would send events to init instead of running scripts themselves with their own perculiar configuration and semantics. A system administrator who only wanted a particular daemon to be run while the computer was on AC power would simply need to edit /etc/event.d/daemon and change “on startup” to “on ac power”. What about compatibility? There’s a lot of systems administrators out there who have learned how Linux works already and will not want to learn again immediately, there’s also a large number of books that cover the existing software and won’t cover upstart for at least a couple of years. For this reason, compatibility is very important. upstart will continue to run the existing scripts for the forseeable future so that packages will not need to be updated until the author wants. Compatibility command-line tools that behave like their existing equivalents will also be implemented, a system administrator would never need to know that crontab -e is actually changing upstart jobs. Does it use D-BUS?

“To D-BUS people, every problem seems like a D-BUS problem.” —Erik Troan

The UNIX philosophy is that something should do just one job, and do it very well. upstart’s one job is starting, supervising and stopping other jobs; D-BUS’s one job is passing messages between other jobs. D-BUS does provide a mechanism for services to be activated when the first message is sent to them, thereby starting other jobs. Some people have taken this idea and extended it to suggest that all a replacement init system need do is register jobs with D-BUS and turn booting into a simple matter of message parsing. This seems wrong to me, D-BUS would need to be extended to supervise these services, provide means for them to be restarted and stopped; as well as deal with being process #1 which means cleaning up after children whose parent’s have died, etc. It seems far simpler to arrange for D-BUS to send an event to init when it needs a service to be started, and focus on being a very good message passing system. The IPC mechanism used by upstart is not currently D-BUS because of various problems, however it’s always been expected that even if init itself doesn’t communicate with D-BUS directly, there would be a D-BUS proxy that would ensure messages about all init jobs and events would be given to D-BUS and D-BUS clients could send messages to init to query and change the state of jobs. What is the implementation plan? Because this is process #1 we are changing, we want to make sure that we get it right. Therefore instead of releasing a fully-featured daemon and configuration to the world, we’re developing it in the following stages:

1. Principal development; at the end of this stage the daemon has been implemented and can manage jobs as described.
2. Replacement of /sbin/init while running the existing sysv-rc scripts. This is the shake-down test of the daemon, can it perform the same job as the existing sysvinit daemon without any regressions?
3. /etc/rcS.d scripts replaced by upstart jobs. These consitute the majority of tasks for booting the system into at least single-user mode, and contain many of the current ordering problems and race conditions. If the daemon solves the problems here, it will be a success.
4. Other daemon’s scripts replaced by upstart jobs on a package-by-package basis; this will be an ongoing effort during which upstart will continue running the existing sysv-rc scripts as well as its own jobs. During this time the event system may be tweaked to ensure it truly solves the problems we need.
5. Replcement of cron, atd, anacron and inetd. This will happen alongside the above and result in a single place to configure system jobs.
6. Modification of other daemons and processes to send events to init instead of trying to run things themselves.

The current plan is that we will be at least part of the way into stage #3 by the time edgy is released, with that release shipping with upstart as the init daemon and the most critical rcS scripts being run by it to correct the major problems For edgy+1 we hope to have completed stage #5 and be at least part of the way into the implementation of stage #6. From the start of development of edgy+2, no new packages will be accepted unless they provide upstart jobs instead of init scripts and init scripts will be considered deprecated. What state is it in now? The init daemon has been written and is able to manage jobs as described above, receiving events on the control socket to start and stop them. This has now been uploaded to the Ubuntu universe component in the upstart package for testing before it becomes the init daemon. We welcome any experienced users who want to help test this; install the package and follow the instructions in /usr/share/doc/upstart/README.Debian to add a boot option that will use upstart instead of init. If your system boots and shut downs normally (other than a slightly more verbose boot without usplash running) then it is working correctly. Other types of events will be added as required during development and testing. Currently only a basic client tool (initctl) has been written, compatibility tools such as shutdown will be written over the next week or two before it replaces our sysvinit package.

Writing your first Asterisk AGI script, putting it in /var/lib/asterisk/agi-bin because that's where another package has put one, and all of the random documentation on the Internet refers to that location (even though it's in violation of the FHS to go sticking scripts there), and after a marathon debugging session, discovering that Debian's Asterisk package ships with astagidir set to /usr/share/asterisk/agi-bin (which it appears you need to create yourself).

You will not be missed.

During the Journ es du Logiciel Libre 2006 (an excellent event, thanks to the organizers), I had long chats with people from Mozilla Europe (in a more constructive way than what others did). This is mostly second-hand information, since the well-informed guys from Mozilla Europe I talked with got their info from other Mozilla developers, so it might be slightly inaccurate, but still help to understand the issue. More details welcomed in comments. The Mozilla Foundation goal is to prevent the web from becoming a proprietary web, by ensuring that Firefox, their product, has a big-enough market-share. I find it a bit disturbing to hear free software people speak about product and market-share, but well, OK. Of course, it’s quite difficult to achieve >20% market-share only with GNU/Linux users. They put a very high emphasis on the User experience. They want Firefox to look exactly the same on all platforms, even if some features (automatic plugin installation, automatic upgrades) do not fit well on a GNU/Linux system (such features don’t work with the Debian build). They want to enforce their trademark to prevent attempts to ruin their reputation to succeed. After a question asked by Dave Neary (GNOME) to Tristan Nitot (president of Mozilla Europe), Tristan gave the example of this hacker who said he found critical security bugs in Firefox before admitting it was a joke. He also gave the example of Microsoft indirectly founding SCO. Dave Neary remarked that some people could probably be trusted (like distro maintainers). But, if I remember correctly, Tristan said that it was difficult to allow some people to use the trademark without allowing everybody to do so. Also, allowing Debian to use the trademark without allowing derivatives to do so is not possible (DFSG#8). Iceweasel is considered a good thing inside Mozilla Europe (”It’s what we have asked them to do for a long time!“). However, some people are bitter about the fact that Debian seems to have chosen to use a gnu.org-hosted fork instead of just renaming the package to iceweasel. During Tristan Nitot’s conference, I asked a question about length of security support (only 18 months for the 1.0 branch). The answer was that distros were supposed to upgrade to newer Firefox releases even on their stable releases (!) or to come and help with security support for stable releases. Also, the fact that Debian chooses to package Firefox following the FHS is considered a source of problems. Somebody claimed that Debian was the only distribution to package Firefox this way (seems strange, but I did not checked this claim). I’m not sure how this fits with other Mozilla-based browsers such as epiphany. On a more technical side, they also claimed that Debian wasn’t properly collaborating with Mozilla, sending unusable 100000-lines patches for validation just before releases (haven’t checked this claim). It’s interesting to note that the same problem exists on the other side, with Mozilla releasing “security and stability” releases instead of just providing patches for the security bugs. My personal conclusion is that Iceweasel is really a good thing, and is quite unavoidable, even if it seems that the Debian/Mozilla collaboration could maybe have been better. Let’s advertise the use of Iceweasel, Epiphany, and other Web browsers! Note : comments open for people willing to fix inaccuracies or provide references, not for a stupid childish flame-war like the one in the comment of this post. Please write your own blog entry and use a trackback if you want to provide a detailed answer! Updates:

• I forgot to mention that the Mozilla people talked about Debian-specific changes that changed frozen APIs, breaking extensions, and causing bug reports on b.m.o or misleading forum posts. (again, not verified)
• Interesting blog post about the trademark issue here
• Mike Hommey addressed most of the points I reported in this blog entry.

I’ve finally finished packaging keytouch, and gotten approval for it by my sponsor. It’s currently in NEW, and probably won’t make it to etch, because of the freeze. Kind of a shame, since I could have finished a little earlier and it would have gotten in. Well, can’t have everything. I did manage to upload a new version of sawfish, which finally documents how to prepare the source tarball. sawfish has so many old bugs I’m at a loss as how to handle them. I’ll be RFHing it soon, to see if someone can help me triage. But still, with upstream almost dead, it’s hard to think of a good approach.

People tend to just ping me on IRC, which is annoying and useless. If people say something like tfheen: I have this problem with blah. Do you know a workaround? I can just respond later when I'm awake or around. Instead, people are going: tfheen: ping? and I pong five hours later and they're not around. Annoying for all parts. To counter this, I have now written a small irssi script which responds to contentless pings with "You sent me a contentless ping. This is a contentless pong. Please provide a bit of information about what you want and I'll respond when I am around." The script is available.

A bit surprising, really.

: tfheen@thosu ~ > history 1 awk ' print $2 ' awk 'BEGIN  FS=" "   print $1 ' sort uniq -c   sort -nr  head -n 10
  21477 ls
  18170 cd
  10640 ssh
   9257 sudo
   5559 less
   3015 grep
   2407 bzr
   2101 ps
   1980 man
   1980 debuild

For those of you wondering why there is no editor on the list; I use emacs and it's in the 11th place (with 1903 runs). I tend to let it run for a while and open more than one file, so it doesn't get that high on the list. I also cd and ls a lot.

Some prominent kernel developers today posted a position paper about the GPLv3. By reading between the lines it is clear that Red Hat and Novell would fork the toolchain and other GNU tools if FSF relicensed them to the GPLv3. Then everybody sane would follow them and the FSF would soon become irrelevant.

upstart is a replacement for the init daemon, the process spawned by the kernel that is responsible for starting, supervising and stopping all other processes on the system. The existing daemon is based on the one found in UNIX System V, and is thus known as sysvinit. It separates jobs into different “run levels” and can either run a job when particular run levels are entered (e.g. /etc/init.d/rc 2) or continually during a particular run level (e.g. /sbin/getty). The /etc/init.d/rc script is also based on the System V one (and is in the sysv-rc package), it simply executes the stop then start scripts found in /etc/rcN.d (where N is the run level) in numerical order. Why change it? Running a fixed set of scripts, one after the other, in a particular order has served us reasonably well until now. However as Linux has got better and better at dealing with modern computing (arguably Linux’s removable device support is better than Windows’ now) this approach has begun to have problems. The old approach works as long as you can guarantee when in the boot sequence things are available, so you can place your init script after that point and know that it will work. Typical ordering requirements are:

• Hard drive devices must have been discovered, initialised and partitions detected before we try and mount from /etc/fstab.

• Network devices must have been discovered and initialised before we try and start networking.

This worked ten years ago, why doesn’t it work now? The simple answer is that our computer has become far more flexible:

• Drives can be plugged in and removed at any point, e.g. USB drives.

• Storage buses allow more than a fixed number of drives, so they must be scanned for; this operation frequently does not block.

• To reduce power consumption, the drive may not actually be spun up until the bus scan so will not appear for an even longer time.

• Network devices can be plugged in and removed at any point.

• Firmware may need to be loaded after the device has been detected, but before it is usable by the system.

• Mounting a partition in /etc/fstab may require tools in /usr which is a network filesystem that cannot be mounted until after networking has been brought up.

We’ve been able to hack the existing system to make much of this possible, however the result is chock-full of race conditions and bugs. It was time to design a new system that can cope with all of these things without any problems. What we needed was an init system that could dynamically order the start up sequence based on the configuration and hardware found as it went along. Design of upstart upstart is an event-based init daemon; events generated by the system cause jobs to be started and running jobs to be stopped. Events can include things such as:

• the system has started,

• the root filesystem is now writable,

• a block device has been added to the system,

• a filesystem has been mounted,

• at a certain time or repeated time period,

• another job has begun running or has finished,

• a file on the disk has been modified,

• there are files in a queue directory,

• a network device has been detected,

• the default route has been added or removed.

In fact, any process on the system may send events to the init daemon over its control socket (subject to security restrictions, of course) so there is no limit. Each job has a life-cycle which is shown in the graph below: The two states shown in red (“waiting” and “running”) are rest states, normally we expect the job to remain in these states until an event comes in, at which point we need to take actual to get the job into the next state. The other states are temporary states; these allow a job to run shell script to prepare for the job itself to be run (“starting”) and clean up afterwards (“stopping”). For services that should be respawned if they terminate before an event that stops them is received, they may run shell script before the process is started again (“respawning”). Jobs leave a state because the process associated with them terminates (or gets killed) and move to the next appropriate state, following the green arrow if the job is to be started or the red arrow if it is to be stopped. When a script returns a non-zero exit status, or is killed, the job will always be stoped. When the main process terminates and the job should not be respawned, the job will also always be stopped. As already covered, events generated by the init daemon or received from other processes cause jobs to be started or stopped; also manual requests to start or stop a job may be received. The communication between the init daemon and other processes is bi-directional, so the status of jobs may be queries and even changes of state to all jobs be received. How does it differ from launchd? launchd is the replacement init system used in MacOS X developed as an “Open Source” project by Apple. For much of its life so far, the licence has actually been entirely non-free and thus it has only become recently interesting with the licence change. Much of the goal of both systems appears initially to be the same; they both start jobs based on system events, however the launchd system severly limits the events to only the following:

• system startup,

• file modified or placed in queue directory,

• particular time (cron replacement),

• connection on a particular port (inetd replacement).

Therefore it does not actually allow us to directly solve the problems we currently have; we couldn’t mount filesystems once the “filesystem checked” event has been recived, we couldn’t check filesystems when the block device is added and we certainly couldn’t start daemons once the complete filesystem (as described by /etc/fstab) is available and writable. The launchd model expects the job to “sit and wait” if it is unable to start, rather than provide a mechanism for the job to only be started when it doesn’t need to wait. Jobs that need /usr to be mounted would need to spin in a loop waiting for /usr to be available before continuing (or use a file in a tmpfs to indicate it’s available, and use that modification as the event). This is not especially surprising given that Apple have a high degree of control over both their hardware and the actual underlying operating system; they don’t need to deal with the wide array of different configurations that we have in the Linux world. Had the licence been sufficiently free at the point we began development of our own system, we would probably have extended launchd rather than implement our own. At the point Apple changed the licence, our own system was already more suitable for our purposes. How does it differ from initng? Initng by Jimmy Wennlund is another replacement init daemon intended to replace the sysvinit system used by Linux. It is a dependency-based system, where upstart is an event-based system. The notion of a dependency-based system is interesting to talk about at this point. Jobs declare dependencies on other jobs that need to happen before the job itself can be started. Starting the job causes its dependencies to be started first, and their dependencies, and so on. When jobs are stopped, if running jobs have no dependencies, they themselves can be stopped. It’s a neat solution to the problem of ordering a fixed boot sequence and the problem of keeping the number of running processes to a minimum needed. However this means that you need to have goals in mind when you boot the system, you need to have decided that you want gdm to be started in order for it, and its dependencies, to be started. Initng uses run levels to ensure this happens, where a run level is a list of goal jobs that should be running in that run level. It’s also not clear how the dependencies interact with the different types of job, a dependency on Apache would need the daemon to be running where a dependency on “checkroot” would need the script to have finished running. Upstart handles this by using different events (“apache running” vs. “checkroot stopping”). Again while interesting, Initng does not solve the problems that we wanted to solve. It can reorder a fixed set of jobs, but cannot dynamically determine the set of jobs needed for that particular boot. A typical example would be that if the only dependency on the job that configures networking is the mount network filesystems job, then should that job fail or notbe a goal (e.g. because there are no network filesystems to be mounted) the result is that network devices themselves will not be configured. You could make everything a goal, and just use the dependencies to determine the order, however this is less efficient than just ordering the existing sysv-rc scripts (which can be done at install time). Another example is that often you simply don’t know whether something is a dependency or not without reading other configuration, for example the mount network filesystems may be a dependency of everything under /usr or may just be a dependency of anything allowing the user to login if it just mounts /home. The difference in model can be summed up as “initng starts with a list of goals and works out how to get there, upstart starts with nothing and finds out where it gets to.” How does it differ from Solaris SMF? SMF is another approach to replacing init developed by Sun for the Solaris operating system. Like initng it’s a dependency-based system, so see above for the differences between those systems and upstart. SMF’s main focus is serive management; making sure that once services are running, they stay running, and allowing the system administrator to query and modify the states of jobs on the system. Upstart provides the same set of functionality in this regard, services are respawned when they fail and system administrators can at any time query the state of running services and adjust the state to their liking. Will it replace cron, inetd, etc? The goal of upstart is to replace those daemons, so that there is only one place (/etc/event.d) where system administrators need to configure when and how jobs should be run. In fact, the goal is that upstart should also replace the “run event scripts” functionality of any daemon on the system. Daemons such as acpid, apmd and Network Manager would send events to init instead of running scripts themselves with their own perculiar configuration and semantics. A system administrator who only wanted a particular daemon to be run while the computer was on AC power would simply need to edit /etc/event.d/daemon and change “on startup” to “on ac power”. What about compatibility? There’s a lot of systems administrators out there who have learned how Linux works already and will not want to learn again immediately, there’s also a large number of books that cover the existing software and won’t cover upstart for at least a couple of years. For this reason, compatibility is very important. upstart will continue to run the existing scripts for the forseeable future so that packages will not need to be updated until the author wants. Compatibility command-line tools that behave like their existing equivalents will also be implemented, a system administrator would never need to know that crontab -e is actually changing upstart jobs. Does it use D-BUS?

“To D-BUS people, every problem seems like a D-BUS problem.” —Erik Troan

The UNIX philosophy is that something should do just one job, and do it very well. upstart’s one job is starting, supervising and stopping other jobs; D-BUS’s one job is passing messages between other jobs. D-BUS does provide a mechanism for services to be activated when the first message is sent to them, thereby starting other jobs. Some people have taken this idea and extended it to suggest that all a replacement init system need do is register jobs with D-BUS and turn booting into a simple matter of message parsing. This seems wrong to me, D-BUS would need to be extended to supervise these services, provide means for them to be restarted and stopped; as well as deal with being process #1 which means cleaning up after children whose parent’s have died, etc. It seems far simpler to arrange for D-BUS to send an event to init when it needs a service to be started, and focus on being a very good message passing system. The IPC mechanism used by upstart is not currently D-BUS because of various problems, however it’s always been expected that even if init itself doesn’t communicate with D-BUS directly, there would be a D-BUS proxy that would ensure messages about all init jobs and events would be given to D-BUS and D-BUS clients could send messages to init to query and change the state of jobs. What is the implementation plan? Because this is process #1 we are changing, we want to make sure that we get it right. Therefore instead of releasing a fully-featured daemon and configuration to the world, we’re developing it in the following stages:

1. Principal development; at the end of this stage the daemon has been implemented and can manage jobs as described.
2. Replacement of /sbin/init while running the existing sysv-rc scripts. This is the shake-down test of the daemon, can it perform the same job as the existing sysvinit daemon without any regressions?
3. /etc/rcS.d scripts replaced by upstart jobs. These consitute the majority of tasks for booting the system into at least single-user mode, and contain many of the current ordering problems and race conditions. If the daemon solves the problems here, it will be a success.
4. Other daemon’s scripts replaced by upstart jobs on a package-by-package basis; this will be an ongoing effort during which upstart will continue running the existing sysv-rc scripts as well as its own jobs. During this time the event system may be tweaked to ensure it truly solves the problems we need.
5. Replcement of cron, atd, anacron and inetd. This will happen alongside the above and result in a single place to configure system jobs.
6. Modification of other daemons and processes to send events to init instead of trying to run things themselves.

The current plan is that we will be at least part of the way into stage #3 by the time edgy is released, with that release shipping with upstart as the init daemon and the most critical rcS scripts being run by it to correct the major problems For edgy+1 we hope to have completed stage #5 and be at least part of the way into the implementation of stage #6. From the start of development of edgy+2, no new packages will be accepted unless they provide upstart jobs instead of init scripts and init scripts will be considered deprecated. What state is it in now? The init daemon has been written and is able to manage jobs as described above, receiving events on the control socket to start and stop them. This has now been uploaded to the Ubuntu universe component in the upstart package for testing before it becomes the init daemon. We welcome any experienced users who want to help test this; install the package and follow the instructions in /usr/share/doc/upstart/README.Debian to add a boot option that will use upstart instead of init. If your system boots and shut downs normally (other than a slightly more verbose boot without usplash running) then it is working correctly. Other types of events will be added as required during development and testing. Currently only a basic client tool (initctl) has been written, compatibility tools such as shutdown will be written over the next week or two before it replaces our sysvinit package.

I am currently working on something which hopefully end up being a good calendar server. The goal is to make it easy to add new frontends such as a CalDAV frontend, a Web frontend, etc. So far, I am just working on getting the data model right. The iCalendar RFC has a data model which I am trying to formulate into SQL, something which is ending up being quite hard. Shannon Clark blogs about why calendars are hard, and all of those issues are issues I am running up against when trying to make the data model. The main problem seems to be related to time zones, more closely: "How do you store timezone information in your database". The easy (but wrong) solution is to just normalise everything to UTC. This will not handle the case of events crossing timezone changes, such as to or from daylight savings. What I am currently considering is making sure all data in the database is stored in UTC, but also store the original time zone information. While a bit more complex, this allows applications to get back the same timezone they put in (I am not convinced Evolution will be happy if the timezones are renamed, for instance) and it allows my application to generate recurring events properly. My main issue with this is having the application responsible for making sure the data in the database makes sense, but without putting half my app in PL/perl, I can not see a way around that. Oh, and if anybody has great ideas on how to represent timezones in a relational database (postgres), please do mail me or grab me on IRC or something similar.

(From a debian-devel@ post arguing against support for 2.4 kernels.) The problem with supporting old kernels is not just the need to maintain a few packages like initrd-tools or modutils, but that every important package cannot rely on features of modern kernels: inotify, sysfs, etc. This means that Debian as a whole will either lack support for features relying on these kernel features or will become more and more complex due to compatibility code. Please consider carefully the effects of advocating support for old kernels.

I suddenly realised that I like receiving bug reports. In a recent bout of structured procrastination I fixed more or less all the bugs I could fix in my packages (except ppp. I don't feel like working on ppp right now, does anybody want to help me?) and now this is a quiet period. Obsessively checking the bugs mailbox for new messages is a clear withdrawal symptom. I am addicted to bugs for a very simple reason: they are a challenge. I stopped working on mutt because it was not interesting anymore to me and then I adopted ppp and packaged udev because I tough that maintaining them would require solving many interesting problems. It's too bad that solving some needs the collaboration of other developers, but this will be the topic of a different rant. But not all bugs are an interesting challenge. Some are just a total waste of my time, which usually is much more valuable than the time of the submitter. I will show some examples.

• Obscure bugs which I cannot debug and need repeated interactions with the upstream developers trying to debug them. OK, you submitted the bug and I am sorry that I cannot help more than this, but I told you where you can find somebody who cares. Now please do not expect me to forward emails back and forth from the BTS and upstream developers. This is not only very inefficient for you and for the people who are trying to help you, but also wastes my precious time.
• Features to be forwarded to the upstream maintainer. I am a Debian developer, not your personal assistant. Forwarding your feature request, which probably will never be implement because it is crazy, useless or just needs more work than it is available wastes my precious time. And then I will waste even more every time I will be distracted by seeing it in the BTS, and curse you for continuing to waste my time.
• Trivial upstream bugs like spelling errors. It is highly improbable that I will consider them important enough to be worth fixing with a patch in the Debian packaging, so what about you do not waste my precious time and just report them to the upstream maintainers?
• Requests for useless changes to the packaging. If it is useless and you do not even care enough to implement it, what makes you believe that I will? (Yes, I will decide what is useful.)
• Multiple reports about the package being totally broken. If an important package is obviously broken (often broken as in "the system will not boot") then probably somebody else will have already submitted a bug report. It is your duty to check for open bugs before reporting a new one, and merging or closing duplicate bugs wastes my precious time.
• Bogus reports showing that you did not RTFM.
• Bogus reports showing that you have no grasp of how the package works. Especially if I explained the issue in the README.Debian file.
• Unreproducible reports where every clue points to a PEBKAC and the reporter still does not promptly follow my instructions for trying to confirm the bug.

So, if you can follow these simple rules I will probably not flame you to ashes the next time you will report a bug on one of my packages.

upstart is a replacement for the init daemon, the process spawned by the kernel that is responsible for starting, supervising and stopping all other processes on the system. The existing daemon is based on the one found in UNIX System V, and is thus known as sysvinit. It separates jobs into different “run levels” and can either run a job when particular run levels are entered (e.g. /etc/init.d/rc 2) or continually during a particular run level (e.g. /sbin/getty). The /etc/init.d/rc script is also based on the System V one (and is in the sysv-rc package), it simply executes the stop then start scripts found in /etc/rcN.d (where N is the run level) in numerical order. Why change it? Running a fixed set of scripts, one after the other, in a particular order has served us reasonably well until now. However as Linux has got better and better at dealing with modern computing (arguably Linux’s removable device support is better than Windows’ now) this approach has begun to have problems. The old approach works as long as you can guarantee when in the boot sequence things are available, so you can place your init script after that point and know that it will work. Typical ordering requirements are:

• Hard drive devices must have been discovered, initialised and partitions detected before we try and mount from /etc/fstab.

• Network devices must have been discovered and initialised before we try and start networking.

This worked ten years ago, why doesn’t it work now? The simple answer is that our computer has become far more flexible:

• Drives can be plugged in and removed at any point, e.g. USB drives.

• Storage buses allow more than a fixed number of drives, so they must be scanned for; this operation frequently does not block.

• To reduce power consumption, the drive may not actually be spun up until the bus scan so will not appear for an even longer time.

• Network devices can be plugged in and removed at any point.

• Firmware may need to be loaded after the device has been detected, but before it is usable by the system.

• Mounting a partition in /etc/fstab may require tools in /usr which is a network filesystem that cannot be mounted until after networking has been brought up.

We’ve been able to hack the existing system to make much of this possible, however the result is chock-full of race conditions and bugs. It was time to design a new system that can cope with all of these things without any problems. What we needed was an init system that could dynamically order the start up sequence based on the configuration and hardware found as it went along. Design of upstart upstart is an event-based init daemon; events generated by the system cause jobs to be started and running jobs to be stopped. Events can include things such as:

• the system has started,

• the root filesystem is now writable,

• a block device has been added to the system,

• a filesystem has been mounted,

• at a certain time or repeated time period,

• another job has begun running or has finished,

• a file on the disk has been modified,

• there are files in a queue directory,

• a network device has been detected,

• the default route has been added or removed.

In fact, any process on the system may send events to the init daemon over its control socket (subject to security restrictions, of course) so there is no limit. Each job has a life-cycle which is shown in the graph below: The two states shown in red (“waiting” and “running”) are rest states, normally we expect the job to remain in these states until an event comes in, at which point we need to take actual to get the job into the next state. The other states are temporary states; these allow a job to run shell script to prepare for the job itself to be run (“starting”) and clean up afterwards (“stopping”). For services that should be respawned if they terminate before an event that stops them is received, they may run shell script before the process is started again (“respawning”). Jobs leave a state because the process associated with them terminates (or gets killed) and move to the next appropriate state, following the green arrow if the job is to be started or the red arrow if it is to be stopped. When a script returns a non-zero exit status, or is killed, the job will always be stoped. When the main process terminates and the job should not be respawned, the job will also always be stopped. As already covered, events generated by the init daemon or received from other processes cause jobs to be started or stopped; also manual requests to start or stop a job may be received. The communication between the init daemon and other processes is bi-directional, so the status of jobs may be queries and even changes of state to all jobs be received. How does it differ from launchd? launchd is the replacement init system used in MacOS X developed as an “Open Source” project by Apple. For much of its life so far, the licence has actually been entirely non-free and thus it has only become recently interesting with the licence change. Much of the goal of both systems appears initially to be the same; they both start jobs based on system events, however the launchd system severly limits the events to only the following:

• system startup,

• file modified or placed in queue directory,

• particular time (cron replacement),

• connection on a particular port (inetd replacement).

Therefore it does not actually allow us to directly solve the problems we currently have; we couldn’t mount filesystems once the “filesystem checked” event has been recived, we couldn’t check filesystems when the block device is added and we certainly couldn’t start daemons once the complete filesystem (as described by /etc/fstab) is available and writable. The launchd model expects the job to “sit and wait” if it is unable to start, rather than provide a mechanism for the job to only be started when it doesn’t need to wait. Jobs that need /usr to be mounted would need to spin in a loop waiting for /usr to be available before continuing (or use a file in a tmpfs to indicate it’s available, and use that modification as the event). This is not especially surprising given that Apple have a high degree of control over both their hardware and the actual underlying operating system; they don’t need to deal with the wide array of different configurations that we have in the Linux world. Had the licence been sufficiently free at the point we began development of our own system, we would probably have extended launchd rather than implement our own. At the point Apple changed the licence, our own system was already more suitable for our purposes. How does it differ from initng? Initng by Jimmy Wennlund is another replacement init daemon intended to replace the sysvinit system used by Linux. It is a dependency-based system, where upstart is an event-based system. The notion of a dependency-based system is interesting to talk about at this point. Jobs declare dependencies on other jobs that need to happen before the job itself can be started. Starting the job causes its dependencies to be started first, and their dependencies, and so on. When jobs are stopped, if running jobs have no dependencies, they themselves can be stopped. It’s a neat solution to the problem of ordering a fixed boot sequence and the problem of keeping the number of running processes to a minimum needed. However this means that you need to have goals in mind when you boot the system, you need to have decided that you want gdm to be started in order for it, and its dependencies, to be started. Initng uses run levels to ensure this happens, where a run level is a list of goal jobs that should be running in that run level. It’s also not clear how the dependencies interact with the different types of job, a dependency on Apache would need the daemon to be running where a dependency on “checkroot” would need the script to have finished running. Upstart handles this by using different events (“apache running” vs. “checkroot stopping”). Again while interesting, Initng does not solve the problems that we wanted to solve. It can reorder a fixed set of jobs, but cannot dynamically determine the set of jobs needed for that particular boot. A typical example would be that if the only dependency on the job that configures networking is the mount network filesystems job, then should that job fail or notbe a goal (e.g. because there are no network filesystems to be mounted) the result is that network devices themselves will not be configured. You could make everything a goal, and just use the dependencies to determine the order, however this is less efficient than just ordering the existing sysv-rc scripts (which
can be done at install time). Another example is that often you simply don’t know whether something is a dependency or not without reading other configuration, for example the mount network filesystems may be a dependency of everything under /usr or may just be a dependency of anything allowing the user to login if it just mounts /home. The difference in model can be summed up as “initng starts with a list of goals and works out how to get there, upstart starts with nothing and finds out where it gets to.” How does it differ from Solaris SMF? SMF is another approach to replacing init developed by Sun for the Solaris operating system. Like initng it’s a dependency-based system, so see above for the differences between those systems and upstart. SMF’s main focus is serive management; making sure that once services are running, they stay running, and allowing the system administrator to query and modify the states of jobs on the system. Upstart provides the same set of functionality in this regard, services are respawned when they fail and system administrators can at any time query the state of running services and adjust the state to their liking. Will it replace cron, inetd, etc? The goal of upstart is to replace those daemons, so that there is only one place (/etc/event.d) where system administrators need to configure when and how jobs should be run. In fact, the goal is that upstart should also replace the “run event scripts” functionality of any daemon on the system. Daemons such as acpid, apmd and Network Manager would send events to init instead of running scripts themselves with their own perculiar configuration and semantics. A system administrator who only wanted a particular daemon to be run while the computer was on AC power would simply need to edit /etc/event.d/daemon and change “on startup” to “on ac power”. What about compatibility? There’s a lot of systems administrators out there who have learned how Linux works already and will not want to learn again immediately, there’s also a large number of books that cover the existing software and won’t cover upstart for at least a couple of years. For this reason, compatibility is very important. upstart will continue to run the existing scripts for the forseeable future so that packages will not need to be updated until the author wants. Compatibility command-line tools that behave like their existing equivalents will also be implemented, a system administrator would never need to know that crontab -e is actually changing upstart jobs. Does it use D-BUS?

“To D-BUS people, every problem seems like a D-BUS problem.”
—Erik Troan

The UNIX philosophy is that something should do just one job, and do it very well. upstart’s one job is starting, supervising and stopping other jobs; D-BUS’s one job is passing messages between other jobs. D-BUS does provide a mechanism for services to be activated when the first message is sent to them, thereby starting other jobs. Some people have taken this idea and extended it to suggest that all a replacement init system need do is register jobs with D-BUS and turn booting into a simple matter of message parsing. This seems wrong to me, D-BUS would need to be extended to supervise these services, provide means for them to be restarted and stopped; as well as deal with being process #1 which means cleaning up after children whose parent’s have died, etc. It seems far simpler to arrange for D-BUS to send an event to init when it needs a service to be started, and focus on being a very good message passing system. The IPC mechanism used by upstart is not currently D-BUS because of various problems, however it’s always been expected that even if init itself doesn’t communicate with D-BUS directly, there would be a D-BUS proxy that would ensure messages about all init jobs and events would be given to D-BUS and D-BUS clients could send messages to init to query and change the state of jobs. What is the implementation plan? Because this is process #1 we are changing, we want to make sure that we get it right. Therefore instead of releasing a fully-featured daemon and configuration to the world, we’re developing it in the following stages:

1. Principal development; at the end of this stage the daemon has been implemented and can manage jobs as described.
2. Replacement of /sbin/init while running the existing sysv-rc scripts. This is the shake-down test of the daemon, can it perform the same job as the existing sysvinit daemon without any regressions?
3. /etc/rcS.d scripts replaced by upstart jobs. These consitute the majority of tasks for booting the system into at least single-user mode, and contain many of the current ordering problems and race conditions. If the daemon solves the problems here, it will be a success.
4. Other daemon’s scripts replaced by upstart jobs on a package-by-package basis; this will be an ongoing effort during which upstart will continue running the existing sysv-rc scripts as well as its own jobs. During this time the event system may be tweaked to ensure it truly solves the problems we need.
5. Replcement of cron, atd, anacron and inetd. This will happen alongside the above and result in a single place to configure system jobs.
6. Modification of other daemons and processes to send events to init instead of trying to run things themselves.

The current plan is that we will be at least part of the way into stage #3 by the time edgy is released, with that release shipping with upstart as the init daemon and the most critical rcS scripts being run by it to correct the major problems For edgy+1 we hope to have completed stage #5 and be at least part of the way into the implementation of stage #6. From the start of development of edgy+2, no new packages will be accepted unless they provide upstart jobs instead of init scripts and init scripts will be considered deprecated. What state is it in now? The init daemon has been written and is able to manage jobs as described above, receiving events on the control socket to start and stop them. This has now been uploaded to the Ubuntu universe component in the upstart package for testing before it becomes the init daemon. We welcome any experienced users who want to help test this; install the package and follow the instructions in /usr/share/doc/upstart/README.Debian to add a boot option that will use upstart instead of init. If your system boots and shut downs normally (other than a slightly more verbose boot without usplash running) then it is working correctly. Other types of events will be added as required during development and testing. Currently only a basic client tool (initctl) has been written, compatibility tools such as shutdown will be written over the next week or two before it replaces our sysvinit package.

Talking about logic, and where people fail at understanding it... Fun with stupid Google queries - Is there any page about Google and not Google? Wonder how big the Google index is? Google OR -Google. (I'm aware that the count is imprecise, it's just funny that this query is actually processed.) Now let's look at MSN. google 64741016
-google 4669973284
okay. that should make "4734714300"... google OR -google 5156927770 - oops, magic new hits.
-google OR google 68735504
So where are the 5 billion pages that have "google or -google" but not "-google or google"? And how about MSN OR -MSN - there can be only one. askdjfhsakdf OR -askdjfhsakdf - top result for this garbage word: Google! askdjfhsakdf OR -askdjfhsakdf Google results are consistent. I wonder what their sorting is in this case... random? hash function? age? Remember Googlestossen? Like Googlewhack, but with scoring. There must be only one hit with both words; score = # of hits with word 1 * # of hits with word 2

This post was part of my answer in a thread about the "sourceless data" general resolution on the debian-project mailing list. Firmwares are an essential part of any modern computer, and it is an annoying but currently hard to change fact of life that we lack the source for almost all of them. I reject the notion that we can ignore some fundamental parts of the computers which we use only because we do not distribute them: this would be hypocritical. Even if Debian stopped distributing sourceless firmwares users would continue to use them, either on a flash chip or by downloading and installing them on their file systems. Such a change on our part would not drive users to buy hardware whose firmwares are accompanied by their source, because with a very small number of exceptions there are no such devices. The claim that vendors would start to distribute the firmware sources is unproven, and indeed I cannot see how it could happen when only a small part of the community would care enough to deliberately make life harder for their customers. Instead this would push the less technically competent users toward other distributions, whose commitment to free software is usually less strong than our own. In this scenario I do not see an advancement for the cause of free software.

Well, two days ago Jason Norwood-Young published in Tectonic the worst article about Ubuntu, Debian and the Ubuntu latest release Dapper Drake until now. I would like to point out where he's wrong here, just to clarify for Debian users my view on the subject.<br /><br />Source: http://www.tectonic.co.za/view.php?id=1026<br /><br /><span style="font-style: italic;" class="viewbody">"I can only guess where Ubuntu went wrong. The first mistake, I think, was its desire to be a bleeding edge distribution, rather than a leading edge distro. Basing itself on Etch, Debian's unstable release, could be a problem. When the early versions of Ubuntu used the then-unstable Sarge as their foundations, it wasn't a risk -- Sarge was on the cusp of being released. Etch, on the other hand, is brand-new, and far from getting a thumbs-up as a stable distribution."</span><br /><br />The truth is that Ubuntu Dapper Drake was never based on Debian testing that we will be our next release, that we already call Etch. Ubuntu spend some time importing packages from Debian Unstable (that we always call sid too) and merging them with the changes they had and applying new changes, adding more packages software and all that. For more information about Debian release cycle, check our <a href="http://www.us.debian.org/releases/">Debian release page</a>.<br /><br /><span style="font-style: italic;" class="viewbody">"I also wonder who is doing the up-stream support. Shuttleworth poached the best-of-the-best Debian developers for Ubuntu. So who's working on Debian? The hald bug has been identified as an up-stream problem -- a Debian problem in other words. Perhaps Debian doesn't have the man-power to run around fixing Ubuntu bugs. "<br /><br /></span>Mr. Jason failed miserably here. Considering that he's right and Mark hired the best-of-the-best Debian developers for Ubuntu, the hald bug that is a software <a href="http://www.freedesktop.org/wiki/Software_2fhal">from freedesktop</a> and not Debian, should be solved by the best-of-the-best, no? Perhaps Ubuntu doesn't have the man-power to run around fixing every freedesktop bug.<br /><br /><span style="font-style: italic;" class="viewbody">"Finally, there's the problem of prioritising. Ubuntu, at its heart, is a community project. Many of the developers are not Canonical employees and don't get paid to work on Ubuntu -- they do it because the want to."<br /><br /><span style="font-style: italic;"></span></span><span class="viewbody">Great, Ubuntu at its heart, is a community project and what Debian is?<br /><br />I think the guy isn't a troll, but he should be more polite with both projects, after all he likes the previous Ubuntu release without considering that we (Debian) had some input there. In other words if it works it's Ubuntu, if it breaks is it upstream's fault? He can't recognize that Ubuntu upstream isn't always Debian.<br /><br />Meanwhile:<br /></span> <span class="body"><b>"Comment added</b></span><br /><span class="body">Thank you for your contribution <b>Gustavo Franco</b>. <br />Your comments will be posted on the site as soon as they have been moderated by the editors."<br /><br />We will see which kind of moderation i'll receive there.<br /></span>

Just in case you missed it before…Fernando Herrera and Lorenzo Gil came up with a lovely bit of code that allows the parts of the GNOME experience written in Python to send traces via bug-buddy when something goes wrong. Pessulus will soon have this and i’m working on doing it for Deskbar. Update: Forgot the link to the code…bug-buddy-integration.py.

I am quite happy to see that we will be able to make GConf FHS-compliant in etch.

Here are the packages still shipping files in /etc/gconf.

That makes really few, given that:

• goobox has been fixed,


• yank will probably be removed from the archive,


• we will probably be able to remove gconf (the GNOME 1 version) now that gnucash 2.0 has entered unstable.

This leaves only 2 bugs : #380728 (apt-watch, I only noticed it today) and #369858 (gnome-panel). Gnome-panel is the only case that couldn't be handled by dh_gconf, and I'll try to push one of the solutions that were thought of ASAP.

Thanks to everyone who made this possible, especially package maintainers who updated their packages so quickly (e.g. one hour for goobox).

Ninth day in Addis SSH To enable remote logins with ssh apt-get install openssh-server Then you can login with:

$ ssh efossnet@proxy.dream.edu.et

To verify the host key fingerprint of a machine:

$ ssh-keygen -l -f /etc/ssh/ssh_host_rsa_key.pub

Note: you need to verify it before logging in! More information at http://www.securityfocus.com/infocus/1806 Example ssh usages To log in:

    $ ssh efossnet@proxy

To run a command in the remote computer:

    $ ssh efossnet@proxy "cat /etc/hosts"

To copy a file to the remote computer:

    $ scp Desktop/july-18.tar.gz efossnet@proxy:

To copy a file from the remote computer:

    $ scp efossnet@proxy:july-18.tar.gz /tmp/

Beware of brute-force login attempts Warning about SSH: there are people who run automated scans for ssh servers and try to login using commonly used easy passwords. If you have an SSH server on the network, use strong passwords, or if you can it's even better to disable password authentication: in /etc/ssh/sshd_config, add:

    PasswordAuthentication no

To log in using public/private keys:

1. Create your key:

   ssh-keygen -t rsa
   

2. Copy your public key to the machine where you want to log in:

   ssh-copy-id -i .ssh/id_rsa.pub efossnet@proxy
   

3. Now you can ssh using your RSA key

If you use ssh often, read these:

• http://mah.everybody.org/docs/ssh
• http://www.securityfocus.com/infocus/1812
• http://www.sshkeychain.org/mirrors/SSH-with-Keys-HOWTO/SSH-with-Keys-HOWTO-6.html

proxy Problems we had today with the proxy: ssl does not work Reason: squid tries to directly connect to the ssl server, but the AAU network wants us to go through their proxy. Ideal solution: none. There is no way to tell squid to use a parent proxy for SSL connections. Solution: update the documentation for the Dream university users telling to setup a different proxy for SSL connections. Longer term solution: get the AAU network admins to enable outgoing SSL connections from the Dream university proxy. Other things that can be done:

• report a bug on squid reporting the need and requesting the feature
• download squid source code and implement the feature ourselves, then submit the patch to the squid people

Browsing normal pages returns an error of 'Connection refused'. In the logs, the line is:

1153294204.912    887 192.168.0.200 TCP_MISS/503 1441 GET http://www.google.com.et/search? - NONE/- text/html

That "/503" is one of the HTTP error codes. Explanation of the error codes:

• http://www.w3.org/Protocols/HTTP/HTRESP.html
• http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html
• http://offline.web.cern.ch/offline/web/http_error_codes.html

Reason: the other proxy is refusing connections from our proxy. Solution: none so far. Will need to get in touch with the admins of the other proxy to try to find out why it refuses connection to our proxy, and how we can fix the problem. postfix on smtp.dream.edu.et Basic information is at http://www.postfix.org/basic.html. Difference between mail name and smarthost:

• The mail name is the name of the mail server you're setting up (TODO: need more details on what's it used for)
• The smarthost is the name of the mail server that will relay mail for you.

Quick way to send test mails:

apt-get install mailx
echo ciao   mail efossnet@localhost

To configure a workstation not to do any mail delivery locally and send all mail produced locally to smtp.dream.edu.et:

1. install postfix choosing "Satellite system"
2. put smtp.dream.edu.et as a smarthost.

To setup a webmail: apt-get install squirrelmail (on a working apache setup). To setup mailing lists: apt-get install mailman, then follow the instructions in /usr/share/doc. Mail server issues we encountered When a mail is sent to efossnet@localhost, the system tries to send it to efossnet@yoseph.org Investigation:

• "yoseph.org" does not appear anywhere in /etc or /var/spool/postfix
• postfix configuration has been reloaded
• postfix logs show that the mail has been 'forwarded'

Cause: the user efossnet had forgotten that he or she had setup a .forward file in the home directory. Solution:

 rm ~efossnet/.forward

Apache To add a new website:

1. cd /etc/apache2/sites-available
2. sudo cp default course
3. sudo vi course:
   1. Remove the first line
   2. Add a ServerName directive with the address of your server: ServerName course.dream.edu.et
   3. Customize the rest as needed: you at least want to remove the support for browsing /usr/share/doc and you want to use a different document root.
4. sudo a2ensite course
5. sudo /etc/init.d/apache2 reload

More VIM Undo: u (in command mode) Redo: ^R (in command mode) You can undo and redo multiple times. To recover a lost password for root or for the ubuntu admin user Boot with a live CD, mount the system on the hard disk (the live CD usually does it automatically), then edit the file /etc/shadow, removing the password:

enrico:$1$3AJfasjJFHa234dfh230:13343:0:99999:7:::

becomes:

enrico::13343:0:99999:7:::

You can edit the file because, in the live CD system, you can always become root. After you do this, reboot the system: you can log in without password, and set yourself a new password using the command passwd. Installing packages not on the CDs To get a package for installing when offline:

1. apt-get --print-uris install dnsmasq
2. Manually download the packages at the URLs that it gives you

Otherwise, apt-get --download-only install dnsmasq will download the package for you in /var/cache/apt/archives. You can install various previously downloaded debian packages with:

dpkg -i *.deb

Backups There are various ways:

• dump (for ext2/ext3 file systems) or xfsdump (for xfs file systems). Makes a low-level dump of the file system. It must be used for every different partition. It makes the most exact backup possible, including inode numbers. It can do full and incremental backups. To see the type of the filesystems, use 'mount' with no parameters. To restore: restore or xfsrestore.
• tar Filesystem independent. It can work accross partitions. It correctly backups permissions and hard links. It can do full and incremental backups. Example:

  tar lzcpf backup.tar.gz /home /var /etc /usr/local
  tar lzcpf root.tar.gz /
  

  To restore:

  tar zxpf backup.tar.gz
  

• faubackup Filesystem independent. Uses hard drive as backup storage. Always incremental. It cannot do compression. Unchanged files in new backups are just links to old backups, and do not occupy space. Any old backup can be deleted at any time without compromising the others. It can be used to provided a "yesterday's files" service to users (both locally and exported as a read-only samba share...). To restore, just copy the files from the backup area.
• amanda

  apt-get install amanda-client amanda-server
  

  It is a network backup system. It can do full and incremental backups. You can have a backup server which handles the storage and various backup clients that send the files to backup to the server. It takes some studying to set up. To restore: it has its own tool.

Some data requires exporting before backing it up:

• To save the list of installed packages and the answer to configuration questions:

  dpkg --get-selections > pkglist
  debconf-get-selections > pkgconfig
  

  To restore:

  dpkg --set-selections < list
  debconf-set-selections < pkgconfig
  apt-get dselect-upgrade
  

  If you do this, they you only need to backup /etc, /home, /usr/local, /var.
• To save the contents of a MySQL database:

  mysqldump name-of-database   gzip > name-of-database.dump.gz
  

  To restore:

  zcat name-of-database.dump.gz   mysql
  

You can schedule these dumps to be made one hour before the time you make backups. Scheduling tasks As a user:

crontab -e

As root: add a file in one of the /etc/cron.* directories. In cron. hourly,daily,weekly,monthly you put scripts. In the other directories you put crontab files (man 5 crontab). If the system is turned off during normal maintainance hours, you can do two things:

1. Change /etc/crontab to use different maintanance hours
2. Install anacron (it's installed by default in ubuntu)

For scheduling one-shot tasks, use at(1):

$ at 17:40
echo "Please tell Enrico that the lesson is finished"   mail efossnet@dream.edu.et
^D

When and how to automate

1. First, you manage to do it yourself
2. Then, you document it
3. Then, you automate it

Start at step 1 and go to 2 or 3 if/when you actually need it. (credits to sto@debian.org: he's the one from which I heard it for the first time, said so well). Interesting programs to schedule during maintanance

• rkhunter, chkrootkit
• checksecurity
• debsecan
• tiger

Important keys to know in a Unix terminal These are special keys that work on Unix terminals:

• ^C: interrupt (sends SIGTERM)
• ^\: interrupt (send SIGQUIT)
• ^D: end of input
• ^S: stop scrolling
• ^Q: resume scrolling

Therefore, if the terminal looks like it got stuck, try hitting ^Q. Problems we had today with postfix

• Problem: mail to efossnet@dream.edu.et is accepted only if sent locally. Reason:

  $ host -t mx dream.edu.et
  Host dream.edu.et not found: 3(NXDOMAIN)
  

  Solution: tell dnsmasq to handle a MX record also for dream.edu.et: mx-host=dream.edu.et,smtp.dream.edu.et,50
• The problem not solved with the previous solution. Reason: postfix was making complaints which mentioned localhost as a domain name. Solution: fixed by changing 'myhostname' in main.cf to something different than localhost. Note: solved by luck. Investigate why this happened.

Problems found yesterday and today

• there is no way to tell squid to use another proxy for SSL connections: it only does them directly
• if you want to configure evolution to get mail from /var/mail/user, you need to explicitly enter the path. It would be trivially easier if evolution presented a good default, since it's easy to compute. It would also be useful if below the "Path" entry there were some text telling what path is being requested: the mail spool? the evolution mail storage?
• In Evolution: IMAP or IMAPv4r1? What is the difference? Why should I care?
• apt-get --print-uris doesn't print the URIs if the package is in the local cache, and there seems to be no way to have it do it.
• in /etc/apache2/sites-available/default, is the NameVirtualHost * directive appropriate there? It gets in the way when using 'default' as a template for new sites. Otherwise, one can add a new (disabled) site that can be used as a template for new sites instead of default.
• the default comments put by crontab -e are not that easy to read.

Happy SysAdmin day to all!

(Specially to Mr Climent (aka [data mooch]), my favourite sysadmin ever!)