Scott James Remnant: Upstart in Universe
upstart is a replacement for the init daemon, the process spawned by the kernel that is responsible for starting, supervising and stopping all other processes on the system.
The existing daemon is based on the one found in UNIX System V, and is thus known as
sysvinit
. It separates jobs into different “run levels” and can either run a job when particular run levels are entered (e.g. /etc/init.d/rc 2
) or continually during a particular run level (e.g. /sbin/getty
).
The /etc/init.d/rc
script is also based on the System V one (and is in the sysv-rc
package), it simply executes the stop then start scripts found in /etc/rcN.d
(where N is the run level) in numerical order.
Why change it?
Running a fixed set of scripts, one after the other, in a particular order has served us reasonably well until now. However as Linux has got better and better at dealing with modern computing (arguably Linux’s removable device support is better than Windows’ now) this approach has begun to have problems.
The old approach works as long as you can guarantee when in the boot sequence things are available, so you can place your init script after that point and know that it will work. Typical ordering requirements are:
- Hard drive devices must have been discovered, initialised and partitions detected before we try and mount from
/etc/fstab
.
- Network devices must have been discovered and initialised before we try and start networking.
- Drives can be plugged in and removed at any point, e.g. USB drives.
- Storage buses allow more than a fixed number of drives, so they must be scanned for; this operation frequently does not block.
- To reduce power consumption, the drive may not actually be spun up until the bus scan so will not appear for an even longer time.
- Network devices can be plugged in and removed at any point.
- Firmware may need to be loaded after the device has been detected, but before it is usable by the system.
- Mounting a partition in
/etc/fstab
may require tools in/usr
which is a network filesystem that cannot be mounted until after networking has been brought up.
- the system has started,
- the root filesystem is now writable,
- a block device has been added to the system,
- a filesystem has been mounted,
- at a certain time or repeated time period,
- another job has begun running or has finished,
- a file on the disk has been modified,
- there are files in a queue directory,
- a network device has been detected,
- the default route has been added or removed.
- system startup,
- file modified or placed in queue directory,
- particular time (cron replacement),
- connection on a particular port (inetd replacement).
/etc/fstab
) is available and writable.
The launchd model expects the job to “sit and wait” if it is unable to start, rather than provide a mechanism for the job to only be started when it doesn’t need to wait. Jobs that need /usr
to be mounted would need to spin in a loop waiting for /usr
to be available before continuing (or use a file in a tmpfs to indicate it’s available, and use that modification as the event).
This is not especially surprising given that Apple have a high degree of control over both their hardware and the actual underlying operating system; they don’t need to deal with the wide array of different configurations that we have in the Linux world.
Had the licence been sufficiently free at the point we began development of our own system, we would probably have extended launchd rather than implement our own. At the point Apple changed the licence, our own system was already more suitable for our purposes.
How does it differ from initng?
Initng by Jimmy Wennlund is another replacement init daemon intended to replace the sysvinit
system used by Linux. It is a dependency-based system, where upstart is an event-based system.
The notion of a dependency-based system is interesting to talk about at this point. Jobs declare dependencies on other jobs that need to happen before the job itself can be started. Starting the job causes its dependencies to be started first, and their dependencies, and so on. When jobs are stopped, if running jobs have no dependencies, they themselves can be stopped.
It’s a neat solution to the problem of ordering a fixed boot sequence and the problem of keeping the number of running processes to a minimum needed.
However this means that you need to have goals in mind when you boot the system, you need to have decided that you want gdm to be started in order for it, and its dependencies, to be started. Initng uses run levels to ensure this happens, where a run level is a list of goal jobs that should be running in that run level.
It’s also not clear how the dependencies interact with the different types of job, a dependency on Apache would need the daemon to be running where a dependency on “checkroot” would need the script to have finished running. Upstart handles this by using different events (“apache running” vs. “checkroot stopping”).
Again while interesting, Initng does not solve the problems that we wanted to solve. It can reorder a fixed set of jobs, but cannot dynamically determine the set of jobs needed for that particular boot.
A typical example would be that if the only dependency on the job that configures networking is the mount network filesystems job, then should that job fail or notbe a goal (e.g. because there are no network filesystems to be mounted) the result is that network devices themselves will not be configured. You could make everything a goal, and just use the dependencies to determine the order, however this is less efficient than just ordering the existing sysv-rc scripts (which
can be done at install time).
Another example is that often you simply don’t know whether something is a dependency or not without reading other configuration, for example the mount network filesystems may be a dependency of everything under /usr
or may just be a dependency of anything allowing the user to login if it just mounts /home
.
The difference in model can be summed up as “initng starts with a list of goals and works out how to get there, upstart starts with nothing and finds out where it gets to.”
How does it differ from Solaris SMF?
SMF is another approach to replacing init developed by Sun for the Solaris operating system. Like initng it’s a dependency-based system, so see above for the differences between those systems and upstart.
SMF’s main focus is serive management; making sure that once services are running, they stay running, and allowing the system administrator to query and modify the states of jobs on the system.
Upstart provides the same set of functionality in this regard, services are respawned when they fail and system administrators can at any time query the state of running services and adjust the state to their liking.
Will it replace cron, inetd, etc?
The goal of upstart is to replace those daemons, so that there is only one place (/etc/event.d
) where system administrators need to configure when and how jobs should be run.
In fact, the goal is that upstart should also replace the “run event scripts” functionality of any daemon on the system. Daemons such as acpid, apmd and Network Manager would send events to init instead of running scripts themselves with their own perculiar configuration and semantics.
A system administrator who only wanted a particular daemon to be run while the computer was on AC power would simply need to edit /etc/event.d/daemon
and change “on startup” to “on ac power”.
What about compatibility?
There’s a lot of systems administrators out there who have learned how Linux works already and will not want to learn again immediately, there’s also a large number of books that cover the existing software and won’t cover upstart for at least a couple of years.
For this reason, compatibility is very important. upstart will continue to run the existing scripts for the forseeable future so that packages will not need to be updated until the author wants.
Compatibility command-line tools that behave like their existing equivalents will also be implemented, a system administrator would never need to know that crontab -e
is actually changing upstart jobs.
Does it use D-BUS?
“To D-BUS people, every problem seems like a D-BUS problem.” —Erik TroanThe UNIX philosophy is that something should do just one job, and do it very well. upstart’s one job is starting, supervising and stopping other jobs; D-BUS’s one job is passing messages between other jobs. D-BUS does provide a mechanism for services to be activated when the first message is sent to them, thereby starting other jobs. Some people have taken this idea and extended it to suggest that all a replacement init system need do is register jobs with D-BUS and turn booting into a simple matter of message parsing. This seems wrong to me, D-BUS would need to be extended to supervise these services, provide means for them to be restarted and stopped; as well as deal with being process #1 which means cleaning up after children whose parent’s have died, etc. It seems far simpler to arrange for D-BUS to send an event to init when it needs a service to be started, and focus on being a very good message passing system. The IPC mechanism used by upstart is not currently D-BUS because of various problems, however it’s always been expected that even if init itself doesn’t communicate with D-BUS directly, there would be a D-BUS proxy that would ensure messages about all init jobs and events would be given to D-BUS and D-BUS clients could send messages to init to query and change the state of jobs. What is the implementation plan? Because this is process #1 we are changing, we want to make sure that we get it right. Therefore instead of releasing a fully-featured daemon and configuration to the world, we’re developing it in the following stages:
- Principal development; at the end of this stage the daemon has been implemented and can manage jobs as described.
- Replacement of
/sbin/init
while running the existing sysv-rc scripts. This is the shake-down test of the daemon, can it perform the same job as the existingsysvinit
daemon without any regressions? /etc/rcS.d scripts
replaced by upstart jobs. These consitute the majority of tasks for booting the system into at least single-user mode, and contain many of the current ordering problems and race conditions. If the daemon solves the problems here, it will be a success.- Other daemon’s scripts replaced by upstart jobs on a package-by-package basis; this will be an ongoing effort during which upstart will continue running the existing sysv-rc scripts as well as its own jobs. During this time the event system may be tweaked to ensure it truly solves the problems we need.
- Replcement of cron, atd, anacron and inetd. This will happen alongside the above and result in a single place to configure system jobs.
- Modification of other daemons and processes to send events to init instead of trying to run things themselves.
upstart
package for testing before it becomes the init daemon.
We welcome any experienced users who want to help test this; install the package and follow the instructions in /usr/share/doc/upstart/README.Debian
to add a boot option that will use upstart instead of init. If your system boots and shut downs normally (other than a slightly more verbose boot without usplash running) then it is working correctly.
Other types of events will be added as required during development and testing. Currently only a basic client tool (initctl
) has been written, compatibility tools such as shutdown
will be written over the next week or two before it replaces our sysvinit
package.