tl;dr: Providing support for all 3 init systems (sysv-rc, Upstart and systemd) isn t hard, and generating the init scripts / Upstart job / systemd using a template system is a lot easier than I previously thought.
As always, when writing this kind of blog post, I do expect that others will not like what I did. But that s the point: give me your opinion in a constructive way (please be polite even if you don t like what you see I had too many times had to read harsh comments), and I ll implement your ideas if I find it nice.
History of the implementation: how we came to the idea
I had no plan to do this. I don t believe what I wrote can be generalized to all of the Debian archive. It s just that I started doing things, and it made sense when I did it. Let me explain how it happened.
Since it s clear that many, and especially the most advanced one, may have an opinion about which init system they prefer, and because I also support Ubuntu (at least Trusty), I though it was a good idea to support all the main init system: sysv-rc, Upstart and systemd. Though I have counted (for the sake of being exact in this blog) : OpenStack in Debian contains currently 64 init scripts to run daemons in total. That s quite a lot. A way too much to just write them, all by hand. Though that s what I was doing for the last years until this the end of this last summer!
So, doing all by hand, I first started implementing Upstart. Its support was there only when building in Ubuntu (which isn t the correct thing to do, this is now fixed, read further ). Then we thought about adding support for systemd. Gustavo Panizzo, one of the contributors in the OpenStack packages, started implementing it in Keystone (the auth server for OpenStack) for the Juno release which was released this October. He did that last summer, early enough so we didn t expect anyone to use the Juno branch Keystone. After some experiments, we had kind of working. What he did was invoking /etc/init.d/keystone start-systemd , which was still using start-stop-daemon. Yes, that s not perfect, and it s better to use systemd foreground process handling, but at least, we had a unique place where to write the startup scripts, where we check the /etc/default for the logging configuration, configure the log file, and so on.
Then around in october, I took a step backward to see the whole picture with sysv-rc scripts, and saw the mess, with all the tiny, small difference between them. It became clear that I had to do something to make sure they were all the same, with the support for the same things (like which log system to use, where to store the PID, create /var/lib/project, /var/run/project and so on ).
Last, on this month of December, I was able to fix the remaining issues for systemd support, thanks to the awesome contribution of Mikael Cluseau on the Alioth OpenStack packaging list. Now, the systemd unit file is still invoking the init script, but it s not using start-stop-daemon anymore, no PID file involved, and daemons are used as systemd foreground processes. Finally, daemons service files are also activated on installation (they were not previously).
Implementation
So I took the simplistic approach to use always the same template for the sysv-rc switch/case, and the start and stop functions, happening it at the end of all debian/*.init.in scripts. I started to try to reduce the number of variables, and I was surprised of the result: only a very small part of the init scripts need to change from daemon to daemon. For example, for nova-api, here s the init script (LSB header stripped-out):
DESC="OpenStack Compute API"
PROJECT_NAME=nova
NAME=$ PROJECT_NAME -api
That is it: only 3 lines, defining only the name of the daemon, the name of the project it attaches (eg: nova, cinder, etc.), and a long description. There s of course much more complicated init scripts (see the one for neutron-server in the Debian archive for example), but the vast majority only needs the above.
Here s the sysv-rc init script template that I currently use:
#!/bin/sh
# The content after this line comes from openstack-pkg-tools
# and has been automatically added to a .init.in script, which
# contains only the descriptive part for the daemon. Everything
# else is standardized as a single unique script.
# Author: Thomas Goirand <zigo@debian.org>
# PATH should only include /usr/* if it runs after the mountnfs.sh script
PATH=/sbin:/usr/sbin:/bin:/usr/bin
if [ -z "$ DAEMON " ] ; then
DAEMON=/usr/bin/$ NAME
fi
PIDFILE=/var/run/$ PROJECT_NAME /$ NAME .pid
if [ -z "$ SCRIPTNAME " ] ; then
SCRIPTNAME=/etc/init.d/$ NAME
fi
if [ -z "$ SYSTEM_USER " ] ; then
SYSTEM_USER=$ PROJECT_NAME
fi
if [ -z "$ SYSTEM_USER " ] ; then
SYSTEM_GROUP=$ PROJECT_NAME
fi
if [ "$ SYSTEM_USER " != "root" ] ; then
STARTDAEMON_CHUID="--chuid $ SYSTEM_USER :$ SYSTEM_GROUP "
fi
if [ -z "$ CONFIG_FILE " ] ; then
CONFIG_FILE=/etc/$ PROJECT_NAME /$ PROJECT_NAME .conf
fi
LOGFILE=/var/log/$ PROJECT_NAME /$ NAME .log
if [ -z "$ NO_OPENSTACK_CONFIG_FILE_DAEMON_ARG " ] ; then
DAEMON_ARGS="$ DAEMON_ARGS --config-file=$ CONFIG_FILE "
fi
# Exit if the package is not installed
[ -x $DAEMON ] exit 0
# If ran as root, create /var/lock/X, /var/run/X, /var/lib/X and /var/log/X as needed
if [ "x$USER" = "xroot" ] ; then
for i in lock run log lib ; do
mkdir -p /var/$i/$ PROJECT_NAME
chown $ SYSTEM_USER /var/$i/$ PROJECT_NAME
done
fi
# This defines init_is_upstart which we use later on (+ more...)
. /lib/lsb/init-functions
# Manage log options: logfile and/or syslog, depending on user's choosing
[ -r /etc/default/openstack ] && . /etc/default/openstack
[ -r /etc/default/$NAME ] && . /etc/default/$NAME
[ "x$USE_SYSLOG" = "xyes" ] && DAEMON_ARGS="$DAEMON_ARGS --use-syslog"
[ "x$USE_LOGFILE" != "xno" ] && DAEMON_ARGS="$DAEMON_ARGS --log-file=$LOGFILE"
do_start()
start-stop-daemon --start --quiet --background $ STARTDAEMON_CHUID --make-pidfile --pidfile $ PIDFILE --chdir /var/lib/$ PROJECT_NAME --startas $DAEMON \
--test > /dev/null return 1
start-stop-daemon --start --quiet --background $ STARTDAEMON_CHUID --make-pidfile --pidfile $ PIDFILE --chdir /var/lib/$ PROJECT_NAME --startas $DAEMON \
-- $DAEMON_ARGS return 2
do_stop()
start-stop-daemon --stop --quiet --retry=TERM/30/KILL/5 --pidfile $PIDFILE
RETVAL=$?
rm -f $PIDFILE
return "$RETVAL"
do_systemd_start()
exec $DAEMON $DAEMON_ARGS
case "$1" in
start)
init_is_upstart > /dev/null 2>&1 && exit 1
log_daemon_msg "Starting $DESC" "$NAME"
do_start
case $? in
0 1) log_end_msg 0 ;;
2) log_end_msg 1 ;;
esac
;;
stop)
init_is_upstart > /dev/null 2>&1 && exit 0
log_daemon_msg "Stopping $DESC" "$NAME"
do_stop
case $? in
0 1) log_end_msg 0 ;;
2) log_end_msg 1 ;;
esac
;;
status)
status_of_proc "$DAEMON" "$NAME" && exit 0 exit $?
;;
systemd-start)
do_systemd_start
;;
restart force-reload)
init_is_upstart > /dev/null 2>&1 && exit 1
log_daemon_msg "Restarting $DESC" "$NAME"
do_stop
case $? in
0 1)
do_start
case $? in
0) log_end_msg 0 ;;
1) log_end_msg 1 ;; # Old process is still running
*) log_end_msg 1 ;; # Failed to start
esac
;;
*) log_end_msg 1 ;; # Failed to stop
esac
;;
*)
echo "Usage: $SCRIPTNAME start stop status restart force-reload systemd-start " >&2
exit 3
;;
esac
exit 0
Nothing particularly fancy here You ll noticed that it s really OpenStack centric (see the LOGFILE and CONFIGFILE things ). You may have also noticed the call to init_is_upstart which is needed for upstart support. I m not sure if it s at the correct place in the init script. Should I put that on top of the script? Was I right with the exit values for it? Please send me your comments
Then I thought about generalizing all of this. Because not only the sysv-rc scripts needed to be squared-up, but also Upstart. The approach here was to source the sysv-rc script in debian/*.init.in, and then generate the Upstart job accordingly, using the above 3 variables (or more as needed). Here, the fun is that, instead of taking the approach of calculating everything at runtime with the sysv-rc, for Upstart jobs, many things are calculated at build time. For each debian/*.init.in script that the debian/rules finds, pkgos-gen-upstart-job is called. Here s pkgos-gen-upstart-job:
#!/bin/sh
INIT_TEMPLATE=$ 1
UPSTART_FILE= echo $ INIT_TEMPLATE sed 's/.init.in/.upstart/'
# Get the variables defined in the init template
. $ INIT_TEMPLATE
## Find out what should go in After=
#SHOULD_START= cat $ INIT_TEMPLATE grep "# Should-Start:" sed 's/# Should-Start://'
#
#if [ -n "$ SHOULD_START " ] ; then
# AFTER="After="
# for i in $ SHOULD_START ; do
# AFTER="$ AFTER $ i .service "
# done
#fi
if [ -z "$ DAEMON " ] ; then
DAEMON=/usr/bin/$ NAME
fi
PIDFILE=/var/run/$ PROJECT_NAME /$ NAME .pid
if [ -z "$ SCRIPTNAME " ] ; then
SCRIPTNAME=/etc/init.d/$ NAME
fi
if [ -z "$ SYSTEM_USER " ] ; then
SYSTEM_USER=$ PROJECT_NAME
fi
if [ -z "$ SYSTEM_GROUP " ] ; then
SYSTEM_GROUP=$ PROJECT_NAME
fi
if [ "$ SYSTEM_USER " != "root" ] ; then
STARTDAEMON_CHUID="--chuid $ SYSTEM_USER :$ SYSTEM_GROUP "
fi
if [ -z "$ CONFIG_FILE " ] ; then
CONFIG_FILE=/etc/$ PROJECT_NAME /$ PROJECT_NAME .conf
fi
LOGFILE=/var/log/$ PROJECT_NAME /$ NAME .log
DAEMON_ARGS="$ DAEMON_ARGS --config-file=$ CONFIG_FILE "
echo "description \"$ DESC \"
author \"Thomas Goirand <zigo@debian.org>\"
start on runlevel [2345]
stop on runlevel [!2345]
chdir /var/run
pre-start script
for i in lock run log lib ; do
mkdir -p /var/\$i/$ PROJECT_NAME
chown $ SYSTEM_USER /var/\$i/$ PROJECT_NAME
done
end script
script
[ -x \"$ DAEMON \" ] exit 0
DAEMON_ARGS=\"$ DAEMON_ARGS \"
[ -r /etc/default/openstack ] && . /etc/default/openstack
[ -r /etc/default/\$UPSTART_JOB ] && . /etc/default/\$UPSTART_JOB
[ \"x\$USE_SYSLOG\" = \"xyes\" ] && DAEMON_ARGS=\"\$DAEMON_ARGS --use-syslog\"
[ \"x\$USE_LOGFILE\" != \"xno\" ] && DAEMON_ARGS=\"\$DAEMON_ARGS --log-file=$ LOGFILE \"
exec start-stop-daemon --start --chdir /var/lib/$ PROJECT_NAME \\
$ STARTDAEMON_CHUID --make-pidfile --pidfile $ PIDFILE \\
--exec $ DAEMON -- --config-file=$ CONFIG_FILE \$ DAEMON_ARGS
end script
" >$ UPSTART_FILE
The only thing which I don t know how to do, is how to implement the Should-Start / Should-Stop in an Upstart job. Can anyone shoot me a mail and tell me the solution?
Then, I wanted to add support for systemd. Here, we cheated, since we only just called the sysv-rc script from the systemd unit, however, the systemd-start target uses exec, so the process stays in the foreground. It s also much smaller than the Upstart thing. However, here, I could implement the After thing, corresponding to the Should-Start:
#!/bin/sh
INIT_TEMPLATE=$ 1
SERVICE_FILE= echo $ INIT_TEMPLATE sed 's/.init.in/.service/'
# Get the variables defined in the init template
. $ INIT_TEMPLATE
if [ -z "$ SCRIPTNAME " ] ; then
SCRIPTNAME=/etc/init.d/$ NAME
fi
if [ -z "$ SYSTEM_USER " ] ; then
SYSTEM_USER=$ PROJECT_NAME
fi
if [ -z "$ SYSTEM_GROUP " ] ; then
SYSTEM_GROUP=$ PROJECT_NAME
fi
# Find out what should go in After=
SHOULD_START= cat $ INIT_TEMPLATE grep "# Should-Start:" sed 's/# Should-Start://'
if [ -n "$ SHOULD_START " ] ; then
AFTER="After="
for i in $ SHOULD_START ; do
AFTER="$ AFTER $ i .service "
done
fi
echo "[Unit]
Description=$ DESC
$AFTER
[Service]
User=$ SYSTEM_USER
Group=$ SYSTEM_GROUP
WorkingDirectory=/var/lib/$ PROJECT_NAME
PermissionsStartOnly=true
ExecStartPre=/bin/mkdir -p /var/lock/$ PROJECT_NAME /var/log/$ PROJECT_NAME /var/lib/$ PROJECT_NAME
ExecStartPre=/bin/chown $ SYSTEM_USER :$ SYSTEM_GROUP /var/lock/$ PROJECT_NAME /var/log/$ PROJECT_NAME /var/lib/$ PROJECT_NAME
ExecStart=$ SCRIPTNAME systemd-start
Restart=on-failure
[Install]
WantedBy=multi-user.target
" >$ SERVICE_FILE
As you can see, it s calling /etc/init.d/$ SCRIPTNAME sytemd-start, which isn t great. I d be happy to have comments from systemd user / maintainers on how to fix it to make it better.
Integrating in debian/rules
To integrate with the Debian package build system, we only need had to write this:
override_dh_installinit:
# Create the init scripts from the template
for i in ls -1 debian/*.init.in ; do \
MYINIT= echo $$i sed s/.init.in// ; \
cp $$i $$MYINIT.init ; \
cat /usr/share/openstack-pkg-tools/init-script-template >>$$MYINIT.init ; \
pkgos-gen-systemd-unit $$i ; \
done
# If there's an upstart.in file, use that one instead of the generated one
for i in ls -1 debian/*.upstart.in ; do \
MYPKG= echo $$i sed s/.upstart.in// ; \
cp $$MYPKG.upstart.in $$MYPKG.upstart ; \
done
# Generate the upstart job if there's no already existing .upstart.in
for i in ls debian/*.init.in ; do \
MYINIT= echo $$i sed s/.init.in/.upstart.in/ ; \
if ! [ -e $$MYINIT ] ; then \
pkgos-gen-upstart-job $$i ; \
fi \
done
dh_installinit --error-handler=true
# Generate the systemd unit file
# Note: because dh_systemd_enable is called by the
# dh sequencer *before* dh_installinit, we have
# to process it manually.
for i in ls debian/*.init.in ; do \
pkgos-gen-systemd-unit $$i ; \
MYSERVICE= echo $$i sed 's/debian\///' ; \
MYSERVICE= echo $$MYSERVICE sed 's/.init.in/.service/' ; \
dh_systemd_enable $$MYSERVICE ; \
done
As you can see, it s possible to use a debian/*.upstart.in and not use the templating system, in the more complicated case (I needed it mostly for neutron-server and neutron-plugin-openvswitch-agent).
Conclusion
I do not pretend that what I wrote in the openstack-pkg-tools is the ultimate solution. But I m convince that it answers our own need as the OpenStack maintainers in Debian. There s a lot of room for improvements (like implementing the Should-Start in Upstart jobs, or stop calling the sysv-rc script in the systemd units), but that this is a very good move that we did to use templates and generated scripts, as the init scripts are a way more easy to maintain now, in a much more unified way. As I m not completely satisfied for the systemd and Upstart implementation, I m sure that there s already a huge improvements on the sysv-rc script maintainability.
Last and again: please send your comments and help improving the above! :)
The last remaining 6 bugs are only affecting OpenStack Essex (which is what Wheezy ships, and which unfortunately I have not enough time to support properly), and the last one is waiting for FTP masters approval after I added Python 3 support to oslo-config (OpenStack is slowly moving to Python3, and I ve tried to add support to Python3 in the packages as well each time it was possible). Also, the Icehouse packages have passed the tempest tests (a set of functional tests) which we run in our continuous integration system.
As I was doing some triaging, it is possible that some bugs have been closed a bit quickly (like for example the Ceilometer FTBFS which I couldn t reproduce, even with sbuild and cowbuilder), so it is well possible that some more QA work will be needed later on. I m also expecting a new set of patches for supporting Ceph in Nova. I m sure there s issues which we will discover later on, however, it is nice to have this result right after the first Icehouse release of OpenStack.
Next up: testing new components that I uploaded for this release: Trove (DB as a Service), Designate (DNS as a Service), Ironic (Metal as a Service, or cloud computing on bare-metal), and TripleO (OpenStack on OpenStack). I unfortunately know already that TripleO and Tuskar wont really work yet, and that it needs some patches to be sent upstream for it to support Debian correctly (let s work it out for Juno!). So please consider it as a technology preview only. Though Trove, Designate and Ironic are supposed to be in good enough shape, I didn t have the chance to test them more than just installing the packages and checking daemons are running and connected to the various components of OpenStack (eg: database, keystone and RabbitMQ). Please do test them and report bugs in the BTS.
I d like to thanks Gustavo Panizzo & Thomas Bechtold who contributed to this release, and also the folks at eNovance (Emilien, Cyril, Seb ) who provided precious help with the CI and package quality. See you in the Juno Atlanta design summit next week (as hopefully, my plane ticket issue will be soon solved).
Day 9 is a bit of a strange one. I had a rare three evenings spare,
but I have been feeling very ill so I didn't achieve much, relatively,
despite the increase.
I update