Antoine Beaupr : Debian considering automated upgrades
The Debian project is looking at possibly making automatic minor
upgrades to installed packages the default for newly installed systems.
While Debian has a reliable and stable package update system that has
been an inspiration for multiple operating systems (the venerable APT),
upgrades are, usually, a manual process on Debian for most users.
The proposal was brought
up during
the Debian Cloud
sprint in
November by longtime Debian Developer Steve McIntyre. The rationale was
to make sure that users installing Debian in the cloud have a "secure"
experience by default, by installing and configuring the
Problems with automated upgrades
An issue raised in the following discussion is that automated upgrades
may create unscheduled downtime for critical services. For example,
certain sites may not be willing to tolerate a master MySQL server
rebooting in conditions not controlled by the administrators. The
consensus seems to be that experienced administrators will be able to
solve this issue on their own, or are already doing so.
For example, Noah Meyerhans, a Debian developer,
argued
that "any reasonably well managed production host is going to be driven
by some kind of configuration management system" where competent
administrators can override the defaults. Debian, for example, provides
the
unattended-upgrades
package within the images.
The unattended-upgrades
package contains a Python program that
automatically performs any pending upgrade and is designed to run
unattended. It is roughly the equivalent of doing
apt-get update; apt-get upgrade
in a cron job, but has special code to
handle error conditions, warn about reboots, and selectively upgrade
packages. The package was originally written for Ubuntu by Michael Vogt,
a longtime Debian developer and Canonical employee.
Since there was a concern that Debian cloud images would be different
from normal Debian installs, McIntyre suggested installing
unattended-upgrades
by default on all Debian installs, so that people
have a consistent experience inside and outside of the cloud. The
discussion that followed was interesting as it brought up key issues one
would have when deploying automated upgrade tools, outlining both the
benefits and downsides to such systems.
Problems with automated upgrades
An issue raised in the following discussion is that automated upgrades
may create unscheduled downtime for critical services. For example,
certain sites may not be willing to tolerate a master MySQL server
rebooting in conditions not controlled by the administrators. The
consensus seems to be that experienced administrators will be able to
solve this issue on their own, or are already doing so.
For example, Noah Meyerhans, a Debian developer,
argued
that "any reasonably well managed production host is going to be driven
by some kind of configuration management system" where competent
administrators can override the defaults. Debian, for example, provides
the policy-rc.d
mechanism
to disable service restarts on certain packages out of the box.
unattended-upgrades
also features a way to disable upgrades on
specific packages that administrators would consider too sensitive to
restart automatically and will want to schedule during maintenance
windows.
Reboots were another issue discussed: how and when to deploy kernel
upgrades? Automating kernel upgrades may mean data loss if the reboot
happens during a critical operation. On Debian systems, the kernel
upgrade mechanisms already provide a /var/run/reboot-required
flag
file that tools can monitor to notify users of the required reboot. For
example, some desktop environments will popup a warning prompting users
to reboot when the file exists. Debian doesn't currently
feature an equivalent warning for command-line operation: Vogt
suggested
that the warning could be shown along with the usual /etc/motd
announcement.
The ideal solution here, of course, is reboot-less kernel upgrades,
which is also known as "live patching" the kernel. Unfortunately, this
area is still in development in the kernel (as was previously discussed
here). Canonical deployed the
feature for the Ubuntu 16.04 LTS release, but Debian doesn't yet have
such capability, since it requires extra infrastructure among other
issues.
Furthermore, system reboots are only one part of the problem. Currently,
upgrading packages only replaces the code and restarts the primary
service shipped with a given package. On library upgrades, however,
dependent services may not necessarily notice and will keep running with
older, possibly vulnerable, libraries. While libc6
, in Debian, has
special code to restart dependent services, other libraries like
libssl
do not notify dependent services that they need to restart to
benefit from potentially critical security fixes.
One solution to this is the needrestart
package which inspects all
running processes and restarts services as necessary. It also covers
interpreted code, specifically Ruby, Python, and Perl. In my experience,
however, it can take up to a minute to inspect all processes, which
degrades the interactivity of the usually satisfying apt-get install
process. Nevertheless, it seems like needrestart
is a key component of
a properly deployed automated upgrade system.
Benefits of automated upgrades
One thing that was less discussed is the actual benefit of automating
upgrades. It is merely described as "secure by default" by McIntyre in
the proposal, but no one actually expanded on this much. For me,
however, it is now obvious that any out-of-date system will be
systematically attacked by automated probes and may be taken over to the
detriment of the whole internet community, as we are seeing with
Internet of Things devices. As Debian Developer Lars Wirzenius
said:
The ecosystem-wide security benefits of having Debian systems keep
up to date with security updates by default overweigh any
inconvenience of having to tweak system configuration on hosts where
the automatic updates are problematic.
One could compare automated upgrades with backups: if they are not
automated, they do not exist and you will run into trouble without
them. (Wirzenius, coincidentally, also works on the
Obnam backup software.)
Another benefit that may be less obvious is the acceleration of the
feedback loop between developers and users: developers like to know
quickly when an update creates a regression. Automation does create the
risk of a bad update affecting more users, but this issue is already
present, to a lesser extent, with manual updates. And the same solution
applies: have a staging area for security
upgrades, the same way updates to
Debian stable are first
proposed before
shipping a point
release. This
doesn't have to be limited to stable security updates either: more
adventurous users could follow rolling distributions like Debian testing
or unstable with unattended upgrades as well, with all the risks and
benefits that implies.
Possible non-issues
That there was not a backlash against the proposal surprised me: I
expected the privacy-sensitive Debian community to react negatively to
another "phone home" system as it did with the Django
proposal. This, however, is different than a phone
home system: it merely leaks package lists and one has to leak that
information to get the updated packages. Furthermore, privacy-sensitive
administrators can use APT over
Tor
to fetch packages. In addition, the diversity of the mirror
infrastructure makes it difficult for a single entity to profile users.
Automated upgrades do imply a culture change, however: administrators
approve changes only a posteriori as opposed to deliberately deciding
to upgrade parts they chose. I remember a time when I had to maintain
proprietary operating systems and was reluctant to enable automated
upgrades: such changes could mean degraded functionality or additional
spyware. However, this is the free-software world and upgrades generally
come with bug fixes and new features, not additional restrictions.
Automating major upgrades?
While automating minor upgrades is one part of the solution to the
problem of security maintenance, the other is how to deal with major
upgrades. Once a release becomes unsupported, security issues may come
up and affect older software. While Debian
LTS extends releases lifetimes
significantly, it merely delays the inevitable major upgrades. In the
grand scheme of things, the lifetimes of Linux systems (Debian: 3-5
years, Ubuntu: 1-5 years) versus other operating systems (Solaris:
10-15 years,
Windows:
10+ years) is fairly short, which makes major upgrades especially
critical.
While major upgrades are not currently automated in Debian, they are
usually pretty simple: edit sources.list
then:
# apt-get update && apt-get dist-upgrade
But the actual upgrade process is really much more complex. If you run
into problems with the above commands, you will quickly learn that you
should have followed the release
notes, a
whopping 20,000-word, ten-section document that outlines all the gory
details of the release. This is a real issue for large deployments and
for users unfamiliar with the command line.
The solutions most administrators seem to use right now is to roll their
own automated upgrade process. For example, the Debian.org system
administrators have their own
process for the
"jessie" (8.0) upgrade. I have also written a
specification of how major
upgrades could be automated that attempts to take into account the wide
variety of corner cases that occur during major upgrades, but it is
currently at the design stage. Therefore, this problem space is
generally unaddressed in Debian: Ubuntu does have a do-release-upgrade
command
but it is Ubuntu-specific and would need significant
changes
in order to work in Debian.
Future work
Ubuntu currently defaults to "no
automation"
but, on install, invites users to enable unattended-upgrades
or
Landscape, a proprietary
system-management service from Canonical. According to Vogt, the company
supports both projects equally as they differ in scope:
unattended-upgrades
just upgrades packages while Landscape aims at
maintaining thousands of machines and handles user management, release
upgrades, statistics, and aggregation.
It appears that Debian will enable unattended-upgrades
on the images
built for the cloud by default. For regular installs, the consensus
that has
emerged
points at the Debian installer prompting users to ask if they want to
disable the feature as well. One reason why this was not enabled
before is that unattended-upgrades
had serious bugs in the past that
made it less attractive. For example, it would simply fail to follow
security
updates, a
major bug that was fortunately promptly fixed by the maintainer.
In any case, it is important to distribute security and major upgrades
on Debian machines in a timely manner. In my long experience in
professionally administering Unix server farms, I have found the upgrade
work to be a critical but time-consuming part of my work. During that
time, I successfully deployed an automated upgrade system all the way
back to Debian woody, using the simpler
cron-apt. This approach is,
unfortunately, a little brittle and non-standard; it doesn't address the
need of automating major upgrades, for which I had to revert to tools
like cluster-ssh or more
specialized configuration management tools like
Puppet. I therefore encourage any effort towards
improving that process for the whole community.
More information about the configuration of unattended-upgrades can be
found in the Ubuntu
documentation
or the Debian wiki.
Note: this article first appeared in the
Linux Weekly News.
The ecosystem-wide security benefits of having Debian systems keep up to date with security updates by default overweigh any inconvenience of having to tweak system configuration on hosts where the automatic updates are problematic.One could compare automated upgrades with backups: if they are not automated, they do not exist and you will run into trouble without them. (Wirzenius, coincidentally, also works on the Obnam backup software.) Another benefit that may be less obvious is the acceleration of the feedback loop between developers and users: developers like to know quickly when an update creates a regression. Automation does create the risk of a bad update affecting more users, but this issue is already present, to a lesser extent, with manual updates. And the same solution applies: have a staging area for security upgrades, the same way updates to Debian stable are first proposed before shipping a point release. This doesn't have to be limited to stable security updates either: more adventurous users could follow rolling distributions like Debian testing or unstable with unattended upgrades as well, with all the risks and benefits that implies.
Possible non-issues
That there was not a backlash against the proposal surprised me: I
expected the privacy-sensitive Debian community to react negatively to
another "phone home" system as it did with the Django
proposal. This, however, is different than a phone
home system: it merely leaks package lists and one has to leak that
information to get the updated packages. Furthermore, privacy-sensitive
administrators can use APT over
Tor
to fetch packages. In addition, the diversity of the mirror
infrastructure makes it difficult for a single entity to profile users.
Automated upgrades do imply a culture change, however: administrators
approve changes only a posteriori as opposed to deliberately deciding
to upgrade parts they chose. I remember a time when I had to maintain
proprietary operating systems and was reluctant to enable automated
upgrades: such changes could mean degraded functionality or additional
spyware. However, this is the free-software world and upgrades generally
come with bug fixes and new features, not additional restrictions.
Automating major upgrades?
While automating minor upgrades is one part of the solution to the
problem of security maintenance, the other is how to deal with major
upgrades. Once a release becomes unsupported, security issues may come
up and affect older software. While Debian
LTS extends releases lifetimes
significantly, it merely delays the inevitable major upgrades. In the
grand scheme of things, the lifetimes of Linux systems (Debian: 3-5
years, Ubuntu: 1-5 years) versus other operating systems (Solaris:
10-15 years,
Windows:
10+ years) is fairly short, which makes major upgrades especially
critical.
While major upgrades are not currently automated in Debian, they are
usually pretty simple: edit sources.list
then:
# apt-get update && apt-get dist-upgrade
But the actual upgrade process is really much more complex. If you run
into problems with the above commands, you will quickly learn that you
should have followed the release
notes, a
whopping 20,000-word, ten-section document that outlines all the gory
details of the release. This is a real issue for large deployments and
for users unfamiliar with the command line.
The solutions most administrators seem to use right now is to roll their
own automated upgrade process. For example, the Debian.org system
administrators have their own
process for the
"jessie" (8.0) upgrade. I have also written a
specification of how major
upgrades could be automated that attempts to take into account the wide
variety of corner cases that occur during major upgrades, but it is
currently at the design stage. Therefore, this problem space is
generally unaddressed in Debian: Ubuntu does have a do-release-upgrade
command
but it is Ubuntu-specific and would need significant
changes
in order to work in Debian.
Future work
Ubuntu currently defaults to "no
automation"
but, on install, invites users to enable unattended-upgrades
or
Landscape, a proprietary
system-management service from Canonical. According to Vogt, the company
supports both projects equally as they differ in scope:
unattended-upgrades
just upgrades packages while Landscape aims at
maintaining thousands of machines and handles user management, release
upgrades, statistics, and aggregation.
It appears that Debian will enable unattended-upgrades
on the images
built for the cloud by default. For regular installs, the consensus
that has
emerged
points at the Debian installer prompting users to ask if they want to
disable the feature as well. One reason why this was not enabled
before is that unattended-upgrades
had serious bugs in the past that
made it less attractive. For example, it would simply fail to follow
security
updates, a
major bug that was fortunately promptly fixed by the maintainer.
In any case, it is important to distribute security and major upgrades
on Debian machines in a timely manner. In my long experience in
professionally administering Unix server farms, I have found the upgrade
work to be a critical but time-consuming part of my work. During that
time, I successfully deployed an automated upgrade system all the way
back to Debian woody, using the simpler
cron-apt. This approach is,
unfortunately, a little brittle and non-standard; it doesn't address the
need of automating major upgrades, for which I had to revert to tools
like cluster-ssh or more
specialized configuration management tools like
Puppet. I therefore encourage any effort towards
improving that process for the whole community.
More information about the configuration of unattended-upgrades can be
found in the Ubuntu
documentation
or the Debian wiki.
Note: this article first appeared in the
Linux Weekly News.
sources.list
then:
# apt-get update && apt-get dist-upgrade
But the actual upgrade process is really much more complex. If you run
into problems with the above commands, you will quickly learn that you
should have followed the release
notes, a
whopping 20,000-word, ten-section document that outlines all the gory
details of the release. This is a real issue for large deployments and
for users unfamiliar with the command line.
The solutions most administrators seem to use right now is to roll their
own automated upgrade process. For example, the Debian.org system
administrators have their own
process for the
"jessie" (8.0) upgrade. I have also written a
specification of how major
upgrades could be automated that attempts to take into account the wide
variety of corner cases that occur during major upgrades, but it is
currently at the design stage. Therefore, this problem space is
generally unaddressed in Debian: Ubuntu does have a do-release-upgrade
command
but it is Ubuntu-specific and would need significant
changes
in order to work in Debian.