Jamie McClelland: Puppet Without Masters
Puppet labs's use of the term puppetmaster is rather clever (in contrast to other un-necessarily offensive uses of "master" in the software world).
While I appreciate the clever name, I'm less impressed with the concept.
At May First/People Link we've spent the last several years (including the last couple months in earnest) working to transition management of our 90-some servers from a collection of hand-written bash scripts to puppet.
Over the years, we've worked hard to keep our servers as secure as possible. We have a team of about a half dozen people who all have root access on all servers. It's all key-based access. To help mitigate a disaster if one person's keys were compromised, we've implemented monkeysphere on all servers, allowing us to easily revoke access.
After spending so much time thinking through our root-access strategy and fully implementing the monkeysphere to reduce our exposure to a single point of vulnerability, I was disappointed by puppet's use of a puppet master. For those less familiar with puppet, it goes something like this:
One server (or god forbid multiple servers), run an externally accessible daemon. Each and every server on your network runs a daemon as root that periodically communicates with the puppet master, receives new instructions, and then (again, as root) executes these instructions.
In other words, if your puppet master is compromised, I'm not sure exactly what you would need to do, short of rebuilding every server in your network.
To make matters worse, it seems as though some users generate and store all server ssh keys (private and public) on the puppet master and then push the private keys to their respective nodes. That means an intruder doesn't need to write to the puppet master, just reading these keys would be enough to compromise all servers in your network.
There must be a better way.
Puppet without masters
After some web-searching, I found a promising thread on the puppet list asking what's lost without a puppet master. This thread lead to a couple other blogs by people who have worked out a system for using puppet without a master.
It turns out that there are two distinct points of centralization with puppet. One is the puppet master (as described above). In addition, there is a concept called storeconfigs - which allows each node in the network to store information in a central database. For example, one server can store a request for an account to be setup on a backup server. The next time the backup server runs, it checks the storeconfigs, finds the request, and creates the user.
It's possible to run puppet with storeconfigs but without running a puppet master (that avoids the hassle of running the puppet daemons, while providing the convenience of centralization). For our purposes, however, we decided to forego both the puppet master and storeconfigs. We did not want any form of centralization that would provide an additional point of vulnerability.
As is common with puppet, we are storing our puppet recipes in a git repository. And, we are publishing to a single, canonical git repository on the Internet. On each node, we have two git repositories - one is a bare repo (that we can push to) and the other is a checked out repo (in /etc/puppet) that is read by puppet. The bare repo has a post-update hook the changes into the /etc/puppet directory, pulls in the changes from the bare repository, and runs puppet against the newly checked out files. Therefore, we can apply new puppet recipes to any server on the network with
git push <server>
git remote update from /etc/puppet. The script then checks the time stamp on the most recent gpg-signed tag and compares it with the time stamp of the current commit. If the most recent gpg-signed tag is newer, it verifies that the tag came from a list of authorized gpg keys (the very same gpg keys used by the monkeysphere to grant root level ssh access). If the gpg signature of the tag can be properly verified, then the changes are merged and puppet is run on the new recipes.
What about privacy?
One of the benefits of a puppet master setup is that nodes get configuration details on a need-to-know basis. The puppet master doesn't share the entire puppet repo - only the compiled manifest for the node with which it's communicating.
Our solution to this problem was to go screaming in the other direction. As you might notice from our support wiki and ticket system, we generally favor transparency. Since we are publishing our entire puppet git repo publicly, there seems little point in trying to hide one node's configuration details from another node.
That also means each node carries around about 4Mb of extra weight in the form of disk space for the git repo. That seems like a small price to pay for the resource savings of not running a puppetd process all the time.
More differences
As I've read the puppet lists, faqs and documentation, I've found yet more ways our use of puppet diverges from the norm.
The first is a little thing really - most people seem to store all their node configurations in a single nodes.pp file. I'm not sure why. Fortunately, puppet's include syntax allows globbing, so we've created a directory and gave each server it's own .pp file. This arrangement makes it much easier to parse the configuration with tools other than puppet (like, Q. How many servers do we have? A. ls wc -l).
Backup and Nagios monitoring without storeconfigs
More significantly - there are some things we can't do since we are not using storeconfigs. While many puppet users add a variable, like $nagios = true before including their sshd class (which then causes the sshd class to store a configuration for the nagios server to monitor ssh on the node in question), we were forced to come up with alternatives.
My first solution was to simply list all the servers that needed to be monitored in the server node configuration file for the nagios server. Ditto for the backup servers. This approach, however, proved cumbersome and error prone. When adding a new node, I now have to edit three files instead of one. And, how can I easily tell if all nodes have their nagios and/or backup configurations set? The solution was rather simple - there's more than one way to store a config for another node. Our nagios server is called jojobe.mayfirst.org and our backup server is luisa.mayfirst.org. A typical node.pp file looks like this:
node pietri.mayfirst.org
# node config goes here
if ( $fqdn == "jojobe.mayfirst.org )
nagios_montior "pietri":
if ( $fqdn == "luisa.mayfirst.org )
backup_access "pietri":
root@$server.mayfirst.org into an authorizeduserids file on the target server.
There's no question - the setup was rather tedious (we're using runit to maintain an ssh-agent for each root user), however, now that's in place (and configured via puppet), it's a breeze to add new servers. The only extra step we have to take is to confirm and sign each new server's keys. This "extra" step not only allows our servers to verify each other, but also allows our users to verify the servers as well, so it's hardly an extra step at all.
Shared modules
There's a vibrant community of third party module developers for puppet. Rather than figure out the intricacies of having puppet configure sshd, for example, you can install a contributed sshd module and then you simply add:
include sshd
Nowadays, many websites provide a mobile version, like
If you reduce the size of your window, you will get something like this:
Browse my 
Background:
Lars Wirzenius was 
A new version of the 
Lately I have been working on the triggers feature of dpkg. I would like to share my plan and what I have done so far. I ll first explain what triggers are, the current problems, and the work I did to try to improve the situation.
Introduction
Dpkg triggers are a neat feature of dpkg that package can use to send/receive notifications to/from other installed packages. Those notifications take the form a simple string.
This feature is heavily used to track changes of packaged files in a list of predefined directories, and to update other files based on this. For instance, man-db is watching the directories containing manual pages so that it can update its cache (in 
From a packager s perspective
The implementation of triggers has several consequences on the status that packages can have.
Let s assume that the package A installs a file in a directory that is watched by package B (and that B is currently in the installed state). When A is unpacked, dpkg adds B to its Triggers-Awaited field and lists the activated trigger in B s Triggers-Pending field. Package A is in unpacked state, but B has been changed to triggers-pending .
When A is configured, instead of going to the installed state, it will go to the triggers-awaited state. In that state the package is assumed to NOT fulfill dependencies. However, B which is still in triggers-pending state does fulfill dependencies.
A and B will switch to installed at the same time when the trigger has been processed.
The fact that the triggers-awaited status does not fulfill dependencies means that some common triggers like man-db have to be processed regularly just to be able to ensure dependencies are satisfied before running the postinst of other installed packages.
But a package which ships a manual page can certainly be considered as configured when its postinst has been run even if man-db has not yet updated its cache to know about the new/updated manual page.
When you activate a trigger with the dpkg-trigger command you have an option 
The 
The lab features four web servers that will be merged into a single
web service with the help of keepalived. Since, keepalived now
supports full IPv6 support, we will also use IPv6 in our lab.
Here is the 
The Debian kernel team regularly backports driver updates to the
Linux kernel in stable releases to add support for new hardware, and
I've prepared several updates intended for point release 6.0.2.
Since the kernel team does not have a large collection of hardware
on which to test driver changes, we would appreciate test reports
from users. It is important to test not just that new devices are
supported properly, but that there are no regressions in support
for older devices.
Changes
I have updated these drivers to the versions found in Linux
2.6.38, modulo driver API changes:
Here is the simplified IDL for these parts:
Let s implement this lab. You should grab the sources first:
The host name
On Unix systems, the host name is kept in memory by the kernel: it is set
and get by the 
This year Debconf is a bit 
P.S. I must say thank you Accenture for paying for my plane tickets.
Here is the configuration that I bought:
