The
Debian CI platform is comprised of 30+ (virtual)
machines. Maintaining this many machines, and being able to add new ones with
some degree of reliability requires one to use some sort of configuration
management.
Until about a week ago, we were using Chef for our
configuration
management. I was, for
several years, the main maintainer of Chef in Debian, so using it was natural
to me, as I had used it before for personal and work projects. But last year I
decided to
request the removal of Chef from
Debian, so that it won't be shipped with
Debian 11 (
bullseye).
After evaluating a few options, I believed that the path of least resistance
was to migrate to
itamae. itamae was inspired by
chef, and uses a DSL that is very similar to the Chef one. Even though the
itamae team claim it's not compatible with Chef, the changes that I needed to
do were relatively limited. The
necessary code
changes
might look like a lot, but a large part of them could be automated or done in
bulk, like doing simple search and replace operations, and moving entire
directories around.
In the rest of this post, I will describe the migration process, starting with
the infrastructure changes, the types of changes I needed to make to the
configuration management code, and my conclusions about the process.
Infrastructure changes
The first step was to add support for itamae to
chake, a configuration management wrapper
tool that I wrote. chake was originally written as a serverless remote executor
for Chef, so this involved a bit of a redesign. I thought it was worth it to
do, because at that point chake had gained several interesting managements
features that we no directly tied to Chef. This work was done a bit slowly over
the course of the several months, starting almost exactly one year ago, and was
completed 3 months ago. I wasn't in a hurry and knew I had time before Debian
11 is released and I had to upgrade the platform.
After this was done, I started the work of migrating the then Chef cookbooks to
itamae, and the next sections present the main types of changes that were
necessary.
During the entire process, I sent a few patches out:
Code changes
These are the main types of changes that were necessary in the configuration
code to accomplish the migration to itamae.
Replace
cookbook_file with
remote_file.
The resource known as
cookbook_file in Chef is called
remote_file in itamae.
Fixing this is just a matter of search and replace, e.g.:
-cookbook_file '/etc/apt/apt.conf.d/00updates' do
+remote_file '/etc/apt/apt.conf.d/00updates' do
Changed file locations
The file structure assumed by itamae is a lot simpler than the one in Chef. The
needed changes were:
Explicit file ownership and mode
Chef is usually design to run as root on the nodes, and files created are owned
by root and have move
0644 by default. With itamae, files are by default
owned by the user that was used to SSH into the machine. Because of this, I had
to review all file creation resources and add owner, group and mode explicitly:
-cookbook_file '/etc/apt/apt.conf.d/00updates' do
- source 'apt.conf'
+remote_file '/etc/apt/apt.conf.d/00updates' do
+ source 'files/apt.conf'
+ owner 'root'
+ group 'root'
+ mode "0644"
end
In the end, I guess being explicit make the configuration code more
understandable, so I take that as a win.
Different execution context
One of the major differences between Chef itamae comes down the execution
context of the recipes. In both Chef and itamae, the configuration is written
in DSL embedded in Ruby. This means that the recipes are just Ruby code, and
difference here has to do with where that code is executed. With Chef, the
recipes are always execute on the machine you are configuring, while with
itamae the recipe is executed on the workstation where you run itamae, and that
gets translated to commands that need to be executed on the machine being
configured.
For example, if you need to configure a service based on how much RAM the
machine has, with Chef you could do something like this:
total_ram = File.readlines("/proc/meminfo").find do l
l.split.first == "MemTotal:"
end.split[1]
file "/etc/service.conf" do
# use 20% of the total RAM
content "cache_size = # ram / 5 KB"
end
With itamae, all that Ruby code will run on the client, so
total_ram will
contain the wrong number. In the Debian CI case, I worked around that by
explicitly declaring the amount of RAM in the static host configuration, and
the above construct ended up as something like this:
file "/etc/service.conf" do
# use 20% of the total RAM
content "cache_size = # node['total_ram'] / 5 KB"
end
Lessons learned
This migration is now complete, and there are a few points that I take away
from it:
- The migration is definitely viable, and I'm glad I picked itamae after all.
- Of course, itamae is way simpler than Chef, and has less features. On the
other hand, this means that it a simple package to maintain, with less
dependencies and keeping it up to date is a lot easier.
- itamae is considerably slower than Chef. On my local tests, a noop
execution (e.g. re-applying the configuration a second time) against local
VMs with itamae takes 3x the time it takes with Chef.
All in all, the system is working just fine, and I consider this to have been a
successful migration. I'm happy it worked out.
TLDR: Freexian s success means that we have resources to invest into Debian projects. Plainly offering money has not worked so far, so I am looking to hire a project manager whose work would be to help spend that money in useful ways. At the same time, Freexian needs to adapt to cope with the growth: with new employees, with new infrastructure and a new offering. I want to give an idea of where we are headed, to try to inspire persons that share our values and our desire to improve Debian. Read on if you are interested.
Note: The original text has been split into 4 blog posts that will be published over a few days.
Introduction
Kubernetes is 
Last weekend Debian Quebec 
We fixed a couple of 
About three months after the 
Too much has been written, a war of support letters is going on, Debian is heading head first like Lemmings into the same war. And then, there is this by the first female President of the American Civil Liberties Union (ACLU):
A long time, no blog entries. What's prompted a new missive?
Two things:
Many quickly reacted to the return of rms to the FSF and asked that he
leaves again; some also asked for the whole board of directors to resign and
some not. Meanwhile, Debian discusses a general resolution on that
matter. Maybe it was not the original intent, but in practice the object of
the GR is about FSF's board of directors. Perhaps we will have the result
after rms resigns? Like many GRs, it will divide Debian and leave scars, at
least a tally sheet of who voted what, and who voted like whom.
I think that most other organisations did not go through a process as
plenary and collegial, but also heavy and cleaving, to decide which way to
go.
What if our two DPL candidates would issue a statement that, if elected,
they would refuse to fund events linked to the FSF until rms quits again
(and also the directors, if that is what the DPL candidate wishes to
propose). This would let Debian be part of the wave of reaction on time, and
maybe allow us to cancel this GR?