Louis-Philippe V ronneau: puppetserver 6: a Debian packaging post-mortem
I have been a Puppet user for a couple of years now, first at work, and
eventually for my personal servers and computers. Although it can have a steep
learning curve, I find Puppet both nimble and very powerful. I also prefer it
to Ansible for its speed and the agent-server model it uses.
Sadly, Puppet Labs hasn't been the most supportive upstream and tends to move
pretty fast. Major versions rarely last for a whole Debian Stable release and
the upstream
.deb
packages are full of vendored libraries.1
Since 2017, Apollon Oikonomopoulos has been the one doing most of the work on
Puppet in Debian. Sadly, he's had less time for that lately and with Puppet 5
being deprecated in January 2021, Thomas Goirand, Utkarsh Gupta and I have been
trying to package Puppet 6 in Debian for the last 6 months.
With Puppet 6, the old ruby Puppet server using Passenger is not supported
anymore and has been replaced by puppetserver
, written in Clojure and running
on the JVM. That's quite a large change and although puppetserver
does reuse
some of the Clojure libraries puppetdb
(already in Debian) uses, packaging it
meant quite a lot of work.
Work in the Clojure team
As part of my efforts to package puppetserver
, I had the pleasure to join the
Clojure team and learn a lot about the Clojure ecosystem.
As I mentioned earlier, a lot of the Clojure dependencies needed for
puppetserver
were already in the archive. Unfortunately, when Apollon
Oikonomopoulos packaged them, the leiningen
build tool hadn't been packaged
yet. This meant I had to rebuild a lot of packages, on top of packaging some
new ones.
Since then, thanks to the efforts of Elana Hashman, leiningen
has been
packaged and lets us run the upstream testsuites and create .jar
artifacts
closer to those upstream releases.
During my work on puppetserver
, I worked on the following packages:
List of packages
If you want to learn more about packaging Clojure libraries and applications,
I rewrote the Debian Clojure packaging tutorial and added a section
about the quirks of using backport9
bidi-clojure
clj-digest-clojure
clj-helper
clj-time-clojure
clj-yaml-clojure
cljx-clojure
core-async-clojure
core-cache-clojure
core-match-clojure
cpath-clojure
crypto-equality-clojure
crypto-random-clojure
data-csv-clojure
data-json-clojure
data-priority-map-clojure
java-classpath-clojure
jnr-constants
jnr-enxio
jruby
jruby-utils-clojure
kitchensink-clojure
lazymap-clojure
liberator-clojure
ordered-clojure
pathetic-clojure
potemkin-clojure
prismatic-plumbing-clojure
prismatic-schema-clojure
puppetlabs-http-client-clojure
puppetlabs-i18n-clojure
puppetlabs-ring-middleware-clojure
puppetserver
raynes-fs-clojure
riddley-clojure
ring-basic-authentication-clojure
ring-clojure
ring-codec-clojure
shell-utils-clojure
ssl-utils-clojure
test-check-clojure
tools-analyzer-clojure
tools-analyzer-jvm-clojure
tools-cli-clojure
tools-reader-clojure
trapperkeeper-authorization-clojure
trapperkeeper-clojure
trapperkeeper-filesystem-watcher-clojure
trapperkeeper-metrics-clojure
trapperkeeper-scheduler-clojure
trapperkeeper-webserver-jetty9-clojure
url-clojure
useful-clojure
watchtower-clojure
leiningen
without a dedicated dh_lein
tool.
Work left to get puppetserver 6 in the archive
Unfortunately, I was not able to finish the puppetserver
6 packaging work.
It is thus unlikely it will make it in Debian Bullseye. If the issues described
below are fixed, it would be possible to to package puppetserver
in
bullseye-backports
though.
So what's left?
jruby
Although I tried my best (kudos to Utkarsh Gupta and Thomas Goirand for the
help), jruby
in Debian is still broken. It does build properly, but the
testsuite fails with multiple errors:
ruby-psych
is broken (#959571)- there are some random java failures on a few tests (no clue why)
- tests ran by
raklelib/rspec.rake
fail to run, maybe because the--pattern
command line option isn't compatible with our version ofrake
? Utkarsh seemed to know why this happens.
jruby
testsuite failures aside, I have not been able to use the jruby.deb
the
package currently builds in jruby-utils-clojure
(testsuite failure). I had the
same exact failure with the (more broken) jruby
version that is currently in
the archive, which leads me to think this is a LOAD_PATH
issue in
jruby-utils-clojure
. More on that below.
To try to bypass these issues, I tried to vendor jruby
into
jruby-utils-clojure
. At first I understood vendoring meant including
upstream pre-built artifacts (jruby-complete.jar
) and shipping them directly.
After talking with people on the #debian-mentors
and #debian-ftp
IRC
channels, I now understand why this isn't a good idea (and why it's not
permitted in Debian). Many thanks to the people who were patient and kind enough
to discuss this with me and give me alternatives.
As far as I now understand it, vendoring in Debian means "to have an embedded
copy of the source code in another package". Code shipped that way still needs
to be built from source. This means we need to build jruby
ourselves, one way
or another. Vendoring jruby
in another package thus isn't terribly helpful.
If fixing jruby
the proper way isn't possible, I would suggest trying to
build the package using embedded code copies of the external libraries jruby
needs to build, instead of trying to use the Debian libraries.2 This
should make it easier to replicate what upstream does and to have a final
.jar
that can be used.
jruby-utils-clojure
This package is a first-level dependency for puppetserver
and is the glue
between jruby
and puppetserver
.
It builds fine, but the testsuite fails when using the Debian jruby
package. I
think the problem is caused by a jruby
LOAD_PATH
issue.
The Debian jruby
package plays with the LOAD_PATH
a little to try use
Debian packages instead of downloading gems from the web, as upstream jruby
does. This seems to clash with the gem-home
, gem-path
, and
jruby-load-path
variables in the jruby-utils-clojure
package. The testsuite
plays around with these variables and some Ruby libraries can't be found.
I tried to fix this, but failed. Using the upstream jruby-complete.jar
instead
of the Debian jruby
package, the testsuite passes fine.
This package could clearly be uploaded to NEW right now by ignoring the
testsuite failures (we're just packaging static .clj
source files in the
proper location in a .jar
).
puppetserver
jruby
issues aside, packaging puppetserver itself is 80% done. Using the
upstream jruby-complete.jar
artifact, the testsuite fails with a weird
Clojure error I'm not sure I understand, but I haven't debugged it for very
long.
Upstream uses git submodules to vendor puppet (agent), hiera (3), facter and
puppet-resource-api for the testsuite to run properly. I haven't touched that,
but I believe we can either:
- link to the Debian packages
- fix the Debian packages if they don't include the right files (maybe in a new binary package that just ships part of the source code?)
puppetserver
really isn't far away from making it in the archive.
Very talented Debian Developers are always eager to work on these issues and
can be contracted for very reasonable rates. If you're interested in
contracting someone to help iron out the last issues, don't hesitate to reach
out via one of the following:
- the debian-puppet mailing list
- the debian-jobs mailing list
- the Debian Consultants index
- fossjobs.net
- For example, the upstream package for the Puppet Agent vendors OpenSSL.
-
One of the problems of using Ruby libraries already packaged in
Debian is that
jruby
currently only supports Ruby 2.5. Ruby libraries in Debian are currently expected to work with Ruby 2.7, with the transition to Ruby 3.0 planned after the Bullseye release. -
If you run Puppet, you clearly should care: the
.deb
packages upstream publishes really aren't great and I would not recommend using them.