Niels Thykier: Annotating the Debian packaging directory
In my previous blog post Providing online reference documentation for debputy,
I made a point about how debhelper documentation was suboptimal on account
of being static rather than online. The thing is that debhelper is not alone
in this problem space, even if it is a major contributor to the number of
packaging files you have to to know about.
If we look at the "competition" here such as Fedora and Arch Linux, they tend to only
have one packaging file. While most Debian people will tell you a long list of cons
about having one packaging file (such a Fedora's spec file being 3+ domain specific
languages "mashed" into one file), one major advantage is that there is only
"the one packaging file". You only need to remember where to find the documentation
for one file, which is great when you are running on wetware with limited storage
capacity.
Which means as a newbie, you can dedicate less mental resources to tracking multiple
files and how they interact and more effort understanding the "one file" at hand.
I started by asking myself how can we in Debian make the packaging stack more
accessible to newcomers? Spoiler alert, I dug myself into rabbit hole and ended up
somewhere else than where I thought I was going.
I started by wanting to scan the debian directory and annotate all files that I
could with documentation links. The logic was that if debputy could do that
for you, then you could spend more mental effort elsewhere. So I combined
debputy's packager provided files detection with a static list of files and
I quickly had a good starting point for debputy-based packages.
Adding (non-static) dpkg and debhelper files to the mix
Now, I could have closed the topic here and said "Look, I did debputy files
plus couple of super common files". But I decided to take it a bit further. I added
support for handling some dpkg files like packager provided files (such as
debian/substvars and debian/symbols). But even then, we all know that
debhelper is the big hurdle and a major part of the omission...
In another previous blog post (A new Debian package helper: debputy),
I made a point about how debputy could list all auxiliary files while
debhelper could not. This was exactly the kind of feature that I would need
for this feature, if this feature was to cover debhelper. Now, I also
remarked in that blog post that I was not willing to maintain such a list.
Also, I may have ranted about static documentation being unhelpful for
debhelper as it excludes third-party provided tooling.
Fortunately, a recent update to dh_assistant had provided some basic
plumbing for loading dh sequences. This meant that getting a list of
all relevant commands for a source package was a lot easier than it used
to be. Once you have a list of commands, it would be possible to check
all of them for dh's NOOP PROMISE hints. In these hints, a command
can assert it does nothing if a given pkgfile is not present. This
lead to the new dh_assistant list-guessed-dh-config-files command
that will list all declared pkgfiles and which helpers listed them.
With this combined feature set in place, debputy could call
dh_assistant to get a list of pkgfiles, pretend they were packager
provided files and annotate those along with manpage for the relevant
debhelper command. The exciting thing about letting debpputy
resolve the pkgfiles is that debputy will resolve "named" files
automatically (debhelper tools will only do so when --name is
passed), so it is much more likely to detect named pkgfiles
correctly too. Side note: I am going to ignore the elephant in the room
for now, which is dh_installsystemd and its package@.service files
and the wide-spread use of debian/foo.service where there is no
package called foo. For the latter case, the "proper" name would
be debian/pkg.foo.service.
With the new dh_assistant feature done and added to debputy,
debputy could now detect the ubiquitous debian/install file.
Excellent. But less great was that the very common debian/docs
file was not. Turns out that dh_installdocs cannot
be skipped by dh, so it cannot have NOOP PROMISE hints. Meh...
Well, dh_assistant could learn about a new INTROSPECTABLE marker
in addition to the NOOP PROMISE and then I could sprinkle that into
a few commands. Indeed that worked and meant that debian/postinst
(etc.) are now also detectable.
At this point, debputy would be able to identify a wide range of
debhelper related configuration files in debian/ and at least
associate each of them with one or more commands.
Nice, surely, this would be a good place to stop, right...?
Adding more metadata to the files
The debhelper detected files only had a command name and manpage
URI to that command. It would be nice if we could contextualize this
a bit more.
Like is this file installed into the package as is like debian/pam
or is it a file list to be processed like debian/install. To make
this distinction, I could add the most common debhelper file types
to my static list and then merge the result together.
Except, I do not want to maintain a full list in debputy.
Fortunately, debputy has a quite extensible plugin infrastructure,
so added a new plugin feature to provide this kind of detail and
now I can outsource the problem! I split my definitions into two and
placed the generic ones in the debputy-documentation plugin and
moved the debhelper related ones to debhelper-documentation.
Additionally, third-party dh addons could provide their own
debputy plugin to add context to their configuration files.
So, this gave birth file categories and configuration features, which
described each file on different fronts. As an example,
debian/gbp.conf could be tagged as a maint-config to signal
that it is not directly related to the package build but more of a
tool or style preference file. On the other hand, debian/install
and debian/debputy.manifest would both be tagged as a
pkg-helper-config. Files like debian/pam were
tagged as ppf-file for packager provided file and so on.
I mentioned configuration features above and those were added
because, I have had a beef with debhelper's "standard"
configuration file format as read by filearray and
filedoublearray. They are often considered simple to
understand, but it is hard to know how a tool will actually read
the file. As an example, consider the following:
So these "simple" files have way too many combinations of how they can be interpreted. I figured it would be helpful if debputy could highlight these difference, so I added support for those as well. Accordingly, debian/install is tagged with multiple tags including dh-executable-config and dh-glob-after-execute. Then, I added a datatable of these tags, so it would be easy for people to look up what they meant. Ok, this seems like a closed deal, right...?
- Will the debhelper use filearray, filedoublearray or none of them to read the file? This topic has about 2 bits of entropy.
- Will the config file be executed if it is marked executable assuming you are using the right compat level? If it is executable, does dh-exec allow renaming for this file? This topic adds 1 or 2 bit of entropy depending on the context.
- Will the config file be subject to glob expansions? This topic sounds like a boolean but is a complicated mess. The globs can be handled either by debhelper as it parses the file for you. In this case, the globs are applied to every token. However, this is not what dh_install does. Here the last token on each line is supposed to be a directory and therefore not subject to globs. Therefore, dh_install does the globbing itself afterwards but only on part of the tokens. So that is about 2 bits of entropy more. Actually, it gets worse...
- If the file is executed, debhelper will refuse to expand globs in the output of the command, which was a deliberate design choice by the original debhelper maintainer took when he introduced the feature in debhelper/8.9.12. Except, dh_install feature interacts with the design choice and does enable glob expansion in the tool output, because it does so manually after its filedoublearray call.
Context, context, context
However, the dh-executable-config tag among other are
only applicable in compat 9 or later. It does not seem
newbie friendly if you are told that this feature exist,
but then have to read in the extended description that
that it actually does not apply to your package.
This problem seems fixable. Thanks to dh_assistant, it is
easy to figure out which compat level the package is using.
Then tweak some metadata to enable per compat level rules.
With that tags like dh-executable-config only appears
for packages using compat 9 or later.
Also, debputy should be able to tell you where packager
provided files like debian/pam are installed. We already
have the logic for packager provided files that debputy
supports and I am already using debputy engine for
detecting the files. If only the plugin provided metadata gave
me the install pattern, debputy would be able tell you
where this file goes in the package. Indeed, a bit of tweaking
later and setting install-pattern to
usr/lib/pam.d/ name , debputy presented me with the
correct install-path with the package name placing the
name placeholder.
Now, I have been using debian/pam as an example, because
debian/pam is installed into usr/lib/pam.d in compat
14. But in earlier compat levels, it was installed into
etc/pam.d. Well, I already had an infrastructure for doing
compat file tags. Off we go to add install-pattern to the
complat level infrastructure and now changing the compat level
would change the path. Great. (Bug warning: The value is
off-by-one in the current version of debhelper. This is
fixed in git)
Also, while we are in this install-pattern business, a
number of debhelper config files causes files to be
installed into a fixed directory. Like debian/docs which
causes file to be installed into /usr/share/docs/ package .
Surely, we can expand that as well and provide that bit of
context too... and done. (Bug warning: The code currently does
not account for the main documentation package context)
It is rather common pattern for people to do debian/foo.in
files, because they want to custom generation of
debian/foo. Which means if you have debian/foo you get
"Oh, let me tell you about debian/foo ". Then you rename it
to debian/foo.in and the result is "debian/foo.in is a
total mystery to me!". That is suboptimal, so lets detect those
as well as if they were the original file but add a tag saying
that they are a generate template and which file we suspect it
generates.
Finally, if you use debputy, almost all of the standard
debhelper commands are removed from the sequence, since
debputy replaces them. It would be weird if these commands
still contributed configuration files when they are not actually
going to be invoked. This mostly happened naturally due to the
way the underlying dh_assistant command works. However,
any file mentioned by the debhelper-documentation plugin
would still appear unfortunately. So off I went to filter
the list of known configuration files against which dh_
commands that dh_assistant thought would be used for this
package.
Wrapping it up
I was several layers into this and had to dig myself out.
I have ended up with a lot of data and metadata. But it
was quite difficult for me to arrange the output in a user
friendly manner.
However, all this data did seem like it would be useful any
tool that wants to understand more about the package.
So to get out of the rabbit hole, I for now
wrapped all of this into JSON and now we have a
debputy tool-support annotate-debian-directory
command that might be useful for other tools.
To try it out, you can try the following demo:
In another day, I will figure out how to structure this output so it is
useful for non-machine consumers. Suggestions are welcome. :)
Limitations of the approach
As a closing remark, I should probably remind people that this feature relies
heavily on declarative features. These include:
You can help yourself and others to better results by using the declarative way rather than using debian/rules, which is the bane of all introspection!
- When determining which commands are relevant, using Build-Depends: dh-sequence-foo is much more reliable than configuring it via the Turing complete configuration we call debian/rules.
- When debhelper commands use NOOP promise hints, dh_assistant can "see" the config files listed those hints, meaning the file will at least be detected. For new introspectable hint and the debputy plugin, it is probably better to wait until the dust settles a bit before adding any of those.
After more than one a year, a new minor FAI version is available, but
it includes some interesting new features.
Here a the items from the NEWS file:
fai (6.2) unstable; urgency=low
This year was hard from a personal and work point of view, which impacted the amount of Free Software bits I ended up doing - even when I had the time I often wasn t in the right head space to make progress on things. However writing this annual recap up has been a useful exercise, as I achieved more than I realised. For previous years see 
Now copy the ISO image to the newly created instance and extract
its data:
Login the recovery console (root without password) and fix its default route to
make it reachable:
Being able to use this procedure for complete disaster recovery within Hetzner
cloud VPS (using off-site backups) gives me a better feeling, too.
Hello from a snowy Montr al! My life has been pretty busy lately
For documentation, please have a look at: