Niels Thykier: Providing online reference documentation for debputy
I do not think seasoned Debian contributors quite appreciate how much knowledge we
have picked up and internalized. As an example, when I need to look up
documentation for debhelper, I generally know which manpage to look in. I suspect
most long time contributors would be able to a similar thing (maybe down 2-3
manpages). But new contributors does not have the luxury of years of experience.
This problem is by no means unique to debhelper.
One thing that debhelper does very well, is that it is hard for users to tell
where a addon "starts" and debhelper "ends". It is clear you use addons, but
the transition in and out of third party provided tools is generally smooth. This
is a sign that things "just work(tm)".
Except when it comes to documentation. Here, debhelper's static documentation
does not include documentation for third party tooling. If you think from a
debhelper maintainer's perspective, this seems obvious. Embedding documentation
for all the third-party code would be very hard work, a layer-violation, etc.. But
from a user perspective, we should not have to care "who" provides "what". As as
user, I want to understand how this works and the more hoops I have to jump through
to get that understanding, the more frustrated I will be with the toolstack.
With this, I came to the conclusion that the best way to help users and solve the
problem of finding the documentation was to provide "online documentation". It
should be possible to ask debputy, "What attributes can I use in
install-man?" or "What does path-metadata do?". Additionally, the lookup
should work the same no matter if debputy provided the feature or some
third-party plugin did. In the future, perhaps also other types of documentation
such as tutorials or how-to guides.
Below, I have some tentative results of my work so far. There are some
improvements to be done. Notably, the commands for these documentation features
are still treated a "plugin" subcommand features and should probably have its own
top level "ask-me-anything" subcommand in the future.
Automatic discard rules
Since the introduction of install rules, debputy has included an automatic
filter mechanism that prunes out unwanted content. In 0.1.9, these filters
have been named "Automatic discard rules" and you can now ask debputy to
list them.
For these rules, the provider can both provide a description but also an example
of their usage.
The example is a live example. That is, the provider will provide debputy
with a scenario and the expected outcome of that scenario. Here is the concrete
code in debputy that registers this example:
When showing the example, debputy will validate the example matches what the plugin
provider intended. Lets say I was to introduce a bug in the code, so that the discard rule
no longer worked. Then debputy would start to show the following:
Obviously, it would be better if this validation could be added directly as a plugin test,
so the CI pipeline would catch it. That is one my personal TODO list. :)
One final remark about automatic discard rules before moving on. In 0.1.9, debputy will also
list any path automatically discarded by one of these rules in the build output to make sure
that the automatic discard rule feature is more discoverable.
$ debputy plugin list automatic-discard-rules
+-----------------------+-------------+
Name Provided By
+-----------------------+-------------+
python-cache-files debputy
la-files debputy
backup-files debputy
version-control-paths debputy
gnu-info-dir-file debputy
debian-dir debputy
doxygen-cruft-files debputy
+-----------------------+-------------+
$ debputy plugin show automatic-discard-rules la-files
Automatic Discard Rule: la-files
================================
Documentation: Discards any .la files beneath /usr/lib
Example
-------
/usr/lib/libfoo.la << Discarded (directly by the rule)
/usr/lib/libfoo.so.1.0.0
api.automatic_discard_rule(
"la-files",
_debputy_prune_la_files,
rule_reference_documentation="Discards any .la files beneath /usr/lib",
examples=automatic_discard_rule_example(
"usr/lib/libfoo.la",
("usr/lib/libfoo.so.1.0.0", False),
),
)
# Output if the code or example is broken
$ debputy plugin show automatic-discard-rules la-files
[...]
Automatic Discard Rule: la-files
================================
Documentation: Discards any .la files beneath /usr/lib
Example
-------
/usr/lib/libfoo.la !! INCONSISTENT (code: keep, example: discard)
/usr/lib/libfoo.so.1.0.0
debputy: warning: The example was inconsistent. Please file a bug against the plugin debputy
Plugable manifest rules like the install rule
In the manifest, there are several places where rules can be provided by plugins. To make
life easier for users, debputy can now since 0.1.8 list all provided rules:
(Output trimmed a bit for space reasons)
And you can then ask debputy to describe any of these rules:
(Output trimmed a bit for space reasons)
All the attributes and restrictions are auto-computed by debputy from information
provided by the plugin. The associated documentation for each attribute is supplied
by the plugin itself, The debputy API validates that all attributes are covered
and the documentation does not describe non-existing fields. This ensures that you as
a plugin provider never forget to document new attributes when you add them later.
The debputy API for manifest rules are not quite stable yet. So currently only
debputy provides rules here. However, it is my intention to lift that restriction
in the future.
I got the idea of supporting online validated examples when I was building this feature.
However, sadly, I have not gotten around to supporting it yet.
$ debputy plugin list plugable-manifest-rules
+-------------------------------+------------------------------+-------------+
Rule Name Rule Type Provided By
+-------------------------------+------------------------------+-------------+
install InstallRule debputy
install-docs InstallRule debputy
install-examples InstallRule debputy
install-doc InstallRule debputy
install-example InstallRule debputy
install-man InstallRule debputy
discard InstallRule debputy
move TransformationRule debputy
remove TransformationRule debputy
[...] [...] [...]
remove DpkgMaintscriptHelperCommand debputy
rename DpkgMaintscriptHelperCommand debputy
cross-compiling ManifestCondition debputy
can-execute-compiled-binaries ManifestCondition debputy
run-build-time-tests ManifestCondition debputy
[...] [...] [...]
+-------------------------------+------------------------------+-------------+
$ debputy plugin show plugable-manifest-rules install
Generic install ( install )
===========================
The generic install rule can be used to install arbitrary paths into packages
and is *similar* to how dh_install from debhelper works. It is a two "primary" uses.
1) The classic "install into directory" similar to the standard dh_install
2) The "install as" similar to dh-exec 's foo => bar feature.
Attributes:
- source (conditional): string
sources (conditional): List of string
A path match ( source ) or a list of path matches ( sources ) defining the
source path(s) to be installed. [...]
- dest-dir (optional): string
A path defining the destination *directory*. [...]
- into (optional): string or a list of string
A path defining the destination *directory*. [...]
- as (optional): string
A path defining the path to install the source as. [...]
- when (optional): manifest condition (string or mapping of string)
A condition as defined in [Conditional rules](https://salsa.debian.org/debian/debputy/-/blob/main/MANIFEST-FORMAT.md#Conditional rules).
This rule enforces the following restrictions:
- The rule must use exactly one of: source , sources
- The attribute as cannot be used with any of: dest-dir , sources
[...]
Manifest variables like PACKAGE
I also added a similar documentation feature for manifest variables such as
PACKAGE . When I implemented this, I realized listing all manifest variables
by default would probably be counter productive to new users. As an example, if you
list all variables by default it would include DEB_HOST_MULTIARCH (the most common
case) side-by-side with the the much less used DEB_BUILD_MULTIARCH and the even
lessor used DEB_TARGET_MULTIARCH variable. Having them side-by-side implies they
are of equal importance, which they are not. As an example, the ballpark number of
unique packages for which DEB_TARGET_MULTIARCH is useful can be counted on two
hands (and maybe two feet if you consider gcc-X distinct from gcc-Y).
This is one of the cases, where experience makes us blind. Many of us probably have the
"show me everything and I will find what I need" mentality. But that requires experience
to be able to pull that off - especially if all alternatives are presented as equals.
The cross-building terminology has proven to notoriously match poorly to people's
expectation.
Therefore, I took a deliberate choice to reduce the list of shown variables by default
and in the output explicitly list what filters were active. In the current version
of debputy (0.1.9), the listing of manifest-variables look something like this:
I will probably tweak the concrete listing in the future. Personally, I am considering to provide
short-hands variables for some of the DEB_HOST_* variables and then hide the DEB_HOST_*
group from the default view as well. Maybe something like ARCH and MULTIARCH, which
would default to their DEB_HOST_* counter part. This variable could then have extended
documentation that high lights DEB_HOST_<X> as its source and imply that there are special
cases for cross-building where you might need DEB_BUILD_<X> or DEB_TARGET_<X>.
Speaking of variable documentation, you can also lookup the documentation for a given
manifest variable:
This was my update on online reference documentation for debputy. I hope you found it
useful. :)
$ debputy plugin list manifest-variables
+----------------------------------+----------------------------------------+------+-------------+
Variable (use via: NAME ) Value Flag Provided by
+----------------------------------+----------------------------------------+------+-------------+
DEB_HOST_ARCH amd64 debputy
[... other DEB_HOST_* vars ...] [...] debputy
DEB_HOST_MULTIARCH x86_64-linux-gnu debputy
DEB_SOURCE debputy debputy
DEB_VERSION 0.1.8 debputy
DEB_VERSION_EPOCH_UPSTREAM 0.1.8 debputy
DEB_VERSION_UPSTREAM 0.1.8 debputy
DEB_VERSION_UPSTREAM_REVISION 0.1.8 debputy
PACKAGE <package-name> debputy
path:BASH_COMPLETION_DIR /usr/share/bash-completion/completions debputy
+----------------------------------+----------------------------------------+------+-------------+
+-----------------------+--------+-------------------------------------------------------+
Variable type Value Option
+-----------------------+--------+-------------------------------------------------------+
Token variables hidden --show-token-variables OR --show-all-variables
Special use variables hidden --show-special-case-variables OR --show-all-variables
+-----------------------+--------+-------------------------------------------------------+
$ debputy plugin show manifest-variables path:BASH_COMPLETION_DIR
Variable: path:BASH_COMPLETION_DIR
==================================
Documentation: Directory to install bash completions into
Resolved: /usr/share/bash-completion/completions
Plugin: debputy
Thanks
On a closing note, I would like to thanks Jochen Sprickerhof, Andres Salomon, Paul Gevers
for their recent contributions to debputy. Jochen and Paul provided a number of real
world cases where debputy would crash or not work, which have now been fixed. Andres
and Paul also provided corrections to the documentation.
Ever since Linux kernel 2.6.31 was released, my AMD Geode LX800-based FIC ION 603 can no longer boot; I get a fatal kernel crash near the end of the initramfs loading phase. At the crux of the issue seems to be a recently added kernel feature for caching filesystem ACL. I don't even use the ACL feature on my hosts, but there you go; it still crashes the kernel. Sigh.
Having filed a 
In an hour the online part of this combined embedded+fai meeting in Badajoz in Extremadura, Spain, will sadly be over, but the sadness will hopefully be compensated by good food (yes, the food at the last dinner was great) and more fun! To me it were three intensive days (plus some two half days of intense travelling) which were really productive. This post is a summary of what I did here, to document how useful these meetings are. I'm only one person out of 18, who did some Debian work here, much more stuff was done, most of it I probably didnt even notice, as 14 people where working on embedded stuff, which I mostly ignored... ;-) That said, I think it was still very nice to have this meeting together, a.) because I'm quite interested in embedded stuff and b.) because the embedded crowd is a fun one to hang around with! 
Another point release for Etch has been done; now it's the time for the CD
team to roll out new images after the next mirror pulse. The official
announcements (prepared by Alexander Reichle-Schmehl, thanks!) will follow
shortly afterwards. FTP master of the day was Joerg Jaspert, who
did his first point release since Woody, as he told us on IRC. We
appreciate your work and you spending your time that shortly before
going to Argentina.
This point release includes the etchnhalf update introducing a new kernel
image (based on 2.6.24) and some driver updates. Additionally the
infamous openssl hole will be fixed for good, even for new installs.
Again I want to present you a list of people who contributed to this
release. It cannot be complete as I got the information out of the
Welcome to this 2nd issue of the DeFuBu contest, the monthly championship of the funniest bug reported to the 
