Johannes Schauer: botch updates
My last update about ongoing development of botch, the bootstrap/build ordering
tool chain, was three months ago with the announcement of
bootstrap.debian.net. Since then a number of things happened, so I thought
an update was due.
New graphs for port metrics
By default, a dependency graph is created by arbitrarily choosing an
installation set for binary package installation or source package compilation.
Installation set vertices and source vertices are connected according to this
arbitrary selection.
Niels Thykier approached me at Debconf13 about the possibility of using this
graph to create a metric which would be able to tell for each source package,
how many other source packages would become uncompilable or how many binary
packages would become uninstallable, if that source package was removed from
the archive. This could help deciding about the importance of packages. More
about this can be found at the thread on debian-devel.
For botch, this meant that two new graph graphs can now be generated. Instead
of picking an arbitrary installation set for compiling a source package or
installing a binary package, botch can now create a minimum graph which is
created by letting dose3 calculate strong dependencies and a maximum graph by
using the dependency closure.
Build profile syntax in dpkg
With dpkg 1.17.2 we now have experimental build profile support in unstable.
The syntax which ended up being added was:
Build-Depends: large (>= 1.0), small <!profile.stage1>
But until packages with that syntax can hit the archive, a few more tools need
to understand the syntax. The patch we have for sbuild is very simple
because sbuild relies on libdpkg for dependency parsing.
We have a patch for apt too, but we have to rebase it for the current apt
version and have to adapt it so that it works exactly like the functionality
dpkg implements. But before we can do that we have to decide how to handle
mixed positive and negative qualifiers or whether to remove this feature
altogether because it causes too much confusion. The relevant thread on
debian-dpkg starts here.
Update to latest dose3 git
Botch heavily depends on libdose3 and unfortunately requires features which are
only available in the current git HEAD. The latest version packaged in Debian
is 3.1.3 from October 2012. Unfortunately the current dose3 git HEAD also
relies on unreleased features from libcudf. On top of that, the GraphML output
of the latest ocamlgraph version (1.8.3) is also broken and only fixed in the
git. For now everything is set up as git submodules but this is the major
blocker preventing any packaging of botch as a Debian package. Hopefully new
releases will be done soon for all involved components.
Writing and reading GraphML
Botch is a collection of several utilities which are connected together in a
shell script. The advantage of this is, that one does not need to understand or
hack the OCaml code to use botch for different purposes. In theory it also
allows to insert 3rd party tools into a pipe to further modify the data. Until
recently this ability was seriously hampered by the fact that many botch tools
communicated with each other through marshaled OCaml binary files which prevent
everything which is not written in OCaml from modifying them. The data that was
passed around like this were the dependency graphs and I initially implemented
it like that because I didnt want to write a GraphML parser.
I now ended up writing an xmlm based GraphML parser so as of now, botch
only reads and writes ASCII text files in XML (for the graphs) and in rfc822
packages format (for Packages and Sources files) which can both easily be
modified by 3rd party tools. The ./tools directory contains many Python
scripts using the networkx module to modify GraphML and the apt_pkg module to
modify rfc822 files.
Splitting of tools
To further increase the ability to modify program execution without having to
know OCaml, I split up some big tools into multiple smaller ones. Some of the
smaller tools are now even written in Python which is probably much more
hackable for the general crowd. I converted those tools to Python which did not
need any dose3 functionality and which were simple enough so that writing them
didnt take much time. I could convert more tools but that might introduce bugs
and takes time which I currently dont have much of (who does?).
Gzip instead of bz2
Since around January 14, snapshot.debian.org doesnt offer bzip2 compressed
Packages and Sources files anymore but uses xz instead. This is awesome for
must purposes but unfortunately I had to discover that there exist no OCaml
bindings for libxz. Thus, botch is now using gzip instead of bz2 until either myself
or anybody else finds some time to write a libxz OCaml binding.
Self hosting Fedora
Paul Wise made me aware of Harald Hoyer's attempts to bootstrap Fedora. I
reproduced his steps for the Debian dependency graph and it turns out that they
are a little bit bigger. I'm exchanging emails with Harald
Hoyer because it might not be too hard to use botch for rpm based distributions
as well because dose3 supports rpm.
The article also made me aware of the tred tool which is part of graphviz and
allows to calculate the transitive reduction of a graph. This can help
making horrible situations much better.
Dose3 bugs
I planned to generate such simplified graphs for the neighborhood of each
source package on bootstrap.debian.net but then binutils stopped building
binutils-gold and instead provided binutils-gold while libc6-dev breaks
binutils-gold (<< 2.20.1-11). This unfortunately triggered a dose3 bug
and thus bootstrap.debian.net will not generate any new results until this is
fixed in dose3.
Another dose3 bug affects packages which Conflicts/Replaces/Provides:bar
while bar is fully virtual and are Multi-Arch:same. Binaries of different
architecture with this property can currently not be co-installed with dose3.
Unfortunately linux-libc-dev has this property and thus botch cannot be used to
analyze cross builds until that bug is fixed in dose3.
I hope I get some free time soon to be able to look at these dose3 issues
myself.
More documentation
Since I started to like the current set of tools and how they work together I
ended up writing over 2600 words of documentation in the past few days.
You can start setting up and running botch by reading the first steps and
get more detailed information by reading about the 28 tools that botch
makes use of as of now.
All existing articles, thesis and talks are linked from the wiki home.
Build-Depends: large (>= 1.0), small <!profile.stage1>
But until packages with that syntax can hit the archive, a few more tools need
to understand the syntax. The patch we have for sbuild is very simple
because sbuild relies on libdpkg for dependency parsing.
We have a patch for apt too, but we have to rebase it for the current apt
version and have to adapt it so that it works exactly like the functionality
dpkg implements. But before we can do that we have to decide how to handle
mixed positive and negative qualifiers or whether to remove this feature
altogether because it causes too much confusion. The relevant thread on
debian-dpkg starts here.
Update to latest dose3 git
Botch heavily depends on libdose3 and unfortunately requires features which are
only available in the current git HEAD. The latest version packaged in Debian
is 3.1.3 from October 2012. Unfortunately the current dose3 git HEAD also
relies on unreleased features from libcudf. On top of that, the GraphML output
of the latest ocamlgraph version (1.8.3) is also broken and only fixed in the
git. For now everything is set up as git submodules but this is the major
blocker preventing any packaging of botch as a Debian package. Hopefully new
releases will be done soon for all involved components.
Writing and reading GraphML
Botch is a collection of several utilities which are connected together in a
shell script. The advantage of this is, that one does not need to understand or
hack the OCaml code to use botch for different purposes. In theory it also
allows to insert 3rd party tools into a pipe to further modify the data. Until
recently this ability was seriously hampered by the fact that many botch tools
communicated with each other through marshaled OCaml binary files which prevent
everything which is not written in OCaml from modifying them. The data that was
passed around like this were the dependency graphs and I initially implemented
it like that because I didnt want to write a GraphML parser.
I now ended up writing an xmlm based GraphML parser so as of now, botch
only reads and writes ASCII text files in XML (for the graphs) and in rfc822
packages format (for Packages and Sources files) which can both easily be
modified by 3rd party tools. The ./tools directory contains many Python
scripts using the networkx module to modify GraphML and the apt_pkg module to
modify rfc822 files.
Splitting of tools
To further increase the ability to modify program execution without having to
know OCaml, I split up some big tools into multiple smaller ones. Some of the
smaller tools are now even written in Python which is probably much more
hackable for the general crowd. I converted those tools to Python which did not
need any dose3 functionality and which were simple enough so that writing them
didnt take much time. I could convert more tools but that might introduce bugs
and takes time which I currently dont have much of (who does?).
Gzip instead of bz2
Since around January 14, snapshot.debian.org doesnt offer bzip2 compressed
Packages and Sources files anymore but uses xz instead. This is awesome for
must purposes but unfortunately I had to discover that there exist no OCaml
bindings for libxz. Thus, botch is now using gzip instead of bz2 until either myself
or anybody else finds some time to write a libxz OCaml binding.
Self hosting Fedora
Paul Wise made me aware of Harald Hoyer's attempts to bootstrap Fedora. I
reproduced his steps for the Debian dependency graph and it turns out that they
are a little bit bigger. I'm exchanging emails with Harald
Hoyer because it might not be too hard to use botch for rpm based distributions
as well because dose3 supports rpm.
The article also made me aware of the tred tool which is part of graphviz and
allows to calculate the transitive reduction of a graph. This can help
making horrible situations much better.
Dose3 bugs
I planned to generate such simplified graphs for the neighborhood of each
source package on bootstrap.debian.net but then binutils stopped building
binutils-gold and instead provided binutils-gold while libc6-dev breaks
binutils-gold (<< 2.20.1-11). This unfortunately triggered a dose3 bug
and thus bootstrap.debian.net will not generate any new results until this is
fixed in dose3.
Another dose3 bug affects packages which Conflicts/Replaces/Provides:bar
while bar is fully virtual and are Multi-Arch:same. Binaries of different
architecture with this property can currently not be co-installed with dose3.
Unfortunately linux-libc-dev has this property and thus botch cannot be used to
analyze cross builds until that bug is fixed in dose3.
I hope I get some free time soon to be able to look at these dose3 issues
myself.
More documentation
Since I started to like the current set of tools and how they work together I
ended up writing over 2600 words of documentation in the past few days.
You can start setting up and running botch by reading the first steps and
get more detailed information by reading about the 28 tools that botch
makes use of as of now.
All existing articles, thesis and talks are linked from the wiki home.
./tools directory contains many Python
scripts using the networkx module to modify GraphML and the apt_pkg module to
modify rfc822 files.
I was bracing myself for a rough night, but Zoe slept through until almost 6am.
Today was playgroup again, so we just took a slow start to the day and headed
out on the bike a bit after 9am.
There was a mix of new faces and people from last week. I packed a lunchbox
this time instead of relying on the snack collection in my "Dad Bag" backpack.
We worked on trying to approach other girls to get them to play.
Unfortunately they're all that little bit younger and more shy, so we haven't had
a huge success. Zoe's been pretty happy just playing with the various toys and
playing with me. Parachute time was a big hit again.
I think there were about 25 adults there (I was the only Dad this week), with
many with multiple children. The youngest baby I saw was a 7 week old.
Zoe had been asking me what a honeydew was, so after Playgroup we dropped by
the
I'm working in a very small web agency with 4 employees, one of them part time and our boss who doesn't do programming. It shouldn't come as a surprise, that our development infrastructure is not perfect. We have many ideas and dreams how we could improve it, but not the time.
Now that
Back in 2006, the compiler in Ubuntu was
Here s today s snapshot of Debian s unstable archive for the past year (at the start of May you can see the archive unfreezing after the Wheezy release; the gaps were my analysis tool failing):
Ubuntu s lines are relatively flat because everything that can be built with hardening already is. Debian s graph is on a slow upward trend as more packages get migrated to dh to gain knowledge of the global flags.
Each line in the graphs represents the count of source packages that contain binary packages that have at least 1 hit for a given category. ELF is just that: a source package that ultimately produces at least 1 binary package with at least 1 ELF binary in it (i.e. produces a compiled output). The Read-only Relocations ( relro ) hardening feature is almost always done for an ELF, excepting uncommon situations. As a result, the count of ELF and relro are close on Ubuntu. In fact, examining relro is a good indication of whether or not a source package got built with hardening of any kind. So, in Ubuntu, 91.5% of the archive is built with hardening, with Debian at 55.2%.
The stack protector and fortify source features depend on characteristics of the source itself, and may not always be present in package s binaries even when hardening is enabled for the build (e.g. no functions got selected for stack protection, or no fortified glibc functions were used). Really these lines mostly indicate the count of packages that have a sufficiently high level of complexity that would trigger such protections.
The PIE and immediate binding ( bind_now ) features are specifically enabled by a package maintainer. PIE can have a noticeable performance impact on CPU-register-starved architectures like i386 (ia32), so it is neither patched on in Ubuntu, nor part of the default flags in Debian. (And bind_now doesn t make much sense without PIE, so they usually go together.) It s worth noting, however, that it probably should be the default on amd64 (x86_64), which has plenty of available registers.
Here is a comparison of default installed packages between the most recent stable releases of Ubuntu (

Couple of weeks ago I was in the situation of having to setup up a new laptop. I decided to go with wheezy s DVD installer. So far so good. I didn t expect (somewhere in the lies-I-tell-myself dept.) to have GNOME as the default Debian desktop. However after install I ve figured it was the new GNOME3 everybody was talking about. Before that I ve seen it in Ubuntu systems used by my classmates. I thought yeah, OK GNOME3, I think that s fine for them as a Linux desktop. Turns out that I ve started using the Debian desktop aka GNOME3 and noticed that it was not as bloated as the Ubuntu desktop I ve seen before looked, so I sticked with it, (I thought for a while).
Turns out that I did like this new so called GNOME3, the non-window based but an application-based system (that is something that sticks in my head). I liked the way it makes sense as a desktop system, like when you look for applications, documents, connect to networks, use pluggable devices or just configure stuff every time with less and less effort. Good practices and concepts learned from Mac OS X-like environments and for sure taking advantage of the new features the Linux kernel and user-space environment got over the years. So, like one month later I stick with it and it makes sense for me to keep it. I had no chance to try the latest XFCE or KDE, my default choices before this experience. Kudos GNOME team, even after the depictions you guys had on GNOME Shell; as I learned.
This whole situation got me into some pondering about the past of the Linux user experience and how we in the community lead people into joining. I remember that when I guy asked: how do I configure this new monitor/VGA card/network card/Etc? the answer was in the lines of: what is the exact chipset model and specific whole product code number that your device has? Putting myself in the shoes of such people or today s people I d say: what are you talking about? what it is a chipset? I mean, like it was too technical that only one guy with more than average knowledge could grasp. From a product perspective this is similar for a car manufacturer tell to a customer to look for the exact layout or design your car s engine has, so that they are able to tell whether is the 82 s model A or 83 s model C. Simplicity on naming and identification was not in the mindset of most of us.
This is funny because as technology advances it also becomes more transparent to the customer. So, for instance, today s kids can become really power users of any new technology as if they had, indeed, many pre-set chipsets in their brain. But when going into the details we had to grasp few years ago they have some hard time figuring out the complexity of the product that presents itself on this clean and simple interface. New times, interesting times. Always good to look back. No, I m not getting old.
Friday was my last day
at
Yesterday, the second milestone of the Icehouse development branch of
Ceilometer has been released and is now available for testing and download.
This means the first half of the OpenStack Icehouse development has
passed!
New features
For the