Midsummer for me is also known as
Noite do Lume Novo (literally New Fire Night ), one of the big calendar
events of the year, marking the end of the school year and the beginning of
summer.
On this day, there are
celebrations
not very unlike the bonfires in the
Guy Fawkes
Night in England or Britain
[1]. It is a bit different in that
it is not a single event for the masses, more of a friends and neighbours thing,
and that it lasts for a big chunk of the night (sometimes until morning).
Perhaps for some people, or outside bigger towns or cities, Guy Fawkes Night is
also celebrated in that way and that's why during the first days of November
there are fireworks rocketing and cracking in the neighbourhoods all around.
Like many other
celebrations around the world involving
bonfires, many of
them also happening around the summer solstice, it is supposed to be a time of
renewal of cycles, purification and keeping the evil spirits away; with rituals
to that effect like jumping over the fire when the flames are not high and it
is safe enough.
So it was fitting that, in the middle of June (almost Midsummer in the northern
hemisphere), I learnt that I was about to leave my now-previous job, which is a
pretty big signal and precursor for renewal (and it might have something to
do with purifying and keeping the evil away as well
;-) ).
Whatever... But what does all of this have to do with
aptitude or Debian, anyway?
For one, it was a question of timing.
While looking for a new job (and I am still at it), I had more spare time than
usual.
DebConf 15 @ Heidelberg was within
sight, and for the first time circumstances allowed me to attend this event.
It also coincided with the time when I re-gained access to commit to
aptitude on the 19th of June.
Which means Renewal.
End of June was also the time of the announcement of the colossal
GCC-5/C++11
ABI transition in
Debian, that was scheduled to start on the 1st of August, just before the
DebConf. Between 2 and 3 thousand source packages in Debian were affected by
this transition, which a few months later is not yet finished (although the most
important parts were completed by mid-end September).
aptitude itself is written in C++, and depends on several libraries written in
C++, like
Boost,
Xapian and
SigC++. All of them had to be compiled with
the new C++11 ABI of GCC-5, in unison and in a particular order, for
aptitude
to continue to work (and for minimal breakage).
aptitude and some
dependencies did not even compile straight away, so this transition meant that
aptitude needed attention just to keep working.
Having recently being awarded again with the Aptitude Hat, attending DebConf for
the first time and sailing towards the Transition Maelstrom, it was a clear sign
that Something Had to Be Done (to avoid the sideways looks and consequent shame
at DebConf, if nothing else).
Happily (or a bit
unhappily for me, but let's pretend...), with the unexpected
free time in my hands, I changed the plans that I had before re-gaining the
Aptitude Hat (some of them involving Debian, but in other ways maybe I will
post about that soon).
In July I worked to fix the problems before the transition started, so
aptitude would be (mostly) ready, or in the worst case broken only for a few
days, while the chain of dependencies was rebuilt. But apart from the changes
needed for the new
GCC-5, it was decided at the last minute that
Boost 1.55
would not be rebuilt with the new ABI, and that the only version with the new
ABI would be 1.58 (which caused further breakage in
aptitude, was added to
experimental only a few days before, and was moved to
unstable after the
transition had started). Later, in the first days of the transition,
aptitude
was affected for a few days by breakage in the dependencies, due to not being
compiled in sequence according to the transition levels (so with a mix of old
and new ABI).
With the critical intervention of
Axel
Beckert (abe / XTaran), things were not so
bad as they could have been. He was busy testing and uploading in the critical
days when I was enjoying a small holiday on my way to DebConf, with minimal
internet access and communicating almost exclusively with him; and he promptly
tended the complaints arriving in the Bug Tracking System and asked for rebuilds
of the dependencies with the new ABI. He also brought the packaging up to
shape, which had decayed a bit in the last few years.
Gruesome Challenges
But not all was solved yet, more storms were brewing and started to appear in
the horizon, in the form of clouds of fire coming from
nearby
realms.
The APT Deities, which had long ago spilled out their
secret, inner
challenge
(just the initial paragraphs), were relentless. Moreover, they were present at
Heidelberg in full force, in or close to their home grounds, and they were
Marching Decidedly towards Victory:
In the talk @ DebConf
This APT has Super Cow
Powers
(video available), by David Kalnischkies, they told us about the niceties of
apt 1.1 (still in
experimental but hopefully coming to
unstable soon), and
they boasted about getting the lead in our
arms race (should I say
bugs
race?) by a few open bug reports.
This act of provocation further escalated the tensions. The fierce competition
which had been going on for some time gained new heights. So much so that APT
Deities and our team had to sit together in the outdoor areas of the venue and
have many a
weissbier together, while discussing and fixing bugs.
But beneath the calm on the surface, and while pretending to keep good
diplomatic relations, I knew that Something Had to Be Done, again. So I could
only do one thing jump over the bonfire and Keep the Evil away, be that
Keep Evil bugs Away or Keep Evil APT Deities Away from winning the challenge, or
both.
After returning from DebConf I continued to dedicate time to the project, more
than a full time job in some weeks, and this is what happened in the last few
months, summarised in another graph, showing the evolution of the BTS for
aptitude:
The numbers for
apt right now (15th November 2015) are:
- 629 open (731 if counting all merged bugs independently)
- 0 Release Critical
- 275 (318 unmerged) with severity Important or Normal
- 354 (413 unmerged) with severity Minor or Wishlist
- 0 marked as Forwarded or Pending
The numbers for
aptitude right now are:
- 488 (573 if counting all merged bugs independently)
- 1 Release Critical (but it is an artificial bug to keep it from migrating to
testing)
- 197 (239 unmerged) with severity Important or Normal
- 271 (313 unmerged) with severity Minor or Wishlist
- 19 (20 unmerged) marked as Forwarded or Pending
The Aftermath
As we can see, for the time being I could keep the Evil at bay, both in terms of
bugs themselves and re-gaining the lead in the
bugs race the Evil APT
Deities were thwarted again in their efforts.
... More seriously, as most of you suspected, the graph above is not the whole
truth, so I don't want to boast
too much. A big part of the reduction in the
number of bugs is because of merging duplicates, closing obsolete bugs, applying
translations coming from multiple contributors, or simple fixes like typos and
useful suggestions needing minor changes. Many of remaining problems are
comparatively more difficult or time consuming that the ones addressed so far
(except perhaps avoiding the immediate breakage of the transition, that took
weeks to solve), and there are many important problems still there, chief among
those is
aptitude offering
very poor solutions to resolve conflicts.
Still, even the simplest of the changes takes effort, and triaging hundreds of
bugs is not fun at all and mostly a thankless effort althought there is the
occasionally kind soul that thanks you for handling a decade-old bug.
If being subjected to the rigours of the BTS and reading and solving hundreds of
bug reports is not Purification, I don't know what it is.
Apart from the triaging, there were 118 bugs closed (or pending) due to
changes made in the upstream part or the packaging in the last few months, and
there are many changes that are not reflected in bugs closed (like most of the
changes needed due to the C++11 ABI transition, bugs and problems fixed that had
no report, and general rejuvenation or improvement of some parts of the code).
How long this will last, I cannot know. I hope to find a job at some point,
which obviously will reduce the time available to work on this.
But in the meantime, for all
aptitude users: Enjoy the fixes and new features!
Notes
[1]
^ Some visitors of the
recent
mini-DebConf @
Cambridge
perhaps thought that the fireworks and throngs gathered were in honour of our
mighty Universal Operating System, but sadly they were not. They might be, some
day. In any case, the reports say that the visitors enjoyed the fireworks.
What happened in the reproducible
builds effort this week:
Toolchain fixes
Long over due post
It s been a long time I haven t written here. And lots of things happened in the OpenStack planet. As a full time employee with the mission to package OpenStack in Debian, it feels like it is kind of my duty to tell everyone about what s going on.
Liberty is out, uploaded to Debian
Since my last post, OpenStack Liberty, the 12th release of OpenStack, was released. In late August, Debian was the first platform which included Liberty, as I proudly outran both RDO and Canonical. So I was the first to make the announcement that Liberty passed most of the Tempest tests with the beta 3 release of Liberty (the Beta 3 is always kind of the first pre-release, as this is when feature freeze happens). Though I never made the announcement that Liberty final was uploaded to Debian, it was done just a single day after the official release.
Before the release, all of Liberty was living in Debian Experimental. Following the upload of the final packages in Experimental, I uploaded all of it to Sid. This represented 102 packages, so it took me about 3 days to do it all.
Tokyo summit
I had the pleasure to be in Tokyo for the Mitaka summit. I was very pleased with the cross-project sessions during the first day. Lots of these sessions were very interesting for me. In fact, I wish I could have attended them all, but of course, I can t split myself in 3 to follow all of the 3 tracks.
Then there was the 2 sessions about Debian packaging on upstream OpenStack infra. The goal is to setup the OpenStack upstream infrastructure to allow packaging using Gerrit, and gating each git commit using the usual tools: building the package and checking there s no FTBFS, running checks like lintian, piuparts and such. I knew already the overview of what was needed to make it happen. What I didn t know was the implementation details, which I hoped we could figure out during the 1:30 slot. Unfortunately, this didn t happen as I expected, and we discussed more general things than I wished. I was told that just reading the docs from the infra team was enough, but in reality, it was not. What currently needs to happen is building a Debian based image, using disk-image-builder, which would include the usual tools to build packages: git-buildpackage, sbuild, and so on. I m still stuck at this stage, which would be trivial if I knew a bit more about how upstream infra works, since I already know how to setup all of that on a local machine.
I ve been told by Monty Tailor that he would help. Though he s always a very busy man, and to date, he still didn t find enough time to give me a hand. Nobody replied to my request for help in the openstack-dev list either. Hopefully, with a bit of insistence, someone will help.
Keystone migration to Testing (aka: Debian Stretch) blocked by python-repoze.who
Absolutely all of OpenStack Liberty, as of today, has migrated to Stretch. All? No. Keystone is blocked by a chain of dependency. Keystone depends on python-pysaml2, itself blocked by python-repoze.who. The later, I upgraded it to version 2.2. Though python-repoze.what depends on version <= 1.9, which is blocking the migration. Since python-repoze.who-plugins, python-repoze.what and python-repoze.what-plugins aren t used by any package anymore, I asked for them to be removed from Debian (see #805407). Until this request is processed by the FTP masters, Keystone, which is the most important piece of OpenStack (it does the authentication) will be blocked for migration to Stretch.
New OpenStack server packages available
On my presentation at Debconf 15, I quickly introduced new services which were released upstream. I have since packaged them all:
It's been at least a year since I last did any work on 
There is a companion logrotate script, to avoid filling the file system:
A new version of 
Writing programs is fun, but making them fast can be a pain. Python programs
are no exception to that, but the basic profiling toolchain is actually not
that complicated to use. Here, I would like to show you how you can quickly
profile and analyze your Python code to find what part of the code you should
optimize.
What's profiling?
Profiling a Python program is doing a dynamic analysis that measures the
execution time of the program and everything that compose it. That means
measuring the time spent in each of its functions. This will give you data
about where your program is spending time, and what area might be worth
optimizing.
It's a very interesting exercise. Many people focus on local optimizations,
such as determining e.g. which of the Python functions 
The variation in accuracy displayed is a result of experimentation with different dataset sizes and algorithm variations. More information and details can be found in the 
I m excited to see other people thinking about userspace-to-kernel attack surface reduction ideas. Theo de Raadt recently published slides describing 
Wishing you all a happy Diwali and Gujarati New Year (V.S. 2072 called Plavanga.)Instead of being something sane like a flat disk supported by four elephants standing on a tortoise, the Earth is an oblate spheroid in heliocentric orbit. The Moon too offends by maintaining an unreasonably non-epicyclic orbit. Combined this means that the Americas (plus Iceland if there any Hindus there) observe Diwali today whereas in India and the rest of the world it is tomorrow. Technically in these longitudes the Ashvayuja amavasya tithi pervades both Tuesday and Wednesday but as Lakshmi Puja takes place at the pradosha kala it means it should be celebrated today though no doubt most of the non-astronomically minded people will just follow India any way. The New Year is on Thursday all over the world.Regardless of when you celebrate, may your year be full of happiness, good fortune and prosperity.
Debian Live has passed on. And it has done so in 
