Launchpad still requires Python 2, which in 2020
is
a bit of a problem.
Unlike a lot of the rest of 2020, though, there s good reason to be
optimistic about progress.
I ve been porting Python 2 code to Python 3 on and off for a long time, from
back when I was on the Ubuntu Foundations team and maintaining things like
the
Ubiquity installer. When I moved to
Launchpad in 2015 it was certainly on my mind that this was a large body of
code still stuck on Python 2. One option would have been to just accept
that and leave it as it is, maybe doing more backporting work over time as
support for Python 2 fades away. I ve long been of the opinion that this
would doom Launchpad to being unmaintainable in the long run, and since I
genuinely love working on Launchpad - I find it an incredibly rewarding
project - this wasn t something I was willing to accept. We re already
seeing some of our important dependencies dropping support for Python 2,
which is perfectly reasonable on their terms but which is starting to become
a genuine obstacle to delivering important features when we need new
features from newer versions of those dependencies. It also looks as though
it may be difficult for us to run on Ubuntu 20.04
LTS (we re currently on
16.04, with an upgrade to 18.04 in progress) as long as we still require
Python 2, since we have some system dependencies that 20.04 no longer
provides. And then there are exciting new features like
type
hints and
async/await that we d like
to be able to use.
However, until last year there were so many blockers that even considering a
port was barely conceivable. What changed in 2019 was sorting out a
trifecta of core dependencies. We
ported our
database layer,
Storm. We
upgraded
to modern versions of our
Zope Toolkit dependencies
(after contributing various fixes upstream, including some substantial
changes to Zope s
test runner
that we d carried as local patches for some years). And we
ported
our Bazaar code hosting infrastructure to
Breezy. With all that in place, a port
seemed more of a realistic possibility.
Still, even with this, it was never going to be a matter of just following
some
standard porting advice and calling it
good. Launchpad has almost a million lines of Python code in its
main git
tree, and around 250 dependencies of
which a number are quite Launchpad-specific. In a project that size, not
only is following standard porting advice an extremely time-consuming task
in its own right, but just about every strange corner case is going to show
up somewhere. (Did you know that
StringIO.StringIO(None) and
io.StringIO(None) do different things even after you account for the
native string vs. Unicode text difference? How about
the behaviour of
.union() on a subclass of
frozenset?)
Launchpad s test suite is fortunately extremely thorough, but even just
starting up the test suite involves importing most of the data model code,
so before you can start taking advantage of it you have to make a large
fraction of the codebase be at least syntactically-correct Python 3 code and
use only modules that exist in Python 3 while still working in Python 2; in
a project this size that turns out to be a large effort on its own, and can
be quite
risky
in places.
Canonical s product engineering teams work on a six-month cycle, but it just
isn t possible to cram this sort of thing into six months unless you do
literally nothing else, and please can we put all feature development on
hold while we run to stand still is a pretty tough sell to even the most
understanding management. Fortunately, we ve been able to grow the
Launchpad team in the last year or so,
and so it s been possible to put Python 3 on our roadmap on the
understanding that we aren t going to get all the way there in one cycle,
while still being able to do other substantial feature development work as well.
So, with all that preamble, what have we done this cycle? We ve taken a
two-pronged approach. From one end, we identified 147 classes that needed
to be ported away from some compatibility code in our database layer that
was substantially less friendly to Python 3: we ve ported 38 of those, so
there s clearly a fair bit more to do, but we were able to distribute this
work out among the team quite effectively. From the other end, it was clear
that it would be very inefficient to do general porting work when any
attempt to even run the test suite would run straight into the same crashes
in the same order, so I set myself a target of getting the test suite to
start up, and started hacking on an
enormous git
branch that I never
expected to try to land directly: instead, I felt free to commit just about
anything that looked reasonable and moved things forward even if it was very
rough, and every so often went back to tidy things up and cherry-pick
individual commits into a form that included some kind of explanation and
passed existing tests so that I could propose them for review.
This strategy has been dramatically more successful than anything I ve tried
before at this scale. So far this cycle, considering only Launchpad s main
git tree, we ve landed 137 Python-3-relevant merge proposals for a total of
39552 lines of
git diff output, keeping our existing tests passing along
the way and deploying incrementally to production. We have about 27000 more
lines of patch at varying degrees of quality to tidy up and merge. Our main
development branch is only perhaps 10 or 20 more patches away from the test
suite being able to start up, at which point we ll be able to get a buildbot
running so that multiple developers can work on this much more easily and
see the effect of their work. With the full unlanded patch stack, about 75%
of the test suite passes on Python 3! This still leaves a long tail of
several thousand tests to figure out and fix, but it s a much more
incrementally-tractable kind of problem than where we started.
Finally: the funniest (to me) bug I ve encountered in this effort was the
one I encountered in the test runner and fixed in
zopefoundation/zope.testrunner#106:
IDs of failing tests were written to a pipe, so if you have a test suite
that s large enough and broken enough then eventually that pipe would reach
its capacity and your test runner would just give up and hang. Pretty
annoying when it meant an overnight test run didn t give useful results, but
also eloquent commentary of sorts.