Sometimes, I m becoming really pestered by the number of issues, especially hardware issues, that remain prominent in the Linux system after so many years. And I m convinced that one of the root causes is the kernel development model. While far to be the only one, you understand a lot when you have a look at it.
The stable API nonsense
To explain the development model, the kernel documentation contains a document written by Greg Kroah-Hartman called
stable_api_nonsense.txt. I think this document gets one thing very right: advertising it is complete nonsense.
Many things in this document can get you a good laugh if you are used to software development, unfortunately they are not funny given how much they affect us as users and developers. Let s start with the executive summary:
You think you want a stable kernel interface, but you really do not, and
you don't even know it. What you want is a stable running driver, and
you get that only if your driver is in the main kernel tree. You also
get lots of other good benefits if your driver is in the main kernel
tree, all of which has made Linux into such a strong, stable, and mature
operating system which is the reason you are using it in the first
place.
What it should say instead is:
You know you want a stable kernel interface, but you don't have
it, and we will never provide it. You want stable interfaces to
focus on fixing bugs in your driver instead of updating it for
interface changes, and to make integration of your driver in the
main kernel tree easy. You would get a lot of good benefits if
your driver was in the main kernel tree, but it won't make it
unless you adapt to our bizarre processes, all of
which have made the Linux kernel into a constantly moving target
which is the reason many hardware vendors don't want to support
it in the first place.
Our compiler does not have the same ABI as yours
This is what happens when the only thing that interests people in charge is to code. Writing this means that these developers have no interest in making the system usable. This becomes more unbearable when such technically sharp people make up false technical arguments:
Depending on the version of the C compiler you use, different kernel
data structures will contain different alignment of structures, and
possibly include different functions in different ways (putting
functions inline or not.)
Really, I wonder how the Microsoft developers, the GNOME guys, the glibc guys, and basically all library developers who know what they are talking about, manage to have stable ABIs despite the compiler changing all the time. One of the incredibly complicated techniques they use is to only rely on guaranteed functionality of the C specification, instead of doing many clever and funky hacks. For things more related to development itself, there are also incredibly complicated techniques like opaque structures, which save a lot of ABI breakage. Actually the only thing you need, if you design your ABI properly, is to be rigorous.
Depending on what kernel build options you select, a wide range of
different things can be assumed by the kernel
This statement of a problem also contains the solution: stop making available build options no one cares about (remember, people install binary packages from their distributions) to guarantee stability instead. Instead of implementing the solution, kernel developers deliberately choose to deal with this insanity in the worst possible way: by encouraging it.
Only lame developers need stable APIs
But the nonsense doesn't stop to the ABI, which after all is merely a problem for distributors. To be more annoying to developers themselves, let s make up reasons to break the API all the time:
Linux kernel development is continuous and at a rapid pace, never
stopping to slow down. As such, the kernel developers find bugs in
current interfaces, or figure out a better way to do things.
Explained this way, it looks like a good thing. If you've done serious
development or integration, you know that actually it is a problem. If
you never slow down to look at what you've done, you're only going to
add new bugs while you're trying to fix the others.
When they do
so, function names may change, structures may grow or shrink, and
function parameters may be reworked. If this happens, all of the
instances of where this interface is used within the kernel are fixed up
at the same time, ensuring that everything continues to work properly.
Everything is said. In the Linux kernel, changes are not done in a
continuous way. Every non-minor change is going to have an impact on
hundreds of different modules. The amount of work needed to accomplish
these changes is absolutely insane. When a project like GNOME decides to
phase-out an API, it takes several years before being actually replaced. In
the kernel, it can simply happen between two minor releases, together with
a rewrite of the whole ATA stack.
The lolution
Of course, Greg KH has a simple solution for you, in the What to do section.
So, if you have a Linux kernel driver that is not in the main kernel
tree, what are you, a developer, supposed to do? Releasing a binary
driver for every different kernel version for every distribution is a
nightmare, and trying to keep up with an ever changing kernel interface
is also a rough job.
No shit !
Simple, get your kernel driver into the main kernel tree.
Simple, isn't it? Unfortunately this is not going to happen while you
are too busy adapting it for the constantly moving interfaces. This is
quite feasible if you are writing a driver for an Ethernet network
adapter, but for a video card this is another story, and for a
virtualization layer? Well, the Xen developers have been struggling for
years to integrate their technology in the kernel, and it is still far
from done. No wonder why you get good drivers only for ethernet cards
and not for 3D cards or even wifi chips.
This doesn t go without other, worse consequences for users and distributors. Not everyone can afford to run Ubuntu, Debian unstable or Fedora, especially if they have simple requirements like not changing the whole system every 6 months. Corporate users using Debian stable, RHEL, SLES or Ubuntu LTS need a stable kernel that doesn t move, and that doesn t break everything when upgraded.
The introduction of projects like
DKMS to distribute drivers in a sourceful way while keeping them easily installable is a step in the good direction, despite being a workaround for a broken situation underneath. However these efforts are ruined by the constantly changing APIs that will go on requiring changes to the sources as well as the binaries.
Why it can still work
In fact, with such processes, you can wonder how it is possible that the system is actually usable. The answer is simple: while being smaller than many other projects, the Linux kernel has much, much more developers.
A rough estimation shows that Linux has 2000 developers, many of which
are paid full-time to work on it, to work on a 15 Mloc codebase. You
can compare it with X.org, which has a few dozens of developers for 3
Mloc. Or with GNOME, 400 developers for 20 Mloc. With OOo, 150
developers for 10 Mloc.
Yes, you need 10 times as many people to maintain the kernel code than
for another comparable project. That's of course what happens when you
keep them busy with a permanent refactoring of the code.
This development method, while certainly having the advantage of bringing lots of innovation and fast integration of new features, is also driving away a very large portion of the open source community to work on rewriting Firewire stacks every 6 months. And as long as they are busy with this, they are not writing correct X drivers nor fixing bugs in the applications.