Simon Richter: Wacky ideas #13: Conflicts-If-in-Same-Dependency-Tree
Well, obviously this one needs a shorter name.
The problem I'm trying to solve here is that SONAMEs, or rather the way they are used today, only handle the issue of a single, fully contained library changing ABI as one of its header files changes.
However, libraries do not exist in a vacuum; as a bare minimum, they use definitions for types and alignment requirements built into the compiler, and if any of these changes, so does the library ABI. Hence, the idea of using a single number to denote the ABI is not really sufficient.
The correct solution would be to embed all ABI information into the soname (libapt-pkg, for example, uses "libapt-pkg-libc6.3-6.so.3.11" on amd64, which is better but still not perfect). Obviously, this would require massive changes to the build scripts in all libraries and educating all library authors, so it is not going to happen (soon).
So, what other options are there?
As the kernel ABI (the syscall interface) is supposed to be really stable, we probably do not have to worry about that. So, we only have a problem if there are libraries with different assumptions about each others' ABI in the same userspace program. This can happen in two ways: The compiler changed (as in the case with the recent C++ transitions), or a library that is an indirect dependency of something that also has another dependency (direct or indirect) changes ABI. I've talked (briefly) about the former in an earlier post already.
To illustrate the latter, a small example:
There is a library, libuseful.so.0 that provides a number of functions, among these:
struct complex_number
float real;
float imaginary;
void complex_multiply(
struct complex_number *result,
struct complex_number const *lhs,
struct complex_number const *rhs);
Now another library, libresourceful.so.0, comes along and implements complex_divide on top of complex_multiply. Since the definition for complex numbers from libuseful makes a lot of sense, the include file for libresourceful reads
#include "libuseful.h"
void complex_divide(
struct complex_number *result,
struct complex_number const *lhs,
struct complex_number const *rhs);
Now one of the more serious people in Debian decides to write pointless 1.0, which takes two complex numbers and calculates (a*b)/a, using both complex_multiply and complex_divide, and links against libresourceful.so.0 and libuseful.so.0.
Some time passes, and shortly before Debian freezes, libuseful's author decides that she needs more precision, so she changes the floats to doubles, bumps the SONAME and releases the new version. Shortly thereafter, someone notices that libresourceful is now using an outdated version of libuseful, and recompiles libresourceful. Since the header of libresourceful did not change, noone thinks of changing the SONAME, after all, changing it would render us incompatible with other distributions.
Now, pointless 1.0 depends on libuseful.so.0 and libresourceful.so.0, which in turn uses libuseful.so.1. It still believes complex numbers are made from floats, however libresourceful now disagrees. The (a*b) expression is calculated correctly, using floats, however the result is then passed to the complex_divide function which expects a pair of doubles. Not good.
Obviously, there is a number of things people have done wrong in this example.
First of all, if you inherit type definitions or anything else from another library, your ABI becomes bound to theirs. As you cannot know statically what version your library will be linked against, you need a script that computes your ABI version at build time. If you do not want that, you need to repeat the type definitions in your header and (deep-)copy the data from one of your structures into one provided by the library you are using (I can already hear the screams of the performance fetishists).
The assumption that, just because a dependency changed, there is not necessarily a need to bump the soname is not fundamentally incorrect; if no interfaces are being propagated, it is certainly correct, and most libraries do not really do this (for example, a HTTP library uses the resolver, but never passes the hostent structures upwards). Detecting this is difficult to do, although Sun has a few tools to help here.
So where does that leave us?
In my opinion, the most sensible thing a distribution can do is to make sure such a constellation never gets installed to an user's system. If there is any doubt, we need to err on the side of caution (aka use a bigger hammer).
This is a "wacky ideas" post, so I also need to propose a totally untested thing that might work (or not), but at least has passed the stage of me sleeping a night over it.
Ideally, dpkg would enforce that. In order to do that, it needs appropriate metadata that tells it that a certain constellation is probably wrong. The thing we want to avoid is an application using multiple versions of the same library at the same time, where "use" is denoted by a dependency relationship. So what we need is a way for a library package to declare that it is wrong to (indirectly) depend on this package and a package out of a certain group at the same time. I can see two ways of doing that, each with its own advantages and disadvantages:
The "explicit" way is to add a new dependency relationship (that is where the subject "Conflicts-If-in-Same-Dependency-Tree" comes from), where you would conflict in this way with a virtual package that is named after your library without the SONAME version, and at the same time provide that virtual package (which is a technique that should not be unfamiliar to Debian people). The hypothetical libuseful above would then state
Package: libuseful0
Provides: libuseful
Conflicts-If-in-Same-Dependency-Tree: libuseful
So installing pointless_1.0 would pull in libuseful0 and libuseful1, dpkg would see the conflict and refuse. For upgrades, it would refuse the libresourceful0 upgrade, as that would create an inconsistent situation. The only way out of there is to recompile pointless.
The advantage is that the modifications to dpkg would be manageable (at least compared with the other option, which I will introduce shortly), the obvious disadvantage is the massive number of virtual packages introduced.
The "implicit" way of doing that is moving the SONAME into its own field in the control file, and teaching dpkg that two packages with the same name and different SONAME fields can be installed at the same time, although not depended on by the same package. This is a really nasty change to dpkg and also requires a syntax for dependencies that includes the desired SONAME version, but it gives us a lot of other advantages.
The one immediately visible is that it does not bloat the Packages file too much, and does not introduce lots of virtual packages, although the net effect inside apt and dpkg is probably going to be the same. A really cool side effect though would be that the soname is no longer part of the Package: field, and thus it becomes possible to insert it using substvars. This, of course, is a cool thing when library authors come to their senses and include ABI information for dependent libraries into their SONAMEs.
Since this has gotten lengthy, thanks for reading this far.
Of course, questions and comments are requested, and if you are interested in pursuing this, please get in touch, either via email or if you can wait that long, at the "Wacky Ideas" BoF at DebConf.
The last thing for me to do here is to welcome Joachim to the "post your ideas" "meme".
Now, while the government may now adopt open standards and open-everything, does not mean that the openness’ forgoes security, either; the government may opt to use well-known encryption protocols and even base their own security infrastructure on them. In fact, they are free to even look inside the source of these well-known standards, to study them, and to branch off new implementation that may even change the way these standards work.
In fact, even without this bill in place, the government can participate in the production and development of FOSS!!!
Wait a minute, wasn’t I supposed to defend this bill?
Eh, well. I suppose this bill is good and all, but like I said earlier, there are those first impressions made by the parties involved in the making of this bill which unsettles me. IMHO I would rather see applications of this first in key cities (the happy works in Munich and in Extremadura are no accident, believe me) which in turn, would allow both local the national government to see just exactly how this FOSS magic works. Then, when the observations have been made, the papers are in, and the workers get hired, then, maybe, we’ll have a nation where FOSS can be mandated as a very strong preference, but never a forced one.
Its all about growing up, really. I’m being reminded of Tom DeMarco’s