As some of you already know, since the larger restructuring in PackageKit for the 1.0 release, I am
rethinking Listaller, the 3rd-party application installer for Linux systems, as well.
During the past weeks, I was playing around with a lot of different ideas and code, to make installations of 3rd-party software easily possible on Linux, but also working together with the distribution package manager. I now have come up with an experimental project, which might achieve this.
Motivation
Many of you know Lennart s
famous blogpost on how we put together Linux distributions. And he makes a lot of good and valid points there (in fact, I agree with his reasoning there). The proposed solution, however, is not something which I am very excited about, at least not for the use-case of installing a simple application[1]. Leaving things like the exclusive dependency on technology like Btrfs aside, the solution outlined by Lennart basically bypasses the distribution itself, instead of working together with it. This results in a duplication of installed libraries, making it harder to overview which versions of which software component are actually running on the system. There is also a risk for security holes due to libraries not being updated. The security issues are worked around by a superior sandbox, which still needs to be implemented (but will definitively come soon, maybe next year).
I wanted to explore a different approach of managing 3rd-party applications on Linux systems, which allows sharing as much code as possible between applications.
Limba Glick2 and Listaller concepts merged
In order to allow easy creation of software packages, as well as the ability to share software between different 3rd-party applications, I took heavy inspiration from Alexander Larssons
Glick2 project, combining it with ideas from the application-directory based Listaller.
The result is
Limba (named after
Limba tree, not the voodoo spirit I needed some name starting with li to keep the prefix used in Listaller, and for a tool like this the name didn t really matter
).
Limba uses
OverlayFS to combine an application with its dependencies before running it, as well as mount
namespaces and
shared subtrees. Except for OverlayFS, which just landed in the kernel
recently, all other kernel features needed by Limba are available for years now (and many distributions ship with OverlayFS on older kernels as well).
How does it work?
In order to to achieve separation of software, each software component is located in a separate container (= package). A software component can be an application, like Kate or GEdit, but also be a single shared library (openssl) or even a full runtime (KDE Frameworks 5 parts, GNOME 3).
Each of these software components can be identified via
AppStream metadata, which is just a few bits of XML. A Limba package can declare a dependency on any other software component. In case that software is available in the distribution s repositories, the version found there can be used. Otherwise, another Limba package providing the software is required.
Limba packages can be provided from software repositories (e.g. provided by the distributor), or be nested in other packages. For example, imagine the software Kate requires a version of the Qt5 libraries, >= 5.2. The downloadable package for Kate can be bundled with that dependency, by including the Qt5 5.2 Limba package in the Kate package. In case another software is installed later, which also requires the same version of Qt, the already installed version will be used.
Since the software components are located in separate directories under /opt/software, an application will not automatically find its dependencies, or be able to locate its own files. Therefore, each application has to be run by a special tool, which merges the directory trees of the main application and it s dependencies together using OverlayFS. This has the nice sideeffect that the main application could override files from its dependencies, if necessary. The tool also sets up a new mount namespace, so if the application is compiled with a certain prefix, it does not need to be relocatable to find its data files.
At installation time, to achieve better system integration, certain files (like e.g. the .desktop file) are split out of the installed directory tree, so the newly installed application achieves almost full system integration.
AQNAY*
Can I use Limba now?
Limba is an experiment. I like it very much, but it might happen that I find some issues with it and kill it off again. So, if you feel adventurous, you can compile the
source code and use the example Foobar application to play around with Limba. Before it can be used in production (if at all), some more time is needed.
I will publish documentation on how to test the project soon.
Doesn t OverlayFS have a maximum stacking depth?
Oh yes it has! The How does it work explanation doesn t tell the whole truth in that regard (mainly to keep the section small). In fact, Limba will generate a runtime for the newly installed software, which is a directory with links to the actual individual software components the runtime consists of. The runtime is identified by an UUID. This runtime is then mounted together with the respective applications using OverlayFS. This works pretty great, and also results in no dependency-resolution to be done immediately before an application is started.
Than dependency stuff gives me a headache
Admittedly, allowing dependencies adds a whole lot of complexity. Other approaches, like the one outlined by Lennart work around that (and there are good reasons for doing that as well).
In my opinion, the dependency-sharing and de-duplication of software components, as well as the ability to use the components which are packaged by your Linux distribution is worth the extra effort.
Can you give an overview of future plans for Limba?
Sure, so here is the stuff which currently works:
- Creating simple packages
- Installing packages
- Very basic dependency resolution (no relations (like >, <, =) are respected yet)
- Running applications
- Initial bits of system integration (.desktop files are registered)
These features are planned for the new future:
- Support for removing software
- Automatic software updates of 3rd-party software
- Atomic updates
- Better system integration
- Integration with the new sandboxing features
- GPG signing of packages
- More documentation / bugfixes
Remember that Limba is an experiment, still
XKCD 927
Technically, I am replacing one solution with another one here, so the situation does not change at all ;-). But indeed, some duplicate work is done due to more people working in this area now on similar questions.
But I think this is a good thing, because the solutions worked on are fundamentally different approaches, and by exploring multiple ways of doing things, we will come up with something great in the end. (
XKCD reference)
Doesn t the use of OverlayFS have an impact on the performance of software running with Limba?
I ran some synthetic benchmarks and didn t notice any problems even the startup speed of Limba applications is only a few milliseconds slower than the startup of the raw native application. However, I will still have to run further tests to give a definitive answer on this.
How do you solve ABI compatibility issues?
This approach requires software to keep their ABI stable. But since software can have strict dependencies on a specific version of a software (although I d discourage that), even people who are worried about this issue can be happy. We are getting
much better at tracking unwanted ABI breaks, and larger projects offer stable API/ABI during a major release cycle. For smaller dependencies, there are, as explained above, stricter dependencies.
In summary, I don t think ABI incompatibilities will be a problem with this approach at least not more than they have been in general. (The libuild facilities from Listaller to minimize dependencies will still be present im Limba, of course)
You are wrong because of $X!
Please leave a comment in this case! I d love to discuss new ideas and find the limits of the Limba concept that s why I am writing C code afterall, since what looks great on paper might not work in reality or have issues one hasn t thought about before. So any input is welcomed!
Conclusion
Last but not least I want to thank
Alexander Larsson for writing Glick2, which Limba is heavily inspired from, and for his patient replies to my emails.
If Limba turns out to be a good idea, you can expect a few more blog posts about it soon.
* Answered questions nobody asked yet
[1]: Don t get me wrong, I would like to have these ideas implemented they offer great value. But I think for simple software deployment, the solution is an overkill.
Almost two months ago I posted our keyring status graphs, showing the progress of the transition to >=2048-bit keys for the different active Debian keyrings. So, here are the new figures.
First, the Non-uploading keyring: We were already 100% transitioned. You will only notice a numerical increase: That little bump at the right is our dear friend T ssia finally joining as a Debian Developer. Welcome! \o/
As for the Maintainers keyring: We can see a sharp increase in 4096-bit keys. Four 1024-bit DM keys were migrated to 4096R, but we did have eight new DMs coming in To them, also, welcome \o/.
Sadly, we had to remove a 1024-bit key, as Peter Miller sadly passed away. So, in a 234-key universe, 12 new 4096R keys is a large bump!
Finally, our current-greatest worry If for nothing else, for the size of the beast: The active Debian Developers keyring. We currently have 983 keys in this keyring, so it takes considerably more effort to change it.
But we have managed to push it noticeably.
This last upload saw a great deal of movement. We received only one new DD (but hey welcome nonetheless! \o/ ). 13 DD keys were retired; as one of the maintainers of the keyring, of course this makes me sad but then again, in most cases it's rather an acknowledgement of fact: Those keys' holders often state they had long not been really involved in the project, and the decision to retire was in fact timely. But the greatest bulk of movement was the key replacements: A massive 62 1024D keys were replaced with stronger ones. And, yes, the graph changed quite abruptly:
We still have a bit over one month to go for our cutoff line, where we will retire all 1024D keys. It is important to say we will not retire the affected accounts, mark them as MIA, nor anything like that. If you are a DD and only have a 1024D key, you will still be a DD, but you will be technically unable to do work directly. You can still upload your packages or send announcements to regulated mailing lists via sponsor requests (although you will be unable to vote).
Speaking of votes: We have often said that we believe the bulk of the short keys belong to people not really active in the project anymore. Not all of them, sure, but a big proportion. We just had a big, controversial GR vote with one of the highest voter turnouts in Debian's history. I checked the GR's tally sheet, and the results are interesting: Please excuse my ugly bash, but I'm posting this so you can play with similar runs on different votes and points in time using the public keyring Git repository:
Unless you ve been living under a firewalled rock, you know that IPv6 is
coming. There s also a good chance that you ve heard that IPv6 doesn t have
NAT. Or, if you pay close attention to the minutiae of IPv6 development,
you ve heard that IPv6 does have NAT, but you don t have to (and
shouldn t) use it.
So let s say we ll skip NAT for IPv6. Fair enough. However, let s say you
have this use case:
Another PSA: something surprising about XML.
As you might all know, XML must be valid UTF-8 (or UTF-16 (or another encoding supported by the parser, but one which yields valid Unicode codepoints when read and converted)). Some characters, such as the ampersand & , must be escaped ( & or & , although & may also work, depending on the domain) or put into a CDATA section ( <![CDATA[&]]> ).
A bit surprisingly, a literal backspace character (ASCII 08h, Unicode U+0008) is not allowed in the text. I 
Today I was walking past the 
If you have a cadence sensor on your bike such as the Garmin GSC-10, you can approximate the number of pedal turns you made on the bike ride using the following script (requires 
Big changes are always a complicated thing to get done and can be the harder the bigger or more diverse an organization is it is taking place in.
Transparency
Ideally every change is well communicated early and openly.
Leaving people in the dark about what will change and when means people have much less time to feeling comfortable about it or arranging with it mentally.
Especially bad can be extending the change later or or shortening transition periods.
Letting people think they have some time to transition only to force them to rush later will remove any credibility you have and severely reduce their ability to believe you are not crossing them.
Making a new way optional is a great way to create security (see below), but making that obligatory before the change even arrives as optional with them will not make them very willing to embrace change.
Take responsibility
Every transformation means costs.
Even if some change did only improve and did not make anything worse once implemented (the ideal change you will never meet in reality), the deployment of the change still costs:
processes have adapted to it, people have to relearn how to do things, how to detect if something goes wrong, how to fix it, documentation has to be adopted and and and.
Even as the change causes more good than costs in the whole organization (let's hope it does, I hope you wouldn't try to do something if the total benefit is negative), the benefits and thus the benefit to cost ratio will differ for the different parts of your organization or the different people within it.
It's hardly avoidable that for some people there will not be much benefit, much less perceived benefit compared to the costs they have to burden for it.
Those are the people whose good will you want to fight for, not the people you want to fight against.
They have to pay with their labor/resources and thus their good will for your benefit the overall benefit.
This is much easier if you acknowledge that fact.
If you blame them for having the costs, claim their situation does not even exist or even ridicule them for not embracing change you only prepare yourself for frustration.
You might be able to persuade yourself that everyone that is not willing to invest in the change is just acting out of malevolent self-interest.
But you will hardly be able to persuade people that it is evil to not help your cause if you treat them as enemies.
And once you ignored or played down costs that later actually happen, your credibility in being able to see the big picture will simply cease to exist at all for the next change.
Allow different metrics
People have different opinions about priorities, about what is important, about how much something costs and even about what is a problem.
If you want to persuade them, try to take that into account.
If you do not understand why something is a reason, it might be because the given point is stupid. But it might also be that you miss something. And often there is simple a different valuation of what is important, what the costs are and what are problems.
If you want to persuade people, it is worth to try to understand those.
If all you want to do is persuade some leader or some majority then ridiculing their concerns might get you somewhere.
But how do you want to win people over if you do not even appear to understand their problems.
Why should people trust you that their costs will be worth the overall benefits if you tell them the costs that they clearly see do not exist? How credible is referring to the bigger picture if the part of the picture they can see does not match what you say the bigger picture looks like?
Don't get trolled and don't troll
There will always be people that might be unreasonable or even try to provoke you. Don't allow being provoked. Remember that for successful changes you need to win broad support. Feeling personally attacked or feeling presented a large amount of pointless arguments easily results in not bringing proper responses or actually looking at arguments.
If someone is only trolling and purely malevolent, they will tease you best if they bring actual concerns of people in a way you likely degrade your yourself and your point in answering.
Becoming impertinent with the troll is like attacking the annoying little goblin hiding next to the city guards with area damage.
When not being able to persuade people, it is also far to easy to consider them in bad faith and/or attacking them personally. This can only escalate even more. Worst case you frustrate someone in good faith. In most cases you poison the discussion so much that people actually in good faith will no longer contribute the discussion. It might be rewarding short term because after some escalation only obviously unreasonable people will talk against you, but it makes it much harder to find solutions together that could benefit anyone and almost impossible to persuade those that simply left the discussion.
Give security
Last but not least, remember that humans are quite risk-averse. In general they might invest in (even small) chances to win, but go a long way to avoid risks.
Thus an important part of enabling change is to reduce risks, real and perceived ones and give people a feeling of security.
In the end, almost every measure boils down to that:
You give people security by giving them the feeling that the whole picture is considered in decisions (by bringing them early into the process, by making sure their concerns are understood and part of the global profit/cost calculation and making sure their experiences with the change are part of the evaluation).
You give people security by allowing them to predict and control things (by transparency about plans, how far the change will go and guaranteed transitions periods, by giving them enough time so they can actually plan and do the transition).
You give people security by avoiding early points of no return (by having wide enough tests, rollback scenarios,...).
You give people security by not letting them alone (by having good documentation, availability of training, ...).
Especially side-by-side availability of old and new is an extremely powerful tool, as it fits all of the above:
It allows people to actually test it (and not some little prototype mostly but not quite totally unrelated to reality) so their feedback can be heard. It makes it more predictable as all the new ways can be tried before the old ones no longer work. It is the ultimate role-back scenario (just switch off the new). And allows for learning the new before losing the old.
Of course giving the people a feeling of security needs resources. But it is a very powerful way to get people to embrace the chance.
Also in my experience people only fearing for themselves will usually mostly be passive by not pushing forward and trying to avoid or escape the changes.
(After all, working against something costs energy, so purely egoistic behavior is quite limiting in that regard).
Most people actively pushing back do it because they fear for something larger than only them. And any measure to making them fear less that you ruin the overall organization, not only avoids unnecessary hurdles rolling out the change but also has some small chance to actually avoid running into disaster with closed eyes.
A common way to build a firewall ruleset is to run a shell script
calling 
This is a reply to Josselin Mouette's blog article 
Like 
There s a lot of discussion / moaning /arguing at this time, so I thought I d post something about how LAVA got into Debian Jessie, the work involved and the lessons I ve learnt. Hopefully, it will help someone avoid the disappointment of having their package missing the migration into a future stable release. This was going to be a talk at the Minidebconf-uk in Cambridge but I decided to put this out as a permanent blog entry in the hope that it will be a useful reference for the future, not just Jessie.
Context
LAVA relies on a number of dependencies which were at the time all this started NEW to Debian as well as many others already in Debian. I d been running LAVA using packages on my own system for a few months before the packages were ready for use on the main servers (I never actually installed LAVA using the old virtualenv method on my own systems, except in a VM). I did do quite a lot of this on my own but I also had a team supporting the effort and valuing the benefits of moving to a packaged system.
At the time, LAVA was based on Ubuntu (12.04 LTS Precise Pangolin) and a new Ubuntu LTS was close (Trusty Tahr 14.04) but I started work on this in 2013. By the time my packages were ready for general usage, it was winter 2013 and much too close to get anything into Ubuntu in time for Trusty. So I started a local repo using space provided by Linaro. At the same time, I started uploading the dependencies to Debian. json-schema-validator, django-testscenarios and others arrived in April and May 2014. (Trusty was released in April). LAVA arrived in NEW in May, being accepted into unstable at the end of June. LAVA arrived in testing for the first time in July 2014.
Upstream development continued apace and a regular monthly upload, with some hotfixes in between, continued until close to the freeze.
At this point, note that although upstream is a medium sized team, the Debian packaging also has a team but all the uploads were made by me. I planned ahead. I knew that I would be going to Macau for Linaro Connect in February a critical stage in the finalisation of the packages and the migration of existing instances from the old methods. I knew that I would be on vacation from August through to the end of September 2014 including at least two weeks with absolutely no connectivity of any kind.
Right at this time, Django1.7 arrived in experimental with the intent to go into unstable and hence into Jessie. This was a headache for me, I initially sought to delay the migration until after Jessie. However, we discussed it upstream, allocated time within the busy schedule and also sought help from within Debian with the RFH tag. Rapha l Hertzog contributed patches for django1.7 support and we worked on those patches upstream, once I was back from vacation. (The final week of my vacation was a work conference, so we had everyone together at one hacking table.)
Still there was more to do, the django1.7 patches allowed the unit tests to complete but broke other parts of the lava-server package and needed subsequent tweaks and fixes.
Even with all this, the auto-removal from testing for packages affected by RC bugs in their dependencies became very important to monitor (it still is). It would be useful if some packages had less complex dependency chains (I m looking at you, uwsgi) as the auto-removal also covers build-depends. This led to some more headaches with libmatheval. I m not good with functional programming languages, I did have some exposure to Scheme when working on Gnucash upstream but it wasn t pleasant. The thought of fixing a scheme problem in the test suite of libmatheval was daunting. Again though, asking for help, I found people in the upstream team who wanted to refresh their use of scheme and were able to help out. The fix migrated into testing in October.
Just for added complications, lava-server gained a few RC bugs of it s own during that time too fixed upstream but awkward nonetheless.
Achievement unlocked
So that s how a complex package like lava-server gets into stable. With a lot of help. The main problem with top-level packages like this is the sheer weight of the dependency chain. Something seemingly unrelated (like libmatheval) can seriously derail the migrations. The package doesn t use the matheval support provided by uwsgi. The bug in matheval wasn t in the parts of matheval used by uwsgi. It wasn t in a language I am at all comfortable in fixing but it s my name on the changelog of the NMU. That happened because I asked for help. OK, when django1.7 was scheduled to arrive in Debian unstable and I knew that lava was not ready, I reacted out of fear and anxiety. However, I sought help, help was provided and that help was enough to get upstream to a point where the side-effects of the required changes could be fixed.
Maintaining a top-level package in Debian is becoming more like maintaining a core package in Debian and that is a good thing. When your package has a lot of dependencies, those dependencies become part of the maintenance workload of your package. It doesn t matter if those are install time dependencies, build dependencies or reverse dependencies. It doesn t actually matter if the issues in those packages are in languages you would personally wish to be expunged from the archive. It becomes your problem but not yours alone.
Debian has a lot of flames right now and Enrico encouraged us to look at what else is actually happening in Debian besides those arguments. Well, on top of all this with lava, I also did what I could to help the arm64 port along and I m very happy that this has been accepted into Jessie as an official release architecture. That s a much bigger story than LAVA yet LAVA was and remains instrumental in how arm64 gained the support in the kernel and various upstreams which allowed patches to be accepted and fixes to be incorporated into Debian packages.
So a roll call of helpers who may otherwise not have been recognised via changelogs, in no particular order:
Noticing that Debian just entered its freeze, I went ahead and changed the APT sources on a spare host that is currently running stable. Then, it was time for this command to be executed: