There are many ways to build Debian distributions from source. None of them are trivial enough for beginners and all of them require complex setups. For instance, the Debian official setup uses the following software components:

• Centralized database with package information, wanna-build.
• Build daemon software along installation documentation.

Around the core components, there are other software components needed to run the Debian distribution:

• Incoming package queues (ftp-master)
• Repository management (dak)

Thanks to Debian ftp-master and buildd team, all software is built for several architectures and several ports. Most Debian infrastructure is managed and maintained by Debian System Administration team. Debian Wiki has been growing different random pages trying to ease the setup and configuration problems concerning to Debian build system infrastructure.

• Debian Wanna Build Infrastructure on one server
• Build Service for Wanna Build
• Buildd Setup

The above infrastructure barely documents what it is involved to create and generate Debian unstable ( sid ) distribution suite. In order to produce Debian stable distribution suite, there are software transitions to happen and yet another Debian team gets involved in the process, our beloved Debian release team. Once distribution reaches its maturity level and gets released as in its stable version, it gets updates also lead by release team and security related updates, which yet another team is responsible for them, the Debian security team. and there are lots of Debian teams doing many other things you might enjoy, have a byte

There are many ways to build Debian distributions from source. None of them are trivial enough for beginners and all of them require complex setups. For instance, the Debian official setup uses the following software components:

• Centralized database with package information, wanna-build.
• Build daemon software along installation documentation.

Around the core components, there are other software components needed to run the Debian distribution:

• Incoming package queues (ftp-master)
• Repository management (dak)

Thanks to Debian ftp-master and buildd team, all software is built for several architectures and several ports. Most Debian infrastructure is managed and maintained by Debian System Administration team. Debian Wiki has been growing different random pages trying to ease the setup and configuration problems concerning to Debian build system infrastructure.

• Debian Wanna Build Infrastructure on one server
• Build Service for Wanna Build
• Buildd Setup

The above infrastructure barely documents what it is involved to create and generate Debian unstable ( sid ) distribution suite. In order to produce Debian stable distribution suite, there are software transitions to happen and yet another Debian team gets involved in the process, our beloved Debian release team. Once distribution reaches its maturity level and gets released as in its stable version, it gets updates also lead by release team and security related updates, which yet another team is responsible for them, the Debian security team. and there are lots of Debian teams doing many other things you might enjoy, have a byte

I've been nagged for a while (by Wookey mainly) about the unhelpfulness of dpkg-checkbuilddeps when it outputs the versions and alternatives alongside the package names which are missing, making it harder to pass the list to the package manager. apt-get build-dep isn't particular helpful either - it doesn't look at the modified package at all.

Of course, once the output is in a suitable format, a new script might as well make it possible to simply pass that output to said package manager. Once it can do that, it can then pass the output to the cross-dependency tools, like xapt.

So, I've refactored the embuilddeps script in the xapt package to do just this.

It's gained a few features in the process:

1. Support for Build-Conflicts resolution


2. Support for virtual packages, swapping the virtual for the first package to provide it (borrowed some code from Dpkg::Deps for that one).


3. Support for Build-Depends alternatives (currently using the buildd default of "first alternative gets first chance")


4. Reads data from debian/control, not the apt cache - to help with the package you're building instead of the one you've already uploaded.


5. Handles cross dependencies (which are always assumed to not currently be installed) and native dependencies. This support is transitory until such time as enough packages are Multi-Arch compatible that Cross-Multi-Arch becomes trivial.


6. Support for being used as a build-dependency resolver in pbuilder, including cross-architecture dependencies with pdebuild-cross.


7. Can locate a debian/control file in a specified directory without needing to be called from that directory


8. Checks your apt-cache policy to see if the required version of a package is available from your current apt sources. Fails completely if not. (The pdebuild-cross usage will need that to be extended a touch to look at the apt-cache policy from within the chroot.)


9. Hector Oron has also been asking me to get embuilddeps working with sbuild, so I'm working on that feature too.


10. verbose and quiet support (so use -q inside other scripts)


11. most output is already translated - more translations are welcome, especially for the documentation, but hold on until this version has actually been uploaded.

More testing is needed, particularly that the extensive refactoring hasn't broken the pbuilder resolution support and then looking at what still needs to be done for sbuild support. The new script is in Emdebian SVN.

The first-choice method for virtuals and alternatives may well bear extension to explicit management via command-line options. I'm unsure yet whether it needs to be a configuration file setting. It could simply be a recursive - try one, move on - model.

The following developers got their Debian accounts in the last month:

• Ludovico Cavedon (cavedon)
• Gilles Filippini (pini)
• Christopher Taylor (ctaylor)
• Oliver Korff (okoli)
• Franck Joncourt (franck)
• Antonio Radici (antonio)
• Emmanuel Bouthenot (kolter)
• Tim Retout (diocles)
• Dario Minnucci (midget)
• Hector Oron (zumbi)
• Albin Tonnerre (lutin)
• Modestas Vainius (modax)
• Christoph Egger (christoph)
• Andrea Veri (and)

Congratulations!

The strongly connected set of the GPG keys graph contains a bit more than 40000 keys now (yes, that’s a lot of geeks!). I wondered what was the biggest clique (complete subgraph) in that graph, and also of course the biggest clique I was in. It’s easy to grab the whole web of trust there. Finding the maximum clique in a graph is NP-complete, but there are algorithms that work quite well for small instances (and you don’t need to consider all 40000 keys: to be in a clique of n keys, a key must have at least n-1 signatures, so it’s easy to simplify the graph — if you find a clique with 20 keys, you can remove all keys that have less than 19 signatures). My first googling result pointed to Ashay Dharwadker’s solver implementation (which also proves P=NP ;). Googling further allowed me to find the solver provided with the DIMACS benchmarks. It’s clearly not the state of the art, but it was enough in my case (allowed to find the result almost immediately). The biggest clique contains 47 keys. However, it looks like someone had fun, and injected a lot of bogus keys in the keyring. See the clique. So I ignored those keys, and re-ran the solver. And guess what’s the size of the biggest “real” clique? Yes. 42. Here are the winners:

CF3401A9 Elmar Hoffmann
AF260AB1 Florian Zumbiehl
454C864C Moritz Lapp
E6AB2957 Tilman Koschnick
A0ED982D Christian Brueffer
5A35FD42 Christoph Ulrich Scholler
514B3E7C Florian Ernst
AB0CB8C0 Frank Mohr
797EBFAB Enrico Zini
A521F8B5 Manuel Zeise
57E19B02 Thomas Glanzmann
3096372C Michael Fladerer
E63CD6D6 Daniel Hess
A244C858 Torsten Marek
82FB4EAD Timo Weing rtner
1EEF26F4 Christoph Ulrich Scholler
AAE6022E Karlheinz Geyer
EA2D2C41 Mattia Dongili
FCC5040F Stephan Beyer
6B79D401 Giunchedi Filippo
74B11360 Frank Mohr
94C09C7F Peter Palfrader
2274C4DA Andreas Priesz
3B443922 Mathias Rachor
C54BD798 Helmut Grohne
9DE1EEB1 Marc Brockschmidt
41CF0322 Christoph Reeg
218D18D7 Robert Schiele
0DCB0431 Daniel Hess
B84EF12A Mathias Rachor
FD6A8D9D Andreas Madsack
67007C30 Bernd Paysan
9978AF86 Christoph Probst
BD8B050D Roland Rosenfeld
E3DB4EA7 Christian Barth
E263FCD4 Kurt Gramlich
0E6D09CE Mathias Rachor
2A623F72 Christoph Probst
E05C21AF Sebastian Inacker
5D64F870 Martin Zobel-Helas
248AEB73 Rene Engelhard
9C67CD96 Torsten Veller

It’s likely that this happened thanks to a very successful key signing party somewhere in germany (looking at the email addresses). [Update: It was the LinuxTag 2005 KSP.] It might be a nice challenge to beat that clique during next Debconf ;) And the biggest clique I’m in contains 23 keys. Not too bad.

http://www.emdebian.org/buildd/

The scripts are in emdebian-tools 1.1.6 and with a few more fixes and tweaks, that will be the main body of v1.2.0 for Debian unstable.

A few issues with ssh and sudo passphrase caching vs running buildd scripts as root - I'll need to improve the documentation of the process before any other buildds can be set up. The need to have emdebian-tools and subversion installed inside the buildd chroot does make things a little more complicated than a standard wanna-build implementation - as does the need to download, build, install and clean-up cross dependencies.

This autobuilder is the Emdebian target package buildd - Hector Oron has another to build the cross-building toolchains. The toolchain autobuilder has fewer packages but each package is larger and is built for more architectures. Adding more architectures to the target autobuilder is quite difficult - packages that use cache files will need new cache files with different values for certain cache values. See the wiki for info on how to fix cache files. Each solves a different problem within the overall design of any buildd. In each case, instead of listening to email from incoming.debian.org, each buildd picks up the incoming data from the apt cache - a useful little time delay that gives me time to head-off certain problematic updates.

The target package autobuilder runs incrementally - a package is repeatedly rebuilt until it works and is then left alone until the next Debian release. There's lots of work available on the frontend if anyone fancies helping with the PHP. (See Emdebian SVN).

The advantage now is that as more packages fix their cross building bugs, more packages will autobuild without intervention which frees up more of my time to fix the more complicated problems. Equally, I can refer to the buildd logs in bug reports, helping everyone see what is going on.

OK, changes in dpkg-dev 1.14.7 have forced my hand a little and dpkg-cross 2.0.0 has now been uploaded to Debian unstable (not experimental).
apt-cross 0.2.9 has been uploaded to Emdebian unstable - dependent on
dpkg-cross >= 1.99
apt-cross 0.3.0 has been uploaded to Debian unstable but will spend some time in the NEW queue (hence the 0.2.9 upload to Emdebian). apt-cross 0.3.0 is functionally identical to 0.2.9
emdebian-tools 0.3.9 has been uploaded to Emdebian unstable - dependent on apt-cross >= 0.2.9 and dpkg-cross >= 1.99
All three comprise extensive rewrites. apt-cross, in particular, is barely recognisable. dpkg-cross is a shadow of the former package and v2.0.0 signals the beginning of the long goodbye to both packages. dpkg developers expect to be able to merge all dpkg-cross functionality into dpkg before Lenny. Once that is done, it should be trivial to merge apt-cross behaviour into apt.
Only the new libcache-apt-perl package (containing the modified NorthernCross code from virtuoso) will persist. Once this whole transition is over, I'll upload Cache::Apt::* to CPAN so that it can become its own source package in Debian once apt-cross is removed.

IMPORTANT
dpkg cross-building support is HIGHLY EXPERIMENTAL!
Significant changes are still pending, including remapping the build paths during the
dpkg-buildpackage build and utilising the updated dpkg-shlibdeps code which is a decade ahead of what we had in dpkg-cross 1.39.

1. dpkg 1.14.7 included dpkg-buildpackage as a perl script instead of shell which breaks dpkg-cross <= 1.39 but the new code to support dpkg-cross >= 1.99 is not implemented yet.


2. The breakage that Zumbi experienced when using the dpkg version from
   experimental will now occur in unstable.
   http://lists.debian.org/debian-embedded/2007/09/msg00067.html. Note that this breakage would be severity "critical" if filed as a bug because it prevents not only cross-builds but normal Debian builds.


3. dpkg-cross 2.0.0 fixes this issue. Hence the urgency in uploading dpkg-cross 2.0.0


4. dpkg-cross 2.0.0 removes the old diversions of dpkg-buildpackage and dpkg-shlibdeps.


5. dpkg is set to support the removal of the dpkg-cross diversions and code to support this is coming this weekend. This means that there will be a GAP where cross-built packages will fail to find the correct shared libraries. HOWEVER, installing dpkg-cross 2.0.0 restores the ability to build normal Debian packages whilst keeping dpkg-cross installed. This is a far more important gain than a temporary problem in packages that hardly anyone is using yet (our Emdebian crossbuilt packages).

When dpkg-dev 1.14.8 becomes available, apt-cross, dpkg-cross and emdebian-tools will be updated to depend on this version.

Everyone: please note - cross-building support in Debian will be "fragile" for the next week or so. The rewritten packages are being uploaded against experimental dpkg support I expect bugs.

The benefits of these changes are:

1. The removal of diversions that are a decade out of date.


2. The implementation of virtuoso's NorthernCross code that transforms apt-cross and fixes lots of awkward bugs.


3. The incorporation of cross-building support in the heart of dpkg.


4. The beginning of the long goodbye to dpkg-cross and apt-cross.

RESULTS
With the new packages installed, some packages will produce:

 emdebuild: ERROR /usr/bin/ contains files for the wrong architecture!

i.e. the build appears to succeed but until dpkg-dev 1.14.8 includes support for looking for the correct libraries in /usr/$arch/lib instead of /usr/lib, we get unusable packages. This at least means that packages can be test built and the patches updated.