It's been a while since I wanted to post about a project I started.
But today when I read Russell Coker's post about his
SE Linux repository I decided it was time to blog about DeBaBaReTools.
So, what is/are DeBaBaReTools?is: a project which tries to unify different packages-building systems (pbuilder, buildd, etc) and different repository tools (debarchiver, reprepro, etc) in a single system by providing an incoming/ system.
are: Debian Bash-Based Repository Tools.
Or in other words:DeBaBaReTools handles new packages by:
- Verifying (signature, size, md5 sums)
- Building (packages are listed to be built under certain architectures and then an other set of scripts take care of building them, so different build systems and hosts can be used)
- Installing (by making use of a tool which handles the repository itself, e.g. reprepro)
DeBaBaReTools is NOT a rewrite of a repository tool (like reprepro, debpool or debarchiver), neither yet an other building tool (like buildd or pbuilder). It is rather a glue between those two kind of tools.
So, how does DeBaBaReTools work?As already said, they are a set of scripts which do all the work.
These are some of the scripts (more might be written later, they are under development):
- checkSums: just like but doesn't allow the signatures check to be skipped (not really a feature), it uses gpgv and only uses the keyrings specified by KEYRING rather than using the debian-keyring keyring if installed. Available checks (in order): signature, size, md5sum
- processIncoming: it takes care of checking the files in pub/UploadQueue, moving them to accepted (and soon to rejected or new), and running the repository's commands to install the packages from incoming/. It also takes care of moving the source/packages to incoming/ when necessary. Packages are verified by checkSums before moving anything to accepted
- needsBuild: this script processes .changes and .dsc files in accepted/ and marks them to be built on the architectures supported by the repository.
- wannaBuild: processes the packages that need to be built and calls the given build system (pbuilder, buildd, etc), and installs (and soon upload) them to the UploadQueue so they are processed by processIncoming and then installed in the repository.
- wannaLock (not yet written): this script will take care of managing a distributed package building lock system
- wannaUpload (not yet written): this script will take care of uploading/installing the built packages to the UploadQueue directory of the repository (will use dput, and similar).
- needsReBuild (not yet written): will test all packages with edos-debcheck and mark the packages to be rebuilt when necessary. This feature will not be in needsBuild because of the long run it requires and also because of some checks performed by needsBuild.
Note that at the moment these are the only supported backends: gpgv, du, md5sums (all built in checkSums), pbuilder and reprepro.
Sounds interesting, doesn't it? :)
So, here's the magic:
The backends systemDeBaBaReTools doesn't try to re-invent the wheel, but it makes use of all the available tools so it works as less as possible.
This is where the backends take action. They are even more shell (even tough DeBaBaReTools clearly specifies they are Bash scripts they should work on any other POSIX-compatible shell) scripts which are loaded on demand and according to the user settings.
This opens a wide range of possible combinations of backends; making DeBaBaReTools work with almost anything.
The probe.sh script searches different directories for the specified backend (and will soon support user-hinted backends).
I think an example says more than a thousand words:
When needsBuild finds a package that needs to be built in a given architecture, the toBuild shell function is called. In order to provide the already said flexibility, needsBuild will at startup time call probeFile "buildQueue" which will then check several directories for the required backend (buildQueue in this case). Once it finds the given backend (backends/buildQueue/) it will check for any user hint (which will specify the real backend to be used) and then for a default one (default.sh which is a symlink to a real backend, plaintext.sh by default).
So, if somebody at some point wants to use a sqlite database to store all that information, that person would have to write a file called backends/buildQueue/sqlite.sh (or make it available in any of the include paths which can be extended via the EXTRA_PROBE_DIRS env variable) and either set a user hint or make default.sh point to it.
The sqlite.sh script can then call any external tools (or even more scripts) to insert the information in the database, read it and more.
I know this is a lot of blahblahblah but no single line of code is shown.
I've already requested an alioth project but it was denied because the package isn't already in the Debian archive.
If you are a DD and you think this sounds like a good project please contact the Alioth admins and convince them to create a project account for DeBaBaReTools.
Once created, many other people will be able to work on it and make it work with virtually any tool.
Any kind of feedback is very welcomed but I hope they are constructive ideas.
Why did I start this crazy project?
Because AFAIR only dak is able to do such job but I think dak is just too complex for a simple system to manage an a repository with an incoming system. And AFAIK there's no script which is able to do this job besides dak.
I hope this post makes sense because I don't want to review it again (I'm tired) :)
:bye: all