Search Results: "Stefan Fritsch"

4 October 2016

Raphaël Hertzog: My Free Software Activities in September 2016

My monthly report covers a large part of what I have been doing in the free software world. I write it for my donators (thanks to them!) but also for the wider Debian community because it can give ideas to newcomers and it s one of the best ways to find volunteers to work with me on projects that matter to me. Debian LTS With the increasing number of paid contributors, easy fixes (CVE with patches available) tend to be processed rather quickly. All the package I worked on had issues that were open for a long time because they were hard to handle. I prepared DLA-613-1 fixing 3 CVE on roundcube. The fix required to manually backport the CRSF handling code which was not available in the wheezy version. I spent almost 8 hours on roundcube. Then I started to work on tiff3. I reviewed many CVE: CVE-2016-3658, CVE-2015-7313, CVE-2015-7554, CVE-2015-8668, CVE-2016-5318, CVE-2016-3625, CVE-2016-5319. I updated their status for tiff3 in wheezy, requested reproducer files to people who reported the CVE when the files were not publicly available and made sure that everything was recorded in the upstream bug tracker. The 4.25 hours I spent on the package were not enough to work on patches, so I put the package back in the work queue. GNOME 3.22 transition I uploaded a new gnome-shell-timer that would work with GNOME 3.21 that had been uploaded to sid. Unfortunately, that new GNOME (and GTK+) version caused many regressions that affected Debian Testing (and thus Kali) users in particular in gnome-control-center. I uploaded a new version fixing some of those issues and I reported a bunch of them to upstream too (#771515, #771517, #771696). Kali I worked on #836211 creating a dpkg patch to work-around the overlayfs limitation (we use it in Kali because persistence of live system relies on overlayfs) and I contacted the upstream overlayfs maintainer to hopefully get a proper fix on the overlayfs side instead. I uploaded radcli 1.2.6-2.1 to fix RC bug #825121 as the package was removed from testing and openvas depends on it in Kali. As part of the pkg-security team, I sponsored/uploaded acccheck and arp-scan for Marcos Fouces, and p0f 3.09b as well. Misc Debian work Distro Tracker. I tested, fixed and merged Paul Wise s patch integrating multiarch hints into tracker.debian.org (#833623). Debian Handbook. I enabled the new Vietnamese translation on debian-handbook.info and updated all translations with Weblate updates. systemd units for apache2. I prepared systemd units for apache2 which I submitted in #798430. With approval of Stefan Fritsch, I committed my work to the git repository and then uploaded the result in version 2.4.23-5. Hindsight packaging. I first packaged lua-sandbox (#838969) which is a dependency of Hindsight and then Hindsight itself (#838968). In this process, I opened a couple of upstream tickets. PIE by default. I uploaded a new version of cpputest compiled with -fPIC so shat executable linking to its static library can be compiled with -fPIE (#837363, forwarded upstream here). Bugs filed. Bad homepage link in haskell-dice-entropy-conduit. Inconsistent options --onlyscripts and --noscripts in debhelper. pidgin entry in security-support-limited is out of date in debian-security-support. New upstream version (2.0.2) in puppet-lint. Thanks See you next month for a new summary of my activities.

No comment Liked this article? Click here. My blog is Flattr-enabled.

2 November 2015

Lunar: Reproducible builds: week 27 in Stretch cycle

What happened in the reproducible builds effort this week: Toolchain fixes Packages fixed The following packages became reproducible due to changes in their build dependencies: maven-plugin-tools, norwegian, ocaml-melt, python-biom-format, rivet. The following packages became reproducible after getting fixed: Some uploads fixed some reproducibility issues but not all of them: The following package is currently failing to build from source but should now be reproducible: Patches submitted which have not made their way to the archive yet: reproducible.debian.net A quick update on current statistics: testing is at 85% of packages tested reproducible with our modified packages, unstable on armhf caught up with amd64 with 80%. The schroot name used for running diffoscope when testing OpenWrt, NetBSD, Coreboot, and Arch Linux has been fixed. (h01ger, Mattia Rizzolo) Documentation update Paul Gevers documented timestamps in unit files created by the Free Pascal Compiler. reproducible-builds.org is now live. It contains a comprehensive documentation on all aspects that have been identified so far of what we call reproducible builds . It makes room for pointers to projects working on reproducible builds, news, dedicated tools, and community events. Package reviews 206 reviews have been removed, 171 added and 196 updated this week. Chris Lamb reported 28 failing to build from source issues. New issues identified this week: timestamps_in_pdf_content, different_encoding_in_html_by_docbook_xsl, timestamps_in_ppu_generated_by_fpc, method_may_never_be_called_in_documentation_generated_by_javadoc. Misc. Andrei Borzenkov has proposed a fix for uninitialized memory in GRUB's mkimage. Uninitialized memory is one source of hard to track down reproducibility errors. Holger Levsen presented the efforts on reproduible builds at Festival de Software Libre in Puerto Vallarta, Mexico.

27 October 2015

Lunar: Reproducible builds: week 26 in Stretch cycle

What happened in the reproducible builds effort this week: Toolchain fixes Mattia Rizzolo created a bug report to continue the discussion on storing cryptographic checksums of the installed .deb in dpkg database. This follows the discussion that happened in June and is a pre-requisite to add checksums to .buildinfo files. Niko Tyni identified why the Vala compiler would generate code in varying order. A better patch than his initial attempt still needs to be written. Packages fixed The following 15 packages became reproducible due to changes in their build dependencies: alt-ergo, approx, bin-prot, caml2html, coinst, dokujclient, libapreq2, mwparserfromhell, ocsigenserver, python-cryptography, python-watchdog, slurm-llnl, tyxml, unison2.40.102, yojson. The following packages became reproducible after getting fixed: Some uploads fixed some reproducibility issues but not all of them: reproducible.debian.net pbuilder has been updated to version 0.219~bpo8+1 on all eight build nodes. (Mattia Rizzolo, h01ger) Packages that FTBFS but for which no open bugs have been recorded are now tested again after 3 days. Likewise for depwait packages. (h01ger) Out of disk situations will not cause IRC notifications anymore. (h01ger) Documentation update Lunar continued to work on writing documentation for the future reproducible-builds.org website. Package reviews 44 reviews have been removed, 81 added and 48 updated this week. Chris West and Chris Lamb identified 70 fail to build from source issues. Misc. h01ger presented the project in Mexico City at the 3er Congreso de Seguridad de la Informaci n where it became clear that we lack academic papers related to reproducible builds. Bryan has been doing hard work to improve reproducibility for OpenWrt. He wrote a report linking to the patches and test results he published.

1 September 2015

Lunar: Reproducible builds: week 18 in Stretch cycle

What happened in the reproducible builds effort this week: Toolchain fixes Aur lien Jarno uploaded glibc/2.21-0experimental1 which will fix the issue were locales-all did not behave exactly like locales despite having it in the Provides field. Lunar rebased the pu/reproducible_builds branch for dpkg on top of the released 1.18.2. This made visible an issue with udebs and automatically generated debug packages. The summary from the meeting at DebConf15 between ftpmasters, dpkg mainatainers and reproducible builds folks has been posted to the revelant mailing lists. Packages fixed The following 70 packages became reproducible due to changes in their build dependencies: activemq-activeio, async-http-client, classworlds, clirr, compress-lzf, dbus-c++, felix-bundlerepository, felix-framework, felix-gogo-command, felix-gogo-runtime, felix-gogo-shell, felix-main, felix-shell-tui, felix-shell, findbugs-bcel, gco, gdebi, gecode, geronimo-ejb-3.2-spec, git-repair, gmetric4j, gs-collections, hawtbuf, hawtdispatch, jack-tools, jackson-dataformat-cbor, jackson-dataformat-yaml, jackson-module-jaxb-annotations, jmxetric, json-simple, kryo-serializers, lhapdf, libccrtp, libclaw, libcommoncpp2, libftdi1, libjboss-marshalling-java, libmimic, libphysfs, libxstream-java, limereg, maven-debian-helper, maven-filtering, maven-invoker, mochiweb, mongo-java-driver, mqtt-client, netty-3.9, openhft-chronicle-queue, openhft-compiler, openhft-lang, pavucontrol, plexus-ant-factory, plexus-archiver, plexus-bsh-factory, plexus-cdc, plexus-classworlds2, plexus-component-metadata, plexus-container-default, plexus-io, pytone, scolasync, sisu-ioc, snappy-java, spatial4j-0.4, tika, treeline, wss4j, xtalk, zshdb. The following packages became reproducible after getting fixed: Some uploads fixed some reproducibility issues but not all of them: Patches submitted which have not made their way to the archive yet: Chris Lamb also noticed that binaries shipped with libsilo-bin did not work. Documentation update Chris Lamb and Ximin Luo assembled a proper specification for SOURCE_DATE_EPOCH in the hope to convince more upstreams to adopt it. Thanks to Holger it is published under a non-Debian domain name. Lunar documented easiest way to solve issues with file ordering and timestamps in tarballs that came with tar/1.28-1. Some examples on how to use SOURCE_DATE_EPOCH have been improved to support systems without GNU date. reproducible.debian.net armhf is finally being tested, which also means the remote building of Debian packages finally works! This paves the way to perform the tests on even more architectures and doing variations on CPU and date. Some packages even produce the same binary Arch:all packages on different architectures (1, 2). (h01ger) Tests for FreeBSD are finally running. (h01ger) As it seems the gcc5 transition has cooled off, we schedule sid more often than testing again on amd64. (h01ger) disorderfs has been built and installed on all build nodes (amd64 and armhf). One issue related to permissions for root and unpriviliged users needs to be solved before disorderfs can be used on reproducible.debian.net. (h01ger) strip-nondeterminism Version 0.011-1 has been released on August 29th. The new version updates dh_strip_nondeterminism to match recent changes in debhelper. (Andrew Ayer) disorderfs disorderfs, the new FUSE filesystem to ease testing of filesystem-related variations, is now almost ready to be used. Version 0.2.0 adds support for extended attributes. Since then Andrew Ayer also added support to reverse directory entries instead of shuffling them, and arbitrary padding to the number of blocks used by files. Package reviews 142 reviews have been removed, 48 added and 259 updated this week. Santiago Vila renamed the not_using_dh_builddeb issue into varying_mtimes_in_data_tar_gz_or_control_tar_gz to align better with other tag names. New issue identified this week: random_order_in_python_doit_completion. 37 FTBFS issues have been reported by Chris West (Faux) and Chris Lamb. Misc. h01ger gave a talk at FrOSCon on August 23rd. Recordings are already online. These reports are being reviewed and enhanced every week by many people hanging out on #debian-reproducible. Huge thanks!

25 August 2015

Lunar: Reproducible builds: week 17 in Stretch cycle

A good amount of the Debian reproducible builds team had the chance to enjoy face-to-face interactions during DebConf15.
Names in red and blue were all present at DebConf15
Picture of the  reproducible builds  talk during DebConf15
Hugging people with whom one has been working tirelessly for months gives a lot of warm-fuzzy feelings. Several recorded and hallway discussions paved the way to solve the remaining issues to get reproducible builds part of Debian proper. Both talks from the Debian Project Leader and the release team mentioned the effort as important for the future of Debian. A forty-five minutes talk presented the state of the reproducible builds effort. It was then followed by an hour long roundtable to discuss current blockers regarding dpkg, .buildinfo and their integration in the archive. Picture of the  reproducible builds  roundtable during DebConf15 Toolchain fixes Reiner Herrmann submitted a patch to make rdfind sort the processed files before doing any operation. Chris Lamb proposed a new patch for wheel implementing support for SOURCE_DATE_EPOCH instead of the custom WHEEL_FORCE_TIMESTAMP. akira sent one making man2html SOURCE_DATE_EPOCH aware. St phane Glondu reported that dpkg-source would not respect tarball permissions when unpacking under a umask of 002. After hours of iterative testing during the DebConf workshop, Sandro Knau created a test case showing how pdflatex output can be non-deterministic with some PNG files. Packages fixed The following 65 packages became reproducible due to changes in their build dependencies: alacarte, arbtt, bullet, ccfits, commons-daemon, crack-attack, d-conf, ejabberd-contrib, erlang-bear, erlang-cherly, erlang-cowlib, erlang-folsom, erlang-goldrush, erlang-ibrowse, erlang-jiffy, erlang-lager, erlang-lhttpc, erlang-meck, erlang-p1-cache-tab, erlang-p1-iconv, erlang-p1-logger, erlang-p1-mysql, erlang-p1-pam, erlang-p1-pgsql, erlang-p1-sip, erlang-p1-stringprep, erlang-p1-stun, erlang-p1-tls, erlang-p1-utils, erlang-p1-xml, erlang-p1-yaml, erlang-p1-zlib, erlang-ranch, erlang-redis-client, erlang-uuid, freecontact, givaro, glade, gnome-shell, gupnp, gvfs, htseq, jags, jana, knot, libconfig, libkolab, libmatio, libvsqlitepp, mpmath, octave-zenity, openigtlink, paman, pisa, pynifti, qof, ruby-blankslate, ruby-xml-simple, timingframework, trace-cmd, tsung, wings3d, xdg-user-dirs, xz-utils, zpspell. The following packages became reproducible after getting fixed: Uploads that might have fixed reproducibility issues: Some uploads fixed some reproducibility issues but not all of them: Patches submitted which have not made their way to the archive yet: St phane Glondu reported two issues regarding embedded build date in omake and cduce. Aur lien Jarno submitted a fix for the breakage of make-dfsg test suite. As binutils now creates deterministic libraries by default, Aur lien's patch makes use of a wrapper to give the U flag to ar. Reiner Herrmann reported an issue with pound which embeds random dhparams in its code during the build. Better solutions are yet to be found. reproducible.debian.net Package pages on reproducible.debian.net now have a new layout improving readability designed by Mattia Rizzolo, h01ger, and Ulrike. The navigation is now on the left as vertical space is more valuable nowadays. armhf is now enabled on all pages except the dashboard. Actual tests on armhf are expected to start shortly. (Mattia Rizzolo, h01ger) The limit on how many packages people can schedule using the reschedule script on Alioth has been bumped to 200. (h01ger) mod_rewrite is now used instead of JavaScript for the form in the dashboard. (h01ger) Following the rename of the software, debbindiff has mostly been replaced by either diffoscope or differences in generated HTML and IRC notification output. Connections to UDD have been made more robust. (Mattia Rizzolo) diffoscope development diffoscope version 31 was released on August 21st. This version improves fuzzy-matching by using the tlsh algorithm instead of ssdeep. New command line options are available: --max-diff-input-lines and --max-diff-block-lines to override limits on diff input and output (Reiner Herrmann), --debugger to dump the user into pdb in case of crashes (Mattia Rizzolo). jar archives should now be detected properly (Reiner Herrman). Several general code cleanups were also done by Chris Lamb. strip-nondeterminism development Andrew Ayer released strip-nondeterminism version 0.010-1. Java properties file in jar should now be detected more accurately. A missing dependency spotted by St phane Glondu has been added. Testing directory ordering issues: disorderfs During the reproducible builds workshop at DebConf, participants identified that we were still short of a good way to test variations on filesystem behaviors (e.g. file ordering or disk usage). Andrew Ayer took a couple of hours to create disorderfs. Based on FUSE, disorderfs in an overlay filesystem that will mount the content of a directory at another location. For this first version, it will make the order in which files appear in a directory random. Documentation update Dhole documented how to implement support for SOURCE_DATE_EPOCH in Python, bash, Makefiles, CMake, and C. Chris Lamb started to convert the wiki page describing SOURCE_DATE_EPOCH into a Freedesktop-like specification in the hope that it will convince more upstream to adopt it. Package reviews 44 reviews have been removed, 192 added and 77 updated this week. New issues identified this week: locale_dependent_order_in_devlibs_depends, randomness_in_ocaml_startup_files, randomness_in_ocaml_packed_libraries, randomness_in_ocaml_custom_executables, undeterministic_symlinking_by_rdfind, random_build_path_by_golang_compiler, and images_in_pdf_generated_by_latex. 117 new FTBFS bugs have been reported by Chris Lamb, Chris West (Faux), and Niko Tyni. Misc. Some reproducibility issues might face us very late. Chris Lamb noticed that the test suite for python-pykmip was now failing because its test certificates have expired. Let's hope no packages are hiding a certificate valid for 10 years somewhere in their source! Pictures courtesy and copyright of Debian's own paparazzi: Aigars Mahinovs.

3 August 2015

Lunar: Reproducible builds: week 14 in Stretch cycle

What happened in the reproducible builds effort this week: Toolchain fixes akira submitted a patch to make cdbs export SOURCE_DATE_EPOCH. She uploded a package with the enhancement to the experimental reproducible repository. Packages fixed The following 15 packages became reproducible due to changes in their build dependencies: dracut, editorconfig-core, elasticsearch, fish, libftdi1, liblouisxml, mk-configure, nanoc, octave-bim, octave-data-smoothing, octave-financial, octave-ga, octave-missing-functions, octave-secs1d, octave-splines, valgrind. The following packages became reproducible after getting fixed: Some uploads fixed some reproducibility issues but not all of them: In contrib, Dmitry Smirnov improved libdvd-pkg with 1.3.99-1-1. Patches submitted which have not made their way to the archive yet: reproducible.debian.net Four armhf build hosts were provided by Vagrant Cascadian and have been configured to be used by jenkins.debian.net. Work on including armhf builds in the reproducible.debian.net webpages has begun. So far the repository comparison page just shows us which armhf binary packages are currently missing in our repo. (h01ger) The scheduler has been changed to re-schedule more packages from stretch than sid, as the gcc5 transition has started This mostly affects build log age. (h01ger) A new depwait status has been introduced for packages which can't be built because of missing build dependencies. (Mattia Rizzolo) debbindiff development Finally, on August 31st, Lunar released debbindiff 27 containing a complete overhaul of the code for the comparison stage. The new architecture is more versatile and extensible while minimizing code duplication. libarchive is now used to handle cpio archives and iso9660 images through the newly packaged python-libarchive-c. This should also help support a couple other archive formats in the future. Symlinks and devices are now properly compared. Text files are compared as Unicode after being decoded, and encoding differences are reported. Support for Sqlite3 and Mono/.NET executables has been added. Thanks to Valentin Lorentz, the test suite should now run on more systems. A small defiency in unquashfs has been identified in the process. A long standing optimization is now performed on Debian package: based on the content of the md5sums control file, we skip comparing files with matching hashes. This makes debbindiff usable on packages with many files. Fuzzy-matching is now performed for files in the same container (like a tarball) to handle renames. Also, for Debian .changes, listed files are now compared without looking the embedded version number. This makes debbindiff a lot more useful when comparing different versions of the same package. Based on the rearchitecturing work has been done to allow parallel processing. The branch now seems to work most of the time. More test needs to be done before it can be merged. The current fuzzy-matching algorithm, ssdeep, has showed disappointing results. One important use case is being able to properly compare debug symbols. Their path is made using the Build ID. As this identifier is made with a checksum of the binary content, finding things like CPP macros is much easier when a diff of the debug symbols is available. Good news is that TLSH, another fuzzy-matching algorithm, has been tested with much better results. A package is waiting in NEW and the code is ready for it to become available. A follow-up release 28 was made on August 2nd fixing content label used for gzip2, bzip2 and xz files and an error on text files only differing in their encoding. It also contains a small code improvement on how comments on Difference object are handled. This is the last release name debbindiff. A new name has been chosen to better reflect that it is not a Debian specific tool. Stay tuned! Documentation update Valentin Lorentz updated the patch submission template to suggest to write the kind of issue in the bug subject. Small progress have been made on the Reproducible Builds HOWTO while preparing the related CCCamp15 talk. Package reviews 235 obsolete reviews have been removed, 47 added and 113 updated this week. 42 reports for packages failing to build from source have been made by Chris West (Faux). New issue added this week: haskell_devscripts_locale_substvars. Misc. Valentin Lorentz wrote a script to report packages tested as unreproducible installed on a system. We encourage everyone to run it on their systems and give feedback!

24 August 2012

Olivier Berger: Generating RDF description of Debian package sources with ADMS.SW

Edit : I ve now managed to roll out my contribution which is now in production on packages.qa.debian.org. See a later post I ve made on the subject, and beware that the generated RDF has changed a bit also. ADMS.SW proposes specifications for description of software present in software catalogues. I ve tried and apply it to the contents of the Debian Package Tracking System (PTS), using transformation of the information known by the PTS to RDF+XML. The result is not yet in production, but here s an example of what can be done, using the Turtle syntax (more readable) :
<style type="text/css"> .T1 color:#000000; font-size:14pt; font-weight:bold; .T2 color:#000000; font-size:12pt; .T3 color:#000000; font-size:8.5pt; .T4 color:#0000ff; font-size:8.5pt; .T5 color:#a020f0; font-size:8.5pt; .T6 color:#228b22; font-size:8.5pt; .T7 color:#b22222; font-size:8.5pt; .T8 color:#008b8b; font-size:8.5pt; .T9 color:#8b2252; font-size:8.5pt; .dp1 .dp2 </style> @base <http://packages.qa.debian.org/> .
@prefix rdf: <http://www.w3.org/1999/02/22 rdf syntax ns#> .
@prefix admssw: <http://purl.org/adms/sw/> .
@prefix doap: <http://usefulinc.com/ns/doap#> .
@prefix dcterms: <http://purl.org/dc/terms/> .
@prefix rdfs: <http://www.w3.org/2000/01/rdf schema#> .
@prefix foaf: <http://xmlns.com/foaf/0.1/> .
@prefix schema: <http://schema.org/> .
@prefix spdx: <http://www.spdx.org/rdf/terms#> .
@prefix : <http://www.w3.org/1999/xhtml> .
@prefix str: <http://exslt.org/strings> . #
# First the packaging project for apache2 in Debian
#
# <http://packages.qa.debian.org/apache2> resource :
<apache2>
a admssw:SoftwareProject ;
doap:name "apache2" ;
doap:description "Debian apache2 source packaging" ;
doap:homepage "http://packages.debian.org/src:apache2" ;
schema:contributor [
a foaf:OnlineAccount ;
foaf:accountName "Debian Apache Maintainers" ;
foaf:accountServiceHomepage <http://qa.debian.org/developer.php?login=debian apache@lists.debian.org>
], [
a foaf:OnlineAccount ;
foaf:accountName "Stefan Fritsch" ;
foaf:accountServiceHomepage <http://qa.debian.org/developer.php?login=sf@debian.org>
], [
a foaf:OnlineAccount ;
foaf:accountName "Steinar H. Gunderson" ;
foaf:accountServiceHomepage <http://qa.debian.org/developer.php?login=sesse@debian.org>
], [
a foaf:OnlineAccount ;
foaf:accountName "Arno T ll" ;
foaf:accountServiceHomepage <http://qa.debian.org/developer.php?login=arno@debian.org>
] ;
# pointer to the release in the different suites :
doap:release <apache2_2.2.16 6+squeeze7>, <apache2_2.2.22 11>, <apache2_2.4.2 2>. #
# Now the different debian package source releases
#
# <http://packages.qa.debian.org/apache2_2.2.16 6+squeeze7> resource
<apache2_2.2.16 6+squeeze7>
a admssw:SoftwareRelease ;
rdfs:label "apache2 2.2.16 6+squeeze7" ;
admssw:project <apache2> ;
dcterms:description "Debian apache2 source package version 2.2.16 6+squeeze7" ;
doap:revision "2.2.16 6+squeeze7" . # This one is the reference version for the PTS as in unstable, so
# contains more details than the others
<apache2_2.2.22 11>
a admssw:SoftwareRelease ;
rdfs:label "apache2 2.2.22 11" ;
admssw:project <apache2> ;
dcterms:description "Debian apache2 source package version 2.2.22 11" ;
doap:revision "2.2.22 11" ;
# this release contains two components
admssw:includedAsset <apache2/apache2_2.2.22 11_debian>, <apache2/apache2_2.2.22_orig>
;
# this release can be downloaded as one package (with dget)
admssw:package <apache2/apache2_2.2.22 11.dsc> ;
# it also has a related release somewhere else
dcterms:relation <https://launchpad.net/ubuntu/+source/apache2/2.2.22 6ubuntu2> .
<apache2_2.4.2 2>
a admssw:SoftwareRelease ;
rdfs:label "apache2 2.4.2 2" ;
dcterms:description "Debian apache2 source package version 2.4.2 2" ; admssw:project <apache2> ;
doap:revision "2.4.2 2" . # Then the .dsc file for the current unstable version
<apache2/apache2_2.2.22 11.dsc>
a admssw:SoftwarePackage ;
dcterms:description "Debian source package descriptor file for apache2 version 2.2.22 11";
schema:downloadUrl "http://cdn.debian.net/debian/pool/main/a/apache2/apache2_2.2.22 11.dsc";
schema:fileSize "2885" ;
spdx:checksum [
a spdx:Checksum ;
spdx:algorithm <apache2#checksumAlgorithm_md5sum> ;
spdx:checksumValue "d7d03719b9f6432beeecd3aa04f7b22c"
] . #
# Then the upstream project
#
<apache2/apache2_orig>
a admssw:SoftwareProject ;
doap:description "The apache2 upstream project" ;
# either its name or homepage can be matched against other ADMS.SW
# or DOAP descriptors
doap:name "apache2" ;
doap:homepage "http://httpd.apache.org/" . # And a upstream release 2.2.22
<apache2/apache2_2.2.22_orig>
a admssw:SoftwareRelease ;
rdfs:label "Upstream apache2 release 2.2.22" ;
dcterms:description "Upstream source release for apache2 version 2.2.22" ;
doap:revision "2.2.22" ;
admssw:project <apache2/apache2_orig> ;
# and a link to the upstream source tarball s description
admssw:package <apache2/apache2_2.2.22.orig.tar.gz> . # now the (potentially re archived) upstream source archive for that release
<apache2/apache2_2.2.22.orig.tar.gz>
a admssw:SoftwarePackage ;
dcterms:description "Upstream source archive for apache2 version 2.2.22 11 (potentially re archived by Debian)";
schema:downloadUrl "http://cdn.debian.net/debian/pool/main/a/apache2/apache2_2.2.22.orig.tar.gz";
# Those 2 bits should help match packagings of the same tarball if needed
schema:fileSize "7200529" ;
spdx:checksum [
a spdx:Checksum ;
spdx:algorithm <apache2#checksumAlgorithm_md5sum> ;
spdx:checksumValue "d77fa5af23df96a8af68ea8114fa6ce1"
] . #
# Now, document specific details of the second component : Debian
# packaging files archive
#
<apache2/apache2_2.2.22 11_debian>
a admssw:SoftwareRelease ;
rdfs:label "Package sources apache2_2.2.22 11" ;
dcterms:description "Debian packaging sources for apache2 version 2.2.22 11" ;
doap:revision "2.2.22 11" ;
admssw:package <apache2/apache2_2.2.22 11.debian.tar.gz> . <apache2/apache2_2.2.22 11.debian.tar.gz>
a admssw:SoftwarePackage ;
dcterms:description "Debian source package files archive for apache2 version 2.2.22 11" ;
schema:downloadUrl "http://cdn.debian.net/debian/pool/main/a/apache2/apache2_2.2.22 11.debian.tar.gz";
schema:fileSize "195980" ;
spdx:checksum [
a spdx:Checksum ;
spdx:algorithm <apache2#checksumAlgorithm_md5sum> ;
spdx:checksumValue "d3d4146ccad51129636b7dbff284a110"
] . #
# Now, we also weave links with the Ubuntu counterparts
#
<https://launchpad.net/ubuntu/+source/apache2>
a admssw:SoftwareProject ;
doap:description "\"apache2\" source package in Ubuntu" ;
# this one promises some trolls and flames
admssw:forkOf <apache2> ;
doap:homepage "https://launchpad.net/ubuntu/+source/apache2" ;
doap:release <https://launchpad.net/ubuntu/+source/apache2/2.2.22 6ubuntu2> . # and its release known by the PTS
<https://launchpad.net/ubuntu/+source/apache2/2.2.22 6ubuntu2>
a admssw:SoftwareRelease ;
rdfs:label "apache2 2.2.22 6ubuntu2" ;
dcterms:description "\"apache2\" 2.2.22 6ubuntu2 source package in Ubuntu" ;
doap:revision "2.2.22 6ubuntu2" ;
admssw:project <https://launchpad.net/ubuntu/+source/apache2> .

4 October 2010

Rapha&#235;l Hertzog: Can Debian offer a Constantly Usable Testing distribution?

Debian s testing distribution is where Debian developers prepare the next stable distribution. While this is still its main purpose, many users have adopted this version of Debian because it offers them a good trade-off between stability and freshness. But there are downsides to using this distribution and the Constantly Usable Testing (CUT) project aims to resolve those. This article will present the project and the challenges involved to make it happen. About Debian unstable & testing Debian unstable is the distribution where developers upload new versions of their packages. It happens frequently that some packages are not installable due to changes in other packages or due to transitions not yet completed. Debian testing, on the contrary, is managed by a tool that ensures the consistency of the whole distribution: it picks updates from unstable only if the package has been enough tested (10 days usually), if it s free of new release-critical bugs, if it s available on all supported architectures, and if it doesn t break any other package already present in testing. The Release Team (RT) controls this tool and provide hints to help it find a set of packages that can flow from unstable to testing. Those rules also ensure that the packages that flow into testing are reasonably free of show-stopper bugs (like a system that doesn t boot, or X that doesn t work at all). This makes it very attractive to users who like to regularly get new upstream versions of their software without dealing with the biggest problems associated to them. This is all very attractive, yet several Debian developers advise people to not use testing. Why is that? Known problems with testing Disappearing software The release team use this distribution to prepare the next stable release and from time to time they remove packages from it. Either because it s needed to ensure that other packages can migrate from unstable to testing, or because they have long-standing release-critical bugs without progress towards a resolution. It also happens that they remove packages on request of the maintainers because they believe that the current version of the software cannot be supported (security-wise) for 2 years or more. The security team also regularly issues such requests. Long delays for security and important fixes Despite the 10-day delay in unstable, there are always some annoying bugs (and security bugs are no exceptions) that are only discovered when the package already has migrated to testing. The maintainer might be quick to upload a fixed package in unstable, and might even raise the urgency to allow the package to migrate sooner, but if the packages gets entangled in a large ongoing transition, it will not migrate before the transition is completed. Sometimes it can take weeks for that to happen. This delay can be avoided by doing direct uploads to testing (through testing-proposed-updates) but this is almost never used, except during a freeze, where targeted bugfixes are the norm. Not always installable With testing evolving daily, updates sometimes break the last installation images available (in particular netboot images that get everything from the network). The debian-installer (d-i) packages are usually quickly fixed but they don t move to testing automatically because the new combination of d-i packages has not necessarily been validated yet. Colin Watson sums up the problem:
Getting new installer code into testing takes too long, and problems remain unfixed in testing for too long. [...] The problem with d-i development at the moment is more that we re very slow at producing new d-i *releases*. [...] Your choices right now are to work with stable (too old), testing (would be nice except for the way sometimes it breaks and then it tends to take a week to fix anything), unstable (breaks all the time).
CUT s history CUT finds its root in an old proposal by Joey Hess: it introduces the idea that the stable release is not Debian s sole product and that testing could become with some work a suitable choice for end-users. Nobody took on that work and there was no visible progress in the last 3 years. But recently Joey brought up CUT again on the debian-devel mailing list and Stefano Zacchiroli (the Debian project leader) challenged him to setup a BoF on CUT for Debconf10. It turned out to be one of the most heavily attended BoF (video recording is here), there is clearly a lot of interest in the topic. There s now a dedicated wiki and an Alioth project with a mailing list. The rest of this article tries to summarize the various options discussed and how they re supposed to address the problems identified. The ideas behind CUT Among all the ideas, there are two main approaches that have been discussed. The first is to regularly snapshot testing at points where it is known to work reasonably well (those snapshots would be named cut ). The second is to build an improved testing distribution tailored to the needs of users who want a working distribution with daily updates, its name would be rolling . Regular snapshots of testing There s general agreement that regular snapshots of testing are required: it s the only way to ensure that the generated installation media will continue to work until the next snapshot. If tests of the snapshot do not reveal any major problem, then it becomes the latest cut . For clarity, the official codename would be date based: e.g. cut-2010-09 would be the cut taken during September 2010. While the frequency has not been fixed yet, the goal is clearly to be on the aggressive side: at the very least every 6 months, but every month has been suggested as well. In order to reach a decision, many aspects have to be balanced. One of them (and possibly the most important) is the security support. Given that the security team is already overworked, it s difficult to put more work on their shoulders by declaring that cuts will be supported like any stable release. No official security support sounds bad but it s not necessarily so problematic as one might imagine. Testing s security record is generally better than stable s one (see the security tracker) because fixes flow in naturally with new upstream versions. Stable still get fixes for very important security issues sooner than testing, but on the whole there are less known security-related problems in testing than in stable. Since it s only a question of time until the fixed version comes naturally from upstream, more frequent cut releases means that users get security fixes sooner. But Stefan Fritsch, who used to be involved in the Debian testing security team, has also experienced the downside for anyone who tries to contribute security updates:
The updates to testing-security usually stay useful only for a few weeks, until a fixed version migrates from unstable. In stable, the updates stay around for a few years, which gives a higher motivation to spend time on preparing them.
So if it s difficult to form a dedicated security team, the work of providing security updates comes back to the package maintainer. They are usually quite quick to upload fixed packages in unstable but tend to not monitor whether the packages migrate to testing. They can t be blamed because testing was created to prepare the next stable release and there is thus no urgency to get the fix in as long as it makes it before the release. CUT can help in this regard precisely because it changes this assumption: there will be users of the testing packages and they deserve to get security fixes much like the stable users. Another aspect to consider when picking a release frequency is the amount of associated work that comes with any official release: testing upgrades from the previous version, writing release notes and preparing installation images. It seems difficult to do this every month. With this frequency it s also impossible to have a new major kernel release for each cut (since they tend to come out only every 2 to 3 months) and the new hardware support that it brings is something worthwhile to many users. In summary, regular snapshots address the not always installable problem and changes the perception of maintainers towards testing, so that hopefully they care more of security updates in that distribution (and in cuts). But they do not solve the problem of disappearing packages. Something else is needed to fix that problem. A new rolling distribution? Lucas Nussbaum pointed out that regular snapshots of Debian is not really a new concept:
How would this differentiate from other distributions doing 6-month release cycles, and in particular Ubuntu, which can already be seen as Debian snapshots (+ added value)?
In Lucas s eyes, CUT becomes interesting if it can provide a rolling distribution (like testing) with a constant flux of new upstream releases . For him, that would be something quite unique in the Free Software world . The snapshots would be used as starting point for the initial installation, but the installed system would point to the rolling distribution and users would then upgrade as often as they want. In this scenario, security support for the snapshots is not so important, what matters is the state of the rolling distribution. If testing were used as the rolling distribution, the problem of disappearing packages would not be fixed. That s why there have been discussions of introducing a new distribution named rolling that would work like testing but with adapted rules, and the cuts would then be snapshots of rolling instead of testing. The basic proposal is to make a copy of testing and to re-add the packages which have been removed because they are not suited for a long term release while they are perfectly acceptable for a constantly updated release (the most recent example being Chromium). Then it s possible to go one step further: during freeze, testing is no longer automatically updated which makes it inappropriate to feed the rolling distribution. That s why rolling would be reconfigured to grab updates from unstable (but using the same rules than testing). Given the frequent releases, it s likely that only a subset of architectures would be officially supported. This is not a real problem because the users who want bleeding edge software tends to be desktop users on mainly i386/amd64 (and maybe armel for tablets and similar mobile products). This choice if made opens up the door to even more possibilities: if rolling is configured exactly like testing but with only a subset of the architectures, it s likely that some packages migrate to rolling before testing when non-mainstream architectures are lagging in terms of auto-building (or have toolchain problems). While being ahead of testing can be positive for the users, it s also problematic on several levels. First, managing rolling becomes much more complicated because the transition management work done by the release team can t be reused as-is. Then it introduces competition between both distributions which can make it more difficult to get a stable release out, for example if maintainers stop caring of the migration to testing once the migration to rolling has been completed. The rolling distribution is certainly a good idea but the rules governing it must be designed to avoid any conflict with the process of releasing a stable distribution. Lastly, the mere existence of rolling would finally fix the marketing problem plaguing testing: the name rolling does not suggest that the software is not yet ready for prime time. Conclusion Whether CUT will be implemented remains to be seen, but it s off for a good start: ftpmaster Joerg Jaspert said that the new archive server can cope with a new distribution, and there s now a proposal shaping up. The project might start quickly: there is already an implementation plan for the snapshot side of the project. The rolling distribution can always be introduced later, once it is ready. Both approaches can complement each other and provide something useful to different kind of users. The global proposal is certainly appealing: it would address the concerns of obsolescence of Debian s stable release by making intermediary releases. Anyone needing something more recent for hardware support can start by installing a cut and follow the subsequent releases until the next stable version. And users who always want the latest version of every software could use rolling after having installed a cut. From a user point of view, there are similarities with the mix of usual and long term releases of Ubuntu. But from the development side, the process followed would be quite different, and the constraints imposed by having a constantly usable distribution are stronger: any wide-scale change must be designed in a way that it can happen progressively in a transparent manner for the user.
This article was first published in Linux Weekly News. If you want to see more articles like this, join Flattr and click on the flattr button below every article that you like.
Flattr this Share/Bookmark 21 comments Support my work

26 July 2008

Philipp Kern: Stable Point Release: Etch 4.0r4 (aka etchnhalf)

Another point release for Etch has been done; now it's the time for the CD team to roll out new images after the next mirror pulse. The official announcements (prepared by Alexander Reichle-Schmehl, thanks!) will follow shortly afterwards. FTP master of the day was Joerg Jaspert, who did his first point release since Woody, as he told us on IRC. We appreciate your work and you spending your time that shortly before going to Argentina. This point release includes the etchnhalf update introducing a new kernel image (based on 2.6.24) and some driver updates. Additionally the infamous openssl hole will be fixed for good, even for new installs. Again I want to present you a list of people who contributed to this release. It cannot be complete as I got the information out of the Changed-by fields of the uploads. From the Release Team we had dann frazier (who drove the important kernel part of etchnhalf), Luk Claes, Neil McGovern, Andreas Barth, Martin Zobel-Helas and me working on it. ;-)