Search Results: "fjp"

11 November 2022

Reproducible Builds: Reproducible Builds in October 2022

Welcome to the Reproducible Builds report for October 2022! In these reports we attempt to outline the most important things that we have been up to over the past month. As ever, if you are interested in contributing to the project, please visit our Contribute page on our website.

Our in-person summit this year was held in the past few days in Venice, Italy. Activity and news from the summit will therefore be covered in next month s report!
A new article related to reproducible builds was recently published in the 2023 IEEE Symposium on Security and Privacy. Titled Taxonomy of Attacks on Open-Source Software Supply Chains and authored by Piergiorgio Ladisa, Henrik Plate, Matias Martinez and Olivier Barais, their paper:
[ ] proposes a general taxonomy for attacks on opensource supply chains, independent of specific programming languages or ecosystems, and covering all supply chain stages from code contributions to package distribution.
Taking the form of an attack tree, the paper covers 107 unique vectors linked to 94 real world supply-chain incidents which is then mapped to 33 mitigating safeguards including, of course, reproducible builds:
Reproducible Builds received a very high utility rating (5) from 10 participants (58.8%), but also a high-cost rating (4 or 5) from 12 (70.6%). One expert commented that a reproducible build like used by Solarwinds now, is a good measure against tampering with a single build system and another claimed this is going to be the single, biggest barrier .
It was noticed this month that Solarwinds published a whitepaper back in December 2021 in order to:
[ ] illustrate a concerning new reality for the software industry and illuminates the increasingly sophisticated threats made by outside nation-states to the supply chains and infrastructure on which we all rely.
The 12-month anniversary of the 2020 Solarwinds attack (which SolarWinds Worldwide LLC itself calls the SUNBURST attack) was, of course, the likely impetus for publication.
Whilst collaborating on making the Cyrus IMAP server reproducible, Ellie Timoney asked why the Reproducible Builds testing framework uses two remarkably distinctive build paths when attempting to flush out builds that vary on the absolute system path in which they were built. In the case of the Cyrus IMAP server, these happened to be: Asked why they vary in three different ways, Chris Lamb listed in detail the motivation behind to each difference.
On our mailing list this month:
The Reproducible Builds project is delighted to welcome openEuler to the Involved projects page [ ]. openEuler is Linux distribution developed by Huawei, a counterpart to it s more commercially-oriented EulerOS.

Debian Colin Watson wrote about his experience towards making the databases generated by the man-db UNIX manual page indexing tool:
One of the people working on [reproducible builds] noticed that man-db s database files were an obstacle to [reproducibility]: in particular, the exact contents of the database seemed to depend on the order in which files were scanned when building it. The reporter proposed solving this by processing files in sorted order, but I wasn t keen on that approach: firstly because it would mean we could no longer process files in an order that makes it more efficient to read them all from disk (still valuable on rotational disks), but mostly because the differences seemed to point to other bugs.
Colin goes on to describe his approach to solving the problem, including fixing various fits of internal caching, and he ends his post with None of this is particularly glamorous work, but it paid off .
Vagrant Cascadian announced on our mailing list another online sprint to help clear the huge backlog of reproducible builds patches submitted by performing NMUs (Non-Maintainer Uploads). The first such sprint took place on September 22nd, but another was held on October 6th, and another small one on October 20th. This resulted in the following progress:
41 reviews of Debian packages were added, 62 were updated and 12 were removed this month adding to our knowledge about identified issues. A number of issue types were updated too. [1][ ]
Lastly, Luca Boccassi submitted a patch to debhelper, a set of tools used in the packaging of the majority of Debian packages. The patch addressed an issue in the dh_installsysusers utility so that the postinst post-installation script that debhelper generates the same data regardless of the underlying filesystem ordering.

Other distributions F-Droid is a community-run app store that provides free software applications for Android phones. This month, F-Droid changed their documentation and guidance to now explicitly encourage RB for new apps [ ][ ], and FC Stegerman created an extremely in-depth issue on GitLab concerning the APK signing block. You can read more about F-Droid s approach to reproducibility in our July 2022 interview with Hans-Christoph Steiner of the F-Droid Project. In openSUSE, Bernhard M. Wiedemann published his usual openSUSE monthly report.

Upstream patches The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of such patches, including:

diffoscope diffoscope is our in-depth and content-aware diff utility. Not only can it locate and diagnose reproducibility issues, it can provide human-readable diffs from many kinds of binary formats. This month, Chris Lamb prepared and uploaded versions 224 and 225 to Debian:
  • Add support for comparing the text content of HTML files using html2text. [ ]
  • Add support for detecting ordering-only differences in XML files. [ ]
  • Fix an issue with detecting ordering differences. [ ]
  • Use the capitalised version of Ordering consistently everywhere in output. [ ]
  • Add support for displaying font metadata using ttx(1) from the fonttools suite. [ ]
  • Testsuite improvements:
    • Temporarily allow the stable-po pipeline to fail in the CI. [ ]
    • Rename the order1.diff test fixture to json_expected_ordering_diff. [ ]
    • Tidy the JSON tests. [ ]
    • Use assert_diff over get_data and an manual assert within the XML tests. [ ]
    • Drop the ALLOWED_TEST_FILES test; it was mostly just annoying. [ ]
    • Tidy the tests/test_source.py file. [ ]
Chris Lamb also added a link to diffoscope s OpenBSD packaging on the diffoscope.org homepage [ ] and Mattia Rizzolo fix an test failure that was occurring under with LLVM 15 [ ].

Testing framework The Reproducible Builds project operates a comprehensive testing framework at tests.reproducible-builds.org in order to check packages and other artifacts for reproducibility. In October, the following changes were made by Holger Levsen:
  • Run the logparse tool to analyse results on the Debian Edu build logs. [ ]
  • Install btop(1) on all nodes running Debian. [ ]
  • Switch Arch Linux from using SHA1 to SHA256. [ ]
  • When checking Debian debstrap jobs, correctly log the tool usage. [ ]
  • Cleanup more task-related temporary directory names when testing Debian packages. [ ][ ]
  • Use the cdebootstrap-static binary for the 2nd runs of the cdebootstrap tests. [ ]
  • Drop a workaround when testing OpenWrt and coreboot as the issue in diffoscope has now been fixed. [ ]
  • Turn on an rm(1) warning into an info -level message. [ ]
  • Special case the osuosl168 node for running Debian bookworm already. [ ][ ]
  • Use the new non-free-firmware suite on the o168 node. [ ]
In addition, Mattia Rizzolo made the following changes:
  • Ensure that 2nd build has a merged /usr. [ ]
  • Only reconfigure the usrmerge package on Debian bookworm and above. [ ]
  • Fix bc(1) syntax in the computation of the percentage of unreproducible packages in the dashboard. [ ][ ][ ]
  • In the index_suite_ pages, order the package status to be the same order of the menu. [ ]
  • Pass the --distribution parameter to the pbuilder utility. [ ]
Finally, Roland Clobus continued his work on testing Live Debian images. In particular, he extended the maintenance script to warn when workspace directories cannot be deleted. [ ]
If you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:

3 December 2016

Ben Hutchings: Linux Kernel Summit 2016, part 1

I attended this year's Linux Kernel Summit in Santa Fe, NM, USA and made notes on some of the sessions that were relevant to Debian. LWN also reported many of the discussions. This is the first of two parts of my notes; part 2 is here. Stable process Jiri Kosina, in his role as a distribution maintainer, sees too many unsuitable patches being backported - e.g. a fix for a bug that wasn't present or a change that depends on an earlier semantic change so that when cherry-picked it still compiles but isn't quite right. He thinks the current review process is insufficient to catch them. As an example, a recent fix for a minor information leak (CVE-2016-9178) depended on an earlier change to page fault handling. When backported by itself, it introduced a much more serious security flaw (CVE-2016-9644). This could have been caught very quickly by a system call fuzzer. Possible solutions: require 'Fixes' field, not just 'Cc: stable'. Deals with 'bug wasn't present', but not semantic changes. There was some disagreement whether 'Fixes' without 'Cc: stable' should be sufficient for inclusion in stable. Ted Ts'o said he specifically does that in some cases where he thinks backporting is risky. Greg Kroah-Hartman said he takes it as a weaker hint for inclusion in stable. Is it a good idea to keep 'Cc: stable' given the risk of breaking embargo? On balance, yes, it only happened once. Sometimes it's hard to know exactly how/when the bug was introduced. Linus doesn't want people to guess and add incorrect 'Fixes' fields. There is still the option to give some explanation and hints for stable maintainers in the commit message. Ideally the upstream developer should provide a test case for the bug. Is Linus happy? Linus complained about minor fixes coming later in the release cycle. After rc2, all fixes should either be for new code introduced in the current release cycle or for important bugs. However, new, production-ready drivers without new infrastructure dependencies are welcome at almost any point in the release cycle. He was unhappy about some big changes in RDMA, but I'm not sure what those were. Bugzilla and bug tracking Laura Abbott started a discussion of bugzilla.kernel.org, talking about subsystems where maintainers ignore it and any responses come from random people giving bad advice. This is a terrible experience for users. Several maintainers are actively opposed to using it, and the email bridge no longer works (or not well?). She no longer recommends Fedora bug submitters to submit reports there. Are there any alternatives? None were proposed. Someone asked whether Bugzilla could tell reporters to use email for certain products/components instead of continuing with the bug entry process. Konstantin Ryabitsev talked about the difficulty of upgrading a customised instance of Bugzilla. Much customisation requires patches which don't apply to next version (maybe due to limitations of the extension mechanism?). He has had to drop many such patches. Email is hard to track when a bug is handed over from one maintainer to another. Email archives are very unreliable. Linus: I'll take Bugzilla over mail-archive. No-one is currently keeping track of bugs across the kernel and making sure they get addressed by an appropriate maintainer. It's (at least) a full-time job but no individual company has business case for paying for this. Konstantin suggested (I think) that CII might pay for this. There was some discussion of what information should be included in a bug report. The Cut here line in oops messages was said to be a mistake because there are often relevant messages before it. The model of computer is often important. Beyond that, there was not much interest in the automated information gathering that distributions do. Distribution maintainers should curate bugs before forwarding upstream. There was a request for custom fields per component in Bugzilla. Konstantin says this is doable (possibly after upgrade to version 5); it doesn't require patches. The future of the Kernel Summit The kernel community is growing, and the invitation list for the core day is too small to include all the right people for technical subjects. For 2017, the core half-day will have an even smaller invitation list, only ~30 subsystem maintainers that Linus pulls from. The entire technical track will be open (I think). Kernel Summit 2017 and some mini-summits will be held in Prague alongside Open Source Summit Europe (formerly LinuxCon Europe) and Embedded Linux Conference Europe. There were some complaints that LinuxCon is not that interesting to kernel developers, compared to Linux Plumbers Conference (which followed this year's Kernel Summit). However, the Linux Foundation is apparently soliciting more hardcore technical sessions. Kernel Summit and Linux Plumbers Conference are quite small, and it's apparently hard to find venues for them in cities that also have major airports. It might be more practical to co-locate them both with Open Source Summit in future. time_t and 2038 On 32-bit architectures the kernel's representation of real time (time_t etc.) will break in early 2038. Fixing this in a backward-compatible way is a complex problem. Arnd Bergmann presented the current status of this process. There has not yet been much progress in mainline, but more fixes have been prepared. The changes to struct inode and to input events are proving to be particularly problematic. There is a need to add new system calls, and he intends to add these for all (32-bit) achitectures at once. Copyright retention James Bottomley talked about how developers can retain copyright on their contributions. It's hard to renegotiate within an existing employment; much easier to do this when preparing to sign a new contract. Some employers expect you to fill in a document disclosing 'prior inventions' you have worked on. Depending on how it's worded, this may require the employer to negotiate with you again whenever they want you to work on that same software. It's much easier for contractors to retain copyright on their work - customers expect to have a custom agreement and don't expect to get copyright on contractor's software.

31 March 2014

John Goerzen: Springtime in the Mountains

The scene: early one morning as the sun has just started to rise. Jacob and Oliver, ages 7 and 4, are the first people to wake up in the house their grandparents in California, where the four of us are visiting for the first time as a family. They have a conversation and decide that would be a good to go find a mystery. They decide to take their flashlights pink and blue, matching each boy s favorite color and slowly, but not very quietly, open their bedroom door and creep out. Brother, you forgot your flashlight! says Oliver. Oh, thanks brother! I ll get it! says Jacob. Meanwhile, Laura s mom wakes up, and notices two boys with flashlights creeping through the living room. Pretty soon they reach the kitchen, open the dishwasher, spy a suspicious-looking bowl, and decide that they have found the mystery a clean bowl! Or, at least that s the story that I pieced together based on what a 4-year-old and the grandma he awakened told me. We were on our first family trip to the Fresno, California area, to visit Laura s parents Jacob and Oliver s new grandparents. They ve played together before, but as this was our first visit to their place, there was quite the excitement. The boys had flown before, but it was several years ago and neither of them remember it well, so they were excited about that, too. The night before, Jacob woke up to tell me Dad, I am too excited to sleep. I think I will go downstairs and watch some TV. He didn t get too far with that plan. But he was excited. We went through security at the laid-back Wichita airport (where the TSA agents smile and there are often no security lines at all). We found our gate with enough time to grab lunch, which we did. The boys and I then did what we often do to kill time: explore. We explored the terminal, watching carpet-layers cut out carpet for the jetway, watching the construction of the new Terminal 3 out the window. And, of course, watching airplanes take off and land from the terminal s large windows. Finally it was our turn to board, and we all got on the plane: Jacob and Oliver with their backpacks of on-board activites, Laura and me with the rest of our carryon luggage, for the short trip to Denver. Jacob and Oliver s noses were pressed against the windows. Or, well, Jacob s was. Oliver s window was a little too high for him, but he was thrilled anyhow. They delighted in the airplane snacks, and the fact that they were allowed to drink pop on the plane. We packed books and some new art supplies for them (colored post-its, pages from a train-themed page-a-day calendar, a notebook, and a set of colored pens really seemed to do the trick.) We had a choice of 35 minutes or 4 hours between flights in Denver, and I had chosen 4 hours, thinking that would be a lot less stressful with boys. And it was. We found a nice corner of the mezzanine to sit for awhile they did art projects and played a game with Laura. Then I took them exploring Denver. We rode the moving sidewalks up and down the terminal, took a train ride to another terminal and back, ate supper all together, and flew to Fresno. We had stopped in the Wichita airport to buy them each a souvenir airplane, and these came out often during the rest of the trip. They enjoyed the mockups of the sequoias in Fresno Yosemite International Airport, enjoyed their beds and their room at the house, and did actually manage to fall asleep eventually. We had a few days there, where they played in a park, with bubbles on the patio, or croquet in the yard (I even discovered Jacob happily using the cast his broken arm is in as a hammer to pound the hoops into the ground!) There are a lot of miniatures in the house, and the boys enjoyed exploring the dollhouses and especially the N-gauge model train. Jacob enjoyed it so much he asked me to record a video of him playing with the trains. Evenings often brought book-reading, from the many children s books in the house. At home, Laura and I and both boys often scrunch onto an oversized chair and read a book and sing a song (one I make up on whatever topic they choose). Over there, we often had Laura, Jacob, Oliver, Laura s mom, and me scrunched up somewhere while the boys heard a story read to them by their grandma. That happened plenty of times other than bedtime, too. (Or Jacob would take his favorite books and read them to himself.) Laura s parents organized a reception Sunday for us, for the people from that area that couldn t make it to our wedding. Jacob and Oliver, predictably, had fun playing and even talked to some of the adults. The adults that didn t ask Jacob about his cast, anyhow (he dislikes talking about it). The boys discovered a live mic at the church where the reception was, and do I detect two future pastors in our midst? We had a great time at Laura s uncle and aunt s place. The boys were happy to discover an orange tree in their backyard, a tetherball post not far away, and an uncle ready to give them a demonstration of a swimming pool vacuum cleaner or sit at the piano with them. Jacob s favorite part, though, was when the hamburger buns his great uncle were toasting were left on the grill during the prayer before the meal, got a bit scorched, and the uncle remarked with a chuckle that I guess the Lord was tired of listening to me drone on! Jacob loved his meal, and cackled at the thought of a prayer causing buns to get scorched. But their highlight was the visit to the sequoias at Kings Canyon National Park the next day. The excitement had been building for that day all weekend. On the way out, we stopped at a fruit stand and bought some delicious strawberries the fresh, juicy, sweet and tasty kind that are red all the way through. We continued up through the foothills, stopping periodically to get out and stretch, look at the sights, take some photos, or borrow grandpa s binoculars. I knew we d be traveling in two cars, so I had the thought to pack some 2-way radios before we left. I gave one to the boys and one to the grandparents. All weekend long, whenever the six of us went somewhere, the boys (and especially Jacob) would give directions to the car that was following. Turn right! The light is green! Catch up, you re going too slow! So all the way into the mountains, Jacob would send back instructions on what to do. We saw Grant Grove, home to the worlds third-largest tree (267ft/81m tall and 3000 years old). It s quite the impressive tree the trunk s diameter near the ground is 29ft or almost 9m. As we walked the trails, their speed kept increasing as they were hunting for the tree tunnel I had told them about a tree that fell centuries ago and had been hollowed out to make a home. That trunk was easily 8ft or more in diameter, and I could stand up completely in places. We found it, to much delight from the boys So this is what it s like to be inside a tree! Our trip home brought a delay in Denver and a missed flight, which excited the boys when I told them now we get to eat supper in the airport! I wonder how long that tactic will work But Jacob was also excited because the plane we were put on in the end was bigger than the one we were scheduled on, so that was another piece of excitement. We got home, and I carried two sleeping boys in from the car, upstairs, tucked them in, pulled off their shoes, and put their favorite stuffed animals in their arms. They were happy to be home, and with memories to treasure for a long time.

2 September 2010

Wouter Verhelst: Frans Pop

I'm shocked to learn that Frans has died. Even more shocked to learn that, due to me sitting with my head in the sand, I almost missed it. You'll be missed, Frans. I didn't always agree with you or your methods, but I deeply respected you for who you were, what you did, and what you were willing to do. May you rest in peace.

1 September 2010

Maximilian Attems: fjp

Frans Pop contributions to Debian has already been honoured: Frans Pop obituary by Steve McIntyre. One less known fact is that he hacked in upstream linux-2.6 too. Latest linux-2.6 git lists him with 80 commits. A bigger part of his work was testing latest linux-2.6 on different architectures. There are lots of patches with "Reported-by: Frans Pop <elendil>" and "Tested-by: Frans Pop <elendil>". Also in this field he was aiming for big coverage and a special responsive tester. I am very sad to have missed the opportunity to meet you in person. You are missed. Rest in peace, my friend.

9 August 2010

Frans Pop: Stranger than fiction

If you haven't seen it, try to. Amazing this movie did not get any major awards. Maggie Gyllenhaal is well worth looking out for in other movies.

30 March 2010

Frans Pop: Debianizing an ARM-based netbook

I got this neat little Chinese netbook after a mail to the debian-arm list where one machine was offered in exchange for porting Debian to it. So I offered to get Debian Installer running on it. In total 20 of these ChiTech PC89E netbooks were bought as a group by various people in different countries. The netbook is based on an ARM S3C6410 SoC, has 256 MB RAM and an excellent 8.9" 1024x600 LCD display. The machine itself is 11" and weighs under 800 grams. Most of the weight is the LCD display; they've even had to add a small extra weight in the base to avoid it toppling over backwards. The goal of getting D-I running was achieved last week, though not without needing to overcome some steep hurdles. Base installation Our main problem is that we don't have the source code to either the kernel they use (2.6.24.2 with patches from Samsung and others), or the u-boot bootloader. SSH was available without password for the user account and various basic errors allowed us to root the system relatively quickly. Brute force cracked the root password a few days later. The only information we did have was from the provided desktop system (which is really quite good and certainly looks very polished, but in the end still way too limited), what's on the flash memory and the contents ofa "firmware upgrade" image they provided. Having that firmware upgrade image proved very important as it gave us a good idea how to boot the system from SD card and made it possible to locate the kernel, u-boot bootloader and other files in flash memory. Googling for some boot messages we found two kernel source trees (for a different device: SmartQ) that looked promising as a basis. After some disassembly work by Luke Kenneth Casson Leighton on the kernel to get LCD timings and correct GPIO data we managed to get a basic working kernel: LCD, USB (and thus keyboard and touchpad), and wired networking. Although there are still loads of things that need improving/fixing, that gave me enough to work with to try to get Debian Installer going. The main limitations for running D-I are that we cannot change the bootloader configuration and that their boot procedure does not use an initrd. But we did work out how to boot a custom kernel from an external SD card by making creative use of their "firmware upgrade" procedure. Essentially I needed to find a way to run D-I by only booting a kernel. Piggy-backing an initrd onto a kernel is possible (using the 'bootpImage' target from the upstream kernel build system), but then the size of kernel plus initrd is limited to 4 MB and that is really too limited for a decent D-I initrd (especially if you want full i18n support). I ended up creating a micro-initrd (only 66 kB!) which only function is to mount the external SD card, load the real D-I root initramfs (which now no longer has any real size limitation) and then run init in that. After adding some relatively minor customizations for this netbook (such as installing the kernel from a custom repository on alioth), the installer was up and running. As the framebuffer worked without problem for the newt frontend, I next wondered if it would be possible to also get the graphical installer working. This should be easier for non-x86 systems after the recent switch from DirectFB to X.Org as backend for the graphical installer. And the image below shows it was. I had to create an /etc/xorg.conf to get the USB keyboard and touchpad working (apparently auto-detection does not quite work for less standard hardware), but that was basically it. Language selection So now all we have to do is get all the remaining kernel issues sorted out...

26 February 2010

Frans Pop: Fun with VMware

Until two years or so ago I used VMware Workstation intensively for development and testing of Debian Installer, using one of the licenses generously made available to the D-I team by VMware. It was much faster than Qemu and thus very welcome. But due to increasing problems keeping it working with new kernels on a desktop system running unstable, I switched to VirtualBox (the OSE version), which has the huge advantage of being packaged for Debian and thus automatically follows the Debian kernel packages. That advantage has disappeared somewhat as I'm currently compiling my own kernels I'm currently running 2.6.33 on a Lenny desktop. So far I've been able without too much trouble to find patches for the VirtualBox modules to keep them compatible with new upstream kernels. This week I had reason to try VMware again. The Server edition this time as that has free licenses. The Server concept, where you connect to the management system and virtual machines via a web browser, is quite nice. It means I could install it on a server and keep my laptop nice and responsive. To keep VMware separate from the rest of the system and not mix it with software installed cleanly from Debian packages, I decided to install it in /opt. And that works beautifully. There are only a few bits installed in /etc, but that's OK. I have /etc managed by etckeeper anyway [1]. More challenging was building WMware modules to match my kernel. VMware wants you to run their vmware-config.pl script every time you switch kernels and build the kernel modules on the system where it runs (which requires the kernel headers). I wanted to build the modules as part of my kernel build process, on a totally different machine. The same as I already do for VirtualBox. I save the custom modules in a tarball that has the same uname and package version as the kernel package I build and have a simple script that installs them correctly. I succeeded by hacking the Makefiles (deleting most of their content) and then integrating them in my kernel build wrapper script. The module source provided by VMware built fine with a Debian 2.6.26 kernel, but not with the 2.6.32.9 kernel on my server (of course). But luckily others had already run into the same problems and solved them, so with some searching I found the required patches (first a full updated source for 2.6.29, then patches needed for .31 and for .32). Isn't open source just wonderful? So with that the VMware server started beautifully (well, the init script needed to be told that module names have changed from *.o to *.ko), but I could not connect to it from my laptop using SSL. Connecting without SSL (using an instance of iceweasel running on the server but started remotely from my laptop using X forwarding SSH) worked fine. That turned out to be a known issue too. After applying all the suggested workarounds in my iceweasel configuration (disable TLS 1.0; set cache size to zero; explicitly disable use of proxy for the server) I can now connect reliably to the VMware Server management system. Phew. [1] I had to exclude a fair number of VMware files for etckeeper as they are updated by VMware itself and thus really belong in /var. Example are DCHP status files...

19 December 2009

Frans Pop: debmirror IV

The Debian FTP-masters recently changed the way gzipped meta files are compressed in order to make them more efficient to update using the rsync option. This was done by adding the --rsyncable option when calling gzip. Consequence was however that when debmirror compressed Packages, Sources and Contents files after updating them by applying diffs, the md5sum of the gzipped file created by debmirror no longer matched the md5sum listed in the Release file (because debmirror did not use --rsyncable). Result was that debmirror would also download the full gzipped Packages, Sources and Contents files from the parent mirror, something the diffs are meant to avoid. Not nice. Anyway, this has been fixed in debmirror 2.4 which now by default also uses --rsyncable when gzipping the updated meta files. I've also uploaded a fixed version for Lenny (20070123lenny1), which should soon be available from proposed-updates and will be included in the next stable point release. For archives that also provide diffs (most archives don't have them) but do not have rsyncable gzipped files, the default options used when calling gzip can be overruled using the new option --gzip-options (only in version 2.4). Tip: if you are using the rsync method to download files, using --diff=none may well be more efficient now that the archive has rsyncable gzipped meta files. Version 2.4 also has a few other improvements and fixes. If you're currently using version 2.3.x an update to the new version is probably a good idea.

10 October 2009

Jaldhar Vyas: Barack Obama is a Keynesian

pointless award I haven't written about politics since the election for several reasons. First and foremost, Obama won fair and square. While I might wish the other guy had won, I can't whine and pout or threaten to move to Canada like some people. The vox populi chose and that choice has to be respected. Secondly, one of the worst aspects of the current American political scene is the paranoid obsession with personalities in which the president is not just an implementor of policies good or bad, but personally responsible for the entire state of the cosmos. It started with Nixon, started getting alarming with Clinton and reached batshit levels of insanity with George W. Bush. But just because the left does it it, doesn't mean we have to follow suit. Although I figured Obamas policies would end up a trainwreck, I wanted to see what he actually did before calling him on it. The Nobel comittee isn't hoping and changing with me. In a decison that is bizarre even by the lax standards of a prize that long since jumped the shark, the President is to receive the Nobel Peace prize for...not being George Bush? It's difficult to see what substantive reasons they could have for their choice. But apparently one of these days he is going to make extraordinary efforts to strengthen international diplomacy and cooperation between peoples...democracy and human rights are to be strengthened. Well let's see now: Perhaps I am being a little unfair in not taking into consideration all the great things the president is going to do real soon now. It's just that I'm worried that this might be an attempt to sway him in the task of winning the war in Afghanistan (and more likely than not carrying it into Iran and Pakistan.) I take solace in the knowledge that heretofore Obama has shown a refreshing inclination to throw people under the bus when they are no longer convenient. If he can do that to the guy who says he wrote his book than some silly Norwegians won't be a problem.

3 October 2009

Frans Pop: debmirror III

debmirror 2.3 should be hitting the mirrors about now. Main change is that it will now use the available diffs to update Contents files, which should give a nice bandwidth reduction for users who mirror those. With that the option --pdiff (for "package diff") no longer really covered its function, so I decided to change it to --diff. There's also a fix for mirroring archives that don't have a Release file. Question for users The option --add-dir has been marked as deprecated (for quite some time now I suspect). I'm considering to remove it in the next release as I cannot see any use cases for it, but it's quite possible I'm missing something and there are still people using it. If you would like that option preserved, then please mail me at debmirror@packages.d.o with an explanation of why and how you use it. Managing the size of a local mirror The archive has grown a lot over the past Debian releases and keeping even a partial local mirror can require quite some disk space. Luckily debmirror offers quite a few options to tune what is mirrored. My own mirror covers testing and unstable 'main' for 6 architectures (i386, amd64, armel, hppa, sparc and s390), no source, no D-I images. It uses only 61G. I say "only" as that's about 33GB less than it could have been without tuning. In other words, I'm saving a bit more than one third! Here are the options I added to achieve this: 
--exclude-deb-section='^debug$'
--exclude='/(xen-)?linux-[a-z]+-2\.6[.0-9]*-[-[:alnum:]]*(openvz vserver xen)[-[:alnum:]]*_'
--exclude='(k/kde g/gnome o/openoffice\.org).*/.*_(armel hppa s390)\.deb'
--exclude='(a/axiom/ d/debian-edu-doc/ e/ember( -media)/ e/eclipse(/ -))'
--exclude='(e/erlang g/(gcl(cvs)? ghc6)/ l/llvm(/ -) p/paraview/ o/openturns/)'
--exclude='(s/scalapack(-doc)?/ f/festvox- g/gcc-snapshot/)'
--exclude='(/acl2-books_ /digikam-doc_ /fluid-soundfont-gm_ /deal.ii-doc_)'
--exclude='(/libxmpp4r-ruby-doc_ /lilypond-doc_ /qt4-doc_ /vtk-doc_)'
--exclude='/i18n/Translation-.*\.bz2' --include='/i18n/Translation-(nl de)\.bz2'
And the explanation is: Obviously I have nothing against any of the packages that I exclude. It's just that I don't need them.

29 September 2009

Jon Dowland: archfs build dependencies

build-dependencies using an XML manpage
build-dependencies using an nroff manpage
archfs has been accepted into unstable! I've been working on this package in between other tasks for quite a long time. I based my package on prior packaging work by Adam Sloboda from mentors.debian.net. The upstream package lacks a manpage, and one of the changes I have made from the initial packaging was to replace an XML-authored manpage by Adam with a roff one (based on the processed output of the XML one). I did this because I find authoring roff documents easier than XML ones. Another consequence is the build dependencies a lot simpler. We can see just how simpler thanks to Frans Pops' debtree.

16 September 2009

Frans Pop: debtree 1.0 - Instant dependency graphs

Yay! I've done it: 1160 lines of bash script are now 1215 lines of perl, and: 
    'debtree aptitude': 1m2.832s -> 0m0.596s
The new release is available as version 0.9.9 from the debtree web site and has been uploaded for the archive as version 1.0.
This was the starting position, the run time for my complete test set: 
    real    22m33.583s
    user    18m29.709s
    sys     4m21.320s
I began with a pure language conversion from bash to perl, i.e. I kept the call-outs to dctrl-tools. This allowed me to easily identify problems in the language conversion by running my test suite, without having to worry that a change might have been caused by getting different data. The language conversion itself was fairly straightforward; most time was spent on finding all the little errors made during the conversion. This resulted in "only" a 10% speedup: 
    real    20m56.368s
    user    18m3.996s
    sys     2m46.986s
So bash itself isn't even horribly slower than perl, even with all the recursion and starting of subshells for calls to grep, sed, etc. Then I replaced the call-outs to dctrl-tools one by one, adding the dependency on libapt-pkg-perl. And that resulted in the amazing: 
    real    0m21.350s
    user    0m19.797s
    sys     0m1.372s
So, from 22 minutes to 21 seconds for 22 graphs, including some pretty complex ones. Not bad. I had to keep a call-out to dctrl-tools for build dependencies as it turned out libapt-pkg-perl does not expose architecture conditions. The full conversion process can be seen in the source repository, which was recently moved from my $HOME on alioth to collab-maint.

12 September 2009

Frans Pop: debtree 0.8.0

debtree 0.8.0, including the new option to display reverse dependencies, is now officially (or rather: unofficially) available. The new feature is of course documented in the man page, but also on the website. And now I think the time has come to port the script to perl. If I manage that I plan to upload the package into the archive as version 1.0. P.S. debtree now also supports generating trivial graphs: 
$ debtree --max-depth=0 dpkg
Funnily enough that same graph is less trivial for apt. Support for --max-depth=0 was added to allow to generate graphs showing only reverse dependencies.

Frans Pop: debmirror II - Overview of new features

I've just uploaded version 2.2 of debmirror, which introduces yet another new feature: mirroring the i18n/Translation files that contain translations of package descriptions. Many thanks to Joerg Jaspert for his quick response to my request to include those files in the Release file. Joerg also implemented the change needed to use the diffs for Contents files but that requires a fairly big code restructuring in debmirror. The package has jumped from version 1.0 to 2.2 in just three weeks (closing 28 bug reports in the process), but I think the changes justify that. Here's an overview.
If you're currently using the Lenny version of debmirror and would like to use the new features: the package from unstable can be installed on Lenny without any problems. The changes have been well tested, but I would advice to do use --dry-run after the upgrade to check there are no unexpected problems. One area where you may experience problems is when using debmirror for other archives than the official Debian mirrors. If you do encounter issues then please file a bug report.
Note that debmirror is not intended to be used for official mirrors. There are different scripts available for that from the Debian mirror team.

8 September 2009

Frans Pop: debtree-next - More steroids

Funny how working on a program immediately inspires to do more. Remember that the initial motivation for debtree was to find out why a package was installed? It can now show that in the dependency graphs! I'm not quite ready to do a new release, but the new version is available from the git repository. Let's start with a simple example (all graphs are based on Lenny). 
$ debtree -I --rdeps-depth=3 apt
reverse deps for apt Only installed packages are displayed here; if the -I option is omitted, debmirror will display all, but that does tend to explode the graphs, especially for common libraries. As for forward dependencies, the color of the arrows indicates Pre-Depends, Depends and Recommends. The reverse dependencies are shown three levels deep (one is default). The graph will always include all direct reverse dependencies (both on the package itself and all virtual packages provided by it. For indirect reverse dependencies there's a cut of that is set at five by default. Example is debconf, that apparently has 9 reverse Pre-Depends and 58 reverse Depends installed on my system. The next one is simply beautiful. 
$ debtree -I --rdeps-depth=20 --no-conflicts libcairo2
reverse deps for libcairo2 Because of the --rdeps-depth=20 this shows the full recursion! I was surprised that this graph remained a reasonable size. Apparently no packages depend on the virtual package libcairo, at least none that I have installed. The final one is extreme, and I must confess that I have cheated a bit by suppressing the least interesting reverse depends (which explains why it does not match the numbers from the apt graph). 
$ debtree -I -R --no-recommends --no-conflicts debconf
reverse deps for debconf The most interesting thing here is how it shows the debconf-2.0 transition. Most packages depend on 'debconf debconf2.0'; tex-common instead has 'debconf cdebconf', while tasksel and exim4 have both combinations (probably one explicitly in debian/control and the other added by debhelper. ucf is missing the alternative; apparently does not use debhelper (no prizes for guessing who the maintainer is :-). Notice anything about iamerican and ibritish? Yes, they really have a double dependency on 'debconf debconf2.0'. The one thing missing is the version info for versioned dependecies. Not sure yet if I want to add that for reverse dependencies. P.S. SVG versions of the images are available in the same directory as the JPGs.

6 September 2009

Frans Pop: debtree 0.7.3 - Oh what tangled webs we weave

It's been quite some time (almost two years) since my previous "release" of debtree, but now version 0.7.3 is available. And it still generates very nice graphs :-) The changes are relatively minor: a few nice fixes for corner cases that were not handled correctly, and an update of the default lists of "skip" and "end" packages which help to limit the size of graphs for a fair number of packages I tried (including konqueror and openoffice.org).
Reason to revisit debtree was a recent nice mail from a debtree user, but also the current discussions about udev and the FHS. I'm on the side of "let's please keep /usr mountable separately". Mostly because I like a (small) encrypted root with a separate (large) unencrypted /usr'. I'm also increasingly unhappy with the default size of Debian's desktop installs, especially now that it looks as if Squeeze will see installation of Recommends by default by tasksel (and thus Debian Installer). For comparison, the size of a default Gnome desktop install for Etch was 1360MB; for Lenny it is 1830MB; for Squeeze it looks like it will be well over 3000MB! Remember that for Sarge we installed both Gnome and KDE from CD1 with both together taking 1390MB? Sure, some of that is real functionality, but a lot is also (IMO) redundant visual effects that only serve to slow the desktop down and junk needed to do stuff automagically. And a heck of a lot is duplicated functionality. One of the main reasons I switched to Linux was because it gave me back control over my systems, but with KDE4 and pervasive stuff like hal and all the various "kits" Linux is on a fast track that's giving priority to flashiness over real functionality and eroding that control. Here's a fairly default dependency graph for hal (click for full image). Looks reasonable, right? hal dependency graph But that's only because most major dependencies, such as dbus, policykit and pm-utils have been pruned. Here's a complete graph, with only libc6 omitted (full image is 1.5MB). Truly a tangled web. Scary. full hal dependency graph One can also look at it from the other side. Today I upgraded my sid chroot and found I suddenly needed to install libavahi-client3, libavahi-common3, libavahi-common-data and libdbus-1-3. Why? Reason turned out to be libcups2, so I checked if I really needed that. And here's why I do. libgtk2.0-0 dependency graph Most of these dependencies of libgtk2.0-0 I can understand, but isn't gtk supposed to be a graphical toolkit library? Couldn't printing support be implemented in some more specialized Gnome printing toolkit library? But I'm probably missing something. Anyway, now that I have a bit more perl experience through my recent work on debmirror, maybe I should finally port debtree from shell script to perl...
See the debtree home page for a full overview of how to read the graphs, but here's a quick intro. Purple arrows are Pre-Depends, blue are Depends and black are Recommends; green connections show Provides. The green packages are currently installed in my sid chroot, while the white ones are not. The diamonds show where the graph has been pruned: dependencies for these packages are not shown.

21 May 2009

Frans Pop: Fighting Debian Mailing List Spam

As a past member of the listmaster team, this is a goal I have to support. I did some work myself a few years ago to clean out at least the most offensive images from the archive. And I can tell you there were a number of really gross ones. This was mostly manual work, replacing the spams by placeholder messages. Since then the listmaster team has implemented an excellent toolset for nominating messages as spam, reviewing the nominations and removing confirmed spam from the archive. My current contribution is twofold: Plug-in that allows reporting new list spam from KMail It's not actually a plug-in, but just uses the filter functionality of KMail. I've documented two alternative filters developed for KMail 1.9.9 (from KDE 3.5). Possibly they can also be (adapted to be) used with KMail from KDE 4, but that has not been tested. There are plug-ins available for several Mail User Agents (MUAs), with a few still to be developed. Spam cleaning campaign for the debian-boot list A few weeks ago I sent out a request for help to clean "our" archive. The response has been great. The work is [tracked on a Wiki page[(http://wiki.debian.org/DebianInstaller/SpamClean) and has already resulted in the removal of 676 spams. By the end of this week I expect that number to be doubled. If you'd like to start a similar project for your own favorite list, I'd suggest you start by reading through the thread linked above.
Update: I was wrong, the number was almost tripled: from 676 to 1801!

1 March 2009

Frans Pop: The case of the self-perpetuating DNS errors

Ingredients: The last couple of days I've been plagued by some DNS errors that kept showing up in the logcheck mails for my home server which I was busy migrating from one box to another, doing an upgrade from etch/i386 to lenny/amd64 at the same time. So, plenty of stuff going on to confuse the issue. I kept getting the following messages every hour (anonymized): 
named: connection refused resolving 'somedomain.org/NS/IN': xxx.yyy.zzz.nnn#53
named: connection refused resolving 'somedomain.org/NS/IN': xxx.yyy.zzz.mmm#53
named: connection refused resolving 'ns1.somedomain.org/AAAA/IN': xxx.yyy.zzz.mmm#53
named: connection refused resolving 'ns2.somedomain.org/AAAA/IN': xxx.yyy.zzz.mmm#53
named: connection refused resolving 'ns1.somedomain.org/AAAA/IN': xxx.yyy.zzz.nnn#53
named: connection refused resolving 'ns2.somedomain.org/AAAA/IN': xxx.yyy.zzz.nnn#53
The times were fairly regular: once just before the hour, most 2 minutes after. I fetch mail at around that time, but also at other times, so possible but unlikely. The 2 minutes after was the first real clue: some cron job maybe? After disabling logcheck the messages no longer appeared in the log. Enable it again, and they were back. Additional confusion was caused by the fact that the domain had "debian" in its name, but it was somewhere obscure. So why was logcheck causing a lookup for that domain? This did confuse me enough to waste some time looking for some silly weird (default) configuration problem in some package. Enter spamassassin. Apparently that was parsing the message body, recognized "somedomain.org" as a host name, and proceded to do a DNS lookup as validity check. So we have the following loop, started off by something causing an initial DNS lookup for the domain, which fails and gets logged: Duh. I remember struggling with probably the same problem a couple of years ago, but then it was a lot more severe: masses of repeating DNS errors for obscure domains. At that time I failed to get to the bottom of it and ended up just ignoring the errors by adding the following option in my bind9 configuration: 
logging  
    category lame-servers   null;  ;
 ;
Anyway, now I just no longer pass logcheck mails through spamassassin. (Although filtering out these DNS errors in bind9 can be perfectly valid.)

Frans Pop: The case of the self-perpetuating DNS-errors

Ingredients: The last couple of days I've been plagued by some DNS errors that kept showing up in the logcheck mails for my home server which I was busy migrating from one box to another, doing an upgrade from etch/i386 to lenny/amd64 at the same time. So, plenty of stuff going on to confuse the issue. I kept getting the following messages every hour (anonymized): 
named: connection refused resolving 'somedomain.org/NS/IN': xxx.yyy.zzz.nnn#53
named: connection refused resolving 'somedomain.org/NS/IN': xxx.yyy.zzz.mmm#53
named: connection refused resolving 'ns1.somedomain.org/AAAA/IN': xxx.yyy.zzz.mmm#53
named: connection refused resolving 'ns2.somedomain.org/AAAA/IN': xxx.yyy.zzz.mmm#53
named: connection refused resolving 'ns1.somedomain.org/AAAA/IN': xxx.yyy.zzz.nnn#53
named: connection refused resolving 'ns2.somedomain.org/AAAA/IN': xxx.yyy.zzz.nnn#53
The times were fairly regular: once just before the hour, most 2 minutes after. I fetch mail at around that time, but also at other times, so possible but unlikely. The 2 minutes after was the first real clue: some cron job maybe? After disabling logcheck the messages no longer appeared in the log. Enable it again, and they were back. Additional confusion was caused by the fact that the domain had "debian" in its name, but it was somewhere obscure. So why was logcheck causing a lookup for that domain? This did confuse me enough to waste some time looking for some silly weird (default) configuration problem in some package. Enter spamassassin. Apparently that was parsing the message body, recognized "somedomain.org" as a host name, and proceded to do a DNS lookup as validity check. So we have the following loop, started off by something causing an initial DNS lookup for the domain, which fails and gets logged: Duh. I remember struggling with probably the same problem a couple of years ago, but then it was a lot more severe: masses of repeating DNS errors for obscure domains. At that time I failed to get to the bottom of it and ended up just ignoring the errors by adding the following option in my bind9 configuration: 
logging  
    category lame-servers   null;  ;
 ;
Anyway, now I just no longer pass logcheck mails through spamassassin. (Although filtering out these DNS errors in bind9 can be perfectly valid.)

Next.