Hello, The past week I ve finished the muon package, available @ git.debian.org/?p=pkg-kde/kde-extras/muon.git on the gsoc-floris-2014 branch (use at your own risk!) Also, I ve finished up the KAuth support when adding/removing repos (already pushed to the master of Muon). Most importantly, I ve started integrating apt-listbugs a most important component of Debian. What does this mean and how it will affect the end-user? Well, having apt-listbugs integrated means that the user will be warned when installing packages if the packages have some serious/grave/critical bugs! I am still working on making it async and I hope to be done by tonight. Also, some formatting work on the dialog is still needed. Check it out bellow: After I finish the apt-listbugs integration, I will get started on writing up some apt unit tests as suggested by Aleix. Also, I will work with Sune on updating the qapt package as well. In a nutshell: This week:
*muon package
*KAuth support implemented
*initial apt-listbugs integration Plans for next week:
*finish apt-listbugs integration (async, with pretty formatting!)
*apt unit tests
*update qapt package Thanks for reading, see you next week!
Floris.

A new minor release 0.11.2 of Rcpp is now on the CRAN network for GNU R, and binaries for Debian have also been uploaded. The release smoothes a few edges on both the Rcpp side itself, as well as on the interaction between Rcpp and R which since release 3.1.0 offers a few new features related to C++11. Kevin added a couple of neat extensions related to vectors, a new ListOf templated list class, as well as a new option to warn on implicit casts. We decided not to make this option the default as it may be too common in some packages. JJ took care of a few buglets related to the wonderful Rcpp Attributes. See the NEWS file section below for details, or the ChangeLog file in the package and on the Rcpp Changelog page. Note that the diffstat reported by CRANberries is very large as Kevin also committed a whitespace cleanup which touched almost all files. As before, we tested this release by building against all CRAN packages which depend upon Rcpp. In fact we did three such runs leading up to the release. Only one package was blacklisted (as I currently don't have CUDA set-up), two had what may be internal errors or tests which were too restrictive, sixteen suffered from missing packages or RGL devices --- but the remaining 202 packages all built and tested cleanly. Detailed results of those tests (and the scripts for it) are in the rcpp-logs repo GitHub. There are a number of other fixes, upgrades and other extensions detailed in NEWS file extract below, in the ChangeLog file in the package and on the Rcpp Changelog page.

Changes in Rcpp version 0.11.2 (2014-06-06

• Changes in Rcpp API:
  • Implicit conversions, e.g. between NumericVector and IntegerVector, will now give warnings if you use #define RCPP_WARN_ON_COERCE before including the Rcpp headers.
  • Templated List containers, ListOf<T>, have been introduced. When subsetting such containers, the return is assumed to be of type T, allowing code such as ListOf<NumericVector> x; NumericVector y = x[0] + x[1] + x[2].
  • In a number of instances, returned results are protected and/or cast more carefully.
• Changes in Rcpp Attributes
  • Trailing line comments are now stripped by the attributes parser. This allows the parser to handle C++ source files containing comments inline with function arguments.
  • The USE_CXX1X environment variable is now defined by the cpp11 plugin when R >= 3.1. Two additional plugins have been added for use with C++0x (eg when using g++ 4.6.* as on Windows) as well as C++1y for compilers beginning to support the next revision of the standard; additional fallback is provided for Windows.
  • compileAttributes() now also considers Imports: which may suppress a warning when running Rcpp.package.skeleton().

Thanks to CRANberries, you can also look at a diff to the previous release. As always, even fuller details are on the Rcpp Changelog page and the Rcpp page which also leads to the downloads, the browseable doxygen docs and zip files of doxygen output for the standard formats. A local directory has source and documentation too. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Julien LIE completed this release a few weeks ago, but between work and other things I'm only finally now posting the announcement. This is primarily a bug-fix release. The most user-visible changes are that the radius.conf file for the nnrpd RADIUS authenticator has been renamed to inn-radius.conf (to avoid a conflict with the configuration of the actual RADIUS server), and the sample innd and nnrpd MOTD files are now installed as samples and not as live configuration on a new server install. controlchan and pgpverify now check the signer of control messages in a way that works properly with UIDs containing spaces and keys that define multiple UIDs, which should fix long-standing problems with verifying the control messages for fido.ger.* and grisbi.*. The nnrpd posting filter now has access to the attributes hash, similar to the authentication and access Perl hooks. In addition, there are fixes for a variety of crash bugs, build issues, log rotation issues, and the normal variety of bug fixes that accumulate over time. You can get the latest version from the official ISC download page or from my personal INN pages. The latter also has links to the full changelog and the other INN documentation. As always, thanks to Julien LIE for doing the lion's share of work on the new release, including investigating and fixing bugs, merging patches, and doing all the hard parts of the release process.

The Debian project is excited to announce that we are now accepting presentations, discussion sessions and tutorials for our DebConf14 conference which will take place in Portland State University, Oregon, USA from 23 to 31 August. Submitting an event To submit an event, first register as an attendee for DebConf14 in the conference management system. If you have any doubts or have problems with the registration process please check the Registration FAQ. After registering, go to the event submission page, or click on the Create an event option from the management system. Describe your submission in the web form. The most common event types are Lecture or Open Discussion (BoF). Please include a short title (to make it easy to produce a compact schedule) and an engaging description of the event. Tracks We will organize some talks into thematic tracks. If you have a proposal for a DebConf track, such as "Debian ARM", "Debian Infrastructure", or "Community Outreach" please contact talks@debconf.org. If you would like to be a track coordinator, please volunteer on the given mail address. Format of the events A regular session will be 45 minutes long, including time for questions. There will be a 15 minute breaks between events. Submissions are not limited to traditional talks: you could propose a performance, art installation, debate, or anything else. If you have any specific requirements for your event, please send an email to talks@debconf.org with the details of your requirements and be sure to mention your event title in the subject. Deadline While we ask speakers to submit their events before the deadline of 7 July 2014, 23:59:59 UTC, late submissions will continue to be accepted for scheduling until the end of DebConf. All attendees will have an opportunity to schedule ad-hoc events during DebConf itself if we have space for them. Very promising late submissions may be considered for inclusion in the main conference. Note that ad-hoc events have a much lower chance of video archiving, and streaming, so if you want these services it's better to get your submissions in early. DebConf official events will be broadcast live on the Internet when possible, and videos of the talks will be published on the web along with the presentation slides and papers. For private communication regarding your talk, or for more general ideas, or questions about the event and talks, please mail us We hope to you see you and share some good times with you this year in Portland during DebConf14!

I m writing this on the train back from the ZuriHac Haskell Hackathon in Z rich, generously sponsored by Better and Google. My goal for this event was to attract new people to work on GHC, the Haskell compiler, so I announced a GHC bugsquashing project . I collected a few seemingly simple ticket that have a good effort/reward ratio for beginners and encouraged those who showed up to pick one to work on. Roughly six people started, and four actually worked on GHC on all three days. The biggest hurdle for them was to get GHC built for the first time, especially those using a Mac or Windows. They also had to learn to avoid recompilation of the whole compiler, which takes an annoying amount of time (~30 minutes for most people). But once such hurdles weren taken all of them managed to find their way around the source code to the place they need to touch and were able to produce a patch, some of which are already merged into GHC master. When I wasn t giving tips and hints I was working on various small tickets myself, but nothing of great impact. I very much hope that this event will pay off and one or two of the newcomers end up being regular contributors to GHC.

We took breaks from our respective projects to listen to interesting talks by Edward Kmett and Simon Marlow, and on Saturday evening we all went to the shores of the Zurisee and had a nice Barbecue. It was a good opportunity to get into contact with more of the attendees (the hacking itself was separated in multiple smaller office rooms) and I was happy to hear about people having read my recent Call Arity paper, and even found it valuable.

Thanks to the organizers and sponsors for this nice opportunity!

I ve been learning about android development over the last few weeks, and I think I m slowly getting the hang of best practices. It s a bit tough, there s a lot of stuff that s not super Java-idomatic that s become Android-idiomatic, so getting over that stuff has been interesting. I ve been finding that Android tends to re-implement most things in a similar enough way, but always with some small tweak that feels kinda funny. It s working well enough, and I m hoping that I can clean up a few android libraries to deal with some of the OpenGov datasets I m interested in. If anyone has any tips on proper handling of what to make a fragment, and what activities should look like, I d really love posts about that. It seems like sometimes I have a 1-to-1 mapping of Fragments to Activity, but I want to keep it a fragment for large devices. Anyway, best practices welcome.

According to the Free Software Foundation, the Debian distribution is not a Free System Distribution. While I, as a Debian Developer, disagree with that point of view, it should be obvious that the opinion of an organization that engages in politics (or, at the very least, in political lobbying) and of which I am not a member should not be my concern. However, it has long irked me the wrong way that these claims exist. A few weeks ago, I found the System distribution commitments page on Wikipedia. This does change the situation somewhat; I cannot say that a page explaining an opinion of the FSF is wrong; but if that opinion is copied nearly verbatim to a website which claims neutrality as one of its "Core content policies", then the situation is significantly different. When I originally found that page, it was immediately obvious where the content came from, but there was no reference listed, so I added a "citation needed" tag, the first time I ever did so on Wikipedia. Now, a few weeks later, a reference has been added, and the "citation needed" tag removed. However, as expected, the only reference which has been added is the highly opinionated document on the FSF's website, which is hardly neutral. I've added another tag (this time a "which" tag, asking for more clarification) and added a Talk page listing my concerns with the article. But I can't really explain in Wikipedia why I think the whole premise is wrong; that's what a blog is for. So, here goes. The FSF lists two reasons why they do not endorse Debian:

1. The non-free and contrib repositories
2. An unsubstantiated claim that "the installer in some cases recommends [...] non-free firmware files for the peripherals on the machine."

Let's have a look at them, each in turn.

Non-free and contrib It is true that Debian provides non-free software in separate repositories, and that software in these repositories does not comply with either the Debian Free Software Guidelines (DFSG) or the FSF's Free Software Definition. However, the non-free and contrib repositories are fairly small:

wouter@carillon:/var/lib/apt/lists$ awk -F':' 'BEGIN total=0 /^Size/ total+=$2 END print total ' ftp.be.debian.org_debian_dists_sid_non-free_binary-amd64_Packages
4398812934

That is, the non-free repository contains just over 4G of software (after installation). Contrib is even smaller:

wouter@carillon:/var/lib/apt/lists$ awk -F':' 'BEGIN total=0 /^Size/ total+=$2 END print total ' ftp.be.debian.org_debian_dists_sid_contrib_binary-amd64_Packages 
306578636

While, combined, this is approximately the same size as some other distributions, free and non-free together, it pales in comparison with the "main" repository:

wouter@carillon:/var/lib/apt/lists$ awk -F':' 'BEGIN total=0 /^Size/ total+=$2 END print total ' ftp.be.debian.org_debian_dists_sid_main_binary-amd64_Packages
60025033622

i.e., the "main" repository contains just shy of 60G of software. Of course, whether non-free is large or not should not matter at all; the fact that it exists in the first place is a problem for free software advocates. But of those 4 gigabytes, what exactly is the non-free repository used for? For that, the Debian popularity contest has a page specifically for the non-free section. Looking at this page, and ignoring the packages that are listed as "Not in sid" (which either means they were installed from third-party suppliers and therefore cannot be attributed to Debian, or were only available in older distributions so should not be held against the Debian of today anymore), we see three main uses of non-free:

1. Various implementations of non-free algorithms (unrar, libfaac0, etc)
2. A lot of packages containing non-free firmware
3. A number of packages containing non-free documentation

The first package which does not fit any of those descriptions is iozone3, currently on the 64th position of the popularity contest (including packages "Not in sid"). The first item in this list is something which most free software advocates will try to avoid. After all, for all of those algorithms, alternatives exist, and so there is no real reason why one would want to use such software, other than "someone sent me a file encoded according to that algorithm, and I want to decode it". Debian gives you the option to do so, but a valid case could be made for not doing so. The second item in that list is really an alternate version of "the installer recommends non-free software", so I'll come back on that in the next section. The 3rd item in the above list deserves some special attention. Just over 10 years ago, Debian clarified its position where it clearly specified that documentation and programs (as both are interpreted as "software") should follow the same set of guidelines regarding freedom. The FSF disagrees with this position, and chose to release some of its documentation under the GFDL with invariant sections, which Debian decided is a non-free license in 2006. Since then, such GNU documentation would no longer be available were Debian to ditch the non-free repository.

Recommending non-free firmware As I said above, the FSF's page claims that the Debian installer recommends non-free firmware "in some cases", without going into much detail. Special care should be taken in understanding the word "recommend" here; wiktionary defines it as follows:

recommend (third-person singular simple present recommends, present participle recommending, simple past and past participle recommended)

1. (transitive) To bestow commendation on; to represent favourably; to suggest, endorse or encourage as an appropriate choice.

The board recommends Philips, given his ample experience in similar positions.

1. (transitive) To make acceptable; to attract favor to.
2. (transitive) To advise, propose, counsel favorably

The therapist recommends resting the mind and exercising the body.

1. (transitive, archaic) To commit, confide to another's care, confidence or acceptance, with favoring representations

A medieval oblate's parents recommended the boy for life to God and the monastery

However, according to the FSF, merely showing a firmware filename in unencrypted form counts as a recommendation. I've always found that use of the word "recommendation" to be dubious, at best. Ignoring that part, we can make the following observations: The debian-installer module ("udeb") which is responsible for configuring apt is called apt-mirror-setup (code in the apt-setup git repository). The actual debconf templates which allow the user to enable the non-free or contrib repositories are the apt-setup/non-free and apt-setup/contrib templates, and they look like this:

Template: apt-setup/non-free
Type: boolean
Default: false
# :sl1:
_Description: Use non-free software?
 Some non-free software has been made to work with Debian. Though this
 software is not at all a part of Debian, standard Debian tools can be used
 to install it. This software has varying licenses which may prevent you
 from using, modifying, or sharing it.
 .
 Please choose whether you want to have it available anyway.
Template: apt-setup/contrib
Type: boolean
Default: false
# :sl1:
_Description: Use contrib software?
 Some additional software has been made to work with Debian. Though this
 software is free, it depends on non-free software for its operation. This
 software is not a part of Debian, but standard Debian tools can be
 used to install it.
 .
 Please choose whether you want this software to be made available to you.

Note the lines Default: false, which imply that the default answer to these questions is to disable the non-free and contrib repositories. A question which defaults to false isn't a recommendation, nor do I read the text as any form of recommendation to install these repositories (quite the contrary, in fact). More important than whether the default is to enable or disable these repositories is whether this question is even shown to the user in normal installations. Whether this is the case isn't decided by any template; instead, it is decided by the code that calls the templates. That code currently looks like this:

STATE=1
while true; do
        case "$STATE" in
            1)
                db_input low apt-setup/non-free   true
                ;;
            2)
                # If they chose not to use non-free, ask about
                # contrib, with a default of no. If they chose to
                # use non-free, they get contrib too..
                db_get apt-setup/non-free
                if [ "$RET" = false ]; then
                        db_fget apt-setup/contrib seen
                        if [ "$RET" = false ]; then
                                db_set apt-setup/contrib false
                        fi
                        db_input low apt-setup/contrib   true
                else
                        db_fget apt-setup/contrib seen
                        if [ "$RET" = false ]; then
                                db_set apt-setup/contrib true
                        fi
                fi
                ;;
            *)
                break
                ;;
        esac
        if db_go; then
                STATE=$(($STATE + 1))
        else
                STATE=$(($STATE - 1))
        fi
done

Note the "low" in the db_input lines; it means the question is only shown at low priority, i.e., in "expert" mode. The result is that, unless you enable expert mode, you won't even see the question about the non-free repository, and it will be disabled by default. There won't even be any mention of it in your sources.list file. Of course, the above isn't about firmware files, which is what the FSF website claims. So, there is a second part to this story. If you run a system with hardware for which there is a non-free firmware package in the non-free repository, and you provided firmware for this hardware to the installer through various means, then the hw-detect module comes into play. This has code which currently looks like this:

# install cached firmware debs
if [ -d /var/cache/firmware ]; then
    for deb in /var/cache/firmware/*.deb; do
            if [ -f "$deb" ]; then
                    cp -a "$deb" /target/tmp
                    # TODO debconf passthrough
                    if ! in-target dpkg -i "/tmp/$(basename "$deb")"; then
                            # dpkg failed, force removal of package
                            in-target dpkg --force-depends --remove "$(deb_package "$deb")"   true
                    fi
                    rm -f "/target/tmp/$deb"
                    need_nonfree=1
            fi
    done
fi
if [ "$need_nonfree" ]; then
    db_set apt-setup/non-free true
fi

As can bee seen, if any firmware .deb files exist under /var/cache/firmware in the installer environment, then these will be installed on the target system and default answer to the apt-setup/non-free question will be changed to enable the non-free repository. How do such packages end up there? Well, that's done by the same hw-detect udeb, in a different script. The code there is a bit too long to quote in this already far too long blog post, but it boils down to:

• If the installer detects missing firmware, the user is asked whether (s)he wants to install firmware, with the following template:

  Template: hw-detect/load_firmware
  Type: boolean
  Default: true
  # :sl2:
  _Description: Load missing firmware from removable media?
   Some of your hardware needs non-free firmware files to operate. The
   firmware can be loaded from removable media, such as a USB stick or
   floppy.
   .
   The missing firmware files are: $ FILES 
   .
   If you have such media available now, insert it, and continue.
  

• If the user agreed to doing so (and to me, the above template does not read as a recommendation), any firmware .deb files that are provided to the installer are tested for the missing firmware files.
• If any of those files are found, these .deb files are copied to the /var/cache/firmware directory, and the firmware files are made available to the running kernel.

Conclusion In my opinion, while some of the claims on the FSF website are correct, others are at the very least misleading. Debian does not recommend the use of non-free software; and while providing non-free firmware to the installer isn't too difficult, it isn't documented very well; and as such installing a system with non-free enabled isn't something you can easily do by accident. Debian and the FSF clearly disagree on certain things, and this disagreement, not any policies by the Debian project, is the main reason why the FSF won't encourage Debian as a free distribution. I don't think the Debian project will easily agree to get rid of the non-free repository while the FSF still produces non-free documentation or while non-free firmware is still a fact of life in this world. However, I do think the following should happen:

1. The FSF should relicense its non-free documentation so it becomes truly free.
2. Once that has happened, Debian should rename its non-free repository to a "hardware-support" repository (or something along those lines), and remove everything that does not exist solely for the support of proprietary hardware (i.e., non-free drivers, non-free firmware, etc).
3. The installer could be clearer on the fact that if non-free firmware is found to be in use for the system, the user is using software that cannot be properly supported by the Debian project.

While the above wouldn't necessarily bring the opinions of the FSF and Debian 100% in line, it would at least bring their positions closer to eachother. Unfortunately, however, I doubt that this will happen; and that's a shame, because in the end, both Debian and the FSF want the same thing: a world where Free Software is the norm, not the exception.

Update

• The first version of this text said that providing firmware is not very easy. As was pointed out to me, it isn't too hard, but it's not very well documented and not part of the 'regular' installer images.
• Add contrib too, when we compare non-free and main
• Clarify when GFDL is non-free

More books. More reading is not yet happening, but I have hope. This particular set is quite random, mostly an excuse to pad out an order since My Real Children came out. Philip Ball Bright Earth (non-fiction)
Anthony Doerr All the Light We Cannot See (mainstream)
Leslie Jamison The Empathy Exams (non-fiction)
Dava Sobel Longitude (non-fiction)
Gore Vidal Julian (historical)
Jo Walton My Real Children (sff)
Markus Zusak The Book Thief (sff, sort of) My immediate inclination is to mark any book in which Death appears as a literal character as at least fantasy, but The Book Thief comes out of the mainstream genre, so we'll see what I think about the genre after reading it.

Today I attended the Don't Spy On Us campaign's Day Of Action at Shoreditch Town Hall in London. I'm not sure how much actual action there was, but the talking was interesting. Retrospective Day of Action drinking game: drink every time you hear the phrase "If you have nothing to hide, you have nothing to fear." The spooks have a really good marketing department. I don't write a lot on the internet any more - something I regret, actually. It can't even be because there are alternative places I am writing, because over the last couple of years I have been closing most of my social media accounts. I just share much less of myself online. On the internet, nothing is ephemeral. Bruce Schneier says so. Choose your words carefully. The thing about blogging is that it's so public. It's often tied to the writer's real-life identity. One of the valuable things about social media services is that they supposedly let you restrict who can read your words - the trade-off being that you must also grant access to advertisers, spies, cybercriminals... Most memorable moments of the day:

• Alan Rusbridger contrasting how he felt as a journalist working on the Snowden stories when in the US versus the UK - when in the UK, it's like "you know something bad is going to happen".
• Being asked to leave a tiny meeting room, in order that I could show my ticket for the next talk, only to retake the seat where I had left my bag. Go bureaucracy!
• Shami Chakrabarti leaving in her sunglasses like a rockstar.

In general, there were lots of famous people walking around as if they were normal. I was in the same room as Cory Doctorow, Jimmy Wales and Bruce Schneier at the same time. Ahem, action (mostly for UK citizens):

• Sign the petition.
• Join the various groups who organized today (Open Rights Group, Liberty, Big Brother Watch, Privacy International, Article 19, English PEN).
• Ask to meet your MP to talk about privacy at their next surgery - you can phone the House of Commons.

The heading might be ambiguous, but I couldn't figure out better heading so let me start by explaining what I'm trying to solve here.

This week went much better. I figured out how to properly implement over an Elektra Keyset and was able to write my first implementation of the Merge function. Right now I need to move on to testing my implementation and fixing the bugs. Already I am aware of a change that will need to be made. I am going to need to use full key names for lookup instead of a cascading lookup like I have now. I plan to use the full name for the root of the keyset in conjunction with the keys full names to change the names to look in the right place for each keyset. Additionally I had a video call with Markus and Felix today and I have a clear plan for what the next two weeks should look like. I really feel like I am getting into the swing of things and better understanding Elektra as well as the development process. So for next week I plan to better test the merge implementation and fix any bugs that crop up. Additionally I plan to write some automated tests for the merge function to make sure it is running correctly. Additionally I am going to work to understand Debian packaging. Also I am going to look at Samba and understand when it checks the configuration files when upgrading the package to see where the merge tool would be used. I want to get a prototype of a package upgrade running the kdb merge tool during an upgrade instead of prompting the user to manually merge the config files. I also plan to spend some extra hours looking at the latest version on Elektra and trying to fix errors with the debian package. I am going to package a version with the augeas_plugin. Additionally I am going to work on fixing some of the lintian errors that are occurring. In other words, it is going to be a very busy week. For reference, here is the proper way to iterate through a keyset. KeySet ks;
Key key;
ks.rewind(); //This is the line I was forgetting. It is necessary to call this function to point the KeySet's internal cursor to the beginning of the KeySet.
while(key=ks.next())
//This will iterate through the KeySet until key is no longer valid (the end of the KeySet).
Anyway, I will update you all next week. Until Then,
Ian

I just got my tickets to Portland, to assist to one of my favorite events: DebConf! So, the only point of this post it to say:
See you there and let s keep Portland Weird

This year's debconf in portland will happen without me being there. As much as I would love to be at home again, I won't be able to afford it. As much as I'd liked to help to keep portland weird, a discussion led to the feeling that I'm not welcome there and along that lines made me miss the deadline for sponsorship request due to not being very motivated to push for it because of that. And without sponsorship I won't be able to afford it, given that I need to save up for my upcoming move. This also means I won't be able to host the Poetry Night. I hope that someone will be picking up that ball and continue it. Personally I am more motivated than ever to start writing again, given that there is currently a Bus Bim Slam (Bus Tram Slam) happening over here in Vienna and I try to attend as much stations as possible, and there will be a Diary Slam during this year's FemCamp Vienna.
I'm indifferent on whether the Debconf Poetry Night should be recorded or not. On the one hand it would be great to see people performing, on the other hand it might shy away certain personal poems that one wouldn't want to have out in the wild. Whoever picks it up, think about that part. I wish everyone luck in Portland, and I'm looking forward to yet another great job by the video team so I can follow a few talks from at home. It sort of breaks my heart to not be able to hug you lot this year, and I wish you a great conference. We'll meet again next year in Heidelberg!

/debian permanent link Comments: 3 Flattr this

Debsources: Live and Historical Views on Macro-Level Software Evolution The paper entitled Debsources: Live and Historical Views on Macro-Level Software Evolution, which I've co-authored with Matthieu Caneill, has been accepted at ESEM 2014: the 8th international symposium on Emprical Software Engineering and Measurement. In the paper we have described Debsources as a software platform for monitoring the evolution of Free Software through the lenses of Debian, and used the main Debsources instance (http://sources.debian.net) to replicate and extend a former study on macro-level software evolution. Now we "just" have to integrate all the nice charts and data we have extracted for the paper into Debsources' stats page... /o\

I must echo John Sullivan's post: GPG keysigning and government identification. John states some very important reasons for people everywhere to verify the identities of those parties they sign GPG keys with in a meaningful way, and that means, not just trusting government-issued IDs. As he says, It's not the Web of Amateur ID Checking. And I'll take the opportunity to expand, based on what some of us saw in Debian, on what this means. I know most people (even most people involved in Free Software development not everybody needs to join a globally-distributed, thousand-people-strong project such as Debian) are not that much into GPG, trust keyrings, or understand the value of a strong set of cross-signatures. I know many people have never been part of a key-signing party. I have been to several. And it was a very interesting experience. Fun, at the beginning at least, but quite tiring at the end. I was part of what could very well constitute the largest KSP ever in DebConf5 (Finland, 2005). Quite awe-inspiring We were over 200 people, all lined up with a printed list on one hand, our passport (or ID card for EU citizens) in the other. Actwally, we stood face to face, in a ribbon-like ring. And, after the basic explanation was given, it was time to check ID documents. And so it began. The rationale of this ring is that every person who signed up for the KSP would verify each of the others' identities. Were anything fishy to happen, somebody would surely raise a voice of alert. Of course, the interaction between every two people had to be quick More like a game than like a real check. "Hi, I'm #142 on the list. I checked, my ID is OK and my fingerprint is OK." "OK, I'm #35, I also printed the document and checked both my ID and my fingerprint are OK." The passport changes hands, the person in front of me takes the unique opportunity to look at a Mexican passport while I look at a Somewhere-y one. And all is fine and dandy. The first interactions do include some chatter while we grab up speed, so maybe a minute is spent Later on, we all get a bit tired, and things speed up a bit. But anyway, we were close to 200 people That means we surely spent over 120 minutes (2 full hours) checking ID documents. Of course, not all of the time under ideal lighting conditions. After two hours, nobody was checking anything anymore. But yes, as a group where we trust each other more than most social groups I have ever met, we did trust on others raising the alarm were anything fishy to happen. And we all finished happy and got home with a bucketload of signatures on. Yay! One year later, DebConf happened in Mexico. My friend Martin Krafft tested the system, perhaps cheerful and playful in his intent but the flaw in key signing parties such as the one I described he unveiled was huge: People join the KSP just because it's a social ritual, without putting any thought or judgement in it. And, by doing so, we ended up dilluting instead of strengthening our web of trust. Martin identified himself using an official-looking ID. According to his recount of the facts, he did start presenting a German ID and later switched to this other document. We could say it was a real ID from a fake country, or that it was a fake ID. It is up to each person to judge. But anyway, Martin brought his Transnational Republic ID document, and many tens of people agreed to sign his key based on it Or rather, based on it plus his outgoing, friendly personality. I did, at least, know perfectly well who he was, after knowing him for three years already. Many among us also did. Until he reached a very dilligent person, Manoj, that got disgusted by this experiment and loudly denounced it. Right, Manoj is known to have strong views, and using fake IDs is (or, at least, was) outside his definition of fair play. Some time after DebConf, a huge thread erupted questioning Martin's actions, as well as questioning what do we trust when we sign an identity document (a GPG key). So... We continued having traditional key signing parties for a couple of years, although more carefully and with more buzz regarding these issues. Until we finally decided to switch the protocol to a better one: One that ensures we do get some more talk and inter-personal recognition. We don't need everybody to cross-sign with everyone else A better trust comes from people chatting with each other and being able to actually pin-point who a person is, what do they do. And yes, at KSPs most people still require ID documents in order to cross-sign. Now... What do I think about this? First of all, if we have not ever talked for at least enough time for me to recognize you, don't be surprised: I won't sign your key or request you to sign mine (and note, I have quite a bad memory when it comes to faces and names). If it's the first conference (or social ocassion) we come together, I will most likely not look for key exchanges either. My personal way of verifying identities is by knowing the other person. So, no, I won't trust a government-issued ID. I know I will be signing some people based on something other than their name, but hey I know many people already who live pseudonymously, and if they choose for whatever reason to forgo their original name, their original name should not mean anything to me either. I know them by their pseudonym, and based on that pseudonym I will sign their identities. But... *sigh*, this post turned out quite long, and I'm not yet getting anywhere ;-) But what this means in the end is: We must stop and think what do we mean when we exchange signatures. We are not validating a person's worth. We are not validating that a government believes who they claim to be. We are validating we trust them to be identified with the (name,mail,affiliation) they are presenting us. And yes, our signature is much more than just a social rite It is a binding document. I don't know if a GPG signature is legally binding anywhere (I'm tempted to believe it is, as most jurisdictions do accept digital signatures, and the procedure is mathematically sound and criptographically strong), but it does have a high value for our project, and for many other projects in the Free Software world. So, wrapping up, I will also invite (just like John did) you to read the E-mail self-defense guide, published by the FSF in honor of today's Reset The Net effort.

ISC has issued an operational notification advising the use of the -fno-delete-null-pointer-checks flag when compiling current versions of BIND with GCC 4.9:

Beginning with GCC 4.9.0, code optimization in GCC now includes (by default) an optimization which is intended to eliminate unnecessary null pointer comparisons in compiled code. Unfortunately this optimization removes checks which are necessary in BIND and the demonstrated effect is to cause unpredictable assertion failures during execution of named, resulting in termination of the server process. Future versions of BIND will be modified so that the optimizer does not incorrectly remove necessary checks when building from source, and until those versions are available multiple immediate workarounds are available.

According to the GCC documentation, -fdelete-null-pointer-checks performs the following optimization, and is enabled by default even when compiling with -O0:

Assume that programs cannot safely dereference null pointers, and that no code or data element resides there. This enables simple constant folding optimizations at all optimization levels. In addition, other optimization passes in GCC use this flag to control global dataflow analyses that eliminate useless checks for null pointers; these assume that if a pointer is checked after it has already been dereferenced, it cannot be null.

This optimization dates back to 1999 but it recently became more aggressive in the GCC 4.9 release:

GCC might now optimize away the null pointer check in code like:

int copy (int* dest, int* src, size_t nbytes)  
  memmove (dest, src, nbytes);
  if (src != NULL)
    return *src;
  return 0;
 

The pointers passed to memmove (and similar functions in <string.h>) must be non-null even when nbytes==0, so GCC can use that information to remove the check after the memmove call. Calling copy(p, NULL, 0) can therefore deference a null pointer and crash. The example above needs to be fixed to avoid the invalid memmove call, for example:

if (nbytes != 0)
  memmove (dest, src, nbytes);

It's interesting that the ISC operational advisory labels this an "incorrect" optimization. I wonder if this is a similar case to the Glibc optimization which exposed incorrect usage of memcpy() in many applications, including BIND. Fortunately, few users should be affected by this bug, since GCC 4.9 is fairly new. (The upcoming Fedora 21 release will probably be the first mainstream distribution to ship with GCC 4.9 as the default compiler.) Update: Debian bug #750760 filed.

My last update about ongoing development of botch, the bootstrap/build ordering tool chain, was four months ago and about several incremental updates. This post will be of similar nature. The most interesting news is probably the additional data that bootstrap.debian.net now provides. This is listed in the next section. All subsequent sections then list the changes under the hood that made the additions to bootstrap.debian.net possible.

bootstrap.debian.net The bootstrap.debian.net service used to have botch as a git submodule but now runs botch from its Debian package. This at least proves that the botch Debian package is mature enough to do useful stuff with it. In addition to the bootstrapping results by architecture, bootstrap.debian.net now also hosts the following additional services:

• History of graph size shows how the dependency graph developed over time for a normal self-contained repository plus for both minimizing strategies, updated every five days
• Crossbuild dependency satisfaction gives an overview of reasons why the crossbuild dependency situation cannot yet be analyzed with bug numbers where applicable
• Crossbuild order modifies the metadata of a repository so that the crossbuild dependency situation can be analyzed and then outputs an overview page as it is done for every architecture on the main page
• Source package importance calculates the port metric for source packages on a daily basis

Further improvements concern how dependency cycles are now presented in the html overviews. While before, vertices in a cycle where separated by commas as if they were simple package lists, vertices are now connected by unicode arrows. Dashed arrows indicate build dependencies while solid arrows indicate builds-from relationships. For what it's worth, installation set vertices now contain their installation set in their title attribute.

Debian package Botch has long depended on features of an unreleased version of dose3 which in turn depended on an unrelease version of libcudf. Both projects have recently made new releases so that I was now able to drop the dose3 git submodule and rely on the host system's dose3 version instead. This also made it possible to create a Debian package of botch which currently sits at Debian mentors. Writing the package also finally made me create a usable install target in the Makefile as well as adding stubs for the manpages of the 44 applications that botch currently ships. The actual content of these manpages still has to be written. The only documentation botch currently ships in the botch-doc package is an offline version of the wiki on gitorious. The new page ExamplesGraphs even includes pictures.

Cross By default, botch analyzes the native bootstrapping phase. That is, assume that the initial set of Essential:yes and build-essential packages magically exists and find out how to bootstrap the rest from there through native compilation. But part of the bootstrapping problem is also to create the set of Essential:yes and build-essential packages from nothing via cross compilation. Botch is unable to analyze the cross phase because too many packages cannot satisfy their crossbuild dependencies due to multiarch conflicts. This problem is only about the dependency metadata and not about whether a given source package actually crosscompiles fine in practice. Helmut Grohne has done great work with rebootstrap which is regularly run by jenkins.debian.net. He convinced me that we need an overview of what packages are blocking the analysis of the cross case and that it was useful to have a crossbuild order even if that was a fake order just to have a rough overview of the current situation in Debian Sid. I wrote a couple of scripts which would run dose-builddebcheck on a repository, analyze which packages fail to satisfy their crossbuild dependencies and why, fix those cases by adjusting package metadata accordingly and repeat until all relevant source packages satisfy their crossbuild dependencies. The result of this can then be used to identify the packages that need to be modified as well as to generate a crossbuild order. The fixes to the metadata are done in an automatic fashion and do not necessarily reflect the real fix that would solve the problem. Nevertheless, I ended up agreeing that it is better to have a slightly wrong overview than no overview at all.

Minimizing the dependency graph size Installation sets in the dependency graph are calculated independent from each other. If two binary packages provide A, then dependencies on A in different installation sets might choose different binary packages as providers of A. The same holds for disjunctive dependencies. If a package depends on A C and another package depends on C A then there is no coordination to choose C so to minimize the overall amount of vertices in the graph. I implemented two methods to minimize the impact of cases where the dependency solver has multiple options to satisfy a dependency through Provides and dependency disjunctions. The first method is inspired by Helmut Grohne. An algorithm goes through all disjunctive binary dependencies and removes all virtual packages, leaving only real packages. Of the remaining real packages, the first one is selected. For build dependencies, the algorithm drops all but the first package in every disjunction. This is also what sbuild does. Unfortunately this solution produces an unsatisfiable dependency situation in most cases. This is because oftentimes it is necessary to select the virtual disjunctive dependency because of a conflict relationship introduced by another package. The second method involves aspcud, a cudf solver which can optimize a solution by a criteria. This solution is based on an idea by Pietro Abate who implemented the basis for this idea back in 2012. In contrast to a usual cudf problem, binary packages now also depend on the source packages they build from. If we now ask aspcud to find an installation set for one of the base source packages (I chose src:build-essential) then it will return an installation set that includes source packages. As an optimization criteria the number of source packages in the installation set is minimized. This solution would be flawless if there were no conflicts between binary packages. Due to conflicts not all binary packages that must be coinstallable for this strategy to work can be coinstalled. The quick and dirty solution is to remove all conflicts before passing the cudf universe to aspcud. But this also means that the solution does sometimes not work in practice.

Test cases Botch now finally has a test target in its Makefile. The test target tests two code paths of the native.sh script and the cross.sh script. Running these two scripts covers testing most parts of botch. Given that I did lots of refactoring in the past weeks, the test cases greatly helped to assure that I didnt break anything in the process. I also added autopkgtests to the Debian packaging which test the same things as the test target but naturally run the installed version of botch instead. The autopkgtests were a great help in weeding out some lasts bugs which made botch depend on being executed from its source directory.

Python 3 Reading the suggestions in the Debian python policy I evaluated the possibility to use Python 3 for the Python scripts in botch. While I was at it I added transparent decompression for gzip, bz2 and xz based on the file magic, replaced python-apt with python-debian because of bug#748922 and added argparse argument parsing to all scripts. Unfortunately I had to find out that Python 3 support does not yet seem to be possible for botch for the following reasons:

• no soap module for Python 3 in Debian (needed for bts access)
• hash randomization is turned on by default in Python 3 and therefore the graph output of networkx is not deterministic anymore (bug#749710)

Thus I settled for changing the code such that it would be compatible with Python 2 as well as with Python 3. Because of the changed string handling and sys.stdout properties in Python 3 this proved to be tricky. On the other hand this showed me bugs in my code where I was wrongly relying on deterministic dictionary key traversal.

Please stop recommending that checking government-issued ID is a good way to verify someone's identity before signing their GPG key. Have you been a US bartender before? Or held any other position where you've had to verify an ID? It's not an easy thing to do. People in those positions have books of valid IDs from different states. They have lights that show the security marks. They still get it wrong regularly. A very amateur fake ID, or borrowed real ID, will fool just about everyone in any informal context. What's even worse is that people have a habit of happily looking at passports from other countries than their own, and nodding knowingly. It's fun, but be honest, you have no idea what you're doing. How about just signing keys with people you would actually say you know well enough to trust? It's not the Web of Amateur ID Checking. ID checking is at best ineffective against the threats it's supposed to address, and is probably actually damaging to the Web of Trust because of the false sense of security. No idea what I'm talking about? Learn to encrypt your email by reading the FSF's new Email Self-Defense Guide, published in honor of today's Reset The Net effort.