Search Results: "cklin"

1 April 2017

Antoine Beaupr : My free software activities, February and March 2017

Looking into self-financing Before I begin, I should mention that I started tracking my time working on free software more systematically. I spend a lot of time on the computer, as regular readers of this blog might remember so I wanted to know exactly how much time was paid vs free work. I was already using org-mode's time clock system to keep track of my work hours, so I just extended this to my regular free software contributions, which also helps in writing those reports. It turns out that over 60% of my computer time is spent working on free software. That's huge! I was expecting something more along the range of 20 to 40% of my time. So I started thinking about ways of financing this work. I created a Patreon page but I'm hesitant into launching such a campaign: the only thing worse than "no patreon page" is "a patreon page with failed goals and no one financing it". So before starting such an effort, I'd like to get a feeling of what other people's experience with it are. I know that joeyh is close to achieving his goals, but I can't compare with the guy that invented git-annex or debhelper, so I'm concerned I wouldn't be able to raise the same level of funding. So any advice you have, feel free to contact me in private or in the comments. If you would be ready to fund my work, I'd love to know about it, obviously, but I guess I wouldn't get real numbers until I actually open up such a page... Now, onto the regular report.

Wallabako I spent a good chunk of time completing most of the things I had in mind for Wallabako, which I mentioned quickly in the previous report. Wallabako is now much easier to installed, with clearer instructions, an easier to use configuration file, more reliable synchronization and read status propagation. As usual the Wallabako README file has all the details. I've also looked at better integration with Koreader, the free software e-reader that forms the basis of the okreader free software distribution which has been able to port Debian to the Kobo e-readers, a project I am really excited about. This project has the potential of supporting Kobo readers beyond the lifetime that upstream grants it and removes a lot of proprietary software and spyware that ships with the Kobo readers. So I have made a few contributions to okreader and also on koreader, the ebook reader okreader is based on.

Stressant I rewrote stressant, my simple burn-in and stress-testing tool. After struggling in turn with Debirf, live-build, vmdebootstrap and even FAI, I just figured maybe it wasn't the best idea to try and reinvent that particular wheel: instead of reinventing how to build yet another Debian system build tool, maybe I should just reuse what's already there. It turns out there's a well known, succesful and fairly complete recovery system called Grml. It is a Debian Derivative, so all I needed to do was to stop procrastinating and actually write the actual stressant tool instead of just creating a distribution with a bunch of random tools shipped in. This allowed me to focus on which tools were the best to stress test different components. This selection ended up being: fio can also be used to overwrite disk drives with the proper options (--overwrite and --size=100%), although grml also ships with nwipe for wiping old spinning disks and hdparm to do a secure erase of SSD disks (whatever that's worth). Stressant still needs to be shipped with grml for this transition to be complete. In the meantime, I was able to configure the excellent public Gitlab CI service to provide ISO images with Stressant built-in as a stopgap measure. I also need to figure out a way to automate starting stressant from a boot menu to automate deployments on a larger scale, although because I have little need for the feature at this moment in time, this will likely wait for a sponsor to show up for this to be implemented. Still, stressant has useful features like the capability of sending logs by email using a fresh new implementation of the Python SMTPHandler (BufferedSMTPHandler) which waits for logging to complete before sending a single email. Another interesting piece of code in there is the NegateAction argparse handler that enables the use of "toggle flags" (e.g. --flag / --no-flag). I'm so happy with the code that I figure I could just share it here directly:
class NegateAction(argparse.Action):
    '''add a toggle flag to argparse

    this is similar to 'store_true' or 'store_false', but allows
    arguments prefixed with --no to disable the default. the default
    is set depending on the first argument - if it starts with the
    negative form (define by default as '--no'), the default is False,
    otherwise True.
    '''
    negative = '--no'
    def __init__(self, option_strings, *args, **kwargs):
        '''set default depending on the first argument'''
        default = not option_strings[0].startswith(self.negative)
        super(NegateAction, self).__init__(option_strings, *args,
                                           default=default, nargs=0, **kwargs)
    def __call__(self, parser, ns, values, option):
        '''set the truth value depending on whether
        it starts with the negative form'''
        setattr(ns, self.dest, not option.startswith(self.negative))
Short and sweet. I wonder why stuff like this is not in the standard library yet - maybe just because no one bothered yet? It'd be great to get feedback of more experienced Pythonistas on this one. I hope that my work on Stressant is complete. I get zero funding for this work, and have little use for it myself: I manage only a few machines and such a tool really shines when you regularly put new hardware online, which is (fortunately?) not my case anymore. I'd be happy, of course, to accompany organisations and people that wish to further develop and use such a tool. A short demo of stressant as well as detailed description of how it works is of course available in its README file.

Standard third party repositories After looking at improvements for the grml repository instructions, I realized there was no real "best practices" document on how to configure an Apt repository. Sure, there are tools like reprepro and others, but those hardly qualify as policy: they are very flexible and there are lots of ways to create insecure repositories or curl sh style instructions, which we of course generally want to avoid. While the larger problem of Unstrusted Debian packages remain generally unsolved (e.g. when you install any .deb file, it can get root on your system), it seemed to me one critical part of this problem was how to add a random third-party repository to your machine while limiting, as much as possible, what possible attackers could do with such a repository. In other words, to solve the more general problem of insecure .deb files, we also need to solve the distribution problem, otherwise fixing the .deb files themselves will be useless. This lead to the creation of standardized repository instructions that define:
  1. how to distribute the repository's public signing key (ie. over HTTPS)
  2. how to name suites and components (e.g. use stable and main unless you have a good reason, and explain yourself)
  3. recommend a healthy does of apt preferences pinning
  4. how to distribute keys (e.g. with a derive-archive-keyring package)
I've seen so many third party repositories get this wrong. For example, a lot of repositories recommend this type of command to intialize the OpenPGP trust path:
curl http://example.com/key.asc   apt-key add -
This has the following problems:
  • the key is transfered in plaintext and can easily be manipulated by an active attacker (e.g. a router on your path to the server or a neighbor in a Wifi cafe)
  • the key is added to the main trust root, which allows the key to authentify as the real Debian archive, therefore giving it all rights over all packages
  • since it's part of the global archive, it's difficult for a package to remove/add the key when a key rollover is necessary (and repositories generally don't provide a deriv-archive-keyring to do that process anyways)
An example of this are the Docker install instructions that, at least, manage to do this over HTTPS. Some other repositories don't even bother teaching people about the proper way of adding those keys. We settled for:
wget -O /usr/share/keyrings/deriv-archive-keyring.gpg https://deriv.example.net/debian/deriv-archive-keyring.gpg
That location was explicitly chosen to be out of the main trust directory, so that it needs to be explicitly added to the sources.list as well:
deb [signed-by=/usr/share/keyrings/deriv-archive-keyring.gpg] https://deriv.example.net/debian/ stable main
Similarly, we highly recommend users setup "apt pinning" to restrict what a given repository can do. Since pinning is so confusing, most people don't actually bother even configuring it and I have yet to see a single repo advise its users to configure those preferences, which are essential to limit what a repository can do. To keep configuration simple, we recommend this:
Package: *
Pin: origin deriv.example.net
Pin-Priority: 100
Obviously, for a single-package repository, the actual package name should be listed, e.g.:
Package: foo
Pin: origin deriv.example.net
Pin-Priority: 100
And the priority should probably be set to 1 unless you want to allow automatic upgrades. It is my hope that this design will get more traction in the years to come and become a de-facto standard that will be a key part in safely adding third party repositories. There is obviously much more work to be done to improve security when installing untrusted .deb files, and I encourage Debian developers to consider contributing to the UntrustedDebs discussions and particularly to the Teams/Dpkg/Spec/DeclarativePackaging work.

Signal R&D I spent a significant amount of time this month struggling with the Signal project on my phone. I'm still ambivalent on Signal: it's a centralized designed, too dependent on phone numbers, but I must admit they get a lot of things right and it's the only free-software platform that allows for easy-to-use, multi-platform videoconferencing that my family can use. I've been following Signal for a while: up until now, I had been using the LibreSignal rebuild of the official client, as it is distributed on a F-Droid repository. Because I try to avoid Google (proprietary) software on my phone, it's basically the only way I could even install Signal. Unfortunately, the repository is out of date and introduces another point of trust in the distribution model: now you not only need to trust the Signal authors to do the right thing, you also need to trust that F-Droid repo not to inject nasty code on your phone. I've therefore started a discussion about how Signal could be distributed outside of the Google Play Store. I'd like to think it's one of the things that led the Signal people to distribute an official copy of Signal outside of the playstore. After much struggling, I was able to upgrade to this official client and will be able to upgrade easily by just downloading the APK. (Do note that I ended up reinstalling and re-registering Signal, which unfortunately changed my secret keys.) I do hope Signal enters F-Droid one day, but it could take a while because it still doesn't work without Google services and barely works with MicroG, the free software alternative to the Google services clients. Moxie also set a list of requirements like crash reporting and statistics that need to be implemented on F-Droid's side before he agrees to the deployment, so this could take a while. I've also participated in the, ahem, discussion on the JWZ blog regarding a supposed vulnerability in Signal where it would leak previously unknown phone numbers to third parties. I reviewed the way the phone number is uploaded and, while it's possible to create a rainbow table of phone numbers (which are hashed with a truncated SHA-1 checksum), I couldn't verify the claims of other participants in the thread. For me, Signal still does the right thing with contacts, although I do question the way "read status" notifications get transmitted, but that belong in another bug report / blog post.

Debian Long Term Support (LTS) It's been more than a year working on Debian LTS, started by Raphael Hertzog at Freexian. I didn't work much in February so I had a lot of hours to catchup with, and was unfortunately unable to do so, partly because I was busy with other projects, and partly because my colleagues are doing a great job at resolving the most important issues. So one my concerns this month was finding work. It seemed that all the hard packages were either taken (e.g. my usual favorites, tiff and imagemagick, we done by others) or just too challenging (e.g. I don't feel quite comfortable tackling the LTS branch of the Linux kernel yet). I spent quite a bit of time trying to figure out what was wrong with pcre3, only to realise the "32" in the report was not about the architecture, but about the character width. Because of thise, I marked 4 CVEs (CVE-2017-7186, CVE-2017-7244, CVE-2017-7245, CVE-2017-7246) as "not-affected", since the 32-bith character support wasn't enabled in wheezy (or jessie, for that matter). I still spent some time trying to reproduce the issues, which require a compiler with an AddressSanitizer, something that was introduced in both Clang and GCC after Wheezy was released, which makes reproducing this fairly complicated... This allowed me to experiment more with Vagrant, however, and I have provided the Debian cloud team with a 32-bit Vagrant box that was merged in shortly after, although it doesn't show up yet in the official list of Debian images. Then I looked at the apparmor situation (CVE-2017-6507), Debian bug #858768). That was one tricky bug as well, since it's not a security issue in apparmor per se, but more an issue with things that assume a certain behavior from apparmor. I have concluded that Wheezy was not affected because there are no assumptions of proper isolation there - which are provided only starting from LXC 1.0 - and Docker is not in Wheezy. I also couldn't reproduce the issue on Jessie, but, as it turns out, the issue was sysvinit-specific, which is why I couldn't reproduce it under the default systemd configuration shipped with Jessie. I also looked at the various binutils security issues: as I reported on the mailing list, I didn't see anything serious enough in there to warrant a a security release and followed the lead of both the stable and Red Hat security teams by marking this "no-dsa". I similiarly reviewed the mp3splt security issues (specifically CVE-2017-5666) and was fairly puzzled by that issue, which seems to be triggered only the same address sanitization extensions than PCRE, although there was some pretty wild interplay with debugging flags in there. All in all, it seems we can't reproduce that issue in wheezy, but I do not feel confident enough in the results to push that issue aside for now. I finally uploaded the pending graphicsmagick issue (DLA-547-2), a regression update to fix a crash that was introduced in the previous release (DLA-547-1, mistakenly named DLA-574-1). Hopefully that release should clear up some of the confusion and fix the regression. I also released DLA-879-1 for the CVE-2017-6369 in firebird2.5 which was an interesting experiment: I couldn't reproduce the issue in a local VM. After following the Ubuntu setup tutorial, as I wasn't too familiar with the Firebird database until now (hint: the default username and password is sysdba/masterkey), I ended up assuming we were vulnerable and just backporting the patch after seeing the jessie folks push out a release just in case. I also looked at updating the ca-certificates package to deal with the pending WoSign/Startcom removal: I made an explicit list of the CAs that need to be removed after reviewing the Mozilla list. I also sent a patch for an unrelated issue where ca-certificates is writing to /usr/local (!!) in Debian bug #843722. I have also done some "meta" work in starting a discussion about fixing the missing DLA links in the tracker, as you will notice all of the above links lead to nowhere. Thanks to pabs, there are now some links but unfortunately there are about 500 DLAs missing from the website. We also discussed ways to Debian bug #859123, something which is currently a manual process. This is now in the hands of the excellent webmaster team. I have also filed a few missing security bugs (Debian bug #859135, Debian bug #859136), partly because I wanted to help the security team. But it turned out that I felt the script needed some improvements, so I submitted a patch to improve the script so it is easier to run.

Other projects As usual, there's the usual mixed bags of chaos: More stuff on Github...

1 March 2017

Thorsten Glaser: New GitHub Terms of Service r e q u i r e removing many Open Source works from it

Please use the correct (perma)link to bookmark this article, not the page listing all wlog entries of the last decade. Thank you.</update> Some updates inline and at the bottom. The new Terms of Service of GitHub became effective today, which is quite problematic there was a review phase, but my reviews pointing out the problems were not answered, and, while the language is somewhat changed from the draft, they became effective immediately. Now, the new ToS are not so bad that one immediately must stop using their service for disagreement, but it s important that certain content may no longer legally be pushed to GitHub. I ll try to explain which is affected, and why. I m mostly working my way backwards through section D, as that s where the problems I identified lie, and because this is from easier to harder. Note that using a private repository does not help, as the same terms apply. Anything requiring attribution (e.g. CC-BY, but also BSD, ) Section D.7 requires the person uploading content to waive any and all attribution rights. Ostensibly to allow basic functions like search to work , which I can even believe, but, for a work the uploader did not create completely by themselves, they can t grant this licence. The CC licences are notably bad because they don t permit sublicencing, but even so, anything requiring attribution can, in almost all cases, not written or otherwise, created or uploaded by our Users . This is fact, and the exceptions are few. Anything putting conditions on the right to use, display and perform the work and, worse, reproduce (all Copyleft) Section D.5 requires the uploader to grant all other GitHub users Note that section D.4 is similar, but granting the licence to GitHub (and their successors); while this is worded much more friendly than in the draft, this fact only makes it harder to see if it affects works in a similar way. But that doesn t matter since D.5 is clear enough. (This doesn t mean it s not a problem, just that I don t want to go there and analyse D.4 as D.5 points out the same problems but is easier.) This means that any and all content under copyleft licences is also no longer welcome on GitHub. Anything requiring integrity of the author s source (e.g. LPPL) Some licences are famous for requiring people to keep the original intact while permitting patches to be piled on top; this is actually permissible for Open Source, even though annoying, and the most common LaTeX licence is rather close to that. Section D.3 says any (partial) content can be removed though keeping a PKZIP archive of the original is a likely workaround. Affected licences Anything copyleft (GPL, AGPL, LGPL, CC-*-SA) or requiring attribution (CC-BY-*, but also 4-clause BSD, Apache 2 with NOTICE text file, ) are affected. BSD-style licences without advertising clause (MIT/Expat, MirOS, etc.) are probably not affected if GitHub doesn t go too far and dissociates excerpts from their context and legal info, but then nobody would be able to distribute it, so that d be useless. But what if I just fork something under such a licence? Only continuing to use GitHub constitutes accepting the new terms. This means that repositories from people who last used GitHub before March 2017 are excluded. Even then, the new terms likely only apply to content uploaded in March 2017 or later (note that git commit dates are unreliable, you have to actually check whether the contribution dates March 2017 or later). And then, most people are likely unaware of the new terms. If they upload content they themselves don t have the appropriate rights (waivers to attribution and copyleft/share-alike clauses), it s plain illegal and also makes your upload of them or a derivate thereof no more legal. Granted, people who, in full knowledge of the new ToS, share any User-Generated Content with GitHub on or after 1 March, 2017, and actually have the appropriate rights to do that, can do that; and if you encounter such a repository, you can fork, modify and upload that iff you also waive attribution and copyleft/share-alike rights for your portion of the upload. But especially in the beginning these will be few and far between (even more so taking into account that GitHub is, legally spoken, a mess, and they don t even care about hosting only OSS / Free works). Conclusion (Fazit) I ll be starting to remove any such content of mine, such as the source code mirrors of jupp, which is under the GNU GPLv1, now and will be requesting people who forked such repositories on GitHub to also remove them. This is not something I like to do but something I am required to do in order to comply with the licence granted to me by my upstream. Anything you ve found contributed by me in the meantime is up for review; ping me if I forgot something. (mksh is likely safe, even if I hereby remind you that the attribution requirement of the BSD-style licences still applies outside of GitHub.) (Pet peeve: why can t I adopt a licence with British spelling? They seem to require oversea barbarian spelling.) The others Atlassian Bitbucket has similar terms (even worse actually; I looked at them to see whether I could mirror mksh there, and turns out, I can t if I don t want to lose most of what few rights I retain when publishing under a permissive licence). Gitlab seems to not have such, but requires you to indemnify them YMMV. I think I ll self-host the removed content. And now? I m in contact with someone from GitHub Legal (not explicitly in the official capacity though) and will try to explain the sheer magnitude of the problem and ways to solve this (leaving the technical issues to technical solutions and requiring legal solutions only where strictly necessary), but for now, the ToS are enacted (another point of my criticism of this move) and thus, the aforementioned works must go off GitHub right now. That s not to say they may not come back later once this all has been addressed, if it will be addressed to allow that. The new ToS do have some good; for example, the old ToS said you allow every GitHub user to fork your repositories without ever specifying what that means. It s just that the people over at GitHub need to understand that, both legally and technically , any and all OSS licences grant enough to run a hosting platform already , and separate explicit grants are only needed if a repository contains content not under an OSI/OKFN/Copyfree/FSF/DFSG-free licence. I have been told that these are important issues and been thanked for my feedback; we ll see what comes from this. maybe with a little more effort on the coders side All licences on one of those lists or conformant to the DFSG, OSD or OKD should do . e.g. when displaying search results, add a note this is an excerpt, click HERE to get to the original work in its context, with licence and attribution where HERE is a backlink to the file in the repository It is understood those organisations never un-approve any licence that rightfully conforms to those definitions (also in cases like a grant saying just use any OSS licence which is occasionally used) Update: In the meantime, joeyh has written not one but two insightful articles (although I disagree in some details; the new licence is only to GitHub users (D.5) and GitHub (D.4) and only within their system, so, while uploaders would violate the ToS (they cannot grant the licence) and (probably) the upstream-granted copyleft licence, this would not mean that everyone else wasn t bound by the copyleft licence in, well, enough cases to count (yes it s possible to construct situations in which this hurts the copyleft fraction, but no, they re nowhere near 100%).

2 February 2017

Steinar H. Gunderson: Not going to FOSDEM but a year of Nageru

It's that time of the year :-) And FOSDEM is fun. But this year I won't be going; there was a scheduling conflict, and I didn't really have anything new to present (although I probably could have shifted around priorities to get something). But FOSDEM 2017 also means there's a year since FOSDEM 2016, where I presented Nageru, my live video mixer. And that's been a pretty busy year, so I thought I'd do a live cap from high up above. First of all, Nageru has actually been used in production we did Solskogen and Fyrrom, and both gave invaluable input. Then there have been some non-public events, which have also been useful. The Nageru that's in git right now is evolved considerably from the 1.0.0 that was released last year. diffstat shows 19660 insertions and 3543 deletions; that's counting about 2500 lines of vendored headers, though. Even though I like deleting code much more than adding it, the doubling (from ~10k to ~20k lines) represents a significant amount of new features: 1.1.x added support for non-Intel GPUs. 1.2.x added support for DeckLink input cards (through Blackmagic's proprietary drivers), greatly increasing hardware support, and did a bunch of small UI changes. 1.3.x added x264 support that's strong enough that Nageru has really displaced VLC as my go-to tool for just video-signal-to-H.264-conversion (even though it feels overkill), and also added hotplug support. 1.4.x added multichannel audio support including support for MIDI controllers, and also a disk space indicator (because when you run out of disk during production without understanding that's what happens, it really sucks), and brought extensive end-user documentation. And 1.5.x, in development right now, will add HDMI/SDI output, which, like all the previous changes, requires various rearchitecting and fixing. Of course, there are lots of things that haven't changed as well; the basic UI remains the same, including the way the theme (governing the look-and-feel of the finished video stream) works. The basic design has proved sound, and I don't think I would change a lot if I were to design something like 1.0.0 again. As a small free software project, you have to pick your battles, and I'm certainly glad I didn't start out doing something like network support (or a distributed architecture in general, really). So what's for the next year of Nageru? It's hard to say, and it will definitely depend on the concrete needs of events. A hot candidate (since I might happen to need it) is chroma keying, although good keying is hard to get right and this needs some research. There's also been some discussion around other concrete features, but I won't name them until a firm commitment has been made; priorities can shift around, and it's important to stay flexible. So, enjoy FOSDEM! Perhaps I'll return with a talk in 2018. In the meantime, I'll preparing the stream for the 2017 edition of Fyrrom, and I know for sure there will be more events, more features and more experiences to be had. And, inevitably, more bugs. :-)

27 December 2016

Gunnar Wolf: Giving up on the Drupal 8 debianization

I am sad (but feel my duty) to inform the world that we will not be providing a Drupal 8 package in Debian. I filed an Intent To Package bug a very long time ago, intending to ship it with Jessie; Drupal 8 was so deep a change that it took their community overly long to achieve and stabilize. Still, Drupal 8 was released over a year ago today. I started working on debianizing the package shortly afterwards. There is also some online evidence As my call for help sent through this same blog. I have been too busy this last year. I let the packaging process lay dormant for too long, without even touching it for even half a year. Then, around September, I started working with the very nice guys of Indava, David and Enrique, and did very good advances. They clearly understood Debian's needs when it comes to full source inclusion (as D8 ships many minified Javascript libraries), attribution (as additionally to all those, many third-party PHP projects are bundled in the infamous vendor/ directory), and system-wide dependency management (as Drupal builds on some frameworks and libraries already available within Debian, chiefly Symfony, Doctrine, Twig... Even more, most of them appeared to work at the version levels we will be shipping, so all was dandy and for some weeks, I was quite optimistic on finishing the package on time and with the needed quality and testing. Yay! But... Reality bites. When I started testing my precious package... It broke in horrible ways. Uncomprehensible PHP errors (and I have to add here, I am a PHP newbie and am reluctant to learn better a language that strikes me as so inconsistent, so ugly), which we spent some time tackling... Of course, configuration changes are more than expected... But, just as we Debianers learnt some important lessons after the way-too-long Sarge freeze (ten years ago, many among you won't remember those frustrating days), Drupal learnt as well. They changed their release strategy Instead of describing it, those interested can read it at its source. What it meant for me, sadly, is that this process does not align with the Debian maintenance model. This means: The Drupal API stays mostly-stable between 8.0.x, 8.1.x, 8.2.x, etc. However, Drupal will incorporate new versions of their bundled libraries. I understood the new versions would be incorporated at minor-level branches, but if I read correctly some of my errors, some dependencies change even at patch-level updates. And... Well, if you update a PHP library, and the invoking PHP code (that is, Drupal) relies in this new version... Sadly, it makes it unmaintainable for Debian. So, long story short: I have decided to drop Drupal8 support in Debian. Of course, if somebody wants to pick up the pieces, the Git repository is still there (although I do plan on erasing it in a couple of weeks, as it means useless waste of project resources otherwise), and you could probably even target unstable+backports in a weird way (as it's software that, given our constraints, shouldn't enter testing, at least during a freeze). So... Sigh, a tear is dropped for every lost hour of work, and my depeest regrets to David and Enrique who put their work as well to make D8 happen in Debian. I will soon be closing the ITP and... Forgetting about the whole issue?

22 June 2016

Satyam Zode: GSoC 2016 Week 4 and 5: Reproducible Builds in Debian

This is a brief report on my last week work with Debian Reproducible Builds. In week 4, I mostly worked on designing an interfaces and tackling different issues related to argument completion feature of diffoscope and in week 5 I worked on adding hiding .buildinfo from .changes files. Update for last week s activities Goal for upcoming week: I am thankful to Shirish Agarwal for helping me through visa process. But, unfortunately I won t get visa till 5th July. So I don t think, I would make it to debconf this year. I will certainly attend Debconf 2017. Good news for me is I have passed mid-term evaluations of Google Summer of Code 2016. I will continue my work to improve Debian. Even, I have post GSoC plans ready for Debian project ;) Have a nice day :)

20 June 2016

Joey Hess: twenty years of free software -- part 1 ikiwiki

I'm forty years old. I've been developing free software for twenty years. A decade ago, I wrote a series of posts about my first ten years of free software, looking back over projects I'd developed. These retrospectives seem even more valuable in retrospect; there are things in the old posts that jog my memory, and other details I've forgotten by now. So, I'm doing it again. Over the next two weeks (with a week off in the middle for summer vacation), I'll be writing one post each day about a free software project I've developed in the past decade.
We begin with Ikiwiki. I started it 10 years ago, and still use it to this day; it's the engine that builds this website, and nearly all my other websites, as well as wikis and websites belonging to tons and tons of other projects, like NetBSD, X.org, Freedesktop.org, FreedomBox and many other users. Indeed I'm often reading a website and find myself wondering "hey.. is this using Ikiwiki?", and glance at the html and find that yes, it is. So, Ikiwiki is a reasonably successful and widely used peice of software, at least in its niche. More important to me, it was a static site generator before we knew what those were. It wasn't the first, but it broke plenty of new ground. I'm particularly proud of the way it combines a wiki with blogging support seamlessly, and the incremental updating of the static pages including updating wikilinks and backlinks. Some of these and other features are still pretty unique to Ikiwiki despite the glut of other static site generators available now. Ikiwiki is written in Perl, which was great for getting lots of other contributions (including many of its 113 plugins), but has also held it back some lately. There are less Perl programmers these days. And over the past decade, concurrency has become very important, but Ikiwiki's implementation is stubbornly single threaded, and multithreading such a Perl program is a losing propoisition. I occasionally threaten to rewrite it in Haskell, but I doubt I will. Ikiwiki has several developers now, and I'm the least active of them. I stepped back because I can't write Perl very well anymore, and am mostly very happy with how Ikiwiki works, so only pop up now and then when something annoys me. Next: twenty years of free software -- part 2 etckeeper

16 April 2016

Matthias Klumpp: Introducing AppStream-Generator

AppStream GeneratorSince mid-2015 we were using the dep11-generator in Debian to build AppStream metadata about available software components in the distribution. Getting rid of dep11-generator Unfortunately, the old Python-based dep11-generator was hitting some hard limits pretty soon. For example, using multiprocessing with Python was a pain, since it resulted in some very hard-to-track bugs. Also, the multiprocessing approach (as opposed to multithreading) made it impossible to use the underlying LMDB database properly (it was basically closed and reopened in each forked off process, since pickling the Python LMDB object caused some really funny bugs, which usually manifested themselves in the application hanging forever without any information on what was going on). Additionally to that, the Python-based generator forced me to maintain two implementations of the AppStream YAML spec, one in C and one in Python, which consumes quite some time. There were also some other issues (e.g. no unit-tests) in the implementation, which made me think about rewriting the generator. Adventures in Go / Rust / D Since I didn t want to write this new piece of software in C (or basically, writing it in C was my last option  ), I explored Go and Rust for this purpose and also did a small prototype in the D programming language, when I was starting to feel really adventurous. And while I never intended to write the new generator in D (I was pretty fixated on Go ), this is what happened. The strong points for D for this particular project were its close relation to C (and ease of using existing C code), its super-flat learning curve for someone who knows and likes C and C++ and its pretty powerful implementations of the concurrent and parallel programming paradigms. That being said, not all is great in D and there are some pretty dark spots too, mainly when it comes to the standard library and compilers. I will dive into my experiences with D in a separate blogpost. What good to expect from appstream-generator? So, what can the new appstream-generator do for you? Basically, the same as the old dep11-generator: It will extract metadata from a distribution s package archive, download and resize screenshots, search for icons and size them properly and generate reports in JSON and HTML of found metadata and issues. LibAppStream-based parsing, generation of YAML or XML, multi-distro support, As opposed to the old generator, the new generator utilizes the metadata parsers and writers of libappstream. This allows it to return the extracted metadata as AppStream YAML (for Debian) or XML (everyone else) It is also written in a distribution-agnostic way, so if someone wants to use it in a different distribution than Debian, this is possible now. It just requires a very small distribution-specific backend to be written, all of the details of the metadata extraction are abstracted away (just two interfaces need to be implemented). While I do not expect anyone except Debian to use this in the near future (most distros have found a solution to generate metadata already), the frontend-backend split is a much cleaner design than what was available in the previous code. It also allows to unit-test the code properly, without providing a Debian archive in the testsuite. Feature Flags, Optipng, The new generator also allows to enable and disable certain sets of features in a standardized way. E.g. Ubuntu uses a language-pack system for translations, which Debian doesn t use. Features like this can be implemented as disableable separate modules in the generator. We use this at time to e.g. allow descriptions from packages to be used as AppStream descriptions, or for running optipng on the generated PNG images and icons. No more Contents file dependency Another issue the old generator had was that it used the Contents file from the Debian archive to find matching icons for an application. We could never be sure whether the contents in the Contents file actually matched the contents of the package we were currently dealing with. What made things worse is that at Ubuntu, the archive software is only updating the Contents file weekly daily (while the generator might run multiple times a day), which has lead to software being ignored in the metadata, because icons could not yet be found. Even on Debian, with its quickly-updated Contents file, we could immediately see the effects of an out-of-date Contents file when updating it failed once. In the new generator, we read the contents of each package ourselves now and store them in a LMDB database, bypassing the Contents file and removing the whole class of problems resulting from missing or wrong contents-data. It can t all be good, right? That is true, there are also some known issues the new generator has: Large amounts of RAM required The better speed of the new generator comes at the cost of holding more stuff in RAM. Much more. When processing data from 5 architectures initially on Debian, the amount of required RAM might lie above 4GB, with the OOM killer sometimes being quicker than the garbage collector That being said, on subsequent runs the amount of required memory is much lower. Still, this is something I am working on to improve. What are symbolic links? To be faster, the appstream-generator will read the md5sum file in .deb packages instead of extracting the payload archive and reading its contents. Since the md5sums file does not list symbolic links, symlinks basically don t exist for the new generator. This is a problem for software symlinking icons or even .desktop files around, like e.g. LibreOffice does. I am still investigating how widespread the use of symlinks for icons and .desktop files is, but it looks like fixing packages (making them not-symlink stuff and rather move the files) might be the better approach than investing additional computing power to find symlinks or even switch back to parsing the Contents file. Input on this is welcome! Deploying asgen I finished the last pieces of the appstream-generator (together with doing lots of other cool things and talking to great people) at the GNOME Software Hackfest in London last week (detailed blogposts about things that happened there will follow many thanks once again for the Ubuntu community for sponsoring my attendance!). Since today, the new generator is running on the Debian infrastructure. If bigger issues are found, we can still roll back to the old code. I decided to deploy this faster, so we can get some good testing done before the Stretch release. Please report any issues you may find!

30 March 2016

Julien Danjou: The OpenStack Schizophrenia

When I started contributing to OpenStack, almost five years ago, it was a small ecosystem. There were no foundation, a handful of projects and you could understand the code base in a few days. Fast forward 2016, and it is a totally different beast. The project grew to no less than 54 teams, each team providing one or more deliverable. For example, the Nova and Swift team each one produces one service and its client, whereas the Telemetry team produces 3 services and 3 different clients. In 5 years, OpenStack went to a few IaaS projects, to 54 different teams tackling different areas related to cloud computing. Once upon a time, OpenStack was all about starting some virtual machines on a network, backed by images and volumes. Nowadays, it's also about orchestrating your network deployment over containers, while managing your application life-cycle using a database service, everything being metered and billed for. This exponential growth has been made possible with the decision of the OpenStack Technical Committee to open the gates with the project structure reform voted at the end of 2014. This amendment suppresses the old OpenStack model of "integrated projects" (i.e. Nova, Glance, Swift ). The big tent, as it's called, allowed OpenStack to land new projects every month, growing from the 20 project teams of December 2014 to the 54 we have today multiplying the number of projects by 2.7 in a little more than a year. Amazing growth, right? And this was clearly a good change. I sat at the Technical Committee in 2013, when projects were trying to apply to be "integrated", after Ceilometer and Heat were. It was painful to see how the Technical Committee was trying to assess whether new projects should be brought in or not. But what I notice these days, is how OpenStack is still stuck between its old and new models. On one side, it accepted a lot of new teams, but on the other side, many are considered as second-class citizens. Efforts are made to continue to build an OpenStack project that does not exist anymore. For example, there is a team trying to define what's OpenStack core, named DefCore. That is looking to define which projects are, somehow, actually OpenStack. This leads to weird situations, such as having non-DefCore projects seeing their doc rejected from installation guides. Again, I reiterated my proposal to publish documentation as part of each project code to solve that dishonest situation and put everything on a level playing field Some cross-projects specs are also pushed without implication of all OpenStack projects. For example, The deprecate-cli spec which proposes to deprecate command-line interface tools proposed by each project had a lot of sense in the old OpenStack sense, where the goal was to build a unified and ubiquitous cloud platform. But when you now have tens of projects with largely different scopes, this start making less sense. Still, this spec was merged by the OpenStack Technical Committee this cycle. Keystone is the first project to proudly force users to rely on openstack-client, removing its old keystone command line tool. I find it odd to push that specs when it's pretty clear that some projects (e.g. Swift, Gnocchi ) have no intention to go down that path. Unfortunately, most specs pushed by the Technical Committee are in the realm of wishful thinking. It somehow makes sense, since only a few of the members are actively contributing to OpenStack projects, and they can't by themselves implement all of that magically. But OpenStack is no exception in the free software world and remains a do-ocracy. There is good cross-project content in OpenStack, such as the API working group. While the work done should probably not be OpenStack specific, there's a lot that teams have learned by building various HTTP REST API with different frameworks. Compiling this knowledge and offering it as a guidance to various teams is a great help. My fellow developer Chris Dent wrote a post about what he would do on the Technical Committee. In this article, he points to a lot of the shortcomings I described here, and his confusion between OpenStack being a product or being a kit is quite understandable. Indeed, the message broadcasted by OpenStack is still very confusing after the big tent openness. There's no enough user experience improvement being done. The OpenStack Technical Committee election is opened for April 2016, and from what I read so far, many candidates are proposing to now clean up the big tent, kicking out projects that do not match certain criteria anymore. This is probably a good idea, there is some inactive project laying around. But I don't think that will be enough to solve the identity crisis that OpenStack is experiencing. So this is why, once again this cycle, I will throw my hat in the ring and submit my candidacy for OpenStack Technical Committee.

2 March 2016

Steinar H. Gunderson: Nageru FOSDEM talk video

I got tired of waiting for the video of my FOSDEM talk about Nageru, my live video mixer, to come out, so I made an edit myself. (This wouldn't have been possible without getting access to the raw video from the team, of course.) Of course, in maximally bad timing, the video team published their own (auto-)edit of mine and a lot of other videos on the very same day (and also updated their counts to a whopping 151 fully or partial lost videos out of a bit over 500!), so now there's two competing ones. Of course, since I have only one and not 500 videos to care about, I could afford to give it a bit more love; in particular, I spliced in digital versions of the original slides where appropriate, modified the audio levels a bit when there are audience questions, added manual transcriptions and so on, so I think it ended up quite a bit better. Nageru itself is also coming pretty nicely along, with an 1.1.0 release that supports DeckLink PCI cards (not just the USB ones) and also the NVIDIA proprietary driver, for much increased oomph. It's naturally a bit more quiet now than it was, though conferences always tend to generate a flurry of interest, and then you get back to other interests afterwards. You can find the talk on YouTube; I'll also be happy to provide a full-quality master of my edit if FOSDEM or anyone else wants. Enjoy :-)

7 November 2015

Wouter Verhelst: Minidebconf Cambridge

My main reason for being here was to join the Debconf video team sprint. But hey, since I'm here anyway, why not join all of it, right? Right. Apart from the video team stuff that I've been involved in, I've been spending the last few days improving NBD: There're a few more things that I'd like to see fixed, but after that I'll probably release 3.12 (upstream and in Debian) sometime later this week. With that, I'll be ready to start tackling Debian bug #796633, to provide proper systemd support in nbd-client.

10 September 2015

Antoine Beaupr : Switched to bootstrap theme

I finally gave up and drank the cool-aid of the Bootstrap theme. The main reason was that I noticed the site was basically unusable on any mobile device, including tablet computers like the iPad, which are unfortunately very, very common. Phones wouldn't render the site in a legible way either, unfortunately. The change is rather drastic, so I figured it was important to mention it here. I am not so satisfied pretty exctied with the result: the theme is very basic, if not absent. I actually like that purified form now, and it works across devices now much better. But I do feel I just made my blog look like everyone else that uses Bootstrap. We seem to enter an era where non-graphic designers (like me) are back to building web pages that all look the same, not very different, in a way, from the old days of plain HTML, like the default ikiwiki theme. I did some work to change the look at least minimally, but the top black navbar is really a killer giveaway this is a bootstrap theme. Bootstrap does a lot for the typography, at least when checking the presslabs checklist or the practical typography checklist, but I still had to change the main body width, which helped a lot I think. Anyways, the old Night City theme is still available for download and I could still flip it back on here if I need to.

What changed The new theme keeps the distractions away, but ironically, it's slower than the previous theme even though there are no images and basically no colors at all. On my tests, it now loads in about 3.42 seconds on the "Regular 2G (250Kbps)" simulation of Chromium with 72KB in 12 requests. Whereas my tests with the previous theme were taking 3.25 seconds with 43.5KB in 17 requests. This is due to the Bootstrap CSS, which takes a whopping 19KB itself, but especially because I now load JQuery, which takes a whopping 32.8KB! And that is probably gzip-compressed, as the original is more around 90KB. But at least the thing is readable on phones and tablets now. Compare:

Samsung S3, before Screenshot of a fake rendering on the Samsung Galaxy S3 in night city

Samsung S3, after Screenshot of a fake rendering on the Samsung Galaxy S3 in bootstrap I like this: much simpler, purer version on smaller devices. And we see what counts: the freaking text. Of course, on tablets, it doesn't fare as well: the top menu gets wrapped around...

Apple iPad, before Screenshot of a fake rendering on the iPad in night city

Apple iPad, after Screenshot of a fake rendering on the iPad in bootstrap Apparently, the fix would be for me to "customize the @grid-float-breakpoint variable" in LESS (which I thought was a pager, but seems to be a CSS preprocessor, graaah). Still, I think it's better that way, more readable, and less crufty.

How it was done After doing a thorough evaluation of all the Bootstrap ikiwiki themes I could find (there are more than 4 at least!), I figured I prefered the Jak Linux one. It seemed to be better implemented than the others, cleaner, and I liked the mean black border on top. Black is cool. Oh, and I liked the subtle footer as well. Subtle is cool. Unfortunately, that theme originally required a custom plugin to have a menu on top, which i thought was silly. So I patched it to make it work with the sidebar plugin, which turned out to be trivially simple: just dump the sidebar content, and make the sidebar page have explicit HTML tags in it that match Bootstrap's required classes. I also added the regular action links in the top navbar, but that does overload it quite a bit. I also did some code cleanups and various other small changes, all of which are available in my personnal git repo.

What doesn't work So of course, there are always problems. The first problem is the extra bandwidth usage. Not sure that can be fixed, other than switching to the upcoming Bootstrap 4, which is smaller than Bootstrap 3, bizarrely. The more annoying problems are weird alignment issues. If the screen gets two narrow without kicking some collapse rules in place (the iPad bug above), the navbar rows overflow and look ugly. Even more bizarre, in some cases the right navbar can just completely disappear, in fact, that's the only fix I could find: to assign the hidden-xs and hidden-sm classes to the navbar-right <UL> element. But then it just goes away, which is pretty darn stupid. Still - I assigned the classes to a few actions that seemed low priority, both in the theme and in the sidebar page. Similarly, the search form at the bottom of the page doesn't seem to want to fit in the footer properly. No idea why and I can't bother to figure that one out. I ended up merging this with the backlinks, tags and trails navigation items. Oh, and amazingly enough: comments are simply not rendered at all right now. Oops. I seemed to have picked the single bootstrap Ikiwiki theme that does *not* have comments rendering... Aaaargh. For now I just copy-pasted stuff from the [ramseydsilva][] theme and it looks pretty ugly, but at least it works. I ended up theming comments the same way i did for Night City, which looks pretty good, but is not in sync with the LESS stuff in Bootstrap. I also add to add back trails, backlinks and favicon... looks like the Jak theme wasn't made for a blog after all... but that's now fixed! :) Another improvement would be to change the font from the canonical Helvetica to something prettier, like suggested in Practical Typography, in the font recommendations section, or the presslabs checklist. So far I have settled on Mozilla's Fira font (which means, yes, one set of objects is actually loading from the CDN, sorry). Feedback welcome.

2 January 2015

John Goerzen: Sound players: Adventures with Ampache, mpd, pulseaudio, Raspberry Pi, and Logitech Media Server

I finally decided it was about time to get my whole-house sound project off the ground. As an added bonus, I d like to be able to stream music from my house to my Android phone. Some Background It was about 2.5 years ago that I last revisited the music-listening picture on Linux. I used Spotify for awhile, but the buggy nature of its support for local music eventually drove me up the wall. I have a large collection of music that will never be on Spotify (local choirs, for instance) and this was an important feature. When Google Play Music added the feature of uploading your local collection, I used that; it let me stream music from my phone in my car (using the Bluetooh link to the car). I could also listen at home on a PC, or plug my phone into various devices to play. But that was a hassle, and didn t let me have music throughout the house. Google Play is reasonable for that, but it has a number of really glaring issues. One is that it often gets album artwork wrong; it doesn t use the ID3 tags embedded in the files, but rather tries to guess . Another is that the sync is only add . Move files to another place in your collection, or re-rip them to FLAC and replace your old MP3s, and suddenly they re in Google Play twice. It won t ever see updated metadata, either quite a hassle for someone that uses Musicbrainz and carefully curates metadata. My Hardware I already have an oldish PC set up as a entertainment box running MythTV. It is a diskless system (boots PAE over the network and has NFS root) and very quiet. It has video output to my TV, and audio output via S/PDIF to my receiver. It is one logical audio frontend. My workstation in my office is another obvious place. My kitchen has a radio with a line in jack, and I also have a small portable speaker with a line in jack to make the last two options. I also have a Raspberry Pi model B that I bought awhile ago and was looking for a purpose, so I thought this should be cheap and easy, right? Well. Cheap, yes. Easy, not so much. First attempt: Ampache The Ampache project produces quite a nice piece of software. Ampache has matured significantly in the last 2.5 years, and the usability of its web-interface with HTML5 and Flash players as options is quite impressive. It is easily as usable as Google s, though its learning curve is rather more steep. There are multiple Android apps for Ampache to stream remotely. And, while most are terribly buggy and broken, there is at least one that seems to work well. Ampache can also output m3u/pls files for a standalone player. It does on-the-fly transcoding. There are some chinks in the armor, however. The set of codecs that are transcoded or passed through is a global setting, not a per-device setting. The bitrates are per-account, so you can t easily have it transcode FLAC into 320Kbps mp3 for streaming on your LAN and 128Kbps MP3 for streaming to your phone. (There are some hacks involving IP address ranges and multiple accounts, but they are poorly documented and cumbersome.) Ampache also has a feature called localplay in which it drives local players instead of remote ones. I tried to use this in combination with mpd to drive music to the whole house. Ampache s mpd interface is a bit odd; it actually loads things up into mpd s queue. Sadly it shares the same global configs as the rest. Even though mpd is perfectly capable of handling FLAC audio, the Ampache web player isn t, so you have to either make it transcode mp3 for everything or forego the web player (or use a second account that has transcode everything set). Frustratingly, not one of the Android clients for Ampache is even remotely compatible with Localplay, and some will fail in surprising ways if you have been using Localplay on the web client. So let s see how this mpd thing worked out. Ampache with mpd for whole-house audio The primary method here is to use mpd s pulseaudio driver. I configured it like so:
audio_output
type "pulse"
name "MPD stream"
#server "remote_server" # optional
sink "rtp" # optional
mixer_type "software"

Then in /etc/pulse/system.pa:
load-module module-null-sink sink_name=rtp format=s16be channels=2 rate=44100 sink_properties="device.description='RTP Multicast Sink'"
load-module module-rtp-send source=rtp.monitor
This tells Pulse to use multicast streaming to the LAN for the audio packets. Pulse is supposed to have latency synchronization to achieve perfect audio everywhere. In practice, this works somewhat poorly. Plus I have to install pulse everywhere, which inserts its tentacles way too deeply into the ALSA stack for my taste. (alsamixer suddenly turns useless by default, for instance.) But I gave it a try. After much fiddling Pulse is rather poorly documented and the interactions of configuration tools with it even more so I found a working configuration. On my MythTV box, I added:
load-module module-rtp-recv
load-module module-native-protocol-tcp auth-ip-acl=127.0.0.1;x.x.x.x
The real annoyance was getting it to set the output through the S/PDIF digital port. Finally I figured out the magic potion:
set-card-profile 0 output:iec958-stereo+input:analog-stereo
This worked reasonably well. The Raspberry Pi was a much bigger challenge, however. I put Raspbian on it easily enough, and installed Pulse. But apparently it is well-known in the Pulse community that Pulse s RTP does not work well with wifi. Multicast itself works poorly with wifi in general, and Pulse won t do unicast RTP. However, Pulse assumes very low latency and just won t work well with wifi at all. Morever, pulseaudio on the rpi is something of a difficult beast to tame. It has crackling audio, etc. I eventually got it working decently with: daemon.conf:
realtime-scheduling = yes
realtime-priority = 5
resample-method = src-sinc-fastest
default-sample-rate = 44100
default-fragments = 4
default-fragment-size-msec=20
and in system.pa, commented out the module-udev-detect and module-detect, and added:
load-module module-alsa-card device_id=0 tsched=0 fragments=10 fragment_size=640 tsched_buffer_size=4194384 tsched_buffer_watermark=262144
plus the rtp-recv and tcp protocol lines as before. This got it working with decent quality, but it was always out of sync by at least a few tenths of a second, if not a whole second, with the other rooms. Brief diversion: mplayer Many players besides pulseaudio can play the RTP streams that pulseaudio generates. mplayer, for instance, can. I found this worked on the Raspberry Pi:
mplayer -really-quiet -cache 64 -cache-min 95 -demuxer rawaudio -rawaudio format=0x20776172 rtp://224.0.0.56:46510
But this produced even worse sync. It became clear that Ampache was not going to be the right solution. Logitech Media Server I have looked at this a few times over the years, but I ve somehow skipped it. But I looked at it again now. It is an open source (GPL) server, and was originally designed to work with Logitech hardware. There are now all sorts of software clients out there. There is a helpful wiki about it, although Logitech has rebranded the thing so many times, it s not even consistent internally on what the heck it s called (is it LMS? or Squeezebox Server? or Slimserver? The answer is: yes.) LMS does no web-based streaming. At all. But I thought I d give it a try. Installation is a little weird (its .deb packages up binary perl modules for things that are already in Debian, for half a dozen architectures and many Perl versions.) I had some odd issues but eventually it worked. It scanned my media collection. I installed squeezelite on my workstation as my first player, and things worked reasonably well out of the box. I proceeded to install squeezelite on the Raspberry Pi and MythTV server (where I had to compile libsoxr for wheezy, due to an obscure error I couldn t find the cause of until I used strace). Log in to the LMS web interface, tell it to enable synchronization, and bam! Perfect synchronization! Incredible. But I ran into two issues: one was that it used a lot of CPU even on my workstation (50% or more, even when idle). Strangely, it used far less CPU on the pi than did pulseaudio. Secondly, I d get annoying clicks from it from time to time. Some debugging and investigation revealed that they were both somewhat related; it was getting out of sync with the pi and correcting. Initially I assumed this to be an issue with the pi, but tracked it down to something else. I told it to upsample to 48kHz and this made the problem go away. My command on PC hardware is:
squeezelite -a 200:10 -o dmix -u hLX -z
And on the Raspberry Pi:
squeezelite -a 800:10:16:0 -n kitchen -z
It is working perfectly now. LMS even has an Android remote control app that can control all devices together, queue up playlists, etc. Very slick. The Raspberry Pi The Raspberry Pi, even the model B+, apparently has notoriously bad audio output. I noticed this first as very low volume on the output. I found a $11 USB DAC on Amazon that seems to do the trick, so we will see. I am also using the $10 Edimax USB Wifi and it seems to work well. Other Options The big-name other option here is Sonos. They have perfect sync audio already working. But they re pricy, closed, and proprietary; you have to buy Sonos speakers, transmitters, etc. for the entire house. Their receivers start at $200, and it s at least $350 to integrate an existing stereo into the system. I d be shelling out over $1000 to use Sonos for this setup. As it is, I ve bought less than $100 of equipment (a second Pi and its accessories) and getting output that, while no doubt not quite as pristine, it still quite nice and acceptable. Quite nice. Update: LMS with Apache It turns out that Logitech Media Server can automatically save created playlists to a spot where Ampache can find them. So an Ampache streaming player can be used to access the same collection LMS uses, with full features. Nice!

3 December 2014

Michael Stapelberg: Announcing the new Debian Code Search Instant

For the last few months, I have been working on a new version of Debian Code Search, and today it s going live! I call it Debian Code Search Instant, for multiple reasons, see below. A lot faster The new Debian Code Search is still hosted by Rackspace (thank you!), but our new architecture finally allows us to use their Performance Cloud Servers hardware generation. The old code search architecture was split across 5 different servers, however the search itself was only performed on a single machine with a network attached block volume backed by SSD. The new Code Search spreads out both the trigram index and the source code onto 6 different servers, and thanks to the new hardware generation, we have a locally connected SSD drive in each of them. So, we now get more than 6 times the IOPS we ve had before, at much lower latency :). Grouping by Debian source package The first feature request we ever got for Code Search was to group results by Debian source package. I ve tried tackling that before, but it s a pretty hard problem. With the new architecture and capacity, this finally becomes feasible. After your query completes, there is a checkbox called Group search results by Debian source package . Enable it, and you ll get neatly grouped results. For each package, there is a link at the bottom to refine the search to only consider that package. In case you are more interested in the full list of packages, for example because you are doing large-scale changes across Debian, that s also possible: Click on the symbol right next to the Filter by package list, and a curl command will be revealed with which you can download the full list of packages. Expensive queries (with lots of results) now possible Previously, we had a 60 second timeout during which queries must complete. This timeout has been completely lifted, and long-running queries with tons and tons of results are now possible. We kindly ask you to not abuse this feature it s not very exciting to play with complexity explosion in regular expression engines by preventing others from doing their work. If you re interested in that sort of analysis, go grab the source code and play with it on your own machine :). Instant results While your query is running, you will almost immediately see the top 10 results, even though the search may still take a while. This is useful to figure out if your regular expression matches what you thought it would, before having to wait until your 5 minute query is finished. Also, the new progress bar tells you how far the system is with your search. Instant indexing Previously, Code Search was deployed to a new set of machines from scratch every week in order to update the index. This was necessary because the performance would severely degrade while building the index, so we were temporarily running with twice the amount of machines until the new version was up. In the new architecture, we store an index for each source package and then merge these into one big index shard. This currently takes about 4 minutes with the code I wrote, but I m sure this can be made even faster if necessary. So, whenever new packages are uploaded to the Debian archive, we can just index the new version and trigger a merge. We get notifications about new package uploads from FedMsg. Packages that are not seen on FedMsg for some reason are backfilled every hour. The time between uploading a package and being able to find it in Debian Code Search therefore now ranges from a couple of minutes to about an hour, instead of about a week! New, beautiful User Interface Since we needed to rewrite the User Interface anyway thanks to the new architecture, we also spent some effort on making it modern, polished and beautiful. Smooth CSS animations make you aware of what s going on, and the search results look cleaner than ever. Conclusion At least to me, the new Debian Code Search seems like a significant improvement over the old one. I hope you enjoy the new features into which I put a lot of work. In case you have any feedback, I m happy to hear it (see the contact link at the bottom of every Code Search site).

24 November 2014

John Goerzen: My boys love 1986 computing

Yesterday, Jacob (age 8) asked to help me put together a 30-year-old computer from parts in my basement. Meanwhile, Oliver (age 5) asked Laura to help him learn cursive. Somehow, this doesn t seem odd for a Saturday at our place. 2014-11-22 18.58.36 Let me tell you how this came about. I ve had a project going on for a while now to load data from old floppies. It s been fun, and had a surprise twist the other day: my parents gave me an old TRS-80 Color Computer II (aka CoCo 2 ). It was, in fact, my first computer, one they got for me when I was in Kindergarten. It is nearly 30 years old. I have been musing lately about the great disservice Apple did the world by making computers easy to learn namely the fact that few people ever bother to learn about them. Who bothers to learn about them when, on the iPhone for instance, the case is sealed shut, the lifespan is 1 or 2 years for many purchasers, and the platform is closed in lots of ways? I had forgotten how finicky computers used to be. But after some days struggling with IDE incompatibilities, booting issues, etc., when I actually managed to get data off a machine that had last booted in 1999, I had quite the sense of accomplishment, which I rarely have lately. I did something that was hard to do in a world where most of the interfaces don t work with equipment that old (even if nominally they are supposed to.) The CoCo is one of those computers normally used with a floppy drive or cassette recorder to store programs. You type DIR, and you feel the clack of the drive heads through the desk. You type CLOAD and you hear the relay click closed to turn on the tape motor. You wiggle cables around until they make contact just right. You power-cycle for the times when the reset button doesn t quite do the job. The details of how it works aren t abstracted away by innumerable layers of controllers, interfaces, operating system modules, etc. It s all right there, literally vibrating your desk. So I thought this could be a great opportunity for Jacob to learn a few more computing concepts, such as the difference between mass storage and RAM, plus a great way to encourage him to practice critical thinking. So we trekked down to the basement and came up with handfulls of parts. We brought up the computer, some joysticks, all sorts of tangled cables. We needed adapters, an old TV. Jacob helped me hook everything up, and then the moment of truth: success! A green BASIC screen! I added more parts, but struck out when I tried to connect the floppy drive. The thing just wouldn t start up right whenever the floppy controller cartridge was installed. I cleaned the cartridge. I took it apart, scrubbed the contacts, even did a re-seat of the chips. No dice. So I fired up my CoCo emulator (xroar), and virtually saved some programs to cassette (a .wav file). I then burned those .wav files to an audio CD, brought up an old CD player from the basement, connected the cassette in plug to the CD player s headphone jack, and presto instant programs. (Well, almost. It takes a couple of minutes to load a program from audio codes.) The picture above is Oliver cackling at one of the very simplest BASIC programs there is: number find. The computer picks a random number between 1 and 2000, and asks the user to guess it, giving a too low or too high clue with each incorrect guess. Oliver delighted in giving invalid input (way too high numbers, or things that weren t numbers at all) and cackled at the sarcastic error messages built into the program. During Jacob s turn, he got very serious about it, and is probably going to be learning about how to calculate halfway points before too long. But imagine my pride when this morning, Jacob found the new CD I had made last night (correcting a couple recordings), found my one-line instruction on just part of how to load a program, and correctly figured out by himself all the steps to do in order (type CLOAD on the CoCo, advance the CD to the proper track, press play on the player, wait for it to load on the CoCo, then type RUN). I ordered a replacement floppy controller off eBay tonight, and paid $5 for a coax adapter that should fix some video quality issues. I rescued some 5.25 floppies from my trash can from another project, so they should have plenty of tools for exploration. It is so much easier for them to learn how a disk drive works, and even what the heck a track is, when you can look at a floppy drive with the cover off and see the heads move. There are other things we can do with more modern equipment Jacob has shown a lot of interest in Arduino projects but I have so far drawn a blank on ways to really let kids discover how a modern PC (let alone a modern phone or tablet) works. Update Nov. 24: Every so often, the world surprises me by deciding to, well, read one of my random blog posts. For the benefit of those of you that don t already know my boys, you might want to know that among their common play activites are turning trees into pretend trains, typing at a manual typewriter, reading, writing their own books, using a cassette recorder, building a PC and learning to use bash or xmonad, making long paper tapes with an adding machine, playing records on a record player, building electric gizmos, and even making mud balls. I am often asked about the role of the computer in the lives, given that my hobby and profession involves computers. The answer: less than that of most of their peers. I look for opportunities for them to learn by doing, discovering, playing, or imagining. I make no presumption that they will develop the passion for computers that I did. What I want is for them to have the curiosity and drive to learn everything there is to know about whatever they do develop a passion for, so they will be great at it.

16 November 2014

Stefano Zacchiroli: Debsources Participation in FOSS Outreach Program

Jingjie Jiang selected as OPW intern for Debsources I'm glad to announce that Jingjie Jiang, AKA sophiejjj, has been selected as intern to work on Debsources as part of the FOSS Outreach Program (formerly known as Outreach Program for Women, or OPW). I'll co-mentor her work together with Matthieu Caneill. I've just added sophiejjj's blog to Planet Debian, so you will soon hear about her work in the Debian blogosphere. I've been impressed by the interest that the Debsources proposal in this round of OPW has spawned. Together with Matthieu I have interacted with more than a dozen OPW applicants. Many of them have contributed useful patches during the application period, and those patches have been in production at http://sources.debian.net since quite a while now (see the commit log for details). A special mention goes to Akshita Jha, who has shown a lot of determination in tackling both simple and complex issues affecting Debsources. I hope there will be other chances to work with her in the future. OPW internship will begin December 9th, fasten your seat belts for a boost in Debsources development!

20 September 2014

Francesca Ciceri: Four Ways to Forgiveness

"I have seen a picture," Havzhiva went on.
The Chosen was impassive; he might or might not know the word. "Lines and colors made with earth on earth may hold knowledge in them. All knowledge is local, all truth is partial," Havzhiva said with an easy, colloquial dignity that he knew was an imitation of his mother, the Heir of the Sun, talking to foreign merchants. "No truth can make another truth untrue. All knowledge is a part of the whole knowledge. A true line, a true color. Once you have seen the larger patttern, you cannot go back to seeing the part as the whole.
I've just finished to read "Four Ways to Forgiveness" by U.K Le Guin.
It deeply resonated within me, it's still there doing its magic in my brain, lingering in the corners of my mind, tickling my view of reality, humming with the beauty of ideas you didn't knew were inside you till you've seen them written on paper.
And then, you know they were there all along, you just didn't know how to make them into words.
Le Guin knows how to do it, wonderfully. I loved the whole book, but the last two stories were eye-openers.
Thanks Enrico for suggesting me this one, thanks dkg for having introduced me to Le Guin's books (with another fantastic book: The Left Hand of Darkness).

5 May 2014

Robert Collins: Distributed bugtracking quick thoughts

Just saw http://sny.no/2014/04/dbts and I feel compelled to note that distributed bug trackers are not new the earliest I personally encountered was Aaron Bentley s Bugs everywhere coming up on it s 10th birthday. BE meets many of the criteria in the dbts post I read earlier today, but it hasn t taken over the world and I think this is in large part due to the propogation nature of bugs being very different to code different solutions are needed. XXXX: With distributed code versioning we often see people going to some effort to avoid conflicts semantic conflicts are common, and representation conflicts extremely common.The idions Take for example https://bugs.launchpad.net/ubuntu/+source/ntp/+bug/805661. Here we can look at the nature of the content:
  1. Concurrent cannot-conflict content e.g. the discussion about the bug. In general everyone should have this in their local bug database as soon as possible, and anyone can write to it.
  2. Observations of fact e.g. the code change that should fix the bug has landed in Ubuntu or Commit C should fix the bug .
  3. Reports of symptoms e.g. Foo does not work for me in Ubuntu with package versions X, Y and Z .
  4. Collaboratively edited metadata tags, title, description, and arguably even the fields like package, open/closed, importance.
Note that only one of these things the commit to fix the bug happens in the same code tree as the code; and that the commit fixing it may be delayed by many things before the fix is available to users. Also note that conceptually conflicts can happen in any of those fields except 1). Anyhow my humble suggestion for tackling the conflicts angle is to treat all changes to a bug as events in a timeline e.g. adding a tag foo is an event to add foo , rather than an event setting the tags list to bar,foo then multiple editors adding foo do not conflict (or need special handling). Collaboratively edited fields would be likely be unsatisfying with this approach though last-writer-wins isn t a great story. OTOH the number of people that edit the collaborative fields on any given bug tend to be quite low so one could defer that to manual fixups. Further, as a developer wanting local access to my bug database, syncing all of these things is appealing but if I m dealing with a million-bug bug database, I may actually need the ability to filter what I sync or do not sync with some care. Even if I want everything, query performance on such a database is crucial for usability (something git demonstrated convincingly in the VCS space). Lastly, I don t think distributed bug tracking is needed it doesn t solve a deeply burning use case offline access would be a 90% solution for most people. What does need rethinking is the hugely manual process most bug systems use today. Making tools like whoopsie-daisy widely available is much more interesting (and that may require distributed underpinnings to work well and securely). Automatic collation of distinct reports and surfacing the most commonly experienced faults to developers offers a path to evidence based assessment of quality something I think we badly need.

25 March 2014

Francesca Ciceri: A crafty roundup: projects from the last months

A quick roundup of things I made during the last months, following tutorials here and there. finished projects for babies 1: a soft bunny more or less based on this one 2: skelly man! You can find pattern and instructions on Chez Beeper Bebe's blog 3: bibs: the embroidered one is based on this pattern by Charlotte Lyons on SouleMama's blog, the other re-uses the same pattern but add a bird applique instead. finished projects for women 1, 4: my first try with the beautiful Weekender Bag ended with something else entirely: apparently, where euclidean geometry principles apply, cutting a piece of fabric 10 cm smaller will result in a bag 10 cm smaller!
Go figure! Fhtang! I'm happy anyway with the final result, but it resembles more a tote bag than a weekender one. I then made also a successful Weekender, but haven't taken a pic yet. If you try that pattern, you may find this blogpost helpful. 2: This is my try at copycatting this model without a pattern and didn't end very well.
However, a couple lessons learned: 3: Blouse with knit-stretchy stuff! (You love me when I speak technical, don't you?)
I did that without pattern, mostly using a similar top I have as inspiration/guide

30 January 2014

Andrew Pollock: [life] Day 3, Playgroup

Today was our first expedition to the local Playgroup at the Bulimba Senior Citizen's Centre, an easy 5 minute bike ride from our place. I told Zoe we were going to Playgroup when Sarah dropped her around this morning, and she got very excited and jumped up and down. When Sarah asked her if she knew what Playgroup was, she said "no!" while continuing to jump up and down excitedly. We all laughed. I must have put this on my calendar when I was in Sydney last week, because I had it down for 8:30am, and we rocked up and there wasn't anyone there. A lady arrived to set up for a music class, and she said that the Playgroup didn't usually start until much later, so I figured I'd had a timezone fail and we went for a little bike ride around Bulimba while we waited. The lady who was running the Playgroup arrived at about 9:15am, and so we helped with the set up. Basically lots of toys, a couple of slides and some basic craft stuff. Heaps of the little IKEA round plastic stools. Parents (almost exclusively mothers) started trickling in from 9:30am. I've got a total name overload. We need name tags. Most of the girls seemed to be about 2 and a half, and a lot of the mothers had very young second children. Apparently a mother's group had joined en masse because they all had second children who were crawling, and so meeting in a park wasn't practical any more. Structurally, the first hour or so was just a free for all, and the last half an hour was some songs, a story and some playing under and on a parachute. All in all it was a good morning's activity, and at $20 for the term, nice and cheap. Apparently you're supposed to join Playgroup Queensland as well, for $40, but that takes care of liability insurance and whatnot. If only their website used SSL. Zoe mostly just wanted to play with me. We did a lot of role playing with some dolls, and she checked out the various toys. She loved that there was a stage, and took it upon herself to give an impromptu dance performance for a bit. We also did a little craft activity. She seemed more interested in helping flap the parachute than going under it, but enjoyed having a ride on it when the time came. I've just finished reading How Not to F*** Them Up by Oliver James. It was quite an interesting read, although not particularly applicable for my situation, as it more dealt with the first three years. It was interesting observing a whole bunch of different mothering techniques in action, both in dealing with their toddler children, and how they responded to their second infants. After Playgroup, we biked over to the local toy shop around the corner from home, and grabbed some birthday presents for the birthday party on Saturday and came home for lunch. Chicken quesadillas and some mango (I hadn't realised how much I've missed Queensland mangoes until we moved back, and I'm glad Zoe loves them as well). I got her down for a nap earlier today, and after she woke up, I asked Zoe if she wanted to write on the birthday cards we'd bought for the twins' birthday presents. She said she didn't know how to write her name (which isn't true) and wanted to paint them instead, so we compromised and used some sponge letters to stamp "Z O E" on the cards in paint. Zoe said that Eva's favourite colour was purple and Layla's was pink (or it may have been the other way around), so we stamped one with purple paint and mixed red and white to make something that looked more orange than pink and stamped the other one. Then Zoe wanted to glue the two paper plates together that we'd been using to hold the paint, so we did that, and then she wanted to "hand paint" a cardboard tube, which seemed to involve me directly squirting various colours into her hands and then her "painting" the tube with what was in her hands. The tube quickly ended up a brown/grey colour. All in all, it was a fabulous mess, and fortunately no walls or doors copped any paint on the way to the bathroom to clean up. After that, we cleaned up Zoe's room a bit, which resulted in her going off and playing with her Magna Doodle with Smudge for a bit, and then we played Doctor with her doctor set and then we did some baking. It was a good afternoon, and she only asked to watch TV once, when she first woke up from her nap, and we managed to successfully divert to other much more fun stuff.

17 January 2014

Chris Lamb: Captain Phillips: Pontius Pirate

https://chris-lamb.co.uk/wp-content/2014/phillips_b.jpg Somalia's chief exports appear to be morally-ambiguous Salon articles about piracy and sophomoric evidence against libertarianism. However, it is the former topic that Captain Phillips concerns itself with, inspired by the hijacking of the Maersk Alabama container ship in 2009. What is truth? In the end, Captain Phillips does not rise above Pontius Pilate in providing an answer, but it certainly tries using more refined instruments than irony or leaden sarcasm. This motif pervades the film. Obviously, it is based on a "true story" and brings aboard that baggage, but it also permeates the plot in a much deeper sense. For example, Phillips and the US Navy lie almost compulsively to the pirates, whilst the pirates only really lie once (where they put Phillips in greater danger). Notice further that Phillips only starts to tell the truth when he thinks all hope is lost. These telling observations become even more fascinating when you realise that they must be based on the testimony of the, well, liars. Clearly, deception is a weapon to be monopolised and there are few limits on what good guys can or should lie about if they believe they can save lives.
https://chris-lamb.co.uk/wp-content/2014/phillips_c.jpg

Even Phillip's nickname ("Irish") is a falsehood he straight-up admits he is an American citizen.

Futhermore, there is an utterly disarming epilogue where Phillips is being treated for shock by clinical efficient medical staff. Not only will this scuttle any "blanket around the shoulders" clich but is probably a highly accurate portrayal of what actually happens post-trauma. This echoes the kind of truth Werner Herzog aims for in his filmmaking as well his guilt-inducing duality between uncomfortable lingering and compulsive viewing. Lastly, a starter for a meta-discussion: can a film based on real-world events even be "spoilered"? Hearing headlines on the radio before you read your newspaper hardly robs you of a literary journey... Captain Phillips does have some quotidian problems. Firstly, the only tool for ratcheting up tension is for the Somalians to launch verbal broadsides at the Americans, with each compromise somehow escalating the situation. This technique is effective but well before the climatic rescue scene where it is really needed it has been subject to the most extreme diminishing returns. (I cannot be the first to notice the "Africans festooned with guns shouting incomphensively" trope I hope it is based on a Babel-esque mechanism of disorientation from miscommunication rather than anything more unsavoury.)
https://chris-lamb.co.uk/wp-content/2014/phillips_a.jpg

The racist idea that Africans prefer an AK-47 rotated about the Z-axis is socially constructed.

Secondly, the US Navy acts like a teacher with an Ofsted inspector observing quietly from the corner of the classroom; far too well-behaved it suspends belief, with no post-kill gloating or even the tiniest of post-arrest congratulations. Whilst nobody wants to see the Navy overreact badly to other military branches getting all the glory, nobody wants to see a suspiciously bland recruitment vehicle either. Paradoxically, this hermetic treatment made me unduly fascinated by them as if they were part of some military "uncanny valley". Two quick observations:
https://chris-lamb.co.uk/wp-content/2014/phillips_d.jpg

The drone footage: I'd love to write an essay about how Call of Duty might have influenced (or even be) cinema.

Finally, despite the title, the film is actually about two captains; the skillful liar Phillips and ... well, that's the real problem. Whilst Captain Muse is certainly no caricatured Hook, we are offered little depth beyond a "You're not just a fisherman" faux-revelation that leads nowhere. I was left inventing reasons for his akrasia so that he made any sense whatsoever. One could charitably argue that the film attempts to stay objective on Muse, but the inability for the film to take any obvious ethical stance actually seems to confuse and then compromise the narrative. What deeper truth is actually being revealed? Is this film or documentary? Worse still, the moral vacuum is invariably filled by the viewer's existing political outlook: are Somali pirates victims of circumstance who are forced into (alas, regrettable) swashbuckling adventures to pacify plank-threatening warlords? Or are they violent and dangerous criminals who habour an irrational resentment against the West, flimsily represented by material goods in shipping containers? Your improvised answer to this Rorschach test will always sit more haphazardly in the film than any pre-constructed treatment ever could. 6/10

Next.

Previous.