I have two annoyances with Appengine's NDB OR queries. One is that the order must be by key, the other is that OR queries are done in series, so ndb.OR(a == 1, a == 2) are done as query(a == 1) then query(a == 2). To work around this, I created a new cursor class that allows sorting by any attribute.

Using this cursor does come with it's own caveats, one is that the order between the queries is not maintained (fixable) and two the limit attribute is per query not for the results of all queries (fixable).

import base64
import json

from google.appengine.ext import ndb

class MultiCursor(object):
@staticmethod
def from_websafe_string(value):
values = None
try:
value = base64.urlsafe_b64decode(str(value))
values = json.loads(value)
except Exception as e:
logging.error('Invalid Cursor %s - %s', e, value)
if not values:
return MultiCursor()

cursors = MultiCursor()
for key, cursor in values.iteritems():
try:
if isinstance(cursor, basestring):
cursors.set(key, ndb.Cursor.from_websafe_string(cursor))
else:
cursors.set(key, cursor)
except:
logging.error('MultiCursor.from_string - bad cursor (%s) for %s in %s',
cursor, key, values)
return cursors

def to_websafe_string(self):
values =
for key, value in self.values:
if isinstance(value, ndb.Cursor):
values[key] = value.to_websafe_string()
else:
values[key] = value
return base64.urlsafe_b64encode(json.dumps(values))

def __init__(self, values=None):
self.values = values or

def get(self, key):
return self.values.get(key), self.values.get(key + _'done')

def set(self, key, cursor, more):
self.values[key] = cursor
self.values[key + _'done'] = not more

def __len__(self):
return len(self.values)

To use the cursor the code should look like:

@ndb.tasklet
def _query_A_by_value_async(value, cursor, limit=100):
start_cursor, done = cursor.get(value)
if not done:
query = Foo.query(Foo.A == value)
query = Foo.order(-Foo.last_access_time)
foos, end_cursor, more = yield query.fetch_page_async(
limit, start_cursor=start_cursor)
cursor.set(value, end_cursor, more)
raise ndb.Return(foos, end_cursor, more)
raise ndb.Return([], None, False)

Debian-installer is a great piece of software, which allows one to install the system in a variety of situations: manually, on a serial console, graphically, automatically, there are a number of scenarios it handles well. However, there is one issue: when installing Debian to a machine that requires firmware just to get at the network, getting things to work can be a hassle. There is a tarball containing the most-often required firmware blobs in order to be able to use the installer on such systems; but since they fail the DFSG, these cannot be part of Debian, for political reasons. To use them, you need to write the required firmware packages to a directory "/firmware" on a USB stick or similar, and put that in the system when asked. This is acceptable when you have just one system to install and have a USB stick handy; but if you have a lot of systems to install, or (as recently happened to me) you don't have a USB stick handy, then things become slightly more problematic. Luckily, there's an easy solution: you can update the initrd image so it contains the firmware packages already on it; when doing that, you don't need to deal with USB sticks anymore. Of course, using such an initrd image will cause the installer to automatically install all provided firmware packages, even if your hardware will function (at a basic level) without said firmware. Additionally, if you pass firmware packages to the installer in any way (through a USB stick, or through the method I'm about to describe, that makes no difference), and the installer determines that it might help to install one of these firmware packages, then the installed system will have them installed, and have the non-free repository enabled, which is not the default. Don't do this if you don't want that. Anyway, on to the actual procedure now:

• First, get yourself a "netboot.tar.gz" tarball from one of Debian's mirrors. This will get you a pxeboot.0 file, a set of pxeboot configuration files, a linux kernel file, and an initrd.gz file. It's the latter file that we need to modify
• Create a directory where you'll unpack the initrd image, 'cd' to it, and run zcat path/to/initrd.gz cpio -vi. This will unpack the initrd into your current directory. You can explore it if you want to see how debian-installer works, but other than that it's not very interesting.
• Add a directory "firmware" in the top-level extracted directory that you just created.
• Into this "firmware" directory, add all the firmware package files that you'll need.
• Now, run find . cpio -ov -H newc gzip --best > path/to/initrd.gz.

Tadaa! You've now created an initrd image that will not ask you for firmware images, instead just install them.

FAT USB sticks are traditionally formatted with FAT 32, because this file system is implemented by almost every operating system and device. Unfortunately, it sucks, as it cannot use more than 2 TiB, store files larger than 2 GiB or store symbolic links for instance. In a word, it is an obsolete and deficient file system.

exFAT Good news: someone addressed that problem. Bad new: that someone is Microsoft. So as you could expect, exFAT, the extended FAT, is a stinking proprietary, secret and patented file system. There are free implementations of that shit, but it is safer to stay away from it.UDF to the rescue! Good news: there is one file system that is implemented almost everywhere as well, and which does not suffer from such limitations. UDF, the Universal Disk Format, is an ISO standard originally designed for DVDs, but it is perfectly usable for USB sticks. It also supports POSIX permissions, with one killer feature for removable media: a file can belong to no specific person or group. So, to use it, assuming you USB stick is /dev/sdc:

$ dd if=/dev/zero of=/dev/sdc bs=1M count=1
$ mkudffs -b 512 --media-type=hd /dev/sdc

The initial dd is to erase the existing partition table or file system information, to prevent you USB stick from being detected as a FAT after it has been formatted with UDF. The -b 512 is to force a file system block size equal to the USB stick's physical block size, as required by the UDF specification. Adapt it if you have the luck of having a USB stick with an more appropriate block size. After that, you USB stick will be usable for reading and writing with GNU/Linux and the other free operating systems of course, but also with current versions of Windows (read-only with the outdated version XP) and with MacOS.

I'm struggling a little to keep to a schedule since the weekend, so I both didn't follow my normal evening exercise schedule nor did I write this up earlier when I could add more details. Ah well. But I'm still going to post something today, since I like this streak of (mostly) providing some useful or at least hopefully entertaining content. Today, I converted the Perl module build system inside WebAuth to use Module::Build instead of ExtUtils::MakeMaker, and it was a rousing success. I highly recommend doing this, and am going to be doing this with all of my other Perl packages, both embedded in larger packages and standalone. I have several packages where Perl XS modules are embedded within a larger Autoconf and Automake project that includes a shared library used by the Perl module. The largest problem with doing this is integrating the build systems in such a way that the Perl module is built against the in-tree version of the shared library, rather than some version of it that may already be on the system. I've managed to do this with ExtUtils::MakeMaker, but it was horribly ugly, involving overriding internal target rules and setting a bunch of other variables. With Module::Build and ExtUtils::CBuilder, it's much easier and even sensible. Both support standard ways of overriding Config settings, which provides just the lever required. So this:

    our $PATH = '@abs_top_builddir@/lib/.libs';
    my $lddlflags = $Config lddlflags ;
    my $additions = "-L$PATH @LDFLAGS@";
    $lddlflags =~ s%(^  )-L% $additions -L%;
    package MY;
    sub const_loadlibs  
        my $loadlibs = shift->SUPER::const_loadlibs (@_);
        $loadlibs =~ s%^(LD_RUN_PATH =.*[\s:])$main::PATH(: \n)%$1$2%m;
        return $loadlibs;
     
    package main;

(comments stripped to just show the code) became this:

    my @LDFLAGS = qw(-L@abs_top_builddir@/lib/.libs @LDFLAGS@);
    my $LDDLFLAGS = join q   , @LDFLAGS, $Config lddlflags ;

plus adding config => lddlflags => $LDDLFLAGS to the standard Module::Build configuration. So much better! At some point, I'll write up a set of guidelines on how to embed Perl XS modules into Automake projects.

So, today an endurance test can be declared as finished. In early 2008, for the first time, I paid for a cellular phone (as my previous ones were all 100% subsidized by the operator in a fixed plan). I got a Nokia N95. And, although at the beginnning I was quite thrilled with my smartphone (when such things were still a novelty), it didn't take much to me to start dumbing it down to what is really useful to me: A phone with a GPS. And the GPS only because it is the only toy I want in such a form factor. Anyway, despite the operator repeatedly offering me newer and more capable models, I kept this one, and as soon as I was free of the forced 18 month rental period, switched to a not-data-enabled, pay-as-you-go plan. I don't want to be connected to the intertubes when I'm away from a usable computer! But yes, five years are over what a modern phone wants to endure. Over time, I first started getting SIM card errors whenever the phone was dropped or slightly twisted - As I'm a non-frequent phone user, I didn't care much. Charging it also became a skill of patience, as getting the Nokia micro-connector to make contact has been less and less reliable. Over one year ago, the volume control (two sensors on the side) died after a phone drop (and some time later I found the switches broken from the mainboard loose) - A nuisance, yes, but nothing too bad. I don't know how, but some time ago the volume went down when using the radio, and as I can't raise it again, my phone became radio-disabled. And today, the screen died (it gets power, but stays black). I can blind-operate the phone, but of course, it is really not meant for that. So, I expect this Saturday to go get a new phone. Between now and then, I'll be cellphone-deprived (in case you wonder why I'm not answering to your messages or whatever). I would love to get a phone with a real keyboard (as I prefer not to look at the screen when writing messages, just to check if everything came out right and fix what's needed). I understand Android phones are more likely to keep me happy as a free software geek, and I'd be delighted to use Cyanogen if it is usable and stable But my phone is *not* my smart computer and it should not attempt to be, so it's not such a big deal. I will look for something with FM radio capability, and GPS. Of course, I want something cheap. It would be great to get it at no cost, but I don't expect I'll find such a bargain. Oh, and I want something I can find at the first Telcel office I come to, am I asking for too much? :) Anyway - I'll enjoy some days of being really disconnected from any wireless bugs (that I am aware off).

Changes:

dput-ng (1.4) unstable; urgency=low
   [ Arno T ll ]
   * Really fix #696659 by making sure the command line tool uses the most recent
     version of the library.
   * Mark several fields to be required in profiles (incoming, method)
   * Fix broken tests.
   * Do not run the check-debs hook in our mentors.d.n profile
   * Fix "[dcut] dm bombed out" by using the profile key only when defined
     (Closes: #698232)
   * Parse the gecos field to obtain the user name / email address from the local
     system when DEBFULLNAME and DEBEMAIL are not set.
   * Fix "dcut reschedule sends "None-day" to ftp-master if the delay is
     not specified" by forcing the corresponding parameter (Closes: #698719)
 .
   [ Luca Falavigna ]
   * Implement default_keyid option. This is particularly useful with multiple
     GPG keys, so dcut is aware of which one to use.
   * Make scp uploader aware of "port" configuration option.
 .
   [ Paul Tagliamonte ]
   * Hack around Launchpad's SFTP implementation. We musn't stat *anything*.
     "Be vewy vewy quiet, I'm hunting wabbits" (Closes: #696558).
   * Rewrote the test suite to actually test the majority of the codepaths we
     take during an upload. Back up to 60%.
   * Added a README for the twitter hook, Thanks to Sandro Tosi for the bug,
     and Gergely Nagy for poking me about it. (Closes: #697768).
   * Added a doc for helping folks install hooks into dput-ng (Closes: #697862).
   * Properly remove DEFAULT from loadable config blocks. (Closes: #698157).
   * Allow upload of more then one file. Thanks to Iain Lane for the
     suggestion. (Closes: #698855).
 .
   [ Bernhard R. Link ]
   * allow empty incoming dir to upload directly to the home directory
 .
   [ Sandro Tosi ]
   * Install example hooks (Closes: #697767).

Thanks to all the contributors! For anyone who doesn t know, you should check out the docs.

The War on Time Whoosh! I ve been incredibly quiet on my blog for the last 2-3 months. It s been a crazy time but I ll catch up and explain everything over the next few entries. Firstly, I d like to get out a few details about the last Ubuntu Developer Summit that took place in Copenhagen, Denmark in October. I m usually really good at getting my blog post out by the end of UDS or a day or two after, but this time it just flew by so incredibly fast for me that I couldn t keep up. It was a bit shorter than usual at 4 days, as apposed to the usual 5. The reason I heard for that was that people commented in previous post-UDS surveys that 5 days were too long, which is especially understandable for Canonical staff who are often in sprints (away from home) for the week before the UDS as well. I think the shorter period works well, it might need a bit more fine-tuning, I think the summary session at the end wasn t that useful because, like me, there wasn t enough time for people to process the vast amount of data generated during UDS and give nice summaries on it. Overall, it was a great get-together of people who care about Ubuntu and also many areas of interest outside of Ubuntu. Copenhagen, Denmark I didn t take many photos this UDS, my camera is broken and only takes blurry pics (not my fault I swear!). So I just ended up taking a few pictures with my phone. Go tag yourself on Google+ if you were there. One of the first interesting things I saw when arriving in Copenhagen was the hotel we stayed in. The origami-like design reminded me of the design of the Quantal Quetzel logo that is used for the current stable Ubuntu release. The Road ahead for Edubuntu to 14.04 and beyond St phane previously posted about the vision we share for Edubuntu 14.04 and beyond, this was what was mostly discussed during UDS and how we ll approach those goals for the 13.04 release. This release will mostly focus on the Edubuntu Server aspect. If everything works out, you will be able to use the standard Edubuntu DVD to also install an Edubuntu Server system that will act as a Linux container host as well as an Active Directory compatible directory server using Samba 4. The catch with Samba 4 is that it doesn t have many administration tools for Linux yet. St phane has started work on a web interface for Edubuntu server that looks quite nice already. I m supposed to do some CSS work on it, but I have to say it looks really nice already, it s based on the MAAS service theme and St phane did some colour changes and fixes on it already. From the Edubuntu installer, you ll be able to choose whether this machine should act as a domain server, or whether you would like to join an existing domain. Since Edubuntu Server is highly compatible with Microsoft Active Directory, the installer will connect to it regardless of whether it s a Windows Domain or Edubuntu Domain. This should make it really easy for administrators in schools with mixed environments and where complete infrastructure migrations are planned. You will be able to connect to the same domain whether you re using Edubuntu on thin clients, desktops or tablets and everything is controllable using the Epoptes administration tool. Many people are asking whether this is planned for Ubuntu / Ubuntu Server as well, since this could be incredibly useful in other organisations who have a domain infrastructure. It s currently meant to be easily rebrandable and the aim is to have it available as a general solution for Ubuntu once all the pieces work together. Empowering Ubuntu Flavours This cycle, Ubuntu is making some changes to the release schedule. One of the biggest changes made this cycle is that the alpha and beta releases are being dropped for the main Ubunut product. This session was about establishing how much divergence and changes the Ubuntu Flavours (Ubuntu Studio, Mythbuntu, Kubuntu, Lubuntu and Edubuntu) could have from the main release cycle. Edubuntu and Kubuntu decided to be a bit more conservative and maintain the snapshot releases. For Edubuntu it has certainly helped so far in identifying and finding some early bugs and I m already glad that we did that. Mythbuntu is also a notable exception since it will now only do LTS releases. We re tempted to change Edubuntu s official policy that the LTS releases are the main releases and treat the releases in between more like technology previews for the next LTS. It s already not such a far stretch from the truth, but we ll need to properly review and communicate that at some point. Valve at UDS and Steam for Linux One of the first plenaries was from Valve where Drew Bliss talked about Steam on Linux. Steam is one of the most popular publishing and distribution systems for games and up until recently it has only been available on Windows and Mac. Valve (the company behind Steam and many popular games such as Half Life and Portal) are actively working on porting games to run natively on Linux as well. Some people have asked me what I think about it, since the system is essentially using a free software platform to promote a lot of non-free software. My views on this is pretty simple, I think it s an overwhelmingly good thing for Linux desktop adoption and it s been proven to be a good thing for people who don t even play games. Since the announcement from Valve, Nvidia has already doubled perfomance in many cases for its Linux drivers. AMD, who have been slacking on Linux support the last few years have beefed up their support drastically with the announcement of new drivers that were released earlier this month. This new collection of AMD drivers also adds support for a range of cards where the drivers were completely discontinued, giving new life to many older laptops and machines which would be destined for the dumpster otherwise. This benefits not only gamers, but everyone from an average office worker who wants snappy office suite performance and fast web browsing to designers who work with graphics, videos and computer aided design. Also, it means that many home users who prefer Linux-based systems would no longer need to dual-boot to Windows or OS X for their games. While Steam will actively be promoting non-free software, it more than makes up for that by the enablement it does for the free software eco-system. I think anyone who disagrees with that is somewhat of a purist and should be more willing to make compromises in order to make progress. Ubuntu Release Changes Last week, there was a lot of media noise stating that Ubuntu will no longer do releases and will become a rolling release except for the LTS releases. This is certainly not the case, at least not any time soon. One meme that I ve noticed increasingly over the last UDSs was that there s an increasing desire to improve the LTS releases and using the usual Ubuntu releases more and more for experimentation purposes. I think there s more and more consensus that the current 6 month cycle isn t really optimal and that there must be a better way to get Ubuntu to the masses, it s just the details of what the better way is that leaves a lot to be figured out. There s a desire between developers to provide better support (better SRUs and backports) for the LTS releases to make it easier for people to stick with it and still have access to new features and hardware support. Having less versions between LTS releases will certainly make that easier. In my opinion it will probably take at least another 2 cycles worth of looking at all the factors from different angles and getting feedback from all the stakeholders before a good plan will have formed for the future of Ubuntu releases. I m glad to see that there is so much enthusiastic discussion around this and I m eager to see how Ubuntu s releases will continue to evolve. Lightning Talks Lightning talks are a lot like punk-rock songs. When it s good, it s really, really amazingly good and fun. When it s bad, at least it will be over soon :) Unfortunately, since it s been a few months since the UDS, I can t remember all the details of the lightning talks, but one thing that I find worth mentioning is that they re not just awesome for the topic they aim to produce (for example, the one lightning talks session I attended was on the topic of Tests in your software ), but since they are more demo-like than presentation-like, you get to learn a lot of neat tricks and cool things that you didn t know before. Every few minutes someone would do something and I d hear someone say something like Awesome! I didn t know you could do that with apt-daemon! . It s fun and educational and I hope lightning talks will continue to be a tradition at future UDSs. Social Stefano Rivera (fellow MOTU, Debianista, Capetonian, Clugger) wins the prize for person I ve seen in the most countries in one year. In 2012, I saw him in Cape Town for Scaleconf, Managua during Debconf, Oakland for a previous UDS and Copenhagen for this UDS. Sometimes when I look at silly little statistics like that I realise what a great adventure the year was! Between the meet n greet, an evening of lightning talks and the closing party (which was viking themed and pretty awesome) there was just one free evening left. I used it to gather with the Debian folk who were at UDS. It was great to see how many Debian people were attending, I think we had around a dozen or so people at the dinner and there were even more who couldn t make it since they work for Canonical or Linaro and had to attend team dinners the same evening. It was as usual, great to put some more faces to names and get to know some people better. It was also great to have a UDS with many strong technical community folk present who is willing to engage in discussion. There were still a few people who felt missing but it was less than at some previous UDSs. I also discovered my face on a few puzzles! They were a *great* idea, I saw a few people come and go to work on them during the week, they seem to have acted as good menial activities for people to fix their brains when they got fried during sessions :) Overall, this was a good and punchy UDS. I ll probably not make the next one in Oakland due to many changes in my life currently taking place (although I will remotely participate), but will probably make the one later this year, especially if it s in Europe. I ll also make a point of live-blogging a bit more, it s just so hard remembering all the details a few months after the fact. Thanks to everyone who contributed their piece in making it a great week!

Finally, phpMyAdmin packages for Debian and Ubuntu do not lag much behind upstream. Today, I've prepared packages for yesterday released bug fix release 3.5.6. For Debian users, the package should be soon available in experimental (sorry no uploads to unstable during freeze). Ubuntu users, can use my phpMyAdmin PPA. After dozens of comments and no help offered, I've still decided to be nice to Ubuntu users and adjusted the package so that it should work on Lucid as well. The downside is that, unlike in Debian, the package includes bundled copy of many PHP and javascript libraries. PS: As soon as Debian is not frozen or 4.0 is officially released, I will start uploading 4.0 to experimental. My current bet is that 4.0 release will come earlier.

Filed under: Debian English Phpmyadmin 0 comments Flattr this!

Indeed, two days was enough time to recover from an afternoon of social, despite having an hour and a half of meetings today. But today was a day of catching up, sorting issues in JIRA, doing lots of planning meetings, and doing some debugging, so I don't have anything new code-wise to give to the world. So have a photograph. The product backlog is now all sorted out for the Account Services project into the new phases that will define the work through pretty much the end of the calendar year, I'm guessing. And everything is now a story, not a bug or enhancement or something else without story points, and a few things are shuffled into a more reasonable order. I'm probably going to find a few hours this week to go play video games or work on personal projects, given that I've been working well over 40 hours a week since the start of the year.

Well, today was another day of sleeping, zoning, and having no brain for programming (and no willpower for that or exercise, which is even more frustrating), so you all get more musing about missions and work. Tomorrow, I will re-engage my normal schedule and turn the television off again, because seriously, two days of recovery should be enough even for a whole afternoon of meetings. Perhaps the best concept in the scrum methodology for agile project management is the concept of the product backlog and the product owner. For those who aren't familiar, a scrum project has, as one of its inputs, a prioritized list of stories that the team will implement. During each planning session (generally between once a week and once a month), the development team estimates and takes from the front of the product backlog the top-priority stories that can be completed during that period of time. The product owner is responsible for building the list of pending development stories based on business needs and keeping them sorted in priority order so that the work of the team follows the goals of the larger organization. But the product owner does not get to say how difficult the stories are; the development team estimates the stories, and the product owner may change order based on those estimates (to do simpler things earlier, for example). We've been doing scrum to some extent or another for quite some time now, and while there are a variety of things that I like about it, I think this is the best part. And I think it has very useful concepts for management in general. The product backlog is effectively a problem stream of the sort that I talked about in my last couple of posts. It's a set of problems that need to be solved, that matter to other people. And, if the team is doing scrum properly, the problems are presented in a form that describe a complete unit of desired functionality with its requirements and constraints, but without assuming any sort of implementation. The problem is the domain of the product owner; the solution is the domain of the development team. I am increasingly convinced that my quality of life at work, and that of many other people doing similar sorts of development and project work, would be drastically improved if managers considered this role the core of their jobs. (Ironically, at least in our initial experiences with scrum, it was quite rare for regular managers to be product owners; instead, individual line staff or project managers act as product owners. I think this speaks to how confused a lot of managers are about their roles in a technical development organization. This seems to be improving.) The core of the sort of work that I do is a combination of ongoing maintenance and solving new problems. Both of those can be viewed as a sequence of (possibly recurring) stories; I know this because that's largely how I do my own time management. Apart from HR and staffing, the core role of a manager is almost exactly what scrum calls "backlog grooming": communicating with everyone who has input into what their group is doing, finding out what their problems are, translating that into a list of things that the organization needs to have done, prioritizing them, breaking them into manageable chunks, ensuring they are fully specified and actionable (read: well-written stories), and communicating their completion back to other stakeholders (and then possibly iterating). This lept out at me when I started thinking about our larger strategic vision. That strategic vision is a sort of product backlog: a set of development stories (or, in this case, epics that would be broken down into a large number of smaller stories). But most strategic plans have glaring flaws if they're evaluated by the standards of scrum product backlogs. They're normally written as marketing statements and aimed at the clients or customers rather than at the staff. From the staff perspective, they're often hopelessly vague, not concrete, actionable epics. Often, they are almost entirely composed of grand, conceptual epics that any scrum team would immediately reject as too large and nebulous to even discuss. And often they're undefined but still scheduled: statements that the organization will definitely complete some underspecified thing within the next year or two. Scrum rightfully rejects any form of scheduling for unspecified stories and epics. Additional vagueness and sketchiness is tolerated the farther the story is from the current iteration, so they can be put into the future backlog, but scrum makes no guarantees about when they'll get done until they've been sized and you can apply a reasonable velocity estimate. If you want to put a date on something, you have to make it concrete. This is exactly the problem that I have with long-range strategic plans. We already know this about technology development: long-range strategic plans are feel-good guesses, half-truths, and lies that we tell other people because we're expected to do so, but which no one really believes. The chance that we can predict anything about the shape of projects and rate of delivery of those projects three years from now is laughable, and certainly none of the work that would be required to make real estimates has normally been done for those sorts of long-term strategic projects. There are a lot of scrum advocates who would go farther than I in asking for a complete revolution of how technical projects are planned. I'm not sure how realistic that is, or how well the whole scrum process would work when rolling up work across dozens, if not hundreds, of scrum-sized teams. But in the specific area of the role of managers in a development organization, I think we could learn a lot from this product backlog concept. Right now, there is a constant tension between managers who feel like they need to provide some sort of visionary leadership and guidance (the cult of Steve Jobs) and line staff who feel like managers don't understand enough about specific technology to lead or guide anything. But if you ask a technical developer whether, instead, it would be useful for managers to provide a well-structured, priority-ordered, and clean problem stream from the larger organization so that the developer can trust the problem stream (and not have to go talk to everyone themselves) but have control over the choice of implementation within the described constraints, I think you would find a great deal of enthusiasm. As a further bonus, scrum has a lot of advice about what that problem stream should look like. I think some of the story structuring is overblown (for example, I can't stand the story phrasing structures, the "AS A... I WANT TO... SO THAT..." shackles), but one thing it does correctly emphasize is the difference between problem and solution. The purpose of the story is to describe the outcome from the perspective of the person who will use the resulting technology. Describing the constraints on the solution, in terms of cost or integrations, is actively encouraged. Describing the implementation is verboten; it's up to the project team to choose the implementation that satisfies the constraints. If managers in general would pick up even that simple distinction, I think it would greatly improve the engagement, excitement, and joy that developers could take in their work. There's very little that's more frustrating than to be given an interesting problem in the guise of a half-designed solution and be told that one is not permitted to apply any of one's expertise in elegance, robustness, or durability to replacing the half-baked partial solution. Or even be allowed to know the actual motivating problem, instead of the half-digested form that the manager is willing to pass on.

I'll be at FOSDEM again this year, arriving in Brussels on Thursday 31st and leaving on Tuesday 5th. I'll be speaking on Sunday in the legal issues devroom at 10:00. If you will be there and want to meet up, let me know. I may be trying to watch the Super Bowl from there, a plan that didn't quite work out last year but seems more likely this year. State of the GNUnion FSF licensing policy challenges in 2013 This talk will cover the main challenges facing the Free Software Foundation's Licensing and Compliance lab in 2013, and will invite discussion of the FSF's work and policies in this area. We'll explore:

• Copyright assignment: Some high-profile GNU maintainers have recently criticized the FSF's copyright assignment policy and system. What are these criticisms, what does the FSF intend to do about them, and what's the point of its assignment process to begin with?
• GPL adoption: Last year here, in "Is copyleft being framed?", I put numbers supposedly showing declining GPL adoption in perspective, showing problems with the data, questioning the conclusions drawn from the data, and presenting different data leading to the opposite conclusions. We'll look at the questions that were raised since then about my data, and at some new data that's been made available, and draw new conclusions.
• App Stores: When Apple's App Store launched, the FSF concluded that its terms were incompatible with the GPL -- and with any kind of strong copyleft. Since then, we have several new App Stores; most notably from Google and Microsoft. Are the Apple terms still incompatible with the GPL? Are the other stores any better? Are these stores undermining GPL adoption, and should copyleft relax its standards in order to get free software to this audience, or should it stand its ground?

This morning I gave a status report on SE Linux. The talk initially didn t go too well, I wasn t in the right mental state for it and I moved through the material too fast. Fortunately Casey Schaufler asked some really good questions which helped me to get back on track. The end result seemed reasonably good. Here s a summary of the things I discussed: Transaction hooks for RPM to support SE Linux operations. This supports signing packages to indicate their security status and preventing packages from overwriting other packages or executing scripts in the wrong context. There is also work to incorporate some of the features of that into dpkg for Debian. Some changes to libraries to allow faster booting. Systems with sysvinit and a HDD won t be affected but with systemd and SSD it makes a real difference. Mostly Red Hat s work. Filename transition rules to allow the initial context to be assigned based on file name were created in 2011 but are not starting to get used. When systemd is used for starting/stopping daemons some hacks such as run_init can be avoided. Fedora is making the best progress in this regard due to only supporting systemd while the support for other init systems will limit what we can do for Debian. This improves security by stopping terminal buffer insertion attacks while also improving reliability by giving the daemon the same inherited settings each time it s executed. Labelled NFS has been accepted as part of the NFSv4.2 specification. This is a big deal as labelled NFS work has been going for many years without hitting such a milestone in the past. ZFS and BTRFS support but we still need to consider management issues for such snapshot based filesystems. Filesystem snapshots have the potential to interact badly with relabelling if we don t develop code and sysadmin practices to deal with it properly. The most significant upstream focus of SE Linux development over the last year is SE Android. I hope that will result in more work on the X Access Controls for use on the desktop. During question time I also gave a 3 minute lightning talk description of SE Linux.

The next point update for Debian 6.0 should include many fixes and improvements to the Linux kernel. Version 2.6.32-47, currently available in the 'squeeze-proposed-updates' suite, includes updates to the following SAS drivers, adding support for new hardware:

• hpsa: HP SmartArray (bug #690100)
• megaraid_sas: LSI Logic MegaRAID, also branded as Dell PERC (bug #666108)
• isci: Intel C600 series (bug #698094)

Note that the binary package names don't change; they are still linux-image-2.6.32-5-flavour. If you are using any of the above SAS controllers, whether or not it is already supported by Debian 6.0.6, please consider testing this kernel version. If it works, please follow up to the bug reports listed above. If you find any regressions, please report bugs against the new package version.

LCA 2013 [1] is starting so it seems like time to finish my write-up of LCA 2012. As usual it was a great conference, although I got sick immediately after getting there which reduced my ability to attend. Android A major unofficial theme of the conference was Android. Most delegates seemed to have Android phones, the Samsung Galaxy S and Galaxy Nexus seemed to be the most popular phones. Many delegates had two or more phones for development purposes. A large portion of the casual conversation at the conference concerned Android. There were a couple of really interesting talks about the Serval mesh networking project [2] which involves Android phones running in ad-hoc Wifi mode for long range communication without any official base station. Serval allows transferring messages, pictures, and voice calls. If you need to get longer range you can mount one phone in a convenient place and other phones will decide to use it as a relay there is no need to have a dedicated relay device (such as a mobile phone tower or Wifi access point). Serval is supposed to work with Wifi access points but due to Java not exposing some networking details to the higher levels of software the code that was available at the time of the conference didn t support networks other than a /24, which meant that the conference Wifi network didn t work with Serval. As an aside most people at the conference who installed Serval were using a development version that was newer than the version on the Android market. I can t remember what the extra features were though. Serval was designed for emergency situations, it can be installed on phones (and pushed to other phones via Wifi) in the field and allow communications when the infrastructure is broken. Also it s designed with some aim of circumventing censorship which among other things means that there are no facilities for tracking use. I think it would be really handy to be able to in some way track viewing of or interest in images that are transported via the mesh (maybe by something similar in concept to Google +1). Then in a crowd sourced environment people who take photos would be encouraged in their work by audience appreciation. One thing that interests me is the possibility of using Serval on a cruise ship. A cruise ship is an environment where mobile phone calls are unreasonably expensive, cabin phones aren t much use (who pays for a cruise and hangs out in their cabin?), and where there is usually a Wifi network installed. If a ship has a single bridged Wifi network that allows connecting to Wifi before authenticating for Internet access (which is probably the common case) then you could transport VOIP over that network without paying and without incurring any expense on the cruise company. One of the Serval developers assured me that this should be possible, of course a cruise ship with 3,000 passengers probably doesn t use a /24 for their Wifi so the current versions of Serval won t work At the geek my dinner event (a party where everyone brought $20 of food/drink which was cooked by volunteers) an artist showed me some art work that she created on her Samsung Galaxy Note (which was the biggest phone on sale at the time), it was very impressive. She recommended Picasso Mirror Draw, Sketch Free, Sketch Book Mobile Express, and Freenote as free graphical programs for Android. Samsung has a Noteworthy Project advertising campaign based on the artistic uses of the Galaxy Note which has some good videos of artists [3]. Chris Neugebauer and Paris Buttfield-Addison gave an informative and amusing talk about Android UI development (this link has the video) [4]. It s a pity that I missed seeing that one live but fortunately the video is of high quality. Accommodation As I ve become interested in Sociology I couldn t help but notice the pictures that accompanied the rules about cleaning the dormitory (which were displayed over the kitchen sink), it seems to imply that cleaning is only women s work. I wonder whether the people who created that poster deliberately chose pictures of women or whether they just chose the first available pictures from a collection of stock photos. Someone who was near my dorm room seemed to not realise how their alarm impacts other people. For the first two mornings I was woken repeatedly after 6AM by someone who was pressing the snooze button on their alarm. When sleeping in close proximity to other people the reasonable options involve some combination of having no loud alarm, immediately turning the alarm off and getting up (not pressing snooze to have it go off repeatedly), and setting the alarm for a time when almost everyone wants to get up (EG 1 hour before the first session). Networking Chris Neugebauer organised the Unprofessional Delegates Networking Session which was a great event. It was an event held at the same time as the Professional Delegates Networking Session with the difference being that you had to pay $5 for food and there was no free drink. A lot of great people attended the UPDNS so I m glad I don t pay for the PDNS. It seems that we won t have a UPDNS this year unfortunately. Conclusion LCA is always great fun and very educational. I recommend attending every year.

• [1] http://www.linux.conf.au/
• [2] http://developer.servalproject.org/site/
• [3] http://noteworthyproject.com/
• [4] http://chris.neugebauer.id.au/2012/01/25/lca2012-android-is-not-vi-user-experience-for-geeks/

We had our tech leads retreat on Friday, including the discussion about "why," and I thought it was excellent, although I'm still trying to understand some parts of it. Part of why I'm still working on that is, well, you know that introversion thing? I had five and a half hours of meetings on Friday, and the tail end was an intense four and a half hour retreat, during which I was a major participant the whole time. And tried to say some difficult and intense things, both about the "why" conversation that I posted about and about why the tail end of last year was particularly difficult for me. All that went very well. I was very happy with the overall meeting. Sometimes, I half-convince myself that I'm overstating the degree that introversion or that interpersonal interaction affects me, or that I'm using it as an excuse to duck out of things that aren't really what I want to be working on. Then I do something like this, and find myself completely dead on my feet at the end of the meeting. Not physically tired, but utterly and completely drained. I went home and did something completely right-brain and creative and needed that so desperately that I stayed with it until two in the morning, slept in today until past noon, and still have, despite a couple of minor attempts, absolutely no capability to write code or focus on accomplishing anything. I'm fairly sure I'll still be feeling the after-effects on Monday and Tuesday. I'm a little annoyed at having both missed a day of releasing something to the world (I'd managed to post something every day this year until yesterday) and blowing off exercising yesterday evening for only the second time this year. But I think part of coming to terms with what introversion means is being kind to myself about this. That's something Cain talked a lot about in her book, and it's the advice I'd give to any other introvert, but it's something I'm not applying as well as I could. Being an introvert doesn't mean to not do meetings. It means to pick the ones that are important, do them with all your heart, and then be kind to yourself afterwards just as if you'd put in extreme effort to do something quite energetic and difficult. Because you have. So, I'm trying not to be annoyed at myself. And I think we need to be better about teaching people about this, because apart from Susan Cain, I don't remember many people writing about this, or understanding it. Occasionally there are screeds against meetings, but not a lot of discussion about when it's worth expending the energy, and about how to be kind to oneself and recharge after putting one's energy into the social interactions that are important. I did want to mention that while retreats seem to have a bad reputation in tech circles (and I've heard of a lot of retreats that I would hate), the ones that we do in our group are great. We spent about an hour talking about the things I posted about on Thursday, then a couple of hours talking about the larger organizational roadmap and searching out things that aren't on our upcoming work roadmap that feel like large missing pieces, and then a couple of hours identifying and enumerating our technical debt and deciding what major pieces of technical debt we're going to tackle in the next six months. It's a lot of concrete, specific discussion of what we're going to spend time on and what we're going to defend resources for by pushing back against other priorities. If I had to sum it up in a sentence, I'd say that these retreats are all about listing vaguely equal things that exceed our total capacity and making the decisions about which of them we're going to do and which of them we're not going to do so that we don't have to keep wasting time on revisiting unresolved decisions. The actual mission discussion turned into a discussion by each of us of what we most value in work, stated in terms very similar to what I used in my post, and that was wonderful. I found out things about my co-workers that I didn't know, and which will help me a great deal in improving their enjoyment of their work. I think we came away with a better idea of how to support each other in creating that engagement and excitement, and I think that's the best possible outcome. Since, like I said, I think that's the only real mission statement a group should have. We did, still, talk about having a mission statement anyway. Apparently having a shared statement is really important to people who aren't me. I'm still struggling with this, since it doesn't make sense to me and doesn't feel useful to me. But maybe it's a sort of validation and confirmation that we're allowed to focus on what we care about? But the group mission statement we came away with isn't about any specific technology and focuses on exactly the things that I care about: elegance, robustness, durability, and solving whatever problems the university has. As part of that discussion, we reached a surprising and really satisfying unanimity of opinion about what values we care deeply about. It's a value statement as opposed to a distillation of job descriptions, so if a mission statement is valuable for other people, that's one I'm quite happy with. Even if I don't understand why people want to have one. I'm less satisfied with the roadmap and strategic plan for our larger organization, but I'm starting to realize that's because I'm utterly not the target audience and it's not going to engage or interest me. It's more of a marketing statement that's focused externally, and my proper role in that is to try to help ensure that it reflects the enthusiasms and engagement of the line staff, not to try to turn it into a document that will be inspiring for the line staff. Which ties into what I was writing about previously: in my ideal world, these sorts of focuses flow upwards from the people who will be doing the work, not downward from a high-level manager deciding on an inspirational direction.

I am building a realtime notification system as my final semester project. The end product is a simple API that can be used by third party to push real time notification to client applications running on multiple platforms that include browsers and devices. Ideally, if your device can connect to the internet, it should be able to receive notifications. The non functional system requirements are pretty ambitious with scalability, availability and latency guarantees being the most important. Maybe little too ambitious for someone like me but then I will learn a lot in this process and its more fun when its challenging. To address the scalability issues, we ll have to distribute the system. A single system will not scale beyond a point and we need to be able to distribute the load over nodes effectively to be able to serve clients while maintaining the quality of service guarantees. With distribution comes a whole lot of other problems like latency, coordination and handling failures. We have a theorem that officially states that you cannot have the pie and eat it too; its called the CAP theorem. It says that you cannot have have a system that is Consistent, Available and Partition tolerant at the same time (Partition tolerance here refers to the ability of a system to continue operating in the event of partial failures). You can have any two but not all three in varying degrees. In a way this is good because you do not go chasing the holy grail because you know it doesn t exist. CAP Twelve Years Later: How the Rules Have Changed is a good read for some sensible advice when applying CAP theorem to real world scenarios. Reading Notes on Distributed Systems for Young Bloods by Jeff Hodges is highly recommended for anyone starting to work on distributed systems. All the commons assumptions that you may have when you are building a distributed system is wrong, so wrong that it is sort of standardized now under the title Fallacies of Distributed Computing. A more detailed article is available here. I will be using concepts as given in the Amazon Dynamo paper to make the system highly available. This one is big and will a get a blog post of its own. The notification service that I am building at its core is a Publish/Subscribe system with soft real time guarantees. Right now I am clueless about how to deal with latency guarantees but I hope to figure it out soon. Websockets will be the application layer protocol that ll be used to push to browsers and for the trasport layer protocol I hear the MQTT protocol fits to a T. More on this later. I will updating my blog as the project progresses, documenting all the design decisions and the rationale behind them. The code will be available online. This is not something new and I am aware of tools that can do this. While I will try to improve on what already exists, the point is, as mentioned earlier, to learn something new and build what will be considered good software. I have never attended a distributed systems theory class, so I am reading up on that as well. Vector clocks are really cool, if you ask me. If you think this is too big, to be accomplished by a single student programmer in a semester s worth of effort, I wouldn t deny that but then when you pick the right tools for a job, things become a lot more easier and you can afford to be ambitious. Erlang for the win! Feel free to comment on what I am building as I may be potentially unaware of a lot of things I should be knowing. I could use all your suggestions/advice to make things better.

The Debian package of the Linux kernel is based on Linux 3.2, but has some additional features. This continues from parts 1, 2 and 3, and covers new and improved hardware support. DRM drivers from Linux 3.4 (proposed) Some recent Intel and AMD graphics chips were not supported well or at all by the DRM (Direct Rendering Manager) drivers in Linux 3.2. Although many bug fixes have been included in 3.2.y stable updates, we are considering updating them to the versions found in Linux 3.4. (We did something similar for Debian 6.0 'squeeze'.) Julien Cristau has been working on this and has prepared binary packages. To install, run as root:

gpg --no-default-keyring --keyring /usr/share/keyrings/debian-keyring.gpg --export 310180050905E40C   apt-key add -
echo deb http://people.debian.org/~jcristau/wheezy-drm34/ ./ > /etc/apt/sources.list.d/jcristau-wheezy-drm34.list
apt-get update
apt-get install linux-image-3.2.0-4.drm-amd64  # or -486, or -686-pae

Please test these and report your results to bug #687442. I would suggest testing suspend/resume (if applicable), use of internal and external displays on laptops, and 3D accelerated graphics (games and compositing window managers such as GNOME Shell). Remember that AMD/ATI graphics chips require the firmware from the firmware-linux-nonfree package for 3D acceleration and many other features. amilo-rfkill driver I wrote a standard rfkill driver for the Fujitsu-Siemens Amilo A1655 and M7440, based on the out-of-tree fsaa1655g and fsam7440 drivers. Unlike those, it should work with the rfkill command, Network Manager, etc. ALPS touchpads Newer touchpads made by ALPS use different protocols for reporting scroll and pinch gestures. Jonathan Nieder backported the changes to support these. Wacom tablets Jonathan Nieder updated the wacom driver to the version in Linux 3.5, adding support for the Intuos5, Bamboo Connect, Bamboo 16FG and various other models. Emulex OneConnect 'Skyhawk' Sarveshwar Bandi at Emulex contributed a backport of the be2net driver from Linux 3.5, adding support for their 'Skyhawk' chip. Marvell Kirkwood Marvell's Kirkwood SoCs have been supported upstream for some time and in Debian since release 6.0 'squeeze'. However new models and boards generally require specific support. Arnaud Patard backported support for the 6282 rev A1 chip found in QNAP TS-x19P II models, and for the Marvell Dreamplug and Iomega Iconnect. Miscellaneous

• ITE IT8728F hardware monitor
• Ralink RT5392, RT5390R and RF5372 wifi chips
• Synaptics USB touchpad devices
• JMicron JM362 SATA controller
• IguanaWorks USB IR transceiver

More to come Missing hardware support is an important bug that can be fixed by kernel updates during a freeze and throughout the lifetime of a stable release. If you know that new hardware isn't supported by the Debian kernel, please open a bug report. I can't promise that it will be fixed, but we need to know what's missing. Hardware vendors that maintain their own drivers upstream (not out-of-tree) are especially welcome to contribute tested backports that add support for the latest hardware. Send mail to debian-kernel@lists.debian.org if you have any questions about this.

After this announcement about the future of GNOME in Debian, I received several requests to look at SolusOS and their Consort fork of GNOME. On the paper, this doesn t look like a too bad idea, since they only forked gnome-panel and metacity, the two key components that are no longer maintained upstream. Still, starting a plain fork at the very moment when the former maintainer decided to give the key to anyone who wanted to, looked like a very destructive way of acting. Even if the fork gains momentum (which remains speculative), the amount of effort to rename packages makes you think twice before such a switch. I decided to go talk about it with them on IRC nevertheless. Mind you, they work on a GNOME fork and a Debian derivative, but deliberately use their own IRC server (you ll soon understand why). Just in case you would want to cooperate with SolusOS, you d have to use their infrastructure. It is an euphemism to say that the conversation didn t go well. This could have ended there, but thankfully for your already widening eyes, Ikey (the charming person I had the opportunity to discuss with) made the log public, in an attempt at public shaming that would soon gather his followers, chanting out loud how the revolutionary SolusOS would quickly replace every other Linux distribution. (As a side note, of the hundreds of such claims that were made public in the last 10 years, only one came true, and it was from a billionnaire who hired dozens of developers. Just saying.) Then you can understand why the specific IRC server: it is to K-line people who don t behave with enough deference to Ikey, because kickban is not enough. This log and the claims made in the comments look very ironical, if only for presenting a person working in a team of 10+ on the most popular desktop on one of the most popular desktop distributions, as a beggar. Even more ironical when it is the person who spent the most time on making GNOME Classic available and working nicely for wheezy. So when a dozen kids go chanting that Consort is so much better than GNOME and will replace it, while it just re-does the work I already did, let me at least smile. So what is SolusOS, after all? If you want to see where this story is going, there s a place that tells you: the consortium and consort-panel repositories. Here you will find interesting things:

• the work started from a tarball, instead of forking the git repositories (just in case you wanted to pull some of their patches);
• there is only one active contributor, who happens to be the founder of SolusOS, and the same person who K-lines people on his IRC channel;
• he seems to commits stuff randomly, without any cleanup, and with a lot of 4-letter words;
• most commits are here to rename everything to the Consort brand;
• even in debian/changelog, he retroactively renamed every mention of the former package names;
• some changes were made prior to creating the repository, so they are completely lost in history;
• among those changes I noted the reversion of a GNOME 3.0 change, which is to require Alt+right-click to edit a panel (and anyone who thinks this is a bug has never had to manage a help desk where disappearing notification areas were in the Top 5 incidents);
• nevermind the change was done in a hackish way and even mentions that in a comment, while the upstream commit is still available in git history;
• this also includes committing directly Debian-specific patches without a further look at why they hadn t been pushed upstream.
• For example, he pushed the menus renaming, which is a very Debuntu-specific stuff which will yield empty menus on other distributions.
• Or a bugfix that will only build with Debian s modified libwnck.

I ve stopped looking there. We re facing an ego issue, but not something that can improve the desktop for our users. In the end, I m sorry for those who genuinely thought we could do better by sharing some maintenance load with SolusOS, but it doesn t look like a big loss anyway.