A few months ago, my company made the decision to switch from our previous mail provider, to use Google Apps. It was a no-brainer really, most of the company were using it already, for the calendars, and Documents, and well we were a bit fed up with our previous mail provider. So we made the move. As an online retailer, we obviously get a lot of Financial related emails, and our Accountancy Department has an email address setup as a group, where the group sends to all the members in the Accounts Team. A pretty simple setup. However, for the last 2 months, I ve been fighting an uphill battle to get this to work properly. When an email gets delivered to a Google Group, it stops there, and then Google re-sends the email to the people in the group. Somehow, this triggers the receiving account to believe that it s originated from the Google Groups servers, rather than the actual originating server. Google Apps emails have a pretty nifty spam filtering service. For those well-known services on the web, it ll check whether it actually came from them (I think through some combination of DKIM and SPF), and bounce it if not. Can you see where I m going yet? Google Groups has a nifty feature to stop it attempting to send emails to addresses that don t exist. If an email address bounces at a certain rate, it ll flag it as undeliverable. If all the emails in a group are flagged as undeliverable, it ll bounce that email to the original party, so that they know no-one received it. The Accounts team are setup to receive information regarding Paypal Payments, Disputes, etc etc. A couple of days in, I had our Financial controller tell us that the Accounts team had stopped receiving emails. Well, being who I am I sent a test email out guess what I got back? Hello <redacted>, Your message could not be delivered because of previous failures during delivery attempts to this mailing list. Please update the list with valid addresses. This is an example of some of the bounces we received: ----------------------------------------------
Recipient: pa.......@mobilefun.co.uk 550 550 5.7.1 Unauthenticated email is not accepted from this domain. u22si29486554yba.55 (state 18).
Message-Id: <redacted> ----------------------------------------------
Recipient: ni........@mobilefun.co.uk 550 550 5.7.1 Unauthenticated email is not accepted from this domain. u22si29486554yba.55 (state 18).
Message-Id: <redacted> ----------------------------------------------
Recipient: lu..........@mobilefun.co.uk 550 550 5.7.1 Unauthenticated email is not accepted from this domain. u22si29486554yba.55 (state 18).
Message-Id: <redacted> *sigh* time to play with the group. I change the members of the group to use one of our alias domains instead of the @mobilefun.co.uk this gets emails working again. 2 days later . our Financial Controller comes to me and complains that clients are complaining that they re receiving bounce emails when sending to his team I check, and once again, get the same message. Time to open a support ticket. That was in October since then we ve tried everything Changing to user-defined groups and switching off the spam filters, whitelisting the groups, setting up DKIM on our domain, changing the spam levels, clearing bounce statuses all to no avail I ve been back and forthing with Google Enterprise Support for 2 months and have still not found a suitable solution *sigh*. I must say however, that the person dealing with the case at Google (Josephine H) has been professional and helpful all the way along, even with my rising frustration at the issue. Today I ended up calling their service unusable number, after having tried a couple of times to change the groups so that the Accounts Team could receive emails again and finding that none of my previous tricks worked. While on hold, I found that I could use a local part extension to an email address, and the group would recognise it as a new email address and therefore have no bounce status. I ve now made a script using the Provisioning APIs, and a bit of python-fu that will generate a local part extension based on the current date/time, and replace the users in the group for the Accounts team with those. Say for example, the primary email address for a user was test@example.com it d add a user of test+20101221235641@example.com. This is set to run each day so it s pretty much the same as resetting the bounce status (which fixes things for a short while) on a daily basis. According to my latest email from Google Enterprise support:-

At the moment there is an incentive going on to fix this outright as messages from Paypal etc have been causing bounce s for other domains. This fix is supposedly due very early in the New Year and will solve this problem indefinitely.

I wait with baited breath but for now, I m happy with my hack.

Debian unstable is not conceived as a product for end-users, and for very good reasons. There seems to be some misunderstanding and people trying to push end-users to use unstable. This blog post tries to address the claims raised and put them into proper light. 1. It contains mainly stable versions of the software The critical part here is the term mainly. Yes, developers are advised to only upload packages to unstable that they deem to be suitable for the next stable release. This is no hard requirement though and no one actually playing a gate-keeper enforcing this recommendation. Also, there is a fine amount of packages that follow either VCS snapshots or development branches, and only time can tell how stable those releases actually are. That's actually why there is a delay of several days before a package can transition over to testing. 2. It doesn't break badly every other day That's right but if you look at it, it also means that it does break badly eventually. And if you don't know how to move on from there, including knowing the location of maintainer scripts and how to edit them in those cases, or even resort to a rescue system, you are in troubles. 3. It's the basis of other distributions That's right in itself too, but it doesn't address the fact that those other distributions do put a lot of effort into quality assurance to work around the most nasty and annoying bugs that do affect unstable every other day. 4. It's not inherently less secure than Stable or Testing It's not inherently more secure than stable or testing either. And this is also ignoring the fact that some security bugs don't even get into stable or testing, they only affect unstable and have to get addressed just there. 5. I use it on my main computer This is the reason that you should ignore as most. People using unstable are often enough deeply involved into Debian maintenance, know how to write maintainer scripts, know where they are located in case of troubles, know how to use a rescue system in case of bootloader or kernel troubles. Just think about whether you would consider yourself to be at the same knowledge level than the person who wrote the blog article you read that used this as a convincing argument. Conclusion: Stories always have several truths attached to them. If you feel adventurous, like to understand what's going on and have a tendency for digging into things that go wrong, you most probably are using unstable already anyway and are reporting bugs that you find along your way. If you on the other hand rather just want to use your computer and don't want to work around (smaller) bugs every other day, you rather should stick with the release that is actually provided as purpose to suit the needs of end-users: stable.

/debian permanent link Comments: 19 Flattr this

Debian Unstable (also known as sid) is one of the 3 distributions that Debian provides (along with Stable and Testing). It s not conceived as a product for end-users, instead it s the place where contributors are uploading newer packages. Daily. Yes that means that Unstable is a quickly moving target and it s not for everybody. But you can use it and your computer won t explode. 1. It contains mainly stable versions of the software Yes, you read it right. Unstable is not full of development versions of the various software. It happens on some software but then it s usually a conscious decision of the maintainer who believes that this specific version is already better than the previous one. The packages in sid are supposed to migrate to testing, the place where the next Debian stable release is prepared. So maintainers are advised to only upload stuff that is of release quality, the rest should be uploaded to experimental instead. 2. It doesn t break badly every other day Breakages happen but they are not a big deal usually. It has been long time since I could not reboot my computer after an upgrade or since the graphical interface was no longer working. The kind of breakages that you have is that one software stops working, or triggers an annoying bug, or that a few packages are uninstallable. In most cases, you can save yourself by downgrading to the version available in Testing. Or by finding a work-around in the bug tracking system. Or by not upgrading because you have apt-listbugs installed and you have been warned about the problem. 3. It s the basis of other distributions If Debian Unstable was really so bad, it would not be a good basis to build a derivative distribution, isn t it? But Ubuntu and SiduxAptosid (to name only two) are based on Debian Sid. 4. It s not inherently less secure than Stable or Testing High impact security vulnerabilities will usually be quickly fixed in Stable and Unstable. The stable upload is done by the security team while the unstable one is made by the maintainer. Testing will usually get the fix through the package uploaded to Unstable, so testing users get security updates with a delay. For less serious vulnerabilities, it s entirely possible that stable does not get any update at all. In that case, unstable/testing users are better served since they will get the fix with the next upstream version anyway. Of course, it happens that maintainers are busy or that something falls through the cracks, but there are other people watching RC bugs who will fix this if the maintainer doesn t react at all. 5. I use it on my main computer And many other people do the same. And you can do the same if you meet the criteria below:

• you can work on the command-line (enough to downgrade a problematic package, to edit configuration files, etc.);
• you know how to work with APT and multiple distributions in /etc/apt/sources.list;
• you are able to read/write English so that you can read/file bug reports when needed;
• you have another computer connected to the Internet that you can use to lookup documentation (or the bug tracking system, or the support mailing lists) when your usual computer is off-line for a reason that you don t understand.

If you feel you are not ready for the jump, click here to subscribe to this blog (or here via the RSS feed), I ll surely teach some of the required skills in future articles. PS: All that said, if you have a working sid installation, do not upgrade it just before an important presentation, or before a trip. It will always break at the most annoying time. Unless you like to live dangerously, of course.

41 comments Liked this article? Click here. My blog is Flattr-enabled.

We had a small celebration with friends at our local Italian restaurant, Mario's Italiano today, to mark Sarah's 30th birthday. The reason this milestone is a big deal is because, to put it simply, two of her direct ancestors didn't make it to it. The undetermined genetic connective tissue disorder that Sarah has, claimed the lives of her father and paternal aunt when they were both in their twenties, suddenly and two weeks apart from each other. The case made the journal of the American College of Chest Physicians at the time. We're just very fortunate that with advances in medical diagnostic and surgical technology, Sarah was able to be diagnosed and have her aortic aneurysm repaired before it dissected. So there's the heart thing. That's a big deal. Then there's all the other crap that she's had to go through in only thirty years of life. She's had to endure more tragedies and medical challenges than some people have to go through in a lifetime. The fact that she's still such a happy, positive person is a testament to just what an amazing person she is, and it is an honour to be able to call her my wife. So happy birthday Sarah, here's to the next thirty years being a whole lot less eventful than the first thirty, and the last three in particular.

In GNU/Linux I usually use an AltGr-based keyboard layout for entering Lithuanian characters. That is, you can enter Lithuanian diacritics by holding down AltGr (right Alt)). I had lived with separate modes for entering English and Lithuanian until I found out that having a single layout was a huge relief.

Windows does ship with a Lithuanian layout where AltGr switches to English characters, but I use the English layout much more, so I need it the other way around. So, I grabbed Microsoft's Keyboard Layout Creator and made the layout I needed. The result is available on a GitHub repository. You may also download the installer directly.

In Linux, keymaps, being plain text files, are more transparent, but to be fair, in this case the Windows way wins. Not only do you get a GUI tool to create the layout, but also an installer can be generated automatically for deployment of the layout.

The Google books ngram datasets are a nice resource for playing with text generation. I've created a program that will, given a starting phrase, blather on by connecting phrases from the datasets. The results are much more varied, and seem to more often make sense than most Markov text generators I've seen before, such as dadadodo. It had better be -- it's working with data derived from 4% of all books ever written! Some examples. Do they tell a story? (The starting phrase I entered is in italics.)

• He is assailed they merge
• To be or not to be so deceived ! She denies abdominal scanning
• Beauty with enormous strides . The hospital was under the control of the communal organization of solitary mammals . In other words the mere failure of an individual .
• Time passes . It is the child's first day in town and at his own level ?
• Time above his left shirt pocket for safe keeping . At first I accepted the old man's work . He settled in that part of Paris .
• Until then no ill Accident would generate hydrogen sulfide
• Random nonsense put together huge boxes with ammunition against it itself stops . Working things out afterwards
• A change in voice and piano . He sat down at last at No . 5 in connection with another or with others ; and that by some strange disaster .
• O God ! What is the condition of morality is that they all hold hands in a chain of propositions . These propositions are the primary bearers of meaning making and the rearing of children

I have cheated slightly by picking where to stop quoting the blather each time. It tends to go on for quite a while otherwise. Although it often does manage some kind of sense at length too:

• Worst Case Scenarios in Search of Reality : The ERA and emendation . I can never recover . She became thoughtful . He looked at me gloomily above the tree tops . The women were asked how they used to behave more competitively . Forecasts should be made

---

This was accomplished using only 99 lines of code. I imported the datasets into Xapian databases using this program. And this program is the text generator. I threw out the metadata, to keep the Xapian databases of reasonable size. By which I mean, only a dozen gigabytes. It might be interesting to include the publication year data in Xapian, so that it could prefer ngrams that were published around the same time as the input text. I'm still importing data. I may put up a web interface later if there's interest. PS:

• What about Spam ? Who caused it ? You have said it so loud that the little lady in a black storm .
• What about Spam ? I think it cannot be any . The Church had the right men . They are our friends and relations who could be depended upon with regard to the UN . Some people paint the portraits of the writers .

PPS:

• Joey Hess was not the real author of these lines .

It seems the current trend is to use short-form dh. Some people have even thought that dh has superseded CDBS. Since I prefer CDBS for my own packaging, I will say that no, CDBS is not being deprecated and in fact has been active. Packaging with CDBS is very simple. I'll try to explain how it works, and how to package with it. This may turn into a blog post series, but I won't promise anything.

CDBS is a set of makefiles that do several tasks that are common while packaging software, so that you don't have to repeat them over and over on each package. The makefiles are classified into 2 groups: classes and rules. Classes implement the rules required for building and installing software. These are classes because they can and do inherit from others. For example, there is the makefile class, and the autotools, qmake and cmake classes inherit from it, and the gnome and kde (which is for kde 3) classes inherits from the autotools class. Rules implement several other general purpose rules that don't depend on the toolchain used to build the package. For example, there is the debhelper rules file, which takes care of creating the debian package using the usual dh_* commands. There are several rules files which do all sorts of useful stuff, from running license checks to downloading new upstream releases. If this post transforms into a series, we may look into some of them. For now, I'll demonstrate how to use CDBS to package some simple software (only the debian/rules file, of course).

The first task while running creating debian/rules is determining which build system the software uses. I will use qutecsound package as a guide. So, first things first, we start using CDBS!

#!/usr/bin/make -f



# Uncomment this to turn on verbose mode.

#export DH_VERBOSE=1



include /usr/share/cdbs/1/rules/debhelper.mk

include /usr/share/cdbs/1/class/qmake.mk

Now this would be all that is required if qutecsound where a standard qmake package where nothing needs to be customized. Alas, this is not the case. First, there are some variables that need to be passed to qmake to appropriately configure the build. Also, the projects .pro file is not standard-named. How do we fix this? The CDBS way of overriding this kind of behavior is through the use of variables. Since we are using the qmake class, we will be overriding variables named DEB_QMAKE_*:

DEB_QMAKE_CONFIG_VAL += build64

DEB_QMAKE_ARGS = qcs.pro

We add a configuration value (because CDBS is setting other config values), and we set the extra arguments to qmake.

Now, there is a problem: the resulting binary is not created with the name we want. How do we change that? We could patch the source to avoid that, but it is far easier to add a rule to the makefile to do that. We will modify the build to rename the file at the end (and then clean it up on clean because the upstream makefile will not spot it):

build/qutecsound::

	[ -f bin/qutecsound ]   mv bin/qutecsound-d bin/qutecsound



clean::

	rm -f bin/qutecsound

So what did we just do? We extended the build and clean rules with extra stuff to do. In CDBS each package listed in debian/control gets a build/package rule (and several others at different stages of the build) so that one can add stuff specific to each package (this will be much more useful when dpkg finally learns about build-arch and build-indep).

For the final touch, we want to do a few more things. First, we want to ensure that we are using QT4's qmake, because the build will fail if QT3's is used. Second, since I created a manpage for the command, I want to install it. Finally, we want to use parallel building when the user has specified it, to build the project faster. How to do that? Again, via variables:

QMAKE = qmake-qt4

DEB_BUILD_PARALLEL = 1

DEB_INSTALL_MANPAGES_qutecsound = debian/qutecsound.1

CDBS creates several variables for each of the packages listed in debian/control, so we can customize each package build. In our case, this variable is passed to dh_installman from the debhelper rules file (CDBS will invoke the debhelper tools once for each binary package).

After all this we now have the complete debian/rules of a relatively simple package. How does it look like? Like this:

#!/usr/bin/make -f



# Uncomment this to turn on verbose mode.

#export DH_VERBOSE=1



include /usr/share/cdbs/1/rules/debhelper.mk

include /usr/share/cdbs/1/class/qmake.mk



QMAKE = qmake-qt4

DEB_QMAKE_CONFIG_VAL += build64

DEB_QMAKE_ARGS = qcs.pro

DEB_BUILD_PARALLEL = 1



DEB_INSTALL_MANPAGES_qutecsound = debian/qutecsound.1



build/qutecsound::

        [ -f bin/qutecsound ]   mv bin/qutecsound-d bin/qutecsound



clean::

        rm -f bin/qutecsound

Tags: cdbs, packaging

<iframe frameborder="0" marginheight="0" marginwidth="0" scrolling="no" src="http://rcm.amazon.com/e/cm?lt1=_blank&amp;bc1=000000&amp;IS2=1&amp;bg1=FFFFFF&amp;fc1=000000&amp;lc1=0000FF&amp;t=dedasys-20&amp;o=1&amp;p=8&amp;l=as1&amp;m=amazon&amp;f=ifr&amp;md=10FE9736YVPPT7A0FBG2&amp;asins=0470929839" style="width: 120px; height: 240px;"></iframe> I got this to read on my trip to Oregon. Initially I was a bit dubious, after reading the chapter headings on Amazon. Indeed, if you read a lot of articles about tech startups, a lot of the material is not particularly novel, and are things that you've already read elsewhere. If, on the other hand, you're looking for a quick introduction to some of the common ideas behind web/tech startups, it's a pretty good list, written by a variety of people, with a lot of practical advice. The list of concepts includes "execution, not ideas", "fail fast", get feedback early", "get a co-founder", and a lot of similar advice that permeates discussions of early stage internet companies. Other people have different tastes, but for me, this passage was a strikingly effective advertisement for living in Boulder, Colorado:

With this as the backdrop, it was natural to make cycling a central part of my work with Everlater's founders, Nate Abbott and Natty Zola. Nate, Natty, and I scheduled regular weekly rides (sometimes several times a week) and while we didn't keep a formal agenda, each ride had a number of topics planned out for us to cover. In this way, we worked through many of the early challenges the business faced as it moved from idea to reality in the course of the summer of 2009. Our typical rides lasted from 60 to 90 minutes during which time we were able to focus on the business without interruption. And while getting some fresh air helps bring perspective, it's hard to avoid thinking about problems in a new way when you're out of the office, surrounded by the beauty of the Boulder foothills while your mind is completely clear. Startups, cycling, mountains... sign me up! Another tidbit I found interesting:

We often talk about the Number One startup killer at TechStars making a product for which there is no interesting market. TechStars accepts just 10 of more than 600 startups that apply and presumably they are among the best. But still, we find that at least one-third of those startups are attempting to build a product that they want, or that no one wants, instead of what the market wants But wait, I thought "execution is everything", and "ideas are worthless". It seems like avoiding bad ideas is potentially worth a whole lot of money in wasted time and energy though, no? The bit about ideas being worthless is a load of crap. The reason there is no market for ideas is that they are not "excludable". There was never a market for lighthouses either, the classic example in economics of a public good. So the government built them, and they were valuable to people operating ships. Ideas for startups may not have a lot of value compared to people executing them well, but to say they are worthless is hyperbole. In any case, though, if you're interested in startups, or especially if you happen to be interested in the TechStars program, it's book full of good advice, even if none of it will come as a revalation to anyone who follows that sort of thing regularly.

This has been bugging me ever since we got our Sony Internet TV, and I think I've finally figured out what's going on. Pandora hasn't been working on the Google TV ever since we got it, it just spits out Unexpected HTTP error occurred code: 417 while StartupAsyncTask and that's the end of that. I have vague recollections seeing this on my phone in the past also, but this doesn't appear to be the case any longer. My phone has version 1.5.2 on it, and the TV has 1.0, so I think I see the problem here. I took a full packet capture while starting up Pandora, and it was a lot more revealing. I could see that it was my Squid proxy (I transparently run all HTTP traffic through Squid) was barfing. That'd explain the errors I was seeing in Squid's cache.log

parseHttpRequest: Unsupported method ''
clientTryParseRequest: FD 153 (172.16.1.5:55001) Invalid Request

Looking at the headers of the HTTP request that the Pandora client is sending out, I see Expect: 100-Continue, and the text that accompanies a HTTP 417 error is "Expectation Failed", so I think this is the culprit. Searching for Expect: 100-Continue squid turns up information for an ignore_expect_100 directive, but this appears to only be available if Squid is compiled with --enable-http-violations, which the Debian Squid package doesn't appear to be compiled with. I may have to adjust my transparent proxying rules to not transparently proxy the TV, but this makes me a bit sad. Update I tried adding ignore_expect_100 on anyway, and Squid took it, and now Pandora isn't barfing. Yay!

So, I've been working on multitouch on and off the past few months (which have included a solid ten weeks of holiday), but have finally posted the third patch series, which I think should be pretty close to final, to the list today.

Touch events look kind of similar to standard Xi 2.1 events (DeviceMotionNotify and friends), but the implementation of grabbing and event delivery is quite different from the traditional X model that we all know and love.

Firstly, touch grabs are not strictly grabs as with other input events. When a synchronous pointer or keyboard grab activates, it 'freezes' the device, stopping delivery of all further events until the grabbing client has decided what to do. The client can either pass on the event to the next client (replay), opt to continue receiving all events with the device freezing again after each event (synchronous mode), or continue receiving all events with the device unfrozen (asynchronous). While this is happening, the grabbing client is the only one receiving events, hence the name.

Touch grabs break both of these assumptions. Firstly, the device never freezes: delivery of all events (pointer, keyboard, touch, whatever) continues at full tilt, and a touch grab never affects pointer or keyboard events, with the possible future exception of pointer events that are emulated from touch events. Secondly, all clients with grabs, as well as the client with the first selection we find<super>0</super>, receive the events. An 'owner' flag is set (or unset) to indicate to the client that it does, or does not, own the touch stream in question.

The owner principle is important: the theory is that all touch streams must be considered as a whole (i.e. from init, through all the motion events, to the touch stream ending), and cannot be separated into component parts. So a client which does not own a touch stream can do gesture recognition or whathaveyou in the background, but must not act upon it until it becomes the owner. We added an extra request, XIAllowTouchEvents, to control ownership: clients can either reject ownership, passing it on to the next in line, or assert their ownership and permanently remove all other clients from the delivery list.

So it's somewhat different, but (hopefully) not without reason. The motivation behind this was to keep latency as low as humanly possible, since people seem to demand lower latencies from direct touch-based devices. Sticking to the traditional grab model would have meant a flurry of round trips between all the clients in the stack and the server, as well as maintaining huge buffers in the server for every touch event, which each client would have to transfer in turn.

This model is a compromise between my original design and Ubuntu's suggested design; both of us had far more simple designs, but oriented towards different usecases. The original Ubuntu usecase was for a global gesture recognition system, which apps would only opt out of in rare cases. My primary usecase was exactly the reverse: apps doing their own gesture recognition, with a very small, if any, global system. Hopefully it's actually the best of both worlds, rather than the worst possible intersection: I guess we'll find out. :)

At the moment, it's all largely theoretical, but some of the Nokia Qt guys are working on a port and have provided really helpful feedback, and hopefully the GTK+ multitouch branch, which was written against a much earlier and, er, speculative, version of the spec, gets updated too.

Finally, some thanks: to both Nokia and my excellent employer Collabora for sponsoring my part of this work; to Chase Douglas from Canonical for his contributions and for not just giving us the finger and vanishing when we rejected the X Gesture Extension, but hanging around and working through it; and to Peter Hutterer, for spec work and review. There were also numerous others involved earlier (including, but not limited to, Henrik Rydberg working on mtdev and others) who deserve credit, so sorry if I forgot anyone.

If you want to get it and play around with it, the patches/repositories are all laid out in my initial mail linked to earlier, as well as the follow-up mail noting the evdev repository location. The spec has more details, and will hopefully be cleaned up to be a lot less clunkily-written.

[0]: The delivery order is unchanged. For grabs, we start at the root window and proceed down the window stack to the innermost child window, activating grabs in that order. For events selected for with XISelectEvents, we start at the innermost child window, work our way up towards the root window, and stop as soon as we find one.

One of the bad aspects of the the C Programing language is that it silently allows many bad C programs. Together with the widespread use of an architecture that is very bad at catching errors (i386), this sometimes leads to common ideoms that are only working accidentially. This is bad as they often break on other architectures and can break with every new optimisations or new feature the compiler adds. Take for example a look at the code there (I tried to leave a comment there but did not succeed): If you see something like this:

     char buffer[1000];
     struct thingy *header;
     header = (struct thingy *)buffer;

then it is time to run. I hope you do not depend on this software, because it is a pure accident if this is doing anything at all. While you can cast a char * to a struct, that is only allowed if that memory actually was this struct (or one compatible, like a struct with the same initial part and you are only accessing that part). In this case it is obviously not (it's just an array of char), so you might see bus errors or random values if the compiler does no optimisations and you are on a architecture where alignment matters. Or the compiler might optimize it to whatever it wants to, because the C compiler is allowed to do everything with code like that. The next problem is the one that post was about: You are not allowed to access an array after its end. Something like

 struct thingy  
     int magic;
     char data[4];
  

means you may only access the first 4 bytes of data. If you access more than that it may work now on your machine, but it can stop tomorrow with the next revision of the compiler or on another machine. If you have a struct with a variable length data block, then you can use the new C99 feature of char data[] or the old gcc extension of char data[0]. Or you can use unions. (Or in some case use the behaviour of structs with the same initial parts). If you use C code with undefined semantics then every new compiler might break it with some optimisation. There is often the tempting option of just using a slightly different code that currently works. But in the not too distant future the compiler (or even some processor) might again get some new optimisations and the code will break again. Fixing it properly might be harder but it's less likely it will fail to compile again and it also reduces the chances that it will not fail to compile but simply do something you did not expect.

Recently gcc (4.5) improved its ability to see the size of various structures. As a result, the FORTIFY protections have suddenly gotten a bit stricter. In the past, you used to be able to do things like this:

struct thingy  
    int magic;
    char data[4];
 
void work(char *input)  
    char buffer[1000];
    int length;
    struct thingy *header;
    header = (struct thingy *)buffer;
    length = strlen(input);
    if (length > sizeof(buffer) - sizeof(*header) - 1) abort();
    strcpy(header->data, input);
    header->magic = 42;
    do_something_fun(header);
 

The problem here is that gcc thinks that header->data is only 4 bytes long. But gcc doesn t know we intentionally overruled this (and even did length checking), so due to -D_FORTIFY_SOURCE=2, the strcpy() checks kick in when input is more than 4 bytes. The fix, in this case, is to use memcpy() instead, since we actually know how long our destination is, we can replace the strcpy(...) line with:

    memcpy(header->data, input, length + 1); /* take 0-term too */

This kind of header and then data stuff is common for protocol handlers. So far, things like Wine, TFTP, and others have been experiencing problems with the change. Please keep an eye out for it when doing testing.

I've recently spent five days debugging a problem in obnam, my backup program. It turned out to be a bug in my B-tree implementation, which I wrote for obnam. The process was pretty long, and it might be instructive to de-brief myself. The bug manifested itself when I ran a backup test that was larger than what I had ran before: one terabyte of live data. The symptom was a key lookup problem in a B-tree. Good: the problem was entirely deterministic, and easy to reproduce. Bad: it took hours to reproduce it. My first step was to fiddle with the obnam parameters to see if I could make it fail faster, but in the same way. I got the time down to half an hour. After this, I started fiddling with the code to see if I can figure out what happens, and perhaps find a quick fix to the bug. After some hours, I realized I was being stupid. I was making essentially random changes to the code, instead of hunting down the actual cause. I had no idea what was going wrong, so trying to fix it was an act of utter stupidty. So I went back to where I had started from. The next step was to run "obnam fsck" on the failing backup store. This did not find any problems. The fsck I've written is a bit simplistic, and merely checks that for every client in the backup store, every generation is found, and within every generation each file's metadata, and data chunks are found. I will improve fsck in the future, but right now it was not helpful. I did not have a tool to verify that the actual B-trees were OK. So I wrote one. It, too, is quite simplistic, but at least it proved that the key was still present in the on-disk version of the B-tree. At this point, I did a code review of the B-tree code. It had been a some months since I'd touched the code, and hence the review brought up a bunch of minor issues: outdated comments, badly formatted code, stupid variable names, etc. One or two buglets, too, but nothing relevant to the case of the missing key. At this point I made a mistake: I concluded that the error was not in the B-tree code, but the obnam code using the B-trees. I spent the next couple of days running the test case, adding more logging, and analyzing the results. Without my mistake, I would not have ignored indicators that would have led me faster into the depths of the B-tree code. I might have saved a day or two of the process. Eventually I narrowed down the problem to a single B-tree insertion call in the obnam code. The call inserted a new value for a given key into one tree, and the value in another tree changed as well. Quite mysterious. My B-trees are an implementation of the Rodeh B-trees. All updates are done using copy-on-write, and this allows efficient cloning of trees. However, as an optimization, a node that is used by only one tree at a time can be modified in-place. This speeds things up a lot when the same tree is updated many times. In obnam, each B-tree represents a backup generation. The specific bug I was hunting was that the key I use for storing the unique identifier for a generation changed in two trees at the same time. Now that I have actually found and fixed the problem, it is clear that I should have realized at once what was happening, but I didn't, so I spent another couple of days adding logging statements into the B-tree insertion code, and tracing where things went wrong. Since B-trees are updated quite heavily by obnam, the log files grew quite big, and the test runs grew quite long. The last run I did resulted in a 94 gigabyte log file, and lasted over 12 hours. less had trouble browsing the log file, probaby due to the rather long lines generated by dumps of the B-trees. I ended up splitting the large file into quite a number of small ones, using csplit. After some more analysis, I figured out the problem: the code that shadowed a B-tree node, i.e., implemented the copy-on-write modification, failed to update the reference counts of the children of a shadowed index node. The Rodeh B-trees use lazy reference counts: when an index node is shadowed, only the immediate children get reference count updates. This turns out to work quite well, and saves quite massive amounts of updates. Unfortunately, my code was too lazy, and did not update any reference counts. After I had come this far, it only took a couple of minutes to write a unit test to catch the problem, fix the code, run the unit tests successfully, and commit. Then I ran the big test case again, without the extra logging I had added, and the code worked fine. At least it worked fine for about 24 hours until it ate up all memory in the VM I was running it on, but that's a separate problem. What can I do to prevent this kind of bug happening in the future? Unit testing didn't catch it, and I have 100% statement coverage with unit tests (not counting 19 excluded statements). Code review by other programmers would surely help. I should perhaps attract others to the project. Please excuse the length of this blog post. It is a distillation of the about ten A4-pages' worth of notes I made during this debugging process. (Always make notes when debugging. Your brain can't remember all details.)

The H Open Source have announced that Oracle have released it s Oracle Open Office 3.3. I was very happy to see that title on my RSS feeder, and checked download.openoffice.org to see that version 3.2.1 is still the main version, and that RC8 is on its way. Then I read the whole item on the H, to learn it not only me (hence I could stop trying to clear my browser cache). I do think having Oracle s support for OpenOffice.org is a good thing, as a lot of organizations are looking to buy OO.org support from a big company. But this doesn t justify what is going here with this press release by Oracle which make me, as an OpenOffice.org member, feel like a there is some kind of a ripoff going on. How can they release an enterprise product before the community version is final? Is there anything Oracle or its developers aren t telling us? I can t prove anything, but this really looks ugly. Many people, including myself, have invested a lot of time and effort in the coming 3.3 release. I believe that Oracle should treat the community better and should think how to motivate the individuals in it instead of demotivating them which things like this. In cases like this, I m happy the project was fork into LibreOffice by the The Document Foundation.
Filed under: Openoffice.org

As I'm not currently working on any suitable paper, I'll just post this to my blog so it does not completely slip off my radar ;-) Also, it might be interesting to my reader. Readers? Oh, there are two of you now? Good! Yesterday, I learnt thanks to Beatriz Busaniche that a group of South American Free Culture activists launched number zero of a magazine that promises to be very interesting: Cultura RWX, cultura en modo lectura, escritura y acci n (culture in reading, writing and action mode). Guys, best luck with this new project! Anyway, reading it, I found this asseveration I want to keep at hand:

( )cuando surge la industria musical aparecen los derechos de autor como forma de defensa de los productores musicales, espec ficamente los m sicos. No tanto frente a los usuarios, porque hasta el cassette no existi posibilidad de copiar una obra musical. Era una defensa frente a las discogr ficas, que buscaban cerrar contratos muchas veces abusivos con los artistas. M sica en Libertad: La industria musical frente al cambio de paradigma; Mat as Lennie, adaptaci n: Sebasti n Vazquez

Yes, yes, translating to English:

( )when the musical industry was born, copyright appeared as a means of defense of the musical producers, specifically of the musicians. Not so much against the users, because up until the invention of the cassette there was no possibility to copy a musical creation. It was a defense against the discographic companies, which tried to close often abusive contracts with the artists. Music in Freedom: Musical industry and the paradigm shift; Mat as Lennie, adaptation: Sebasti n V zquez

I have argued (i.e. in here) in this same line regarding the birth of copyright itself It was an arrangement that had to be made between writers and printers, back in the XVI/XVII centuries. Simple individuals were just unable to get anything of value out of the copying technology they had at hand. Copyright was born in a time where reproduction required specialized equipment. Today, massive reproduction technology is a given for a good portion of the planet's population. Copyright now only defends big corporations And will inevitably fade away as anachronic. Of course, it refuses to go without a fight... But it cannot win long-term. We cannot afford to allow it!

It's been an intense couple of days out at the Hollow. Did it only start to snow yesterday? It feels like a long time in winter. Yesterday was the first day that the solar panels produced no power, it was too cloudy and snowing most of the time. I cleaned an inch of slushy snow off the panels to no avail, and wondered if the power would last. But the wood stove has kept it nice and warm, so yesterday was spent cosily in the living room enjoying the snowfall, nethack, and podcasts. Also, I'm very much enjoying cooking on the wood stove. I've never been in a position to cook on one before; when I lived at Wortroot the stove was in the cabin, and the cold kitchen in a different building. So cooking on the stove, or even keeping a kettle on, was mostly only done in emergencies. Here, I am constantly using it, and the propane stove is still on its first 20 pound tank. Long slow cooked beany or roastish stuff does especially well. Yesterday I did a great pork loin roast with homemade quick sauerkraut. Today, after cleaning another 3 inches of snow off the solar panels, and scraping off a layer of ice, I am getting a decent amount of power for a cloudy day with some snow still falling. Turns out it was good I cleaned them yesterday after all, since a part I missed has a inch+ layer of hard ice, which I don't dare chip away at for fear of damaging the panels. I'd not want to live someplace where it snowed a lot with solar panels that didn't have some kind of a heating system. I wish I had a camera to show the long walk down the driveway, through the winter woods, and past the field where the wind blows gusts of snow into the air. But my cell phone is on a Caribbean cruise. I have been recording data ever since the fall equinox, and it's almost to the winter solstice now. Time to geek out with some graphs! Solar production has surely been falling off with the waning days, but it seems like it will get through the shortest day with enough, and from then on, no worries.

Highest watt production seen each day I've been here.
Zero all day yesterday.

Power system voltages.
Charge controller limits it to a maximum of 15 volts during the day.
Originally, it was limiting it to 13 volts.

Back in April, I wrote about Jacob and I building his first computer together and how it s running just the Linux command-line interface no graphics at all. And he loves it. Well, as is often the case with him, his interest in various things waxes and wanes over time. He hadn t spent much time with his computer lately, until I suggested he try it one day. He did, and rediscovered sl (a simple program that draws an animated steam locomotive using ASCII characters) again. I showed him how to string sl commands together with the semicolon to get lots of trains, and he loved that. Since then, he s discovered that the up arrow accesses bash history and saves him typing. But more impressive, the other day while I was at work, he was playing with his computer and it looked to him like it had locked up. (I believe he had hit Scroll Lock from talking with Terah.) He was fussy and bothered by that, but then decided he could fix it. He pushed to power button to turn it off, waited for it to shut down, then turned it back on. And logged himself back in. And got right back to his steam locomotives. I run Linux, so he had never been exposed to a power cycle as a troubleshooting mechanism. You re looking at one very impressed dad. And perhaps another future Debian developer?

I received a trackback from Tim Bray s Saving Data Safely post on the Android Developer s blog to my Don t fear the fsync! blog entry, so I guess the cat s out of the bag. Starting with Gingerbread, newer Android phones (starting with the Nexus S) will be using the ext4 file system. Very cool! So just as IBM used to promote Linux by saying that it was scalable enough to run on everything between watches and mainframes, I can now talk about ext4 as running in production on cell phones to Google data centers. How much am I worried about Tim Bray s caution to Android programmers that they use fsync() correctly? Not a lot. Sure, they should make sure they use fsync(), or if they want to be clever, sync_file_range(), to make sure files are appropriately written to disk (or, in Android s case, to flash). But unlike Ubuntu s running on random PC s, with users downloading the latest (possibly buggy) Nvidia drivers, handset manufacturers test their systems very carefully before they let them ship. So if there are any problems, they tend to be found before the phone ships to end-users. So at least in my experience, my Nexus One has been very reliable; it s never crashed on me. So the chances of random crashes when you exit the 3D game (as one Ubuntu user reported and considered acceptable?!? I d be roasting Ubuntu and/or Nvidia under a slow fire if that was a reliably reproducible bug, not considering it par for the course) are very remote. And fsync() is important if systems crash or aren t shut down cleanly. Still, if users are randomly ripping out their batteries to turn off their cell phone in a hurry, because they re too impatient to wait for a controlled shutdown, then sure, we might have problems, and it s a good reason for application writers to use fsync() correctly. By the way, I had nothing to do with the choice to use ext4 on Android. So if you re curious about why ext4 was chosen, I can t say anything authoritatively, since I wasn t consulted before the decision was made (although obviously I m delighted). As far as I can tell after the fact, one of the reasons for choosing ext4 was better performance, especially in the light of dual-core ARM CPU s which are becoming available in large quantities in the near future; YAFFS is single threaded, so it would have been a bottleneck on dual-core systems. Why not btrfs? Well, for all of btrfs s features, it s not out of beta yet, and Gingerbread is shipping today. (Well, in less than a week to Best Buy, if we want to be precise.) This is another reason why I m glad to see ext4 being used on Android is that it validates my decision to keep working on ext4 2-3 years ago, even though newer filesystems like btrfs were promised to be coming down the pike. As I ve said many times before, file systems are like fine wine, and they take many years to mature. So having ext4 ready today is a great way of giving more choices to developers and system administrators about what file system they want to use. No related posts.

I received a trackback from Tim Bray s Saving Data Safely post on the Android Developer s blog to my Don t fear the fsync! blog entry, so I guess the cat s out of the bag. Starting with Gingerbread, newer Android phones (starting with the Nexus S) will be using the ext4 file system. Very cool! So just as IBM used to promote Linux by saying that it was scalable enough to run on everything between watches and mainframes, I can now talk about ext4 as running in production on cell phones to Google data centers. How much am I worried about Tim Bray s caution to Android programmers that they use fsync() correctly? Not a lot. Sure, they should make sure they use fsync(), or if they want to be clever, sync_file_range(), to make sure files are appropriately written to disk (or, in Android s case, to flash). But unlike Ubuntu s running on random PC s, with users downloading the latest (possibly buggy) Nvidia drivers, handset manufacturers test their systems very carefully before they let them ship. So if there are any problems, they tend to be found before the phone ships to end-users. So at least in my experience, my Nexus One has been very reliable; it s never crashed on me. So the chances of random crashes when you exit the 3D game (as one Ubuntu user reported and considered acceptable?!? I d be roasting Ubuntu and/or Nvidia under a slow fire if that was a reliably reproducible bug, not considering it par for the course) are very remote. And fsync() is important if systems crash or aren t shut down cleanly. Still, if users are randomly ripping out their batteries to turn off their cell phone in a hurry, because they re too impatient to wait for a controlled shutdown, then sure, we might have problems, and it s a good reason for application writers to use fsync() correctly. By the way, I had nothing to do with the choice to use ext4 on Android. So if you re curious about why ext4 was chosen, I can t say anything authoritatively, since I wasn t consulted before the decision was made (although obviously I m delighted). As far as I can tell after the fact, one of the reasons for choosing ext4 was better performance, especially in the light of dual-core ARM CPU s which are becoming available in large quantities in the near future; YAFFS is single threaded, so it would have been a bottleneck on dual-core systems. Why not btrfs? Well, for all of btrfs s features, it s not out of beta yet, and Gingerbread is shipping today. (Well, in less than a week to Best Buy, if we want to be precise.) This is another reason why I m glad to see ext4 being used on Android is that it validates my decision to keep working on ext4 2-3 years ago, even though newer filesystems like btrfs were promised to be coming down the pike. As I ve said many times before, file systems are like fine wine, and they take many years to mature. So having ext4 ready today is a great way of giving more choices to developers and system administrators about what file system they want to use. No related posts.

Went to the Chicago Symphony yesterday to see Regina Carter as well as Jazz wunderkind Esperanza Spalding perform one seat each with their respective bands. Regina Carter was presenting material from her current record 'Reverse Thread'. This was a real nice set of African-themed world music featuring Carter herself on violin, Yacouba Sissoko on kora, Will Holshouser on accordion, Chris Lightcap on bass and Alvester Garnett on drums. Some of pieces were really, really nicely done and I particularly enjoyed Holshouser on the accordion. After the break, Esperanza Spalding come on for her Chamber Music Society'. Lovely setup with Spalding on acoustic bass and vocals, Leo Genovese on piano/keyboards, Sara Caswell on violin, Lois Martin on viola, Jody Redhage on cello, the always impressice Terry Lyne Carrington on drums and Leala Cyr on backing vocals (and one co-lead in a really nice duet with Spalding). This was clearly more experimental and a chunk of the audience left during the act. But there is room for improvided chamber music, and it was a good modern music act. And Spalding is really quite impressive and I will gladly go and see her again.