For Debian s birthday, Francesca Ciceri of the Debian Publicity team suggested that developers blog about their first experiences with Debian . I found this a good idea so I m going to share my own early experience. It s quite different from what happens nowadays Before speaking of my early Debian experience, I have to set some context. In my youth, I have always been a Windows user and a fan of Bill Gates. That is until I got Internet at home at that point, I got involved in Usenet and made some friends there. One of those made me discover Perl and it has been somewhat of a revelation for me who had only been programming in Visual Basic, Delphi or ObjectPal. Later the same friend explained me that Perl was working much better on Linux and that Debian Linux installs it by default so I should try this one. I had no idea of what Linux was, but given how I loved Perl, I was eager to try his advice. So I got myself a Tri-Linux CD with Debian/RedHat/Slackware on it and started the installation process (which involved preparing boot floppies). But I did not manage to get the graphical interface working despite lots of fiddling with Xfree86 s configuration file. So I ended up installing RedHat and used it for a few months. But since many of the smart guys in my Usenet community were Debian users, I persisted and finally managed to get it to work! After a few months of usage, I was amazed at everything that was available for free and I wanted to give back. I filed my first bug report in July 1998, I created my first Debian packages in August 1998 and I got accepted as an official Debian developer in September 1998 (after a quick chat over the phone with Martin Schulze or James Troup I never understood the name of my interlocutor on the phone and I was so embarassed to have to use my rusty English over the phone that I never asked). That s right, it took me less than 3 months to become a Debian developer (I was 19 years old back then). I learned a lot during those months by reading and interacting with other Debian developers. Many of those went away from Debian in the mean time but some of them are still involved (Joey Hess, Manoj Srivastava, Ian Jackson, Martin Schulze, Steve McIntyre, Bdale Garbee, Adam Heath, John Goerzen, Marco D Itri, Phil Hands, Lars Wirzenius, Santiago Vila, Matthias Klose, Dan Jacobowitz, Michael Meskes, ). My initial Debian work was centered around Perl: I adopted dpkg-ftp (the FTP method for dselect) because it was written in Perl and had lots of outstanding bug reports. But I also got involved in more generic Quality Assurance work and tried to organize the nascent QA team. It was all really a lot of fun, I could take initiatives and it was clear to me that my work was appreciated. I don t know if you find this story interesting but I had some fun time digging through archives to find out the precise dates if you want to learn more about what I did over the following years, I maintain a webpage for this purpose.

One comment Liked this article? Click here. My blog is Flattr-enabled.

I've temporarily got hold of a Retina Macbook Pro. Fedora 17 boots fine on it with the following kernel arguments: nointremap drm_kms_helper.poll=0 video=eDP-1:2880x1800@45e, although you'll want an updated install image. The nointremap requirement is fixed by this patch, and the others are being tracked here. The gmux chip that controls the multiple graphics chipsets has changed - patches here. That just leaves Thunderbolt.

Thunderbolt is, to a first approximation, PCIe tunnelled over a Displayport connector. It's a high-bandwidth protocol for connecting external devices. Things are complicated by it also supporting Displayport over the same connection, which I'm sure will be hilarious when we get yet another incompatible display protocol. But anyway. I picked up a Thunderbolt ethernet adapter and started poking.

Booting with the device connected looked promising - an ethernet device appeared. Pulling it out resulted in the pcie hotplug driver noticing that it was missing and cleaning up, although the b44 ethernet driver sat around looking puzzled for a while. So far so good. Unfortunately plugging it in again didn't work, and rebooting without any Thunderbolt devices connected not only did nothing on hotplug, it also meant that the Thunderbolt controller didn't show up in lspci at all.

It turns out that there's several problems at play here. The first is the missing Thunderbolt controller. The problem here is the following code in Apple's ACPI tables:

        Method (RMCR, 0, Serialized)
         
            If (LNot (OSDW))
             
                If (LAnd (LEqual (\_SB.PCI0.PEG1.UPSB.DSB1.UPS0.AVND, 0xFFFF), 
                    LEqual (\_SB.PCI0.PEG1.UPSB.DSB2.UPS0.AVND, 0xFFFF)))
                 
                    Store ("RMCR: Disable Link and Power Off Cactus Ridge Chip",
                           Debug)
                    Store (0x01, \_SB.PCI0.PEG1.LDIS)
                    Sleep (0x07D0)
                    Store (0x00, GP23)
                 
             
         

This checks if the system claims to be Darwin (the core of OS X). If not, it checks whether two PCIe bridges have been configured. If both are still in an unconfigured state, it cuts the PCIe link to the upstream PCIe link and then sets GP23 to 0. Looking further, GP23 turns out to be a GPIO line. Setting it to 0 appears to cut power to the Thunderbolt controller. In summary, unless your OS claims to be OS X, booting without an attached Thunderbolt device will result in the chipset being powered down so hard that it's entirely invisible to the OS.

Fixing that is as easy as adding Darwin to the list of operating systems Linux claims to be. So, now I could boot without a connected device and still see the controller. Since ACPI seemed to be important here, I started looking at the other ACPI methods associated with the device. The first one that drew attention was Name(_GPE, 0x14). The _GPE method (not to be confused with the _GPE object at the top of the global ACPI namespace) is defined as providing the ACPI general purpose event associated with the device. Unfortunately, it's only defined as doing this for embedded controllers - Apple's use of it on a PCI device is massively non-standard. But, still, easy enough to handle. Intel chipsets map GPE 0x10 to 0x1f to GPIO lines 0-15, so this GPE is associated with GPIO 4. Looking through the ACPI code showed a bunch of other references to GP04, and it turns out that if the chip is powered down (via setting GP23 to 0) then plugging a device in to the port will generate a GPE. This makes sense - a powered down controller isn't going to notice hotplug events itself, so you need some kind of out of band notification mechanism. There's also another ACPI method that flips the polarity of the GPIO line so it doesn't keep generating events for the entire time a device is connected. It didn't take long to come up with a stub driver that would power down the controller if nothing was connected at boot, and then wake it up again on a hotplug event.

Unfortunately that's as far as I've got. I'd been hoping that everything beyond this was just PCIe hotplug, but it seems not. Apple's Thunderbolt driver is rather too large to just be handling ACPI events, and this document on Apple's website makes it pretty clear that there's OS involvement in events and device configuration. Booting with a device connected means that the firmware does the setup, but if you want to support hotplug then the OS needs to know how to do that - and Linux doesn't.

Getting this far involved rather a lot of irritation at Apple for conspiring to do things in a range of non-standard ways, but it turns out that the real villains of the piece are Intel. The Thunderbolt controller in the Apples is an Intel part - the 82524EF, according to Apple. Given Intel's enthusiasm for Linux and their generally high levels of engagement with the Linux development community, it's disappointing[1] to discover that this controller has been shipping for over a year with (a) no Linux driver and (b) no documentation that would let anyone write such a driver. It's not even mentioned on Intel's website. So, thanks Intel. You're awful.

Anyway. This was my attempt to spend a few days doing something more relaxing than secure boot, and all I ended up with was eczema and liver pain. Lesson learned, hardware vendors hate you even more than firmware vendors do.

[1] By which I mean grotesquely infuriating

comments

A new version 0.2.7 of RInside is now available via CRAN. RInside provides a set of convenience classes which facilitate embedding of R inside of C++ applications and programs, using the classes and functions provided by the Rcpp R and C++ integration package. This release adds two new examples subdirectories demonstrating use of RInside with, respectively, RcppArmadillo and RcppEigen. We extended the 'web application' example using the Wt toolkit by adding CSS and XML support, and added another new example motivated by a StackOverflow question. CMake support has been added for Windows as well thanks to Peter Aberline---he also contributed CMake code for the two new example directories but that contribution made it only into SVN and not this release. All changes since the last release are summarized below in the NEWS file entry:

Changes in RInside version 0.2.7 (2012-08-12)

• New fifth examples subdirectory 'armadillo' with two new examples showing how to combine RInside with RcppArmadillo
• New sixth examples subdirectory 'eigen' with two new examples showing how to combine RInside with RcppEigen
• Prettified the Wt example 'web application' with CSS use, also added and XML file with simple headers and description text
• New example rinside_sample12 motivated by StackOverflow question on using sample() from C
• Added CMake build support on Windows for the examples

CRANberries also provides a short report with changes from the previous release. More information is on the RInside page. Questions, comments etc should go to the rcpp-devel mailing list off the Rcpp R-Forge page.

Some of you may have noticed that I am not a concise author. Perhaps that has something to do with the fact that I am not a concise reader. I like facts, details, and lots of them. So as a recent Nexus 7 purchaser, here you go. Genesis I ve long used Android devices, and last year had a company-issued Motorola Xoom, which was the first Google Experience tablet with Honeycomb. That tablet has specs roughly similar to iPads; its 10.1 screen was the same, the 1280 800 screen was better than the iPad available at the time, and its 730g weight identical to the early iPads (though 10% higher than the current iPad). I lost access to it when I changed jobs, and had been without a tablet until recently. Other devices I own are the Galaxy Nexus, sporting a 4.65 screen; and what s now called the Kindle Keyboard, with an eInk screen. I had been somewhat interested in the Kindle Fire, but the closed nature and limited capability of the system kept me away. The Nexus 7 reviews, however, were stunning, as was the price. $200 for a great tablet. I wound up buying the $250 16GB model. But not until after I spent a great deal of time thinking about size. Physical Size My main concern was that the Nexus 7 would be too small to be useful. I had never been particularly pleased with my input speed on the Xoom. I tried to touch type on it, but was just never fast enough to surpass frustratingly slow. I have long been a fast and accurate typist on keyboard; well over 100 words per minute, and it is frustrating when my fingers can t maintain that speed. I figured the situation would be even worse on the Nexus 7, given its smaller size. I also found the Xoom to sometimes feel a little small with the 10 screen, and was concerned about that as well. And finally, 7 doesn t sound all that much larger than 4.65 . However, having actually had the Nexus 7 for a little while now, I m very pleased with the size, and may even prefer it. The 10 tablets are just too big and heavy to comfortably hold in one hand, and I ve realized that part of my Xoom frustration was the fact that I had to set it down and prop it up for anything beyond very brief use. At 340g, the Nexus 7 is less than half the weight of the Xoom or iPad, and it makes a huge difference. While still nowhere near where I d be with a keyboard, two-thumb typing in portrait mode, or even something approaching touch typing in landscape mode, is possible on the Nexus 7. The screen size hasn t been a bother, at all. This may be due to the fact that it s higher resolution (it s 1280 800 like the Xoom, but those pixels are crammed into only 7 ). I think it s also partly due to the fact that the browser in Jelly Bean is significantly better than the one in Honeycomb, and perhaps that websites are better at tablet-friendliness, too. Overall, the Nexus 7 feels a lot farther from the size of a laptop than did the Xoom, and as such is more prone to come with me in lots of situations, I think. It works reasonably well with foldable Bluetooth keyboards, so when thinking about a laptop replacement or alternative, that might be the way I go. A Bluetooth mouse also works with it, though I found it didn t provide near the utility that a BT keyboard does. Display The display is both amazing and disappointing. Browse some photos and some of them will show up in eye-popping clarity. Websites display fine. But the screen can also take on a washed-out appearance at times. I am notoriously picky in my displays, and this bothered me enough that I researched it. Analysis has shown that poor firmware calibration has lead to the compression of highlights, which mirrors what I was seeing. I am mostly used to it by now, but it s a disappointment. Most of the time, though, the screen is excellent. In comparison to my eInk Kindle, however, I don t think any tablet will ever be as good for book reading. The eInk screen truly is easier on the eyes, and the reflection of overhead lights on the Nexus 7 display can be distracting at first. I have had occasional issues with it not registering touches properly. This is always cleared up by touching the power button to put the unit to sleep, then waking it back up. Other Hardware There are three hardware buttons: power and volume up/down. Physically, the device fits my hand well, though I might wish it was a little lighter like my Kindle. Charging is accomplished via high-power 2A micro-USB, and there is, of course, a headphone port. There is no alert LED like my Galaxy Nexus has, and no vibration feature. The speaker is on the back, and the microphones along the left side a position which, it appears, many Nexus 7 cases are blocking. Battery Life I am astonished at how good this device is battery-wise, especially compared to the battery disaster that is the Galaxy Nexus. Google claims the Nexus 7 can survive 8 hours of solid screen-on use, and I don t doubt it. Mine s never gotten low enough to get a solid measurement. Wifi The wifi works well, as far as it goes. The wifi doesn t support 802.11n in 5GHz, which although somewhat common for devices like this, is a bit of a disappointment. Software The big story about the Nexus 7 is Jelly Bean. I had used Honeycomb on the Xoom, and Ice Cream Sandwich on my Galaxy Nexus, so I m familiar with its predecessors. Let s take a look. Project Butter Much has been made of Project Butter, Google s attempt to optimize Android to improve its responsiveness and the smoothness of things like scrolling. I can say they have done quite well. This device is so smooth you don t notice how smooth it is. It wasn t until I had been using it for a bit that I really noticed. That s a job well-done. Chrome The browser in Jelly Bean is now called Chrome. I am not sure if this is just marketing or not. It doesn t really feel all that different from previous versions of the Android browser, and the changes have been along the lines of incremental changes Google has introduced before. One of the very best new features happens when you touch a link that is close to other links on a page. Rather than getting a pretty much random page, Chrome pops up a partial-screen zoom box showing the part of the page near where your finger touched. With everything showing up huge, it is now easy to touch the precise link you want. Do so, and the box goes away, and your page loads. I am amazed at how much improvement this one change brings. Compared to ICS Browser, bookmarks can be brought up quicker, and the tab interface is nicer. All is not perfect in the land of Chrome, however. It contains several regressions from the Ice Cream Sandwich browser. I have two complaints about bookmarks. One is that previous versions of the browser would show thumbnails of sites in the bookmark viewer. This was a nice navigation aid. Chrome shows only favorites icons, if one is available, or a generic icon if not. Also, the bookmarks synced with other Android devices are called, confusingly enough, Desktop Bookmarks now, and require an extra tap to access. I have had occasional trouble with Chrome not wanting to prompt for credentials for servers on my LAN that use HTTP auth. Chrome has also removed the ICS browser s ability to save a page, including all its elements, for offline viewing. Good for things like an airline checkin screen and such. I have no idea why Chrome removed this. I installed the Firefox Beta for Android, which also doesn t have the offline save feature, but it does have a save to PDF feature. Soft Keyboard The on-screen soft keyboard in Jelly Bean is a significant regression from previous versions of Android. My biggest complaint is the lack of visual feedback for keypresses. On earlier versions of Android, when you push a key, you ll see an image of it pop up on the screen, offset a little from the location of the key itself. In JB, all that happens is that the key itself changes colors. Not very helpful, because it is under your finger at the time. This small thing frustrates me to no end. The keyboard in ICS introduced some nice features as well, mainly long-presses as shortcuts to other features. For instance, you can long-press a key on the top row of letters to get numerals without having to switch to the number mode. Similarly, long press the period and you get other common punctuation. The JB keyboard removed both of those features. Thankfully, in the Market, there is an app called Ice Cream Sandwich Keyboard. It appears geared towards people running earlier versions of Android. Sadly, it is also a step up over what we have in JB. Google Now and Voice Recognition The other main headline feature in Jelly Bean is Google Now. The somewhat-competitor to Apple s Siri, Google Now takes a bit of a different approach than Apple. It is said that Siri is better than Google Now at responding to queries, but Google Now is better at predicting what you want to know before you ever ask. I haven t ever used Siri, but I would buy that explanation. Google Now is available with a swipe up from the bottom of the screen, or with a single touch from any Home screen. Bring it up and it shows you current information about what it thinks you need to know. Examples include weather and forecast information, time to get to home or work from your current location, alerts that you need to leave soon to get to a certain place on time, flight schedules, sports scores, etc. Google Now has been mostly a gimmick to me, but that may be because I fall outside its target demographic in significant ways. I live nowhere near a public transportation system, work at home for the most part, haven t flown wince I ve had the Nexus 7, don t follow sports, and already know how long it takes to get places (and when it varies, it s because of muddy roads or harvest neither things that traffic services know about.) The weather widget always seems to show the temperature from a couple of hours ago. It does show the weather in your current location. Well, mostly. I was in Newton, KS one day. I tapped on the icon for more detail. That simply took me to a Google search for weather Newton . Which showed me the weather for Newton, Massachusetts 1600 miles away. Fail. Speech recognition in JB is definitely improved. It is somewhat useful with Google Now. I like being able to simply say set alarm for 30 minutes. And it does it a lot quicker than I could in the interface. It s supposed to be able to let me bring up my contacts in the same way, but it is much more likely to try turning such an attempt into a Google search than an actual display of a contact. It s picky on the precise language used for setting an alarm too; say it slightly differently, and it s another Google search. JB also supports limited offline speech recognition. I say limited because it s a bit strange. I have, for instance, a Remember the Milk widget on my home screen. It has a microphone icon to use to speak a new reminder. Tap it, and you can t use it offline. It also has a button that brings up the on-screen keyboard. Do that, then touch the microphone on the keyboard, and you can use offline recognition. I have no idea how to explain this difference, since both are clearly using Google s engine. The speech recognition is indeed better, and might make it suitable for use instead of a keyboard for composing short texts and such. But it rarely produces even a sentence that I don t have to correct in some way, even now. Conclusion Despite some of its shortcomings, I am very fond of the Nexus 7. It is an excellent device. And at $200-$250, it is an AMAZING device. I am truly impressed with it, and don t regret my purchase at all.

Review: Design Patterns, by Erich Gamma, et al.

Author:	Erich Gamma
Author:	Richard Helm
Author:	Ralph Johnson
Author:	John Vlissides
Publisher:	Addison-Wesley
Copyright:	1995
Printing:	September 1999
ISBN:	0-201-63361-2
Format:	Hardcover
Pages:	374

Design Patterns: Elements of Reusable Object-Oriented Software by the so-called "Gang of Four" (Gamma, Helm, Johnson, and Vlissides) is one of the best-known books ever written about software design, and one of the most widely cited. The language introduced here, including the names of specific design patterns, is still in widespread use in the software field, particularly with object-oriented languages. I've had a copy for years, on the grounds that it's one of those books one should have a copy of, but only recently got around to reading it. The goal of this book is to identify patterns of design that are widely used, and widely useful, for designing object-oriented software. It's specific to the object-oriented model; while some of the patterns could be repurposed for writing OO-style programs in non-OO languages, they are about inheritance, encapsulation, and data hiding and make deep use the facilities of object-oriented design. The patterns are very general, aiming for a description that's more general than any specific domain. They're also high-level, describing techniques and methods for constructing a software system, not algorithms. You couldn't encapsulate the ideas here in a library and just use them; they're ideas about program structure that could be applied to any program with the relevant problem. With the benefit of seventeen years of hindsight, I think the primary impact of this book has been on communication within the field. The ideas in here are not new to this book. Every pattern in Design Patterns was already in use in the industry before it was published; the goal was taxonomy, not innovation. One would not come to Design Patterns to learn how to program, although most introductory texts on object-oriented programming now borrow much of the pattern terminology. Rather, Design Patterns is as influential as it is because it introduced a shared terminology and a rigor around that terminology, allowing writers and programmers to put a name to specific program structures and thus talk about them more clearly. This also allows one to take a step back and see a particular structure in multiple programs, compare and contrast how it's used, and draw some general conclusions about where it would be useful. I have the feeling that the authors originally hoped the book would serve as a toolbox, but I think it's instead become more of a dictionary. The pattern names standardized here are widely used even by people who have never read this book, but I doubt many people regularly turn to this book for ideas for how to structure programs. Design Patterns is divided into two parts: a general introduction to and definition of a software pattern followed by a case study, and then a catalog of patterns. The catalog is divided into creational patterns (patterns for creating objects), structural patterns (patterns for composing objects into larger structures), and behavioral patterns (patterns for interactions between objects). Each pattern in turn follows a very rigid presentation structure consisting of the pattern name and high-level classification, its basic intent, other common names, a scenario that motivates the pattern, comments on the applicability of the pattern, the structure and classes or objects that make up the pattern, how those participants collaborate, how the pattern achieves its goals, comments on implementation issues, sample code, known uses of the pattern in real-world software, and related patterns. As with a dictionary, the authors go to great lengths to keep the structure, terminology, and graphical representations uniform throughout, and the cross-referencing is comprehensive (to the point of mild redundancy). As for the patterns themselves, their success, both as terminology and as useful design elements, varies. Some have become part of the core lexicon of object-oriented programming (Factory Method, Builder, Singleton), sometimes to the point of becoming syntactic elements in modern OO languages (Iterator). These are terms that working programmers use daily. Others aren't quite as widespread, but are immediately recognizable as part of the core toolkit of object-oriented programming (Adapter, Decorator, Proxy, Observer, Strategy, Template Method). In some cases, the technique remains widespread, but the name hasn't caught on (Command, for example, which will be immediately familiar but which I rarely hear called by that name outside of specific uses inside UI toolkits due to ambiguity of terms). Other patterns are abstract enough that it felt like a bit of a reach to assign a name to them (Bridge, Composite, Facade), and I don't think use of those names is common, but the entries are still useful for definitional clarity and for comparing similar approaches with different implications. Only one pattern (Interpreter) struck me as insufficiently generic to warrant recording in a catalog of this type. So far, so good, but the obvious question arises: if you've not already read this book, should you read it? I think the answer to that is debatable. The largest problem with Design Patterns is that it's old. It came late enough in the development of object-oriented programming that it does capture much of the foundation, but OO design has continued to change and grow, and some patterns have either been developed subsequently or have become far more important. For example, Model-View-Controller is conspicuous by its absence, mentioned only in passing in the discussion of the Observer pattern. Any pattern catalog written today would have an extensive discussion. Similarly absent are Inversion of Control and Data Access Object, which are much more central to the day-to-day world of the modern programmer than, say, Memento or Visitor. One could easily go on: Lazy Initialization, Mock Object, Null Object... everyone will have their own list. A related problem is that all the implementation examples are shown in either C++ or Smalltalk (occasionally both). Those were probably the best languages to use at the time, but it's doubtful a modern rewrite would choose them. Smalltalk, in particular, I found nearly incomprehensible for the uninitiated, to the point where I ignored the code and only read the surrounding English description. C++ fares better, but Design Patterns occasionally drifts off into tedious discussions of how to work around C++'s limitations in ways that are irrelevant to the pattern itself and would not be necessary in, say, Java or Python. (This is ameliorated by the fact that C++, unlike Smalltalk, is still in widespread use, so those discussions remain moderately helpful for some readers.) Design Patterns is not, therefore, a very good source for a working knowledge of the most common patterns in use today. It has also become somewhat obsolete via its own success: the concept of a design pattern has become so popular that nearly all introductory texts include at least a basic discussion of design patterns and an introduction to the most notable and useful patterns. I think that's a more comfortable and more efficient way to pick up the basics than reading through this book, which is somewhat dense and which expects from the reader a reasonably good working knowledge of object-oriented programming. And, once you have the basics, MVC, DAO, and similar design patterns are probably more important than the more subtle design patterns presented here. That said, I think the rigor of description and the comparisons and discussions here still have some value. Design Patterns encourages the reader to look at patterns from a higher-level perspective, to think about meta-patterns such as the balance between data hiding and access, or between structure designed for the present purpose and structure that's adaptable to future needs. It's also mildly interesting from a historical standpoint; one can see the inspiration for future language designers in how problems are described here, and see how many of the implementation issues and negative consequences have been corrected or simplified by richer language designs. Overall, I would hesitate to recommend buying this book today, particularly at new textbook prices. But if you're a working object-oriented software designer or programmer, I think it's worth checking out from a library (and, thanks to its influence, any library with a decent software design section will almost certainly have a copy). Read the overall discussion, skim the catalog, and read the discussion of the patterns that strike your interest. It may help provide some additional structure and perspective to how you think about OO design. Rating: 6 out of 10

Most Debian source packages are using the 3.0 (quilt) format. This means that Debian changes to upstream files are managed in a quilt patch series. Knowledge of quilt is thus a must if you want to get involved in some serious packaging work. Don t worry, this tutorial will teach you how to use quilt in the context of Debian packaging. Pre-requisites Install the packaging-dev package to get a decent set of packages to do Debian packaging. It includes the quilt package.

$ sudo apt-get install packaging-dev

What s quilt? Read the description of the Debian package:

Quilt manages a series of patches by keeping track of the changes each of them makes. They are logically organized as a stack, and you can apply, un-apply, update them easily by traveling into the stack (push/pop). Quilt is good for managing additional patches applied to a package received as a tarball or maintained in another version control system. The stacked organization is proven to be efficient for the management of very large patch sets (more than hundred patches). As matter of fact, it was designed by and for Linux kernel hackers (Andrew Morton, from the -mm branch, is the original author), and its main use by the current upstream maintainer is to manage the (hundreds of) patches against the kernel made for the SUSE distribution.

The various files involved To better understand the various quilt commands, you should have a basic idea of how the tool works. The stack of patches is maintained in a dedicated directory ( patches by default, but in Debian packages we override this value to debian/patches ). This directory contains the patch files and a series file that gives an ordered list of patches to apply. Example:

$ ls debian/patches/
01_use_stdlib_htmlparser_when_possible.diff
02_disable-sources-in-sphinxdoc.diff
03_manpage.diff
06_use_debian_geoip_database_as_default.diff
series
$ cat debian/patches/series
01_use_stdlib_htmlparser_when_possible.diff
02_disable-sources-in-sphinxdoc.diff
03_manpage.diff
06_use_debian_geoip_database_as_default.diff

When quilt is used, it also maintains some internal files in a directory of its own (it s named .pc ). This directory is used to know what patches are currently applied (.pc/applied-patches) and to keep backup copies of files modified by the various patches. Configuring quilt Before going further, you should put this in your ~/.quiltrc file:

for where in ./ ../ ../../ ../../../ ../../../../ ../../../../../; do
    if [ -e $ where debian/rules -a -d $ where debian/patches ]; then
        export QUILT_PATCHES=debian/patches
        break
    fi
done

This ensures that quilt will always use debian/patches instead of patches when your current directory is within a Debian source package where debian/patches exists. If you only use quilt for debian packaging, then you can be more expedient and put a simple export QUILT_PATCHES=debian/patches in that file. As a matter of personal preferences, I also have those lines in my ~/.quiltrc:

QUILT_PUSH_ARGS="--color=auto"
QUILT_DIFF_ARGS="--no-timestamps --no-index -p ab --color=auto"
QUILT_REFRESH_ARGS="--no-timestamps --no-index -p ab"
QUILT_DIFF_OPTS='-p'

It enables syntax coloring for the output of several commands and customizes the generated patches to get rid of useless information. I recommend you to use those settings too. Applying and unapplying patches, navigating in the stack of patches When you already have a patch series, you can navigate in the stack of patches so that any subset of consecutive patches (starting from the bottom) can be applied. quilt series will list all patches known by quilt. You can apply all patches with quilt push -a or unapply them all with quilt pop -a. You can also verify what patches are applied (quilt applied) or unapplied (quilt unapplied). quilt push applies the next unapplied patch (i.e. the patch returned by quilt next) and quilt pop unapplies the last applied patch (i.e. the patch returned by quilt top). You can give a patch name as parameter to quilt push/pop and it will apply/unapply all the patches required until the given patch is on the top. Here are some examples of navigation in a quilt patch series

$ quilt series
02_disable-sources-in-sphinxdoc.diff
03_manpage.diff
06_use_debian_geoip_database_as_default.diff
$ quilt applied
No patches applied
$ quilt next
02_disable-sources-in-sphinxdoc.diff
$ quilt push
Applying patch 02_disable-sources-in-sphinxdoc.diff
patching file docs/conf.py
Now at patch 02_disable-sources-in-sphinxdoc.diff
$ quilt push
Applying patch 03_manpage.diff
patching file docs/man/django-admin.1
Now at patch 03_manpage.diff
$ quilt applied
02_disable-sources-in-sphinxdoc.diff
03_manpage.diff
$ quilt unapplied
06_use_debian_geoip_database_as_default.diff
$ quilt pop -a
Removing patch 03_manpage.diff
Restoring docs/man/django-admin.1
Removing patch 02_disable-sources-in-sphinxdoc.diff
Restoring docs/conf.py
No patches applied
$ quilt push 03_manpage.diff
Applying patch 02_disable-sources-in-sphinxdoc.diff
patching file docs/conf.py
Applying patch 03_manpage.diff
patching file docs/man/django-admin.1
Now at patch 03_manpage.diff
$ quilt top
03_manpage.diff

Creating a new patch If there s no quilt series yet, you want to create the debian/patches directory first. If you already have one, you need to decide where to insert the new patch. Quilt will always add the new patch just after the patch which is currently on top. So if you want to add the patch at the end of the series, you need to run quilt push -a first. Then you can use quilt new name-of-my-patch.diff to tell quilt to insert a new empty patch after the current topmost patch. In this operation quilt does almost nothing except updating the series file and recording the fact that the new patch is applied (even if still empty at this point!). Now to add changes in this patch, you re supposed to modify files but only after having informed quilt of your intent to modify those files. You do this with quilt add file-to-modify. At this point quilt will make a backup copy of that file so that it can generate the final patch when you re done with your changes. It s quite common to forget this step and to be unable to generate the patch afterward. That s why I recommend you to use quilt edit file-to-modify which is a shorthand for doing quilt add and then opening the file in your favorite text editor. If you want, you can review your work in progress with quilt diff. When you re done with the changes, you should call quilt refresh to generate the patch (or to update it if it was already existing). And since you re a good packager, you call quilt header --dep3 -e to add DEP-3 meta-information to your patch header. Importing an external patch If someone else already prepared a patch, you can just import it right away with quilt import /tmp/the-patch. If you want to import it under a better name you can use the option -P better-patch-name . Like quilt new, it inserts the patch after the topmost patch. Updating patches for a new upstream version With some luck, your patches will still apply with some offsets in line numbers (quilt displays those offsets) and sometimes with some fuzz:

$ quilt push
[...]
Hunk #1 succeeded at 1362 (offset 11 lines).
Hunk #2 succeeded at 1533 with fuzz 1 (offset 4 lines).
[...]

While offsets are nothing to worry about (it means some lines were added and/or removed before the patched part), fuzz means that patch had to ignore some context lines to find the place where to apply the changes. In that case, you need to double check that patch did the right thing because it might have made changes somewhere where it shouldn t. Also, you ll have to update those patches because dpkg-source doesn t accept any fuzz. If you re confident that all patches are correctly applied by quilt, you can refresh them to get rid of those warnings:

$ quilt pop -a
[...]
$ while quilt push; do quilt refresh; done

That was for the easy case. Now let s deal with the case where some of the patches no longer apply. There s one case that is usually nice to have:

$ quilt push
Applying patch 04_hyphen-manpage.diff
patching file docs/man/django-admin.1
Hunk #1 FAILED at 194.
1 out of 1 hunk FAILED -- rejects in file docs/man/django-admin.1
Patch 04_hyphen-manpage.diff can be reverse-applied

When the patch can be reverse-applied, it means that the upstream authors included the Debian patch (or that they made the same change even though you forgot to forward the patch). You can thus get rid of it:

$ quilt delete -r 04_hyphen-manpage.diff
Removed patch 04_hyphen-manpage.diff

Note that without the -r the patch is only dropped from the series file. With -r the patch file is also removed. But there s a less desirable case where the patch is still relevant but it doesn t apply any longer:

$ quilt push
Applying patch 01_disable_broken_test.diff
patching file tests/regressiontests/test_utils/tests.py
Hunk #1 FAILED at 422.
1 out of 1 hunk FAILED -- rejects in file tests/regressiontests/test_utils/tests.py
Patch 01_disable_broken_test.diff does not apply (enforce with -f)

In that case, you should follow quilt s advice to force the patch application, manually apply the parts of the patch that were rejected, and then refresh the patch.

$ quilt push -f
Applying patch 01_disable_broken_test.diff
patching file tests/regressiontests/test_utils/tests.py
Hunk #1 FAILED at 422.
1 out of 1 hunk FAILED -- saving rejects to file tests/regressiontests/test_utils/tests.py.rej
Applied patch 01_disable_broken_test.diff (forced; needs refresh)
$ vim tests/regressiontests/test_utils/tests.py    
$ quilt refresh
Refreshed patch 01_disable_broken_test.diff

Other quilt commands You should probably read quilt s manual page too to learn about the various other commands and options that exist. There s at least quilt rename new-name that you can also find useful to rename the topmost patch (you can use -P patch-to-rename to rename a patch which is not currently at the top). Feedback Please leave comments if you have suggestions of improvements, or if there are some tips that are good to know. I might incorporate them in this article. Feel free to share this article with newbie packagers which are struggling with quilt. For your convenience, you can also refer to this article with this URL: http://raphaelhertzog.com/go/quilt

Found it useful? Be sure to not miss other packaging tips (or lessons), click here to subscribe to my free newsletter and get new articles by email.

2 comments Liked this article? Click here. My blog is Flattr-enabled.

Monthly DPL bits, fresh from the oven.

---

Tip to feel good about the release #476: before reading this, grab and fix one of the RC bugs affecting Wheezy. Done? Now you're ready for a slightly less exciting report of DPL activities. Highlights

• I've wrapped up a couple of years of discussions with FSF representatives, proposing a joint working group on FSF' assessment of Debian Free-ness. Some news coverage ensued. The discussion list that has been set up welcomes participants and, in particular, could use some more DDs, fond of Debian principles but also interested in external review of our free/non-free separation.
• DebConf12 has happened!, and many of us attended it. Due to family reasons I've been able to attend only DebCamp and DebConf day-0, just in time to deliver the traditional "bits from the DPL" speech. Among other topics, in the speech I've discussed governance scaling issues in our governance, and in particular about the role of the DPL --- issue which I'm partly trying to address with the "DPL helpers" initiatives. Slides and videos of the talk are available. As many others DebConf people, I took part in Nicaraguan TV shows about DebConf, showing a good deal of interest by the local media in our presence. I don't have links to the recordings yet, but I suspect DebConf orga people will provide them as soon as they're available.
• Many others DebConf12 events are relevant for future Project directions. I went through many of the events I've missed thanks to the work of the video team and of the gobby note takers. I recommend doing the same. Some events are particularly relevant to recent project-wide discussions, such as EFI, hijacking, and the release process and I'm sure you'll find more If you're a DebConf12 speaker and produced slides, please upload them to penta, so that they could be used as future reference.

Assets

• On the hardware front, we bought another server according to DSA's "5-year plan". It will be used as a ganeti node at the man-da hosting location. We bought it thanks to donated money, for a total of ~6'600 EUR.
• (After ~8 months of pestering,) we got from SPI a data dump of Debian financial transactions there. Thanks for this result goes to the SPI Board and in particular to Robert Brockway and Joerg Jaspert. This was the last blocker to produce detailed financial reports for the past couple of years of Debian activities. Unfortunately, it still is a blocker: Debian auditors found out that the data dump is not complete. SPI will send us remaining data soon , but I don't have an ETA, unfortunately.
• An ftpmasters sprint is being organized and I've been happy to approve its budget. Help is still needed on the -sprints list to keep up with the organization part of sprints, including activities like documenting sprints to show donors what we do with their help.

Logo & trademark

• to stop the absurdity of not using the official logo ("with Debian") as part of our official theme, I advanced a bit on the topic of relicensing the logo under a DFSG-free license. I sought a second legal advice to SFLC (as requested by the SPI board). They confirm we can safely do the relicensing (without undermining our rights on contained marks) under a license like dual LGPLv3+ / CC-BY-SA, provided that a suitable trademark policy is in place
• which is why I've posted to -project a trademark policy draft, based on legal advice from SFLC. I've gather useful feedback already, but more is welcome. Note that, as Russ very eloquently put it, we do not particularly enjoy thinking about which restrictions we put on the usage of our marks. But given we do have (and use) those marks, we should better empower the legal owners (SPI in this case) to properly defend them. Our specification for the policy to SFLC has been "as free as possible", and that is what we're trying to achieve.

Note that the actual logo relicensing should be done by SPI, via their board, upon request of mine (as Debian Project liaison). We won't make it for the next SPI board meeting, as it is on 4 days away. But we can aim for the subsequent meeting, on September 13th. If all goes well (big "if"), we will enjoy a DFSG-free logo after that date. Internal organization Recent flurry of re-organization in the tech-ctte --- which I somewhat triggered pushing for periodic meetings --- seems to be proceeding well. I'm very happy about it, as we all need to trust that tech-ctte decisions will be not only sound, but also prompt. If you're interested into this topic, some recent evidence of the ongoing reorganization and its result can be found in their DebConf12 BoF, minutes of the last meeting, a set of forthcoming GRs, and the recent great decision of posting decisions results to d-d-a (as it happened for node/nodejs). Kudos to tech-ctte members for the recent activism! I haven't worked on it myself directly, but I highlight Enrico's work on server side archival of NM conversations. It has the potential of enabling automatic detection of stuck NM processes. I'm a bit rusty as AM now, but I think missing that ability is one of the main remaining causes of frustration when joining Debian. So, if you are an AM, please opt-in and use this feature with your appicants. If you are not an AM why not? Miscellanea Some misc legal stuff:

• I got the legal advice we asked about the possibility of turning mentors.debian.net into an official service. The worry there was about the potential copyright infringements by mentees. In the end, we don't need to worry about that, provided we've clear contact points (and "reasonable" timing ) for removing infringing material once it's been reported, and that we declare our public policy about that. I've a draft policy document to start with. I'll soon contact mentors.d.n to adapt it where needed and have it vetted by SPI before going live.
• I've worked with GNOME maintainers (thanks in particular to Jordi Mallach) to rework the mixed Debian/GNOME logo on their alioth project page, to stay on the safe side and avoid any ambiguity wrt GNOME trademark policy.

and some misc "political" stuff:

• we've been happy to accept an invitation to attend the GNOME Advisory Board meeting at GUADEC: Jord (thanks, again!) kindly accepted to represent Debian there. An early report has been posted on his blog, a more detailed one is forthcoming to -project.
• OSI has discussed with their affiliates the possibility of signing the Declaration of Internet Freedom. On behalf of Debian, I've vouched for it. By the way, I'm serving pro tempore as Debian liaison at OSI, but if there are people interested in doing that, I'll be happy to consider handing over the responsibility to other (it's a quite lightweight one, at present).

Happy RC bug squashing,
Cheers.

---

PS the boring day-to-day activity log for July 2012 is available at master:/srv/leader/news/bits-from-the-DPL.txt.201207

A copy of this post is sent to soc-coordination@lists.alioth.debian.org as well as to debian-bootstrap@lists.mister-muffin.de.

Diary

July 2

• playing around with syntactic dependency graphs and how to use them to flatten dependencies

July 4

• make work with dose 3.0.2
• add linux-amd64 to source architectures
• remove printing in build_compile_rounds
• catch Not_found exception and print warning
• use the whole installation set in crosseverything.ml instead of flattened dependencies
• detect infinite loop and quit in crosseverything.ml
• use globbing in _tags file
• use wildcards and patsubst in makefile

July 5

• throw a warning if there exist binary packages without source packages
• add string_of_list and string_of_pkglist and adapt print_pkg_list and print_pkg_list_full to use them
• fix and extend flatten_deps - now also tested with Debian Sid

July 6

• do not exclude the crosscompiled packages from being compiled in crosseverything.ml
• clean up basebuildsystem.ml, remove old code, use BootstrapCommon code
• clean up basenocycles.ml, remove unused code and commented out code
• add option to print statistics about the generated dependency graph
• implement most_needed_fast_wrong as well as most_needed_slow_correct and make both available through the menu

July 7

• allow to investigate all scc, not only the full graph and the scc containing the investigated package
• handle Not_found in src_list_from_bin_list with warning message
• handle the event of the whole archive actually being buildable
• replace raise Failure with failwith
• handle incorrectly typed package names
• add first version of reduced_dist.ml to create a self-contained mini distribution out of a big one

July 8

• add script to quickly check for binary packages without source package
• make Debian Sid default in makefile
• add *.d.byte files to .gitignore
• README is helpful now
• more pattern matching and recursiveness everywhere

July 9

• fix termination condition of reduced_dist.ml
• have precise as default ubuntu distribution
• do not allow to investigate an already installable package

July 10

• milestone: show all cycles in a graph
• add copyright info (LGPL3+)

July 11

• advice to use dose tools in README

July 16

• write apt_pkg based python filter script replacing grep-dctrl

July 17

• use Depsolver.listcheck more often
• add dist_graph.ml
• refactor dependency graph code into its own module

July 18

• improve package selection for reduced_dist.ml
• improve performance of cycle enumeration code

July 20

• implement buildprofile support into dose3

July 22

• let dist_graph.ml use commandline arguments

July 23

• allow dose3 to generate source package lists without Build- Depends Conflicts -Indep

July 29

• implement crosscompile support into dose3

Results

Readme There is not yet a writeup on how everything works and how all the pieces of the code work together but the current README file provides a short introduction on how to use the tools.

• build and runtime dependencies
• compile instructions
• execution examples for each program
• step by step guide how to analyze the dependency situation
• explanation of general commandline options

A detailed writeup about the inner workings of everything will be part of a final documentation stage.

License All my code is now released under the terms of the LGPL either version 3, or (at your option) any later version. A special linking exception is made to the license which can be read at the top of the provided COPYING file. The exception is necessary because Ocaml links statically, which means that without that exception, the conditions of distribution would basically equal GPL3+.

reduced_dist.ml Especially the Debian archive is huge and one might want to work on a reduced selection of packages first. Having a smaller selection of the archive would be significantly faster and would also not add thousands of packages that are not important for an extended base system. I call a reduced distribution a set of source packages A and a set of binary packages B which fulfill the following three properties:

• all source packages A must be buildable with only binary packages B being available
• all binary packages B except for architecture:all packages must be buildable from source packages A

The set of binary packages B and source packages A can be retrieved using the reduced_dist program. It allows to either build the most minimal reduced distribution or one that includes a certain package selection. To filter out the package control stanzas for a reduced distribution from a full distribution, I originally used a call to grep-dctrl but later replaced that by a custom python script called filter-packages.py. This script uses python-apt to filter Packages and Sources files for a certain package selection.

dist_graph.ml It soon became obvious that there were not many independent dependency cycle situation but just one big scc that would contain 96% of the packages that are involved in build dependency cycles. Therefor it made sense to write a program that does not iteratively build the dependency graph starting from a single package, but which builds a dependency graph for a whole archive.

Cycles I can now enumerate all cycles in the dependency graph. I covered the theoretical part in another blog post and wrote an email about the achievement to the list. Both resources contain more links to the respective sourcecode. The dependency graph generated for Debian Sid has 39486 vertices. It has only one central scc with 1027 vertices and only eight other scc with 2 to 7 vertices. All the other source and binary packages in the dependency graph for the archive are degenerate components of length one. Obtaining the attached result took 4 hours on my machine (Core i5 @ 2.53GHz). 1.5 h of that were needed to build the dependency graph, the other 2.5 hours were needed to run johnson's algorithm on the result. Memory consumption of the program was at about 700 MB. It is to my joy that apparently the runtime of the cycle finding algorithm for a whole Debian Sid repository as well as the memory requirements are within orders of magnitude that are justifiable when being run on off-the-shelf hardware. It must also be noted that nothing is optimized for performance yet. A list of all cycles in Debian Sid up to length 4 can be retrieved from this email. This cycle analysis assumes that only essential packages, build-essential and dependencies and debhelper are available. Debhelper is not an essential or build-essential package but 79% of the archive build-depends on it. The most interesting cycles are probably those of length 2 that need packages that they build themselves. Noticeable examples for these situations are vala, python, mlton, fpc, sbcl and ghc. Languages seem love to need themselves to be built.

Buildprofiles There is a long discussion of how to encode staged build dependency information in source packages. While the initial idea was to use Build-Depends-StageN fields, this solution would duplicate large parts of the Build-Depends field, which leads to bitrot as well as it is inflexible to possible other build "profiles". To remedy the situation it was proposed to use field names like Build-Depends[stage1 embedded] but this would also duplicate information and would break with the rfc822 format of package description files. A document maintained by Guillem Jover gives even more ideas and details. Internally, Patrick and me decided for another idea of Guillem Jover to annotate staged build dependencies. The format reads like:

Build-Depends: huge (>= 1.0) [i386 arm] <!embedded !bootstrap>, tiny

So each build profile would follow a dependency in <> "brackets" an have a similar format as architecture options. Patrick has a patch for dpkg that implements this functionality while I patched dose3.

Dropping Build-Depends-Indep and Build-Conflicts-Indep When representing the dependencies of a source package, dose3 concatenates its Build-Depends and Build-Depends-Indep dependency information. So up to now, a source package could only be compiled, if it manages to compile all of its binary packages including architecture:all packages. But when bootstrapping a new architecture, it should be sufficient to only build the architecture dependent packages and therefor to only build the build-arch target in debian/rules and not the build-indep target. Only considering the Build-Depends field and dismissing the Build-Depends-Indep field, reduced the main scc from 1027 vertices to 979 vertices. The amount of cycles up to length four reduced from 276 to 206. Especially the cycles containing gtk-doc-tools, doxygen, debiandoc-sgml and texlive-latex-base got much less. Patrick managed to add a Build-Depends-Indep field to four packages so far which reduced the scc further by 14 vertices down to 965 vertices. So besides staged build dependencies and cross building there is now a third method that can be applied to break dependency cycles: add Build-Depends-Indep information to them or update existing information. I submitted a list of packages that have a binary-indep and/or a build-indep target in their debian/rules to the list. I also submitted a patch for dose3 to be able to specify to ignore Build-Depends-Indep and Build-Conflicts-Indep information.

Dose3 crossbuilding So far I only looked at dependency situations in the native case. While the native case contains a huge scc of about 1000 packages, the dependency situation will be much nicer when cross building. But dose3 was so far not able to simulate cross building of source packages. I wrote a patch that implements this functionality and will allow me to write programs that help analyze the cross-situation as well.

Debconf Presentation Wookey was giving a talk at debconf12 for which I was supplying him with slides. The slides in their final version can be downloaded here

Future Patrick maintains a list of "weak" build dependencies. Those are dependencies that are very likely to be droppable in either a staged build or using Build-Depends-Indep. I must make use of this list to make it easier to find packages that can easily be removed of their dependencies. I will have to implement support for resolving the main scc using staged build dependencies. Since it is unlikely that Patrick will be fast enough in supplying me with modified packages, I will need to create myself a database of dummy packages. Another open task is to allow to analyze the crossbuilding dependency situation. What I'm currently more or less waiting on is the inclusion of my patches into dose3 as well as a decision on the buildprofile format. More people need to discuss about it until it can be included into tools as well as policy. Every maintainer of a package can help making bootstrapping easier by making sure that as many dependencies as possible are part of the Build-Depends-Indep field.

On a lark, I signed up for the pilot program of the Google Cr-48 about a week ago. To my surprise, I found the distinctive box on my doorstep a week later. After initially mistaking it for an overpacked, late-model OLPC XO-1 I would have to repair, I sort of did a double-take as I realized what I had received.
Using Chrome OS


Wow, they weren't kidding about the boot time. Chrome OS responds (to suspend / resume and boot / power-off) very quickly. Responsiveness elsewhere in the OS of course varies on system load and other factors. On the "terms" page, there is a "system security setting" which tells me I have a Trusted Platform Module, and that my TPM has a randomly generated password. The dialog could use some rewording; I'm pretty familiar with the idea of a TPM, but I'm still not sure what I'm to do with that information.

Cloud print is really a problem for me, right now. I use school printers for the majority of my work, which I was previously able to use by adding them by IP address. Unless there's some magic I'm missing here, I can't do that with Chrome OS, and such a feature will not be supported. (except by "connectors" on Windows and Mac computers)
General thoughts
I can't imagine using a system running Chrome OS as a primary computer. The biggest missing feature is PGP (or any sort of encryption) support for email. This is probably not terribly difficult to implement as an extension, but the idea of my cryptographic software being automatically updated is rather unsettling. I think I'll have to agree with Paul Buchheit that ChromeOS will have to merge with Android; there is a utility for local apps, even if they're becoming less and less critical.

Playing around with LLC controls on Sandybridge today. What I wanted to know was how caching backbuffers affects application performance. The reason for this question is that when the driver uses page flipping to display a new frame, we force the new scanout buffer to be uncached because the display hardware only scans out from main memory. However, when we go back to using this buffer as the next back-buffer, we don t turn the caching bits back on as that requires writing GTT entries to change caching modes. This means that swapping via page-flipping and swapping via copying has a large change in main memory access patterns during rendering page flipping applications use an uncached back buffer while copying applications use a cached back buffer. What effect does the back-buffer caching-mode have on application performance? The changes In the kernel, I have three modes 1) Existing code. Set buffer objects to cache mode none when preparing them as scanout buffers. The first time the application flips to a back buffer, caching is disabled on that buffer forever 2) Flip back and forth. Set buffers to cache mode none when used as scanout. When that stops, flip them back to cache mode llc. This involves a whole lot of clflushing and GTT rewriting, obviously. 3) Flush to memory, but don t disable caching. In this mode, the buffer gets flushed (from both CPU and GPU) before being used as a scanout, but the cache mode isn t changed. Benchmarking with glxgears The first test was pretty simple I hacked up glxgears (yes, glxgears is not a benchmark) to redraw the same frame multiple times (including the clear) per swap to provide a scalable amount of work per swap. No pixel operations, so this is just writing to the back buffer. Thes numbers are the median of 5 runs, except for those obviously limited by the refresh rate.

Reps	Existing	Flip	Flush
10	60	60	60
20	60	60	60
40	42.070	34.113	37.130
80	21.259	17.957	18.658

Note the linear scaling over reps (once you get below refresh). I think this means that the swap overhead is in the noise, including flushing and rewriting GTT entries. Given that my screen is (yuck) 1366x768 pixels, or just over 4MB, and my total LLC is 4MB, it s seems like the performance changes here are likely caused by the rendering ejecting every other object from the cache. What s wrong with benchmarking with glxgears? Of course, glxgears is not a benchmark, and I realized while collecting this data that, in particular, it never reads from the backbuffer, and most pixels are only written once by the clear operation at the start of the rendering code. So, it forces everything else out of the cache at the start of each drawing cycle, but doesn t stick anything useful into the cache. Benchmarking nexuiz, a (more) real application For the next test, I ran nexuiz demo1.dem in all three modes. Numbers out of that weren t nearly as stable as glxgears, and I m afraid I didn t write all of them down, but I got numbers between 45 and 50 frames per second in all three modes. None significantly faster, none significantly slower. So, for these two tests, caching has no positive effect on overall rendering performance. Obviously, I need to collect data from more applications to see if the effect is general. I sure hope so, because the alternative will be to find some heuristic to direct when to enable caching. The patch This is on top of kernel version 3.5.

diff --git a/drivers/gpu/drm/i915/i915_drv.h b/drivers/gpu/drm/i915/i915_drv.h
index b0b676a..f69fba8 100644
--- a/drivers/gpu/drm/i915/i915_drv.h
+++ b/drivers/gpu/drm/i915/i915_drv.h
@@ -1335,6 +1335,10 @@ int __must_check
 i915_gem_object_pin_to_display_plane(struct drm_i915_gem_object *obj,
                     u32 alignment,
                     struct intel_ring_buffer *pipelined);
+
+void
+i915_gem_object_unpin_from_display_plane(struct drm_i915_gem_object *obj);
+
 int i915_gem_attach_phys_object(struct drm_device *dev,
                struct drm_i915_gem_object *obj,
                int id,
diff --git a/drivers/gpu/drm/i915/i915_gem.c b/drivers/gpu/drm/i915/i915_gem.c
index 288d7b8..b606bd2 100644
--- a/drivers/gpu/drm/i915/i915_gem.c
+++ b/drivers/gpu/drm/i915/i915_gem.c
@@ -2851,6 +2851,9 @@ int i915_gem_object_set_cache_level(struct drm_i915_gem_object *obj,
    return 0;
  
+#define JUST_FLUSH 0
+#define FLIP_CACHING   0
+
 /*
  * Prepare buffer for display plane (scanout, cursors, etc).
  * Can be called from an uninterruptible phase (modesetting) and allows
@@ -2883,7 +2886,11 @@ i915_gem_object_pin_to_display_plane(struct drm_i915_gem_object *obj,
     * of uncaching, which would allow us to flush all the LLC-cached data
     * with that bit in the PTE to main memory with just one PIPE_CONTROL.
     */
+#if JUST_FLUSH
+   ret = i915_gem_object_finish_gpu(obj);
+#else
    ret = i915_gem_object_set_cache_level(obj, I915_CACHE_NONE);
+#endif
    if (ret)
        return ret;
@@ -2913,6 +2920,16 @@ i915_gem_object_pin_to_display_plane(struct drm_i915_gem_object *obj,
    return 0;
  
+void
+i915_gem_object_unpin_from_display_plane(struct drm_i915_gem_object *obj)
+ 
+   i915_gem_object_unpin(obj);
+#if FLIP_CACHING
+   if (HAS_LLC(obj->base.dev))
+       i915_gem_object_set_cache_level(obj, I915_CACHE_LLC);
+#endif
+ 
+
 int
 i915_gem_object_finish_gpu(struct drm_i915_gem_object *obj)
  
diff --git a/drivers/gpu/drm/i915/intel_display.c b/drivers/gpu/drm/i915/intel_display.c
index a8538ac..76a2012 100644
--- a/drivers/gpu/drm/i915/intel_display.c
+++ b/drivers/gpu/drm/i915/intel_display.c
@@ -1821,7 +1821,7 @@ err_interruptible:
 void intel_unpin_fb_obj(struct drm_i915_gem_object *obj)
  
    i915_gem_object_unpin_fence(obj);
-   i915_gem_object_unpin(obj);
+   i915_gem_object_unpin_from_display_plane(obj);
  
 static int i9xx_update_plane(struct drm_crtc *crtc, struct drm_framebuffer *fb,

Sometimes I have things nicely organized. Power adapters for radios in one drawer, for cameras in the next. And sometimes not so much. Sometimes I m not sure if things are in the basement or the attic. It seems like technology should be able to help solve this problem, but as far as I can tell, no such solution exists yet. So I m planning to build one. Here s my general idea. Feedback, of course, is welcome. Each item to be tracked can have an RFID tag of some sort attached to it. These tags can be read by an RFID reader at a distance of somewhere between 1ft and 1m in typical conditions. The reader could then act as a proximity alert to an object being searched for a you re close beep, for instance. That helps, but is only part of the battle. It doesn t help if you don t even know which room to look in. So the second part of the plan is that the RFID reader is constantly talking to an object database. Besides the obvious association between RFID tag IDs and object descriptions, the database will also capture background reads of RFID tags. Logged with accurate timestamps, we can then conjecture that RFID reads that occurred within a few seconds of each other are probably physically nearby. If boxes have RFID tags on them, then I can probably get a reasonable idea, down to a box or two, of which box that elusive book is in. If I further put tags on certain immovable physical locations in the house, such as every few feet along shelving, then these will also be captured in the background and hopefully associated with objects nearby, giving a good physical idea of where things are. What s more, the simple act of looking for things using the RFID reader would help keep the proximity tables up-to-date, since it could of course log the RFID tags it sees on the way. The only discipline required to keep this info current is to periodically hover nearby storage areas when moving things around. I like this concept a lot. That doesn t mean it s necessarily simple. Implementation requirements

• The cost of tagging an item must be less than $0.25 each. Ideally it would be $0.10 or less. We re talking hundreds or thousands of items here, so even the $2.50 RFID tags for sale on hobbyist sites are way too expensive. RFID tags in industrial bulk quantities are needed, and cheaply.
• Read range must be at least 1ft, and ideally 3ft, and ideally even around as many obstacles as possible. Shorter than that and it can t even take in a whole box without pulling things out.
• Cost kept as reasonable as possible.
• Must be simple and unobtrusive, requiring little manual effort or discipline to maintain current data. This is the reason for RFID instead of barcodes; barcodes require much more specific action (scan the bin, scan the item) rather then just turn on the scanner and wave it around a bit.

Software side I m envisioning the software being split into two components. One would run on an embedded system with the RFID reader. Its job is simple transmitting of scanned RFIDs to the server, receiving instructions from the server, and generating a tone if it s in search mode and the item being searched is nearby. On the server lies a database. The database would contain descriptions for the objects that are tagged (so that the tag ID can be looked up). It would also contain the timestamped scan logs. I envision a simple CGI-based frontend to it that is mobile-friendly, so a laptop, phone, tablet, etc. could be the user interface for the thing saving the cost of a display and input device by reusing what most people already have. This is the part I feel most qualified to work on already. Hardware The first question is what kind of RFID tags to use. Optimizing for cost per tag, the 800/900MHz UHF tags (EPC gen 2) seem ideal. I have found them in costs approaching $0.10 per tag when bought in rolls of 500 or 1000 Avery RFID labels. That s reasonable. The RFID reader is the more complicated part. UHF RFID readers are a lot more costly than their HF or LF counterparts. So far, the cheapest solution I have found started with a post about an Arduino UHF reader. It used a SolidDigi UHF reader board with UART interface for $177. That same board is also available with a USB interface at the same cost. There is also the need for an antenna. There is a small 5dBi one rated 0-50cm for $8, or a $100 8dBi version rated 1-6m designed for wall mounting. They sell these in kit form that include power adapter, USB cables, etc. as well. The kit with the small antenna runs $192, including reader board. Next is the compute platform. A DreamPlug might work for this at $160, though both the cost and the power consumption (5VDC 3A) are high. A Raspberry Pi seems perfect, though for whatever reason seem to be backordered by months everywhere and don t include wifi. The Pi is $35, and the USB wifi is another $30, plus a cheap SD card, so we re around $80 of computing. The running total of the project, then, is at least $272. Add on provisions for batteries, some sort of case, etc. and we re probably past the $300 mark. That s a lot cheaper than the $1000 for handheld RFID readers. The final item is tags. Bulk tags can be found in the $100 to $200 range, making the total cost of the project $400 to $500. Higher than I d like, but providing some valuable experience building something. Risks There are a few major risks to the project. First among them is read distance. If it s not long enough, the usefulness of the project will be low. Fortunately, $100 gets a more high-gain antenna and I d only be out the $8 if the cheaper one doesn t pan out. But that s something of a cold comfort, as it s another, well, $100. Having a bunch of RFID hardware can be used for all sorts of other interesting things, though, so it could perhaps be reused or repurposed. Another risk is that RFID collisions wouldn t be handled as intelligently as I d like, meaning that the read range required to be useful would activate so many tags that collision algorithms break down. I don t know enough about RFID collision algorithms to know if this is a serious councern.

A random tip for shell script hackers. You know that with set -x you can turn on tracing, so that every command is printed before being executed. In bash, you can also customize the output prefix by setting the PS4 variable. The default is PS4='+ '. Here is an example. I wanted to "profile" a deployment script, to see why it took so long. Ordinarily, I might have sprinkled it with date calls. Instead, I merely added

set -x
PS4='+\t '

near the top. \t stands for time stamp. (The script was already using bash explicitly, as opposed to /bin/sh.) That way, every line is prefixed by a time stamp, and the logs could easily be analyzed to find a possible performance bottleneck.

Once again Debian day was somewhat compromised: this time by more DIY. I still managed a few bits and pieces, amortized over the rest of the week. I benefited from some train time this weekend. A new version of flactag and catching up the flactag packaging vcs repository. A maintainer endorsement for one of the flactag maintainers and a request to NM for another maintainer to be granted access to the collab-maint repository. More work on the musicbrainz 'transition': First, usertag tracking of packages to update, with upstream bugs where necessary. Tested and confirmed Michael Biebl's sushi patch; found and fixed a sushi bug along the way; wrote patches for sound-juicer ( merged upstream) and goobox.

In my long trip to Nicaragua I made progress in my reading: Quicksilver, by Neal Stephenson. In the Spanish edition the title is Azogue. But I m assuming that you are not a Spanish speaker. Here is a small fragment (in English) I found there:

You must remember that the planters are short-sighted. They re all desperate to get out of Jamaica they wake up every day expecting to find themselves, or their children, in the grip of some tropical fever. To import female Neegers would cost nearly as much as to import males, but the females cannot produce as much sugar particularly when they are breeding. Daniel had finally recognized this voice as belonging to Sir Richard Apthorp the second A in the CABAL.

It s a bit embarrassing when I discovered myself realizing where the word cabal comes from. And I m posting this as a head-up for everyone who know there is no cabal in Debian; but they don t know which is the origin of the word cabal. Stephenson changed the name of the historic cabal, a group of high councillers of King Charles II of England, Scotland and Ireland, in 1668. In the novel, they are: John Comstock (Earl of Epsom), Louis Anglesey (Duke of Gunfleet), Knott Bolstrood (Count Penistone), Sir Richard Apthorp and Hugh Lewis (Duke of Tweed). In the real world they had been:

Thomas Clifford, 1st Baron Clifford of Chudleigh (1630-1673).	Henry Bennet, 1st Earl of Arlington (1618-1685).	George Villiers, 2nd Duke of Buckingham (1628-1687).	Anthony Ashley Cooper, 1st Baron Ashley of Wimborne St Giles (1621-1683).	John Maitland, 1st Duke of Lauderdale (1616-1682).

This group shared the effective power in a royal council rather than the King.

Not much progress ATM (most translations are complete and D-I has been frozen to prepare its release anyway).

• Arabic completes level 3
• Progress for Uyghur in level 2

Status for D-I level 1 (core D-I files):

• 55 languages 100%: ar ast be bg bn bs ca cs cy da de el eo es et eu fa fi fr gl gu he hi id it ja kk km kn ko lo lv mr nb nl pa pl pt pt_BR ro ru si sk sl sr sv ta te th tr ug uk vi zh_CN zh_TW
• 1 language 98%: ga
• 2 languages 97%: hu lt
• 4 languages 95%: dz hr is mk
• 1 language 92%: ml
• others are 90% or below

Status for D-I level 2 (packages that have localized material that may appear during default installs, such as iso-codes, tasksel, etc.):

• 39 languages 100%: ast be bg ca cs cy da de eo es eu fa fr gl gu he id is it ja kk km lt lv mr nb nl pl pt ru si sk sl sv th tr uk zh_CN zh_TW
• 3 languages 99%: fi hr vi
• 4 languages 98%: dz pt_BR ro ug
• 1 languages 97%: ta
• 2 languages 94%: ar el
• 1 language 92%: sr
• 5 languages 92%: bn bs hu ko ne
• others are 90% or below

Status for D-I level 3 (packages that have localized material that may appear during non-default installs, such as win32-loader)

• 42 languages 100%: ar ast be bg bs ca cs cy da de el eo es fa fi fr ga gl he hr id is it ja kk km lt lv nb nl pl pt ru sk sl sr sv th tr vi zh_CN zh_TW
• 4 languages 98%: hu pt_BR ug uk
• others are 90% or below

Full 100% completeness (hall of fame) for 32 languages: Asturian, Belarusian, Bulgarian, Catalan, Czech, Welsh, Danish, German, Esperanto, Spanish, Persian, French, Galician, Hebrew, Indonesian, Italian, Japanese, Kazakh, Khmer, Latvian, Norwegian Bokm l, Dutch, Polish, Portuguese, Russian, Slovak, Slovenian, Swedish, Thai, Turkish, Simplified Chinese, Traditional Chinese

Sitting in the airport during a layover on my way back from DebConf is one of the few times of year that time and motivation come together and routinely lead me to blog. Each year's DebConf has been special in its own way. This year at DebConf 12, in addition to the usual routine of catching up with all my far-away Debian friends, and of making new Debian friends in a country I've never been before, I got a special treat: my first opportunity to experience the practice of medicine in a foreign country. Last Saturday after arriving in Managua, I discovered that the very pretty paving stones making up the hotel walkway were also very slippery when wet, and the thin rubber strips that had been helpfully put in place to make them less slippery did not measurably succeed in doing so. So as it was raining that afternoon (as it does nearly every afternoon this time of year in Nicaragua), I slipped, and made the critical error of not falling. Instead, I stopped my slide - as it happens, by using the big toe of my left foot as if it were a roller skate brake. I don't recommend this, particularly if, as in my case, you already had it on your todo list to consult a podiatrist after DebConf regarding an ongoing feeling of weakness in precisely that part of precisely that foot. So while everything seemed fine the next day, Monday I woke up early with excruciating pain in my foot which, having forgotten about the incident two days prior, seemed to come from nowhere. After favoring the foot for another day and trying to keep the pain at bay using just OTC drugs, by Tuesday my foot was quite swollen and a source of concern. So I visited the clinic on the university campus, where the doctor diagnosed me with a dropped arch plus cellulitis due to bacteria entering through some invisible cut. Conclusion: university campus clinics are equally useless the world over. But at least in addition to pointlessly prescribing antibiotics, the doctor also prescribed some anti-inflammatory drugs to help bring the swelling down. Nevertheless, my questionable mobility kept both me and Patty from attending the day trip on Wednesday, which was a real shame as I had been keen to climb a volcano that day with my fellow Debian Developers (which in the end didn't happen for anyone, but that's a story for someone else's blog). Instead I took it easy for the day at the hotel, together with a few others who had opted not to attend the day trip, catching up on email and other work and hoping matters would improve. But by the end of the day the swelling was as bad as ever, if not worse, and I had no idea what was going on. Was it broken? Was something dislocated? And so I decided that the following day I would go to the hospital. The nearest hospital, as given in the DebConf local information page, is the Hospital Militar. It's a strange sort of thing as an American who remembers the 1980s to be seeking medical care from a facility operated by the Nicaraguan military; but accompanied by Fitoria, my native escort from the local team I had nothing to worry about; and besides, as a frequent international traveler I long ago learned the secret truth that people are people everywhere, no matter what flag they were born under. The highlight of the hospital visit was definitely when a member of the hospital staff, in full military uniform, shooed me towards a chair instead of standing to talk to her because "every patient has the right to sit, no matter what country they're in". After six x-rays and a doctor's consult, the conclusion was that it's probably not a broken bone and that it will probably clear up in just a couple of days with an even-better anti-inflammatory prescription and taking care of it by not walking on it. And indeed, my foot was already feeling better by Friday and aside from a little tenderness is now just about 100%. There are plenty of other entries on Planet Debian talking about the conference itself, and I don't think I have much to add to what's already been said there. But as for Nicaragua, I can conclude that it must be one of the best countries in the world to break your foot in - between the help of the hotel staff, the top-notch medical professionals, and above all the support of the DebConf local team (with a very big thank you again to Fitoria!), this is one brand of socialized medicine I can appreciate.

A new neighbourhood to explore today: the South-West/West area. Indeed, my original plan was going towards Lagunas de Nejapa and Asososca. However, I now understood that such non urban areas in Managua are almost inaccessible. It sound like walking and hiking in the nature is not something that urban inhabitants of Managua do often. So, in short, these places are....just impossible to go to. Anyway, I wanted to see this, just in case. As a consequence, I ended heading westward from Hotel Seminole through Avenida Miguel Obando Y Bravo (the busy highway close to the hotel, where we boarded the Day Trip buses). Running along this one is not a bad thing as it is not that busy, particularly at 05:45..:-) After crossinb Pista de la Unan that goes north towards the lake, I went 200m east then headed north up to reach the very busy Pista Juan Pablo II (or Pista de la Resistencia). For those who went to the day trip, this is the highway we went through at the beginning of the journey to Leon. It sounds crazy to run along a busy highway but:

• it was not that busy at that time
• this is the only way to not get lost in the barrios as we'll see later on..:-)

I followed that one during about 4km until it ends facing a cone-style mountain, about 150m high (former volcano crater?) After that long run straight, I ended at a big crossing, where the highways go either north or south....but a small street continues westward and is apparently going around the Laguna de Nejapa, at least according to Google Maps. However, there is indeed, as too often here, a barrier (which OSM would have told me, indeed). Hence, disappointed, I only could go back to the busy streets. I however decided that I would NOT come through the same way but rather try to run in smaller streets. This is where the problem lies. Indeed, Managua "barrios" are not always connected by streets to the large and busy streets that surround them. This is mostly because several rivers run south to north, towards the lake. And these are canalyzed to avoid floodings, which means there are not that many bridges to cross them. So, it often ends that a barrio is only connected on one or two of its sides, which makes travelling through them particularly difficult and puzzling (especially going in the east-west directions). So, I happened to search my way many times in these places, particularly in Barrio Pablo Sexto where the rivers directions are really confusing..:-) I finally had no other choice than going down to the big Pista Juan Pablo II and come back the same way to the hotel. The final result is a 15km run in 1h20. Not that a bad pace in these weather conditions. GPS trace

Conrad released version 3.2.4 of Armadillo yesterday. It contains a workaround for g++ 4.7.0 and 4.7.1 which have a regression triggered by the Armadillo codebase for small fixed-sized matrices. The corresponding RcppArmadillo package 0.3.2.4 arrived on CRAN earlier today. The short NEWS entry follows below.

0.3.2.4  2012-07-11
    o   Upgraded to Armadillo release 3.2.4
          * workaround for a regression (bug) in GCC 4.7.0 and 4.7.1

Courtesy of CRANberries, there is also a diffstat report for 0.3.2.4 relative to 0.3.2.3 As always, more detailed information is on the RcppArmadillo page. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page.

A couple of days ago, I had posted a short Python script to convert numpy files into a simple binary format which R can read quickly. Nice, but still needing an extra file. Shortly thereafter, I found Carl Rogers cnpy library which makes reading and writing numpy files from C++ a breeze, and I quickly wrapped this up into a new package RcppCNPy which was released a few days ago. This post will show a quick example, also summarized in the short pdf vignette describing the package, and provided as a demo within the package.

R> library(RcppCNPy)
Loading required package: Rcpp
R> library(rbenchmark)
R> 
R> n <- 1e5
R> k <- 50
R> 
R> M <- matrix(seq(1.0, n*k, by=1.0), n, k)
R> 
R> txtfile <- tempfile(fileext=".txt")
R> write.table(M, file=txtfile)
R> 
R> pyfile <- tempfile(fileext=".py")
R> npySave(pyfile, M)
R> 
R> pygzfile <- tempfile(fileext=".py")
R> npySave(pygzfile, M)
R> system(paste("gzip -9", pygzfile))
R> pygzfile <- paste(pygzfile, ".gz", sep="")
R> 
R> 

We first load the new package (as well as the rbenchmark package used for the benchmarking example) into R. We then create a large matrix of 100,000 rows and 50 columns. Not quite big data by any stretch, but large enough for ascii reading to be painfully slow. We also write two npy files and compress the second one. Next, we use the benchmark function to time the three approaches:

R> res <- benchmark(read.table(txtfile),
+                  npyLoad(pyfile),
+                  npyLoad(pygzfile),
+                  order="relative",
+                  columns=c("test", "replications", "elapsed", "relative"),
+                  replications=10)
R> print(res)
                 test replications elapsed relative
2     npyLoad(pyfile)           10   1.241  1.00000
3   npyLoad(pygzfile)           10   3.098  2.49637
1 read.table(txtfile)           10  96.744 77.95649
R> 

As shown by this example, loading a numpy file directly beats the pants off reading the data from ascii: it is about 78 times faster. Reading a compressed file is somewhat slower as the data stream has to be passed through the uncompressor provide by the zlib library. So instead of reading a binary blob in one go (once the file header has been parsed) we have to operate piecemeal---which is bound to be slower. It does however save in storage space (and users can make this tradeoff between speed and size) and is still orders of magnitude faster than parsing the ascii file. Finally, and not shown here, we unlink the temporary files. Summing up, this post demonstrated how the RcppCNPy package can be a useful to access data in numpy files (which may even be compressed). Data can also be written from R to be accessed later by numpy.