Search Results: "ema"

18 August 2012

David Welton: Simple phone alerts with Gmail and Android

I have some long running server processes that I am going to launch soonish, and I want to be alerted when they're done. Using Gmail and Android, it's pretty easy:

In Gmail, set up a new label, "Alerts" or something like that.
Set up a filter in Gmail that matches messages along the lines of codered@example.com, and adds the Alerts label, does not skip the inbox and are always marked as important.
Now, on your Android phone, in Gmail -> Settings -> Account settings (select the account) -> Sync inboxes and labels, and select your Alerts label.
Then, go back and select Labels to Notify in settings, select "Alerts", and add it to "Notify in Status bar", select an appropriate ringtone, set vibrate to 'always', and deselect 'Notify once' - because we want to be notified any time an alert comes in.

That should do it - on your server, set up an alias in something like /etc/aliases that redirects email from the alert alias - codered in this case - to your own email address. Now, you can script alerts that will get their own ringtone on your phone like so: echo "Dave, I can't let you do that" mail -s "Warning, computer malfunction" codered@example.com Pretty simple and effective if you need a simple way for your computer to let you know that it needs attention right now.

17 August 2012

Gunnar Wolf: More on hacker conferences and sexual harassment

At the beginning of this year, I blogged about a Mexican security-minded hacker conference scaring away its female audience by advertising in a sexist way. I don't know if it is the need to be l33t or kewl, to show off that hackers are really socially inept, or what... but this seems to carry on. I know many are familiar with the red/yellow card project (and followup) by kdotcdot. I am (rightfully? naively?) proud that at DebConf we have achieved a clean conference environment, without such problems... Yes, I know that, during the last ~year at DebConf11 we discussed an anti-harassment policy (look at the thread, it was quite interesting!), came up with standards of respect And Debian as a whole voted on a GR that ratifies a diversity statement. The fact that we had those very positive discussions, documents and events shows we needed to have them. But, again, this shows that being a hacker does not necessarily mean being a jerk. And I'm very proud to be part of this community. I recently stumbled across a very nice, insightful post by Valerie Aurora on The Ada Initiative: Supporting women in open technology and culture DEFCON: Why conference harassment matters. Take a good read at it. I hope it helps shape other hacker groups in a less-aggressive, more welcoming way. Oh! And before closing: Be sure to at least skim through both Valerie Aurora's and kdotcdot's comments. LOTS of insight in them.

16 August 2012

Matthew Garrett: Secure Boot is almost certainly not causing you problems yet

There's been a few links to this story of someone buying a system that turned out to have UEFI firmware and also turned out not to boot. Given all the press, it's unsurprising that people would assume that problems they have with UEFI booting are related to Secure Boot, but it's very unlikely that this is the actual problem here. First, nobody's shipping an appropriately signed operating system yet. A hardware vendor that enabled secure boot out of the box would be selling a machine that wouldn't boot any OS you could buy. That's a poor way to make money. Second, the system booted a Fedora 17 CD. Fedora 17 isn't signed, so if the firmware booted it then the firmware isn't enforcing Secure Boot. Third, it didn't boot the installed OS. That's really at the point of it sounding like a hardware problem - selling systems that don't run the OS you sold them with is a guaranteed way of getting enough support calls that you wouldn't make any money on them, ever.

To be fair, Linux compatibility with UEFI systems is still not as good as it is with BIOS systems. Fedora 18 will be using a new UEFI boot process and so far in our testing it's been significantly more reliable than Fedora 17. There's still some remaining issues that we're aware of and working on, but right now it's hugely more likely that failures to boot Fedora 17 on UEFI systems are down to our bugs rather than Secure Boot.

comments

Gunnar Wolf: 19 years of Debian

Happy 19th birthday, Debian!

The Debian project is 19 years old now. Following Francesca's invitation (and Rapha l's lead, and using Leandro's image, collaboratively as it always happens here), I will tell a bit of my memories: How I got to Debian. I am a Debian user since ~2000, and a Debian Developer since April 2003. But, just as Rapha l's, my history must go somewhat further back in time. In 1992, I got my first 1200bps modem, and almost immediately became an avid BBS user (what's that? Javier Matuk talks [in Spanish] about BBSes in his newspaper column back in 1994). By mid 1993 I started operating my own BBS, CatarSYS. One of the key points that defined CatarSYS is that my focus was large-scale communication. I started connecting to several BBS networks, allowing messages to be relayed to distant people, mainly in the USA and Spain, and getting some feeds that could be seen as the poor man's Usenet. At some point during the year I kept CatarSYS going, I got connected through a strange set of gateways to pure gold: A UUCP feed! This means, during some months, I operated the first (free, hobbyist) service that offered its users a free Internet mail address in Mexico. Yes, it was completely different to what we are used to today. I tried to connect to my provider at least three times a week, but this meant less than one week turnover time for messages sent to people anywhere in the world! But, back to Debian: Via this UUCP feed, I also got some real Usenet newsgroups Including several on the comp.os category. I remember reading about Linux back then, and learning some commands, but didn't really get hooked into it. I was in Israel from July 1994 to June 1995, doing completely non-computer-related tasks. Came back in 1995, and due to my father being an academic, got dialup internet access at home. WOW, *real* Internet! It didn't take me long to start downloading Linux information and floppy images. One failed after another. But before the end of the year, I found in a bookshop a book (that included a CD-ROM) called "Build your own web server with Linux". WOW again Remember this was still 1995! I bought it, and shortly afterwards, I had a Slackware system (Linux kernel 1.0.9) running. That meant many sleepless nights full of joy and frustration (as getting hardware to work was cloe to impossible). By 1996, I got (within one week) my two first real jobs: A systems administrator at a small ISP and a highschool teacher. At the ISP, I got a spare computer to play with Linux, as of course the Big Server was running with Windows NT 3.51. Poor little machine... By then I was already a part of the Mexican Linux User Group. This group had just printed a batch of Linux CDs RedHat 4.0. This was the first release that really made me happy and allowed me to do good work. Together with a friend I took to work with us, Juan Pablo Romero, we installed over a weekend a full replacement for our buggy NT machine, in much cheaper hardware. Of course, Linux was nowhere near corporate-recognition, and our project remained a project, not touching the Windows machine. Anyway... Several years passed, and I was happy with my RedHat choice. I won't mention the milestones and job changes, as it would get boring and leave the point completely aside. By the year 2000, I was quite more involved with the LUG, as well as with the computer security group in DGSCA-UNAM. I became also an OpenBSD user, and had got so hooked up in free software that I felt the need to collaborate: To be a little part of one of those Big Projects that had given me so much. But which one? I have never been much of a programmer Yes, I can solve my everyday needs and have fun with it, and sometimes a bit beyond that. I enjoy programming. But all of my projects have begun little and stayed little. I wanted to join OpenBSD, as it was a community I really believed in, but given my skillset (and given a flame-prone, aggressive developer community), I lost motivation to do so. By 2000, I had also lost faith in RedHat. I don't have the exact dates, so I might be some months off But after RedHat's IPO, I felt a sharp change. Version 7.0 was really demotivating It tried to offer a polished desktop experience, but was really buggy, unstable, and full of bad decisions. In Mexico, Pepe Neif had taken up the job of making a derivative distribution of RedHat (called LinuxPPP), pressing hundreds of CDs and making a teaching program I was part of several times. Talking with Pepe (who continued to release based on RedHat 6), he told me he was interested in switching over to become a Debian-based distribution, but the job of migrating his installed base made the project stall LinuxPPP reached only version 6.4. But I installed Debian in early 2000, and loved it. I started getting familiar with its social philosophy and foundation documents at the same time I started migrating my servers from RedHat to Debian This must have been by Spring 2000, as I installed Potato while it was frozen but not yet stable. By January 2002 I applied for NM. My process took a long time, as my AM got MIA when he had already approved me (but before sending the AM report), so basically I had to go through AM twice And by April 16, 2003, I got accepted as a DD. Contrary to what is acceptable today, I requested the full process to be done before starting to maintain any packages, as I didn't want to bother people with package sponsorship requests, so my whole process was done evaluating packages I would eventually upload. Since becoming a DD, my main involvement in the project has been in packaging groups (I was a pkg-perl founder and member for many years, and am currently working in the pkg-ruby-extras group). But, as I said, my main strength is not programming So my main involvement in Debian has been more social than technical: I have been a DebConf organizer since 2005, a very interesting, stressing, rewarding and (for some months) time-demanding role, and since 2009 I am part of the keyring maintainence team, which is much easier workload, although carries important ramifications. So, after 19 years of Debian, and after nine years of me being part of it, Debian is clearly my strongest link to the Free Software community, a project I have grown to love and whose way of being I share and enjoy studying and explaining. And it is a technically excellent product, and a great place to start and keep learning both about how every aspect and layer of an operating system works, and how human-to-human interaction works in such a diverse, almost impossible environment happens. Congratulations, Debian!

Attachment	Size
debian-19_500.png	149.81 KB

Matthew Garrett: Building personal trust with UEFI Secure Boot

The biggest objection that most people have to UEFI Secure Boot as embodied in the Windows 8 certification requirements is the position of Microsoft as the root of trust. But we can go further than that - putting any third party in a position of trust means that there's the risk that your machine will end up trusting code that you don't want it to trust. Can we avoid that?

It turns out that the answer is yes, although perhaps a little more complicated than ideal. The Windows 8 certification requirements insist that (on x86, at least) the key databases be completely modifiable. That means you can delete all the keys provided by your manufacturer, including Microsoft's. However, as far as the spec is concerned, a system without keys is in what's called "Setup Mode", and in this state it'll boot anything - even if it doesn't have a signature.

So, we need to populate the key database. This isn't terribly difficult. James Bottomley has a suite of tools here, including support for generating keys and building an EFI binary that will enrol them into the key databases. So, now you have a bunch of keys and the public half of them is in your platform firmware. What next?

If you've got a system with plug-in graphics hardware, what happens next is that your system no longer has any graphics. The firmware-level drivers for any plug-in hardware also need to be signed, and won't run otherwise. That means no graphics in the firmware. If you're netbooting off a plug-in network card, or booting off a plug-in storage controller, you're going to have similar problems. This is the most awkward part of the entire process. The drivers are all signed with a Microsoft key, so you can't just enrol that without trusting everything else Microsoft have signed. There is a way around this, though. The typical way to use Secure Boot is to provide a list of trusted keys, but you can also provide trusted hashes. Doing this involves reading the device ROM, generating a SHA256 hash of it and then putting that hash in the key database.

This is, obviously, not very practical to do by hand. We intend to provide support for this in Fedora by providing a tool that gets run while the system is in setup mode, reads the ROMs, hashes them and enrols the hashes. We'll probably integrate that into the general key installation tool to make it a one-step procedure. The only remaining problem is that swapping your hardware or updating the firmware will take it back to a broken state, and sadly there's no good answer for that at the moment.

Once you've got a full set of enrolled keys and the hashes of any option ROMs, the only thing to do is to sign your bootloader and kernel. Peter Jones has written a tool to do that, and it's available here. It uses nss and so has support for using any signing hardware that nss supports, which means you can put your private key on a smartcard and sign things using that.

At that point, assuming your machine implements the spec properly everything that it boots will be code that you've explicitly trusted. But how do you know that your firmware is following the spec and hasn't been backdoored? That's tricky. There's a complete open implementation of UEFI here, but it doesn't include the platform setup code that you'd need to run it on any x86 hardware. In theory it'd be possible to run it on top of Coreboot, but right now that's not implemented. There is a complete port to the Beagleboard hardware, and there's instructions for using it here. The remaining issue is that you need a mechanism for securing access to the system flash, since if you can make arbitrary writes to it an attacker can just modify the key store. On x86 this is managed by the flash controller being switched into a mode where all writes have to be carried out from System Management Mode, and the code that runs there verifies that writes are correctly authenticated. I've no idea how you'd implement that on ARM.

There's still some work to be done in order to permit users to verify the entire stack, but Secure Boot does make it possible for the user to have much greater control over what their system runs. The freedom to make decisions about not only what your computer will run but also what it won't is an important one, and we're doing what we can to make sure that users have that freedom.

comments

15 August 2012

Gintautas Miliauskas: Why Software Engineering Is Not Politics

A recent post by Steve Yegge has made quite a splash. The main thesis of his post is:

Software engineering has its own political axis, ranging from conservative to liberal.

Software engineering, like any other social enterprise, does have a political aspect, but I do not think this thesis is apt. The essential property of politics is that you can not have it both ways. A political problem is a coordination problem, and political decisions by its nature apply for an entire political body. Software engineering as a whole does not have these properties. There is absolutely no reason why any two engineering projects have to follow the same ideals and guidelines. Politics in engineering only appears within boundaries of a project (or, at a slightly higher level, a company), at which point it does not really warrant the name of politics, but rather character traits or personal preferences. If Johnny goes for the puck more often than Louie, that does not really warrant a label of a "liberal".

The fundamental quality that Steve is looking at is actually risk aversiveness, as he noted himself. He should have stopped there. Making that next step of conceptualization in political language does not bring any new insights and just muddles the matter.

It seems that the political labels of "liberal" and "conservative" were picked not for their meaning, but rather for their connotations, as political views are perceived to be 1) very stable, part of a person's self-identity and (therefore) hard to change, 2) inherently polarizing, and 3) difficult or impossible to evaluate objectively. Again, these do not apply to software engineering. For example, probably like many engineers, I am clearly a liberal on personal and small projects, and a conservative on larger and more fundamental ones; so much for the idea of a core identity. As for polarization, it may be prevalent in internet echo chambers, but I have not seen much of that in my professional life, especially when working with experienced engineers. When they do have objections, those objections are grounded in engineering. This leads to the third point, claiming that the approaches can not be compared objectively. This one is the most pernicious of all, because it provides an escape hatch from arguments with substance on what is an engineering decision.

Here is my, admittedly less catchy and impressive, counter thesis: Conservativeness of a software engineering project is an objective decision to be based on circumstances. TL;DR: It depends. (Very original, isn't it?) It is a risk management decision, and those are very tricky, but not subjective.

One thing to note is that the architecture of the system should correspond to the implementation style. Schemaless, heavy on magic, untyped approaches are suited for modular systems where each individual module is of limited complexity, and where the modules are isolated from each other and can recover from failures. The more complex the individual modules (and sometimes the complexity is inherent to the problem), the more benefits from "conservative" techniques. (By the way, this lines up nicely with Steve being liberal and a proponent of service-oriented architectures). If you have the freedom to pick between these two approaches, the decision making does get a bit more ideological. However, neither the "liberal" and "conservative" labels, nor risk averseness by itself is enough for a decision here. Say, if you are developing a service, what is more risky: having a "traditional", monolithic service that is inevitably sensitive to failures and is likely to scale poorly, or a cluster of components that are generally more resistant, but have more complicated aggregate behavior?

Despite the criticisms, I have to give fat credit to Steve. His post was thought-provoking, and it brought attention to the subjective side of engineering, one which is often overlooked. That side should not be left to grow - on the contrary - but the only way to reduce its domain it is to be aware of it. I know I will think twice before making another gut decision where risk averseness plays an important factor.

14 August 2012

Michal Čihař: Weblate 1.2

Quite on schedule, Weblate 1.2 has been released today. It comes with lot of improvements, especially for project admins, handling of po files and improved performance. Full list of changes for 1.2:

Weblate now uses South for database migration, please check upgrade instructions if you are upgrading.
Fixed minor issues with linked git repos.
New introduction page for engaging people with translating using Weblate.
Added widgets which can be used for promoting translation projects.
Added option to reset repository to origin (for privileged users).
Project or subproject can now be locked for translations.
Possibility to disable some translations.
Configurable options for adding new translations.
Configuration of git commits per project.
Simple antispam protection.
Better layout of main page.
Support for automatically pushing changes on every commit.
Support for email notifications of translators.
List only used languages in preferences.
Improved handling of not known languages when importing project.
Support for locking translation by translator.
Optionally maintain Language-Team header in po file.
Include some statistics in about page.
Supports (and requires) django-registration 0.8.
Caching of counted units with failing checks.
Checking of requirements during setup.
Documentation improvements.

You can find more information about Weblate on it's website, the code is hosted on Github. If you are curious how it looks, you can try it out on demo server. You can login there with demo account using demo password or register your own user. Ready to run appliances can be found in SUSE Studio Gallery. Weblate is also being used https://l10n.cihar.com/ as official translating service for phpMyAdmin, Gammu, Weblate itself and others. If you are free software project which would like to use Weblate, I'm happy to help you with set up or even host Weblate for you (this will be decided case by case as my hosting space is limited). Update: Weblate appliace has been updated to 1.2 as well.

Filed under: English Weblate 0 comments Flattr this!

Matthew Garrett: Playing with Thunderbolt under Linux on Apple hardware

I've temporarily got hold of a Retina Macbook Pro. Fedora 17 boots fine on it with the following kernel arguments: nointremap drm_kms_helper.poll=0 video=eDP-1:2880x1800@45e, although you'll want an updated install image. The nointremap requirement is fixed by this patch, and the others are being tracked here. The gmux chip that controls the multiple graphics chipsets has changed - patches here. That just leaves Thunderbolt.

Thunderbolt is, to a first approximation, PCIe tunnelled over a Displayport connector. It's a high-bandwidth protocol for connecting external devices. Things are complicated by it also supporting Displayport over the same connection, which I'm sure will be hilarious when we get yet another incompatible display protocol. But anyway. I picked up a Thunderbolt ethernet adapter and started poking.

Booting with the device connected looked promising - an ethernet device appeared. Pulling it out resulted in the pcie hotplug driver noticing that it was missing and cleaning up, although the b44 ethernet driver sat around looking puzzled for a while. So far so good. Unfortunately plugging it in again didn't work, and rebooting without any Thunderbolt devices connected not only did nothing on hotplug, it also meant that the Thunderbolt controller didn't show up in lspci at all.

It turns out that there's several problems at play here. The first is the missing Thunderbolt controller. The problem here is the following code in Apple's ACPI tables:

        Method (RMCR, 0, Serialized)
         
            If (LNot (OSDW))
             
                If (LAnd (LEqual (\_SB.PCI0.PEG1.UPSB.DSB1.UPS0.AVND, 0xFFFF), 
                    LEqual (\_SB.PCI0.PEG1.UPSB.DSB2.UPS0.AVND, 0xFFFF)))
                 
                    Store ("RMCR: Disable Link and Power Off Cactus Ridge Chip",
                           Debug)
                    Store (0x01, \_SB.PCI0.PEG1.LDIS)
                    Sleep (0x07D0)
                    Store (0x00, GP23)

This checks if the system claims to be Darwin (the core of OS X). If not, it checks whether two PCIe bridges have been configured. If both are still in an unconfigured state, it cuts the PCIe link to the upstream PCIe link and then sets GP23 to 0. Looking further, GP23 turns out to be a GPIO line. Setting it to 0 appears to cut power to the Thunderbolt controller. In summary, unless your OS claims to be OS X, booting without an attached Thunderbolt device will result in the chipset being powered down so hard that it's entirely invisible to the OS.

Fixing that is as easy as adding Darwin to the list of operating systems Linux claims to be. So, now I could boot without a connected device and still see the controller. Since ACPI seemed to be important here, I started looking at the other ACPI methods associated with the device. The first one that drew attention was Name(_GPE, 0x14). The _GPE method (not to be confused with the _GPE object at the top of the global ACPI namespace) is defined as providing the ACPI general purpose event associated with the device. Unfortunately, it's only defined as doing this for embedded controllers - Apple's use of it on a PCI device is massively non-standard. But, still, easy enough to handle. Intel chipsets map GPE 0x10 to 0x1f to GPIO lines 0-15, so this GPE is associated with GPIO 4. Looking through the ACPI code showed a bunch of other references to GP04, and it turns out that if the chip is powered down (via setting GP23 to 0) then plugging a device in to the port will generate a GPE. This makes sense - a powered down controller isn't going to notice hotplug events itself, so you need some kind of out of band notification mechanism. There's also another ACPI method that flips the polarity of the GPIO line so it doesn't keep generating events for the entire time a device is connected. It didn't take long to come up with a stub driver that would power down the controller if nothing was connected at boot, and then wake it up again on a hotplug event.

Unfortunately that's as far as I've got. I'd been hoping that everything beyond this was just PCIe hotplug, but it seems not. Apple's Thunderbolt driver is rather too large to just be handling ACPI events, and this document on Apple's website makes it pretty clear that there's OS involvement in events and device configuration. Booting with a device connected means that the firmware does the setup, but if you want to support hotplug then the OS needs to know how to do that - and Linux doesn't.

Getting this far involved rather a lot of irritation at Apple for conspiring to do things in a range of non-standard ways, but it turns out that the real villains of the piece are Intel. The Thunderbolt controller in the Apples is an Intel part - the 82524EF, according to Apple. Given Intel's enthusiasm for Linux and their generally high levels of engagement with the Linux development community, it's disappointing[1] to discover that this controller has been shipping for over a year with (a) no Linux driver and (b) no documentation that would let anyone write such a driver. It's not even mentioned on Intel's website. So, thanks Intel. You're awful.

Anyway. This was my attempt to spend a few days doing something more relaxing than secure boot, and all I ended up with was eczema and liver pain. Lesson learned, hardware vendors hate you even more than firmware vendors do.

[1] By which I mean grotesquely infuriating

comments

13 August 2012

Robert Collins: minimising downtime for schema changes with postgresql

Two years ago Launchpad did schema changes once a month. Everyone would cross their fingers and hope while the system administrators took all the application servers offline, patched the database with a months worth of work and brought up the servers again running the new QA d codebase. This had two problems:

due to the complexity of the system something like 300 processes have to be stopped or inhibited to take everything offline the downtime duration was often about 90 minutes long irrespective of the schema patch duration. [Some of the processes don't like being interrupted at all].
We simply could not deliver any change in less than 1 week, with the on average latency for something that jumped all the queues still being 2 weeks.

About a year ago we wanted to increase the rate at which schema changes could be carried out the efforts to speed Launchpad up had consumed most low hanging fruit and more and more schema patches were required. We didn t want to introduce additional 90 minute downtime windows though. Adopting incremental migrations the sort of change process described in various places on the internet seemed like a good way to make it possible to apply the schema changes without this slow shutdown-and-restart step, which was required because the pre-patch codebase couldn t speak to the new schema. We could optimise each patch to be very fast by avoiding anything that causes a full table scan or table rewrite (such as adding indices, adding columns with a non-NULL default value). That would let us avoid the 90 minutes of downtime caused by stopping and restarting everything. However, that wasn t sufficient the reason Launchpad ended up doing monthly downtime is that previous attempts to do more frequent schema changes had too high a failure rate. A key reason for patch deployment time blowing out when everything wasn t shut down was due to Launchpad being a very busy system with the use of Slony, schema changes require an exclusive lock on all tables. [More recent versions of Slony only lock some tables, but it still requires very widespread locks for most DDL operations]. We re doing nearly 10 thousand transactions per minute, at any point in time there are always locks open on some table in the system: it was highly improbably and effectively impossible for slonik to get an exclusive lock on all tables in a reasonable timeframe. Background tasks that take many minutes to complete exacerbate this we can t just block new transactions long enough to deliver all the in-flight web pages and let locks clear that way. PGBouncer turns out to be an ideal tool here. If you route all your connections through PGBouncer, you have a single point you can deliberately interrupt to clear all database locks in a second or so (it takes time for backends to all notice that their clients have gone). So we combined these things to get what we called Fast Down Time or FDT. We set the following rules for developers:

Any schema patch had to complete in <= 15 seconds in our schema staging environment (which has a full copy of the production DB), or we d roll it back and redesign.
Any patch could change either code or schema, never both. schema patches were to land on a separate branch and would be promoted to trunk only after deployment. That branch also receives automated merges from trunk after every commit to trunk, so its running the latest code.

This meant that we could be confident in QA: we would QA the new schema and the application process with the current live code (we deploy trunk multiple times a day). We published some documentation about how to write fast schema patches to help socialise the approach. Then we wrote an automated tool that would:

Check for known fragile processes and abort if any were found.
Check for very long transactions and abort if any were found.
Shutdown pgbouncer, disconnecting all clients instantly.
Use slonik to apply one or more schema patches.
Start pgbouncer back up again.

The code for this (call it FDTv1) is in the Launchpad source code history its pretty entangled but its there for grabbing if you need it. Read on to see why its only available in the history

The result was wonderful we immediately were able to deploy schema changes with <= 90 seconds of downtime, which was significantly less than the 5 minutes our stakeholders had agreed to as a benchmark if we were under 5 minutes, we could schedule downtime once a day rather than once a month. We had to fix some API client code to retry more reliably, and likewise fix a few minor bugs in the database connection handling logic in the appservers, but all in all it was a pretty smooth project. Along the way we spun off a small python helper to run and control pgbouncer, which let us write effective tests for the connection handling code paths. In This gave us the following workflow for making schema changes:

Land and deploy an incremental schema change.
Land and deploy any indices that need to be added these are deployed live using CREATE INDEX CONCURRENTLY.
Land and deploy code changes to populate any additional fields/tables from both application servers, and from cron we do a bulk backfill that does many small transactions while walking over the entire dataset that needs to be updated / populated.
Land and deploy code changes to drop references to the old schema, whatever it was.
Land and deploy an incremental schema change to finalise the change such as making a new column NOT NULL once the backfill is complete.

This looks long and unwieldy but its worth noting that its actually just repeated applications of a smaller primitive:

Make a schema change that is fast and compatible with existing code.
Change code to take advantage of the changed schema

Pretty much any change that is desired can be done using this single primitive. We wanted to go further though the multiple stages required for complex migrations became a burden with one change a day. Fortunately PostgreSQL now includes its own replication engine, which replicates the WAL logs rather than installing triggers on all tables like Slony. Stuart, our intrepid DBA migrated Launchpad to PostreSQL 9.1, updated the FDT tool to work with native replication, and migrated Launchpad off of Slony. The result is again wonderful the overhead in doing a schema patch, with all the protection I described above, is now ~5 seconds. We can do incremental changes in less time than it takes your browser to figure out that a given server is offline. We re now negotiating with the Launchpad stakeholders to get multiple downtime windows each day, with this almost unnoticable, super reliable process in place. Reliability wise, FDT has been superb. We ve had 2 failures: one where we believe we encountered a bug in Slony when dropping two tables at once, and one where we landed a patch that worked on staging but led to lock contention in production so the patch applied, but the system was very unhealthy after that until we fixed it. Thats after doing approximately 60 patches over a 1 year period. We re partway through extracting the patching logic from Launchpad s code base into a reusable tool, but the basic principles will apply to any PostgreSQL environment.

11 August 2012

Julian Andres Klode: Implicit preferences in OR dependencies

Debian packages commonly use or dependencies of the form a b to mean that a or b should be installed, while preferring option a over b. In general, for resolving an or dependency, we will try all options from the left to the right, preferring the left-most option. We also prefer real packages over virtual ones. If one of the alternatives is already installed we use that.

def solve_or(or):
  best_real = None
  best_virtual = None
  for dep in or:
     for target in dep:
        if target.name == dep.name and best_real is None:
           best_real = target
        if target.name != dep.name and best_virtual is None:
           best_virtual = target        
        if target.is_installed():
          return target
  return best_real if best_real is not None else best_virtual

Now, this way of solving dependencies is slightly problematic. Let us consider a package that depends on: a b, b. APT will likely choose to install a to satisfy the first dependency and b to satisfy the second. I currently have draft code around for a future version of APT that will cause it to later on revert unneeded changes, which means that APT will then only install b . This result closely matches the CUDF solvers and cupt s solver. On the topic of solving algorithms, we also have the problem that optimizing solvers like the ones used with apt-cudf do not respect the order of dependencies, rather choosing to minimise the number of packages installed. This causes such a solver to often do stuff like selecting an sqlite database as backend for some service rather then a larger SQL server, as that installs fewer packages. To make such solvers aware of the implicit preferences, we can introduce a new type of dependency category: Weak conflicts, also known as Recommends-Not. If a package P defines a Recommends-Not dependency against a package Q, then this means that Q should not be installed if P is installed. Now, if we have a dependency like: Depends: a b c we can encode this as: Recommends-Not: c, c, b Causing the solver to prefer a, then b, and then c. This should be representable as a pseudo-boolean optimization problem, as is common for the dependency problem, although I have not looked at that yet it should work by taking the standard representation of conflicts, adding a relaxation variable and then minimising [or maximising] the number of relaxation variables.
Filed under: APT2, Debian, General

Russ Allbery: Review: Design Patterns

Review: Design Patterns, by Erich Gamma, et al.

Author:	Erich Gamma
Author:	Richard Helm
Author:	Ralph Johnson
Author:	John Vlissides
Publisher:	Addison-Wesley
Copyright:	1995
Printing:	September 1999
ISBN:	0-201-63361-2
Format:	Hardcover
Pages:	374

Design Patterns: Elements of Reusable Object-Oriented Software by the so-called "Gang of Four" (Gamma, Helm, Johnson, and Vlissides) is one of the best-known books ever written about software design, and one of the most widely cited. The language introduced here, including the names of specific design patterns, is still in widespread use in the software field, particularly with object-oriented languages. I've had a copy for years, on the grounds that it's one of those books one should have a copy of, but only recently got around to reading it. The goal of this book is to identify patterns of design that are widely used, and widely useful, for designing object-oriented software. It's specific to the object-oriented model; while some of the patterns could be repurposed for writing OO-style programs in non-OO languages, they are about inheritance, encapsulation, and data hiding and make deep use the facilities of object-oriented design. The patterns are very general, aiming for a description that's more general than any specific domain. They're also high-level, describing techniques and methods for constructing a software system, not algorithms. You couldn't encapsulate the ideas here in a library and just use them; they're ideas about program structure that could be applied to any program with the relevant problem. With the benefit of seventeen years of hindsight, I think the primary impact of this book has been on communication within the field. The ideas in here are not new to this book. Every pattern in Design Patterns was already in use in the industry before it was published; the goal was taxonomy, not innovation. One would not come to Design Patterns to learn how to program, although most introductory texts on object-oriented programming now borrow much of the pattern terminology. Rather, Design Patterns is as influential as it is because it introduced a shared terminology and a rigor around that terminology, allowing writers and programmers to put a name to specific program structures and thus talk about them more clearly. This also allows one to take a step back and see a particular structure in multiple programs, compare and contrast how it's used, and draw some general conclusions about where it would be useful. I have the feeling that the authors originally hoped the book would serve as a toolbox, but I think it's instead become more of a dictionary. The pattern names standardized here are widely used even by people who have never read this book, but I doubt many people regularly turn to this book for ideas for how to structure programs. Design Patterns is divided into two parts: a general introduction to and definition of a software pattern followed by a case study, and then a catalog of patterns. The catalog is divided into creational patterns (patterns for creating objects), structural patterns (patterns for composing objects into larger structures), and behavioral patterns (patterns for interactions between objects). Each pattern in turn follows a very rigid presentation structure consisting of the pattern name and high-level classification, its basic intent, other common names, a scenario that motivates the pattern, comments on the applicability of the pattern, the structure and classes or objects that make up the pattern, how those participants collaborate, how the pattern achieves its goals, comments on implementation issues, sample code, known uses of the pattern in real-world software, and related patterns. As with a dictionary, the authors go to great lengths to keep the structure, terminology, and graphical representations uniform throughout, and the cross-referencing is comprehensive (to the point of mild redundancy). As for the patterns themselves, their success, both as terminology and as useful design elements, varies. Some have become part of the core lexicon of object-oriented programming (Factory Method, Builder, Singleton), sometimes to the point of becoming syntactic elements in modern OO languages (Iterator). These are terms that working programmers use daily. Others aren't quite as widespread, but are immediately recognizable as part of the core toolkit of object-oriented programming (Adapter, Decorator, Proxy, Observer, Strategy, Template Method). In some cases, the technique remains widespread, but the name hasn't caught on (Command, for example, which will be immediately familiar but which I rarely hear called by that name outside of specific uses inside UI toolkits due to ambiguity of terms). Other patterns are abstract enough that it felt like a bit of a reach to assign a name to them (Bridge, Composite, Facade), and I don't think use of those names is common, but the entries are still useful for definitional clarity and for comparing similar approaches with different implications. Only one pattern (Interpreter) struck me as insufficiently generic to warrant recording in a catalog of this type. So far, so good, but the obvious question arises: if you've not already read this book, should you read it? I think the answer to that is debatable. The largest problem with Design Patterns is that it's old. It came late enough in the development of object-oriented programming that it does capture much of the foundation, but OO design has continued to change and grow, and some patterns have either been developed subsequently or have become far more important. For example, Model-View-Controller is conspicuous by its absence, mentioned only in passing in the discussion of the Observer pattern. Any pattern catalog written today would have an extensive discussion. Similarly absent are Inversion of Control and Data Access Object, which are much more central to the day-to-day world of the modern programmer than, say, Memento or Visitor. One could easily go on: Lazy Initialization, Mock Object, Null Object... everyone will have their own list. A related problem is that all the implementation examples are shown in either C++ or Smalltalk (occasionally both). Those were probably the best languages to use at the time, but it's doubtful a modern rewrite would choose them. Smalltalk, in particular, I found nearly incomprehensible for the uninitiated, to the point where I ignored the code and only read the surrounding English description. C++ fares better, but Design Patterns occasionally drifts off into tedious discussions of how to work around C++'s limitations in ways that are irrelevant to the pattern itself and would not be necessary in, say, Java or Python. (This is ameliorated by the fact that C++, unlike Smalltalk, is still in widespread use, so those discussions remain moderately helpful for some readers.) Design Patterns is not, therefore, a very good source for a working knowledge of the most common patterns in use today. It has also become somewhat obsolete via its own success: the concept of a design pattern has become so popular that nearly all introductory texts include at least a basic discussion of design patterns and an introduction to the most notable and useful patterns. I think that's a more comfortable and more efficient way to pick up the basics than reading through this book, which is somewhat dense and which expects from the reader a reasonably good working knowledge of object-oriented programming. And, once you have the basics, MVC, DAO, and similar design patterns are probably more important than the more subtle design patterns presented here. That said, I think the rigor of description and the comparisons and discussions here still have some value. Design Patterns encourages the reader to look at patterns from a higher-level perspective, to think about meta-patterns such as the balance between data hiding and access, or between structure designed for the present purpose and structure that's adaptable to future needs. It's also mildly interesting from a historical standpoint; one can see the inspiration for future language designers in how problems are described here, and see how many of the implementation issues and negative consequences have been corrected or simplified by richer language designs. Overall, I would hesitate to recommend buying this book today, particularly at new textbook prices. But if you're a working object-oriented software designer or programmer, I think it's worth checking out from a library (and, thanks to its influence, any library with a decent software design section will almost certainly have a copy). Read the overall discussion, skim the catalog, and read the discussion of the patterns that strike your interest. It may help provide some additional structure and perspective to how you think about OO design. Rating: 6 out of 10

10 August 2012

Craig Small: Australia Data Retention Scheme Stalled

It s been reported in the mainstream media that the Australian Data Retention Scheme has been referred to another committee, effectively delaying it to after the next election. The Scheme was part of a broad review of the various security agencies powers and how to streamline them. The initial paper originally stated that retention would be for up to 2 years for parts of the data set, without really specifying what the data was. There has been various ideas of what this data is, depending who you ask and when. Some documents state it is only the accounting data; that user X used IP address Y at time Z, while others state it is email logs and a third have been the previous two plus web logs. There are many problems with a scheme such as this. 2 years of everyones web-browsing history is a desirable target for both legitimate and illegitimate access to that data. Leaving aside would the access to the agencies get the right level of access; who else would want this data? I m sure that AFACT (Australia s MPAA effectively) would like this data to go trawling. The recent exposure of AAPT s data by Anonymous shows that there may be some other means of obtaining this data. I actually wonder, if the scheme was in, what they would find? Take email for example; my email leaves my computer and directly communicates with the destination mail server. If someone pulled my email records they d show very little information. If it is too hard to run your own mailserver, I m sure the enterprising no-gooders that they are so interested in finding can work out how to buy a remote mailserver somewhere. If that s too hard, there s always gmail with https, or VPNs, or TOR, or.. well there are plenty of options. It probably comes down to how resourceful (if thats the word) people are. By the way, mentioning https reminded me of the great little plug-in called HTTPS Everywhere by EFF. I m glad to see that this legislation is stalled. To me the paperwork I saw appeared to be saying that they got some bad guys with what powers they have so they will catch more with more powers; therefore more powers are good, OK? It s a rather simplistic view of the world. My only worry is its not stopped, just stalled so it might be re-animated in future.

9 August 2012

Don Armstrong: Migrating from Subversion to git with git-annex

Recently, I've started converting many of my subversion repositories to git, some of which contain fairly large files (2-3G). However, git can be slow to deal with repositories with large files, and it also isn't able to selectively discard unneeded files when disk space is pressing. Thankfully, git-annex resolves most of these problems with git, but the process required to use git-annex on a converted subversion repository is slightly complicated. Basic conversion of svn to git The basic conversion of svn to git is done using git-svn:

 git svn clone file:///srv/svn/foo --no-metadata -A authors.txt -T trunk foo

where /srv/svn/foo is the subversion repository, authors.txt is a list of login = Full Name <email@example.com> pairs matching each of the subversion commit authors, and foo is the git repository to create. git-svn has a ton of useful options, but the basic invocation above is all I'm concerned with. Migrating large files from git into git-annex In order to migrate from git to a git+git-annex setup, we'll have to walk the entire commit history, and edit each commit to instead store large files in git-annex, replacing the large file with a symlink, and finally eliminate all of the references to the old large objects, and do garbage collection. Because we may have the same file move around, we're going to use the git-annex SHA1 backend instead of the default WORM backend which is based on filename and size, and init git-annex.

  cd foo; echo '* annex.backend=SHA1' > .git/info/attributes
  git annex init

Then, we're going to filter out the large files using

git
filter-branch

. To do that, we'll first, we'll create a little helper script git_annex_add.sh, which will remove the file from the git repository, add to git annex, and fix up the symlinks:

 #!/bin/bash
 f="$1";
 git rm --cached "$ f ";
 git annex add "$ f ";
 annexdest="$(/bin/readlink -v $ f )";
 ln -sf "$ annexdest#../../ " "$ f ";
 echo -n "Added: "
 ls -l "$ f ";

Then we will run filter-branch, and annex all files larger than 5 megabytes. [Tweak the find command if you want to do something different.]

 git filter-branch  --tag-name-filter cat --tree-filter \
'find . -ipath \*.git\* -prune -o -path \*.temp\* -prune -o -size +5M -type f -print0 xargs -0 -r -n1 ~/git_annex_add.sh;
 git reset HEAD .git-rewrite; :' -- master

This operation will take a while. [It would be better to do this during the initial svn git conversion, but since that requires more knowledge of git-svn, svn, git, and git-annex internals than I have, and I only have to do this once for each repository, it's not worth my time.] Now we have successfully switched everything to using git-annex, and we need to clean out the old references to the files:

 rm .git/svn -rf;
 rm -rf .git/refs/original .git/refs/remote/trunk .git/refs/remote/git-svn;
 git reflog expire --expire=now --all
 git gc --prune=now
 git gc --prune=now --aggressive

(I'm not sure if the last two commands need to be separate; I'm cargo culting a bit there.) Storing all git-annex files in a remote repository Because git-annex allows you to easily throw away files which are no longer referred to by the tip of any branch using git annex unneeded (and because I'd like all of the files on my central remote repository), I'm going to shove all of the git annex files into the remote bare repository. Normally, you would use

git annex copy
--to=remote;

to do this, but because that only copies needed files, not everything, we'll have to do it manually. First, create the remote repository:

 git init --bare /srv/git/foo.git
 cd /srv/git/foo.git; git annex init foo.example.com

Add the remote to the local repository, push to the remote, and sync the objects and sync the annex:

 git remote add origin ssh://foo.example.com/srv/git/foo.git
 git push origin master
 rsync -avP .git/annex/objects ssh://foo.example.com/srv/git/foo.git/annex/.;
 git annex sync

Finally, on the remote, run git annex fsck to clean up the links to the imported objects:

 cd /srv/git/foo.git; git annex fsck;

Unresolved issues I don't know if the above works properly for branches. I suspect that it does not. I also have not exhaustively tested this methodology to verify that all of the history is present in every case. But hopefully this post (or some modification of it) will be helpful to you. Credit Many of the methodologies described here I originally found in tyger's git-annex forum post, the git gc stuff came from random google searches about shrinking git repositories, and the rsync suggestion came from joeyh (author of git-annex) and the other helpful denizens of #vcs-home on irc.oftc.net.

John Sullivan: Tor UK goes DRM-free

UK Readers Can Now Purchase DRM-Free Books From Tor UK

We believe that making our Tor ebooks DRM-free is the best for our readers, allowing you to use legitimately-purchased ebooks in perfectly legal ways, like moving your library from one ereader to another, says Jeremy Trevathan, Publisher at Pan Macmillan. We understand that DRM can make your ebooks less easy to read. It also makes building and maintaining your digital library more complicated. For these reasons, we are committed to remaining DRM-free.

I'm really happy to see this and I hope more ebook sellers follow suit. Some of the others already doing DRM-free can be found on our list at Defective by Design. Also see StoryBundle, which just launched and offers a set of DRM-free ebooks on a pay-what-you-want basis, with some of the money going to charity.

C.J. Adams-Collier: Linus on Instantiation and Armadaification

I feel a sense of pride when I think that I was involved in the development and maintenance of what was probably the first piece of software accepted into Debian which then had and still has direct up-stream support from Microsoft. The world is a better place for having Microsoft in it. The first operating system I ever ran on an 08086-based CPU was MS-DOS 2.x. I remember how thrilled I was when we got to see how my friend s 80286 system ran BBS software that would cause a modem to dial a local system and display the application as if it were running on a local machine. Totally sweet. When we were living at 6162 NE Middle in the nine-eight 292, we got an 80386 which ran Doom. Yeah, the original one, not the fancy new one with the double barrel shotgun, but it would probably run that one, too. It was also totally sweet and all thanks to our armadillo friends down south and partially thanks to their publishers, Apogee. I suckered my brothers into giving me their allowance from Dad one time so that we could all go in on a Sound Blaster Pro 16 sound card for the family s 386. I played a lot of Team Fortress and Q2CTF on that rig. I even attended the Quake 3 Arena launch party that happened at Zoid s place. I recall that he ported the original quake to Linux. I also recall there being naughty remarks included in the README.txt. When my older brother, Aaron turned 16, he was gifted a fancy car. When asked what type of car I would like when I turned 16, I said that I d prefer a computer instead. So I got a high-end 80486 with math co-processor. It could compile the kernel in 15 minutes flat. With all the bits turned on on in /usr/src/linux/.config. But this was later. I hadn t even heard of linux when I got my system. I wanted to be entertained by the thing. I made sure to get a CD-Rom and a sound card. I got on the beta for Ultima Online and spent a summer as a virtual collier. Digging stuff out of mines north of Britannia and hauling them to town to make weapons and armor out of them. And then setting out in said armor only to be PK d because I forgot healing potions and I was no good at fighting. While I was in the middle of all this gaming, my friend Lucas told me that I should try out this lynx thing that they run at the University of Washington. He heard that it was reported to run doom faster than it ran on MS-DOS. It turns out that it did, but that it was not, in fact, called lynx. Or pine. The Doom engine ran so fast that the video couldn t keep up. This was probably because they didn t use double buffering for frame display, since they didn t want to waste the time maintaining and switching context. I think I downloaded the boot/root 3.5 disk pair and was able to get the system to a shell with an on-phone assist from the Rev. I then promptly got lost in bash and the virtual terminals (OMG! I GET SIX CONSOLES!?) and bought a book on the subject. It shipped with slackware. Which I ran. Until Debian came along. Lucas also recommended that I try out this IRC thing, so I did. And I m still doing it on #linpeople just like I did back then. I learned to write Pascal on dos. Then I learned c while they were trying to teach me c++. I learned emacs and vi when I was attending North Kitsap High School. I learned sed and only a little awk when I took Running Start classes in Lynnwood at Edmonds Community College and perl & x.509 while attending Olympic Community College and simultaneously jr-administering Sinclair Communications. I studied TCP/IP, UNP, APUE, C and algorithms & data structures while preparing for an interview with a company whose CEO claimed to have invented SCSI. I learned PGP and PHP while writing web-based adware for this company. I didn t want to write ads and instead wanted to work in security, so took a job with Security Portal. While there, I wrote what one might call a blogging platform. It worked and made it possible for authors to write prose and poetry. Editors didn t have to manage a database in order to review and publish the posts that were ready. Everyone but me was able to avoid html and cgi. Then I sold pizza. Then I helped bring the bombay company onto the interwebs using the Amazon ECS (now AWS) platform. Then I helped support MaxDB. Then I helped develop and maintain the Amazon blogging platform. And then attempted to reduce the load on the Amazon pager system by doing and enforcing code reviews. It turns out that they prefer to run their support team at full bore and a load average of 16.

I am now, still, fully employed in an effort to make hard things possible. The hard thing we re working on now is the implementation and ongoing operations of distributed x.500 infrastructure. This includes request handling, processing and delivery of response ( la HTTP, SMTP, IMAP, SIP, RTP, RTSP, OCSP) including authentication, authorization and auditing (AAA) of all transactions. It s a hard thing to get right, but our product development team gets it right. Consistently and reliably. We make mistakes sometimes (sorry Bago), but we correct them and make the product better. I m the newest member of an R and d team (note: big R, little d) called NTR, which sits behind the firewall that is Product Development, out of production space. In a manner that reminds me of Debian Testing. We try new things. Our current project is to allow users to compare their current (cloud-based or iron-based) IT system with what their system would be like with a BIG-IP in front of it. I can probably come up with a demo if anyone s interested in checking it out. I ll go work on that now.

C.J. Adams-Collier: Linus on Instantiation and Armadaification

8 August 2012

Raphaël Hertzog: How to use quilt to manage patches in Debian packages

Most Debian source packages are using the 3.0 (quilt) format. This means that Debian changes to upstream files are managed in a quilt patch series. Knowledge of quilt is thus a must if you want to get involved in some serious packaging work. Don t worry, this tutorial will teach you how to use quilt in the context of Debian packaging. Pre-requisites Install the packaging-dev package to get a decent set of packages to do Debian packaging. It includes the quilt package.

$ sudo apt-get install packaging-dev

What s quilt? Read the description of the Debian package:

Quilt manages a series of patches by keeping track of the changes each of them makes. They are logically organized as a stack, and you can apply, un-apply, update them easily by traveling into the stack (push/pop). Quilt is good for managing additional patches applied to a package received as a tarball or maintained in another version control system. The stacked organization is proven to be efficient for the management of very large patch sets (more than hundred patches). As matter of fact, it was designed by and for Linux kernel hackers (Andrew Morton, from the -mm branch, is the original author), and its main use by the current upstream maintainer is to manage the (hundreds of) patches against the kernel made for the SUSE distribution.

The various files involved Stack of files Picture

To better understand the various quilt commands, you should have a basic idea of how the tool works. The stack of patches is maintained in a dedicated directory ( patches by default, but in Debian packages we override this value to debian/patches ). This directory contains the patch files and a series file that gives an ordered list of patches to apply. Example:

$ ls debian/patches/
01_use_stdlib_htmlparser_when_possible.diff
02_disable-sources-in-sphinxdoc.diff
03_manpage.diff
06_use_debian_geoip_database_as_default.diff
series
$ cat debian/patches/series
01_use_stdlib_htmlparser_when_possible.diff
02_disable-sources-in-sphinxdoc.diff
03_manpage.diff
06_use_debian_geoip_database_as_default.diff

When quilt is used, it also maintains some internal files in a directory of its own (it s named .pc ). This directory is used to know what patches are currently applied (.pc/applied-patches) and to keep backup copies of files modified by the various patches. Configuring quilt Before going further, you should put this in your ~/.quiltrc file:

for where in ./ ../ ../../ ../../../ ../../../../ ../../../../../; do
    if [ -e $ where debian/rules -a -d $ where debian/patches ]; then
        export QUILT_PATCHES=debian/patches
        break
    fi
done

This ensures that quilt will always use debian/patches instead of patches when your current directory is within a Debian source package where debian/patches exists. If you only use quilt for debian packaging, then you can be more expedient and put a simple export QUILT_PATCHES=debian/patches in that file. As a matter of personal preferences, I also have those lines in my ~/.quiltrc:

QUILT_PUSH_ARGS="--color=auto"
QUILT_DIFF_ARGS="--no-timestamps --no-index -p ab --color=auto"
QUILT_REFRESH_ARGS="--no-timestamps --no-index -p ab"
QUILT_DIFF_OPTS='-p'

It enables syntax coloring for the output of several commands and customizes the generated patches to get rid of useless information. I recommend you to use those settings too. Applying and unapplying patches, navigating in the stack of patches When you already have a patch series, you can navigate in the stack of patches so that any subset of consecutive patches (starting from the bottom) can be applied. quilt series will list all patches known by quilt. You can apply all patches with quilt push -a or unapply them all with quilt pop -a. You can also verify what patches are applied (quilt applied) or unapplied (quilt unapplied). quilt push applies the next unapplied patch (i.e. the patch returned by quilt next) and quilt pop unapplies the last applied patch (i.e. the patch returned by quilt top). You can give a patch name as parameter to quilt push/pop and it will apply/unapply all the patches required until the given patch is on the top. Here are some examples of navigation in a quilt patch series

$ quilt series
02_disable-sources-in-sphinxdoc.diff
03_manpage.diff
06_use_debian_geoip_database_as_default.diff
$ quilt applied
No patches applied
$ quilt next
02_disable-sources-in-sphinxdoc.diff
$ quilt push
Applying patch 02_disable-sources-in-sphinxdoc.diff
patching file docs/conf.py
Now at patch 02_disable-sources-in-sphinxdoc.diff
$ quilt push
Applying patch 03_manpage.diff
patching file docs/man/django-admin.1
Now at patch 03_manpage.diff
$ quilt applied
02_disable-sources-in-sphinxdoc.diff
03_manpage.diff
$ quilt unapplied
06_use_debian_geoip_database_as_default.diff
$ quilt pop -a
Removing patch 03_manpage.diff
Restoring docs/man/django-admin.1
Removing patch 02_disable-sources-in-sphinxdoc.diff
Restoring docs/conf.py
No patches applied
$ quilt push 03_manpage.diff
Applying patch 02_disable-sources-in-sphinxdoc.diff
patching file docs/conf.py
Applying patch 03_manpage.diff
patching file docs/man/django-admin.1
Now at patch 03_manpage.diff
$ quilt top
03_manpage.diff

Creating a new patch If there s no quilt series yet, you want to create the debian/patches directory first. If you already have one, you need to decide where to insert the new patch. Quilt will always add the new patch just after the patch which is currently on top. So if you want to add the patch at the end of the series, you need to run quilt push -a first. Then you can use quilt new name-of-my-patch.diff to tell quilt to insert a new empty patch after the current topmost patch. In this operation quilt does almost nothing except updating the series file and recording the fact that the new patch is applied (even if still empty at this point!). Now to add changes in this patch, you re supposed to modify files but only after having informed quilt of your intent to modify those files. You do this with quilt add file-to-modify. At this point quilt will make a backup copy of that file so that it can generate the final patch when you re done with your changes. It s quite common to forget this step and to be unable to generate the patch afterward. That s why I recommend you to use quilt edit file-to-modify which is a shorthand for doing quilt add and then opening the file in your favorite text editor. If you want, you can review your work in progress with quilt diff. When you re done with the changes, you should call quilt refresh to generate the patch (or to update it if it was already existing). And since you re a good packager, you call quilt header --dep3 -e to add DEP-3 meta-information to your patch header. Importing an external patch If someone else already prepared a patch, you can just import it right away with quilt import /tmp/the-patch. If you want to import it under a better name you can use the option -P better-patch-name . Like quilt new, it inserts the patch after the topmost patch. Updating patches for a new upstream version With some luck, your patches will still apply with some offsets in line numbers (quilt displays those offsets) and sometimes with some fuzz:

$ quilt push
[...]
Hunk #1 succeeded at 1362 (offset 11 lines).
Hunk #2 succeeded at 1533 with fuzz 1 (offset 4 lines).
[...]

While offsets are nothing to worry about (it means some lines were added and/or removed before the patched part), fuzz means that patch had to ignore some context lines to find the place where to apply the changes. In that case, you need to double check that patch did the right thing because it might have made changes somewhere where it shouldn t. Also, you ll have to update those patches because dpkg-source doesn t accept any fuzz. If you re confident that all patches are correctly applied by quilt, you can refresh them to get rid of those warnings:

$ quilt pop -a
[...]
$ while quilt push; do quilt refresh; done

That was for the easy case. Now let s deal with the case where some of the patches no longer apply. There s one case that is usually nice to have:

$ quilt push
Applying patch 04_hyphen-manpage.diff
patching file docs/man/django-admin.1
Hunk #1 FAILED at 194.
1 out of 1 hunk FAILED -- rejects in file docs/man/django-admin.1
Patch 04_hyphen-manpage.diff can be reverse-applied

When the patch can be reverse-applied, it means that the upstream authors included the Debian patch (or that they made the same change even though you forgot to forward the patch). You can thus get rid of it:

$ quilt delete -r 04_hyphen-manpage.diff
Removed patch 04_hyphen-manpage.diff

Note that without the -r the patch is only dropped from the series file. With -r the patch file is also removed. But there s a less desirable case where the patch is still relevant but it doesn t apply any longer:

$ quilt push
Applying patch 01_disable_broken_test.diff
patching file tests/regressiontests/test_utils/tests.py
Hunk #1 FAILED at 422.
1 out of 1 hunk FAILED -- rejects in file tests/regressiontests/test_utils/tests.py
Patch 01_disable_broken_test.diff does not apply (enforce with -f)

In that case, you should follow quilt s advice to force the patch application, manually apply the parts of the patch that were rejected, and then refresh the patch.

$ quilt push -f
Applying patch 01_disable_broken_test.diff
patching file tests/regressiontests/test_utils/tests.py
Hunk #1 FAILED at 422.
1 out of 1 hunk FAILED -- saving rejects to file tests/regressiontests/test_utils/tests.py.rej
Applied patch 01_disable_broken_test.diff (forced; needs refresh)
$ vim tests/regressiontests/test_utils/tests.py    
$ quilt refresh
Refreshed patch 01_disable_broken_test.diff

Other quilt commands You should probably read quilt s manual page too to learn about the various other commands and options that exist. There s at least quilt rename new-name that you can also find useful to rename the topmost patch (you can use -P patch-to-rename to rename a patch which is not currently at the top). Feedback Please leave comments if you have suggestions of improvements, or if there are some tips that are good to know. I might incorporate them in this article. Feel free to share this article with newbie packagers which are struggling with quilt. For your convenience, you can also refer to this article with this URL: http://raphaelhertzog.com/go/quilt

Found it useful? Be sure to not miss other packaging tips (or lessons), click here to subscribe to my free newsletter and get new articles by email.

2 comments Liked this article? Click here. My blog is Flattr-enabled.

Martín Ferrari: Debian whois: a proposal to increase visibility and recognition of contributors

Another great DebConf is over. Two weeks ago, I was lingering a little longer in Nicaragua, enjoying some tropical holidays, and having a few To as with some fellow debconfers in Granada, and I exposed some rough ideas I've been incubating in the back of my head for some days. Since it had a positive initial reaction from the table (Gregor already volunteered to help!), I want to show it to a larger audience, in the hopes of getting some useful feedback, ideas, and maybe even some volunteers! The trigger was Zack's Bits from the DPL talk, and Enrico's post on the same topic. We need to recognise more what people is doing in the project, much valuable work is done in the shadows, and it is in our interest to make the people doing that work feel acknowledged, feel they are part of the project. The core of my idea is to create a web service that, for each person known to the project (let's say, http://whois.debian.net/tincho), shows a simple page with information about the person, optionally a mugshot/hackergotchi, what are they working on, maybe some stats and links (the Debian portfolio stuff, for example), and the core of this idea badges of merit, and attribution for their contributions. What I picture is having badges that say: Closed X RC bugs , Active translator for Esperanto , #1 video team camera operator in DC12 , Created 100 bugs .. You get the idea. In this ugly mock I created with my null designing skills, the idea might be a little more clear.

An important feature of the project is that it will be just a collector of information distributed in different places:

Personal information stored in special files in the person's alioth account (if any). If the person does not have an account or does not add any information, the page will only show the user name, and maybe the full name.
Automated links to other sources of information: PTS, Debian Portfolio, UDD Dashboard, etc.
A collection of badges and detailed statistics gathered from independent sources:

Automated data from UDD, PentaBarf, BTS.
Statistics from commits, changes to wikis, mailing lists, etc.
Lists created by specific teams to recognise their members, specially when these are not using automated tools that can generate reports.

This way, every team or project in Debian that wants to give recognition to its volunteers would easily do it, without needing to dive into the details of the code, or dealing with much bureaucracy. Just a signed mail with the pointer to the source of the data would be sufficient. I think the more complex part of this project would be matching different identities to real persons, in a way similar to carnivore, so we can match between different email addresses, a name, a Debian or alioth user-id (if applicable), IRC nickname, or any other identification that might be used to distinguish a person inside Debian. This might sound scary, but I don't plan on exposing or crawling any identities that are not currently publicly exposed, I just want to be able to see in a single page all the information associated both with my debian.org email address and with my gmail one. I have some basic design ideas, but I don't want to make this post even longer than what it is now. Please get in touch if you have comments, ideas, or if you want to flame me. Tags: Planet Debian

5 August 2012

Stefano Zacchiroli: bits from the DPL for July 2012

Monthly DPL bits, fresh from the oven.

Tip to feel good about the release #476: before reading this, grab and fix one of the RC bugs affecting Wheezy. Done? Now you're ready for a slightly less exciting report of DPL activities. Highlights

I've wrapped up a couple of years of discussions with FSF representatives, proposing a joint working group on FSF' assessment of Debian Free-ness. Some news coverage ensued. The discussion list that has been set up welcomes participants and, in particular, could use some more DDs, fond of Debian principles but also interested in external review of our free/non-free separation.
DebConf12 has happened!, and many of us attended it. Due to family reasons I've been able to attend only DebCamp and DebConf day-0, just in time to deliver the traditional "bits from the DPL" speech. Among other topics, in the speech I've discussed governance scaling issues in our governance, and in particular about the role of the DPL --- issue which I'm partly trying to address with the "DPL helpers" initiatives. Slides and videos of the talk are available. As many others DebConf people, I took part in Nicaraguan TV shows about DebConf, showing a good deal of interest by the local media in our presence. I don't have links to the recordings yet, but I suspect DebConf orga people will provide them as soon as they're available.
Many others DebConf12 events are relevant for future Project directions. I went through many of the events I've missed thanks to the work of the video team and of the gobby note takers. I recommend doing the same. Some events are particularly relevant to recent project-wide discussions, such as EFI, hijacking, and the release process and I'm sure you'll find more If you're a DebConf12 speaker and produced slides, please upload them to penta, so that they could be used as future reference.

Assets

On the hardware front, we bought another server according to DSA's "5-year plan". It will be used as a ganeti node at the man-da hosting location. We bought it thanks to donated money, for a total of ~6'600 EUR.
(After ~8 months of pestering,) we got from SPI a data dump of Debian financial transactions there. Thanks for this result goes to the SPI Board and in particular to Robert Brockway and Joerg Jaspert. This was the last blocker to produce detailed financial reports for the past couple of years of Debian activities. Unfortunately, it still is a blocker: Debian auditors found out that the data dump is not complete. SPI will send us remaining data soon , but I don't have an ETA, unfortunately.
An ftpmasters sprint is being organized and I've been happy to approve its budget. Help is still needed on the -sprints list to keep up with the organization part of sprints, including activities like documenting sprints to show donors what we do with their help.

Logo & trademark

to stop the absurdity of not using the official logo ("with Debian") as part of our official theme, I advanced a bit on the topic of relicensing the logo under a DFSG-free license. I sought a second legal advice to SFLC (as requested by the SPI board). They confirm we can safely do the relicensing (without undermining our rights on contained marks) under a license like dual LGPLv3+ / CC-BY-SA, provided that a suitable trademark policy is in place
which is why I've posted to -project a trademark policy draft, based on legal advice from SFLC. I've gather useful feedback already, but more is welcome. Note that, as Russ very eloquently put it, we do not particularly enjoy thinking about which restrictions we put on the usage of our marks. But given we do have (and use) those marks, we should better empower the legal owners (SPI in this case) to properly defend them. Our specification for the policy to SFLC has been "as free as possible", and that is what we're trying to achieve.

Note that the actual logo relicensing should be done by SPI, via their board, upon request of mine (as Debian Project liaison). We won't make it for the next SPI board meeting, as it is on 4 days away. But we can aim for the subsequent meeting, on September 13th. If all goes well (big "if"), we will enjoy a DFSG-free logo after that date. Internal organization Recent flurry of re-organization in the tech-ctte --- which I somewhat triggered pushing for periodic meetings --- seems to be proceeding well. I'm very happy about it, as we all need to trust that tech-ctte decisions will be not only sound, but also prompt. If you're interested into this topic, some recent evidence of the ongoing reorganization and its result can be found in their DebConf12 BoF, minutes of the last meeting, a set of forthcoming GRs, and the recent great decision of posting decisions results to d-d-a (as it happened for node/nodejs). Kudos to tech-ctte members for the recent activism! I haven't worked on it myself directly, but I highlight Enrico's work on server side archival of NM conversations. It has the potential of enabling automatic detection of stuck NM processes. I'm a bit rusty as AM now, but I think missing that ability is one of the main remaining causes of frustration when joining Debian. So, if you are an AM, please opt-in and use this feature with your appicants. If you are not an AM why not? :-)

Miscellanea Some misc legal stuff:

I got the legal advice we asked about the possibility of turning mentors.debian.net into an official service. The worry there was about the potential copyright infringements by mentees. In the end, we don't need to worry about that, provided we've clear contact points (and "reasonable" timing ) for removing infringing material once it's been reported, and that we declare our public policy about that. I've a draft policy document to start with. I'll soon contact mentors.d.n to adapt it where needed and have it vetted by SPI before going live.
I've worked with GNOME maintainers (thanks in particular to Jordi Mallach) to rework the mixed Debian/GNOME logo on their alioth project page, to stay on the safe side and avoid any ambiguity wrt GNOME trademark policy.

and some misc "political" stuff:

we've been happy to accept an invitation to attend the GNOME Advisory Board meeting at GUADEC: Jord (thanks, again!) kindly accepted to represent Debian there. An early report has been posted on his blog, a more detailed one is forthcoming to -project.
OSI has discussed with their affiliates the possibility of signing the Declaration of Internet Freedom. On behalf of Debian, I've vouched for it. By the way, I'm serving pro tempore as Debian liaison at OSI, but if there are people interested in doing that, I'll be happy to consider handing over the responsibility to other (it's a quite lightweight one, at present).

Happy RC bug squashing,
Cheers.

PS the boring day-to-day activity log for July 2012 is available at master:/srv/leader/news/bits-from-the-DPL.txt.201207

Craig Small: Context sensitive menus with GtkTreeView

A project I now maintain called GJay uses the GTK+ toolkit and has a GtkTreeView that shows the directories and the songs in them. I was wanting to add a context sensitive menu on that view so that you could do things to the item you have right-clicked on; generally file specific things like give me information about this song or queue it to play. My only problem was, how to do it? A few Google searches showed things that were almost, but not quite, what I was after. The main problem being either the documentation was incomplete or it assumed I was after the selected field (the row I may of left-clicked on) and not the one I right-clicked on to get this event. This GtkTreeView uses a store where NAME_COLUMN is defined at 0, meaning it is the first column. The program uses this column to store the filename of the song. So first I needed to create an event to kick things off, we need to connect the button-press-event to our tree_view, which calls a function in this case view_on_button_pressed.

 g_signal_connect(tree_view, "button-press-event", 
   G_CALLBACK(view_on_button_pressed), NULL);

gboolean view_on_button_pressed(GtkWidget *tree_view, 
                                GdkEventButton *event, 
                                gpointer user_data)  
  GtkTreeSelection *selection;
  GtkTreePath *tree_path;
  GtkTreeModel *tree_model;
  GtkTreeIter tree_iter;
  gchar *name;
 
  if (event-&gt;type == GDK_BUTTON_PRESS &amp;&amp; event-&gt;button == 3)  
    tree_model = gtk_tree_view_get_model(GTK_TREE_VIEW(tree_view));
    if (gtk_tree_view_get_path_at_pos(GTK_TREE_VIEW(tree_view),
          (int)event-&gt;x, (int)event-&gt;y,
          &amp;tree_path, NULL, NULL, NULL))  
      if (gtk_tree_model_get_iter(tree_model, &amp;tree_iter, tree_path))  
        gtk_tree_model_get(tree_model, &amp;tree_iter, NAME_COLUMN, &amp;name, -1);
        g_print("rc name is %s\n", name);

So a breakdown of what my callback is doing:

Line 1:The callback s widget is the GtkTreeView widget we connected the signal with
Line 10:Checks the event was a button press and button #3 (right button)
Line 11: Find the GtkTreeModel from the given GtkTreeView
Lines 12-14: Get the GtkTreePath that the right-click was sitting at by using the event coordinates.
Line 15: Convert the GtkTreePath into a GtkTreeIter
Line 16: Find the actual value we put into this row.

The program, now that it knows the song title, can then go off and generate a menu for that song.

Next.

Previous.

Search Results: "ema"

18 August 2012

17 August 2012

16 August 2012

15 August 2012

14 August 2012

13 August 2012

11 August 2012

10 August 2012

9 August 2012

8 August 2012

5 August 2012

Related articles GtkTreeview and GtkListstore in python (daniweb.com)

Related articles

GtkTreeview and GtkListstore in python (daniweb.com)