Search Results: "ron"

6 February 2012

David Welton: BikeChatter.com for sale

What with two kids, a new house, and LiberWriter getting some good traction, I've been looking around for things to give to a good home so as to have less stuff to deal with. So, on the auction block goes BikeChatter.com : https://flippa.com/2696023-professional-cyclists-on-twitter-plus-2-years-of-history

BikeChatter.com is the place to go on the web to follow professional cyclists on twitter. With 500+ racers, and nearly half a million status updates from racers like Lance Armstrong, Alberto Contador, Mark Cavendish, and many, many more, this site is the best place to find out what's going on in the world of professional cycling, directly from the participants. Since I like following the site myself, I really want to see it go to people who will take it and make it even better.

5 February 2012

Russell Coker: Reliability of RAID

ZDNet has an insightful article by Robin Harris predicting the demise of RAID-6 due to the probability of read errors [1]. Basically as drives get larger the probability of hitting a read error during reconstruction increases and therefore you need to have more redundancy to deal with this. He suggests that as of 2009 drives were too big for a reasonable person to rely on correct reads from all remaining drives after one drive failed (in the case of RAID-5) and that in 2019 there will be a similar issue with RAID-6. Of course most systems in the field aren t using even RAID-6. All the most economical hosting options involve just RAID-1 and RAID-5 is still fairly popular with small servers. With RAID-1 and RAID-5 you have a serious problem when (not if) a disk returns random or outdated data and says that it is correct, you have no way of knowing which of the disks in the set has good data and which has bad data. For RAID-5 it will be theoretically possible to reconstruct the data in some situations by determining which disk should have it s data discarded to give a result that passes higher level checks (EG fsck or application data consistency), but this is probably only viable in extreme cases (EG one disk returns only corrupt data for all reads). For the common case of a RAID-1 array if one disk returns a few bad sectors then probably most people will just hope that it doesn t hit something important. The case of Linux software RAID-1 is of interest to me because that is used by many of my servers. Robin has also written about some NetApp research into the incidence of read errors which indicates that 8.5% of consumer disks had such errors during the 32 month study period [2]. This is a concern as I run enough RAID-1 systems with consumer disks that it is very improbable that I m not getting such errors. So the question is, how can I discover such errors and fix them? In Debian the mdadm package does a monthly scan of all software RAID devices to try and find such inconsistencies, but it doesn t send an email to alert the sysadmin! I have filed Debian bug #658701 with a patch to make mdadm send email about this. But this really isn t going to help a lot as the email will be sent AFTER the kernel has synchronised the data with a 50% chance of overwriting the last copy of good data with the bad data! Also the kernel code doesn t seem to tell userspace which disk had the wrong data in a 3-disk mirror (and presumably a RAID-6 works in the same way) so even if the data can be corrected I won t know which disk is failing. Another problem with RAID checking is the fact that it will inherently take a long time and in practice can take a lot longer than necessary. For example I run some systems with LVM on RAID-1 on which only a fraction of the VG capacity is used, in one case the kernel will check 2.7TB of RAID even when there s only 470G in use! The BTRFS Filesystem The btrfs Wiki is currently at btrfs.ipv5.de as the kernel.org wikis are apparently still read-only since the compromise [3]. BTRFS is noteworthy for doing checksums on data and metadata and for having internal support for RAID. So if two disks in a BTRFS RAID-1 disagree then the one with valid checksums will be taken as correct! I ve just done a quick test of this. I created a filesystem with the command mkfs.btrfs -m raid1 -d raid1 /dev/vg0/raid? and copied /dev/urandom to it until it was full. I then used dd to copy /dev/urandom to some parts of /dev/vg0/raidb while reading files from the mounted filesystem that worked correctly although I was disappointed that it didn t report any errors, I had hoped that it would read half the data from each device and fix some errors on the fly. Then I ran the command btrfs scrub start . and it gave lots of verbose errors in the kernel message log telling me which device had errors and where the errors are. I was a little disappointed that the command btrfs scrub status . just gave me a count of the corrected errors and didn t mention which device had the errors. It seems to me that BTRFS is going to be a much better option than Linux software RAID once it is stable enough to use in production. I am considering upgrading one of my less important servers to Debian/Unstable to test out BTRFS in this configuration. BTRFS is rumored to have performance problems, I will test this but don t have time to do so right now. Anyway I m not always particularly concerned about performance, I have some systems where reliability is important enough to justify a performance loss. BTRFS and Xen The system with the 2.7TB RAID-1 is a Xen server and LVM volumes on that RAID are used for the block devices of the Xen DomUs. It seems obvious that I could create a single BTRFS filesystem for such a machine that uses both disks in a RAID-1 configuration and then use files on the BTRFS filesystem for Xen block devices. But that would give a lot of overhead of having a filesystem within a filesystem. So I am considering using two LVM volume groups, one for each disk. Then for each DomU which does anything disk intensive I can export two LVs, one from each physical disk and then run BTRFS inside the DomU. The down-side of this is that each DomU will need to scrub the devices and monitor the kernel log for checksum errors. Among other things I will have to back-port the BTRFS tools to CentOS 4. This will be more difficult to manage than just having an LVM VG running on a RAID-1 array and giving each DomU a couple of LVs for storage. BTRFS and DRBD The combination of BTRFS RAID-1 and DRBD is going to be a difficult one. The obvious way of doing it would be to run DRBD over loopback devices that use large files on a BTRFS filesystem. That gives the overhead of a filesystem in a filesystem as well as the DRBD overhead. It would be nice if BTRFS supported more than two copies of mirrored data. Then instead of DRBD over RAID-1 I could have two servers that each have two devices exported via NBD and BTRFS could store the data on all four devices. With that configuration I could lose an entire server and get a read error without losing any data! Comparing Risks I don t want to use BTRFS in production now because of the risk of bugs. While it s unlikely to have really serious bugs it s theoretically possible that as bug could deny access to data until kernel code is fixed and it s also possible (although less likely) that a bug could result in data being overwritten such that it can never be recovered. But for the current configuration (Ext4 on Linux software RAID-1) it s almost certain that I will lose small amounts of data and it s most probable that I have silently lost data on many occasions without realising. Related posts:
  1. Some RAID Issues I just read an interesting paper titled An Analysis of...
  2. ECC RAM is more useful than RAID A common myth in the computer industry seems to be...
  3. Software vs Hardware RAID Should you use software or hardware RAID? Many people claim...

Petter Reinholdtsen: Saving power with Debian Edu / Skolelinux using shutdown-at-night

Since the Lenny version of Debian Edu / Skolelinux, a feature to save power have been included. It is as simple as it is practical: Shut down unused clients at night, and turn them on again in the morning. This is done using the shutdown-at-night Debian package. To enable this feature on a client, the machine need to be added to the netgroup shutdown-at-night-hosts. For Debian Edu, this is done in LDAP, and once this is in place, the machine in question will check every hour from 16:00 until 06:00 to see if the machine is unused, and shut it down if it is. If the hardware in question is supported by the nvram-wakeup package, the BIOS is told to turn the machine back on around 07:00 +- 10 minutes. If this isn't working, one can configure wake-on-lan to try to turn on the client. The wake-on-lan option is only documented and not enabled by default in Debian Edu. It is important to not turn all machines on at once, as this can blow a fuse if several computers are connected to the same fuse like the common setup for a classroom. The nvram-wakeup method only work for machines with a functioning hardware/BIOS clock. I've seen old machines where the BIOS battery were dead and the hardware clock were starting from 0 (or was it 1990?) every boot. If you have one of those, you have to turn on the computer manually. The shutdown-at-night package is completely self contained, and can also be used outside the Debian Edu environment. For those without a central LDAP server with netgroups, one can instead touch the file /etc/shutdown-at-night/shutdown-at-night to enable it. Perhaps you too can use it to save some power?

John Goerzen: Rain, A Funeral, and Excitement

Rain Friday was something of a rare day for February in Kansas. Starting at about 2AM, the wind picked up, blowing so hard that our windows rattled. That part isn t so rare. Then the cold rain started, dropping almost 2.5 throughout the day. As I worked, I had the blinds on the windows open, but they didn t let in very much light. Still, the wind had calmed down, so the intermittent rain outside was peaceful. Jacob went out to play for a little while, so every so often I saw a warmly-dressed and excited-looking 5-year-old run past my window. A little while after he came in, I told Jacob, I saw you playing outside. His response: Oh good! I got wet! Which, despite the fact that it was about 50 degrees, seemed to excite him. After the blustery start, the calm, slow, and peaceful rain was a pleasant thing to see throughout the day. Funeral My great aunt Alice Goerzen passed away last Sunday. So today, for the third time in a little over a year, I was at the funeral of a Goerzen relative and neighbor. Alice s husband, Milt, passed away in late 2010, and it was while I was at his funeral that Jacob got run over by a tractor. That memory certainly came back to me today. But I think I should set the stage and explain what funerals are like in this small, rural Kansas community. At the church, while people file in, family and close friends generally defined as loosely as desired meet in some other room before the funeral. Memories may be shared, or songs sung, or maybe just a brief meditation or prayer. Then the man from the funeral home there s only one in town will step in. Ivan Miller owned the business for decades, and although he s now retired, his replacement seems pretty similar. Kindly, respectful, and pretty much unchanging. This group then files into the church sanctuary to sit up front, while the rest of the congregation is standing and music is played. We typically sing some hymns, hear memories from the family, a message from a pastor, and then do downstairs for faspa: an light meal with coffee, zwieback, funeral cheese , and some relishes and dessert. You can, by the way, go to the local grocery store and find a product labeled funeral cheese . It s a sharp cheddar, sliced thick and cut into pie piece-shaped wedges. After everyone has picked up their food, microphones are passed around, and anybody that wants to can share memories and stories. These are often hilarious, or touching, and can be more random than anyone could expect. Today we heard a lot about how Aunt Alice loved her flowers and garden. We even saw a video of her giving a tour of her garden, with Milt s mower in the background occasionally accidentally causing a distraction (or maybe it wasn t so accidental; he d never miss an opportunity to cause some mischief ) I tend to think of attending funerals around here as a good time. Sadness is inevitable, but there are so many amazing stories that it is hard to leave feeling sad. Excitement This afternoon, Jacob found me in the office and as he often does, said, Dad, I want to do something with you. Usually I ask him what he d like to do, but his first instinct is usually to ask for watching train videos on Youtube. So sometimes I make other suggestions. Today we played hide and seek with radios, in which the person that is counting is supposed to radio to the other person when they are done. Today was the first time that Jacob came up with the trick of talking into the radio while I was hiding so he could hear where I was. I was sort of proud of him, and he failed to completely hide his smile when I told him I had to turn off my radio or else he d find me too fast. Then later, we played with Jacob s computer, a Linux-based command-line-only machine. I have set up a few shell scripts and aliases for him. Since it doesn t play videos, he doesn t use it as much as he does mine, but it is really fun to watch how his interaction with it changes as he gets older. He can now read amazingly well for a 5-year-old, and is starting to learn how to spell. He loves word games, writing, and typing. I thought I would install an ASCII art program for him. I told Jacob I had some ideas for a new game, and he was irresistibly intrigued. I offered him a choice between figlet and toilet. And, as is probably no surprise to anyone with a 5-year-old, he chose toilet based on its name, Jacob and Oliver both loved typing things and seeing them displayed bigger. I showed Jacob how I could make a freight train by typing ,<@-[]-(*)-@> (that s the comma-shaped snowplow, engine, boxcar [], tank car (*), and caboose @>). Then toilet drew them big, and though abstract, caused great excitement. I hooked up one of the speech synthesizers in Debian to a simple shell script named talk , which is a huge hit with the boys. They enjoy typing in nonsense and hearing the funny result, or in typing in real words and hearing how the computer says them right (or doesn t). All told, we had a good hour s worth of excitement up there.

4 February 2012

Stefano Zacchiroli: bits from the DPL for January 2012

Fresh from the oven, monthly report of what I've been working on as DPL during January 2012.
Dear Developers,
here is another monthly report of what happened in DPL-land, this time for January 2012. There's quite a bit to report about --- including an insane amount of legal-ish stuff --- so please bear with me. Or not. Legal stuff Most of the above wouldn't have been possible without the precious help of folks at SFLC working for SPI and Debian. Be sure to thank SFLC for what they're doing for us and many other Free Software projects. Coordination Nobody stepped up to coordinate the artwork collection for Wheezy I've mentioned last month, so I've tried to do a little bit of that myself. The -publicity team is now preparing the call for artwork and hopefully we'll send it out RSN. In case you want to help, there is still a lot of room for that; just show up on the debian-desktop mailing list. Sprints A Debian Med sprint has happened in January, and Andreas Tille has provided a nice and detailed report about it. Some more sprints are forthcoming this spring, how about yours? Money Important stuff going on Other important stuff has been going on in various area of the project in January. I'd like to point your attention to a couple of things: Miscellanea In the unlikely case you've read thus far, thanks for your attention! Happy Debian hacking.
PS as usual, the boring day-to-day activity log is available at master:/srv/leader/news/bits-from-the-DPL.*

Bartosz Fe&#324;ski: Quick howto for the most stable Linux kernel with Zenbook

Quick howto for everyone having Asus UX31e aka Zenbook and wants the most stable vanilla kernel with long-time battery life and working touchpad made by Sentelic.
This should work for every Debian based distro, but has been tested only on the latest Linux Mint 12 (aka Lisa).
Tutorial is aimed mainly at beginners and should work in a copy&paste manner. Of course you have to change every occurence of linux-3.2.4 to whichever version you re going to install.
Just make sure that all commands are entered in the correct directories, prompts should look similar to these from examples.
You will need about 10GB of free disk space. Let s install some dependencies first:
fenio@zenbook ~ $ sudo apt-get install build-essential kernel-package fakeroot libncurses5-dev git iasl
Now let s download and unpack the latest kernel (3.2.4 at the time it was written):
fenio@fenio ~ $ mkdir kernel
fenio@fenio ~ $ cd kernel
fenio@fenio ~/kernel $ wget http://www.kernel.org/pub/linux/kernel/v3.0/linux-3.2.4.tar.bz2
fenio@fenio ~/kernel $ tar jxvf linux-3.2.4.tar.bz2
Now we take current kernel configuration and put in the kernel tree:
fenio@fenio ~/kernel $ cp /boot/config- uname -r  linux-3.2.4/.config
Now we have to download the latest driver for Sentelic touchpad (this is why we needed git as a dependency):
fenio@fenio ~/kernel $ git clone git://github.com/saaros/sentelic.git
fenio@fenio ~/kernel $ cp sentelic/src/sentelic.* linux-3.2.4/drivers/input/mouse/
It s time to fix broken DSDT table (part of ACPI). This is why we had iasl in dependencies.
fenio@fenio ~/kernel $ wget http://files.benesovi.eu/ux31e/ux31e_dsdt.dsl
fenio@fenio ~/kernel $ iasl -tc ux31e_dsdt.dsl
And include it in our kernel configuration:
fenio@fenio ~/kernel $ sed -ie 's/# CONFIG_ACPI_CUSTOM_DSDT is not set/CONFIG_ACPI_CUSTOM_DSDT=y/g' linux-3.2.4/.config 
fenio@fenio ~/kernel $ sed -ie "s@CONFIG_ACPI_CUSTOM_DSDT_FILE.*@CONFIG_ACPI_CUSTOM_DSDT_FILE=\" pwd /ux31e_dsdt.hex\"@g" linux-3.2.4/.config
Now ensure that all our options are set up correctly (you can change some other options if you want).
If you don t want to change anything then simply exit saving configuration.
fenio@fenio ~/kernel $ cd linux-3.2.4/
fenio@fenio ~/kernel/linux-3.2.4 $ make menuconfig
We re ready to start compilation now!
fenio@fenio ~/kernel/linux-3.2.4 $ fakeroot make-kpkg clean
fenio@fenio ~/kernel/linux-3.2.4 $ fakeroot make-kpkg --jobs=4 --initrd --append-to-version=-fenio --revision=20120204 kernel_image kernel_headers modules_image
Of course you can change append-to-version option for something own. In the meantime (kernel compilations takes about one hour) you can modify Grub options to enable powersaving features RC6:
fenio@zenbook ~ $ sudo sed -ie 's/GRUB_CMDLINE_LINUX=.*/GRUB_CMDLINE_LINUX="i915.powersave=1 i915.semaphores=1 i915.i915_enable_rc6=1"/g' /etc/default/grub
Be sure to do that before installation of kernel, otherwise you will have to run update-grub. After compilation you can finally install your new kernel:
fenio@zenbook ~/kernel/linux-3.2.4 $ sudo dpkg -i ../*.deb
Reboot and you re done! Feel free to comment this tutorial if something went wrong.

2 February 2012

Joey Hess: unicode ate my homework

I've just spent several days trying to adapt git-annex to changes in ghc 7.4's handling of unicode in filenames. And by spent, I mean, time withdrawn from the bank, and frittered away. In kindergarten, the top of the classrom wall was encircled by the aA bB cC of the alphabet. I'll bet they still put that up on the walls. And all the kids who grow up to become involved with computers learn that was a lie. The alphabet doesn't stop at zZ. It wouldn't all fit on a wall anymore. So we're in a transition period, where we've all learnt deeply the alphabet, but the reality is much more complicated. And the collision between that intuitive sense of the world and the real world makes things more complicated still. And so, until we get much farther along in this transition period, you have to be very lucky indeed to not have wasted time dealing with that complexity, or at least having encountered Mojibake. Most of the pain centers around programming languages, and libraries, which are all at different stages of the transition from ascii and other legacy encodings to unicode. My most recent pain is because the haskell GHC compiler is moving along in the transition, getting closer to the end. Or at least finishing the second 80% and moving into the third 80%. (This is not a quick transition..) The change involves filename encodings, a situation that, at least on unix systems, is a vast mess of its own. Any filename, anywhere, can be in any encoding, and there's no way to know what's the right one, if you dislike guessing. Haskell folk like strongly typed stuff, so this ambiguity about what type of data is contained in a FilePath type was surely anathama. So GHC is changing to always use UTF-8 for operations on FilePath. (Or whatever the system encoding is set to, but let's just assume it's UTF-8.) Which is great and all, unless you need to write a Haskell program that can deal with arbitrary files. Let's say you want to delete a file. Just a simple rm. Now there are two problems:
  1. The input filename is assumed to be in the system encoding aka unicode. What if it cannot be validly interpreted in that encoding? Probably your rm throws an exception.
  2. Once the FilePath is loaded, it's been decoded to unicode characters. In order to call unlink, these have to be re-encoded to get a filename. Will that be the same bytes as the input filename and the filename on disk? Possibly not, and then the rm will delete the wrong thing, or fail.
But haskell people are smart, so they thought of this problem, and provided a separate type that can deal with it. RawFilePath hearks back to kindergarten; the filename is simply a series of bytes with no encoding. Which means it cannot be converted to a FilePath without encountering the above problems. But does let you write a safe rm in ghc 7.4. So I set out to make something more complicated than a rm, that still needs to deal with arbitrary filename encodings. And I soon saw it would be problimatic. Because the things ghc can do with RawFilePaths are limited. It can't even split the directory from the filename. We often do need to manipulate filenames in such ways, even if we don't know their encoding, when we're doing something more complicated than rm. If you use a library that does anything useful with FilePath, it's not available for RawFilePath. If you used standard haskell stuff like readFile and writeFile, it's not available for RawFilePath either. Enjoy your low-level POSIX interface! So, I went lowlevel, and wrote my own RawFilePath versions of pretty much all of System.FilePath, and System.Directory, and parts of MissingH and other libraries. (And noticed that I can understand all this Haskell code.. yay!) And I got it close enough to working that, I'm sure, if I wanted to chase type errors for a week, I could get git-annex, with ghc 7.4, to fully work on any encoding of filenames. But, now I'm left wondering what to do, because all this work is regressive; it's swimming against the tide of the transition. GHC's change is certainly the right change to make for most programs, that are not like rm. And so most programs and libraries won't use RawFilePath. This risks leaving a program that does a fish out of water. At this point, I'm inclined to make git-annex support only unicode (or the system encoding). That's easy. And maybe have a branch that uses RawFilePath, in a hackish and type-unsafe way, with no guarantees of correctness, for those who really need it.
Previously: unicode eye chart wanted on a bumper sticker abc boxes unpacking boxes

31 January 2012

Pau Garcia i Quiles: LibreTab not?

In a perfectly orchestrated marketing campaign for a 100% free-libre tablet called Spark that will run KDE Plasma Active, Aaron Seigo writes today about the problems they are facing with GPL-violations. Apparently, every Chinese manufacturer is breaking the GPLv2 by not releasing the sources for their modified Linux kernel. Conversations and conversations with Zenithink (designers of the Spark), Synrgic (designers of the Dreambook W7), etc have arrived nowhere. To the point that CordiaTab, another similar effort using Gnome instead of KDE, has been cancelled. I have to say I am very surprised at the lack of the kernel sources. What is the Free Software Foundation doing? Why don t we seek ban of all imports of tablets whose manufacturers don t release the full GPL source? Apple got the Samsung GalaxyTab imports blocked in Germany and Australia for something as ethereal as patents covering the external frame design. We are talking about license infringement, which is easier to demonstrate in court. China may ignore intellectual property but they cannot ignore business, and no imports means no business. Let s get all GPL-infringing tablet imports banned and we will get more source in two weeks than we can digest in two years. Heck, I m surprised Apple is not trying this in court to block Android!

Andrea Veri: A few useful Puppet snippets

I ve been playing with Puppet lately both on my home network and within the Fedora s Infrastructure team and I thought some of the work I did might be useful for anyone out there being stuck with a Puppet s manifest or an ERB template. Snippet #1: Make sure the user foo is alwais created with its own home directory, password, shell, and full name.
class users  
    users::add   "foo":
        username        => 'foo',
        comment         => 'Foo's Full Name',
        shell           => '/bin/bash',
        password_hash   => 'pwd_hash_as_you_can_see_in_/etc/shadow'
     
define users::add($username, $comment, $shell, $password_hash)  
    user   $username:
        ensure => 'present',
        home   => "/home/$ username ",
        comment => $comment,
        shell  => $shell,
        managehome => 'true',
        password => $password_hash,
     
   
 
Snippet #2: Make sure the user foo gets added into /etc/sudoers.
class sudoers  
file   "/etc/sudoers":
      owner   => "root",
      group   => "root",
      mode    => "440",
      
 
augeas   "addfootosudoers":
  context => "/files/etc/sudoers",
  changes => [
    "set spec[user = 'foo']/user foo",
    "set spec[user = 'foo']/host_group/host ALL",
    "set spec[user = 'foo']/host_group/command ALL",
    "set spec[user = 'foo']/host_group/command/runas_user ALL",
  ],
 
Snippet #3: Make sure that openssh-server is: installed, running on Port 222 and accepting RSA authentications only.
class openssh-server  
  package   "openssh-server":
      ensure => "installed",
   
    service   "ssh":
        ensure    => running,
        hasstatus => true,
        require   => Package["openssh-server"],
     
augeas   "sshd_config":
  context => "/files/etc/ssh/sshd_config",
    changes => [
    "set PermitRootLogin no",
    "set RSAAuthentication yes",
    "set PubkeyAuthentication yes",
    "set AuthorizedKeysFile	%h/.ssh/authorized_keys",
    "set PasswordAuthentication no",
    "set Port 222",
  ],
  
 
Snippet #4: Don t apply a specific IPTABLES rule if an host is tagged as staging in the relevant node file. On templates/iptables.erb:
<% if environment == "production" %>
-A INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 25 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 80 -j ACCEPT
<% unless defined?(staging).nil? %>
-A INPUT -s X.X.X.X -j REJECT --reject-with icmp-host-prohibited
<% end -%>
# Allow unlimited traffic on eth0
-A INPUT -i eth0 -j ACCEPT
-A OUTPUT -o eth0 -j ACCEPT
# Allow unlimited traffic from trusted IP addresses
-A INPUT -s 192.168.1.1/24 -j ACCEPT
<% end -%>
On the manifest file:
class iptables  
    package   iptables:
        ensure => installed;
     
    service   "iptables":
        ensure    => running,
        hasstatus => true,
        require   => Package["iptables"],
     
    file   "/etc/sysconfig/iptables":
        owner   => "root",
        group   => "root",
        mode    => 644,
        content => template("iptables/iptables.erb"),
        notify  => Service["iptables"],
     
 
That s all for now!

Paul Tagliamonte: python-sunlight (or: get at some awesome US Political data programmatically)

I ve spent a few days during work, after work and on the weekend working on python-sunlight, a unified API implementation of a few Sunlight services. This is very unstable, and not released yet (so please don t rely on it yet), but it will be shortly. Be sure to sign up for an API Key, and dump the key to ~/.sunlight.key there s a simple script to help with some of this in bin/, but nothing solid yet. I do, however, encourage you to use it and play around with it, and report your bugs. Contributions (in the form of code) are also very welcome, so please do fork the project and play around with it. Just to give everyone a taste of how cool this is this will pull up a list of twitter IDs of people who mention free market more then anyone else according to the congressional record:
from sunlight import capitolwords
from sunlight import congress
for person in capitolwords.phrases_by_entity(
    "legislator",
    phrase="free market",
    sort="count",
):
    n = congress.legislators( bioguide_id=person['legislator'],
        all_legislators="true" )
    if len(n) >= 1:
        n = n[0]
        if n['twitter_id'] != "":
            print n['twitter_id']
And the output:
RepRonPaul
stevekingia
SenSherrodBrown
JacksonLeeTX18
SenatorLeahy
ChuckGrassley
SenJohnMcCain
OrrinHatch
DanaRohrabacher
JudgeTedPoe
McConnellPress
edtowns
SenatorDurbin
SenatorSessions
SenatorHarkin
SenChrisDodd
russfeingold
RosLehtinen
sencarllevin
senjonkyl
Dennis_Kucinich
senatorsanders
SenatorReid
RepDanBurton
RepMikePence
SenSamBrownback
joebiden
FrankPallone
ToddAkin
senatorboxer
RepTrentFranks
JohnCornyn
Have fun! Show off what cha got, and please let me know if you do something cool!

30 January 2012

Matthew Garrett: The ongoing fight against GPL enforcement

GPL enforcement is a surprisingly difficult task. It's not just a matter of identifying an infringement - you need to make sure you have a copyright holder on your side, spend some money sending letters asking people to come into compliance, spend more money initiating a suit, spend even more money encouraging people to settle, spend yet more money actually taking them to court and then maybe, at the end, you have some source code. One of the (tiny) number of groups involved in doing this is the Software Freedom Conservancy, a non-profit organisation that offers various services to free software projects. One of their notable activities is enforcing the license of Busybox, a GPLed multi-purpose application that's used in many embedded Linux environments. And this is where things get interesting

GPLv2 (the license covering the relevant code) contains the following as part of section 4:

Any attempt otherwise to copy, modify, sublicense or distribute the Program is void, and will automatically terminate your rights under this License.

There's some argument over what this means, precisely, but GPLv3 adds the following paragraph:

However, if you cease all violation of this License, then your license from a particular copyright holder is reinstated (a) provisionally, unless and until the copyright holder explicitly and finally terminates your license, and (b) permanently, if the copyright holder fails to notify you of the violation by some reasonable means prior to 60 days after the cessation

which tends to support the assertion that, under V2, once the license is terminated you've lost it forever. That gives the SFC a lever. If a vendor is shipping products using Busybox, and is found to be in violation, this interpretation of GPLv2 means that they have no license to ship Busybox again until the copyright holders (or their agents) grant them another. This is a bit of a problem if your entire stock consists of devices running Busybox. The SFC will grant a new license, but on one condition - not only must you provide the source code to Busybox, you must provide the source code to all other works on the device that require source distribution.

The outcome of this is that we've gained access to large bodies of source code that would otherwise have been kept by companies. The SFC have successfully used Busybox to force the source release of many vendor kernels, ensuring that users have the freedoms that the copyright holders granted to them. Everybody wins, with the exception of the violators. And it seems that they're unenthusiastic about that.

A couple of weeks ago, this page appeared on the elinux.org wiki. It's written by an engineer at Sony, and it's calling for contributions to rewriting Busybox. This would be entirely reasonable if it were for technical reasons, but it's not - it's explicitly stated that companies are afraid that Busybox copyright holders may force them to comply with the licenses of software they ship. If you ship this Busybox replacement instead of the original Busybox you'll be safe from the SFC. You'll be able to violate licenses with impunity.

What can we do? The real problem here is that the SFC's reliance on Busybox means that they're only able to target infringers who use that Busybox code. No significant kernel copyright holders have so far offered to allow the SFC to enforce their copyrights, with the result that enforcement action will grind to a halt as vendors move over to this Busybox replacement. So, if you hold copyright over any part of the Linux kernel, I'd urge you to get in touch with them. The alternative is a strangely ironic world where Sony are simultaneously funding lobbying for copyright enforcement against individuals and tools to help large corporations infringe at will. I'm not enthusiastic about that.

comment count unavailable comments

Colin Watson: APT resolver bugs

I've managed to go for eleven years working on Debian and nearly eight on Ubuntu without ever needing to teach myself how APT's resolver works. I get the impression that there's a certain mystique about it in general (alternatively, I'm just the last person to figure this out). Recently, though, I had a couple of Ubuntu upgrade bugs to fix that turned out to be bugs in the resolver, and I thought it might be interesting to walk through the process of fixing them based on the Debug::pkgProblemResolver=true log files. Breakage with Breaks The first was Ubuntu bug #922485 (apt.log). To understand the log, you first need to know that APT makes up to ten passes of the resolver to attempt to fix broken dependencies by upgrading, removing, or holding back packages; if there are still broken packages after this point, it's generally because it's got itself stuck in some kind of loop, and it bails out rather than carrying on forever. The current pass number is shown in each "Investigating" log entry, so they start with "Investigating (0)" and carry on up to at most "Investigating (9)". Any packages that you see still being investigated on the tenth pass are probably something to do with whatever's going wrong. In this case, most packages have been resolved by the end of the fourth pass, but xserver-xorg-core is causing some trouble. (Not a particular surprise, as it's an important package with lots of relationships.) We can see that each breakage is:
Broken xserver-xorg-core:i386 Breaks on xserver-xorg-video-6 [ i386 ] < none > ( none )
This is a Breaks (a relatively new package relationship type introduced a few years ago as a sort of weaker form of Conflicts) on a virtual package, which means that in order to unpack xserver-xorg-core each package that provides xserver-xorg-video-6 must be deconfigured. Much like Conflicts, APT responds to this by upgrading providing packages to versions that don't provide the offending virtual package if it can, and otherwise removing them. We can see it doing just that in the log (some lines omitted):
Investigating (0) xserver-xorg-core [ i386 ] < 2:1.7.6-2ubuntu7.10 -> 2:1.11.3-0ubuntu8 > ( x11 )
  Fixing xserver-xorg-core:i386 via remove of xserver-xorg-video-tseng:i386
Investigating (1) xserver-xorg-core [ i386 ] < 2:1.7.6-2ubuntu7.10 -> 2:1.11.3-0ubuntu8 > ( x11 )
  Fixing xserver-xorg-core:i386 via remove of xserver-xorg-video-i740:i386
Investigating (2) xserver-xorg-core [ i386 ] < 2:1.7.6-2ubuntu7.10 -> 2:1.11.3-0ubuntu8 > ( x11 )
  Fixing xserver-xorg-core:i386 via remove of xserver-xorg-video-nv:i386
OK, so that makes sense - presumably upgrading those packages didn't help at the time. But look at the pass numbers. Rather than just fixing all the packages that provide xserver-xorg-video-6 in a single pass, which it would be perfectly able to do, it only fixes one per pass. This means that if a package Breaks a virtual package which is provided by more than ten installed packages, the resolver will fail to handle that situation. On inspection of the code, this was being handled correctly for Conflicts by carrying on through the list of possible targets for the dependency relation in that case, but apparently when Breaks support was implemented in APT this case was overlooked. The fix is to carry on through the list of possible targets for any "negative" dependency relation, not just Conflicts, and I've filed a patch as Debian bug #657695. My cup overfloweth The second bug I looked at was Ubuntu bug #917173 (apt.log). Just as in the previous case, we can see the resolver "running out of time" by reaching the end of the tenth pass with some dependencies still broken. This one is a lot less obvious, though. The last few entries clearly indicate that the resolver is stuck in a loop:
Investigating (8) dpkg [ i386 ] < 1.15.5.6ubuntu4.5 -> 1.16.1.2ubuntu5 > ( admin )
Broken dpkg:i386 Breaks on dpkg-dev [ i386 ] < 1.15.5.6ubuntu4.5 -> 1.16.1.2ubuntu5 > ( utils ) (< 1.15.8)
  Considering dpkg-dev:i386 29 as a solution to dpkg:i386 7205
  Upgrading dpkg-dev:i386 due to Breaks field in dpkg:i386
Investigating (8) dpkg-dev [ i386 ] < 1.15.5.6ubuntu4.5 -> 1.16.1.2ubuntu5 > ( utils )
Broken dpkg-dev:i386 Depends on libdpkg-perl [ i386 ] < none -> 1.16.1.2ubuntu5 > ( perl ) (= 1.16.1.2ubuntu5)
  Considering libdpkg-perl:i386 12 as a solution to dpkg-dev:i386 29
  Holding Back dpkg-dev:i386 rather than change libdpkg-perl:i386
Investigating (9) dpkg [ i386 ] < 1.15.5.6ubuntu4.5 -> 1.16.1.2ubuntu5 > ( admin )
Broken dpkg:i386 Breaks on dpkg-dev [ i386 ] < 1.15.5.6ubuntu4.5 -> 1.16.1.2ubuntu5 > ( utils ) (< 1.15.8)
  Considering dpkg-dev:i386 29 as a solution to dpkg:i386 7205
  Upgrading dpkg-dev:i386 due to Breaks field in dpkg:i386
Investigating (9) dpkg-dev [ i386 ] < 1.15.5.6ubuntu4.5 -> 1.16.1.2ubuntu5 > ( utils )
Broken dpkg-dev:i386 Depends on libdpkg-perl [ i386 ] < none -> 1.16.1.2ubuntu5 > ( perl ) (= 1.16.1.2ubuntu5)
  Considering libdpkg-perl:i386 12 as a solution to dpkg-dev:i386 29
  Holding Back dpkg-dev:i386 rather than change libdpkg-perl:i386
The new version of dpkg requires upgrading dpkg-dev, but it can't because of something wrong with libdpkg-perl. Following the breadcrumb trail back through the log, we find:
Investigating (1) libdpkg-perl [ i386 ] < none -> 1.16.1.2ubuntu5 > ( perl )
Broken libdpkg-perl:i386 Depends on perl [ i386 ] < 5.10.1-8ubuntu2.1 -> 5.14.2-6ubuntu1 > ( perl )
  Considering perl:i386 1472 as a solution to libdpkg-perl:i386 12
  Holding Back libdpkg-perl:i386 rather than change perl:i386
Investigating (1) perl [ i386 ] < 5.10.1-8ubuntu2.1 -> 5.14.2-6ubuntu1 > ( perl )
Broken perl:i386 Depends on perl-base [ i386 ] < 5.10.1-8ubuntu2.1 -> 5.14.2-6ubuntu1 > ( perl ) (= 5.14.2-6ubuntu1)
  Considering perl-base:i386 5806 as a solution to perl:i386 1472
  Removing perl:i386 rather than change perl-base:i386
Investigating (1) perl-base [ i386 ] < 5.10.1-8ubuntu2.1 -> 5.14.2-6ubuntu1 > ( perl )
Broken perl-base:i386 PreDepends on libc6 [ i386 ] < 2.11.1-0ubuntu7.8 -> 2.13-24ubuntu2 > ( libs ) (>= 2.11)
  Considering libc6:i386 -17473 as a solution to perl-base:i386 5806
  Added libc6:i386 to the remove list
Investigating (0) libc6 [ i386 ] < 2.11.1-0ubuntu7.8 -> 2.13-24ubuntu2 > ( libs )
Broken libc6:i386 Depends on libc-bin [ i386 ] < 2.11.1-0ubuntu7.8 -> 2.13-24ubuntu2 > ( libs ) (= 2.11.1-0ubuntu7.8)
  Considering libc-bin:i386 10358 as a solution to libc6:i386 -17473
  Removing libc6:i386 rather than change libc-bin:i386
So ultimately the problem is something to do with libc6; but what? As Steve Langasek said in the bug, libc6's dependencies have been very carefully structured, and surely we would have seen some hint of it elsewhere if they were wrong. At this point ideally I wanted to break out GDB or at the very least experiment a bit with apt-get, but due to some tedious local problems I hadn't been able to restore the apt-clone state file for this bug onto my system so that I could attack it directly. So I fell back on the last refuge of the frustrated debugger and sat and thought about it for a bit. Eventually I noticed something. The numbers after the package names in the third line of each of these log entries are "scores": roughly, the more important a package is, the higher its score should be. The function that calculates these is pkgProblemResolver::MakeScores() in apt-pkg/algorithms.cc. Reading this, I noticed that the various values added up to make each score are almost all provably positive, for example:
         Scores[I->ID] += abs(OldScores[D.ParentPkg()->ID]);
The only exceptions are an initial -1 or -2 points for Priority: optional or Priority: extra packages respectively, or some values that could theoretically be configured to be negative but weren't in this case. OK. So how come libc6 has such a huge negative score of -17473, when one would normally expect it to be an extremely powerful package with a large positive score? Oh. This is computer programming, not mathematics ... and each score is stored in a signed short, so in a sufficiently large upgrade all those bonus points add up to something larger than 32767 and everything goes haywire. Bingo. Make it an int instead - the number of installed packages is going to be on the order of tens of thousands at most, so it's not as though it'll make a substantial difference to the amount of memory used - and chances are everything will be fine. I've filed a patch as Debian bug #657732. I'd expected this to be a pretty challenging pair of bugs. While I certainly haven't lost any respect for the APT maintainers for dealing with this stuff regularly, it wasn't as bad as I thought. I'd expected to have to figure out how to retune some slightly out-of-balance heuristics and not really know whether I'd broken anything else in the process; but in the end both patches were very straightforward.

Pau Garcia i Quiles: HTML5 for everything?

Apparently HTML5 applications are the best thing after sliced bread. HTML5 is the first platform any mobile vendor supports: iPhone, Android, Windows Phone, BlackBerry, Symbian. All of them. Windows 8 is said to promote HTML5 as the preferred application development solution. I used to look kindly at that. But about a month ago I started to get worried: is HTML5 good for everything? Long-lived applications In military, industrial, warehouse management, medical, etc is not rare that bespoke applications are developed and stay in use for many years (and I really mean many: 10, 20 or even more!) with barely an update. It s not rare that those applications only receive very small updates once very 5 years. Those applications, not Angry Birds, are what keeps the world running: troops know what supplies they can count on, iPhones are manufactured, FedEx is able to deliver your package and your doctor is able to check your health. But now that everybody seems to be moving to HTML5 webapps, what happens when my warehouse management application is a webapp and the additions in the newest browsers make the webapp no longer work? Are vain upgrades the future? Say my webapp is released in 2014 and it works fine with Firefox 14.0 and Chrome 26.0, the newest browsers when I release the application in 2014. Fast-forward to 2020 and Firefox 14.0 and Chrome 26.0 do not even install on Windows 10 computer! What s the solution? Should the customer pay for a huge update and redesign to make it work with Firefox 27.1 and Chrome 41.0 in 2020? A virtual machine with Windows 8 and Firefox 14.0? A portable Mozilla Firefox 14.0 on Windows 10 in 2020 to be able to use that line-of-business application that only requires a small update once or twice every 5 years? How are the virtual machine and/or Portable Firefox 14.0 different from or better than a fat client? What s the advantage? I d say none! Native applications usually do not have that kind of problems because APIs are much more stable. You can still run Win16 applications on Windows 7! You don t believe me? We may soon be developing for 76 browsers! While HTML5 may be fine for applications which are updated very often, it makes me feel very uneasy to see it used in environments where applications will be rarely updated, such as SCADAs, warehouse management, control system, medical records, etc. A solution is needed It looks like that choice of technology is going to make those applications much more expensive in the medium and long term, paying for adaptations to new browsers (sorry, I resist to call update or upgrade to something that adds zero value other than being able to run on a newer browser). Or maybe it s about time to define actual HTML5 profiles . ACID3 seems to be too weak of a profile: two very different browsers may pass ACID3 yet a webapp would work with one browser and fail with the other due to bugs, missing features/added features, etc. Something needs to be done.

29 January 2012

Don Armstrong: Changing Grub2's default boot entry and halting

I have a mythtv box which (when working) records television shows for me. As I'm not interested in the vast majority of shows shown on US television, it spends most of it's time off, waiting for a show that I want to record. This requires using nvram-wakeup, and one of the oddities of my machine's bios is that it wants to be rebooted after setting the nvram. [This is likely due to Debian writing to the RTC after the nvram being updated, but not setting the RTC seems stupid.] After the reboot, the machine should halt, and grub should be configured to start the machine normally once the bios starts. As grub2 now supports named default entries, this is fairly straightforward. We create a menu entry like the following in /etc/grub.d/40_custom:
 menuentry 'halt'  
         set saved_entry=0;
         save_env saved_entry;
         load_env;
         halt;
  
make sure that GRUB_DEFAULT="saved" in /etc/default/grub; and set MythShutdownNvramRestartCmd to /usr/sbin/grub-set-default halt:
 mysql mythdb -e "UPDATE settings SET data='/usr/sbin/grub-set-default halt' WHERE value='MythShutdownNvramRestartCmd'";
and viola, the machine now behaves properly with grub2.

28 January 2012

Russ Allbery: Summary of C++ symbols experience

This is a follow-up to my previous post about the way that Debian's library symbols analysis support interacts with C++ libraries, based on further debian-devel discussion. The primary outcome of that discussion was to point me at the pkg-kde-tools utilities for handling maintenance of these files, which solved a bunch of my problems. As before, this was also posted on debian-devel, which is where I expect most of the discussion to be. First of all, for those who haven't explored the pkg-kde-tools infrastructure for this, it looks like the effective process goes something like this:
  1. Do a local build of the package and then generate an initial symbols template for that architecture. So, for example, I did:
        fakeroot debian/rules build install
        pkgkde-gensymbols -plibxml-security-c16 -v1.6.1 -Osymbols.i386 \
            -edebian/libxml-security-c16/usr/lib/*/libxml-security-c.so.*.*
        pkgkde-symbolshelper create -o debian/libxml-security-c16.symbols \
            -v 1.6.1 symbols.i386
    
    This generates an initial symbols file that will work for i386.
  2. Build and upload this version of the package. Now wait for all the buildds to fail (because they will, on probably nearly every other architecture than your local one).
  3. Download all of the build logs and feed them back into pkgkde-symbolshelper so that it can adjust your symbols template for all of the architecture variations. So, for example, I did:
        pkgkde-getbuildlogs
        pkgkde-symbolshelper batchpatch -v 1.6.1 *_unstable_logs/*.build
    
    This will generate rather nice annotated symbols files with appropriate arch lists and using subst tags where appropriately (such as when an argument to a function is a size_t).
  4. Modify your package to use pkg-kde-tools during the build, because subst tags are a really nice way to handle a lot of C++ symbol patterns but #533916 against dpkg-dev is wontfix, so you need the pkg-kde-tools replacement for dpkg-gensymbols. I added a dependency on pkg-kde-tools and then added the (undocumented) pkgkde_symbolshelper add-on to my dh --with flag.
  5. Build and upload the package again. Now it will hopefully build on all architectures.
This is, as you might expect, a fair bit of work, but it does work, and what comes out the other end is a symbols file that works across the current buildd architectures and detects any mistakes by upstream that change the symbols exported. However, it does have some problems, some obvious, and some more subtle. Here's a list of the issues that I see:
  1. It feels like this symbols file is still likely to be fragile. While pkg-kde-tools detects a lot of template instances and marks them optional, I was still seeing a lot of "leaked" symbols from other C++ libraries that my package build-depends on, and which I suspect may also disappear. There's also the problem of optimizers eliminating some things that are inlined; for example, that appears to be the variation that my package sees on arm. And for a substantial C++ library, the symbols file is HUGE. Over 12,000 lines for one of my libraries. It's kind of scary to think of how many fragility landmines are buried under there.
  2. By the nature of this process, it's very sensitive to the current supported architectures. I expect any package with a symbols template built this way stands a good chance of FTBFS on a new architecture. For example, my symbols file has a lot of !armel !armhf patterns, which don't appear to be anything unique to arm except that the C++ compiler behaves slightly differently there due to inlining, and those are the only two architectures in the current set with that pattern.
  3. The lack of dpkg-gensymbols support for subst is very unfortunate, since it's used by the pkg-kde-tools utilities and it's the Right Thing To Do for symbols that take, for example, size_t as an argument. I would strongly encourage the dpkg maintainers to accept the work in #533916 except maybe for the vt work, since the alternative is to generate regexes that are significantly worse at capturing mistakes (since they're going to accept either int or long, rather than only whatever one corresponds to size_t, for example), or to embed a lot of specific architecture restrictions listing, for instance, all 64-bit architectures and guarantee FTBFS for every new architecture.
  4. It's a little hard on the infrastructure, since adding a symbols file requires uploading a package that's guaranteed to fail to build almost everywhere just so that you can get all the build logs to feed into the machinery. I always feel guilty about doing that.
  5. It's still not clear that the benefit is worth the amount of effort, since I expect most C++ libraries to require frequent SONAME changes anyway, which means that the long-term binary compatibility angle of symbols is probably futile. Mostly, all of this is to give you a tool to do ABI checking, which will catch some mistakes but not all of them. And I'm guessing it's going to be substantially more work than maintaining symbols for a C library if one actually does it properly and checks every missing symbol and change in new builds.
I'm going to try this for the Shibboleth packages for a while since I want to see what it's like across a new upstream release, but I'm still very much not convinced that this is a useful use of a packager's time.

Don Armstrong: Using wicd on UCR's WPA network

UC Riverside's wireless network uses WPA-EAP for the encrypted network. [The unencrypted network does a https based browser capture.] Unfortunately, none of the default wicd encryption templates support the precise brand of WPA that the network does, so you have to make your own template. Luckily, wicd makes this fairly simple: Create a new template, say, /etc/wicd/encryption/templates/eap-only, with appropriate contents.
 name = EAP
 author = Don Armstrong
 version = 1
 require identity *Identity passwd *Password 
 -----
 ctrl_interface=/var/run/wpa_supplicant
 network= 
    ssid="$_ESSID"
    key_mgmt=WPA-EAP
    identity="$_IDENTITY"
    password="$_PASSWD"
  
Then tell wicd about this new template by editing /etc/wicd/encryption/templates/active and adding eap-only to the existing list of templates, and restart wicd /etc/init.d/wicd restart. [I'm not sure if restarting wicd is necessary, but it shouldn't hurt.] Finally, configure the network using the appropriate wicd interface as usual.

27 January 2012

Rapha&#235;l Hertzog: People Behind Debian: Josselin Mouette, founder of the Debian GNOME team

Josselin Mouette is one the leaders of the pkg-gnome team, he takes sound technical decisions and doesn t fear writing code to work-around upstream issues. He deserves kudos for the work he has put into packaging GNOME over the years. He can also be very sarcastic (sometimes he even enjoys participating to flamewars on debian lists), and there are quite a few topics where we have long agreed to disagree. But this kind of diversity is also what makes Debian a so interesting place Read on to learn more about the pkg-gnome team, its plans for Wheezy, Josselin s opinion on the GNOME 3 switch, and much more. Raphael: Who are you? Josselin: I am a 31 years old Linux systems engineer. I started in life with physics, which I studied at the ENS Lyon. I started a thesis on experimental and numerical models for optoelectronics, but when it became clear that research was not for me, I abandoned it and accepted a job at the CEA, which holds the largest computing center in Europe. Working on these machines has been the most awesome job ever (except for it being near Paris). After that I worked a bit on system monitoring technologies. I am married, currently living in Lyon, and working for EDF (the French historical electricity company) on scientific workstations using Debian. EDF is using Debian on more than a thousand workstations and holds the fastest Debian supercomputer in the world (200 Tflops), which makes it another obvious place for Debian developers. Raphael: How did you start contributing to Debian? Josselin: I discovered Debian in 1999 while studying at the ENS, which is one of the biggest nests of Debian developers while being a small place, it is producing almost one Debian developer per year on average. After wondering for a while what it could be useful for, hacking on a slink snapshot made me think that it was for, well, everything except for gaming. Later, in 2002, when I was working on optoelectronics computing codes, I started to package them for Debian in order to make them easier to install, for us as well as other labs over the world. I started the NM process, and it was going smoothly but also going to take time. However, at that moment, the frozen-bubble game went out and made quite some buzz. Since I knew a guy who knew the game s developer, he asked me to package it. The package found 3 sponsors in a very short time and was fast-tracked into the archive at a speed that was unseen before. After which the NM process was completed very quickly. At that time, I was a heavy WindowMaker user, but I didn t like the direction the project was taking (actually, I wonder if there was one). GNOME was starting to become attractive, but its packaging in Debian was very ineffective, with many inconsistent packages maintained by people who didn t ever talk to each other some of them didn t speak English, and some of them didn t talk at all. Together with awesome people, among which Jordi Mallach, Gustavo Noronha Silva, JHM Dassen, Ross Burton and S bastien Bacher, we started the GNOME team in 2003, introducing consistent packaging practices, and initiating synchronized uploads. Releasing a completely integrated GNOME 2.8 in sarge was a considerable achievement; proving (together with the Perl team) that a team was the best way to maintain large package sets changed the way people work on Debian.
Proving [ ] that a team was the best way to maintain large package sets changed the way people work on Debian.
Raphael: You re one of the most active contributors of the team which is packaging GNOME for Debian. What would you suggest to a new contributor who would like to help the team? Josselin: There are several ways to contact the team, but the recommended one has always been IRC. We hang on #debian-gnome on the OFTC network, so just come around and ask for us. The real question is what you want to do in the team. Of course, most new volunteers want to help packaging the latest and greatest version of GNOME into unstable as soon as possible, but unless they already have Debian background, this is not the easiest task. Since there are already people working on this, the big packages are usually waiting on dependencies. I used to direct newcomers towards bug triage, but it is a tedious task and I m now convinced that our huge bug backlog will never be dealt with. The most useful thing to do for newcomers now is probably to find a GNOME or GNOME-related package that needs improvement or is lagging behind, and simply try to work on it. You can also come and fix the bugs you find annoying. Find a patch on the GNOME bugzilla, or cook it yourself, propose it, and if it s worthy enough you ll soon get commit access.
Our huge bug backlog will never be dealt with.
At this point I feel worth mentioning that if no one answers in 10 minutes, it doesn t mean that no one will answer in 2 hours, so please stay on the channel after asking. Raphael: There s been some controversy about GNOME 3 and the direction that the project is taking. What s your personal stance on GNOME 3? And what s the position of the pkg-gnome team? Josselin: The controversy is not new to GNOME 3, but the large-scale changes made with it have put it more prominently. The criticism usually boils down to a few categories:
  1. General lack of configurability
  2. Strange design decisions
  3. Red Hat centric development
  4. Hardware requirements
  5. Change resistance
The lack of configuration options has been an ongoing criticism since GNOME 2.0 has decided to rip off most of them. Of course, when the control center was redesigned again for 3.0, there was a surge of horrified exclamations from people who missed their favorite buttons. On this topic, I fully concur with GNOME developers. The configuration option that is useful for you is not necessarily useful for someone else. Of course, sometimes developers go a bit too far, but the general direction is right. At work, we found that only a minority of users actually configure anything on their desktops: they just want something that works to launch their applications. Apple and Google have sold millions of devices by making them the simplest possible and without any configuration. Design decisions are, on the contrary, individual decisions, and each of them, while having reasons behind it, can be questioned. I remember seeing a lot of complaints when the OK and Cancel buttons were reversed in dialog boxes, something that nobody questions anymore. GNOME Shell is full of such changes; some are easy to get accustomed with, some others just make eyebrows raise. The most obvious example is the user menu in GNOME 3.2, which contains an entry to configure your Google account, but no entry to shutdown the computer. Both decisions were taken independently, each of them with (good or bad) reasons, but the result is simply ridiculous. The default configuration in Debian will contain an extension to make it a bit better, but on the whole we don t intend to diverge from the upstream design, on which a lot of good work has been done.
On the whole we don t intend to diverge from the upstream design, on which a lot of good work has been done.
Point 3 is more complex. Red Hat being the company spending the most on GNOME, it is obvious that their employees work on making things work for their distribution. An example is the recurring discussions about relying on system services that are currently only implemented by systemd. Since there is a lot of (mostly unjustified) resistance against systemd in Debian, and since it won t work on kFreeBSD anyway, someone needs to develop an alternative implementation of these services for upstart and sysvinit. Everything is in place for someone else to do the job but it has to be done, and this can be frustrating. Especially since it can also be hard to integrate changes needed for other distributions . Hardware requirements are mostly a consequence of the previous criticism: there s hardware that most distributions just don t want to bother supporting. We ve seen it in squeeze with the introduction of a hard dependency on PulseAudio. The Debian GNOME team (together with the Gentoo maintainers) made this dependency optional, carrying heavy patches, in order to cover the cases where it does not work. Now that it has gained more maturity, making this effort obsolete, the new tendency is to require 3D acceleration. For various reasons, it is not available to everyone . On this matter, the position of the Debian GNOME team has always been to support as much different configurations as possible with reasonable effort. Thanks to efforts from the incredible Vincent Untz, upstream supports a so-called fallback mode , which is the GNOME panel from 2.x with a lot of its bugs fixed. We intend to support this mode for as long as reasonably possible in Debian, possibly even after upstream ends up dropping it. However, other applications are going to require 3D because GStreamer is moving to clutter too, affecting video playback performance on non-accelerated systems . For epiphany this is not a problem; only embedded video will be affected. But for totem, this is a major issue; because of that we will probably keep totem 3.0 in wheezy. Finally, there is a natural human tendency to dislike change (I have it too), and it applies a lot to desktop users habits. Needless to say a change of such a scale as introducing GNOME Shell can trigger reactions. However, I don t think it is reasonable, because of this resistance, to keep gnome-panel 2.x in Debian. This would be a lot of work on obsolete technology, and would prevent the upcoming removal of a lot of deprecated libraries. This time is much better spent improving gnome-panel 3.x in Debian and keeping the fallback mode great. One of the change that was made in Debian was to make it easier to find, being available as GNOME Classic directly from the login manager, instead of having to find it in an obscure configuration panel. In all cases, I would recommend to actually try GNOME Shell for a few hours before ditching it. I had never been accustomed to a new environment as quickly ever before.
In all cases, I would recommend to actually try GNOME Shell for a few hours before ditching it.
Having seen several of my GDM patches reverted without a warning, I know we are not finished with carrying patches in Debian packages.
Scientific workstations are a non-trivial example, since there is a measurable effect of using 3D in the window manager on heavy 3D applications.
On the other hand, on accelerated systems, this feature should end up improving performance a lot. Raphael: What are your plans for Debian Wheezy? Josselin: The first goal of the GNOME team is, of course, to provide again a great desktop environment to work on. For wheezy it will probably be based on GNOME 3.4. There also needs to be some work on package management interfaces. Upstream bases everything on PackageKit, but it is not as featureful as the aptdaemon Ubuntu technology. If I have time, I would also like to improve HTTP proxy support, since currently it is based on a stack of terrible hacks. Raphael: If you could spend all your time on Debian, what would you work on? Josselin: Obviously I would like to make GNOME in Debian even better. That would imply working on underneath dependencies (what we now like to call plumbing) to make sure everything is working great. This would also imply working more as GNOME upstream to make it more suitable for our needs. I would also work on large-scale improvements on the distribution, like conditional recommends which I d love to see implemented , or automatic build-dependency generation. I would also work on the installer to make it better for desktops machines. The idea is to automatically install language packs, or glues between two packages when both packages are installed. Raphael: What s the biggest problem of Debian? Josselin: The obvious answer is the same as the one most people you interviewed before gave: not enough members in core teams. A lot of developers join Debian to work on a small number of pet packages, and don t necessarily want to be involved with existing teams. It is probably still not obvious enough that the primary way to start contributing to Debian is to join an existing team. But if there is one thing that is preventing Debian from gaining more momentum now, it is a completely different one: the too short support timeframe. 3 years is really not enough for corporate users. One year to migrate from one version to another is too short, and it is not possible to skip a release. It is definitely possible to change that with reasonable effort: the long-term support after 3 years doesn t have to cover the same perimeter as the short-term one. For example, we could upgrade the kernel to the version in the current stable release, and stop fixing all non-remote security holes. The important thing is to cover the most basic needs: companies are ready to take the risk of having less support if it allows skipping a version, but not the risk of having no support at all. And even more important is to say that you do something. Red Hat says they support a release for 10 years, but of course after 5 years the supported perimeter is extremely small.
3 years [of support] is really not enough for corporate users.
Long-term support will not magically fix all problems in Debian, but it will bring more corporate users into the picture. And with corporate users come paid Debian developers, who can work on critical pieces of the system. Debian was built on the synergy between individuals and companies, and in recent years perhaps as a reaction against what happened with Ubuntu we ve kind of forgot the latter. A lot of individuals have joined the project, and they are actively working, for example, on shortening the release cycle, which goes against the interest of professionals. We should embrace again such users and developers, and that means adapting to the current needs of larger entities. Raphael: You re the maintainer of python-support, a packaging helper that was competing with python-central. Both helpers are now deprecated in favor of dh_python2. Does this mean that the situation of Python in Debian is now sane? Or are there remaining problems? Josselin: dh_python2 (and the Python3 version, dh_python3) has a sane enough design. It fixes a lot of issues in python-central and also python-support, at the expense of somehow reduced functionality for developers. However, just like the previous tools, it merely works around design mistakes in the Python interpreter. For example it is not possible to split binary modules, pure-Python modules and byte-compiled modules in different directory trees, like Perl does although PEP 3147 introduces a way to do so. There is still no sane and standardized way to deal with module versions. There is no difference made between the module (which is a part of language semantics) and the file containing it (an information which depends on the implementation). Developers heavily rely on introspection features and make assumptions based on the implementation, that make it impossible to work around problems with module files. Such problems are not restricted to Python. Those who fought against Ruby gems could tell even worse stories. While introducing GObject introspection packages in Debian (they can be used in JavaScript and Python to provide modules based on GObject libraries), I was pleased to see a clear distinction between file and module, but I was again struck by the fact you are not forced to declare API versions in your Python/JS code. In all cases, there is no reliable way to detect runtime dependencies in a given Python or JavaScript file, which leaves the maintainer to declare them by hand, and of course, often be wrong about them. Add to that the fact that most errors cannot be detected before runtime. For all these reasons, and while still being fond of Python for scripts and prototyping, I ve become really skeptical of using purely interpreted languages to write real applications. Some GNOME developers are moving away from Python and JavaScript, mostly towards Vala; I can only approve of that move and hope the same happens to other projects. Raphael: Is there someone in Debian that you admire for their contributions? Of course there is the never-sleeping, never-stopping, Michael Biebl who can upload a whole GNOME release in a single week-end. But there are a lot of awesome people who make Debian something that simply works. I could talk about Cyril Brulebois from the X strike force, Julien Cristau from the release team, Sjoerd Simons for his sound advice and work on plumbing, Luca Falavigna who is so fast at processing NEW, to quote only a few of those I work with frequently. And of course, Jordi and Sam for their humor.
Thank you to Josselin for the time spent answering my questions. I hope you enjoyed reading his answers as I did. Note that you can find older interviews on http://wiki.debian.org/PeopleBehindDebian.

Subscribe to my newsletter to get my monthly summary of the Debian/Ubuntu news and to not miss further interviews. You can also follow along on Identi.ca, Google+, Twitter and Facebook .

8 comments Liked this article? Click here. My blog is Flattr-enabled.

26 January 2012

Aigars Mahinovs: Zombies. Wait, don t run away, yet

I never got the craze for the zombie this and zombie that. I saw the premise as thinner than vampire storyline which in my book only had a good day with Buffy franchise. But then I got Kindle and run out of stuff to read and decided to pick up something less lively and more shambling. And you know what I found there are a few great books hiding behind the zombie label. I read them in the worng order, so to help you guys out, here is the way I would recommend to check out if the zombie genre is interesting to you: Start with World War Z: An Oral History of the Zombie War. It is written as a historical book documenting the memories of people that have survived the world wide zombie apocalipse is a series of escalating short stories. This is a perfect bathroom or bus reading. Each story focuses on how people react (or fail to react) to a radical change in their environment. Zombies are a tool to expose the human condition for this book. The next you could read (or you could even skip this if it get a bit boring for you) is The Zombie Survival Guide: Complete Protection from the Living Dead this is a problem solving book, it basically takes a problem ( ZOMBIES! ) a goal ( I want to survive! ) and describes the logical steps needed to reach the goal in the context of the problem. It is very educational on the issue of problem solving, especially with unusual problems. But it can get boring for people if so, don t give up, but just skip this book for now and move on the the best of them all The third is a trilogy, two books are already published, the third is coming soon first books is Feed (Newsflesh, Book 1) and the second is Deadline (Newsflesh, Book 2). This is a rare gem of modern literature. Basically this is a book set in the world long time after a zombie outbreak (with a good reason why and how it happened) where people have already lived a few decades with the problema and the kids are very used to it. So the book is written from a perspective of a young woman and her brother, who were born after Rising and are now into one of the most dangerous profession journalism. :) Early on they become the first bloggers to be picked to follow a presidential election campagn (book was written before Obama campaign really took off) and it only get wilder from there. I really liked these books. So much so, that I will now explore the other zombie fiction books to see if there possibly is something good there as well. Any recommendations?

Russ Allbery: Do symbols files make sense for C++?

I also posted this to debian-devel, which is where the discussion will be, but I wanted to post it on my journal as well so that others who may be interested will see it on Planet Debian. For those not familiar with Debian, this concerns the implications of a new feature for how Debian tracks shared library dependencies. I'm currently working on the Policy modification to document (and recommend) use of symbols instead of shlibs, but I'd only personally used symbols with C libraries. Today I decided that I should try adding a symbols file to a C++ library, particularly if I'm going to recommend everyone do it. I tried this exercise with xml-security-c, which is, I think, a reasonably typical C++ library. Not the sort of core C++ library that would sit at the center of the distribution, but a random software package that's in Debian because other things use it. The experience was rather interesting, and I ended up uploading the new version without a symbols file and continuing to just use shlibs. That's for the following reasons:
  1. The generated symbols file was HUGE. Hundreds of lines. This is a marked difference from the typical C symbols file, which is of quite manageable size. Some of that is that the library provides a lot of different classes, but some of it is that C++ just generates a lot of exported symbols. There's no way that I could do what I would do with a C library and understand those symbols, why they're there, and whether they are likely to have changed between revisions.
  2. Generating a reasonable symbols file was a pain. Generating an unreasonable symbols file that just contains all of the mangled symbols is largely mechanical and uninteresting, but that symbols file doesn't seem to me to convey useful information. So I did some scripting to translate the symbols back with c++filt, and add (c++) tags, and then try to understand what I was looking at and figure out whether I should sort the symbols list because the default sort is by mangled name, which is meaningless. This is a rather unappealing process. It's not particularly difficult, but it's very awkward and feels like it's missing vital tools.
  3. The resulting symbols file is incomprehensible to someone without strong knowledge of C++. It's full of opaque entries that don't make sense to the non-C++ programmer, which I suspect is a substantial number of people who package C++ libraries for Debian. I know enough C++ from school that I can evaluate security fixes, make simple patches, and review upstream changes, and I think that's all that should be needed to package things for Debian. But I'm deeply uncomfortable producing a symbols file on my own that contains entries for things that I know nothing about and cannot evaluate when they've last changed, like "non-virtual thunk to FooClass::~FooClass@Base".
  4. Once I had a symbols file that resulted in a successful build and that I could have uploaded, I started thinking about how I was going to maintain it. With a C program, I would change the symbols file versions when the underlying function implementation changes in a way that may not offer eqiuvalence, similar to bumping shlibs. I realized that I was going to have no idea when that happened, and the only way that I would maintain the symbols file would be to either trust upstream to maintain ABI equivalence and therefore only change the symbols file when upstream changes the SONAME, or not trust upstream to maintain ABI equivalence and therefore change all the versions with each new upstream release. That gives me exactly the same semantics as a shlibs file, so what's the point in having a symbols file?
  5. The exported symbols of the library contained many symbols that obviously weren't really from that library, but instead were artifacts of the C++ compilation process, things like instantiations of std::vector. Do those go into the symbols file? Do they change from architecture to architecture? If they disappear again, is that actually an ABI break? How do I know? It's all very mysterious, and while shlibs provides the same semantics as just ignoring this, at least I'm not then including in the package data, generated by me, things that I'm just blindly ignoring.
I came away from this experience thinking that I should revise the Policy amendment to say that symbols files are really for C libraries and for C++ libraries with either a tightly maintained symbol export list or maintained by a C++ expert, and that most C++ library maintainers should just not bother with this and use shlibs, bumping the shlibs version or not based on their impression of how good upstream is at maintaining ABI equivalence. But that feels like a result contrary to what I had previously thought was the intended direction, so I wanted to ask the Debian development community as a whole: am I missing something? Are these symbols files actually useful? Am I missing some trick to make them useful?

24 January 2012

Charles Plessy: Strange Megumi

It is since two days that all my emails, @debian.org included, are in parked somewhere between the sender and my MX, as my server broke at a moment where I strictly had no time (dealine for a grant application). It is an OpenRD Ultimate with a second hand SSD. I was quite happy to have an ARM system available to test some packages like T-COFFEE, but it looks like I will have to do without. Apparently, the machine permanently reboots. I did not manage to access to the USB serial port; there are no 10 seconds where the USB port is not disconnected / reconnected. Both ethernet port LEDs blink together synchronously with the disconnections. Removing the drive did not solve the problem. The SSD itself is fine, and I did not find in the logs hints for software problems. Strange coincidence, the last log file that was modified is daemon.log, which indicates the connection of so-called megumi-PC to our unprotected, but to my knowledge never visited, wireless network. After this, nothing.

Next.

Previous.