Having had a little discussion with Stefano Zacchiroli aboutdebian/changelog editing and bug number completion, Idecided I'll make the 'apt-listbugs list' command a bit moreuseful.It now has two new features.--severityall option to specify all severities, and you canspecify the current version with PACKAGE/VERSIONsyntax. These functionalities should be available in the0.0.79 upload.It also uses the shiny new BTS SOAPinterface, and no longer relies on merkel.debian.org, thanks to help from Don Armstrong.To find a bug that is not yet closed on apt-listbugs 0.0.79, you'll ask for it like this: apt-listbugs list -s all apt-listbugs 0.0.79

Those of you that look at photos on my blog may have noticed that I recently switched from a self-hosted Gallery2 installation to the popular photo-sharing site Flickr.

I initially decided to try Flickr for a simple reason: digikam's image upload to Gallery2 has never worked for me, and I hoped that its Flickr upload would work (it did).

I had heard good things about Flickr from friends and colleagues, so I gave it a try.

First Impressions

Almost the first thing I noticed about Flickr was that there are people there that are interested in the same things I'm interested in. I discovered the Anabaptist History group. I've been contacted by a person in North Dakota that is interested German-speaking Russians, as my ancestors were. There are groups about Debian, classic-style black & white photography on digital cameras, and really just about any topic. These groups contain both discussions and a photo "pool" to which any group member can post.

In one sense, Flickr is social networking via photography.

Flickr also seems to be a company with a clue. It has tons of open APIs, permitting everything from flickrfs (mount flickr as a FUSE filesystem) to countless uploaders, mashups, scripts, etc. Despite a couple of recent high-publicity controversies, they seem to be laid back overall and haven't been turned into a corporate drone by Yahoo. There's an RSS feed for just about everything, too.

What is Flickr?

Depends on what you want it to be. It could be that place where you post your cell phone's snapshots, or a place to share photos with family in private, or just a place to find other's work.

I found sharing and talking with others to be infectious. I went from never getting comments on photos on my Gallery2 site to wondering what people will think -- and looking at their photos too.

Let's look at its different aspects in more detail.

Uploading Photos

You can upload photos manually with a HTTP form, or you can use any of the numerous uploading tools. Most of these will let you scale down your images on the fly, should you wish to. Most will also let you assign tags and write descriptions as they're being posted, as well as define who can view the photos. Uploaders are available for every common platform.

Flickr has a AJAX tool called the Organizr that helps you assign tags, add photos to groups, put them in sets, etc. It's fairly nice.

Sharing Photos

Making photos available to the public is easy. You can just them them a link to your photostream. You can also tag photos and give people a link to photos with a specific tag. And you can create sets -- photos around a specific theme.

Security

Each photo can be made visible to: just you, just your contacts marked "family", just your contacts marked "friend", any of your contacts, or anyone at all. This can be defined per photo.

On a global basis, you can define whether you make EXIF information visible on the site, default security for new uploads, who can post comments, who can post photo notes or add tags, and who can use the "all sizes" button.

You can also control how much of your profile to make public. Settings range from making everything public to concealing even your email address.

For sets, you can also send out a "Guest Pass" (GP). A GP lets someone that is not a Flickr member see non-public photos. I use GPs to share family photos, so that family doesn't have to create a Flickr account to see them. GPs can be tied only to sets.

Groups

Flickr groups appear to have been designed for discussions, but I find them most attractive because each group has its own dedicated photo pool. Members of the group can add their photos to the pool. Some groups have tens of thousands of photos, and others a few dozen. It's an interesting way to find images. There are a lot of group management tools as well.

Free vs. Paid Accounts

Free account holders can upload up to 100MB of photos per month. There is no cap on bandwidth consumed by views of photos or management of them. Free users can create up to 3 sets and put photos in up to 10 groups.

The photostream for free account holders will show only their 200 most recent photos. Additionally, the "All Sizes" button will max out at the Large size, which generally will be 1024 pixels along the longest side.

Paid accounts are $25/year and include unlimited upload bandwidth (10MB max per photo though), unlimited photostream size, unlimited sets, and removal of all ads on the site. Additionally, the original upload is made available to you. If you have enabled the All Sizes button, it's also made available to the public. One nice thing is that with a free account, all that data is saved, even if it isn't available. So if you later upgrade to Pro, it's all there.

Prints

Flickr sells prints of your photos for $0.15 each. As with viewing, you can designate who can purchase prints of your photos. I ordered a test batch and they turned out very nice -- approximately the same quality as Shutterfly, which is one of the better consumer-quality labs. They are a bit pricier than Shutterfly when buying in bulk, though the convenience of all the Flickr uploaders may make up for that.

Annoyances

There are a few things that bug me.

One is that the "All Sizes" button is either on or off. Once I upgraded to pro, suddenly my original size images were available. There is no way with a pro account to limit image availability to the 1024xwhatever size like there is with free -- either you give people the medium size only (500xwhatever) or you give them everything. Those that want to have the originals on Flickr generally upload twice: once in original size, kept private; and another resized to 1024xwhatever, made public.

It is also complex for some people to join Flickr. Several of our family and friends had trouble with the signup process, since you first have to create a Yahoo account, then a Flickr one, which can use a different username. It's a nice flexibility, but overly complex for people that just want to sign up quickly. Guest Passes have received rave reviews however -- with the one caveat that the guest pass URLs aren't bookmarkable (they set a cookie and then redirect to the real target).

There is also no built-in way to see all the new photos in the groups you're a member of. However, each group's pool has an RSS feed so I have just subscribed to them with bloglines.

Overall

Overall I am happy with Flickr. I will be migrating my Gallery2 install to it, and will not miss having to maintain Yet Another PHP Script on my server. The best part about Flickr is the community, which has already taught me about mpix.com, which I'll write about tomorrow.

I’m just sitting in the night venue of Debconf7 and do several tasks in the same moment, that includes: * Getting kind of a spanish course on IRC in #debian-qt-kde * Get some news from friends in .ch via MSN (there were several unweathers and other troubles) * Think about tdbackup which is the topic of this blog entry. So what is tdbackup? It is just another backup implementation but it should work kind of distributed. I manage several servers and in that farm are 3 backup servers with a tape drive, every sunday i mount parts of the other machines via NFS and write them with tar to the tape. Until I did that by hand because there was no implementation which fits my needs unfortunately. This is the point where tdbackup jumps in. I want to create a GUI which edits an ini file in the form of: [backupmachine 1] mount = bla1:/home mount = bla2:/home tdbackup will then distribute a specifically generated configfile to each backuphost and a little python script which is called by cron and mounts the file systems, write some information into mysql and generate warning messages via e-mail if the tape is almost full. So, I will be able to sleep better because the stuff is auto-documented in a way. I’m still thinking about how exactly i want to implement that, if there are any interested people who would like to join that project, please just tell me.

So, I'm back from Holidays, to find we now have a new Government in Ireland. (An unexpected choice of government, of which more later.) We were camping in Fouesnant, in Brittany; Family & Friends might like to see the holiday snaps here). Camping in France went well. With three children aged 7, 4 and 2, there is only so much you can expect from a family holiday. Like it or not, you have to mind the toddler fairly consistently, and the older kids would like your full attention too. This is a holiday ''for'' the kids; plan also a break ''from'' the kids later (a long-weekend, with kids at grandparents, for example). This was our first time camping abroad, and its well recommended. While we went touring with a camper van with the two older children two years ago, this is different in that we were at the campsite for almost two weeks: the kids got to meet and play with others in a way they couldn't before. Also, there were good facilities at the site in Fouesnant, and the swimming pools in particular were appreciated (Caveat emptor though: there were no lifeguards or supervision. We went off-peak, and there were apparently two accidents in the time we were there: a broken kneecap and chin when people came down the slides too soon after others and collided). It was the first time the children (a) had pocket money to spend, and (b) were able to walk to a (campsite) shop to spend it. Lessons were learnt in the finiteness of pocket money, especially by Caoimhe . The weather was fairly good: two wet days (though not overly so, compared to what we're used to in Galway); enough to give the 'authentic camping experience'. And for the children especially, just warm enough. Long drives South were not an option with young kids, and we wanted to see Carnac and Brittany in general. We went touring to two places: Mont St. Michel, on the way down, Carnac, and locally around Quimper and Concarneau. At seven, Cormac is beginning to be old enough to enjoy this, but Sadhbh at 2 is too young, and easily aggravated. Caoimhe can appreciate some of it, such as the well recommended Breton museum in Quimper, seeing the local costumes and clothing is just her style at the moment. We're looking forward to going back when the kids are older, or we can enjoy touring on our own. The camp site is dominated by English / Dutch / Irish. It is still term-time in Britain and Ireland, which meant the children on the site was full of 2-4 year-olds, with a shortage of playmates for Cormac. Next time, synchronise with English school times, etc. We went during Cormacs term-time because it was much cheaper (1000 Euro) than high-season. If Cormac was having difficulty in school things would have been different, as it is, we had no qualms in doing so. Things will probably be different next year with Caoimhe also in school at the same time, and we would go later. Suffice it to say I think synchonisation of school timetables is a bad idea, as it has led to these huge price differentials and parents taking their kids from school. As a friend with experience of the French school system points out, they've spent the last 40 years desynchronising their timetables ... So would we go again, or buy our own tents, etc? possibly. The brochure for next year arrived already, with large discounts for booking early ... Tags holiday, family, france, brittany, cormac, coimhe, sadhbh

After my brother's wedding, a bunch of us Goerzens got together out at our creek. We had a hot dog roast, some smores, and then eight of us remained to camp out at the creek. That included Terah, Jacob, and me.

My uncle loves camping and has organized camping at the creek many times over the years. This was the first time it worked out for me to join them in quite awhile. Jacob slept well overnight, but woke up 5 minutes late -- 6:35. Never mind that he got to bed hours later than his usual time, he still woke up at the same time as always. He's funny that way.

My uncle brought with him supplies for making pancakes over the campfire in the morning. They were great. After breakfast, it was time to take down the tent and get back home to shower before going to church.

Now, a word about Terah and camping here. Terah had only gone camping with me once since we were married, and that was two years ago. This time, we had a larger tent, warmer weather, and most importantly, a complete lack of raccoon fights all around us.

I had visited a Coleman Factory Outlet store in Wichita in the morning, before the wedding, and bought a new Coleman Crestline tent (the one on the left above) and some other supplies. It's a very nice tent with a lot of ventilation options and plenty of little pockets to store things. Jacob slept in his car seat (which he likes to do) in the tent with us.

I think we might not have to wait two years before going camping again. Though I'm sure Terah will correct me if I'm wrong on that.

The source-searching system I was talking about previously is progressing slowly. So far I've synced the source to Etch to my local machine, total size 29Gb, and this evening I've started unpacking all the source. I'm still in the "a" section at the moment, but thanks to caching I should be able to re-sync the source archive and unpack newer revisions pretty speedily. The big problem at the moment is that the unpacking of all the archives is incredibly slow. Still I do have one new bug to report aatv: Buffer overflow in handling environmental variables.. That was found with:

rgrep getenv /mnt/mirror/unpacked   grep sprintf

(A very very very slow pair of greps. Hopefully once the unpacking has finished it will become faster. ha!) The only issue I see at the moment is that I might not have the disk space to store an unpacked tree. I've got 100Gb allocated, with 29Gb comprised of the source. I'll just have to hope that the source is less than 70Gb unpacked or do this in stages.) I've been working on a list of patterns and processes to run, I think pscan, rats, and its should be the first tools to run on the archive. Then after that some directed use of grep. If anybody else with more disk space and connectivity than myself be interested I can post the script(s) I'm using to sync and unpack .. Failing that I'll shut up now.

So, I'm happy to see that commercial wiki provider WetPaint now supports OpenID for logging into their site. It's great to have more wiki services supporting this single sign-on standard. A couple of nits, though: first, I logged in with my normal OpenID, http://evan.prodromou.name/ , which delegates to http://certifi.ca/evan . WetPaint's OpenID service used the delegated URL, which is just wrong. They need to fix that. The delegate is the "backend", and users should be able to change it over their lifetime. If relying parties store that backend URL instead of the "main" OpenID URL, that undermines the process. Second, they have a signup page after the validation, and it has a lot of fields that could be pre-populated using the OpenID Simple Registration extension. I shouldn't have to type in my preferred user name, real name, or email address. That information is already available through my OpenID provider. Other than that, it seems to work OK. I'm glad that it exists at all, but I wish it was a smoother user experience. tags: wetpaint wiki openid sso

Wikispaces and SourceForge I haven't already mentioned the announcement that SourceForge has implemented Wikispaces wikis for every project on their site. I think that's great news for the wiki community, and probably a good thing for SourceForge users. I feel churlish mentioning it, but it has to be said: with hundreds of available WikiEngines, the vast majority of which are Open Source, and many of which are hosted on SourceForge, why did SourceForge go with a commercial service provider? It doesn't seem like they've justified that decision very well. tags: wikispaces sourceforge wiki announcement

For a long time I've been vexed by the gender bias of computing towards men. It's clear that computing has multiple issues, ranging from its perception to the nature of its society. As organizer of computing conferences, I care about ensuring that participation is meritocratic and inclusive.
As various debates have taken place over blogs in the past on women in computing, I have stayed silent. Not because I don't care, but because I really didn't know what to say. I try to understand, but at the end of the day I'm a man, I tend to misunderstand important points, and I suspect my contributions would come across as at best naïve, and more likely, patronising. So many conversations between men and women on this topic end up in mutual uncomprehension. Better to stay silent than be a fool.
This is why I'm very grateful for Jeni Tennison's analysis of women in computing. It is the most helpful blog post I have read recently on the topic. Perhaps it's because Jeni's British, and we share some of the same cultural assumptions and constraints. But more importantly, it's because her article builds bridges to my experience and understanding.
An argument of femininity Rather than simply appealing to tribes of men and women, Jeni places the discussion in terms of feminine and masculine traits. Like many men, I identify with certain attitudes and preferences more closely allied to my feminine side. By not excluding me, her wording helped me view the points she unfolded in terms that my own experience, however poor, could identify with. I've often found contemporary computing communities to be suffocatingly macho. There's a strong pressure to succeed in terms of attention from the chattering bloggers, privileged inner circles and the capacity for vicious excoriation of those who fall from grace.As I grew up through school and university, I held these sorts of social forms in contempt, as well as occasionally suffering at their hands. Yet now, bafflingly, I find myself under the impression I need to belong and compete in these spheres.Self-efficacy One of Jeni's central points is concern about one's own self-efficacy. This was the real revelation to me. I tend to assume most capable people I meet are aware of their own capability. Jeni turned this on its head for me when she wrote

... look at how Ruby on Rails is marketed. A big play is made of how easy it is. But if a language or framework is easy then people with low self-efficacy can’t win: if they manage to do something with it then they haven’t really achieved very much because anyone can do it; if they don’t manage to do something with it then they’re complete idiots. I’m not saying that we should advertise languages or frameworks as being hard, because obviously that can put people off as well, but a recognition of the barriers that people might face may, in a strange way, make them more approachable.

As Jeni brings her article to a close, it's with some shock and shame that I get the punchline loud and clear: "this isn't about you."It's about empathy, inclusivity and selflessness. Human qualities that are unrestricted to either gender.

Does anyone fancy hacking for an hour on a neat little feature for Sound Juicer, as a way of learning the code base a little? This shouldn't take more than an hour or so, and would be good for the world. When Musicbrainz can't find the track listing for a CD, it automatically queries FreeDB and will attempt to return something from the garbled nonsense that FreeDB generally contains. The user cannot tell the difference, unless they notice that the track listings are badly formatted, the album artist is incorrect, or the encoding is wrong. The user will not notice any trivial issues, fix any obvious problems and then rip the CD, or just give up now if the encoding is wrong. This is bad. What should happen is that Sound Juicer should subtly point out to the user if the data is proxied from FreeDB, so that they can import it into MusicBrainz where more data can be added and verified. What I'd like to see is a little pane appear if a CD is from FreeDB (SJ knows this already) with a link to the relevant import URL, so the user can quickly import it once for the benefit of everyone else who rips the album in the future. Does anyone fancy hacking this? As I said, it sounds like an hour of coding for someone who can code GTK+ but hasn't touched SJ before. NP: This Book Is About Words, Ahmad Szabo

While Subversion is for many the source code revision system of choice, I've never stopped long with it myself. Why? Because it's mostly a fixed-up CVS, and doesn't fix the really gnarly problem with revision control: merging.Mark Shuttleworth has written an insightful little article on the key aspects of merging source code. Some of the points he makes underpin my choice of revision control system.For several years now I've been a very contented user of Bazaar, in its first arch-based incarnation, and now in its latter form. The key reason has been ease of merging.While most articles praising Bazaar are reports of open source development, I've also found it very handy for small in-house teams working on web applications.There are very often multiple arcs of development going on at once, and the ease of merging makes it easy to ensure that long-running development arcs don't get ridiculously far from the main trunk of development. Bazaar's merging also makes it easier for lead developers to police the merging of the code base.Using Bazaar with Rails and other toolsWorking mostly with Rails, one repeatedly runs up against the assumption that Subversion is the golden path for revision control. This is perhaps a little telling about the level of participation most Rails-based projects have reached.The tools story for a Bazaar user is both good and bad.The good

• Deployment: the Capistrano deployment system has support for Bazaar. I've written previously about the technique I use to deploy my Rails apps.
  
• Collaboration: Trac is a wonderful integrated bug database, wiki and source code browser. Although it's a little hard to find, the trac-bzr integration works very well, and happily moves Trac out of the Subversion-based assumption that you can only have one repository root per project.
  

The bad

• Continuous integration: with Expectnation, we've had to roll our own continuous integration system ('system' being rather a grand word for a shell script on a cronjob...) Most of the CI systems currently popular seem to assume Subversion.
• Editor/IDE integration: some early work exists, but it's early days. I've not yet found anything making me want to leave the command line.
  

It's now shortly before 5am (Argh!) and I'll be leaving home now. If everything goes as planned this will be my journey:

5:20 - Bus to station
5:43 - Train to Hanover
6:21 - Train to Cologne
9:21 - Train to the Airport (CGN)
10:50 - Plane to EDI
Around noon - Arrival at Teviot.

If I don't show up, something went wrong.

Bye bye for now ... *yawn*

For some days at Debconf, I've been somewhat stressed and frustrated. You see, the first DebCamp/DebConf I took part in (Oslo), I was really productive - Besides meeting lots and lots of people and having a great time, during DebConf3 I learnt a scary lot about the project in many aspects and, although I was already a DD, it really decided me to get more involved.
When we were at Brazil, I had a great time as well, but (code-wise, bugs-wise and so on) I was much less productive - I started peeking at the processes that involve running such a strange and intensive conference, I was more active talking with other Latin Americans, and so on.
So, for DebConf5, I was a full orga team member - Man, did that make me feel powerful! Of course, as there was so much orga work to be done, I basically didn't hack into anything. During the day it was organizational work, and during the night time-after-22:00 (how do you people live without a night!?) it was sauna. What a great invention of the Finns! After getting relaxed by sauna, nights were socially intensive
About DebConf 6... Well, it was hectic and chaotic. It is probably the most stressful two-week period (two week? Two months at least! Well, anyway) period of my life. It is the only Debconf so far I don't even remember having had much social interaction. But still, for reasons I fail to understand, I am really happy we held DebConf in Mexico. It is one of the best, most exciting and (hopefully!) unrepeatable experiences in life.
What about this year? Well, I took a couple of steps back, and while I'm formally still part of the always-great orga team, there are way too many things I don't know or can answer. Partially, I blame that impossible dialect of English they speak in the UK - Please, next time you feel a bit tired, lock yourself in a room with six hurried English and Scottish people, add a couple of Germans to the mix (no, I don't have problem with the other accents spoken in there), and try to get the general lines discussed. Impossible.
As I took a couple of steps back, I thought I would be able to hack a bit. There were some great ideas of things to work on - But so far, I've practically not started with any! I wanted to do some QA rounds in the pkg-perl repository, kill a couple of long-standing bugs, and then there is thismetainit idea we have been discussing... So we held the MetaInit BoF, and Nomeata has been actively hacking it - But I must recognize I have done little besides documenting and cleaning a bit a minor module of it. The second day I was here, I started working on something that was completely outside of my plan (hacking up a subsystem for Pentabarf to help the video team, as I previously complained). Just after we finished with it, DebConf properly started - So I've been basically taping the talks, reviewing the files, in the orga and video meetings... Until yesterday, I was quite frustrated not to have been able to work a bit on the couple of things I wanted to do. I have not yet been able even to work on three things I need for my Real Life, which still counts, even at DebConf.
But then, yesterday I read a post by Wouter - Man, you are completely right, and I must keep this in mind at all times: DebCamp to me isn't about being productive, so much as it is about meeting people and getting to know the environment. Well, I've terribly even failed to get to know the environment (I'll reserve some time on Thursday or Friday, I hope), but so far this has been quite fun, and completely out of what I usually do. And, even without me considering this, DebConf has been quite productive for me in ways I didn't expect - I've met and had a couple of beers with the pkg-perl group (and today I'll meet with the pkg-ruby-extras group), have plotted several things with the Spanish-Speaking group (which, by the way, has tremendously grown since we were at Oslo), I even started hacking a bit my way into Pentabarf, which will be very useful once I get back to work!
Anyway, in short: Wouter, you made me reconsider, you made my day - Thanks! And now, I'm a happy man again :)

One of the most interesting thing about slippery slopes is how you never seem to be standing at the top of them, looking down. The slope seems fine at the top, and it’s only once you start down it that you realise this could end up with some broken limbs. When Ubuntu was formed, Debian were having a debate about how to treat GFDL documentation. It was their opinion that the GFDL was inherently non-free, and they’ve since taken steps to remove all such licensed documentation from their main distribution. We took a more pragmatic approach, and decided that it maintained the spirit of freedom, and thus we continue to this day to ship that documentation in our main distribution. A similar discussion resulted in the handling of data files such as graphics, icons, fonts, etc. We decided that such things didn’t necessarily need to ship with corresponding source code, as frequently they don’t have any such thing or when they do, it’s just as easy to modify the data file directly. The slope didn’t seem at all slippery back then. Then came the issue of firmware, binary blobs in the kernel which are uploaded into a flash (or similar) chip in the hardware. Could we distribute these? On one hand, these blobs have always existed, they just used to be in ROM in the hardware; the move to firmware doesn’t change that. On the other hand, they’re machine code and if we had the source, we could improve the hardware as well. And what if we didn’t distribute them? Our users would be stuck without being able to use some fairly (to them) critical parts of their computer. In the end, the argument that firmware isn’t inherently any less free on the disk than in the ROM won, so we opted to continue to ship it. Perhaps that slope is a bit slippery, but we’ve got a good foothold. Of course, at that point somebody notices the binary “Hardware Access Layer” in the Atheros WiFi card driver. It’s not firmware, it’s run on the host processor, and is separate to “comply with FCC law”. (The ipw3495 driver has a binary daemon that allegedly performs the same legal function). Again, if we don’t distribute that, a large section of laptop users will not be able to use their WiFi cards. A compromise was reached; because the driver is necessary we’d ship it, but in a special restricted component that makes it absolutely clear that it’s not completely free. Users could choose to remove that component and any packages from it, to keep their system untainted. Ok, foothold wasn’t as strong as we thought; tumbled a bit, but we’re definitely on solid ground now! That’s what we thought, anyway. Unfortunately it seems that there’s a point a little bit lower down the slope which has a fantastic vista. The views from there are just incredible, people are saying, much prettier than where we are now. The only trouble is that we’re not sure there’s a foothold down there, if we try for the better view, we could end up broken at the bottom. I’m talking, of course, about the NVIDIA binary X driver. (Some reports/blogs/etc. indicate we’re also considering the ATI fglrx driver, this isn’t true – that driver doesn’t support AIGLX, so it’s not being considered.) We’ve shipped this driver in our restricted section, but not enabled it by default. It’s been there for people who want it to switch on, if they know how, but the default driver has always been the free (albeit obfuscated) one in the Xorg distribution. The problem is that users do not need this driver, they can get decent enough 2D graphics support from the free(ish) driver. In the long term, they may even get decent 3D graphics support from the nouveau driver effort. What’s the problem then? Simple, other operating systems use the 3D GPU to make the desktop seriously beautiful. If Linux doesn’t catch up and do the same, then we’ll be considered obsolete again. And just to drive the point home, some of our Linux friends shipped similar support in their last releases. They don’t enable the NVIDIA binary driver, but this means that a large percentage of their user base can’t get the bling without manual hackery. We needed a way to catch up with both the commerical operating systems and other Linux distributions; we have a policy of not doing our own software development, but only packaging what others have developed, so the only way for us to get ahead was to package something that others wouldn’t. Which brings us back to the NVIDIA binary driver. If we install that by default, we’ll be bringing a 3D desktop to more people. And we’ll gain a step ahead of the other distributions. Will our users care? To be brutally honest, I think the answer is no! In fact, I suspect our users will largely love us for this decision. Most probably already install the NVIDIA driver anyway, because they think it’s better, or because (sadly, like me) they have a card combination not supported by the free one. Will this make any difference to the effort to get NVIDIA to free up the driver, or at least the specs? Sorry, but to be honest again, I don’t think it’ll make one little difference. Linux distributions have been refusing to install it for years, and yet NVIDIA haven’t budged in their position. Perhaps a new tactic is required. Maybe if we do install it, we’ll be more likely to be chosen by OEMs as we can actually support the hardware they install. Then later, we may be able to actually affect their decision as to what hardware they install, and maybe then NVIDIA will pay attention. Will this change the perception of Ubuntu in the Linux developer community? I’m not sure about this one, I think that those who already feel strongly about the distribution of binary drivers are probably already pretty grumpy at us distributing things like the Atheros and ipw3495 drivers. I suspect this will change the opinion of a lot of people who’ve been on the fence until now, probably equal in both directions. Will we be able to sleep at night? Despite all of the above, personally I still think that installing and using the nvidia driver by default, when the nv driver would do, is the wrong decision. If the nv driver doesn’t work, I’m willing to accept the nvidia driver being used; provided that there’s some message informing the user what’s happened, why it has happened, and which alternate graphics cards they can purchase if they aren’t willing to accept a non-free binary driver. If the nv driver is good enough for 2D, I would prefer that we instead disabled the 3D desktop effects for this group of users (by default). A similar message could explain why this is disabled, again which alternate graphics cards provide this by default, but also provide a button for the user to enable it if they wish. While we should make the correct moral decision for the defaults, we shouldn’t stand in the way of users who wish to make a different decision for themselves. Later as the nouveau driver becomes stable, we may be able to activate 3D support for nvidia users by default. I think that preserves our current foothold, we’d only activate it if there is no free alternative. Where there is, we’d be educating users about why they may wish to consider alternatives to NVIDIA in future, while at the same time not getting too much in their way if they want to see the better view down the slope. Unfortunately it’s not my decision, and I suspect that the lure of the bling will win out. With any luck, we’ll find a foothold there, and the fallout of doing so won’t be too bad. I’m just worried that once we compromise on this, we’ll start compromising on other things … would we replace Firefox with a non-free web browser that rendered web pages “better”? The slippery slope only gets steeper from here on …

For the last couple of months, both at the Ubuntu Developer Summit in Mountain View and on the #upstart IRC channel, we’ve been discussing the changes we want to make to upstart for the Feisty Fawn release of Ubuntu. This will ship with a version of upstart based on the 0.3 series (it may end up getting called 0.5 before release); the primary goal for this are to have an init system that is suitable for general standalone list in any Linux distribution. I’ll be giving a talk at linux.conf.au 2007 in Sydney with that aim, I hope to persuade at least one other major Linux distribution that it’s the right solution. A complete list of the specifications and bugs being targeted for the 0.3 release can be found in Launchpad. The rest of this post will introduce some of the shiniest new things. Writing Jobs Upstart takes care of starting, supervising and stopping daemons itself; unlike in the init script system where you have to write code to do that yourself, often using a helper like start-stop-daemon. All you need to is give the path to, and arguments for, the binary you wish to be started.

exec /usr/bin/dbus-daemon

Some jobs, especially quick tasks, will usually be written as shell scripts. To save having to write a separate file and invoke it, you can include shell script code directly in the job file instead of using the exec stanza.

script
    echo /usr/share/apport/apport > /proc/sys/kernel/crashdump-helper
end script

Usually it’s not sufficient to just start a binary and wish it well; you frequently need something to be run before it is started to prepare the system, and sometimes something after it terminates to clean up again. For these purposes, additional snippets of shell code can be given – to be run before the binary is started, and after it has finished. Unlike init scripts, these do not need to start or stop the daemon itself; that’s done automatically based on the exec stanza.

pre-start script
    mkdir -p /var/run/dbus
    chown messagebus:messagebus /var/run/dbus
end script
post-stop script
    rm -f /var/run/dbus/pid
end script

For consistency, executables may be specified with pre-start exec and post-start exec instead of shell scripts as above. It’s sometimes useful to be able to run something after the binary has been started; for example, you may wish to attempt to connect to the daemon to determine whether it is ready to serve requests. post-start script or post-start exec can be used to this.

post-start script
    # wait for listen on port 80
    while ! nc -q0 localhost 80 </dev/null >/dev/null 2>&1; do
        sleep 1;
    done
end script

It’s also useful to be able to notify a daemon that it may be about to be stopped, or delay it for a while. pre-stop script or pre-stop exec can be used for this.

pre-stop script
    # disable the queue, wait for it to become empty
    fooctl disable
    while fooq >/dev/null; do
        sleep 1
    done
end script

Events Events are now quite a bit more detailed than in previous versions; they’re still named with simple strings that are up to the system sending the event, but they can now include arguments and environment variables which are passed through to jobs being started or stopped as a result.

initctl emit network-interface-up eth0 -DIFADDR=00:11:D8:98:1B:37

This command will now output all of the effects of this event, and will not terminate until the event has been fully handled inside upstart. Events such as the above can be used by jobs that examine the event arguments and environment within their script:

start on network-interface-up
script
    [ $1 = lo ] && exit 0
    grep -q $IFADDR /etc/network/blacklist && exit 0
    # etc.
 end script

or matched directly in the start on and stop on stanzas:

start on block-device-added sda*

The events generated by job state changes have also changed. Previously both jobs and events shared the same namespace, which not only caused confusion but actually caused some problems when one accidentally named a job after an event. The two primary events generated are now simply called started and stopped; they inform you that a job is fully up and running, or fully shut down again. The name of the job is received as an argument to this event.

start on started dbus

The started event is not emitted until the post-start task (described above) has finished; so the post-start task can delay other jobs from starting because they can’t yet connect to the daemon. Likewise the stopped event is not emitted until after the post-stop task has finished. The other two events emitted by a job are special; they are the starting and stopping events. The reason they are special is that the job is not permitted to start or stop until the event has been handled. This means that if you have a task to perform when your database server is stopped, but before it’s actually terminated, it’s as simple as:

start on stopping mysql
exec /usr/bin/backup-db.py

MySQL won’t be terminated until the backup has finished. This is especially useful for daemons that depend on each other, for example HAL needs DBUS, it shouldn’t be started until DBUS is running and DBUS should not be stopped until HAL has been terminated. All the HAL job needs is:

start on started dbus
stop on stopping dbus

Likewise if tomcat is installed, Apache should not be started until tomcat is running; and tomcat should not be stopped until apache has been terminated. All the tomcat job needs is:

start on starting apache
stop on stopped apache

Failure Nothing goes smoothly all of the time, sometimes tasks the job runs will fail, or the daemon itself will die. As well as providing the ability for a crashed daemon to be automatically restarted, upstart ensured that other jobs are notified with a special failed argument to the stopping and stopped events.

start on stopped typo failed
script
    echo "typo failed again :-("   mail -s "typo failed" root
end script

And if any job started or stopped by an event fails, it’s possible to discover that the event itself failed.

start on network-interface-up/failed

States While tasks such as configuring a network interface, or checking and mounting a block device are usually performed as a result of events; services are more complicated. Services normally need to be running while the system is in a certain state, not just when a particular event occurs. Therefore upstart allows you to describe arbitrarily complex system states by referring to events that define their changes. For example, many services should be running only while the filesystem is mounted, and at least one network device is up. We have events to indicate the changes into and out of these dates, we just need to combine them:

from fhs-filesystem-mounted until fhs-filesystem-unmounted
and from network-up until network-down

The until operator defines a period between two events, the and operator ensures we’re within both of these periods. Perhaps we need to be running while any display manager is:

from started gdm until stopping gdm
or started kdm until stopping kdm

Or maybe we only want to be run if a network interface comes up before bind9 has been started:

on network-interface-up and from startup until started bind9

These “complex event configurations” can appear in any job file; and any job file itself can serve as a reference for other jobs. They will be started and stopped at the same time as the named job:

with apache

Omitting the exec or script stanza from a job file means that it simply defines a state that can serve as a reference for others. As such, the multiuser state is simply a job file that defines it. As an added bonus, these states can still have pre-start, post-stop, etc. scripts.

There are many articles on the Interwobble telling you how to set the process title on Linux; they all concentrate on the problem of placing an arbitrarily long string in argv[0] to report status information in the process list. But what if argv[0] is fine, and what you want to do is remove the following arguments from the process list? Perhaps they contain sensitive information, or perhaps they’re just likely to be surprising. A fictional example may look like this (I’ve added the $ to demonstrate a later problem):

11234 ?        S      0:00 some-program --username=foo --passphrase=bar$

How not to do it The first thing you’ll probably try is just setting those arguments in the argv list to NULL. For example, with code like this:

for (i = 1; i < argc; i++)
    argv[i] = NULL;

That won’t work because that array is just a list of pointers to the real area in which the arguments are stored, and is local to your main function. Modifying it has no effect what ps and similar see. The next thing you might try, as a C programmer, is setting the first character of each argument to a NULL byte. It’s not unreasonable to assume that these strings are likely to be NULL-terminated. You might use code like:

for (i = 1; i < argc; i++)
    strcpy (argv[i], "");

But that doesn’t seem to work either, in fact, what you get in the process list is something rather odd and unexpected!

    11234 ?        S      0:00 some-program  -username=foo  -passphrase=bar$

All that’s done is lost the first character of each argument. In frustration, you might try overwriting the entire argument with NULL bytes, e.g.

for (i = 1; i < argc; i++)
    memset (argv[i], '\0', strlen (argv[i]));

Checking the process list, that will have seemed to have worked…

    11234 ?        S      0:00 some-program                                $

On closer inspection though, it’s not perfect. You’ll notice that the line ends with spaces in place of where the arguments used to be. If the number or length of arguments you’re trying to hide is particularly long, you can end up with strange blank lines in ps output. Not ideal. Why this happens To understand why this happens, we need to understand how the kernel reports the command line to ps. ps reads the /proc/PID/cmdline file, with contains the entire command line as a single character array, with each argument terminated by a NULL byte. Looking at it with od -t c, we see something like:

0000000   s   o   m   e   -   p   r   o   g   r   a   m  \0   -   -   u
0000020   s   e   r   n   a   m   e   =   f   o   o  \0   -   -   p   a
0000040   s   s   p   h   r   a   s   e   =   b   a   r  \0

When we changed just the first character to a NULL byte, we ended up with:

0000000   s   o   m   e   -   p   r   o   g   r   a   m  \0  \0   -   u
0000020   s   e   r   n   a   m   e   =   f   o   o  \0  \0   -   p   a
0000040   s   s   p   h   r   a   s   e   =   b   a   r  \0

ps is just assuming that \0 represents any argument gap, so outputs an ordinary space character every time it encounters one. This is why the first character of each argument appeared to be replaced by a space, and the rest were still printed. When we overwrite the arguments, we ended up with:

0000000   s   o   m   e   -   p   r   o   g   r   a   m  \0  \0  \0  \0
0000020  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0
0000040  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0

Writing all those zeros didn’t change the length of the command line character array, so ps got all those NULLs and output a space character for each one, assuming it was an argument break. What we need to is decrease the length of this character array, so ps only sees the first argument. How to do it The kernel maintains internal pointers to the start and end of the arguments array in the process address space. When you ask for the cmdline file in /proc, it replies with all of the characters between these two positions and a length of the difference. The kernel doesn’t provide us with any method of moving this pointer, so we’d initially appear to be stuck. However there turns out to be a very nasty solution. In order to support processes changing their title, the kernel has to support the arguments possibly overrunning into the environment space. If it thinks this has happened, it concatenates the environment space onto the end of the buffer and returns the entire area with the length set to the length up to the first NULL byte. All we need to do is fool the kernel into thinking we’ve overrun into the environment space, it’ll then just call strlen on the arguments list and return only the first argument; exactly what we want! What technique does the kernel use for deciding whether or not the arguments have overrun into the environment space? It checks whether the final character in the argument space is NULL or not. Thus paradoxically, to hide the argument list from ps, we remove the NULL …

if (argc > 1)  
    char *arg_end;
    arg_end = argv[argc-1] + strlen (argv[argc-1]);
    *arg_end = ' ';
 

The check against argc is necessary, as we don’t want to remove the NULL character on the end of argv[0] as that would cause the first environment variable to be inadvertently concatenated. Now when we look at ps, we see just the program name:

    11234 ?        S      0:00 some-program$

And even when we look in /proc/PID/cmdline, we only see the program name and no additional arguments.

0000000   s   o   m   e   -   p   r   o   g   r   a   m  \0

One of my pet dislikes are those people that pay lip-service to a particular problem, such as Accessibility, Internationalisation or Usability, without actually thinking about them. My favourite example can be found at the Trafford Centre in Manchester, England. Somebody there has clearly realised that accessibility is a problem, and ensured that every single sign has Braille added to it so that partially-sighted or blind people are able to read what they say. They’ve only been paying lip-service to the problem though, and not thinking about it. The evidence? The following sign, mounted directly on a door, with Braille on it: Do not stand directly in front of this door.

Just prior to the “State of the Coffee Cup” talk on Sunday, Tom Marble sent a quick invitation to Steve Langasek, Sam Hocevar and myself to have a chat with Simon Phipps about the CDDL and some of the concerns that have been raised on the debian-legal list lately – particularly about choice of venue. Unfortunately in the ten or fifteen minutes between the mail and the meeting, we couldn’t find Sam, but we managed to grab Don Armstrong to join in instead. It’s a couple of days later, and I’m going from memory, not notes, so this is paraphrasing. Simon’s primary concern was one of predictability – Debian accepted the Mozilla Public License back when it was released (and prior to Mozilla and other software tending to get dual licensed under the GPLv2) which is what the CDDL was based on and contains almost the exact same terms. Simon’s comment was that if you’ve accepted that then, reviewing new licenses from scratch and coming up with different results really makes it hard to work with you, and worse makes it look like you’re basing your conclusions more on who proposed the license rather than what the license actually says. Don and Steve seemed to mostly appreciate that point, but still thought that choice of venue was harmful and unnecessary, and that it’d be nice if Sun – as pretty much the primary organisation currently behind any of these licenses – would take some steps to reduce that harm: limiting choice of venue to when both parties in a dispute are multinational, for example. Simon’s concern there was that updating the CDDL would require a lot of effort internally – getting buy in from all the groups in Sun using the CDDL, as well as getting the wording changes approved from the appropriate legal advisors, and convincing the lawyers of Sun’s commercial partners that the changes weren’t going to be a problem later; and committing to all that work when a new versions of the GPL is about to be released and it might be a better thing to work on getting the relevant software available under that, or even the GPLv2. On the other hand, Simon thought that making (binding) statements about Sun’s interpretation of the license seemed to (eventually) work reasonably well for the DLJ, so might be something that can be done again for this issue. And while that only counts for Sun, we can fairly easily ask if other authors of CDDLed stuff agree with Sun’s interpretation for their software, and only worry about possible problems if it turns out they don’t. That’s pretty much where we ended up I think, with Simon suggesting that we work out what exactly needs to be clarified with Tom, and then get it passed back up to Simon to get signed-off on. And hopefully that’s what’ll happen over the next little while. :) (The somewhat more interesting question of whether we can combine CDDL’ed OpenSolaris (in particular Solaris libc) with GPLed userspace tools such as dpkg to produce a Debian OpenSolaris distro is still up in the air – Sam’s currently waiting on a response from the FSF legal team on one of the questions that concerns Debian, aiui; but they’re currently flat out working on GPLv3 as you might imagine. Oh well – in the meantime, there’s always Nexenta!)

In a comment on my previous blog post a question was asked as to who benefits when customers are able to use cheap electricity. The answer is that the electricity company benefits the most! When electricity sources such as wind and solar power which can vary in capacity are used the electricity would be cheaper when there is adequate or excess supply. The electricity would be cheaper then to encourage customers to use power hungry devices at those times rather than when the electricity supply is reduced. The larger the capacity of back-up power plants such as gas-fired plants the larger the overall cost of the system (having extra peak-load capacity that is unused most of the time is a waste of money). However the company that I once worked for on a project related to this was not an electricity company. I can’t name the company due to the confidentiality agreement but if the project ever goes into production I’ll blog about it. Recently some of the major investment banks have been focussing on how climate change affects business. I think that with these developments there will be a lot of new investment in environment related technologies. Probably if a few people started work on an embedded Linux box for scheduling power use they could have a good chance of getting some investment. The ideal feature list would include control and monitoring over the Internet, the ability to schedule operations based on power price (received from the grid via a technology similar to X10) and local conditions (how charged the batteries are in your photo-voltaic system). It would control devices via standard X10 modules (which can control the power to the device) as well as directly interfacing with machines that need to be turned on all the time. I would be happy to offer more suggestions via private email to anyone who is interested in implementing this.

Share This