There’s been some controversy generated over my use of the terminology of “Organic” and “Non-Organic” Open Source. Asa Dotzler noted that it wasn’t Mozilla’s original intent to “make a distinction between how Mozilla does open source and how others do open source”. Nessance complained that he didn’t like the term “Non-Organic”, because it was “raw and vague - is it alien, poison, silicon-based?” and suggested instead the term “Synthetic Open Source”, referencing a paper by Siobh n O Mahony, ” What makes a project open source? Migrating from organic to synthetic communities”. Nessance referenced a series of questions and answers by Stephen O’ Grady from Red Monk, where he claimed the distinction between the two doesn’t matter. (Although given that Sun is a paying customer of Red Monk, Stephen admits that this might have influenced his thinking and so he might be “brainwashed” :-). So let’s take some of these issues in reverse order. Does the distinction matter? After all, if the distinction doesn’t matter, then there’s no reason to create or define specialized terminology to describe the difference. Certainly, Brian Aker, a senior technologist from MySQL, thinks it does, as do folks like me and Amanda McPherson and Mike Dolan; but does it really? Are we just saying that because we want to take a cheap shot at Sun? Well, to answer that, let’s go back and ask the question, “Why is Open Source a good thing in the first place?” It’s gotten to the point where people just assume that it’s a good thing, because everybody says it is. But if we go back to first principals maybe it will become much clearer why this dinction is so important. Consider the Apache web server; it was able to completely dominate the web server market, easily besting all of its proprietary competitors, including the super-deep-pocketed Microsoft. Why? It won because a large number of volunteers were able to collaborate together to create a very fully featured product, using a “stone soup” model where each developer “scratched their own itch”. Many, if not most, of these volunteers were compensated by their employers for their work. Since their employers were not in the web server business, but instead needed a web server as means (a critical means, to be sure) to pursue their business, there was no economic reason not to let their engineers contribute their improvements back to the Apache project. Indeed, it was cheaper to let their engineers work on Apache collaboratively than it was to purchase a product that would be less suited for their needs. In other words, it was a collective “build vs. buy” decision, with the twist that because a large number of companies were involved in the collaboration, it was far, far cheaper than the traditional “build” option. This is a powerful model, and the fact that Sun originally asked Roy Felding from the Apache Foundation to assist in forming the Solaris community indicates that at least some people in Sun appreciated why this was so important. There are other benefits of having code released under the Open Source license, such as the ability for others to see the implementation details of your operating system — but in truth, Sun had already made the Source Code for Solaris available for a nominal fee years before. And, of course, there are plenty of arguments over the exact licensing terms that should be used, such as GPLv2, GPLv3, CDDL, the CPL, MPL, etc., but sometimes those arguments can be a distraction from the central issue. While the legal issues that arise from the choice of license are important, at the end of the day, the most crucial issue is the development community. It is the strength and the diversity of the development community which is the best indicator for the health and the well-being of an Open Source project. But what about end-users, I hear people cry? End users are important, to the extent that they provide ego-strokes to the developers, and to the extent that they provide testing and bug reports to the developers, and to the extent that they provide an economic justification to companies who employ open source developers to continue to do so. But ultimately, the effects of end-users on an open source project is only in a very indirect way. Moreover, if you ask commercial end users what they value about Open Source, a survey by Computer Economics indicated that the number one reason why customers valued open source was “reduced dependence on software vendors”, which end users valued 2 to 1 over “lower total cost of ownership”. (Which is why Sun Salescritters who were sending around TCO analysis comparing 24×7 phone support form Red Hat with Support-by-email from Sun totally missed the point.) What’s important to commercial end users is that they be able to avoid the effects of vendor lock-in, which implies that if all of the developers are employed by one vendor, it doesn’t provide the value the end users were looking for. This is why whether a project’s developers are dominated by employees from a single company is so important. The license under which the code is released is merely just the outward trappings of an open source project. What’s really critical is the extent to which the development costs are shared across a vast global community of developers who have many different means of support. This saves costs to the companies who are using a product being developed in such a fashion; it gives choice to customers about whether they can get their support from company A or company B; programmers who don’t like the way things are going at one company have an easier time changing jobs while still working on the same project; it’s a win-win-win scenario. In contrast, if a project decides to release its code under an open source license, but nearly all the developers remain employed by a single company, it doesn’t really change the dynamic compared to when the project was previously under a closed-source license. It is a necessary but not sufficient step towards attracting outside contributors, and eventually migrating towards having a true open source development community. But if those further steps are not taken, the hopes that users will think that some project is “cool” because it is under an open-source license will ultimately be in vain. The “Generation Y”/Millennial Generation in particular are very sensitive indeed to Astroturfing-style marketing tactics. Ok, so this is why the distinction matters. Given that it does, what terms shall we use? I still like “Organic” vs “Non-organic”. While it may not have been intended by the Mozilla Foundation, the description in their web page, “only a small percentage of whom are actual employees [of the Mozilla Foundation]“, is very much what I and others have been trying to describe. And while I originally used the description “Projects which have an Open Source Development Community” vs “Projects with an Open Source License but which are dominated by employees from a single company”, I think we can all agree these are very awkward. We need a better shorthand. When Brian Aker from MySQL suggested “Organic” vs “Non-Organic” Open Source, and I think those terms work well. If some folks think that “Non-Organic” is somehow pejorative (hey, at least we didn’t say “genetically modified Open Source” :-), I suppose we could use Synthetic Open Source. I’m not really convinced that is any much more appetizing, myself, however. So what would be better terms to use? Please give me some suggestions, and maybe we can come up with a better set of words that everyone is happy with.

There’s been some controversy generated over my use of the terminology of “Organic” and “Non-Organic” Open Source. Asa Dotzler noted that it wasn’t Mozilla’s original intent to “make a distinction between how Mozilla does open source and how others do open source”. Nessance complained that he didn’t like the term “Non-Organic”, because it was “raw and vague - is it alien, poison, silicon-based?” and suggested instead the term “Synthetic Open Source”, referencing a paper by Siobh n O Mahony, ” What makes a project open source? Migrating from organic to synthetic communities”. Nessance referenced a series of questions and answers by Stephen O’ Grady from Red Monk, where he claimed the distinction between the two doesn’t matter. (Although given that Sun is a paying customer of Red Monk, Stephen admits that this might have influenced his thinking and so he might be “brainwashed” :-). So let’s take some of these issues in reverse order. Does the distinction matter? After all, if the distinction doesn’t matter, then there’s no reason to create or define specialized terminology to describe the difference. Certainly, Brian Aker, a senior technologist from MySQL, thinks it does, as do folks like me and Amanda McPherson and Mike Dolan; but does it really? Are we just saying that because we want to take a cheap shot at Sun? Well, to answer that, let’s go back and ask the question, “Why is Open Source a good thing in the first place?” It’s gotten to the point where people just assume that it’s a good thing, because everybody says it is. But if we go back to first principals maybe it will become much clearer why this dinction is so important. Consider the Apache web server; it was able to completely dominate the web server market, easily besting all of its proprietary competitors, including the super-deep-pocketed Microsoft. Why? It won because a large number of volunteers were able to collaborate together to create a very fully featured product, using a “stone soup” model where each developer “scratched their own itch”. Many, if not most, of these volunteers were compensated by their employers for their work. Since their employers were not in the web server business, but instead needed a web server as means (a critical means, to be sure) to pursue their business, there was no economic reason not to let their engineers contribute their improvements back to the Apache project. Indeed, it was cheaper to let their engineers work on Apache collaboratively than it was to purchase a product that would be less suited for their needs. In other words, it was a collective “build vs. buy” decision, with the twist that because a large number of companies were involved in the collaboration, it was far, far cheaper than the traditional “build” option. This is a powerful model, and the fact that Sun originally asked Roy Felding from the Apache Foundation to assist in forming the Solaris community indicates that at least some people in Sun appreciated why this was so important. There are other benefits of having code released under the Open Source license, such as the ability for others to see the implementation details of your operating system — but in truth, Sun had already made the Source Code for Solaris available for a nominal fee years before. And, of course, there are plenty of arguments over the exact licensing terms that should be used, such as GPLv2, GPLv3, CDDL, the CPL, MPL, etc., but sometimes those arguments can be a distraction from the central issue. While the legal issues that arise from the choice of license are important, at the end of the day, the most crucial issue is the development community. It is the strength and the diversity of the development community which is the best indicator for the health and the well-being of an Open Source project. But what about end-users, I hear people cry? End users are important, to the extent that they provide ego-strokes to the developers, and to the extent that they provide testing and bug reports to the developers, and to the extent that they provide an economic justification to companies who employ open source developers to continue to do so. But ultimately, the effects of end-users on an open source project is only in a very indirect way. Moreover, if you ask commercial end users what they value about Open Source, a survey by Computer Economics indicated that the number one reason why customers valued open source was “reduced dependence on software vendors”, which end users valued 2 to 1 over “lower total cost of ownership”. (Which is why Sun Salescritters who were sending around TCO analysis comparing 24×7 phone support form Red Hat with Support-by-email from Sun totally missed the point.) What’s important to commercial end users is that they be able to avoid the effects of vendor lock-in, which implies that if all of the developers are employed by one vendor, it doesn’t provide the value the end users were looking for. This is why whether a project’s developers are dominated by employees from a single company is so important. The license under which the code is released is merely just the outward trappings of an open source project. What’s really critical is the extent to which the development costs are shared across a vast global community of developers who have many different means of support. This saves costs to the companies who are using a product being developed in such a fashion; it gives choice to customers about whether they can get their support from company A or company B; programmers who don’t like the way things are going at one company have an easier time changing jobs while still working on the same project; it’s a win-win-win scenario. In contrast, if a project decides to release its code under an open source license, but nearly all the developers remain employed by a single company, it doesn’t really change the dynamic compared to when the project was previously under a closed-source license. It is a necessary but not sufficient step towards attracting outside contributors, and eventually migrating towards having a true open source development community. But if those further steps are not taken, the hopes that users will think that some project is “cool” because it is under an open-source license will ultimately be in vain. The “Generation Y”/Millennial Generation in particular are very sensitive indeed to Astroturfing-style marketing tactics. Ok, so this is why the distinction matters. Given that it does, what terms shall we use? I still like “Organic” vs “Non-organic”. While it may not have been intended by the Mozilla Foundation, the description in their web page, “only a small percentage of whom are actual employees [of the Mozilla Foundation]“, is very much what I and others have been trying to describe. And while I originally used the description “Projects which have an Open Source Development Community” vs “Projects with an Open Source License but which are dominated by employees from a single company”, I think we can all agree these are very awkward. We need a better shorthand. When Brian Aker from MySQL suggested “Organic” vs “Non-Organic” Open Source, and I think those terms work well. If some folks think that “Non-Organic” is somehow pejorative (hey, at least we didn’t say “genetically modified Open Source” :-), I suppose we could use Synthetic Open Source. I’m not really convinced that is any much more appetizing, myself, however. So what would be better terms to use? Please give me some suggestions, and maybe we can come up with a better set of words that everyone is happy with.

Brian Aker dropped by and replied to my previous essay by making the following comment:

I believe you are hitting the nail on the organic vs nonorganic open source. I do not believe we have a model for going from one to the other. Linux and Apache both have very different models for contribution but I don t believe either are really optimized at this point. Optimization to me would lead to a system of less priests and more inclusion.

I made an initial reply as comment, and then decided it was so long that I should promote it to a top-level post. I assume that when Brian talks about “organic open source” what he means is what I was calling an “open source development community”. Some googling turned up the following definition from Mozilla Firefox’s organic software page: “Our most well-known product, Firefox, is created by an international movement of thousands, only a small percentage of whom are actual employees.” This puts it in contrast with “non-organic” software, where all or nearly all of the developers are employed by one company. (And anyone who proves talented at adding features to that source base soon gets a job offer by that one company. By that definition we can certainly see projects like Wine, Mysql, Ghostscript (at one time), and others as fitting into that model, and being quite successful. There’s nothing really wrong with the non-organic software model, although many of them have struggled to make enough money when competing with pure proprietary softare competitors, with MySQL perhaps being the exception which proves the rule. In most of these cases, though, the project started more as an organic open source, and then transitioned into the non-organic model when there was a desire to monetize the project and/or when the open source programmers decided that it would be nice if they could turn their avocation into a vocation, and let their hobby put food on the family table. Solaris, of course, is doing something else quite different, though. They are trying to make the transition from a proprietary customer/supplier relationship to trying to develop an Open Source community and what John s candidate statement pointed out is that they weren t really interested in creating an organic open source developer community at all, but they wanted the fruits of an open source community with plenty of application developers, end-users, etc., all participating in that community. We don t have a lot of precedent for projects who try to go in this direction, but I suspect they are skipping a step when they try to go to the end step without bothering to try to make themselves open to outside developers. And by continuing to act like a corporation, they end up shooting themselves in the foot. For example, the OpenSolaris license still prohibits people from publishing benchmarks or comparisons with other operating systems. Very common in closed-source operating systems and databases, but it discourages people from even trying to make things better, both within and outside of the Open Solaris core team. Instead, they respond to posts like David Miller s with Have you ever kissed a girl? . (Thanks, Simon, for that quote; I had seen it before, but not for a while, and it pretty well sums up the sheer arrogance of the Open Solaris development team.) So while Linux may not be completely optimized in terms of less priests and more inclusion, at least over 1200 developers contributed to 2.6.25 during its development cycle. Compared to that, Open Solaris is positively dominated by high priests and with a you may not touch the holy-of-holies attitude; heck, they won t even allow you to compare them to other religions without branding you a heretic and suing you for licensing violations!

Brian Aker dropped by and replied to my previous essay by making the following comment:

I believe you are hitting the nail on the organic vs nonorganic open source. I do not believe we have a model for going from one to the other. Linux and Apache both have very different models for contribution but I don t believe either are really optimized at this point. Optimization to me would lead to a system of less priests and more inclusion.

I made an initial reply as comment, and then decided it was so long that I should promote it to a top-level post. I assume that when Brian talks about “organic open source” what he means is what I was calling an “open source development community”. Some googling turned up the following definition from Mozilla Firefox’s organic software page: “Our most well-known product, Firefox, is created by an international movement of thousands, only a small percentage of whom are actual employees.” This puts it in contrast with “non-organic” software, where all or nearly all of the developers are employed by one company. (And anyone who proves talented at adding features to that source base soon gets a job offer by that one company. By that definition we can certainly see projects like Wine, Mysql, Ghostscript (at one time), and others as fitting into that model, and being quite successful. There’s nothing really wrong with the non-organic software model, although many of them have struggled to make enough money when competing with pure proprietary softare competitors, with MySQL perhaps being the exception which proves the rule. In most of these cases, though, the project started more as an organic open source, and then transitioned into the non-organic model when there was a desire to monetize the project and/or when the open source programmers decided that it would be nice if they could turn their avocation into a vocation, and let their hobby put food on the family table. Solaris, of course, is doing something else quite different, though. They are trying to make the transition from a proprietary customer/supplier relationship to trying to develop an Open Source community and what John s candidate statement pointed out is that they weren t really interested in creating an organic open source developer community at all, but they wanted the fruits of an open source community with plenty of application developers, end-users, etc., all participating in that community. We don t have a lot of precedent for projects who try to go in this direction, but I suspect they are skipping a step when they try to go to the end step without bothering to try to make themselves open to outside developers. And by continuing to act like a corporation, they end up shooting themselves in the foot. For example, the OpenSolaris license still prohibits people from publishing benchmarks or comparisons with other operating systems. Very common in closed-source operating systems and databases, but it discourages people from even trying to make things better, both within and outside of the Open Solaris core team. Instead, they respond to posts like David Miller s with Have you ever kissed a girl? . (Thanks, Simon, for that quote; I had seen it before, but not for a while, and it pretty well sums up the sheer arrogance of the Open Solaris development team.) So while Linux may not be completely optimized in terms of less priests and more inclusion, at least over 1200 developers contributed to 2.6.25 during its development cycle. Compared to that, Open Solaris is positively dominated by high priests and with a you may not touch the holy-of-holies attitude; heck, they won t even allow you to compare them to other religions without branding you a heretic and suing you for licensing violations!

Previously, my journal was managed by Movable Type. Originally, Six Apart seemed like a cool company, but I should have known better than to trust anything that wasn't open source. They started screwing with the license, it was never packaged properly, and then of course it became far too painful to upgrade. I've been wanting to fix this for some time, and have occasionally poked at other blog software. Serendipity looked like a winner for a while, but then I spent two days fighting with it and while I got it to mostly look like what I want, that was demoralizing. I hate spending large quantities of time fighting with someone else's software to get it to do what I want, particularly when I then have to write plugins and additions to get what feels to me like basic functionality. It would be one thing if I had time and inclination to get involved as a developer, but usually what Web 2.0 software developers want to pursue and what I care about are almost entirely at odds. During my recent vacation, I finally realized that I just wanted something that used all the infrastructure I already have. I have a macro language that I like for generating web pages, I have a page design that I like; the journal one isn't bad, but it doesn't match. I don't care about nearly all of the features of blog software. I could use something that's much better-designed (and free software) like ikiwiki, but I don't particularly like wikis either. (That's another story that I should write up sometime.) I really just wanted something that works like my book reviews work and lets me maintain the whole thing in spin. So I just gave up and wrote it. It probably took me more time than it would take to beat some piece of blog software into submission, but now it's done, I know how it works, it knows what I want to do, and I don't have to fight with it again with every upgrade. So this is my first post with my new journal software, with an RSS feed generated by my own code. There will probably be bugs for a while to iron out, but it will also make it far easier for me to maintain multiple RSS feeds with different topics, post reviews to my journal even more automatically, and add automatic posting for other things. The main loss is comments. With this system, my entire web site can be static, which I love. But that means no comment scripts. Since almost no one was commenting on my journal anyway (as opposed to mailing me or discussing posts elsewhere), I think I'm okay with that. Eventually I may mirror my journal somewhere else where there are comments. All of the old posts have been converted over, including old comments. Old reviews with no comments have become redirects to the permanent review pages. Let me know if you see anything weird.

I’ve noted in the past that Debian has deliberately enshrined in its constitution some rather serious principal-agent problems. By and large this isn’t a bad thing, since there isn’t the consensus within the Debian community to support the “benevolent dictator for life” model of decision-making—if you want that, well that’s what Ubuntu and Daddy Warbucks is for. But it does mean that sometimes the caca hits the fan when a Debian project leader does exercise his powers, as our now-former DPL did earlier this week just before the end of his term of office (by my estimate, just over one hour and 27 minutes before). John Adams would be proud. So we have three related issues in my mind:

• As a matter of general principle, lame-duck DPLs shouldn’t be making appointments. This issue is ameliorated somewhat because DPL delegations—unlike “midnight judges”—can be revoked at any time, but it strikes me that whatever legitimacy a DPL has from the developers evaporates once a new DPL-elect has been designated. I can only speculate why this happened in this case, so I won’t bother.
• Second, while Debian has a very strong tradition of developer sovereignty, with many aspects of the project being self-organized rather than originating with appointments from upon high, it seems to me that certain aspects of core infrastructure can’t be managed in this way.
• Third, the appointment does little to relieve the excessive concentration of power in the core of Debian; if anything, Anthony Towns’ apparent resignation in the wake of Jörg’s appointment worsens the situation. Ensuring there is proper vetting of people with access to important infrastructure is important, but at the same time I find it difficult to believe that there are only a half-dozen or so Debian developers who are trustworthy enough to be system administrators, account managers, or archive maintainers (several of them occupying overlapping positions). That, rather than a lack of technical tools, has been a problem of note within Debian since, oh, the days of my youthful vigor within Debian (which are long since past).

In any event, congratulations to all the new Debian developers—and I’ll avoid pondering for too long why one person’s appointment to an unrelated group would suddenly break the logjam of developer application approvals.

I was recently checking to see what, if any follow-up there had been from Sun’s ham-handed handling of the Open Solaris Trademark, and I ran across this very interesting comment from Jon Plocher’s Candidate Statement for the Open Solaris Governing Board:

“I also think there was a misunderstanding about what Sun desired when it launched the community (in part) to encourage developers to adopt and use Solaris. My take is that, while there *is* value in getting more kernel, driver and utility developers contributing to and porting the (open) Solaris operating system, there is significantly *more* value in having a whole undivided ecosystem based on a compatible set of distributions, where application developers, university students, custom distro builders and users are all able to take advantage of each other’s work. Put these two things together, and you can see Sun’s predicament. Sun *wanted* a community that empowered application developers, but *got* a community aimed squarely at kernel hackers. Whether you see this as the “kernel.org -vs- Ubuntu” fight, or the “fully open -vs- MySQL model” argument, in my opinion, it all is simply a reflection of the above mismatched expectations.”

So that explains why it’s take three long years to try to get basic open source development tools (such as putting Open Solaris source code in a distributed SCM located outside of the Sun firewall) for Open Solaris. It was never was Sun’s intention to try to promote a kernel engineering community, or at least, it was certainly not a high priority for them to do so. This can be shown by the fact that as of this writing they still are using the incredibly clunky requester/sponsor system for getting patches into Solaris; setting up a git or mercurial server is not rocket science. This lack explains why Linus gets more contributions while brushing his teeth than Open Solaris gets in a week. So if you run into a Sun salescritter or a Sun CEO claiming that OpenSolaris is just like Linux, it’s not. Fundamentally, Open Solaris has been released under a Open Source license, but it is not an Open Source development community. Maybe it will be someday, as some Sun executives have claimed, but it’s definitely not a priority by Sun; if it was, it would have been done before now. And why not? After all, they are getting all of the marketing benefit of claiming that Solaris is “just like Linux”, without having to deal with any of the messy costs of working with an outside community. As a tactical measure, astroturfing is certainly a valid marketing trick. But after three years, the excuse of “just you wait a little longer, we’re just trying to figure this open source community stuff out”, is starting to wear a little thin. Furthermore, if (as Jon Ploncher claims) this was about “empowering application programmers”, why was it that Sun’s first act was to trumpet how wonderful it was to release the Solaris source code under a Open Source license? This only seems to make sense if the Open Solaris initiative was really a cynical marketing tactic to try to save Solaris from being viewed as irrelevant. If that was Sun’s intention, I think it is fair to say that from a marketing point of view, the tactic has been at least partially successful — although as Jon has admitted, the goal of creating a full community with application developers, university students, and so on, hasn’t materialized for Open Solaris. Sun has the dream; the Linux community is living it. However, from business standpoint, I wonder if Sun will really be able to sustain their Solaris engineering team if they will really be doing all of the work themselves, and outside contributions continue at the rate of 0.6 patches per day. After all, the margins when you are selling low-cost AMD servers are much lower than when you are selling ber-expensive SPARC servers. With Linux, we have a major advantage in that kernel improvements are coming from multiple companies in the ecosystem, instead of being paid for by a single company. And given that 70-80% of Sun’s AMD servers are running Linux, not Solaris, it’s not clear how Sun justifies their Solaris engineering costs to their shareholders. Furthermore, if Solaris on x86_64 were to actually take off, there’s nothing to stop competitors from selling Solaris support — except the competitors won’t have to pay the engineering costs to maintain and improve Solaris, so they would be able to provide the support much more cheaply than Sun could. So while Sun’s marketing tactics have kept Solaris alive in some verticals, I have to question how successful Sun will be in the long-term.

when I woke up today (after sleeping in for the first time with my new roll-top in front of my bedroom window) I was surprised & confused by a couple of "congratulations!" messages in my irc away-log. it took me a bit of time & coffee (& looking into my mailbox) to begin to realize that my Debian account had indeed been created while I was asleep. — in fact I guess I still haven't completely realized my new status as Debian Developer.

as others I'd like to follow the good tradition of taking the opportunity to thank some of the people who helped me to get there:

• first & foremost Tony Mancill, my advocate, permanent sponsor, & long-time co-maintainer; for all his support in preparing & quickly uploading packages, tracking down bugs, encouraging & advocating me but also for our very pleasant "chats" via email. — we have already agreed to continue co-maintaining each other's packages.
• the friendly & helpful guys from the Debian Perl Group: Gunnar Wolf, for always encouraging me (to join the group, to apply for the New Maintainer Process, ...); Damyan Ivanov, for teaching me so much (even if he called it "nitpicking"); Gunnar, Damyan, Krzysztof Krzyżaniak, Niko Tyni, Roberto C. Sánchez, Stephen Gran, Frank Lichtenheld, Jaldhar H. Vyas, Russ Allbery, Raphaël Hertzog, & some others for uploading packages I've prepared & helping me when I had questions or problems; Martín Ferrari, for all his help, expertise, & especially for the fun & the good times we share on IRC.
• my Application Manager Wouter Verhelst; for guiding me through the New Maintainer process not only fast but also in a very helpful, at the same time demanding & supporting way.
• the Debian Women sub-project; for making Debian a better & friendlier place not only for women but in general.
• finally: the people who used the time while I was asleep to actually create my account tonight :)

some final thoughts about the NM process from my point of view:

• it took me some time to actually apply for NM; what deterred me was not that I knew it would take some time but that I didn't know how long the time would be; & that I knew that the bigger part of the overall time would consist of waiting.
• I applied on 2007-04-03, my account was created on 2008-04-18; 380 days is not bad altogether in my opinion; the actual work with my AM was from 2007-08-12 until 2007-11-28 (i.e. 108 days, or 28% of the whole duration).
• I did enjoy the actual checks; I had to read & think a lot, & I learned a lot in that period.
• Front Desk (i.e. Christoph Berg) was very quick on all necessary steps.
• I was never demotivated about my Debian work because of all the people supporting me & taking the burden of uploading packages I had prepared. but I have to admit that I got a little bored in the last months of waiting after the report had been submitted by my Application Manager.

& now it's time first to celebrate & then to try to fully grasp my new rights & responsibilities.

One thing that's been bothering me since quite a while is the fact that apt and friends are not very verbose when things go wrong. Usually that's just fine; if a package can't be installed, there isn't much you can do about it anyway. However, for those of us who have to maintain buildd machines, a message like the following is not very helpful:

ome packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:
The following packages have unmet dependencies:
  kdelibs4-dev: Depends: kdelibs4c2a (= 4:3.5.9.dfsg.1-2) but it is not going tobe installed
                Depends: libarts1-dev (>= 1.5.0) but it is not going to be installed
                Depends: libavahi-qt3-dev (>= 0.4) but it is not going to be installed
                Depends: libopenexr-dev (>= 1.2.2-3) but it is not going to be installed
                Depends: libqt3-mt-dev (>= 3:3.3.5) but it is not going to be installed
E: Broken packages

If you're not convinced, let me tell you that the culprit is libopenexr2ldbl. If you've noticed that libopenexr2 is not mentioned anywhere in the above output, you understand why I hate this. Usually, figuring out this stuff requires you to manually walk the dependency tree, which is silly. So today, I finally spent some time to sit down and write a couple of perl scripts to help me with that:

wouter@country:~/code/perl$ ./cache-packages Packages 
Reading data... done (20492 packages). Closing cache... ready. Have a nice day!
wouter@country:~/code/perl$ ./check-dep-waits 
kdelibs4-dev
checking dependencies of kdelibs4-dev
 checking dependencies of kdelibs4c2a (= 4:3.5.9.dfsg.1-2)
   checking dependencies of kdelibs-data (>> 4:3.5.9.dfsg.1)

[... copious output snipped ...]

  checking dependencies of  libopenexr2ldbl (>= 1.2.2)
libopenexr2ldbl is not available in any version

So, there. Obviously the "Packages" file in the first command line above came from a Debian mirror. The "cache-packages" script writes a file "~/.packages-dbm", which contains an MLDBM tiehash of all the packages, indexed on package name. If you have no clue what that means, just remember that it parses the Packages file into something my laptop can load in a split-second rather than requiring 45 seconds to open and parse the Packages file. Once you've ran that, you can call 'check-dep-waits' as many times as you want, that is, until the Packages file is outdated and you need to download a new one. The second command will just sit and wait. On the first line, you give it a piece of Depends: information, and it will parse that and see whether the dependency can be resolved. There's a lot of output; but given how I want to verify dependencies, I thought I'd rather have too much than too little information. If there are version numbers involved, this script will call dpkg --compare-versions to do the actual verification, so I don't have to watch policy for any possible changes, which is helpful; but it does mean that it'll only work on a Debian machine. The code is available for download at http://comp.uter.be/perl-Packages, FTWCA buildds. Update: While on the train, I realized that I'd totally forgotten about virtual packages. That's now been fixed; if you downloaded the code before, you may want to re-fetch it...

Dear Jaldhar, I wrote the post you’re referring to knowing that only one person had been stupid enough to title his post “Is Debian dying?”, so before you go on writing I’m insulting a group of people (at least I’m reading your post this way), please do your research. Lucas’ post is about the worst PR we can imagine. There’s nothing more stupid to do than what he did. That’s the PR equivalent of committing suicide, mostly. It’s seriously hindering the work some of us are doing, publicizing Debian, going to trade shows, etc. It’s not only stupid, it’s hurting people who do this work. What’s worse, the post comes with comments from random nobodies, who have nothing to do with Debian, don’t have the first clue about how the Project works internally, yet they can tell that Debian is dying and grinding to a halt. Not to mention it’s been publicized yet again by some “journalist”. I’ve known Lucas for a long time now. I know where I stand. And I maintain my previous post in full, like it or not. Kthxbye.

If only I had this book My Beautiful Mommy when I was a youthful spring chicken, I may have grown with a less traumatized soul. It is a “book that helps kids cope with mommy’s plastic surgery” and without such an aid, I was lost, so lost.

I've been on holidays for one week now, breaking my usual workflow. Well, that allowed me to rest and to think about what I'd like to do and things I need to handle during next weeks. The urgent things this next days will be my primary server replacement. It is currently dying, and I already had to change its power supply twice in a month. Unfortunately, I'm now at a point where I do not have any spare piece so if things go wrong again, I'm screwed. I need to collect some money and buy a new server, or maybe get a server if someone have an old or spare one to give me, I do not know yet. On the awesome front, I'm about to release awesome 2.3, which will be the final minor release of the major branch 2. This will lead me to work on awesome 3 at a slower rate and a cooler pace. Then, the thing I do not have to hurry for is awesome 3. There's no big problems in awesome 2, and the xcb-util stuff are not stabilized yet. After my gentle yelling on XCB mailing list, it seems that things will move but will slowly. So I do have time to make things right and do what I want on that branch, making bugfix release on awesome 2 if needed. You can read more about futur on my last post about awesome. All this should give me some more spare time to work on the upcoming Debian release, lenny, which I'd like to work on. Two years ago (my god, time flies), we've done good work with the french cabal squashing critical bugs and I'd like to go back on this and squash asses again.

So I am nearly done with teaching this year, which is a relief, not because I don’t like it but because after a full year of teaching, one naturally wants a break. But before it is even over, you have to start thinking about your fall courses, mostly so you can order books in time and because developing a syllabus also requires more than a few days or weeks of work. Next fall, I am re-teaching a first year course Introduction to Human Communication and Culture as well as a new course on computer hacking. While I have an old version of the syllabus, I am going to spend the next few weeks revising, reshuffling, and updating. If there is anything you think should be included in this type of course drop me a line. Below is the new description of the course.

The Culture and Politics of Computer Hacking. This course takes as its object computer hackers to interrogate not only the ethics and practices of hacking, but to examine more broadly how hackers and hacking have transformed the politics of computing and the Internet more generally. We will examine how hacker values are realized and constituted by different legal, technical, and ethical activities of computer hacking for example, free software production, cyberactivism and hactivism, cryptography, and the pranksih games of hacker underground. We will pay close attention to how ethical principles are variably represented and thought of by hackers, journalists, and academics and will use the example of hacking to address various topics on law, order, and politics on the Internet such as: free speech and censorship, computer gaming, privacy and security, and intellectual property. This will allow us to critically 1) problematize thinking on computer hackers as a socio-cultural group guided by a singular ethic and set of practices 2) examine the multiple ways hackers draw on and reconfigure dominant ideas of property, freedom, and privacy through their diverse moral codes and technical activities 3) broaden our understanding of politics of the Internet by evaluating the various political effects and ramifications of hacking.

Uwe Hermann has described how to resize a root filesystem after booting from a live-cd or recovery disk [1]. He makes some good points about resizing an LVM PV (which I hadn’t even realised was possible). The following paragraph is outdated, see the update at the end:
Incidentally it should be noted that if your root filesystem is an LVM logical volume then it can’t be resized without booting from a different device because the way LVM appears to work is that the LV in question is locked, then files under /etc/lvm/ are written, and then the LV is unlocked. If the LV in question contains /etc/lvm then you deadlock your root filesystem and need to press reset. Of course if your root filesystem is on an LV which has been encrypted via cryptsetup (the LV is encrypted not the PV) then a live resize of the root filesystem can work as locking the LV merely means that write-backs from the encryption layer don’t get committed. I’m not sure if this means that data written to an encrypted device is less stable (testing this is on my todo list). If your root filesystem is on a partition of a hard drive (such as /dev/hda2) then it is possible to extend it without booting from different media. There is nothing stopping you from running fdisk and deleting the partition of your root filesystem and recreating it. When you exit fdisk it will call an ioctl() to re-read the partition table, the kernel code counts the number of open file handles related to the device and if the number is greater than 1 (fdisk has one open handle) then it refuses to re-read the table. So you can use fdisk to change the root partition and then reboot to have the change be noticed. After that ext2online can be used to take advantage of the extra space (if the filesystem has a recent enough version of the ext3 disk format). One thing he didn’t mention is that if you do need to boot from another device to manipulate your root filesystem (which should be quite rare if you are bold and know the tricks) then you can always use your swap space. To do this you simply run swapoff and then run mkfs on the device that had been used for swap (incidentally there is nothing really special about the swap space, but it does tend to often be used for such recovery operations simply because it has no data that persists across a reboot). The minimum set of files that need to be copied to a temporary filesystem is usually /bin, /sbin, /dev, /lib (excluding all directories under /lib/modules apart from the one related to the kernel you are using), and /etc. Also you should make directories /proc, /sys, and /selinux (if the machine in question runs SE Linux). The aim is not to copy enough files for the machine to run in a regular manner, merely enough to allow manipulating all filesystems and logical volumes. Often for such recovery I boot with init=/bin/bash as a kernel parameter to skip the regular init and just start with a shell. Note that when you use init=/bin/bash you end up with a shell that has no job control and ^C is not enabled, if you want to run a command that might not terminate of it’s own accord then the command “openvt /bin/bash” can be used to start another session with reasonable terminal settings. I recommend that anyone who wants to work as a sys-admin experiment with such procedures on a test machine. There are lots of interesting things that you can learn and interesting ways that you can break your system when performing such operations. Update: Wouter points out that the LVM bug of deadlocking the root filesystem has been fixed and that you can also use ext2online to resize the mounted filesystem [2].

• [1] http://www.hermann-uwe.de/blog/resizing-a-dm-crypt-lvm-ext3-partition


• [2] http://www.grep.be/blog/en/computer/cluebat/actually_you_can

Share This

I think most people reading my blog would agree that backups are extremely important. So much important data is on computers these days: family photos, emails, financial records. So I take backups seriously.
A little while back, I purchased two identical 400GB external hard disks. One is kept at home, and the other at a safe deposit box in a bank in a different town. Every week or two, I swap drives, so that neither one ever becomes too dated. This process is relatively inexpensive (safe deposit boxes big enough to hold the drive go for $25/year), and works well.
I have been using rdiff-backup to make these backups for several years now. (Since at least 2004, when I submitted a patch to make it record all metadata on MacOS X). rdiff-backup is quite nice. It is designed for storage to a hard disk. It stores on the disk a current filesystem mirror along with some metadata files that include permissions information. History is achieved by storing compressed rdiff (rsync) deltas going backwards in time. So restoring "most recent" files is a simple copy plus application of metadata, and restoring older files means reversing history. rdiff-backup does both automatically.
This is a nice system and has served me well for quite some time. But it has its drawbacks. One is that you always have to have the current image, uncompressed, which uses up lots of space. Another is that you can't encrypt these backups with something like gpg for storage on a potentially untrusted hosting service (say, rsync.net). Also, when your backup disk fills up, it takes forever to figure out what to delete, since rdiff-backup --list-increment-sizes must stat tens of thousands of files. So I went looking for alternatives.
The author of rdiff-backup actually wrote one, called duplicity. Duplicity works by, essentially, storing a tarball full backup with its rdiff signature, then storing tarballs of rdiff deltas going forward in time. The reason rdiff-backup must have the full mirror is that it must generate rdiff deltas "backwards", which requires the full prior file available. Duplicity works around this.
However, the problem with duplicity is that if the full backup gets lost or corrupted, nothing newer than it can be restored. You must make new full backups periodically so that you can remove the old history. The other big problem with duplicity is that it doesn't grok hard links at all. That makes it unsuitable for backing up /sbin, /bin, /usr, and my /home, in which I frequently use hard links for preparing CD images, linking DVCS branches, etc.
So I went off searching out other projects and thinking about the problem myself.
One potential solution is to simply store tarballs and rdiff deltas going forward. That would require performing an entire full backup every day, which probably isn't a problem for me now, but I worry about the load that will place on my hard disks and the additional power it would consume to process all that data.
So what other projects are out there? Two caught my attention. The first is Box Backup. It is similar in concept to rdiff-backup. It has its own archive format, and otherwise operates on a similar principle to rdiff-backup. It stores the most recent data in its archive format, compressed, along with the signatures for it. Then it generates reverse deltas similar to rdiff-backup. It supports encryption out of the box, too. It sounded like a perfect solution. Then I realized it doesn't store hard links, device entries, etc., and has a design flaw that causes it to miss some changes to config files in /etc on Gentoo. That's a real bummer, because it sounded so nice otherwise. But I just can't trust my system to a program where I have to be careful not to use certain OS features because they won't be backed up right.
The other interesting one is dar, the Disk ARchive tool, described by its author as the great grandson of tar -- and a pretty legitimate claim at that. Traditionally, if you are going to back up a Unix box, you have to choose between two not-quite-perfect options. You could use something like tar, which backs up all your permissions, special files, hard links, etc, but doesn't support random access. So to extract just one file, tar will read through the 5GB before it in the archive. Or you could use zip, which doesn't handle all the special stuff, but does support random access. Over the years, many backup systems have improved upon this in various ways. Bacula, for instance, is incredibly fast for tapes as it creates new tape "files" every so often and stores the precise tape location of each file in its database.
But none seem quite as nice as dar for disk backups. In addition to supporting all the special stuff out there, dar sports built-in compression and encryption. Unlike tar, compression is applied per-file, and encryption is applied per 10K block, which is really slick. This allows you to extract one file without having to decrypt and decompress the entire archive. dar also maintains a catalog which permits random access, has built-in support for splitting archives across removable media like CD-Rs, has a nice incremental backup feature, and sports a host of tools for tweaking archives -- removing files from them, changing compression schemes, etc.
But dar does not use binary deltas. I thought this would be quite space-inefficient, so I decided I would put it to the test, against a real-world scenario that would probably be pretty much a worst case scenario for it and a best case for rdiff-backup.
I track Debian sid and haven't updated my home box in quite some time. I have over 1GB of .debs downloaded which represent updates. Many of these updates are going to touch tons of files in /usr, though often making small changes, or even none at all. Sounds like rdiff-backup heaven, right?
I ran rdiff-backup to a clean area before applying any updates, and used dar to create a full backup file of the same data. Then I ran apt-get upgrade, and made incrementals with both rdiff-backup and dar. Finally I ran apt-get dist-upgrade, and did the same thing. So I have three backups with each system.
Let's look at how rdiff-backup did first.
According to rdiff-backup --list-increment-sizes, my /usr backup looks like this:

        Time                       Size        Cumulative size
-----------------------------------------------------------------------------
Sun Apr 13 18:37:56 2008         5.15 GB           5.15 GB   (current mirror)
Sun Apr 13 08:51:30 2008          405 MB           5.54 GB
Sun Apr 13 03:08:07 2008          471 MB           6.00 GB

So what we see here is that we're using 5.15GB for the mirror of the current state of /usr. The delta between the old state of /usr and the state after apt-get upgrade was 471MB, and the delta representing dist-upgrade was 405MB, for total disk consumption of 6GB.
But if I run du -s over the /usr storage area in rdiff, it says that 7.0GB was used. du -s --apparent-size shows 6.1GB. The difference is that all the tens of thousands of files each waste some space at the end of their blocks, and that adds up to an entire gigabyte. rdiff-backup effectively consumed 7.0GB of space.
Now, for dar:

-rw-r--r-- 1 root root 2.3G Apr 12 22:47 usr-l00.1.dar
-rw-r--r-- 1 root root 826M Apr 13 11:34 usr-l01.1.dar
-rw-r--r-- 1 root root 411M Apr 13 19:05 usr-l02.1.dar

This was using bzip2 compression, and backed up the exact same files and data that rdiff-backup did. The initial mirror was 2.3GB, much smaller than the 5.1GB that rdiff-backup consumes. The apt-get upgrade differential was 826MB compared to the 471MB in rdiff-backup -- not really a surprise. But the dist-upgrade differential -- still a pathologically bad case for dar, but less so -- was only 6MB larger than the 405MB rdiff-backup case. And the total actual disk consumption of dar was only 3.5GB -- half the 7.0GB rdiff-backup claimed!
I still expect that, over an extended time, rdiff-backup could chip away at dar's lead... or maybe not, if lots of small files change.
But this was a completely unexpected result. I am definitely going to give dar a closer look.
Also, before I started all this, I converted my external hard disk from ext3 to XFS because of ext3's terrible performance with rdiff-backup.

I can t possibly summarize all that has happened to me since my last post. The short explanation is that I had to serve in Brazilian army for a year, and that kind of disorganized my life. While I was in the army, I was forbidden to blog, and when that year ended, my life was too messy to keep up and I ve been procrastinating my come back! Now, finally, I am back. Just in time, I would say. fisl9.0 is about to begin. For this come back, somethings changed: my blog engine (from Rodin to Mephisto), my Gallery (it s gone sorry!), my domain (I ve moved from my company s webserver to my own), I will be publishing some entries in portuguese (in a separate section)... Hope you all enjoy. Happy hacking.

I'm pretty impressed with the new Geotag Icon Project. The well-known feed icon has become ubiquitous, and it makes sense to have a free-to-use icon for geographically-tagged information. I can see it being very useful for hcard and geo microformats, for example. tags: geotagicon geo microformats

Relayed for your review without comment For all those who think Open Content guidebooks like Wikitravel are by definition less reliable than proprietary ones: "Travel writer tells newspaper he plagiarized, dealt drugs". "A Lonely Planet author says he plagiarized or made up portions of the popular travel guidebooks and dealt drugs to supplement poor pay. [...] He didn't travel to Colombia to write the guidebook on the country because 'they didn't pay me enough. I wrote the book in San Francisco. I got the information from a chick I was dating.'" Cough, cough. tags: lonelyplanet travel guidebook plagiarism

While looking at poll about Czech open source for first time, I was surprised that two of my projects got in - phpMyAdmin and Wammu. After looking again at that page I noticed there is a typo in phpMyAdmins name, it was written as phpMinAdmin. Then I looked even more carefully and even URL it is pointing to is something different. In this time I realized that it is something completely different with just very similar name. phpMinAdmin is some minimalistic clone of phpMyAdmin, but why they could not choose a bit more different name? PS: Anyway you can still vote for Wammu in Czech open source :-).