When I explain the importance of free software, I often use some variation of the following example:

Suppose I see a beautiful sunset and I want to describe it to a loved one on the other side of the world. Today's communication technology makes this possible. In the process, however, the technology in question puts constraints on message communicated. For example, if I pick up my cellphone, my description of the sunset will be limited to words and sounds that can be transmitted by phone. If I happen to have a camera phone and the ability to send a picture message, I will be able to communicate a very different type of description. If I'm limited to 150 characters in an SMS message, my message will be constrained differently again.

The point of the example is this: the technology I use to communicate puts limits and constraints on my communication. Technology defines what I can say, how I can say it, when I can say it, and even who I can say it to. This is neither good nor bad. It is simply the nature of technology. But it means that those who control our technology control us, to some degree. As information technology becomes increasingly central to our lives, the way we experience, understand, and act in the world is increasingly controlled by technology and, by extension, by those who control technology. I believe that the single most important struggle for freedom in the twenty first century is over the question of who will set these terms. Who will control the technology that controls our lives? Free software can be understood as an answer to this question: An answer in the form of an unambiguous statement that technology must be under the control of its users. When free software triumphs, we will live in a world where users control their technological destiny. We simply cannot afford to fail. The Free Software Foundation is the most important organization fighting for the rights of users in this struggle. Here are some of the ways that I plan to direct the FSF to support software freedom in the coming year:

Network Services Last year, the FSF organized a meeting on software freedom and network services that lead to the creation of the Autonomo.us group and the release of the Franklin Street Statement on Freedom and Network Services. As network services --- like those built by Facebook, Google, and others --- continue rise in popularity, progress in this conversation is of increasing importance. This year, I will direct the FSF to build on the work of Autonomo.us to launch the first of what I hope will be several FSF position statements on software freedom and network services. More importantly, the FSF will begin to provide support and planning for solutions --- technologies, social campaigns, and legal steps --- that will protect computer users whose freedom is currently threatened by network services. Enforcing free software licenses Early on, people who decided to work on free software did so because they agreed with --- or, at the very least, were willing to abide by --- the principles and rules laid out in our licenses. In our push for software freedom, we have created software of immense value and attracted companies and individuals to our community who are less willing, or simply less interested, in protecting users' freedom. In December 2008, the FSF went to court for the first time in the organization's history to force Cisco to uphold the freedom of the users of FSF copyrighted software. This lawsuit asks Cisco to live up to its obligations under the GPL and to ensure that it does so in the future. The FSF needs the support of its community in this and in future enforcement actions. The FSF has operated a compliance lab for several years and has ensured software freedom for countless users. As free software becomes more successful in the next year, the FSF will be playing an increased role in protecting software freedom from those who do not share its principles. Continuing the fight against software patents As I said last year, one cannot write non-trivial software today without running a serious risk of infringing patents. The software patent minefield we've found ourselves in is a very fundamental threat to the success of free software and we've already begun to see the first casualties and costs. We must eliminate software patents. Now. The FSF will continue its work toward this end. The FSF's End Software Patents project's major contribution this year was a brief submitted in the In Re Bilski case at the U.S. Court of Appeals for the Federal Circuit. Recently, an opinion was issued which seems to mark an important shift away from broad protection for software patents. With this victory, now is the time to keep up this momentum. The FSF is drawing up a strategy to do just this and will be announcing the relaunch of its campaign on February 9,2009 led by Ciaran O'Riordan.

Every recent fundraising appeal seems to mention the difficult economic climate and it seems to cliche to do so here. That said, the effect of a bad economy is, in fact, felt most strongly in non-profits dependent on donations. The FSF is not immune. The FSF's work is essential for success on the issues I've described here and on all of its other campaigns and projects. Although the cost of a membership or donation may be less easy to afford this year for many of you, the free software movement cannot afford a weakened FSF at this important moment. If you are not an FSF associate member, now is the time to become one. If you are already a member, please join me in giving generously through a tax-deductible donation. The FSF is a very lean, very humble organization of passionate and dedicated individuals working tirelessly for software freedom. Every single gift makes a difference.

Our power alone cannot protect us, nor does it entitle us to do as we please. . . Our security emanates from the justness of our cause; the force of our example; the tempering qualities of humility and restraint. - Barack Obama, inaugural speech, Jan. 20, 2009

This, right here, is why, for the first time in my life, I actually feel good about an American president. Why I have hope about our government for the first time in years. Why I m glad I used a vacation day to sit on the couch and watch TV yesterday. On the occasion, once every 4 or 8 years, that is a celebration of American strength, power, and pride, we see our new president speaking of humility, of peace, of moral leadership, this is something remarkable. Past presidents have used occasions such as these to speak of crushing our enemies, of wanting people dead or alive, of grand government promises that turned out to triple the national debt. Obama spoke of extending the hand of friendship to anyone that would unclench their fist. He spoke that we had kicked the can down the road too far, and now we ve reached the end of the road. We have to stop thinking that we can have everything: low taxes, expensive programs, and a large military, simply by mortgaging our future. And he leveled with us: we all are in this together, and all have to work to make it better. Conventional politicians assumed it would be political suicide to say even half of what Obama has said. Yet he went out there and did it. He was blasted during the campaign by people on both sides of the political spectrum for being just words . He s the first presidential candidate that meant what he said about bringing Americans, and their representatives, together. The shock in Washington has he invited gasp! both an openly gay bishop and an anti-gay evangelical minister to give prayers was telling. It s as if people were saying, Wait, he really MEANT that? Yes, he did. Let s hope he can pull it off. And as Rev. Lowry concluded with his benediction:

With your hands of power and your heart of love, help us then, now, Lord, to work for that day when nations shall not lift up sword against nation, when tanks will be beaten into tractors, when every man and every woman shall sit under his or her own vine and fig tree and none shall be afraid, when justice will roll down like waters and righteousness as a mighty stream. . . Let all those who do justice and love mercy say Amen. (amen) Say Amen! (Amen!) Say Amen! (AMEN!)

I keep writing code to talk to databases in perl and I'm forever forgetting the correct runes for talking to databases, so I thought I'd stick it here for easy reference.

use DBI;
my $db_driver = "Pg" # Pg or mysql (or others)
my $db_name = "database";
my $db_host = "localhost";
my $db_user = "username";
my $db_pass = "password";
my $dbh = DBI->connect("dbi:$db_driver:dbname=$db_name;host=$db_host", 
   $db_user, $db_pass);

It's probably handy to give an example of a common database read operation

my $sth = $dbh->prepare( "SELECT * FROM table WHERE id  = ?") 
      or die $dbh->errstr;
$sth->execute($id) or die $dbh->errstr;
while (my $hashref = $sth->fetchrow_hashref)  
   print $hashref-> id ;
 

Before I get started, I want to make it quite clear that I have no problems with Github - au contraire, they provide a very nice service. What my speculation centers around is the social dynamics of Github. One of the Github guys writes a nice response to my original post, here: http://ozmm.org/posts/linux_vs_classic_dev_style.html First of all, apologies for the comment bug. In theory, it has been fixed by the Typo guys, I just need to find the time to upgrade. Chris mentions that it's possible to create a "SourceForge style" project on Github. I see that it's possible to add collaborators, so I guess that's what he means. I still find it a bit ugly to have things at .../davidw/... but that's not very important if the URL is at least stable, rather than hopping around to whoever has the latest fork and updates of the code in question. In my other post, I talked about people being less interested in doing what it takes to contribute to existing projects, rather than forking them, unless it's really called for. Here's a concrete example of how things might go wrong. At my current consulting gig, we were looking at utilizing actionwebservice to do some SOAP stuff. That code is not part of Rails anymore, but that's not the problem - surely someone has decided to take up maintenance of the gem, right? Well, yes... but let's look:

root@fortrock:~# gem1.8 search -r actionwebservice
*** REMOTE GEMS ***
actionwebservice (1.2.6)
datanoise-actionwebservice (2.2.2)
dougbarth-actionwebservice (2.1.1)
nmeans-actionwebservice (2.1.1)

So we have the original, clearly out of date, and three others. Yuck... I guess we'll use the 2.2.2 one, but who knows if perhaps the others contain some good stuff too? Let's Google it. http://www.google.com/search?hl=en&q=actionwebservice The first link is to a RubyForge project, which is basically dead. A few links down, there is a link talking about a more recent version, at the Datanoise site. However, it's one of many. Confusing. If you go to the github page for the datanoise version of actionwebservice, the README has this:

The latest Action Web Service version can be downloaded from http://rubyforge.org/projects/actionservice

Hrm. We were just there and that doesn't seem quite right... Sure, it's an easy enough oversight to ignore, but it adds to the confusion. Now, this is not really a serious problem, but do you see how it could get worse over time, as various people fork the code, write blog entries, and so on? It would get harder for someone interested in utilizing the code in question to ascertain which is "the" version. It's not hard to imagine situations where two forks add different bits of useful code. Typically, in situations like this, new users, those who don't know the code at all, are also the people least able to look through all the diffs and changes and log entries and what not in an attempt to merge any changes that might be useful. This leads to a "paradox of choice" type of problem. Another consideration, this one more positive, is that perhaps people are putting code in Github that, in the past, might have simply remained on their own computers, with no effort whatsoever to share it, so in that sense, Github might be doing people a service by at least making the code public. However, even in that case, if the service fills up with code that people have simply "thrown over the wall", rather than creating a genuine open source project, with a community, communication channels, and so on, it could get a reputation for having code that you have to handle with care, as a lot of is just stuff slapped up there without much thought to quality or continuity. Anyway, the Github guys have done a nice job with the service, and I'm sure they'll continue to improve it. Who knows, perhaps they have some good ideas about mitigating some of these potential problems that they'll be rolling out soon.

Fabio Tranchitella recently posted about his new filesystem which really reminded me of an outstanding problem I have. I do some email filtering, and that is setup in a nice distributed fashion. I have a web/db machine, and then I have a number of MX machines which process incoming mail rejecting spam and queuing good mail for delivery. I try not to talk about it very often, because that just smells of marketting. More users would be good, but I find explicit promotion & advertising distasteful. (It helps to genuinly consider users as users, and not customers even though money changes hands.) Anyway I handle mail for just over 150 domains (some domains will receive 40,000 emails a day others will receive 10 emails a week) and each of these domains has different settings, such as "is virus scanning enabled?" and "which are the valid localparts at this domain?", then there are whitelists, blacklists, all that good stuff. The user is encouraged to fiddle with their settings via the web/db/master machine - but ultimately any settings actually applied and used upon the MX boxes. This was initially achieved by having MySQL database slaves, but eventually I settled upon a simpler and more robust scheme: Using the filesystem. (Many reasons why, but perhaps the simplest justification is that this way things continue to work even if the master machine goes offline, or there are network routing issues. Each MX machine is essentially standalone and doesn't need to be always talking to the master host. This is good.) On the master each domain has settings beneath /srv. Changes are applied to the files there, and to make the settings live on the slave MX boxes I can merely rsync the contents over. Here is an anonymized example of a settings hierarchy.. So a user makes a change on the web machine. That updates /srv on the master machine immediately - and then every fifteen minutes, or so, the settigngs are pushed accross to the MX boxes where the incoming mail is actually processed. Now ideally I want the updates to be applied immediately. That means I should look at using sshfs or similar. But also as a matter of policy I want to keep things reliable. If the main box dies I don't want the machines to suddenly cease working. So that rules out remotely mounting via sshfs, nfs or similar. Thus far I've not really looked at the possabilities, but I'm leaning towards having each MX machine look for settings in two places:

• Look for "live" copies in /srv/
• If that isn't available then fall back to reading settings from /backup/

That way I can rsync to /backup on a fixed schedule, but expect that in everyday operation I'll get current/live settings from /srv via NFS, sshfs, or something similar. My job for the weekend is to look around and see what filesystems are available and look at testing them. Obmovie:Alive

My son Andrew is now ten weeks old, and already I ve learned so much. For example Being a good parent occasionally means being a proper adversary It s much harder than you might expect, and happens more often than you might expect. A pacifier is an excellent way to calm him down, especially when he has gas pain (and a pacifier can also reduce the risk of SIDS); however, he would let it fall out of his mouth when he wasn t sucking on it. Rather than hold it in, if you tug on the pacifier a little bit, it teaches him how to suck on it so it doesn t fall out. Sometimes what s best is actually not so good Andrew has a food allergy, but we re not exactly sure what he s allergic to. For the last couple weeks, he s been on Nutramigen, a hypoallergenic formula specifically for babies with food allergies, and has been responding really well. Meanwhile, Lisa s been on an elimination diet, eating mainly chicken and rice, trying to get everything he might possibly be allergic to out of her system so she can start nursing again. Recently, Lisa started nursing at an occasional feeding, and he coincidentally seems to have gotten more fussy. He s not screaming in pain like he was before going on the formula, and it s hard to say that it s because of the breast milk. Still, we re thinking about going back to just formula and seeing if he improves again. I read too much When I was a child, and still to this day, I had pretty severe sleep problems. I didn t sleep through the night until I was 18 months old. So before Andrew was born, Lisa and I read a few books about how to establish a healthy sleep pattern. All of them had various theories about why children develop sleep problems, and how to quiet a baby and establish a routine. Now we re so full of conflicting theories about problems that will develop if we do or don t follow some guideline, that it s hard to make a decision about what s best sometimes. It doesn t help that, in part due to his food allergy, he ll sleep through the night one night, and the next night wake up every few hours. Fortunately, only one feeding in the middle of the night has been the norm for quite some time, so it hasn t been too bad.

It s been a while now since the 2008 Summer of Code ended. This year, twelve (?) projects were selected. That s twelve students working full time on a Debian-related project during the summer. The Google Summer of Code has sometimes been criticized in the past for having a poor student-developer retention rate inside the host projects. One of the goals of the program has always been to bring new people to budding or established free software organizations and it s a pity that some would leave the project as soon as the program ends. On the other end, poor integration of created code within the project leads to work that is hard to merge in, or worse, doesn t get merged in at all. That s a waste of time and resources and a probably cause of global warming as well. Hopefully, it s not always the case. Some people choose to stay committed within the organization in the long-term. Useful code gets merged in and pushed to the public. I am going to give a talk about this at FOSDEM (go to FOSDEM!) so I m giving you a little preview. I need your help to collect information for my talk. As you know, information is always hard to come by with these kinds of projects so anything can be useful. Without further ado, let s have a look at the cast of the Debian Google Summer of Code 2008: Netconf, a network configuration management system Presentation Netconf is a network configuration management system designed with modern network infrastructures and the needs of roaming users in mind. It is a personal project of Martin Krafft that he started in 2007. He did some presentations about it that you can find on the dev website. The project proposal was introduced by the mentor. The work was mainly about completing the roadmap items for version 1.0. Most of the design was done and code fleshed out. The stated goal was to have netconf ready for lenny. Martin noted that due to lack of regular free time, he couldn t reach that goal by himself. Student Jonathan Roes was a computer science graduate student from the University of North Carolina at Charlotte. He had programming experience as a hobby for a long time and wrote some free games and libraries for the Nintendo DS and some proprietary webapps. He submitted a few little patches in mid-march right after the publication of accepted mentoring organizations and went on to work from mid-may to mid-august. He wrote a lot of code right into the trunk since the whole project was a prototype. Result The last commit by Jonathan was also the last to date in the main netconf git repository. No further progress has been made and obviously netconf didn t get into lenny. I couldn t find any further public involvement of Jonathan within Debian. The ultimate Debian database, all things Debian in a SQL database Presentation The Ultimate Debian database wants to reunite all Debian data sources in a SQL database The project was mentored by Lucas Nussbaum and co-mentored by Stefano Zacchiroli and Marc HE Brockschmidt. The project proposal was introduced by Lucas. Student Christian Neronus Von Essen is.. well, there wasn t much information readily available on him. Result The whole project is coded up and working well with a whole bunch of data sources. There will be a talk at FOSDEM about this so I ll leave it to Lucas to talk about it in detail. I couldn t find any further public involvement of Christian within Debian. Security-beta, a beta testing for Debian security updates Presentation The task is to improve the quality assurance process for security updates by providing a public security update beta test program in addition to the existing QA done for security updates. During the preparation of security updates, there s an inherent delay between the initial upload of the fixed packages and the time until the packages have been built on porter machines. This time gap will be used for a new security update beta program. The project was supposed to be mentored by Moritz M hlenhoff. Student The project was supposed to be done by Nico Nion Golde. He is studying computer sciences at the Technical University of Berlin. He s also a DD. Result There s no nothing. Nico, what happened ? And obviously, he s still developing for Debian. Debgraph, a generic infrastructure for the development of packages management tools Presentation In a large software ecosystem such as Debian Linux, there is the potential for dependencies among software packages to create complex management and technical problems. For example, dependency loops (cycles) in which a package directly or indirectly depends on itself can confuse package management tools as they determine the proper order of package installation. debgraph helps developers to solve this problem by enabling generic queries (e.g., Give me all the nodes that depend on package X ) against the graph of packages and thus automating much of the manual labor that is typically involved in resolving dependency problems. The project was mentored by Robert Lemmen, who introduced the project proposal. The project was already started and the C++ code foundation was done by the time it was proposed. Student The project was executed by Adam Jensen, research assistant in the Software Engineering and Network Systems Laboratory at Michigan State University. Result Adam maintained a blog about his progress and finished ahead of schedule. However, the resulting work seems to be unused, which is a pity since the code could be used within other programs (package managers?). I couldn t find any further public involvement of Adam within Debian.
That s all for now. The information is quite fragmented I admit. Most of it was pulled from Google, mailing lists, commit logs, blogs, whatever. If some projects are lacking in information here, it s because I couldn t find it readily (which is an issue in itself!). In my next post, I ll try to analyze the success and failures to extract some insight. Teaser: pet projects!
If you re a student or a mentor mentioned above, feel free to fill any of the blanks in my report. It s much appreciated. You re not a student or mentor mentioned above and have an opinion on how to improve the next Debian Summer of Code ? Feel free to comment.
Sledge, ping! Here s a list of projects to be described in my next posts:

• Debian NAS, improve support of Debian on NAS devices
• Cran2deb, generate Debian packages from R packages
• Mergemaster, interactively merge changes in configuration files
• PAS NSS Debian Installer, improve support of PAM and NSS at install-time
• Jigdo-ivory, a JavaScript Jigdo client
• Aptitude-gtk, usability and GTK+ GUI for the Aptitude package manager
• Lintian for fuller automated setups
• Debexpo, a generic web-based package repository

From Today's Astronomy & Astrophysics there were two papers from the Geneva Observatory / Grenoble team on The HARPS search for southern extra-solar planets:

• XIII. A planetary system with 3 super-Earths (4.2, 6.9, and 9.2 M_e )
• XIV. Gl 176b, a super-Earth rather than a Neptune, and at a different period

These are probably part of the 45 super-Earths tentively announced by M. Mayor last June at the Nantes conference on super-Earths. He showed radial-velocity diagrams for 45 or so planets in an impressive display, but only gave details of three at the time, while the rest were being peer-reviewed, etc. Now they're drip-feeding out. Interestingly, this now makes 3 out of 7 super-Earths do far discovered as being around M dwarf stars, and another multiple planetary system. It looks more like the "metallicity" relation (larger and more frequent planets around more metallic stars) seems to be more true for gas giants than terrestrial planets. Super-Earths (potentially rocky planets, though heavier than Earth) look more abundant around low mass and less-metal stars. Now, down to work examining the characteristics of these planets. Rory Barnes and colleagues at LLPL, U. of Arizona have also been looking at these three planets around HD 40307, to determine if they are super-Earths or mini-Neptunes. They have an interesting technique of modelling the tidal circularisation of the planet backwards in time, based on its current measured or estimated eccentiricity. Tidal circularisation makes the orbit of close-in planets more circular (decreasing the eccentricity, e, to zero), dumping the energy into tidal heating of the planet (and the star). Doing this for Gl 437, for example, they showed that it would be hotter than Io, with a volcanic surface and couldn't be habitable. For HD 403007b, Hence they believe the planets are not terrestrial-like, but may be more Neptune-like. So, could these planets have oceans ? Tags astronony, super-earths, neptunes, exoplanets

Asheesh Laroia explained how to use ssh as a SOCKS proxy if you re behind a firewall that allows you to ssh to some machine not restricted by it: ssh -D. (I don t know how common such firewall setups would be, but the one at my Uni does indeed allow ssh to the outside over authenticated wireless.) With -D, ssh will listen in a local port, and behave as a SOCKS proxy. Set it up like this:

  % ssh -N -f -D localhost:4444 external-machine.example.com

And then you can point your SOCKS-capable application at localhost:4444. Chances are, however, that the application you want to use doesn t support SOCKS (like, in my case, Git). You can use tsocks then, which LD_PRELOADS a library that will divert an app s TCP traffic through the SOCKS proxy. To use it:

  % cat >~/.tsocks.conf
  server = 127.0.0.1
  server_type = 5
  server_port = 4444
  % tsocks git pull

AIUI, this is better than setting -L forwarding because you don t need a forward for each host whose port you want to access. If you re interested in starting ssh in SOCKS mode from .ssh/config, the configuration item is DynamicForward.

I ve been subscribed to the Git mailinglist for three weeks now, and I ve finally seen in action what was already imaginable by looking at their git log history: they not only review the code for the patches that go in (as in many other projects, that is), but also review the accompanying commit messages, in order to ensure they meet certain quality standards. More than once already I ve seen an otherwise-fine patch get resubmitted in order to gain a more meaningful or thorough commit message. I m sure there are more projects that do this, but alas, I haven t been in any of them (nor in that many projects in general, for that matter). It s very exciting for me to see it in action, because quality commit messages are high in my Grokking VCS list. Ideally, a commit message conveys all the necessary information the maintainer of a project needs to understand your patch and its benefits, in particular those bits that don t fit as comments in the code. If you have to accompany your patch with rationales, wouldn t it make sense to include a condensed version of them in your commit message? If you re writing messages for a project of your own, please think of future people dealing with your code, particularly if it s a public project. (It takes discipline in addition to being convinced of it, though.) Commit messages are also great because they allow you to be verbose, without cluttering the code with information that s only relevant in the context of the change, and not to the final code. There is however an (actually-not-so-fine) line between what should be in the commit message, and what should be as comments in the code, and I think we commit message enthusiasts should watch out for that. Take, for example, 39c19ce. I think a short comment saying Don t call rev-parse for each blamed line would have been a perfect companion. There must be somewhere a document explaining with more detail what makes a good commit message. If you would recommend any, please let me know (but I m not interested in documents that explain how to format your commit messages).

For a few weeks Laurent Vivier, Blue Swirl and myself have been working on getting QEMU PowerPC working correctly with recent distributions. QEMU used to rely on OpenHackWare for the OpenFirmware implementation on PowerPC. It is a very limited implementation (for example it as no Forth support), which is unable to boot most 2.6.x kernels with the OldWorld emulation. It is able to boot recent kernels with the PReP emulation, but things like the PCI bus emulation are not working correctly. Moreover the PReP kernels are gone with the removal of the arch/ppc tree. OpenBIOS was already used for the OpenFirmware implementation of Sparc 32 and Sparc 64 targets. It now supports PowerPC for the OldWorld emulation. As a result it is now possible to use Debian PowerPC under QEMU emulating an OldWorld machine. What works?

• Display (partly), keyboard, hard disk, network;
• Booting from CD-ROM or from disk using Quik;
• Installation of Debian Etch or Lenny (but due to a bug in debian-installer quik.conf has to be fixed manually);
• Standard Debian kernels;
• G3 CPU emulation: the testsuite results of the GNU libc 2.7 and of GCC 4.3 (Debian packages) are the same than on a real machine.
• virtio devices

What doesn t work / has to be done?

• A few devices part of the MacIO chipset are not emulated and/or are replaced by other devices: IDE, SCSI, Ethernet and sound;
• The red and green colors are reversed, in some modes only (in debian-installer for example);
• X only outputs some strange images;
• PCI devices using I/O ports don t work (like the RTL8029 card, or the RTL8139 card with the 8139too driver).

For those who want to test, an Etch image is available. You will need to compile QEMU by manually given that the version in Debian is too old and that openbios-ppc is still in the NEW queue.

Someone has performed a month of sysadmin blogging over at http://sysadvent.blogspot.com/: some good stuff in there. In particular, I found Day 2 - Windows Powershell interesting (hotwire is an example of something similar in the UNIX world) and Day 24 - Message Brokers is relevant for me as we're looking at implementing some message queueing stuff at work.

Developers seem to do a lot of fighting with bleeding-edge libraries installed in a --prefix, so I thought I'd share how I develop telepathy-glib (a library) and telepathy-gabble (a program using that library). ~/Collabora/telepathy/tpglib on my laptop is a git clone of telepathy-glib.git, branched using the process I documented on http://telepathy.freedesktop.org/wiki/Git, which is basically:

• clone the upstream repository (git+ssh://git.collabora.co.uk/git/telepathy-glib.git in this case) into your home directory on the server (git+ssh://people.collabora.co.uk/home/smcv/public_html/git/telepathy-glib-smcv.git in this case)
• clone that onto your laptop
• on your laptop, add "upstream" as a remote, tracking the upstream repository
• this reduces mistakes, because "git push" pushes to origin (which is in my home directory) for review, and "git push upstream" is necessary when I actually want to commit to the upstream repository

~/Collabora/telepathy/gabble is a git clone of telepathy-gabble.git, using the same process. In my Gabble checkout, I used to configure like this:

cd ~/Collabora/telepathy/gabble
./autogen.sh \
TP_GLIB_CFLAGS="-I$ HOME /Collabora/telepathy/tpglib" \
TP_GLIB_LIBS="$ HOME /Collabora/telepathy/tpglib/telepathy-glib/libtelepathy-glib.la"

This would make libtool link Gabble against the copy of telepathy-glib built by my bleeding-edge git checkout, without ever needing to install it. This allowed me to try out my latest telepathy-glib changes with Gabble's more extensive regression tests, which was a great win. It also meant I could develop Gabble branches that required a not-yet-released telepathy-glib, and all without touching my system copy of telepathy-glib (the one I use to run Empathy and talk to people - which is the packaged version from Debian unstable or experimental). The reason I could do this is that I laid out the telepathy-glib source tree in The Right Way , which is: if a header is designed to be used via #include <my-library/foo.h>, then it goes in a my-library/ directory in the source tree, and includes other headers (even from the same project) with #include <my-library/other.h>. This means you can just add the top directory of the source tree to your include path (which automake does by default) and the right thing will happen. (Actually, this is a slight simplification of the truth: because telepathy-glib contains generated headers, you have to add the top directory of the build tree to your include path too, if you want to support out-of-tree builds. Which, of course, you do.) Other projects that do this correctly include GLib and Avahi. Avahi is a particularly interesting example because it has numerous libraries in the same tarball (including avahi-common, avahi-core, avahi-glib and avahi-gobject, which form a dependency chain). Projects that don't do this correctly include telepathy-mission-control (which is on my hit list for further swamp-draining) and libtelepathy (which is obsolete). Today I committed a patch to telepathy-glib, inspired by GStreamer, to generate telepathy-glib-uninstalled.pc. The presence of this patch simplifies the process further, to:

./autogen.sh PKG_CONFIG_PATH=$HOME/Collabora/tpglib/telepathy-glib

This has the same benefits as setting TP_GLIB_CFLAGS and TP_GLIB_LIBS, but is easier to remember, and is more future-proof against changes to telepathy-glib. It only works because pkg-config has a special case for this situation - if you ask for the package "foo", pkg-config will always look for foo-uninstalled.pc before foo.pc. (Edited 2009-01-13: putting environment variables on the ./configure or ./autogen.sh command line like ./autogen.sh PKG_CONFIG_PATH=foo is better than putting them in the environment like PKG_CONFIG_PATH=foo ./autogen.sh, since it works around Debian bug #398901 and bugs like it)

Dear Lazyweb, I m looking for recommendations for Open Source backup solutions which track incremental backups using a database, and which do not use hard link directories. Someone gave me a suggested OSS backup program at UDS, but it s slipped my memory; so I m fairly sure that at least one or more such OSS backup solutions exist, but don t know their names. Can some folks give me some suggestions? Thanks!

There are a number of very popular Open Source backup solutions that use a very clever hack of using hard link trees to maintain incremental backups. The advantage of such schemes is that they are very easy to code up, and it allows you to easily browse incremental backups by simply using cd and ls . The disadvantage of such schemes is that it creates very large number of directories blocks which must be validated by an fsck operation. As I ve discussed previously, this causes e2fsck to consume a vast amount of memory; sometimes more than can be supported by 32-bit systems. Another problem which has recently been brought home to me, is how much time it can take to fsck such file systems. This shouldn t have come as a surprise. Replicating the directory hierarchy for each incremental backup is perhaps the most inefficient way you could come up with for storing information for an incremental backup. The filenames are replicated in each directory hierarchy, and even if an entire subtree hasn t changed, the directories associated with that subtree must be replicated for each snapshot. As a result, each incremental snapshot results in a large number of additional directory blocks which must be tracked by the filesystem and checked by fsck. For example, in one very extreme case, a user reported to me that their backup filesystem contained 88 million inodes, of which 77 million of them were directories. Even if we assume that every directory was only a single block long, that still means that during e2fsck s pass 2 processing, 77 million times 4k, or 308 gigabytes, worth of directory blocks must be read into memory by e2fsck s pass 2. Worse yet, these 308 GB of directory blocks are scattered all across the filesystem, which means the time to simply read all of the directory blocks so they can be validated will take a very, very, long time indeed. The real right way to do implement tracking incremental backups is to use a database, since it can much more efficiently store and organize the information of what file is located where, for each incremental snapshot. If the user wants to browse an incremental snapshot via cd and ls , this could be done via a synthetic FUSE filesysem. There s a reason why all of the industrial-strength, enterprise-class backup systems use real databases; it s the same reason why enterprise class databases use their own data files, and not try to store relational tables in a filesystem, even if the filesystem supports b-tree directories and efficient storage of small files. Purpose written-and-opimzied solutions can be far more efficient than general purpose tools. So, can anyone recommend OSS backup solutions which track incremental backups using some kind of database? It doesn t really matter whether it s MySQL, Postgresql, SQLite, so long as it s not using/abusing a file systems hard links to create directory trees for each snapshot. For bonus points, there would be a way to browse a particular incremental snapshot via a FUSE interface. What suggestions can you give me, so I can pass it on to ext3/4 users who are running into problems with backup solutions such as Amanda, Backup-PC, dirvish, etc.? Update: A feature which a number of these hard-link tools do right is that they do de-duplication; that is, they will create hard links between multiple files that have the same contents, even if they are located in different directories with different names (or they have been renamed or their directory structure reorganized since they were originally backed up) and even if the two files are from different clients. This basically means the database needs to include a checksum of the file, and a way to look up to see if the contents of that file have already been backed up, perhaps with a different name. Unfortunately, it looks like many of the tape-based tools, such as Bacula, assume that tape is cheap, so they don t have these sorts of de-duplication features. Does anyone know of non-hard-link backup tools that do de-dup and which are also Open Source?

I have never felt inclined to create content for Youtube due to the low resolution of the display and the fact that only one format is supported (Flash which is totally non-free). The existence of blip.tv has inspired my interest in creating videos for distribution on the net. blip.tv has higher resolution than Youtube and supports multiple formats. Currently the multiple format support is limited to allowing users to upload whatever they want and then Flash (FLV) will be provided for download in addition to the original file(s). So for example I could upload MPEG4 and Quicktime movie files for the same content which would result in three download options (MPG, MOV, and FLV). It would be nice if they could support more formats as the default (maybe support transcoding to OGG), but it s certainly a lot better than Youtube. So the question then is how to create content. One option that I plan to use is Istanbul - the GNOME desktop session recorder [1]. It will record everything that happens on the desktop (or a section of the desktop) to an OGG file while also optionally recording audio input. This allows the creation of tutorials for performing various tasks on the computer. Another option is to do animation, either completely computer generated or by combining a set of JPEG files into an animation file. This doesn t interest me to the degree necessary to make me invest the necessary time. But the most common way of producing video content is via a camera. So the question becomes what camera to get. My current camera is a Sony Cyber-shot DSC-T5 [2]. It is a 5.1 mega-pixel camera with 3* optical zoom and supports 640*480 resolution movies at up to 30 fps. The quality of photos and video that it produces seems to be quite good, I don t have any particular problem with the camera that I want to address by purchasing a new one, so a major aim of buying a new camera was the fact that it s a few years old and something better must be available at a low price! I asked for advice about this on my local LUG mailing list. One piece of advice that was offered by several people was to use a Digital Video camera with Firewire output to record quality video. They spoke about common digital cameras as point and shoot with the acronym P&S - which seems to be used as a term of abuse by people who are really into cameras. If using a DV camera with Firewire support (which is apparently a really important feature) then apparently Kino [4] is the program to use to capture the raw video data. While the advice in regard to a digital video camera sounds good I m not going to take it. Such DV cameras start at about $300 and appear to be physically larger. While a larger camera (with a larger lens) will give better quality pictures and higher frame rates with lower levels of ambient light I am planning to do my serious video recording in a controlled environment with reasonable lighting. Really I m not THAT serious about video recording, so I don t plan to spend $300 or more - I ll accept the lower quality as a trade-off for lower price and greater convenience. I have seen an advert for a camera described as a DC-777 at a good price from DirectBuy.com.au [3]. The camera in question is one that has no reviews anywhere on the net and seems to be a re-badged version of someone else s camera. It initially sounded quite good with 11mega-pixel resolution, but the fine print shows that it has a 7MP CCD and uses interpolation to get a higher resolution. Also the movie capability is 640*480 at 30fps (the same as my Sony). So the benefit of using the DC-777 would be the rotating display which can face the same direction as the lens thus allowing someone who is making a video of themself to see what they are creating. This doesn t seem to be $200 worth of benefit, but the camera might be a reasonable deal for someone who doesn t already have a decent P&S digital camera. Another interesting piece of advice I received from my local LUG was that the amount of time taken to get a good image (which determines the shutter speed on a good camera or the maximum frame rate for video) is determined in part by the size of the elements in the CCD. So with all other things being equal a camera that supports higher resolution will require longer exposure. For my use the benefit of higher resolution photos is often for the purpose of cropping pictures of objects that were too far away to allow completely zooming in. 3* optical zoom (as implemented in my current camera) is a limit when photographing objects that are far away in good light conditions (in poor light it s often a factor of the lens size). In the past when looking at digital cameras I would make the resolution of the picture the prime criteria - with the idea that bigger is better. But now it seems that the smart thing to do is to make optical zoom the prime criteria for general purpose camera use. Also having a larger lens is a significant benefit, but as the overall volume of the camera is going to be roughly proportional to the radius of the lens cubed it can t get that big without some serious compromises on portability. Obviously an SLR [5] offers significant quality benefits. But if you are unwilling to spend so much money a good result can be obtained by what are sometimes called Big Zoom Digital cameras. These are cameras with a body shape that resembles a SLR, they vary in size between the P&S cameras and SLRs, and have fixed lenses that are significantly larger than those of P&S cameras. I gave such a camera to a relative for Christmas and I was very impressed by the quality of pictures and movies that it produced. I chose a Kodak camera with 12* optical zoom which was quite impressive (unfortunately the 15* model had sold out). One of the benefits of a Big Zoom Digital camera over a DSLR is that as there is no option to replace the lens there is also no possibility to get dust inside the camera (which is apparently a problem with SLR cameras). For my own use I plan to stick with my Sony Cybershot. I can t get anything significantly better for an amount of money that I am prepared to spend, and I also don t really want a larger camera. So I think I ll buy a tripod (or maybe two tripods of different sizes), that should deliver a real benefit for my photography without costing much. Update: One thing I forgot to mention is the film speed rating (AKA the ISO number) [6]. A higher ISO number means that the camera can take faster pictures in less light. The Wikipedia page reports that camera manufacturers are permitted to interpret the ISO scheme in several different ways, so they can essentially just make stuff up. Probably the best that can be said for the ISO number is that any camera which advertises it is aimed at more serious users and is therefore likely to be of higher quality. I m sure that if I was to spend $700 or more then I would get a camera with an advertised ISO number that actually means something, but I don t have any plans to ever do that. Matt made a good comment that the quality of the glass matters. For an SLR you can consider the lens quality independently of the quality of the rest of the camera. But for cheap cameras it s just a matter of the entire package.

• [1] http://live.gnome.org/Istanbul


• [2] http://www.dcresource.com/reviews/sony/dsc_t5-review/


• [3] http://www.directbuy.com.au/moreinfo.php?txt_itemcode=DC-777&txt_searchstring=DC-777&txt_from=3&catcode=45


• [4] http://www.kinodv.org/


• [5] http://en.wikipedia.org/wiki/Single-lens_reflex_camera


• [6] http://en.wikipedia.org/wiki/Film_speed

Okay, so I had a simple job to be done. I have two mail accounts: A private mail account and a company mail account, which has two folders containing private mails. I want to synchronize these folders to my private account. Which is the right tool to choose? I thought that one way would be to fire up a graphical mail user agent, like lets say Thunderbird, setup the accounts and simply move the mails between them. But this has some implications, like:

• I don't use such a MUA (instead I use mutt), which makes it unneccessary hard, because I really need to setup the accounts
• Most graphical MUAs I know are bad at working with a great number of mails. At least Thunderbird does not even properly show what he is doing and how long he is supposed to doing it

So I decided that I needed some small tool, that UN*X admins are used to have. After a small apt-cache search, I found two tools:

• imapcopy
• imapsync

I installed both and looked at them. For the impatient, a spoiler: I decided to use imapsync.
I had a quick look at imapcopy and it did not have a proper manpage.
Instead it refers to the built-in help (imapcopy -h) which is not useful either and to examples in /usr/share/doc.
After that I had look at imapsync. It comes with a pretty good manpage and a pretty good built-in usage information. Appearently I rate it very important that either the manpage or the built-in help are good enough to get started with a tool. Certainly I know that tools exist where a manpage is simply not enough, but I guess a tool to sync imap folders is not one of them.

After studying the manpage for about 2 minutes I was ready to construct a command line and give it a --dry try. This parameter lets me see what the tool would do if I would ommit it. That one looked good and so I gave it a shot. It then started to work. It has two flaws.

1. Unfortunately it does not indicate its progress and the normal messages are not a good help either, because they contain numbers that actually do not refer to mails in one of the mailboxes (they are soon literally higher as the number of mails in both mailboxes) and I do understand what it is referring to.
   
2. It sometimes crashed at random locations with random messages. I didn't look deeper into it, because restarting the script helped and therefore I cannot speak of a easy reproducible problem. In 3000 mails it happened about 1-2 times, so not a great deal but still annoying.

Anyway, it did the job, which took some time, because of my bandwith.