(I love memes.) Here s my snowing contribution: Unicode-impaired people may refer to the following codepoints: 0x2603, 0x2744, 0x2745, 0x2746. (Not.)

Gintautas Miliauskas blogs about a computer program that can generate something which reportedly really sounds like composer-written music; apparently it has passed a 'turing test' where people listened to music generated by this thing, and music written by a human composer. I'm afraid, however, I'll have to disappoint him a bit, really. If we ignore the effect that music can have on the emotional state of a living being for a moment, then it's a plain and simple fact that music is pure math:

• 'An octave' is the musician's term for 'a doubling in frequency'. As many people know, a center A on a piano is tuned at 440Hz; it follows that an A one octave higher is tuned at 880Hz, and that the one which is one octave lower will be 220Hz.
• 'Chords' are based around that, too. A regular C chord, for example, contains the C, E, and G notes; and usually, a C at one octave above the first C is included, too. The frequency of a G is exactly halfway between the two C notes, and the E is about halfway between those. Most chords are built in such a manner; the sound waves of those notes will build an interference pattern that is perceived as a chord.
• A melody usually starts with the base note of the key in which the piece is written, and almost always (except during a short time in the early baroque) ends with that same base note. Note that the 'base' note is not the same thing as the "bass" note (although the base is often used in the bass).

There are many other things in music that can be expressed as mathematical rules; learning to how to compose music involves learning those rules, which is a long and tedious process. Now I'm not saying that following those rules will necessarily lead you to an interesting piece of music; the fact that it's possible to create something ugly while still using chords etc. But since so much of it already is math, I can imagine it not being extremely hard to figure out what the other rules are (the ones a composer figures out by imself as opposed to being taught them), transferring them into a computer program, and using that to generate music. That's not to say that such a thing is easy to do, and I'm sure it's still an impressive feat to create a computer program which can create "nice" sounding music; but I don't think this would qualify as 'artificial intelligence'. At least not any more than Deep Blue

Last spring, I posted If Version Contol Systems Were Airlines, which I really enjoyed. Now, because I seem to have a desire to take a good joke way too far, it’s time for: IF PROGRAMMING LANGUAGES WERE CHRISTMAS CAROLS I apologize in advance. (Feel free to add your own verses/carols in the comments.) Away in a Pointer (C) (to Away in a Manger) Away in a pointer, the bits in a row.
A little dereference to see where they go.
I look down upon thee, and what do I see?
A segfault and core dump, right there just for me. I saw thy init there, a reaping away
My process, from its address space, so sorry to say.
I thought I had saved thee, from void pointers all,
But maybe I missed one, and doomed you to fall. Be near me, debugger, I ask thee to stay
Close by my terminal, and help me, I pray;
To find all the bugs and the void pointers too,
And if my kernel oopses, help me reboot for you. Joy to the Wall (Perl) (to Joy to the World) Joy to the Wall, the Perl is come!
Let awk receive her King;
Let every grep prepare him room,
And bash and sed shall sing,
And bash and sed shall sing,
And bash, and bash, and sed shall sing. Joy to the keyboard, we’ll use it all!
Let men, shift keys, employ;
Implicit variables, and globals never fall.
Repeat the line noise now,
Repeat the line noise now,
Repeat, repeat, the line noise now. Perl rules the world with truth and ASCII,
And makes the doctors prove
The glories of carpal tunnel hands,
And we do it more than one way,
And we do it more than one way,
And we do it, and we do it, more than one way. Hark! The Herald Coders Sing (Haskell) (to Hark! The Herald Angels Sing) Hark! The herald coders sing,
“Map and fold, recursive King;
Recursion and patterns wild,
Pure and IO — they’re reconciled!”
Joyful, all ye functions rise,
Join the typeclasses of the types,
With recursion, do proclaim,
“Laziness is born in this domain.” Refrain
Hark! The herald coders sing,
“Map and fold, recursive king!” Monads, by highest Heav’n adored;
Monads, their depths still unexplored;
Late in time, behold they’re good,
Never once were understood.
Veiled in functions, the Monads stay,
Used for IO, and more, each day,
With excitement, Monads say,
“Arrows are stranger, so with us stay.” (Refrain) Hail the glorious compiler of Glasgow!
Hail the threaded run-time system!
Join the beautiful Cabal of Hackage,
Upload there thy perfect package.
We know best, what we will Handle,
You’re safe with us: no pointers, no vandals.
Born to make your exceptions throw,
Unless you unsafePerformIO. (Refrain) Lispy the Paren (to Frosty the Snowman) Lispy the paren was a jolly happy soul,
With a lot of cars and a little cons
And two ends made out of curves.
Lispy the paren is a fairy tale, they say,
He was just common, but the children know
how he came to life one day.
There must have been some magic in that
Old Symbolics they found.
For when they placed him on its disk,
It recursed around and ’round. O, Lispy the paren,
Was recursive as can be.
And the coders say it would take a day
To put his parens away.
Clunkety clunk clunk,
Clunkety clunk clunk,
Look at Lispy go.
Clunkety clunk clunk,
Clunkety clunk clunk,
Consing on the car. Lispy the snowman knew
The keyboard was hot the day,
So he said, “Let’s cons and we’ll have some fun
now before they Scheme away.”
Down to the function,
With a list there in his RAM,
Running here and there,
all around the LAN, saying
“cdr me if you can.”
He led them down the streets of disk
Right to the traffic bus.
And only paused a moment when
He heard them holler (quit). Oh BASIC Night (to O Holy Night) Oh BASIC night, the LEDs are brightly glinting;
It is the night of the dear GOSUB’s birth!
Long lay the world in sin and error printing,
Till you appeared and the RAM felt its worth.
Shiver of fear, line numbers do inspire,
For yonder breaks a mostly harmless GOTO.
Fall on your bits, O hear the Visual voices!
O BASIC divine, O BASIC where GOTO was born!
O BASIC, O Holy BASIC, O BASIC, you’re mine! Some want to say, “GOTO is harmful always,”
But what of them, in their post-modern world.
We PRINT the truth, in the line-numbered goodness,
But Dijkstra appeared, and the faith, it was lost.
A thrill of hope, when .NET BASIC announces,
But Visual BASIC, what kind of thing are you?
Fall on your GUI, O see the old line numbers!
Behold BASICA, O BASIC when DOS was born!
O numbers, O lines, spaghetti divine! Guido We Have Heard on High (Python) (to Angels We Have Heard on High) Guido we have hard on high
Sweetly indenting o’re the code,
And the functions in reply
Their exceptions sweetly flowed. Refrain Indent….. in your whitespace careful!
Indent…… in your whitespace careful! Spaces, why this jubilee?
Why semicolons have you so wronged?
What backslashes must we use
If we want our lines so long? (Refrain) Come to Guido here to see
“One Right Way” is good, of course.
There’s no need for Perl, you know,
We have to be more verbose. (Refrain) Now the PEP will show the way
To the future, we shall see.
Banish lambda and the rest
Of the things we liked the best. (Refrain)

Thanks to Koha, TTLLP is partly a library services worker cooperative and just now there’s a massive flame-storm about the largest library services consumer cooperative - OCLC, the Online Computer Library Center - because it just updated the terms for sharing its book data collection. OCLC seems to be treating it as a private asset that it can exploit, not the common resource that many librarians thought it was. I’ve been watching this fire with interest - because, at best, I don’t think it’s helping other cooperatives that sell to libraries - and here is my summary. OCLC and the Great Library Scandal is a good introduction and there’s a summary of the Talis Podcast about OCLC WorldCat Record Use Policy with Karen Clahoun and Roy Tennant which reveals some of the OCLC thinking. Just like CC and Open Source? Still No is a great illustration of how some librarians think OCLC is reaching the wrong conclusion, which then leads to asking Is OCLC truly cooperative? and What would it look like if OCLC was broken up? A commenter on Tom Watson: Library data asks: “Why would libraries play this game?” Well, the road to hell is paved with good intentions and this road has taken 40 years. First off, OCLC is essentially a good idea - libraries cooperating to act as a counter-weight to the large library service companies. However, I believe OCLC predates many recent company rule innovations (stronger common asset locks) and data licensing innovations (Share-Alike, Fiduciary Licensing and the general free and open source software movements), so there’s nothing to stop them trying to privatise the assets (library data) that the libraries have given to OCLC and it was only a matter of time before someone tried. I’m having a similar discussion with another organisation that has been given data by its members under vague terms and is now apparently about to exploit that data for the organisation’s benefit, to the detriment of some members. TTLLP is a member but has not contributed much data because I was suspicious of the lack of terms. In time, hopefully, asset locks and more awareness of data licensing will eliminate these problems, but there’s going to be a lot more people getting burnt first. Finally, a quick shout-out to the Open Library Environment as a possible emerging alternative to OCLC for practical services. Raw Thought: Stealing Your Library: The OCLC Powergrab covers some of its motivation (and includes a link to a petition to OCLC, if you want something to sign), but I share Stefano’s (now there’s someone I’d not read in 7 or so years) analysis of some of the challenges:

“any grass-root approach that will get big enough to take on OCLC on the metadata collection and redistribution service that libraries need will have to incorporate under the pressure of its users (if only for legal liability protection) and will have to find an answer to the same set of problems (policy, governance, financial sustainability) that OCLC has.”

This might be a good message to write in Spanish... But then again, a long time ago I decided this is an English-posting site. So be it, I'll only have to give more background information.
This day marks the date when, 98 years ago, Francisco I. Madero started the Mexican Revolution - About a decade of unrest, civil war and ideologies. The revolution is what created the violent, uncivil image of the Mexican, which accompanied us for long years in many foreigners' minds. The revolution brought to an end 30 years of a single-man rule, the Porfiriato. But that's only the major symptom - The Revolution had many, many other consequences. About one million (out of a 10 million population) people died. There was a very significative rearrangement of the society, a rearrangement that took about half a century to settle. But I won't write more background - You can always ask the wikipedia about our Revolution.
The reason I am posting this is that, as it usually happens in this time of year, several so-called analysts in the media have started asking, was the Revolution really worth it? Did it change anything at all? Did the Revolution in the end win, or was it defeated from within? Should we still celebrate it?
And there are, yes, reasons to doubt it. Renato Leduc, at the same time a great journalist and a delicious poet, says it as many - while at the same time, as nobody else: Tiempos en que era Dios omnipotente / y el se or Don Porfirio presidente / Tiempos, ay! tan iguales al presente, or ya se est n muriendo todos / Jes s qu desilusi n...! / se est volviendo gobierno / Ay dios...! La revoluci n.
Anyway... Our media overlords insist on us forgetting the struggles and the real changes that came from them, on rewriting the history... Probably they will push us later on to have the cristeros as the real fathers of the Nation?
Even if so many bits of reality didn't change after Porfirio Diaz's regime fell in 1910, I find it insulting to think that even 70 years of PRI -with very sharp differences between periods, with huge differences between the PRI-born governments- are comparable to 30 years of a one-man rule; even with brutal repressions such as the dirty war against so many subversive movements in the 50s-80s (as officially There Was No Armed Struggle Anymore, just some pesky communist subversives), it cannot be compared to the Porfirian Peace (ask Cananea and R o Blanco). Today we might have a shameful concentration of money and power in very few hands (including the world's richest man), but it certainly does not reach the point of 1910 where most of the Mexican soil was owned by less than 30 families, with latifundios as big as many states...
Anyway - So far, nothing new - just bits I heard here and there, and my reactions to them. But this morning, around 8:25, I tuned in to Noticias IMER, the news program of one of the few public, non-gubernamental, independent radio stations. An interview was under way, but I could not get the interviewed person's name (I guess, a historian - will write to ask for his data). His comments were very interesting, and very worth echoing. I'll try not to distort him.
The Revolution started off very organized, and with a very simple goal: Get Porfirio Diaz out, and call for real federal elections. Sufragio efectivo, no reelecci n. Of course, the fight was very short, and Madero became the president, with an overwhelming majority. Of course, also, the reactionary sectors set up a coup and killed Madero. Victoriano Huerta seized the power - and that's where the real revolution really began. Groups all over the country (some of which were at unrest since Madero, as they were not seeing the changes they needed - changes that would bring an end to the huge class differences and disrespect to the native Mexican population) rose in arms, and forced Huerta into exile. Then, they battled each other for many more years. It became known as la bola - When somebody joined the revolutionary forces, people said he went to fight with the crowd. But, inside the crowd, there were very different points of view. No, Carranza, Villa and Zapata (the foremost leaders in the hardest part of the fight) were not power-hungry barbarians - much to the contrary. They had very full, very complex views of the problem and possible solutions. I won't delve much into them, also, as I'm not an expert...
Villa and Zapata had the most compatible approaches, seeking an aggresive land redistribution, a communal property system (closest to most of the indigenous population's roots, what we would now call usos y costumbres). For the government, both favored going towards a Europe-like parliamentary system, where the parliament were the real force, and the president (or prime minister or whatever) would only be the designated person to implement the parliament's decision. Both Villa and Zapata feared the evil stemming from the unlimited power that the Presidential Chair symbolized (Fui soldado de Francisco Villa / de aquel hombre de fama mundial, / que aunque estuvo sentado en la silla / no envidiaba la presidencial). They met at the Aguascalientes convention, and were quite close to each other - but were defeated by the superior Venustiano Carranza (Constitucionalista) army.
Carranza, although vilified for his corruption (nowadays, carrancear is still a synonim for stealing), had an opposite view - also originating from a very deep analysis. Carranza saw that what brought down Madero was, in the end, the lack of power of the President to rule the country without support from the legislative power. So, he pushed a political program making the President the strongest man in Mexico. He and his people wrote and passed the 1917 Constitution, valid today. This constitution goes to great lengths pushing revolutionary ideals - Land and wealth redistribution, universal and free education, keeps a complete separation between state and church, ensures state control over strategic areas... The 1917 constitution is one of our history's greatest achievements.
But, of course, it is not perfect - it paved the way for a hegemonic party controlling the real power behind it all. PRI started as a very heterogeneous mixture of the whole revolutionary family, but slowly became a bureaucratic, stagnated monolith.
And in a somehow ironic twist of destiny, the forces that today push for deepest changes, and precisely in the same direction that Villa and Zapata wished, are... Ej rcito Zapatista de Liberaci n Nacional (EZLN) and Frente Popular Francisco Villa (FPFV). EZLN is far more successful and advanced in its social experiment. Again, I won't comment further in what I don't really understand.
As a last point, the commenter I'm quoting (and whose name I must get, to update this post!), said that practically in every country that has transited from any sort of dictatorship towards a more-or-less believable democracy (say, everywhere in South America, or Spain, or Eastern Europe, or...), one of the first steps has been to update or replace the constitution with a new one, preventing the mistakes overlooked by the previous one from being reinstated. In our country, we have long heard about the "Reforma del Estado", a very nice-sounding-term which nobody believes in. After the 2006 electoral mess (no matter who won in the end, everybody will agree it was a mess that should be prevented from happening again, we had high hopes of real changes being introduced. A parliamentary, or at least semi-presidentialist regime was strongly suggested as a way forward. Changing the electoral system towards having second-rounds if needed. _anything!_ But no, we were stuck with... The same as always.
So, did the Revolution win or lose? It is clear to me. It won, and it really shaped -for better- what would happen in the next 100 years. However, in a century, we have been able to twist the law to make it turn against itself. I have to agree with my EZLN-minded friends (I sympathize with EZLN's general goals, but don't think its way forward is the right way to go): Pushing the change from within the government is just wishful thinking, but a strong delusion. However, is there a way to push our country forward without repeating a violent cycle? I really hope so. Our current situation is simply pathetic.
I lack a good closing for this post... So I'll let good old Jefe Pluma Blanca, Renato Leduc, do it for me.
Tiempos de Pancho Villa
y de la guerra de mentadas y tiros en la sierra.
Tiempos de fe
no en Dios sino en la tierra Por el cerro de la Pila
fueron entrando a Torre n
mi general Pancho Villa
y atr s la revoluci n...
Ay jijos...! ya se nos hizo
cu nto diablo bigot n... Ya viene Toribio Ortega
subiendo y bajando cerros
y no te enredes ni enga es
que ah anda Pablito Se ez
haciendo ladrar los perros. Cu nto usurero barb n...!
Ay jijos... c mo les vuela
de la levita el fald n...!
Ay jijos... ya se nos hizo:
triunf la revoluci n...! Tenemos camino andado...
No hay que juntarse con rotos
siempre te juegan traici n
ya Madero est vengado
ya muri la usurpaci n. En su caballo retinto
lleg Emiliano Zapata
bonita su silla charra
y sus botones de plata
pero mucho m s bonito
su famoso Plan de Ayala... Este gallo es de navaja
y no es gallo de espol n
si quieres tierra trabaja
trabaja no seas huev n... Ya lleg don Venustiano
con sus anteojos oscuros
y Villa y Zapata gritan:
No s que tengo en los ojos...
porque ya en Pablo Gonz lez
se vislumbra la traici n
Ay reata no te revientes
que es el ltimo jal n...! ya se est n muriendo todos
Jes s qu desilusi n...!
se est volviendo gobierno
Ay dios...! La revoluci n.

I have been using Serendipity on my blog for some time now. Overall, I’ve been pleased with it, but the conversion was a pain. Serendipity is a simple blog engine, and has a wonderful built-in plugin system. It can detect what plugins need upgrading, and install those upgrades, all from directly within the management interface. There’s no unzipping stuff in install directories as with WordPress.

On the other hand, Serendipity has had some long-standing issues. #1 in the list is spam protection. I’ve been using the Captha system, but as some others have commented on here periodically, it can be a hassle for legitimate humans. The Serendipity implementation in particular, I mean. Another issue was with the plugins themselves. Because the software does updates itself, and plugins tend to write things in the install directory, your Serendipity install directory has to be writable by the webserver. Morever, it is difficult to keep track of local changes — forcing me to use git on the tree, and even then, it’s difficult. Finally, there has been some long-standing weirdness with the RSS feeds. I switched to WordPress over the weekend. It doesn’t solve everything, but it is a nice improvement. It has a ton of themes available, though many of them are not as high-quality as the themes in Serendipity, and it is hard to pick out the good ones quickly. WordPress has a larger development community and seems to be getting new features and improvements at a more rapid rate than Serendipity. I particularly like how it automatically saves drafts as I’m typing. That is a wonderful feature. I can’t tell how many draft posts I’ve lost by accidentally clicking on the wrong thing. Serendipity has a ton of plugins as well. I looked for Twitter plugins today. There are maybe a dozen of them. But there is no central tracking of them, so I have no idea which ones might be nicely peer-reviewed and which ones might rm -rf / on my server. Also, I don’t know which ones are going to be supported and upgraded over time. I’m happy with the way most WordPress themes work nicely with widgets. Though I am baffled that some WP themes hard-code in things like the calendar or archives when they could just let people insert a widget for it. The Conversion Process This was a real pain. I started with a Serendipity to WordPress converter. But alas, it only supports Serendipity databases in MySQL and mine’s in PostgreSQL. I used SQuirreL SQL with the DBCopy plugin to convert it. This worked… sorta. A few tables didn’t convert, but I didn’t need those. Some others had indexes that wouldn’t convert, and after re-trying it enough times, it went. Then I ran the converter. Alas it had some bugs and had to be fixed up with current table names and some things like that. Also I hacked it to preserve ID numbers. So eventually this migrated over the posts and categories. It did not preserve the category hierarchy. It also didn’t recognize when nl2br was disabled, and munge the post appropriately. And it didn’t migrate over tags, so I wrote a Haskell program to do that. Then it was a matter of crafting a bunch of RedirectPermanent and RewriteRule entries for Apache to try to redirect the old article pages and RSS feeds to their new locations as much as possible. Final Thoughts I’m glad I did it, and I don’t plan to do this again for a long time.

Looking at:

http://www.iht.com/articles/ap/2008/11/13/america/Gay-Marriage.php

It looks like Connecticut now has what we had a week or so ago. And isn't staring a proposition in the eyes to get rid of it.

Someone explain to be how gay San Francisco can't have weddings and somewhere-by-DC Connecticut can?

I wouldn't have wanted folks in Connecticut to have to wait any longer, but I admit that I'm having trouble overcoming my sadness at Prop 8 enough to celebrate this the way it should be. It's an awesome ruling, and I wish people many long and happy marriages. I love being married and think that everyone should be able to have the same shot at this.

If I’m not mistaken, this is at least the second time I see this meme going on, though I didn’t participate the first time… Here are the rules:

• Grab the nearest book.
• Open it to page 56.
• Find the fifth sentence.
• Post the text of the sentence in your journal along with these instructions.
• Don t dig for your favorite book, the cool book, or the intellectual one: pick the CLOSEST.

And the result:

From 12 by

Effective immediatly -- If you're annoyed by the spelling of something I've written, I will accept a patch -- preferably generated by git-format-patch, but diff -u is also acceptable. Any other communication about spelling mistakes will be ignored, unless the mistake has ramifications that will cause undue pain and suffering to people who are not English majors. Furthermore, if the spelling "mistake" is that I spelled "-ize" as "-ise", it's not a mistake -- I prefer to use the latter form for obscure reasons, with a few exceptions. PS, I realise that these are entirely arbitrary rules forced upon you willy-nilly. Teh irony..

(Written on Friday 16th, posted on Monday)

As I write this, I'm on a plane off the coast of the Netherlands, on the way back from a couple of days designing APIs with Rob McQueen and the RTCom people at NRC Helsinki (who we work with on the chat and VoIP functionality for Nokia internet tablets).

We've had a very useful couple of days, mainly designing the next-generation channel requesting and dispatch API for Telepathy (the same APIs I've been doing preparatory work for on the Telepathy mailing list in the last couple of weeks). API sketches for review and comment are either on are on Merge Monkey or on the way there.

Before even reaching Nokia, Rob and I did a lot of design on the plane over to Helsinki, and I took some notes: here they are, by way of a glimpse at how the Telepathy design process works when we don't have a whiteboard[1] :-)

• Geoloc looks good in principle, needs some more work
  • add more docstrings
  • add a way to ask people for uncached info (RequestLocations)
  • move access control to Co.I.Presence, call it RichPresenceAccessControl?
• DeliveryReporting looks good, ish
  • improve wording of delivery-echo and rationale
  • when sending a delivery report, add a reason for the status (using Channel_Text_Send_Error?)
• Uniqueness requirement for (channel, handle_type != 0, handle) is in the way - let's drop it
• MSN should use "authentic" 1-1 chats, and use Ch.I.UpgradeableToRoom (name tbd) to migrate the underlying switchboard to be a room
• XMPP should use Ch.I.UpgradeableToRoom too, but the 1-1 channel may also stay open
• Mutable handle => handle 0 must stay (for requestotron's benefit)
• bundles ftw
• renaming: we want SelfHandleChanged in the core
• TpC* - add ability to subscribe to ifaces ay construct time (blocks Ready) or later
• Dispatching: when registering a c'handler, specify everything it can handle
• If a new bundle can be handled in its entirety by the same handler (call it 'the bundle handler'), do it. Otherwise split it up completely (there is no 'bundle handler').
• If a channel in an existing bundle can be handled by the bundle handler, send it there, else send to the default handler.
• Group - add SelfHandleChanged, maybe? Also Connection
• Add HandleOwners: prop a uu , HandleOwnersChanged(a uu , au)
• Use TargetHandleType, TargetHandle
• Stringified initiator handle as a separate property
• Maybe even a FirstMessage property, although this stretches the definition of "immutable"
• org.freedesktop.Telepathy.Client interface

Nokia's Mikhail Zabaluev kindly took notes for some further discussion we had with him and Alberto Mardegan about the dispatch API, which he already posted to the mailing list.

We hope this API will make it possible to integrate Telepathy into the desktop better - it should make it a lot easier to share connections between processes in a useful way and hand out incoming channels to the appropriate handlers, which was always a major goal for Telepathy. It should also let us and others implement exciting new notification mechanisms (hi to the people on IRC and the mailing list wanting to patch their media players to integrate with Telepathy! :-)

[1]	When we do have a whiteboard, the design process is: "Robot101 scribbles on a whiteboard, smcv writes down <tp:rationale> elements"

With a small mail to our mirrors-announce list, I just opened the work of more than one month to the public. Even better, I strongly encourage every mirror admin to use my scripts, or at least scripts that follow a minimal set of rules I wrote down… Some long time (somewhere in April or May this year) I got added to the Debian mirroradm group. First it was only to get some problems with packages.debian.org solved, but later on I started doing more. This group is basically the one that handles the archvsync user on .debian.org machines and also manages the official Debian mirrors listing. As the archvsync user exists on multiple .debian.org machines, and many of them do mirror a copy of our archive for various reasons, all of them had some version of a mirroring script, and sometimes also a version of a pushmirror script. Unfortunately, as they all had been maintained mostly on their own per machine - they sometimes had evolved a bit different. Additionally all of them had a few problems which could leave us with a broken mirror behind, despite it looking all well. So on Sept. 26 I decided to trash it all and rewrite the script and put it all in a VCS (and selected git). For that I set myself a few goals, like (in no particular order)

• It is all proper set -u,
• should have as less external dependencies as possible and should work by only using bash (only the runmirrors script does need egrep),
• should be as flexible as possible and support as many different configs as one can imagine without the need to change the script itself,
• should replace the existing mirror scripts of Debian and not only be for the archvsync user alone,

Now, 109 commits and more than a month later, I was finally able to open the scripts to the wide public. (Had multiple people test them before this, but thats always different than a real “release”). The scripts are in use on all .debian.org machines and some other mirrors already, so I’m fairly sure they work.

---

As I already wrote, Debian mirror admins are strongly encouraged to use this scriptset and not to invent their own. In case you miss a feature either send me a mail describing it, or better, a patch implementing it. I’m happy to implement as much as possible (and sensible :) ) And while it is discouraged to use own scripts, sometimes people have reasons to have their own, so there is now a minimal set of functionality that such a script MUST support.

• It must perform a 2-stage sync. The archive mirroring must be done in 2 stages. The first rsync run must ignore the index files. The correct exclude options for the first rsync run are:

  --exclude Packages* --exclude Sources* --exclude Release* --exclude ls-lR*
  

  The first stage must not delete any files. The second stage should then transfer the above excluded files and delete files that no longer belong on the mirror. Rationale: If archive mirroring is done in a single stage, there will be periods of time during which the index files will reference files not yet mirrored. And that will lead to a lot of confusion for our users.
• Must not ignore pushes whilst running. If a push is received during a run of the mirror sync, it must not be ignored. The whole synchronization process must be rerun. Rationale: Most implementations of Debian mirror scripts will leave the mirror in an inconsistent state in the event of a second push being received while the first sync is still running. It is likely that in the near future, the frequency of pushes will increase.
• Should understand multi-stage pushes. The script should parse the arguments it gets via ssh, and if they contain a hint to only sync stage1 or stage2, then only those steps should be performed. Rationale: This enables us to coordinate the timing of the first and second stage pushes and minimize the time during which the archive is desynchronized. This is especially important for mirrors that are involved in a round robin or GeoDNS setup. The minimum arguments the script has to understand are:
  • sync:stage1 Only sync stage1
  • sync:stage2 Only sync stage2
  • sync:all Do everything. Default if none of stage1/2 are present. There are more possible arguments, for a complete list see the ftpsync script in our git repository.

All of those requirements are easy to fulfill. Well, at least the MUST ones. The first one is basically a given, all scripts I know are doing this already. The second one sounds more complicated, but actually might just be about 4 lines of actual code changes. Just touch an extra file before you gain a lockfile (to avoid two scripts running at the same time) and then rerun rsync as long as that extra file is around. (It helps to rm this trigger file right before you call rsync :) ). The should directive is actually a little more work to implement correctly. I won’t show source here, read the ftpsync script if you want to see it, but I try some small explanation. Background: Debian mirror synchronisation is usually done by scripts that get started via ssh. As mirror operators (understandably) do not like their upstream mirror to be able to login, this is done using “ssh triggers”, a feature I wrote about earlier. Those are sshkeys using a forced command, so nothing else than this command will be run, ever. SSH, if used with such forced commands, has the helpful feature that it sets the environment variable SSH_ORIGINAL_COMMAND containing the original arguments. Which means that, if the mirror pushes you with “ssh yourhost -l mirror sleep 1” you will see *sleep 1 in that variable. It is what many mirror operators currently use when pushing their downstreams. Now, we can use that to tell the receiving end all things we want it to know. If you look in my script you will see that I defined a little more than the minimal arguments. But none of the additional ones are a requirement, they only make life much easier. My script currently understands, additional to the 3 minimal ones:

• sync:archive:$FOO This allows your upstream to tell you what archive needs an update. This way you can receive many different archives from your upstream, but only need to setup one ssh key. Adding another archive to the list you get notified about from them is now just copying the config file in place. (But the main reason is to not need dozens of keys and easy management of the authorized_keys files)
• sync:callback A feature I don’t yet use, but have implemented, are callbacks. Initially I planned to do my “update DNS RR listed mirrors at the same time” using callbacks, but then I found a better way for doing that, see further below. The idea is that you can configure a user, a target host and an ssh key. And “call back”, ie. tell your upstream you finished mirroring the archive. Could be used for statistics. Or, thinking a little more complicated, your upstream could only allow rsync access for the time between his push to you and you calling back. Or whatever, I’m sure people will come up with ideas.

---

As if that wasn’t enough, I also had to rewrite the runmirrors script. The old 10-line one just didn’t cope with the more general approach. This script is used to “push” your downstream mirrors, telling them that they need to update. (Which is the only sane way to run a mirror. cron is just not made for mirrors that can update at random times!). One nice new feature of it is the ability to update multiple mirrors nearly at the same time. This is extremely helpful in case of DNS Round Robin entries like security.debian.org (multiple hosts offering the same archive under one hostname). You do not want to have such hosts out of sync. Not even for a minute, if you can avoid that. So we now push the mirrors with sync:stage1 as argument. Then we wait until all of them have finished the stage1 sync before we push them all again, this time with sync:stage2 as argument. This will erase any update delays thanks to network latency / low bandwidth one of the security mirrors might have, as the large transfers (all the binary and source packages) are done in stage1. They can take time, a normal user won’t notice it - the Package and Source indices are not transferred yet. They arevery small and as such transfered pretty fast whenever stage1 is done everywhere and we tell mirrors to do stage2.

---

All in all it was a pretty nice exercise writing those scripts. Comments: 0

I opened my eyes this morning at 7:00, and immediately felt he was also awake. He got up and briefly wandered around. I felt this incredible expectation, even while barely awake. Something important had happened. As he returned to bed, I asked Who won? . Monsieur Obama , he replied as he slipped back into bed. I broadly smiled and he cuddled up against me, hugging me from behind. I peeked through the open bedroom door and could see the cats smiling in the living-room. Still one more hour before I had to get up for work and life was already good. That early in the morning! Definitely good.

Last night they could, and Yes! they did. And I am truly thankful.

(or Taxation without Representation) Wow, it's finally over, and best and right man won. I'm very excited and happy with the result. It has been very interesting being able to observe the entire process from start to finish. All the primaries and caucuses, the protracted battle between Clinton and Obama in the Democratic primaries, the record breaking money Obama raised, the campaigning, the misinformation. It'll be very interesting to see what the next four years brings. It's an exciting time to be living in America. California is an interesting state. It has "ballot initiatives". Its constitution can (and has been) amended by essentially referendums whenever there's some other excuse for a vote. Anyone can propose a ballot initiative with sufficient backing by petition. Wikipedia has a good writeup of all of California's propositions. Probably the most controversial one this time around is Proposition 8, which seeks to essentially reverse the same-sex marriage right that the Californian Superior Court ruled on earlier this year. This concept of ballot initiatives seems to trickle all the way down to the city level, where they're called Measures. San Francisco has some ridiculous number of them this time around. They use letters, and I think they go all the way to V. There's one to rename a sewage treatment plant after George W Bush, there's one to decriminalise prostitution. A co-worker who lives in San Francisco brought the official voter guide book to work with her today. It's about the size of a phone book and about 1.5 centimetres thick. I don't see how it's possible to make an informed decision on everything you're expected to vote on, when there's that much reading. The observation I've made is that Americans seem to have an inherent distrust of their governments, so that's why so many bits of it are directly elected by the people. Judges are voted for, county sheriffs are voted for. You vote for your school board. It seems like very few positions of any power are appointed by someone. Coming from Australia, where it's really the opposite, I'm not sure what I think is better. I think I like the simplicity of the Australian system. You vote for someone, and you essentially delegate power to and place trust in them.

My church is one in which politics are checked at the door. Some church members wear their politics on their yard, or on their blog -- and just about every opinion is represented in the church. But you rarely hear politics mentioned in church. When it is mentioned, it's issue-oriented rather than candidate-oriented or policy-oriented -- we'll hear updates on efforts to create a peace tax fund, for instance.

But today, hearing about politics is just about unavoidable.

The relationship between Christianity and government has been uneasy and troubled all the way back to the religion's founding. Many Christians, and I count myself in this, believe that our first loyalty is to Jesus, and on those grounds, refuse to say the pledge of allegiance. What, we wonder, would our word be worth if we were forced to disobey our government because of a law that was unjust or immoral? How could we even say the words "one nation, under God, with liberty and justice for all" when those words were written at a time when the KKK was active, lynchings were common, and are said today at a time when people treat Muslims and immigrants with modern disdain?

In short, we believe we are called to be citizens of a different kingdom first.

So, today, our pastor deliberately picked a difficult scripture passage for us: Romans 13:1-7, which reads, in part:

Everyone must submit himself to the governing authorities. . . The authorities that exist have been established by God. . . Consequently, he who rebels against the authority is rebelling against what God has instituted, and those who do so will bring judgment on themselves. For rulers hold no terror for those who do right, but for those who do wrong. . . Therefore, it is necessary to submit to the authorities, not only because of possible punishment but also because of conscience.

What an extraordinary set of statements. This was written during the time of the Roman Empire, which could hardly be said to have been a just and benign government. It's hard for me to imagine the Roman Legion being established by God.

In more modern times, it would seem to denounce the American revolution as a rebellion against authority and therefore a rebellion against God. It would also seem to denounce the protests that we see all over the world -- striking workers in France, human rights seekers in Burma, war protesters in the United States. Would it even have condemned the protests in the 1960s over civil rights in this country, or the protests against war today?

One commentator notes that "Paul is not stating that this will always be true but is describing the proper, ideal function of rulers. When civil rulers overstep their proper function, the Christian is to obey God rather than human authorities" -- a theme Paul mentioned more than once in Acts.

What relevance does this have for us today? It seems that we are to help our rulers act in a just way, even if we disagree with them -- no matter who wins the election. It is also a reminder that a superficial reading of the Bible, taken out of context and without a deep understanding to understand the author's point, can potentially lead to very strange conclusions.

The American National Council of Churches has issued a non-partisan voting guide, which we found in our bulletin today. It is an interesting read, and probably not what you think; it begins with, "War is contrary to the will of God." Thought-provoking stuff.

I find it interesting that there are a lot of people out there that say that religion is responsible for a lot of ill in this country, then proceed to hold pretty much the same opinions I do for pretty much the same reasons. I just point out that the Bible is deeper than intolerance and submission.

One significant dividing factor between mailing lists is the difference between summary lists (where the person who asks a question receives replies off-list and then sends a summary to the list) and the majority of mailing lists which are discussion lists (where every reply goes to the list by default). I have seen an argument put forward that trusting the answers on a mailing list that operates under the summary list model is inherently risky and that peer review is required. It could be argued that the process of sending a summary to the list is the peer review. I’m sure that if someone posts a summary which includes some outrageously bad idea then there will be some commentary in response. Of course the down-side to this is that it takes a few days for responses to the question to arrive and as it’s common that computer problems need to be solved in hours not days the problem will be solved (one way or another) before the summary message is written. But the idea of peer review in mailing lists seems to fall down in many other ways. The first problem with the idea of peer review is that the usual aim of mailing lists is that most people will first ask google and only ask the list if a reasonable google search fails (probably most mailing lists would fall apart under the load of repeated questions otherwise). Therefore I expect that the majority of such problems to be solved by reading a web page (with no peer review that is easily accessible). Some of those web pages contain bad advice and part of the skill involved in solving any problem relates to recognising which advice to follow. Also it’s not uncommon for a question on a discussion list to result in a discussion with two or more radically different points of view being strongly supported. I think that as a general rule there is little benefit in asking for advice if you lack any ability to determine whether the advice is any good, and which of the possible pieces of good advice actually apply to your situation. Sometimes you can recognise good advice by the people who offer it, in a small community such as a mailing list it’s easy to recognise the people who have a history of offering reasonable advice. It seems that the main disadvantage of asking google when compared to asking a mailing list is that the google results will in most cases contain links to web sites written by people who you don’t know. Sometimes the advice is easy to assess, for example if someone recommends a little-known and badly documented command-line option for a utility it’s easy to read the man page and not overly difficult to read the source to discover whether it is a useful solution. Even testing a suggested solution is usually a viable option. Also it’s often the case that doing a google search on a recommended solution will be very informative (sometimes you see web pages saying “here’s something I tried which failed”). Recommendations based on personal experience are less reliable due to statistical issues (consider the the regular disagreements about the reliability of hard disks where some people claim that RAID is not necessary due to not having seen failures while others claim that RAID-5 is inadequate because it has failed them). There are also issues of different requirements, trivial issues such as the amount of money that can be spent will often determine which (if any) of the pieces of good advice can be adopted. The fact that a large number of people (possibly the majority of Internet users) regularly forward as fact rumors that are debunked by Snopes.com (the main site for debunking urban legends) seems to indicate that it is always going to be impossible to increase the quality of advice beyond a certain level. A significant portion of the people on the net are either unwilling to spend a small amount of effort in determining the accuracy of information that they send around or are so gullible that they believe such things beyond the possibility of doubt. Consider that the next time you ask for advice on a technical issue, you may receive a response from someone who forwarded a rumor that was debunked by Snopes. Sometimes technical advice is just inherently dangerous because it is impossible to verify the integrity of some code that is being shared, or because it may be based on different versions of software. In a previous blog post I analyse some issues related to security of the Amazon EC2 service [1]. While the EC2 service is great in many ways (and implements a good well-documented set of security features on the servers) the unsigned code for managing it and the old versions of the images that they offer to customers raise some serious issues that provide avenues for attack. Getting the EC2 management tools to work correctly on Debian is not trivial, I have released patches but will not release packages for legal reasons. It seems most likely to me that someone will release packages based on my patches (either because they don’t care about the legal issues or they have legal advice suggesting that such things are OK - maybe due to residing in a different jurisdiction). Then people who download such packages will have to determine whether they trust the person who built them. They may also have the issue of Amazon offering a newer version of the software than that which is packaged for Debian (for all I know Amazon released a new version yesterday). The term integrity when applied to computers refers to either accidental or malicious damage to data [2]. In the context of mailing list discussions this means both poorly considered advice and acts of malice (which when you consider spam and undisclosed conflicts of interest are actually quite common). If you ask for advice in any forum (and I use the term in it’s broadest sense to cover web “forums”, IRC, twitter, etc) then getting a useful result will depend on having the majority of members of the forum possessing sufficient integrity and skill, being able to recognise the people whose advice should be followed, or being able to recognise good advice on it’s own. I can think of few examples of forums of which I have been involved where the level of skill was sufficient to provide quality answers (and refutations for bad answers) for all areas of discussion that were on topic. People whose advice should generally be followed will often offer advice on areas where their skills are less well developed, someone whose advice can be blindly followed in regard to topic A may not be a reliable source for advice on topic B - which can cause confusion if the topics in question are closely related. Finally a fundamental difference between “peer review” (as applied to conferences and academic journals) is that review for conferences and journals is conducted before the presentation. Not only does the work have to be good enough to pass the review, but the people doing it will never be sure what the threshold is (and will generally want to do more than a minimal effort) so the quality will be quite high. While peer review in mailing lists is mostly based around the presence or absence of flames. A message which doesn’t attract flames will either have some minimal quality or be related to a topic that is not well known (so no-one regards it as being obviously wrong). Update: The “peer review” process of publishing a post on my blog revealed that I had incorrectly used who’s instead of whose.

• [1] http://etbe.coker.com.au/2008/10/13/ec2-security/


• [2] http://en.wikipedia.org/wiki/Data_integrity

Share This

The Debian Lenny release has still quite a big list of Release Critical bugs, and as Debian developer I feel that I should do my share, and at least look at the list of bugs and see if there's any I could do something for.

Almost always though, I find that I won't touch the remaining bugs in the list.because of one or more of these reasons:

1. a package I really don't care about
2. hugely complex package or might also use something obscure like cddb
3. it seems like people are already looking into it
4. it requires a political solution, not a technical one
   

As part of the effort to get Lenny released, Joss Mouette started the Debian Bug Sprint.
This is really a cool concept, and it is a shame not more people participated.

The reason for me it is cool is that it forced me to break the rules I mentioned above, because I got a bug which fitted 1 and 4 of my list above!

Turned out this bug is really a border case in interpretation of Debian policy.

The package is perfectly usable without any extern dependencies, hence it is currently in the main section of Debian. However it doesn't end there. The package also has a download script that can fetch firmware images for certain printers. It appears for people with these printers the package is NOT usable without external files.

Aparantly though, the current, as one person involved in the bug calls it, "spirit" of Debian is to tolerate this, as it is good enough that it is usable without external dependencies for SOME persons.

However the bug submitter doesn't agree, and thinks this is a case that needs addressing withing the Debian project.

I suggested splitting up the package and moving the download script to contrib, but this was mostly dismissed as idea.

The end result is that the maintainer decided to escalate the problem to the Debian CTTE.
This means it probable won't be solved by today, soo I will have to bake cookies for someone :)

I'm now reaching the very bottom of the Debian localization radar that points out packages with pending debconf l10n bug reports. About 20 packages are currently still under work (either waiting for the maintainer's answer, or in a translation update round...or waiting for the package to go out from DELAYED). That should be completed in about 10 days. The remaining packages are those with very recent l10n bug reports and I can't decently nag maintainers for bugs sent since less than one month. The result is now that we have 3 languages that reached near 100% completeness: German, French, Portuguese. Czech is over 97%. Swedish has been dramatically revived and now over 85%. And some more translation teams such as the Basque, Russian, Italian and others have taken this last opportunity to get as many translations as possible in the game. The overall quality of several packages I NMU is not optimal. I often refrained myself from trivial fixes as this is not the policy during freezes but that's really sad. I might initiate a QA NMU campaign when we enter the squeeze development round, to fix all those trivial lintian warnings (obsolete debhelper compat level, maintainer scripts ignoring errors, debian/rules ignoring make errors, typos here and there, etc.) Anyway, once this l10n NMU stuff is done, the question becomes: what will I do now and until we release lenny? Maybe come back to the French translation of the Developer's Reference which I worked on during the trip to Debconf8, then left aside since then....

Spent last weekend in Mountain View where Google had invited a number of mentors for the Summer of Code project that Google once again graciously sponsored. A rather impressive list of projects sent up to two people each, giving a probably unparalled sample of major Open Source projects. I had a blast. Chris, Leslie and the rest of the Google's Open Source Programs Office facilitated a really nice unconference that spawned a few really nice sessions, and they took very good care of us. And just about everybody met a number of folks in person that were previously known only via email or irc. As the saying goes: nothing like the bandwidth of a face-to-face meeting... Last but not least I should issue a health warning. Sharing a room with the fearless Debian DPL is not for the faint of heart: His snooring is truly world-class.