Today is Ada Lovelace day, a project to celebrate the women involved in technology and make it easier to give examples of female role models in the industry. I've had the opportunity to meet many women working on Linux over the past few years, and if there's one thing that's tended to overshadow the achievements and dedication to their work it's the sheer amount of effort they've had to go to in order to gain equal recognition. And that made me realise that in many ways, the woman who's had the greatest impact on my career is Hanna Wallach[1]. About ten years ago she spent a ridiculous amount of effort teaching me that it didn't matter how much I professed to be entirely free of sexism if I then proceeded to do things that implicitly excluded women from being involved in computing communities. I've gone to some amount of effort to repay that (contains profanity and a man with a raccoon tail covering his crotch), but I'm very aware of how different my life might have been if Hanna hadn't gone to the trouble of ensuring that I knew not to be a dick.

I think the Linux community has become more welcoming since then, and Hanna and people like her have been instrumental in helping that happen. However, we still have people being driven away by the behaviour of others or arguments that technical contributions are more important than social behaviour, and while that's true we need role models for social change just as much as we need role models for technical achievement. Thanks, Hanna[2]. The Linux world's a better place because of you.

[1] And while she's a role model of mine for social reasons, she's also got a PhD in machine learning, was British computer science student of the year in 2001 and came top of her MSc class at Edinburgh. So there.

[2] "Thanna"

debootstrap has only ever been designed to work with a single repository and the current code includes a comment:

# XXX: I can't think how to deal well with dependency resolution and

#      lots of Packages files. -- aj 2005/06/12

The main reason for that, AFAICT, is that debootstrap has a perfectly valid goal of trying to work without needing dpkg or apt themselves, doing everything in the chroot. Theoretically, anything that can run POSIX shell should be able to run debootstrap.

Emdebian has a need for a different script that can assume that both dpkg and apt are fully functional already and behave as normal Debian packages which opens the door to supporting as many repositories as apt itself can handle and, until now, that support was --foreign only. i.e. it was explicitly for cross-debootstrap (armel, etc.) for use with Emdebian Grip and actually used debootstrap for some of the work. That code has been simplified and there is no need for debootstrap now in multistrap. (multistrap itself is written in perl.)

I recognise that native support could be useful so I've been working on a fix for the native support - partially achieved by dropping the need to use debootstrap at all and porting the existing fix for #518719 from the emdebian-rootfs package (shell) into perl.

In effect, multistrap allows you to collapse the following stages into one:

1. $ sudo debootstrap $suite $dir $mirror


2. $ sudo chroot $dir


3. # nano /etc/apt/sources.list


4. # apt-get update


5. # apt-get install $package_list


6. # apt-get clean


7. $ sudo $my_config_script

That much is architecture-neutral (due to the support for working around #518719 which allows packages to be unpacked without running the configure scripts) - what I've added for native support is that sudo chroot $dir dpkg --configure -a can be automatically run before the config_script. In the typical Emdebian case, the configure stage would happen immediately after unpacking the multistrap filesystem onto the device (certainly before first boot).

That gives you a single method for creating a complete root filesystem with everything you need installed, everything configured and the config_script hook allows you to put in things like kernels and kernel modules etc. too - all ready for tar -czf and unpacking onto your device. Instead of a plain debootstrap, multistrap can give you a complete filesystem with all the packages you'd only install as soon as debootstrap had finished.

In addition, multistrap allows apt to decide which versions to install from the collection of Packages files so you can implement fallbacks and other permutations of mirrors and packages. Emdebian is using this functionality to mix private packages with official packages from Emdebian Grip 1.0 and ensure that not only are all dependencies met but that each dependency has the chance to be the latest version without needing any configuration or manual parsing. Just specify the repositories and which packages you need at the top level from each one. apt does the rest.

So, the question:

Does this deserve to be a separate source package ? For now, it's heading into Emdebian as part of the emdebian-rootfs package, version 1.6.1.

The advantages will be that multistrap itself doesn't need to be tied to the still rapidly developing emdebian-tools source package and doesn't need to carry the quite large changelog from emdebian-tools (which is just going to keep on growing).

 3014 2009-03-17 20:54 ./usr/share/doc/multistrap/README

 2265 2009-02-18 18:32 ./usr/share/doc/multistrap/TODO

 1536 2008-08-20 14:25 ./usr/share/doc/multistrap/copyright

21067 2009-03-17 21:58 ./usr/share/doc/multistrap/changelog.gz

 1233 2008-10-28 13:13 ./usr/share/doc/multistrap/NEWS.Debian.gz

Disadvantages, it's very small (a lot of the current content is actually POD documentation) and probably doesn't warrant a source package all alone.

 17774 2009-03-22 13:41 ./usr/sbin/multistrap

Details:

 Package: multistrap

 Source: emdebian-tools

 Version: 1.6.1

 Architecture: all

 Maintainer: Neil Williams 

 Installed-Size: 120

 Depends: perl, apt, debhelper (>= 5), 

  libconfig-auto-perl, libparse-debian-packages-perl

 Replaces: emdebian-rootfs (<= 1.6.0)

 Section: admin

 Priority: extra

 Homepage: http://www.emdebian.org/

 Description: multiple repository bootstrap based on apt

  debootstrap replacement with multiple repository support

  using apt to handle all dependency issues and conflicts.

  .

  Includes support for foreign architecture multistraps that need

  only minimal configuration on the final device. Supports cleaning

  up the generated bootstrap filesystem to remove downloaded packages

  and hooks to modify the files in the bootstrap filesystem after

  the packages have been unpacked but before being configured.

Is there a better home? devscripts? Ideas? (It would need to be a package I can join as Uploader: and I'd prefer if that package used subversion.)

This is my fourth CLT report and they surely are becoming repetitive! But every good event deserves a good report, so here goes: Contrary to previous years we already arrived on Friday this year. Since we were also not up for another stint with the gym (Aaah the sissifying effects of marriage!) we booked a room with a nice little guesthouse about 15 bus-minutes from the venue. The conference was great as ever:

• Speakers, booth personel and visitors were taken very good care of, there was nothing left to be desired
• Organisation was exicellent, no (noticable) technical difficulties, no scheduling problems, a helpful staff member literally every 5 meters
• A broad variety of very well-chosen talks from total beginner's to very sophisticated topics
• Several accompanying social events including the 2nd Night of Free Films
• A broad variety of exhibitors from small to large projects, both commercial and non commercial. CLT manages very well to be a community event that feels fun and almost like a family meeting but at the same time instills a sense of seriousness and respectability

I usually visit the CLT mostly for networking but I also managed to see a few talks. I moderated the session on High Performance and got to see three very nice talks on GPU programing, cluster computing and parallel programming. This year also featured the first talk on (the lack of) women in Free Software. Unusually enough this one was even given by a man. I was quite curious about that talk, which was (how fitting!) scheduled in direct succession to my own. I am kind of torn on that topic myself. On one hand women in Free Software is a topic that is very important to me and deserves any attention it can get. On the other hand instead of talking about how evenly capable women are in Free Software I'd rather just demonstrate it, for instance by giving a good Free Software related talk. And instead of talking about how there should be more women in Free Software I'd rather just be one and try to encourage others with my example. The usual quota of female speakers at linux events is somewhere between 2-4% and I just don't like the idea of having a conference with n men talking about Free Software and 1 woman talking about women. I've had these settings before and it just doesn't sit well with me. Because of this I was quite please to find the topic being tackled by a man this time.
The talk itself, in my impression, left a few open issues though. It was rather brief and focussed mainly on stating the usual numbers, asserting that the Women in FLOSS movement wasn't about affirmative action or discriminating men and explaining how women feel discriminated by sexist behaviour and advertising and how objection to such things should not be mistaken as prudery. It's general advice on how to improve the quota of women in FLOSS mostly boiled down to the linuxchix slogan "Be Polite. Be Helpful.". What I missed most was practical advice to projects wishing to attract more female contributors, such as mentoring programs or low-threshold entry points. Also I felt that the talk lacked a real motivation beyond "gender balance is a Good Thing". However, I was glad the topic finally found its way into the CLT as well and I had a couple of very interesting discussions afterwards. Concerning my own talk I was rather satisfied as well. Attendance was - as usually in Chemnitz - very good, according to the organisers I got around 200 people. Since I designed the talk as a collaborative project and its feedback was predominantly good I'll continue developing it with the feedback I got and submit it again to other events. I think the topic is very worthwile and there's still a lot in it. Some people asked for a more collaborative way of contributing their thoughts and ideas so I'll just create a wiki to collect the new ideas. I'll of course announce it here as soon as it's in place. The slides and audio recording (both German) are as usually available from the talks page at CLT or my own talks section.

and now for something completely different... sometimes in a shell script you need to do stuff that modifies the positional parameters ($@, $1, $2, etc) from the cmdline, usually via the builtin shift, maybe from using getopt(1) or similar. for example, it's quite common to see the following in a shell script that parses cmdline flags with getopt

# parse cmdline options with getopt
TEMP= getopt -o h123 -n $0 -- "$@" 
if [ $? != 0 ] ; then usage >&2 ; exit 1 ; fi
# overwrites $@ with getopt-generated/sanitized arguments
eval set -- "$TEMP"
# destructively iterates across $@
while true; do
  case "$1" in
        -h)
          # handle option h
          shift
        ;;
        ...
        --)
          shift
          break
        ;;
      esac
...
done

but sometimes you want to be able to restore the original positional parameters later, or otherwise quote them in a manner where they can be passed along to another script/function, log them, embed them in a string/file, etc. you can't always just stash them in another variable to use that later, because "$@" has a special meaning to the shell that can't otherwise be reproduced with another variable. one way you could try to store/save them would be to save the original copy of $@ and use eval set -- later on to change it back to the saved variable. using a slightly simplified version of the above:

#!/bin/sh
# save the cmdline
saved="$@"
echo original command line, $# arguments: "$@"
while test -n "$*"; do echo arg: "$1"; shift; done
echo mangled command line, $# arguments: "$@"
# restore the cmdline
eval set -- "$saved"
echo restored command line, $# arguments: "$@"
while test -n "$*"; do echo arg: "$1"; shift; done

and at first glance, it seems to work:

rangda[/home/sean] ./bad.sh one two three
original command line, 3 arguments: one two three
arg: one
arg: two
arg: three
mangled command line, 0 arguments:
restored command line, 3 arguments: one two three
arg: one
arg: two
arg: three

but you may run into problems when some parameters have spaces, or quotes:

rangda[/home/sean] ./bad.sh "1: one" "2: two"
original command line, 2 arguments: 1: one 2: two
arg: 1: one
arg: 2: two
mangled command line, 0 arguments:
restored command line, 4 arguments: 1: one 2: two
arg: 1:
arg: one
arg: 2:
arg: two
rangda[/home/sean] ./bad.sh "one's one"
original command line, 1 arguments: one's one
arg: one's one
mangled command line, 0 arguments:
./bad.sh: eval: line 12: unexpected EOF while looking for matching  ''

or worse, shell meta characters:

rangda[/home/sean] ./bad.sh 'i wonder what  pwd  would do'
original command line, 1 arguments: i wonder what  pwd  would do
arg: i wonder what  pwd  would do
mangled command line, 0 arguments:
restored command line, 6 arguments: i wonder what /home/sean would do
arg: i
arg: wonder
arg: what
arg: /home/sean
arg: would
arg: do

the solution is to build up the saved value a bit more manually, single-quoting each parameter (which should prevent any unintended side effects of space/quotes/metacharacters), and escaping existing single quotes inside of parameters.

# escape any single quotes in an argument
quote() 
  echo "$1"   sed -e "s,','\\\\'',g"
 
# save up a properly quoted/escaped version of "$@"
for arg in "$@"; do
  saved="$ saved:+$saved  '$(quote "$arg")'"
done
echo original command line, $# arguments: "$@"
while test -n "$*"; do echo arg: "$1"; shift; done
echo mangled command line, $# arguments: "$@"
# restore the cmdline
eval set -- "$saved"
echo restored command line, $# arguments: "$@"
while test -n "$*"; do echo arg: "$1"; shift; done

let's see how it behaves:

rangda[/home/sean] ./good.sh "1 one" "2: two"
original command line, 2 arguments: 1 one 2: two
arg: 1 one
arg: 2: two
mangled command line, 0 arguments:
restored command line, 2 arguments: 1 one 2: two
arg: 1 one
arg: 2: two
rangda[/home/sean] ./good.sh "one's one"
original command line, 1 arguments: one's one
arg: one's one
mangled command line, 0 arguments:
restored command line, 1 arguments: one's one
arg: one's one
rangda[/home/sean] ./good.sh 'i wonder what  pwd  would do'
original command line, 1 arguments: i wonder what  pwd  would do
arg: i wonder what  pwd  would do
mangled command line, 0 arguments:
restored command line, 1 arguments: i wonder what  pwd  would do
arg: i wonder what  pwd  would do

while not entirely pretty, it does to do the trick. i'd be interested to know if anyone has a better way of doing this (or if there are hidden problems in what i've posted above).

There has been a lot of hype about ext4 lately, following an Ubuntu bug which rose a lot of concerns about ext4 security. I know these are no news, and I know also that I am not a kernel guru, but let s resume some important concepts.
Ext4 implements something called allocate-on-flush. This means that kernel will decide how to write to disk in an efficient way by batching allocations together in larger runs. It increases performance and reduces fragmentation. But if a system crash happens between metadata and data is actually written to disk a 0-size length file can be found in the filesystem. With ext3 journal had a default timeout of 5 seconds and then was flushed to disk. With ext4 this time is unknown, so appplications that want to ensure everything is on disk have to call fsync() on file and directory after every operation. To avoid this, a series of patches have been queued for 2.6.30 so when a file is going to be replaced, it is written to disk with no delay.
Another concern raised is the need for atomicity and durability of files. For achieving this applications have first to write a temporary file and then rename() it over the old file. This will ensure that new or old file will be found in the filesystem after a crash, instead of a corrupted file that can be found when it is opened with O_TRUNC.
Anyway, my opinion is that when choosing a filesystem, one have to know what the computer is going to be used for. Using binary propietary drivers is an option, which can lead your Linux installation to behave most like Vista and filesystem cannot be blamed for losing files if it is making its job as designed. Perhaps, it s better to use ext3 in this situation, but at the cost of missing performance.
If you want to learn more about this issues I recommend you to read both articles by Theodore Tso, Delayed allocation and the zero-length file problem and Don t fear the fsync! and also AlessanderAlexander Larsson s one ext4 vs fsync, my take as well as comment in all of them.

I am not aware of any Linux Users Group (LUG) being active in informing it s members of how the policies of the various political parties compare with regard to free software and the other issues that are of interest to most members. I believe that this is a grave mistake. Shortly before an election there are many social groups that send lists of questions to all the parties. They ask about the policies the parties have in regard to the issues that they care about, and helpfully mention the number of members that will receive the response. This of course doesn t mean that every member of the group in question will cast their vote in the same way, merely that they will take note of the answers. The committee members of the parties in question will then decide how to answer the questions and whether policy should be tweaked to allow answers that the lobby groups will like. So this process not only helps members of a group make informed voting decisions related to issues that they care about, but it also helps political parties choose policies that are least offensive to the group in question. Here is a draft of a list of questions that I think should be asked of all political parties on behalf of Linux users:

1. It is important for all citizens to access all government data without being forced to buy new software or hardware, open standards allow everyone to access the data with free software. Do you support the use of open standards for data on government web sites and other forms of electronic communication between government agencies and citizens?


2. For long term archival of records it is important that file formats remain readable. The only effective way of doing this is to use open file formats that are implemented in free software. Do you support mandating that all data submitted to government agencies (by citizens or corporations) be in open file formats wherever possible?


3. In these difficult economic times there is a great interest in keeping jobs in the country instead of sending money overseas. To what extent do you support the use of free software that is installed and managed by locals (keeping the money in the economy) instead of importing software at great taxpayer expense?


4. Commercial software has a limited support period, after that time has elapsed there is no further support and systems become increasingly unreliable. Do you support mandating that all systems relating to the emergency services run on free software to allow quality long-term support by local citizens?


5. There has been a lot of concern recently about the spread of child-porn. The best available evidence shows that insecure home PCs that run Trojan Horse programs are a key part of distributing it and other illegal material. Do you support the introduction of government programs to train parents in installing one of the more secure free operating systems on their home PC to protect their children?

This is just a rough draft. Obviously there needs to be local differences (EG don t use point 3 in the US because MS brings money into the US economy). Does anyone have any suggestions for other questions?

I went to see Fruit Fly at the Castro Theatre tonight. It was a really fun movie and very local. Oh yeah, it's a musical, I don't think I've had that much fun watching a movie since I saw Jesus Christ Vampire Hunter the first time. The opening number is a song about public transit and not needing a car in San Francisco. The previous film by H.P. Mendoza is Colma: The Musical, it's got a lot of the same cast and also look like a lot of fun. Unfortunatly, movies are behind music and I can't find a stream or download on hulu, amazon or youtube of Colma the musical.

Anyway, if anyone wants to go see Fruit Fly it is playing in Berkeley on friday. I don't think I'll make it out to that since the Castro Theatre is having a triple bill "nerds with balls" Midnites for maniacs on friday.

In other good news, I couldn't get either a Petzl size 1 or size 2 harness since my legs are size two, but my waist is size 1.

No, you re not reading it wrong but what about 1.1.2? , you ask. Well, we made a small mistake with the soversion there (if mistakes regarding soversion are ever small, that is =/), so we had to quickly release a fixed up version before packages start showing up for distributions with an incorrect library soname. Apologies. With this release we have a shiny new website at http://webkitgtk.org/, hosted by Igalia and designed by Christian Dywan, who s one of the contributors to WebKitGTK+ s code. This new release sports many interesting features and fixes. You may remember me talking about the download support initially contributed by Collabora, which I was fixing up and pushing for inclusion. Well, it is now available, and already implemented in Epiphany 2.27.0. Christian Dywan tells me he won t let Midori stay behind Epiphany on this one, so expect support for downloading correctly integrated in Midori as well. Some more pet peeves got fixed with this release, such as scrolling with keys such as space bar, home, end, not behaving the way people expect. We also found more improvements which are in order for libsoup, and we ll be pushing them, even though we have added small workarounds to WebKitGTK+ itself for the time being. We have added proper logging support for our port, providing a WEBKIT_BUILD environment variable. Documentation on how to use this will follow soonish, but for now you can look at the LoggingGTK.cpp file to know what values you can add to that variable. Also notice that only debug builds use logging to its full extent! WebKitGTK+ 1.1.1 has now hit Debian experimental, by the way, and I plan to have 1.1.3 uploaded early this week. An epiphany-webkit package based on 2.27 is in the works, and should be in soon, too. There seems to be lots of interest in having a PPA for Ubuntu, as well, so we may see that happening soonish. We plan to get more testing this way, and be able to uncover and fix more bugs, so that we can get a rock solid release ready in time for GNOME 2.28.

Today, I was sitting on the couch. Jacob crawled up, and pushed me forward, saying I go there. He crawled behind me. Then, ready for one of his favorite games, yelled out: QUISH, daddy! So I leaned back gently a bit, and said squish! Jacob yelled, laughing, QUISH! Quish again, daddy! So I d lean back a bit, gently, again, this time reaching behind me to tickle him a bit as I leaned. Squash! Louder laughing. QUASH!!! Right in my ear ouch, but I didn t mind. Do it again, daddy! Right now, being a dad seems complicated enough. You ve got to have the right touch to squish a 2-year-old without really squishing him. Or have the presence of mind to realize that when Jacob was happily playing outside, then suddenly comes running over, very upset, saying Go inside! it means he probably needs to use the potty urgently, or just had an accident. (Or both, as it happened today.) Or to recognize that little walk that means he really does need to use the potty even though he d rather not. And, of course, there s figuring out what he s saying, when his words can still be a bit garbled. But these all seem simple to me, compared to what will come. How will we help Jacob to grow as a person of good character? How will we meet his need to be challenged intellectually? Will we be able to maintain a good relationship, and yet still have the judgment to have the right set of rules, when he gets to high school? Will I have a good relationship with him as an adult? And how am I going to react when the day comes when he tells me I ought to move into the nursing home? Jacob, of course, doesn t care about any of that right now. Each night, when I put him down for the night, he wants me to cover him up with blankets. Once I ve done that, he peeks out and says, Have a good night, dad! I always reply with You have a good night too, Jacob! If we can get along that well for the next 60 years, I guess we ll do all right.

Edited to add: Sorry, it turns out that ext4 does now have a heuristic to deal with this case. However, it seems that btrfs doesn't. My point remains, though - designing a filesystem in such a way that a useful behaviour is impossible is unhelpful for the vast majority of application writers, even if it does improve performance in some other use cases.

Further edit: It's mentioned in the comments that btrfs will force a flush of data when a rename is performed as of 2.6.30. That prevents the data loss, but it means that we're still stuck with disk access when we'd be happy with that being batched up for the future. It feels like we're optimising for the wrong case here.

Original text:

There's been a certain amount of discussion about behavioural differences between ext3 and ext4[1], most notably due to ext4's increased window of opportunity for files to end up empty due to both a longer commit window and delayed allocation of blocks in order to obtain a more pleasing on-disk layout. The applications that failed hardest were doing open("foo", O_TRUNC), write(), close() and then being surprised when they got zero length files back after a crash. That's fine. That was always stupid. Asking the filesystem to truncate a file and then writing to it is an invitation to failure - there's clearly no way for it to intuit the correct answer here. In the end this has been avoided by avoiding delayed allocation when writing to a file that's just been truncated, so everything's fine.

However, there's another case that also breaks. A common way of saving files is to open("foo.tmp"), write(), close() and then rename("foo.tmp", "foo"). The mindset here is that a crash will either result in foo.tmp being zero length, foo still being the original file or foo being your new data. The important aspect of this is that the desired behaviour of this code is that foo will contain either the original data or the new data. You may suffer data loss, but you won't suffer complete data loss - the application state will be consistent.

When used with its (default) data=ordered journal option, ext3 provided these semantics. ext4 doesn't. Instead, if you want to ensure that your data doesn't get trampled, it's necessary to fsync() before closing in order to make sure it hits disk. Otherwise the rename can occur before the data is written, and you're back to a zero length file. ext4 doesn't make guarantees about whether data will be flushed before metadata is written.

Now, POSIX says this is fine, so any application that expected this behaviour is already broken by definition. But this is rules lawyering. POSIX says that many things that are not useful are fine, but doesn't exist for the pleasure of sadistic OS implementors. POSIX exists to allow application writers to write useful applications. If you interpret POSIX in such a way that gains you some benefit but shafts a large number of application writers then people are going to be reluctant to use your code. You're no longer a general purpose filesystem - you're a filesystem that's only suitable for people who write code with the expectation that their OS developers are actively trying to fuck them over. I'm sure Oracle deals with this case fine, but I also suspect that most people who work on writing Oracle on a daily basis have very, very unfulfilling lives.

But anyway. We can go and fix every single piece of software that saves files to make sure that it fsync()s, and we can avoid this problem. We can probably even do it fairly quickly, thanks to us having the source code to all of it. A lot of this code lives in libraries and can be fixed up without needing to touch every application. It's not the end of the world.

So why do I still think it's a bad idea?

It's simple. open(),write(),close(),rename() and open(),write(),fsync(),close(),rename(), are not semantically equivalent. One is "give me either the original data or the new data"[2]. The other is "always give me the new data". This is an important distinction. fsync() means that we've sent the data to the disk[3]. And, in general, that means that we've had to spin the disk up.

So, on the one hand, we're trying to use things like relatime to batch data to reduce the amount of time a disk has to be spun up. And on the other hand, we're moving to filesystems that require us to generate more io in order to guarantee that our data hits disk, which is a guarantee we often don't want anyway! Users will be fine with losing their most recent changes to preferences if a machine crashes. They will not be fine with losing the entirity of their preferences. Arguing that applications need to use fsync() and are otherwise broken is ignoring the important difference between these use cases. It's no longer going to be possible to spin down a disk when any software is running at all, since otherwise it's probably going to write something and then have to fsync it out of sheer paranoia that something bad will happen. And then probably fsync the directory as well, because what if someone writes an even more pathological filesystem. And the disks sit there spinning gently and chitter away as they write tiny files[4] and never spin down and the polar bears all drown in the bitter tears of application developers who are forced to drink so much to forget that they all die of acute liver failure by the age of 35 and where are we then oh yes we're screwed.

So. I said we could fix up applications fairly easily. But to do that, we need an interface that lets us do the right thing. The behaviour application writers want is one which ext4 doesn't appear to provide. Can that be fixed, please?

[1] xfs behaves like ext4 in this respect, so the obvious argument is that all our applications have been broken for years and so why are you complaining now. To which the obvious response is "Approximately anyone who ever used xfs expected their data to vanish if their machine crashed so nobody used it by default and seriously who gives a shit". xfs is a wonderful filesystem for all sorts of things, but it's lousy for desktop use for precisely this reason.

[2] Yes, ok, we've just established that it actually isn't that in the same way that GMT isn't UTC and battery refers to a collection of individual cells and so you don't usually put multiple batteries in your bike lights, but the point is that this is, for all practical intents and purposes, an unimportant distinction and not one people should have to care about in their daily lives.

[3] The disk is free to sit there bored for arbitrary periods of time before it does anything, but that's fine, because the OS is behaving correctly. Sigh.

[4] Dear filesystem writers - application developers like writing lots of tiny files, because it makes a large number of things significantly easier. This is fine because sheer filesystem performance is not high on the list of priorities of a typical application developer. The answer is not "Oh, you should all use sqlite". If the only effective way to use your filesystem is to use a database instead, then that indicates that you have not written a filesystem that is useful to typical application developers who enjoy storing things in files rather than binary blobs that end up with an entirely different set of pathological behaviours. If I wanted all my data to be in oracle then I wouldn't need a fucking filesystem in the first place, would I?

With any luck, someone has seen this problem before and can fix it easily. I don t seem to be able to find the fix by searching, but the search terms feel like they re either too general or too specific. I was trying to install some perl software on a debian lenny system that was upgraded from etch (and previously from sarge). After installing the dependencies, I started getting errors like this:-

$ perl -MYAML::Syck -e print
Can't locate YAML/Syck.pm in @INC (@INC contains:
/usr/local/lib/perl5/5.10.0/i686-linux /usr/local/lib/perl5/5.10.0
/usr/local/lib/perl5/site_perl/5.10.0/i686-linux
/usr/local/lib/perl5/site_perl/5.10.0 .).
BEGIN failed--compilation aborted.

Update: Well done to Raphael who spotted the local that I d overlooked. D oh! Now to see if I can discover where that came from. Update 2: Also thanks to Florian who emailed in at about the same time (I don t read my email as much as my website dashboard, usually).

Then I discovered that the debian mirror being used (which is in the same city, and in a nearby network) is marked as bad on the health check. I switched mirrors to a known-good one that I ve used for other systems recently, but no joy. I think a key symptom is this bit near the end of the output of perl -V:

Compiled at Aug 4 2008 09:48:59
@INC:
/usr/local/lib/perl5/5.10.0/i686-linux
/usr/local/lib/perl5/5.10.0
/usr/local/lib/perl5/site_perl/5.10.0/i686-linux
/usr/local/lib/perl5/site_perl/5.10.0

Our good systems show a December compiled at date and a much longer @INC list. I ve tried reinstalling pretty much every package I think might be connected, including libc6, libperl5.10, perl, perl-base and perl-modules (those last four are all version 5.10.0-19). I really don t want to reinstall the whole system because it s behind an irritating firewall that will make it a 10-hour round-trip site visit. I m currently hunting and removing obsolete packages to try to make sure they re not causing problems, because I m running out of straws to clutch at. How would you recover a debian system from this?

I've just uploaded Policy 3.8.1.0 with the first batch of post-lenny changes. There isn't anything that earth-shattering in this set, but I'm hoping that we'll be able to get a lot more done during the squeeze release cycle. Policy has well over a hundred open bugs, quite a few of which we should be able to resolve. I also uploaded Lintian 2.2.8, which is synchronized against Policy 3.8.1 and also reverts some changes around install-menu due to a reverted change in debhelper 7.2.5, and fixes diagnosis of diversions. The last few versions of Lintian have all been a little bit buggy due mostly to the current pace of change. Hopefully this version will have fewer unexpected problems. Here is the summary of normative Policy changes, taken from upgrading-checklist.txt.gz in the debian-policy package.

• Care should be taken when adding functionality to essential and such additions create an obligation to support that functionality in essential forever unless significant work is done.
• Changelog files must be encoded in UTF-8.
• Tighten some format requirements for changelog files from a should to a must.
• Remove alternative changelog formats. Debian only supports one changelog format for the Debian Archive.
• New nocheck option for DEB_BUILD_OPTIONS indicating any build-time test suite provided by the package should not be run.
• All control files must be encoded in UTF-8.
• debian/control allows comment lines starting with # with no preceding whitespace.
• Init scripts ending in .sh are not handled specially. They are not sourced and are not guaranteed to be run by /bin/sh regardless of the #! line. This brings Policy in line with the long-standing behavior of the init system in Debian.
• The start action of an init script must exit successfully and not start the daemon again if it's already running.
• /var/run and /var/lock may be mounted as temporary filesystems, and init scripts must therefore create any necessary subdirectories dynamically.
• /bin/sh scripts may assume that local can take multiple variable arguments and supports assignment.
• User mailboxes may be mode 600 and owned by the user rather than mode 660, owned by user, and group mail.

Many people answered to my previous rant stating that it is upstream's prerogative to refuse integration of external patches. I agree and I never said they must integrate patches if they open their source.

What I find unacceptable, and the only reason for the rant, is the fact that upstream deleted my posts from the forum when it was clear that none of the fake reasons that were given was the real reason, since I answered to all of them with sane, logic and reasonable arguments.


I find acceptable to refuse external patches, although undesirable.

Still, I find it unacceptable to try to erase all tracks of a conversation from which it is clear that you were wrong, and try to keep face by erasing traces.

I would have found it hard to deal with, but still acceptable, if upstream would have said simply "please don't try to make patches for glest with the purpose of integrating them later on, we don't want any patches from outsiders". That would have been a decent thing to do.

This release had more annoying globalish changes than any release that I remember recently. There were lots of little changes across all files, lots of internal reorganization, and lots of tweaks and little changes to match behavior changes elsewhere in the infrastructure. It's all movement towards a more maintainable code base, but it's not as much fun as fixing lots of bugs. This release should fix the occasional problems that people saw with checking source packages (sorry about that, my fault) and somewhat faster performance due to a bunch of work by Raphael Geissert. It also incorporates changes from debhelper 7.2.3, including removing the requirement that one call install-docs and update-menus in maintainer scripts. That work is now handled by triggers (except for menu-methods files). This was supposed to be the Policy 3.8.1 release, but I ran out of time today. I'm still hoping to get to that very soon (before 2.2.7 propagates into testing), but there were enough important fixes in this that I didn't want to delay it further. I'm running into a bit of a problem with Lintian at the moment. It's a lot of fun, and we're making a ton of progress, but it's taking me roughly a day a week to really keep up with the incoming bugs and patch submissions and do a proper job of merging them (which is more than just applying them). This really isn't sustainable. I have a bunch of other things that I want to do, both inside Debian and outside, and I'm going to have to reduce the amount of time that I'm spending on Lintian soon. I hate to do that, since it's great to stay this up-to-date and see this high pace of change, but things may start backing up a bit in the near future. As usual, thank you very much to Adam D. Barratt and Raphael Geissert for all of their work, without which this release wouldn't have been possible.

Today is the International Woman's Day (IWD) so I will take advantage of my blog to congratulate all women either into the Debian Project and in General.

The women's history, from my point of view, has been involved of many difficult situations. Women have had to face the inequality in many ways and levels. On the other hand, it has changed a lot but there still happening some cases nowadays.

I got surprised, when I read the article in Wikipedia, by some controversial facts. Sometimes it is a shame our "human" behavior...

It s been quite the week. Stomach Flu Last Friday, my stomach was just starting to feel a little odd. I didn t think much off it a little food that didn t go over well or stress, I thought. Saturday I got out of bed and almost immediately felt like throwing up. Ugh. I probably caught some sort of stomach flu. I was nauseous all day and had some terrible diarrhea to boot. I spent parts of Saturday, Saturday night, Sunday, and Sunday night supervising some emergency downloads as the BOFH would say. By Sunday afternoon, I thought I was doing good enough to attend a practice of the Kansas Mennonite Men s Choir. I made it through but it wasn t quite as up to it as I thought. Monday morning I woke up and thought the worst was behind me, so I went to work. By evening, the worst clearly was not behind me. I was extremely cold, and then got very hot a few hours later. Tuesday I left work a little early because of not feeling well. Servers Wednesday a colleague called me at home before I left to say that the ERP database had a major hiccup. That s never good. The database is this creaky old dinosaur thing that has a habit of inventing novel ways to fail (favorite pastime: exceeding some arbitrary limit to the size of files that no OS has cared about for 5 years, then hanging without telling anybody why). My coworkers had been working on it since 5. I went into the office and did what I could to help out, though they had mostly taken care of it. Then we went to reboot the server. It didn t come back. I/O error on sda just after init started, and it hung. Puzzled, as it just used that disk to boot from. Try rebooting again. This time, I/O error as the fibre channel controller driver loads. Again, puzzled as it just used that controller to load grub. Power cycle this time. And now the server doesn t see the fibre channel link at all. Eep. Check our fiber optic cables, and power cycle again. And THIS time, the server doesn t power back up. Fans whir for about a second, then an ominous red light I never knew was there shows up. Eeep! So I call HP. They want me to remove one CPU. Yes, remove one CPU. I tried, and long story short, they dispatch a local guy with a replacement motherboard. Can you send along a FC controller, in case it s dead too? Nope, not until we diagnose a problem with it. Local guy comes out. He s a sharp guy and I really like him. But the motherboard wasn t in stock at the local HP warehouse, so he had to have it driven in from Oklahoma City. He gets here with it by about 4:30. At this point the single most important server to the company s business has been down almost 12 hours. He replaces the motherboard. The server now powers up yay! And it POSTs, and it . doesn t see the disks. !#$!#$ He orders the FC controller, which is so very much not in stock that they can t get it to us until 8:30AM the next morning (keep in mind this thing is on a 4-hour 24/7 contract). Next morning rolls around. Outage now more than 24 hours. He pops the FC controller in, we tweak the SAN settings appropriately, we power up the machine, and . still doesn t see any disks, and the SAN switch still doesn t see any link. EEP! Even the BIOS firmware tool built into the controller doesn t see a link, so we KNOW it s not a software issue. We try plugging and unplugging cables, trying different ports, everything. Nothing makes a difference. At this point, while he ponders what else he can replace while we start migrating the server to a different blade. We get ERP back up on its temporary home an hour later, and he basically orders us every part he can think of while we ve bought him some room. Several additional trips later, he s replaced just about everything at least once, some things 2 or 3 times, and still no FC link. Meanwhile, I ve asked my colleague to submit a new ticket to HP s SAN team so we can try checking of the switch has an issue. They take their sweet time answering until he informs them this morning that it s been *48 HOURS* since we first reported the outage. All of a sudden half a dozen people at HP take a keen interest in our case. As if they could smell this blog post coming So they advise us to upgrade the firmware in the SAN switch, but they also say we really should send this to the blade group; the problem can t be with the SAN and of course the blade people are saying the problem s GOT to be with the SAN . We try to plan the firmware upgrade. In theory, we can lose a switch and nobody ever notices due to multipathing redundancy. In practice, we haven t tested that in 2 years. None of this equipment had even been rebooted in 390 days. While investigating this, we discovered that one of the blade servers could only see one path to its disks, not two. Strange. Fortunately, THAT blade wasn t mission-critical on a Friday, so I power cycled it. And it powered back up. And it promptly lost connection to its disks entirely, causing the SAN switches to display the same mysterious error they did with the first blade the one that nobody at HP had heard of, could find in their documentation, or even on Google. Yes, that s right. Apparently power cycling a server means it loses access to its disks. Faced with the prospect of our network coming to a halt if anything else rebooted (or worse, if the problem started happening without a reboot), we decided we d power cycle one switch now and see what would happen. If it worked out, our problems would be fixed. If not, at least things would go down in our and HP s presence. And that worked? What? Yes. Power cycling the switch fixed every problem over the course of about 2 minutes, without us having to do anything. Meanwhile, HP calls back to say, Uhm, that firmware upgrade we told you to do? DON T DO IT! We power cycle the other switch, and have a normal SAN life again. I let out a WOOHOO! My colleague, however, had the opposite reaction. Now we ll never be able to reproduce this problem to get it fixed! Fair point, I suppose. Then began the fairly quick job of migrating ERP back to its rightful home it s all on Xen already, designed to be nimble for just these circumstances. Full speed restored 4:55PM today. So, to cap it all off, within the space of four hours, we had fail:

• One ERP database
• ERP server s motherboard
• Two fiber optic switches but only regarding their ability to talk to machines recently rebooted
• And possibly one FC controller

Murphy, I hate you. The one fun moment out of this was this conversation: Me to HP guy: So yeah, that machine you ve got open wasn t rebooted in 392 days until today. HP guy: WOW! That s INCRED oh wait, are you running Linux on it? Me: Yep. HP: Figures. No WAY you d get that kind of uptime from Windows. And here he was going to be all impressed.

all programs that interact with e-mail are broken in one way or another. Please be careful. - Lars Wirzenius

I seem to have a cunning knack of finding problems with configuring server software, particularly involving e-mail, where a) I can t find answers in Google, b) most people I go and ask for help say they d usually ask me such things, and c) if I go onto IRC or mailing lists I end up helping other people and not getting any help with my problem. It s quite likely this is just because I m something of a perfectionist, so the ridiculous crappy hacks people come up with and seem content to entrust their mail to are unacceptable to me for one reason or another. Anyway, in my ongoing quest for the perfect mail system, I ve painted myself into a corner again. (I m currently running with postfix, postgrey, clamav-milter, dspam, dovecot using LDA, managesieve and the cmusieve & antispam plugins. If I can get the current incarnation working, I ve had enough requests to write up a full HOWTO, and seen enough around with pretty questionable content, that I ll probably do it before too long.) I ve got postfix s local transport configured to hand mail to dspam over LMTP, using mailbox_transport = lmtp:unix:/dspam/dspam.sock. dspam is configured to listen here, add X-DSPAM-Result and signature headers, and then deliver the mail with dovecot s deliver LDA (which I ve set to 4750 root:dspam). From dspam.conf: ServerParameters "--deliver=innocent,spam -d %u"
ServerDomainSocketPath "/var/spool/postfix/dspam/dspam.sock"
...
Preference "spamAction=tag"
Preference "signatureLocation=headers"
...
TrustedDeliveryAgent "/usr/lib/dovecot/deliver" My dovecot configuration is pretty standard, using PAM for both passdb and userdb, and provides the auth-master socket that deliver needs. The problem I have is that postfix s local transport is qualifying the local username with the FQDN of the machine before delivering it to dspam with LMTP (the local mail transfer protocol), even for locally-originated mail which was only addressed with a bare username! dspam doesn t mangle it or care if the user is local or not, and then cheerfully invokes deliver -d robot101@omega.example.co.uk, which returns EX_NOUSER (addressee unknown) because my username is just robot101. From mail robot101: Mar 6 02:25:40 omega postfix/pickup[13607]: DAA4942F41F: uid=1000 from=<robot101>
Mar 6 02:25:40 omega postfix/cleanup[13637]: DAA4942F41F: message-id=<20090306022540.DAA4942F41F@omega.example.co.uk>
Mar 6 02:25:40 omega postfix/qmgr[12552]: DAA4942F41F: from=<robot101@omega.example.co.uk>, size=339, nrcpt=1 (queue active)
Mar 6 02:25:40 omega dovecot: auth(default): passwd(robot101@omega.example.co.uk): unknown user
Mar 6 02:25:40 omega dspam[13527]: Delivery agent returned exit code 67: /usr/lib/dovecot/deliver -d robot101@omega.example.co.uk
Mar 6 02:25:40 omega postfix/lmtp[13640]: DAA4942F41F: to=<robot101@omega.example.co.uk>, orig_to=<robot101>, relay=omega.example.co.uk[/dspam/dspam.sock], delay=0.08, delays=0.05/0.01/0.01/0.03, dsn=4.3.0, status=deferred (host omega.example.co.uk[/dspam/dspam.sock] said: 421 4.3.0 <robot101@omega.example.co.uk> Delivered (in reply to end of DATA command)) So, no e-mail for me. Dearest lazyweb, which of the three components is behaving wrongly, and how can I fix it? (And no, I m not just going to switch to GMail. I store my data on hard drives, which are sometimes in servers, not in the cloud . Until about a month ago, most people I knew spoke about clouds which were made of particles of water in the sky, rather than as a data storage media. What if it rains? ;)

all programs that interact with e-mail are broken in one way or another. Please be careful. - Lars Wirzenius

I seem to have a cunning knack of finding problems with configuring server software, particularly involving e-mail, where a) I can t find answers in Google, b) most people I go and ask for help say they d usually ask me such things, and c) if I go onto IRC or mailing lists I end up helping other people and not getting any help with my problem. It s quite likely this is just because I m something of a perfectionist, so the ridiculous crappy hacks people come up with and seem content to entrust their mail to are unacceptable to me for one reason or another. Anyway, in my ongoing quest for the perfect mail system, I ve painted myself into a corner again. (I m currently running with postfix, postgrey, clamav-milter, dspam, dovecot using LDA, managesieve and the cmusieve & antispam plugins. If I can get the current incarnation working, I ve had enough requests to write up a full HOWTO, and seen enough around with pretty questionable content, that I ll probably do it before too long.) I ve got postfix s local transport configured to hand mail to dspam over LMTP, using mailbox_transport = lmtp:unix:/dspam/dspam.sock. dspam is configured to listen here, add X-DSPAM-Result and signature headers, and then deliver the mail with dovecot s deliver LDA (which I ve set to 4750 root:dspam). From dspam.conf: ServerParameters "--deliver=innocent,spam -d %u"
ServerDomainSocketPath "/var/spool/postfix/dspam/dspam.sock"
...
Preference "spamAction=tag"
Preference "signatureLocation=headers"
...
TrustedDeliveryAgent "/usr/lib/dovecot/deliver" My dovecot configuration is pretty standard, using PAM for both passdb and userdb, and provides the auth-master socket that deliver needs. The problem I have is that postfix s local transport is qualifying the local username with the FQDN of the machine before delivering it to dspam with LMTP (the local mail transfer protocol), even for locally-originated mail which was only addressed with a bare username! dspam doesn t mangle it or care if the user is local or not, and then cheerfully invokes deliver -d robot101@omega.example.co.uk, which returns EX_NOUSER (addressee unknown) because my username is just robot101. From mail robot101: Mar 6 02:25:40 omega postfix/pickup[13607]: DAA4942F41F: uid=1000 from=<robot101>
Mar 6 02:25:40 omega postfix/cleanup[13637]: DAA4942F41F: message-id=<20090306022540.DAA4942F41F@omega.example.co.uk>
Mar 6 02:25:40 omega postfix/qmgr[12552]: DAA4942F41F: from=<robot101@omega.example.co.uk>, size=339, nrcpt=1 (queue active)
Mar 6 02:25:40 omega dovecot: auth(default): passwd(robot101@omega.example.co.uk): unknown user
Mar 6 02:25:40 omega dspam[13527]: Delivery agent returned exit code 67: /usr/lib/dovecot/deliver -d robot101@omega.example.co.uk
Mar 6 02:25:40 omega postfix/lmtp[13640]: DAA4942F41F: to=<robot101@omega.example.co.uk>, orig_to=<robot101>, relay=omega.example.co.uk[/dspam/dspam.sock], delay=0.08, delays=0.05/0.01/0.01/0.03, dsn=4.3.0, status=deferred (host omega.example.co.uk[/dspam/dspam.sock] said: 421 4.3.0 <robot101@omega.example.co.uk> Delivered (in reply to end of DATA command)) So, no e-mail for me. Dearest lazyweb, which of the three components is behaving wrongly, and how can I fix it? (And no, I m not just going to switch to GMail. I store my data on hard drives, which are sometimes in servers, not in the cloud . Until about a month ago, most people I knew spoke about clouds which were made of particles of water in the sky, rather than as a data storage media. What if it rains? ;) Update: The problem is fixed! Even though arguably the problem is dspam s for not knowing which users are local or not, it s fixable in dovecot 1.1.x using the auth_username_format = %n option. Thanks so much to Angel Marin for helping me out. Update 2: There s also a patch for dspam floating around which adds a StripRcptDomain option, which makes the LMTP server truncate the e-mail address at the @, so essentially assumes everyone to be a local user. The problem with both of these fixes is that they re both blunt instruments which will break virtual users on the same host. I think the real fix would be something more like a LocalDomains option in dspam, to choose which domains should be considered local and truncated from the e-mail addresses for delivery purposes.

At some point in the past, when I was still thinking that Glest can be packaged in a sane way, I had a conversation with upstream developer martinho on the Glest forum (they call it board) about them integrating a patch in order to support big endian machines.

After some messages on their board, it was clear that they don't want to integrate any patches and started to throw out random fake arguments against the integration of my patch and other patches.

When he figured out that I replied and debunked all of his fake arguments, he went on and actually removed my posts from the forum. It was so weird since I have never seen that kind of behaviour before and some of his posts and other people's posts were replies to my posts and things didn't made any sense after the deletion.

At that moment I have managed to save one small bit of the conversation via Google's cache, but now I have finally decided to make it public.


Note: Back then, I kept this post private in the hope that I was maybe too involved in the conversation at that moment to make a rational decision about it, and hoped things would get better in time. They didn't. Meanwhile they migrated to another forum software and they said they lost most/all(?) of the previous content from the old forum.



Here is the conversation bit I managed to save.
Original text is in italic font. Regular font is for my current comments for clearing up the context of the discussion. The text retains the BB syntax and [quote]-d text is martinho's own words.



[quote="martinho"]thebohemian, eddyd,

I think you guys are missing the point, this is not the place for any [/quote]

Congrats, you are avoiding to answer the points I have raised.

[quote]
- The Glest project is made by a bunch of friends for fun, we ask for nothing, we answer to no one. So I beg you not co complain, because it is out of order.
[/quote]

In that case I don't see the point of the "development" forum. care to explain?
And, from my PoV, these are your users you're arrogantly dismissing.


The following was a reply to my request to add big endian support through a patch I provided at that time. The patch wasn't complete to actually make glest work on BE machines, but it didn't broke little endian support at all. He was trying to argue that there were no valid big endian machines on which the game would run. Integrating the patch would have made my life a lot easier in continuing the work on the patch.

[quote]
- Our target plarforms are PCs running Windows (I'm responsible for this) or Linux (matze is the boss here). We don't care if Glest doesn't run in a big endian dishwasher with Solaris.
[/quote]

Linux on PowerPC is a viable platform. So is Mac OS X on PowerPC.



I made the mistake of saying that some parts of the code where not that clean and could be improved. More on that below.

[quote]
- One of the keys of Glest success is how extremely clean our code is. Bringing random people in would break this, and we are not going to do it. [/quote]

So you are the only keepers of true knowledge of clean code? Hahahaha!

There were at least 3 patches that provided this through word/double-word swapping. I don't see how embedding the patch can affect the unity of the code.

BTW, you are aware you can "carve" people into coding in your style before accepting in, aren't you?



I suggested using int8, uint8, int16, uint16 in order to make sure that the parameters are of the expected fixed size since plain char, int, long, long long have different sizes depending on the architecture. The usage of int and int16 was inconsistent and I also suggested using always the numbered variants.

I said the approach that was then taken in code was not portable and that was the reason for the proposed change.

[quote]
- Glest has been coded with portability in mind, some of your comments about portability are really naive (compilers don't care if you declare function parameters with fixed byte length, they just use whatever is the fastest int in the platform). Fixed length data types are used when needed,
[/quote]

What? When did I say anything about that? I said the code uses inconsistently "int" and "int16" and should use the later always because the code assumes the sizes on i386, which are not always applicable - see long on amd64.


If you mean that I can't enforce the width of the parameter, then you should know better, those types are defined in a platform dependent way to be platform independent. Yes, there are cases where [b]on the stack[/b] the size of the passed data is bigger (they might be padded), but that is transparent to the functions and, in spite of that, the functions will see the proper values for the parameters.

Also, the packing behaviour is compiler and architecture dependent, and some compilers produce the weirdest packing (if you are allowed access to such docs, look for Green Hills compilers' for V850 and you'll be stunned).


Please, don't make a fool of yourself by trying to discredit me, just not to loose face.

Later he just removed my posts not to loose face.




As a sane programmer, I made my patch to do the swapping only on big endian platforms, and the code was enabled in through the precompiler, so there was no performance hit on the little endian arches.

Of course, any more-than-newbie programmer would agree that having two data sets for a single game is stupid and is better to do the swapping in the application itself to avoid space wastage and maintaing an alternate big endian data set.

[quote]
no more, no less. About endianness, we don't care. If we wanted to fix the problem anyway, the right way is to do the byte swap in the data, this is how it is done in commercial games, since it is more efficient.
[/quote]

You just proved how much bullshit you can deliver in one phrase just for the sake of winning an argument. All portable games and all applications that have data files prefer to do swapping on the spot just because swapping the data on the disk (or providing BE and LE data) is plain stupid and inefficient from a disk and bandwidth PoV.

For Glest, swapping the data after reading from disk is not even affecting performance in a manner that could be significant or noticed, not even for a processor running at 500MHz, since that would be done only on load, once, and just for the bits of data which are relevant; these are mostly headers, since the actual data, in most cases, is just a long byte vector. This translates in not more than 3 or 4 instructions in assembler (an init, an exchange, an increment and/or a loop instruction), and that would happen before starting a game, which is dead time anyway and wouldn't be noticed unless the data would be huge (order of hundreds of MB or GB) and the whole data set would need swapping.

Even if the data itself (not headers) represents data which needs to be swapped, that is still irrelevant in terms of speed.


BTW, the reason that in commercial games it might be done like that is because there the games are distributed in a package per platform anyway and having two data sets is not an issue since they are using the same amount of CDs/DVDs, but doing the same for a game like Glest would mean doubling the storage used and it would increase the complexity for a release, not to count the confusion of the users.

I still doubt commercial games do this, but I might be wrong.

So you do computer stuff for work, and what do you do in the weekend? Right, more computer stuff. Not that I dislike it, obviously. But, well. The computer stuff that I did this weekend was situated mainly around dvswitch. During their talk at the Debian devroom at FOSDEM, Ben and Holger mentioned that one of the things they'd still like to see implemented in DVswitch is the ability to do tally lights. Such lights are useful, not only for the benefit of those in front of the camera, but also for the benefit of those behind the camera allowing them to know when they're free to roam about in order to find something interesting to show, or when they should keep the camera somewhat stead because they're live. Since I was planning on using dvswitch to record the concert (dutch only) of the choir group that I'm part of, and since a concert has different requirements than does a talk, I set about implementing tally light functionality. For now, the actual tally light is just a laptop screen which turns red when you're live, or green when you're not but there's nothing stopping anyone from changing that into, say, a USB-controlled light or some such. In any case, it works now. It's not entirely pretty (involving two processes working with the same socket) and not entirely finished yet, either (it only works for primary sources, not secondary sources), but it gets the job done. It did require me to refresh knowledge that I hadn't used in literally 10 years, however. I had some C++ at school, but haven't used it since my final exam on that course, and while the code is still readable, it did require me to refresh my memory at some levels. Oh my... Anyway, that was yesterday, for the most part. Today, a friend came over with a DV camera (since I don't have any), and we ran some tests. Most of it worked immediately, but there were a few bugs, which I ironed out today. Right now, it works correctly the parts that are implemented, at least. This video stuff is rather fun to play with.