There is a lot of career advice related to writing CVs in books and floating around the net. Unfortunately almost all of it is tailored to permanent positions. Some advice that I have seen includes researching the company you are applying for and emphasising your experience in related fields. This probably makes some sense if you have targetted a major company to be your employer for the rest of your life. But if you are after contract work (which incidentally tends to pay better than permanent positions) then it’s not going to work. About half the time when applying for contract work you won’t know the name of the company that is receiving your CV until after they have decided whether to grant you an interview. Contractors therefore need to write their CVs to have broad appeal (emphasising common technologies such as C programming and skills such as debugging), and to appeal to a recruiting agent. Some people recommend writing your CV in MS-Word to support agencies and potential employers who are not competent enough to open other file formats (such as PDF and HTML). I strongly disagree with this advice, most contract positions are advertised by multiple recruiting agencies and it would be best for you to be represented by the most competent agency. Therefore having an agency staffed by incompetent people not represent you because they can’t figure out how to read a HTML or PDF file is a good thing! The same applies for employers, while applying for the best paying contracts will often result in some strange and unpleasant working conditions having a manager who can’t open a PDF or HTML file is likely to result in something that is unusually bad - run away! Recruiting agents will often give specific requirements for a CV. Sometimes these requirements are a little odd (such as demanding that references to C programming be removed from a CV when applying for a Unix sys-admin job), but being merely a little odd is no reason to boycot the agency. To solve this I wrote my CV in M4 macros to generate HTML code. When I wanted to apply for a position I would just type make to compile the M4 source into five or more different CVs. Sometimes a recruiting agent would request two versions when applying for one position, it’s a strange request but when five versions were compiled in one pass it was easy to satisfy it. If I liked PDF I could have written M4 macros that generate TeX and have the make process convert TeX to PS and then PS to PDF. Unfortunately I can’t share any macros because the code was not of particularly high quality (which is a little ironic) and because it’s tightly tied in to my CV. The best piece of advice that I can give to anyone trying this is to get columns sorted out as the first issue. If you want to have some lists of programming languages, lists of OSs, and lists of hardware, etc in your CV then it usually works best to have two columns in the page. If the largest list is removed then the other lists need to be shuffled around to get the column lengths to match. Doing this in HTML required some ugly M4 macros. There probably is a better way of doing this in TeX (but I’m really not good at TeX and can’t give advice about it). James has some interesting advice about writing CVs. I think that his most useful points are:

1. Use file names based on your name so that agents can find your CV in the directory full of CVs


2. Include complete contact details. One thing he doesn’t mention is that you should buy a mobile phone for the purpose of getting a job even if you don’t want one for any other reason. A 5% difference in pay for a year’s work would cover the cost of many mobile phones and the difference between a good job and a bad one is likely to be more than 5%. If you are afraid of brain cancer then don’t worry, if you only spend a few minutes a month talking to recruiting agents then you are at no risk.


3. Don’t start a sentence in the career history section with the word worked - everything in that section will be related to where you worked in the past. It seems obvious in retrospect but I realised that I have made this mistake on my CV.

Share This

Ben Fowler blogs about the issues related to running IRC as root. Google searches for (irc client exploit) and (irc client “buffer overflow”) give a number of interesting web pages. Many of the exploits require the user to perform an action that’s slightly unusual, but why take a chance? The advice to not run as root while generally sensible (run everything with minimum privileges as much as possible) is IMHO not very useful in recent times (and probably was never very useful). Generally when a user is worried about system compromise they are not worried about attackers having direct hardware access, the ability to corrupt system files, etc. They are worried that the attacker might read their email and access other personal files. Therefore the instruction should be “don’t run IRC as root or as any account that has access to data which is important to you“. It’s not difficult to start an X-term that runs “exec su - ircuser irc” or “ssh -t ircuser@localhost irc“. Note that the -t option is required for ssh to make it allocate a pty even when receiving a command to run. Note also that in the case of su you need the exec option so that if the irc client is compromised and tries to perform a ioctl(0, TIOCSTI… based attack then it won’t succeed. In any of these methods make sure that X access is not granted. Until we get Security Enhanced X working in a viable manner any process that can display an X window on your screen can own you totally. There are of course relatively safe ways of doing X, I have previously documented how to configure the Xephyr X server (replacement for Xnest) to allow a process with a different security context to safely display a graphical window on your desktop. Generally I recommend not using a graphical X client on an untrusted network (IE anything other than an Intranet IRC server). I prefer to do my IRC in an account that’s not even on a machine that I care about and have it run screen so I can disconnect and re-connect from anywhere in the world. When I first got SE Linux in Debian to be useful (when I could boot and run all programs without problem) I logged on to some IRC channels related to Debian with the security context of root:user_r:user_t. I admit that my actions in this regard could possibly be described as trolling, but I wanted to demonstrate what SE Linux can do. Unfortunately of the many people who told me off for logging in to IRC as root, none of them wanted to hear an explanation of why user_r is safe in this regard. I expect that most of them were running their IRC client in the same Unix account that was used for their email etc (and probably most of them had GPG keys accessible from such an account). Sigh, it’s so easy to run IRC as a different user - in fact it’s probably the easiest of all network client programs to run in such a manner. There’s no reason not to.

Share This

A few days ago, Joe Reagle was telling me about the rise of (sometimes very arcane) policies and bureaucratic imperatives that now characterize Wikipedia. A few days later I stumbled across a few interesting posts on the topic, posts (especially Nick Carr’s) seeping with almost celebratory gloom and doom:

‘But, given human nature, is it really so “incredible” that Wikipedia has evolved as it has? Although writers like Yochai Benkler have presented Wikipedia as an example of how widescale, volunteer-based “social production” on the Internet can exist outside hierarchical management structures, the reality is very different. As Wikipedia has grown, it has developed a bureaucracy that is remarkable not only for the intricacies of its hierarchy but for the breadth and complexity of its rules. The reason Deletionism has triumphed so decisively over Inclusionism is pretty simple: It’s because Deletionism provides a path toward ever more elaborate schemes of rule-making - with no end - and that’s the path that people prefer, at least when they become members of a large group. The development of Wikipedia’s organization provides a benign case study in the political malignancy of crowds.”

It is without question that a problem has arisen in Wikipedia, a problem composed of a thick web and net of rules that can be helpful as guidelines but often are often confusing and clearly work to ensnarl new users. But can we so quickly put blame on so-called human nature? Or is it not a problem of human organization, which as the anthropological and sociological record shows, can take many forms? And is it not just that: a problem begging for a solution instead of an opportunity to declare the fundamental nature of wikipedia (and that of human nature?). Not only may history prove him wrong, other large scale collaborative projects at least prove that solutions can be found to deal with problems of growth and scale. It is as if Carr just wants to see a project like Wikipedia fail, which somehow, this morning hit a raw nerve. What I find exciting about large scale projects of (at times unwieldy) collaboration are not just the explicit outputs of the projects (an encyclopedia or operating system) but the social worlds they create. And there is no inevitable path they *must* follow. These groups have a choice to react to and respond to these sort of problems and enact solutions that will hopefully solve them and allow these projects to change. Debian, a slightly older project than Wikipedia, has gone through many growing pains and there was even a period when the the whole process of integrating new Debian developers was shut down and if my memory serves me correctly, for 2 years! At the time, it could have been possible to say: “This signals the end of Debian” but eventually a solution was found, the New Maintainer process, which while not perfect (what is?) allowed the project to grow and produce a great operating system for years to come. Debian today faces new problems and is working to find solutions. I hope that Wikipedia can and will do the same. And instead of declaring its death, why not wait and see, and offer something a little more constructive and illuminating, than destructive?

After much wading through possible names, none of which really excited me, I have finally chosen "DAViCal" as the new name for my CalDAV server that was previously called RSCDS, or the "Really Simple CalDAV Store". In the end, I chose DAViCal because it:

• seems easy to pronounce.
• combines the 'Cal' and 'DAV'.
• returns < 1000 results on google.
• doesn't make me cringe.
• didn't have a domain name registered.

That was about the hardest part of preparing for the 0.8.1 release, and now that I've done that I should manage to make the changes to the packaging, though I have no doubt that the old name will appear in all sorts of places for a while yet. Choosing names is an important business, and I should know that from the length of time we spent agonising over names for our children, discarding all sorts of things because they had silly abbreviation collisions (like the "Royal Scottish Country Dance Society" :-) Even then, I think we got the kids names wrong, and the big one should be called "Thumper" with the little one called "Sly", but perhaps that's just a temporary annoyance and in time the names that we registered for them will fit them better. I also recall Grant once saying that you should never use the word "Simple" in the name of your project, and he should know. DAViCal is no longer particularly simple, although I have attempted to hide the complexity from the user as far as that is possible, and will continue to do so. Once I get out version 0.8.1 of DAViCal I will finally upload it to Debian, proper. This version has some important enhancements to its DAV spec compliance which are going to be needed by some future versions of Mozilla, and probably other things too, so it's important to push it out as soon as possible now.

I just completed a very successful and long overdue dist-upgrade of my unstable machine, using mainly DebTorrent for downloading the packages, so I thought I'd post some of my thoughts and experiences. The download consisted of 1294 packages to upgrade, totalling 1350 MB and taking 2h12m to download. Here are some of the good and bad things I observed. Good things There was a single other peer with me in the same torrent, and I managed to download 182 MB from him, which is about 13.5% of the total download. This is the first time I have noticed this downloading from peers occurring, as usually there are too few peers, too many torrents and too many possible packages to download for any sharing to occur. This will hopefully change in the future when more people start using DebTorrent, and when unique piece numbers are introduced to make the torrents last longer. However, it does show how the use of the backup HTTP downloader can seamlessly integrate with downloading from peers to provide a good user experience, even for early adopters. The CPU time used was only 10m46s, which translates to an average CPU usage of 8%, which is very reasonable. The average download speed was 174 KB/s, which is 58% of my maximum download speed. Though this may seem like a bad thing, my goal all along has been to make sure that the download time would not be more than twice as long as using HTTP. Of course, using DebTorrent may never be as fast as a straight HTTP download from a well-provisioned server, but that it not the point. The idea is to reduce the bandwidth needs of hosting a debian archive. But in the future, when there are many peers in a single torrent the download speed may be even faster than using HTTP, especially for peers with very high download rates that could not be matched by a single server. Things that need improvement The completion percentage reported by APT during the download was fairly inaccurate. Here are some sample readings I noted, compared with the actual completion percentage from the DebTorrent status page:

APT Reports	Actual Completion
10%	40%
20%	70%
30%	85%
40%	90%
50%	93%
60%	96%
70%	97.5%
80%	98%
90%	99%

The discrepancy occurs because larger packages are broken up into pieces, so they can be partially downloaded without APT knowing about it (since only fully downloaded packages are passed to APT). Clearly this situation is far from ideal, and can lead to the user feeling that the download is progressing very slowly, or not at all. There are some plans to add status updates to the communication between APT and the DebTorrent client, but they require changes to the APT code to support them, so it may take some time to implement. Another problem is the memory usage I saw during the download, which was approximately 213 MB. This is obviously unnecessarily large, though the metainfo that DebTorrent needs to be aware of is quite large (stored in a text file it is about 3 MB). There does seem to be a memory allocation bug in Python 2.4 which causes increased memory usage, so moving to python 2.5 might help. However, my preliminary tests show this only saves you about 20% for DebTorrent. I think I will have to delve deeper into which parts are using all this memory, and unfortunately python doesn't seem to have a good memory profiler to help with this. I will be looking at both PySizer and Guppy/Heapy to start, but if anyone knows of a better solution, please let me know.

I was recently talking to a client about the lack of guidelines for acceptable personal use of office resources in his company. He rejected the suggestion that he provide any real rules or guidelines (apart from some old rules that most employees were not aware of and of which there was no procedure to remind them to periodically read). He said that he only wanted to hire motivated people who wanted to work, and that people who need rules should work elsewhere. The results of his approach seem reasonable, everyone who works for him works really hard. In return his attitudes towards employees are more relaxed than most employers, among other things he is very supportive of employees who desire career opportunities greater than his company can offer. I have been wondering whether rules alone can make people lose interest in working, or whether it’s the type of person who doesn’t want to impose rules that inspires hard work. My experience is that when working for people who have a rule-free environment I tend tend to work really hard, and that when working for companies that have lots of rules I find it difficult enough to get out of bed in the morning - let alone become motivated to do any work. I am interested to see comments from other people, both workers and managers. Is my experience common in this regard?

Share This

The Google Summer of Code coding period has ended earlier this week, so it seems to be a good time to stop and look what was done (and what wasn't). Let's start with an objective analysis, following the list of planned features:

Advanced query of bugs on the Debian BTS

In a 1-10 scale, I would give myself something like 6 or 7 here. bug-triage can query bugs by number, source package, package, maintainer and submitter. These parameters can be combined to make an query (reports must match all parameters) and more than one query may be added, subtracted and and'ed. A clearly missing feature is querying by usertag, support for it has been added on btsutils already and is just a question of adding a new input box on the query builder dialog (it should work if you enter the query manually in the bug-triage query line AFAICT).

Show all the information related to a bug report

bug-triage has a "goto" button which opens the bug in the user's preferred browser. Missing is the support to open the bug in a MUA, which I have promissed to Lo c.

Send additional information to a bug report

bug-triage has a "followup" button which opens the user's MUA with the right address. Maybe setting the subject would be nice...

Manipulate the tags of a bug report

I haven't had time to implement this at all. I've started it, the bugs have a context-sensitive menu which have each of the available tags as checkboxes on the right current status, but clicking on them does nothing currently.

Search for similar bugs in upstream developer BTS

This one would value something like 3 or 4 in a 1-10 scale. It searches upstream bugzillas by product and component, but doesn't do any cool trick like searching words in the bug subjects. The plus is that it does support KDE bugzilla, even though I haven't been brave enough to promisse it before SoC :P

Forward a bug report to the upstream developer BTS

There isn't really a good interface to report bugs programatically on GNOME bugzilla, which was disappointing. <rant>There is the bug-buddy xmlrpc(?) interface, but the developers say they will actively break anything using it, which I find very free-software-unlike.</rant>. The current bug-triage solution opens a browser with some of the fields filled. It would be nice to fill the full bug description, but its limited by the maximum size of an URL, so it fills only an introductory text (something like This bug was filled on Debian BTS #XXX) currently. Let's hope bugzilla 3 get on the GNOME world soon.

Conclusions

Unfortunately, this project hasn't evolved as much as I would like in the second half of the GSoC. As I said elsewhere, a lot of stuff has happened, some good, some bad, but on the end everything I hadn't been able to dedicate myself to anything I should. Even so, I think the project may be considered sucessful.

Maybe the more interesting product of this project, at least currently, is still on python-btsutils. As I said in the middle-term report, it'll be integrated in the debian_bundle soon, and might be powering important stuff like the devscripts in the future.

The bug-triage application has a lot to evolve yet, but is also an interesting tool IMHO. There are a lot of hurdles in the way to get different BTSes integrated, but the effort will for sure pay in the future. I plan to keep developing it (evon though keeping the GSoC pace is impossible) and am anxious to see the results ;)

Finally, as I said before, one of the main things on GSoC for me was the potential to learn cool stuff. The best choice I've made during all of this was to use Python. Even though it's still a new language for me, and i still have to search embarassingly simple things in the tutorial/reference, it was a lot more productive than if I was developing in C. Learning to use git was also a great. This goal was the more fulfilled one, I've really learnt a lot of things which will surely be useful in the future. BTOH, I still can't get used to glade :/

python-gammu 0.22 has been just released. New features:

• Complete support for managing calls.
• Compatibility with current gammu.

Thursday morning I arrived Chubu airport near Nagoya and at this time started my first visit to Japan. It will take quite long time (3-6 months), so I should have enough time to do as much sightseeing as I want. But there was not much time for sightseeing so far. Besides work and having rest after 24 hours of travelling, we only went today to see a bit of city. No real sightseeing, it was rather finding out some things. The most important was to find ATM accepting our credit cards. I didn't expect it will be that difficult, but it turned out that most ATMs accept only Japanese Visa and MasterCard cards and not ours. Fortunately post office has ATMs accepting foreign cards, so we finally have enough cache to survive next week ;-). The next step was to buy something we can start to learn Japanese a bit. It would be great to be able to read at least some signs and menu in restaurant. We bought few books, but later in restaurant it turned out to be almost useless. Fortunately the cook can a bit English, so we managed to order something we wanted. But this is the largest problem here - our poor Japanese knowledge as there was not time to prepare for this back at home...

This is the first post of a series on componentization, starting with a brief explanation of the problem(s) we are trying to solve. I expect some familiarity with programming in general, but no specific knowledge of any particular system; the terms "DLL" and "shared library" are used interchangeably. When you are writing a DLL, you are essentially committing to support whatever interface you are exporting forever. So it better be good on the first try. DLL hell is what happens if it was not good, and you end up making an incompatible change. The obvious way an interface can be "not good" is when you need to touch the header files that define it in order to implement some new functionality. You can get away with a few things, like the addition of new functions, if you can somehow make sure that users will always have a recent enough version of your DLL around, but in general any change to the interface is going to be incompatible. The traditional Unix way of solving the issue has been renaming the library. When you link a program against a shared library on a system that uses the ELF format, the linker looks inside the library for the SONAME field, which tells it what name the library wants to be known under, and the program will then attempt to load this file at startup (which usually is a symlink to the latest compatible version). This "official" name of the library usually contains a number (sometimes more than one), and when there is an incompatible change, you simply change that number, and the new version of the library can be installed in parallel with the old. There are additional mechanisms, like symbol versioning support, that allow a single library to provide different "versions" of exported functions, and have the program specify the desired version for each symbol. To run a system like this, one generally needs to keep track of programs' dependencies; this generally happens in the package manager with more (Debian) or less (Red Hat) support from package build scripts. If you built your system from scratch, you end up doing this work yourself. That's why you had highly paid system administrators in the old days, and very short shell scripts to replace them (with various degrees of success) in the current age. In the Windows world, things have been a bit different, because except in large installations, you generally do not have a clueful, benevolent system administrator who will take care of these things; rather, you have users that expect your software to work without having an insight into the inner workings. In general, software developers have not been taking chances and simply shipped all the DLLs they needed along with the main program. When they installed these DLLs into the same directory as the program, things were fine, if they installed them into the WINDOWS directory, not so much, because when you had two conflicting versions of the same DLL, you had to decide which one to keep, and a program that would only work with the one that you threw away would simply fail in curious and interesting ways. This is the original definition of DLL hell — you cannot do it right. To make matters worse, this is not the only way an interface could change. When you define a platform, you write up an "Application Binary Interface" (ABI) document that details conventions that should be followed; for example, the ARM processor does not have a dedicated stack pointer register, so the ABI document for Linux on ARM says that R13 is to be used as a stack pointer, functions may thus expect it to point to memory that is readable and writeable, and memory with an address lower than the stack pointer is considered "free". When the function is done, it is supposed to jump to the address in R14, which happens to be the register the ARM's "Branch with Link" instruction sets to the address of the instruction following it. If the function takes arguments, the ABI says where to find them, and if it returns a value, it also tells where to place that. For everything not in the ABI document, the compiler author gets to choose. The ABI document usually attempts to be rather minimal, so it defines how C functions are to be called, and calls it a day. Anything not in the C standard is not guaranteed to work between different compilers (or even different versions of the same compiler). That includes C language extensions such as the "long double" data type, language features in C++ and Java such as namespaces, nested types and exceptions, and the data structures used by the C library's malloc() function. The latter makes life difficult even for pure C programmers, as it disallows releasing memory from a different module than the one it was allocated in, because the different modules may link to different C libraries. In essence, this means you are not supposed to export anything from a DLL that is not a C function, and even then, there are additional requirements. You could in theory export data objects, but that is discouraged, as it exposes an implementation detail. Now, we're in a bit of a dilemma here. There is lots of interesting code out there that is not necessarily written in C, we'd like to reuse it and we don't want stuff to break. The Linux approach so far has been "we simply pretend the problem does not exist", and this has been working to a large extent, simply because everyone uses gcc, and when the ABI extensions the gcc people had made changed, recompiling everything that uses these extensions makes the problem go away. This has several drawbacks. For one, it mandates that the tools used to rebuild everything and keep the installation consistent be written in C or handle the transition gracefully (for the curious who have a Debian system handy: look at libapt-pkg's SONAME), and it also requires that we have the full source code for everything not just available but buildable in an automated way. Within the context of a Linux distribution, both of these are true, so we have survived the last major ABI transitions without too many losses. So we need an extension of the ABI document that specifies how modules written by different people and compiled with different compilers can interact. These are commonly called component models, and, to quote Andrew Tanenbaum, "[t]he good thing about standards is that there are so many to choose from". Next: "componentization tutorial #2: CORBA and COM"

And no, I don't mean Mao.
How was Thursday 23 for me?
6:30 Wake up, knowing it will be a long day. No, don't get those delicious extra 15 minutes of sleep.
8:00 Arrive to work, ~90min earlier than usual, to finish the Rails app you have to demo at 10:00
10:00 Meeting with the Institute's Director, Academic Secretary and Technic Secretary to show them the system for following the internal inter-tematic colloquium information workflow. Thankfully, they were happy with it.
11:20 (20 minutes after scheduled): As several academics have said they are interested in getting to know Linux, I'm giving a series of introductory sessions on Debian to them. So, I went with some of them through the Etch installation. Of course, mostly everything was fine. What was not? Resizing a Windows partition is still a step that requires several jumps through hoops (i.e. booting Debian LiveCDs, some console-based interactivity which I wanted to avoid...), and... Well, d-i does not time out if the selected mirror is not responding. Will file a bug, yes. Anyway...
14:30 Managed to leave the lab working, with no damages. Went to the bank to check some data and to have a quick lunch.
16:00 Get back to the institute. Jos Luis, from Difusi n y extensi n was already waiting for me to show a set of components for the Comas-based system we use for tracking attendance to the Institute's activities
16:45 The people from AMECIDER arrived so I could show them my proposal for their new website. Can you believe most of the websites I host are still completely static HTML, and, what's even worse, generated by the $ despiseful_adjective Dreamweaver? Yes, I am proposing them to move over to Drupal
17:30 I finally received another person, one of the people that's attending my Thursday lessons, wanting me to help with some Linux questions. What's Linux's killer app for my Institute's researchers? Believe it or not, a not-brain-dead PDF reader. They are amazed that, using Gnome's Evince, they can just copy-paste bits of PDFs into their documents (and the fact that pasting tabular data into OpenOffice's Calc allows for an easy import feature - Yesterday I spent ~1hr with this researcher importing data generated by the International Monetary Fund into his work), and they are even more amazed with the gnome-screenshot they can directly include graphics from such documents without redrawing them.
19:45 (45 minutes behind schedule): Went to the Astronomy Research Institute, to meet with Sergio, H ctor and Eric, to decide on bits of the implementation of the new gear we will be putting in nisamox, our dear and faithful ftp.mx.debian.org mirror. Thanks to everybody! We will soon tell you what we got and what we did with it, it's mostly settled by now.
20:30 Had the first of some... Four calls regarding yet another system I have been working with for the ICME conference, to be held in Mexico in 2008. Several pushing details that need my attention and work, bah :-/ Have to work on them...
21:00 Got home. Did a couple housekeeping activities (i.e. take out the trash, buy some food...). I was too burnt out to start making my bags for going to Vienna (I'm leaving tonight!), work on the pending systems I have to work on, or even stay awake.
Leaving work for a week to go a conference is always fun and great... But man, it's stressing!

So it would seem that the forces of "justice" in Canada might not be above a little shit-stirring to cast their opponents in a bad light. Considering all of the hoo-hah around the security of the upcoming APEC meetings in Sydney, I wonder if the local constabulary has any similar plans? Remember, if you're planning on protesting the senseless waste of money that is APEC, carry your ID at all times, to prove you're a legitimate protester and not a security shill! <grin>

Here’s an interesting CSPAN interview with Dick Cheney from 1994. It’s amazing how accurate Dick’s analysis of the Iraq situation was in 1994 when justifying the decision to merely destroy Saddam’s army in Kuwait and not try to occupy Iraq or catpure Saddam. It’s a pity that he didn’t stick to that idea. Since the old CSPAN interview became popular the MSNBC show Countdown with Keith Olbermann covered it (youtube link). Keith initially said some ridiculous things about heart surgery affecting people’s emotions (that was the medical opinion about 2000 years ago, doctors have learned a lot since and Keith should learn from them). Then John Nichols of The Beat blog and author of Dick: The Man Who is President (Dick Cheney) makes some interesting comments. John interviewed the professors who taught Dick at university and their opinion of him matches the current observations - that he believes that the US government can do whatever it wants with no consequences.

Share This

Some friends and I are considering to set up something similar to Steve s xen-hosting.org. One issue that came up was: Who get s to be the main admin and how can we trust him? At first I found this an odd issue I ve never heard anyone considering this when using one of those anonymous virtual server hosting offers.After all, when you have something valuable to store somewhere, e.g. your family store of gold coins, you d rather ask your good friends instead of some random stranger. But it seems to be different for information: Imagine you have some valuable piece of information (e.g. your dirty secret from ten years ago) and you had to share it with someone suddenly the completely anonymous stranger seems to be a better choice than the persons that know you...I think this effect is also the reason why people are so little troubled by known and unknown facts such as

• E-mail-communication is open to any one having control over a server inbetween you and your recepient.
• Messages sent over AIM are (or were, it seems), according to the Terms of Services, legally owned by AOL.
• Google uses the contents of your gmail e-mails to do stuff.

This is just a small part of the list of situations where people give information to some unknown stranger that they would probably not give their friends.Is this way of thought valid? If not, what are good arguments to convince people to be more careful? And under what circumstances would you trust your friend to manage the host machine for your Xen server instance?

It's something north of 100 degrees F (I think that's 40+ in celcius) and sticky. The museum was way cool. I'll post about it when I can upload the pictures as well.

We're crashing with Stephanie Truax this evening in Indiana. It's time to head west.

Firstly, a big hello to Planet Ubuntu! As an Ubuntu community member and Ubuntu core-dev, I've decided to add my livejournal to there. Since I usually don't make any effort to sensor my thoughts on here, I chose not to add it while I was still with Canonical in order to make sure that there was no confusion as to the OfficialStatus(tm) of anything I write.

The last two days have taken us from Toronto through to Cincinnati. Andrew took us down to the Buffalo airport, we met Jason and Brian from Koolu on the way day and finally wound up down there at 22h30 despite having left NewMarket at 11am. When we stopped at Babur for lunch, we wound up seeing Nancy and Alexis walk past.

We stayed overnight at the Mum's Red Carpet Inn, just over the Pennsylvania border. It didn't have Red Carpet. It *did* have Orange Shag on the side of the bed, however. High class. The continental breakfast was Tang, Bananas, vegan doughnuts and twinkies. The vegan doughnuts were full of palm oil, I don't think my heart rate has recovered yet.

On the drive towards Kentucky, we got hungry (and tired of Subway) when we approached Columbus. We first tried to go to a place called "Leaf Restaurant", which it turns out can feed us exactly zero things. Down to the pre-made chef salad, and the spaghetti sauce having meat in it. We went a bit further, and decided to try the university district. We pulled over after seeing an Ethiopian place, and saw Ben Sostrom, whom we'd just met at Opus the week before! We spent the evening hanging out with him, michael dobson, amber, and the rest of their YA group. They meet on Tuesday nights a couple blocks from where we parked! It was nice to have a post-Opus day like that, but their stories of carjackings and cops randomly pulling people over are hopefully overstated by several orders of magnitude!

We continued the drive down to just outside of Cincinnati. We're meeting up with Stephanie Truax today for our tour of the Creationism Museum. After this, we'll turn west and try to find our way to California.

[Long post, sorry. If you're short on time skip to the end for the juicy parts.]

It all started about a week ago when I decided to find out how many files were unique in the whole Debian archive, and how many were duplicated in the same package or in other packages. I had done some work on duplication detection before, and I knew that the process involves getting some kind of hash value of every document, then finding duplicate values in the list of hashes.

I already had a local Debian mirror so the raw data was all there. I basically had two options: either compute the hash value of every file myself from the packages in my mirror, or find some other (ideally faster) way to determine if a file is unique. I quickly decided that the best option was to use the md5sums embedded in most Debian packages (in the DEBIAN/md5sums control file). That would give me MD5 hashes of all regular files, excluding conffiles.

So the first step was to check how many packages have embedded md5sums, and a simple script showed that less than 3% don't have them. This first check exposed a bug in python-debian, which was duly reported. Along the way, my post prompted a discussion on debian-devel about the state of md5sums, and I set up a daily check to keep track of things.

The next step was to make sure that all md5sum-enabled packages had usable information. It turned out that python-debian choked on 103 packages because of embedded spaces in filenames, and that it also had a blocker bug when used in Python 2.4. I filed two more bugs.

All that remained was the easiest part: write the program to find duplicate files. I did that, and I now have all kinds of funny statistics:

• The dataset consists of 20170 packages with md5sums, shipping a total of 2069830 files. That gives an average ratio of 102 files per package, excluding conffiles.
• There are 113732 duplicates in the archive. 1556 files are duplicated more than 10 times, and 14 files are duplicated more than 300 times.
• The empty file is present 8325 times in the archive, spread over 874 packages. This isn't surprising since it's used for all kinds of purposes like Python's init.py files, Perl's .bs files, etc. I also learned (among other oddities) that the python2.4-doc package ships a few zero-byte .png files. Uh?
• Also popular is the file with just one newline character in it: 343 occurrences. In the same vein, we have 461 occurrences of the "deny from all\n" file.
• Most of the hits are for Doxygen images in -dev and -doc packages, namely doxygen.png, tab_b.gif, tab_l.gif, etc (about 350 hits each). In the same category, gjdoc CSS files (149 hits).
• The partlibrary package is our worst offender. It ships a total of 9680 non-empty files, and only 4833 of them are unique. 6 files are duplicated more than 400 times each in the package.

If you're still reading so far, thanks! You can see the report file, and the program itself. If you want to run it, note that it's optimized for speed rather than memory efficiency; it runs in under 3 minutes but uses up to 1.5GB of memory (my home desktop has 4GB).

The grand conclusion is that all things considered, there is very little file duplication in the archive: the #1 duplicate represents less than 0.4% of the two million analyzed files, and it doesn't actually use any space since it's an empty file... :)

Problems for shared desktop computers. I share a home desktop computer with my wife. Our computer use is probably pretty typical of a home computer shared between a small number of users, where only one user is logged in at a time, and users have some shared files and some private files. I've found many things that don't work as well as they could in this situation. We use Gnome on Debian GNU/Linux, but most other desktops have similar problems. I know fixes or workarounds for most of the problems below, but novice users would probably have a hard time finding them. Some of the problems below are relatively easy to fix. Most already have bugs filed in the appropriate places, and some already have patches available. A few problems may require hard work or fundamental design changes to solve. Here is the complete list of problems: User Switching:

• If I select my name from FUSA while I'm not already logged in, GDM appears and I have to type my username. (The username should be filled in automatically.)
• When GDM is started by FUSA, the X server has slightly settings, causing the fonts on the login screen to be a different size than my normal settings.
• When I log out, I am taken either to a login screen or to another user's session, depending on the order in which users originally logged in. This is arbitrary and unpredictable.
• If I switch to another user, my screen is locked. Later, if the computer is displaying a login screen and I type my username and password, I need to type my password a second time to unlock the screen.
• When I shut down the computer, it does not warn me if other users are still logged in.

Filesystem:

• By default, users can read all of each other's files, and write to none of each other's files (umask 0002). This is a reasonably sensible default, but most users on shared systems will want to change this to some degree.
• It's very difficult for a user to change which other users can read which of her files. (Some strategies include making some or all files world writeable; adding the other users to her primary group; creating a new shared group and a share setgid directory owned by that group; changing her umask; and/or manually changing permissions of individual files.)
• There is no good location by default for shared files. Various programs complain (correctly) about insecurity if my home directory is group-writeable, so I must create a world- or group-writeable (and setgid) directory elsewhere.
• Copying from other locations, or using programs that do not respect umask, can cause users to inadvertently put non-group-writeable files in shared setgid directories. Other users of the shared directory have no easy way to fix this.
• Some programs modify permissions or groups of files when overwriting them. (Usually this happens if the program moves the original file to a backup location, then writes a new version using the default settings instead of the original file's settings.)

Application data:

• Applications that maintain a database of files (e.g. photo managers, music players) must be updated for each user when one user adds new files to a shared directory. For example, when I rip a CD to our shared music directory, my wife can't see the songs in her music player until she adds the new files to her database.
• Applications that store their data in hard-coded locations in user home directories (e.g. Tomboy, Miro) can't share files between users. There is no setting or permission that will let my wife open Tomboy and view my Tomboy notes, for example.

Installing software:

• When I install software using the system admin tools, it changes the application menu for all other users too.