My mother just asked me to get Google Earth (link to download page [1]) working on her machine so she can see where my sister [2] lives. So I download all 20 megs of the sucker (of course it had a horrible license agreement that precludes packaging for Debian or any other sane distribution) and ran it (under a different account of course because I don’t trust non-free software). It’s unfortunate that Google weren’t prepared to put in the extra effort of making Debian and RPM packages for it. When I examined the contents of the file there were sym-links from FreeBSD, NetBSD, and OpenBSD to Linux and from amd64 and x86_64 to x86. So much for portability, just assume that everything is an i386 running Linux! The first problem I encountered is that it doesn’t support installing in text mode and demands X access, have the Google people never heard of sys-admins who do their work over low-bandwidth links and don’t use X? What happens if your sys-admin is using a braille terminal? When I enabled X by “ssh -X” I encountered the error below (Error 1). It seems that the installation program was not written with the X11 Security Restrictions in mind and only works with “ssh -Y“. The same applies to actually running the program once it’s installed. Finally it gave me a GNOME error dialogue about the Bitstream Vera Sans font not being installed. If they had made a Debian package then it could have depended on the package that provides the font in question. Now I’m left wondering which package provides it, and whether it’s even available (maybe they depend on non-free fonts). Once I got it working I found it to be incredibly slow on my Pentium-M laptop with 1.5G of RAM. I ran it on a Pentium-D desktop system in a Xen DomU (it’s only 32bit and I couldn’t figure out how to get the AMD64 version of Debian to run i386 programs without a chroot or a Xen instance). But it still wasn’t particularly fast. One problem is that the GUI is not well written, so I drag the Earth to rotate it from the initial view and the Earth moves slowly long after I have released my mouse button. Another is that my X server started using significantly more memory while it was running (and has not returned to it’s previous size). Google Earth is a really neat program that does some nice things. But I’m not going to buy a new Thinkpad to run it, and the experience of running it on a Pentium-D server is not that great either. I’ve just rm’d it from my laptop, I’ll leave it installed on the server but probably won’t run it often. I’ll try installing it on my mother’s computer (Celeron 2.4GHz with 512M of RAM) but I am not expecting it to run well. The machine is still new to her, I replaced her Pentium3 800MHz with 384M of RAM about a week ago. She never found the Pentium3 to be slow (and I don’t think that Google Earth is a good reason for an upgrade).

• [1] http://www.earth.google.com/download-earth.html


• [2] http://helen.coker.com.au/

(more…)

Share This

Homebank has had a few problems getting through NEW due to legacy problems from the original Amiga version (and it isn't ready yet) but I've been trying it out anyway and comparing to gnucash. Bearing in mind that I have done upstream work in gnucash (i.e. I like gnucash enough to tackle the massively scary gnucash source code) and have (almost) got used to the regular dependency hell that it imposes on my sid installation, I'm finding homebank quite friendly.
(See #441110, #441103, #441129 and #441363 for recent examples of gnucash causing dependency hell - again.)
(1:0 to Homebank).
Homebank is closer to the Quicken model with categories but internal transfers between accounts are supported to give some impression of double-entry. (1:1) My main interest is in data import/export because I've long been looking for a way to create invoices from PIM data e.g. in a Palm Pilot or similar. Homebank supports a CSV data import for data that can be handled as an invoice. OK, there's no fancy invoice printing but that doesn't bother me (I never used the gnucash invoice print support) (2:1 to Homebank) - my invoices are all electronic and I have to use the client system (MSDOS!). What I need is to create my own records of the invoices within my financial accounts and Homebank may be able to do that with far greater ease than gnucash. My data exists primarily in my Palm because I take and change my bookings on-the-move. I want to be able to HotSync my Palm through a query interface (pilot-qof) and XSL (datafreedom-qsfxsl) direct into a financial application that can correlate that data with the payments made into my bank by the client and thence with data from my bank statements to prepare data suitable for my tax report.
I'm thinking that if I can get half of the ease-of-use of Homebank into the rigorous double-entry support of gnucash in a small program (gpe-cash), I'll be doing very well.
One limitation of Homebank is the "wording" - the translations appear awkward and unfamiliar but that can be easily fixed. The link between "Archives" and "automated transactions" isn't clear - I think it means "templates" and "scheduled transactions". (2:2)
Homebank can't do all the things that gnucash can do but that is a weakness in gnucash, not homebank. (3:2 to Homebank) IMNSHO, GnuCash tries to do too much and that leads to an oversized and overly complex codebase as well as a dependency list that puts OOo to shame. The graph support in homebank is a lot better than gnucash and that is not the fault of libgoffice, it is entirely down to the choices made by gnucash upstream developers. Homebank graphs are fast, simple, intuitive and informative, displaying useful info at the first attempt instead of having to wade through the Options dialogue in gnucash. (4:2 to Homebank).
If I can work out some simple scripting that combines these external tools into 'HomeOffice' support in Homebank, everyone wins.
I've never liked CSV as a data format and the homebank main files are XML (like gnucash). Unlike gnucash, the XML appears sane and usable for XSLT - raising the prospect of creating reports directly from the homebank data files.
Homebank upstream have been very supportive of the various requests to assist getting Homebank through NEW so I'm hopeful that if an XML import/export method was proposed it could be included in future - especially if the donkey work is done by external tools.
This would tie in well with datafreedom-qsfxsl and datafreedom-perl. Initially, I'll look at XML->CSV - ugly, I know but at least Homebank can import data to be used as invoices; gnucash cannot. (5:2 to Homebank).

At the end of round one:
Homebank : 5 GnuCash: 2.

I spent much of my early childhood at the video arcade. I was fortunate enough to have a best friend who 1) Lived in the old quarters of San Juan where you could walk everywhere 2) Had three older teenage sisters who would take us when we were still quite young to the arcade, which also doubled as a fantastic ice-cream parlor. My favorite game there was galaga, and I played it incessantly (and had to use a foot stool too!!) As a result of all that childhood gaming, I am still pretty half decent at it. For example, I more often than not beat my partner (except under less than auspicious conditions, like being sleep deprived), which is pretty unbelievable as he is much more of a video game wizard than I and otherwise beats me in everything.
Back in my first year in grad school, a cohort of us would play Galaga after our large theory class, and I would also always win (and by a lot and then get crushed at foosball). So of course when I heard that there was a documentary on the very genre of arcade games that consumed so much of my early childhood, King of Kong and a movie that was rumored to leave you at the edge of your seat (which seemed like an impossibility to me), I decided to fork over the cash and check it out in the movie theatre last night. And it was well worth every penny. The movie somehow managed to capture a story and set of subjective experiences that I think are incredibly hard to portray via film: the individual and social intimacy that comes with machine interaction. The story lines revolves around two very different characters who are masters at conquering one of the hardest video games in the world, Donkey Kong and are trying to clinch the world record. The tale of rivalry between them is really good in and of itself and probably without this drama, the movie would have not worked. But what I also found amazing was how the movie conveyed the persistence of the (older) game. They live on in the lives of individuals and collectives, despite the rise of a whole, new class of games that are much more popular today. I am not sure how much longer they will live on, or if the movie was also inadvertently portraying the rise and slow decline of an era that will, in another 50 years, become part of the archive of dead history. Whatever the case, if you had a loving relationship with these games (or your friends or parents did), check out the movie. It really has some incredible footage and moments and will make you want to run to that old arcade, once again.

Thanks to a patch from Mathieu Geli, live-initramfs as of version 1.99.1-1 supports a boot parameter called fetch. That means, that it is enough to have a bootloader (syslinux, grub, whatever) and a kernel with the initrd image on a medium (cdrom, usb-stick, whatever) to boot a Debian Live system directly from the Internet or the local network without needing to setup a netboot environment (as in PXE). Just type:

	live fetch=http://example.com/my_squashfs.img

at the boot prompt. Whithin the boot process of the live system, the squashfs image will be once downloaded into RAM. After that point, no network access is required anymore. This is also the reason why it was invented initially, it is an alternative to the conventional netboot (PXE with tftp for boot and shared root over a network filesystem such as cifs, nfs or smb) where permanent network access is required, not a replacement. Temporary limitations

• At the moment, the amount of RAM required to run a web booted live session is equal to the size of the squashfs image. Because squashfs compression is quite good, this is not so much of a problem on reasonably modern machines (512MB for a complete Xfce Debian desktop, 758MB for a complete GNOME or KDE Debian desktop system). Nevertheless, improvements to use a local swap partition to store the image are on the way (including caching and updating).
  
  
• Due to a still pending move of the machine where live.debian.net is hosted, there are not yet autobuild squashfs images available. Later, you can just boot with something similar as: live fetch=http://live.debian.net/webboot/etch.

A fairly common request is to be able to duplicate a Xen instance. For example you might have a DomU for the purpose of running Wordpress and want another DomU to run MediaWiki. The difference in configuration between two DomU’s for running web based services that are written in PHP and talking to a MySQL back-end is quite small, so copying the configuration is easier than a clean install. It is a commonly held opinion that a clean install should be done every time and that Kickstart on Red Hat, FAI on Debian and comparable technologies on other distributions can be used for a quick automatic install. I have not yet got FAI working correctly or got Kickstart working on Xen (it’s on my todo list - I’ll blog about it when it’s done). Regardless of whether it’s a good idea to copy a Xen DomU, there are often situations where clients demand it or when it’s impractically difficult to do a fresh install. I believe that the most sensible way to store block devices with Xen is to use LVM. It is a requirement for a Xen system that you can easily create new block devices while the machine is running and that the size of block devices can be changed with minimal effort. This rules out using Linux partitions and makes it unreasonably difficult to use LUNs on a fiber-channel SAN or partitions on a hardware RAID. LVM allows creating new block devices and changing the size of block devices with minimal effort. Another option would be to use files on a regular filesystem to store the filesystem data for Xen DomU’s, if choosing this option I recommend using XFS [1] filesystem (which delivers good performance with large filesystems and large files). If you use XFS to store the block devices for the DomU that you want to copy then you will need to halt the DomU for the duration of the copy as there is no other way of getting an atomic copy of the filesystem while it’s in use. The way of doing this would be to run the command “xm console foo ; cp /mnt/whatever/foo-root /mnt/whatever/bar-root ; xm create -c foo” where “foo” is the name of the DomU and “/mnt/whatever/foo-root” is the file that is used to store the root device for the DomU (note that multiple cp commands would be needed if there are multiple block devices). The reason for having the two xm commands on the one line is that you initially login to the DomU from the console and type halt and then the xm command will terminate when the DomU is destroyed. This means that there is no delay from the time the domain is destroyed to the time that the copy starts. If you use LVM to store the block device then things are a little easier (and you get no down-time). You simply run the command “lvcreate -s -L 300m -n foo-snap /dev/V0/foo-root” to create a snapshot with the device name /dev/V0/foo-snap which contains a snapshot the of the LV (Logical Volume) /dev/V0/foo-root. The “-L 300m” option means to use 300Meg of storage space for the snapshot - if the writes to /dev/V0/foo-root exceed 300Meg of data then your snapshot breaks. There is no harm in setting the allocated space for the snapshot to be the same as the size of the volume that you are going to copy - it merely means that more disk space is reserved and unavailable for other LVM operations. Note that V0 needs to be replaced by the name of the LVM VG (Volume Group) Once you have created the snapshot you can create a new LV with the command “lvcreate -n new-root -L X /dev/V0” where X is the size of the device (must be at least as big as the device you are copying) and then copy the data across with a command similar to “dd if=/dev/V0/foo-snap of=/dev/V0/new-root bs=1024k“. After the copy is finished you must remove the snapshot with the command “lvremove /dev/V0/foo-snap” (please be very careful when running this command - you really don’t want to remove an LV that has important data). Note that in normal operation lvremove will always give a prompt “Do you really want to remove active logical volume“. If you made the new device bigger then you must perform the operations that are appropriate for your filesystem to extend it’s size to use the new space. There is no need to copy a swap device, it’s easier to just create a new device and run mkswap on it. After copying the data you will need to create the new Xen config (by copying /etc/xen/foo to the new name). Make sure that you edit the Xen config file to use the correct block devices and if you are specifying the MAC address [2] by a “vif” line in the config file make sure that you change them to unique addresses for your LAN segment (reference [2] has information on how to select addresses). Now you must mount the filesystem temporarily to change the IP address (you really don’t want two DomU’s with the same IP address). If your Dom0 has untrusted users or services that are accessed by untrusted users (IE any Internet facing service) then you want to mount the filesystem in question with the options nosuid and nodev so that if the DomU has been cracked it won’t allow cracking of the Dom0. After changing the configuration files to change the IP address(es) of the DomU you can then umount the filesystem and start it with the xm create command. If instead of creating the clone DomU on the same Dom0 you want to put it on a different system you can copy the block devices to files on a regular filesystem on removable media (EG an IDE disk with USB attachment). When copying the block devices you also need to copy the Xen configuration and edit it to reflect the new paths to block devices for the data once it’s copied to the new server, but you won’t necessarily need to change the MAC address if you are copying it to a different LAN segment. References:

1. http://en.wikipedia.org/wiki/XFS


2. http://en.wikipedia.org/wiki/MAC_address

Share This

Some people at DebConf 7 were (like Mario) wondering how many users Debian has. They realised that while we have a distributed mirror network we do have all installed systems use security.debian.org by default and we do have control over that. Since apt will only download files when they have been changed we can make a minimal estimate by looking at the number of downloads of the release files while they remain unchanged. Obviously, this will only be a minimum - for example, some users won’t have network access, some users will use caching proxies or local mirrors - but it’s a start. Unfortunately I can’t actually remember the numbers they came up with but perhaps someone could enlighten us.

A significant problem with the old-fashioned media is that as a general rule they don’t cite references for anything. Some of the better TV documentaries and non-fiction books cite references, but this is the exception not the norm. Often documentaries only cite references in DVD extras which are good for the people who like the documentary enough to buy it but not for people who want to rebut it (few people will pay for a resource if they doubt the truth and accuracy of it’s claims). I can understand newspapers not wanting to publish much in the way of background information in the paper version as every extra line of text in an article is a line of advertising that they can’t sell. So they have financial pressure to produce less content, and the number of people like me who want to check the facts and figures used in articles is probably a small portion of the readership. Another issue with newspapers is that they are often considered as primary authoritative sources (by themselves and by the readers). It is often the case that journalists will interview people who have first-hand knowledge of an issue and the resulting article will be authoritative and a primary source in which case all they need to do is to note that they interviewed the subject. However the majority of articles published will be sourced from elsewhere (news agencies [ http://en.wikipedia.org/wiki/News_agency ] such as Reuters are commonly used). Also articles will often be written based on press releases - it is very interesting to read press releases and see how little work is done by some media outlets to convert them to articles, through a well written press release a corporation or interest group can almost write it’s own articles for publication in the old media. One way of partially addressing the problem of citing references in old media would be to create a web site of references, then every article could have a URL that is a permanent link to the references and calculations to support the claims and numbers used. Such a URL could be produced by any blogging software, and a blog would be an ideal way of doing this. For bloggers however it’s much easier to cite references and readers have much higher expectations of links to other sites to support claims and of mathematical calculations shown to indicate how numbers are determined. But there is still room for improvement. Here are some of the most common mistakes that I see in posts by people who are trying to do the right thing:

1. Indirect links. When you refer to a site you want to refer to it directly. In email (which is generally considered a transient medium) a service such as TinyURL [ www.TinyURL.com ] can be used to create short URLs to refer to pages that have long URLs. This is really good for email as there are occasions when people will want to write the address down and type it in to another computer. For blogging you should assume that your reader has access to browse the web (which is the case most of the time). Another possibility is to have the textual description of a link include a reference to the TinyURL service but to have the HREF refer to the real address. Any service on the net may potentially go away at some future time. Any service on the net may have transient outages, and any reader of your blog may have routing problems that make parts of the net unavailable to them. If accessing a reference requires using TinyURL (or a similar service) as well as the target site then there are two potential things that might break and prevent your readers from accessing it.
   One situation where indirect links are acceptable is for the printed version. So you could have a link in the HTML code for readers to click on to get to the reference page directly and a TinuURL link for people who have a printed version and need to type it in.
   Also when linking to a blog it’s worth considering the fact that a track-back won’t work via TinyURL and track-backs may help you get more readers…


2. Links that expire. For example never say “there’s a good article on the front page of X” (where X is a blog or news site). Instead say “here’s a link to a good article which happens to be on the front page now” so that someone who reads your post in a couple of years time can see the article that you reference.
   Another problem is links to transient data. For example if you want to comment on the features of a 2007 model car you should try to avoid linking to the car manufacturer page, next year they will release a new car and delete the old data from their site.
   A potential problem related to this is the Google cache pages which translate PDF to HTML and high-light relevant terms and can make it much easier to extract certain information from web pages. It can provide value to readers to use such links but AFAIK there is no guarantee that they will remain forever. I suggest that if you use them you should also provide the authoritative link so that if the Google link breaks at some future time then the reader will still be able to access the data.


3. Not giving the URLs of links in human readable form. Print-outs of blog pages will lose links and blog reading by email will also generally lose links (although it would be possible to preserve them). This counts for a small part of your readership but there’s no reason not to support their needs by also including links as text (either in the body or at the end of the post). I suggest including the URL in brackets, the most important thing is that no non-URL text touch the ends of the URL (don’t have it in quotes and have the brackets spaced from it). Email clients can generally launch a web browser if the URL is clear. Note that prior to writing this post I have done badly in this regard, while thinking about the best advice for others I realised that my own blogging needed some improvement.
   I am not certain that the practice I am testing in this post of citing URLs inline will work. Let me know what you think via comments, I may change to numbering the citations and providing a list of links in the footer.


4. Non-specific links. For example saying “Russell Coker wrote a good post about the SE Linux” and referring to my main blog URL is not very helpful to your readers as I have written many posts on that topic and plan to write many more (and there is a chance that some of my future posts on that topic may not meet your criteria of being “good”). Saying “here is a link to a good post by Russell Coker, his main blog URL is here” is more useful, it gives both the specific link (indicating which post you were referring to) and the general information (for people who aren’t able to find it themselves, for the case of deleted/renamed posts, and for Google). The ideal form would be “<A HREF=”http://etbe.coker.com.au/whatever”>here is a link to a good post by Russell Coker [ http://etbe.coker.com.au/whatever ]</A>, his main blog URL is <A HREF=”http://etbe.coker.com.au/”> [ http://etbe.coker.com.au ]</A>” (note that this is an example of HTML code as a guide for people who are writing their own HTML, people who use so-called WYSIWYG editors will need to do something different).


5. Links that are likely to expire. As a rule of thumb if a link is not human readable then the chance of it remaining long-term is low. Companies with content management systems are notorious for breaking links.


6. Referencing data that you can’t find. If you use data sourced from a web site and the site owner takes it down then you may be left with no evidence to support your assertions. If data is likely to be removed then you should keep a private copy off-line (online might be an infringement of copyright) for future reference. It won’t let you publish the original data but will at least let you discuss it with readers.


7. Referencing non-public data. The Open Access movement [ http://en.wikipedia.org/wiki/Open_access ] aims to make scholarly material free for unrestricted access. If you cite papers that are not open access then you deny your readers the ability to verify your claims and also encourage the companies that deny access to research papers.
   An insidious problem is with web sites such as the New York Times [ www.nytimes.com ] which need a login and store cookies. As I have logged in to their site at some time in the past I get immediate access to all their articles. But if I reference them in a blog post many readers will be forced to register (some readers will object to this). With the NYT this isn’t such a problem as it’s free to register so anyone who is really interested can do so (with a fake name if they wish). But I still have to keep thinking about the readers for such sites.
   I should probably preview my blog posts from a different account without such cookies.


8. Failing to provide calculations. My current procedure is to include the maths in my post, for example if you have a 32bit data type used to store a number of milliseconds then it can store 2^32/1000 seconds which is 2^32/1000/60/60/24 = 49.7 days, in this example you can determine with little guessing what each of the numbers represent. For more complex calculations an appendix could be used. A common feature of blogs is the ability to have a partial post sent to the RSS feed and the user has the ability to determine where the post gets cut. So you could cut the post before the calculations, the people who want to see them will find it’s only one click away, and the people who are happy to trust you will have a shorter post.


9. Linking with little reason. Having a random word appear highlighted with an underline in a blog post is often not very helpful for a reader. It sometimes works for Wikipedia links where you expect that most readers will know what the word means but you want to link to a reference for the few who don’t (my link for the word Wikipedia is an example). In the case where most readers are expected to know what you are referring to then citing the link fully (with a description of the link and a human-readable form for an email client) is overkill and reduces the readability of the text.
   The blogging style of “see here and here for examples” does not work via email and does not explain why a reader should visit the sites. If you want to include random links in a post then having a section at the footer of related links would probably be best.


10. Linking to a URL as received. Many bloggers paste URLs from Google, email, and RSS feeds into their blog posts. This is a bad idea because it might miss redirection to a different site. If a Google search or an email gives you a URL that is about to go away then it might redirect to a different site. In that case citing the new URL instead of the old one is a service to your readers and will decrease the number of dead-links in your blog over the long-term. Also using services such as www.feedburner.com may cause redirects that you want to avoid when citing a blog post, see my previous post about Feedburner [ http://etbe.coker.com.au/2007/08/20/feedburner-item-link-clicks/ ].

Here are some less common problems in citing posts:

1. Inappropriately citing yourself. Obviously if there is a topic that you frequently blog about then there will be benefit to linking to old posts instead of covering all the background material, and as long as you don’t go overboard there should not be any problems (links to your own blog are assumed to have the same author so there is no need for a disclaimer). If you write authoritative content on a topic that is published elsewhere then you will probably want to blog about it (and your readers will be interested). But you must mention your involvement to avoid giving the impression that you are trying to mislead anyone. This is particularly important if you are part of a group that prepares a document, your name may not end up on the list of authors but you have a duty to your readers to declare this.
   Any document that you helped prepare can not be used by itself as a support of claims that you make in a blog post. You can certainly say “I have previously demonstrated how to solve this problem, see the following reference”. But links with comments such as “here is an example of why X is true” are generally interpreted to be partly to demonstrate the popular support for an idea.


2. Citing secret data. The argument “if you knew what I know then you would agree with me” usually won’t be accepted well. There are of course various levels of secrecy that are appropriate. For example offering career advice without providing details of how much money you have earned (evidence of one aspect of career success) is acceptable as the readers understand the desire for some degree of financial secrecy (and of course in any game a coach doesn’t need to be a good player). Arguing the case for a war based on secret data (as many bloggers did) is not acceptable (IMHO), neither is arguing the case for the use of a technology without explaining the science or maths behind it.


3. Not reading the context of a source. For example I was reading the blog of a well regarded expert in an area of computer science, and he linked to another blog to support one of his claims. I read the blog in question (more than just the post he cited) and found some content that could be considered to be racially offensive and much of the material that I read contained claims that were not adequately supported by facts or logic. I find it difficult to believe that the expert in question (for whom I have a great deal of respect) even casually inspected the site in question. In future I will pay less attention to his posts because of this. I expect a blogger to pay more attention to the quality of their links than I do as a reader of their blog.

While writing this post I realised that my own blogging can be improved in this regard. Many of my older posts don’t adequately cite references. If you believe that any of my future posts fail in this regard then please let me know.

Share This

Earlier this year I blogged about using the twitter bot for Remember the Milk. I still use it fairly well, even though the SMS messages from the RTM come late a bit. It's working for me in order to get a good chain going for the thesis proposal. There are a couple other bots that I'm using which are on twitter as well.

The first bot that I use is the bot at timer. The timer bot will let you set a timer (surprisingly). You only need to send it a direct message with the amount of minutes as well as a reminder for you about what it was set for. For example, if you want to remember to plug the meter before it runs out in 40 minutes, just twitter 'd timer 38 plug the meter' to timer and in 38 minutes it will inform you that your timer is up. You can have multiple timers running at once, which is useful if you are perhaps having tea at a coffee shop while you are parked on the street.

Another bot I use is the gcal. Gcal is a connection to your google calendar through twitter. You can set up new appointments by messaging the gcal bot with something like 'd gcal Pick up joe at 7pm' or 'd gcal set up web application for judy tomorrow' and it will
add it to your calendar. I use it in the morning to track my weight. I just pull out my phone while I am at the scale and SMS 'd gcal my weight is XXX', and it adds the event as a day event on the day it was sent.

The last bot I will go over today is the mymm bot. It is slightly more complicated but still useful. When you're at the pump, you can direct message mymm a little info and get back the MPG that you have done since the last fillup. It helps me keep a tab on how good or bad I'm driving, which is a good thing. The car we drive usually gets about 24 MPG, but I learned through this that when I'm carrying a bunch of crap in the back, the MPG drops 3-4, so it's better for me to have an empty trunk.

I'm sure there are a bunch of other useful bots out there (ququoo is one, which I may cover in another post), but these are the most useful that I've found so far. Twitter is turning out to be much more useful than just random status updates.

Anyone who has done an operating systems 101 course is familiar with the standard hierarchical page-table structure. This is essentially a radix-tree where bits of the virtual address are taken as indexes into an array of pointers, each pointing to the next level down or, at the leaves, to a physical entry. One potential issue with this style of page-table is with a very large, very sparse address space (such as in a single address-space operating system) there will be many intermediate steps with possibly only a single translation at the end (i.e. lots of single pages scattered all around your address-space). Thus we can add a bit-field guard to each of the entries in the page table (Liedtke, 1994) to help indicate only those paths through the tree which are valid (similar to a Patricia tree). Translation first proceeds as usual; the top bits are used as an index into the top level table. However, each entry now contains extra information about further valid addresses. The guard is masked against the next bits in the virtual address; if it matches (e.g. in our example is 0x2) the next step can proceed directly to the leaf node, bypassing the middle look-up. The entry can be marked as either final (i.e. pointing to a leaf node) or intermediate, in which case the process continues with the next level (which also has its own guard, and so forth). Any entry that does not match the guard is known to be false, and therefore a fault can be raised. To examine this further, consider that we know that each page in our 24-bit address space above is 4KiB, thanks to the 2¹² = 4KiB offset. Therefore each of the 16 possible values of the top four bits selects 1MiB of the 2²⁴ = 16MiB address space; e.g. 0x0 selects the first megabyte, 0x1 the second and so forth. The second 4 bits selects a 64KiB region of this 1MiB, and the last 4 bits selects a 4KiB region of this 64KiB region. This scheme could work quite well if we were using only 16 pages each separated by 1MiB. Each of these pages would only require an entry in the top level with an appropriate guard which could then be marked as final and point directly to a physical frame. Once things become less sparse however, we need to start filling in middle levels. If you have 2 pages within a 16MiB region, you need at least one extra bit of address to tell them apart. If you have 3 or 4 pages, you need at least 2 bits to tell them apart, and so forth. It would therefore be beneficial to have variable level sizes to dynamically adapt and create the most efficient lookup possible given the address space layout of a process. For example, if you knew your process was only going map pages 0x123 (100100011) and 0x124 (100100100) a good option would be to make your top-level only two entries (i.e. check the top bit) with a guard on the entry at index 0x1 of 00100. You then need a second level with 8 pointers to differentiate the remaining 3 bits (each of these would be final and hence point directly to a physical page). The more pages you start mapping, the less bits you can discard with the guard. If you follow this through to a guard of 0 bits you end up with a linear page-table. It has been shown (Elphinstone, 1999) that fixed level sizes with 16 entries tends to work well. For a 32-bit address-space with 4KiB pages (e.g. 12 bits offset) this leaves 20 bits to be mapped by the page-tables; with each level mapping 2⁴ bits this means a 5 level tree. The guard values provide enough path-compression that the table is not constantly walked. The disadvantage is that it may "over-allocate" meaning more space is dedicated to page-tables than strictly required. Deciding where and when to split levels, what size indexes to put at each level and updating all the guard values dynamically to make the smallest, most efficient page-table would create a variable-radix page-table (Szmajda and Heiser, 2003). In summary, a guarded page-table is more useful the more sparse your address space is. A variable-radix guarded page-table is complex, but could offer advantages for implementing variable page-size support and thus having positive effects on TLB coverage.

I previously posted about Interesting Ideas from George Monbiot, one of which was to establish individual emissions trading. Gyros Geier disagrees with this and cites the current emission trading schemes as evidence. There are several fundamental differences between George’s idea and the current implementations of emission trading. The biggest flaw in current emission trading schemes is that the emission credits are assigned to the worst polluters. George is proposing that an equal amount be assigned to all citizens. Assigning credits to the worst polluters is another form of by the polluting industries. The way to solve these problems through emission trading is to start by fairly assigning the credits (and what better way than to equally distribute them among all citizens) and to then reduce the amounts assigned over time. Gyros claims that emission trading which allows people who use little emissions to get large credits will cause people to have resources used in their name which they would not otherwise use. The solution to this is to assign to each citizen in a country a set of credits that is equal to the use by someone on the median income. Note specifically that setting credits equal to average use is not the right thing to do, the vast majority of the population produce significantly less emissions than average. The result of such a policy would be that people who produce median emissions (and most of whom would be close to the median income) would reduce their emissions as much as possible so that they could sell the credits, they would even have an incentive to spend money to reduce their emissions (for example by installing better insulation in their home) as it would be an investment. Then people who produce more emissions than the median would be forced to buy credits to support their extravagant lifestyle. This would give a significant reduction in emissions (the median income is about half the average income and I presume that the emissions produced are in line with income). Gyros also makes the startling claim that emissions trading increases emissions. I can’t imagine that being possible, in fact I can’t imagine how the coal industry could do more damage to the environment if they tried. Finally, taking a positive approach to blogging is a really good idea. I welcome discussion with people who want to claim that my ideas (and the ideas that I quote) are bad, but if you are going to do this please describe something that you consider to be better.

Share This

In the last week, I have been witness to and part of many conversations and probably one of my favorite ones was about coffee. My friend reasoned that coffee is as wonderful as it is because of its dependability (unlike, for example, your relatives). You know that for a moderate sum of money, you can drink a drink that makes you happy, alert, and, for some of us, allows us to face the rest of the day on an even keel. It is pure comfort that derives from a form of almost ritualistic dependability. This morning, as I was sipping my coffee, I came across a short blog post by Stanley Fish who certainly does not make me as happy as my morning cup of joe, but I do admire him for his dependability and consistency when it comes to reporting on matters of liberalism. For over 20 years he has dependably written on the quandaries and limit of liberal political ideology and his most recent installment, which focuses primarily on a new book by Paul Starr, is no different. Well, his conclusion strikes just a little differently than the tone of some of his previous works. In the past (or perhaps in some of his longer academic works), Fish’s solution to the problem of competing ideologies is that there are no solutions, just incommensurable ideologies and you gotta sort of duke it out, and the strong man/woman/group wins (see Terry Eagleton for this characterization of Fish’s work. But the ending to this piece is subtly different, a tone and stance I rather prefer: So again, what to do? Lilla s answer is pragmatic rather than philosophical (and all the better for that). All we can do, he says, is cope ; that is, employ a succession of ad hoc, provisional strategies that take advantage of, and try to extend, moments of perceived mutual self-interest and practical accommodation. We need to recognize that coping is the order of the day, not defending high principles. Now there s a principle we can live with, maybe. What I like about his ending is that it acknowledges there are times when compromise is possible, where a common meeting ground can be forged, however provisional these may be. As someone interested in the politics of consensus and accommodation, I think it is important to recognize that human beings are not simply molded by one set of values but are are often dwelling within various systems (of sometimes contradictory) values. And it is because of this multiplicity that forms of accommodation and consensus emerge and can emerge, signaling a more hopeful politics that derive not from abstract adherence to precept such as tolerance, but from the far messier realm of actual life experience.

On cleaning and distcleaning ... ... and CDBS and "FTBFS if built twice in a row" Yesterday I closed two bugs of mine (424331 and 424392) of the "FTBFS if built twice in a row" class. Shame on me, they were open since too long. Both packages use autotools and, on the Debian side, CDBS. For both packages the fix was as simple as adding:

DEB_MAKE_CLEAN_TARGET = clean

to debian/rules. The rationale for that is rather obvious (now that I've thought about it of course). CDBS per default uses "distclean" as the upstream target which is invoked when "debian/rules clean" is executed. Unfortunately, "distcheck" (used by upstream to create the distribution tarball) does not check that "distclean" leave the distribution tarball as it was before the build. So nothing grants a priory that stuff removed by "distclean" is rebuildable with just the build-deps. A lot of packages come with generated sources (for example by XSLT or by some template toolkits) included in the source package, if that stuff is removed by mistake during distclean you'll see yet another instance of this kind of bugs. I bet a lot of other "FTBFS if built twice in a row" bugs out there have been fixed this way or can still be fixed with the addition of the line above. Update: changed this post, the first writing was made thinking at "maintainer-clean" instead of "distclean"

The Belgian Navy recently acquired a new frigate, baptized the Leopold I; and it's leaving soon on a training trip. The TV news just had an interview with the captain of this vessel, who declared that the function of a frigate is to protect "other vessels"; one of the examples he gave was "a carrier vessel". What he neglected to mention is that the Belgian Navy, in all her glory, doesn't have any carriers. In fact, it doesn't have any ships that are larger than frigates. But hey, at least this frigate has a helicopter pad, so maybe it qualifies. Then again, maybe not.

Fighting Spam Lots of people get spam and don't want to fight it themselves. Many of those people elect to use a service such as Google Mail, which does an adequate job of filtering most of the time. For people who wish to host their own mailservers, and off-load their anti-spam support to another entity, there are big companies such as MessageLabs. But if you can't/won't pay for somebody else how would you handle it? Right now I run my own mailserver and mostly it rocks. However I also manage a couple of servers for other people, and this mostly means that I have identical configuration(s) scattered across other machines. (Or as I recently discovered "mostly identical" setups scattered across N hosts.) After viewing a recent thread upon the forums of my employers I'm wondering how hard it would be to setup an anti-spam mail host. The idea being that users of domain example.org would continue to run their own mailserver (+optional POP/IMAP server). However this server wouldn't be listed in an MX record. Instead the MX record would point at the antispam server, which would then process and forward the mail on. (I'm assuming that the mailserver at example.org would be firewalled such that it would only accept SMTP connections from the anti-spam host). The only challenges I see would be:

• The anti-spam server knowing where to forward mail for domain X - since there would be no MX record. Maybe a A record 'incoming.example.org'; maybe a hardwired list.
• Having enough resources on the anti-spam server.
• The pain that would ensue if legitimate mail were dropped.

The last one is the killer point which makes me wonder if I'm wasting my time on the general case; I don't believe I could persuade people to pay for such a service (on the grounds that I wouldn't!) and if ever a mail were dropped I'd be the first person who would be blamed. And probably rightly so. The simple solution would be to always accept mail - but just tag it. Then you still have the same issue, but you have the excuse that "I still delivered your mail; just tagged wrongly". The downside of that is that the recipient still has to filter the mail; and still eats the bandwidth cost. The main gain is the lack of spam-processing CPU that is eaten locally. Anti-spam, and to a lesser extent anti-virus, topics are very interesting to me, and I'd love to hear any thoughts - or even what you use for yourself. Right now I'm 50/50 on setting up my main mailserver as the proxy for a few domains I have to manage (family, friends) to see how it works, but I'm not quite convinced yet. (Single point of failure == bad. Though centralised configuration == good.)

I have just been asked for advice on whether SE Linux is Linux specific, and therefore whether code related to SE Linux should always be stored with other Linux specific code instead of being in the main branch of certain free software projects. One example of SE Linux access controls being implemented on a different OS is the work to port SE Linux to Mac OS/X. Here is a paper on the topic presented at the SE Linux Symposium 2007, and the main site is at http://sedarwin.org. One thing I have been doing is trying to get some friends interested in doing similar work for GNU Hurd (there are some similarities between Darwin and HURD so the work done on Mac OS/X “Darwin” will help the HURD effort). I believe that The HURD has the potential to offer significant security benefits due to the micro-kernel design. One significant problem area in computer security is kernel security flaws, if the kernel can be split into a set of independent processes that run with minimal privileges then the scope of such problems is dramatically decreased - and the possibility of upgrading parts of a kernel on a live machine is provided. As people such as Linus point out there is a performance overhead to micro-kernels, but most machines are idle most of the time anyway. I believe that reliability and security are more important than getting the last 10% of system performance for most machines. The success of Xen is evidence that features other than maximum performance are desired. Another example of SE Linux access controls on a non-Linux platform is the MAC framework in the TrustedBSD project. This implements SE Linux access controls on top of FreeBSD. From reading the documentation it seems that the amount of changes required to the SE Linux code base for implementation on TrustedBSD was significantly smaller than the changes required for Darwin. Sun is also apparently considering adding type-enforcement to Solaris. It’s yet to be seen whether this happens and if so whether it is compatible with SE Linux. So it seems that a significant portion of the SE Linux code base is portable, and in particular the user-space code should port well. The interfaces for and methods labelling files etc should port well between platforms. Therefore I recommend not having SE Linux code split into Linux specific trees and instead having a compile option to enable SE Linux support.

Share This

I just needed to test something so I mounted the filesystem of one of my Xen domains in the Dom0 and chroot’d into it (I didn’t need the overhead of running a DomU for a quick test). Then strangely I found that my chroot environment had no apt-get and no dpkg installed. After a small amount of thought I realised that I had accidentally mounted the filesystem for a Fedora image instead of a Debian image. It seems to me that it might be a good idea for distributions to have shell scripts replacing the package tools for other distributions. For example dpkg on a Fedora box could be a shell script that runs echo “This system is running Fedora, you want to use rpm instead of dpkg”. Such a command would have saved me a couple of minutes of thinking. This may sound trivial, but if you consider the number of people who make such mistakes (and the even larger number of people who don’t realise that there is even more than one package tool in existence) then wasting a few K of disk space on every system to help them seems like a good idea. The minimum size of new hard drives that you can purchase seems to be 36G nowadays. So it’s not as if this would really cost anything. Please leave comments to tell me if I’m being sensible or silly. If the idea is regarded as good I’ll start filing bug reports.

Share This

People usually uses Google, among other things, to look for hints about how to do small tasks. Hopefully, you will find the hint you were looking for in somebody’s blog, because he or she had the same problem time ago and decided to talk about it. Last night, I was a bit bored and started writing a dumb set of files to introduce people how static and dynamic linking work, so trying to put my two cents and add a new hint to the wild wild web, I’m sharing it. The C source files are practically useless, therefore pay attention to the Makefile, the magic is in there. If you’re downloading it only to compile, execute and see what happens forget about it, the program is completely silly. Otherwise, probably the Makefile is buggy somewhere so bug reports are welcome. For teaching purposes, try, for example, breaking the ABI and see what happens. Google, index this post please! Update: This example does not cover autotools/libtool

GPL and Business FAQ I sometimes happen to be in a dialogue like this:

X: According to GPL, you can sell copies of your software at any price you want, right? Me: Right. X: But your customers can in turn sell copies of your software at any price they want, right? Me: Right. X: So, the only way a GPL developer can make money is by charge for development of their software? Me: Not only for development: also for maintenance, improvements, customisation, installation, training, gadgets, support...

The market is quite big, actually. In fact, one can even make business by selling the software in a box, if one makes a box that people would like to have (for example, including things like useful documentation, a few gadgets, a support contract and a pony). The idea is that the business of selling software alone is a useless extortion that provides no particular benefit to the user (and in many cases, no particular benefits to the developer either). If you put that aside, there is a plethora of ways to charge the customers more, and for things and services that are useful to them and make them happy. Actually, if you develop a GPL software for someone who is then happy about it, they may then pass it on to other people in the same sector. Who then maybe need help with installation, support, maintenance, custom features, training, gadgets and ponies. If your software hits the spot, a niche market can end up growing by itself. The last discussion on this topic then ended up with another question:

X: Ok. But to do a good job you have to be many people at once: for example you sometimes need to be a sysadmin as well as a developer. Me: Yes. Being many people at once is what starting a company is all about.

He complained that answers like these could not be found in the Selling Free Software article of FSF: in the meantime I'll publish them here. The license of this post is GPL. The FSF and any of their regional chapters are welcome to use all or part of this text, with or without modification as they see fit, alone or included in documents that use any license they see fit, even some of the not really free ones they use sometimes, such as the one used by Selling Free Software itself :-P

Recently, Josh Berkus blogged about The Myth of the Benevolent Dictator. In it he complained about people who try to posit that a benevolent dictator was "an unalloyed asset" to their project, and that there were many successful and valid forms project governance other than the singleton project leader. Furthermore, he argues that the best way is to form a community, and then ends by suggesting that people who say "dictatorship good" is really saying "democracy bad", and perhaps the motives of the person using that argument should be questioned.

Given that I was one of the first people (and probably the first person) to use that phrase to describe Linus's governance model, obviously I feet the need to respond to his thoughts on the matter.

First of all, the term "benevolent dictator" comes from an old saying, "the best form of government is the benevolent dictator --- there's only one problem, finding the benevolent dictator"[1]. And in my presentations where I explained the Linux project management model, I was always careful to say that there were other models that were used by other projects, and that it was interesting to study why certain management models worked for some projects but not others.

So by definition, if a project is successful with a single project leader ultimately responsible for making decisions to settle conflicts, then that leader is a "benevolent dictator"; if they aren't successful, then that would be an example of a dictator who wasn't able to be "benevolent", eh? :-)

In all seriousness, though, I agree with Josh when he says that the key part of an open source project being successful is gathering together a healthy community. But let's unpack that a little. In order for a community to be healthy, there needs to be general consensus on the values and cultural norms for that community. So for example, in the case of Debian, which Josh holds up as a successful community, not everyone has the ability to vote; you have to pass a fairly stringent process which tests your technical skills and your willingness to subscribe to Debian's core values as embedded to the Debian Social Contract. People who do not pass the New Membership process do not have the right to vote in Debian, even if they are the most rabid, fanatical users of Debian.

This brings up a key point about governance analogies; they all don't map cleanly into the real world for two closely related and extremely important reasons: (1) No one has a fundamental human right to be a member in some OSS project, and (2) No one is forced to stay in a particular OSS project. These are really two sides of the same coin, and what it means is that, as in Debian, "democracy" does not mean that everyone is entitled to political franchise, or the right to vote, and in a project which has a "benevolent dictator", the powers of the dictator is fundamentally limited by the simple fact that if he is too unreasonable, people are always free to leave or fork the project. These differences are extremely important when trying to reason about "democracy" or "dictatorship" as being good or bad as applied to open source projects.

Democracy is rarely the best way for an open source project to make decisions. For technical issues, you want the right answer, and majority vote isn't necessarily way to resolve a technical issue. As far as policy decisions, if the decision can not be made by consensus, then some percentage of the community will be unhappy with how the decision, and that can cause a huge amount of arguments and dissension. If the community has very clearly expressed values, then the number of times where such votes need to be made will be minimal.

For example, Debian has an explicit declaration of its values and mission in its Debian Social Contract; people who don't agree to the Social Contract aren't allowed to become Debian Developers and thus are not allowed to vote. But, one of the more controversial disputes that have threatened to tear apart Debian is the question of the non-free packages, and that's fundamentally because of an ambiguity in the Social Contract. On the one hand, Debian has declared that it is fundamentally 100% about Free Software, to the extent that it has adopted Richard Stallman's rather petulant request that the Linux operating system be called GNU/Linux (which admittedly is better than Lignux.) On the other hand, the Social Contract also explicitly specifies the existent of the non-free section in the distribution. Over time, this ambiguity have been the source of a lot of highly passionate debate and controversial votes that have satisfied neither the FSF fanatics nor the more Linux-like moderate position tolerating the use of proprietary as well as open source software. I submit that if Debian had either explicitly stated an FSF-centric position, and only accepted members that supported point of view, or explicitly subscribed to a position that users should be able to use whatever software they feel meets their needs, as Ubuntu has, it would avoided many arguments that have threatened to tear apart Debian.

More recently the fact that the Social Contract has been silent about whether or not Debian Developers should be paid or purely be 100% volunteer. This was never an issue in the Linux Kernel community, but for some reason it has been a highly divisive and disruptive issue inside Debian. Democratic votes have not really resolved the conflicts, and in fact some might claim that these votes have if anything hardened various Debian Developers' positions.

Going back to Josh's final point, "Next time you hear the words 'benevolent dictator' you probably want to think about the motives of the person saying it. Because the corollary to 'dictator good' is 'democracy bad', and is that a place you really want to go?", I am quite proud to stand by my convictions that for Open Source Projects, pure democracy is a bad idea. If a projects wants to be technically successful, it needs to be a meritocracy, and chose the best technical decisions; Linus does not decide by abiding with majority vote whether or not a patch should be accepted, and that's as it should be.

Even Debian's "raw democracy" doesn't let any random luser get to vote on who should be the Debian Project Leader. Decisions need to be made by people who are competent to make a valid choice. Even in real life, one could argue that the choice made by Americans, over of which 70% believed that Saddam Hussein was "very likely" or "somewhat likely" to be personally involved with the 9/11 attacks even though there has been absolutely no evidence to support this "gut feeling", made a rather disastrous choice in voting to re-elect George Bush to the presidency. Democracy without sufficient education, literacy, and intellectual curiosity to learn about what is happening in the world can result in truly disastrous choices being made. So certainly, democracy is not a panacea. In the real world, it may be the best choice of many bad choices, given how much people have suffered under military dictatorships. However, in the Open Source world, things are different because people are always free to leave one project and affiliate with another --- not an alternative most people living in dictatorships have historically had. Given these key differences, I think it is clear that for open source projects, democracy is indeed often a bad idea, and that if you if a project has a single person with a strong technical vision and good interpersonal skills, the benevolent dictator model can indeed be a very viable and successful.

Does that mean that other forms of governance can't work? Of course not! The rotating dictatorship used by Perl[2], the core-team-with-strongarm leader used by OpenBSD, the corporate dominated scheme of MySQL, the self-selecting core team model of Samba, all are successful models. I just have to strongly disagree with Josh's claim the single strong leader approach is somehow inferior to "democracy" as if democracy was somehow a magic bullet.


[1] For am early example of this quote, nine years ago, please see this message, where I'm pretty sure I authored the colon prefixed quote, and I know I was using the term "benevolent dictator" much earlier than 1998.

[2] In the early days of Perl, it also used the benevolent dictator model. Larry Wall is a great exemplar of how someone who has a strong vision of what he wanted to accomplish, and was humble and easy to work with, both on-line and in person, can make all the difference in the world for an up-and-coming open source project.