I have ordered my new MacBook today. The model silently got an update yesterday and that was what I was waiting for. While my “old” PowerBook (4 years old) is still working fine, it has a broken hinge and unexplainable display-startup problems. Combined with the fact that it always connected to all my peripheral devices, external drives and a second screen, it is less then ideal to just pick it up and go. (Although we have XRandr 1.2 now, the whole desktop environment is not integrated with these features yet). Moreover, because I somehow ended up in a 5 year cycle of needing a computer system update and something portable at the same time, I always bought a heavy laptop. This time, I want to break this cycle, and buy a light, portable laptop and maybe later some workstation. And why Apple or a MacBook… well, I have good experiences with it, well-designed and thought-through, and if you look for portable laptops, 1000 , with a 12-14” screen, quality hardware and good specs, the MacBook is on of the few options that is left, actually.

Dearest Lazyweb, I wouldn't mind something that put in-your-face announcements on the desktops of our users. Something exactly like sysnews would be ideal. If it were GNOME-centric, it'd pop up some sort of notification thingy down on the bottom, right-hand corner, on top of everything and require explicit dismissal, but in an ideal world, it'd be desktop environment agnostic. Anything like that already out there, or do I have to put my coding pants on? Love, and sloppy kisses Andrew Update libnotify-bin has a very promising looking notify-send -t 0. That might address the GNOME and KDE users, but I'm not sure how say FVWM2 users would fare.

Other Unix systems apparently calculate the load average differently to Linux. According to the Wikipedia page about Load(computing) [1] most Unix systems calculate it based on the average number of processes that are using a CPU or available for scheduling on a CPU while Linux also includes the count of processes that are blocked on disk IO (uninterruptible sleep). There are three load average numbers, the first is for the past minute, the second is for the past 5 minutes, and the third is for the past 15 minutes. In most cases you will only be interested in the first number. What is a good load average depends on the hardware. For a system with a single CPU core a load average of 1 or greater from CPU use will indicate that some processes may perform badly due to lack of CPU time - although a long-running background process with a high “nice” value can increase the load average without interfering with system performance in most cases. As a general rule if you want snappy performance then the load average component from CPU use should be less than the number of CPU cores (not hyper-threads). For example a system with two dual-core CPUs can be expected to perform really well with a load average of 3.5 from CPU use but might perform badly with a load average of 5. The component of the load average that is due to disk IO is much more difficult to interpret in a sensible manner. A common situation is to have the load average increased by a NFS server with a network problem. A user accesses a file on the NFS server and gets no response (thus giving a load average of 1), they then open another session and use “ls” to inspect the state of the file - ls is blocked and gives a system load average of 2. A single user may launch 5 or more processes before they realise that they are not going to succeed. If there are 20 active users on a multi-user system then a load average of 100 from a single NFS server that has a network problem is not uncommon. While this is happening the system will perform very well for all tasks that don’t involve the NFS server, the processes that are blocked on disk IO can be paged out so they don’t use any RAM or CPU time. For regular disk IO you can have load average incremented by 1 for each non-RAID disk without any significant performance problems. For example if you have two users who each have a separate disk for their home directory (not uncommon with certain systems where performance is required and cooperation between users is low) then each could have a single process performing disk IO at maximum speed with no performance problems for the entire system. A system which has four CPU cores and two hard drives used for separate tasks could have a load average slightly below 6 and the performance for all operations would be quite good if there were four processes performing CPU intensive tasks and two processes doing disk intensive tasks on different disks. The same system with six CPU intensive programs would under-perform (each process would on average get 2/3 of a CPU), and if it had six disk intensive tasks that all use the same disk then performance would be terrible (especially if one of the six was an interactive task). The fact that a single load average number can either mean that the system is busy but performing well, under a bit of load, or totally overloaded means that the load average number is of limited utility in diagnosing performance problems. It is useful as a quick measure, if your server usually has a load average of 0.5 and it suddenly gets a load average of 10 then you know that something is wrong. Then the typical procedure for diagnosing it starts with either running “ps aux grep D” (to get a list of D state processes - processes that are blocked on disk IO) or running top to see the percentages of CPU time idle and in IO-wait states. Cpu(s): 15.0%us, 35.1%sy,  0.0%ni, 49.9%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
7331 rjc      25  0  2868  640  312 R  100  0.0  0:21.57 gzip Above is a section of the output of top showing a system running gzip -9 < /dev/urandom > /dev/null. Gzip is using one CPU core (100% CPU means 100% of one core - a multi-threaded program can use more than one core and therefore more than 100% CPU) and the overall system statistics indicate 49.9% idle (the other core is almost entirely idle). Cpu(s):  1.3%us,  3.2%sy,  0.0%ni, 50.7%id, 44.4%wa,  0.0%hi,  0.3%si,  0.0%st   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
7425 rjc      17  0  4036  872  588 R    4  0.1  0:00.20 find Above is a section of the output of top showing the same system running find /. The system is registering 44% IO wait and 50.7% idle. The IO wait is the percentage of time that CPU core is waiting on IO, so 44% of the total system CPU time (or 88% of one CPU core) is idle while the system is waiting for disk IO to complete. A common mistake is to think that if the IO was faster then more CPU time would be used, in this case with the find program using 4% of one CPU core if all the IO was instantaneous (EG in cache) then the command would complete 25 times faster with 100% CPU use. But if the disk IO performance was doubled (a realistic possibility given that the system has a pair of cheap SATA disks in a RAID-1) then find would probably use 8% of CPU time. Really the only use for load average is for getting an instant feel for whether there are any performance problems related to CPU use or disk IO. If you know what the normal number is then a significant change will stand out. Dr. Neil Gunther has written some interesting documentation on the topic [2], which goes into more technical detail including kernel algorithms used for calculating the load average. My aim in this post is to educate Unix operators as to the basics of the load average. His book The Practical Performance Analyst gives some useful insights into the field. One thing I learned from his book is the basics of queueing theory. One important aspect of this is that as the rate at which work arrives approaches the rate at which work can be done the queue length starts to increase exponentially, and if work keeps arriving at the same rate when the queue is full and the system can’t perform the work fast enough the queue will grow without end. This means that as the load average approaches the theoretical maximum the probability of the system dramatically increasing it’s load average increases. A machine that’s bottlenecked on disk IO for a task where there is a huge number of independent clients (such as a large web server) may have it’s load average jump from 3 to 100 in a matter of one minute. Of course this won’t mean that you actually need to be able to serve 30 times the normal load, merely slightly more than the normal load to keep the queues short. I recommend reading the book, he explains it much better than I do.

• [1] http://en.wikipedia.org/wiki/Load_(computing)


• [2] http://www.teamquest.com/resources/gunther/display/5/index.htm

Share This

I'd been using Emacs for quite awhile, and about 8 months ago I decided I would try using Vim. I'd only used vi for system emergency work, but knew a number of people that swore by it for regular work. So I decided I would learn Vim and use it for my regular work. I figure that with things like this, I don't get a real feel for how well they work unless I use them for all my work. So I haven't really opened Emacs at all in the past 8 months.

Yesterday I finally decided that Vim was not living up to my expectations and I'm in the process of switching back to Emacs. I thought I ought to write down why I'm doing that, for my own future reference... and since nobody has ever written about Emacs vs. Vim, I might as well post it where everyone can see it.

So here we are.

Original Reasons for Using Vim

It would lead to more comfortable typing. Lots of Vim users mention that you don't have to hold down keys while hitting other keys as much in Vim as in Emacs, and that the movement keys are all on the home row. That's true, but I didn't find it to be that big of an improvement, since Esc is a farther reach than anything in Emacs, and let me tell you, you're hitting Esc all the time in Vim. I found that removing the armrests from my chair made my hands happier than Vim ever did, and swapping Ctrl and CapsLock in Emacs will probably help there too.

It starts faster. I'm not sure if that really was true even when I switched, but it certainly isn't true on any of my machines today. Both Vim and Emacs have had major version upgrades (v7 and v22, respectively) since I started using Vim. People seem to say that Emacs 22 feels faster, though I don't know if that's true. The startup times of the two, if they're different, are imperceptible.

Vim would use less RAM. Frankly, these days, both Emacs and Vim are way down on the list of things that use up RAM. Heck, kmail has 141MB resident, and each of its two IMAP processes is using more than 30MB. Emacs in X right after start has 16MB resident, 10MB of which is shared, and 25MB VSS. gvim right after start has 8MB resident, 5MB of which is shared, and a 43MB VSS. Emacs tends to use fewer processes for things that vim. So they're not all that different, and Emacs could come out smaller in certain situations. But the difference is irrelevant on today's machines, and modern Gnome and KDE apps are many times larger than both of them.

It will make me more comfortable in rescue environments where I have only traditional vi available. Actually, the vi on AIX is so different from modern Vim that this didn't really help.

It would make me more productive. There are some editing commands that did, but as you'll see below, it was more than balanced out by other problems.

Things I Liked about Vim

The commands dt, dT, df, dF. Wonderful little things those. Emacs now has M-z (Zap), which is similar to df but can actually go to other lines (a nice addition). And there are easy ways to bind keys to the others as well, though that doesn't make it a pervasive convention like it is in Vim.

Antialiased fonts. It's crazy that Emacs doesn't have this yet. But not a showstopper; I still like good ole 10x20 just fine.

Regexp search-and-replace. Emacs actually has this now, and maybe it had it back then too. M-C-%. Apparently in Emacs22 the replacement expression can also have lisp code in it, which sounds really slick but I can't see myself using it regularly.

Annoying Things in Vim

Syntax highlighting. The syntax highlighting for most languages in Vim felt like it was about as smart as it was in Emacs about 10 years ago. Strings like "Hello!\"" (in languages where \" inserts a literal ") often confused it. Sometimes quotes within comments confused it. Sometimes it would be confused permanently. Other times, just until I scrolled around in the file or reloaded it.

Indentation. This is much more annoying than the syntax highlighting, really. In many languages -- and especially the two modes I've used most recently, XML and Haskell -- it really, really stinks. The indentation there isn't aware of syntax, or not very much. Sometimes it is smart enough to know that if an XML line starts with </ that it moves left and if it starts with an opening tag, that the next line moves right. But it's not smart enough to do this reliably. Not only that, but indentation is not handled with consistent configuration between languages. And even though Vim ships with a ton of language modes, the central docs only cover indentation for C.

I've asked Vim experts about this, and have tried all sorts of various tweaks, have read through Vim indentation mode source files, etc. There is just no way to get it anywhere near the intelligence of Emacs for most languages, short of writing my own mode, it appears. This is even worse because when using the backspace key in insert mode, for awhile it deletes individual spaces, and then all of a sudden deletes a big chunk of whitespace back to the beginning of the line. (And no, the insertion of Tab characters is disabled.) Indentation is my #1 complaint about Vim, and something that shows no progress towards being fixed any time soon.

And forget about anything like Emacs M-x reindent-region. This is a syntax-aware indenter. You can write out an entire source file with no indentation whatsoever, and it will indent the entire thing according to the indentation rules you've defined and the syntax of the language you're using. The best I've seen in Vim are commands that add or remove space at the beginning of every line in a region.

In short, Emacs seems to "understand" the file format on a much deeper level than Vim, and can automate things to a much better extent because of it.

Too many things disrupt the paste buffer. I can use Y or y to yank some text in Vim, and it's really, really easy to overwrite that buffer with other things. Yes, I know that I can yank it into a named buffer, but that's inconvenient and I don't usually know in advance that I'll have that need. In Emacs, only C-k and other "large area" commands disrupt it.

Vim doesn't like you having lots of files open at once. It's surprisingly convoluted to do this. If you use the basic documented command to edit another file, :e, it closes the file you're working on. The normal way to open multiple files at once is to use split windows. Well, I don't like split windows all that well, and often just want to make a quick change in one file -- in full screen -- and then go back to another. Even though I use set hidden in my ~/.vimrc, it still is annoying and more convoluted than it should be.

Vim can't create new top-level X windows. In Emacs, I can press C-x 5 2, and poof, I have a second Emacs window in X, and it's tied to the same editing session and Emacs process. Not a new process, with a different set of files, its own buffers, etc. The same process, same set of files. Just like a split window, but with a new top-level X window instead. gvim simply has no way to do that. This is also a large annoyance.

gqap stinks. This has burned me more than once. I'll be editing an XML document, and insert some text in the middle of a paragraph. Now I have a really wide line. So I type gqap to reformat the paragraph. My cursor is near the bottom of the screen, so I don't really see much past the current line. I then save the document and exit. Later I discover that vim considered the entire rest of the document part of the single paragraph, and removed all the different indentation levels at </para> and the like, so it's completely messed up. Emacs is smart enough to know what is a paragraph in XML mode, and M-q does the right thing. Oh, and Emacs reindent-region can fix the Vim gqap-induced mess.

I made a new release of the Chronicle blog compiler the other day, which seems to be getting a suprising number of downloads from my apt repository. The apt repository will be updated shortly to drop support for Sarge, since in practise I've not uploaded new things there for a while. In other news I made some new code for the Debian Administration website! The site now has the notion of a "read-only" state. This state forbids new articles from being posted, new votes being cast, and new comments being posted. The read-only state is mostly designed for emergencies, and for admin work upon the host system (such as when I'm tweaking the newly installed search engine). In more coding news I've been updating the xen-shell a little recently, so it will shortly have the ability to checksum the filesystem of Xen guests - and later validate them. This isn't a great security feature because it assumes you trust dom0 - and more importantly to checksum files your guest must be shutdown. However as a small feature I believe the suggestion was an interesting one. Finally I've been thinking about system exploitation via temporary file abuse. There are a couple of cases that are common:

• Creation of an arbitrary (writeable) file upon a host.
• Creation of an arbitrary (non-writable) file upon a host.
• Truncation of an existing file upon a host.

Exploiting the first to go from user to root access is trivial. But how would you exploit the last two? Denial Of Service attacks are trivial via the creation/truncation of /etc/nologin, /etc/shadow, (or even /boot/grub/menu.lst! But gaining privileges? I can't quite see how. Comments welcome!

I like to record NASA TV and watch it in fast foward. Generally the best bits are long exterior views, watching solar panels turn to catch the light as the earth rotates below, but I also sometimes see amusing things. This is Clay Anderson on the ISS. He spent quite a while floating around in this makeshift cape today. Of course even vampire astronauts have the ever-present clipboard which just makes him look more scary.

The Sith arrive at last, courtesy of the British government. (Watch the first 30s)

Finally this testing series is stabilizing and there are no known regressions. The latest big regression with encoding PDU data has been just fixed. Full list of changes:

• Fixed wrong PDU encoding (bug #412).
• Provide API for cleaning SMS backup structure.
• Added testing of SMS PDU encoding.

You can download from usual place: http://cihar.com/gammu/ I hope this will be really the last one before 1.14.0.

Even though Adrian von Bidder suggests that I'm starting a bit early with the DPL campaigning, I thought I could add a few more thoughts to the blagosphere.

Adrian suggested to make the discussion of our of "vision of Debian" a regular event on debconf - I think it's a nice idea, but that doesn't mean we should keep silent about that topic the rest of the year. I think one of the problems we see in Debian is that we are missing an aim to strive for. We are currently just moving from release to release, while more and more developers have actually lost interest in stable releases. The impression that a common ground like Debian 4.0 is irrelevant seems to prevail, because the people most vocal in our user base usually want to see the newest and shiniest software on their desktop. We have been spending more and more effort to cater to their wishes and have ignored the fact that a majority of Debian-based installations are not run by such people, but by system administrators who do mass roll-outs and neither want nor need the newest X.org with more *bling*.

Ubuntu, the biggest Debian-based distribution, has found its users by packing up new software behind a nice installer, adding some branding elements and then shipping it. Stability is an issue for Ubuntu, but they don't spend as much time as we do on eliminating the last seemingly important bug in some esoteric software package before actually releasing. That observation is the ground for one of my ideas: Our current metric to determine how far testing is away from a release is counting the number of rc bugs - it doesn't matter if the rc bug is X.org crashing for all people or if it is a policy violation in a package with 12 users. Perhaps we should start thinking about a new metric, keeping in mind that for a majority of users, some bugs matter a lot and some matter not at all. With the release of etch, the number of installations of popcon has risen a lot, so for the first time, we have actual usage data that could help us with such a metric. There are a few problems with that approach, though:

• Popcon data is inherently wrong, because single computer installations tend to have popcon, while large installations (like the one in Extremadura) don't.
• A majority of popcon users uses the last stable release, not the one currently in development.
• 3 bugs in packages seldomly used can be more annoying for our users than one central bug - you might be able to find a work around somewhere for something that happens to a lot of people, but a bug only hitting a few users will stay unfixed for a long time.

I don't have more time right now, so the rest of my ideas will follow in the next few days, so stay tuned or add me to your killfile.

Last week I installed Ubuntu Gutsy onto Heather's laptop. While Gutsy seems to be an easy task for most situations, installing it onto a Pentium 366 laptop with 200M of RAM and (particularly) an 800x600 screen was harder than it perhaps should have been. I'm sure that most installations these days aren't 800x600, but the graphical installer in Gutsy seems determined to make this painful. I had to move the toolbars to the sides of the screen, and then I could see the top half of the buttons on each page. It was like the page was sized for 600 vertical pixels, but the designer had forgotten about toolbars and title bars - not that I could see any screens in the process I followed that needed more than 5/6 of that screen anyway. Eventually I got it installed, and it even seemed to run OK once we booted into it. That's "OK for a 200M P366 with an 800x600 screen" though. Looking around at the price of a new laptop made putting up with that sort of performance a whole lot less palatable. The Acer Aspire 5310 (with free RAM upgrade) was $898 at Dick Smith, with a $99 cashback offer. A quick google shows that it's using the Broadcom 43xx wireless which isn't even close to being the best, but can be made to work with Linux. Everything else seemed likely to work, so we bought it. Installing Gutsy on it was nearly trivial, though I had to install bcm43xx-fwcutter on a different PC (my laptop, which is running Debian, in fact) to get the firmware for the WLAN before I could get the wireless working. I'm surprised that Broadcom still don't make that firmware publicly available somewhere, rather than forcing people to jump through the sort of hoops that would get them wanting an Intel chipset next time. Anyway, everything installed very easily, and the laptop is working quite nicely. Strangely neither sound, nor suspend to ram are working out of the box, but perhaps I'll get them going in due course. They're not so important in this case fortunately, but perhaps in due course I'll try and get them working and post some details about it. Much harder has been getting the fabled 'cashback' from Acer. I think I now know what I'm being paid $99 for. Firstly the only way to get your cashback is by registering through a webpage. Heather's first attempt to do this resulted in an error from our proxy about a malformed request, so I got called in. I tried registering using on my laptop, but couldn't even get to the cashback page. I then tried using IE6, with similar results. So perhaps it's my PC? I tried using a different PC, with the same result again! We tried ringing them up, but they were absolutely determined that (even after 20 minutes on the phone) they were not going to accept that information over the phone. So the only way to get the cashback from Acer was via their thoroughly broken website. Even their Contact Acer page is broken in firefox just showing a blank. Firefox users need not apply. Eventually, while spending some time in front of Heather's main computer (which had made it all the way through to submitting their on-line form before failing) I realised that the error she was getting was a proxy error from some in-form javascript submitting an invalid request, so I disabled the proxy, the form finally worked, and I managed to apply for the cashback. Now we just have to send the printed form in, along with some blood from our firstborn, the ashes of my grandmother, various barcodes, receipts and toenail clippings and we're sweet. They say they'll send us some money within 30 days. I think we should maybe frame it or something. I just know I'm going to feel really inclined to take advantage of cashback offers in future. In Other News: DVD Slideshow Meanwhile I've been playing with DVD Slideshow which seems to be just what my parents have been after for a while, so they don't have to keep their favourite photos on the camera to be able to show them off on someone's TV. It's great! At least it is great now after I changed all the calls to ffmpeg to add a 'k' after the bitrate parameter. But that's Open Source Software, I guess. I'll send a patch to them... :-)

Today, it's been two years I'm stuck in the NM queue. I've registered on oct 24, 2005, because Sam Hocevar told me that it could last a little, so if I was interested I should register early. I wondered for a long time if I should resign on this two years birthday, but I dropped this eventuality. I still want to be a Debian Developer, and if I resign I don't think I'll really want to still work on Xfce packages. But on the other hand I wouldn't really like to use an Xfce not packaged by me (no offense, but I guess I'd still want to hack on it) and I don't think I would be able to be on another distro. So I'm still there, sitting in NM queue, waiting for the next step. The only required step to be accepted as a Debian Developer is to have its gpg key added to the debian keyring (then an account is created on debian boxes, there's a -private subscription etc.). This is done by a team, called DAM, "Debian Accounts Managers". Because at one time they had a lot of work, they could not examine each candidate and decide if he was a potential good Debian Developer, so people imagined the NM Process: some volunteers would help them asking questions to candidates, then write a report, so DAM would only have to read it and decide if it was ok or not to create an account. I've already passed this step. I am "DAM Approved". One could imagine that, if the Debian *Accounts* Managers reviewed and approved my application, the account creation should follow. But no, the next step (according to NM pages) is "DAM creates account or rejects application". The problem here is that there is no "DAM Team", responsible for Debian accounts, but various people which are not really equivalent in this team. This is really a problem in Debian, where teams aren't really a "team". Not every team I guess, but in some cases one can trust a team to do a job. Debian is a volunteer organisation, so people think nobody can be forced to do a job if he doesn't have time to it. I'm fine with that, real life can be really time-consuming sometime. But my opinion is that teams may have to be forced to do their job. If nobody in the team has time, fine, it means somebody should join the team. But the team itself shouldn't fail, and this should be the only thing visible from outside: the "team", and the team actions. There are some proposition about this on lists at the moment, but I don't really know how it'll end. It's not easy doing social stuff, there's a lot of people involved, with many different views. But sometimes the consensus is *not* findable, and sometimes, somebody has to decide. That's why why need team leaders, and that's the project leader role. I guess people are quite opposed to have a "strong" leader in a volunteer organization, and I don't have concrete proposals but I really thing team management is not really perfect and a lot of people (users and developers) suffer from that (not only NM applicants, I mean) and from "elephants". As I said, I have no good solution, nor even concrete proposals (except "sometimes we need a leader to make decisions") but I really think there are things Debian (as a project) needs to improve.

I made a new release of YADA, the packaging helper different than others. New version of YADA supports Homepage field in debian/control file which appeared in dpkg last time. Wow. The YADA supported Home-Page field many years ago. YADA is very different that other Debian packaging helpers. Most important is that no Makefile have to be edited at all. This is the strongest and the weakest point of the YADA. People loves it or hate it. Some people even have a vendetta against YADA. Other people find the tool is very simple. Well, I think the YADA is some kind of prosthesis for native Debian packaging system, but even this prototype is much better than any Makefile-based aids just because I don't need to edit any Makefile. YADA had a black PR and much of bad FUD. There was opinion that it violates Debian policy, it is problematic for NMUs and security updates and maintaining of Debian patches is very difficult. This tool is very powerful and has some pitfalls but it doesn't mean it is evil by design. I really don't know how to make i.e. the iceape-l10n package without using YADA and its macro engine. It have to be more complicated and with much redundancy for metainformations. I think I could switch from YADA to another packaging tool, but please... Just create it! :)

After over a year I have finally got a SE Linux Play Machine online again. The details for logging in are at this link [1]. I’ve created T-shirt and mug designs with the login details too, they are on cafepress.com LINK [2]. For fun wear such a shirt to a conference (or even when shopping at your local electronics store. ;)

• [1] http://www.coker.com.au/selinux/play.html


• [2] http://www.cafepress.com/rufa/

Share This

Is it a sign that Sam should post the next Bits? ;-) I particularly like Marc's idea of a “liaison officer” to the other distributions in the Debian crowd. Can't say anything about that buildd and architecture issues, not knowing enough. Force-feeding the core teams more members is obviously a bad solution, but it's equally obvious that the current state of affairs should not continue. The “Vision of Debian” stuff should probably become a regular session at Debconf (based on the idea that face to face discussions are better than electronic ones, and that Debconf is truly the official meeting point for Debian, and that all important people should come to Debconf.) Only losely related to how to deal with child projects: there are two solutions: lock them out or outrun them. (Somewhere I can't remember I read something like this.) Obviously, we don't want to lock them out. So ... I should get working. Gah. It always comes back to me not doing enough for Debian. Sorry, folks.

I'll probably get laughed at by git/mercurial/darcs/whatever fanboys for even mentioning that I use subversion, but whatever. Just so I don't forget: If, in subversion, you want to merge changes on a branch to the mainline, then the following is wrong:

svn merge http://svnserver/branches/foo/ http://svnserver/trunk/ .

Instead, you should use this:

svn merge http://svnserver/branches/foo@REV http://svnserver/branches/foo@HEAD .

With the current directory being a checkout from /trunk/, and REV being the revision where /branches/foo and /trunk started to diverge; that is, either the revision where the branch was made, or the one where they were brought back in sync with a merge in either direction. Now hopefully I don't forget this for next time.

Recently it has come up on the LKML and other blogs about how C is not thread safe. The nice piece of example code given is At first glance it seems OK, and I wasn't exactly sure what the problem was until I pulled it apart. GCC is well within its rights to produce the following code (annotated for clarity). This code uses a bit of a trick to avoid a branch around code. It uses the carry flag, which is simple enough to understand when adding. Consider if you are adding 0001 + 1111 you end up with 11111, where the top bit is the carry bit. You can then feed this carry bit into another part of the addition operation if you were, say, using two 32 bit numbers to represent a 64 bit number on a 32 bit system. The trick uses the dual nature of the carry flag as a "borrow" bit; if a < b when a - b then the bit is set. Consider the simplest case of 0 - 1 -- the carry bit is set to indicate to the next operation that a bit needs to be borrowed. Imagine for a moment we have a four bit machine and want to do the 8-bit unsigned operation First we do 0000 - 0001 which leaves us 1111 with the borrow bit set. Then we do 1111 - 0000 but take note that the borrow bit is set, so get rid of the borrowed bit, leaving us 1110. Putting the two together gives us 1110 1111 or 239. Therefore the cmp $0x1, instruction (which is really just result - 1 with no output but setting flags) can only set the carry flag (i.e. borrow a bit) if the value is 0; any other value in result means there will be no need to borrow and the carry flag is not set. Consequently the add with carry (adc) will add one to the result only if the carry bit was set. A neat trick which avoids a jump, but not so much thread safe: if you are interrupted after the cmp but before the mov store you can write back the wrong value. Most people (including me till 5 minutes ago) looking at the code would assume that it was safe on the theory that acquires_count is not touched if you don't have the lock because it is inside the if statement. Interestingly, the Intel compiler does the simpler jump method Frankly I have no idea which one is faster/better (I would guess the second one because you don't always dirty a cache line, but you'd have to measure that against the cost of the branch) but obviously GCC thinks one thing and Intel thinks another and I bet if you put them in a room together you would end up with 3 opinions! I think the major point is that once upon a time it was mostly people writing kernels doing things like accessing I/O ports or other memory mapped areas that needed to be very careful the compiler didn't optimise out (or in) something incorrectly. With multi-threaded C code that problem moves up to the application developer too. A different instruction set can help here. The Itanium version is possibly less prone to this sort of thing because predication makes it faster to implement conditional branches -- any instruction with a predicate that is not set is skipped. The load and add is safe because it will only be done with the lock held. However, if anyone is wondering why Itanium come with such big caches, the difference between the 6 line version and the Itanium version below should answer that. I guess the moral of the story is that multi-threading is hard, but we knew that anyway.

I recently attended the Play! concert in Oslo, with a friend who was visiting Oslo from Bergen. The concert took place in an expo hall just outside Oslo, which was being used for the Electroworld electronics exposition. Wow, has that ever changed! I was there the first time, three-odd years ago, when it was arranged in Oslo Spektrum, and I was bitterly disappointed! It was nothing but a slightly larger Expert Electronics store. I was angry that I had actually been conned into paying for this. This year, it was completely different. They had a gigantic venue, with lots of interesting booths. Yamaha showed off some truly gorgeous digital pianos, and electronic drum sets, NRK was there, to show off their beautiful HD-1 HDTV production bus, and all kinds of interesting things. Even so, it was very clear that Expert was trying to get you to buy stuff. Something just doesn’t sit right with me about paying money to enter a store which is marketed as an expo. My previous experience with Electroworld meant I was not at all interested in repeating it. I wanted to catch the concert which was being held there. But they wouldn’t let me in unless I bought a ticket for the expo. Having already paid 70 EUR for a concert ticket, I was not enthusiastic about paying 15 more EUR to be allowed to enter the venue. But I relented, and was duly fleeced. I only got around 30 minutes of time before the concert started, so I didn’t get all that much time to look around. The first part of the concert started. The orchestra was the wonderful KORK (AKA Kringkastingsorkesteret, AKA “The Norwegian Radio Orchestra”). They started off with a beautiful fanfare by Nobuo Uematsu. However, their choice of music frustrates me deeply. They played some of the obvious classics, like Mario, Sonic The Hedgehog, some Zelda, but most of the time was wasted on some really lame music. It seems that their repertoire was not chosen based on the musical merit of the games, but on the current popularity. Why else would they be devoting time to some dull, insipid incidental music from WoW, Halo, and Silent Hill, without even an identifiable motif, as opposed to more of Koji Kondo’s wonderful Mario or Zelda tunes, or - damnit - Nobuo Uematsu’s Final Fantasy music? This is the thing that has left me the most disappointed. They practically ignored the elephant in the room of game music, Nobuo Uematsu. His compositions are absolutely beautiful, and they really shine when performed by a classical orchestra. You can practically tell that they’re meant to be performed by them, rather than some dinky Super Nintendo synthesizer chip. The Final Fantasy VI Opera, most of the Final Fantasy VII soundtrack… They then proceed to play One by Metallica. …WTF, you ask? WTF, indeed. Their justification was that it’s in the soundtrack from Guitar Hero. That’s absurd. And it wasn’t even a very good performance. It was still pretty funny that this very serious-looking classically trained vocalist slowly walked up on stage and gave a really good performance - he really did sound like the Metallica vocalist. Almost as an afterthought, at the very last, they play One-Winged Angel, one of the most dramatic and beautiful pieces from Final Fantasy VII. The audience breaks into absolutely wild applause. But - what - they bring in a METAL BAND on stage. A GODDAMNED METAL BAND! And they proceed to absolutely slaughter the whole piece! The choir and orchestra are drowned out by loud electric guitars with riffs that don’t make sense, and a drummer that can’t even keep the damned beat - the bass drum was all over the place! Seemingly not content with simply omitting his music, they have to absolutely demolish the performance of it. I actually have to say that this is by far the most disappointing concert I have ever gone to. Granted, my expectations were high, but just to witness a live performance by this wonderful orchestra of the Prelude, Anxious Heart, Tifa’s Theme, Eyes On Me, Fighting!, Still More Fighting! or Aerith’s Theme would’ve made the 85 EUR worth it. Judging from the audience’s applause when he first mentioned the Final Fantasy series, I think they agreed with me.

My daughters are perhaps among the first who will grow up seeing much more YouTube than POTV (my newly coined acronym, “Plain Old TV.”) Esther, now 2 , has still never seen anything on a television (recorded, broadcast, or cable). (Havi, a week old today, may not have seen anything at all at this point.) In addition to frequently watching videos from her own blog, Esther loves YouTube. Her most requested videos are embedded below, thanks to the excellent recently-installed Viper’s Video Quicktags. Thus Esther may never know anything but on-demand and user-generated culture. At least those will be her basic assumptions about how it’s supposed to work. It’s possible that this next generation will spell the end of network-scheduled and traditional advertising-interrupted media. I can never understand why anyone would bother to arrange their schedule to watch a show on TV and sit through commercials when they can rent it from Netflix or buy it from Amazon or iTunes and watch it when and how they want — and I grew up, to some extent, with the old TV model. Esther can barely wait for whichever video she wants now. I can’t see her putting up with content on someone else’s schedule as she grows up. Now the videos, as promised (both strong indicators that Esther adheres to the Long Tail theory): Actually, it turns out I didn’t coin POTV. It’s already defined here. Technorati Tags: Culture, Kids, YouTube, UGC

Share This

I’ve just updated the dgap packages with a new (beta) release from Digi. This release accommodates newer kernels and introduces a udev script, though the previous releases worked just fine with udev too. The packages are now built for Etch. On Sarge systems, you should keep version 1.3.6-1. The following apt source line can be used (sources available too): deb http://debian.technologeek.org/ etch non-free You’ll find the old packages at http://debian.technologeek.org/pool/non-free/d/dgap/old/ for Sarge systems or should anything go wrong with the latest packages. They can be installed by apt if you request the exact version. Bug reports and feedback should be directed at my inbox, as usual.

I have previously posted about the difference between using a chroot and using SE Linux [1]. Theo de Raadt claims that virtualisation does not provide security benefits [2] based on the idea that the Xen hypervisor may have security related bugs. From my understanding of Xen a successful exploit of a Xen system with a Dom0 that is strictly used for running the DomU’s would usually start by gaining local root on one of the DomU instances. From there it is possible to launch an attack on the Xen Dom0. One example of this is the recent Xen exploit (CVE-2007-4993) [3] where hostile data in a grub.conf in a DomU could be used to execute privileged commands in the Dom0. Another possibility would be to gain root access to a DomU and then exploit a bug in the Xen API to take over the hypervisor (I am not aware of an example of this being implemented). A final possibility is available when using QEMU code to provide virtual hardware where an attacker could exploit QEMU bugs, an example of this is CVE-2007-0998 where a local user in a guest VM could read arbitrary files in the host [4] - it’s not clear from the advisory what level of access is required to exploit it (DomU-user, DomU-root, or remote VNC access). VNC is different from other virtual hardware in that the sys-admin of the virtual machine (who might be untrusted) needs to access it. Virtual block devices etc are only accessed by the DomU and Xen manages the back-end. The best reference in regard to these issues seems to be Tavis Ormandy’s paper about hostile virtualised environments [5]. Tavis found some vulnerabilities in the QEMU hardware emulation, and as QEMU code is used for a fully virtualised Xen installation it seems likely that Xen has some vulnerabilities in this regard. I think that it is generally recommended that for best security you don’t run fully virtualised systems. The remote-console type management tools are another potential avenue of attack for virtualised servers in the case where multiple users run virtual machines on the same host (hardware). I don’t think that this is an inherent weakness of virtualisation systems. When security is most important you have one sys-admin running all virtual machines - which incidentally seems to be the case for most implementations of Xen at the moment (although for management not security reasons). In ISP hosting type environments I doubt that a remote console system based on managing Xen DomU’s is going to be inherently any less secure than a typical remote console system for managing multiple discrete computers or blades. I have just scanned the Xen hypervisor source, the file include/asm-x86/hypercall.h has 18 entries for AMD64 and 17 for i386 while include/xen/hypercall.h has 18 entries. So it seems that there are 35 or 36 entry points to call the hypervisor, compared to 296 system calls on the i386 version of Linux (which includes the sys_socketcall system call which expands to many system calls). This seems to be one clear indication that the Linux kernel is inherently more complex (and therefore likely to have a higher incidence of security flaws) than the Xen hypervisor. Theo’s main claim seems to be that Xen is written by people who aren’t OpenBSD developers and who therefore aren’t able to write secure code. While I don’t agree with his strong position I have to note the fact that OpenBSD seems to have a better security history than any other multi-user kernel for which data is available. But consider a system running Xen with Linux in Dom0 and multiple para-virtualised OpenBSD DomU’s. If the Linux Dom0 has OpenSSH as the only service being run then the risk of compromise would be from OpenSSH, IP based bugs in the Linux kernel (either through the IP address used for SSH connections or for routing/bridging to the OpenBSD instances), and from someone who has cracked root on one of the OpenBSD instances and is attacking the hypervisor directly. Given that OpenSSH comes from the OpenBSD project it seems that the above scenario would only add the additional risk of an IP based Linux kernel attack. While a root compromise of an OpenBSD instance (consider that a typical OpenBSD system will run a lot of software that doesn’t come from the OpenBSD project - much of which won’t have a great security history) would only lose that instance unless the attacker can also exploit the hypervisor (which would be a much more difficult task than merely cracking some random daemon running as root that the sys-admin is forced to install). Is the benefit of having only one instance of OpenBSD cracked due to a bad daemon enough to outweigh the risk of a Linux IP stack? I’m sure that the OpenBSD people would consider that a better option would be OpenBSD in the Dom0 and in the DomU. In which case the risk of damage from a root compromise due to one badly written daemon that didn’t come from OpenBSD is limited to a single DomU unless the attacker also compromises the hypervisor. When working as a sys-admin I have been forced by management to install some daemons as root which were great risks to the security of the system, if I had the ability to install them in separate DomU’s I would have been able to significantly improve the security of the system. Another flaw in Theo’s position is that he seems to consider running a virtual machine as the replacement of multiple machines - which would be an obvious decrease in security. However in many cases the situation is that no more or less hardware is purchased, it is just used differently. If instead of a single server running several complex applications you have a Xen server running multiple DomU’s which each have a single application then things become much simpler and more secure. Upgrades can be performed on one DomU at a time which decreases the scope of failure (which often means that you only need one business unit to sign-off on the upgrade) and upgrades can be performed on an LVM snapshot (and rolled back with ease if they don’t succeed). A major problem with computer security is when managers fear problems caused by upgrades and prohibit their staff from applying security fixes. This combined with the fact that on a multiple DomU installation one application can be compromised without immediate loss of the others (which run in different DomU’s and require further effort by the attacker for a Xen compromise) provides a significant security benefit. It would be nice for security if every application could run on separate hardware, but even with blades this is not economically viable - not even for the biggest companies. I have converted several installations from a single overloaded and badly managed server to a Xen installation with multiple DomU’s. In all cases the DomU’s were easier to upgrade (and were upgraded more often) and the different applications and users were more isolated. Finally there is the possibility of using virtualisation to monitor the integrity of the system, Bill Broadley’s presentation from the 2007 IT Security Symposium [6] provides some interesting ideas about what can be done. It seems that having a single OpenBSD DomU running under a hypervisor (maybe Xen audited by the OpenBSD people) with an OpenBSD Dom0 would offer some significant benefits over a single OpenBSD instance.

• [1] http://etbe.coker.com.au/2007/08/22/se-linux-vs-chroot/


• [2] http://kerneltrap.org/OpenBSD/Virtualization_Security


• [3] http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2007-4993


• [4] http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2007-0998


• [5] http://taviso.decsystem.org/virtsec.pdf


• [6] http://shell.cse.ucdavis.edu/~bill/virt/virt.pdf

Share This