Search Results: "rul"

Other Unix systems apparently calculate the load average differently to Linux. According to the Wikipedia page about Load(computing) [1] most Unix systems calculate it based on the average number of processes that are using a CPU or available for scheduling on a CPU while Linux also includes the count of processes that are blocked on disk IO (uninterruptible sleep). There are three load average numbers, the first is for the past minute, the second is for the past 5 minutes, and the third is for the past 15 minutes. In most cases you will only be interested in the first number. What is a good load average depends on the hardware. For a system with a single CPU core a load average of 1 or greater from CPU use will indicate that some processes may perform badly due to lack of CPU time - although a long-running background process with a high “nice” value can increase the load average without interfering with system performance in most cases. As a general rule if you want snappy performance then the load average component from CPU use should be less than the number of CPU cores (not hyper-threads). For example a system with two dual-core CPUs can be expected to perform really well with a load average of 3.5 from CPU use but might perform badly with a load average of 5. The component of the load average that is due to disk IO is much more difficult to interpret in a sensible manner. A common situation is to have the load average increased by a NFS server with a network problem. A user accesses a file on the NFS server and gets no response (thus giving a load average of 1), they then open another session and use “ls” to inspect the state of the file - ls is blocked and gives a system load average of 2. A single user may launch 5 or more processes before they realise that they are not going to succeed. If there are 20 active users on a multi-user system then a load average of 100 from a single NFS server that has a network problem is not uncommon. While this is happening the system will perform very well for all tasks that don’t involve the NFS server, the processes that are blocked on disk IO can be paged out so they don’t use any RAM or CPU time. For regular disk IO you can have load average incremented by 1 for each non-RAID disk without any significant performance problems. For example if you have two users who each have a separate disk for their home directory (not uncommon with certain systems where performance is required and cooperation between users is low) then each could have a single process performing disk IO at maximum speed with no performance problems for the entire system. A system which has four CPU cores and two hard drives used for separate tasks could have a load average slightly below 6 and the performance for all operations would be quite good if there were four processes performing CPU intensive tasks and two processes doing disk intensive tasks on different disks. The same system with six CPU intensive programs would under-perform (each process would on average get 2/3 of a CPU), and if it had six disk intensive tasks that all use the same disk then performance would be terrible (especially if one of the six was an interactive task). The fact that a single load average number can either mean that the system is busy but performing well, under a bit of load, or totally overloaded means that the load average number is of limited utility in diagnosing performance problems. It is useful as a quick measure, if your server usually has a load average of 0.5 and it suddenly gets a load average of 10 then you know that something is wrong. Then the typical procedure for diagnosing it starts with either running “ps aux grep D” (to get a list of D state processes - processes that are blocked on disk IO) or running top to see the percentages of CPU time idle and in IO-wait states. Cpu(s): 15.0%us, 35.1%sy, 0.0%ni, 49.9%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
7331 rjc 25 0 2868 640 312 R 100 0.0 0:21.57 gzip Above is a section of the output of top showing a system running gzip -9 < /dev/urandom > /dev/null. Gzip is using one CPU core (100% CPU means 100% of one core - a multi-threaded program can use more than one core and therefore more than 100% CPU) and the overall system statistics indicate 49.9% idle (the other core is almost entirely idle). Cpu(s): 1.3%us, 3.2%sy, 0.0%ni, 50.7%id, 44.4%wa, 0.0%hi, 0.3%si, 0.0%st PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
7425 rjc 17 0 4036 872 588 R 4 0.1 0:00.20 find Above is a section of the output of top showing the same system running find /. The system is registering 44% IO wait and 50.7% idle. The IO wait is the percentage of time that CPU core is waiting on IO, so 44% of the total system CPU time (or 88% of one CPU core) is idle while the system is waiting for disk IO to complete. A common mistake is to think that if the IO was faster then more CPU time would be used, in this case with the find program using 4% of one CPU core if all the IO was instantaneous (EG in cache) then the command would complete 25 times faster with 100% CPU use. But if the disk IO performance was doubled (a realistic possibility given that the system has a pair of cheap SATA disks in a RAID-1) then find would probably use 8% of CPU time. Really the only use for load average is for getting an instant feel for whether there are any performance problems related to CPU use or disk IO. If you know what the normal number is then a significant change will stand out. Dr. Neil Gunther has written some interesting documentation on the topic [2], which goes into more technical detail including kernel algorithms used for calculating the load average. My aim in this post is to educate Unix operators as to the basics of the load average. His book The Practical Performance Analyst gives some useful insights into the field. One thing I learned from his book is the basics of queueing theory. One important aspect of this is that as the rate at which work arrives approaches the rate at which work can be done the queue length starts to increase exponentially, and if work keeps arriving at the same rate when the queue is full and the system can’t perform the work fast enough the queue will grow without end. This means that as the load average approaches the theoretical maximum the probability of the system dramatically increasing it’s load average increases. A machine that’s bottlenecked on disk IO for a task where there is a huge number of independent clients (such as a large web server) may have it’s load average jump from 3 to 100 in a matter of one minute. Of course this won’t mean that you actually need to be able to serve 30 times the normal load, merely slightly more than the normal load to keep the queues short. I recommend reading the book, he explains it much better than I do.

I'd been using Emacs for quite awhile, and about 8 months ago I decided I would try using Vim. I'd only used vi for system emergency work, but knew a number of people that swore by it for regular work. So I decided I would learn Vim and use it for my regular work. I figure that with things like this, I don't get a real feel for how well they work unless I use them for all my work. So I haven't really opened Emacs at all in the past 8 months.

Yesterday I finally decided that Vim was not living up to my expectations and I'm in the process of switching back to Emacs. I thought I ought to write down why I'm doing that, for my own future reference... and since nobody has ever written about Emacs vs. Vim, I might as well post it where everyone can see it.

So here we are.

Original Reasons for Using Vim

It would lead to more comfortable typing. Lots of Vim users mention that you don't have to hold down keys while hitting other keys as much in Vim as in Emacs, and that the movement keys are all on the home row. That's true, but I didn't find it to be that big of an improvement, since Esc is a farther reach than anything in Emacs, and let me tell you, you're hitting Esc all the time in Vim. I found that removing the armrests from my chair made my hands happier than Vim ever did, and swapping Ctrl and CapsLock in Emacs will probably help there too.

It starts faster. I'm not sure if that really was true even when I switched, but it certainly isn't true on any of my machines today. Both Vim and Emacs have had major version upgrades (v7 and v22, respectively) since I started using Vim. People seem to say that Emacs 22 feels faster, though I don't know if that's true. The startup times of the two, if they're different, are imperceptible.

Vim would use less RAM. Frankly, these days, both Emacs and Vim are way down on the list of things that use up RAM. Heck, kmail has 141MB resident, and each of its two IMAP processes is using more than 30MB. Emacs in X right after start has 16MB resident, 10MB of which is shared, and 25MB VSS. gvim right after start has 8MB resident, 5MB of which is shared, and a 43MB VSS. Emacs tends to use fewer processes for things that vim. So they're not all that different, and Emacs could come out smaller in certain situations. But the difference is irrelevant on today's machines, and modern Gnome and KDE apps are many times larger than both of them.

It will make me more comfortable in rescue environments where I have only traditional vi available. Actually, the vi on AIX is so different from modern Vim that this didn't really help.

It would make me more productive. There are some editing commands that did, but as you'll see below, it was more than balanced out by other problems.

Things I Liked about Vim

The commands dt, dT, df, dF. Wonderful little things those. Emacs now has M-z (Zap), which is similar to df but can actually go to other lines (a nice addition). And there are easy ways to bind keys to the others as well, though that doesn't make it a pervasive convention like it is in Vim.

Antialiased fonts. It's crazy that Emacs doesn't have this yet. But not a showstopper; I still like good ole 10x20 just fine.

Regexp search-and-replace. Emacs actually has this now, and maybe it had it back then too. M-C-%. Apparently in Emacs22 the replacement expression can also have lisp code in it, which sounds really slick but I can't see myself using it regularly.

Annoying Things in Vim

Syntax highlighting. The syntax highlighting for most languages in Vim felt like it was about as smart as it was in Emacs about 10 years ago. Strings like "Hello!\"" (in languages where \" inserts a literal ") often confused it. Sometimes quotes within comments confused it. Sometimes it would be confused permanently. Other times, just until I scrolled around in the file or reloaded it.

Indentation. This is much more annoying than the syntax highlighting, really. In many languages -- and especially the two modes I've used most recently, XML and Haskell -- it really, really stinks. The indentation there isn't aware of syntax, or not very much. Sometimes it is smart enough to know that if an XML line starts with </ that it moves left and if it starts with an opening tag, that the next line moves right. But it's not smart enough to do this reliably. Not only that, but indentation is not handled with consistent configuration between languages. And even though Vim ships with a ton of language modes, the central docs only cover indentation for C.

I've asked Vim experts about this, and have tried all sorts of various tweaks, have read through Vim indentation mode source files, etc. There is just no way to get it anywhere near the intelligence of Emacs for most languages, short of writing my own mode, it appears. This is even worse because when using the backspace key in insert mode, for awhile it deletes individual spaces, and then all of a sudden deletes a big chunk of whitespace back to the beginning of the line. (And no, the insertion of Tab characters is disabled.) Indentation is my #1 complaint about Vim, and something that shows no progress towards being fixed any time soon.

And forget about anything like Emacs M-x reindent-region. This is a syntax-aware indenter. You can write out an entire source file with no indentation whatsoever, and it will indent the entire thing according to the indentation rules you've defined and the syntax of the language you're using. The best I've seen in Vim are commands that add or remove space at the beginning of every line in a region.

In short, Emacs seems to "understand" the file format on a much deeper level than Vim, and can automate things to a much better extent because of it.

Too many things disrupt the paste buffer. I can use Y or y to yank some text in Vim, and it's really, really easy to overwrite that buffer with other things. Yes, I know that I can yank it into a named buffer, but that's inconvenient and I don't usually know in advance that I'll have that need. In Emacs, only C-k and other "large area" commands disrupt it.

Vim doesn't like you having lots of files open at once. It's surprisingly convoluted to do this. If you use the basic documented command to edit another file, :e, it closes the file you're working on. The normal way to open multiple files at once is to use split windows. Well, I don't like split windows all that well, and often just want to make a quick change in one file -- in full screen -- and then go back to another. Even though I use set hidden in my ~/.vimrc, it still is annoying and more convoluted than it should be.

Vim can't create new top-level X windows. In Emacs, I can press C-x 5 2, and poof, I have a second Emacs window in X, and it's tied to the same editing session and Emacs process. Not a new process, with a different set of files, its own buffers, etc. The same process, same set of files. Just like a split window, but with a new top-level X window instead. gvim simply has no way to do that. This is also a large annoyance.

gqap stinks. This has burned me more than once. I'll be editing an XML document, and insert some text in the middle of a paragraph. Now I have a really wide line. So I type gqap to reformat the paragraph. My cursor is near the bottom of the screen, so I don't really see much past the current line. I then save the document and exit. Later I discover that vim considered the entire rest of the document part of the single paragraph, and removed all the different indentation levels at </para> and the like, so it's completely messed up. Emacs is smart enough to know what is a paragraph in XML mode, and M-q does the right thing. Oh, and Emacs reindent-region can fix the Vim gqap-induced mess.

Amusing tidbits you pick up in the mailing industry...

No, you can't mail running chainsaws. At least I'm pretty sure about that; I don't know a specific regulation against them, but USPS takes a dim view of anything that puts its employees and machinery at risk.

On the other hand, you can mail live scorpions if you package them appropriately. Note that, however, that this is limited to the US; apparently doing so in Australia is not a good idea. The same page also gives the rules for mailing baby chicks (which must be not more than 24 hours old) as well as adult chickens (Express Mail only, please; and no refund if the chicken dies in transit unless the container was damaged).

Is it a sign that Sam should post the next Bits? ;-) I particularly like Marc's idea of a “liaison officer” to the other distributions in the Debian crowd. Can't say anything about that buildd and architecture issues, not knowing enough. Force-feeding the core teams more members is obviously a bad solution, but it's equally obvious that the current state of affairs should not continue. The “Vision of Debian” stuff should probably become a regular session at Debconf (based on the idea that face to face discussions are better than electronic ones, and that Debconf is truly the official meeting point for Debian, and that all important people should come to Debconf.) Only losely related to how to deal with child projects: there are two solutions: lock them out or outrun them. (Somewhere I can't remember I read something like this.) Obviously, we don't want to lock them out. So ... I should get working. Gah. It always comes back to me not doing enough for Debian. Sorry, folks.

I recently attended the Play! concert in Oslo, with a friend who was visiting Oslo from Bergen. The concert took place in an expo hall just outside Oslo, which was being used for the Electroworld electronics exposition. Wow, has that ever changed! I was there the first time, three-odd years ago, when it was arranged in Oslo Spektrum, and I was bitterly disappointed! It was nothing but a slightly larger Expert Electronics store. I was angry that I had actually been conned into paying for this. This year, it was completely different. They had a gigantic venue, with lots of interesting booths. Yamaha showed off some truly gorgeous digital pianos, and electronic drum sets, NRK was there, to show off their beautiful HD-1 HDTV production bus, and all kinds of interesting things. Even so, it was very clear that Expert was trying to get you to buy stuff. Something just doesn’t sit right with me about paying money to enter a store which is marketed as an expo. My previous experience with Electroworld meant I was not at all interested in repeating it. I wanted to catch the concert which was being held there. But they wouldn’t let me in unless I bought a ticket for the expo. Having already paid 70 EUR for a concert ticket, I was not enthusiastic about paying 15 more EUR to be allowed to enter the venue. But I relented, and was duly fleeced. I only got around 30 minutes of time before the concert started, so I didn’t get all that much time to look around. The first part of the concert started. The orchestra was the wonderful KORK (AKA Kringkastingsorkesteret, AKA “The Norwegian Radio Orchestra”). They started off with a beautiful fanfare by Nobuo Uematsu. However, their choice of music frustrates me deeply. They played some of the obvious classics, like Mario, Sonic The Hedgehog, some Zelda, but most of the time was wasted on some really lame music. It seems that their repertoire was not chosen based on the musical merit of the games, but on the current popularity. Why else would they be devoting time to some dull, insipid incidental music from WoW, Halo, and Silent Hill, without even an identifiable motif, as opposed to more of Koji Kondo’s wonderful Mario or Zelda tunes, or - damnit - Nobuo Uematsu’s Final Fantasy music? This is the thing that has left me the most disappointed. They practically ignored the elephant in the room of game music, Nobuo Uematsu. His compositions are absolutely beautiful, and they really shine when performed by a classical orchestra. You can practically tell that they’re meant to be performed by them, rather than some dinky Super Nintendo synthesizer chip. The Final Fantasy VI Opera, most of the Final Fantasy VII soundtrack… They then proceed to play One by Metallica. …WTF, you ask? WTF, indeed. Their justification was that it’s in the soundtrack from Guitar Hero. That’s absurd. And it wasn’t even a very good performance. It was still pretty funny that this very serious-looking classically trained vocalist slowly walked up on stage and gave a really good performance - he really did sound like the Metallica vocalist. Almost as an afterthought, at the very last, they play One-Winged Angel, one of the most dramatic and beautiful pieces from Final Fantasy VII. The audience breaks into absolutely wild applause. But - what - they bring in a METAL BAND on stage. A GODDAMNED METAL BAND! And they proceed to absolutely slaughter the whole piece! The choir and orchestra are drowned out by loud electric guitars with riffs that don’t make sense, and a drummer that can’t even keep the damned beat - the bass drum was all over the place! Seemingly not content with simply omitting his music, they have to absolutely demolish the performance of it. I actually have to say that this is by far the most disappointing concert I have ever gone to. Granted, my expectations were high, but just to witness a live performance by this wonderful orchestra of the Prelude, Anxious Heart, Tifa’s Theme, Eyes On Me, Fighting!, Still More Fighting! or Aerith’s Theme would’ve made the 85 EUR worth it. Judging from the audience’s applause when he first mentioned the Final Fantasy series, I think they agreed with me.

I’ve been asked for my opinion of senatoronline.org.au which claims to be Australia’s only internet-based political party. The claim may be correct depending on what you consider to be “Internet based“. Here is a copy of their platform from their web site:
Senator On-Line is not aligned to any other political party it is neither Liberal nor Labor. Senator On-Line ( SOL ) is a truly democratic party which will allow everyone on the Australian Electoral roll who has access to the internet to vote on every Bill put to Parliament and have its Senators vote in accordance with a clear majority view. We will be running candidates for the upcoming federal Upper House (Senate) elections. When a SOL senator is elected a web site will be developed which will provide:

Accurate information and balanced argument on each Bill and important issues

The vast majority of those registered on the Australian Electoral roll the chance to have their say by voting on bills and issues facing our country

A tally of all votes which will then count in Parliament

Each person on the Australian Electoral roll will be entitled to one vote and only be allowed to vote once on each bill or issue. SOL senators will have committed in writing to voting in line with the clear majority view of the SOL on-line voters. Senator On-Line will enable broader community involvement in the political process and the shaping of our country. If you like the concept, please register your details and tell others about SOL.
Now at first glance it sounds like a good idea, the Liberal party (which is similar in all the bad ways to the US Republican party) has demonstrated what happens when a party gets away with entirely ignoring the wishes of the voters. But there are three significant problems with the Senator Online idea. The first is the issue of providing “Accurate information and balanced argument on each Bill“. We have seen many media outlets claiming that there is a debate about global warming (the debate ended years ago, the vast majority of scientists have agreed that global warming existed for a long time) and now the same media outlets are claiming that there is a debate about whether it will cause any harm to us in the near future (ignoring all the dams that are running low). One of the aims of the democratic process is that representatives who spend all their time working on politics can fully analyse the issues and also gain access to resources that are not available to the average citizen, thus being able to make better informed decisions. The next problem is that it can degenerate into mob-rule. The idea of having a tabloid TV show being able to directly control votes in the senate is not an exciting one. The final problem is how to ensure that each citizen gets the opportunity to have exactly one vote, solving this requires good security and identity checks involving viewing photo-id. The checks used for voting (merely asserting your name and residential address) might be considered adequate for an election, but are grossly inadequate for online voting where one false registration allows many false votes. I think it would be more interesting to have a populist party started that campaigns for the issues that have the most impact on the majority of citizens. Issues such as the fact that a couple on the median income can’t afford the median house price [1], the lack of job security that is caused by recent industrial relations legislation, the price of health-care, and the fact that any car which is affordable on a single median income (after money is spent on rent and other basic living expenses) will lack basic safety features such as air-bags. While the Green party has decent policies on these issues they have many other policies to consider. A party that solely concentrated on allowing more people to have decent health care, not risk losing their job, own their own home, and drive a safe car would get a lot of interest from people who don’t normally take much notice of politics.

[1] http://etbe.coker.com.au/2007/08/06/housing-prices/

Share This

the co-operative half-yearly meeting, Bristol

Belatedly, as promised, here are the questions from the floor at last week's half-year meeting of the Co-operative Group Kennet and Avon area. All errors and omissions are mine, of course.

How do we defend against demutualisation and private equity attacks?: Our rules are now written so that demutualisation would require assets to be transferred to other co-operatives. Please comment on other ways to the current constitutional review consultation.
How can ordinary members get informed and involved in running the group?: Get items into network, call the regional office.
The consitutional review should give more power to meetings and make it easier for motions and binding promises.: will forward this to the review.
Why did the group secretary suddenly leave?: The job had changed post-merger, with different responsibilities.
How can we stop the payments to voluntarily-retiring directors?: The area has made its point. The chief executive of CooperativesUK will look into this and there was a congress motion to look into this issue.
There's been no mortgage advisor in one local bank branch for months. Is this indicative of the bank's attitude to mortgages?: No, it's disappointing. Please send details to the area secretary.
Please can we use geographic numbers instead of the expensive 0845 for the membership enquiries?: Will forward and ask for a geographic number.
Is the constitutional review being rushed?: It's streamlined because the merger gives a pressing need, but we want member input.
Can we or do we promote credit unions in-store?: Yes, we can and do promote them to members already. Can set aside some space in-store if needed.
Does the co-operative travel insurance discriminate against over 75s?: will forward the answer when we have it.
The top management post-merger are all from United: is this a take-over?: no, 2 executives had clear intentions to retire already and the lower levels are more mixed.
Which? gave the co-operative food a D for organic produce: why and what will we do about it?: Report from today's Guardian not yet seen. Expect to respond in next few days. The report may have compared convenience stores with larger stores.
Will ex-United travel agents become the co-operative travel stores?: Yes. There's a big big change coming, to be more of a community travel agent, restructured, with more phone and internet options.
Is the energy-saving policy to include refrigeration, lighting and IT?: This is a top goal, to improve refrigeration and lighting control. It saves the earth and saves money. Nearly all electricity is already "greener" electricity.
Do we own the freehold on the land previously occupied by a store at Shirehampton?: Yes. Only a non-food retailer will be allowed in.
Why do elected representatives have to retire at 70 if still capable?: Please submit comments on it to the constitutional review?
Why was there no transport from Weston-super-Mare?: Will check the office records for requests.

Anyone see these themes appear at other meetings? Any comments on these themes?

The last five days I've spent at the combined Debian Edu, FAI and CDD meeting in M rida in Extremadura, together with 17 others. The Debian Edu / Skolelinux crowd was the biggest, followed by FAI, followed by CDD people, though many were part of more than one project. "As usual" it was great to meet many known faces and some unknown ones and to be able to discuss stuff in RL.

Besides various small different things, I worked on the Debian Edu etch r1 release, our lenny roadmap (see the bottom of our roadmap page which also covers etch r1) and the (technical side of the) german translation of the Debian Edu documentation. Non edu related the new setupharddisk2 for FAI took most of my brain activity. And I've signed all keys allready :-)

We stayed at a very nice hotel in the even much nicer city of Merida, which features a lot of roman ruins and history. For example, we walked over the biggest roman bridge in the world! The only thing that was bigger was the amount of good food we were forced to eat every day... Incredible. This was my second visit to Merida and this time I finally managed to visit the amphitheatre (with seats for 15000 people) and the Roman theatre. (Unfortunatly the english page about Emerita Augusta is a stub.) If you happen to come around that area make sure to spend some time sight seeing, it is really worth it! People living in Merida literally fear to rebuild their houses, because as soon as they start to dig, they find something historical and have to turn their property into a museum :-)

In case you have not seen it yet, you should watch this small movie from euro news about the free software efforts in Extremadura. Insightful and very good promotion, also to show around.

Many thanks to C sar, Jos Luis and the Junta de Extremdura for hosting such a great meeting! You rock!

On a unrelated note, spam has gone crazy yesterday while traveling back, 15000 mails were not catched by my so far well working setup. ARG. I still used to have a catchall setting, which was useful to give out addresses like $company@mydomain, but now I will explicitly list those addresses and disable catchall. And luckily for you I will not blog about every spam mail I got... :) Generally I dont consider spam (the source of) evil, it's money: as long as money rules the world, there will be spam. So since it's a social problem, it is hardly solvable by technology. (That is, as long as you want to receive mails from strangers, which I do.) IMO what's worse than spammers are autoresponders to spam. Most of those 15000 mails were auto-replies to spam. OMFSM.

I have just read an interesting post speculating about the possibility of open source hardware [1]. To some extent things have been following a trend in that direction. Back in the bad old days every computer manufacturer wanted to totally control their market segment and prevent anyone else from “stealing their business”. Anti-competitive practices were standard in the computer industry, when you bought a mainframe you were effectively making a commitment to buy all peripherals and parts from the same company. The problems were alleviated by government action, but the real change came from the popularity of PC clones. White-box clones where every part came from a different company truly opened up the hardware development, and it wasn’t all good. When running a simple single-tasking OS such as MS-DOS the problems were largely hidden. But when running a reliable multi-tasking OS such as Linux hardware problems became apparent. The PCI bus (which autoconfigured most things) reduced the scope of the problem but there are still ways that white-box machines can fail you. Now when I get a white-box machine I give it away to members of my local LUG. My time is too valuable to waste on debugging white-box hardware, I would rather stick to machines from IBM and HP which tend to just work. Nowadays I buy only name-brand machines, all the parts were designed and tested to work together - this doesn’t guarantee that the machine will be reliable but it does significantly improve the probability. Fortunately modern hardware is much faster than I require for the work I do, so buying second-hand name-brand machines (for less money than a new white-box machine) is a viable option. The PCI bus [2] standard from Intel can be compared to some of the “Open Source” licenses from companies where anyone can use the software but only one company can really be involved in developing it. One significant impediment to open hardware development is the proprietary nature of CPU manufacture. Currently there are only a few companies that have the ability to fabricate high-end CPUs so projects such as OpenRISC [3] which develop free CPU designs will be limited to having their CPUs implemented with older technology (which means lower clock speeds). However this doesn’t mean that they aren’t useful, tailoring factors such as the number of registers, the bus width of the CPU, and the cache size to match the target application has the potential to offset the performance loss from a lower clock speed. But this doesn’t mean that an OpenRISC or similar open core would be ideal for your typical desktop machine. If companies such as Intel and AMD were compelled to fabricate any competing CPU design at a reasonable cost (legislation in this regard is a possibility as the two companies collectively dominate the world computer industry and it would be easy for them to form a cartel) then designs such as OpenRISC could be used to implement new CPUs for general purpose servers. Another issue is the quality of support for some optional extra features which are essential for some operations. For example Linux software RAID is quite good for what it does (basic mirroring, striping, RAID-5 and RAID-6), but it doesn’t compare well with some hardware RAID implementations (which actually are implemented in software with a CPU on the RAID controller). For example with a HP hardware RAID device you can start with two disks in a RAID-1 and then add a third disk to make it a RAID-5 (I’ve done it). Adding further disks to make it a larger RAID-5 is possible too. Linux software RAID does not support such things (and I’m not aware of any free software RAID implementation which does). It would certainly be possible to write such code but no-one has done so - and HP seem happy to make heaps of money selling their servers with the RAID features as a selling point. Finally there’s the issue of demand. When hardware without free software support (such as some video cards which need binary-only drivers for best performance) is discussed there is always a significant group of people who want it. The binary-only drivers in question are of low quality, often don’t support the latest kernels, and have such a history of causing crashes that kernel developers won’t accept bug reports from people who use them, but still people use them. In the short-term at least I expect that an open hardware design would deliver less performance and in spite of the fact that it would have the potential to offer better reliability the majority of the market would not accept it. The production volume of electronics gear is the major factor determining the price so it would cost more. I think that both IBM and HP provide hardware that is open enough for my requirements, they both have engineers working on their Linux support and the interfaces are well enough documented that we generally don’t have any problems with them. Both Intel and AMD and the major system vendors are all working on making things more open, so I expect some small bug significant improvements in the near future.

[1] http://www.raiden.net/?cat=2&aid=315

[2] http://en.wikipedia.org/wiki/Peripheral_Component_Interconnect

[3] http://en.wikipedia.org/wiki/OpenRISC

Share This

Two recent print-media articles begging to be linked: First, the Autumn of the Multitaskers in this month’s Atlantic. Despite the age-of-Internet theme, the full article is only available to subscribers. The thesis is one I’ve seen before (e.g., in GTD). Namely: although we might feel like we’re accomplishing more when multitasking, we actually end up less productive, and eventually we’ll lose our minds if we continue to overload our sense. The writer takes the idea further than I’ve seen before, incorporating several astonishing indicators from the zeitgeist, including this NSFW quote from Jennifer Connelly that I won’t reprint, and several foreign policy and domestic political examples of how multitasking is failing our culture and our country. There is a generational divide at work here. Most in previous generations (say, people over 40) didn’t come online until the Internet was nearly full-blown with spam, entertainment news, popups, advertising, streaming stock quotes, rich media services, IM, Facebook, Flash, etc.. These people generally deal poorly with information and sensory overflow because they had no incremental introduction. The younger generations (people under 25) don’t remember the world before the Internet and multipurpose portable devices, and thus think it entirely normal to watch TV, surf the web with a few open windows, talk on the phone (perhaps VoIP), and IM at the same time. At least. And they’re bored if they don’t have at least have three or four tasks going simultaneously. This isn’t to say that these gen-Yers are actually doing a good job at it — I find myself often answering computer-related questions from my younger siblings because they can’t actually read a focused page or two of dense text on the computer (e.g., a readme file) in one sitting — but rather that they don’t experience the flow as psychologically overwhelming. That leaves my (in between) generation, generally people born in the 1970’s. Those of us who grew up with early low-bandwidth wide area networks — I started on BITNET, USENET, and BBS’s in the early- to mid-1980’s — have better mechanisms for coping and prioritizing. Perhaps because our multitasking options were relatively limited, we haven’t leaped on the bandwagon to the same extent as others. I personally don’t mind two simultaneous tasks — e.g., perusing RSS headlines and listening to MP3s — but more than that doesn’t interest me. (I don’t count it as “multitasking” in the same sense where one of the tasks is just waiting, e.g., for code to compile or a large upgrade to download). Please feel free to destroy my broad generalizations here in the comments. While we’re on the subject of multitasking, I fall into the “subtle exception” category described in Getting Things Done:

The Multitasking Exception There’s a subtle exception to the one-item-at-a-time rule. Some personality types really need to shift their focus away from some task for at least a minute in order to make a decision about it. When I see this going on with someone, I let him take two or sometimes three things out at once as he’s processing. It’s then easier and faster for him to make a choice about the action required.

I find this trick particularly helpful in getting unstuck. Frequently, when drafting briefs, I’ve rewritten a sentence eight or nine times and it still doesn’t seem to get across the necessary idea efficiently and without making the reader do any hard work. Switching to another (usually simpler) task, if only for one or two minutes, is often enough to come back and do it right in one swoop. In this situation (and probably only in this situation), multitasking actually saves me time. Second, the Well Tempered Web (by Alex Ross, music critic with a blog) in this week’s New Yorker. This one is more naturally available in complete form online. Ross manages to weave together Wagner, Snakes on a Plane, the Long Tail, and Rick Rubin. Worth reading. (And while you’re at it, check out Adam Gopnik’s piece on abridgments, which unfortunately is not available online.) Lest you all think all I read is the New Yorker and the Atlantic: it’s not true. I’m just not at liberty to disclose my other sources at this time.

Share This

Yesterday night, a taxi driver was killed in Madrid. The rest decided to go on strike. No taxis are available in the whole city. I had to get my plane to Lyon at half past seven, the check-in finished at 07:00. Metro starts at 6:00, and the airport is quite far from here. Thus, I got to the airport some minutes after seven. Only some minutes, 4 or 5 maybe, but enough for the check-in to be closed. Damn computers, when they decide something, it seems that all the humans can do is obbey. who is the tool and who the user then? I couldn’t get my plane. It doesn’t matter who I got to ask for help, no one seemed to be able to do anything. Computers rule. Check-in is already closed. I’m not the oly one who missed the plane anyway. I’d really love to be able to be in Lyon now. Next time, I just won’t trust public transport at all, I won’t try to get a plane that early and I will do the check-in through the Internet. Have a lot of fun all of you joining JDLL.

The second movement of Mozart’s String Quartet in C «Dissonance», K. 465 has an interesting two bars where it appears that the violin theme is accidentally missing. As highlighted in red on the score, the violin part is devoid of the crotchet/four quavers that the other parts present - the other “missing” bar is at this section’s mirror in the sub-dominant, bar 75. Some printed editions have added the missing notes, claiming that Mozart made a mistake (or overruling him on musical grounds!), but it seems bizarre that Mozart—who even likened the composition of the “Haydn” quartets to a long, labourious fatigue (”lungha e laboriosa fatica“)—would make such an error. The preface to the Neue Mozart Ausgabe claims:

It is unthinkable that Mozart [could] have made a mistake or from absent-mindedness made a slip of the pen no less than four times in such an obvious and important detail.

(The preparation of the original first edition, which Mozart himself presided over, presumably accounts for the other two.) At least one scholar has claimed that the notes aren’t missing at all; they are simply contained within the cello’s notes. I don’t find this explanation particulary satisfying. The NMA goes further by stressing its wish that the original version should be used in performances, which seems to suggest that adding the notes is (or at least, was) a common practice, but I cannot find many recordings with the added notes: my Leipziger, Emerson, Alban Berg, Ysa e, Italiano and Amadeus Quartet interpretations all omit the notes, whilst only the Hagen Quartet add them.

...at least, in my opinion (which is obviously biased, let's admit that). There are several reasons for this loss:

Strategic choices: Bernard Laporte deliberately chose to use the same tactic against England and New Zealand. However, the tactic that worked against New Zealand failed against England, mostly because that tactic...is just the play of the English team. So, as we say in French, "les Bleus" entered the game of England and therefore made the task easier for them.
The choice of players. This is the result of the previous item. Reconducting the entire team that won against NZ was IMHO a bad idea. It is not a scoop that I dislike Rapha l Ibanez or Fabien Pelous and I would rather have seen the punch of Dimitri Szarzewski or, for once, S bastien Chabal beginning the game (actually, he entered pretty quickly because of Pelous' injury).
Several failures to occupy the English side with kick play. In many occasions, kick shots by either Beauxis, Traille or Jauzion missed their target. This is IMHO the main reason for the defeat.
Too many drop tentatives. When they were in the English side, why the hell did they choose to kick drops several times instead of just play. That sounds like a fear of not scoring and the French lost several key balls with that miserable tactic.
Too many lost balls in mauls. That is not only a failure by the French but also good defense play by the English team.
Discipline. It happened only once (high tackle by Dimitri Szarzewski) but that was *the* moment where no error was possible and it offered an ideal penalty kick to Wilkinson.
Bad luck. We can't rule out that one. The difference is 5 points: this stupid 1st minute try....and the missing 50 centimeters on Chabal's action in the last minutes.

Some *bad* reasons for the English victory:

Johnny Wilkinson influence: this is untrue. Apart from his drop goal, he missed penalty kick tentatives and his kick play was not brilliant during the same. I much preferred Andy Gomarsall during that game.
Scrums. They didn't crush the French scrum which is one of the satisfactions of that game. The English team even had to cheat slightly during 1st half time to win their scrums.
Good play. No. Watching the game again, there was as many bad choices in the English side than in French side. I can only enhance their defense, not their attack. So, as I wrote immediately after the game, that one was one of the worst matches in the entire WC.
The referee. Before the game, some French newspapers were doubting of Mr. Kaplan because he's South-African (funnily, some English newspapers were also doubting because he was supposed to advantage England's opponents in the past). He was indeed perfect. Anyway, everybody knows I never blame referees in sports.

Thanks Russell for the explanations on the execmem bits.

Now I am trying to go further and set up my system to really work with SELinux enabled because, although the promise of the targeted policy is to allow you to do your job mostly as you did before, that "mostly" has a really wide meaning, more than you'd think you bargained for.

Examples from my laptop: hald does not start by default (various denials), resolvconf is denied some getattr operations on tmpfs, hald-addon-dell-backlight is denied access to some character device I just know it should have access to, etc. Of course, this means that automatic mounting does not work anymore and there is a decrease in usability just because of the "mostly" part.

I am sure that some of the denials are correct (see the execstack stuff or the memexec), but there are cases where this "mostly" is stretched way too much. IMHO, desktop installs should suffer no restrictions when using the targeted policy (and I mean "no restrictions that would make my system less than it was before enabling and enforcing SELinux").

But let me tell a reason why SELinux sucks without any help from others:

The interface sucks big time.

Probably there is are good reasons, but if you need to create and use a new policy based on the denials found in the logs, you need to use no less than 4 (four) different tools with the right incantation (although you can use just 2, if you don't want to customize the rules[1]):

audit2allow -m local -l
checkmodule -M -m -o local.mod local.te
semodule_package -o local.pp -m local.mod
semodule -i local.pp

In the end, you still hit the bad interface:

bounty:~# semodule -i local.pp
libsepol.check_assertion_helper: assertion on line 0 violated by allow hald_t memory_device_t:chr_file read ;
libsepol.check_assertions: 1 assertion violations occured
libsemanage.semanage_expand_sandbox: Expand module failed
semodule: Failed!

Now what? That error message doesn't tell me much, except that some assertion failed. But where does the assertion come from? Why is it an assertion? What is bad about that line?
To me that really looks like an internal error or an inconsistency in what the SELinux tools generate.

[1] still, why advertise the longer path as the Fedora FAQ does? I would have done it the other way around.

There's this thing going around on Facebook suggesting you are smart if you can read a paragaph where the middle letters of the words are scrambled. Everything old is new again, since I remember reading about this in 2003. As far as I remember being able to read the scrambled text was not related to intelligence in any way. Anyway, as a Friday afternoon distraction I wrote a little javascript to scramble text. Scramble It Ahh, the good-olde Fisher-Yates shuffle algorithm, friend of the first year tutorial!

Introduction I gave a joint presentation with Manoj at Debconf7 about using distributed version control for Debian packaging, and I volunteered to do an on-line workshop about using Git for the task, so it's about time that I should know how to use Git for Debian packaging, but it turns out that I don't. Or well, didn't. After I made a pretty good mess out of the mdadm packaging repository (which is not a big problem as it's just ugly history up to the point when I start to get it right), I decided to get down with the topic and figure it out once and for all. I am writing this post as I put the pieces together. It's been cooking for a week, simply so I could gather enough feedback. I am aware that Git is not exactly a showcase of usability, so I took some extra care to not add to the confusion. It may be the first post in a series, because this time, I am just covering the case of mdadm, for which upstream also uses Git and where I am the only maintainer, and I shall pretend that I am importing mdadm to version control for the first time, so there won't be any history juggling. Future posts could well include tracking Subversion repositories with git-svn, and importing packages previously tracked therewith, but this ain't no promise! (well, that last post is already being drafted, but far from finished; you have been warned!) I realise that git-buildpackage exists, but imposes a rather strict branch layout and tagging scheme, which I don't want to adhere to. And gitpkg (Romain blogged about it recently), deserves another look since, according to its author, it does not impose anything on its user. But in any case, before using such tools (and possibly extending them to allow for other layouts), I'd really rather have done it by hand a couple of times to get the hang of it and find out where the culprits lie. Now, enough of the talking, just one last thing: I expect this blog post to change quite a bit as I get feedback. Changes shall be highlighted in bold typeface.

Setting up the infrastructure First, we prepare a shared repository on git.debian.org for later use (using collab-maint for illustration purposes), download the Debian source package we want to import (version 2.6.3+200709292116+4450e59-3 at time of writing, but I pretend it's -2 because we shall create -3 further down ), set up a local repository, and link it to the remote repository. Note that there are other ways to set up the infrastructure, but this happens to be the one I prefer, even though it's slightly more complicated:

$ ssh alioth
$ cd /git/collab-maint
$ ./setup-repository pkg-mdadm mdadm Debian packaging
$ exit
$ apt-get source --download-only mdadm
$ mkdir mdadm && cd mdadm
$ git init
$ git remote add origin ssh://git.debian.org/git/collab-maint/pkg-mdadm
$ git config branch.master.merge refs/heads/master

Now we can use git-pull and git-push, except the remote repository is empty and we can't pull from there yet. We'll save that for later. Instead, we tell the repository about upstream's Git repository. I am giving you the git.debian.org URL though, simply because I don't want upstream repository (which lives on an ADSL line) hammered in response to this blog post:

$ git remote add upstream-repo git://git.debian.org/git/pkg-mdadm/mdadm

Since we're using the upstream branch of the pkg-mdadm repository as source (and don't want all the other mess I created in that repository), we'll first limit the set of branches to be fetched (I could have used the -t option in the above git-remote command, but I prefer to make it explicit that we're doing things slightly differently to protect upstream's ADSL line).

$ git config remote.upstream-repo.fetch \
    +refs/heads/upstream:refs/remotes/upstream-repo/upstream

And now we can pull down upstream's history and create a local branch off it. The "no common commits" warning can be safely ignored since we don't have any commits at all at that point (so there can't be any in common between the local and remote repository), but we know what we're doing, even to the point that we can forcefully give birth to a branch, which is because we do not have a HEAD commit yet (our repository is still empty):

$ git fetch upstream-repo
warning: no common commits
[ ]
  # in the real world, we'd be branching off upstream-repo/master
$ git checkout -b upstream upstream-repo/upstream
warning: You appear to be on a branch yet to be born.
warning: Forcing checkout of upstream-repo/upstream.
Branch upstream set up to track remote branch
  refs/remotes/upstream-repo/upstream.
$ git branch
* upstream
$ ls   wc -l
77

Importing the Debian package Now it's time to import Debian's diff.gz remember how I pretend to use version control for package maintenance for the first time. Oh, and sorry about the messy file names, but I decided it's best to stick with real data in case you are playing along: Since we're applying the diff against version 2.6.3+200709292116+4450e59, we ought to make sure to have the repository at the same state. Upstream never "released" that version, but I encoded the commit ID of the tip when I snapshotted it: 4450e59, so we branch off there. Since we are actually tracking the git.debian.org pkg-mdadm repository instead of upstream, you can use the tag I made. Otherwise you could consider tagging yourself:

$ #git tag -s mdadm-2.6.3+200709292116+4450e59 4450e59
$ git checkout -b master mdadm-2.6.3+200709292116+4450e59
$ zcat ../mdadm_2.6.3+200709292116+4450e59-2.diff.gz   git apply

The local tree is now "debianised", but Git does not know about the new and changed files, which you can verify with git-status. We will split the changes made by Debian's diff.gz across several branches.

The idea of feature branches We could just create a debian branch, commit all changes made by the diff.gz there, and be done with it. However, we might want to keep certain aspects of Debianisation separate, and the way to do that is with feature branches (also known as "topic" branches). For the sake of this demonstration, let's create the following four branches in addition to the master branch, which holds the standard Debian files, such as debian/changelog, debian/control, and debian/rules:

upstream-patches will includes patches against the upstream code, which I submit for upstream inclusion.
deb/conffile-location makes /etc/mdadm/mdadm.conf the default over /etc/mdadm.conf and is Debian-specific (thus the deb/ prefix).
deb/initramfs includes the initramfs hook and script, which I want to treat separately but not submit upstream.
deb/docs similarly includes Debian-only documentation I add to the package as a service to Debian users.

If you're importing a Debian package using dpatch, you might want to convert every dpatch into a single branch, or at least collect logical units into separate branches. Up to you. For now, our simple example suffices. Keep in mind that it's easy to merge two branch and less trivial to split one into two. Why? Well, good question. As you will see further down, the separation between master and deb/initramfs actually makes things more complicated when you are working on an issue spanning across both. However, feature branches also bring a whole lot of flexibility. For instance, with the above separation, I could easily create mdadm packages without initramfs integration (see #434934), a disk-space-conscious distribution like grml might prefer to leave out the extra documentation, and maybe another derivative doesn't like the fact that the configuration file is in a different place from upstream. With feature branches, all these issues could be easily addressed by leaving out unwanted branches from the merge into the integration/build branch (see further down). Whether you use feature branches, and how many, or whether you'd like to only separate upstream and Debian stuff is entirely up to you. For the purpose of demonstration, I'll go the more complicated way.

Setting up feature branches So let's commit the individual files to the branches. The output of the git-checkout command shows modified files that have not been committed yet (which I trim after the first example); Git keeps these across checkouts/branch changes. Note that the ./debian/ directory does not show up as Git does not know about it yet (git-status will tell you that it's untracked, or rather: contains untracked files since Git does not track directories at all):

$ git checkout -b upstream-patches mdadm-2.6.3+200709292116+4450e59
M Makefile
M ReadMe.c
M mdadm.8
M mdadm.conf.5
M mdassemble.8
M super1.c
$ git add super1.c     #444682
$ git commit -s
  # i now branch off master, but that's the same as 4450e59 actually
  # i just do it so i can make this point 
$ git checkout -b deb/conffile-location master
$ git add Makefile ReadMe.c mdadm.8 mdadm.conf.5 mdassemble.8
$ git commit -s
$ git checkout -b deb/initramfs master
$ git add debian/initramfs/*
$ git commit -s
$ git checkout -b deb/docs master
$ git add RAID5_versus_RAID10.txt md.txt rootraiddoc.97.html
$ git commit -s
  # and finally, the ./debian/ directory:
$ git checkout master
$ chmod +x debian/rules
$ git add debian
$ git commit -s
$ git branch
  deb/conffile-location
  deb/docs
* master
  upstream
  upstream-patches

At this time, we push our work so it won't get lost if, at this moment, aliens land on the house, or any other completely plausible event of apocalypse descends upon you. We'll push our work to git.debian.org (the origin, which is the default destination and thus needs not be specified) by using git-push --all, which conveniently pushes all local branches, thus including the upstream code; you may not want to push the upstream code, but I prefer it since it makes it easier to work with the repository, and since most of the objects are needed for the other branches anyway after all, we branched off the upstream branch. Specifying --tags instead of --all pushes tags instead of heads (branches); you couldn't have guessed that! See this thread if you (rightfully) think that one should be able to do this in a single command (which is not git push refs/heads/* refs/tags/*)

$ git push --all
$ git push --tags

Done. Well, almost

Building the package (theory) Let's build the package. There seem to be two (sensible) ways we could do this, considering that we have to integrate (merge) the branches we just created, before we fire off the building scripts:

by using a temporary (or "throw-away") branch off upstream, where we integrate all the branches we have just created, build the package, tag our master branch (it contains debian/changelog), and remove the temporary branch. When a new package needs to be built, we repeat the process.
by using a long-living integration branch off upstream, into which we merge all our branches, tag the branch, and build the package off the tag. When a new package comes around, we re-merge our branches, tag, and build.

Both approaches have a certain appeal to me, but I settled for the second, for two reasons, the first of which leads to the second:

When I upload a package to the Debian archive, I want to create a tag which captures the exact state of the tree from which the package was built, for posterity (I will return to this point later). Since the throw-away branches are not designed to persist and are not uploaded to the archive, tagging the merging commit makes no sense. Thus, the only way to properly identify a source tree across all involved branches would be to run git-tag $branch/$tagname $branch for each branch, which is purely semantic and will get messy sooner or later.
As a result of the above: when Debian makes a new stable release, I would like to create a branch corresponding to the package in the stable archive at the time, for security and other proposed updates. I could rename my throw-away branch, if it still existed, or I could create a new branch and merge all other branches, using the (semantic) tags, but that seems rather unfavourable.

So instead, I use a long-living integration branch, notoriously tag the merge commits which produced the tree from which I built the package I uploaded, and when a certain version ends up in a stable Debian release, I create a maintenance branch off the one, single tag which corresponds to the very version of the package distributed as part of the Debian release. So much for the theory. Let's build, already!

Building the package (practise) So we need a long-living integration branch, and that's easier done than said:

$ git checkout -b build mdadm-2.6.3+200709292116+4450e59

Now we're ready to build, and the following procedure should really be automated. I thus write it like a script, called poor-mans-gitbuild, which takes as optional argument the name of the (upstream) tag to use, defaulting to upstream (the tip):

#!/bin/sh
set -eu
git checkout master
debver=$(dpkg-parsechangelog   sed -ne 's,Version: ,,p')
git checkout build
git merge $ 1:-upstream 
git merge upstream-patches
git merge master
for b in $(git for-each-ref --format='%(refname)' refs/heads/deb/*); do
  git merge -- $b
done
git tag -s debian/$debver
debuild   # will ignore .git automatically
git checkout master

Note how we are merging each branch in turn, instead of using the octopus merge strategy (which would create a commit with more than two parents) for reasons outlined in this post. An octopus-merge would actually work in our situation, but it will not always work, so better safe than sorry (although you could still achieve the same result). If you discover during the build that you forgot something, or the build script failed to run, just remove the tag, undo the merges, checkout the branch to which you need to commit to fix the issue, and then repeat the above build process:

$ git tag -d debian/$debver
$ git checkout build
$ git reset --hard upstream
$ git checkout master
$ editor debian/rules    # or whatever
$ git add debian/rules
$ git commit -s
$ poor-mans-gitbuild

Before you upload, it's a good idea to invoke gitk --all and verify that all goes according to plan:

screenshot of gitk after the above steps

When you're done and the package has been uploaded, push your work to git.debian.org, as before. Instead of using --all and --tags, I now specify exactly which refs to push. This is probably a good habit to get into to prevent publishing unwanted refs:

$ git push origin build tag debian/2.6.3+200709292116+4450e59-3

Now take your dog for a walk, or play outside, or do something else not involving a computer or entertainment device.

Uploading a new Debian version If you are as lucky as I am, the package you uploaded still has a bug in the upstream code and someone else fixes it before upstream releases a new version, then you might be in the position to release a new Debian version. Or maybe you just need to make some Debian-specific changes against the same upstream version. I'll let the commands speak for themselves:

$ git checkout upstream-patches
$ git-apply < patch-from-lunar.diff   #444682 again
$ git commit --author 'J r my Bobbio <lunar@debian.org>' -s
  # this should also be automated, see below
$ git checkout master
$ dch -i
$ dpkg-parsechangelog   sed -ne 's,Version: ,,p'
2.6.3+200709292116+4450e59-3
$ git commit -s debian/changelog
$ poor-mans-gitbuild
$ git push
$ git push origin tag debian/2.6.3+200709292116+4450e59-3

That first git-push may require a short explanation: without any arguments, git-push updates only the intersection of local and remote branches, so it would never push a new local branch (such as build above), but it updates all existing ones; thus, you cannot inadvertedly publish a local branch. Tags still need to be published explicitly.

Hacking on the software Imagine: on a rainy Saturday afternoon you get bored and decide to implement a better way to tell mdadm when to start which array. Since you're a genius, it'll take you only a day, but you do make mistakes here and there, so what could be better than to use version control? However, rather than having a branch that will live forever, you are just creating a local branch, which you will not publish. When you are done, you'll feed your work back into the existing branches. Git makes branching really easy and as you may have spotted, the poor-mans-gitbuild script reserves an entire branch namespace for people like you:

$ git checkout -b tmp/start-arrays-rework master

Unfortunately (or fortunately), fixing this issue will require work on two branches, since the initramfs script and hook are maintained in a separate branch. There are (again) two ways in which we can (sensibly) approach this:

create two separate, temporary branches, and switch between them as you work.
merge both into the temporary branch and later cherry-pick the commits into the appropriate branches.

I am undecided on this, but maybe the best would be a combination: merge both into a temporary branch and later cherry-pick the commits into two additional, temporary branches until you got it right, and then fast-forward the official branches to their tips:

$ git merge master deb/initramfs
$ editor debian/mdadm-raid                     #  
$ git commit -s debian/mdadm-raid
$ editor debian/initramfs/script.local-top     #  
$ git commit -s debian/initramfs/script.local-top
[many hours of iteration pass ]
[  until you are done]
$ git checkout -b tmp/start-arrays-rework-init master
  # for each commit $c in tmp/start-arrays-rework
  # applicable to the master branch:
$ git cherry-pick $c
$ git checkout -b tmp/start-arrays-rework-initramfs deb/initramfs
  # for each commit $c in tmp/start-arrays-rework
  # applicable to the deb/initramfs branch:
$ git cherry-pick $c

This is assuming that all your commits are logical units. If you find several commits which would better be bundled together into a single commit, this is the time to do it:

$ git cherry-pick --no-commit <commit7>
$ git cherry-pick --no-commit <commit4>
$ git cherry-pick --no-commit <commit5>
$ git commit -s

Before we now merge this into the official branches, let me briefly intervene and introduce the concept of a fast-forward. Git will "fast-forward" a branch to a new tip if it decides that no merge is needed. In the above example, we branched a temporary branch (T) off the tip of an official branch (O) and then worked on the temporary one. If we now merge the temporary one into the official one, Git determines that it can actually squash the ancestry into a single line and push the official branch tip to the same ref as the temporary branch tip. In cheap (poor man's), ASCII notation:

- - - O             >> merge T >>     - - - = - - OT
         - - T      >>  into O >>

This works because no new commits have been made on top of O (if there would be any, we might be able to rebase, but let's not go there quite yet; rebasing is how you shoot yourself in the foot with Git). Thus we can simply do the following:

$ git checkout deb/initramfs
$ git merge tmp/start-arrays-rework-initramfs
$ git checkout master
$ git merge tmp/start-arrays-rework-init

and test/build/push the result. Or well, since you are not an mdadm maintainer (We^W I have open job positions! Applications welcome!), you'll want to submit your work as patches via email:

$ git format-patch -s -M origin/master

This will create a number of files in the current directory, one corresponding for each commit you made since origin/master. Assuming each commit is a logical unit, you can now submit these to an email address. The --compose option lets you write an introductory message, which is optional:

$ git send-email --compose --to your@email.address <file1> <file2> < >

Once you've verified that everything is alright, swap your email address for the bug number (or the pkg-mdadm-devel list address). Thanks (in advance) for your contribution! Of course, you may also be working on a feature that you want to go upstream, in which case you'd probably branch off upstream-patches (if it depends on a patch not yet in upstream's repository), or upstream (if it does not):

$ git checkout -b tmp/cool-feature upstream
[ ]

when a new upstream version comes around After a while, upstream may have integrated your patches, in addition to various other changes, to give birth to mdadm-2.6.4. We thus first fetch all the new refs and merge them into our upstream branch:

$ git fetch upstream-repo
$ git checkout upstream
$ git merge upstream-repo/master

we could just as well have executed git-pull, which with the default configuration would have done the same; however, I prefer to separate the process into fetching and merging. Now comes the point when many Git people think about rebasing. And in fact, rebasing is exactly what you should be doing, iff you're still working on an unpublished branch, such as the previous tmp/cool-feature off upstream. By rebasing your branch onto the updated upstream branch, you are making sure that your patch will apply cleanly when upstream tries it, because potential merge conflicts would be handled by you as part of the rebase, rather than by upstream:

$ git checkout tmp/cool-feature
$ git rebase upstream

What rebasing does is quite simple actually: it takes every commit you made since you branched off the parent branch and records the diff and commit message. Then, for each diff/commit_message pair, it creates a new commit on top of the new parent branch tip, thus rewrites history, and orphans all your original commits. Thus, you should only do this if your branch has never been published or else you would leave people who cloned from your published branch with orphans.

If this still does not make sense, try it out: create a (source) repository, make a commit (with a meaningful commit message), branch B off the tip, make a commit on top of B (with a meaningful message), clone that repository and return to the source repository. There, checkout the master, make a commit (with a ), checkout B, rebase it onto the tip of master, make a commit (with a ), and now git-pull from the clone; use gitk to figure out what's going on.

So you should almost never rebase a published branch, and since all your branches outside of the tmp/* namespace are published on git.debian.org, you should not rebase those. But then again, Pierre actually rebases a published branch in his workflow, and he does so with reason: his patches branch is just a collection of branches to go upstream, from which upstream cherry-picks or which upstream merges, but which no one tracks (or should be tracking). But we can't (or at least will not at this point) do this for our feature branches (though we could treat upstream-patches that way), so we have to merge. At first, it suffices to merge the new upstream into the long-living build branch, and to call poor-mans-gitbuild, but if you run into merge conflicts or find that upstream's changes affect the functionality contained in your feature branches, you need to actually fix those. For instance, let's say that upstream started providing md.txt (which I previously provided in the deb/docs branch), then I need to fix that branch:

$ git checkout deb/docs
$ git rm md.txt
$ git commit -s

That was easy, since I could evade the conflict. But what if upstream made a change to Makefile, which got in the way with my configuration file location change? Then I'd have to merge upstream into deb/conffile-location, resolve the conflicts, and commit the change:

$ git checkout deb/conffile-location
$ git merge upstream
CONFLICT!
$ git-mergetool
$ git commit -s

When all conflicts have been resolved, I can prepare a new release, as before:

$ git checkout master
$ dch -i
$ dpkg-parsechangelog   sed -ne 's,Version: ,,p'
2.6.3+200709292116+4450e59-3
# git commit -s debian/changelog
$ poor-mans-gitbuild
# git push
$ git push origin tag debian/2.6.3+200709292116+4450e59-3

Note that Git often appears smart about commits that percolated upstream: since upstream included the two commits in upstream-patches in his 2.6.4 release, my upstream-patches branch got effectively annihilated, and Git was smart enough to figure that out without a conflict. But before you rejoice, let it be told that this does not always work.

Creating and using a maintenance branch Let's say Debian "lenny" is released with mdadm 2.7.6-1, then:

$ git checkout -b maint/lenny debian/2.7.6-1

You might do this to celebrate the release, or you may wait until the need arises. We've already left the domain of reality ("lenny" is not yet released), so the following is just theory. Now, assume that a security bug is found in mdadm 2.7.6 after "lenny" was released. Upstream is already on mdadm 2.7.8 and commits deadbeef and c0ffee fix the security issue, then you'd cherry-pick them into the maint/lenny branch:

$ git checkout upstream
$ git pull
$ git checkout maint/lenny
$ git cherry-pick deadbeef
$ git cherry-pick c0ffee

If there are no merge conflicts (which you'd resolve with git-mergetool), we can just go ahead to prepare the new package:

$ dch -i
$ dpkg-parsechangelog   sed -ne 's,Version: ,,p'
2.7.6-1lenny1
$ git commit -s debian/changelog
$ poor-mans-gitbuild
$ git push origin maint/lenny
$ git push origin tag debian/2.7.6-1lenny1

Future directions It should be trivial to create the Debian source package directly from the repository, and in fact, in response to a recent blog post of mine on the dispensability of pristine upstream tarballs, two people showed me their scripts to do it. My post also caused Joey Hess to clarify his position on pristine tarballs, before he went out to implement dpkg-source v3. This looks very promising. Yet, as Romain argues, there are benefits with simple patch management systems. Exciting times ahead! In addition to creating source packages from version control, a couple of other ideas have been around for a while:

create debian/changelog from commit log summaries when you merge into the build branch.
integrate version control with the BTS, bidirectionally:
- given a bug report, create a temporary branch and apply any patches found in the bug report.
- upon merging the temporary branch back into the feature branch it modifies, generate a patch, send it to the BTS and tag the bug report + pending patch.

And I am sure there are more. If you have any, I'd be interested to hear about them!

Wrapping up I hope this post was useful. Thank you for reading to the end, this was probably my longest blog post ever. I want to thank Pierre Habouzit, Johannes Schindelin, and all the others on the #git/freenode IRC channel for their tutelage. Thanks also to Manoj Srivastava, whose pioneering work on packaging with GNU arch got me started on most of the concepts I use in the above. And of course, the members of the the vcs-pkg mailing list for the various discussions on this subject, especially those who participated in the thread leading up to this post. Finally, thanks to Linus and Junio for Git and the continuously outstanding high level of support they give. If you are interested in the topic of using version control for distro packaging, I invite you to join the vcs-pkg mailing list and/or the #vcs-pkg/irc.oftc.net IRC channel. NP: Aphex Twin: Selected Ambient Works, Volume 2 (at least when I started writing )

Problem present in openscenegraph 1.9.x and 2.x series: bug #425837. Looking at the build logs of unstable or experimental for the hppa architecture, there's a linking problem around. Shorten error message:

/usr/bin/g++ -fPIC -shared -Wl,-soname,libosgUtil.so.6 
    -o ../../lib/libosgUtil.so.6.0.0 "many *.o"
    -L/build/buildd/openscenegraph-2.2/build/osg/lib  -losg
    -lOpenThreads -lGLU -lGL -lSM -lICE -lX11 -lXext -lm -lpthread
    -Wl,-rpath,/build/buildd/openscenegraph-2.2/build/osg/lib 
/usr/bin/ld: CMakeFiles/osgUtil.dir/Optimizer.o(.text._ZNSt8_Rb_tree 
     IjSt4pairIKjSt3setIPN3osg9Texture2DESt4lessIS5_ESaIS5_EEESt10_S 
     elect1stISA_ES6_IjESaISA_EE9_M_insertEPSt18_Rb_tree_node_baseSH 
     _RKSA_("unmangled name"): cannot reach 0000393f__ZNSt10_Select1 
     stISt4pairIKjSt3setIPN3osg9Texture2DESt4lessIS5_ESaIS5_EEEEC1Ev+0,
    recompile with -ffunction-sections
/usr/bin/ld: final link failed: Bad value

First, just in case demangling is needed (that's not the case, but imagine), there's a nice tool to get the signature back (output skipped, it is very huge, that C++!). That's c++filt from the binutils package:

c++filt _ZNSt8_Rb_treeIjSt4pairIKjSt3setIPN3osg9Texture2DESt4lessIS\
5_ESaIS5_EEESt10_Select1stISA_ES6_IjESaISA_EE9_M_insertEPSt18_Rb_tr\
ee_node_baseSH_RKSA_

Then, one tends to trust the compiler. Added the following to the cmake command in debian/rules:

-D CMAKE_CXX_CFLAGS='-ffunction-sections'

But not much better. Oh wait, there's a quite interesting string in the build log now: cannot handle R_PARISC_PCREL17F, which looks like quite arch-dependant and perfect to be searched through the internets. Some old bugreports came out: bug #334959 and bug #416496 and the solution: use -mlong-calls. From http://www.projectblackdog.com/3rdParty/gcc/HPPA-Options.html, the relevant parts: Generate code that uses long call sequences. This ensures that a call is always able to reach linker generated stubs. [ ] It is normally not desirable to use this option as it will degrade performance. However, it may be useful in large applications, particularly when partial linking is used to build the application. It looks like it might do the job. And indeed it does. Conditionally added for the hppa architecture. And -ffunction-sections isn't even needed. Thanks to the http://www.parisc-linux.org/ folks for the access to their cluster, which really helped finding out a fix very quickly! And it looks like the number of blockers for the transition to testing has been reduced. \o/

In a comment on my previous post about SE Linux and worms/trojans [1] a user enquired about which methods of gaining local root are prevented by SE Linux. A local exploit is one that can not be run remotely. An attack via TCP or UDP is generally considered a remote exploit - even though in some cases you might configure a daemon to only bind to localhost (in which case the TCP or UDP attack would only work locally). When compromising a machine it’s not uncommon for a local exploit to be used after a remote exploit or social engineering has been used to gain non-root privileges. The two most common types of local root exploit seem to be those which attack the kernel and those which attack a SETUID process. For a non-SE Linux system it usually doesn’t matter much how the local exploit is run. But a SE Linux system in a default configuration will be running the Targeted policy that has almost no restrictions on a user shell session. So an attacker who wants to escalate their privileges from local user to local root on a typical SE Linux system has a significant benefit in starting from a user account instead of starting from a web server or other system process. In the SE Linux model access is granted to a domain, and the domain which is used for a process is determined by the policy based on the domain of the parent process and the labelling of the executable. Some domains are not permitted to transition to any other domains, such as the domain dhcpd_t (used for a DHCP server). Other domains are only permitted to transition to a small set of domains, for example the domain httpd_t (used for a web server) can only transition to a small number of domains none of which has any significant privileges. On a machine running without SE Linux a compromise of a DHCP server is game-over as the server runs as root. A compromise of a daemon such as Apache on a machine without SE Linux gives unrestricted access to run applications on the systems - if a SETUID-root program has a security flaw then you lose. The same bug in a SETUID program on a machine running SE Linux is not fatal because SE Linux will prevent the program from doing anything that it’s parent could not do - even if an attacker made Apache run a buggy SETUID program the broken program in question could do nothing other than what Apache is normally permitted to do. A security flaw in a SETUID-root program on a SE Linux system can still be exploited by a local user (someone who has logged in) when running the Targeted policy. When running the Strict or MLS policies many such vulnerabilities will not be exploitable by local users (for example exploiting PING would only permit network access). As a rule of thumb you should consider that a kernel security flaw will make it possible to bypass all other security features. However there are some situations where SE Linux can prevent local exploits. One example is a bug discovered in July 2006 which allowed the creation of SETUID files under /proc [2], the targeted policy of SE Linux prevented this from working. Another example is CVE-2003-0127 [3] (a kernel security flaw that was exploited by triggering a module load and then exploiting a race condition in the kernel module load process), the commonly used exploit for this did not work on a SE Linux system because the user process was not permitted the socket access used to trigger the module load (it is believed that an attacker could have written a version of the exploit to work on a SE Linux system - but AFAIK no-one ever did so).

[1] http://etbe.coker.com.au/2007/10/08/selinux-storm/

[2] http://lwn.net/Articles/191954/

[3] http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2003-0127

Share This

I'm sure I am only have legs and a brain so that I can keep my stomach happy. Food rules my life. I don't understand people that can miss breakfast; my stomach would be moaning at me if I did. It does have a good side though as it means I really enjoy food and I enjoy cooking it too. Since I have some more time on my hands these days, I've been experimenting a little more and trying some more things out so I thought I should post about them. Bread maker I asked for a bread maker for my birthday. I worked from home and ate bread all the time for lunch but it always went off quickly and supermarket bread isn't always amazing. The machine has been great. The even rapid bake loaves it churns out in just two hours are great. It has a timer so you can put a loaf on over night and come down to lovely fresh bread smells. The dry ingredients are easy to keep around and you just bung them all in the pan, hit a few buttons and later on a loaf appears as if by magic. I especially like the granary loaves especially if I add a little rye flour but all the standard recipes seem good. The French bread one seems to give the right texture and taste but it's odd eating it in a loaf shape so I might have to make that one as dough and then shape and bake it separately one day. I didn't like the olive and passata loaf; it came out an interesting orange colour flecked with the chopped olives but didn't really taste of tomatoes or olives. There's a tomato foccacia that uses sundried tomatoes that I might try out though. There's a croissant recipe which seems to be a quantity of French dough that you put butter in, wrap around butter, and then more butter. How can it possibly go wrong? I need a free morning to try it but I think I will soon. Basically if you don't have a bread machine you probably want one. Pasta I watched a series of cooking programmes on BBC (Kitchen Criminals) and they seemed to be making pasta from scratch every two minutes so I was intrigued to try it. I mentioned it to my Mum and she had a machine that she'd never used that I borrowed. Sadly it's not quite as easy as it looks after TV editing. Basically the first bit is really easy: whizzing flour, olive oil, an egg and some salt in a food processor, kneading it a bit and making it into a ball. Bung it in the fridge to rest and then the fun begins of trying to make it into a flat sheet. Lots more flour (in fact my kitchen was covered with it) and lots of patience and many attempts got me something that was vaguely what I wanted. Every time I got close it would stick together or it would rip or it would go in at an angle and I didn't have enough hands to fix all these problems at once. Even using smaller amounts of the dough is tricky. I was making ravioli the first time and it was nice but took lots of time and I didn't feel it was entirely worth the effort. I tried just simple tagliatelle last night but again it took a long time and you end up thinking that you could have just bought fresh pasta in a bag and be grabbing a handful of it instead of spending 40 minutes faffing just to get some strands. It might be worth it if you had some amazing idea for ravioli filling but I think I'll give Mum her machine back and not buy one. Cheescake I've always really loved Pizza Express's cheesecake but never really understood how they did it especially with the texture at the edges. I'd always thought cheesecakes were just an assembly job really; my standard one is lime and mascarpone and icing sugar and it is tasty but not the same. Chatting to various people it seems the Pizza Express one is baked so I tried a recipe from a book Mum had lying around that I was flicking through. It was a chocolate baked cheesecake and it sort of worked but was almost a little too chocolately and didn't have enough other interest. It probably wants some zest or some stem ginger or something through it. It's a terrible hardship but I think I might have to make another one to perfect it. Plans Like I said, I want to try croissants and I've always been looking for a good recipe for gingerbread; I really love ginger as a flavour. I never really cook with pastry much so I should perhaps try some pies or similar whilst I'm not working and have time to do these things. Any other suggestions welcome.

MJ, each French NPO is free to establish its own regulation rules, which may include a differential voting system if they want to. That should of course respect the French law (for instance, no discrimination based on sex, religion or any other genetic factor) but, as far as I know, nothing prevents deciding that "personnes morales" (ie non physical people) have a higher or lower vote than others. However, this is certainly not common, but many NPO simply don't have non physical members anyway (some don't authorize non physical people to be members). For those who care, Debian France (which is a NPO, but not regulated by 1901 law, but by the special NPO law for Alsace and Lorraine, inherited from the 1810-1918 years) admits physical and "moral" members with an equal voting right. The only special case is a rule that allows Debian developers to have a veto right on any decision of the organisation. See that official regulation rules of Debian France for details (French only). PS: Debian France is currently inactive, due to a longstanding dispute with the French Tax collecting organisation (Direction des Imp ts) who initially refused to consider us as a non profit organisation...because "we sell an operating system" (doh!).

Search Results: "rul"

1 November 2007

Russell Coker: Load Average

John Goerzen: So long, Vim. I'm returning to Emacs

31 October 2007

Daniel Burrows: Mail

29 October 2007

Adrian von Bidder: DPL Campaigning

28 October 2007

Tore S. Bekkedal: Play! - A Video Game Symphony

27 October 2007

Russell Coker: Senator Online

23 October 2007

MJ Ray: the Cooperative Group Half-Yearly Meeting: Q+A

22 October 2007

Holger Levsen: Small report from the meeting in Extremadura

21 October 2007

Russell Coker: Open Hardware - What Would Change

20 October 2007

Adam Rosi-Kessel: Multitasking and Classical Music

Miriam Ruiz: Public transport is not reliable

17 October 2007

Chris Lamb: Missing notes in the Mozart s Dissonance quartet K. 465?

14 October 2007

Christian Perrier: RWC: why did France lose...

12 October 2007

Eddy Petrișor: SELinux: unusable from a newbie perspective

Ian Wienand: Scrambled text

10 October 2007

Martin F. Krafft: Packaging with Git

Cyril Brulebois: Linking on hppa

9 October 2007

Russell Coker: How SE Linux Prevents Local Root Exploits

7 October 2007

Simon Huggins: My stomach and me

4 October 2007

Christian Perrier: No institutional pro-corp bias in French NPO (afaik)

Plain	Scrambled
Hello This is a paragraph that you will still be able to read even after it is scrambled, the only rule being the first and last letters remain in place. This javascript will try and be fairly smart about not moving around non-alphabetic characters too.