... because they are bouncing.

In my ongoing attempt to investigate and fix problems of our mailinglist-server, i found that we only had a working bounce-detection for lists starting debian-* minus debian-private and the three digest-lists.

So i rewrote some parts of the bounce detection:

We now have four categories of lists:

• high (30 or more mails a day)
• medium (20 or more mails a week)
• low (10 or more mails a month)
• very-low (the rest)

for these four categories we calculate a spam-rate depending on the number of distributed messages and the number of bounces in the timerange of 24 hours for 'high' lists, 7 days for 'medium' lists, 30 days for 'low' lists and 90 days for 'very-low' lists.

If the Spamrate exceeds 80% for unmoderated, and 60% for moderated lists, and bounces have been seen for more than 24 hours we kick. (these rates are subject to change, when we have implemented the warning mechanism)

So these changes implemented a bounce-detection for debian-changes-digest and today we reached the level of 60% bounces for more than 500 subscribers.

So what next?

To finish the bounce-detection, i want to implement something which tries to notify subscribers about bouncing mails. This is useless for 'hard'-bouncers, but people with some smaller problems like mail2news-gatewayers this could be helpful. I also want to implement something which sends out unsubscription notices for three weeks after the kick, so kicked out people can resubscribe after they have sorted out the problems.

Both should fire once a week per address.

What do you think?

Firstly, I haven't been following debian-devel as closely as in recent months and some things have passed me by. Secondly, I am rather pre-occupied with my own small world of Embedded Debian. One result of this is that I wasn't really paying attention to arguments over the source package format and took longer than I expected to respond to a request to take a look at tslib - a touchscreen library for embedded devices. When I did start work on it I managed to upset Joey Hess by implementing dpatch support which he had recently decried as unhelpful. I also added some extra fuel by getting myself in a two-and-eight over svn-buildpackage and it's broken bias towards native packages.
I happen to disagree with the idea that maintainers should alter upstream files directly, merely relying on the .diff.gz or some weird $RCS runes. Instead, I seek to ensure that the .diff.gz contains no changes outside the debian/ directory. I'm not happy keeping the entire upstream source in yet another RCS (for any reason) and would much rather make updates to my Debian packages by simply copying the debian/ contents into an unpacked pristine upstream source (preferably via uscan). I find it really useful to simply check out the latest CVS/SVN/$RCS, dump debian/ into place and call 'debuild' when testing various upstream development issues. Maybe this is because I have relatively few Debian packages that need debian/patches/* and certainly few that need more than a couple of patch files. It certainly has a lot to do with my embedded work where I need to make changes to the contents of debian/ to make an Emdebian package whilst retaining the same upstream code as is used in existing embedded projects like Open Embedded and Familiar. I don't want to be chasing changes through the upstream source - I want all the changes in a single place so that I can create Emdebian patches against the Debian source. (I'd rather not have patches in the Debian source at all but this is only realistic for either my own packages or those released as part of GPE for various embedded-native compatibility reasons.) I certainly do not want any Debian $RCS to be lumbered with yet another set of upstream sources that will always be out of date compared to the real upstream sources.
I am also a great fan of CDBS - much to the horror of some in Debian - primarily for one very simple reason, it cross builds so easily. Considering my workload, it should not be a surprise that the ability to support cross building without requiring any changes to the Debian package is incredibly persuasive when selecting a build system for a package of my own. debhelper really is woeful at cross building - each package needs customised cross building support painstakingly devised and tested, making script-based mass bug filing impossible. CDBS shows that with a little wrapping, debhelper can support cross building in a clean and completely non-intrusive manner.
This means I can concentrate on those changes that are actually important - the changes that provide the benefits of cross-building and justify the effort of actually cross building the entire dependency chain: the ability to fully control Debian-specifics like "Essential: yes" and to disable, modify or adapt components of any package such that the installed system has a fraction of the dependencies of the Debian version of precisely the same package. TDeb support is good too but that can be implemented against any build, it doesn't need to be a cross build.
Sadly, the end result of all this for tslib is that it will be removed from collab-maint very soon, Joey does not wish to be associated with the new version because although it includes some features which he is happy to see implemented, my use of dpatch and the hacks I had to devise to workaround svn-buildpackage do not fit with his workload patterns. Once tslib is removed from collab-maint, these hacks become redundant and I can use my normal packaging methods. Equally, his methods (in particular the lack of debian/patches/*) do not fit with my workload patterns and although I do not have as many packages as Joey, I have enough (when the 200 Emdebian source packages are also considered) to make the issue into a blocker. The new version of tslib uses the upstream SONAME and adds a -dbg package so it will spend some time in NEW. I am also likely to migrate it to CDBS once removed from collab-maint.
I suspect these issues will be revisited during discussions at Fosdem 2008 where I'll be presenting two talks - one in the Debian developer room and one in the Embedded developer room. Feel free to harangue me if you feel strongly that there is a sane way of cross building that does a better job than CDBS.

Robert A. Heinlein: Starship Troopers (Putnam, 1959)
Paul Verhoeven (director): Starship Troopers (TriStar, 1997) The trouble with movies is that they fit only a short story. You can make a terrific movie out of a short story, but I have never seen a movie made of a novel that was at the same time good and faithful – most movies made of books fail in both. The novel Starship Troopers chronicles the evolution of a high-school kid into a mobile infantry officer in a world which had seen modern democracies fail and be replaced by a veterans’ rule (veterans of also noncombat and civilian service, not just military) – Juan Rico gets in just to impress a girl, and maybe to get to vote some day, goes through boot camp, makes combat drops from orbit as a private and then as a NCO, gets opted for officer training and by the end of the book is a competent officer, helping his old drill sergeant who had found the prize. Along the way we get political sermons (now I know where John Ringo got his tendency to have characters lecture on politics!), some interesting characters and just a hint of romance. The movie paints in broad strokes and primary colors. The first half of it is in fact a fairly decent redesign of the book as a movie, though I did not like at all how much in the face the romance (and two interconnected love triangles!) was played. Johnnie Rico gets in to impress a girl, goes through boot camp, makes some combat drops – And then it transforms into a horror movie (and loses any resemblance with the book). They make a combat drop, Lieutenant Rico drops his mission and goes rescue his girlfriend (the two triangles having been – eliminated by now) and totally misses his old drill sergeant finding the prize. Allegedly, the director never finished reading the book. Well, it shows. In the book, Rico would have been hanged by the neck until dead, dead, dead well before the end of the movie just for striking a superior officer; in the movie, everybody just shrugs it off. The book is an enjoyable military story, the founding father of a subgenre consisting of lots of newer books; the movie is just silly. If you have seen the movie, read the book. Don’t bother the other way around.

… I used to write in this blog about random things. Let me check what I have done lately … well, first of all I spent 10 days in San Francisco in November, I had a really great time, we spent most of the time there but also paid a visit to the Googleplex, Stanford, Berkeley, … and went several days to the Yosemite National Park, a great trip overall Besides that I gave up one of my jobs (the boring and not challenging one …) two months ago (although it seems like it was ages ago) and started working full-time in my company. In addition, I spent New Year’s Eve in London to begin the year in a different way Right, that was all

Nick Leeson: The financial storm is not over yet

Surreal to see his byline on an article the same day as a rogue trader comes to light... "Cutting interest rates has stemmed the flow for now, but 2008 will see higher energy costs, higher food costs and higher importing costs which will damage consumer confidence. Right now we can only look to Davos and hope that the global economic leaders can keep positive and stop the markets from spiralling further downwards"

SFinfo TV Schedule for Sunday

SFinfo is carrying some of the Davos meeting live again (where corporations tell governments how to direct their populations). Only interesting thing left in my opinion is the Climate Change forum on Sunday morning. Watch and see where our glorious overlords will be taking us.

How to block all Facebook application and message spam

Practical freedom from Matt Lee's Exploring Freedom.

Valuing Users by Allowing Comments : David Lee King

It makes sense - you know it does.

Internet Psychology: Podcasting? That's old-fashioned - video is the online future by Graham Jones, Internet Psychologist

Maybe I just forgot the obvious, or maybe we haven't really seen what people can do with podcasting yet.

Gaba en San Pancho Blog Archive what things to check when buying an used car

Worst used car I ever saw had a split hose leaking petrol from the carb feed onto the engine block - switching that one off when I asked was the fastest I ever saw a car salesman move!

Modern Communicator: Imarco Activ-Media have been acquired

Personally, I find one of the benefits of working in cooperatives is that it's very difficult for us to be acquired and good collaboration is much more common than merging.

sab39 ... O noes, somebody I don't like did something I agree with!

I feel the point about the bozo bit is well-made. Is one of the few features of my weak memory a low probability that I keep the bozo bit set in the long term?

How You Can Fine-Tune Your Blogger Personality Perception

How do you repair a broken perception? Can you repair it, or is it time to delete the blog and start again at a new address? Can you start again, or does web.archive mean past mistakes will hurt you forever?

robmyers - My Ogg Player

I never succeeded in accessing anything besides basic file upload/removal on my YP-U2 and Samsung Support broke promises they made to me.

another blog is possible Give cyclists a break

"Those of you drive should read this article and change your behavior towards cyclists. You may not like cyclists in the road, but they have every right to be there."

New bts command: followup In Debian I'm pretty sure we have many perfectionists. Sadly, I'm one of them: I can't help loosing lots of time fixing silly things totally useless with respect to a more important task which is waiting to be completed. An example? Following up on BTS bugs. If I have the original bug report in my mailbox that's easy, I just hit r on mutt and reply as usual (ok, I will still loose some time getting rid of unneeded quoted text, but that's not too much). If I don't have it ... tragedy! I then have to open the bug log in the browser and compose from scratch a new mail including several information which I have to manually grab from the bug log web page:

• proper recipients (usually one or more of xxxxxx -submitter,-done,-forwarded, @bugs.debian.org)
• proper headers:
  • a Subject starting with Re: and then continuing with the original subject
  • a In-Reply-To header containing the original Message-Id (can I dare breaking threading, of course not!)
• proper body:
  • which should start with a On ... Random J Developer wrote: ...
  • ... and continue with the quoted text of the original bug report

(Yeah, I know, it looks like I'm hopeless, but I'm confident some of the people reading do actually share this and similar dramas.) Missing (or maybe overlooking) a tool helping me out on this, I've patched devscripts's bts to automate the above steps. Now I can do:

bts followup 430986

to end up in a mutt mail composing session with proper (according to the definition of "proper" above) recipients, headers, and quoted text:

From: Stefano Zacchiroli <zack@debian.org>
To: 430986@bugs.debian.org, 430986-submitter@bugs.debian.org
Cc: 
Bcc: 
Subject: Re: when reporting bugs by maintainer, please include Uploaders
Reply-To: 
In-Reply-To: <20070628174236.13848.57178.reportbug@aquarium.takhisis.invalid>
On Thu, 28 Jun 2007 19:42:36 +0200, Stefano Zacchiroli wrote:
> Package: bugs.debian.org
> Severity: wishlist
> 
> Browsing bugs by maintainer (and with this I mean something like
> http://bugs.debian.org/cgi-bin/pkgreport.cgi?maint=zack@debian.org, or
> it shorter form http://bugs.debian.org/zack@debian.org) used to be
> terribly useful as a TODO list. Using that I used to have a simple entry
> point for knowing what I have to do "for Debian" regarding my packages.
> 
> This usefulness has diminished a lot with the spreading of team
> maintenance. To know all bugs that apply to me indeed I know have to
> repeat browse by maintainer for each team I'm a member of.
> 
> It would be terribly useful to include bugs which apply to packages in
> which I'm mentioned as an Uploader in the report by maintainer pages.
<snip>

Now I just need to actually perform the original task, the problem now being that in the meantime I've forgotten what was it about ...

---

Try it out:

• bts with super followup powers
• the patch which adds super followup powers to bts of devscripts version 2.10.13

Comments?
Anyone else find this useful? If so I can just go ahead and commit it to devscripts mainline ...

---

Update: some changes to the code, thanks to Adam Barrat for the feedback.
Changelog:

• safer handling of RFC822 line continuation
• instead of always mailing -submitter, mail the author of the message we are replying to (the bug report is always on Cc though)
• bug message number is now 1-based, following the convention of the BTS get_bug_log method
• add the possibility to specify "last" (and even "last-1" and so on) for pointing to messages in the bug log starting from log bottom

samba.grep.be. This machine has been my trusty publically-accessible server for about four years now. It runs my website, including this blog, is my primary MX, contains my subversion repositories, and my gallery. For slightly less than the first year of Planet Grep, it was the only machine running that site (it has now been migrated to two machines running in two different data centers, to cope with the ever-increasing bandwidth usage of that site). It runs the bacula-director for our backup system, and it contains the modem behind our fax number, allowing us both to receive faxes and to log in using PPP to our local network. Most importantly, I've been using it as an SSH jumphost and IRC box to allow me to connect to other machines whenever I'm behind an overly paranoid firewall. But now it's time to retire the machine. The high number of services combined with the, to today's standards, low number of available system resources, is beginning to be a problem. A few times now, already, I've had to reboot the machine because a load spike was making it unresponsive. So it has to go. Many people will be shocked to find out that the machine powering samba up to now was an IBM SurePos 500. No, really. The reasons are long and complex, but suffice to say that there was a point in time where I wanted to set up a server for myself, and I had an idle, never-to-be-used-anymore, 2000+ priced, machine standing by. So there I went, usurped this machine as server, and have given it near-100% uptime. If you want to know just how near-100% that is:

  9 Power_On_Hours          0x0032   058   058   000    Old_age   Always       -       31170
[...]
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       99

Or, in plain English: according to SMART, the hard disk has seen just 99 power cycles in four years. This machine was never designed to do that. Side note: yes, that is a touch screen on top of that box. The new machine is much, much better. We're going from this:

model name	: Intel(R) Celeron(TM) CPU                1200MHz

to this:

model name	: Dual-Core AMD Opteron(tm) Processor 1210

which is much, much better. The box actually containing this processor is also much, much better. However, there was one problem: the old processor was a PentiumIV-class processor, requiring Debian's i386 port, whereas the new one (obviously) contains a processor that will allow me to run the amd64 port; and I did intend to do this. Additionally, I wanted to migrate from a simple "root-on-partitions" system to a "root-on-LVM" system, with Xen somewhere in between. As such, simply rsync'ing the entire hard disk on the old system over to the new system (my usual way to migrate a non-critical server) wasn't going to work. Luckily Debian isn't too hard to migrate from one machine to the other, however.

• Installed Debian on the new machine, giving it the same hostname (samba) as the old one, but with root on LVM.
• Ran 'dpkg --get-selections' on the old machine, feeding the output to 'dpkg --set-selections' on the new one. Installed packages.
• Rsynced over /home
• Rsynced /etc over to a separate directory on the new box, and copy some files (such as fstab) from the live /etc on the new box into the separate directory.
• Rsynced /srv over
• Rsynced /var over to a separate LVM volume.
• Created a snapshot volume of that /var LVM volume, moved most of the live /var over to some place else, and added the original LVM volume (i.e., not the snapshot) to fstab. Obvious exceptions were such things as databases, which usually have an architecture-specific on-disk format, and the dpkg directory.
• Brought the original system to runlevel 1.
• Rsynced /home, /srv, and /var over again to get the last-minute changes in.
• Rsynced (with --delete) the separate /etc over the live /etc.
• Brought the original system down.
• Rebooted, checked which services died because their on-disk format also differed between i386 and amd64, utterly and completely killed those, copied the files from the original amd64 /var back again, and made some type of plain-text dump that I then imported into that service.
• Rinse, repeat until all those services work. Then, removed the /var snapshot (didn't need that backup anymore, then), and voila.

Or, well, that was the idea. Because I forgot about one silly detail: if I wanted to make everything work out correctly, I would have had to keep UID numbers in synch. Unfortunately, I forgot about that; and as a result, a number of files were created with the wrong owner. This made all kinds of things fail horribly; and rather than stop and think about it, which would have made me copy the old passwd file over from the old installation, add some extra lines for users that were on the old server but were not yet created on the new one, and then finally leave it at that, everything would've been fixed. Instead, I started to change ownership of files all over the filesystem, creating a mess of things. Silly me. But, well, in the end I did get everything to work correctly; even if it took me longer than expected. No worries.

There are a lot of Pugs in NYC. Really, a lot. They are the right size and since they have tubing issues and thus breathing issues, NYers don’t feel bad that they have subject their dogs to a more or less sedentary life minus small jaunts in the park, unless of course, you send you dog to doggie day camp. Whatever the case, there are many. SO New Yorkers with pugs, a present . Use it at work and then it is just like having your pug with you.

I’ve just been working with a Flash device used as /dev/hda (the root filesystem) for a router. The aim is much the same as with my idea of using an EeePC as a router [1]. The client in question may consider the EeePC for future deployments but are concerned about the ability of ASUS to supply when needed. The typical customer at a computer store probably isn’t going to be too upset if they have to order their computer and wait a week, but when rolling out a network for a few corporate offices it’s a major problem if there aren’t enough devices to get them all online at once! One problem I encountered is that the default configuration had IDE DMA enabled and this didn’t work well on the system, rumour suggests that it might be related to the bridge chipset used to run the PATA controller which talks to the device that houses a SD Flash module and connects to the IDE cable. But I discovered that it would work reasonably well (at about 75% full speed) if I turned off DMA. When I run the Flash device with no DMA and 16bit IO I get about 2.4MB/s, I get 4.4MB/s if I use 32bit IO, and 6MB/s with occasional system hangs if I use DMA. The machine in question is running CentOS so the trick was to run hdparm early enough in the boot process to turn off DMA before the machine could hang. In most cases the machine would lock up solidly when udev was started. So I added the following lines to the start of /etc/rc.d/rc.sysinit (just after #!/bin/bash):
/sbin/restorecon -R /dev 2> /dev/null
/sbin/hdparm -d0 -c1 /dev/hda Now the system seems to be working correctly. The flash device seems like a good concept, no moving parts means less heat and more reliability, and the seek time is really good. But slow performance for bulk IO and problems with DMA increase the difficulty of implementing it. The advantage of having it in a pre-packaged device such as an EeePC is that it’s all one unit that is known to work together and has the drivers configured.

• [1] http://etbe.coker.com.au/2008/01/23/asus-eeepc-as-a-router/

Share This

Anyone who knows me, even remotely, knows that I have a huge beef with the American health insurance industry, which was greatly magnified after my own unpleasant run-ins with them over the course of the last two years. But of course while the insurance industry is certainly a ruthless predator and thus deserving much of the blame, they are not the sole culprit in sustaining what is a lousy health care system for Americans. A key player, I think, are American doctors, and I would now like to spread that beef patty of disgust to them, especially since they are now the one’s giving me problems dealing with the final portion of BCBS bill. The story is indicative of lager issues and problems that center on the problematic silence “spoken” by doctors. To make a long story short, BCBS of NJ was not ponying up some serious cash (at least for me) for 2 mole surgeries because they were claiming it was pre-existing condition. After having the public relations director email me (thanks to that post) and a lot of research and letters etc, it was more or less resolved. Soon after they overturned the pre-existing “bs” (and I am not talking blue shield here), they promptly started to pay all sorts of bills (and I even got money back from doctors I had paid). But there is one pesky $1600 bill that lingers like a bad smell and the question is why? I am not entirely sure but my interactions with my ex-dermatologists, Affilated Dermatology in NJ, I think reveal some important lessons as to why and how the medical establishment are complicit. Basically, BCBS is telling me that it looks like they have been double billed because the cost is exactly the same for two procedures that are also exactly the same. I explained that I had did indeed have two surgeries on my scalp (they were really close to each other) and they let me know that the provider needs to call back and send information that clearly shows there were two surgeries. This actually seems somewhat legitimate and potential mistake (and it is not like medical billing is known for their lack of mistakes and integrity either). So I call Affiliated Dermatology and while I would say they have been patient, in so far as patiently waiting for my money, they have been completely unhelpful in any meaningful sense to help me getting this resolved. Now, given how difficult it is to deal with the health insurance industry, I understand they can’t provide fine-tuned, fine-grained personal attention. The health insurance industry engages in some real hefty politics of foot dragging and well, there are so many foot soldiers at doctor’s offices to keep up with the web of knots that the health insurance constantly entangles us in. With that caveat in place, they can however do two things: 1. Inform you that you can contact the Department of Housing and Insurance and start some sort of formal complain process. 2. Give you some small clue as to whether the doctor has sent any of the additional requested information (they are usually mum, or enigmatic, or totally confusing with regards to that) so you know what the heck is going on. So a few days ago when I was speaking with health insurance person at the dermatology office, she claimed there was nothing else she could do about this last bill. Every time she calls BCBS, she claims that they claim it is a problem with being a pre-existing condition, and I was like “how can that be when that has been cleared and every other bill, and there were many, has been paid?” And then she also said that there has never been this confusion before where 2 surgeries look like one, blah blah and blah (and in retrospect, I forgot to tell her that my double surgery, according to the doc, was in fact highly unusual, because they would usually do them on two separate days because they were so close to each other. They made an exception because I was literally on my way to Canada and begged them to do so but anyway). Ok, so there I was bickering with this woman and feeling stuck between a rock and a hard place. Finally I let her know, point blank, that they have been remarkably patient with the billing but totally unhelpful in, well, helping me out. I let her know that they should let people like me know, for example, that they can file a formal complaint against the health insurance company and this would at least get the ball rolling. I then let her know that I had no other option but to file another complaint with the Dept of Banking and Insurance and they would just have to wait for that, at which point, she all of a sudden became more helpful. She suggested that we can do a three way call to resolve this. Well there you go. Why did it take a heated 15 minute conversation topped with a threat to get what I think actually makes perfect sense? Get the 3 parties involved on the phone with each other so as to all get on the same damn page about what needs to be done (not sure if BCBS will agree to this… at which point I will pester the public relations director again and in the end, I think this was resolved because he was sick of my emails). Now, why why why why why why are doctors and their staff so unhelpful? I think that main answer is they are overwhelmed and don’t have the staff to deal with the enormous amount of foot dragging and coy tactics deployed by the health insurance. That said, there are 2 things worth mentioning that I think don’t paint a very flattering picture of many American doctors. If you pay attention to your medical bills (and you always should), it is immediately evident that the doctors and hospitals get paid LESS, sometimes a lot less, if the health insurance company foots the bill because of the pre-arranged negotiated rate. So there is an actual incentive on that side of things to get paid via the consumer as opposed to the health insurance company. Now. I am sure that there are health care economic studies and reports justifying this strangeness but that does not make it right either. There is a clear incentive in place to get your green dollars, as opposed to those of the insurance company. Second and this is a much bigger issue, is that I think that for any meaningful change to happen, we need the active support of a large percentage of the medical community, especially doctors. If they are mum, or actively opposed to health care reform (as was the case when the AMA helped derail the Clinton effort at instituting universal health care in the 1990s), well then it becomes all that much harder for the general populace and the politicians to initiate real change. Doctors are the ones, after all, with the moral weight and capacity to make claims that can STICK. If they are saying this system negatively impacts how we care for the ill, it is a system that is is immoral, well, one should listen to them as they are the ones, after all, who dedicated their LIFE to healing the ill, right? Yes? No? Maybe? So their silence is nothing short of grave. The good news is that it seems like more doctors are on board than ever before clamoring for change in the right direction, but we need a lot more to come aboard… A number of years ago, I wanted to write a controversial and critical article that claimed hackers are more ethical than doctors. I thought it would be fun to claim that those who are usually seen as ethical (doctors) are less so than those that are usually portrayed as bad-as* unethical tricksters. Some people were offended by this because doctors deal with the great burden of life, death, sickness, and thus suffering while geeks and hackers are “just” geeking out on their computers. True. But at least a cadre of hackers have sustained a social realm and a real ethics–free software–in order to guarantee their own autonomy and also create the conditions for what is right for software. Doctors, on the other hand, do not carry the torch of ethics as they should. While individually I am sure they deal with a oodles of difficult ethical choices and decisions, it is about time they they turn as a collective to the larger structural conditions that seriously cause a lot of harm to millions of Americans, especially the droves of uninsured. They have the moral weight to do something about it and it is about time they carry their weight in this battle and take some burden of suffering off others.

could have just been a nice Summer morning, uneventful as they usually are, were it not for the fact that it was already 19:00, and it was mid-winter. (with due apologies, of course, to Bob Dylan and Les Luthiers) Nadezhda called me, warning me about a very dark sky and strong winds, and went off for her meeting. I told her weather looked decent from mx office... But started paying attention. And yes, wind was crazed. Hints of an upcoming power outage were felt in the Institute. I left my office as soon as I could. Cycling back home was quite a challenge - The 3.1Km route back home, usually much easier than the way there as it's mostly downhill, was quite a challenge: Biking with eyes almost-closed because of the flying leaves and dust, and little but menacing raindrops... Scary, all in all. Anyway, it gets scary... At least one person was killed because of the winds not far from here. This city has the fame that febrero loco y los vientos otro poco (February is crazy, but the winds are even more). And the first crazy wind of the season are always scary.[update] One person dead because of the strong winds in Mexico City; Power outages in several areas; Three more hours of strong winds expected; Heavy winds caused by the #24 cold front(?); Winds cause mayhem on DF streets;

We have a problem with our tools in Debian, and it's this: Above the core tools in dpkg-dev, and the nearly core tools in debhelper and devscripts, and the many different -- but all slowly converging -- revision control systems, there is the potential for infinite cruft to pollute the source package. Note that one developer's cruft is another developer's use of an excellent tool. If you don't believe that, you haven't read a lot of crufy upstream build systems, because Debian isn't the only project with this problem. The source of this cruft is a near-constant stream of new tools that are each developed in good faith by someone who needs that tool, and then adopted and used in packages of a generally small percentage of developers, who think that tool might be a good fit for their problems. The tool isn't yet suitable for everyone to use, and most of these tools never become suitable for everyone, because

1. making the leap from a special-purpose, poorly-designed hack to a general-purpose tool is hard
2. getting the project to standardise on a tool is hard
3. a tool often doesn't become general purpose until enough people use it, no matter how well designed it is to start out with

So after a while, a better, or at least different, solution to some of the same problems will come along, and some generally small percentage of developers will choose to use it in their source packages. Meanwhile, the developers who chose the old thing will be set in their ways and not want to use the new thing, and generally cannot be budged to stop using the old thing by any amount of argument. Which in a sense is normal, and fine, until you have to collaborate with them. I don't care if someone I'm collaborating with uses emacs to my vim, but I do care if they use cdbs to my debhelper. I care because their tool-choice encrufts the source that I'm supposed to be working on. Once a significant percentage of all the source packages in Debian are encrufted for a wide array of different tools, it will become harder and harder to work with Debian's source packages. We're already seeing this happen. Another way of looking at it is that we've invented dozens of source package formats that all work differently, aside from a small shared core, and we expect our developers to be able to use all of them. At this point I see this as a huge, potentially killer problem, that is making Debian development become worse with every passing year. But I have no real solution. Obviously pointing out that tools are obsoleted or bad doesn't work; we still have dozens of packages using yada, which everyone agrees is horrible. That doesn't bode well, does it? One partial solution is to design a category killer, and get it embedded in acknowledged core tools. That's what I have tried, and so far, failed, to do with an evolutionary change to the Debian source package format, which I'd hope would give people a reason to stop using dpatch, dbs, and their ilk. As noted, this approach is Hard.

---

For now, I have no interest in being on a team that requires I use any of the following things:

• dpatch
• dbs
• quilt
• Any similar system that is used to move all patches to under the debian/ directory.
• Any system that nests the actual original upstream .tar.gz inside an .orig.tar.gz.
• yada
• cdbs
• Any similar system that obfuscates debian/rules.
• Any special script that is required to be used to build a package as a wrapper around dpkg-buildpackage.

This means that I can't be on the security or release teams, and even teams like the perl team are getting sticky to be on.

---

PS, this doesn't mean I dislike quilt. I dislike quilt encrufting Debian source packages, which is entirely different. PPS, In case you're wondering why debhelper gets a pass, it's because it's gone through the three numbered steps above to become a standard, general purpose tool, and has no serious competators. (Though it does have parasites, like cdbs.)

Having had a number of hard drives fail over the years I use RAID whenever possible to reduce the probability of data loss caused by hardware failure. It’s unfortunate that some machines make it impractically difficult to install a second hard drive (my laptop and some small form factor desktop machines I have given to relatives). But when it’s possible I have two hard drives running software RAID-1. I use two partitions, one for /boot and one as a LVM physical volume (PV). When using RAID I make both /boot and the PV be software RAID-1 devices (the vast majority of machines that I install don’t have budget available for hardware RAID). /boot is a small partition, approximately 100M. For a machine with only one disk I make the second partition take all the available space as there is no benefit in doing otherwise (LVM allows me to create an arbitrary number of block devices out of the space at run-time). When using software RAID I often make the PV take less than the full capacity of the disks. When the disks are 40G or more I usually use less than half the capacity. For most machines that I install or run the full capacity of the disks is not required. One deficiency of Linux software RAID is that on a power failure the entire RAID device must be checked to ensure that all disks have matching data. Reading the entire contents of a pair of disks can take a large amount of time if the disks are large, and as the size of disks is increasing at a greater rate than the speed of disks this problem is getting worse. See my ZCAV benchmark results page for graphs of the contiguous IO performance of some disks I’ve owned [1]. It seems that with a 10G disk you may expect to be able to read it all in less than 1000 seconds, for a 46G disk it’ll be something greater than 1,500 seconds, and for 300G disks you are looking at something well in excess of 5,000 seconds. Almost all disks (and all IDE and SATA disks for which I’ve seen benchmark results) have the lower block numbers mapped to the outside tracks which are longer and give a higher contiguous IO speed. So by using the first half of the disk the RAID synchronisation time is reduced to less than half what it might be (in the absence of motherboard bottlenecks). When there is no need for more than about 10G of storage space there’s no benefit in making a large RAID device and wasting recovery time. While the system can operate while the RAID is synchronising the performance will be significantly lower than normal. If the usage pattern of the machine changes such that it needs more space it’s easy to create new partitions, make a new software RAID, and then add it to the LVM volume group (I’ve done this before). So the down-side to this is minimal. When creating LVM logical volumes (LVs) I create volumes for the root filesystem and swap space when doing the install. This should result in the swap space being near enough to the start of the disk to get reasonable performance (but I haven’t verified this). I make the root filesystem reasonably large (EG 6G) as disk space is plentiful and cheap nowadays and the root filesystem is the only one that you can’t easily extend when running LVM (trying to do so deadlocks the disk IO). After the base install is done I create other LVs.

• [1] http://www.coker.com.au/bonnie++/zcav/results.html

Share This

My mothers computer had died. Well, the Linux on it was still working, so she was able to open her document files, read her email and use chat. Pretty much everything works under Linux, with the exception of using the Fax functionality of the AVM Fritz!Box via CAPI-over-TCP. (Ironically, the Fritz!Box itself runs Linux, so it's just because AVM doesn't bother to supply a Linux client too).Anyway. Logging in remotely, I quickly found out that the hard drive had serious issues, with lots of bad sectors on the main Windows partition (fortunately, we kept system data and user data separate!). So I'm not blaming Microsoft for dying (but Samsung; since this is the second time within three years the drive has died, I guess I won't be buying a Samsung HD anytime soon).Last weekend, I bought a new hard disk and went to my parents to fix it. I copied over the partitions (some 64k data could not be read any more), and tried to get Windows up again, too. (Linux was trivial; copy, boot record, done)So I ended up doing a "rescue" Windows installation, using the XP service pack 2 CD. I figure this is still the latest setup for XP available? It went through the usual driver-loading, did the first part of the installation, rebooted into the new system, did this "estimated time 39 Minutes" setup screen. After like 10-15 Minutes, it had a bluescreen, dumped memory and rebooted. And cycled through that again and again.It took me like 6 hours total to get the Windows system fixed (mostly waiting for the Microsoft setup to run through again). I figure that there aren't many users out there that actually could have fixed that Windows. The reason is simple: Microsoft crappy setup. Probably to protect it's serial number thingy or just to display fancier graphics, setup refuses to work in safe mode. And the bluescreen even suggested to try the safe mode, just to display the message "setup.exe doesn't work in safe mode, click OK to reboot".If I could have used that "safe mode", fixing would have been a lot easier. The bluescreen mentioned a device driver ati2something.sys; apparently part of the ATI video driver. So I tried the rescue console of the XP CD to just kill that driver, but setup kept on actually reinstalling the file.The way I fixed it in the end is simple: Linux to the rescue. I booted the (working like a charm) Linux installation, accessed the main Windows partition using ntfs-3g, found all references to the ATI driver in the Windows\Inf driectory and removed them.After I had removed these references, Microsofts setup.exe completed, and I had a working-again Windows in VGA mode. Then I could install the ATI drivers again. Microsoft really needs to work on that setup, it needs to allow disabling certain drivers and similar rescue things.Maybe Microsoft should start shipping a Knoppix CD with Windows, for repair.

How often have you said to yourself "this programming language's if statement is too flexible, I need to make it more rigid, and with a syntax nobody else understands"? If you've ever said that to yourself, then take a look at this Rails plugin. From the brief description of the plugin:

Allows developers to execute or not execute given code block depending on a set of conditions, written in intuitive manner. Especially useful within view templates, where it can eliminate the need for multiple if ... else ... statements.

That screams "overkill" to me, but for the sake of my morbid curiousity (and on the very, very off chance that it's something not entirely ridiculous) I took a look at the detailed documentation. At least there is detailed documentation for this plugin. I didn't get any further than the first example, though. Take a look at this monster:

How often do you find yourself writing something like that in your views:

	  <% if current_user != :false && (current_user.administrator?   current_user == @profile.user) %>
	    <%= button_to('Big Red Button', destroy_everything_path) %>
	  <% else %>
	    <%= button_to('Hands off!!!', access_denied_path) %>
	   <% end %>
	

It works, but, you know, it's not what you would call beautiful, is it? What about something like that:

	  <% access.to_administrator.or_user_of(@profile) do %>
	    <%= button_to('Big Red Button', destroy_everything_path) %>
	  <% end.else do %>
	    <%= button_to('Hands off!!!', access_denied_path) %>
	  <% end %>
	

At least conditions are looking much better, aren't they?

Hopefully, most people out there answered a resounding "no" to that last question. A plugin which causes you to write just as many lines and very nearly as many characters, whilst using a completely different minilanguage to what every Ruby programmer already knows, does not sound "much better" to me. Ruby people tend to love little minilanguages to get things done, but this... I just don't have the words to properly describe it. I'm going to start a list of "poison plugins" -- ones that, if I see any of them in or around the code of a Rails project I'm looking at working on, immediately doubles the hourly rate and the time estimates. Stupidity is toxic, and working with hazardous substances is always more expensive. Other current candidates include the old-world auth generator I gutted while I was at Vquence, and annotate_models -- the former because it's shocking code, and the latter because anyone who uses it has some seriously scary ideas about what constitutes acceptable documentation and development practices.

Tracing python programs. Today, Evan Dandrea asked a general question "Where is set -x for python". A quick google for sys.settrace found: Some code snippets. I thought this was nice, but surely you want to be able to just trace an arbitrary program. So I present a 'quick hack' (5 minutes precisely :)) to do that based on the previous links final version:

#!/usr/bin/env python
 import linecache
import os
import os.path
import sys
 def traceit(frame, event, arg):
    if event == "line":
        lineno = frame.f_lineno
        filename = frame.f_globals["__file__"]
        if (filename.endswith(".pyc") or
            filename.endswith(".pyo")):
            filename = filename[:-1]
        name = frame.f_globals["__name__"]
        line = linecache.getline(filename, lineno)
        print "%s:%s: %s" % (name, lineno, line.rstrip())
    return traceit
 def main():
    search_path = os.environ.get('PATH',
        '').split(os.path.pathsep)
    argv = sys.argv[1:]
    if not argv:
        raise Exception("No command to trace supplied")
    args = argv[1:]
    command = argv[0]
    if os.path.sep not in command:
        for path in search_path:
            if os.path.exists(os.path.join(path, command)):
                command = os.path.join(path, command)
                break
    del sys.argv[0]
    source = open(command, 'rt')
    exec_symbols = dict(globals())
    exec_symbols['__name__'] = '__main__'
    sys.settrace(traceit)
    exec source in exec_symbols, exec_symbols
 main()
 

When I have to read things like drive-by pharming (via Heise, Symlink article), I’m really happy that there are free 3rd party router firmwares out there, that don’t need any shitty web interface. My ASUS WL-500g Premium runs FreeWRT and the only possibility to change the configuration is to login via ssh and edit the configuration files as root. I really pity all those out there who have to cope with the partially really sleazy web interfaces home routers currently offer.

Zack's pam-hooks module got me thinking that the whole "start stuff at user login/logout" thing should be simplified a huge lot. Currently, there are just too many mechanisms depending on the user's shell, ?dm, distribution, session manager/desktop environment etc. Should this be a topic for the freedesktop.org people (or is it already - it might fit into Portland in some way)? A standardized way to hook into session setup/teardown might also be good for security: if there's only few ways to get stuff started at login, that means fewer places to look where malware could be hiding. Of course, it also means that said malware can spread more easily, being more independent of the user's desktop environment.

Chris Lamb has suggested storing a GPG key on a RAID-5 device [1]. The idea is that it can be stored on several physical block devices such that losing just one will not give the key to an attacker. A default GPG secret key will be about 1.2K in size (3 sectors of a hard drive). A minimal key (with 1024 bit DSA keypair) will be 0.9K (2 sectors). I expect that very few people have secret keys greater than 4K in size. To create a software RAID-5 device under Linux the tool mdadm is used. The default chunk size is 64K, so a 1.2K file will probably be on a single device. If you use the -c option of mdadm to specify a smaller chunk size then the smallest that is accepted is 4K which still permits a default GPG secret key to be on a single device. The Ext2 and Ext3 filesystems will always align such file data to a 4K boundary unless the device is smaller than a certain size (or a special mkfs option is used) to give a 1K block size for the filesystem. If an Ext2 or Ext3 filesystem is used with 1K blocks then you might get a 1.2K file split across multiple 4K RAID chunks. So storing a GPG key on RAID-5 won’t prevent an attacker who steals one part from getting all the most valuable data. It will make it more inconvenient for them (if you are lucky it will prevent them getting all the data) and it will also make it difficult for the owner of the GPG key to determine which of the devices actually contains the secret data (probably all of them will end up having copies if you edit the secret key). Now if RAID-5 did allow chunk sizes that were smaller than the secret key or if you have Ext2/3 with 1K blocks and get lucky with file fragmentation then the problem still isn’t solved. The reason is that you don’t require N-1 of the N disks to get some useful data out of a RAID-5 array (run strings on one element of a RAID-5 array to verify this). A single disk on it’s own will have some data that can be used, as file(1) can recognise GPG secret keys so you could just copy 1K chunks of data into separate files and use file to determine which (if any) has the data in question. The really exciting question is, what do you get if you have the first 1K of a 1.2K GPG secret key? If it could be proved that the first 1K does not give an attacker any advantage then this might provide some benefit. But I think that this is a very dubious assumption, when dealing with such things it’s best to assume the worst. Assume that an attacker who has 1K out of 1.2K of secret data has the ability to reconstruct the rest. In that case the Linux kernel RAID-5 provides no benefit for storing a GPG secret key. Just try not to get your devices that contain secret data stolen. Maybe a watch with a built-in USB device is a good idea. Thieves seem to be targetting mobile phones instead of watches nowadays and something that’s strapped to your wrist is difficult to lose.

• [1] http://chris-lamb.co.uk/2008/01/23/xorly-losing-your-gpg-private-key-safely/

Share This

It seems to me that the Asus EeePC (a $AU499 ultra-light laptop with only flash storage) would make a decent router. Often full desktop PCs are used as routers because they run the most common software and have standard interfaces. There have been dedicated router devices with flash for a long time, but without the ability to connect a standard monitor and keyboard they were always more difficult to manage than general purpose PCs. Also dedicated routers have limited RAM and storage which often does not permit running a standard OS. According to the best review I could find [1] EeePC has a minimum of 256M of RAM, a Celeron-M CPU (32bit Intel), and a minimum of 2G of flash storage. This hardware is more than adequate to run most server software that you might want to run (my current router/firewall/server has 192M of RAM). It could run a web server, a mail server, or any other general server stuff. It comes pre-loaded with a modified version of Debian so you get all the Debian software (which incidentally means more pre-packaged software than is available for any other distribution). Bigger versions are common, I believe that the $AU499 version has 512M of RAM and 4G of flash - I’m not sure that I could even obtain a lesser EeePC in Australia. The up-side of flash is that it doesn’t take much power as having a low power device in whatever confined space ends up housing your router is a good thing (the EeePC is listed as using less than 20W no matter what the load and idling at as little as 14W) and that it doesn’t tend to break when dropped or have any moving parts to wear out. The down-side of flash is that a sufficient number of writes will destroy it. Obviously swap is not a suitable use for a flash storage device. But a small mail server (suitable for the needs of a home or a small office) should be fine with it. Squid is commonly run on router devices, to run it on an EeePC I would be inclined to buy 1G USB flash device for the cache, then if Squid’s use destroyed the flash storage it would be easy to spend $20 and buy another. The EeePC has three USB ports and a built-in Ethernet port. I believe that the minimum number of Ethernet ports for a firewall is three, this means either one for the uplink, one for the DMZ, and one for your LAN, or two for uplinks (with a redundant uplink) and one for the LAN. The three USB ports allow using two USB Ethernet devices to provide the minimum three Ethernet ports and one for USB flash storage. One notable advantage of using a laptop as a server is the built-in UPS (the laptop battery). Many people have put old laptops into service as servers for this reason, but usually an old battery gives no more than about 30 minutes of power, while a new EeePC should be able to last for more than 3 hours without mains power. Using a second-hand laptop as a server is usually not viable in a corporate environment as laptops are all different. Repairing an old desktop PC is easy, repairing an old laptop is unreasonably difficult and often more expensive than replacing it. The low price of the EeePC makes it easily affordable (cheaper than some of the desktop machines that might be used for such a purpose) and the fact that it is purchased new at such a price means that you get warranty support etc. It seems to me that a significant disadvantage of using an EeePC (or anything other than a real server) for a server task is that it lacks ECC RAM [2], but as I’m not aware of any device sold for the purpose of running as a low-end router or sold at a price-point such that it could be used as a low-end router which has ECC RAM I guess that’s not a big disadvantage for that purpose. But the lack of ECC RAM and the lack of useful support for RAID does make it a little less suitable for use as a mail server. Finally if you are going to run a VPN then it would be handy to be able to secure the hardware against malicious activity when the office is closed. An EeePC will fit nicely in most safes… Please let me know what you think about this idea. I’ll probably start installing some such machines for clients in the near future if ASUS can keep up with the demand.

• [1] http://www.hothardware.com/articles/Asus_Eee_PC_Full_Retail_Review_Showcase/?page=1


• [2] http://etbe.coker.com.au/2007/07/15/desktop-machines-and-ecc-ram/

Share This