Don Marti has written his own equivalent to Technorati based on links from blogs that he reads, and my blog comes in at #40 in the list (last place) [1]. Don does note the fact that such lists mean little and links to a post by Doc Searls [2] which makes the same point more strongly. But it’s still interesting to note. Also if Don chose to release the Perl script in question (or host is as a cgi-bin script) to allow other people to make their own top 40 list then I’m sure that many people would appreciate it (I’ll write such a script myself and release it if he doesn’t). He notes when publishing his list that a blog may be included even if he never reads it. I believe that if a blog is highly ranked according to links from blogs that you read then it’s quite likely that you will be interested in reading it, it may be a blog that is opposed to your beliefs (for example I’m sure I’m not the only person in the Linux community to link to an official Microsoft blog) but if it gets enough links then it would be worth reading at least once (if only to discover why you don’t like it). While I can’t tell Don to read every blog on his top 40 list I’ll certainly read every blog on my top 40 list at least once when I create the list.

• [1] http://www.linuxworld.com/community/?q=node/1447


• [2] http://blogs.law.harvard.edu/doc/2007/10/04/ranklings/

Share This

In a previous post, I had recommended the 4130 NIST-Traceable Temperature Controller to control the temperature in a slow cooker. Unfortunately, that particular controller has a range that tops out at 60 degrees C / 140 degrees F, which is enough for cooking beef for long periods of time, but not enough for say, cooking duck confit, which for which a sous vide temperature of 80 degrees C is recommended. In addition, the 4130 is pretty expensive; almost $150. It’s possible to add a resistor to change the range of the 4130, but the temperature displayed by the controller is no longer correct, and you have manually create a conversion table between true temperature and the temperature as seen by the controller. I’ve recently come across a cheaper and better possibility, the Ranco ETC-111000-000 Temperature Controller which is only half the price and comes with a much larger working range (-30 to 220 degrees F). The price with the AC cord already wired in is $75; and the version which just has a 120VAC SPDT relay is only $60. A bit more about food safety. There has always been a lot of concern about bacteria growth and botulism, for good reason — and so therefore the recommendations for cooking temperature have a lot of safety margin in them — to the point now that the USDA recommends that steaks be cooked to at least 145 degrees F, which is well within what had traditionally been called “medium”, and chicken to at least 165 degrees F, which is enough to really destroy taste and texture. Sous vide cooking, especially some of the more low temperature variants, have raised a lot of concerns, to the point where a few years ago New York City (temporarily) banned it, causing a great outcry in the foodie community, since many top restaurants use sous vide techniques. First of all, any recommendation about internal temperatures and food safety that doesn’t also factor in time is massively oversimplifying the problem. Here is a table taken from “Food Safety Hazards and Controls for the Home Food Preparer”, published by the Hospitality Institute of Technology in 1994:

Temperature, F	Time, 5D kill	Time, 6.5D kill
130	86.42 minutes	112.34 minutes
135	27.33 minutes	35.53 minutes
140	8.64 minutes	11.23 minutes
145	2.73 minutes	3.55 minutes
150	51.85 seconds	1.12 minutes
155	16.40 seconds	21.32 seconds
160	5.19 seconds	6.74 seconds
165	1.64 seconds	2.13 seconds

This table lists the time to reduce bacteria concentrations of Salmonella and E. coli from 100,000 to 1 (5D) or 3,162,277 to 1 (6.5D). The FDA and USDA recommend cooking hamburger to 5D destruction. Since it is extremely unlikely for there to be more than 100 Salmonella organisms per gram of meat, a 5D kill will reduce Salmonella concentrations to no more than 1 organism per kilogram. So whether you cook a piece of meat to an internal temperature of 165 degrees for 1.64 seconds, or hold it at 130 degrees for 90 minutes, the effect on Salmonella and E. coli bacteria will be the same.Of course, one concern is that in many forms of cooking, particularly oven roasting and grilling, the temperature of the food as it heats up may not be even; so how do you guarantee that all parts of the food product has been brought up to the requisite 165 degrees? One way that recipe authors, fearful of liability concerns, have done so is to tell people to cook meat to much higher internal temperatures, to provide that extra safety margin at the expense of dessicated, horribly tasting turkey or chicken. But the advantage of sous vide cooking is by immersing the food in a water bath, the excellent heat conductivity of water helps guarantee that the entire body of meat will get raised to desired temperature relatively quickly. (How quickly depends on the thickness of the meat, obviously). So if you hold a roast beef that has been vacuum packed in a Foodsaver bag for five or six hours, there should be no question that all of the common bacteria has been inactivated, and that amount of time at 130 degrees F should be sufficient to inactivate 99.9 percent of all botulism toxin molecules (not there should be any in a fresh piece of meat, of course!) However, sous vide temperatures are not enough to kill bacterial spores, in particular C. botulinum, which is responsible for botulism. This requires temperatures far in excess of boiling water at sea level. For example, home canning protocols recommend holding the food product at 250 degrees F for at least 15 minutes. This is an issue in the restaurant business because very often food would be cooked sous vide, and then stored in the vacuum sealed bags for potentially weeks (yes, you could be eating an extremely expensive meal at a top-end French restaurant that had been cooked several weeks ago, and reheated just before serving; yummy, no?). If the food packages aren’t cooled quickly enough, and then allowed to warm to the danger zone, it’s possible that in the anerobic environment the C. botulinum spores could germinate and then start producing toxin. But, if you are cooking home sous vide where you are serving the food right after it has been cooked, this shouldn’t be a concern.

When running SE Linux you will notice that most applications are not permitted to run with an executable stack. One example of this is libsmpeg0 which is used by the game Freeciv [1]. When you attempt to run the Freeciv client program on a Debian/Etch system with a default SE Linux configuration (as described in my post on how to install SE Linux on Debian in 5 minutes [2]) then you will find that it doesn’t work. When this happens the following will be logged to the kernel log and is available through dmesg and usually in /var/log/kern.log (Debian/Etch doesn’t have auditd included, the same problem on a Fedora, RHEL, or CentOS system in a typical configuration would be logged to /var/log/audit/audit.log):
audit(1191741164.671:974): avc: denied execstack for pid=30823 comm=”civclient” scontext=rjc:system_r:unconfined_t:s0 tcontext=rjc:system_r:unconfined_t:s0 tclass=process The relevant parts are in bold. The problem with this message in the message log is that you don’t know which shared object caused the problem. As civclient is normally run from the GUI you are given no other information. So the thing to do is to run it at the command-line (the avc message tells you that civclient is the name of the failing command) and you get the following result:
$ civclient
civclient: error while loading shared libraries: libsmpeg-0.4.so.0: cannot enable executable stack as shared object requires: Permission denied This makes it clear which shared object is at fault. The next thing to do is to test the object by using execstack to set it to not need an executable stack. The command execstack -q /usr/lib/libsmpeg-0.4.so.0.1.4 will give an “X” as the first character of the output to indicate that the shared object requests an executable stack. The command execstack -c /usr/lib/libsmpeg-0.4.so.0.1.4 will change the shared object to not request an executable stack. After making such a change to a shared object the next thing to do is to test the application and see if it works correctly. In every case that I’ve seen the shared object has not needed such access and the application has worked correctly. As an aside, there is a bug in execstack in that it will break sym-links. Make sure that the second parameter it is given is the shared object not the sym-link to it which was created by ldconfig. See Debian bug 445594 [3] and CentOS bug 2377 [4]. The correct thing to do is to fix the bug in the source (not just modify the resulting binary). On page 8 of Ulrich Drepper’s document about non-SE Linux security [5] there is a description of both the possible solutions to this problem. One is to add a line containing “.section .note.GNU-stack,"",@progbits” to the start of the assembler file in question (which is what I suggested in Debian bug report 445595 [6]). The other is to add the parameters “-Wa,--execstack” to the command-line for the GNU assembler - of course this doesn’t work if you use a different assembler. In the near future I will establish an apt repository for Debian/Etch i386 packages related to SE Linux. One of the packages will be a libsmpeg0 package compiled to not need an executable stack. But it would be good if bug fixes such as this one could be included in future updates to Etch.

• [1] http://en.wikipedia.org/wiki/Freeciv


• [2] http://etbe.coker.com.au/2006/12/08/se-linux-on-debian-in-5-minutes/


• [3] http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=445594


• [4] http://bugs.centos.org/view.php?id=2377


• [5] http://people.redhat.com/drepper/nonselsec.pdf


• [6] http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=445595

Share This

There seems to be a recent trend towards home-schooling. The failures of the default school system in most countries are quite apparent and the violence alone is enough of a reason to keep children away from high-schools, even without the education (or lack therof). I have previously written about University degrees and whether they are needed [1]. The university I attended (which I won’t name in this context) did an OK job of teaching students. The main thing that struck me was that you would learn as much as you wished at university. It was possible to get really good marks without learning much (I have seen that demonstrated many times) or learn lots of interesting things while getting marks that are OK (which is what I did). So I have been considering whether it’s possible to learn as much as you would learn at university without attending one, and if so how to go about it. Here are the ways I learned useful things at university:

1. I spent a lot of time reading man pages and playing with the various Unix systems in the computer labs. It turned out that sys-admin work was one of my areas of interest (not really surprising given my history of running Fidonet BBS systems). It was unfortunate that my university (like almost all other universities) had no course on system-administration and therefore I was not able to get a sys-admin job until several years after graduating.

I read lots of good text books (university libraries are well stocked).

There were some good lectures that covered interesting material that I would not have otherwise learned (there were also some awful lectures that I could have missed - like the one which briefly covered computer security and mentioned NOTHING other than covert channels - probably the least useful thing that they could cover).

I used to hang out with the staff who were both intelligent and friendly (of which there were unfortunately a small number). If I noticed some students hanging out in the office of one of the staff in question I would join them. Then we would have group discussions about many topics (most of which were related to computers and some of which were related to the subjects that we were taking), this would continue until the staff member decided that he had some work to do and kicked us out. Hanging out with smart students was also good.

I did part-time work teaching at university. Teaching a class forces you to learn more about the subject than is needed to basically complete an assignment. This isn’t something that most people can do.

I expect that Children who don’t attend high-school will have more difficulty in getting admitted to a university (the entrance process is designed for the results of high-school). Also if you are going to avoid the public education system then it seems useful to try and avoid it for all education instead of just the worst part. Even for people who weren’t home-schooled I think that there are still potential benefits in some sort of home-university system. Now a home-university system would not be anything like an Open University. One example of an Open University is Open Universities Australia [2], another is the UK Open University [3]. These are both merely correspondence systems for a regular university degree. So it gives a university degree without the benefit of hanging out with smart people. While they do give some good opportunities for people who can only study part-time, in general I don’t think that they are a good thing (although I have to note that there are some really good documentaries on BBC that came from Open University). Now I am wondering how people could gain the same benefits without attending university. Here are my ideas of how the four main benefits that I believe are derived from university can be achieved without one (for a Computer Science degreee anyway):

1. Computers are cheap, every OS that you would ever want to use (Linux, BSD, HURD, OpenSolaris, Minix, etc) is free. It is quite easy to install a selection of OSs with full source code and manuals and learn as much about them as you desire.

University libraries tend not to require student ID to enter the building. While you can’t borrow books unless you are a student or staff member it is quite easy to walk in and read a book. It may be possible to arrange an inter-library loan of a book that interests you via your local library. Also if a friend is a university student then they can borrow books from the university library and lend them to you.

There are videos of many great lectures available on the net. A recent resource that has been added is Youtube lectures from the University of California Berkely [4] (I haven’t viewed any of the lectures yet but I expect them to be of better than average quality). Some other sources for video lectures are Talks At Google [5] and TED - Ideas Worth Spreading [6].

To provide the benefits of hanging out with smart people you would have to form your own group. Maybe a group of people from a LUG could meet regularly (EG twice a week or more) to discuss computers etc. Of course it would require that the members of such a group have a lot more drive and ambition than is typical of university students. Such a group could invite experts to give lectures for their members. I would be very interested in giving a talk about SE Linux (or anything else that I work on) to such a group of people who are in a convenient location.

The benefits of teaching others can be obtained by giving presentations at LUG meetings and other forums. Also if a group was formed as suggested in my previous point then at every meeting one or more members could give a presentation on something interesting that they had recently learned.

The end result of such a process should be learning more than you would typically learn at university while having more flexible hours (whatever you can convince a group of like-minded people to agree to for the meetings) that will interfere less with full-time employment (if you want to work while studying). In Australia university degrees don’t seem to be highly regarded so convincing a potential employer that your home-university learning is better than a degree should not be that difficult. If you do this and it works out then please write a blog post about it and link to this post.

• [1] http://etbe.coker.com.au/2007/01/24/university-degrees/


• [2] https://www.open.edu.au/wps/portal


• [3] http://www.open.ac.uk/


• [4] http://youtube.com/ucberkeley


• [5] http://www.youtube.com/user/AtGoogleTalks


• [6] http://www.ted.com/

Share This

In a lot of talks or blog posts (like Sam’s talk at RMLL, or Rapha l’s blog posts - both in french), people have been talking about what people could do inside Debian, and how it would help Debian. That doesn’t sound like the best approach to me. When describing tasks with the objective of getting potential contributors to pick them up, we should try to make them sexy, to tell users what is exciting about them, what they will learn doing those tasks, where satisfaction will come from. We really need to sell them better. Of course, some Debian tasks are mainly grunt work. And for some of them, people just do them because someone has to do them. But I believe that most tasks inside Debian are actually more interesting than outsiders would expect. For example, I would be very interested in reading why an i18n expert (hint: Christian!) finds i18n sexy … and I should probably try to write about QA myself. (As you might have noticed now, the subject of this blog post was misleading on purpose — chosen so that a lot of people would read the post :P)

Is it possible, under Linux, to request a Kerberos ticket-granting ticket from one Kerberos realm without destroying your TGTs from another realm that are already in your credentials cache? Otherwise, this happens:

Ticket cache: FILE:/tmp/krb5cc_2000_O5THYS
Default principal: wouter@EXAMPLE.COM
Valid starting     Expires            Service principal
10/02/07 19:29:53  10/03/07 05:29:53  krbtgt/EXAMPLE.COM@EXAMPLE.COM
	renew until 10/03/07 19:29:50
10/02/07 19:29:56  10/03/07 05:29:53  HTTP/exampleserver.example.com@
	renew until 10/03/07 19:29:50
10/02/07 20:52:54  10/03/07 05:29:53  host/exampleserver.example.com@
	renew until 10/03/07 19:29:50
Kerberos 4 ticket cache: /tmp/tkt2000
klist: You have no tickets cached
wouter@country:~$ kinit wouter@GREP.BE
Password for wouter@GREP.BE: 
wouter@country:~$ klist
Ticket cache: FILE:/tmp/krb5cc_2000_O5THYS
Default principal: wouter@GREP.BE
Valid starting     Expires            Service principal
10/02/07 20:59:01  10/03/07 06:59:01  krbtgt/GREP.BE@GREP.BE
	renew until 10/03/07 20:58:57
Kerberos 4 ticket cache: /tmp/tkt2000
klist: You have no tickets cached

Note the complete absense of any reference to the first realm in the second klist output, which is annoying. (Yes, I do know about the possibility to create trust paths between two realms; But I'm not going to give customers access to my personal mailserver...)

Madduck wondered why a package using revision control needed pristine tarballs at all. Firstly, the example he gave of tagging an upstream release and creating a tarball of it to build the package against is exactly what pristine-tar does. Except it preserves permissions and timestamps and other tar cruft. Sometimes timestamp or permissions issues do cause issues with package builds, so being able to build against the original files from the pristine tarball is potentially useful to catch and fix such problems. As to other uses of pristine tarballs... It's hard to list the advantages to using pristine tarballs, since Debian (rpm too) has had them forever, and lots of people may depend on them in ways that we don't anticipate. For example, I've seen FreeBSD point at ftp.debian.org to get tarballs for their packages, since they don't archive tarballs on their own, and sometimes the tarball isn't available upstream (or ftp,debian.org is the upstream). That's a wacky example, but I've seen it more often than I've seen pristine tarballs used for the often-cited md5sum comparison case. Rather than try to enumerate all the possible reasons for pristine tarballs, and then try to whittle the list down to reasons that actually make sense, and cost-benefit analise those, it seemed better to just try to write a solution to the problem. (Or at least, to most of it -- a hopefully small subset of the set of reasons for pristine tarballs are not satisfied by pristine-tar.) I've also been theoretically interested in this problem of generating pristine tarballs from version controlled source ever since Scott Remnant mentioned at a meeting (140 mb video of meeting here) that he had a tool to do it -- and AFAIK never shared the tool! So the challange factor was another reason to write it. However, pristine-tar has already saved me hundreds of megabytes of disk space, sped up my release process (avoiding bug #225483) and simplified the way I work, so it's already a net win for me, even if it turns out we don't need pristine tarballs for anything at all!

I’ve just started getting a lot of traffic referred by live.com. It seems that my post Porn for Children [1] is the second link returned from a live.com search for “porn” and my post Porn vs Rape [2] is the third link. These results occur in two of the three settings for “safe search” (the most safe one doesn’t return any matches for a query about “porn”). A query for “porn” and “research” (which would reasonably be expected to match a blog post concerning scientific research made my page the 8th listing (behind http://www.news.com.au/404, http://www.news.com.au/couriermail/404, and http://www.theaustralian.news.com.au/404). It seems strange that a query which should match my page gives it a lower ranking than three 404 error pages while a query which shouldn’t match my page (no-one who searches for “porn” on it’s own wants to read about scientific research) gives it a high ranking. One very interesting thing about the live.com search is that it doesn’t filter out some of the least effective ways of gaming search engines. For example the URL http://gra.sdsu.edu/research.php has a huge number of links to porn pages and pages that apparently sell illegal pharmecuticals) that are not visible (view page source to see them). The links that I tested were all broken so it seems that the other sites (including http://www.hcs.harvard.edu/~hraaa, http://base.rutgers.edu/pgforms, http://www.wccs.edu/employees, http://base.rutgers.edu/mirna, http://www.calstatela.edu/faculty/jperezc/students/oalamra, and http://institute.beacon.edu/), were cleaned up long ago. There is probably some money to be made in running a service that downloads all content from a web site and/or has a firewall device that sniffs all content that is sent out and makes sure that it seems to be what is intended (about half the URLs in question appear to relate to content that is illegal under US law). As an aside, I did a few other live.com searches for various sites and the word “porn” and found one Australian university running a forum with somewhat broken HTTP authentication that has some interesting posts about porn etc. I’m not going to provide a link because the content didn’t appear to violate Australian law and you expect some off-topic content on a forum. But to be fair, live.com have significantly improved their service since last time I tested it [3]. Now a search for “bonnie” or “bonnie++” will give me the top two spots which is far better than the previous situation. Although I have to admit that the Google result of not giving me a high ranking for “bonnie” is probably better.

• [1] http://etbe.coker.com.au/2007/08/12/porn-for-children/


• [2] http://etbe.coker.com.au/2007/07/30/porn-vs-rape/


• [3] http://etbe.coker.com.au/2007/04/02/livecom-malware/

Share This

Keeping pristine upstream tarballs around is a pain. Wouldn't it be nice to be able to keep them in revision control? Except, it would use far too much disk... Here's a solution. It generates an binary delta between the pristine upstream tarball and a tarball created using files checked out of the repository. The delta should be quite small, and any checkout of the repository that has the same file contents as that used to create the delta can be used to regenerate the pristine tarball. Example (which would presumably be more fun and faster if I used git):

joey@kodama:~package/uqm-voice>svn switch svn+ssh://uqm.debian.net/srv/svn/uqm/tags/uqm-voice/upstream_version_0.3
D    debian
A    comm/blackur/black041.ogg
U    comm/shofixt/shofi040.ogg
U    comm/starbas/starb182.ogg
D    comm/slyland/slyla030.ogg
U    comm/chmmr/chmmr035.ogg
U    comm/supox/supox031.ogg
Updated to revision 223.
joey@kodama:~package/uqm-voice> pristine-tar extract ~/uqm-voice.delta ../uqm-voice_0.3.orig.tar.gz

The generated tarball is bit-identical to the 19 MB upstream tarball, though you have to gunzip them both to check this, since the gzip compression differs. The delta file is all of 41k large. The uqm-voice.delta file was created earlier as follows:

joey@kodama:~> pristine-tar stash lib/debian/unstable/uqm-voice_0.3.orig.tar.gz uqm-voice.delta

I've uploaded pristine-tar to incoming.

I've just "released" debtree 0.6.0. The number of options has almost doubled! It's now possible to suppress Conflicts and to generate full dependency graphs (including packages that are excluded by default). Unknown packages are now marked with a light reddish shading; this could possibly help finding obsolete Conflicts. I've also added verbosity options and fixed a few minor issues. But I'm most exited by two completely new features. Build dependency graphs Besides dependency graphs for binary packages, debtree now also generates build dependency graphs for source packages. It shows both Build-Depends and Build-Depends-Indep dependencies. It also takes into account architecture conditionals (such as [!s390]). It is possible to show such conditionals in the graph, but by default it will just omit any build dependencies not relevant for the selected architecture. Here's an example for dpkg. I expected this to be a lot harder to implement, but it proved relatively straightforward. I've been quite pleased with how extensible this script has proven. Show installed packages With this feature debtree gets back to its original purpose: providing information about what packages will get installed and why. When this option is activated, debtree will shade packages that are already installed light green. So if a graph is generated for a package that is not yet installed, it is easy to see what dependencies are already satisfied and what new packages will be installed. More importantly, you can also see which dependencies could be avoided because they are only recommended or which extra suggested goodies could be installed. Here's an example for clamav. This feature can of course also be used for build dependency graphs. Want to give it a try? The script or the Debian package can be downloaded from the debtree web page. That also has usage information and more examples. P.S. I've found yet another example of an undesired dependency on gconf2.

---

Update: fixed broken link

It seems that the majority of blog traffic (at least in blogs I read) is time-based. It is personal-diary entry posts, references to current events, predictions about future events, bug reports, and other things that either become obsolete or for which it’s important to know the date. For such posts it makes sense to have the date be part of the Permalink URL, and in the normal course of events such posts will tend not to be updated after release. Another type of blog traffic is posts that have ongoing reference value which will (ideally) be actively maintained to keep them current. For such posts it makes sense to have no date stamp in the Permalink - for example if I update a post about installing SE Linux on Etch once Lenny is released (a significant update) I don’t want people ignoring it when it comes up in search engines (or worse having search engines score it down) because the URL indicates that it was written some time before the release of Etch. Wordpress supports Pages as separate entities to Posts, and the names of Pages are direct links under the root of the Wordpress installation. However there is no RSS feed for Pages (AFAIK - I may have missed something) and the Wordpress themes often treat Pages differently (which may not be what you want for timeless posts). Also it is not unreasonable to have Pages and timeless posts. I’m thinking of creating a separate Wordpress installation for posts that I intend to manage for long periods of time with updates (such as documenting some aspects of software I have written). The management options for a blog server program provide significant benefits over editing HTML files. The other option would be to use a different CMS (a blog server being a sub-category of a CMS) to store such things. What I want is a clear way of presenting the data with minimal effort from me (an advantage of Wordpress for this is that I have already invested a significant amount of effort in learning how it works) and editing from remote sites (the offline blog editing tools that are just coming out is a positive point for using a blog server - particularly as I could use the same editor for blog posts and documents). Any suggestions as to how to do this? Then of course there’s the issue of how to syndicate this. For my document blog (for want of a better term) I am thinking of updating the time-stamp on a post every time I make a significant change. If you subscribe to the document feed than that would be because you want to receive new copies of the documents as they are edited. The other option would be to not change the time-stamp and then include the feed along with my regular blog feed (making two feeds be served as one is not a technical challenge). If I was to update the time stamps then I would have to write posts announcing the release of new documents. Does anyone know of someone who writes essays or howto documents in a similar manner to Rick Moen [1] or Paul Graham [2] who also does daily blog posts? I’d like to see some examples of how others have solved these problems (if there are any).

• [1] http://linuxmafia.com/


• [2] http://www.paulgraham.com/

Share This

Randomisation is a good thing for media players, but it must be done properly. What do I mean by this? I mean randomness which only works going forwards. As an example consider the case where you have several films/videos in a directory and you run this:

xine --loop=shuffle *.mpeg *.avi

This appears to do what you want, showing each film in a random order but it is subtly doing the wrong thing. If you're watching film "two", having just finished film "five" you'd expect to be able to return to film "five" by pressing "previous"/Page-Up - however that doesn't happen. Xine decides that "moving backwards" in the playlist means selecting another film at random, rather than the previously viewed film which was selected at random - the two are different - and the current behaviour is both counter-intuitive and plain wrong IMHO. I forget the name of the media player that I used briefly which also behaved like this, but it was enough to make me abandon it almost immediately. So, think of the children. If you have a random mode, make sure that going to previous items actually does return to previously played items and not random ones.

I use a rather simple CRM114-based setup to handle spam in my incoming mail. Nothing really funky here as I essentially used examples from the Debian package's examples/ directory. The only personal touch is using a two-step method for this. All mails are received on my home server and delivered to the "bubulle" user there. A non aggressive CRM114 setup runs there for this user. Mails identified as spam are sorted in a specific folder on the server and are theoretically examined regularly. Indeed, this is something I can't do anymore, given the number of identified spams (about 1000 per day, most of them in Debian mailing lists and Debian BTS). After a quick look, this spam is re-teached as being spam to CR114, then stored as backup in a giant mail folder, which allows me to retrieve a possible false positive later. All other mails are left in /var/mail/bubulle waiting for my laptop to grab them. This gives very few false positives, but still lets a lot of spam slip through. This is why I also run CRM114 on my laptop when mail is grabbed from my home server (I use fetchmail between my laptop and my home server, wherever I am). There, spams are again identified (more aggressively than what's done on my home server) by CRM114. The identified spams are sorted in a folder which I routinely investigate carefully for false positives (there are several a day, but the numbers are low because of the prefiltering done on my home server). Spams are teached as spam to CRM114 and, of course, ham is teached as "ham". I recently added the following to my mutt setup so that SPAM/UNSURE/HAM mails are easy to identify in mutt's index:

spam "X-Spam-Status: Yes"                     "90+/SA"
spam "X-CRM114-Status: SPAM"                     "90+/CRM"
spam "X-CRM114-Status: UNSURE"                     "40+/CRM"
set spam_separator=", "
color index red       default    ~H90    # Spam
color index cyan       default    ~H40    # Unsure

With this, messages identified as "Spam" are shown in red in the index, "unsure" messages are showed in cyan and regular messages have their normal colors. This is really helpful as I can spot messages that are "unsure" and very quickly tell CRM114 that they're ham or spam, with:

macro index \es ""s/home/bubulle/Mail/Administration_systeme/Spams\ncrmlearn --learnspam\n" "crm114 learn as spam"
macro index \eh "crmlearn --learnnonspam\n" "crm114 learn as ham"
macro pager \es ""s/home/bubulle/Mail/Administration_systeme/Spams\ncrmlearn --learnspam\n" "crm114 learn as spam"
macro pager \eh "crmlearn --learnnonspam\n" "crm114 learn as ham"

with the "crmlearn" script being:

#!/bin/sh
 grep -a -v "^X-CRM114"   \
 $HOME/.crm114/mailfilter.crm -u $HOME/.crm114/ $1 --force   \
 grep -a "^X-CRM114"

Shortly speaking, for any cyan message I see in mutt, I hit "ESC-h" if I consider it to be ham or "ESC-s" if I consider it to be spam. Finally, all mails identified as spam are grouped in a dedicated mail folder (/home/bubulle/Mail/Administration_systeme/Spams above) which is sent daily to my home server, where it is used to teach CRM114 about spam, then deleted on my laptop. All this makes spam handling pretty easy with a not so complicated setup (the trickiest part is the nightly laptop->home server synchronization).

(or: Why Everyone Should Learn Some Basic Programming) I am an organisational disaster area. I just can't remember "event oriented" information (like "I need to do $foo" or "I need to do $bar at $time"). I think I've accumulated such a huge collection of technical trivia that it's taken over the event planning part of my brain -- although, since I've never been able to remember event oriented information, even before I became a computer geek, it's more likely that I was just built without that bit of the brain. So I guess I'm the perfect target market for some sort of external brain pack (a la "Ratbert the Consultant"). I've tried paper-based organisers (the mighty Filofax), a few pre-written apps, and I even bought an SL-C3200 Zaurus a while ago (although, to be fair, that was really just for the hack factor, regardless of what justification I may have given to others at the time). None have worked, for various reasons. I can't grep paper, and Other People's Apps bug me because they've got their idea of workflow, and I've got mine, and when they don't match up (and they never do) I just get irritated -- not to mention the fact that the ubiquitous relational database, while wonderful in some ways, does tend to get in the way of ad-hoc faffing with your data. But this article is a success story, not a rant (gasp!), so it's safe for you to assume that I've got something that's working for me. My method is real simple -- plain text files, managed by vim-outliner and a simple shell one-liner I've named 'pda', which just runs vim ~/pda/$1.otl. At the moment I've got a monster todo.otl, which is my world's todo list, and a deeply underused calendar.otl. I'm also using vim-outliner to do some project planning, which has been moderately successful, although I'm a bit freaked out by the idea of trying to manage a large-scale project using it. The really, really cool part of this whole shebang, though, is that because all of my data is (structured) plain text, writing little programs to parse the data and do funky things with it is dead simple. For example, in my project outlines, I put estimated, actual, and remaining times in for all tasks (a la Joel Spolsky's Painless Software Schedules) so I can keep track of my estimates and where they differ from reality (because one day I'd like to not Suck at estimating times, and feedback is the only way I'm going to do that). So, about an hour's worth of Ruby gave me "estimator", a script that reads all that data and does a pile of mathematical gubbins about how far over my estimate I am so far (or under estimate, if that were to ever happen), the percentage of the project complete (based on the estimates and number of tasks completed), and things like that. Similarly, today I wrote a small script (run out of cron in the wee hours of the morning) to read my calendar and send me an e-mail if I've got anything in there for today. Since I practically live in mutt, a morning e-mail is going to be far more effective than my trying to remember to check my calendar every morning (especially since, at present, I'm not using the calendar as often as I should, so there's even less motivation to check it). This, in my opinion, is why everybody (literally, every single person) should learn to program at some fairly simple level, like learning to read and write. By using plain text and small scripts, I've got a workflow that works for me, and it's cost me less time than I'd spend learning some large pre-written app and putting all my data into it. Being able to manipulate data like this is, I think, a fairly important tool in the modern world, and I don't think it should be left to any sort of priesthood of developers -- it should be as universal as most countries try to make literacy and numeracy.

Package dependency graphs on steroids

How an innocent and simple question can lead to strange development.

I was looking for a simple command that would tell me what the dependency path is from one package to some other package. That is, without having to go by trial-and-error using e.g. aptitude. Actually, I am getting quite annoyed by the fact that I regularly see e.g. gconf2 being pulled in when I install something on my (KDE-based) systems and I wanted to see how exactly they got pulled in. It should be possible to install e.g. xchat without being forced to install gconf2 too, right? For Sarge I had a similar issue with the gimp. Anyway, possibly there already is something to print dependency paths, but I did not find it. As I had some time free after dropping out of the Installer team, I decided to scratch my own itch. To get the paths I decided to recurse through dependencies using grep-available and friends, basically building a tree. An old wish was to play with dot, so I thought it would be nice to visualize the tree and so verify my code (conveniently forgetting about apt-cache dotty...). I soon had something basic working and after that it turned out to be irresistible (and surprisingly easy) to keep adding functionality. The result is that I can now produce graphs like: This is a fairly simple example. A lot of graphs are much more interesting. Oh, yes, it also answers my original question:

$ debtree xchat gconf2
xchat -> xchat-common (D) -> gconf2 (D)

The script is quite flexible and now looks solid enough for a first (Beta) release. After a few optimizations the shell script is even acceptably fast (or at least not slow) on my box. Euhh, what? Shell script? Yes. As I don't really know any of the popular languages used in Open Source, writing this in e.g. perl would just have taken too much time. OTOH, after some three years of working on D-I, I have become quite fluent in shell script. After all, about 80% of the installer (almost all functionality that does the actual installations — including the partitioner) is written in shell, so complex systems in shell are perfectly possible; thanks to Joey we even have a web server in shell! And it keeps surprising me how structured you can work in shell if you try a little. So there. Of course I'd like to see debtree re-implemented in another language, mainly to improve performance. Probably in C++ as that is what apt-cache is written in and its 'dotty' function could be a nice starting point. You'll understand from the above that I'm not going to do this myself, so if anybody is interested in working with me to do this, please contact me. debtree seems like it would be a nice addition to Debian, maybe even alongside apt-cache in apt... Enjoy!

There is a wide-spread myth that swap space should be twice the size of RAM. This might have provided some benefit when 16M of RAM was a lot and disks had average access times of 20ms. Now disks can have average access times less than 10ms but RAM has increased to 1G for small machines and 8G or more for large machines. Multiplying the seek performance of disks by a factor of two to five while increasing the amount of data stored by a factor of close to 1000 is obviously not going to work well for performance. A Linux machine with 16M of RAM and 32M of swap MIGHT work acceptably for some applications (although when I was running Linux machines with 16M of RAM I found that if swap use exceeded about 16M then the machine became so slow that a reboot was often needed). But a Linux machine with 8G of RAM and 16G of swap is almost certain to be unusable long before the swap space is exhausted. Therefore giving the machine less swap space and having processes be killed (or malloc() calls fail - depending on the configuration and some other factors) is probably going to be a better situation. There are factors that can alleviate the problems such as RAID controllers that implement write-back caching in hardware, but this only has a small impact on the performance requirements of paging. The 512M of cache RAM that you might find on a RAID controller won’t make that much impact on the IO requirements of 8G or 16G of swap. I often make the swap space on a Linux machine equal the size of RAM (when RAM is less than 1G) and be half the size of RAM for RAM sizes from 2G to 4G. For machines with more than 4G of RAM I will probably stick to a maximum of 2G of swap. I am not convinced that any mass storage system that I have used can handle the load from more than 2G of swap space in active use. The reason for the myths about swap space size are due to some old versions of Unix that used to allocate a page of disk space for every page of virtual memory. Therefore having swap space less than or equal to the size of RAM was impossible and having swap space less than twice the size of RAM was probably a waste of effort (see this reference [1]). However Linux has never worked this way, in Linux the virtual memory size is the size of RAM plus the size of the swap space. So while the “double the size of RAM” rule of thumb gave virtual memory twice the size of physical RAM on some older versions of Unix it gave three times the size of RAM on Linux! Also swap spaces smaller than RAM have always worked well on Linux (I once ran a Linux machine with 8M of RAM and used a floppy disk as a swap device). As far as I recall some time ago (I can’t remember how long) the Linux kernel would by default permit overcommitting of memory. For example if a program tried to malloc() 1G of memory on a machine that had 64M of RAM and 128M of swap then the system call would succeed. However if the program actually tried to use that memory then it would end up getting killed. The current policy is that /proc/sys/vm/overcommit_memory determines what happens when memory is overcommitted, the default value 0 means that the kernel will estimate how much RAM and swap is available and reject memory allocation requests that exceed that value. A value of 1 means that all memory allocation requests will succeed (you could have dozens of processes each malloc 2G of RAM on a machine with 128M of RAM and 128M of swap). A value of 2 means that a different policy will be followed, incidentally my test results don’t match the documentation for value 2. Now if you run a machine with /proc/sys/vm/overcommit_memory set to 0 then you have an incentive to use a moderately large amount of swap, safe in the knowledge that many applications will allocate memory that they don’t use, so the fact that the machine would deliver unacceptably low performance if all the swap was used might not be a problem. In this case the ideal size for swap might be the amount that is usable (based on the storage speed) plus a percentage of the RAM size to cater for programs that allocate memory and never use it. By “moderately large” I mean something significantly less than twice the size of RAM for all machines less than 7 years old. If you run a machine with /proc/sys/vm/overcommit_memory set to 1 then the requirements for swap space should decrease, but the potential for the kernel to run out of memory and kill some processes is increased (not that it’s impossible to have this happen when /proc/sys/vm/overcommit_memory is set to 0). The debian-administration.org site has an article about a package to create a swap file at boot [2] with the aim of making it always be twice the size of RAM. I believe that this is a bad idea, the amount of swap which can be used with decent performance is a small fraction of the storage size on modern systems and often less than the size of RAM. Increasing the amount of RAM will not increase the swap performance, so increasing the swap space is not going to do any good.

• [1] http://wombat.san-francisco.ca.us/faqomatic/cache/53.html


• [2] http://www.debian-administration.org/articles/550

Share This

My patch has been merged during my night, yay! And it even was merged in nice revision number 1234 :-). If you want to see example, you can check digiKam screenshot, which Gilles Caulier has quickly generated.

Sometimes you need direct make to progress through various phases which may not have a particular file target associated with them. For example the first step might be to extract a compressed tarball of data and run a script over it in preparation for compiling that data into something else. Equally you might need to patch some files or maybe download some data. The general idiom to accomplish this sort of thing is the stamp file. The general principle is to make a phony target that depends on a file called target-stamp. The target-stamp target can then do what it needs to do and simply create a dummy file via touch. This file indicates to make that the phase is complete, and targets that depend on this phase being complete can now depend on the stamp file. A small example follows One place you may notice extensive use of this is the -stamp files in the /debian directory of packages built with CDBS.