From Debian I downloaded, installed and toyed around with Liquidsoap. It took me a few moments to understand the language, but once I did, I got very enthusiastic about it. I always wanted something more than just streaming a set for my friends now and then and this tool allows me to created a quite contrived webradio setup. More about that later… To Debian unstable I finally uploaded Camping, a small Ruby web framework for MCV type applications. It is in NEW now. I’ve been writing (or at least started writing) quite a few apps based on Camping the past few months. Primarily because it is fun, but also because I like its flexibility and strength very much (see also some examples). Once Camping is in Debian, I’d like to put more Camping apps out in the open.

Switching from aterm to xterm I found it annoying that xterm didn’t select a full url when i did a double-click somewhere on it, like aterm did. Now, doing a little search on it I found something way stronger. Take a url like http://foo.bar/baz for the following example. If you now put something like the following into your .Xdefaults (and load it with xrdb -load .Xdefaults), you get xterms selection in a more useful way. A double click on foo will select foo.bar, clicking three times it selects the full url (well, better speak is to say it selects everything separated by whitespace). Click four times and it selects everything up to the start of the prompt (default prompt that is). 5 clicks selects the whole line.

XTerm*on2Clicks:  regex [^/@ \n]+
XTerm*on3Clicks:  regex [^ \n]+
XTerm*on4Clicks:  regex [^#$]+
XTerm*on5Clicks:  line

Recently, GozRita unveiled the names of our two Falleretes majors for 2008's Falles festivities. All the free newspapers did some extensive coverage, with reports on who they are and what they do. Qu Valencia interviewed the little Fallera major, and then posted this:

Little Victoria learns Valencian So, Victoria Bl zquez speaks English and Valencian "nearly perfectly". Great! I think having newspapers treat Valencian as if it were just another foreign language that students are forced to learn is a great example of the dark future our language will face in just a few generations.

Yesterday's post about switching back to Emacs saw quite a few comments from people (most of them useful, even). I learned a few things.

My biggest gripe about Vim was that for the file types I worked with most, its indentation and syntax highlighting was inferior to that of Emacs. I'd like to illustrate that with an example.

Let's consider one of those file types: XML containing DocBook markup.

Vim has a DocBook mode. It doesn't autodetect DocBook files, so I have this at the top of each one:

<!-- vim: set filetype=docbkxml shiftwidth=2 autoindent expandtab tw=77 : -->

Now, why should Vim need a separate DocBook mode? DocBook is just XML or SGML, and these things have a well-formed nature. Well, part of the reason is that /usr/share/vim/vim71/syntax/docbk.vim has a ton of lines like this:

syn keyword docbkKeyword chapter citation citerefentry citetitle city contained

Yes, they are hard-coding all the DocBook element names into the editing mode. It's probably used for completion, highlighting, maybe indentation. I'm not sure, really. I remember that editing these files without the DocBook mode was much more painful anyway, but that was 8 months ago and I can't quite remember why.

Now, what about Emacs? I don't know if Emacs even has a DocBook mode, mainly because I don't have to care. The Emacs psgml mode actually parses the DTD for your XML or SGML files. It knows exactly what the valid tags are from doing so. This means it has full functionality not just for DocBook, but for any XML or SGML file with a DTD.

Not only that, but it knows more about the files than Vim does. For instance, both Emacs and Vim can do completion of various things. Vim </ C-x C-o (ooo, sounds like Emacs!) can complete my closing tags. But it can't autocomplete my opening tags, and it certainly isn't aware

Not only can Emacs autocomplete opening and closing tags, but it knows exactly what tags are valid at a given place in the document (thanks to the DTD) and will only consider those tags for completion. Moreover, depending on how you have configured it, it could also insert spots for you to add any required attributes. So, for instance, if you're editing XHTML and autocompletion gives you an <img> tag, it would add src="" in it for you, as a reminder that src is required.

There are a host of other smart things that Emacs can do with XML or SGML documents. For instance, you can get a list of all tags valid at the current point with C-c C-t or Shift-RightClick -- useful if you've forgotten the name of a tag for a moment.

The difference isn't as great with everything. But it sure is noticable as I work with XML and Haskell files.

Other Unix systems apparently calculate the load average differently to Linux. According to the Wikipedia page about Load(computing) [1] most Unix systems calculate it based on the average number of processes that are using a CPU or available for scheduling on a CPU while Linux also includes the count of processes that are blocked on disk IO (uninterruptible sleep). There are three load average numbers, the first is for the past minute, the second is for the past 5 minutes, and the third is for the past 15 minutes. In most cases you will only be interested in the first number. What is a good load average depends on the hardware. For a system with a single CPU core a load average of 1 or greater from CPU use will indicate that some processes may perform badly due to lack of CPU time - although a long-running background process with a high “nice” value can increase the load average without interfering with system performance in most cases. As a general rule if you want snappy performance then the load average component from CPU use should be less than the number of CPU cores (not hyper-threads). For example a system with two dual-core CPUs can be expected to perform really well with a load average of 3.5 from CPU use but might perform badly with a load average of 5. The component of the load average that is due to disk IO is much more difficult to interpret in a sensible manner. A common situation is to have the load average increased by a NFS server with a network problem. A user accesses a file on the NFS server and gets no response (thus giving a load average of 1), they then open another session and use “ls” to inspect the state of the file - ls is blocked and gives a system load average of 2. A single user may launch 5 or more processes before they realise that they are not going to succeed. If there are 20 active users on a multi-user system then a load average of 100 from a single NFS server that has a network problem is not uncommon. While this is happening the system will perform very well for all tasks that don’t involve the NFS server, the processes that are blocked on disk IO can be paged out so they don’t use any RAM or CPU time. For regular disk IO you can have load average incremented by 1 for each non-RAID disk without any significant performance problems. For example if you have two users who each have a separate disk for their home directory (not uncommon with certain systems where performance is required and cooperation between users is low) then each could have a single process performing disk IO at maximum speed with no performance problems for the entire system. A system which has four CPU cores and two hard drives used for separate tasks could have a load average slightly below 6 and the performance for all operations would be quite good if there were four processes performing CPU intensive tasks and two processes doing disk intensive tasks on different disks. The same system with six CPU intensive programs would under-perform (each process would on average get 2/3 of a CPU), and if it had six disk intensive tasks that all use the same disk then performance would be terrible (especially if one of the six was an interactive task). The fact that a single load average number can either mean that the system is busy but performing well, under a bit of load, or totally overloaded means that the load average number is of limited utility in diagnosing performance problems. It is useful as a quick measure, if your server usually has a load average of 0.5 and it suddenly gets a load average of 10 then you know that something is wrong. Then the typical procedure for diagnosing it starts with either running “ps aux grep D” (to get a list of D state processes - processes that are blocked on disk IO) or running top to see the percentages of CPU time idle and in IO-wait states. Cpu(s): 15.0%us, 35.1%sy,  0.0%ni, 49.9%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
7331 rjc      25  0  2868  640  312 R  100  0.0  0:21.57 gzip Above is a section of the output of top showing a system running gzip -9 < /dev/urandom > /dev/null. Gzip is using one CPU core (100% CPU means 100% of one core - a multi-threaded program can use more than one core and therefore more than 100% CPU) and the overall system statistics indicate 49.9% idle (the other core is almost entirely idle). Cpu(s):  1.3%us,  3.2%sy,  0.0%ni, 50.7%id, 44.4%wa,  0.0%hi,  0.3%si,  0.0%st   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
7425 rjc      17  0  4036  872  588 R    4  0.1  0:00.20 find Above is a section of the output of top showing the same system running find /. The system is registering 44% IO wait and 50.7% idle. The IO wait is the percentage of time that CPU core is waiting on IO, so 44% of the total system CPU time (or 88% of one CPU core) is idle while the system is waiting for disk IO to complete. A common mistake is to think that if the IO was faster then more CPU time would be used, in this case with the find program using 4% of one CPU core if all the IO was instantaneous (EG in cache) then the command would complete 25 times faster with 100% CPU use. But if the disk IO performance was doubled (a realistic possibility given that the system has a pair of cheap SATA disks in a RAID-1) then find would probably use 8% of CPU time. Really the only use for load average is for getting an instant feel for whether there are any performance problems related to CPU use or disk IO. If you know what the normal number is then a significant change will stand out. Dr. Neil Gunther has written some interesting documentation on the topic [2], which goes into more technical detail including kernel algorithms used for calculating the load average. My aim in this post is to educate Unix operators as to the basics of the load average. His book The Practical Performance Analyst gives some useful insights into the field. One thing I learned from his book is the basics of queueing theory. One important aspect of this is that as the rate at which work arrives approaches the rate at which work can be done the queue length starts to increase exponentially, and if work keeps arriving at the same rate when the queue is full and the system can’t perform the work fast enough the queue will grow without end. This means that as the load average approaches the theoretical maximum the probability of the system dramatically increasing it’s load average increases. A machine that’s bottlenecked on disk IO for a task where there is a huge number of independent clients (such as a large web server) may have it’s load average jump from 3 to 100 in a matter of one minute. Of course this won’t mean that you actually need to be able to serve 30 times the normal load, merely slightly more than the normal load to keep the queues short. I recommend reading the book, he explains it much better than I do.

• [1] http://en.wikipedia.org/wiki/Load_(computing)


• [2] http://www.teamquest.com/resources/gunther/display/5/index.htm

Share This

Some months ago, I commented about a weak implementation in a fancy captcha. Today I would like to comment about other bad implementations, but in other ways.
The goodA captcha should have big Shannon entropy, finite, but big. The session ID and the challenge must not be reused. The images must be resistant to OCR but should be understandable by a human.
The badHere is the first example:

Believe it or not.. This is a real case. So incredible eh?

The uglyThe victim, in this case, is this one:
This is an implementation of captchanumbers, by Hadar Porat. This captcha and many others generated by captchanumbers can be read with this script.

The idea is simple. As the numbers are nearly in the same place, they can be cut. Those parts can compared independently, reducing the entropy. May be the script and this image would be more enlightening than my limited English:

The 10,000 possibilities was reduced to 159. No OCR, 100% deterministic.

Second moral: Understand the fundamentals first, write code later.

Today, it's been two years I'm stuck in the NM queue. I've registered on oct 24, 2005, because Sam Hocevar told me that it could last a little, so if I was interested I should register early. I wondered for a long time if I should resign on this two years birthday, but I dropped this eventuality. I still want to be a Debian Developer, and if I resign I don't think I'll really want to still work on Xfce packages. But on the other hand I wouldn't really like to use an Xfce not packaged by me (no offense, but I guess I'd still want to hack on it) and I don't think I would be able to be on another distro. So I'm still there, sitting in NM queue, waiting for the next step. The only required step to be accepted as a Debian Developer is to have its gpg key added to the debian keyring (then an account is created on debian boxes, there's a -private subscription etc.). This is done by a team, called DAM, "Debian Accounts Managers". Because at one time they had a lot of work, they could not examine each candidate and decide if he was a potential good Debian Developer, so people imagined the NM Process: some volunteers would help them asking questions to candidates, then write a report, so DAM would only have to read it and decide if it was ok or not to create an account. I've already passed this step. I am "DAM Approved". One could imagine that, if the Debian *Accounts* Managers reviewed and approved my application, the account creation should follow. But no, the next step (according to NM pages) is "DAM creates account or rejects application". The problem here is that there is no "DAM Team", responsible for Debian accounts, but various people which are not really equivalent in this team. This is really a problem in Debian, where teams aren't really a "team". Not every team I guess, but in some cases one can trust a team to do a job. Debian is a volunteer organisation, so people think nobody can be forced to do a job if he doesn't have time to it. I'm fine with that, real life can be really time-consuming sometime. But my opinion is that teams may have to be forced to do their job. If nobody in the team has time, fine, it means somebody should join the team. But the team itself shouldn't fail, and this should be the only thing visible from outside: the "team", and the team actions. There are some proposition about this on lists at the moment, but I don't really know how it'll end. It's not easy doing social stuff, there's a lot of people involved, with many different views. But sometimes the consensus is *not* findable, and sometimes, somebody has to decide. That's why why need team leaders, and that's the project leader role. I guess people are quite opposed to have a "strong" leader in a volunteer organization, and I don't have concrete proposals but I really thing team management is not really perfect and a lot of people (users and developers) suffer from that (not only NM applicants, I mean) and from "elephants". As I said, I have no good solution, nor even concrete proposals (except "sometimes we need a leader to make decisions") but I really think there are things Debian (as a project) needs to improve.

Recently it has come up on the LKML and other blogs about how C is not thread safe. The nice piece of example code given is At first glance it seems OK, and I wasn't exactly sure what the problem was until I pulled it apart. GCC is well within its rights to produce the following code (annotated for clarity). This code uses a bit of a trick to avoid a branch around code. It uses the carry flag, which is simple enough to understand when adding. Consider if you are adding 0001 + 1111 you end up with 11111, where the top bit is the carry bit. You can then feed this carry bit into another part of the addition operation if you were, say, using two 32 bit numbers to represent a 64 bit number on a 32 bit system. The trick uses the dual nature of the carry flag as a "borrow" bit; if a < b when a - b then the bit is set. Consider the simplest case of 0 - 1 -- the carry bit is set to indicate to the next operation that a bit needs to be borrowed. Imagine for a moment we have a four bit machine and want to do the 8-bit unsigned operation First we do 0000 - 0001 which leaves us 1111 with the borrow bit set. Then we do 1111 - 0000 but take note that the borrow bit is set, so get rid of the borrowed bit, leaving us 1110. Putting the two together gives us 1110 1111 or 239. Therefore the cmp $0x1, instruction (which is really just result - 1 with no output but setting flags) can only set the carry flag (i.e. borrow a bit) if the value is 0; any other value in result means there will be no need to borrow and the carry flag is not set. Consequently the add with carry (adc) will add one to the result only if the carry bit was set. A neat trick which avoids a jump, but not so much thread safe: if you are interrupted after the cmp but before the mov store you can write back the wrong value. Most people (including me till 5 minutes ago) looking at the code would assume that it was safe on the theory that acquires_count is not touched if you don't have the lock because it is inside the if statement. Interestingly, the Intel compiler does the simpler jump method Frankly I have no idea which one is faster/better (I would guess the second one because you don't always dirty a cache line, but you'd have to measure that against the cost of the branch) but obviously GCC thinks one thing and Intel thinks another and I bet if you put them in a room together you would end up with 3 opinions! I think the major point is that once upon a time it was mostly people writing kernels doing things like accessing I/O ports or other memory mapped areas that needed to be very careful the compiler didn't optimise out (or in) something incorrectly. With multi-threaded C code that problem moves up to the application developer too. A different instruction set can help here. The Itanium version is possibly less prone to this sort of thing because predication makes it faster to implement conditional branches -- any instruction with a predicate that is not set is skipped. The load and add is safe because it will only be done with the lock held. However, if anyone is wondering why Itanium come with such big caches, the difference between the 6 line version and the Itanium version below should answer that. I guess the moral of the story is that multi-threading is hard, but we knew that anyway.

I have previously posted about the difference between using a chroot and using SE Linux [1]. Theo de Raadt claims that virtualisation does not provide security benefits [2] based on the idea that the Xen hypervisor may have security related bugs. From my understanding of Xen a successful exploit of a Xen system with a Dom0 that is strictly used for running the DomU’s would usually start by gaining local root on one of the DomU instances. From there it is possible to launch an attack on the Xen Dom0. One example of this is the recent Xen exploit (CVE-2007-4993) [3] where hostile data in a grub.conf in a DomU could be used to execute privileged commands in the Dom0. Another possibility would be to gain root access to a DomU and then exploit a bug in the Xen API to take over the hypervisor (I am not aware of an example of this being implemented). A final possibility is available when using QEMU code to provide virtual hardware where an attacker could exploit QEMU bugs, an example of this is CVE-2007-0998 where a local user in a guest VM could read arbitrary files in the host [4] - it’s not clear from the advisory what level of access is required to exploit it (DomU-user, DomU-root, or remote VNC access). VNC is different from other virtual hardware in that the sys-admin of the virtual machine (who might be untrusted) needs to access it. Virtual block devices etc are only accessed by the DomU and Xen manages the back-end. The best reference in regard to these issues seems to be Tavis Ormandy’s paper about hostile virtualised environments [5]. Tavis found some vulnerabilities in the QEMU hardware emulation, and as QEMU code is used for a fully virtualised Xen installation it seems likely that Xen has some vulnerabilities in this regard. I think that it is generally recommended that for best security you don’t run fully virtualised systems. The remote-console type management tools are another potential avenue of attack for virtualised servers in the case where multiple users run virtual machines on the same host (hardware). I don’t think that this is an inherent weakness of virtualisation systems. When security is most important you have one sys-admin running all virtual machines - which incidentally seems to be the case for most implementations of Xen at the moment (although for management not security reasons). In ISP hosting type environments I doubt that a remote console system based on managing Xen DomU’s is going to be inherently any less secure than a typical remote console system for managing multiple discrete computers or blades. I have just scanned the Xen hypervisor source, the file include/asm-x86/hypercall.h has 18 entries for AMD64 and 17 for i386 while include/xen/hypercall.h has 18 entries. So it seems that there are 35 or 36 entry points to call the hypervisor, compared to 296 system calls on the i386 version of Linux (which includes the sys_socketcall system call which expands to many system calls). This seems to be one clear indication that the Linux kernel is inherently more complex (and therefore likely to have a higher incidence of security flaws) than the Xen hypervisor. Theo’s main claim seems to be that Xen is written by people who aren’t OpenBSD developers and who therefore aren’t able to write secure code. While I don’t agree with his strong position I have to note the fact that OpenBSD seems to have a better security history than any other multi-user kernel for which data is available. But consider a system running Xen with Linux in Dom0 and multiple para-virtualised OpenBSD DomU’s. If the Linux Dom0 has OpenSSH as the only service being run then the risk of compromise would be from OpenSSH, IP based bugs in the Linux kernel (either through the IP address used for SSH connections or for routing/bridging to the OpenBSD instances), and from someone who has cracked root on one of the OpenBSD instances and is attacking the hypervisor directly. Given that OpenSSH comes from the OpenBSD project it seems that the above scenario would only add the additional risk of an IP based Linux kernel attack. While a root compromise of an OpenBSD instance (consider that a typical OpenBSD system will run a lot of software that doesn’t come from the OpenBSD project - much of which won’t have a great security history) would only lose that instance unless the attacker can also exploit the hypervisor (which would be a much more difficult task than merely cracking some random daemon running as root that the sys-admin is forced to install). Is the benefit of having only one instance of OpenBSD cracked due to a bad daemon enough to outweigh the risk of a Linux IP stack? I’m sure that the OpenBSD people would consider that a better option would be OpenBSD in the Dom0 and in the DomU. In which case the risk of damage from a root compromise due to one badly written daemon that didn’t come from OpenBSD is limited to a single DomU unless the attacker also compromises the hypervisor. When working as a sys-admin I have been forced by management to install some daemons as root which were great risks to the security of the system, if I had the ability to install them in separate DomU’s I would have been able to significantly improve the security of the system. Another flaw in Theo’s position is that he seems to consider running a virtual machine as the replacement of multiple machines - which would be an obvious decrease in security. However in many cases the situation is that no more or less hardware is purchased, it is just used differently. If instead of a single server running several complex applications you have a Xen server running multiple DomU’s which each have a single application then things become much simpler and more secure. Upgrades can be performed on one DomU at a time which decreases the scope of failure (which often means that you only need one business unit to sign-off on the upgrade) and upgrades can be performed on an LVM snapshot (and rolled back with ease if they don’t succeed). A major problem with computer security is when managers fear problems caused by upgrades and prohibit their staff from applying security fixes. This combined with the fact that on a multiple DomU installation one application can be compromised without immediate loss of the others (which run in different DomU’s and require further effort by the attacker for a Xen compromise) provides a significant security benefit. It would be nice for security if every application could run on separate hardware, but even with blades this is not economically viable - not even for the biggest companies. I have converted several installations from a single overloaded and badly managed server to a Xen installation with multiple DomU’s. In all cases the DomU’s were easier to upgrade (and were upgraded more often) and the different applications and users were more isolated. Finally there is the possibility of using virtualisation to monitor the integrity of the system, Bill Broadley’s presentation from the 2007 IT Security Symposium [6] provides some interesting ideas about what can be done. It seems that having a single OpenBSD DomU running under a hypervisor (maybe Xen audited by the OpenBSD people) with an OpenBSD Dom0 would offer some significant benefits over a single OpenBSD instance.

• [1] http://etbe.coker.com.au/2007/08/22/se-linux-vs-chroot/


• [2] http://kerneltrap.org/OpenBSD/Virtualization_Security


• [3] http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2007-4993


• [4] http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2007-0998


• [5] http://taviso.decsystem.org/virtsec.pdf


• [6] http://shell.cse.ucdavis.edu/~bill/virt/virt.pdf

Share This

I wrote a few months ago about how Ryanair had started charging for web check-in despite it being a cost saving for them — and how this was not the Ryanair that I understood. Well, I was proved right last week when they switched surcharges. Now you have to pay if you want to check in at the desk, and you get priority boarding and online check-in for free. If only they would start putting some effort into maintaining the uptime of the web check-in site, I would be happy. That said, it beats Aer Lingus which refuses to let you print out a replacement boarding pass if (for example), Firefox crashes on getting the PDF the first time around.

I have a theory. There are two kinds of people in the world: those who see conspiracies everywhere, and those who don't. Personally, I'm part of the latter group; and it annoys me endlessly when people in the former group use conspiracy theories to explain what they don't understand—much in the same way that I don't really understand people who use religion to explain what they don't understand. An example of what I mean: Marc 'HE' Brockschmidt wrote about the stuff he'd like to change in Debian, since he feels many things go wrong. Now even though he doesn't mention the word 'Cabal' anywhere in there, it still reads as someone who sees conspiracy theories in Debian. No hard feelings, Marc, but...

• It doesn't really help to 'enforce' addition of new architectures. Arguments like "I've heard that foo said bar" don't really help, either; please go find out for sure. If you can't or won't, then please do assume that those people who did care enough to find out know what they're doing.
• Adding more official suites, such as volatile or backports, increases the burden on the security team and others. This isn't something that should be done lightly, and so "just" adding a CNAME won't really fix it.
• I don't see why the practice of porting machines should be "resurrected". Yeah, sure, some of them are down, but we all know hardware sucks. That doesn't mean there's malicious intent or a change of policy or anything.
• I'll agree that it's a bad thing that some people in core teams aren't delegates, especially when they don't do their job. There are many core teams of which the members aren't delegates, such as the security team, the listmasters, the maintainers of our default MTA, the d-i team, the kernel team, the X strike force, and a huge bunch of other teams; if and when they stop doing their jobs, that would be a serious problem.
  I also really don't believe it's a good idea to go to a specific team that you don't have any experience with, and start dictating terms.
• With my m68k buildd maintainer hat on, I can tell you that adding more buildd maintainers to any architecture is a bad idea. I'll gladly explain the details in more detail later, if there's any interest.
• You do know about the draft updates to the FDL, right? We've put forward our position, now it's the FSF's turn; and though they considered updating the GPL to be of higher priority, they didn't ignore our opinion, which one can easily see if they read the discussion drafts.

I could go on, but I guess you got the point. Like I said, conspiracy theories annoy the hell out of me. Marc, it isn't personal, you're just the last in a long list. Oh, and BTW: comments to this post will be moderated away.

apt-xapian-index: smart way of querying tags I've recently posted:

• an introduction of apt-xapian-index;
• an example of how to query it;
• a way to add simple result filters to the query;
• a way to suggest keywords and tags to use to improve the query.
• a way to search for similar packages.
• a way to implement an adaptive cutoff on result quality.

Note that I've rewritten all the old posts to only show the main code snippets: if you were put off by the large lumps of code, you may want to give it another go. Today I'll show how to implement a really good way of searching for Debtags tags. When I say really good, I mean the sort of good that after you run it you wonder how could it possibly manage to do it. The idea is simple: you run a package search, but instead of showing the resulting packages, you ask Xapian to suggest tags like we saw in axi-query-expand.py. For extra points, I'll use an adaptive cutoff in chosing the packages that go in the rset. So, let's ask the user to enter some keywords to look for tags, and use them to run a normal package query: Now, instead of showing the results of the query, we ask Xapian what are the tags in the index that are most relevant to this search. First, we pick some representative packages for the expand: Then we define the filter that only keeps tags: Then we print the tags: That's it. We turned a package search into a tag search, and this allows us to search for tags using keywords that are not present in the tag descriptions at all:

$ ./axi-query-tags.py explore the dungeons
27.50 game::rpg:rogue
26.14 use::gameplaying
17.53 game::rpg
10.27 uitoolkit::ncurses
...
$ ./axi-query-tags.py total world domination
7.55 use::gameplaying
5.68 x11::application
5.35 interface::x11
5.05 game::strategy
...

You can use the wsvn interface to get to the full source code and the module it uses. You can see a similar technique working in the Debtags tag editor: enter a package, then choose "Available tags: search".

I have just read Career Development for Geeks [1] by Erik de Castro Lopo [2]. It makes some interesting points about a traditional approach to an IT career. The path I followed for most of my career (after I had a few years experience) was to work as a contractor and happily leave jobs without having anything else lined up. Erik suggests getting Engineers rather than managers to give references, it’s an interesting idea. Engineers can give better references about quality of work, while managers can speak on behalf of their employer (in theory). In practice a sane manager won’t give a bad review for legal reasons so the value of a reference from a manager is probably limited. Of course one problem with reviews is that I have never heard of a recruiting agent or employer actually verifying the ID of a reference. I could be listed on a friend’s CV as a senior manager in a multi-national company (which doesn’t have a major Australian presence) and give a good review and it seems unlikely that the typical recruiter would work it out. For someone with plenty of spare time and no significant assets (no risk of being sued) it could be entertaining to apply for a bunch of positions that they are not qualified for with friends using pre-paid mobile phones to give references. This could be done as a documentary on corporate hiring practices, or to simply try and get the highest paid job possible. Based on observing some former colleagues it seems that little skill is required to get a job and that when people do really badly they get paid for at least a few months. I am constantly amazed when reading reports about so-called “con artists” who commit crimes for what are often small amounts of money. Getting an income significantly greater than average without knowing anything about how to do the work is very common and is never treated as fraud (the classic example was a former colleague who wanted to write his own encryption algorithm but who didn’t even know about binary and therefore couldn’t use operations such as XOR). Erik’s main suggestion for dealing with recruiting agents is to talk about projet management issues (recruiters don’t understand technology). My way of dealing with them has been to assure them that I know it all and tell them to forward my CV to the client. Another suggestion is to learn new skills and diversify your skills. I don’t supoprt this because I believe that the majority of people who read my blog are significantly more skillful than the typical programmer. If an area of technology starts to go out of fashion then it’s the people with the least skills who suffer the most. If you are good at your work and enjoy it then it shouldn’t matter much if people around you are being laid off. Of course to rely on this you have to be working in a reasonably large field. For example if you develop software in a language which has hundreds of programmers then you may be at risk, but if there are tens of thousands of programmers using the language then you only need to be in the most skillful 10% to be assured of employment for the next decade or two. That said there are benefits to moving around, meeting new people, and working on new contracts. One thing you don’t want is to have 10 years of experience which are the same year repeated 10 times! Update: Here is a paper Erik submitted to OSDC on the same topic [3]. Mostly the advice is the same but with more detail and in a better format for reading.

• [1] http://www.mega-nerd.com/Career-dev/career-dev.pdf


• [2] http://www.mega-nerd.com/erikd/Blog


• [3] http://www.mega-nerd.com/erikd/cddg.pdf

Share This

apt-xapian-index: adaptive quality cutoff I've recently posted:

• an introduction of apt-xapian-index;
• an example of how to query it;
• a way to add simple result filters to the query;
• a way to suggest keywords and tags to use to improve the query.
• a way to search for similar packages.

Note that I've rewritten all the old posts to only show the main code snippets: if you were put off by the large lumps of code, you may want to give it another go. Today I'll show how to implement an adaptive cutoff to get rid of the worse results. Recall that Xapian shows results by decreasing order of quality, and as we pull more an more results out, we reach a point where the matches are so approximated that they look random. This can be a problem if we want to change the order of the result, for example we may want to sort by package size, or by popcon popularity. There are many scenarios in which a really bad match could end up at the top of the results. For most cases, you just want to say "discard all results whose quality is less than 70%". But sometimes you have queries that OR lots of terms, and even your top result, while still being a very good result, may be below the cutoff you decided. Implementing an adaptive cutoff is extremely simple: first, you get the quality estimate of the top result: Then you tell Xapian that you want a cutoff value that is, for example, 70% of that: Finally, you repeat the query. If you want, you can go for bigger result sets, as the cutoff will make it so that if you have lots of results, they will very likely be all good results: This is it. You can use the wsvn interface to get to the full source code and the module it uses. Next in the series: smart way of querying tags.

Jonathan Schwartz: “[L]ater this week, we’re going to use our defensive portfolio to respond to Network Appliance, filing a comprehensive reciprocal suit. As a part of this suit, we are requesting a permanent injunction to remove all of their filer products from the marketplace, and are examining the original NFS license - on which Network Appliance was started. By opting to litigate vs. innovate, they are disrupting their customers and employees across the world. “In addition to seeking the removal of their products from the marketplace, we will be going after sizable monetary damages. And I am committing that Sun will donate half of those proceeds to the leading institutions promoting free software and patent reform (in specific, The Software Freedom Law Center and the Peer to Patent initiative), and to the legal defense of free software innovators. We will continue to fund the aggressive reexamination of spurious patents used against the community (which we’ve been doing behind the scenes on behalf of several open source innovators). Whatever’s left over will fuel a venture fund fostering innovation in the free software community.” Bravo.

apt-xapian-index: searching for similar packages I've recently posted:

• an introduction of apt-xapian-index;
• an example of how to query it;
• a way to add simple result filters to the query;
• a way to suggest keywords and tags to use to improve the query.

Today I'll show how to abuse Xapian to show a list of packages similar to a given one. This time I'll try just linking to the code in wsvn and showing in the blog only show the most important bits. So, we have a package name, and we want to show what are the packages similar to that one. To do it, we simply build a big OR query with all the terms indexed for that package: Xapian will show us the packages whose terms are most similar, and that does the trick. This works because Xapian gives us the best results first, therefore even if no package except the given one will give an exact match, we still get the nearest matches first. In order to get the list of indexed terms given a package name we need to do two things:

1. Get the Xapian document for the package.
2. Get the termlist of the document.

To get the Xapian document we search for a term that only that document can have. In the index, the package name is indexed with the special prefix "XP", so we can search for that: Then we build the big term list, by iterating the termlist of the document: Note that it's trivial to fetch terms from more than one document, if you want to query "all packages a bit like this one and a bit like that one", although that's less of a useful feature. Lastly, we build the final query: I add an AND_NOT part with the input package name so that we don't get in the output the package that we asked for. This is it:

$ ./axi-query-similar.py debtags
20309 results found.
Results 1-20:
33% debtags-edit - GUI browser and editor for Debian Package Tags
27% tagcolledit - GUI editor for tagged collections
25% libtagcoll2-dev - Functions used to manipulate tagged collections (development version)
24% tagcoll - Commandline tool to perform operations on tagged collections
19% packagesearch - GUI for searching packages and viewing package information
18% doodle - Desktop Search Engine (client)
18% doodled - Desktop Search Engine (daemon)
18% libept0 - High-level library for managing Debian package information
18% upgrade-system - system upgrader from Konflux
18% libept-dev - High-level library for managing Debian package information
17% ept-cache - Commandline tool to search the package archive
16% tracker-utils - metadata database, indexer and search tool - commandline tools
[...]

You can use the wsvn interface to get to the full source code and the module it uses. Next in the series: adaptive quality cutoff.

I've written a short essay about anti-features. An anti-feature, I argue, is functionality that technology producers charge you to turn off. Apple's new, "pay-more to get DRM-free" is one example of an anti-feature but one can find them everywhere. It's a quick read and, I believe, an important but largely missing argument in most free software advocates' arsenal. I've posted it the on the FSF blog here:

http://www.fsf.org/blogs/community/antifeatures

I have always had some trouble with my eyes. One bizarre side effect of this is that switching fonts is very unpleasant. I recently got fed up with fighting things to live in the stone age of pre-Unicode fonts and locales and started hunting for a new console font. First I looked for nice new ones already designed for Unicode, with a selection of accented characters and so forth. The nicest I found was Terminus, but I’ve had trouble adjusting to it. So eventually I decided to stick with the font I’d been using - at least for now. I have been using the same font for my consoles since before 1997. This is, or at least was, the default Linux console font back when I did all my work on a VT on Linux/PPC: /usr/share/consolefonts/default8x16.psf.gz. When I started using X routinely I missed it quite a lot. Mostly that was because I used BitchX, and BitchX is written for an IBM437 code page set of line drawing characters. So I opened up that console font in xmbdfed and converted it to a PCF file. If you have Debian’s BitchX package installed, you’ve got a copy of this work in /usr/share/fonts/X11/misc/default8x16.pcf.gz. So I’ve done the same trick again. Here’s the steps I followed:

• Open the PCF font file in FontForge.
• Choose Add Enconding Name… from the Encoding menu.
• Enter IBM437, which is the iconv name for the traditional PC ROM character set.
• Choose Force Encoding and select IBM437.
• Choose Reencode and select ISO 10646 (BMP).
• Choose Generate Fonts… and save it as a BDF file, since PCF wasn’t an option.
• Create a fonts.dir file by running mkfontdir.
• Add the directory to the font path using xset +fp.
• Attempt to start an xterm.

Actually, after all that I was having trouble with the font’s name and eventually getting scary resource errors from X. I opened the font in gbdfed and saved it, which added some attributes and modified the bitmaps. Then I tried again and it worked. And now, I can use a UTF-8 locale and gcc’s error messages come out with foo’ (”angled” quotes) instead of display corruption! The resulting font is uni8×16, if anyone else would like it. I have switched to using it for all my terminals other than the one I run BitchX in; BitchX still needs the IBM437 version in order to display correctly. Of course, as soon as I finished this Paul Brook informed me that Konsole includes basically the same sort of font. I do not know the origins of KDE’s console8×16, but I do know that it’s a slightly different font (the @ has a smaller center, for example). And I know that I’m mighty annoyed it didn’t get added to my font path correctly when I installed konsole ages ago, since I might otherwise have found it!

apt-xapian-index: suggesting new terms for improving the query I've recently posted an introduction of apt-xapian-index, an example of how to query it and a way to add simple result filters to the query. Today I'll show how to use a very interesting feature of Xapian: computing a list of the best terms to use to improve the query. I'll expand apt-query-pkgtype.py to write, after the results, the suggested terms. If you feel like reimplementing my examples in another language, let me know and I'll include it to the post. Let's start with a few definitions taken from the Xapian glossary:

RSet (Relevance Set) The Relevance Set (RSet) is the set of documents which have been marked by the user as relevant. They can be used to suggest terms that the user may want to add to the query (these terms form an ESet), and also to adjust term weights to reorder query results. ESet (Expand Set) The Expand Set (ESet) is a ranked list of terms that could be used to expand the original query. These terms are those which are statistically good differentiators between relevant and non-relevant documents.

What I'm doing now is: after printing the normal result, I build an rset with all the results. For better results one can ask the user to pick the results they like (make a GUI with that: it should rock!), but starting with the top results in the RSet is a good enough default: Then I use the RSet to compute the ESet, and display the results to the user: You can also abuse this feature to show what are the tags that are most related to the search results. This allows you to turn a search based on keywords to a search based on semantic attributes, which would be an absolutely stunning feature in a GUI. We can do it thanks to Xapian allowing to specify a filter for the output of get_eset. This filter filters out all the keywords that are not tags, or that were in the list of query terms: I just rerun get_eset like above, but adding the filter, to show only a suggestion of tags: This is it. You can use the wsvn interface to get to the full source code and the module it uses. Try running it as:

axi-query-expand.py --type=gui image editor
axi-query-expand.py --type=cmdline image editor
axi-query-expand.py --type=game image editor

you will see that as the results change, the suggestions change as well. I'm happy to see that Xapian often suggests tags: this means that tags are useful discriminators in search results, and that all the work done on Debtags has been a good work. Notice that the suggestions change as you change any of the terms, the custom filter, and the tags you put in the search. Try to imagine how such a thing could improve Synaptic. What if I told you that Xapian is fast enough that all of this can happen live while you type? Next in the series: searching for similar packages.

A while back I posted about a couple of my irritations with GNU Screen. One of my irritations was the failure to reattach to sessions by name, if common prefixes were in use. For example with the following two (detached) sessions:

There are screens on:
        24419.abc       (Detached)
        24395.abcd      (Detached)
2 Sockets in /var/run/screen/S-skx.

The naive "screen -R abc" fails. Yesterday whilst looking over the screen bug list I came up with a patch. It isn't ideal as it introduces a new failure case, but I believe it is a step in the right direction and better than the current situation. See attachment to #361274 for the code. Also I patched screen so that #330036 is now fixed, and the blankerprg primitive works as expected. Finally I closed #317450 (with a version) as it has been fixed since Etch. Fun stuff. In the spirit of completeness I should say I had a stab at #447210 which is tilde (~) expansion in the chdir primitive, but gave up after a while as the code got too messy even for me. The trivial s/~/getenv("HOME")/ approach works fine for the simple case, but dealing with the expansion of strings such as ~foo/bar/ gets messy quickly. I can offer my patch if there is any interest though as a stop-gap measure. Now I'm almost tempted to look over another package's bugs, but I think I'd rather eat pie & drink beer... must. stop. talking. about. pies. Update: Patch for tilde expansion submitted to #447210 - tested and seemed to cover all cases. Now time for beer!