I saw that the inimitable Steve Kemp has cranked out an RBL for hosts that do brute force SSH attacks within seconds of thinking about doing it. As I have a passing interest in such things, and discovered that his sample submission client is really more geared towards people using iptables with the LOG target (and I use the ULOG target), I cranked out this moderately flexible Python client for logs written by ulogd this evening. If you use Netfilter rules like mine, you can use my script after /var/log/ulog/syslogemu.log gets rotated with:

$ ./report_iptables_ulog.py --log /var/log/ulog/syslogemu.log.1 --prefix SSH_brute_force

It may come as no surprise to some of you that I actually enjoy -- yes, enjoy -- watching political conventions. I've spent some time watching them over the past two weeks.

McCain has yet to speak tonight as I write this, but I've got to get out a few comments.

First of all, both parties of course were attacking the presidential candidate of the other. But did you notice a key difference? The Democrats attacked McCain on policy, while the McCain people attacked Obama on biography. VP nominee Palin even went so far as to mock Obama's work 20 years ago as a community organizer. I think that is incredibly telling. It seems to me to indicate that the Republicans know that their policies aren't working, and are grasping at straws to find something else. There is plenty in McCain's biography that the Democrats could go after -- his involvement in one of the worst scandals in the Senate, for instance -- but they aren't.

What about the other difference? Hope vs. fear. Obama's speech last week was truly inspiring. I don't think I've ever been inspired by a politician before. Make no mistake about it, he set the bar for himself incredibly high. I don't think anybody since Kennedy has done that. Obama realizes that there are things we must do as a country, and he also emphasized that the "ask not what your country can do for you" theme. Watching Giulianni and Palin was an exercise in fear-mongering. It's a constant stream of "9/11 MUSLIM OIL SHORTAGE TERRORISM NUCLEAR BIN LADEN IRAN GAS PRICES RED ALERT PANIC PANIC PANIC MEXICANS 9/11 9/11 9/11 SCARY WAR WAR WAR." It's as if they have discovered M-x spook RET in Emacs.

The Democrats, on the other hand, are talking about reducing health care costs, making sure everyone is covered, ending war, making the economy better for working-class people. It's an interesting disconnect that Obama highlighted well: McCain has said that the American economy has done well over the past 8 years. And indeed, by some "traditional" measures, such as productivity or GDP, it has. But the vast majority of Americans are not better off today than they were 8 years ago. Obama argues that the way McCain measures economic success is way off; our economy is not doing well if the vast majority of regular Americans are not doing well.

Also telling was how both candidates promised to reach across party lines (Bush promised that too). One actually is working to do it. Take abortion, for example. Obama said that he doesn't agree with the pro-lifers, but surely everyone can agree that providing better funding and support for alternatives to abortion such that it becomes incredibly rare is a good thing. Where's McCain looking for common ground? And then, of course, we have Palin wanting her family to be off limits -- but only when it's convenient to her.

I've voted in the past, but usually as a vote against someone rather than a vote for someone. This year, I'll make a single vote and do both.

Like Jaldhar, I too am thrilled at the news that McCain has chosen Palin as his running mate. Finally, a candidate who embodies the truly American spirit of the Republican party! Haven't we had enough political circus in Washington? Isn't it time to give Flying Circus a chance?

When Graham Bell invented the telephone in 1876, he probably didn t expect the rise of the cellphones we ve been seeing over the last 20 years. At first, there were C-net phone too huge to carry, but mobile still, as they communicated wireless. Then, devices became smaller, networks faster and ubiquitous, and today, the number of cellphones sold worldwide has exceeded the population. Much like everything else, it used to be better back then . When phone calls were still ridiculously expensive, people were able to enjoy their peace and life progressed slow for everyone to think enough, not do or say without engaging their brains. Then, when the first cellphones stuffed people s pockets, they did their job pretty much from the start: you could make phone calls. Some genius discovered SMS as a splendid tool to rip off customers, so phones grew pager abilities, but other than that, they just worked; I remember my first phone, which didn t break in years. Obviously, if you re a phone manufacturer, you don t like that, because once you sell a phone that works, the customer won t come back to give more monies in ten years. Clever as you are, you devised two schemes to ensure your cash flow: make phones more brittle and crap, so that they break within a year, at most two; and drag ever younger people into the debt trap. For the truly stupid, sites and services offer ring tones and games and what not, and the lesser challenged you keep close by the continuous addition of new features that noone needs. So these days, almost every phone can play music files, which is mighty convenient to spice up your work commute with some tunes, but our youngsters are overburdened by that, it seems. I almost soiled myself laughing at a group of five Italians at Ezeiza Airport in Buenos Aires, who were sitting around waiting for a bus, every one of them ear plugged and grooving to the beats (you know how dorky it looks when people silently sing along rap songs?). That s not the funny part. The funny part is that every minute, one of them would say something, which would cause the others to unplug one ear, and form their lips to bleat what (making sure to add just a little bit extra of the tone of general disinterest, which is cool ). This elicited one of two responses: either the original speaker would say oh, nothing and everyone nodded, or he d repeat his wisdom, causing everyone to laugh and nod before in both cases they replugged and returned to luff themselves. I wonder how they made it to the airport themselves, and why they travel as a group. Worse than that, however, is that cellphone manufacturers remembered that their phones had speakers (for fancy ringtones) and consequently added the ability to blast tunes through them. As a result, groups of kiddies walk around or sit in trains, with one (or more) of those cellphones blaring into the environment. Apart from being generally inconsiderate, what I don t understand is how they put up with the sound quality. It s mostly hip-hop music you know that genre that makes some homies out there install 5000 watt subwoofers into their cars so make sure the windows rattle with the base but these phones have a frequency spectrum comparable in width to that of your grandfather s, way further up the scale (meaning they just don t do base). Playing hip-hop through those is like putting a flute concerto on a subwoofer, just worse, because high-pitch tones are harder to filter by those who don t want to hear them. And yet, I see it all over the place, kiddies listening to music through cellphone speakers. Is it because silence would kill them? NP: 65daysofstatic: The Fall of Math

The current method of patching into place a series of ./configure variables as $(DEB_HOST_GNU_TYPE).cache files is becoming a pain - it blocks native builds of Emdebian packages for a start. So, I've been experimenting with a different method of support.
Centralised cache value support
Initially, I've collated all the current cache values needed (i.e. beyond the ones already provided for specific architectures) into /etc/dpkg-cross/cross-config.cache and have a script in emdebian-tools called emcache that will manage the central values by comparing them to the real values used in each build tree.
So, cache values, what are they?

1. Cross-builds fail if ./configure tries to compile a test program and then try to execute it because it will have been compiled with the cross-compiler - instead, the cross-build passes the value that the test was trying to calculate so that ./configure does not need to try to execute any cross-compiled code.


2. The values are derived from native builds, written to a $(DEB_HOST_GNU_TYPE).cache file that is passed to ./configure via debian/rules.


3. The values turn out to be mostly (completely?) architecture-independent, so are eminently suitable for collation into a central cache file.


4. Quite package-specific. There is some overlap but most are customised for a specific package. This led to a development of the central cache file: package-maintained cache files to support extensibility.

Package maintained cache files
I'm now experimenting with cache files written (and maintained) by packages during normal builds and dropped into /etc/dpkg-cross/cross-config.d/ - which would later migrate into dpkg support. Ideas welcome!
The basic idea will be that IF cross-building, the build process would check for /etc/dpkg[-cross]/cross-config.d/$source_package and, if found, add that to the cache value list for that build.
Cache files are shell scripts so this it be manageable to bring some intelligence to the process.
Where does MIPS come in?
Well, all the builds, so far, have been ARM or armel (where the cache values are likely to be identical) and i386 (which doesn't need a cross-compiler at all and therefore does not need cache files). MIPS is the first chance to check the cache values and ensure that the above criteria really are true.
Now that we have a volunteer looking at MIPS, the current MIPS builds are a suitable test bed.
What does this have to do with Debian?
Once Lenny is released, I'll be looking to implement this support via the long term mass bug filing for cross-build support into the packages themselves. It is possible that the process can be entirely automated - as long as the script knows which values are relevant. A normal cache file, post-build, can be very, very long whereas most cache files for Emdebian are just a few lines long - the longest is about 20 lines.
The current process is outlined on the Emdebian website, including a list of packages that currently need cache files to cross-build.
Oh, and just a note - when I start this phase of the long term mass bug filing, I'll first do delayed NMUs for the existing bug reports - hence the delay until Lenny is released.
In order to speed things along, I have also been doing some RC bugs - downgrading some, fixing others etc. (I've finished 6 since coming back from DebConf, so far, IIRC.) All efforts to get Lenny released will help Emdebian too, so if you want to help Emdebian, then help Lenny get out of the door and let's get to work on Squeeze.

(Squeeze is a great name for a release with lots of fixes for embedded stuff, don't you think?)

From Roderich Schupp I received the following instructions:

cp /usr/share/doc/hal/examples/10-x11-input.fdi /etc/hal/fdi/policy/

And in order to set a default keymap:

<deviceinfo version="0.2">
  <device>
    <match key="input.xkb.rules" contains="base">
      <merge key="input.xkb.layout" type="string">de</merge>
      <merge key="input.xkb.variant" type="string">nodeadkeys</merge>
    </match>
  </device>
</deviceinfo>

Into yet another custom file in this directory.Thank you, I'm going to try that on my next reboot (which may take a week).

According to the Slashdot article and the StopSoftwarePatents.org website itself and on Digg, the anti-software-patent activists are attempting a world-wide event on the 24th of September as a world-wide day against software patents. USA has them via a weird court ruling, Japan has them as well (not sure why), there have been efforts to force software patents on EU, India, Australia and many other countries either by Microsoft lobbies or even via US trade treaty pressure. While lobbying by local Microsoft branches and their pet companies can be countered locally, like I and many others did in EU a couple years ago (mostly thanks to FFII), trade negotiations are very secretive affairs and it is very hard to lobby there directly. The public needs to be aware of the issue, otherwise the politicians will not be aware of its importance to their voters. Software patents are a threat to free software as it circumvents the power of GPL and other copyright licenses if enough money is thrown at the lawyers. If a software patent is generic enough, it can easily stop development of a whole class of free software applications on a whim of the patent holder. And there are plenty of granted software patents with a very broad scope (progress bar, anyone?). I am aware that to get Debian support for such an initiative, a GR is needed, but how about a personal word of support from the DPL or support from the SPI? It is in direct area of interest for SPI - software patents create an ever present threat of legal action against any and all software in Debian (in USA and Japan, at least) and, as the legal umbrella of Debian in the USA, SPI would be a prime target. What do others think about this?

As per the original press release:

In Quebec, access to public markets is the rule while contracts attribution without invitation to tender is the exception. A public market should be transparent, fair and most importantly, open to all. The solutions as well as the propositions must be evaluated objectively on known and accepted criteria. Furthermore, the regulation implies that public markets have to enhance the local economic development as well as the Quebec technologies. From February to June 2008, FACIL has noticed sales of proprietary software for more than 25 million dollars. These purchases were made for products offered by large multinational enterprises, with no regard to suppliers in Quebec.

Much to my surprise, news about this lawsuit have spread as far as Vietnam! Something tells me that this court case might have lasting repercussions...

As I start this blog post, I'm on the train between Amersfoort and Osnabr ck, on my way to Kiel, where there'll be a meeting of the m68k porters, kindly organized by m68k kernel maintainer Christian Steigies. I took the 8:40 train in Mechelen this morning, and if all goes well, I'll be arriving in Kiel at 17:22 tonight. Some train trip... but I certainly prefer that to any flight. Anyway. The days of Debconf were very nice. Debconf is a lot of things to me. Travel. Hacking. Mao, evolving into a drinking game. Beer. Attending talks. Giving talks. Meeting people. Face-to-face non-flaming. Gesturing for an extra knife at luunch. Hangover. Whiskey. Talking to the person next to you—over IRC. Getting killed before knowing what the rules of the assassins game are. Filing bugreports in person. Pictures. Kilts. Streaming video. Not being sick, hopefully. Name badges. Flying. Sleeping. Yes, sleeping. Most of all, though, this year, Debconf was just great. Thanks. You know who you are. After debconf, I flew to Buenos Aires, where I slept for one night in a Youth Hostel somewhere downtown. They were affordable, but the bed wasn't great—the mattress sorely needed replacement. Since I had a day in BA, and since there were apparently not enough people on the schedule, I'd agreed to hold a talk, and came up with the idea of a 'debian secrets' talk—about Debian-specific commands, such as dpkg-divert and update-alternatives etc—so that people could learn how to use their Debian system more efficiently. As I was sitting in Andreas' talk, who was right before me on the schedule, suddenly Dag walked up to me and said hi. This was unexpected; Dag is a fellow Belgian, who's involved with the CentOS project, and who maintains a positively huge RPM repository at his site—if you maintain an RPM-based system somewhere, you'll probably know about that site. So while I recognized him, I immediately wondered what he was doing there. Turned out he was invited to hold a talk at the Free Software event to which Debianday was attached (and which would not start for another day or two), and that when he saw my name on the schedule, he decided to attend. Fun. He learned a thing or two from my talk, and was talking later about writing some tools for RPM-based systems that would perform the same or similar functionality implemented by some of the Debian tools I talked about. Great. We had a picture together (in front of the DebConf banner—hah) that I still should upload, and then went out for lunch together with some other CentOS guys. The talk itself seems to have been well-received, and I'm glad about that; I only gave it because there was a need for more talks, not because I felt confident I was very knowledgeable about that subject. In fact, I did have to ask on the debconf-discuss mailinglist for some input (which I received) so that I could make sure the talk would actually be useful to people. That helped a lot, I guess. After that, we played some mao in the lobby of the DebianDay venue. While doing so, I overheard Dag talking about CentOS, advocating it to some of the people there, which I must say felt pretty weird on a Debian event. Not that there's anything wrong with it, of course. Except that Dag was supposed to be writing his talk slides. How did that work out, Dag? ;-) Eventually, I got in a cab to the EZE airport, and flew home. That wasn't fully without issue, but I did get there. And now, I've been, eh, overloaded. Still have to follow up on a question a customer asked me, but I've barely been home or at the office, just enough to sleep really. Should do something about that, I guess... Two days ago, I also managed to forget my camera somewhere. For 12 hours, I was worried, even though I had a pretty good idea of where it was; but since I'm getting quite good at taking pictures now, and since I really like doing so as well, I really didn't want to lose my camera. Fortunately, when I called this morning to the place where I thought I'd lost it, they told me it'd been found, and where to get it. In other words, it's safe, it's taken care of, Philip went to get it, and I just need to arrange to get it when I get back. Which is a relief. Let's not be so stupid anymore in the future... Finishing up now, a day later, and I'm at Christian's office, playing with his and my coldfire board. As it happens, it appears that Freescale has brought out a new BSP for these boards, so that's nice. Let's see whether we can get those to boot again...

The Problem: A problem with virtual machines is the fact that one rogue DomU can destroy the performance of all the others by inappropriate resource use. CPU scheduling is designed to allow reasonable sharing of computational resources, it is unfortunately not well documented, the XenSource wiki currently doesn’t document the “credit” scheduler which is used in Debian/Etch and CentOS 5 [1]. One interesting fact is that CPU scheduling in Xen can have a significant effect on IO performance as demonstrated in the paper by Ludmila Cherkasova, Diwaker Gupta and Amin Vahdat [2]. But they only showed a factor of two performance difference (which while bad is not THAT bad). A more significant problem is managing virtual memory, when there is excessive paging performance can drop by a factor of 100 and even the most basic tasks become impossible. The design of Xen is that every DomU is allocated some physical RAM and has it’s own swap space. I have previously written about my experiments to optimise swap usage on Xen systems by using a tmpfs in the Dom0 [3]. The aim was to have every Xen DomU swap data out to a tmpfs so that if one DomU was paging heavily and the other DomUs were not paging then the paging might take place in the Dom0’s RAM and not hit disk. The experiments were not particularly successful but I would be interested in seeing further research in this area as there might be some potential to do some good. I have previously written about the issues related to swap space sizing on Linux [4]. My conclusion is that following the “twice RAM” myth will lead to systems becoming unusable due to excessive swapping in situations where they might otherwise be usable if the kernel killed some processes instead (naturally there are exceptions to my general rule due to different application memory use patterns - but I think that my general rule is a lot better than the “twice RAM” one). One thing that I didn’t consider at the time is the implications of this problem for Xen servers. If you have 10 physical machines and one starts paging excessively then you have one machine to reboot. If you have 10 Xen DomUs on a single host and one starts paging heavily then you end up with one machine that is unusable due to thrashing and nine machines that deliver very poor disk read performance - which might make them unusable too. Read performance can particularly suffer in a situation when one process or VM is writing heavily to disk due to the way that the disk queuing works, it’s not uncommon for an application to read dozens or hundreds of blocks from disk to satisfy a single trivial request from a user, if each of these block read requests has to wait for a large amount of data to be written out from the write-back cache then performance will suck badly (I have seen this in experiments on single disks and on Linux software RAID - but have not had the opportunity to do good tests on a hardware RAID array). Currently for Xen DomUs I am allocating swap spaces no larger than 512M, as anything larger than that is likely to cause excessive performance loss to the rest of the server if it is actually used. A Solution for Similar Problems: A well known optimisation technique of desktop systems is to use a separate disk for swap, in desktop machines people often use the old disk as swap after buying a new larger disk for main storage. The benefit of this is that swap use will not interfere with other disk use, for example the disk reads needed to run the “ps” and “kill” programs won’t be blocked by the memory hog that you want to kill. I believe that similar techniques can be applied to Xen servers and give even greater benefits. When a desktop machine starts paging excessively the user will eventually take a coffee break and let the machine recover, but when an Internet server such as a web server starts paging excessively the requests keep coming in and the number of active processes increases so it seems likely that using a different device for the swap will allow some processes to satisfy requests by reading data from disk while some other processes are waiting to be paged in. Applying it to Xen Servers: The first thing that needs to be considered for such a design is the importance of reliable swap. When it comes to low-end servers there is ongoing discussion about the relative merits of RAID-0 and RAID-1 for swap. The benefit of RAID-0 is performance (at least in perception - I can imagine some OS swapping algorithms that could potentially give better performance on RAID-1 and I am not aware of any research in this area). The benefit of RAID-1 is reliability. Now there are two issues in regard to reliability, one is continuity of service (EG being able to hot-swap a failed disk while the server is running), and the other is the absence of data loss. For some systems it may be acceptable to have a process SEGV (which I presume is the result if a page-in request fails) due to a dead disk (reserving the data loss protection of RAID for files). One issue related to this is the ability to regain control of a server after a problem. For example if the host OS of a machine had non-RAID swap then a disk failure could prevent a page-in of data related to sshd or some similar process and thus make it impossible to recover the machine without hardware access. But if the swap for a virtual machine was on a non-RAID disk and the host had RAID for it’s swap then the sysadmin could login to the host and reboot the DomU after creating a new swap space on a working disk. Now if you have a server with 8 or 12 disks (both of which seem to be reasonably common capacities of modern 2RU servers) and if you decide that RAID is not required for the swap space of DomUs then it would be possible to assign single disks for swap spaces for groups of virtual machines. So if one client had several virtual machines they could have them share the same single disk for the swap, so a thrashing server would only affect the performance of other VMs from the same client. One possible configuration would be a 12 disk server that has a four disk RAID-5 array for main storage and 8 single disks for swap. 8 CPU cores is common for a modern 2RU server, so it would be possible to lock 8 groups of DomUs so that they share CPUs and swap spaces. Another possibility would be to have four groups of DomUs where each group had a RAID-1 array for swap and two CPU cores. I am not sure of the aggregate performance impact of such a configuration, I suspect that a group of single disks would give better performance for swap than a single RAID array and that RAID-1 would outperform RAID-5. For a single DomU it seems most likely that using part of a large RAID array for swap space would give better performance. But the benefit in partitioning the server seems clear. An architecture where each DomU had it’s own dedicated disk for a swap space is something that I would consider a significant benefit if renting a Xen DomU. I would rather have the risk of down-time (which should be short with hot-swap disks and hardware monitoring) in the rare case of a disk failure than have bad performance regularly in the common situation of someone else’s DomU being overloaded. Failing that, having a separate RAID array for swap would be a significant benefit. If every process that isn’t being paged out could deliver full performance while one DomU was thrashing then it would be a significant benefit over the situation where any DomU can thrash and kill the file access performance of all other DomUs. A single RAID-1 array should handle all the swap space requirements for a small or medium size Xen server One thing that I have not tested is the operation of LVM when one disk goes bad. In the case of a disk with bad sectors it’s possible to move the LVs that are not affected to other disks and to remove the LV that was affected and re-create it after removing the bad disk. The case of a disk that is totally dead (IE the PV header can’t be read or written) might cause some additional complications.

• [1] http://wiki.xensource.com/xenwiki/Scheduling


• [2] http://www.cs.ucsd.edu/~dgupta/papers/per07-3sched-xen.pdf


• [3] http://etbe.coker.com.au/2008/05/22/xen-and-swap/


• [4] http://etbe.coker.com.au/2007/09/28/swap-space/

Share This

The Linux kernel has a number of code sections which look at the apparent size of the machine and determine what would be the best size for buffers. For physical hardware this makes sense as the hardware doesn’t change at runtime. There are many situations where performance can be improved by using more memory for buffers, enabling large buffers for those situations when the machine has a lot of memory makes it convenient for the sysadmin. Virtual machines change things as the memory available to the kernel may change at run-time. For Xen the most common case is the Dom0 automatically shrinking when memory is taken by a DomU - but it also supports removing memory from a DomU via the xm mem-set command (the use of xm mem-set seems very rare). Now a server that is purchased for the purpose of running Xen will have a moderate amount of RAM. In recent times the smallest machine I’ve seen purchased for running Xen had 4G of RAM - and it has spare DIMM slots for another 4G if necessary. While a non-virtual server with 8G of RAM would be an unusually powerful machine dedicated for some demanding application, a Xen server with 8G or 16G of RAM is not excessively big, it’s merely got space for more DomU’s. For example one of my Xen servers has 8 CPU cores, 8G of RAM, and 14 DomUs. Each DomU has on average just over half a gig of RAM and half of a CPU core - not particularly big. In a default configuration the Dom0 will start by using all the RAM in the machine, which in this case meant that the buffer sizes were appropriate for a machine with 8G of RAM. Then as DomUs are started memory is removed from the Dom0 and these buffers become a problem. This ended up forcing a reboot of the machine by preventing Xen virtual network access to most of the DomUs. I was seeing many messages in the Dom0 kernel message log such as “xen_net: Memory squeeze in netback driver” and most DomUs were inaccessible from the Internet (I didn’t verify that all DomUs were partially or fully unavailable or test the back-end network as I was in a hurry to shut it down and reboot before too many customers complained). The solution to this is to have the Dom0 start by using a small amount of RAM. To do this I edited the GRUB configuration file and put “dom0_mem=256000” at the end of the Xen kernel line (that is the line starting with “kernel /xen.gz“). This gives the Dom0 kernel just under 256M of RAM from when it is first loaded and prevents allocation of bad buffer sizes, it’s the only solution to this network problem that a quick Google search (the kind you do when trying to fix a serious outage before your client notices (*)) could find. One thing to note is that my belief that it’s kernel buffer sizes that are at the root cause of this problem is based on my knowledge of how some of the buffers are allocated plus an observation of the symptoms. I don’t have a test machine with anything near 8G of RAM so I really can’t do anything more to track this down. There is another benefit to limiting the Dom0 memory, I have found that on smaller machines it’s impossible to reduce the Dom0 memory below a certain limit at run-time. In the past I’ve had problems in reducing the memory of a Dom0 below about 250M, while such a reduction is hardly desirable on a machine with 8G of RAM, when running an old P3 machine with 512M of RAM there are serious benefits to making Dom0 smaller than that. As a general rule I recommend having a limit on the memory of the Dom0 on all Xen servers. If you use the model of having no services running on the Dom0 there is no benefit in having much ram assigned to it. (*) Hiding problems from a client is a bad idea and is not something I recommend. But being able to fix a problem and then tell the client that it’s already fixed is much better than having them call you when you don’t know how long the fix will take.

Share This

Docking the FreeRunner into the laptop Earlier in the day, I wrote:

So yes, my laptop can now be turned into a phone charger with networking, DNS and apt cache services. I shall look into hooking that script into dbus to have it run automatically when the phone is plugged and unplugged.

It's not dbus, it's udev, and I've managed to do it. It's an udev rule:

# cat /etc/udev/rules.d/z60_openmoko_net.rules
ACTION=="add", SUBSYSTEM=="net", ATTRS idVendor =="1457", ATTRS idProduct =="5122", RUN+="/root/bin/share-openmoko"
ACTION=="remove", SUBSYSTEM=="net", INTERFACE="usb0", RUN+="/root/bin/share-openmoko"

And a script:

# cat bin/share-openmoko 
#!/bin/sh
if [ "$ACTION" == "add" ]
then
    ACTION=start
fi
if [ "$ACTION" == "remove" ]
then
    ACTION=stop
fi
INTERFACE=$ INTERFACE:-"usb0" 
ACTION=$ ACTION:-"$1" 
case "$ACTION" in
    start)
        logger -t openmoko "Connected, setting up network"
        iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE -s 192.168.0.200/29
        ifconfig usb0 192.168.0.200 netmask 255.255.255.248
        /etc/init.d/dnsmasq start
        echo 1 > /proc/sys/net/ipv4/ip_forward
        ;;
    stop)
        logger -t openmoko "Disconnected, bringing down network"
        echo 0 > /proc/sys/net/ipv4/ip_forward
        /etc/init.d/dnsmasq stop
        iptables -t nat -F POSTROUTING
        ifconfig usb0 down
        ;;
esac

The script has extra cruft that makes it double as a udev script and as a init.d style script, just because I didn't feel like abandoning the init.d style interface. However, udev handles it perfectly, so there's probably no use at all for the start/stop part. This means that the interface and all supporting services will be brought up and down when the phone is connected/disconnected, and also when the phone is suspended/resumed. Of course, more fancy things can be plugged into the script, like syncing PIM info, file systems, turning on applets in panels, mounting phone file systems and whatnot.

According to a recent article, 6 Common Website Mistakes That Cost Money are:

1. JavaScript or other crawler-unfriendly navigation that may impede indexing

This one is best avoided at design-time, by including Web Content Accessibility Guidelines 1.0 Level A in the design brief, but if you’ve ended up with JavaScript-based navigation on your site (check by doing “View Source” and searching for the code for your home page link and so on - if you can find it, then it’s probably not javascript) and it’s based on some template system, a webmaster can probably do a whole-site edit to put the navigation links in the page properly - or at least add a useful <noscript> tag.

2. Navigation that buries important pages within the site architecture.

The structure of the website’s files and the structure of the link menus do not need to match, so if there’s a page that you feel is important, get your webmaster to add it to the navigation links across the whole site.

3. Duplicate “pages” getting indexed under multiple URLs.

This usually happens for one of two reasons: one is inappropriately-parked domains, which is mistake 6 below and often fairly easy to fix; the other is a misbehaving web application, which you’ll need to get a programmer to fix. You can do a simple test of your web application by starting at your homepage and following links to a particular page; then open a new browser window and try to reach the same page by different links (or a site search) and compare the address bars (the bit of your browser showing http://) - do they match? If not, you’ve got this problem.

4. No keyword phrase focus in the content or conversely, keyword phrase stuffing

You can use a good word-counter on the text to see how common different keywords and keyphrases, or use the “webmaster tools” section of some search engines to see what they’re focusing on. Keyword-stuffing can be fairly easy to see. If you “View Source” on a page and there’s a large block of keyword-intensive text somewhere in it that doesn’t appear it when viewed in a browser, then it’s probably stuffing. Many sites regard stuffing as a sort of spam, so you don’t want to be found doing this. To fix these problems, rewrite the page text appropriately.

5. An optimized home page, but that’s it

Repeat your checks from problem 4 on a few pages other than the home page to discover whether you suffer from this. Also, see whether your website statistics show search engine visitors arriving at a variety of different pages (these are sometimes called “Entry Page” statistics).

6. Additional domains owned by the company are not properly redirected

If you usually use .co.uk, but you also have a .com domain, try visiting a random page on your website, then click in the address bar and replace the .co.uk with .com - what happens? If the page is Not Found, then your domains aren’t properly redirected and you need a Redirect adding to the second one. If the page displays but the address doesn’t change itself back, then your domains are probably pointing at the same webspace but aren’t properly redirected, which will mean you’re probably making mistake 3 above. Usually, the simplest way to fix this is with a conditional redirect. On Apache webservers, you can add a .htaccess file containing something like:-

RewriteEngine On
RewriteCond % HTTP_HOST  !^www\.yourdomain\.co\.uk$ [NC]
RewriteRule (.*) http://www.yourdomain.co.uk/$1 [R=permanent,L]

That even works on BT Business web hosting, by the way. If it doesn’t work, try asking your web hosting provider to enable mod_rewrite for you or ask them to suggest how to achieve the same result.

The above list of mistakes was published on Search Engine Land last week. The correction methods are all things used by my webmaster cooperative.

I had a problem where the email address circsales@washpost.com spammed a Request Tracker (RT) [1] installation (one of the rules for running a vaction program is that you never respond twice to the same address, another rule is that you never respond to automatically generated messages). Deleting these tickets was not easy, the RT web interface only supports deleting 50 tickets at a time. To delete them I first had to find the account ID in RT, the following query does that:
select id from Users where EmailAddress='circsales@washpost.com'; Then to mark the tickets as deleted I ran the following SQL command (where X was the ID):
update Tickets set Status='deleted' where Creator=X; Finally to purge the deleted entries from the database (which was growing overly large) I used the RTx-Shredder [2] tool. RTx-Shredder doesn’t seem to support deleting tickets based on submitter, which is why I had to delete them first. I am currently using the following command to purge the tickets. The “limit,500” directive tells rtx-shredder to remove 500 tickets at one time (the default is to only remove 10 tickets).
./sbin/rtx-shredder --force --plugin 'Tickets=status,deleted;limit,500' There are currently over 34,000 deleted tickets to remove, and rtx-shredder is currently proceeding at a rate of 9 tickets per minute, so it seems that it will take almost three days of database activity to clear the tickets out. I also need to purge some tickets that have been resolved for a long time, I’m running the following command to remove them:
./sbin/rtx-shredder --force --plugin 'Tickets=status,resolved;updated_before,2008-03-01 01:01:34;limit,500' With both the rtx-shredder commands running at once I’m getting a rate of 15 tickets per minute, so it seems that the bottleneck is more related to rtx-shredder than MySQL (which is what I expected). Although with two copies running at once I have mysqld listed as taking about 190% of CPU (two CPUs running two capacity). The machine in question has two P4 CPUs with hyper-threading enabled, so maybe running two copies of rtx-shredder causes mysqld to become CPU bottlenecked. I’m not sure how to match up CPU use as reported via top to actual CPU power in a system with hyper-threading (the hyper-threaded virtual CPUs do not double the CPU power). I wonder if this means that the indexes on the RT tables are inadequate to the task. I tried adding the following indexes (as suggested in the rtx-shredder documentation), but it didn’t seem to do any good - it might have improved performance by 10% but that could be due to sampling error. CREATE INDEX SHREDDER_CGM1 ON CachedGroupMembers(MemberId, GroupId, Disabled);
CREATE INDEX SHREDDER_CGM2 ON CachedGroupMembers(ImmediateParentId, MemberId);
CREATE UNIQUE INDEX SHREDDER_GM1 ON GroupMembers(MemberId, GroupId);
CREATE INDEX SHREDDER_TXN1 ON Transactions(ReferenceType, OldReference);
CREATE INDEX SHREDDER_TXN2 ON Transactions(ReferenceType, NewReference);
CREATE INDEX SHREDDER_TXN3 ON Transactions(Type, OldValue);
CREATE INDEX SHREDDER_TXN4 ON Transactions(Type, NewValue);

• [1] http://bestpractical.com/rt


• [2] http://search.cpan.org/~ruz/RTx-Shredder-0.07/

Share This

I’ve had little time in the four years I’ve lived in Roslindale to get involved in community volunteer efforts. My wife Rachele, however, has devoted several years to efforts such as the Roslindale Community Center, Roslindale Village Main Streets, and Roslindale Clean and Green. She has served as chair of both RCC and one of the RVMS committees. Over the past few months, I’ve been disappointed to see the RCC board endure repeated senseless attacks by a perhaps overzealous community member, Walter Michalik, who has effectively paralyzed the organization. I have nothing personal against this guy, but I thought I could do my part here by publishing an example of an email he broadcast to the committee, which I think is embarrassing enough on its own merits to need no editorializing from me, except to say any email that starts with “just so you won’t be surprised when the IRS launches its investigation” can be neither productive nor taken seriously. The context is a response to an announcement sent out pursuant to the Community Centers By-Laws scheduling a special meeting to discuss the recent upheaval on the Council. There’s no reason this should not be in the public record:

Rachele Just so you won’t be surprised when the IRS launches its investigation, be advised that YOU DO NOT HAVE THE AUTHORITY TO CALL FOR A SPECIAL MEETING. The By Laws do not allow the President/Chair to call for a Special meeting - that right is reserved for the MEMBERS to have a power over a hostile or non-performing President/Chair. It was created by a Membership wary of annimosity within its ranks in order to protect itself from falling into a useless dysfunctional body. That’s why our By Laws require the majority of the membership request it through the Chair and not the other way around. So you screwed that up too. Your continuous violation of rules and laws have proven beyond a doubt that this current Roslindale Community Center Council has violated the trust behind its IRS-designated not-for-profit status. The failures are on so many significant levels over 6 months it is readily apparent that the Roslindale Community is better served without a Community Center Council. There is no longer any excuse to allow the residents of Roslindale to be denied the services they deserve that you and the officers have no idea how to deliver. Because that’s what you are supposed to do - serve the residents of Roslindale and not preseide over a self-serving board. You’ve taken us from dysfunctional to non-functional over the past 60 days. You and your officers have already driven away our Archdale partners and now the merger is off. Congratulations. You got what you wanted. You and the officers have withheld information from the Members of the Council imposing gag orders along the way and in more than one instance spreading lies within this community. There are so very many lapses on your part and the rest of the officers that it is very apparent that the Roslindale Community Center exists IN SPITE OF the Roslindale Community Center Council - not because of it. It would be far better indeed, if the Center were run without the Council. What have you done in 6 months? All you do is avoid the work of leadership. Please, just gracefully step aside and let a real leader emerge - or dissolve the Council and let the Center run itself under BCYF before the IRS makes that happen. Of course this is my stated opinion. You can choose not to belive it. You can huddle with a group of self-serving officers and invent excuses or change the agenda away from the message and continue with your ad hominem attacks against the messenger. I’m thick-skinned. Feelings aren’t important to me - services to residents are important to me. While I have, in fact, enjoyed the confidences of others within this community and their true displesure with this Council, I will produce that to the appropriate authorities as I see fit along with the Bill of Particulars I have produced with documentation for all charges. As a founder of the Roslindale Community Center Council I can no longer support you or the officers as any legitimate authority in this community.

Technorati Tags: Boston, Roslindale, Walter Michalik

Share This

Ah, Dell. Seeming inventors of the tech support pit of bureaucratic indifferences, inventors of the flamingo pink Inspiron, perpetrators of fraud in New York...

I have, for over a year now, been on a crusade trying to get them to stop sending me their Dell Home and Home Office catalog to my mailbox. It has been a bundle of fun, let me tell you.

They have a nice-sounding privacy policy. It says you can opt out of all their mailings by filling out a form online. Yeah, good luck with that. First of all, there are different forms for different departments at Dell. I've filled out them all, multiple times. They do nothing whatsoever. Perhaps they use them as lists of known-good addresses to send new advertisements to, rather than lists of people to remove. Oh well.

Now, unfortunately I feel compelled to bore you with the saga so far, involving telephone hang-ups, broken privacy policies, and the like. But there is a silver lining at the end, in which I submitted a request to the postal service asking them to block Dell from sending me any more mail, and it appears that they are very likely to violate Federal Law any day now.

I have called them about it. Dealt with the old "let me transfer you to the correct department" then hang up on me ploy. Spoken to people that have promised up and down that I'll be off their list in 30-60 days. It's always 30-60 days, isn't it? Very convenient that I can't tell for 2 months whether or not they've processed my request.

I've tried their online chat. One of my attempts went like this:

Session Started with Agent (Sneha Ranga)

Agent (Sneha Ranga): "Due to circumstances that have affected Dell Communications I am temporarily unable to pull up any information. The down time is temporary. We apologize for the inconvenience, as we value your time as a customer. Please contact us back after an hour."

Session Ended

Ah Dell, only you could reach such a pinnacle of customer service. /kicking someone out of a chat room before they have a chance to say a word.

Finally, last fall, I blogged about the situation (that's the link above). Debbie from Dell red the post and emailed me. Great, I thought. She asked for my address information and catalog information and sent me a removal confirmation:

From: Debbie@Dell.com
Date: Thu, 18 Oct 2007 13:08:18 -0500
To: jgoerzen@complete.org
Subject: RE: Dell mailing list

Thank you, Mr. Goerzen, your request to have the below address
information removed from our marketing lists has been received:

[ my address here ]

We will process your request promptly. However, it may take several
weeks for some changes to take effect. If you are still receiving
catalogs after thirty (30) days feel free to email me. Sorry for any
inconvenience you may have experienced.

Thank you,
Debbie

So that was October. In December, I replied to that message, saying: "I received another mailing today, and it's been nearly 2 months since your initial message. If there's anything further you can do, I'd appreciate it." Debbie said, "I am very sorry Mr. Goerzen, I will resubmit your request." Guess how successful that was.

So in February, I manage to figure out a way to send in a support ticket without having a Dell system serial number. I wrote:

I keep getting your Dell Home and Home Office catalog. I have tried for months to get off your mailing list. I have called in, talked to people in multiple departments, who have promised to remove me from the list. I have contacted you online. NOTHING IS HELPING. This has gone on for MONTHS.

PLEASE NEVER SEND ME MAIL AGAIN, EVER, AND DO NOT RENT OR SHARE MY NAME.

My address is above.

The code on the mailing I received is: [ snipped ]

The form letter I got back said:

If you are currently receiving our catalog or mailings and would like to be removed, please visit the following web page and select the appropriate link under the "Opt-out of direct mail, phone or fax communications" heading: http://www.dell.com/OptOut

I replied, saying that form didn't work. Guess what I got back?

Thank you for signing up for Dell Email Subscriptions. Please save this email for your records.

Yes, that's right. Asking them to take me off their postal mailing lists got them to PUT ME ON their email lists. ARGH.

So they eventually manage to correctly take me off the email list, and of course promise to do the same with the postal list. This back in February.

I contacted them again in March and July, only to have a similar stupidity-laced run-in with clueless form-answer-laden Dell support reps. Each one claimed to have now, finally, and permanently removed me from the list. It never happened, and none of them lifted a finger to find out way, and no amount of begging could make them.

So, here's the good part.

Junkbusters has spent years educating people on how to get rid of unwanted mail, and documents getting a prohibitory order against the sender. It was originally designed for people that didn't want to receive obscene advertising mailings, but thanks to the happy fact that one non-adult-mailer challenged a prohibitory order all the way to the Supreme Court, you can now get prohibitory order against anyone. Yes, even Dell. (The supreme court's ruling even gave an example: you can prohibit a clothing catalog if you want.)

And last month, that's exactly what I did. The USPS sent me back a copy of the letter they sent to Dell, as well as a second page with instructions on reporting violations. Here's the letter they sent to Dell:



Somehow I get a chuckle over some Dell mail clerk trying to figure out how an 11-pound laptop is sexually provocative.

From August 25 on, it is a federal offense for Dell to send me another Home and Home Office catalog. This is a branch of criminal law, not civil law. That is, it's the maybe-go-to-jail branch of law.

How disappointed I was to receive yet another catalog from them today. If only they had waited 5 more days, I could have turned them in now.

Oh well. There's always next month's catalog. Let's just hope the clerk that received the USPS letter removed my name with a better system than everyone else at Dell uses, eh?

Update: There are more notes on how aptitude performs as a Sudoku solver here. Russell Coker recently wrote a post entitled Ownership of the Local SE Linux Policy. This post has nothing to do with the substance of his post, which is a discussion of how distributions should configure SELinux by default. I know nothing about SELinux, but something that Russell said in passing caught my attention:

I am not aware of the Debian package dependencies (or those of any other distribution) being about to represent that the postfix package depends on selinux-policy-default-postfix if and only if the selinux-policy-default package is installed. Please note that I am not suggesting that we add support for such things, a package management system that can solve Sudoku based on package dependency rules is not something that I think would be useful or worth having.

(emphasis added) As it happens, a little-known fact about the Debian packaging system is that you can, in fact, describe Sudoku puzzles in it! Note for the impatient: If you want to skip all this talk and jump to the code, see debsudoku.py. For those of you who, like me, are not Sudoku-heads, here's a quick summary of the rules (for the common case of a game with 3 blocks):

1. The game is played by filling in a 9-by-9 grid with numbers between 1 and 9, inclusive. The board is divided into 9 3-by-3 blocks of grid cells in the obvious way.
2. All the cells in a column must contain different numbers.
3. All the cells in a row must contain different numbers.
4. All the cells in a block must contain different numbers.

There are many ways that you could convert a Sudoku problem into a Debian package archive, but here's a particularly simple one that is close to my summary of the rules. Create 9 * 9 * 9 = 729 packages, named cellR.C-V where R stands for row, C for column, and V for value. Each of these packages represents writing the value V in the cell (R, C). To describe the relationships of each cell to the other cells in the problem, we can create several virtual packages:

• The virtual package cellR.C represents the statement The cell (R, C) has been filled in with some value. It will be used to ensure that only one value is placed in any particular cell, and to ensure that all the cells in the puzzle are filled in with something.
• The virtual package blockR.C-V represents the statement A cell in the block (R, C) has been filled in with the value V. The block coordinates are the block-row and block-column containing the cell; for instance, the cell at (5, 8) is in block (2, 3). This package will be used to ensure that only one cell in a given block contains the same value.
• The virtual package rowR-V represents the statement A cell in the row R has been filled in with the value V. It will be used to ensure that no two cells in any given row have the same value.
• The virtual package colC-V represents the statement A cell in the column C has been filled in with the value V. It will be used to ensure that no two cells in any given column have the same value.

Converting these requirements to package dependencies is quite straightforward. For the package cellR.C-V, in block row BR and block column BC, we add these lines to the package's definition:

Provides: cellR.C, blockBR.BC-V, rowR-V, colC-V
Conflicts: cellR.C, blockBR.BC-V, rowR-V, colC-V

For instance, the cell at (5, 8) will get these provides/conflicts for V=2:

Provides: cell5.8, block2.3-2, row5-2, col8-2
Conflicts: cell5.8, block2.3-2, row5-2, col8-2

This relies on the fact that self-conflicts are ignored (see Debian Policy section 7.4). To actually describe the puzzle we want to solve, we need to add another package that requires two things:

1. Every cell contains a value.
2. The cells that the problem fixes have their fixed values.

For instance,

Package: puzzle
Depends: cell1.1, cell1.2, ..., cell5.9-3, cell6.1, ....

if we want to require that the cell at (5, 9) contains the value 3. Proof-of-concept As a proof-of-concept, I have written a hacky Python script, named debsudoku.py, that can convert ksudoku saved games into Packages files suitable for use with apt-get or aptitude. In fact, it has two modes: conflicts, which implements the conversion described above, and depends, which implements a different but logically equivalent conversion using only Depends instead of Conflicts. To use the script, save a ksudoku board without doing any work on it, then run

python debsudoku.py /path/to/the/ksudoku/board

I've only done minimal testing on this, so don't use it to fly airplanes (in fact, please don't use any Sudoku player to fly airplanes), but I did use it, in combination with aptitude, to solve a single ksudoku puzzle. Real-world evaluation Interestingly, in a test with an easy puzzle generated by ksudoku, aptitude was able to solve the puzzle in just a few seconds when it was expressed with the conflicts reduction, but failed to find a solution at all using depends (I stopped it after several minutes). I think this is due to how the depends reduction is performed. Take rows: instead of generating conflicts between cells in a row, it requires that each row contain every number between 1 and 9. The problem with this is that aptitude's resolver functions poorly when there are non-obvious conflicts in the dependency structure. It will decide to try a combination of packages that can't be installed together, but it won't realize it. As a result of this, it ends up trying many possible combinations of packages. For instance, it might try to place the number 3 in both (2, 5) and (5, 5). There is no dependency outlawing this in the depends reduction, so aptitude will happily continue trying to resolve dependencies. Since it has used 3 twice in the fifth column, it will not be able to find any way to resolve all the dependencies that state that all the numbers between 1 and 9 appear in the 5th column, but it lacks the higher-level reasoning that would allow it to break out; it will end up trying many or most of the possible ways of filling cells in before it gives up on the offending re-use of 3. Luckily, real-world package archives (what the aptitude resolver is designed to run on) don't usually look like this.

Some interesting two days have just passed. I took the airplane from Mar del Plata to Buenos Aires on the 17th, slept one night in a hostel on a mattress that felt like stone, and went to the 'Circulo del Officieles de la Mar' (or some such, I don't know spanish) where I had to give a talk. The oral feedback I've gotten so far indicates that people did appreciate it, which is good. I had originally planned to do some sightseeing in Buenos Aires after my talk, but me being tired and not immediately finding anyone else to go with me changed those plans. Instead, I just stayed at the C.O.M., where we played some Mao. Amongst other things. About an hour before I left the place, Holger Levsen gave me a tripod, with the request to take it with me to Belgium, and keep it until FOSDEM. As that would also give me the use of an tripod, and being the photography hobbyist that I am, I was too happy to oblige. There was just one problem: the suitcase that contained my luggage already had a bulge from here to tokio, so adding it was a no-no. No problem, I thought. I'll just keep it in my carry-on luggage. There's some space left there, and I certainly didn't have 5kg of carry-on luggage yet. Problem solved. The 'security' in EZE is a joke; the security officer doesn't even look at the screen while X-raying my luggage. Well done. So I fly from EZE to MAD on the first, 11-and-a-half-hour leg of my flight. Madrid Barajas is a crazy airport, with its own private metro. Somewhere between gates R-something (where IB6844 from EZE arrived) and J40 (where the flight to BRU will depart from), we need to go through a security checkpoint. Strange, since I hadn't had to do that on my way in from BRU to EZE; but of course you have no choice, so I comply. As I've gotten used to by now, what with the mess of cabling in my laptop bag and the other metal objects there, I get a baggage check. The security officer who goes through my very messy bag asks me to open the bag containing the tripod, so he can see it. He then promptly decides that it is a "sharp" object, and that it is "not acceptible". Here the fun begins. Of course, trying to argue with a security guard isn't the best idea if you want to be allowed on any flight, ever, so I don't even try that. I ask him 'can I check that in, instead?' and he tells me how to go back, to the exit, and to the departures hall where I can have that little tripod get checked in. So I go there. The lady at the check-in desk tells me that she cannot check the tripod in, because the flight has already been closed and I'm too late. I tell her I'm from a connecting flight, and that I can't help that. She says she knows. So I ask her what my other options are. She tells me I could try to get on the next flight to Brussels, which is at 20:00 (rather than the 16:20 flight that I was booked on). I consider that an acceptable compromise, so ask her how to get that done, and she refers me to the ticket sales desk, some 10 meters to my left. Unfortunately, the 20:00 flight turns out to be completely booked, and there are no other flights on the same day. The best alternative he can give me is a flight to Amsterdam at 19:20, which would cost 'a little more'. I ask him 'how much more'. He starts looking up things in his computer, and after a few minutes tells me that he cannot refund my ticket for the 16:20 flight and that a flight to amsterdam would cost me 429. At this point I get somewhat angry, and tell him that this is not acceptable. He refers me to the customer service desk right across. The lady at the customer service desk is friendly, but firm: she cannot help me. Either I leave the tripod behind, or I pay whatever the ticket service asks me—she takes my word that it is 429, because she cannot check it, and cannot change it, either. I get more agitated, and ask to see the supervisor. She points me to a lady a little across who can 'call' the supervisor. At this point it is 15:55 As I reach that lady, apparently someone else had just asked for the same thing, and starts arguing with a guy in black uniform, who appears to be the supervisor, about the fact that he's got more than 30kg of luggage which he isn't allowed to take along with him. Dude, not a single airline will let you do that, no matter how much you argue. I let them argue for about 10 minutes, all the while nervously looking at my watch, but then I interrupt with 'please, I don't have much time left before my flight leaves'. That gets his attention. In our two-minute conversation, the guy in black backs up everything the other three people have told me, adding that '429 is reasonable; your luggage is already on the flight, and it will cost Iberia a lot of money to delay the flight and get it off'. I curse. He also seems surprised at the notion that I'm not allowed to take a tripod in my carry-on luggage. Hmm. Figuring I have nothing to loose, I curse one more time, and make a run for it. When I reach the security checkpoint, I ask to be allowed to skip the queue in front of it, on the grounds that I have only 15 minutes left before my flight leaves. Mind you, this is not the same security checkpoint as the one where I was rejected about an hour earlier; this is the security checkpoint between the departures hall and the secure area, whereas the other security checkpoint was one between two different terminals. I am granted my request, and quickly throw everything in the X-ray boxes. They need to go through my bag again. I help them, showing where everything conspicuous is, and opening my flute box so they know what's in there. He looks at everything, and signals that I can move on. Hang on, I think. This can't be right. 'Okay,' I ask? 'Yes,' he says. I run for the nearest info desk, show them my boarding pass, and tell them 'please, my flight leaves in 10 minutes, can you tell me where to go?' They are quick and efficient. Gate J40. I make a run for it again, and just make it in time. Of course, I could forget about my plans of finding a power outlet, plugging in my cell phone and calling my parents, but that's not a big deal—there's a train station at Brussels Airport. So what have we learned? Security on airports is random, arbitrary, non-standardized, and utterly stupid. Items that pose no danger at all—plastic bottles containing more than 100ml of water, small quantities of yoghurt that are over 100grammes but not over 100ml, tripods—are not allowed, while on the other hand I've occasionally gotten through security carrying knives and tools that are clearly forbidden by the rules. We've also learned that the rules very much depend on the particular person checking you. If you're refused for an item that is not a knife or a gun or something else similarly problematic, it may help to just find a different security checkpoint and try again. It's not as if they make any sense, anyway.

When you have a server on the Internet, you get lots of "brute force" attacks on the SSH daemon, trying plausible logins with a variety of passwords. Even with good passwords, these attacks might eventually succeed (and they're annoying even when they don't), so you want to thwart them. One way is to use fail2ban, a script that monitors the failed connections, and sets up firewall rules (for instance) blocking further connections from the attacking IP addresses. It's good, but it fills your logs with messages about IPs getting banned and unbanned after a while. And you're still at risk that the multiple connections crash the SSH daemon, or trigger a bug in it, or whatever. A second layer of protection can be to block all SSH connection attempts except when they come from known IP addresses, but that doesn't work when you're away from home, and you're locked out. Been there, done that. So, some wise people have devised a trick called "port-knocking". It's similar to only opening the door to people who use a special knock (think "That's all, folks"): the firewall stays closed, but it opens a tiny targeted hole to some IP addresses for a limited length of time, based on a secret handshake. The window for attack is therefore very small, and the SSH daemon stays idle most of the time. And you can still log on your hosted server when you're attending conferences. There are a variety of implementations for this concept. Some could be web-based (you need to submit the right password to a web page), some could use other services or a dedicated daemon. But when I started investigating port-knocking, I wanted something simple, preferably with no dependencies on a daemon that would need to be exposed to the net and potentially crash. I found an article on the Debian Administration website, but I wasn't entirely satisfied with it. The principles appealed to me (netfilter-only, secret handshake in the form of opening connections to secret ports), though, so I evolved it into my own implementation, which I proudly present to you today. The goals of this implementation were:

• flexibility;
• robustness;
• resistance to replay attacks;
• no dependency on user-land daemons;
• simple, easily auditable code;
• must not be a pain in the neck to use.

The bulk of the work therefore stays in the kernel's netfilter (that's for robustness and no user-land dependency), but the control interface is integrated with the usual firewalling script. Resistance to replay attacks is achieved by choosing hard-to-predict ports. So if someone snoops the wireless while I'm at a conference and catches my secret handshake, it'll only be valid for a short period of time, hopefully short enough to prevent dictionary attacks. The handshake is therefore calculated as a function of the current date and time, with an added secret seed. The following shell function calculates 5 port numbers within a given range (requires dc to be installed, for big-integer arithmetic):

calc_knock_ports ()  
    secret=$1
    bottomport=$2
    topport=$3
    nbports=$(( $topport - $bottomport + 1 ))
    hash=$(TZ=UTC date +%Y-%m-%d-%H-$secret   md5sum   awk ' print $1 '   tr a-z A-Z)
    num=$(echo 16i $hash f   dc)
    pk_port1=$(echo $num $nbports 0 ^ / $nbports % $bottomport + f   dc)
    pk_port2=$(echo $num $nbports 1 ^ / $nbports % $bottomport + f   dc)
    pk_port3=$(echo $num $nbports 2 ^ / $nbports % $bottomport + f   dc)
    pk_port4=$(echo $num $nbports 3 ^ / $nbports % $bottomport + f   dc)
    pk_port5=$(echo $num $nbports 4 ^ / $nbports % $bottomport + f   dc)
 

Okay. So this function calculates ports, now what? Now we're going to define a few chains by which netfilter will store states of IP addresses as they progress through the handshake:

setup_portknocking_tables ()  
    iptables -N portknock_into_phase1
    iptables -A portknock_into_phase1 -m recent --name PK_PHASE1 --set
    # iptables -A portknock_into_phase1 -j LOG --log-level notice --log-prefix "INTO PK_PHASE1: "
    iptables -N portknock_into_phase2
    iptables -A portknock_into_phase2 -m recent --name PK_PHASE1 --remove
    iptables -A portknock_into_phase2 -m recent --name PK_PHASE2 --set
    # iptables -A portknock_into_phase2 -j LOG --log-level notice --log-prefix "INTO PK_PHASE2: "
    iptables -N portknock_into_phase3
    iptables -A portknock_into_phase3 -m recent --name PK_PHASE2 --remove
    iptables -A portknock_into_phase3 -m recent --name PK_PHASE3 --set
    # iptables -A portknock_into_phase3 -j LOG --log-level notice --log-prefix "INTO PK_PHASE3: "                     
    iptables -N portknock_into_phase4
    iptables -A portknock_into_phase4 -m recent --name PK_PHASE3 --remove
    iptables -A portknock_into_phase4 -m recent --name PK_PHASE4 --set
    # iptables -A portknock_into_phase4 -j LOG --log-level notice --log-prefix "INTO PK_PHASE4: "
    iptables -N portknock_into_phase5
    iptables -A portknock_into_phase5 -m recent --name PK_PHASE4 --remove
    iptables -A portknock_into_phase5 -m recent --name PK_PHASE5 --set
    iptables -A portknock_into_phase5 -m recent --name PK_ESTABLISHED --set
    # iptables -A portknock_into_phase5 -j LOG --log-level notice --log-prefix "INTO PK_PHASE5: "
    iptables -N portknock_accept
    iptables -A portknock_accept -m limit -j LOG --log-level notice --log-prefix "ACCEPTED AFTER PORTKNOCKING: "
    # iptables -A portknock_accept -m recent --name PK_PHASE5 --remove
    iptables -A portknock_accept -j ACCEPT
    iptables -N portknocking
 

These chains use the recent module, which seems to be commonly available in standard kernels. You'll notice how, as one packet goes through these rules, its originating IP address moves from one set of "recent" addresses to the next. But no logic exists yet to make the packet actually go through these rules, so here comes the glue:

refresh_portknocking ()  
    calc_knock_ports f00b4r 10000 10999
    iptables -F portknocking
    iptables -A portknocking -p tcp --dport $pk_port1 -m state --state NEW                                                 -j portknock_into_phase1
    iptables -A portknocking -p tcp --dport $pk_port2 -m state --state NEW -m recent --rcheck --name PK_PHASE1 --seconds 5 -j portknock_into_phase2
    iptables -A portknocking -p tcp --dport $pk_port3 -m state --state NEW -m recent --rcheck --name PK_PHASE2 --seconds 5 -j portknock_into_phase3
    iptables -A portknocking -p tcp --dport $pk_port4 -m state --state NEW -m recent --rcheck --name PK_PHASE3 --seconds 5 -j portknock_into_phase4
    iptables -A portknocking -p tcp --dport $pk_port5 -m state --state NEW -m recent --rcheck --name PK_PHASE4 --seconds 5 -j portknock_into_phase5
    # echo clear > /proc/net/ipt_recent/PK_DONE
    echo clear > /proc/net/ipt_recent/PK_PHASE1
    echo clear > /proc/net/ipt_recent/PK_PHASE2
    echo clear > /proc/net/ipt_recent/PK_PHASE3
    echo clear > /proc/net/ipt_recent/PK_PHASE4
    echo clear > /proc/net/ipt_recent/PK_PHASE5
 

Right. This function adds rules to the portknocking chain. A packet injected into this ruleset will, depending on its destination port and whether its source IP address has already been seen, end up in one of the PK_PHASE* sets. All we have to do now is therefore to send some packets to this portknocking chain, and use the port-knocking sets to decide whether to accept incoming connections or not:

iptables -A INPUT -j portknocking
iptables -A INPUT -m recent --rcheck --seconds 5 --name PK_PHASE5 -m state --state NEW -p tcp --dport ssh -j portknock_accept

This example only mentions accepting incoming SSH connections, but it's in no way a limitation: a server of mine uses similar rules to DNAT certain ports to internal IP addresses. And there we have it for the server part: incoming SSH connections are usually ignored (well, handled by the rest of the firewall script, but let's assume that it drops these packets by default), but if one IP address knows the appropriate ports and sends a connection attempt to them in order, then it'll be able to open SSH connections for a little while after that. Of course, it's going to be boring if one has to send these packets by hand, but it can be easily automated by a script. Here's a ~/bin/portknock.sh I have:

#! /bin/sh
host=$1
port=$2
calc_knock_ports ()  
[...]
 
calc_knock_ports f00b4r 10000 10999
for i in $pk_port1 $pk_port2 $pk_port3 $pk_port4 $pk_port5 ; do
    nc -w 1 $host $i < /dev/null > /dev/null 2>&1
done
nc $host $port

It's designed to be called with two parameters, a host and a port, and it needs netcat in addition to dc. Why the last line, I hear you cry? Because then I can just add the following lines to my ~/.ssh/config:

Host blahblah
  IdentityFile foobar
  ProxyCommand /home/roland/bin/portknock.sh %h %p

...and SSH will automagically tunnel its network socket through the script, which will in turn happily tunnel that through netcat after completing the secret handshake. And when I type ssh myserver on my laptop, interesting stuff happens behind the scenes, and a special, just-for-me hole is opened in the server firewall, just for the few seconds I need to establish the SSH session (packets belonging to established TCP sockets are allowed by the firewall's connection tracking). Note: This article is deliberately short on details and ready-to-run scripts. Firstly because firewall scripts vary wildly so any script would have to be adapted anyway, but mostly because security is best handled with one's brain switched on. Fiddling with a firewall can easily open gaping holes or lock everyone out. So please make sure you understand what goes on before blindly pasting stuff into your own setup. Some of the lines that are commented out may also be of interest, and were left as an exercise for the reader. Other lines were not included, and are also left as a rather important exercise to the reader; note in particular how the netfilter rules as currently established do not mitigate the replay attacks...

A large part of the disagreement about the way to manage the policy seems to be based on who will be the primary “owner” of the policy on the machine. This isn’t a problem that only applies to SE Linux, the same issue applies for various types of configuration files and scripts throughout the process of distribution development. Having a range of modules which can be considered configuration data that come from a single source seems to make SE Linux policy unique among other packages. The reasons for packaging all Apache modules in the main package seem a lot clearer. One idea that keeps cropping up is that as the policy is modular it should be included in daemon packages and the person maintaining the distribution package of the policy should maintain it. The reason for this request seems to usually be based on the idea that the person who packages a daemon for a distribution knows more about how it works than anyone else, I believe that this is false in most cases. When I started working on SE Linux I had a reasonable amount of experience in maintaining Debian packages of daemons and server processes, but I had to learn a lot about how things REALLY work to be able to write good policy. Also if we were to have policy modules included in the daemon packages, then those packages would need to be updated whenever there were serious changes to the SE Linux policy. For example Debian/Unstable flip-flopped on MCS support recently, changing the policy packages to re-enable MCS was enough pain, getting 50 daemon packages updated would have been unreasonably painful. Then of course there is the case where two daemons need to communicate, if the interface which is provided with one policy module has to be updated before another module can be updated and they are in separate packages then synchronised updates to two separate packages might be required for a single change to the upstream policy. I believe that the idea of having policy modules owned by the maintainers of the various daemon packages is not viable. I also believe that most people who package daemons would violently oppose the idea of having to package SE Linux policy if they realised what would be required of them. Caleb Case seems to believe that ownership of policy can either be based on the distribution developer or the local sys-admin with apparently little middle-ground [1]. In the section titled “The Evils of Single Policy Packages” he suggests that if an application is upgraded for a security fix, and that upgrade requires a new policy, then it requires a new policy for the entire system if all the policy is in the same package. However the way things currently work is that upgrading a Debian SE Linux policy package does not install any of the new modules. They are stored under /usr/share/selinux/default but the active modules are under /etc/selinux/default/modules/active. An example of just such an upgrade is the Debian Security Advisory DSA-1617-1 for the SE Linux policy for Etch to address the recent BIND issue [2]. In summary the new version of BIND didn’t work well with the SE Linux policy, so an update was released to fix it. When the updated SE Linux policy package is installed it will upgrade the bind.pp module if the previous version of the package was known to have the version of bind.pp that didn’t allow named to bind() to most UDP ports - the other policy modules are not touched. I think that this is great evidence to show that the way things currently work in Debian work well. For the hypothetical case where a user had made local modifications to the bind.pp policy module, they could simply put the policy package on hold - I think it’s safe to assume that anyone who cares about security will read the changelogs for all updates to released versions of Debian, so they would realise the need to do this. Part of Caleb’s argument rests on the supposed need for end users to modify policy packages (IE to build their own packages from modified source). I run many SE Linux machines, and since the release of the “modular” policy (which first appeared in Fedora Core 5, Debian/Etch, and Red Hat Enterprise Linux 5) I have never needed to make such a modification. I modify policy regularly for the benefit of Debian users and have a number of test machines to try it out. But for the machines where I am a sysadmin I just create a local module that permits the access that is needed. The only reason why someone would need to modify an existing module is to remove privileges or to change automatic domain transition rules. Changing automatic domain transitions is a serious change to the policy which is not something that a typical user would want to do - if they were to do such things then they would probably grab the policy source and rebuild all the policy packages. Removing privileges is not something that a typical sysadmin desires, the reference policy is reasonably strict and users generally don’t look for ways to tighten up the policy. In almost all cases it seems best to consider that the policy modules which are shipped by the distribution are owned by the distribution not the sysadmin. The sysadmin will decide which policy modules to load, what roles and levels to assign to users with the semanage tool, and what local additions to add to the policy. For the CentOS systems I run I use the Red Hat policy, I don’t believe that there is a benefit for me to change the policy that Red Hat ships, and I think that for people who have less knowledge about SE Linux policy than me there are more reasons not to change such policy and less reasons to do so. Finally Caleb provides a suggestion for managing policy modules by having sym-links to the modules that you desire. Of course there is nothing preventing the existence of a postfix.pp file on the system provided by a package while there is a local postfix.pp file which is the target of the sym-link (so the sym-link idea does not support the idea of having multiple policy packages). With the way that policy modules can be loaded from any location, the only need for sym-links is if you want to have an automatic upgrade script that can be overridden for some modules. I have no objection to adding such a feature to the Debian policy packages if someone sends me a patch. Caleb also failed to discuss how policy would be initially loaded if packaged on a per-module basis. If for example I had a package selinux-policy-default-postfix which contains the file postfix.pp, how would this package get installed? I am not aware of the Debian package dependencies (or those of any other distribution) being about to represent that the postfix package depends on selinux-policy-default-postfix if and only if the selinux-policy-default package is installed. Please note that I am not suggesting that we add support for such things, a package management system that can solve Sudoku based on package dependency rules is not something that I think would be useful or worth having. As I noted in my previous post about how to package SE Linux policy for distributions [3] the current Debian policy packages have code in the postinst (which I believe originated with Erich Schubert) to load policy modules that match the Debian packages on the system. This means that initially setting up the policy merely requires installing the selinux-policy-default package and rebooting. I am inclined to reject any proposed change which makes the initial install of of the policy more difficult than this. After Debian/Lenny is released I plan to make some changes to the policy. One thing that I want to do is to have a Debconf option to allow users to choose to automatically upgrade their running policy whenever they upgrade the Debian policy package, this would probably only apply to changes within one release (IE it wouldn’t cause an automatic upgrade from Lenny+1 policy to Lenny+2). Another thing I would like to do is to have the policy modules which are currently copied to /etc/selinux/default/modules/active instead be hard linked when the source is a system directory. That would save about 12M of disk space on some of my systems. I’ve taken the unusual step of writing two blog posts in response to Caleb’s post not because I want to criticise him (he has done a lot of good work), but because he is important in the SE Linux community and his post deserves the two hours I have spent writing responses to it. While writing these posts I have noticed a number of issues that can be improved, I invite suggestions from Caleb and others on how to make such improvements.

• [1] http://www.calebcase.com/node/2


• [2] http://lists.debian.org/debian-security-announce/2008/msg00201.html


• [3] http://etbe.coker.com.au/2008/08/18/se-linux-policy-packaging-distribution/

Share This