Last night here at BSP I released nagios-plugins (1.4.14-1), which has a couple of fixes and enhancements. For more information, see the changelog.
Good people steal ideas, right? On that basis I setup a static domain to host the javascript and icons I use upon a few different sites & projects. This was preempted by the release of a new version of the excellent jQuery library.
I also managed to put together a tremendous hack to solve a pretty annoying problem running multiple distributions from a single external kernel under KVM.
Ubuntu users, in particular, will be well aware of dmesg SPAM coming from the use of CONFIG_SYSFS_DEPRECATED.
In short the way that the kernel presents information beneath the /sys tree has changed over the life of the kernel - and this has a knock-on effect to the userspace supplied by different distributions and releases of GNU/Linux.
Some distributions need an "old" kernel and an "old" udev with "old" udev rules in order to create the appropriate device nodes such that the kernel will boot & mount its filesystems. (i.e. These need CONFIG_SYSFS_DEPRECATED to be set.) Conversely some distributions mandate a "new" minimum kernel version, and supply a "new" version of udev with "new" udev rules and they absolutely will not function when presented with an "old" kernel. (i.e. They must have kernels without CONFIG_SYSFS_DEPRECATED set.)I've solved this problem via a kernel patch which is both evil and genius. The details are a little me-specific, but in short:
I know this is quite premature as we're just about to release 3.3, however
development on 3.4 has already started in SVN and we've already implemented
more than 20 requests from our features tracker. This is even bigger
number than release 3.3 will have!
Well the fact is that most of these things are cosmetic or tiny improvements,
on the other side some of them were requested for very long time (the oldest
one currently dates to 2003).
The new features bring things such as direct blob download (thing we definitely
should have earlier), links to documentation for SQL commands in highlighted
SQL (what was quite easy, but nobody requested such thing and I added the
feature after seeing that Adminer does this), possibility to directly
bookmark most pages or exporting user privileges. See ChangeLog for more
detailed list of features.
You can try all new features on our demo server.
In the last few days, I found the time to spend some with KVM and libvirt. Unfortunately, there is a subject that I
haven t yet found a satisfying solution: Naming of block devices in guest instances.
This is surely a common issue, but solutions are rare. Neither an article on Usenet (in German) nor the German version
of this blog article has found solutions for the main question. I should have written this in English in the first place
and am thus translating from German to english, hoping that there will be some answers and suggestions.
KVM is quite inflexible when it coms to configure block devices. It is possible to define on the host, which files or
whole devices from the host should be visible in the guest. The documentation suggests that devices should be brought
into the guest with the virtio model, which needs suppport in the guest kernel. Importing a device as emulated ATA or
SCSI device brings a performance penalty.
The devices brought into the guest via virtio appear in the guest s dev as /dev/vd<x> and do also have their
corresponding entries in /dev/disk/by-uuid and /dev/disk/by-path. The vd<x> node is simply numbered in consecutive
order as hd<x> and sd<x>. /dev/disk/by-uuid is the correct UUID of the file system found on the device, at
least if it s a block device partitioned inside the guest and formatted with ext3 (I didn t try anything
else yet). The terminology of the /dev/disk/by-path node is not yet understood, and I am somewhat reluctant to assume
the PCI paths of emulated hardware as stable.
Looks like I am forced to configure inside the guest when I have changed the host s mass storage system (for
example, after migration to a different file system or after new block devices have been added) to accommodate for the
new order of the /dev/vd<x> or to have the UUID correctly configured. This is like calling for configuration
errors.
This is a reincarnation of the same issue that has been killed by LVM on Linuxes running on bare metal :
The block device itself has a mnemonic name which is constant even in migration and copy actions. This works without
file-system specific stuff like UUID or label, and wouldn t even be possible with a UUID (which won t be
unique in this case). The mnemonic names of LVs are also available when the data is directly written to the raw device
such as a CNFS buffer of a news server.
I would love to have something like a paravirtualized device mapper interface which would allow to say in
the host s configuration which of the host s LVs should be visible in the guest with which name in
/dev/mapper. That way, the guest s configuration could remain stable during data wrangling operations on the
host.
One solution is to have a single LV for each guest and import this LV as /dev/vda into the guest. /dev/vda would then
be partitioned like a real disk and have its own LVM installation. This would however need kpartx if one wants to access
the data from the host, and loses flexibility when resizing file systems.
These issues appear in every installation of KVM virtualization, and I would expect that there are gazillions of other
possible solutions. I am interested in knowing how other people have tackled this issue and whether there are more
possiblity that I haven t thought of before. Maybe there is a solution that doesn t leave me with the
feeling of having implemented something ugly. Does the interface between the host s KVM and the guest s
device mapper that I have been dreaming of maybe exist. Or is there any other possibility of configuring the device
node s name in the guest Linux on the host side?
I received news a little while ago that my visa has been approved by the Brasilian consulate so I will be heading to Sao Paulo tomorrow for Campus Party. This enormous event takes place in a conference center filled with thousands of hackers camping out in tents. Apparently the connection speed is on the order of 10 gigabits.
I'll be giving a talk on Open for Business, Open Source for America, a panel discussion with Simon Phipps (http://www.webmink.net) about the future of Open Source and a panel discussion about whether the corporate sponsored community model of Ubuntu has rendered the entirely organic community model of Debian obsolete (oooh, a hot one!)
Since I'll have a smoking hot net connection at the show and will be camping there (should be interesting) I'll try to send updates. Stay tuned!
$ gcc -g -shared -o libtest.so libtest.c $ objcopy --only-keep-debug libtest.so libtest.debug $ objcopy --add-gnu-debuglink=libtest.debug libtest.so $ objdump -s -j .gnu_debuglink libtest.so libtest.so: file format elf32-i386 Contents of section .gnu_debuglink: 0000 6c696274 6573742e 64656275 67000000 libtest.debug... 0010 52a7fd0a R...
$ readelf --wide --sections ./libtest.so grep build [ 1] .note.gnu.build-id NOTE 000000d4 0000d4 000024 00 A 0 0 4 $ objdump -s -j .note.gnu.build-id libtest.so libtest.so: file format elf32-i386 Contents of section .note.gnu.build-id: 00d4 04000000 14000000 03000000 474e5500 ............GNU. 00e4 a07ab0e4 7cd54f60 0f5cf66b 5799b05c .z.. .O .\.kW..\ 00f4 2d43f456 -C.V
uint32 name_size; /* size of the name */ uint32 hash_size; /* size of the hash */ uint32 identifier; /* NT_GNU_BUILD_ID == 0x3 */ char name[name_size]; /* the name "GNU" */ char hash[hash_size]; /* the hash */
(gdb) show debug-file-directory The directory where separate debug symbols are searched for is "/usr/lib/debug".
$ objdump -s -j .note.gnu.build-id /bin/ls
/bin/ls: file format elf32-i386
Contents of section .note.gnu.build-id:
8048168 04000000 14000000 03000000 474e5500 ............GNU.
8048178 c6fd8024 2a11673c 7c6a5af6 2c65b1b5 ...$*.g< jZ.,e..
8048188 d7e13fd4 ..?.
... [running gdb /bin/ls] ...
access("/usr/lib/debug/.build-id/c6/fd80242a11673c7c6a5af62c65b1b5d7e13fd4.debug", F_OK) = -1 ENOENT (No such file or directory)
Cool, eh? Okay, not that tofrodos has any new features, but at least the
following will return zero tomorrow: wget -q
http://ftp.de.debian.org/debian/dists/sid/main/binary-i386/Packages.bz2
-O- bzgrep Schmehl grep -v -c Reichle.
Just took me a bit more than a year and a half to find a reason to upload all of my
packages at least once...
Now let's take a look at the team maintained packages, which are
still left..
This is the story of a weirdly unfriendly/non-compliant IMAP server, and some nice interactions that arose from a debugging session around it. Over the holidays, i got to do some computer/network debugging for friends and family. One old friend (I'll call him Fred) had a series of problems i managed to help work through, but was ultimately basically stumped based on the weird behavior of an IMAP server. Here's the details (names of the innocent and guilty have been changed), just in case it helps other folks in at least diagnosing similar situations. the diagnosis The initial symptom was that Fred's computer was "very slow". Sadly, this was a Windows machine, so my list of tricks for diagnosing sluggishness is limited. I went through a series of questions, uninstalling things, etc, until we figured it would be better to just have him do his usual work while i watched, kibitzing on what seemed acceptable and what seemed slow. Quite soon, we hit a very specific failure: Fred's Thunderbird installation (version 2, FWIW) was sometimes hanging for a very long period of time during message retrieval. This was not exhaustion of the CPU, disk, RAM, or other local resource. It was pure network delay, and it was a frequent (if unpredictable) frustrating hiccup in his workflow. One thought i had was Thunderbird's per-server max_cached_connections setting, which can sometimes cause a TB instance to hang if a remote server thinks Thunderbird is being too aggressive. After sorting out why Thunderbird was resetting the values after we'd set them to 0 (grr, thanks for the confusing UI, folks!), we set it to 1, but still had the same occasional, lengthy (about 2 minutes) hang when transfering messages between folders (including the trash folder!), or when reading new messages. Sending mail was quite fast, except for occasional (similarly lengthy) hangs writing the copy to the sent folder. So IMAP was the problem (not SMTP), and the 2-minute timeouts smelled like an issue with the networking layer to me. At this point, i busted out wireshark, the trusty packet sniffer, which fortunately works as well on Windows as it does on GNU/Linux. Since Fred was doing his IMAP traffic in the clear, i could actually see when and where in the IMAP session the hang was happening. (BTW, Fred's IMAP traffic is no longer in the clear: after all this happened, i switched him to IMAPS (IMAP wrapped in a TLS session), because although the IMAP server in question actually supports the STARTTLS directive, it fails to advertise it in response to the CAPABILITIES query, so Thunderbird refuses to try it. arrgh.) The basic sequence of Thunderbird's side of an initial IMAP conversation (using plain authentication, anyway) looks something like this:
1 capability 2 login "user" "pass" 3 lsub "" "*" 4 list "" "INBOX" 5 select "INBOX" 6 UID fetch 1:* (FLAGS)What i found with this server was that if i issued commands 1 through 5, and then left the connection idle for over 5 minutes, then the next command (even if it was just a 6 NOOP or 6 LOGOUT) would cause the IMAP server to issue a TCP reset. No IMAP error message or anything, just a failure at the TCP level. But a nice, fast, responsive failure -- any IMAP client could recover nicely from that by just immediately opening a new connection. I don't mind busy servers killing inactive connections after a reasonable timeout. If it was just this, though, Thunderbird should have continued to be responsive. the deep weirdness But if i issued commands 1 through 6 in rapid succession (the only difference is that extra 6 UID fetch 1:* (FLAGS) command), and then let the connection idle for 5 minutes, then sent the next command: no response of any kind would come from the remote server (not even a TCP ACK or TCP RST). In this circumstance, my client OS's TCP stack would re-send the data repeatedly (staggered at appropriate intervals), until finally the client-side TCP timeout would trigger, and the OS would report the failure to the app, which could turn around and do a simple connection restart to finish up the desired operation. This was the underlying situation causing Fred's Thunderbird client to hang. In both cases above (with or without the 6th command), the magic window for the idle cutoff was a little more than 300 seconds (5 minutes) of idleness. If the client issued a NOOP at 4 minutes, 45 seconds from the last NOOP, it could keep a connection active indefinitely. Furthermore, i could replicate the exact same behavior when i used IMAPS -- the state of the IMAP session itself was somehow modifying the TCP session behavior characteristics, whether it was wrapped in a TLS tunnel or not. One interesting thing about this set of data is that it rules out most common problems in the network connectivity between the two machines. Since none of the hops between the two endpoints know anything about the IMAP state (especially under TLS), and some of the failures are reported properly (e.g. the TCP RST in the 5-command scenario), it's probably safe to say that the various routers, NAT devices, and such were not themselves responsible for the failures. So what's going on on that IMAP server? The service itself does not announce the flavor of IMAP server, though it does respond to a successful login with
You are so in, and to a logout with
IMAP server logging out, mate. A bit of digging on the 'net suggests that they are running a perdition IMAP proxy. (clearly written by an Aussie, mate!) But why does it not advertise its STARTTLS capability, even though it is capable? And why do some idle connections end up timing out without so much as an RST, when other idle connections give at least a clean break at the TCP level? Is there something about issuing the UID command that causes perdition to hand off the connection to some other service, which in turn doesn't do proper TCP error handling? I don't really know anything about the internals of perdition, so i'm just guessing here. the workaround I ultimately recommended to Fred to reduce the number of cached connections to 1, and to set Thunderbird's interval to check for new mail down to 4 minutes. Hopefully, this will keep his one connection active enough that nothing will timeout, and will keep the interference to his workflow to a minimum. It's an unsatisfactory solution to me, because the behavior of the remote server still seems so non-standard. However, i don't have any sort of control over the remote server, so there's not too much i can do to provide a real fix (other than point the server admins (and perdition developers?) at this writeup). I don't even know the types of backend server that their perdition proxy is balancing between, so i'm pretty lost for better diagnostics even, let alone a real resolution. some notes I couldn't have figured out the exact details listed above just using Thunderbird on Windows. Fortunately, i had a machine with a decent OS available, and was able to cobble together a fake IMAP client from a couple files (imapstart contained the lines above, and imapfinish contained 8 LOGOUT), bash, and socat. Here's the bash snippet i used as a fake IMAP client:
spoolout() while read foo; do sleep 1 && printf "%s\r\n" "$foo" ; done ( sleep 2 && spoolout < imapstart && sleep 4 && spoolout < imapfinish && sleep 500 ) socat STDIO TCP4:imap.fubar.example.net:143To do the test under IMAPS, i just replaced TCP4:imap.fubar.example.net:143 with OPENSSL:imap.fubar.example.net:993. And of course, i had wireshark handy on the GNU/Linux machine as well, so i could analyze the generated packets over there. One thing to note about user empowerment: Fred isn't a tech geek, but he can be curious about the technology he relies on if the situation is right. He was with me through the whole process, didn't get antsy, and never tried to get me to "just fix it" while he did something else. I like that, and wish i got to have that kind of interaction more (though i certainly don't begrudge people the time if they do need to get other things done). I was nervous about breaking out wireshark and scaring him off with it, but it turned out it actually was a good conversation starter about what was actually happening on the network, and how IP and TCP traffic worked. Giving a crash course like that in a quarter of an hour, i can't expect him to retain any concrete specifics, of course. But i think the process was useful in de-mystifying how computers talk to each other somewhat. It's not magic, there are just a lot of finicky pieces that need to fit together a certain way. And Wireshark turned out to be a really nice window into that process, especially when it displays packets during a real-time capture. I usually prefer to do packet captures with tcpdump and analyze them as a non-privileged user afterward for security reasons. But in this case, i felt the positives of user engagement (how often do you get to show someone how their machine actually works?) far outweighed the risks. As an added bonus, it also helped Fred really understand what i meant when i said that it was a bad idea to use IMAP in the clear. He could actually see his username and password in the network traffic! This might be worth keeping in mind as an idea for a demonstration for workshops or hacklabs for folks who are curious about networking -- do a live packet capture of the local network, project it, and just start asking questions about it. Wireshark contains such a wealth of obscure packet dissectors (and today's heterogenous public/open networks are so remarkably chatty and filled with weird stuff) that you're bound to run into things that most (or all!) people in the room don't know about, so it could be a good learning activity for groups of all skill levels. Tags: debugging, imap, perdition, wireshark
I wrote this for the FSF's annual membership drive where it was originally published. I am reposting it here.At its core, I think of free software as about the ability of computer users to take control of their technology. Insofar as our software defines our experience of the world and each other, software freedom is an important part of what allows us to determine the way we live, work, and communicate. Free software is not really about software in this fundamental sense; it's about bringing freedom to users through software. In free software's incredible success over the last two decades, many people have lost sight of this simple fact. We have created an incredible array of applications, libraries, and tools. We have created vibrant development and support communities. We have created new development methodologies, powerful copyleft licenses, and massive collaborative projects. But these are all how we give users freedom. They are not freedom itself. They are not what we were trying to achieve. They are our instruments, not our goal. This distinction becomes central in a world where technology is in flux. Indeed, we live in such a world. We can see signs of this in how, as most users' primary computers become mobile phones and new types of network services make up most of many users' interactions with computers, the free software movement's old applications, communities, development methodologies, and licenses can become ill-suited to, or ineffective at, protecting user freedoms. And indeed, in the next few years, bringing freedom to computer users will need to involve new software and new forms of advocacy. It will need to involve new licenses and new techniques for their enforcement. It will need to involve new forms of collaboration and organization. If the free software movement is to succeed, it must stay focused on computer users' freedom -- on the question of why we do what we do -- and then work creatively on how to best respect and protect the freedom we are working toward. If we are overly focused on how we've done things in the past, we may lose sight of the most fundamental goal of supporting users' control over their technology in general. There are many organizations that support the how of today's free software in various ways -- they are law firms and companies and nonprofit organizations supporting various free software projects. The Free Software Foundation is, by far, the most important organization focused on why --- on the underlying principle of software freedom. As such, it plays an essential role in keeping our broader community focused on the key issues, threats, and challenges that will affect the success of every free software project, and every computer user, in the present and in the future. In this period of rapid change in computer technology, its role is more vital than ever. The consequence of any failure is more dire. Here are some of the ways that I will be encouraging the FSF to serve the free software movement in the coming year: Mobile Phones In a short essay I wrote earlier this year, I pointed out that there are now billions of mobile phones and that, although these phones are increasingly powerful computers, they represent one of the most locked-down, proprietary, and unfree technologies in wide use. The implications of this fact for users' control over their technology are dire. Although some widely used phones make extensive use of free software, most free phones are locked down and Tivoized and their users remain fettered, divided, and helpless. We must raise awareness of free software issues among users of phones, communicate to users that phones are powerful general purpose computers, and explain that control over these devices has critical implications for individual autonomy in the future. Toward this end, the FSF staff will launch an advocacy campaign around mobile phones and software freedom in the coming year. Network Services As network services -- like those built by Facebook, Google, and others -- have continued to grow both in scope and penetration over the last year, the importance of a meaningful free software responses grows as well. The launch of products like Google's network-centric ChromeOS offers one glimpse of what a future computing platform may look like. The implications for user freedom, and for the effectiveness of traditional free software approaches, are frightening. The fact that many network services are built using free software does not make the effect of these services on users' autonomy and freedom any less catastrophic. In the next year, the FSF is planning to release the first of what I hope will be several statements on software freedom and network services. Building off the work of the FSF-supported group Autonomous, the Foundation will help provide guidelines for those implementing network services, for users deciding whether to use services, and for developers trying to build services that go further to respect their users' freedom. Reaching beyond our traditional communities Successfully fighting for user freedom is going to mean successfully reaching out to users outside the FSF's historical base . The FSF continues to do so with its Defective By Design anti-DRM campaign and its End Software Patents work. In the last year, the FSF has also reached out to younger users through its "GNU Generation" campaign run by and for high school students. Additionally, the FSF convened a summit this year on women in free software. The FSF plans to build on these successes in the coming year and to expand similar outreach projects. Of course, fighting for and promoting software freedom is more work than today's FSF has the resources to accomplish. Each of my three points above represents an ambitious undertaking, and yet just a portion of the items on the plate of the FSF's small but dedicated staff. Even just continuing its existing projects will require that the FSF adds hundreds of new members by the end of this period. Your membership and donations help make goals like this possible. A strong free software movement focused on the principled issues of software freedom -- and a strong FSF in particular -- will determine what freedoms the next generation of computer users will enjoy. At stake is no less than that next generation's autonomy. I know that this is not the first fundraising appeal you've read this season and I know that the weakened economy makes giving difficult for many. I understand that the cost of a membership or donation may be less easy to afford this year. But we also cannot afford a weakened FSF at this important point of technological transition. If you are not an FSF associate member, now is the time to become one. If you've read my appeal the last two years and decided to wait, now is the time to take the plunge. Membership is $120 per year ($60 for students) and payable monthly. If you are already a member, please join me in giving generously through a tax-deductible donation, or encourage a friend to sign up. The FSF is a small, humble organization of passionate individuals working tirelessly for our software freedom. I've seen firsthand that even small gifts make a difference.
There are more webpages I need to fill in... mksh's TaC, arc4random (which needs some major redesign as well) and BSD::arc4random, the RANDEX protocol (entropy exchange over IRC) and its plugins and patches, ... Not just Mac OSX (and, I hope, iPhoneOS) will soon come with mksh(1), but also Android (I prepared patches to make it /bin/sh, which works quite well although I need to find out how to make a hardlink so that #!/bin/mksh scripts will run) and Maemo, for which I wrote an mksh package in a garage project, which also needs some love w.r.t. testing on actual devices, menu integration, etc. (Please contact me if you can help with either of the three.) We also have lewellyn:#ksh making a package for the new OpenSolaris system (thanks again). People persuading Apple to put it on the jesusPhone are also welcome. (This does not mean I endorse any of these right now, I'd probably get the most of a WinCE PDA with built-in GPSr, WLAN and maybe GSM/GPRS.) English and French native speakers, please review, and Dutch native speakers may contribute a translation of, our flyers. (Source code for these is not available, sorry. Benny makes them in Quark on System 7 in Basilisk II, used to be Classic until Apple yanked it. But still, they use only free fonts, free imagery or such the MirOS Project is allowed to use, and beat every single other FOSS project flyer I've ever seen by far!) There's probably more I could write, I bet I forgot half of it anyway, but I'll leave it at that for now. Get yourself a nice cup of hot chocolate, pour an Espresso into it, and enjoy the mix with a piece of cake (I'd say strawberry or mousse-pear but all they had was cassis-cr me) and pity me for not knowing any French next month.
I ve been asked why I dislike surveymonkey surveys, I don t want to reply fully in that forum (because it s off-topic there) and I want to publish this for your feedback now.
surveymonkey really frustrates me. In short, it s often confusing, I think it doesn t follow UK accessiblity laws and I m unsure if it complies with our privacy laws. It seems like another triumph of marketing over good sense. It s a free market failure, possibly based on lack of information on the drawbacks of this option.
If a survey is on Surveymonkey, it s easiest for able-bodied users of full-fat browsers with low security and privacy settings to answer. Using NoScript or a privacy tool? You ll get an error message (which contradicts the VPAT_SurveyMonkey.pdf description). Unable to see graphics clearly? The forms will be hard to read because they ve replaced the browser form widgets (such as checkboxes and radio buttons) with javascript-driven images for no good reason.
Those sort of faults will probably bias results a bit how much will depend on the subject. It s a particular case of a problem I first mentioned in the last-but-one bullet point on my debian surveys page back in 2006. Basically, I think you should keep open surveys as open as possible. Statistics literature is littered with bizarre spurious results from when the survey method accidentally favours some particular subset of the population.
Finally, surveymonkey surveys can be left with answer requirements which make no sense, such as ticking Other requiring a text input, but the text box not accepting input. Is that the fault of the survey owner not being able to work the admin interface, or the interface having some subtle flaws? I m not completely sure, but I ve had that experience a few times.
So, can I persuade people to use limeservice, doodle or even Google Docs rather than Surveymonkey, please?
I think Limeservice is better because it s based in Germany (so EU data protection and privacy laws apply) and is cooperatively-developed FOSS software. Doodle and Google aren t, but are quicker and easier to use for straw polls.
If you want something to install on your own site and the downloadable limesurvey software seems a bit heavy, then askpeople seems another good option. Our co-op can host either of those if you re willing to pay. (If someone would be willing to sponsor it, we d host some free surveys too let me know.)
Lots of people use surveymonkey, but popularity doesn t always mean it s the best tool. Please consider the alternatives and pick the best tool for the job.

sudo pon orange but I wanted a GUI.Next.