Search Results: "ag"

1 March 2009

Sven Mueller: Rescueing files from lost+found

Oh well, I tried to search for it but failed miserably, so I m blogging about it hoping the original author might read my post on planet.debian.org (where I read the original post): A friend of mine might need exactly the sort of scripts someone posted here:
  1. Creating a list of file checksum+pathname/filename
  2. Reading files from lost+found and moving them to their original filename by identifying from the list in (1)
However, I can t recall who posted about this, and don t want to do the same work again. Please, Dear Lazyweb, help me ;-)

Gustavo Noronha Silva: WebKitGTK+ 1.1.1 released

A new version of WebKit s GTK+ port is here, as also announced by Xan! Before I start listing the new features and fixes, let me give you a download link =). For this version we have changed the binary interface slightly, which requires a soversion bump, so rebuilding applications is in order. The API is still stable, though. Another major change worthy of note is that we have made libsoup the only HTTP backend available for WebKitGTK+, removing the CURL related code from our port (the CURL backend remains in the tree, though, as it is used by other ports). This should simplify implementation and allows us to expose Soup objects in the API, reducing the amount of wrapping code that is necessary. We are already exposing the SoupSession we use internally, providing yet more power and flexibility to applications using our API. In this release we finally get to see the official debut of the Web Inspector in the GTK+ port. Can t wait to see releases of our dear browsers featuring the inspector =). Numerous improvements to hading policy decisions regarding navigations and window creation requests to the application have also come with this release. We worked hard on finding and fixing crashes for this release, but also numerous fixes for those pet peeves that bother us just a bit, but enough to add up got landed. There are more of those to kill, but things look promising. We are hoping to release again in two weeks, so expect more fixes soon. So, that s it. Grab the new version, test it, report bugs, and enjoy!

Frans Pop: The case of the self-perpetuating DNS errors

Ingredients: The last couple of days I've been plagued by some DNS errors that kept showing up in the logcheck mails for my home server which I was busy migrating from one box to another, doing an upgrade from etch/i386 to lenny/amd64 at the same time. So, plenty of stuff going on to confuse the issue. I kept getting the following messages every hour (anonymized):
named: connection refused resolving 'somedomain.org/NS/IN': xxx.yyy.zzz.nnn#53
named: connection refused resolving 'somedomain.org/NS/IN': xxx.yyy.zzz.mmm#53
named: connection refused resolving 'ns1.somedomain.org/AAAA/IN': xxx.yyy.zzz.mmm#53
named: connection refused resolving 'ns2.somedomain.org/AAAA/IN': xxx.yyy.zzz.mmm#53
named: connection refused resolving 'ns1.somedomain.org/AAAA/IN': xxx.yyy.zzz.nnn#53
named: connection refused resolving 'ns2.somedomain.org/AAAA/IN': xxx.yyy.zzz.nnn#53
The times were fairly regular: once just before the hour, most 2 minutes after. I fetch mail at around that time, but also at other times, so possible but unlikely. The 2 minutes after was the first real clue: some cron job maybe? After disabling logcheck the messages no longer appeared in the log. Enable it again, and they were back. Additional confusion was caused by the fact that the domain had "debian" in its name, but it was somewhere obscure. So why was logcheck causing a lookup for that domain? This did confuse me enough to waste some time looking for some silly weird (default) configuration problem in some package. Enter spamassassin. Apparently that was parsing the message body, recognized "somedomain.org" as a host name, and proceded to do a DNS lookup as validity check. So we have the following loop, started off by something causing an initial DNS lookup for the domain, which fails and gets logged: Duh. I remember struggling with probably the same problem a couple of years ago, but then it was a lot more severe: masses of repeating DNS errors for obscure domains. At that time I failed to get to the bottom of it and ended up just ignoring the errors by adding the following option in my bind9 configuration:
logging  
    category lame-servers   null;  ;
 ;
Anyway, now I just no longer pass logcheck mails through spamassassin. (Although filtering out these DNS errors in bind9 can be perfectly valid.)

Frans Pop: The case of the self-perpetuating DNS-errors

Ingredients: The last couple of days I've been plagued by some DNS errors that kept showing up in the logcheck mails for my home server which I was busy migrating from one box to another, doing an upgrade from etch/i386 to lenny/amd64 at the same time. So, plenty of stuff going on to confuse the issue. I kept getting the following messages every hour (anonymized):
named: connection refused resolving 'somedomain.org/NS/IN': xxx.yyy.zzz.nnn#53
named: connection refused resolving 'somedomain.org/NS/IN': xxx.yyy.zzz.mmm#53
named: connection refused resolving 'ns1.somedomain.org/AAAA/IN': xxx.yyy.zzz.mmm#53
named: connection refused resolving 'ns2.somedomain.org/AAAA/IN': xxx.yyy.zzz.mmm#53
named: connection refused resolving 'ns1.somedomain.org/AAAA/IN': xxx.yyy.zzz.nnn#53
named: connection refused resolving 'ns2.somedomain.org/AAAA/IN': xxx.yyy.zzz.nnn#53
The times were fairly regular: once just before the hour, most 2 minutes after. I fetch mail at around that time, but also at other times, so possible but unlikely. The 2 minutes after was the first real clue: some cron job maybe? After disabling logcheck the messages no longer appeared in the log. Enable it again, and they were back. Additional confusion was caused by the fact that the domain had "debian" in its name, but it was somewhere obscure. So why was logcheck causing a lookup for that domain? This did confuse me enough to waste some time looking for some silly weird (default) configuration problem in some package. Enter spamassassin. Apparently that was parsing the message body, recognized "somedomain.org" as a host name, and proceded to do a DNS lookup as validity check. So we have the following loop, started off by something causing an initial DNS lookup for the domain, which fails and gets logged: Duh. I remember struggling with probably the same problem a couple of years ago, but then it was a lot more severe: masses of repeating DNS errors for obscure domains. At that time I failed to get to the bottom of it and ended up just ignoring the errors by adding the following option in my bind9 configuration:
logging  
    category lame-servers   null;  ;
 ;
Anyway, now I just no longer pass logcheck mails through spamassassin. (Although filtering out these DNS errors in bind9 can be perfectly valid.)

28 February 2009

Daniel Baumann: dosfstools 3.0.2

I have uploaded dosfstools version 3.0.2 that contains another few fixes. Source tarballs can be downloaded as usual from its homepage. The aim for 3.0 was to unify all the patches out there that have been accumulated over the years. As this is mostly completed now, I will soon start doing real work which will lead to dosfstools 3.1 eventually.

Mike Hommey: ERROR: This RRD was created on other architecture

I can t believe we still have such things in widespread software in 2009 But it happened to me on the BTS graphs I kind of maintain on people.d.o. It so happens that ravel was upgraded to Lenny, and switched from x86 to x86-64 in the process Also, it s nice to see, once again, that the mailing list specially created to announce such changes, namely, debian-infrastructure-announce (which I still believe is useless as such announces should go to d-d-a) got no notification of this change

Uwe Hermann: Teeworlds - a fun, fast-paced, open-source, online multiplayer shooter

Teeworlds CTF screenshot I have no idea how such a great open-source game as Teeworlds has been able to exist without me hearing about it until recently. Teeworlds is a fast-paced realtime multiplayer shooter. You control a small "Tee" which can hold various weapons (hammer, gun, shotgun, laser-rifle, rocket launcher, ninja-sword) while running and jumping around frantically on the map, trying to frag as many other Tees as you can before you're killed by some other guy. Easy, eh? Teeworlds server list screenshot There's are many game servers to choose from, as well as various game modes (death match, team death match, capfure the flag and some unofficial "mods"). You can join servers on the Internet, or create your own server, be it a public one or a LAN server. Installation Usually I would suggest apt-get install teeworlds, but for now the packages in unstable are an older 0.4.x version, whereas upstream released a much-improved 0.5.1 version. I have already filed a bug and I'm optimistic there'll be a new version in unstable soon. In the mean-time however, you can manually build the game from source via:
 $ apt-get install zlib-dev libsdl1.2-dev
   (maybe also libgl, libglu, and python, if not already installed)
 $ wget 'http://teeworlds.com/trac/bam/browser/releases/bam-0.2.0.zip?format=raw' -O bam-0.2.0.zip
 $ wget http://teeworlds.com/files/teeworlds-0.5.1-src.zip
 $ unzip bam-0.2.0.zip
 $ unzip teeworlds-0.5.1-src.zip
 $ cd bam-0.2.0
 $ ./make_unix.sh
 $ cd ../teeworlds-0.5.1-src
 $ ../bam-0.2.0/src/bam release
Teeworlds stats screenshot Running the game
 $ ./teeworlds
You'll obviously need a working OpenGL/DRI setup (check if "glxinfo grep direct" says "Yes"), otherwise the game will be way too slow. In case you experience graphics glitches and distortions, first exit the game, then:
 $ vi ~/.teeworlds/settings.cfg
Change the "gfx_noclip 0" option there to "gfx_noclip 1" and restart the game. Teeworlds DM screenshot Firewalls If you use a local firewall as I do, you need to open at least ports 8300-8303 (UDP), even better 8300-8310 for more choice in game servers:
 $ iptables -A OUTPUT -m state --state NEW -p udp --dport 8300:8310 -j ACCEPT
Further resources Have fun!

27 February 2009

Jordi Mallach: natura upgraded to lenny

natura.oskuro.net, the home server which still serves this blog, has been suffering hardware problems for some weeks. Apparently the hard drive is failing intermittently, so every now the kernel starts spewing out noisy errors about its main disk dying. If I notice this quickly, it can be rebooted and that normally fixes it for a few more days. But if I don't, it'll end up giving nasty bus errors which will make remote logins a challenge. Most processes still work, but the filesystem appears to be gone. It's easy to know what's going on if you visit the blog's url and get some 404, and in that case I can only phone my father and tell him to press the reset button (I've tried sysrqd, but I need to open the port in the router and haven't had chance to do that yet). So it was time to do something about it, and the other day I installed a dirty 40GB drive on the second IDE controller, in case I could find the time to do somethng about it. Being with an endless pharyngitis that doesn't seem to get cured entirely, I've had some time today to look at it. This evening, I was about to transfer all the system to the new disk (it's half the size as the broken one, and probably slower, but it hopefully has no bad sectors), but I decided to upgrade the system first. natura was first installed in late 1997 or at the beginning of 1998, using the Debian bo install media on a Pentium 150MHz, and has gone through seven dist-upgrades which, as far as I can remember, have always worked out without major problems. The upgrade to lenny hasn't been an exception. The server has gradually lost many of the services it once hosted, so there aren't too many services to take care of anymore. All the mail services I setup for my father ended up being deprecated as they started to get used to Hotmail, GMail and so on, and the frequent hardware crashes made me switch them to the Linksys based DHCP server. In the end, the problems I saw after the upgrade were very similar to what I faced when I upgraded to etch: Such an ancient install will clearly have old, obsolete packages. I installed apt-show-versions to find out what didn't match my package sources. I found I had every single version of cpp, gcc and g++ from 2.95 to 4.3, and a myriad of obsolete libs. But there were also real gems:
defrag 0.73pjm1-7 installed: No available version in archive
figlet 2.2.1-3 installed: No available version in archive
ipmasqadm 0.4.2-2 installed: No available version in archive
isapnptools 1.26-5 installed: No available version in archive
ms-sys 2.1.0-1 installed: No available version in archive
queso 0.980922b-3 installed: No available version in archive
update 2.11-4 installed: No available version in archive
Spaniards will remember queso because it was written by Jordi Murg and became a classic tool to find out what OS was running on a remote host. update was apparently needed to flush your filesystems prior to Linux 2.2.8, and defrag is obvious, although leaves me wondering why it was needed at the time. With the upgrade done successfully, next step is trying to get the system transfered to the spare hard drive. For this, I first partitioned it creating a primary partition using up more or less half of the available space, and setup a LVM volume, leaving some free PE's in the volume group just in case I want to do snapshots in the future, and formatted it using ext3. I then transfered the system to the new disk and now face the boot challenge. I haven't created a boot partition and that should be a double problem: the BIOS is buggy and will only boot from the first 1024 cylinders, and my root is on LVM and GRUB legacy might not like it (but I'm not sure). However, I've become a big fan of GRUB2, and know I will be able to boot no matter what my BIOS thinks of my disks and regardless the complex root partition setup I throw at it. The plan is to install GRUB onto the new drive's MBR, and set it up using the ata module, which should allow to ignore what the BIOS says, and read beyond cylinder 1024 or even boot from CD-ROM. However, this isn't a setup I haven't tried before, and a single failure will result in me taking a train to fix it on-site. So, GRUB experts out there, any suggestions? Of course, for now I guess I can install GRUB in the current drive's MBR and make it boot the old kernel using the new system as root, but that's dirty and would just postpone the problem.

Neil Williams: blogging using 2.6.28 on Aspire1

This is my first entry from my Acer Aspire1 since installing the latest Emdebian Grip and the 2.6.28 kernel that now provides free wireless support. A few tweaks still needed to get a few packages sorted out so I've got a Debian source on this box too. Still, with my usual packages (iceweasel, sylpheed, grisbi, geany), all my email cached data, liferea data, a few documents and a few Debian packages and I'm still at only 753Mb used.

Using wicd from squeeze, I've got wireless working - there's no led lit but I'm sure that's fixable.

Copying a few files around from ~/.gnome2/ on my other laptop and suddenly I have drivel, sylpheed and liferea all pre-configured.

I'll be blitzing this install again numerous times in the next few days but by keeping the ~/.gnome2/ files on a USB stick, it will be simple to restore the system, especially as I'm using IMAP email.

(If you're wondering, the drivel upload is delayed by the churn in Debian Sid - waiting for the libmtp7 8 transition to make rhythmbox available in a clean chroot.)

Daniel Kahn Gillmor: redundant DHCP service for a sprawling LAN?

I'm supporting a medium-sized (~70 regular clients and a half-dozen servers) LAN. It's a single logical ethernet segment, but it's built as a tree of cascaded switches. DHCP service is provided by a single host running ISC's DHCPD (via dhcp3-server). DHCP itself is one of the single points of failure in the network layout. i'd really like to make this DHCP server redundant (so that i can take that host down for service if needed and leave the rest of the network intact). However, reading dhcpd.conf(5) makes me pretty worried that the failover stuff is not well-tested or widely deployed. I've read Paul Heinlein's Failover with ISC DHCP, which makes it look not unreasonable, but i was wondering if people have other preferred mechanisms for providing DHCP redundancy. Do you have failover DHCP set up for any LAN that you manage? If so, what do you use? Are there any gotchas to watch out for? I'm also concerned about the security implications. On a network that's not using IPSEC, i don't see any mechanism for the two DHCP servers to properly mutually authenticate. Is it really just by IP address? Could someone spoofing the IP address of one host corrupt the state of the other DHCP server? (i'm less concerned about them keeping network traffic private, since most of what they communicate is likely to go out in the clear on the wire anyway). Am i missing some clever authentication technique? From a security point of view, i understand that there are more severe security problems with DHCP itself, of course (the protocol requires that the client trust the (unauthenticated) server), but that doesn't seem likes a good reason to introduce an opportunity to compromise any given server directly. Your thoughts on DHCP redundancy?Tags: dhcp, failover, redundancy

Bastian Venthur: git-svn branch

Dear Lazyweb, how do I push a local git branch to a new svn branch? The other way round is quite easy and well documented but I didn t find a single document describing what I want to do. I m working with an svn repository with standard layout. I use git-svn to be able to use git locally. Until now I ve always had my master branch reflect the svn s trunk. For every new feature, I created a local git branch, merged with the master branch when I was done and dcommit ed it to the svn s trunk. But now I need to publish an experimental feature which should not yet go into the trunk so I have to publish my local git branch to an svn branch. I tried git-svn branch foo in the local git branch and it created a svn branch/foo for me but not from the current git branch but from the current master (remote/trunk). That s of course not what I wanted, I wanted to push the current branch where I m in and have it linked to the remote svn branch. Neither the manpage of git-svn nor some documentation in the web helped me. I even found a pretty detailed document which describes each and every aspect of git-svn but unfortunately not mine. Is this use case so unusual that no one else uses git-svn this way or is it my weak git-FU?

Michael Banck: 27 Feb 2009

Yesterday evening, I mistyped my GPG passphrase a couple of times. I wasn't very worried back then and didn't try further, after all I was at a pub before and had one or two beers. Today, I wasn't able to correctly type my GPG passphrase, either. At this point, I got a slight panic. I don't have my GPG passphrase written down anywhere, didn't make up any mnemonics for it and I wouldn't be able to easily write it down anyway - it just flowed naturally through my fingers until today. In fact, I couldn't even tell how many characters/digits there are exactly. Maybe once or twice a month I would make a typo, but always get it right on second attempt. And now, from one day to the other, my fingers just couldn't remember anymore. I literally tried hundreds of times, with different characters and character combinations, but once I started thinking about what my passphrase might be, I couldn't just type it in sub-consciously anymore. After a while I became really worried - are these the first signs of Alzheimer or something? I even tried doing something else for a while as a distraction and then suddently jumping back at the keyboard, hoping the magic would return into my fingers when they got taken by surprise. A couple of hours and countless retries later, I suddently typed in the correct passphrase once. Luckily, I was even able to reproduce it after a couple of more retries! However, it took me another five minutes to figure out what was the problem, now that it seemed natural to correctly type my passphrase again. In the end, it turned out my fingers forgot to capitalize a letter.

MJ Ray: New #ukgovOSS Action Plan

Earlier this week, Tom Watson, Cabinet Office Parliamentary Secretary, published the Open Source, Open Standards and Re Use: Government Action Plan. It s had a pretty mixed reaction, with mild scepticism (niq s soapbox: Is gov.uk going open-source?) being the average reaction from what I ve read. I particularly liked the kind offer by Bristol Wireless to debianise Tom Watson s laptop. I think commentators are bang-on that the procurement process needs to change and that this sounds positive. I ll love it if I m wrong, but this looks like the lip service as usual which I ve seen in the last 10 years working on FOSS in the UK. I want to see the action that comes from this plan! When government actually starts buying FOSS from typical FOSS service providers and not just the IBMs of this world, then I ll believe it. My suspicion is re-ignited by some of the activites around this action plan. For example, does anyone know why the Cabinet Office didn t select FOSS to run their special public FOSS Aggregation page? (Actually, what s the best FOSS tag aggregator web service out there? I know we can set them up in Wordpress widgets, but what hosted services are there?) I m also a bit bemused that their page requires users to accept cookies until they expire, yet its privacy policy explicitly says You may suppress cookies after your visit or configure your Internet browser to prevent them. Yes, I can prevent them, but then it does nothing useful! (Based on a comment I made at the OSS Watch team blog and discussions with a few user groups.)

Neil Williams: Goodbye network-manager, hello wicd

I've not had that many problems with network-manager, until now. The new version in unstable is just completely broken on my machine. Sadly, I can't report why it failed as I inadvertently closed the terminal with the relevant output. :-(

Suffice to say, nm repeatedly got the wrong IP address, an invalid route and failed to set a usable nameserver. I've had those problems before with wireless but when this started happening with a wired connection (which had always previously worked), it was time to move on.

I've noticed people using wicd instead and decided I didn't have time to debug nm, I have a worksession starting in Cambridge on Sunday and a shed load of work to do before that, all of which requires a stable network connection.

Installing wicd replaced network-manager which was nice, (nm has always had problems working with any other network config so this allowed me to skip the removal of nm myself). wicd initially had a few issues - which I managed to pin down to the remnants of the broken nm config and steps that I'd taken to try to fix it.

Turns out that I may have some unresolved issues with my router - even with wicd I had problems getting a route over wireless. However, wicd allows suitable configuration that both works and is easy to keep discrete from other configs. Nice work. I have to set a static IP over wireless in order to set a valid route (and netmask). Looks like the DHCP implementation on the router is borked.

I'm sticking with wicd though - the interface is better for what I need, the icon has a useful traffic indication as well as signal strength and it's worth pursuing to see if the previous intermittent nm errors affect wicd.

Sune Vuorela: VPS with large storage

Dear Lazyweb I m looking to retire my basement server. It is a athlon 700MHz processor with 384mb ram and 200G harddrive (of which 40 or so is used) I don t currently need a more beefy machine for this, so I kind of thought of just going for a small virtual private server.
Unfortunately, in order to get a VPS with more than 40G space on places like linode, gandi and slicehost I also need to buy much more ram and cpu power than I actually need. Are there any vps sites out there that are a bit cheaper when talking about disk space? And native IPv6 would be nice as well.

Yves-Alexis Perez: Xfce 4.6 on Debian

As some people may have noticed (especially people in #debian-xfce and other Xfce fans), Xfce 4.6.0 has just been released. Packages are already prepared, since the packaging is done in parallel with the release cycle. I don't really plan to upload anything to my repository, now that Lenny is released and we can break unstable. That being said, I won't upload to unstable directly, as we have some packages which need to go through NEW. NEW is quite huge this days, so those packages won't leave it soon, and for consistency I'd prefer not breaking unstable completely, with a mix of 4.4 and 4.6. So first uploads targeted at experimental, and then when all packages are out of NEW, unstable. In the meantime, I already uploaded a new desktop-base version with config files useful for the new configuration system in 4.6. One of theses files will conflict with one in xfce4-session. Yeah, I shipped that file in xfce4-session when it was private, now it's time to go official. So yes, this morning you may have a conflict with desktop-base. It's not a problem in desktop-base (you don't need to report a bug). Just an inconsistency with an un-official, un-supported repository package, and an official package from main. Just force the overwrite or something like that. Or hold the upgrade until xfce4-session is available. As I already said, no special dependencies informations will be used in packages uploaded to Debian, so it may break your system at upgrade. The best solution would be to clean the packages from my repository before trying to upgrade to the ones in Debian (but it should not be too broken, in the end). For the record, besides the desktop-base / xfce4-session conflict, I didn't have any problem upgrading to the packages I should upload to experimental. More on this when things have settle down and we start uploading to unstable.

26 February 2009

Riku Voipio: The not-slug: SheevaPlug

popcon lists approximately 1000 arm/armel Debian installations. There is also listed 864 installations of nslu2-utils, letting us estimate that approximately 85% of debian/arm(el) installations are Linksys nslu2's. It is nicked sympathetically as "The Slug", which pretty accurately describes the performance of nslu2. Still, people have found absolutely amazing amount of ways to use their slugs. What would you do with something approximately 10x more powerful with same prize/size range?

Enter the Marvell SheevaPlug




What Slug SheevaPlug
CPU 266Mhz 1.2Ghz
Cache 32KB 32+256KB
Flash 8MB 512MB
MEM 32MB 512MB
Net 100Mb 1Gb


And that's not everything - SheevaPlug comes with SDIO slot and miniusb to be used as a serial console (and JTAG). No soldering needed for hacking.

Some more details on the LinuxDevices article.

For those of you who think that has one port too few of something, or don't like the wall-wart design, Other devices based on kirkwood SoC (which SheevaPlug is based on) are on the way from various ODM/OEM houses.

Michal Čihař: Fun with software catalogues

Recently, I made available information about Gammu and Wammu in machine readable formats (currently DOAP and PAD). In order to fully use advantage of PAD, I decided to push both Gammu and Wammu to some PAD enabled sites I found. At least it should give it a bit more visibility in Windows world. When I started looking for these sites, I was surprised that there is so much of them. Yes it is quite easy to setup such site as ASP provides PHP scripts to make such site, but I expected that people would be a bit sane and would not start 1001th software catalogue. But anyway, let's submit my software there to spread the world a bit ;-). The important site (at least as far as I remember them from times when I was using Windows) like Tucows or cnet downloads, have some lengthy review process, what is okay for me. But more funny are random small sites, where you immediately get 5/5 stars or sticker 100% Vista Compatible. Who cares, let's see if it will really attract some users.

Theodore Ts'o: Fast ext4 fsck times, revisited

Last night I managed to finish up a rather satisfying improvement to ext4 s inode and block allocators. The ext4 s original allocator was actually a bit more simple-minded than ext3 s, in that it didn t implement the Orlov algorithm to spread out top-level directories for better filesystem aging. It also was buggy in certain ways, where it would return ENOSPC even when there were still plenty of inodes in the file system. So I had been working on extending ext3 s original Orlov allocator so it would work well with ext4. While I was at it, it occurred to me that one of the tricks I could play with ext4 s flex groups (which are higher-order collection of block groups), was to bias the block allocation algorithms such that the first block group in a flexgroup would be preferred for use by directories, and biased against data blocks for regular files. This meant that directory blocks would get clustered together, which cut a third off the time needed for e2fsck pass2:
Comparison of e2fsck times on an 32GB partition
Pass ext4 old allocator ext4 new allocator
time (s) I/O time (s) I/O
real user system MB read MB/s real user system MB read MB/s
1 6.69 4.06 0.90 82 12.25 6.70 3.63 1.58 82 12.23
2 13.34 2.30 3.78 133 9.97 4.24 1.27 2.46 133 31.36
3 0.02 0.01 0 1 63.85 0.01 0.01 0.01 1 82.69
4 0.28 0.27 0 0 0 0.23 0.22 0 0 0
5 2.60 2.31 0.03 1 0.38 2.42 2.15 0.07 1 0.41
Total 23.06 9.03 4.74 216 9.37 13.78 7.33 4.19 216 15.68

As you may recall from my previous observations on this blog, although we hadn t been explicitly engineering for this, a file system consistency check on an ext4 file system tends to be a factor of 6-8 faster than the e2fsck times on an equivalent ext3 file system, mainly due to the elimination of indirect blocks and the uninit_bg feature reducing the amount of disk reads necessary in e2fsck s pass 1. However, the ext4 layout optimizations didn t do much for e2fsck s pass 2. Well, the optimization of the block and inode allocators is complementary to the original ext4 fsck improvements, since it focuses on what we hadn t optimized the first time around: e2fsck pass 2 times have been cut by a third, and the overall fsck time has been cut by 40%. Not too shabby! Of course, we need to do more testing to make sure we haven t caused other file system benchmarks to degrade, although I m cautiously optimistic that this will end up being a net win. I suspect that some benchmarks will go up by a little, and others will go down a little, depending on how heavily the benchmark exercises directory operations versus sequential I/O patterns. If people want to test this new allocator, it is in the ext4 patch queue. If all goes well, I will hopefully be pushing it to Linus after 2.6.29 is released, at the next merge window.

horizontal separator For comparison s sake, here is a comparison of the fsck time of the same collection of files and directories, comparing ext3 and the original ext4 block and inode allocator. The file system in question is a 32GB install of Ubuntu Jaunty, with a personal home directory, a rather large Maildir directory, some linux kernel trees, and an e2fsprogs tree. It s basically the emergency environment I set up on my Netbook at FOSDEM. In all cases the file systems were freshly copied from the original root directory using the command rsync -axH / /mnt. It s actually a bit surprising to me that ext3 s pass 2 e2fsck times was that much better than e2fsck time under the old ext4 allocator. My previous experience has shown that the two are normally about the same, with a write throughput of around 9-10 MB/s on for e2fsck s pass 2 for both ext3 file systems and ext4 file systems with the original inode/block allocators. Hence, I would have expected ext3 s pass2 time to have been 12-13 seconds, and not 6. I m not sure how that happened, unless it was the luck of draw in terms of how things ended up getting allocated on disk. So I m not too sure what happened there, but overall things look quite good for ext4 and fsck times!
Comparison of e2fsck times on an 32GB partition
Pass ext3 ext4 old allocator
time (s) I/O time (s) I/O
real user system MB read MB/s real user system MB read MB/s
1 108.40 13.74 11.53 583 5.38 6.69 4.06 0.90 82 12.25
2 5.91 1.74 2.56 133 22.51 13.34 2.30 3.78 133 9.97
3 0.03 0.01 0 1 31.21 0.02 0.01 0 1 63.85
4 0.28 0.27 0 0 0 0.28 0.27 0 0 0
5 3.17 0.92 0.13 2 0.63 2.60 2.31 0.03 1 0.38
Total 118.15 16.75 14.25 718 6.08 23.06 9.03 4.74 216 9.37
Vital Statistics of the 32GB partition
312214 inodes used (14.89%)
263 non-contiguous files (0.1%)
198 non-contiguous directories (0.1%)
# of inodes with ind/dind/tind blocks: 0/0/0
Extent depth histogram: 292698/40
4388697 blocks used (52.32%)
0 bad blocks
1 large file
263549 regular files
28022 directories
5 character device files
1 block device file
5 fifos
615 links
20618 symbolic links (19450 fast symbolic links)
5 sockets
312820 files
Related posts (automatically generated):

  1. Fast ext4 fsck times This wasn t one of the things we were explicitly engineering...
  2. Wanted: Incremental Backup Solutions that Use a Database Dear Lazyweb, I m looking for recommendations for Open Source backup...
  3. Ext4 is now the primary filesystem on my laptop Over the weekend, I converted my laptop to use the...

Theodore Ts'o: Fast ext4 fsck times, revisited

Last night I managed to finish up a rather satisfying improvement to ext4 s inode and block allocators. The ext4 s original allocator was actually a bit more simple-minded than ext3 s, in that it didn t implement the Orlov algorithm to spread out top-level directories for better filesystem aging. It also was buggy in certain ways, where it would return ENOSPC even when there were still plenty of inodes in the file system. So I had been working on extending ext3 s original Orlov allocator so it would work well with ext4. While I was at it, it occurred to me that one of the tricks I could play with ext4 s flex groups (which are higher-order collection of block groups), was to bias the block allocation algorithms such that the first block group in a flexgroup would be preferred for use by directories, and biased against data blocks for regular files. This meant that directory blocks would get clustered together, which cut a third off the time needed for e2fsck pass2:
Comparison of e2fsck times on an 32GB partition
Pass ext4 old allocator ext4 new allocator
time (s) I/O time (s) I/O
real user system MB read MB/s real user system MB read MB/s
1 6.69 4.06 0.90 82 12.25 6.70 3.63 1.58 82 12.23
2 13.34 2.30 3.78 133 9.97 4.24 1.27 2.46 133 31.36
3 0.02 0.01 0 1 63.85 0.01 0.01 0.01 1 82.69
4 0.28 0.27 0 0 0 0.23 0.22 0 0 0
5 2.60 2.31 0.03 1 0.38 2.42 2.15 0.07 1 0.41
Total 23.06 9.03 4.74 216 9.37 13.78 7.33 4.19 216 15.68

As you may recall from my previous observations on this blog, although we hadn t been explicitly engineering for this, a file system consistency check on an ext4 file system tends to be a factor of 6-8 faster than the e2fsck times on an equivalent ext3 file system, mainly due to the elimination of indirect blocks and the uninit_bg feature reducing the amount of disk reads necessary in e2fsck s pass 1. However, the ext4 layout optimizations didn t do much for e2fsck s pass 2. Well, the optimization of the block and inode allocators is complementary to the original ext4 fsck improvements, since it focuses on what we hadn t optimized the first time around: e2fsck pass 2 times have been cut by a third, and the overall fsck time has been cut by 40%. Not too shabby! Of course, we need to do more testing to make sure we haven t caused other file system benchmarks to degrade, although I m cautiously optimistic that this will end up being a net win. I suspect that some benchmarks will go up by a little, and others will go down a little, depending on how heavily the benchmark exercises directory operations versus sequential I/O patterns. If people want to test this new allocator, it is in the ext4 patch queue. If all goes well, I will hopefully be pushing it to Linus after 2.6.29 is released, at the next merge window.

horizontal separator For comparison s sake, here is a comparison of the fsck time of the same collection of files and directories, comparing ext3 and the original ext4 block and inode allocator. The file system in question is a 32GB install of Ubuntu Jaunty, with a personal home directory, a rather large Maildir directory, some linux kernel trees, and an e2fsprogs tree. It s basically the emergency environment I set up on my Netbook at FOSDEM. In all cases the file systems were freshly copied from the original root directory using the command rsync -axH / /mnt. It s actually a bit surprising to me that ext3 s pass 2 e2fsck times was that much better than e2fsck time under the old ext4 allocator. My previous experience has shown that the two are normally about the same, with a write throughput of around 9-10 MB/s on for e2fsck s pass 2 for both ext3 file systems and ext4 file systems with the original inode/block allocators. Hence, I would have expected ext3 s pass2 time to have been 12-13 seconds, and not 6. I m not sure how that happened, unless it was the luck of draw in terms of how things ended up getting allocated on disk. So I m not too sure what happened there, but overall things look quite good for ext4 and fsck times!
Comparison of e2fsck times on an 32GB partition
Pass ext3 ext4 old allocator
time (s) I/O time (s) I/O
real user system MB read MB/s real user system MB read MB/s
1 108.40 13.74 11.53 583 5.38 6.69 4.06 0.90 82 12.25
2 5.91 1.74 2.56 133 22.51 13.34 2.30 3.78 133 9.97
3 0.03 0.01 0 1 31.21 0.02 0.01 0 1 63.85
4 0.28 0.27 0 0 0 0.28 0.27 0 0 0
5 3.17 0.92 0.13 2 0.63 2.60 2.31 0.03 1 0.38
Total 118.15 16.75 14.25 718 6.08 23.06 9.03 4.74 216 9.37
Vital Statistics of the 32GB partition
312214 inodes used (14.89%)
263 non-contiguous files (0.1%)
198 non-contiguous directories (0.1%)
# of inodes with ind/dind/tind blocks: 0/0/0
Extent depth histogram: 292698/40
4388697 blocks used (52.32%)
0 bad blocks
1 large file
263549 regular files
28022 directories
5 character device files
1 block device file
5 fifos
615 links
20618 symbolic links (19450 fast symbolic links)
5 sockets
312820 files
Related posts (automatically generated):

  1. Fast ext4 fsck times This wasn t one of the things we were explicitly engineering...
  2. Wanted: Incremental Backup Solutions that Use a Database Dear Lazyweb, I m looking for recommendations for Open Source backup...
  3. Ext4 is now the primary filesystem on my laptop Over the weekend, I converted my laptop to use the...

Next.

Previous.