Search Results: "ron"

2 March 2009

Alastair McKinstry: Titan, by Stephen Baxter

Titan, by Stephen Baxter describes a near-future manned mission to Titan, Saturn's moon, by Space Shuttle, no less. This is the first book i've read by Stephen Baxter, and will probably not be the last. It's well written, the most science-based outer planets mission story i've read in since Arthur C. Clarke, though with a pessimistic edge. The story turns on a waning NASA using its remaining hardware on one big mission, to Titan, where life may have been discovered. Most of the book concentrates naturally enough on getting to Titan, rather than what they found when they get there, but Baxters ammonia-based chemistry is intriguing. The physics of what it would be like to walk on the moon are a little underdone, though. Its an understatement to say that Titan is cold: serious work on insulation would be needed to stop your base melting the "permafrost" and sinking. While we don't know what life might be present, it would be good to imagine the real tasks involved in being there. But it's his idea that NASA and others really don't want space that is most interesting. That the military want space as their baliwick, and would rather frustrate the development of technologies so that no other nations or groups interfere is well worth thinking about. To a degree, I would agree: frustrate, but not he puts it, stop. Sooner or later the technology has other uses on Earth to the point where even amateurs would launch their own satellites. The vast bulk of technology development for future space use will be done not only on Earth but for Earth use: the minaturization that NASA and the military did for the space race in the 1960's now happens for consumer electronics. 3 D fabricators are being built for prototyping, and the prospect of getting a self-contained space mission together (not reliant on Earth) draws closer. I fully expect private groups, rather than national efforts, will be the first to colonise outer space, assembling pre-existing technologies. Tags , ,

Bdale Garbee: TeleMetrum Power Supplies

After a long hiatus for various reasons, I finally had the chance to try my hand at loading a TeleMetrum board today! Sadly, I wasn't able to completely load any boards, because I somehow ended up with the wrong Digi-Key part numbers for 4 capacitors, one of which is a critical value. I placed an order for the missing parts and a few other bits we'll need eventually, hopefully they'll be here in a couple days and I can try to load a board with all the parts. Instead, what I did today was a a partial load of a board with the goal of testing the various power supplies. Seemed like a good idea, since I've had my share of odd problems with power supplies in the past. This also gave me a chance to try out my solder paste stencil, get some experience hand-placing the tiny 0402 passive parts, and an excuse to see if I could hit the solder reflow temperature profile adequately with my electric cooking skillet and IR non-contact thermometer. The good news is that it all worked right the first time! We're successfully charging a LiPo battery from USB power, and successfully making 3.3 volts for the electronics from that. The resistor divider designed to allow the LiPo battery voltage to be sampled by the CPU's analog to digital converter is also working fine. A few observations are in order, however. Getting the right amount of solder paste down on the board requires some finesse with the spreading blade and the stencil. I think the paste was a bit heavy around the CC1111 footprint, as I ended up with some bridged pads. Since I didn't load the actual CPU part, it could just be that not having something real there to wick up some solder meant there was enough to form the bridges. Don't know. The amount of solder on the ground pad in the middle of the chip looks good, though! The 0603 sized LED in the power supply circuit was the only part that didn't self-align correctly. I may have had it a bit too far to one side. The pads on the part have notches in the end, and it looks like one side of each is more or less centered, so I think surface tension did the best it could with what I gave it to work with. The LED soldered ok, isn't shorting to anything else, and clearly works fine... but I'd be happier if it were aligned better. Found and fixed a solder bridge between two pins on the surface-mount USB connector before first application of power. Could be further data that the paste layer was a bit too thick. On the other hand, I had to wiggle the connector around a bit to get it aligned correctly, so I may have smudged the paste into a bridge while doing that. I also note that there's some visible flux left on the surface of the solder, particularly on big paste areas like the feet of the USB connector. I suspect this means I didn't dwell long enough at either or both of the preheat or ultimate reflow temperatures. Since all the parts clearly soldered adequately to function, I'm still pretty happy... but since the rockets we fly experience violent accelerations during boost, this is something I'll play with on future reflows. All in all, a great way to spend an afternoon, and a big step forward for the TeleMetrum project!

Theodore Ts'o: SSD s, Journaling, and noatime/relatime

On occasion, you will see the advice that the ext3 file system is not suitable for Solid State Disks (SSD s) due to the extra writes caused by journaling and so Linux users using SSD s should use ext2 instead. However, is this folk wisdom actually true? This weekend, I decided to measure exactly what the write overhead of journaling actually is in actual practice. For this experiment I used ext4, since I recently added a feature to track the amount of writes to the file system over its lifetime (to better gauge the wear and tear on an SSD). Ext4 also has the advantage that (starting in 2.6.29), it can support operations with and without a journal, allowing me to do a controlled experiment where I could manipulate only that one variable. The test workload I chose was a simple one: For the first test, I ran the test using no special mount options, and the only difference being the presence or absence of the has_journal feature. (That is, the first file system was created using mke2fs -t ext4 /dev/closure/testext4, while the second file system was created using mke2fs -t ext4 -O ^has_journal /dev/closure/testext4.)
Amount of data written (in megabytes) on an ext4 filesystem
Operation with journal w/o journal percent change
git clone 367.7 353.0 4.00%
make 231.1 203.4 12.0%
make clean 14.6 7.7 47.3%
What the results show is that metadata-heavy workloads, such as make clean, do result in almost twice the amount data written to disk. This is to be expected, since all changes to metadata blocks are first written to the journal and the journal transaction committed before the metadata is written to their final location on disk. However, for more common workloads where we are writing data as well as modifying filesystem metadata blocks, the difference is much smaller: 4% for the git clone, and 12% for the actual kernel compile. The noatime mount option Can we do better? Yes, if we mount the file system using the noatime mount option:
Amount of data written (in megabytes) on an ext4 filesystem mounted with noatime
Operation with journal w/o journal percent change
git clone 367.0 353.0 3.81%
make 207.6 199.4 3.95%
make clean 6.45 3.73 42.17%
This reduces the extra cost of the journal in the git clone and make steps to be just under 4%. What this shows is that most of the extra meta-data cost without the noatime mount option was caused by update to the last update time for kernel source files and directories. The relatime mount option There is a newer alternative to the noatime mount option, relatime. The relatime mount option updates the last access time of a file only if the last modified or last inode changed time is newer than the last accessed time. This allows programs to be able to determine whether a file has been read size it was last modified. The usual (actually, only) example that is given of such an application is the mutt mail-reader, which uses the last accessed time to determine if new mail has been delivered to Unix mail spool files. Unfortunately, relatime is not free. As you can see below, it has roughly double the overhead of noatime (but roughly half the overhead of using the standard Posix atime semantics):
Amount of data written (in megabytes) on an ext4 filesystem mounted with relatime
Operation with journal w/o journal percent change
git clone 366.6 353.0 3.71%
make 216.8 203.7 6.04%
make clean 13.34 6.97 45.75%
Personally, I don t think relatime is worth it. There are other ways of working around the issue with mutt for example, you can use Maildir-style mailboxes, or you can use mutt s check_mbox_size option. If the goal is to reduce unnecessary disk writes, I would mount my file systems using noatime, and use other workarounds as necessary. Alternatively, you can use chattr +A to set the noatime flag on all files and directories where you don t want noatime semantics, and then clear the flag for the Unix mbox files where you care about the atime updates. Since the noatime flag is inherited by default, you can get this behaviour by setting running chattr +A /mntpt right after the filesystem is first created and mounted; all files and directories created in that file system will have the noatime file inherited. Comparing ext3 and ext2 filesystems
Amount of data written (in megabytes) on an ext3 and ext2 filesystem
Operation ext3 ext2 percent change
git clone 374.6 357.2 4.64%
make 230.9 204.4 11.48%
make clean 14.56 6.54 55.08%
Finally, just to round things out, I tried the same experiment using the ext3 and ext2 file systems. The difference between these results and the ones involving ext4 are the result of the fact that ext2 does not have the directory index feature (aka htree support), and both ext2 and ext3 do not have extents support, but rather use the less efficient indirect block scheme. The ext2 and ext3 allocators are also someone different from each other, and from ext4. Still, the results are substantially similar with the first set of Posix-compliant atime update numbers (I didn t bother to do noatime and relatime benchmark runs with ext2 and ext3, but I expect the results would be similar.) Conclusion So given all of this, where did the common folk wisdom that ext3 was not suitable for SSD s come from? Some of it may have been from people worrying too much about extreme workloads such as make clean ; but while doubling the write load sounds bad, going from 4MB to 7MB worth of writes isn t that much compared to the write load of actually doing the kernel compile or populating the kernel source tree. No, the problem was that first generation SSD s had a very bad problem with what has been called the write amplification effect , where a 4k write might cause a 128k region of the SSD to be erased and rewritten. In addition in order to provide safety against system crashes, ext3 has more synchronous write operations that is where ext3 waits for the write operation to be complete before moving on, and this caused a very pronounced and noticeable stuttering effect which was fairly annoying to users. However, the next generation of SSD s, such as Intel s X25-M SSD, have worked around the write amplification affect. What else have we learned? First of all, for normal workloads that include data writes, the overhead from journaling is actually relatively small (between 4 and 12%, depending on the workload). Further, than much of this overhead can be reduced by enabling the noatime option, with relatime providing some benefit, but ultimately if the goal is to reduce your file system s write load, especially where an SSD is involved, I would strongly recommend the use of noatime over relatime. Related posts (automatically generated):
  1. Should Filesystems Be Optimized for SSD s? In one of the comments to my last blog entry,...
  2. Aligning filesystems to an SSD s erase block size I recently purchased a new toy, an Intel X25-M SSD,...
  3. Fast ext4 fsck times This wasn t one of the things we were explicitly engineering...

1 March 2009

Julian Andres Klode: debimg update - creating images


Much happened since the last time I wrote about debimg. The project is now registered on Alioth and has a mailing list. On the code side, there have also been several changes. First of all, the repository module has been merged into the master branch. This was the first step towards the creation of the image building, which happened today by introducing the image module. The code should be treated as Beta quality, but the project as a whole is Alpha, because the application utilizing debimg.core is still missing. As always, I hereby encourage to try out debimg, have a look at the examples, and help to develop it. Patches should be sent to the mailing list, created with git format-patch, and its usual settings (eg. prefixed with eg. [PATCH 1/2] ). The patches should be inline and validate in pyflakes and using pep8.py. See the README for some more recommendations. Because NEW seems to be really busy at the moment (and doko uploaded python2.6 and python3.1), I m not uploading the current state as 0.1.0~a1 to experimental, but wait until NEW is smaller (maybe it will be 0.1.0~b1 then). What is wanted/planned?
  • Compatibility configuration formats
    • (debimg.config.simple_cdd) A reimplementation of simple-cdd using debimg. This probably needs to wait until there is a module for dealing with the installer, but I want to implement an application to support simple-cdd configuration files.
    • (debimg.config.debian_cd) There could be an implementation of debian-cd s configuration format. This would allow people to easily try out debimg. We can only support a subset, though.
  • New configuration formats
    • (debimg.config.yaml) Debimg s native configuration format. This is the only one supported by the graphical frontends. Where possible, configuration between compatibility formats and this format will be provided.
  • Graphical Frontends:
    • (debimg.frontend.gtk2) The graphical GTK+ frontend for inexperienced users who just want to create their own disk.
    • (debimg.frontend.qt4) The graphical frontend written in QT4.
  • Text frontends
    • (debimg.frontend.text) The basic command-line frontend.
    • (debimg.ubuntu.frontend.text) A script to build Ubuntu images, which could be used to build the official Ubuntu Images. This requires interaction with the germinate tool for dependency resolution, and more. It s also not flexible enough for building custom images.
Requirements for 0.1 Beta
  • Implement at least one configuration format and the build frontend.
  • Have at least one external contributor.
More Changes Julian Andres Klode (6):
  • debimg/core/files: When creating a file object allow filename as source
  • debimg/core/resolver.py: Introduce Package.fullname, Package.component
  • debimg/core/repository.py: Merge the repository module.
  • debimg/core/resolver.py: Improve handling of certain dependency types
  • debimg/core/repository.py: Allow Repository.add_group to take a distro parameter
  • debimg/core/image.py: Introduce the image module.
Posted in Debian, debimg, Ubuntu

Gustavo Noronha Silva: WebKitGTK+ 1.1.1 released

A new version of WebKit s GTK+ port is here, as also announced by Xan! Before I start listing the new features and fixes, let me give you a download link =). For this version we have changed the binary interface slightly, which requires a soversion bump, so rebuilding applications is in order. The API is still stable, though. Another major change worthy of note is that we have made libsoup the only HTTP backend available for WebKitGTK+, removing the CURL related code from our port (the CURL backend remains in the tree, though, as it is used by other ports). This should simplify implementation and allows us to expose Soup objects in the API, reducing the amount of wrapping code that is necessary. We are already exposing the SoupSession we use internally, providing yet more power and flexibility to applications using our API. In this release we finally get to see the official debut of the Web Inspector in the GTK+ port. Can t wait to see releases of our dear browsers featuring the inspector =). Numerous improvements to hading policy decisions regarding navigations and window creation requests to the application have also come with this release. We worked hard on finding and fixing crashes for this release, but also numerous fixes for those pet peeves that bother us just a bit, but enough to add up got landed. There are more of those to kill, but things look promising. We are hoping to release again in two weeks, so expect more fixes soon. So, that s it. Grab the new version, test it, report bugs, and enjoy!

Frans Pop: The case of the self-perpetuating DNS errors

Ingredients: The last couple of days I've been plagued by some DNS errors that kept showing up in the logcheck mails for my home server which I was busy migrating from one box to another, doing an upgrade from etch/i386 to lenny/amd64 at the same time. So, plenty of stuff going on to confuse the issue. I kept getting the following messages every hour (anonymized):
named: connection refused resolving 'somedomain.org/NS/IN': xxx.yyy.zzz.nnn#53
named: connection refused resolving 'somedomain.org/NS/IN': xxx.yyy.zzz.mmm#53
named: connection refused resolving 'ns1.somedomain.org/AAAA/IN': xxx.yyy.zzz.mmm#53
named: connection refused resolving 'ns2.somedomain.org/AAAA/IN': xxx.yyy.zzz.mmm#53
named: connection refused resolving 'ns1.somedomain.org/AAAA/IN': xxx.yyy.zzz.nnn#53
named: connection refused resolving 'ns2.somedomain.org/AAAA/IN': xxx.yyy.zzz.nnn#53
The times were fairly regular: once just before the hour, most 2 minutes after. I fetch mail at around that time, but also at other times, so possible but unlikely. The 2 minutes after was the first real clue: some cron job maybe? After disabling logcheck the messages no longer appeared in the log. Enable it again, and they were back. Additional confusion was caused by the fact that the domain had "debian" in its name, but it was somewhere obscure. So why was logcheck causing a lookup for that domain? This did confuse me enough to waste some time looking for some silly weird (default) configuration problem in some package. Enter spamassassin. Apparently that was parsing the message body, recognized "somedomain.org" as a host name, and proceded to do a DNS lookup as validity check. So we have the following loop, started off by something causing an initial DNS lookup for the domain, which fails and gets logged: Duh. I remember struggling with probably the same problem a couple of years ago, but then it was a lot more severe: masses of repeating DNS errors for obscure domains. At that time I failed to get to the bottom of it and ended up just ignoring the errors by adding the following option in my bind9 configuration:
logging  
    category lame-servers   null;  ;
 ;
Anyway, now I just no longer pass logcheck mails through spamassassin. (Although filtering out these DNS errors in bind9 can be perfectly valid.)

Frans Pop: The case of the self-perpetuating DNS-errors

Ingredients: The last couple of days I've been plagued by some DNS errors that kept showing up in the logcheck mails for my home server which I was busy migrating from one box to another, doing an upgrade from etch/i386 to lenny/amd64 at the same time. So, plenty of stuff going on to confuse the issue. I kept getting the following messages every hour (anonymized):
named: connection refused resolving 'somedomain.org/NS/IN': xxx.yyy.zzz.nnn#53
named: connection refused resolving 'somedomain.org/NS/IN': xxx.yyy.zzz.mmm#53
named: connection refused resolving 'ns1.somedomain.org/AAAA/IN': xxx.yyy.zzz.mmm#53
named: connection refused resolving 'ns2.somedomain.org/AAAA/IN': xxx.yyy.zzz.mmm#53
named: connection refused resolving 'ns1.somedomain.org/AAAA/IN': xxx.yyy.zzz.nnn#53
named: connection refused resolving 'ns2.somedomain.org/AAAA/IN': xxx.yyy.zzz.nnn#53
The times were fairly regular: once just before the hour, most 2 minutes after. I fetch mail at around that time, but also at other times, so possible but unlikely. The 2 minutes after was the first real clue: some cron job maybe? After disabling logcheck the messages no longer appeared in the log. Enable it again, and they were back. Additional confusion was caused by the fact that the domain had "debian" in its name, but it was somewhere obscure. So why was logcheck causing a lookup for that domain? This did confuse me enough to waste some time looking for some silly weird (default) configuration problem in some package. Enter spamassassin. Apparently that was parsing the message body, recognized "somedomain.org" as a host name, and proceded to do a DNS lookup as validity check. So we have the following loop, started off by something causing an initial DNS lookup for the domain, which fails and gets logged: Duh. I remember struggling with probably the same problem a couple of years ago, but then it was a lot more severe: masses of repeating DNS errors for obscure domains. At that time I failed to get to the bottom of it and ended up just ignoring the errors by adding the following option in my bind9 configuration:
logging  
    category lame-servers   null;  ;
 ;
Anyway, now I just no longer pass logcheck mails through spamassassin. (Although filtering out these DNS errors in bind9 can be perfectly valid.)

27 February 2009

MJ Ray: New #ukgovOSS Action Plan

Earlier this week, Tom Watson, Cabinet Office Parliamentary Secretary, published the Open Source, Open Standards and Re Use: Government Action Plan. It s had a pretty mixed reaction, with mild scepticism (niq s soapbox: Is gov.uk going open-source?) being the average reaction from what I ve read. I particularly liked the kind offer by Bristol Wireless to debianise Tom Watson s laptop. I think commentators are bang-on that the procurement process needs to change and that this sounds positive. I ll love it if I m wrong, but this looks like the lip service as usual which I ve seen in the last 10 years working on FOSS in the UK. I want to see the action that comes from this plan! When government actually starts buying FOSS from typical FOSS service providers and not just the IBMs of this world, then I ll believe it. My suspicion is re-ignited by some of the activites around this action plan. For example, does anyone know why the Cabinet Office didn t select FOSS to run their special public FOSS Aggregation page? (Actually, what s the best FOSS tag aggregator web service out there? I know we can set them up in Wordpress widgets, but what hosted services are there?) I m also a bit bemused that their page requires users to accept cookies until they expire, yet its privacy policy explicitly says You may suppress cookies after your visit or configure your Internet browser to prevent them. Yes, I can prevent them, but then it does nothing useful! (Based on a comment I made at the OSS Watch team blog and discussions with a few user groups.)

Neil Williams: Goodbye network-manager, hello wicd

I've not had that many problems with network-manager, until now. The new version in unstable is just completely broken on my machine. Sadly, I can't report why it failed as I inadvertently closed the terminal with the relevant output. :-(

Suffice to say, nm repeatedly got the wrong IP address, an invalid route and failed to set a usable nameserver. I've had those problems before with wireless but when this started happening with a wired connection (which had always previously worked), it was time to move on.

I've noticed people using wicd instead and decided I didn't have time to debug nm, I have a worksession starting in Cambridge on Sunday and a shed load of work to do before that, all of which requires a stable network connection.

Installing wicd replaced network-manager which was nice, (nm has always had problems working with any other network config so this allowed me to skip the removal of nm myself). wicd initially had a few issues - which I managed to pin down to the remnants of the broken nm config and steps that I'd taken to try to fix it.

Turns out that I may have some unresolved issues with my router - even with wicd I had problems getting a route over wireless. However, wicd allows suitable configuration that both works and is easy to keep discrete from other configs. Nice work. I have to set a static IP over wireless in order to set a valid route (and netmask). Looks like the DHCP implementation on the router is borked.

I'm sticking with wicd though - the interface is better for what I need, the icon has a useful traffic indication as well as signal strength and it's worth pursuing to see if the previous intermittent nm errors affect wicd.

26 February 2009

Theodore Ts'o: Fast ext4 fsck times, revisited

Last night I managed to finish up a rather satisfying improvement to ext4 s inode and block allocators. The ext4 s original allocator was actually a bit more simple-minded than ext3 s, in that it didn t implement the Orlov algorithm to spread out top-level directories for better filesystem aging. It also was buggy in certain ways, where it would return ENOSPC even when there were still plenty of inodes in the file system. So I had been working on extending ext3 s original Orlov allocator so it would work well with ext4. While I was at it, it occurred to me that one of the tricks I could play with ext4 s flex groups (which are higher-order collection of block groups), was to bias the block allocation algorithms such that the first block group in a flexgroup would be preferred for use by directories, and biased against data blocks for regular files. This meant that directory blocks would get clustered together, which cut a third off the time needed for e2fsck pass2:
Comparison of e2fsck times on an 32GB partition
Pass ext4 old allocator ext4 new allocator
time (s) I/O time (s) I/O
real user system MB read MB/s real user system MB read MB/s
1 6.69 4.06 0.90 82 12.25 6.70 3.63 1.58 82 12.23
2 13.34 2.30 3.78 133 9.97 4.24 1.27 2.46 133 31.36
3 0.02 0.01 0 1 63.85 0.01 0.01 0.01 1 82.69
4 0.28 0.27 0 0 0 0.23 0.22 0 0 0
5 2.60 2.31 0.03 1 0.38 2.42 2.15 0.07 1 0.41
Total 23.06 9.03 4.74 216 9.37 13.78 7.33 4.19 216 15.68

As you may recall from my previous observations on this blog, although we hadn t been explicitly engineering for this, a file system consistency check on an ext4 file system tends to be a factor of 6-8 faster than the e2fsck times on an equivalent ext3 file system, mainly due to the elimination of indirect blocks and the uninit_bg feature reducing the amount of disk reads necessary in e2fsck s pass 1. However, the ext4 layout optimizations didn t do much for e2fsck s pass 2. Well, the optimization of the block and inode allocators is complementary to the original ext4 fsck improvements, since it focuses on what we hadn t optimized the first time around: e2fsck pass 2 times have been cut by a third, and the overall fsck time has been cut by 40%. Not too shabby! Of course, we need to do more testing to make sure we haven t caused other file system benchmarks to degrade, although I m cautiously optimistic that this will end up being a net win. I suspect that some benchmarks will go up by a little, and others will go down a little, depending on how heavily the benchmark exercises directory operations versus sequential I/O patterns. If people want to test this new allocator, it is in the ext4 patch queue. If all goes well, I will hopefully be pushing it to Linus after 2.6.29 is released, at the next merge window.

horizontal separator For comparison s sake, here is a comparison of the fsck time of the same collection of files and directories, comparing ext3 and the original ext4 block and inode allocator. The file system in question is a 32GB install of Ubuntu Jaunty, with a personal home directory, a rather large Maildir directory, some linux kernel trees, and an e2fsprogs tree. It s basically the emergency environment I set up on my Netbook at FOSDEM. In all cases the file systems were freshly copied from the original root directory using the command rsync -axH / /mnt. It s actually a bit surprising to me that ext3 s pass 2 e2fsck times was that much better than e2fsck time under the old ext4 allocator. My previous experience has shown that the two are normally about the same, with a write throughput of around 9-10 MB/s on for e2fsck s pass 2 for both ext3 file systems and ext4 file systems with the original inode/block allocators. Hence, I would have expected ext3 s pass2 time to have been 12-13 seconds, and not 6. I m not sure how that happened, unless it was the luck of draw in terms of how things ended up getting allocated on disk. So I m not too sure what happened there, but overall things look quite good for ext4 and fsck times!
Comparison of e2fsck times on an 32GB partition
Pass ext3 ext4 old allocator
time (s) I/O time (s) I/O
real user system MB read MB/s real user system MB read MB/s
1 108.40 13.74 11.53 583 5.38 6.69 4.06 0.90 82 12.25
2 5.91 1.74 2.56 133 22.51 13.34 2.30 3.78 133 9.97
3 0.03 0.01 0 1 31.21 0.02 0.01 0 1 63.85
4 0.28 0.27 0 0 0 0.28 0.27 0 0 0
5 3.17 0.92 0.13 2 0.63 2.60 2.31 0.03 1 0.38
Total 118.15 16.75 14.25 718 6.08 23.06 9.03 4.74 216 9.37
Vital Statistics of the 32GB partition
312214 inodes used (14.89%)
263 non-contiguous files (0.1%)
198 non-contiguous directories (0.1%)
# of inodes with ind/dind/tind blocks: 0/0/0
Extent depth histogram: 292698/40
4388697 blocks used (52.32%)
0 bad blocks
1 large file
263549 regular files
28022 directories
5 character device files
1 block device file
5 fifos
615 links
20618 symbolic links (19450 fast symbolic links)
5 sockets
312820 files
Related posts (automatically generated):

  1. Fast ext4 fsck times This wasn t one of the things we were explicitly engineering...
  2. Wanted: Incremental Backup Solutions that Use a Database Dear Lazyweb, I m looking for recommendations for Open Source backup...
  3. Ext4 is now the primary filesystem on my laptop Over the weekend, I converted my laptop to use the...

Theodore Ts'o: Fast ext4 fsck times, revisited

Last night I managed to finish up a rather satisfying improvement to ext4 s inode and block allocators. The ext4 s original allocator was actually a bit more simple-minded than ext3 s, in that it didn t implement the Orlov algorithm to spread out top-level directories for better filesystem aging. It also was buggy in certain ways, where it would return ENOSPC even when there were still plenty of inodes in the file system. So I had been working on extending ext3 s original Orlov allocator so it would work well with ext4. While I was at it, it occurred to me that one of the tricks I could play with ext4 s flex groups (which are higher-order collection of block groups), was to bias the block allocation algorithms such that the first block group in a flexgroup would be preferred for use by directories, and biased against data blocks for regular files. This meant that directory blocks would get clustered together, which cut a third off the time needed for e2fsck pass2:
Comparison of e2fsck times on an 32GB partition
Pass ext4 old allocator ext4 new allocator
time (s) I/O time (s) I/O
real user system MB read MB/s real user system MB read MB/s
1 6.69 4.06 0.90 82 12.25 6.70 3.63 1.58 82 12.23
2 13.34 2.30 3.78 133 9.97 4.24 1.27 2.46 133 31.36
3 0.02 0.01 0 1 63.85 0.01 0.01 0.01 1 82.69
4 0.28 0.27 0 0 0 0.23 0.22 0 0 0
5 2.60 2.31 0.03 1 0.38 2.42 2.15 0.07 1 0.41
Total 23.06 9.03 4.74 216 9.37 13.78 7.33 4.19 216 15.68

As you may recall from my previous observations on this blog, although we hadn t been explicitly engineering for this, a file system consistency check on an ext4 file system tends to be a factor of 6-8 faster than the e2fsck times on an equivalent ext3 file system, mainly due to the elimination of indirect blocks and the uninit_bg feature reducing the amount of disk reads necessary in e2fsck s pass 1. However, the ext4 layout optimizations didn t do much for e2fsck s pass 2. Well, the optimization of the block and inode allocators is complementary to the original ext4 fsck improvements, since it focuses on what we hadn t optimized the first time around: e2fsck pass 2 times have been cut by a third, and the overall fsck time has been cut by 40%. Not too shabby! Of course, we need to do more testing to make sure we haven t caused other file system benchmarks to degrade, although I m cautiously optimistic that this will end up being a net win. I suspect that some benchmarks will go up by a little, and others will go down a little, depending on how heavily the benchmark exercises directory operations versus sequential I/O patterns. If people want to test this new allocator, it is in the ext4 patch queue. If all goes well, I will hopefully be pushing it to Linus after 2.6.29 is released, at the next merge window.

horizontal separator For comparison s sake, here is a comparison of the fsck time of the same collection of files and directories, comparing ext3 and the original ext4 block and inode allocator. The file system in question is a 32GB install of Ubuntu Jaunty, with a personal home directory, a rather large Maildir directory, some linux kernel trees, and an e2fsprogs tree. It s basically the emergency environment I set up on my Netbook at FOSDEM. In all cases the file systems were freshly copied from the original root directory using the command rsync -axH / /mnt. It s actually a bit surprising to me that ext3 s pass 2 e2fsck times was that much better than e2fsck time under the old ext4 allocator. My previous experience has shown that the two are normally about the same, with a write throughput of around 9-10 MB/s on for e2fsck s pass 2 for both ext3 file systems and ext4 file systems with the original inode/block allocators. Hence, I would have expected ext3 s pass2 time to have been 12-13 seconds, and not 6. I m not sure how that happened, unless it was the luck of draw in terms of how things ended up getting allocated on disk. So I m not too sure what happened there, but overall things look quite good for ext4 and fsck times!
Comparison of e2fsck times on an 32GB partition
Pass ext3 ext4 old allocator
time (s) I/O time (s) I/O
real user system MB read MB/s real user system MB read MB/s
1 108.40 13.74 11.53 583 5.38 6.69 4.06 0.90 82 12.25
2 5.91 1.74 2.56 133 22.51 13.34 2.30 3.78 133 9.97
3 0.03 0.01 0 1 31.21 0.02 0.01 0 1 63.85
4 0.28 0.27 0 0 0 0.28 0.27 0 0 0
5 3.17 0.92 0.13 2 0.63 2.60 2.31 0.03 1 0.38
Total 118.15 16.75 14.25 718 6.08 23.06 9.03 4.74 216 9.37
Vital Statistics of the 32GB partition
312214 inodes used (14.89%)
263 non-contiguous files (0.1%)
198 non-contiguous directories (0.1%)
# of inodes with ind/dind/tind blocks: 0/0/0
Extent depth histogram: 292698/40
4388697 blocks used (52.32%)
0 bad blocks
1 large file
263549 regular files
28022 directories
5 character device files
1 block device file
5 fifos
615 links
20618 symbolic links (19450 fast symbolic links)
5 sockets
312820 files
Related posts (automatically generated):

  1. Fast ext4 fsck times This wasn t one of the things we were explicitly engineering...
  2. Wanted: Incremental Backup Solutions that Use a Database Dear Lazyweb, I m looking for recommendations for Open Source backup...
  3. Ext4 is now the primary filesystem on my laptop Over the weekend, I converted my laptop to use the...

Ian Wienand: rdtsc - now even less useful!

An interesting extract from the latest IA32 SDM (18.20.5)
The TSC, IA32_MPERF, and IA32_FIXED_CTR2 operate at the same, maximum-resolved frequency of the platform, which is equal to the product of scalable bus frequency and maximum resolved bus ratio.
For processors based on Intel Core microarchitecture, the scalable bus frequency is encoded in the bit field MSR_FSB_FREQ[2:0] at (0CDH), see Appendix B, "Model-Specific Registers (MSRs)". The maximum resolved bus ratio can be read from the following bit field:
  • If XE operation is disabled, the maximum resolved bus ratio can be read in MSR_PLATFORM_ID[12:8]. It corresponds to the maximum qualified frequency.
  • IF XE operation is enabled, the maximum resolved bus ratio is given in MSR_PERF_STAT[44:40], it corresponds to the maximum XE operation frequency configured by BIOS.
In summary, TSC increment = (scalable bus frequency) * (maximum resolved bus ratio). This implies the TSC is incrementing based on some external bus source (any hardware engineers explain what happened for Core here?), and is a departure from simply assuming that the TSC increments once for each CPU cycle. The interesting bit is that if XE operation is disabled, the bus ratio is assumed to be the maximum qualified frequency. This seems to mean that if you overclock your CPU and your processor is running at higher than the qualified frequency, attempts to measure the CPU speed by counting TSC ticks over a given time may yeild the wrong results (well, will yield the rated result; i.e. the speed of the processor you bought out of the box). While interesting, this divergence is probably has little practical implications because using the TSC for benchmarking is already fraught with danger. You have to be super careful to make sure the compiler and processor don't reschedule things around you and handle other architectural nuances. If you need this level of information, you're much better using the right tools to get it (my favourite is perfmon2).

25 February 2009

Russ Allbery: Expectation and obligation

Some necessary background: George R.R. Martin is a science fiction and fantasy author who's been writing (and editing) in the field for quite a while. In 1996, he started a new epic fantasy series entitled The Song of Ice and Fire, a hugely complex and sprawling series telling the story of an entire medieval world, with four volumes published and something on the order of 19 viewpoint characters to date. The fourth and last book to date in the series was published in 2005. It contained only half the viewpoints expected (and in my opinion and that of at least some other readers, the least interesting half). There was an afterword saying basically that the book was cut somewhat artificially in half for length reasons (it was 753 hardcover pages for what was there), the next book would have the other viewpoints, and it was basically done. Three years and some later, and after a few other optimistic updates, it's not yet done. Some of Martin's fans are increasingly getting antsy and sometimes abusive about this. This has increasingly been getting to Martin, and he's responded in various ways in his personal blog, most recently February 19th. See also John Scalzi and Charlie Stross reacting to his post. Now, I'm not one of the people who is rabid about this series. I'm a little disappointed that I haven't gotten to read the next bits with the characters I really like in the series, and I didn't much care for A Feast for Crows because of how the book was split. But I have a thousand other books I want to read (literally I've read less than half the books I own) and if I don't get to see more of the series, oh well. I am, however, very interested in the fan reaction to this delay and Martin's reaction to that, because it has started reminding me of similar things in my own life. Some of the fans are just batshit insane, which is one of the things that happens to a best-selling author; I feel for Martin there but there isn't much anyone can really do about that. But among the sane people, there are a bunch who say something like "I understand why the constant nagging doesn't help, but on the other hand, when you start a big story with unresolved plots, you have an obligation to your readers to finish it." I call bullshit on that. This same effect happens in the free software world. If you write some program, you're perceived to have an obligation to support it, fix it, release new versions, and so forth. I've also had this happen in other volunteer contexts, such as Usenet. Often that obligation is based on some comments that one makes about future plans, things that will go into the next release, e-mail messages where you say that you'll look at something, and so forth. It's not entirely groundless, and it's not groundless with Martin either. However. As any software developer knows, time estimates for tasks are horribly difficult and almost always too short. This is clearly what happened to Martin, and bitching about the time estimates doesn't help at all. Some problems turn out to be so much harder than you expect that time estimates fail entirely. With creative work (including some programming) it's particularly hard to know how much work is remaining. You may know roughly how much you have left to write, but you don't know how hard each chunk will be until you start it, and sometimes until you finish it. With some projects, the only thing you can do is say "sorry, my estimates were completely off and I don't know how to estimate this" and then stop promising dates. Which Martin has already done. He was "wrong" in some sense for setting bad expectations, but estimation is insanely hard and the world usually doesn't let you get away with just refusing to estimate anything. So much for time. What about whether Martin has an obligation to write the next book at all? The equivalent question in the free software world is whether one can just walk away from a project, or away from a volunteer position, leaving a bunch of stuff undone that some people who were using the software were counting on having finished. It's a bit easier to deal with in free software, since usually just promises are at stake, whereas with Martin he's sold a commercial product and people get weird about money. There is a reasonable expectation that a series of this type will have a conclusion (although I'll point out that the SFF world is filled with unfinished series). Maybe you wouldn't have bought any of the series if you had known that it wouldn't end, so now you're out $100 for the hardcovers that you feel like you bought on false pretenses. Let's examine that. First, I have a serious problem with people who use the economic transaction as leverage in this sort of argument. This is a really piddly amount of money for each individual person, generally less than I get paid for one hour of work. Writing a new novel is way, way more than one hour of work. Martin's fans are demanding that he do work that's worth many orders of magnitude more than they're conceivably out monetarily. (And the reactions are individual reactions of anger; the level of upsetness of each individual shouldn't change based on the total size of the fan base.) Furthermore, their economic transaction was with the publisher, and they had no control and no say over how Martin got paid. From multiple different directions, they're not his employer, have no contractual control over how he spends his time, and have no legitimate economic basis for asserting any such control. They can make a decision to not buy his future work because they're disappointed, and that's it. Second, it is absolutely mandatory to give other people room to be wrong or change their mind. Human beings cannot keep every promise that they make. We fail at things that we fully intend to do, particularly around estimates, and particularly around one's dedication to decade-long major life projects. People are wrong about marriages and children, which involve far more serious consequences than a fiction series. I think it's a good reality check here to note that his fans are placing way more restrictions on what he can do with his life than would be legal for any employer. Third, when someone doesn't meet your expectations, the response should be proportional and rational. This is the one that really gets to me. Choosing not to read the next book in the series because it's been too long is entirely reasonable. Trying to make Martin feel like crap is not. I would go further: nothing that an artist could promise to their general public could warrant intentionally making them feel like crap for failing to do it. That level of emotional consequence should be reserved for things like marriage, parenting, or life-long friendship, not disappointment that you didn't get to hear the end of a story you were enjoying. I speak from personal experience here: this type of punishment by guilt is a nasty, nasty business. It burns people out faster than any other possible reaction, it makes it much less likely that Martin will continue the series at all, and it's probably poisoning his feelings about the entire world. It's burning down the bridge you're standing on. It always hurts and never helps. People already create feelings of obligation around things they've promised, implicitly or explicitly, to do, and generate plenty of their own guilt. That guilt is one of the major obstacles that one has to overcome to be productive; if you don't believe me, read any good productivity book, such as Allen's Getting Things Done, and notice how much it talks about getting past guilt. Intentionally going out of your way to contact someone you do not know personally and pile on more guilt is vicious, cruel, and tactically counterproductive. Trust me, Martin didn't fail to notice that he missed his own deadlines. I think Martin's post and the subsequent discussion is a great opportunity for all of us who create or use creative and technical works to think about how we apply the Golden Rule. There's a natural tendency to hold people tightly to promises that we'll get something that we want. There's a natural tendency for all of us to make those promises, since people are grateful for them, and then pile guilt on ourselves for not meeting them. All of the emotion wrapped up in both sides of that tension of expectation and obligation can be exhausting and poisonous. It makes relationships brittle because we don't let people renegotiate. We don't let people say they were wrong. We don't let people change their minds. And we don't let ourselves change our minds. Here's my promise: the next time someone doesn't meet an expectation I had, I will remember this discussion, give them space to be wrong, and not pile on more guilt. Like all promises, I'll try to keep it, but I might not be able to.

24 February 2009

Russell Coker: Tragedy and Profit

Every time something goes wrong there will be someone who tries to take advantage of the situation. The recent bushfires in Australia that have killed hundreds of people (the count is not known yet) are a good example. Pastor Nalliah of Catch the Fire Ministries [1] claims that it is due to legalising abortion. This is astoundingly wrong. In a more extreme example representatives of the Westboro Baptist Church were planning to visit Australia to launch a protest in support of the bushfires [2]. I have not yet found any news reports about whether they actually visited Australia or protested - it s most likely that they decided not to visit due to the Australian laws being very different to US laws regarding the relative importance of freedom of speech and incitement to violence. Apparently the insane Westboro Baptist Church people (who are best known for GodHatesFags.com and GodHatesAmerica.com) believe that God hates Australia and caused the fires (presumably due to Australia not persecuting homosexuals). Danny Nalliah has permanently damaged his own reputation by acting in a similar way to the Westboro Baptist Church. The reputation of Catch The Fire now depends on how quickly they get a new pastor Please note well that the vast majority of Christians have nothing in common with Westboro or Catch The Fire. I don t recall the last time I met an Australian Christian who was strongly opposed to homosexuality or abortion. Now we do have to try and investigate ways of avoiding future tragedies, and the work to do this needs to begin immediately. John Brumby (the Premier of Victoria) has announced that Victoria will get new strict building codes for fire resistant buildings [3]. There have been many anecdotes of people who claim to have been saved by attaching sprinkler systems to their homes, by building concrete bunkers to hide in while the fire passes, and using other techniques to save their home or save themselves. Some more research on the most effective ways of achieving such goals would be worthwhile, an increase in funding for the CSIRO to investigate the related issues would be a good thing. The article also has an interesting quote As the fallout from the disaster widened, the union representing the nation s 13,000 firefighters warned both the federal and state governments to take global warming seriously to prevent a repeat of last weekend s lethal firestorm . However given that traditionally Australia and the US have been the two nations most opposed to any efforts to mitigate global warming it seems unlikely that anything will change in this regard in a hurry. The attempts to link bushfires to abortion and homosexuality are offensive, but can be ignored in any remotely serious debate about politics. However there are some other groups trying to profit from the tragedy that make claims which are not as ridiculous. On the 9th of February the Australian Green party was compelled to release an official statement from Spokesperson Scott Ludlam, Sarah Hanson-Young, Rachel Siewert, Christine Milne, and Bob Brown following some political discussion about Greens policies [4]. There have been attempts to blame the Greens for the tragedy which were politically motivated, some of which came from groups that traditionally oppose the Greens for other reasons (I m not going to provide the detail - anyone who is really interested can do google searches on the people in question). On the 16th of February Bob Brown (the leader of the Green party) felt obliged to make another media release reiterating the fact that the Greens support prescribed burn-offs to limit the scope of wild fires [5], he also decried the hate mongering that has been occurring in the wake of the disaster. One of the strange memes that seems to be spread by opponents to the Greens is that the Greens are all supposedly from the city and know nothing about the country. To avoid being subject to such attack I feel obliged to note that on one of the bad fire days I visited my parents. I spent the morning with my father and some friends at a park that was not far from the fire area, my friends then returned to their home which was not far from the fire area. I then had lunch with my parents and watched the smoke through the dining room window. After that my friends didn t respond to email for a while and I was concerned that they may have lost their house or maybe suffered injury or death. I didn t know them well enough to feel it appropriate to try a dozen different ways of contacting them (I m sure that many other people were doing so), but I was rather concerned until my wife received an email from them. But I don t base my political beliefs on what I personally observe or my connections to people on the edge of the fire zone. I believe in the Green principles of Peace and Non Violence, Grassroots Democracy, Social and Economic Justice, Ecological Sustainability and the use of science and statistics to determine the best ways of achieving those goals.

Adrian von Bidder: New Toy

Last week, I couldn't resist and bought myself an Acer Aspire One (AOA 150Ab) netbook. It has 1G RAM, 120G HDD, unfortunately needs a fan, and is the model without 3G modem. It comes with Linpus Linux pre-installed. Looks quite nice, but is obviously ultimately the wrong OS ... Besides, it's locked down quite a bit, there's not even an easy way to start a terminal :-) I still have kept it, in a dual-boot configuration, to play around or show to people. Installing Lenny went very well, and to make things more fun I'm running quite a few things from experimental: KDE 4.2, xorg 7.4 (1:7.4~5 right now) and Oo.org 3. And, to get DRI2, also the 2.6.28 kernel from the newer-than-sid repository of the kernel team (2.6.28-2~snapshot.12850, but I hear 2.6.28 has now been uploaded.) While the whole thing is fun to use and didn't make any real problems, there are a few remaining issues (yes, this is a Dear Lazyweb posting, feel free to comment. I'll try to add updates to this article as this progresses): Ok, this has become rather a long list. But at least, as you can see, it's mostly minor issues, and hopefully a few where it's just missing configuration. One other issue is battery life: I see that there should be a large battery available for this thing. If it doesn't cost me as much as the whole netbook again, I'll seriously have to think about this... The 3-cell battery does last about 2.5h (and has, right now, uncovered a little bug where the battery monitoring applet tells me No AC adaptor plugged in, battery capacity: 50%, charging . Whee! I have a perpetuum mobile! I'm gonna be RICH!)

Matt Brown: The government listened!

I was very pleased to wake up this morning to the news that National has delayed the introduction of S92A via an order-in-council. It s a nice short-term victory, but I ll save the champagne until the law is fundamentally rewritten. The most pleasing aspect of the decision is simply that it was made at all. Within two weeks, a small band of protesters were able to harness the power of the Internet to direct international attention and place enough pressure on a Government, whose Prime Minister admitted to not having read the bill prior, that he then took the time to understand the issues and personally announce the delay in implementation of the law. We owe much thanks to the Creative Freedom Foundation for all the effort they put into co-ordinating the protest and ensuring that a single coherent message was presented. Just a little bit of my cynicism and belief that politicians never listen to public opinion outside of election campaigns was chipped away today. The reason I m not breaking out the champagne yet is that we ve only achieved a temporary reprieve in the commencement of the law. While those present at the press conference seem somewhat confident that John Key didn t like what he found in the law and would have repealed it if given the chance, all that has actually been done is delay it in the hopes of an agreement between the TCF and the rights holders (aka big media companies) on how to implement the still fundamentally broken law. The Government has given until late March for that to occur. To put this into a more global context. My happiness as I took the bus to work after reading about the decision to delay the law was short lived as the front page of the local paper declared that Eircom (Ireland s equivalent of Telecom) has voluntarily agreed to block sites such as The Pirate Bay upon request by the media companies (this comes a week after they also announced an agreement to, again voluntarily , implement a 3-strikes S92A style policy). Now, with the biggest ISP in their pocket (so to speak), the media companies have sent threatening letters to the remaining ISPs in the country demanding they implement the same procedure. To me, this illustrates one of the fundamental problems with S92. The concept that an ISP is liable for the conduct of its users, or for policing where on the Internet users should and shouldn t be able to connect to does not belong in our laws. Most ISPs already have provision to disconnect customers for illegal activity in their terms and conditions. If an end-user is doing something illegal, that is an issue between the rights holder and the end-user to take up in the courts just like every other sector of society must do when wronged, at which point the existing ISP terms and conditions can be invoked and access terminated. The big media companies, having decided that it is too expensive/hard/inconvenient to follow standard legal procedures to resolve their grievances are launching multi-pronged attacks to shift the playing field in their favour. In countries like New Zealand, where our politicians yearn for a Free Trade Agreement with America, they use their lobbyists to ensure that S92 style laws are part of the conditions. In other jurisdictions, like Ireland, they use strong-arm, divide and conquer style bully tactics outside of the political and legal process. I don t support copyright infringement. I rely on copyright to protect much of the work I place on the Internet, I want strong laws that protect me when my rights have been infringed. I don t believe that such laws should come at the expense of due process, our legal tradition and the basic principle of fairness! I don t believe that copyright infringement is such a heinous crime that it demands punishments stronger than those we deliver to paedophiles, stalkers or any other class of criminal who uses the Internet to enable their crimes. To me, today s (yesterday s - depending on your timezone) decision is only the first step in clawing New Zealand back from the dangerous path that the big media companies have been leading our law makers down. From here we need to press on and demonstrate to the Government over the next month that even if the TCF and rights holders are able to come up with some sort of workable code of practice, the law is still fundamentally flawed. It is based on premise that we are guilty by accusation. Even if guilt were to be proved by a competent legal body (eg. court or copyright tribunal) we don t need laws placing further liabilites onto ISPs (and remember the definition of ISP under this amendment act includes businesses who provide Internet access to staff, libraries, schools and hospitals) when their existing terms and conditions already prohibit illegal activity. Finally, and most importantly of all, we need to remember that laws exist to serve all sectors of society. Yes, copyright infringement is against the law and rights holders are reasonable in expecting the law to protect their content and allow them to make a fair profit. On the other side of the fence, average New Zealanders are not being unreasonable in their desire to have media available electronically, on demand and non-inhibited by DRM following a legal purchase. The failure of the media businesses to adequately cater to this change in market demand and usage of technology is obviously a contributing factor to the widespread copyright problems that they are facing today. Obviously, I m not condoning copyright infringement simply because the media companies are failing to address demand. Even stupid laws must be obeyed (and the concept of copyright is far from stupid). What I want to see is the Government acknowledging that the problem is not solely with consumers infringing copyright for malicious purposes, and therefore that the solutions do not lie solely in increasing the enforcement and punishments available. Copyright has always been a balancing act between the rights of content producers and consumers. S92 and the act it is contained within are taking us far too far down the road of catering to big business and their outdated business models with far too little concern for the rights of the individual consumer. Despite the many submissions made on this act last year when it was first passing through parliament, there was no comprehensive debate on what copyright means and how it should balance the rights of content producers and consumers in our digital century where copying is a zero-cost, zero-thought activity. Without such a debate we re doomed to continue wasting time arguing over the symptoms of the problem, like S92. So, I m saving my champagne for the day when we as a country address these issues and come up with a fair and workable interpretation of what copyright means today.

23 February 2009

Ingo Juergensmann: Performance Tweaking with Drupal

Isotopp just pointed me to the blog of Ronald Bradford, a MySQL consultant, writing about a Drupal installation and some performance observations.

He observes a large table "watchdog":

Presently, volume and content is of a low volume, but expecting to ramp up. I do however find 90% of disk volume in one table called watchdog ;


Well, yes, the watchdog table logs system messages. No big surprise so far, I guess. So, turning off database logging (Administer -> Site building -> Modules -> turn off Database logging module) or reduce the maximum number of entries in the database log at Administer -> Site Configuration -> Logging and alerts -> Database Logging will reduce the size of the watchdog table.

Robert continues with some "page not found" errors. Of course these have to be fixed somewhere, either by editing the content or fixing theme related bugs. Not really a Drupal issue, I guess.

Then he mentions repeated SQL queries:

Looking at just 1 random second of SQL logging shows 1200+ SELECT statements.
355 are SELECT changed FROM node
[...]
There is plenty of information regarding monitoring the Slow Queries in MySQL, but I have also promoted that s it not the slow queries that ultimately slow a system down, but the 1000 s of repeating fast queries.

MySQL of course has the Query Cache to assist, but this is a course grade solution, and a high volume read/write environment this is meaningless.

There is a clear need for either a application level caching, or a database redesign to pull rather then poll this information, however without more in depth review of Drupal I can not make any judgment calls.


Drupal offers some kind of application level caching at Administer -> Site configuration -> Performance, so turning it on might already help, although I'm not sure whether that minimizes the SQL queries or to what degree.

Anyway, there are some more Drupal performance tips here:

  1. http://drupal.org/node/97347

  2. http://wimleers.com/article/improving-drupals-page-loading-performance

  3. http://2bits.com/articles/drupal-performance-tuning-and-optimization-for-large-web-sites.html



Hope this helps... :-)

Theodore Ts'o: Reflections on a complaint from a frustrated git user

Last week, Scott James Remnant posted a series of Git Sucks on his blog, starting with this one here, with follow up entries here and here. His problem? To quote Scott, I want to put a branch I have somewhere so somebody else can get it. That s the whole point of distributed revision-control, collaboration. He thought this was a mind-numbingly trivial operation, and was frustrated when it wasn t a one-line command in git. Part of the problem here is that for most git workflows, most people don t actually use git push . That s why it s not covered in the git tutorial (this was a point of frustration for Scott). In fact, in most large projects, the number of people need to use the scm push command is a very small percentage of the developer population, just as very few developers have commit privileges and are allowed to use the svn commit command in a project using Subversion. When you have a centralized repository, only the privileged few will given commit privileges, for obvious security and quality control reasons. Ah, but in a distributed SCM world, things are more democratic anyone can have their own repository, and so everyone can type the commands git commit or bzr commit . While this is true, the number of people who need to be able to publish their own branch is small. After all, the overhead in setting up your own server just so people can pull changes from you is quite large; and if you are just getting started, and only need to submit one or two patches, or even a large series of patches, e-mail is a far more convenient route. This is especially true in the early days of git s development, before web sites such as git.or.cz, github, and gitorious made it much easier for people to publish their own git repository. Even for a large series of changes, tools such as git format-patch and git send-email are very convenient for sending a patch series, and on the receiving side, the maintainer can use git am to apply a patch series sent via e-mail. It turns out that from a maintainer s point of view, reviewing patches via e-mail is often much more convenient. Especially for developers who are just starting out with submitting patches to a project, it s rare that a patch is of sufficiently high quality that it can be applied directly into the repository without needing fixups of one kind or another. The patch might not have the right coding style compared to the surrounding code, or it might be fundamentally buggy because the patch submitter didn t understand the code completely. Indeed, more often than not, when someone submits a patch to me, it is more useful for indicating the location of the bug more than anything else, and I often have to completely rewrite the patch before it enters into the e2fsprogs mainline repository. Given that, publishing a patch that will require modification in a public repository where it is ready to be pulled just doesn t make sense for many entry-level patch submitters. E-mail is in fact less work, and more appropriate for review purposes. It is only when a mid-level to senior developer is trusted to create high quality patches that do not need review that publishing their branch in a pull-ready form really makes sense. And that is fairly rare, and why it is not covered in most entry-level git documentation and tutorials. Unfortunately, many people expect to see the command scm push in a distributed SCM, and since git pull is a commonly used command for beginning git users, they expect that they should use git push as well not realizing that in a distributed SCM, push and pull are not symmetric operations. Therefore, while most git users won t need to use git push , git tutorials and other web pages which are attempting to introduce git to new users probably do need to do a better job explaining why most beginning participants in a project probably don t need their own publically accessible repository that other people can pull from, and which they can push changes for publication. There is one exception to this, of course, and this is a developer who wants to get started using git for a new project which he or she is starting and is the author/maintainer, or someone who is interested in converting their project to git. And this is where bzr has an advantage over git, in that bzr is primarily funded by Canonical, which has a strong interest in pushing an on-line web service, Launchpad. This makes it easier for bzr to have relatively simple recipes for sharing a bzr repository, since the user doesn t need to have access to a server with a public IP address, or need to set up a web or bzr server; they can simply take advantage of Launchpad. Of course, there are web sites which make it easy for people to publish their git repositories; earlier, I had mentioned git.or.cz, github, and gitorious. Currently, the git documentation and tutorials don t mention them since they aren t formally affiliated with the git project (although they are used by many git users and developers and the maintainers of these sites have contributed a large amount of code and documentation to git). This should change, I think. Scott s frustrations which kicked off his git sucks complaints would have been solved if the Git tutorial recommended that the easist ways for someone to publicly publish their repository is via one of these public web sites (although people who want to set up their own server certainly free to do so). Most of these public repositories probably won t have much reason to exist, but they don t do much harm, and who knows? While most of the repositories published at github and gitoriuous will be like the hundreds of thousands of abandoned projects on Sourceforge, one or two of the new projects which someone starts experimenting on at github or gitorious could turn out to be the next Ruby on Rails or Python or Linux. And hopefully, they will allow more developers to be able to experiment with publishing commits on their own repositories, and lessen the frustrations of people like Scott who thought they needed their own repositories; whether or not a public repository is the best way for them to do what they need to do, at least this way they won t get as frustrated about git. :-) Related posts (automatically generated):
  1. Git and hg John Goerzen recently posted about Git, Mercurial and Bzr that...
  2. Batches of patched batches of patches I found the following from the Risks Digest, authored by...

Theodore Ts'o: Reflections on a complaint from a frustrated git user

Last week, Scott James Remnant posted a series of Git Sucks on his blog, starting with this one here, with follow up entries here and here. His problem? To quote Scott, I want to put a branch I have somewhere so somebody else can get it. That s the whole point of distributed revision-control, collaboration. He thought this was a mind-numbingly trivial operation, and was frustrated when it wasn t a one-line command in git. Part of the problem here is that for most git workflows, most people don t actually use git push . That s why it s not covered in the git tutorial (this was a point of frustration for Scott). In fact, in most large projects, the number of people need to use the scm push command is a very small percentage of the developer population, just as very few developers have commit privileges and are allowed to use the svn commit command in a project using Subversion. When you have a centralized repository, only the privileged few will given commit privileges, for obvious security and quality control reasons. Ah, but in a distributed SCM world, things are more democratic anyone can have their own repository, and so everyone can type the commands git commit or bzr commit . While this is true, the number of people who need to be able to publish their own branch is small. After all, the overhead in setting up your own server just so people can pull changes from you is quite large; and if you are just getting started, and only need to submit one or two patches, or even a large series of patches, e-mail is a far more convenient route. This is especially true in the early days of git s development, before web sites such as git.or.cz, github, and gitorious made it much easier for people to publish their own git repository. Even for a large series of changes, tools such as git format-patch and git send-email are very convenient for sending a patch series, and on the receiving side, the maintainer can use git am to apply a patch series sent via e-mail. It turns out that from a maintainer s point of view, reviewing patches via e-mail is often much more convenient. Especially for developers who are just starting out with submitting patches to a project, it s rare that a patch is of sufficiently high quality that it can be applied directly into the repository without needing fixups of one kind or another. The patch might not have the right coding style compared to the surrounding code, or it might be fundamentally buggy because the patch submitter didn t understand the code completely. Indeed, more often than not, when someone submits a patch to me, it is more useful for indicating the location of the bug more than anything else, and I often have to completely rewrite the patch before it enters into the e2fsprogs mainline repository. Given that, publishing a patch that will require modification in a public repository where it is ready to be pulled just doesn t make sense for many entry-level patch submitters. E-mail is in fact less work, and more appropriate for review purposes. It is only when a mid-level to senior developer is trusted to create high quality patches that do not need review that publishing their branch in a pull-ready form really makes sense. And that is fairly rare, and why it is not covered in most entry-level git documentation and tutorials. Unfortunately, many people expect to see the command scm push in a distributed SCM, and since git pull is a commonly used command for beginning git users, they expect that they should use git push as well not realizing that in a distributed SCM, push and pull are not symmetric operations. Therefore, while most git users won t need to use git push , git tutorials and other web pages which are attempting to introduce git to new users probably do need to do a better job explaining why most beginning participants in a project probably don t need their own publically accessible repository that other people can pull from, and which they can push changes for publication. There is one exception to this, of course, and this is a developer who wants to get started using git for a new project which he or she is starting and is the author/maintainer, or someone who is interested in converting their project to git. And this is where bzr has an advantage over git, in that bzr is primarily funded by Canonical, which has a strong interest in pushing an on-line web service, Launchpad. This makes it easier for bzr to have relatively simple recipes for sharing a bzr repository, since the user doesn t need to have access to a server with a public IP address, or need to set up a web or bzr server; they can simply take advantage of Launchpad. Of course, there are web sites which make it easy for people to publish their git repositories; earlier, I had mentioned git.or.cz, github, and gitorious. Currently, the git documentation and tutorials don t mention them since they aren t formally affiliated with the git project (although they are used by many git users and developers and the maintainers of these sites have contributed a large amount of code and documentation to git). This should change, I think. Scott s frustrations which kicked off his git sucks complaints would have been solved if the Git tutorial recommended that the easist ways for someone to publicly publish their repository is via one of these public web sites (although people who want to set up their own server certainly free to do so). Most of these public repositories probably won t have much reason to exist, but they don t do much harm, and who knows? While most of the repositories published at github and gitoriuous will be like the hundreds of thousands of abandoned projects on Sourceforge, one or two of the new projects which someone starts experimenting on at github or gitorious could turn out to be the next Ruby on Rails or Python or Linux. And hopefully, they will allow more developers to be able to experiment with publishing commits on their own repositories, and lessen the frustrations of people like Scott who thought they needed their own repositories; whether or not a public repository is the best way for them to do what they need to do, at least this way they won t get as frustrated about git. :-) Related posts (automatically generated):
  1. Git and hg John Goerzen recently posted about Git, Mercurial and Bzr that...
  2. Batches of patched batches of patches I found the following from the Risks Digest, authored by...

Andrew Pollock: [tech] Upgrading MythTV to Debian 5.0 (Lenny)

I've been dying to upgrade my MythTV setup to use Lenny for ages, ever since I got a HD HomeRun and discovered my primary MythTV front end couldn't play back the HD recordings for some reason (I just got a blue box and the audio, and a lot of errors in the X server log). I figured it was either the kernel or X, and I hadn't been able to get the lirc kernel module to build against anything newer than Etch's 2.6.18 kernel (as I discovered today, that was because of some contamination in module-assistant's build directory, so I could have tried a newer kernel ages ago - drat), so I was unable to determine if it was the kernel or X.org itself. An added bonus of upgrading X was it talks to the TV better. I've got the PC hooked up to the TV's VGA input, and previously, the best I could do was get it to talk 1024x768, with the TV set to a 4:3 aspect ratio. When I initially upgraded, for whatever reason, the TV was very unhappy with what it was receiving on the VGA input and wouldn't display anything, so in desperation, I tried ditching the xorg.conf entirely, and then everything just worked. X came up with a 1280x768 resolution, I think despite the TV thinking it's getting 1024x768, so now we can have the TV in 16:9 mode, and things don't look all stretched, even when the playback is only 4:3. And HD, oh it looks glorious. We just watched the Oscars, and it completely filled the TV and looked absolutely fantastic. Playing back HD content seems to really peg the CPU though. The playback was unbearably choppy until we paused the commercial flagging job that was madly trying to flag commercials in the same recording we were currently watching (i.e. the Oscars). The only thing that seems to be acting a bit weird is notification pop ups from the D-Bus notification daemon. I've got a Griffin PowerMate that runs a home-grown script to re-pair the Bluetooth keyboard and mouse, and that uses the D-Bus notification system to provide feedback while it's running. It seems that while the MythTV front end is running, those pop ups aren't displaying correctly. I haven't gotten to the bottom that one yet. I also had some problems with a couple of init scripts and splashy causing bootup to hang, but I'm not sure that's completely reproducible (I don't reboot this machine all that often) so it may have been operator error. Overall, very happy with the outcome. The upgrade was fairly smooth, and I've got the additional functionality I've been waiting for.

Next.

Previous.