Search Results: "ag"

3 March 2009

Lucas Nussbaum: Creating a large file without zeroing it: update

Given the large number of comments I got (26!), I feel obliged to post a summary of what was said. First, the problem:
I want to create a large file (let s say 10 GB) to use as swap space. This file can t be a sparse file (a file with holes, see wikipedia if you don t know about sparse files).
Since I m going to mkswap it, I don t care about the data that is actually in that file after creating it. The stupid way (but only solution on ext3) to create it is to fill it with zeroes, with is very inefficient. Theodore Tso provided more information in a comment, which I m copying here:
Yes, it will work on ext4. A convenient which makes this easy to use can be found here at http://sandeen.fedorapeople.org/utilities/fallocate.c. It was written by Eric Sandeen, a former XFS developer who now works for Red Hat, who has been a big help making sure ext4 will be ready for Fedora and Red Hat Enterprise Linux. (Well, I guess I shouldn t call him a former XFS developer since he still contributes patches to XFS now and then, but he s spending rather more time on ext4 these days.) One warning about the program; it calls the fallocate system call directly, and it doesn t quite have the right magic architecture-specific magic for certain architectures which have various restrictions on how arguments need to be passed to system calls. In particular, IIRC, I believe there will be issues on the s390 and powerpc architectures. The real right answer is to get fallocate into glibc; folks with pull into making glibc do the right thing, please talk to me. Glibc does have posix_fallocate(), which implements the POSIX interface. posix_fallocate() is wired to use the fallocate system call, for sufficiently modern versions of glibc. However, posix_fallocate() is probablematic for some applications; the problem is that for filesystems that don t support fallocate(), posix_fallocate() will simulate it by writing all zeros to the file. However, this is not necessarily the right thing to do; there are some applications that want fallocate() for speed reasons, but if the filesystem doesn t support it, they want to receive the ENOSPC error message, so they can try some other fallback which might or might not involve writing all zero s to the file. The other shortcoming with posix_fallocate() is that it doesn t support the FALLOC_FL_KEEP_SIZE flag. What this flag allows you to do is to allocate disk blocks to the file, but not to modify the i_size parameter. This allows you to allocate space for files such as log files and mail spool files so they will be contiguous on disk, but since i_size is not modified, programs that append to file won t get confused, and tail -f will continue to work. For example, if you know that your log files are normally approximately 10 megs a day, you can fallocate 10 megabytes, and then the log file will be contiguous on disk, and the space is guaranteed to be there (since it is already allocated). When you compress the log file at the end of the day, if the log file ended up being slightly smaller than 10 megs, the extra blocks will be discarded when you compress the file, or if you like, you can explicitly trim away the excess using ftruncate().
fallocate works fine: creating a 20 GB file is almost immediate. Also, syncing or umounting the filesystem is also immediate, and reading the file returns only zeros. I m not sure how it is implemented, but it looks nice :-). However, it still doesn t solve my initial problem: mkswap works, but not swapon:
:/tmp# touch tmp
:/tmp# /root/fallocate -l 10g tmp
:/tmp# ls -lh tmp
-rw-r--r-- 1 root root 10G Mar  3 11:01 tmp
:/tmp# du tmp
10485764	tmp
:/tmp# mkswap tmp
Setting up swapspace version 1, size = 10737414 kB
no label, UUID=a316ce8e-cf33-412b-8dc0-e10d9f2ebdbb
:/tmp# strace swapon tmp
[...]
swapon( /tmp/tmp )                      = -1 EINVAL (Invalid argument)
write(2,  swapon: tmp: Invalid argument\n , 30swapon: tmp: Invalid argument
) = 30
exit_group(-1)
(swapon works fine if the file is created normally without using fallocate()). Any other ideas?

Michal Čihař: Summer of Code in phpMyAdmin

After quite successful debut in Google Summer of Code last yaer (even not directly, but though MySQL project), where both projects ended quite successfully and brought great addition to our code base, it's time to try to get again on the track. Hopefully we will manage to be direct participants this year. If you have some great ideas which are missing in phpMyAdmin, feel free to fill them either in feature tracker or directly into our ideas page.

Marco d'Itri: The files in /etc/modprobe.d/

The module-init-tools upstream maintainers decided that in the future the files in /etc/modprobe.d/ will be processed only if they have a .conf suffix. The latest module-init-tool release complains loudly for each one and still processes them, but this will change.

Please update your packages and remember that the files in this directory are opened and parsed every time modprobe is run (and it is run very often at boot time!), so try to look at the big picture and install a file there only if it is really needed. Do not install a file if it only contains comments, if it is only useful for some unusual environment or if it not needed by the kernel currently in the archive (nowadays most drivers have their own aliases built in, check /lib/modules/$KVER/modules.alias and submit a kernel patch if they are missing).

Sune Vuorela: Playing with gnash

Recently, a new version of Gnash was uploaded to Debian by Miriam Ruiz. The most important thing in the changelog from my point of view is the following: So I kind of had to test how the KDE 4 thingies works.
Youtube: The videos being watched right now -animation works, but I can t actually watch the movies.
Random flash advertisements plays nicely. Unfortunately. I was happy to avoid those. Conclusion: Some things works, but mostly the annoying things.

Obey Arthur Liu: Google Summer of Code 2009 at Debian needs you

2009-summer-of-code-logo-final-r3-no-url-011 In case you ve been living under a rock these past years, I shouldn t have to tell you what this is about :). Well, just in case.
The Google Summer of Code (GSoC) is an international program that offers student developers stipends to write code for various open source software projects. Debian has participated since 2006, mentoring dozens of students on Debian projects. The important part of the 2009 edition of the Google Summer of Code is going to start next week with the Organizations application period (March 9th). By that time, we should have listed a reasonable number of ideas on the dedicated wiki page. We will try this year to improve on the performance of the previous years, starting first with more and more dedicated manpower. If you have some time to spare to help manage our Summer of Code bid this year, we can make it a much better experience for the students and for Debian. Help is immediately needed in many areas. Some examples:

By mid-March, we will need: Further in the spring: In the summer: As you can see, there is much work ahead. It s time to pitch out your favorite pet project :)
Remember! The Summer of Code has become a prominent outreach forum for open source organizations. It puts a spotlight on each of them to provide a well-rounded and deep experience of open source development. Right now, you can help with the Wiki, join us on IRC on #debian-soc on OFTC or join the mailing-list at soc-coordination on alioth. The short flyers-friendly URL for the Summer of Code at Debian is : <http://wiki.debian.org/gsoc>. Publicize it! See you this summer!

2 March 2009

Ingo Juergensmann: HowTo: Migrate RAID1 to RAID5

As I wrote last week (German), I had the plan to make a RAID5 out of my existing RAID1 after I received my third 1 TB harddisk. Migrating to RAID5 doesn't work online yet, because mdadm doesn't support the use of --grow and --level together, as you can read in the man page when looking for "-l, --level":


-l, --level=
Set raid level. When used with --create, options are: linear, raid0, 0, stripe, raid1, 1, mirror, raid4,
4, raid5, 5, raid6, 6, raid10, 10, multipath, mp, faulty. Obviously some of these are synonymous.

When used with --build, only linear, stripe, raid0, 0, raid1, multipath, mp, and faulty are valid.

Not yet supported with --grow.


But there's always a way to obtain your goal. If you're brave enough you can find the solution by a quick internet search. Ok, I'm not that brave to try this without a backup, but you shouldn't do that kind of work without having a good backup anyway.

But before I describe the way how to do it, as usual a strong disclaimer: use this howto on your own risk! There's no warranty that this will work for you and you should take care to have a backup of your data. Whoever is doing this migration without having a backup risks the loss of all the data!

I'll describe the migration using an example... there are three logical volumes (LVs) with 1 GB in size each. This is enough for testing purposes and demonstrates how the migration works. You can use real partitions like swap partitions if you want. The LVs are r1, r2, and r3 and are within the Volume Group (VG) /dev/vg:


r1 vg -wi-a- 1.00G
r2 vg -wi-a- 1.00G
r3 vg -wi-a- 1.00G


Two of these LVs are being setup for a RAID1 /dev/md5:

CODE:
muaddib:/home/ij# mdadm --create /dev/md5 -l 1 -n 2 /dev/vg/r[12]
mdadm: /dev/vg/r1 appears to be part of a raid array:
level=raid1 devices=2 ctime=Thu Feb 26 16:16:59 2009
mdadm: /dev/vg/r2 appears to be part of a raid array:
level=raid1 devices=2 ctime=Thu Feb 26 16:16:59 2009
Continue creating array? yes
mdadm: array /dev/md5 started.


The warning to be a part of another raid array can be ignored. It's just a testing array. The result should look similar to:


md5 : active raid1 dm-12[1] dm-11[0]
1048512 blocks [2/2] [UU]
[========>............] resync = 40.2% (422900/1048512) finish=0.7min speed=14096K/sec


Now a filesystem can be created on the raid array and some data can be stored there:

CODE:
muaddib:/home/ij# mkfs.xfs -f /dev/md5
meta-data=/dev/md5 isize=256 agcount=4, agsize=65532 blks
= sectsz=512 attr=2
data = bsize=4096 blocks=262128, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0
log =internal log bsize=4096 blocks=1200, version=2
= sectsz=512 sunit=0 blks, lazy-count=0
realtime =none extsz=4096 blocks=0, rtextents=0
muaddib:/home/ij# mount /dev/md5 /mnt
muaddib:/home/ij# cp pics/backdrops/*.jpg /mnt/
muaddib:/home/ij# dir /mnt/
amigager_back1.jpg Dansk-Dunes14.jpg Dansk-Nature-2.jpg
CapeCod_085.jpg Dansk-Dunes15.jpg Dansk-Priel.jpg
ct-1-p1600.jpg Dansk-Dunes16.jpg Dansk-Woods-1.jpg
ct-2-p1600.jpg Dansk-Dunes17.jpg earth_lights_lrg.jpg
Dansk-Beach1.jpg Dansk-Dunes18.jpg HanseSail-AlterStrom2.jpg
Dansk-Beach2.jpg Dansk-Dunes2.jpg HanseSail-AlterStrom.jpg
Dansk-Beach3.jpg Dansk-Dunes3.jpg HanseSail-Mast.jpg
Dansk-Beach4.jpg Dansk-Dunes4.jpg HanseSail-Warnemuende.jpg
Dansk-Beach5.jpg Dansk-Dunes5.jpg LinuxInside_p1600.jpg
Dansk-Beachstones.jpg Dansk-Dunes6.jpg prerow.jpg
Dansk-Cliffs.jpg Dansk-Dunes7.jpg sgi-1440.jpg
Dansk-Dunes10.jpg Dansk-Dunes8.jpg sgi.jpg
Dansk-Dunes11.jpg Dansk-Dunes9.jpg Sonnenuntergang-2.jpg
Dansk-Dunes12.jpg Dansk-Dunes.jpg Sonnenuntergang.jpg
Dansk-Dunes13.jpg Dansk-Nature-1.jpg
muaddib:/home/ij# umount /mnt


After umounting the real challenge arise: creation of a RAID5 label! Normally this is no issue, but I guess you like your data and think twice before proceeding... but hey! You have a backup anyway, don't you?!

CODE:
muaddib:/home/ij# mdadm --create /dev/md5 --level=5 --raid-devices=2 /dev/vg/r[12]
mdadm: /dev/vg/r1 appears to be part of a raid array:
level=raid1 devices=2 ctime=Mon Mar 2 18:18:09 2009
mdadm: /dev/vg/r2 appears to be part of a raid array:
level=raid1 devices=2 ctime=Mon Mar 2 18:18:09 2009
Continue creating array? yes
mdadm: array /dev/md5 started.
muaddib:/home/ij# cat /proc/mdstat
md5 : active (auto-read-only) raid5 dm-12[2](S) dm-11[0]
1048512 blocks level 5, 64k chunk, algorithm 2 [2/1] [U_]
muaddib:/home/ij# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md5 : active raid5 dm-12[2] dm-11[0]
1048512 blocks level 5, 64k chunk, algorithm 2 [2/1] [U_]
[>....................] recovery = 3.1% (33152/1048512) finish=0.5min speed=33152K/sec


After stopping /dev/md5 the new RAID5 label can be written. The data ought to be safe because of the RAID5 algorithm when running with 2 disks. To verify the theory you can mount /dev/md5 again. The next steps can be done with the array online.


muaddib:/home/ij# mount /dev/md5 /mnt
muaddib:/home/ij# dir /mnt/
amigager_back1.jpg Dansk-Dunes14.jpg Dansk-Nature-2.jpg
CapeCod_085.jpg Dansk-Dunes15.jpg Dansk-Priel.jpg
ct-1-p1600.jpg Dansk-Dunes16.jpg Dansk-Woods-1.jpg
ct-2-p1600.jpg Dansk-Dunes17.jpg earth_lights_lrg.jpg
Dansk-Beach1.jpg Dansk-Dunes18.jpg HanseSail-AlterStrom2.jpg
Dansk-Beach2.jpg Dansk-Dunes2.jpg HanseSail-AlterStrom.jpg
Dansk-Beach3.jpg Dansk-Dunes3.jpg HanseSail-Mast.jpg
Dansk-Beach4.jpg Dansk-Dunes4.jpg HanseSail-Warnemuende.jpg
Dansk-Beach5.jpg Dansk-Dunes5.jpg LinuxInside_p1600.jpg
Dansk-Beachstones.jpg Dansk-Dunes6.jpg prerow.jpg
Dansk-Cliffs.jpg Dansk-Dunes7.jpg sgi-1440.jpg
Dansk-Dunes10.jpg Dansk-Dunes8.jpg sgi.jpg
Dansk-Dunes11.jpg Dansk-Dunes9.jpg Sonnenuntergang-2.jpg
Dansk-Dunes12.jpg Dansk-Dunes.jpg Sonnenuntergang.jpg
Dansk-Dunes13.jpg Dansk-Nature-1.jpg


The next step is to add the third "disk":

CODE:
muaddib:/home/ij# mdadm --add /dev/md5 /dev/vg/r3
mdadm: added /dev/vg/r3
muaddib:/home/ij# cat /proc/mdstat
md5 : active raid5 dm-13[2](S) dm-12[1] dm-11[0]
1048512 blocks level 5, 64k chunk, algorithm 2 [2/2] [UU]


That was the first step. As you can see the third "disk" has been added as a spare device. Because we don't want a spare but a RAID5 we need to expand the number of disks to 3 and tell the RAID to grow it's capacity:

CODE:
muaddib:/home/ij# mdadm --grow /dev/md5 --raid-devices=3
mdadm: Need to backup 128K of critical section..
mdadm: ... critical section passed.
muaddib:/home/ij# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md5 : active raid5 dm-13[2] dm-12[1] dm-11[0]
1048512 blocks super 0.91 level 5, 64k chunk, algorithm 2 [3/3] [UUU]
[>....................] reshape = 3.1% (33472/1048512) finish=3.0min speed=5578K/sec


The last two commands were made with the mounted filesystem that just needs to get enlarged. With XFS you make this happen by invoking xfs_growfs:

CODE:
muaddib:/home/ij# df -h /mnt
Filesystem Size Used Avail Use% Mounted on
/dev/md5 1020M 21M 1000M 2% /mnt
muaddib:/home/ij# xfs_growfs /mnt
meta-data=/dev/md5 isize=256 agcount=4, agsize=65532 blks
= sectsz=512 attr=2
data = bsize=4096 blocks=262128, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0
log =internal bsize=4096 blocks=1200, version=2
= sectsz=512 sunit=0 blks, lazy-count=0
realtime =none extsz=4096 blocks=0, rtextents=0
data blocks changed from 262128 to 524256
muaddib:/home/ij# df -h /mnt
Filesystem Size Used Avail Use% Mounted on
/dev/md5 2.0G 21M 2.0G 1% /mnt


Done!

If you have an LVM on your RAID you'll need to use the appropriate LVM tools to enlarge the PVs and VGs as well. But the hardest work is already done. The reshaping can take a lot of time, as you can see here:


md4 : active raid5 sda6[0] sdc6[2] sdb6[1]
969161152 blocks super 0.91 level 5, 64k chunk, algorithm 2 [3/3] [UUU]
[>....................] reshape = 2.8% (27990592/969161152) finish=1485.0min speed=10562K/sec


There might be other fallpits if you change your setup from 2-disk arrays to 3-disk arrays. For example you'll need to adopt the changes to your configuration files like /etc/mdadm/mdadm.conf. But this has not much to do with the migration from RAID1 to RAID5 itself.

Many thanks to Goswin Brederlow for his advice and writing a wishlist bug: #517731

Antti-Juhani Kaijanaho: Hear, hear!

I would like to applaud Colin Watson s excellent post on bad bug triage technique. Thank you, Colin. It needed to be said I have experienced most of those problems myself, in Debian.

Julian Andres Klode: debian-cd, dependencies, debimg, comments


Looking at the amd64 CD images of Lenny, I just saw that there is the package gnome on disk 1, while some of its dependencies are on disk 2. I don t think that this is good. Same also applies to K3B on disk 3, and openoffice.org-kde on KDE-disk 1. In my opinion, all packages which are located on a disk X, should only depend on a disk N (N<X). This means that the package gnome would be moved to disk 2, or its dependencies to disk 1. This is exactly the way debimg works. Debimg uses so-called package groups, which simply represents a set of packages with cyclic dependencies. These package groups are returned in a specific order, so that a group N only requires a group <N. This order is kept when the groups are added to the disk. Furthermore, we treat the groups as one when adding them to the disk, ie. we check whether the whole group fits on the disk (and add it) or not (create a new disk). This way, we ensure that all dependencies can be satisfied. Today, I decided to try it out and therefore wrote a small script reading the debian-cd task files and comparing the file list (of the packages added to the disk) with the file list of the official KDE disk. The results look very good, debimg adds 11 packages and removes 26 which is not that much, and mostly caused by wrong size limits, etc. You can look at the results your self, and regenerate them using: Anyway, debimg still needs real configuration and handling of the debian installer and source packages, but this should give you an idea of how debimg works. And a tarball, for those who don t want to use git all the time. BTW, I have now enabled comment threading here, a new WordPress feature. I hope that it works, as I have not tried it out yet. Posted in Debian, debimg

Joerg Jaspert: DPL Debate

So its this time of year again, we are soon going to kick Steve and welcome another one as DPL (unless he is crazy enough to run again and also win). And as people asked if the DPL IRC Debate we had the last years, some 3 hours of online chat trying to focus on one thing, is useful, there is currently a poll running for it. Feel free to participate:

Michal &#268;iha&#345;: Gammu stable version 1.23.1

After recently introduced tradition, I again come with bugfix update for 1.23.0 called 1.23.1. This time there is no critical fix as in 1.22.1, but few fixes, which might annoy users, for example forgotten installation of gammu-smsd-monitor tool. Full list of changes: You can download from usual place: http://cihar.com/gammu/, Debian users will find packages in unstable soon, SUSE, Fedora and Mandriva users can check repository in OpenSuse Build Service and Ubuntu in my PPA.

Colin Watson: Bug triage rants

I hate to say this, but often when somebody does lots of bug triage on a package I work on, I find it to be a net loss for me. I end up having to go through all the things that were changed, correct a bunch of them, occasionally pacify angry bug submitters, and all the rest of it, and often the benefits are minimal at best. I would very much like this not to be the case. Bug triage is supposed to help developers be more efficient, and I think most people who do bug triage are generally well-intentioned and eager to help. Accordingly, here is a series of mini-rants intended to have educational value.

Alastair McKinstry: Titan, by Stephen Baxter

Titan, by Stephen Baxter describes a near-future manned mission to Titan, Saturn's moon, by Space Shuttle, no less. This is the first book i've read by Stephen Baxter, and will probably not be the last. It's well written, the most science-based outer planets mission story i've read in since Arthur C. Clarke, though with a pessimistic edge. The story turns on a waning NASA using its remaining hardware on one big mission, to Titan, where life may have been discovered. Most of the book concentrates naturally enough on getting to Titan, rather than what they found when they get there, but Baxters ammonia-based chemistry is intriguing. The physics of what it would be like to walk on the moon are a little underdone, though. Its an understatement to say that Titan is cold: serious work on insulation would be needed to stop your base melting the "permafrost" and sinking. While we don't know what life might be present, it would be good to imagine the real tasks involved in being there. But it's his idea that NASA and others really don't want space that is most interesting. That the military want space as their baliwick, and would rather frustrate the development of technologies so that no other nations or groups interfere is well worth thinking about. To a degree, I would agree: frustrate, but not he puts it, stop. Sooner or later the technology has other uses on Earth to the point where even amateurs would launch their own satellites. The vast bulk of technology development for future space use will be done not only on Earth but for Earth use: the minaturization that NASA and the military did for the space race in the 1960's now happens for consumer electronics. 3 D fabricators are being built for prototyping, and the prospect of getting a self-contained space mission together (not reliant on Earth) draws closer. I fully expect private groups, rather than national efforts, will be the first to colonise outer space, assembling pre-existing technologies. Tags , ,

Ian Wienand: YUI ButtonGroup Notes

Some tips and things to check if your YUI ButtonGroup isn't behaving as you wish it would. Hopefully, this will save someone else a few hours!

Russell Coker: A Police SMS about Fire Risk

My wife and I have each received SMS messages from Vic.Police that say: Extreme weather expected tonight (Monday) & tomorrow. High wind & fire risk. Listen to the ABC local radio for emergency update. Do not reply to this message. Presumably the police are trying to contact everyone in Victoria. The problem seems to be related to the high wind speed that is forecast, the temperature is only predicted to be 32C (as opposed to the 38C that they were forecasting a few days ago and the temperatures of 46C or more a few weeks ago). The last reports were that the firefighters were still working on putting out fires, and the unclear news coverage seemed to suggest that some of the fires had been burning since the 7th of February. A day of extreme fire danger that starts without any fires would be bad enough, but starting with some fires that are already out of control is destined to give a very bad result. Below is the link to my previous post about people trying to take advantage of a tragedy to benefit their own political causes. For anyone who wants to rail against abortion, homosexuality, or the Greens party, please show some decency and do so based on relevant facts and do it at an appropriate time. I suggest that anyone who writes later this week about ways to avoid bushfires should be careful to check their claims for accuracy and scientific evidence (hint - the CSIRO and NASA have published a lot of useful background information). http://etbe.coker.com.au/2009/02/25/tragedy-and-profit/

Joey Hess: re: Declarative .deb maintainer scripts

Lars wonders about debhelper's approach of inserting script fragments into package maintainer scripts. It's really very interesting to survey debhelper's collection of fragments, in its autoscripts directory. Of the 55 fragments, the mean length is 4.5 lines, and the mode (and also shortest[1]) length is 3 lines. The 32 scripts that are each three lines follow a very simple pattern:
if [ "$1" = something ] && exists some-script; then
    some-script some-parameters
fi
It would be nice to get that down to Lar's more declarative one line approach:
some-script some-parameters $@
In practice though, many of these scripts are not always available, and so the postinst has to test for them before running them. And making them test the postinst's parameters to see if they should do anything makes them less general and less flexible (ie, not able to be run easily by a sysadmin), and so keeping the "$1" test in the postinst is reasonable. So, I think I've done a pretty good job at keeping debhelper script fragments short and keeping most of the possibly buggy bits that might need maintenace out of debhelper. The exceptions tend to be either older scripts, from a time when I was less careful about doing do, and fragments of code that I could not pawn off onto some other package, often because there was no package that owned the issue. (The dh_usrlocal code is the best example of that.) The most promising way to improve things seems to be using dpkg triggers, which is why I was very enthuisiastic when we finally got them, over a year ago. Triggers let many of these hooks into maintainer scripts go away completely. Rather than each package having to test for and call update-menus, menu can simply trigger on the modification of any menu file. Once menu finishes the conversion to using triggers, debhelper can drop those fragments entirely. Triggers seem likely to be able to eliminate well over 90% of the things we need to put into maintainer scripts, and I hope at least half of debhelper's script fragments can eventually go away thanks to them. The others seem more difficult. You probably cannot, for example, make a trigger that does something sane when an init script is installed. Debhelper has at least three different fragements that it can put into postinst scripts to handle init scripts, and these only cover the most common three cases (start, don't start, don't restart on upgrade). Moving even these things into a declarative form has long been a dream of mine. Maybe it could be done with a control file that specifies which of the methods for running the package's init script to use for this package. But it seems we can't do away with maintainer scripts entirely, no matter how hard we try. That's because maintainer scripts are our only way to deal with other crippling bugs in old versions of a package, and with upgrade issues, and with all the other myriad things that we put custom code in them to deal with. Pity.
[1] Slight oversimplification; two scripts cram the 3-line form in one line for no good reason and are counted as 3-lines above.

Bdale Garbee: TeleMetrum Power Supplies

After a long hiatus for various reasons, I finally had the chance to try my hand at loading a TeleMetrum board today! Sadly, I wasn't able to completely load any boards, because I somehow ended up with the wrong Digi-Key part numbers for 4 capacitors, one of which is a critical value. I placed an order for the missing parts and a few other bits we'll need eventually, hopefully they'll be here in a couple days and I can try to load a board with all the parts. Instead, what I did today was a a partial load of a board with the goal of testing the various power supplies. Seemed like a good idea, since I've had my share of odd problems with power supplies in the past. This also gave me a chance to try out my solder paste stencil, get some experience hand-placing the tiny 0402 passive parts, and an excuse to see if I could hit the solder reflow temperature profile adequately with my electric cooking skillet and IR non-contact thermometer. The good news is that it all worked right the first time! We're successfully charging a LiPo battery from USB power, and successfully making 3.3 volts for the electronics from that. The resistor divider designed to allow the LiPo battery voltage to be sampled by the CPU's analog to digital converter is also working fine. A few observations are in order, however. Getting the right amount of solder paste down on the board requires some finesse with the spreading blade and the stencil. I think the paste was a bit heavy around the CC1111 footprint, as I ended up with some bridged pads. Since I didn't load the actual CPU part, it could just be that not having something real there to wick up some solder meant there was enough to form the bridges. Don't know. The amount of solder on the ground pad in the middle of the chip looks good, though! The 0603 sized LED in the power supply circuit was the only part that didn't self-align correctly. I may have had it a bit too far to one side. The pads on the part have notches in the end, and it looks like one side of each is more or less centered, so I think surface tension did the best it could with what I gave it to work with. The LED soldered ok, isn't shorting to anything else, and clearly works fine... but I'd be happier if it were aligned better. Found and fixed a solder bridge between two pins on the surface-mount USB connector before first application of power. Could be further data that the paste layer was a bit too thick. On the other hand, I had to wiggle the connector around a bit to get it aligned correctly, so I may have smudged the paste into a bridge while doing that. I also note that there's some visible flux left on the surface of the solder, particularly on big paste areas like the feet of the USB connector. I suspect this means I didn't dwell long enough at either or both of the preheat or ultimate reflow temperatures. Since all the parts clearly soldered adequately to function, I'm still pretty happy... but since the rockets we fly experience violent accelerations during boost, this is something I'll play with on future reflows. All in all, a great way to spend an afternoon, and a big step forward for the TeleMetrum project!

Theodore Ts'o: SSD s, Journaling, and noatime/relatime

On occasion, you will see the advice that the ext3 file system is not suitable for Solid State Disks (SSD s) due to the extra writes caused by journaling and so Linux users using SSD s should use ext2 instead. However, is this folk wisdom actually true? This weekend, I decided to measure exactly what the write overhead of journaling actually is in actual practice. For this experiment I used ext4, since I recently added a feature to track the amount of writes to the file system over its lifetime (to better gauge the wear and tear on an SSD). Ext4 also has the advantage that (starting in 2.6.29), it can support operations with and without a journal, allowing me to do a controlled experiment where I could manipulate only that one variable. The test workload I chose was a simple one: For the first test, I ran the test using no special mount options, and the only difference being the presence or absence of the has_journal feature. (That is, the first file system was created using mke2fs -t ext4 /dev/closure/testext4, while the second file system was created using mke2fs -t ext4 -O ^has_journal /dev/closure/testext4.)
Amount of data written (in megabytes) on an ext4 filesystem
Operation with journal w/o journal percent change
git clone 367.7 353.0 4.00%
make 231.1 203.4 12.0%
make clean 14.6 7.7 47.3%
What the results show is that metadata-heavy workloads, such as make clean, do result in almost twice the amount data written to disk. This is to be expected, since all changes to metadata blocks are first written to the journal and the journal transaction committed before the metadata is written to their final location on disk. However, for more common workloads where we are writing data as well as modifying filesystem metadata blocks, the difference is much smaller: 4% for the git clone, and 12% for the actual kernel compile. The noatime mount option Can we do better? Yes, if we mount the file system using the noatime mount option:
Amount of data written (in megabytes) on an ext4 filesystem mounted with noatime
Operation with journal w/o journal percent change
git clone 367.0 353.0 3.81%
make 207.6 199.4 3.95%
make clean 6.45 3.73 42.17%
This reduces the extra cost of the journal in the git clone and make steps to be just under 4%. What this shows is that most of the extra meta-data cost without the noatime mount option was caused by update to the last update time for kernel source files and directories. The relatime mount option There is a newer alternative to the noatime mount option, relatime. The relatime mount option updates the last access time of a file only if the last modified or last inode changed time is newer than the last accessed time. This allows programs to be able to determine whether a file has been read size it was last modified. The usual (actually, only) example that is given of such an application is the mutt mail-reader, which uses the last accessed time to determine if new mail has been delivered to Unix mail spool files. Unfortunately, relatime is not free. As you can see below, it has roughly double the overhead of noatime (but roughly half the overhead of using the standard Posix atime semantics):
Amount of data written (in megabytes) on an ext4 filesystem mounted with relatime
Operation with journal w/o journal percent change
git clone 366.6 353.0 3.71%
make 216.8 203.7 6.04%
make clean 13.34 6.97 45.75%
Personally, I don t think relatime is worth it. There are other ways of working around the issue with mutt for example, you can use Maildir-style mailboxes, or you can use mutt s check_mbox_size option. If the goal is to reduce unnecessary disk writes, I would mount my file systems using noatime, and use other workarounds as necessary. Alternatively, you can use chattr +A to set the noatime flag on all files and directories where you don t want noatime semantics, and then clear the flag for the Unix mbox files where you care about the atime updates. Since the noatime flag is inherited by default, you can get this behaviour by setting running chattr +A /mntpt right after the filesystem is first created and mounted; all files and directories created in that file system will have the noatime file inherited. Comparing ext3 and ext2 filesystems
Amount of data written (in megabytes) on an ext3 and ext2 filesystem
Operation ext3 ext2 percent change
git clone 374.6 357.2 4.64%
make 230.9 204.4 11.48%
make clean 14.56 6.54 55.08%
Finally, just to round things out, I tried the same experiment using the ext3 and ext2 file systems. The difference between these results and the ones involving ext4 are the result of the fact that ext2 does not have the directory index feature (aka htree support), and both ext2 and ext3 do not have extents support, but rather use the less efficient indirect block scheme. The ext2 and ext3 allocators are also someone different from each other, and from ext4. Still, the results are substantially similar with the first set of Posix-compliant atime update numbers (I didn t bother to do noatime and relatime benchmark runs with ext2 and ext3, but I expect the results would be similar.) Conclusion So given all of this, where did the common folk wisdom that ext3 was not suitable for SSD s come from? Some of it may have been from people worrying too much about extreme workloads such as make clean ; but while doubling the write load sounds bad, going from 4MB to 7MB worth of writes isn t that much compared to the write load of actually doing the kernel compile or populating the kernel source tree. No, the problem was that first generation SSD s had a very bad problem with what has been called the write amplification effect , where a 4k write might cause a 128k region of the SSD to be erased and rewritten. In addition in order to provide safety against system crashes, ext3 has more synchronous write operations that is where ext3 waits for the write operation to be complete before moving on, and this caused a very pronounced and noticeable stuttering effect which was fairly annoying to users. However, the next generation of SSD s, such as Intel s X25-M SSD, have worked around the write amplification affect. What else have we learned? First of all, for normal workloads that include data writes, the overhead from journaling is actually relatively small (between 4 and 12%, depending on the workload). Further, than much of this overhead can be reduced by enabling the noatime option, with relatime providing some benefit, but ultimately if the goal is to reduce your file system s write load, especially where an SSD is involved, I would strongly recommend the use of noatime over relatime. Related posts (automatically generated):
  1. Should Filesystems Be Optimized for SSD s? In one of the comments to my last blog entry,...
  2. Aligning filesystems to an SSD s erase block size I recently purchased a new toy, an Intel X25-M SSD,...
  3. Fast ext4 fsck times This wasn t one of the things we were explicitly engineering...

1 March 2009

Dirk Eddelbuettel: Rcpp 0.6.4

A new maintenance version of Rcpp (now at 0.6.4) was just pushed to CRAN and has been uploaded to Debian. Rcpp is a set of utility classes that provide interfaces for transferring the major R data types to C++ and back which makes it easier to extend R with dynamically loadable code written in C or C++. This version changes how use the std namespace: all usage is now properly prefixed. Likewise, we now define R_NO_REMAP to not let R define utility functions as length() or error() which occassionally creates trouble with other include files -- so now use the more explicit forms Rf_length(), Rf_error() etc. Also, starting from this release, C++ class documentation created by Doxygen is included; it can also be seen from my box as both browsable html and a pdf file. Lastly, this version also adds a minor correction to the Windows build (spotted by Uwe and Simon).

Manoj Srivastava: Manoj: Rethinking ucf redux

I have been thinking some more about how to improve ucf. One of the things that struck me was that based on my earlier analysis there are only five actions that ucf can take, and the decision about the actions depends on the state it finds the configuration file in on the target machine, and there are only eight of those. Now, thinking back to my days as a VLSI designer back in the halcyon days of electrical engineering, This is a pretty simple state machine. It is not as neat as it could be (where just three variables would be needed to keep track of things, but still, it bears investigation. This would be a way for converting the current procedural ucf into a functional programming model. Hop over here for a look at how that went it was fun, and afforded me an opportunity to demonstrate how well org handles LaTeX.png snippets.

Julian Andres Klode: debimg update - creating images


Much happened since the last time I wrote about debimg. The project is now registered on Alioth and has a mailing list. On the code side, there have also been several changes. First of all, the repository module has been merged into the master branch. This was the first step towards the creation of the image building, which happened today by introducing the image module. The code should be treated as Beta quality, but the project as a whole is Alpha, because the application utilizing debimg.core is still missing. As always, I hereby encourage to try out debimg, have a look at the examples, and help to develop it. Patches should be sent to the mailing list, created with git format-patch, and its usual settings (eg. prefixed with eg. [PATCH 1/2] ). The patches should be inline and validate in pyflakes and using pep8.py. See the README for some more recommendations. Because NEW seems to be really busy at the moment (and doko uploaded python2.6 and python3.1), I m not uploading the current state as 0.1.0~a1 to experimental, but wait until NEW is smaller (maybe it will be 0.1.0~b1 then). What is wanted/planned?
  • Compatibility configuration formats
    • (debimg.config.simple_cdd) A reimplementation of simple-cdd using debimg. This probably needs to wait until there is a module for dealing with the installer, but I want to implement an application to support simple-cdd configuration files.
    • (debimg.config.debian_cd) There could be an implementation of debian-cd s configuration format. This would allow people to easily try out debimg. We can only support a subset, though.
  • New configuration formats
    • (debimg.config.yaml) Debimg s native configuration format. This is the only one supported by the graphical frontends. Where possible, configuration between compatibility formats and this format will be provided.
  • Graphical Frontends:
    • (debimg.frontend.gtk2) The graphical GTK+ frontend for inexperienced users who just want to create their own disk.
    • (debimg.frontend.qt4) The graphical frontend written in QT4.
  • Text frontends
    • (debimg.frontend.text) The basic command-line frontend.
    • (debimg.ubuntu.frontend.text) A script to build Ubuntu images, which could be used to build the official Ubuntu Images. This requires interaction with the germinate tool for dependency resolution, and more. It s also not flexible enough for building custom images.
Requirements for 0.1 Beta
  • Implement at least one configuration format and the build frontend.
  • Have at least one external contributor.
More Changes Julian Andres Klode (6):
  • debimg/core/files: When creating a file object allow filename as source
  • debimg/core/resolver.py: Introduce Package.fullname, Package.component
  • debimg/core/repository.py: Merge the repository module.
  • debimg/core/resolver.py: Improve handling of certain dependency types
  • debimg/core/repository.py: Allow Repository.add_group to take a distro parameter
  • debimg/core/image.py: Introduce the image module.
Posted in Debian, debimg, Ubuntu

Next.

Previous.