Ingo Juergensmann: Confusion about mkfs.xfs and log stripe size being too big
Recently I bought some new disks, placed them into my computer, and built a RAID5 on these 3x 4 TB disks. Creating a physical device (PV) with pvcreate, a volume group (VG) with vgcreate and some logical volumes (LV) with lvcreate was as easy and well-known as creating an XFS filesystem on the LVs... but something was strange! I never saw this message before, when creating XFS filesystems with mkfs.xfs:
log stripe unit (524288 bytes) is too large (maximum is 256KiB)Usually I don't mess around with the parameters of mkfs.xfs, because mkfs.xfs is smart enough to find near to optimal parameters for your filesystem. But apparently mkfs.xfs wanted to use a log stripe unit of 512 kiB, although its maximum size for this is 256 kiB. Why? So I started to google and in parallel asked on #xfs@freenode. Erik Sandeen, one of the core developers of XFS, suggested that I write that issue to the mailing list. He did already face this issue himself, but couldn't remember details. So I collected some more information about my setup and wrote to the XFS ML. Of course I included information about my RAID5 setup:
log stripe unit adjusted to 32KiB
muaddib:/home/ij# mdadm --detail /dev/md7Apparently, mkfs.xfs takes the chunk size of the RAID5 and want to use this for its log stripe size setting. So, that's the explanation why mkfs.xfs wants to use 512 kiB, but why is the chunk size 512 kiB at all? I didn't messed around with chunk sizes when creating the RAID5 either and all of my other RAIDs are using chunk sizes of 64 kiB. The reason was quickly found: the new RAID5 has a 1.2 format superblock, whereas the older ones do have a 0.90 format superblock. So, it seems that somewhen the default setting in mdadm, which superblock format is to be used for its metadata, has been changed. I asked on #debian.de@ircnet and someone answered that this was changed in Debian after release of Squeeze. Even in Squeeze the 0.90 format superblock was obsolete and has been only kept for backward compatibility. Well, ok. There actually was a change of defaults, which explains the behaviour of mkfs.xfs now, wanting to set log stripe size to 512 kiB. But what is the impact of falling back to 32 kiB log stripe size? Dave Chinner, another XFS developer explains:
/dev/md7:
Version : 1.2
Creation Time : Sun Jun 24 14:58:21 2012
Raid Level : raid5
Array Size : 7811261440 (7449.40 GiB 7998.73 GB)
Used Dev Size : 3905630720 (3724.70 GiB 3999.37 GB)
Raid Devices : 3
Total Devices : 3
Persistence : Superblock is persistent Update Time : Tue Jun 26 05:13:03 2012
State : active, resyncing
Active Devices : 3
Working Devices : 3
Failed Devices : 0
Spare Devices : 0 Layout : left-symmetric
Chunk Size : 512K Resync Status : 98% complete Name : muaddib:7 (local to host muaddib)
UUID : b56a714c:d193231e:365e6297:2ca61b65
Events : 16 Number Major Minor RaidDevice State
0 8 52 0 active sync /dev/sdd4
1 8 68 1 active sync /dev/sde4
2 8 84 2 active sync /dev/sdf4
Best thing in general is to align all log writes to theAnd in another mail, when being asked if it's possible to raise the 256 kiB limit to 512 kiB because of the mdadm defaults to 512 kiB as well:
underlying stripe unit of the array. That way as multiple frequent
log writes occur, it is guaranteed to form full stripe writes and
basically have no RMW overhead. 32k is chosen by default because
that's the default log buffer size and hence the typical size of
log writes.
If you increase the log stripe unit, you also increase the minimum
log buffer size that the filesystem supports. The filesystem can
support up to 256k log buffers, and hence the limit on maximum log
stripe alignment.
You can't, simple as that. The maximum supported is 256k. As it is,So, reducing the log stripe size is in fact a good thing[TM]. If anyone will benefit from larger log stripe sizes, s/he would be knowledgeable enough to play around with mkfs.xfs parameters and tune them to needs of the workload. Erik Sandeen suggested, though, to remove the warning in mkfs.xfs. Dave objects and maybe it's a good compromise to extend the warning by giving an URL for a FAQ entry explaining this issue in more depth than a short warning can do? Maybe someone else is facing the same issue and searches for information and find this blog entry helpful in the meantime...
a default chunk size of 512k is probably harmful to most workloads -
large chunk sizes mean that just about every write will trigger a
RMW cycle in the RAID because it is pretty much impossible to issue
full stripe writes. Writeback doesn't do any alignment of IO (the
generic page cache writeback path is the problem here), so we will
lamost always be doing unaligned IO to the RAID, and there will be
little opportunity for sequential IOs to merge and form full stripe
writes (24 disks @ 512k each on RAID6 is a 11MB full stripe write).
IOWs, every time you do a small isolated write, the MD RAID volume
will do a RMW cycle, reading 11MB and writing 12MB of data to disk.
Given that most workloads are not doing lots and lots of large
sequential writes this is, IMO, a pretty bad default given typical
RAID5/6 volume configurations we see....
Kategorie:
I was looking for an excuse to try



The
I ve begun work on
To:
Game on!
Sarah and I are running in the 5K
Upgrading the local version of subversion against a large subversion repository has so far taken three DAYS. It goes through this multi-gigabyte repository in no perceivable order, it goes through every single directory in every single tag and every single branch and refuses to run in any sub-directory. (Even after clearing out some older tags, the tags/ directory is still nearly 8Gb and that doesn't include any binary files.) Each tag within tags/ is 500Mb and contains nearly 6000 sub-directories. Yes, it's large but svn upgrade should still be able to handle it without crippling every machine. 

(This blog post is a more firm version of a series of tweets and forum posts I made a few weeks ago. This should also be considered a refresh of
I will sound monothematic, but I have been devoting quite a bit of work to this topic lately: Trying to stop the advance of e-voting in Mexico, Latin America and the world.
Why trying to stop it? Isn't technology supposed to help us, to get trustable processes? Yes, it's supposed to... but it just cannot achieve it, no matter how hard it is tried I won't get into explanations in this blog post, but there is plenty of information. Feel free to ask me for further details.
Anyway Yesterday (Sunday, 2012-06-17) was the fifth simulated voting that will lead to the first wide-scale deployment of electronic voting booths in my country: About 10% of the population of the state of Jalisco (that means, ~500,000 people) will cast their votes on July 1st electronically.
This particular case illustrates how simulated votings can be used to forge a lie: Pounce Consulting, the company that won the e-voting project for IEPC (Jalisco's voting authority), delivered their booths over 40 days late, just before the deadline for the project to be canceled. Oh, and by the way, it's the same company that just failed to deliver on time for another planned local authority (10% of the booths in the Federal District, where I live, where fortunately 100% of the votes will be cast on traditional, auditable and cheap paper).
After this delay, five voting simulations were programmed, to get the local population acquinted with them. The first ones just
A copy of this post is sent to
Once again Debian day was somewhat compromised: this time by more DIY. I
still managed a few bits and pieces, amortized over the rest of the week.
I benefited from some train time this weekend.