Search Results: "tar"

8 August 2012

Russell Coker: Hard Drives for Backup

The general trend seems to be that cheap hard drives are increasing in capacity faster than much of the data that is commonly stored. Back in 1998 I had a 3G disk in my laptop and about 800M was used for my home directory. Now I have 6.2G used for my home directory (and another 2G in ~/src) out of the 100G capacity in my laptop. So my space usage for my home directory has increased by a factor of about 8 while my space available has increased by a factor of about 30. When I had 800M for my home directory I saved space by cropping pictures for my web site and deleting the originals (thus losing some data I would rather have today) but now I just keep everything and it s still doesn t take up much of my hard drive. Similar trends apply to most systems that I use and that I run for my clients. Due to the availability of storage people are gratuitously using a lot of disk space. A relative recently took 10G of pictures on a holiday, her phone has 12G of internal storage so there was nothing stopping her. She might decide that half the pictures aren t that great if she had to save space, but that space is essentially free (she couldn t buy a cheaper phone with less storage) so there s no reason to delete any pictures. When considering backup methods one important factor is the ability to store all of one type of data on one backup device. Having a single backup span multiple disks, tapes, etc has a dramatic impact on the ease of recovery and the potential for data loss. Currently 3TB SATA disks are really cheap and 4TB disks are available but rather expensive. Currently only one of my clients has more than 4TB of data used for one purpose (IE a single filesystem) so apart from that client a single SATA disk can backup anything that I run. Benefits of Hard Drive Backup When using a hard drive there is an option to make it a bootable disk in the same format as the live disk. I haven t done this, but if you want the option of a quick recovery from a hardware failure then having a bootable disk with all the data on it is a good option. For example a server with software RAID-1 could have a backup disk that is configured as a degraded RAID-1 array. The biggest benefit is the ability to read a disk anywhere. I ve read many reports of tape drives being discovered to be defective at the least convenient time. With a SATA disk you can install it in any PC or put it in a USB bay if you have USB 3.0 or the performance penalty of USB 2.0 is bearable a USB 2.0 bay is great if you want to recover a single file, but if you want terabytes in a hurry then it won t do. A backup on a hard drive will typically use a common filesystem. For backing up Linux servers I generally use Ext3, at some future time I will move to BTRFS as having checksums on all data is a good feature for a backup. Using a regular filesystem means that I can access the data anywhere without needing any special software, I can run programs like diff on the backup, and I can export the backup via NFS or Samba if necessary. You never know how you will need to access your backup so it s best to keep your options open. Hard drive backups are the best solution for files that are accidentally deleted. You can have the first line of backups on a local server (or through a filesystem like BTRFS or ZFS that supports snapshots) and files can be recovered quickly. Even a SATA disk in a USB bay is very fast for recovering a single file. LTO tapes have a maximum capacity of 1.5TB at the moment and tape size has been increasing more slowly than disk size. Also LTO tapes have an expected lifetime of only 200 reads/writes of the entire tape. It seems to me that tapes don t provide a great benefit unless you are backing up enough data to need a tape robot. Problems with a Hard Drive Backup Hard drives tend not to survive being dropped so posting a hard drive for remote storage probably isn t a good option. This can be solved by transferring data over the Internet if the data isn t particularly big or doesn t change too much (I have a 400G data set backed up via rsync to another country because most of the data doesn t change over the course of a year). Also if the data is particularly small then solid state storage (which costs about $1 per GB) is a viable option, I run more than a few servers which could be entirely backed up to a 200G SSD. $200 for a single backup of 200G of data is a bit expensive, but the potential for saving time and money on the restore means that it can be financially viable. Some people claim that tape storage will better survive a Carrington Event than hard drives. I m fairly dubious about the benefits of this, if a hard drive in a Faraday Cage (such as a regular safe that is earthed) is going to be destroyed then you will probably worry about security of the food supply instead of your data. Maybe I should just add a disclaimer this backup system won t survive a zombie apocalypse . ;) It s widely regarded that tape storage lasts longer than hard drives. I doubt that this provides a real benefit as some of my personal servers are running on 20G hard drives from back when 20G was big. The fact that drives tend to last for more than 10 years combined with the fact that newer bigger drives are always being released means that important backups can be moved to bigger drives. As a general rule you should assume that anything which isn t regularly tested doesn t work. So whatever your backup method you should test it regularly and have multiple copies of the data to deal with the case when one copy becomes corrupt. The process of testing a backup can involve moving it to newer media. I ve seen it claimed that a benefit of tape storage is that part of the data can be recovered from a damaged tape. One problem with this is that part of a database often isn t particularly useful. Another issue is that in my experience hard drives usually don t fail entirely unless you drop them, drives usually fail a few sectors at a time. How to Implement Hard Drive Backup The most common need for backups is when someone deletes the wrong file. It s usually a small restore and you want it to be an easy process. The best solution to this is to have a filesystem with snapshots such as BTRFS or ZFS. In theory it shouldn t be too difficult to have a cron job manage snapshots, but as I ve only just started putting BTRFS and ZFS on servers I haven t got around to changing my backups. Snapshots won t cover more serious problems such as hardware, software, or user errors that wipe all the disks in a server. For example the only time I lost a significant amount of data from a hosted server was when the data center staff wiped it, so obviously good off-site backups are needed. The easiest way to deal with problems that wipe a server is to have data copied to another system. For remote backups you can rsync to a local system and then use cp -rl or your favorite snapshot system to make a hard linked copy of the tree. A really neat feature is the ZFS ability to send a filesystem snapshot (or the diff between two snapshots) to a remote system [1]. Once you have regular backups on local storage you can then copy them to removable disks as often as you wish, I think I ll have to install ZFS on some of my servers for the sole purpose of getting the send feature! There are NAS devices that provide similar functionality to the ZFS send/receive (maybe implemented with ZFS), but I m not a fan of cheap NAS devices [2]. It seems that the best way to address the first two needs of backup (fast local restore and resilience in the face of site failure) is to use ZFS snapshots on the server and ZFS send/receive to copy the data to another site. The next issue is that the backup server probably won t be big enough for all the archives and you want to be able to recover from a failure on the backup server. This requires some removable storage. The simplest removable backup is to use a SATA drive bay with eSATA and USB connectors. You use a regular filesystem like Ext3 and just copy the files on. It s easy, cheap, and requires no special skill or software. Requiring no special skill is important, you never know who will be called on to recover from backups. When a server is backing up another server by rsync (whether it s in the same rack or another country) you want the backup server to be reliable. However there is no requirement for a single reliable server and sometimes having multiple backup servers will be cheaper. At current purchase prices you can buy two cheap tower systems with 4*3TB disks for less money than a single server that has redundant PSUs and other high end server features. Having two cheap servers die at once seems quite unlikely so getting two backup servers would be the better choice. For filesystems that are bigger than 4TB a disk based backup would require backup software that handles multi part archives. One would hope that any software that is designed for tape backup would work well for this (consider a hard drive as a tape with a very fast seek), but often things don t work as desired. If anyone knows of a good Linux backup program that supports multiple 4TB SATA disks in eSATA or USB bays then please let me know. Conclusion BTRFS or ZFS snapshots are the best way of recovering from simple mistakes. ZFS send/receive seems to be the best way of synchronising updates to filesystems to other systems or sites. ZFS should be used for all servers. Even if you don t currently need send/receive you never know what the future requirements may be. Apart from needing huge amounts of RAM (one of my servers had OOM failures when it had a mere 4G of RAM) there doesn t seem to be any down-side to ZFS. I m unsure of whether to use BTRFS for removable backup disks. The immediate up-sides are checksums on all data and meta-data and the possibility of using built-in RAID-1 so that a random bad sector is unlikely to lose data. There is also the possibility of using snapshots on a removable backup disk (if the disk contains separate files instead of an archive). The down-sides are lack of support on older systems and the fact that BTRFS is fairly new. Have I missed anything? Related posts:
  1. New Storage Developments Eweek has an article on a new 1TB Seagate drive....
  2. IDE hard drives I just lent two 80G IDE drives to a friend,...
  3. Hot-swap Storage I recently had to decommission an old Linux server and...

Johannes Schauer: Bootstrappable Debian - How to help

TLDR: multiarch, multiarch, multiarch, cross buildability, staged build dependencies, wiki page, corrections/hints/requests to debian-bootstrap at lists.mister-muffin.de This summer (and this year's GSoC) is nearing its end and to make it easier for people to make use of the information my tools produced so far, I created a page in the Debian wiki. It lists not only the open issues I see but also statistics that I gathered using the output of my GSoC project. I want to use this blog post to make people aware of that page as well as to get some feedback on it and anything related to it. The biggest blocker my tools face, is that many packages are still missing multiarch information. As long as at least the basic packages do not have their cross build dependencies satisfied via multiarch for an existing foreign architecture, automated tools can not properly analyze the dependency situation in the bootstrapping case, when many packages of the new foreign architecture do not even exist yet. If Debian is supposed to be bootstrappable, then the first stage is to make a set of basic packages cross compile for an existing foreign architecture. Once this is possible, a tool of mine can analyze the cyclic build dependency situation that might occur when cross compiling for an architecture that does not exist yet. Then, staged cross builds can be used to cross compile a minimal foreign system. Due to missing multiarch classification, it is not known yet how big the cyclic build dependency situation is for the base packages. It is not only the conversion of packages to multiarch that is needed but also the adding of the :any (and rare cases :native) qualifier to build dependencies on M-A: allowed packages. Prominent build dependencies that should (but are not yet) be M-A: allowed are python and gettext. Both are needed as a build dependency by many packages of the base system. Unfortunately wanna-build does not understand qualifiers like :any and :native yet. Until it does, no package can be marked :any or :native and cross compilation of many base packages can not succeed. Once the point is reached, where a base system can be cross compiled from nothing, native compilation can start. Since native compilation doesnt depend on multiarch, the dependency situation when trying to natively compiling all of Debian from nothing is understood much better. Unfortunately, the cyclic build dependency situation is also much worse in the native case and there exists a big 1000 node strongly connected component of binary and source packages that all interdepend on each other. This dependency mess can be solved using three approaches: The wiki page gives many hints on how to find packages that each method can be applied to. Stage building is a tool that might be useful for cross building (we dont know for sure yet) but is definitely needed for native compilation. It is needed for native compilation because after all possible dependencies are moved to Build-Depends-Indep, the only other alternative to stage building for breaking dependency cycles is to cross build source packages. Since building a package without one of its build dependencies "staged" is often much easier than making the package in question cross compile, it is a preferred alternative. Once more packages have been made multiarch, it might be possible to prove that there is no alternative to introducing a notion of staged builds. Some people (wookey, Patrick McDermott, Guillem Jover, myself) decided that the following format to mark staged build dependencies would be preferred over others: 
Build-Depends: huge (>= 1.0) [i386 arm] <!embedded !bootstrap>, tiny
The <> format was proposed by Guillem Jover in bug#661538. Patches for dpkg and dose3 are done. More people need to discuss about this format for a final decision on how to indicate staged build dependencies. For more information on the topic, have a look at the corresponding wiki page. Feel free to direct any comments/critique/hints to debian-bootstrap at lists.mister-muffin.de or directly to me.

6 August 2012

Vincent Sanders: Education is when you read the fine print. Experience is what you get if you don't.

Given my title I guess I just got a load of "experience" from Gandi. Previously I have written about building the NetSurf Continuous Integration (CI) server and my misjudgement of the resources it would require to be useful.

I contacted Gandi and they confirmed my suspicion that my only options to extend the VPS were to spend a great deal more money. The total spend would have been in excess of 300 per year to get a minimally satisfactory configuration.

So I contacted Mythic Beasts and they offered the NetSurf project a selection of deals they could not refuse. Once we decided on the right deal for us the system was in place, ready to use, within a morning.

Thanks to the generous open source project discount the offering from Mythic is almost double what Gandi were offering for less than half the price. I was, as previously concluded, dumb to go with Gandi in the first place when Mythic were an option.

Once the system was transferred to the shiny new server I went back to Gandi and asked how much my mistake of buying a years service for a single month worth of VPS service was going to cost. The reply was somewhat disappointing I was simply referred to section 8.1.2 of the contract (tl;dr We have your money now, tough luck, your problem). It would appear my mistake was going to cost me the full 114 I had paid.

I was somewhat disheartened at this turn of events but then Daniel Silverstone offered to take on the VPS if I could transfer it to his Gandi handle so he could administer it. this seemed like a good solution, at least the resource would be used and not wasted. Gandi response to this was initially promising but this is the final reply:

I'm really sorry to inform you that my supervisor refused the share transfer request to the other account. He told me that we do not transfer products either in those cases. I deeply apologize since I thought it was possible. I thank you for your confidence in Gandi, and hope that this occurrence does not deter you from continuing with us.
Now I must stress that Gandi are completely within their contractual rights to do this and are not obligated to provide any assistance to help make my blunder less costly to me, but they have firmly shown they have no willingness to be flexible in any way which means I know where I will not be spending my money in future.

Oh and then Pete from Mythic mentioned that they have a full refund and flexible policy on their services and we can always switch to the monthly billing rate and get a refund on the outstanding balance or move it or...

Basically my recommendation for an open source project looking for hosting, VPS or any network services has to be to go talk to the nice people at Mythic and steer clear of Gandi. Maybe in future all providers will realise that they are in a competitive market and certainly a little customer service flexibility might have made this posting much more about my mistake and not the attitude of the suppliers involved.

Vincent Sanders: Pimm's and Punting

The Sun came out in Cambridge the other week (yes I know, a rare occurrence) and because Robert is a generous soul he decided the office should go punting on the cam.

I had not previously been out on the river but Robert proved to be the consummate punter.

We even got fruit based drinks and some serenading, although perhaps that was not a benefit.


I took many photos of the event including the rather nice meal afterwards, some were even in focus and not blurry.

Mike Hommey: Building a Linux kernel module without the exact kernel headers

Imagine you have a Linux kernel image for an Android phone, but you don t have the corresponding source, nor do you have the corresponding kernel headers. Imagine that kernel has module support (fortunately), and that you d like to build a module for it to load. There are several good reasons why you can t just build a new kernel from source and be done with it (e.g. the resulting kernel lacks support for important hardware, like the LCD or touchscreen). With the ever-changing Linux kernel ABI, and the lack of source and headers, you d think you re pretty much in a dead-end. As a matter of fact, if you build a kernel module against different kernel headers, the module will fail to load with errors depending on how different they are. It can complain about bad signatures, bad version or other different things. But more on that later. Configuring a kernel The first thing is to find a kernel source for something close enough to the kernel image you have. That s probably the trickiest part with getting a proper configuration. Start from the version number you can read from /proc/version. If, like me, you re targeting an Android device, try Android kernels from Code Aurora, Linaro, Cyanogen or Android, whichever is closest to what is in your phone. In my case, it was msm-3.0 kernel. Note you don t necessarily need the exact same version. A minor version difference is still likely to work. I ve been using a 3.0.21 source, which the kernel image was 3.0.8. Don t however try e.g. using a 3.1 kernel source when the kernel you have is 3.0.x. If the kernel image you have is kind enough to provide a /proc/config.gz file, you can start from there, otherwise, you can try starting from the default configuration, but you need to be extra careful, then (although I won t detail using the default configuration because I was fortunate enough that I didn t have to, there will be some details further below as to why a proper configuration is important). Assuming arm-eabi-gcc is in your PATH, and that you have a shell opened in the kernel source directory, you need to start by configuring the kernel and install headers and scripts: 
$ mkdir build
$ gunzip -c config.gz > build/.config # Or whatever you need to prepare a .config
$ make silentoldconfig prepare headers_install scripts ARCH=arm CROSS_COMPILE=arm-eabi- O=build KERNELRELEASE= adb shell uname -r
The silentoldconfig target is likely to ask you some questions about whether you want to enable some things. You may want to opt for the default, but that may also not work properly. You may use something different for KERNELRELEASE, but it needs to match the exact kernel version you ll be loading the module from. A simple module To create a dummy module, you need to create two files: a source file, and a Makefile. Place the following content in a hello.c file, in some dedicated directory: 
#include <linux/module.h>       /* Needed by all modules */
#include <linux/kernel.h>       /* Needed for KERN_INFO */
#include <linux/init.h>         /* Needed for the macros */
static int __init hello_start(void)
 
  printk(KERN_INFO "Hello world\n");
  return 0;
 
static void __exit hello_end(void)
 
  printk(KERN_INFO "Goodbye world\n");
 
module_init(hello_start);
module_exit(hello_end);
Place the following content in a Makefile under the same directory: 
obj-m = hello.o
Building such a module is pretty straightforward, but at this point, it won t work yet. Let me enter some details first. The building of a module When you normally build the above module, the kernel build system creates a hello.mod.c file, which content can create several kind of problems: 
MODULE_INFO(vermagic, VERMAGIC_STRING);
VERMAGIC_STRING is derived from the UTS_RELEASE macro defined in include/generated/utsrelease.h, generated by the kernel build system. By default, its value is derived from the actual kernel version, and git repository status. This is what setting KERNELRELEASE when configuring the kernel above modified. If VERMAGIC_STRING doesn t match the kernel version, loading the module will lead to the following kind of message in dmesg: 
hello: version magic '3.0.21-perf-ge728813-00399-gd5fa0c9' should be '3.0.8-perf'
Then, there s the module definition. 
struct module __this_module
__attribute__((section(".gnu.linkonce.this_module"))) =  
 .name = KBUILD_MODNAME,
 .init = init_module,
#ifdef CONFIG_MODULE_UNLOAD
 .exit = cleanup_module,
#endif
 .arch = MODULE_ARCH_INIT,
 ;
In itself, this looks benign, but the struct module, defined in include/linux/module.h comes with an unpleasant surprise: 
struct module
 
        (...)
#ifdef CONFIG_UNUSED_SYMBOLS
        (...)
#endif
        (...)
        /* Startup function. */
        int (*init)(void);
        (...)
#ifdef CONFIG_GENERIC_BUG
        (...)
#endif
#ifdef CONFIG_KALLSYMS
        (...)
#endif
        (...)
(... plenty more ifdefs ...)
#ifdef CONFIG_MODULE_UNLOAD
        (...)
        /* Destruction function. */
        void (*exit)(void);
        (...)
#endif
        (...)
This means for the init pointer to be at the right place, CONFIG_UNUSED_SYMBOLS needs to be defined according to what the kernel image uses. And for the exit pointer, it s CONFIG_GENERIC_BUG, CONFIG_KALLSYMS, CONFIG_SMP, CONFIG_TRACEPOINTS, CONFIG_JUMP_LABEL, CONFIG_TRACING, CONFIG_EVENT_TRACING, CONFIG_FTRACE_MCOUNT_RECORD and CONFIG_MODULE_UNLOAD. Start to understand why you re supposed to use the exact kernel headers matching your kernel? Then, the symbol version definitions: 
static const struct modversion_info ____versions[]
__used
__attribute__((section("__versions"))) =  
	  0xsomehex, "module_layout"  ,
	  0xsomehex, "__aeabi_unwind_cpp_pr0"  ,
	  0xsomehex, "printk"  ,
 ;
These come from the Module.symvers file you get with your kernel headers. Each entry represents a symbol the module requires, and what signature it is expected to have. The first symbol, module_layout, varies depending on what struct module looks like, i.e. depending on which of the config options mentioned above are enabled. The second, __aeabi_unwind_cpp_pr0, is an ARM ABI specific function, and the last, is for our printk function calls. The signature for each function symbol may vary depending on the kernel code for that function, and the compiler used to compile the kernel. This means that if you have a kernel you built from source, modules built for that kernel, and rebuild the kernel after modifying e.g. the printk function, even in a compatible way, the modules you built initially won t load with the new kernel. So, if we were to build a kernel from the hopefully close enough source code, with the hopefully close enough configuration, chances are we wouldn t get the same signatures as the binary kernel we have, and it would complain as follows, when loading our module: 
hello: disagrees about version of symbol symbol_name
Which means we need a proper Module.symvers corresponding to the binary kernel, which, at the moment, we don t have. Inspecting the kernel Conveniently, since the kernel has to do these verifications when loading modules, it actually contains a list of the symbols it exports, and the corresponding signatures. When the kernel loads a module, it goes through all the symbols the module requires, in order to find them in its own symbol table (or other modules symbol table when the module uses symbols from other modules), and check the corresponding signature. The kernel uses the following function to search in its symbol table (in kernel/module.c): 
bool each_symbol_section(bool (*fn)(const struct symsearch *arr,
                                    struct module *owner,
                                    void *data),
                         void *data)
 
        struct module *mod;
        static const struct symsearch arr[] =  
                  __start___ksymtab, __stop___ksymtab, __start___kcrctab,
                  NOT_GPL_ONLY, false  ,
                  __start___ksymtab_gpl, __stop___ksymtab_gpl,
                  __start___kcrctab_gpl,
                  GPL_ONLY, false  ,
                  __start___ksymtab_gpl_future, __stop___ksymtab_gpl_future,
                  __start___kcrctab_gpl_future,
                  WILL_BE_GPL_ONLY, false  ,
#ifdef CONFIG_UNUSED_SYMBOLS
                  __start___ksymtab_unused, __stop___ksymtab_unused,
                  __start___kcrctab_unused,
                  NOT_GPL_ONLY, true  ,
                  __start___ksymtab_unused_gpl, __stop___ksymtab_unused_gpl,
                  __start___kcrctab_unused_gpl,
                  GPL_ONLY, true  ,
#endif
         ;
        if (each_symbol_in_section(arr, ARRAY_SIZE(arr), NULL, fn, data))
                return true;
        (...)
The struct used in this function is defined in include/linux/module.h as follows: 
struct symsearch  
        const struct kernel_symbol *start, *stop;
        const unsigned long *crcs;
        enum  
                NOT_GPL_ONLY,
                GPL_ONLY,
                WILL_BE_GPL_ONLY,
          licence;
        bool unused;
 ;
Note: this kernel code hasn t changed significantly in the past four years. What we have above is three (or five, when CONFIG_UNUSED_SYMBOLS is defined) entries, each of which contains the start of a symbol table, the end of that symbol table, the start of the corresponding signature table, and two flags. The data is static and constant, which means it will appear as is in the kernel binary. By scanning the kernel for three consecutive sequences of three pointers within the kernel address space followed by two integers with the values from the definitions in each_symbol_section, we can deduce the location of the symbol and signature tables, and regenerate a Module.symvers from the kernel binary. Unfortunately, most kernels these days are compressed (zImage), so a simple search is not possible. A compressed kernel is actually a small bootstrap binary followed by a compressed stream. It is possible to scan the kernel zImage to look for the compressed stream, and decompress it from there. I wrote a script to do decompression and extraction of the symbols info automatically. It should work on any recent kernel, provided it is not relocatable and you know the base address where it is loaded. It takes options for the number of bits and endianness of the architecture, but defaults to values suitable for ARM. The base address, however, always needs to be provided. It can be found, on ARM kernels, in dmesg: 
$ adb shell dmesg   grep "\.init"
<5>[01-01 00:00:00.000] [0: swapper]      .init : 0xc0008000 - 0xc0037000   ( 188 kB)
The base address in the example above is 0xc0008000. If like me you re interested in loading the module on an Android device, then what you have as a binary kernel is probably a complete boot image. A boot image contains other things besides the kernel, so you can t use it directly with the script. Except if the kernel in that boot image is compressed, in which case the part of the script that looks for the compressed image will find it anyways. If the kernel is not compressed, you can use the unbootimg program as outlined in this old post of mine to get the kernel image out of your boot image. Once you have the kernel image, the script can be invoked as follows: 
$ python extract-symvers.py -B 0xc0008000 kernel-filename > Module.symvers
Symbols and signature info could also be extracted from binary modules, but I was not interested in that information so the script doesn t handle that. Building our module Now that we have a proper Module.symvers for the kernel we want to load our module in, we can finally build the module: (again, assuming arm-eabi-gcc is in your PATH, and that you have a shell opened in the kernel source directory) 
$ cp /path/to/Module.symvers build/
$ make M=/path/to/module/source ARCH=arm CROSS_COMPILE=arm-eabi- O=build modules
And that s it. You can now copy the resulting hello.ko onto the device and load it. and enjoy 
$ adb shell
# insmod hello.ko
# dmesg   grep insmod
<6>[mm-dd hh:mm:ss.xxx] [id: insmod]Hello world
# lsmod
hello 586 0 - Live 0xbf008000 (P)
# rmmod hello
# dmesg   grep rmmod
<6>[mm-dd hh:mm:ss.xxx] [id: rmmod]Goodbye world

Richard Hartmann: The problem with copyright maximalism

The gradual descent into a [permission culture](http://www.ibiblio.org/ebooks/Lessig/Free_Culture/Free Culture.htm#pxiii) has many ugly heads. Be they copyright maximalism, patent wars, cementing antiquated business models by suing, or a myriad of other things. One of said ugly heads is that people pretend to have a copyright just to milk others. The name Righthaven will be known to most people as a prime example of this going badly for the offenders. From how it looks, the people behind Righthaven, a company founded just to sue others, are being pursued through that shell. Many of you will know the infamous "You have to watch this on YouTube as Rightster filed a copyright claim" notice. What this means is that YouTube will (try to) display ads and Rightster will get a share of any revenue. I can't comment on the validity of those claims, but some of them struck me as dubious in the past. We reached a new low, though. Scripps Local News, which publishes world-renowned papers such as Ventura County Star and Corpus Christi Caller-Times, temporarily blocked NASA's video of Mission Control during MSL's seven minutes of radio silence with their copyright claim. As NASA-produced video featuring NASA, published on NASA's YouTube account. Blocked: Turns out something is borked here and I don't have time to fix it right now. I uploaded it to imgur as a quick fix. The video has been made accessible again in the meantime, but this is, yet again, proof that we are sliding into a society where content and knowledge is locked down by default. Don't let that happen. Pester your politicians, inform your friends, participate in demonstrations, and don't give up. It's our culture, not theirs.

5 August 2012

Stefano Zacchiroli: bits from the DPL for July 2012

Monthly DPL bits, fresh from the oven.
Tip to feel good about the release #476: before reading this, grab and fix one of the RC bugs affecting Wheezy. Done? Now you're ready for a slightly less exciting report of DPL activities. Highlights Assets Logo & trademark Note that the actual logo relicensing should be done by SPI, via their board, upon request of mine (as Debian Project liaison). We won't make it for the next SPI board meeting, as it is on 4 days away. But we can aim for the subsequent meeting, on September 13th. If all goes well (big "if"), we will enjoy a DFSG-free logo after that date. Internal organization Recent flurry of re-organization in the tech-ctte --- which I somewhat triggered pushing for periodic meetings --- seems to be proceeding well. I'm very happy about it, as we all need to trust that tech-ctte decisions will be not only sound, but also prompt. If you're interested into this topic, some recent evidence of the ongoing reorganization and its result can be found in their DebConf12 BoF, minutes of the last meeting, a set of forthcoming GRs, and the recent great decision of posting decisions results to d-d-a (as it happened for node/nodejs). Kudos to tech-ctte members for the recent activism! I haven't worked on it myself directly, but I highlight Enrico's work on server side archival of NM conversations. It has the potential of enabling automatic detection of stuck NM processes. I'm a bit rusty as AM now, but I think missing that ability is one of the main remaining causes of frustration when joining Debian. So, if you are an AM, please opt-in and use this feature with your appicants. If you are not an AM why not? :-) Miscellanea Some misc legal stuff: and some misc "political" stuff: Happy RC bug squashing,
Cheers.
PS the boring day-to-day activity log for July 2012 is available at master:/srv/leader/news/bits-from-the-DPL.txt.201207

Craig Small: Context sensitive menus with GtkTreeView

A project I now maintain called GJay uses the GTK+ toolkit and has a GtkTreeView that shows the directories and the songs in them. I was wanting to add a context sensitive menu on that view so that you could do things to the item you have right-clicked on; generally file specific things like give me information about this song or queue it to play. My only problem was, how to do it? A few Google searches showed things that were almost, but not quite, what I was after. The main problem being either the documentation was incomplete or it assumed I was after the selected field (the row I may of left-clicked on) and not the one I right-clicked on to get this event. This GtkTreeView uses a store where NAME_COLUMN is defined at 0, meaning it is the first column. The program uses this column to store the filename of the song. So first I needed to create an event to kick things off, we need to connect the button-press-event to our tree_view, which calls a function in this case view_on_button_pressed. 
 g_signal_connect(tree_view, "button-press-event", 
   G_CALLBACK(view_on_button_pressed), NULL);
gboolean view_on_button_pressed(GtkWidget *tree_view, 
                                GdkEventButton *event, 
                                gpointer user_data)  
  GtkTreeSelection *selection;
  GtkTreePath *tree_path;
  GtkTreeModel *tree_model;
  GtkTreeIter tree_iter;
  gchar *name;
 
  if (event-&gt;type == GDK_BUTTON_PRESS &amp;&amp; event-&gt;button == 3)  
    tree_model = gtk_tree_view_get_model(GTK_TREE_VIEW(tree_view));
    if (gtk_tree_view_get_path_at_pos(GTK_TREE_VIEW(tree_view),
          (int)event-&gt;x, (int)event-&gt;y,
          &amp;tree_path, NULL, NULL, NULL))  
      if (gtk_tree_model_get_iter(tree_model, &amp;tree_iter, tree_path))  
        gtk_tree_model_get(tree_model, &amp;tree_iter, NAME_COLUMN, &amp;name, -1);
        g_print("rc name is %s\n", name);
So a breakdown of what my callback is doing: The program, now that it knows the song title, can then go off and generate a menu for that song.
Related articles

Vasudev Kamath: Rewriting SILPA with Flask microframework

As a introduction on what is SILPA you can read my previous post This application was written by Santhosh about 6-7 years ago in May 2010 2009 ( Correct me if I'm wrong Santhosh ;-) Thanks to Santhosh for correcting me ) in Python without using any web framework. He crafted whole system single handed and with very nice ideas. He wrote all the modules available in it for all possible Indian languages even though he is native Malayalam speaker. I joined him 3 years back and I was providing some or other improvements and he ultimately made me as the Co-Developer of SILPA. Recently I came across the Flask microframework and thought of rewriting SILPA to use Flask. You might ask me why Flask? So here are my reasonings 
 1. Why reinvent Python webframework when we can use existing? Besides maintaining a not so well known framework code which is not written by you will be difficult    and time consuming.
 2. All Python modules are useful even outside SILPA project why make them dependent on SILPA code?
 3. Templating system used is very old and can't be ported to new Python version lets use something which is new and portable
 4. If new contributor wanted to add a module he need to understand SILPA framework to write and integrate it. This might be discouraging new
    contributions so lets use pure python module and make integration with SILPA straight forward.
 5. I was overwhelmed by Flask :P
I had previously tried bottle microframework but for some reason I couldn't crack it so choosing of Flask is not because it was better among other available microframework or something like that. I choose it because I could easily understand it and it suited my needs better. So I proposed about this to Santhosh and he asked me for a POC code. I started working on April 8 2012 and today by August 5 I can say I've first stable version of SILPA rewritten using Flask deployable over WSGI on Apache2. It can also be deployed on other servers supporting WSGI interface. It took me quite a bit of time but to be frank in the middle I completely had stopped coding because of some other works. Here is the commit graph and code frequency graphs from Github.

Changes in New Version Here is some major changes in current version. 
1. Newly developed framework provides plug and play features for new modules.
2. You need to install the modules which you want to run on it separately using PIP and only minimal code change is required. Only configuration and webbridge.py 
   will be changed. Ofcourse you need to add html file to *templates* folder.
3. This version uses decent routing logic from Flask for page serving.
4. It also provides JSONRPC interface which is compatible with previous version but only with method name changed.
5. This version separates out run time module loading and routing the request to modules.
6. Dynamic module loading now happens only at the beginning of application, so if I want to implement caching for particular services it will be straight forward
   now.
7. Routing requests to proper module is adapted from old version with few modifications
8. All previous pages are now converted to use *Jinja2* template system.
Things Remaining TODO Below is some stuffs still pending before this version can completely replace http://silpa.org.in 
1. Separate all modules from original SILPA and get it uploaded to pypi
2. Provide RESTful interface
3. Provide proper LICENSE, AUTHORS file for all modules 
4. Prepare a Nice UI
5. Find a better name than *SILPA*
...
Update 
Forgot to provide the github link and demo link for the new version of SILPA. Thanks to Aravinda for pointing
it out. Here it goes
Github: http://github.com/copyninja/Silpa-Flask
Demo: http://flasksilpa-indic.rhcloud.com

4 August 2012

Gerfried Fuchs: Mermaids Reloaded, Addendum

First things first: The issue with the feedback form is fixed. Sorry for the inconvenience. I'm not sure if you followed the schedule of this year's debconf closely, but there was an International Poetry Night happening. Given that Mermaids still is quite important for me, but also, that I like to write poetry every now and then, I had the idea spinning around in my head to present Mermaids there. The simple thought of that made me extremely nervous. Things happened as they liked to happen: the video streaming setup for debconf still was giving me a bit of headaches, and suddenly it was too late to print out the poem because I didn't even had CUPS installed on my new laptop. Also, having reread it recently I noticed that it wasn't flowing too well, so I was uncertain whether it would be possible to properly perform it. I attended nevertheless. It was rather cosy in El Panal, but still a fair amount of people there. And to my surprise, quite a lot of Debian people were presenting poetry. Some in Spanish, some in English, but also one in Japanese, one in German and even one Esperanto. I had the feeling that I really should present something too. I still was extremely nervous, but during the day I had opened some of my poems in the browser, so I started to write down The Girl and the Boy, and when Fito called me to the microphone, I think most people were able to tell how nervous I was about it. But it went well. It even went so well that I felt the need to perform another poem. I chose that's what friends are for, quickly scribbling it down in bad light. The choice was easy, given that it is one that means a lot to me and that an haiku already got presented. It was really a nice experience to not only write these poems and publishing them on deviantART and in my blog, but also to present them to a live audience. Then there was a break. Actually I thought it was over already, but they started a second round. And I somehow liked the way it worked out, so I started to dig for something else to present. I settled for Strange and Jonathan insisted on translating it into Spanish and present that version, too. His version wasn't in Haiku style, but I think it still was a very nice idea. It seem to have been received well, but given that I don't speak Spanish I can only hope he was able to catch the feelings that I did put into this short piece. I am confident he did well. :) As final piece I settled for another short piece, this time it was in German, Wahre Liebe. I think I was able to transport the feelings of it with adding pauses in certain spots, although I fear most people didn't get it because it was ... well, in German. Again, a fair amount of the people who presented poetry were from Debian, actually more than half of it I think. And I am wondering: When will the CfP for next year's debconf open so we can try to establish this event as regular debconf event? And who knows, maybe I'll find the courage then to perform Mermaids.

/personal permanent link Comments: 0 Flattr this

Russell Coker: Love of Technology at First Sight

After seeing the Retina display I ve been thinking about the computer products that I ve immediately desired. Here is the list of the ones I can still remember:
  1. My first computer which was the TEC-1 [1], in 1982 or 1983.
  2. A computer with a full keyboard and a monitor (Microbee), in about 1984. A hex-only keypad is very limiting.
  3. Unix, initially SunOS 4.0 in 1991. Primarily the benefits of this were TCP/IP networking, fast email (no multi-day delay for Fidonet mail), IRC, and file transfer from anywhere in the world. Not inherent benefits to Unix, but at the time only Unix systems did TCP/IP at all well.
  4. OS/2 2.0 in 1992. At the time OS/2 had the best GUI of any system available (IMHO) and clearly the best multitasking of DOS and Windows programs.
  5. Linux in 1992. I started with the TAMU and MCC Interim distributions and then moved to SLS when it was released. The first kernel I compiled was about 0.52. At the time the main use of Linux for almost everyone was to learn about Unix and compile kernels. In 1993 I started running a public access Linux server.
  6. Trinitron monitors in 1996. I first saw an IBM Trinitron monitor when working on an IBM project and had to buy one for home use, at the time a 17 Trinitron monitor beat the hell out of any other display device that one could reasonably afford. A bigger screen allowed me to display more code at once which allowed easier debugging.
  7. Thinkpad laptops from 1998 until now. They just keep working well and seem to be better than other products every time I compare them. I also like the TrackPoint. 1998 was when a Thinkpad dropped to a mere $3,800 for a system that could run with 96M of RAM, enough compute power for the biggest compiles and it cost less than most cars!
  8. The KDE desktop environment in 1998. In 1998 I switched my primary workstation from a PC running OS/2 to a Thinkpad running Linux because of KDE. Prior to KDE nothing on Linux was user-friendly enough.
  9. The iPaQ hand-held PC. I got one in 2002 and ran the Familiar distribution of Linux on it. I had it running SE Linux and used it for writing an article for Linux Journal. Being able to get a computer out on public transport to do some work really saved some time. In some ways the iPaQ hardware and the Familiar OS beat modern Android systems.
  10. The EeePC 701 which I bought in 2008 [2]. In the last 4 years someone has probably released a system that s no larger or heavier and has the same amount of compute power (enough for web browsing, email, and ssh). But most Netbooks that I ve seen don t compete. The EeePC allowed me to take laptops to places where it previously wasn t convenient.
  11. Android, before using Android I never had a smart phone that I used for anything other than taking photos. The other smart phone OSs are either locked down or don t have the app support that Android has. I listed lots of problems with my first phone the Sony Ericsson Xperia X10, but I still really enjoyed using it a lot [3]. Since getting an Android phone I ve read a lot of email while on the go, this means I can respond faster when necessary and use time that might otherwise be wasted. The ssh client means that I don t need to carry a laptop with me when there s a risk that emergency sysadmin work may be required.
  12. Cheap rented servers, Amazon defined cloud computing with EC2, Linode offers great deals for small virtual servers, and Hetzner offers amazing deals on renting entire servers. Getting your own Internet connection or running your own physical server in someone s data-center is a lot of effort and expense. Being able to just rent servers is so much easier and allows so many new projects. I can t remember when I first started using such services, maybe 5 years ago.
  13. The Apple Retina Display [4] a few days ago.
For the period between 1998 and 2008 I can t think of anything that really excited me apart from the iPaQ. Computers became a lot smaller, faster, cheaper, etc. But it was never a big exciting change. The AMD64 architecture wasn t particularly exciting as most systems didn t need more than 4G of RAM and the ones that did could use PAE. What are the most exciting computer products you have seen? Related posts:
  1. Old PDA vs New Mobile Phone for PDA use Since about 2002 I have been using a iPaQ [1]...
  2. CyanogenMod and the Galaxy S Thanks to some advice from Philipp Kern I have now...
  3. Moving from a Laptop to a Cloud Lifestyle My Laptop History In 1998 I bought my first laptop,...

Andreas Metzler: proofreading would have helped.

I have recently read ECB/2011/14 (GUIDELINE OF THE EUROPEAN CENTRAL BANK of 20 September 2011 on monetary policy instruments and procedures of the Eurosystem) and have stumbled upon the following gem.
An institution may access the Eurosystem's standing facilities and open market operations based on standard tenders only through the NCB of the Member State in which it is incorporated. If an institution has establishments (its head office or branches) in more than one Member State, each establishment has access to these operations through the NCB of the Member State in which it is established, notwithstanding the fact that the bids of an institution may only be submitted by one establishment (either the head office or a designated branch) in each Member State.
At first glance this fine, it is easily readable. (All abbreviations are explained at the start of document.) However a more attentive examination corrects this impression. Compare the two highlighted versions below:
An institution may access the Eurosystem's standing facilities and open market operations based on standard tenders only through the NCB of the Member State in which it is incorporated. If an institution has establishments (its head office or branches) in more than one Member State, each establishment has access to these operations through the NCB of the Member State in which it is established [...].
An institution may access the Eurosystem's standing facilities and open market operations based on standard tenders only through the NCB of the Member State in which it is incorporated. If an institution has establishments (its head office or branches) in more than one Member State, each establishment has access to these operations through the NCB of the Member State in which it is established [ ].
Perfect contradiction. Either only the "own" NCB works or any NCB where we have a establishments. Ex falsum quodlibet and Bob's your uncle. ;-)

3 August 2012

Paul Tagliamonte: monomoy (the next generation of qlight)

I ve been working on a replacement to qlight since I ve blogged about it something a bit more self-contained and clean. I ve settled on the name monomoy, since I had a lot of the ideas i ve been using come to me on the real monomoy. In particular, I d like for it to support having remote build-nodes, and non-archive upload patterns I d like to be able to upload pkg-foo/1.1.2-1 (UNRELEASED) 20 times without bumping the changelog. Now, my approach and local test-code has been unorthodox. I ve decided to throw caution to the wind, and store as much as I can in mongodb, including parsed debian control files. I ve recently (this afternoon) added support for build queues, which I hope to build out. More to come once it s sane, but it s working-ish!

1 August 2012

Bdale Garbee: Early Retirement

I am pleased to announce that HP accepted my application for participation in this year's Enhanced Early Retirement program. After 26.5 years, my last day at HP will be 31 August 2012. This isn't because I have a problem with HP, quite the contrary! I feel immensely privileged to have worked on so many interesting things with so many smart people for so long. And there are many exciting things happening right now, including high profile projects like Odyssey and Moonshot expanding both ends of the server scaling continuum with Linux, and the immense commitment to and investment in Open Source represented by the use of Open Stack and Ubuntu LTS in building HP Cloud. The temptation to stay and continue to help guide HP's Open Source engagement was strong. I'm certainly under no pressure to leave... in fact, I think I managed to shock my management with my decision to retire! But I'm very excited about "taking a break" from full-time employment for a while to focus more time on things that matter a lot to me. My son and I have a number of projects we'd like to work on together as he enters high school. The side business I started two years ago with Keith Packard to design, build, and sell avionics solutions for high power model rockets is growing and could use more attention. I'm in the midst of designing the processor board for another amateur radio satellite. And of course, I'm looking forward to putting more "quality time" towards Free Software projects like FreedomBox and Debian! Since they do not depend on my employment status, I do not expect any immediate changes in the various roles I play outside of HP. I will continue to serve on the board of the Linux Foundation representing individual affiliates, on the board of the FreedomBox Foundation, as Chairman of the Debian Technical Committee, and as President of Software in the Public Interest. And while I hope to travel less than I have in recent years, I look forward to continuing to speak and otherwise participate in key Free Software conferences around the world from time to time. After I retire, the primary point of contact with HP on Open Source matters will be the Director of HP's Open Source Program Office, Phil Robb. But if there are open issues regarding HP and Open Source that I can help with over the next month, feel free to contact me and I'll see what I can do to help.

Rapha&#235;l Hertzog: My Debian Activities in July 2012

This is my monthly summary of my Debian related activities. If you re among the people who made a donation to support my work (72.65 , thanks everybody!), then you can learn how I spent your money. Otherwise it s just an interesting status update on my various projects. This month has been a short one since I have been away for 2 weeks of vacation. Dpkg My dpkg work encompasses a bunch of small tasks: Packaging I updated nautilus-dropbox to version 1.4.0 and python-django-registration to version 0.8. Both have been uploaded to unstable and I initially wanted to request an unblock for the latter, but it turns out it has gained reverse dependencies and version 0.8 introduces API changes so it s not an option at this point of the freeze. QA work I investigated and fixed #678356 where it had been reported that the PTS static news were no longer working as expected. At the start of the month, I also unblocked the mostly-unknown but important mole service it was out of date of several weeks and several people were annoyed that the information about new upstream versions was no longer up-to-date. Vacation Almost no Debian work during my vacation but the lack of Wifi nearby made me look for solutions to connect my computer through my Nokia N900 3G/GPRS connection. I discovered the Mobile Hotspot application (homepage) and it worked like a charm (although it required Maemo s non-default devel repository to be able to install the alternative kernel for power users ). The Debian Handbook Michal iha proposed us to host a Weblate instance to help translate the book with a web interface. He kindly agreed to implement some improvements to better suit my requirements. Those have been completed and the weblate instance is now live at debian.weblate.org. There s no requirement to use Weblate for translations teams but for those that do, it sure makes it easier to recruit volunteers who have no prior knowledge of Git and PO files. If you want to help, please checkout this page first though, you should not start using Weblate without getting in touch with the respective translations teams. Apart from translations, I also had the pleasure to merge some patches from Philipp Kern who improved the section covering IPv6 and a few other parts. We can make the book even better if more people share their expertise in the part of the book where they know better than me and Roland. :-) Thanks See you next month for a new summary of my activities.

No comment Liked this article? Click here. My blog is Flattr-enabled.

Dirk Eddelbuettel: RcppCNPy 0.2.0

Version 0.2.0 of the recently introduced RcppCNPy package for reading/writing NumPy data in R arrived on CRAN earlier today. The main change are the added ability to also write gzip-ed npy files, to suppress an automatic transposition as well as the correction of one memory leak on data read. The NEWS file entry is below:
Changes in version 0.2.0 (2012-07-30)
  • Support for writing of gzip-ed npy files has been added.
  • A new option dotranspose has been added to npyLoad() to support data sets that do not need to be transposed to be used in R.
  • A memory leak in reading files has been corrected.
CRANberries also provides a diffstat report for 0.2.0 relative to 0.1.0. As always, feedback is welcome and the rcpp-devel mailing list off the R-Forge page for Rcpp is the best place to start a discussion.

31 July 2012

Russell Coker: Cheap SATA Disks in a Dell PowerEdge T410

A non-profit organisation I support has just bought a Dell PowerEdge T410 server to be mainly used as a file server. We need a reasonable amount of space and really good reliability features because the system may have periods without being actively monitored, it also has to be relatively cheap. Dell servers are quite cheap, but disks are not cheap at all when Dell sells them. I get the impression that disks and RAM are major profit centers for Dell and that the profit margins on the basic servers are quite small. So naturally we decided to buy some SATA disks from a local store, one advantage of this is that Dell sells nothing bigger than 2TB while 3TB disks are available cheaply everywhere else. So we bought 4 cheap 3TB (2.7TiB) SATA disks, connected them to the server, and found that only 2TiB was accessible. The Dell Tech Center says that some of the RAID controllers don t support anything larger than 2TiB [1]. Obviously we have one of the older models. There s lots of SATA sockets on the motherboard that could be used, however there is one problem. View of the open side of a PowerEdge T410 The above picture is the side view of the T410, it was taken with a Samsung Galaxy S so the quality is a little poor (click for the original picture). The server is quite neat, not many cables for a system with 6 disks, not the 12 separate cables you would get in a typical white-box system. Disks in a PowerEdge T410 The above picture shows the disk enclosure. You can see that each disk has a single connector for power and data, also the disks aren t separate, multiple disks have the same power wires and the data cables are paired. SAS controller in a PowerEdge T410 Above you can see the SAS controller, it has two large connectors that can each handle the data cables for 4 disks, nothing like the standard cables. It s easy to buy SATA data cables and connect them, but there are no spare power cables in the box. The connector that supplies power to all the disks appears to be something proprietary to Dell which goes straight to the double-connectors on each disk that supply power and data. This setup makes cabling very neat but also provides no good option for cabling regular disks. I m sure I could make my own cables and if I hunted around the net enough I could probably buy some matching power cables, but it would be a hassle and the result wouldn t be neat. So the question was then whether to go to more effort, expense, and possibly risk the warranty to get the full 3TB or to just use the SAS controller and get 2TiB (2.2TB) per disk. One factor we considered is the fact that the higher sector numbers typically give much slower access times due to being on shorter tracks (see my ZCAV results page for test results from past disks [2]). We decided that 2.2TB (2TiB) out of 3TB (2.7TiB) was adequate capacity and that losing some of the slow parts of the disk wasn t a big deal. I ve now setup a RAID-Z2 array on the disks, and ZFS reports 3.78TiB available capacity, which isn t a lot considering that we have 4*3TB disks in the array. But the old server had only 200G of storage, so it s a good improvement in capacity, performance, and RAID-Z2 should beat the hell out of RAID-6 for reliability. Related posts:
  1. Dell PowerEdge T105 Today I received a Dell PowerEDGE T105 for use by...
  2. ZFS vs BTRFS on Cheap Dell Servers I previously wrote about my first experiences with BTRFS [1]....
  3. How I Partition Disks Having had a number of hard drives fail over the...

Martin Pitt: My impressions from GUADEC

I have had the pleasure of attending GUADEC in full this year. TL;DR: A lot of great presentations + lots of hall conversations about QA stuff + the obligatory be er,ach = . Last year I just went to the hackfest, and I never made it to any previous one, so GUADEC 2012 was a kind of first-time experience for me. It was great to put some faces and personal impressions to a lot of the people I have worked with for many years, as well as catching up with others that I did meet before. I discussed various hardware/software testing stuff with Colin Walters, Matthias Clasen, Lennart Poettering, Bertrand Lorentz, and others, so I have a better idea how to proceed with automated testing in plumbing and GNOME now. It was also really nice to meet my fellow PyGObject co-maintainer Paolo Borelli, as well as seeing Simon Schampier and Ignacio Casal Quinteiro again. No need to replicate the whole schedule here (see for yourself on the interwebs), so I just want to point out some personal highlights in the presentations: There were a lot of other good ones, some of them technical and some amusing and enlightening, such as Frederico s review of the history of GNOME. On Monday I prepared and participated in a PyGObject hackfest, but I ll blog about this separately. I want to thank all presenters for the excellent program, as well as the tireless GUADEC organizer team for making everything so smooth and working flawlessly! Great Job, and see you again in Strasbourg or Brno!

Russell Coker: ZFS on Debian/Wheezy

As storage capacities increase the probability of data corruption increases as does the amount of time required for a fsck on a traditional filesystem. Also the capacity of disks is increasing a lot faster than the contiguous IO speed which means that the RAID rebuild time is increasing, for example my first hard disk was 70M and had a transfer rate of 500K/s which meant that the entire contents could be read in a mere 140 seconds! The last time I did a test on a more recent disk a 1TB SATA disk gave contiguous transfer rates ranging from 112MB/s to 52MB/s which meant that reading the entire contents took 3 hours and 10 minutes, and that problem is worse with newer bigger disks. The long rebuild times make greater redundancy more desirable. BTRFS vs ZFS Both BTRFS and ZFS checksum all data to cover the case where a disk returns corrupt data, they don t need a fsck program, and the combination of checksums and built-in RAID means that they should have less risk of data loss due to a second failure during rebuild. ZFS supports RAID-Z which is essentially a RAID-5 with checksums on all blocks to handle the case of corrupt data as well as RAID-Z2 which is a similar equivalent to RAID-6. RAID-Z is quite important if you don t want to have half your disk space taken up by redundancy or if you want to have your data survive the loss or more than one disk, so until BTRFS has an equivalent feature ZFS offers significant benefits. Also BTRFS is still rather new which is a concern for software that is critical to data integrity. I am about to install a system to be a file server and Xen server which probably isn t going to be upgraded a lot over the next few years. It will have 4 disks so ZFS with RAID-Z offers a significant benefit over BTRFS for capacity and RAID-Z2 offers a significant benefit for redundancy. As it won t be upgraded a lot I ll start with Debian/Wheezy even though it isn t released yet because the system will be in use without much change well after Squeeze security updates end. ZFS on Wheezy Getting ZFS to basically work isn t particularly hard, the ZFSonLinux.org site has the code and reasonable instructions for doing it [1]. The zfsonlinux code doesn t compile out of the box on Wheezy although it works well on Squeeze. I found it easier to get a the latest Ubuntu working with ZFS and then I rebuilt the Ubuntu packages for Debian/Wheezy and they worked. This wasn t particularly difficult but it s a pity that the zfsonlinux site didn t support recent kernels. Root on ZFS The complication with root on ZFS is that the ZFS FAQ recommends using whole disks for best performance so you can avoid alignment problems on 4K sector disks (which is an issue for any disk large enough that you want to use it with ZFS) [2]. This means you have to either use /boot on ZFS (which seems a little too experimental for me) or have a separate boot device. Currently I have one server running with 4*3TB disks in a RAID-Z array and a single smaller disk for the root filesystem. Having a fifth disk attached by duct-tape to a system that is only designed for four disks isn t ideal, but when you have an OS image that is backed up (and not so important) and a data store that s business critical (but not needed every day) then a failure on the root device can be fixed the next day without serious problems. But I want to fix this and avoid creating more systems like it. There is some good documentation on using Ubuntu with root on ZFS [3]. I considered using Ubuntu LTS for the server in question, but as I prefer Debian and I can recompile Ubuntu packages for Debian it seems that Debian is the best choice for me. I compiled those packages for Wheezy, did the install and DKMS build, and got ZFS basically working without much effort. The problem then became getting ZFS to work for the root filesystem. The Ubuntu packages didn t work with the Debian initramfs for some reason and modules failed to load. This wasn t necessarily a show-stopper as I can modify such things myself, but it s another painful thing to manage and another way that the system can potentially break on upgrade. The next issue is the unusual way that ZFS mounts filesystems. Instead of having block devices to mount and entries in /etc/fstab the ZFS system does things for you. So if you want a ZFS volume to be mounted as root you configure the mountpoint via the zfs set mountpoint command. This of course means that it doesn t get mounted if you boot with a different root filesystem and adds some needless pain to the process. When I encountered this I decided that root on ZFS isn t a good option. So for this new server I ll install it with an Ext4 filesystem on a RAID-1 device for root and /boot and use ZFS for everything else. Correct Alignment After setting up the system with a 4 disk RAID-1 (or mirror for the pedants who insist that true RAID-1 has only two disks) for root and boot I then created partitions for ZFS. According to fdisk output the partitions /dev/sda2, /dev/sdb2 etc had their first sector address as a multiple of 2048 which I presume addresses the alignment requirement for a disk that has 4K sectors. Installing ZFS deb http://www.coker.com.au wheezy zfs I created the above APT repository (only AMD64) for ZFS packages based on Darik Horn s Ubuntu packages (thanks for the good work Darik). Installing zfs-dkms, spl-dkms, and zfsutils gave a working ZFS system. I could probably have used Darik s binary packages but I think it s best to rebuild Ubuntu packages to use on Debian. The server in question hasn t gone live in production yet (it turns out that we don t have agreement on what the server will do). But so far it seems to be working OK. Related posts:
  1. Discovering OS Bugs and Using Snapshots I m running Debian/Unstable on an EeePC 701, I ve got an...
  2. Starting with BTRFS Based on my investigation of RAID reliability [1] I have...
  3. ZFS vs BTRFS on Cheap Dell Servers I previously wrote about my first experiences with BTRFS [1]....

30 July 2012

Russell Coker: Links July 2012

The New York Times has an interesting article about hacker hostels [1]. I had an idea for similar things after watching a Japanese movie about Tokiwa-s a shared apartment for Manga artists which among others inspired the creator of Astro Boy [2]. The TED blog has an interesting interview with William Noel about open access to art and historical data [3]. Most of his talk concerns an Archimedes codex which has been recovered and published on the Internet. He advocates publishing all manner of art and historical data under a Creative Commons license. The education system is often criticised for trying too hard to make children feel successful and not teaching them skills needed to be successful, it seems that the US military fails the same way in it s war games [4]. Webroot.com published an interesting article last year about the first BIOS rootkit in the wild [5]. I really wish that they would design motherboards with a switch to enable BIOS writing which would default to off . I recently did a poll at a LUG meeting and found that only half the audience had updated the BIOS on most systems they owned, if the most technical people generally don t need a dangerous feature then it should probably be disabled by default. Matthew Wright wrote an interesting article about the costs of upgrading the electricity grid in Australia vs the costs of upgrading air-conditioners [6]. It seems that it would be a lot cheaper for the government to buy everyone a new air-conditioner than to upgrade the grid. Owl City has a post of 10 Myths About Introverts [7]. That could probably be titled 10 Myths About Aspies and still be correct. Susan Cain gave the most popular talk of TED 2012 about The Power of Introverts , here is an interesting interview about the talk and Introversion [8]. Related posts:
  1. Links April 2012 Karen Tse gave an interesting TED talk about how to...
  2. Links July 2011 The Reid Report has an article about the marriage pledge...
  3. Links March 2012 Washington s Blog has an informative summary of recent articles about...

Next.

Previous.