Search Results: "tar"

24 April 2012

Robert Collins: Less SPOFs: pyjunitxml, testscenarios

I ve made the Testtools committers team own both the project and the trunk branch for both pyjunitxml and testscenarios. This removes me as a SPOF if anything needs doing in those projects any Testtools committer can now do it. (Including code review and landing). If you are a testtools committer and need PyPI release rights, ping me and I ll add you. (I wish PyPI had group management).

23 April 2012

Christian Perrier: Boston marathon 2012

I just read reports about Boston marathon 2012. Incredibly though race this year as the heat raised up to 32-35 C (around 90 F?) for the last runners and 26 C at 10:00 at marathon start. The outcome is indeed impressive: the 2012 winner won with a "not so good" time of 2h12 while last year saw an unofficial world best time of 2h03'02" (there can't be any world record in Boston because of the race being one-way, thus prone to too favourable wind, plus a negative slope between start and finish). Even though this year is less likely to see great records (because this is an olympic year and the best runners won't run another marathon before the Olympics), we still had times around 2h05 in Paris, Rotterdam....and this year's best time in London with 2h04'44" (Dubai time is 2h04'23" but I do not count this one as a mass marathon). So, 2h12 in Boston just says how hard this race was. But, still, that's my dream marathon, even more than NYC marathon. And I'll do it, I promise....though I now have to run below 3h30 to qualify....unless I'm still running marathons when I reach 55, where the qualificaiton mark will be only 3h40 (which is exactly my PB as of now). Will I still be able to run in 3h40 in 4 years? Or will I succeed in running in less than 3h30 in the meantime? We'll see..:-) Until then....my first marathon for this year will be on June 9th, in Caen, as planned. Target: less than 3h40, despite the quite hard profile (and possible windy conditions as the first half of the race is along the famous Normandy beaches code named Sword, Gold and Juno, between Courseulles sur Mer and Ouistreham). So, until then, I'll forget about my nice races in forests and plains, and concentrate on interval trainings, boring sequences of 400m or 1000m runs, etc, etc. That will be the first time I'm doing this but everybody keeps telling me I need to do that if I want to lower my PB (and every marathoner wants to!).

Clint Adams: Work in advertising!

The marketing industry divides itself up into a number of businesses. Some are boutique agencies, but most are owned by one of the world's four major holding companies. These conglomerates pursue an aggressive M&A strategy, snapping up smaller companies all the time. There are other sets of services, as well as variations on the above, particularly in terms of targeting certain age groups or ethnic groups, but also particular industries or sectors (movies, politics). There is direct marketing (spamming) through all the imaginable media. There is also paying people to blog about certain things without disclosing their sponsorship.

Russell Coker: Links April 2012

Karen Tse gave an interesting TED talk about how to stop police torture as an investigative tool [1]. Mostly it s about training and empowering public defenders. Phil Plait gave an interesting TED talk about how to defend the Earth from asteroids [2]. Julian Baggini wrote an interesting article for the Financial Times about the persecution of Atheists in the US [3]. Charlie Todd of Improv Everywhere gave an amusing TED talk about Improv events that he has run [4]. He is most famous for organising people to wear blue shirts and khaki pants in Best Buy, but he s done lots of other funny things. Paul Zak gave an interesting TED talk about trust, morality, and oxytocin [5]. One of the many interesting fact that he shared is that oxytocin levels can significantly increase when using social networking sites. So people who use Facebook etc are likely to be more trustworthy as well as more trusting. The Occupy the Judge Rotenberg Center movement aims to stop the torture of Autistic children in the US [6], Anonymous is involved in that too. Paul Lewis gave an insightful TED talk about the use of crowdsourced data in news reporting [7]. A lot of the analysis of citizen journalism is based on comparing bloggers with full-time paid journalists, but Paul describes how professional full-time journalistic work can be greatly assisted by random people filming and photographing things that seem noteworthy. Make sure your next phone has the best possible camera phone cameras will never be great but the quality of the camera you have with you is what matters. Sam Harris published an interesting interview with Tim Prowse who is a Baptist minister who faked belief for two years after becoming an atheist [8A]. He also references The Clergy Project a support group for atheists who are current or former members of the clergy [8B]. Cracked has an insightful article about 6 things that rich people need to stop saying [9]. How do the 1% not understand these things? Barack Obama and Nichelle Nichols (who played Lt Uhura in the original Star Treck) give the Vulcan Salute in the White-House [10]. Gabriel Arana wrote an insightful article about his experiences with the ex-gay movement [11]. The therapist who hurt him so much is still doing the same to other victims. S#!T Ignorant People Say To Autistics is an interesting youtube video about ignorant and annoying people [12]. Strangely I ve received little of that myself, I wonder whether women on the Autism Spectrum get a lot more of that than men. At GManCaseFile an ex-FBI agent has written an informative post about how the TSA is failing [13]. The Nieder Family has an interesting article about how patents are being used to prevent the creation of assisted communication (AAC) devices for children [14]. Apparently the company that has the patents wants all AAC devices to be really expensive and profitable for them. This is yet another example of patents doing harm not good. Renew Economy has an informative article by Giles Parkinson about the affect that solar power generation will have on power prices [15]. In short as solar systems produce power when it s most needed (during the day and at the hottest time of the day for warm climates) it will dramatically reduce the auction price for wholesale power. That will hurt the business of the power companies and also allow lower prices on the retail market. Related posts:
  1. Links March 2012 Washington s Blog has an informative summary of recent articles about...
  2. Links February 2012 Sociological Images has an interesting article about the attempts to...
  3. Links April 2010 Sam Harris gave an interesting TED talk about whether there...

22 April 2012

Joey Hess: moving my email archives and packages to git-annex

I've recently been moving some important data into git-annex, and finding it simplifies things while also increasing my flexability. email archives I've kept my email archives in git for years. This works ok, just choose the right file format (compressed mbox) and number of files (one archive per mailbox per month or so) and git can handle this well enough, as email is not really large. But, email is not really small either. Keeping my email repository checked out on my netbook consumes 2 gigabytes of its 30 gigabyte SSD, half of which is duplication in .git. Also, I have only kept it at 2 gigabytes through careful selection of what classes of mail I archive. That made sense when archival disk was more expensive, but what makes sense these days is to archive everything. For a while I've wanted to have a "raw" archive, that includes all email I receive. (Even spam.) This protects against various disasters in mail filtering or reading. Setting that up was my impetus for switching my mail archives to git-annex today. The new system I've settled on is to first copy all incoming mail into a "raw" maildir folder. Then mailfilter sorts it into the folders I sync (with offlineimap) and read. Each day, the "raw" folder is moved into a mbox archive, and that's added to the git annex. Each month, the mail I've read is moved into a monthly archives, and added to the git annex. A simple script does the work. I counted the number of copies that existed of my mail when it was stored in git, and found 7 copies spread amoung just 3 drives. I decided to slim that back, and configured git-annex to require only 5 copies. But those 5 copies will spread amoung more drives, including several offline archival drives, so it will be more robust overall. My netbook will have an incomplete checkout of my mail, omitting the "raw" archive. If I need to peek inside a spam folder for a lost mail, I can quickly pull it down; if I need to free up space I can quickly drop older archives. This is the flexability that git-annex fans love. :) By the way, this also makes it easier to permanantly delete mail, when you really need to (ie, for contractual reasons). Before, I'd have to do a painful git-filter-branch if I needed to get rid of eg, mail for old jobs. Now I can git annex drop --force. Pro Tip: If you're doing this kind of migration to git-annex, you can save bandwidth by not re-transferring files to machines that already have a copy. I ran this command on my netbook to inject the archives it had in the old repository into the new repository, verifying checksums as it goes:
cd ~/mail/archive; find -type l -exec git annex reinject ~/mail.old/archive/    \;
Note on mairix compatability: I use mairix to index and search my mail. But it refuses to follow git-annex's symlinks to the content. So I have to point it at .git/annex/objects/. I also configured annex.backend to SHA256E, which keeps the extensions on my compressed mailbox files, which is necessary for mairix to realize they're compressed. debian packages I'd evolved a complex and fragile chain of personal apt repositories to hold Debian packages I've released. I recently got rid of the mess, which looked like this: dput local mini-dinstall repo dput→mini-dinstallrepo on my server →dput Debian The point of all that was that I could "upload" a package locally while offline and batch transfer it later. And I had a local and a public apt repository of just the packages I've uploaded. But these days, packages uploaded to Debian are available nearly immediately, so there's not much reason to do that. My old system also had a problem: It only kept the most recent single copy of each package. Again, disk is cheap, so I'd rather have archives of everything I have uploaded. Again I switched to git-annex. My new system is simplicity itself. I release a package by checking it into a "toupload" directory in my git annex repository on my netbook. Items in that directory are dput to Debian and moved to "released". I have various other clones of that repository, which I git annex move packages to periodically to free up SSD space. In the rare cases when I build a package on a server, I check it into the clone on the server, and again rely on git-annex to copy it around. Now, does anyone know a good way to download a copy of every package you're ever released from archive.debian.org? (Ideally as a list of urls I can feed to git annex addurl.) conclusion My email and Debian packages were the last large files I was not storing in git-annex. Even backups of my backups end up checked into git-annex and archived away. Now that I'm using git-annex in every place I can, my goal with it is to make it as easy as possible for as many of you to use it as possible, too. I have some inotify tricks up my sleeve that seem promising. Kickstarter may be involved. Watch this space!

Gregor Herrmann: RC bugs 2012/16

this week I've started to look at the (not yet release critical) gcc 4.7 bugs. luckily there are still enough patches for them in the BTS but at the time of writing, at least 138 are not fixed & don't have a patch yet. so still some work ahead.

& for those who prefer to fix debian/rules files, lucas' last archive rebuild (where build-arch was used) still leaves enough work, & many bugs are not difficult to fix with a basic understanding of makefiles & debian build sytems.

Christian Perrier: Bug #670000

Julien Cristau reported Debian bug #670000 on Sunday April 22th, against libgtkada. This bug is indeed part of a MBF for packages that are "marked as Multi-Arch: same, but contain files in arch-independent paths with arch-specific contents" (doh, these multi-arch things are sometimes giving me headaches!) Bug #660000 was reported as of February 15th: 2 months and 7 days for 10,000 bugs. As expected, QA work and release preparation do trigger an increase in bug report rate and we're now again quite close from the "10,000 bugs in two months mark". I forgot to report that bug #666666 has been reported in the meantime. This time by one of the top bug reporters in Debian, namely Lucas Nussbaum, during one of his archive-wide package rebuilds, against psimedia. Next step will be bug #680000, targeted in June 2012. With this rate, bug #700000 should be reported around mid-November 2012 and Kartik Mistry would win the Bug #700000 prediction contest". In the meantime, Launchpad will have reached bug #1000000 (the highest bug number is currently #986849....which will be no longer true by the time you read this). The rate in LP is currently about 20,000 bugs per month. I wonder how many of these are closed as I feel like this is humanly impossible to process that many bug reports.

Steve Kemp: Work on my bathroom continues

Work on my bathroom continues. I had expected it to be complete on Friday but it lookse like the new completion date will be Tuesday or Wednesday. It will be very nice to have a working shower, and a toilet I don't need to "flush" with a bucket. The only interesting thing I've been doing is planning on a kernel to support long-term for some poor unfortunate souls who cannot use Debian-based kernels. I know building, supporting, and maintaining a kernel will be a pain. But at the moment it looks like a valid thing to be suggesting. Otherwise life goes on. I've been hit by the DIY bug, largely as the result of seeing all the bathroom work going on, and figuring I was going to live with dust/mess anyway - so I've erected some more book-shelves, and ordered a pair of ceiling fans. The only other thing to report is that I've started deleting emails. (shock!) I look after a few machines for myself, friends and so on. Each one will send me automated emails at times. I've got them going back years - but no longer:
#
#  Prune old mailboxes.
#
for maildir in .Automated.backups .machines.spotlight .machines.skx; do
   nrecent --keep=150 $HOME/Maildir/$ maildir /cur
done
Another new use for my nrecent tool. (Note: Only delete files from ./cur, to avoid deleting messages I've not read. Maildir is your friend!) ObQuote: "I'm sorry I'm sweating on you... " - Knocked Up.

Iustin Pop: Running: Z rich city run rain, rain, rain

This year was the 10th anniversary of the Z rich marathon, so they had a new category: 10km city run (at least, I think it was a new category). So I went along, thinking 10k would be a nice Sunday run. But oh, even the famed Swiss organisational skills fail some time. And when they fail, they fail hard. First, if you have 3 events all going on at the same time (marathon, team run, city run), please, please mark the road appropriately. The finish line was marked only marathon and team run, so I didn't even knew when I finished, I only realised well, that must have been it when I reached the place where I gave back the transponder chip. This marking issue was on-going for the last ~500-600 metres, as I thought I was lost when running along a road marked (again) only with marathon and team run and almost exited the road to look for my actual trail. Of course, if I had remembered that the finish is common to all three categories, all would have been good, but don't ask me to remember such stuff at the end of a race. Second, clothes storage. They had clothes storage organised nicely, in railroad freight cars, 500 people per car. This went well, except that they underestimated the number of people registering, so the last car had well over 700 bags, which created a huge bottleneck for this single car, which of course was where I also had the clothes (as I registered late). So I run for ~1 hour and then spend 30 minutes getting my clothes back. This wouldn't have been an issue, if the weather would have been fine. But it started raining seriously 500 meters into the race, and by the time I finished I was already fully drenched, just in a t-shirt (thanks fully with long pants) and waiting in the rain for 30 minutes was painful. I think also unhealthy, due to the abrupt switch from running to just standing in the rain. I haven't been so cold in a long while. By the time I got home my hands were shaking so hard I couldn't unzip my jacket (and I couldn't stop shaking, which is not a good sign). So yeah, I learned a few things. Try not to use the clothes storage if you can (it was the first time I used it), especially if you can pick up the stuff they give you a day early (and I could have gone to the race directly dressed) and I hope the organisers learned to provide better conditions just in case it rains. Which in Z rich happens, well, quite often. People were joking in the line: "1 hour running, 1 hour picking the clothes, 1 week sick". I hope I won't get sick (a hot tea after a hot bath can do wonders), but still, at least my voice will sound like after whole-night partying. OK, rant over. What about the race? It's nice (except the very dark overcast & rain, as said), through the centre of the city; as a friend says: Running on Bahnhofstrasse is cool . Otherwise, 10k seem to go quite quick nowadays . Having ~1,600 people all in red t-shirts was funny, especially if you're trying to look for friends. It was a very diverse crowd, you could see more varied people than in the other races (where more running-focused people seem to come); here there were all kind of people, children, teenagers, adults and so on, no per-category start or such. As to myself, I still am bothered by some small injuries, so I can't enjoy the nice result I got: 47m26.4s, for a 4m.44s/km! Yay! Of course, the reason I run so fast was because I was cold and needed to run to heat up. Also, again a flat race. I should start getting friends with hills. And due to the fact that it was a more diverse crowd, for the first time I didn't class (in my age category) around the ~66% down the list, but around ~44% only (~13 minutes slower than the winner). Yay!

Stefano Zacchiroli: deferred bits from the DPL for March 2012

Posted a week ago, already deferred back then, this report is even more deferred now! But as there are people interest in knowing what the "DPL job" is about even among non debian-devel-announce subscribers, here is a blog-conveyed reproduction, for the records.
Dear project members,
here is my monthly DPL activity report, this time for last March. It is delayed by a couple of weeks because, myself being both incumbent and candidate DPL, I preferred not to use d-d-a during the voting period unless really needed. Apologies for the delay (or the unneeded paranoia, you name it). As a side effect of the delay, the results of the DPL election are now known. I'd like to thank all the people who took part in the elections: voters, people who asked questions on -vote, the secretary, and obviously Gergely and Wouter, without whom the campaign wouldn't have allowed to discuss relevant aspects of Debian "politics". Thanks for your trust. I'll do my best to match your expectations. ... and just to remind you what you've just asked for, here goes the BigMonthlyBlurb! Highlight: long-term hardware replacement planning The highlight for this month is long term planning of hardware replacement. It's something I've been discussing with DSA for quite a while and on which DSA has worked hard during the recent sprint. As a result, we now have a quite ambitious 5-year hardware replacement plan that will guarantee that all machines in production are under warranty at any given time (with the nice side effect of generally better performances, as they go hand in hand with newer hardware). The current estimated cost per year is 29'000 USD. That does not yet include buildds and porter-boxes, so it is expected to increase a bit to cover all our hardware needs. But we expect it not increase too much, as we tend to get explicit hardware donations to cover arch-specific needs. Given the current state of Debian finances and donation trends, the plan looks sustainable for at least 2-3 years. But this assessment still needs to be refined as soon as, together with the auditors, we'll manage to obtain the history of past Debian transactions, in particular from SPI. We've been waiting for this for about 5 months now, but I'm positive it could become a reality in the next weeks. In the meantime, it is surely safe to start with the plan for the next 1-1.5 years, so I'll give green light to DSA for the first acquisitions as soon as they're ready for it. When implemented, this plan will increase our ability to rely on hardware. But it also means we will need to become a bit more organized about fund-raising. The discussion started with the sprint report has some insights about how to do that. As part of this, we'll also need to share resources (e.g. contact databases, people, etc.) among the yearly DebConf fund-raising initiatives and the initiatives mentioned in the aforementioned discussion. Ongoing discussions Summer of Code Debian has been accepted as an organization for the Google Summer of Code. At the time these bits go out, the student application deadline has also elapsed. In March I've contributed a few project ideas and chased potential mentors for them, when I thought the project could be important for Debian and the prospective student. I'm happy that one (a dak building block needed for the implementation of PPAs and more) has found both mentors and students. We'll see if any of the corresponding student proposal is retained and how it goes. Communication I've given an interview, about Debian and Free Software in general, to La Repubblica, one of the major newspapers in Italy. The interview is available online, but only in Italian. If some kind (and Italian-speaking) soul would like to translate it into English, I'll be happy to publish the translation as well. (update 22/04/2012: Matteo Cortese has contributed an English translation of the interview, which I'll make available shortly) Legal stuff In order to transfer ownership of the Debian trademark in Japan to SPI, I've contacted the current owners (all Japanese Debian Developers or contributors) to do the needed paperwork. I've been blessed by the help of Kenshi Muto that has taken the matter in his hands. He is now navigating through Japanese trademark procedures, a subject neither myself nor SPI lawyers were familiar with. Thanks also to Jonathan McDowell who has done the needed paperwork, SPI-side. Sprints Plenty of sprints and sprint reports in March!: Debian Med, DSA, DAM/FrontDesk. Everything should also be available from the wiki sprint page where you can find info to organize your team sprint. Assets miscellanea Cheers.
PS the boring day-to-day activity log for March is available at master:/srv/leader/news/bits-from-the-DPL.txt.201203

Robert Collins: Reading list

I ve recently caught up on a bunch of reading some of which are worth commending. Read them all, or none. I enjoyed them all.

21 April 2012

Andreas Metzler: balance sheet snowboarding season 2011/12

Finally a year with real snow again. Winter started rather late this year, I was still hiking at above 2000m end of November with zero snow there. First day of snowboarding was December 21st. Starting from there we never had too little snow. At home we topped out at 175cm, and had more than 125cm for a long time. Start of February was very cold, I had to skip a good day of good weather as I got a tiny bit of frostburn on my nose. (1200m of descent was a little to much at below -20 C.) A very warm end of March/start of April neverless quickly killed the snow here down at the bottom of the valley. There is still lots of snow in the mountains, although the ski-lifts have closed April 15th. Judging from the number of snowboarding days the season was a bad one. It started late and ended early (31st of March). In spring I simply need good weather to have fun. - Cloudy nights are too warm, the slope does not freeze properly, which is perfect only for breaking one's leg. Nevertheless if I could choose between a repetition of 2010/11 or 2011/2012, I would always go for the latter. :-) On a sidenote, my local hill has switched from www.hoehenmeter.com to www.skiline.cc to provide statistical information. Here is the balance sheet:
2005/06 2006/07 2007/08 2008/09 2009/10 2010/11 2011/12
number of (partial) days25172937303025
Dam ls1010510162310
Diedamskopf154242313414
Warth/Schr cken0304131
total meters of altitude12463474096219936226774202089203918228588
highscore10247m8321m12108m11272m11888m10976m13076m
# of runs309189503551462449516

Paul Tagliamonte: gcc 4.7 patchathon part III

S more days, s more patches: Hacked out a few over the last few days, this time, to some good eats. I m making a habit of this tv-hack-time thing. Consider helping out with the gcc-4.7 ftbfss, there s a bunch!

20 April 2012

Keith Packard: fireworks

LED Fireworks. Or, parent helpers go overboard My friend, Mike Ward, and I help out at our kids school by teaching 5th and 6th graders programming in conjunction with a Lego class. For several years, I was teaching Logo on ancient Apple II machines. The teacher had a great curriculum and the kids learned a lot, but this year we decided to switch things around and try them out on Arduinos. This spring, the kids are building a Lego display centered around a birthday party theme for the star elephant at the Oregon Zoo (Packy), who turned 50 last weekend. In a moment of fun, I suggested that perhaps we should have some fireworks, along with the presents and birthday cake. How hard could it be to wire up a couple of LEDs to an Arduino to make something that looked like fireworks? Selecting parts. Mike and I sat down to order some parts from Digikey. We found these nice, cheap, LED drivers from Sharp. Then we found a selection of LEDs from Cree along with current-programming resistors (12.7k ) to drive the LEDs at 20mA, and some bypass caps (0.1 F) to keep the parts happy. The LED drivers are simple shift registers, with separate clock and data. They have both input and output pins, so you can daisy chain them and drive as many as you like from a single pair of CPU pins. We figured 160 LEDs would be plenty, so we ordered 10 chips. The LED drivers came in a through-hole, DIP package; it seemed like we d be able to just stuff them into a breadboard and quickly wire things up. Prototyping the design The LED driver chips turn out to use a much finer pitch package than the usual 0.1 spacing. There s no way we could stuff them into a breadboard and quickly wire things up. Obviously, a bit more care while ordering would have uncovered this, but my experience with through-hole parts is limited to parts from the mid 1970s I soldered extension wires onto one chip to test the circuit, and managed to get one chip talking to an Arduino. Here s a short movie of that in action: You can see the LED driver chip sitting on a set of extension leads plugged into the breadboard. It s hooked to the SPI port on the Arduino board, with the clock set as fast as it will go. If you watch the video, you can see that the LEDs actually vary in brightness. They re actually being pulse-width modulated. Clocked at 2MHz, there s enough bandwidth in the SPI bus to drive 160 LEDs with 127 discrete pulse widths at nearly 100Hz (100 * 127 * 160 = 2032000). Is bread boarding practical? With the chip propped up on its extension wires, and the LEDs plugged in, the whole setup was not surprisingly fragile. Getting five of these chips wired up and running for the length of the Lego show (one day at school and two days at the zoo) seemed unlikely. So, we decided to build some custom circuit boards. Using the GEDA tools to make circuit boards My business partner, Bdale Garbee, and I build rocketry electronics through Altus Metrum. He s the hardware guy and I spend my weekends writing firmware for various micro-controllers. Of course, I use free software tools for the firmware, and Bdale uses the free gEDA tools for the hardware. I ve gotten vaguely familiar with those by helping Bdale review circuit diagrams and PC board layout over the years. So I figured I d be able to learn a bit more about them and come up with some designs. I decided to stick two LED drivers on each board. As the chips can drive each LED with up to 50mA of current, each board could potentially consume 1.6A at 5V. Sticking a linear regulator on each board capable of supplying that much current would have required a hefty heat sink. Instead, I decided to stick a mini USB connector on the board and use some cheap USB power supplies. Of course, there wasn t a pre-packaged schematic symbol or footprint for the LED driver, so I first had to construct those. But, with symbols for all of the parts I used available, I drew this circuit: Wiring up 32 LEDs It s really easy to draw wires showing 32 LEDs hooked up to each board. Actually selecting connectors that can be easily hooked up took a bit more time. The idea of crimping individual two-pin connectors for 160 LEDs didn t seem like fun, plus individual connectors were going to cost a small fortune (the best option I found would have cost about $0.40 per LED). Bdale suggested using ribbon cable with a crimp-on connector crimp a connector across the whole cable and then split out two-wire sets for each LED. I d still have to individually solder each LED to the wire, but on the board end, all I d have to do is crimp the connector on to the cable. Designing the circuit boards The gEDA tools separate schematic design and PCB layout into two separate tools; the schematic program, gschem, exports a netlist and set of components which the schematic program, PCB, can import. This sounded a bit clunky to me, as you have to re-export and re-import the schematic data into the PCB each time you make changes, but it seems to work reasonably well in practice. The first time the schematic data is imported to the PCB tool, all of the component footprints are simply stacked on top of one another. With those moved to sensible locations, the tool will show rats for each netlist. Having drawn the schematic with symbols that strongly resembled the actual components, it was pretty easy to work through the rats one at a time by painting copper on the board. I d noticed that all of the signals could easily be routed on one side of the board, so I flooded the back side with a ground plane to keep things quiet. The final circuit board design looks like this: Circuit boards. All Altus Metrum products are manufactured by Advanced Circuits, and we ve been quite happy with them. They offer a cheap and fast prototyping service through their barebonespcb.com site. These boards are two layers with no silk screen or solder mask. I uploaded my design and in a couple of days I got back some shiny circuit boards: The footprint I designed for the LED drivers turned out to have a very small gap between the pads for each pin and the ground plane. This lead to a couple of shorted pins in the five boards, which were pretty easy to diagnose as the LED driven by that pin would be stuck on. Here s another picture with the ribbon cable connected and an LED lit up: Mounting the LEDs I started by writing a small nickle program to simulate the trajectory of a ballistic fireworks shell followed by an explosion at apogee and subsequent ballistic tracks for the resulting pieces. I made the initial conditions of the launch somewhat random, and Mike and I sat and watched that draw things until we saw a couple that looked good . The output from that was a series of X/Y locations spaced uniformly in time. Mike used those to drill holes in a piece of acrylic from TAP plastics. He glued the LEDs into that with some white glue (so that we have a chance of getting them back out): A custom maple box? I spent last weekend visiting my father for his birthday. Meanwhile, Mike was busy back home building a case for the project. I got home and found that he had fabricated a solid maple case. It s beautiful, and holds all of the LEDs, boards, wires and a couple of 4-outlet USB power supplies. Here s a picture of the joint he made in the box corners: The finished product Here s the inside of the box: And here s a movie of the whole thing in action. It s missing 8 white LEDs; I hadn t ordered enough from Digikey. Downloading schematics and PCB artwork The hardware design is licensed under the TAPR Open Hardware License. It s all stored in git (of course):
git://keithp.com/git/hw/fireworks
All of the symbols I created are also available through git:
git://keithp.com/git/hw/keithp
Things I d do differently One of the frustrating things about moving atoms around instead of just bits is that once you ve got a physical artifact, it s really expensive to change it. There are a couple of minor things I would change if I were going to build more of these: Things that worked well There were a couple of things that went better than I had expected:

19 April 2012

Raphaël Hertzog: People behind Debian: Samuel Thibault, working on accessibility and the Hurd

Samuel Thibault is a French guy like me, but it took years until we met. He tends to keep a low profile, even though he s doing lots of good work that deserves to be mentioned. He focuses on improving Debian s accessibility and contributes to the Hurd. Who said he s a dreamer? :-) Checkout his interview to have some news of Wheezy s status on those topics. Raphael: Who are you? Samuel: I am 30 years old, and live in Bordeaux, France. During the workday, I teach Computer Science (Architecture, Networking, Operating Systems, and Parallel Programming, roughly) at the University of Bordeaux, and conduct researches in heterogeneous parallel computing. During the evening, I play the drums and the trombone in various orchestra (harmonic/symphonic/banda/brass). During the night, I hack on whatever fun things I can find, mainly accessibility and the Hurd at the moment, but also miscellaneous bits such as the Linux console support. I am also involved in the development of Aquilenet, an associative ISP around Bordeaux, and getting involved in the development of the network infrastructure in Bordeaux. I am not practicing Judo any more, but I roller-skate to work, and I like hiking in the mountains. I also read quite a few mangas. Saturday mornings do not exist in my schedule (Sunday mornings do, it s Brass Band rehearsal :) ). Raphael: How did you start contributing to Debian? Samuel: Bit by bit. I have been hacking around GNU/Linux since around 1998. I installed my first Debian system around 2000, as a replacement for my old Mandrake installation (which after all my tinkering was actually no longer looking like a Mandrake system any more!). That was Potato at the time, which somebody offered me through a set of CDs (downloading packages over the Internet was unthinkable at the time with the old modems). I have been happily reading and hacking around documentation, source code, etc. provided on them. Contribution things really started to take off when I went to the ENS Lyon high school in 2001: broadband Internet access in one s own student room! Since sending a mail was then really free, I started submitting bugs against various packages I was using. Right after that I started submitting patches along them, and then patches to other bugs. I did that for a long time actually. I had very little knowledge of all packaging details at the time, I was just a happy hacker submitting reports and patches against the upstream source code. At ENS Lyon, I met a blind colleague with very similar hacking tastes (of course we got friends) and he proposed me, for our student project, to work on a brlnet project (now called brlapi), a client/server protocol that lets applications render text on braille devices themselves. Along the way, I got to learn in details how a blind person can use a Unix system and the principles that should be followed when developing Accessibility. That is how I got involved in it. We presented our project at JDLL, and the Hurd booth happened to be next to our table, so I discussed with the Hurd people there about how the Hurd console could be used through braille. That is how I got into the Hurd too. From then on, I progressively contributed more and more to the upstream parts of both accessibility software and the Hurd. And then to the packaging part of them. Through patches in bug reports first, as usual, as well as through discussions on the mailing lists. But quickly enough people gave me commit access so I could just throw the code in. I was also given control over the Hurd buildds to keep them running. It was all good at that stage: I could contribute in all the parts I was caring about. People however started telling me that I should just apply for being a Debian Developer; both from accessibility and Hurd sides. I had also seen a bunch of my friends going through the process. I was however a bit scared (or probably it was just an excuse) by having to manage a gpg key, it seemed like a quite dangerous tool to me (even if I already had commit access to glibc at the time anyway ). I eventually applied for DM in 2008 so as to at least be able to upload some packages to help the little manpower of the Accessibility and Hurd teams. Henceforth I had already a gpg key, thus no excuse any more. And having it in the DM keyring was not enough for e.g. signing the hurd-i386 buildd packages. So I ended up going through NM in 2009, which went very fast, since I had already been contributing to Debian and learning all the needed stuff for almost 10 years! I now have around 50 packages in my QA page, and being a DD is actually useful for my work, to easily push our software to the masses :) So to sum it up, the Debian project is very easy to contribute to and open to new people. It was used during discussions at the GNU Hackers Meeting 2011 as an example of a very open community with public mailing lists and discussions. The mere fact that anybody can take the initiative of manipulating the BTS (if not scared by the commands) without having to ask anybody is an excellent thing to welcome contributions; it is notable tha the GNU project migrated to the Debbugs BTS. More generally, I don t really see the DD status as a must, especially now that we have the DM status (which is still a very good way to drag people into becoming DDs). For instance, I gave a talk at FOSDEM 2008 about the state of accessibility in Debian. People did not care whom I was, they cared that there was important stuff going on and somebody talking about it. More generally, decisions that are made through a vote are actually very rare. Most of the time, things just happen on the mailing lists or IRC channels where anybody can join the discussion. So I would recommend beginners to first use the software, then start reporting bugs, then start digging in the software to try fix the bugs by oneself, eventually propose patches, get them reviewed. At some point the submitted patches will be correct already most of the time. That s when the maintainers will start getting bored of just applying the patches, and simply provide with commit access, and voil , one has become a main contributor. Raphael: You re one of the main contributors to the Debian GNU/Hurd port. What motivates you in this project? Samuel: As I mentioned above, I first got real contact with the Hurd from the accessibility point of view. That initially brought me into the Hurd console, which uses a flexible design and nice interfaces to interact with it. The Hurd driver for console accessibility is actually very straightforward, way simpler than the Windows or Linux drivers. That is what caught me initially. I have continued working on it for several reasons. First, the design is really interesting for users. There are many things that are natural in the Hurd while Linux is still struggling to achieve them, such as UID isolation, recently mentioned in LWN. What I really like in the Hurd is that it excels at providing users with the same features as the administrator s. For instance, I find it annoying that I still can not mount an ISO image that I build on e.g. ries.debian.org. Linux now has FUSE which is supposed to permit that, but I have never seen it enabled on an ssh-accessible machine, only on desktop machines, and usually just because the administrator happens to be the user of the machine (who could as well just have used sudo ) For me, it is actually Freedom #0 of Free Software: let the user run programs for any purpose, that is, combining things together all the possible ways, and not being prevented from doing some things just because the design does not permit to achieve them securely. I had the chance to give a Hurd talk to explain that at GHM 2011, whose main topic was extensibility , I called it GNU/Hurd AKA Extensibility from the Ground, because the design of the Hurd is basically meant for extensibility, and does not care whether it is done by root or a mere user. All the tools that root uses to build a GNU/Hurd system can be used by the user to build its own GNU/Hurd environment. That is guaranteed by the design itself: the libc asks for things not to the kernel, but to servers (called translators), which can be provided by root, or by the user. It is interesting to see that it is actually also tried with varying success in GNU/Linux, through gvfs or Plash. An example of things I love being able to do is: $ zgrep foo ~/ftp://cdn.debian.net/debian/dists/sid/main/Contents-*.gz On my Hurd box, the ~/ftp: directory is indeed actually served by an ftpfs translator, run under my user uid, which is thus completely harmless to the system. Secondly and not the least, the Hurd provides me with interesting yet not too hard challenges. LWN confirmed several times that the Linux kernel has become very difficult to significantly contribute to, so it is no real hacking fun any more. I have notably implemented TLS support in the Hurd and the Xen and 64bit support in the GNU Mach kernel used by the Hurd. All three were very interesting to do, but were already done for Linux (at least for all the architectures which I actually know a bit and own). It happens that both TLS and Xen hacking experience became actually useful later on: I implemented TLS in the threading library of our research team, and the Xen port was a quite interesting line on my CV for getting a postdoc position at XenSource :) Lastly, I would say that I am used to lost causes :) My work on accessibility is sometimes a real struggle, so the Hurd is almost a kind of relief. It is famous for his vapourware reputation anyway, and so it is fun to just try to contribute to it nevertheless. An interesting thing is that the opinion of people on the Hurd is often quite extreme, and only rarely neutral. Some will say it is pure vapourware, while others will say that it is the hope of humanity (yes we do see those coming to #hurd, and they are not always just trolls!). When I published a 0.401 version on 2011 April 1st, the comments of people were very diverse, and some even went as far as saying that it was horrible of us to make a joke about the promised software :) Raphael: The FTPmasters want to demote the Hurd port to the debian-ports.org archive if it doesn t manage a stable release with wheezy. We re now at 2 months of the freeze. How far are you from being releasable ? Samuel: Of course, I can not speak for the Debian Release team. The current progress is however encouraging. During Debconf11, Michael Banck and I discussed with a few Debian Release team members about the kind of goals that should be achieved, and we are near completion of that part. The Debian GNU/Hurd port can almost completely be installed from the official mirrors, using the standard Debian Installer. Some patches need some polishing, but others are just waiting for being uploaded Debian GNU/Hurd can start a graphical desktop and run office tools such as gnumeric, as well as the iceweasel graphical web browser, KDE applications thanks to Pino Toscano s care, and GNOME application thanks to Emilio Pozuelo Monfort s care. Of course, general textmode hacking with gcc/make/gdb/etc. just works smoothly. Thanks to recent work on ghc and ada by Svante Signell, the archive coverage has passed 76%. There was a concern about network board driver support: until recently, the GNU Mach kernel was indeed still using a glue layer to embed the Linux 2.2 or even 2.0 drivers (!). Finding a network board supported by such drivers had of course become a real challenge. Thanks to the GSoC work of Zheng Da, the DDE layer can now be used to embed Linux 2.6.32 drivers in userland translators, which was recently ACCEPTed into the archive, and thus brings way larger support for network boards. It also pushes yet more toward the Hurd design: network drivers as userland process rather than kernel modules. That said, the freeze itself is not the final deadline. Actually, freeze periods are rests for porters, because maintainers stop bringing newer upstream versions which of course break on peculiar architectures. That will probably be helpful to continue improving the archive coverage. Raphael: The kfreebsd port brought into light all the packages which were not portable between different kernels. Did that help the Hurd port or are the problems too different to expect any mutual benefit? Samuel: The two ports have clearly helped each other in many aspects. The hurd-i386 port is the only non-Linux one that has been kept working (at least basically) for the past decade. That helped to make sure that all tools (dpkg, apt, toolchain, etc.) were able to cope with non-Linux ports, and keep that odd-but-why-not goal around, and evidently-enough achievable. In return, the kFreeBSD port managed to show that it was actually releasable, at least as a technological preview, thus making an example. In the daily work, we have sometimes worked hand in hand. The recent porting efforts of the Debian Installer happened roughly at the same time. When fixing some piece of code for one, the switch-case would be left for the other. When some code could be reused by the other, a mail would be sent to advise doing so, etc. In the packaging effort, it also made a lot of difference that a non-Linux port is exposed as released architecture: people attempted by themselves to fix code that is Linuxish for no real reason. The presence of the kFreeBSD is however also sometimes a difficulty for the Hurd: in the discussions, it sometimes tends to become a target to be reached, even if the systems are not really comparable. I do not need to detail the long history of the FreeBSD kernel and the amount of people hacking on it, some of them full-time, while the Hurd has only a small handful of free-time hackers. The FreeBSD kernel stability has already seen long-term polishing, and a fair amount of the Debian software was actually already ported to the FreeBSD kernel, thanks to the big existing pure-FreeBSD hackerbase. These do not hold for the GNU/Hurd port, so the expectations should go along. Raphael: You re also very much involved in the Debian Accessibility team. What are the responsibilities of this team and what are you doing there? Samuel: As you would expect it, the Debian Accessibility team works on packaging accessibility-related packages, and helping users with them; I thus do both. But the goal is way beyond just that. Actual accessibility requires integration. Ideally enough, a blind user should be able to just come to a Debian desktop system, plug his braille device, or press a shortcut to enable speech synthesis, and just use the damn computer, without having to ask the administrator to install some oddly-named package and whatnot. Just like any sighted user would do. He should be able to diagnose why his system does not boot, and at worse be able to reinstall his computer all by himself (typically at 2am ). And that is hard to achieve, because it means discussing about integration by default of accessibility features. For instance, the Debian CD images now beep during at the boot menu. That is a precious feature that has been discussed between debian-boot and debian-accessibility for a few weeks before agreeing on how to do it without too much disturbance. Similarly, my proposition of installing the desktop accessibility engines has been discussed for some time before being commited. What was however surprisingly great is that when somebody brought the topic back for discussion, non-debian-accessibility people answered themselves. This is reassuring, because it means things can be done durably in Debian. On the installation side, our current status is that the stable Debian installer has a high contrast color theme, and several years ago, I have pushed toward making standard CD images automatically detect braille devices, which permits standalone installation. I have added to the Wheezy installer some software speech synthesis (which again brought discussion about size increase vs versatility etc.) for blind people who do not have a braille device. I find it interesting to work on such topic in Debian rather than another distribution, because Debian is an upstream for a lot of distributions. Hopefully they just inherit our accessibility work. It at least worked for the text installer of Ubuntu. Of course, the Accessibility team is looking for help, to maintain our current packages, but also introduce new packages from the TODO list or create some backports. One does not need to be an expert in accessibility: tools can usually be tested, at least basically, by anybody, without particular hardware (I do not own any, I contributed virtual ones to qemu). For new developments and ideas, it is strongly recommended to come and discuss on debian-accessibility, because it is easy to get on a wrong track that does not bring actual accessibility. We still have several goals to achieve: the closest one is to just fix the transition to gnome3, which has been quite bad for accessibility so far :/ On the longer run, we should ideally reach the scenario I have detailed above: desktop accessibility available and ready to be enabled easily by default. Raphael: What s the biggest problem of Debian? Samuel: Debian is famous for its heated debian-devel discussions. And some people eventually say this no fun any more . That is exemplified in a less extreme way in the debian-boot/accessibility discussions that I have mentioned above. Sometimes, one needs to have a real stubborn thick head to continue the discussion until finding a compromise that will be accepted for commit. That is a problem because people do not necessarily have so much patience, and will thus prefer to contribute to a project with easier acceptance. But it is also a quality: as I explained above, once it is there, it is apparently for good. The Ubuntu support of accessibility in its installer has been very diverse, in part due to quite changing codebase. The Debian Installer codebase is more in a convergence process. Its base will have almost not changed between squeeze and wheezy. That allowed the Debian Accessibility team to continue improving its accessibility support, and not have to re-do it. A wiki page explains how to test its accessibility features, and some non-debian-accessibility people do go through it. A problem I am much more frightened by is the manpower in some core teams. The Debian Installer, grub, glibc, Xorg, gcc, mozilla derivatives, When reading the changelogs of these, we essentially keep seeing the same very few names over and over. And when one core developer leaves, it is very often still the same names which appear again to do the work. It is hard to believe that there are a thousand DDs working on Debian. I fear that Debian does not manage to get people to work on core things. I often hear people saying that they do not even dare thinking about putting their hands inside Xorg, for instance. Xorg is complex, but it seems to me that it tends to be overrated, and a lot of people could actually help there, as well as all the teams mentioned above. And if nobody does it, who will? Raphael: Do you have wishes for Debian Wheezy? Samuel: That is an easy one :) Of course I wish that we manage to release the hurd-i386 port. I also wish that accessibility of gnome3 gets fixed enough to become usable again. The current state is worrying: so much has changed that the transition will be difficult for users already, the current bugs will clearly not help. I also hope to find the time to fix the qt-at-spi bridge, which should (at last!) bring complete KDE accessibility. Raphael: Is there someone in Debian that you admire for their contributions? Samuel: Given the concerns I expressed above, I admire all the people who do spend time on core packages, even when that is really not fun everyday. Just to alphabetically name a few people I have seen so often here and there in the areas I have touched in the last few years: Aur lien Jarno, Bastian Blank, Christian Perrier, Colin Watson, Cyril Brulebois, Frans Pop, J rg Jaspert, Joey Hess, Josselin Mouette, Julien Cristau, Matthias Klose, Mike Hommey, Otavio Salvador, Petr Salinger, Robert Millan, Steve Langasek. Man, so many things that each of them works on! Of course this list is biased towards the parts that I touched, but people working in others core areas also deserve the same admiration.
Thank you to Samuel for the time spent answering my questions. I hope you enjoyed reading his answers as I did. Note that older interviews are indexed on wiki.debian.org/PeopleBehindDebian.

Subscribe to my newsletter to get my monthly summary of the Debian/Ubuntu news and to not miss further interviews. You can also follow along on Identi.ca, Google+, Twitter and Facebook.

3 comments Liked this article? Click here. My blog is Flattr-enabled.

18 April 2012

Wouter Verhelst: Screen scraping sucks

At a customer, I've migrated a number of manually-maintained servers to having them be maintained through puppet not so long ago. Since then, some more machines have been added, and getting them up and running properly was a breeze: do a base install, install puppet, sign the certificate, restart puppet, and then wait and twiddle thumbs while puppet did its magic. Easy as pie. Now, a few months later, we needed to install a number of windows machines for a lab (not my choice), and the person involved asked me to figure out some diskspace so we could start creating images for those. Not a chance. Instead, I suggested looking for a configuration management system, similar to puppet. Since we're using Samba 3 to run the Windows network here, dropping everything in Active Directory was not an option. But a short while later, he came back with the note that puppet, in its 2.7 version, actually does support Windows as a platform for the managed machines. Interesting. The unfortunate bit was that puppet supports creating files and installing software when it is distributed as an MSI file, but not when it's distributed as a .exe file. This is not unexpected; MSI files can be installed noninteractively; but when something is distributed as a .exe file, it means it needs to be installed interactively; and puppet does not have the ability to interact with GUI software. The workaround: use something that does have that ability (in my case, autoit), and use an exec block in puppet to make it call those scripts. In effect, that's a bit like screenscraping. Add a creates stanza to the block, so that the installer isn't started again if the software at hand has already been installed. This 'autoit' thing also comes with a recording utility, allowing one to create an initial script by just doing the installation, and having the tool just record stuff. With that, the machines are installed 99% automated. I say 99%, because there are still some issues: I'll have to think about this some more, I guess. First, it's clear that while puppet does have some Windows functionality, it's not entirely ready yet. And somehow, using autoit to add to Puppet functionality feels like an ugly hack. We'll see what the future brings.

Paul Tagliamonte: gcc 4.7 patchathon part II

Another day, more patches. This time, it was to the rockn vibes of Chopped. Patch for: In the mean-time, I put together a script for GCC 4.7 transition bugs that are owned by the QA team with patches in the BTS. It s easy as cake to throw together a QA upload, so I encourage anyone out there to do some easy hacking :)

17 April 2012

Russell Coker: The Most Important things for running a Reliable Internet Service

One of my clients is currently investigating new hosting arrangements. It s a bit of a complex process because there are lots of architectural issues relating to things such as the storage and backup of some terabytes of data and some serious computation on the data. Among other options we are considering cheap servers in the EX range from Hetzner [1] which provide 3TB of RAID-1 storage per server along with reasonable CPU power and RAM and Amazon EC2 [2]. Hetzner and Amazon aren t the only companies providing services that can be used to solve my client s problems, but they both provide good value for what they provide and we have prior experience with them. To add an extra complication my client did some web research on hosting companies and found that Hetzner wasn t even in the list of reliable hosting companies (whichever list that was). This is in some ways not particularly surprising, Hetzner offers servers without a full management interface (you can t see a serial console or a KVM, you merely get access to reset it) and the best value servers (the only servers to consider for many terabytes of data) have SATA disks which presumably have a lower MTBF than SAS disks. But I don t think that this is a real problem. Even when hardware that s designed for the desktop is run in a server room the reliability tends to be reasonable. My experience is that a desktop PC with two hard drives in a RAID-1 array will give a level of reliability in practice that compares very well to an expensive server with ECC RAM, redundant fans, redundant PSUs, etc. My experience is that the most critical factor for server reliability is management. A server that is designed to be reliable can give very poor uptime if poorly maintained or if there is no rapid way of discovering and fixing problems. But a system that is designed to be cheap can give quite good uptime if well maintained, if problems can be repidly discovered and fixed. A Brief Overview of Managing Servers There are text books about how to manage servers, so obviously I can t cover the topic in detail in a blog post. But here are some quick points. Note that I m not claiming that this list includes everything, please add comments about anything particularly noteworthy that you think I ve missed.
  1. For a server to be well managed it needs to be kept up to date. It s probably a good idea for management to have this on the list of things to do. A plan to check for necessary updates and apply them at fixed times (at least once a week) would be a good thing. My experience is that usually managers don t have anything to do with this and sysadmins either apply patches or not at their own whim.
  2. It is really ideal for people to know how all the software works. For every piece of software that s running it should either have come from a source that provides some degree of support (EG a Linux distribution) or be maintained by someone who knows it well. When you install custom software from people who become unavailable then it puts the reliability of the entire system at risk if anything breaks then you won t be able to get it fixed quickly.
  3. It should be possible to rapidly discover problems, having a client phone you to tell you that your web site is offline is a bad thing. Ideally you will have software like Nagios monitoring the network and reporting problems via a SMS gateway service such as ClickaTell.com. I am not sure that Nagios is the best network monitoring system or that ClickaTell is the best SMS gateway, but they have both worked well in my experience. If you think that there are better options for either of those then please write a comment.
  4. It should be possible to rapidly fix problems. That means that a sysadmin must be available 24*7 to respond to SMS and you must have a backup sysadmin for when the main person takes a holiday, or ideally two backup sysadmins so that if one is on holiday and another has an emergency then problems can still be fixed. Another thing to consider is that an increasing number of hotels, resorts, and cruise ships are providing net access. So you could decrease your need for backup sysadmins if you give a holiday bonus to a sysadmin who uses a hotel, resort, or cruise ship that has good net access. ;)
  5. If it seems likely that there may be some staff changes then it s a really good idea to hire a potential replacement on a casual basis so that they can learn how things work. There have been a few occasions when I started a sysadmin contract after the old sysadmin ceased being on speaking terms with the company owner. This made it difficult for me to learn what s going on.
  6. If your network is in any way complex (IE it s something that needs some skill to manage) then it will probably be impossible to hire someone who has experience in all the areas of technology at a salary you are prepared to pay. So you should assume that whoever you hire will do some learning on the job. This isn t necessarily a problem but is something that needs to be considered. If you use some unusual hardware or software and want it to run reliably then you should have a spare system for testing so that the types of mistake which are typically made in the learning process are not made on your production network.
Conclusion If you have a business which depends on running servers on the Internet and you don t do all the things in the above list then the reliability of a service like Hetzner probably isn t going to be an issue at all. Related posts:
  1. Servers vs Phones Hetzner have recently updated their offerings to include servers with...
  2. Why Internet Access in Australia Sucks In a comment on my post about (relatively) Cheap Net...
  3. The National Cost of Slow Internet Access Australia has slow Internet access when compared to other first-world...

Russell Coker: ZFS vs BTRFS on Cheap Dell Servers

I previously wrote about my first experiences with BTRFS [1]. Since then I ve been using BTRFS on more systems and have had good results. The main problem I want to address is with the reliability of RAID [2]. Requirements for a File Server Now one of my clients has a need for a new fileserver. They need to reliably store terabytes of data (currently 6TB and growing) which is mostly comprised of data files in the 10MB 15MB size range. The data files will almost never be re-written and I anticiapte that the main bottleneck will be the latency of NFS and other network file sharing protocols. I would hope that saturating a GigE network when sending 10MB data files from SATA disks via NFS, AFS, or SMB wouldn t be a technical challenge. It seems that BTRFS is the way of the future. But it s still rather new and the lack of RAID-5 and RAID-6 is a serious issue when you need to store 10TB with today s technology (that would be 8*3TB disks for RAID-10 vs 5*3TB disks for RAID-5). Also the case of two disks entirely failing in a short period of time requires RAID-6 (or RAID-Z2 as the ZFS variant of RAID-6 is known). With BTRFS at it s current stage of development it seems that to recover from two disks failing you need to have BTRFS on another RAID-6 (maybe Linux software RAID-6). But for filesystems based on concepts similar to ZFS and BTRFS you want to have the filesystem run the RAID so that if a block has a filesystem hash mismatch then the correct copy can be reconstructed from parity. ZFS seems to be a lot more complex than BTRFS. While having more features is a good thing (BTRFS seems to be missing some sysadmin friendly features at this stage) complexity means that I need to learn more and test more before going live. But it seems that the built in RAID-5 and RAID-6 is the killer issue. Servers start becoming a lot more expensive if you want more than 8 disks and even going past 6 disks is a significant price point. As 3TB disks are available an 8 disk RAID-6 gives something like 18TB usable space vs 12TB on a RAID-10 and a 6 disk RAID-6 gives about 12TB vs 9TB on a RAID-10. With RAID-10 (IE BTRFS) my client couldn t use a 6 disk server such as the Dell PowerEdge T410 for $1500 as 9TB of usable storage isn t adequate and the Dell PowerEdge T610 which can support 8 disks and costs $2100 would be barely adequate for the near future with only 12TB of usable storage. Dell does sell significantly larger servers such that any of my clients needs could be covered by RAID-10, but in addition to costing more there are issues of power use and noise. When comparing a T610 and a T410 with a full set of disks the price difference is $1000 (assuming $200 per disk) which is probably worth paying to delay any future need for upgrades. Buying Disks The problem with the PowerEdge T610 server is that it uses hot-swap disks and the biggest disks available are 2TB for $586.30! 2TB*8 in RAID-6 gives 12TB of usable space for $4690.40! This compares poorly to the PowerEdge T410 which supports non-hot-swap disks so I can buy 6*3TB disks for something less than $200 each and get 12TB of usable space for $1200. If I could get hot-swap trays for Dell disks at a reasonable price then the T610 would be worth considering. But as 12TB of storage should do for at least the next 18 months it seems that the T410 is clearly the better option. Does anyone know how to get cheap disk trays for Dell servers? Implementation In mailing list discussions some people suggest using Solaris or FreeBSD for a ZFS server. ZFS was designed for and implemented on Solaris, and FreeBSD was the first port. However Solaris and FreeBSD aren t commonly used systems so it s harder to find skilled people to work with them and there is less of a guarantee that the desired software will work. Among other things it s really convenient to be able to run software for embedded Linux i386 systems on the server. The first port of ZFS to Linux was based on FUSE [3]. This allows a clean separation of ZFS code from the Linux kernel code to avoid license issues but does have some performance problems. I don t think that I will have any performance issues on this server as the data files are reasonably large, are received via an ADSL link, and which require quite a bit of CPU time to process them when they are accessed. But ZFS-FUSE doesn t seem to be particularly popular. The ZFS On Linux project provides source for a ZFS kernel module which you can compile and load [4]. As the module isn t distributed with or statically linked to the kernel the license conflict of the CDDL ZFS code and the GPL Linux kernel code is apparently solved. I ve read some positive reports from people who use this so it will be my preferred option. Related posts:
  1. ECC RAM in a Cheap Machine In a comment on my previous post about ECC RAM...
  2. Starting with BTRFS Based on my investigation of RAID reliability [1] I have...
  3. Dell PowerEdge T105 Today I received a Dell PowerEDGE T105 for use by...

16 April 2012

Wouter Verhelst: DPL vote, 2012

So, the vote is over, and Stefano won. During many past DPL elections, I've made my vote public, and this one is no different:
V: 1223 		597c362e6156ec7e37b334837161da26
That's me, in this list. Obviously I wouldn't run if I'm not serious about it, so I voted myself first. As to the other part: I thought long and hard about that, but eventually came to the conclusion that both Stefano and Gergely had properties as a candidate that I liked, and properties that I didn't like, and that therefore I couldn't prefer either of them over the other. I found Gergely's platform to be fairly similar to my own, which is a good thing; but there were a few details that made me have some pause about his candidacy. And while I stand by the things I said during campaigning about Stefano as a DPL, the truth is that the project could be far worse off than to have him re-elected. As to the outcome... I can't say it's entirely unexpected. I knew it was a long shot even before I started, and then campaigning didn't excactly go as I would have hoped. I expected to lose, but not by such a margin what Stefano did wasn't winning, it's called 'trashing the opposition'. Congratulations, zack, for a truly exceptional performance; and thanks, also to Gergely, for being a worthy opponent. In closing, I'll say that I don't think I'll run again. I've gone through the process three times now, and have never gotten very close to winning; this probably means that what I feel about the position of DPL is somewhat removed from what the project as a whole thinks about it. So, absent some radical changes in either the project itself or in the way I look upon it, another candidacy from me is highly unlikely. I guess I'll have to find other ways to spend my time...

Next.

Previous.