Search Results: "spa"

22 April 2012

Joey Hess: moving my email archives and packages to git-annex

I've recently been moving some important data into git-annex, and finding it simplifies things while also increasing my flexability. email archives I've kept my email archives in git for years. This works ok, just choose the right file format (compressed mbox) and number of files (one archive per mailbox per month or so) and git can handle this well enough, as email is not really large. But, email is not really small either. Keeping my email repository checked out on my netbook consumes 2 gigabytes of its 30 gigabyte SSD, half of which is duplication in .git. Also, I have only kept it at 2 gigabytes through careful selection of what classes of mail I archive. That made sense when archival disk was more expensive, but what makes sense these days is to archive everything. For a while I've wanted to have a "raw" archive, that includes all email I receive. (Even spam.) This protects against various disasters in mail filtering or reading. Setting that up was my impetus for switching my mail archives to git-annex today. The new system I've settled on is to first copy all incoming mail into a "raw" maildir folder. Then mailfilter sorts it into the folders I sync (with offlineimap) and read. Each day, the "raw" folder is moved into a mbox archive, and that's added to the git annex. Each month, the mail I've read is moved into a monthly archives, and added to the git annex. A simple script does the work. I counted the number of copies that existed of my mail when it was stored in git, and found 7 copies spread amoung just 3 drives. I decided to slim that back, and configured git-annex to require only 5 copies. But those 5 copies will spread amoung more drives, including several offline archival drives, so it will be more robust overall. My netbook will have an incomplete checkout of my mail, omitting the "raw" archive. If I need to peek inside a spam folder for a lost mail, I can quickly pull it down; if I need to free up space I can quickly drop older archives. This is the flexability that git-annex fans love. :) By the way, this also makes it easier to permanantly delete mail, when you really need to (ie, for contractual reasons). Before, I'd have to do a painful git-filter-branch if I needed to get rid of eg, mail for old jobs. Now I can git annex drop --force. Pro Tip: If you're doing this kind of migration to git-annex, you can save bandwidth by not re-transferring files to machines that already have a copy. I ran this command on my netbook to inject the archives it had in the old repository into the new repository, verifying checksums as it goes: 
cd ~/mail/archive; find -type l -exec git annex reinject ~/mail.old/archive/    \;
Note on mairix compatability: I use mairix to index and search my mail. But it refuses to follow git-annex's symlinks to the content. So I have to point it at .git/annex/objects/. I also configured annex.backend to SHA256E, which keeps the extensions on my compressed mailbox files, which is necessary for mairix to realize they're compressed. debian packages I'd evolved a complex and fragile chain of personal apt repositories to hold Debian packages I've released. I recently got rid of the mess, which looked like this: dput local mini-dinstall repo dput→mini-dinstallrepo on my server →dput Debian The point of all that was that I could "upload" a package locally while offline and batch transfer it later. And I had a local and a public apt repository of just the packages I've uploaded. But these days, packages uploaded to Debian are available nearly immediately, so there's not much reason to do that. My old system also had a problem: It only kept the most recent single copy of each package. Again, disk is cheap, so I'd rather have archives of everything I have uploaded. Again I switched to git-annex. My new system is simplicity itself. I release a package by checking it into a "toupload" directory in my git annex repository on my netbook. Items in that directory are dput to Debian and moved to "released". I have various other clones of that repository, which I git annex move packages to periodically to free up SSD space. In the rare cases when I build a package on a server, I check it into the clone on the server, and again rely on git-annex to copy it around. Now, does anyone know a good way to download a copy of every package you're ever released from archive.debian.org? (Ideally as a list of urls I can feed to git annex addurl.) conclusion My email and Debian packages were the last large files I was not storing in git-annex. Even backups of my backups end up checked into git-annex and archived away. Now that I'm using git-annex in every place I can, my goal with it is to make it as easy as possible for as many of you to use it as possible, too. I have some inotify tricks up my sleeve that seem promising. Kickstarter may be involved. Watch this space!

Russell Coker: Autism as an Excuse

A Polish geek going by the handle of mmemuar has recently written a blog post claiming that people use Autism as an excuse for bad behavior [1]. He gets enough things wrong in one short post to make it worth debunking it. It seems that Google s translation of Polish isn t as good as some other languages, but unless Google mistranslated about a dozen sections such that they had the exact opposite meaning then mmemuar s post has a lot of wrong ideas. Does Anyone use Autism as an Excuse? I ve read a lot of blog posts written by people on the Autism Spectrum, read many forum discussions, and talked to more than a few people in person. So far I haven t yet encountered any evidence of people using an Autism Spectrum Disorder (ASD) diagnosis as an excuse. There are probably about 70,000,000 people who meet the diagnostic criteria (most of whom have not been diagnosed due to not having access to anyone who is qualified to do an assessment). The number of people who have been diagnosed is large enough that I couldn t claim that none of them have ever used it as an excuse. The number of self-diagnosed people is also large enough that there has to be some people who wouldn t get diagnosed if professionally assessed. But I don t see any evidence that using an ASD as an excuse is at all common. I imagine that some people would take someone merely mentioning the fact that they have an ASD diagnosis in a public place (EG a mailing list or a blog post) as some sort of an excuse. One problem with such an interpretation is that for every way in which people on the Autism Spectrum annoy other people it s the ones who aren t diagnosed (or who reject a diagnosis) that will do it the most. Being diagnosed with an ASD is correlated with annoying other people less. Another problem is that keeping quiet about such things when they get raised for discussion so often takes a psychological toll. When someone is diagnosed as an adult a fairly common reaction is to study Psychology and Sociology (usually through web sites such as Sociological Images which I highly recommend [2]) and try to get a better understanding of other people. Any time you assume that everyone else thinks like you then you will get things wrong, when someone gets an ASD diagnosis they will probably take more care to avoid making such mistakes. Is Autism Obvious? Mmemuar says if you were full of autistic, which is very easy to overheat the brain from excess signals at the input, it would be very obvious to all . There are some people who are utterly incapable of acting like an NT. But the majority of people on the Autism Spectrum have some ability to pass as NT, it just takes a lot of effort. So if someone is spending all their effort to walk, talk, and make eye contact in a way that most people expect then they will have little spare effort for other things. This can result in them having little patience for other people. The solution to this is to not require people to look average. Even apart from Autism there are people who fidget, don t make eye contact in the way you expect, and do other things a little differently. Being tolerant of such things won t hurt you and will generally make things easier for everyone. TheAnMish gave a good Youtube presentation about the way that she has to act normal [3]. Note that while her video represents her own personal experiences (which differ slightly from those of other Aspies particularly male Aspies) they are regarded as representative enough for her video to be shown by Tony Attwood (a world renowned expert on Asperger Syndrome) at a conference about women on the Autism Spectrum. As an aside I disagree with her use of the word normal without scare-quotes. Learning about Psychology through Sci-Fi In the end of the blog post and in some of the comments there is discussion about learning to understand people through reading sci-fi books. The first problem with this is that fiction books generally have a range of characters that is determined by the author s understanding of people, if you read multiple books by an author then you will usually notice the same character types. The second problem is that characters in fiction books are simplified to fit into a reasonable sized book, in real life people have lots of really boring reasons for doing what they do, in fiction only stuff that is interesting ends up in print. But the biggest problem is that fiction books just aren t a good way of learning about people. Whatever lessons might be in a fiction book will probably be missed by a reader who is concentrating on the plot. You can learn things about people by reading an analysis of literature by a Psychologist or a Sociologist, but apart from that you probably won t be able to learn things if you don t already know them. As an aside, my experience of reading sci-fi books suggests that some of the popular sci-fi authors have such a poor understanding of people that it impairs their ability to write believable fiction about human characters. If I was going to try and learn about people by reading fiction I d choose something that s been popular for a long time. If a book has been popular for more than 100 years and sold well in different times, cultures, and languages then it probably has something to say about the human condition. Catching up on Youth Mmemuar says absolutely nothing prevents you to the age of twenty he began to catch up on some of youth . Actually one significant thing is that the human brain develops in particular stages, the older you get the more difficult it becomes to learn things. So even if it was just a matter of learning things someone who was behind at age 20 would have some significant difficulty in catching up. But some things just can t be learned, for example someone who has extreme discomfort in making eye contact can t just learn to be happy with it. Also there are some things which people would have learned if it was possible. For example people who have Prosopagnosia (an inability to recognise faces) suffer extreme bullying in school, if they could just learn to recognise people then they surely would do so, so if they complete school without learning then it s probably going to be impossible for them. Prosopagnosia is one of many conditions which can contribute to social difficulties and therefore contribute to an ASD diagnosis. Can Aspies get Married? Some people think that people on the Autism Spectrum can t get married. In fact this belief is so widely held that some people who seem very obviously Autistic are convinced that they are NT simply because they are married! The fact that there are more than few books offering advice to people who have married someone who is on the Autism Spectrum is clear proof that such claims are bogus. The Relevance to Geek Communities People who meet the diagnostic criteria for Asperger Syndrome will almost certainly be Geeks due to the Restricted, repetitive patterns of behavior, interests, or activities section of the diagnostic criteria (the proposed revision for the DSM-V is the best reference I know for this [4]) as the modern use of the term Geek applies to anyone who has an extreme interest in something. The more Geeky a community is the greater the incidence of people who could be diagnosed with an ASD. Spreading ideas such as those of mmemuar will lead to people not being assessed for an ASD. I would have been assessed earlier if it wasn t for hearing an influential member of the Linux community say some things which were similar in concept (although not as deliberate). Having people not be assessed is bad for the individuals in question and bad for the community. Related posts:
  1. Autism Awareness and the Free Software Community It s Autism Awareness Month April is Autism Awareness month, there...
  2. Autism vs Asperger Syndrome Diagnostic Changes for Autism Spectrum Disorders Currently Asperger Syndrome (AS)...
  3. Communication Shutdown and Autism The AEIOU Foundation The AEIOU Foundation [1] is a support...

Stefano Zacchiroli: deferred bits from the DPL for March 2012

Posted a week ago, already deferred back then, this report is even more deferred now! But as there are people interest in knowing what the "DPL job" is about even among non debian-devel-announce subscribers, here is a blog-conveyed reproduction, for the records.
Dear project members,
here is my monthly DPL activity report, this time for last March. It is delayed by a couple of weeks because, myself being both incumbent and candidate DPL, I preferred not to use d-d-a during the voting period unless really needed. Apologies for the delay (or the unneeded paranoia, you name it). As a side effect of the delay, the results of the DPL election are now known. I'd like to thank all the people who took part in the elections: voters, people who asked questions on -vote, the secretary, and obviously Gergely and Wouter, without whom the campaign wouldn't have allowed to discuss relevant aspects of Debian "politics". Thanks for your trust. I'll do my best to match your expectations. ... and just to remind you what you've just asked for, here goes the BigMonthlyBlurb! Highlight: long-term hardware replacement planning The highlight for this month is long term planning of hardware replacement. It's something I've been discussing with DSA for quite a while and on which DSA has worked hard during the recent sprint. As a result, we now have a quite ambitious 5-year hardware replacement plan that will guarantee that all machines in production are under warranty at any given time (with the nice side effect of generally better performances, as they go hand in hand with newer hardware). The current estimated cost per year is 29'000 USD. That does not yet include buildds and porter-boxes, so it is expected to increase a bit to cover all our hardware needs. But we expect it not increase too much, as we tend to get explicit hardware donations to cover arch-specific needs. Given the current state of Debian finances and donation trends, the plan looks sustainable for at least 2-3 years. But this assessment still needs to be refined as soon as, together with the auditors, we'll manage to obtain the history of past Debian transactions, in particular from SPI. We've been waiting for this for about 5 months now, but I'm positive it could become a reality in the next weeks. In the meantime, it is surely safe to start with the plan for the next 1-1.5 years, so I'll give green light to DSA for the first acquisitions as soon as they're ready for it. When implemented, this plan will increase our ability to rely on hardware. But it also means we will need to become a bit more organized about fund-raising. The discussion started with the sprint report has some insights about how to do that. As part of this, we'll also need to share resources (e.g. contact databases, people, etc.) among the yearly DebConf fund-raising initiatives and the initiatives mentioned in the aforementioned discussion. Ongoing discussions Summer of Code Debian has been accepted as an organization for the Google Summer of Code. At the time these bits go out, the student application deadline has also elapsed. In March I've contributed a few project ideas and chased potential mentors for them, when I thought the project could be important for Debian and the prospective student. I'm happy that one (a dak building block needed for the implementation of PPAs and more) has found both mentors and students. We'll see if any of the corresponding student proposal is retained and how it goes. Communication I've given an interview, about Debian and Free Software in general, to La Repubblica, one of the major newspapers in Italy. The interview is available online, but only in Italian. If some kind (and Italian-speaking) soul would like to translate it into English, I'll be happy to publish the translation as well. (update 22/04/2012: Matteo Cortese has contributed an English translation of the interview, which I'll make available shortly) Legal stuff In order to transfer ownership of the Debian trademark in Japan to SPI, I've contacted the current owners (all Japanese Debian Developers or contributors) to do the needed paperwork. I've been blessed by the help of Kenshi Muto that has taken the matter in his hands. He is now navigating through Japanese trademark procedures, a subject neither myself nor SPI lawyers were familiar with. Thanks also to Jonathan McDowell who has done the needed paperwork, SPI-side. Sprints Plenty of sprints and sprint reports in March!: Debian Med, DSA, DAM/FrontDesk. Everything should also be available from the wiki sprint page where you can find info to organize your team sprint. Assets miscellanea Cheers.
PS the boring day-to-day activity log for March is available at master:/srv/leader/news/bits-from-the-DPL.txt.201203

21 April 2012

Christian Perrier: 2012 update 20 for Debian Installer localization

Status for D-I level 1 (core D-I files): Status for D-I level 2 (packages that have localized material that may appear during default installs, such as iso-codes, tasksel, etc.): Status for D-I level 3 (packages that have localized material that may appear during non-default installs, such as win32-loader) Full 100% completeness (hall of fame) for 22 languages: Asturian, Belarusian, Bulgarian, Catalan, Czech, German, Esperanto, Spanish, French, Indonesian, Italian, Japanese, Kazakh, Khmer, Dutch, Polish, Portuguese, Russian, Slovak, Swedish, Turkish, Simplified Chinese

20 April 2012

Keith Packard: fireworks

LED Fireworks. Or, parent helpers go overboard My friend, Mike Ward, and I help out at our kids school by teaching 5th and 6th graders programming in conjunction with a Lego class. For several years, I was teaching Logo on ancient Apple II machines. The teacher had a great curriculum and the kids learned a lot, but this year we decided to switch things around and try them out on Arduinos. This spring, the kids are building a Lego display centered around a birthday party theme for the star elephant at the Oregon Zoo (Packy), who turned 50 last weekend. In a moment of fun, I suggested that perhaps we should have some fireworks, along with the presents and birthday cake. How hard could it be to wire up a couple of LEDs to an Arduino to make something that looked like fireworks? Selecting parts. Mike and I sat down to order some parts from Digikey. We found these nice, cheap, LED drivers from Sharp. Then we found a selection of LEDs from Cree along with current-programming resistors (12.7k ) to drive the LEDs at 20mA, and some bypass caps (0.1 F) to keep the parts happy. The LED drivers are simple shift registers, with separate clock and data. They have both input and output pins, so you can daisy chain them and drive as many as you like from a single pair of CPU pins. We figured 160 LEDs would be plenty, so we ordered 10 chips. The LED drivers came in a through-hole, DIP package; it seemed like we d be able to just stuff them into a breadboard and quickly wire things up. Prototyping the design The LED driver chips turn out to use a much finer pitch package than the usual 0.1 spacing. There s no way we could stuff them into a breadboard and quickly wire things up. Obviously, a bit more care while ordering would have uncovered this, but my experience with through-hole parts is limited to parts from the mid 1970s I soldered extension wires onto one chip to test the circuit, and managed to get one chip talking to an Arduino. Here s a short movie of that in action: You can see the LED driver chip sitting on a set of extension leads plugged into the breadboard. It s hooked to the SPI port on the Arduino board, with the clock set as fast as it will go. If you watch the video, you can see that the LEDs actually vary in brightness. They re actually being pulse-width modulated. Clocked at 2MHz, there s enough bandwidth in the SPI bus to drive 160 LEDs with 127 discrete pulse widths at nearly 100Hz (100 * 127 * 160 = 2032000). Is bread boarding practical? With the chip propped up on its extension wires, and the LEDs plugged in, the whole setup was not surprisingly fragile. Getting five of these chips wired up and running for the length of the Lego show (one day at school and two days at the zoo) seemed unlikely. So, we decided to build some custom circuit boards. Using the GEDA tools to make circuit boards My business partner, Bdale Garbee, and I build rocketry electronics through Altus Metrum. He s the hardware guy and I spend my weekends writing firmware for various micro-controllers. Of course, I use free software tools for the firmware, and Bdale uses the free gEDA tools for the hardware. I ve gotten vaguely familiar with those by helping Bdale review circuit diagrams and PC board layout over the years. So I figured I d be able to learn a bit more about them and come up with some designs. I decided to stick two LED drivers on each board. As the chips can drive each LED with up to 50mA of current, each board could potentially consume 1.6A at 5V. Sticking a linear regulator on each board capable of supplying that much current would have required a hefty heat sink. Instead, I decided to stick a mini USB connector on the board and use some cheap USB power supplies. Of course, there wasn t a pre-packaged schematic symbol or footprint for the LED driver, so I first had to construct those. But, with symbols for all of the parts I used available, I drew this circuit: Wiring up 32 LEDs It s really easy to draw wires showing 32 LEDs hooked up to each board. Actually selecting connectors that can be easily hooked up took a bit more time. The idea of crimping individual two-pin connectors for 160 LEDs didn t seem like fun, plus individual connectors were going to cost a small fortune (the best option I found would have cost about $0.40 per LED). Bdale suggested using ribbon cable with a crimp-on connector crimp a connector across the whole cable and then split out two-wire sets for each LED. I d still have to individually solder each LED to the wire, but on the board end, all I d have to do is crimp the connector on to the cable. Designing the circuit boards The gEDA tools separate schematic design and PCB layout into two separate tools; the schematic program, gschem, exports a netlist and set of components which the schematic program, PCB, can import. This sounded a bit clunky to me, as you have to re-export and re-import the schematic data into the PCB each time you make changes, but it seems to work reasonably well in practice. The first time the schematic data is imported to the PCB tool, all of the component footprints are simply stacked on top of one another. With those moved to sensible locations, the tool will show rats for each netlist. Having drawn the schematic with symbols that strongly resembled the actual components, it was pretty easy to work through the rats one at a time by painting copper on the board. I d noticed that all of the signals could easily be routed on one side of the board, so I flooded the back side with a ground plane to keep things quiet. The final circuit board design looks like this: Circuit boards. All Altus Metrum products are manufactured by Advanced Circuits, and we ve been quite happy with them. They offer a cheap and fast prototyping service through their barebonespcb.com site. These boards are two layers with no silk screen or solder mask. I uploaded my design and in a couple of days I got back some shiny circuit boards: The footprint I designed for the LED drivers turned out to have a very small gap between the pads for each pin and the ground plane. This lead to a couple of shorted pins in the five boards, which were pretty easy to diagnose as the LED driven by that pin would be stuck on. Here s another picture with the ribbon cable connected and an LED lit up: Mounting the LEDs I started by writing a small nickle program to simulate the trajectory of a ballistic fireworks shell followed by an explosion at apogee and subsequent ballistic tracks for the resulting pieces. I made the initial conditions of the launch somewhat random, and Mike and I sat and watched that draw things until we saw a couple that looked good . The output from that was a series of X/Y locations spaced uniformly in time. Mike used those to drill holes in a piece of acrylic from TAP plastics. He glued the LEDs into that with some white glue (so that we have a chance of getting them back out): A custom maple box? I spent last weekend visiting my father for his birthday. Meanwhile, Mike was busy back home building a case for the project. I got home and found that he had fabricated a solid maple case. It s beautiful, and holds all of the LEDs, boards, wires and a couple of 4-outlet USB power supplies. Here s a picture of the joint he made in the box corners: The finished product Here s the inside of the box: And here s a movie of the whole thing in action. It s missing 8 white LEDs; I hadn t ordered enough from Digikey. Downloading schematics and PCB artwork The hardware design is licensed under the TAPR Open Hardware License. It s all stored in git (of course): 
git://keithp.com/git/hw/fireworks
All of the symbols I created are also available through git: 
git://keithp.com/git/hw/keithp
Things I d do differently One of the frustrating things about moving atoms around instead of just bits is that once you ve got a physical artifact, it s really expensive to change it. There are a couple of minor things I would change if I were going to build more of these: Things that worked well There were a couple of things that went better than I had expected:

19 April 2012

James Morrison: Issues with Appengine

I like App Engine, my wedding website runs on App Engine as well as a couple recent commercial projects. However, there are some annoyances that make a lot of the useful things I learned at Google not applicable.

  1. Chunked responses
    App Engine doesn't have a way of doing chunked responses.
    1.5. Processing after sending the responseMy annoyance here is that I can't send an early response and wait for some resource. Say I want to do 5 url fetches that I cache for a short period of time. If I want to reply in 250ms, I want to use any of the results I've gotten and reply to the user. Then once the rest of the fetches come in, I want to put them in the cache, so they will come back quickly the next time.
  2. Counters
    App Engine has crappy support for counters. You can shard your counters into the datastore: https://developers.google.com/appengine/articles/sharding_counters#counter_python or you can use taskqueues: http://blog.appenginefan.com/2009/10/non-sharded-counters-part-2-using-task.html . Now, how do you graph these counters over time, well, that is left as an exercise for the reader. Google has special datastores for stores and retrieving counters.
  3. Backends
    Backends in appengine are versioned separately from frontends. This sounds like a good idea, but to run the devserver with backends, the backends need to be in the same source tree. Thus backends share the same url space. Backends also share the same app.yaml, so again the url space is shared. Thus, if you make a change that you want to coordinate between the frontend and the backend, you need to change the application version in app.yaml, then rename all the backends in backends.yaml. If backends are supposed to be in the same source tree as frontends, then they should be prefixed with the application version. If backends should be versioned separately from frontends, then they shouldn't share url handlers from app.yaml.
    3.5 Uploading backends is slow
    Each backend is uploaded through appcfg.py separately. So if you are crazy and have one backend per backend level, then you need to upload 4 copies of the source code each time the backends get updated.
  4. Logging
    I want structured logs. Instead what you have to do is use the python logging module to write out human readable logs, then parse them out into the structure you want.
    http://code.google.com/p/google-app-engine-samples/source/browse/trunk/logparser/logparser.py

As a preemptive counter, 1.5 could be dealt with using backends and a background thread, except backends don't scale to the number of queries, so my use case above isn't fixed.

Pablo Lorenzzoni: Timeout a process in Bash > v4

Just for reference, this is really useful: 
( cmdpid=$BASHPID; (sleep 10; kill $cmdpid) & exec some_command )
Update Apr 20, 2012 @ 16:54: As pointed in a comment by Timo Juhani Lindfors, if some_command exits early and the interval is long, another process can reuse its process number and get killed once the sleep runs out. Does anybody know a better way of doing that without using timeout from coreutils (better yet: using just bash)?

18 April 2012

Wouter Verhelst: Screen scraping sucks

At a customer, I've migrated a number of manually-maintained servers to having them be maintained through puppet not so long ago. Since then, some more machines have been added, and getting them up and running properly was a breeze: do a base install, install puppet, sign the certificate, restart puppet, and then wait and twiddle thumbs while puppet did its magic. Easy as pie. Now, a few months later, we needed to install a number of windows machines for a lab (not my choice), and the person involved asked me to figure out some diskspace so we could start creating images for those. Not a chance. Instead, I suggested looking for a configuration management system, similar to puppet. Since we're using Samba 3 to run the Windows network here, dropping everything in Active Directory was not an option. But a short while later, he came back with the note that puppet, in its 2.7 version, actually does support Windows as a platform for the managed machines. Interesting. The unfortunate bit was that puppet supports creating files and installing software when it is distributed as an MSI file, but not when it's distributed as a .exe file. This is not unexpected; MSI files can be installed noninteractively; but when something is distributed as a .exe file, it means it needs to be installed interactively; and puppet does not have the ability to interact with GUI software. The workaround: use something that does have that ability (in my case, autoit), and use an exec block in puppet to make it call those scripts. In effect, that's a bit like screenscraping. Add a creates stanza to the block, so that the installer isn't started again if the software at hand has already been installed. This 'autoit' thing also comes with a recording utility, allowing one to create an initial script by just doing the installation, and having the tool just record stuff. With that, the machines are installed 99% automated. I say 99%, because there are still some issues: I'll have to think about this some more, I guess. First, it's clear that while puppet does have some Windows functionality, it's not entirely ready yet. And somehow, using autoit to add to Puppet functionality feels like an ugly hack. We'll see what the future brings.

Richard Hartmann: Potpourri I

Google rant Seems venting can be a worthwhile once in a while; the content of this post made its way into Google. No promises of any kind were made, but it's good to know that this is getting some exposure within Google :) That was quick. moreutils et al Steve Kemp is right; moreutils is very useful in some situations. And yes, it's a pity that we all have our own scripts without a central repository to host them all. The problem is one of diminishing returns and information overload; if any given collection becomes too large, it becomes tedious to look through it. You may have the Most Awesome Ever solution to any given problem, but unless whoever needs it can find it quickly, it's useless to them. Thorsten Glaser suggested this collection, but it suffers from the same basic problem: Not enough content? You won't find what you need. Too much content? You won't find what you need. Finding the right balance is highly non-trivial. Linux storage Russell Coker shared yet another chapter of his continuing quest for modern, reliable, software-based storage on Linux. The short version is "ZFS is not that good on Linux while btrfs is not ready for prime time, yet." That's, unfortunately, not much of a surprise. fsck.btrfs is still too new and the lack of RAID 5/6 is an absolute show-stopper as far as I am concerned. COW solves the writing speed issues so, hopefully, that old argument can finally be laid to rest. And on top of the "wastng" of more disks for RAID 10, the minimum number of failed disks that can lead to data loss on RAID 10 is two whereas it's three for RAID 6. No-brainer, imo. Anyone who does not agree with Dell's pricing structure should seriously consider looking at SuperMicro, by the way. Cheap(ish), reliable, high disk densities. Password hashes Saku Ytti talks about password hashing and the relative merits of different algorithms. While I agree that bcrypt incorporates a good approach inasmuch it's designed to be slow to compute and can be made slower as needed, I think he focuses on one aspect while disregarding all others. Still, the premise that deterring brute force attacks on known hashes is the only concern is wrong. Avoiding collisions, reasonable certainty that there are no computational shortcuts, irreversibility, pseudo-random output, full use of the output's possible values, and thorough analysis by experts in the field are important concerns, as well. Brute forcing known hashes is hardly the only attack. When brute forcing a service, i.e. against an unknown hash, no matter if the system is remote or local, the computational cost of hashing is mostly irrelevant. Rainbow tables help mitigate computational cost when trying to extract clear text in case the hash is known. And if there are a lot of collisions or an easy way to find them, knowing the clear text password is not needed. Simply use whatever generates the correct output and you're done. Again, I am not saying that bcrypt is a bad idea; it's just that there are more concerns than computational cost alone. Travel If you know your way around Novosibirsk, please contact me. We have almost no idea what to do while there other than visiting the railroad museum; part of the reason why we are stopping there is to have a break after ~50 hours on the train and because Novosibirsk is just an awesome name. Our focus while there will be on seeing interesting industrial or research sites. Given that there are large industrial complexes and formerly secret cold war secret research cities situated around Novosibirsk it would be a pity not to see any of them. We may end up hiring a taxi via the hotel and having that drive us around a bit. Somewhat boring and potentially costly, but a feasible backup. Blogging frequency I have to admit being am a bit worried about spamming the various aggregators which are fed from this blog. Writing this catchall post is an attempt to mitigate this. While I am getting a surprising amount of positive feedback, I would be interested to hear anything negative, if applicable, as well. You know how to reach me.

17 April 2012

Russell Coker: The Most Important things for running a Reliable Internet Service

One of my clients is currently investigating new hosting arrangements. It s a bit of a complex process because there are lots of architectural issues relating to things such as the storage and backup of some terabytes of data and some serious computation on the data. Among other options we are considering cheap servers in the EX range from Hetzner [1] which provide 3TB of RAID-1 storage per server along with reasonable CPU power and RAM and Amazon EC2 [2]. Hetzner and Amazon aren t the only companies providing services that can be used to solve my client s problems, but they both provide good value for what they provide and we have prior experience with them. To add an extra complication my client did some web research on hosting companies and found that Hetzner wasn t even in the list of reliable hosting companies (whichever list that was). This is in some ways not particularly surprising, Hetzner offers servers without a full management interface (you can t see a serial console or a KVM, you merely get access to reset it) and the best value servers (the only servers to consider for many terabytes of data) have SATA disks which presumably have a lower MTBF than SAS disks. But I don t think that this is a real problem. Even when hardware that s designed for the desktop is run in a server room the reliability tends to be reasonable. My experience is that a desktop PC with two hard drives in a RAID-1 array will give a level of reliability in practice that compares very well to an expensive server with ECC RAM, redundant fans, redundant PSUs, etc. My experience is that the most critical factor for server reliability is management. A server that is designed to be reliable can give very poor uptime if poorly maintained or if there is no rapid way of discovering and fixing problems. But a system that is designed to be cheap can give quite good uptime if well maintained, if problems can be repidly discovered and fixed. A Brief Overview of Managing Servers There are text books about how to manage servers, so obviously I can t cover the topic in detail in a blog post. But here are some quick points. Note that I m not claiming that this list includes everything, please add comments about anything particularly noteworthy that you think I ve missed.
  1. For a server to be well managed it needs to be kept up to date. It s probably a good idea for management to have this on the list of things to do. A plan to check for necessary updates and apply them at fixed times (at least once a week) would be a good thing. My experience is that usually managers don t have anything to do with this and sysadmins either apply patches or not at their own whim.
  2. It is really ideal for people to know how all the software works. For every piece of software that s running it should either have come from a source that provides some degree of support (EG a Linux distribution) or be maintained by someone who knows it well. When you install custom software from people who become unavailable then it puts the reliability of the entire system at risk if anything breaks then you won t be able to get it fixed quickly.
  3. It should be possible to rapidly discover problems, having a client phone you to tell you that your web site is offline is a bad thing. Ideally you will have software like Nagios monitoring the network and reporting problems via a SMS gateway service such as ClickaTell.com. I am not sure that Nagios is the best network monitoring system or that ClickaTell is the best SMS gateway, but they have both worked well in my experience. If you think that there are better options for either of those then please write a comment.
  4. It should be possible to rapidly fix problems. That means that a sysadmin must be available 24*7 to respond to SMS and you must have a backup sysadmin for when the main person takes a holiday, or ideally two backup sysadmins so that if one is on holiday and another has an emergency then problems can still be fixed. Another thing to consider is that an increasing number of hotels, resorts, and cruise ships are providing net access. So you could decrease your need for backup sysadmins if you give a holiday bonus to a sysadmin who uses a hotel, resort, or cruise ship that has good net access. ;)
  5. If it seems likely that there may be some staff changes then it s a really good idea to hire a potential replacement on a casual basis so that they can learn how things work. There have been a few occasions when I started a sysadmin contract after the old sysadmin ceased being on speaking terms with the company owner. This made it difficult for me to learn what s going on.
  6. If your network is in any way complex (IE it s something that needs some skill to manage) then it will probably be impossible to hire someone who has experience in all the areas of technology at a salary you are prepared to pay. So you should assume that whoever you hire will do some learning on the job. This isn t necessarily a problem but is something that needs to be considered. If you use some unusual hardware or software and want it to run reliably then you should have a spare system for testing so that the types of mistake which are typically made in the learning process are not made on your production network.
Conclusion If you have a business which depends on running servers on the Internet and you don t do all the things in the above list then the reliability of a service like Hetzner probably isn t going to be an issue at all. Related posts:
  1. Servers vs Phones Hetzner have recently updated their offerings to include servers with...
  2. Why Internet Access in Australia Sucks In a comment on my post about (relatively) Cheap Net...
  3. The National Cost of Slow Internet Access Australia has slow Internet access when compared to other first-world...

Russell Coker: ZFS vs BTRFS on Cheap Dell Servers

I previously wrote about my first experiences with BTRFS [1]. Since then I ve been using BTRFS on more systems and have had good results. The main problem I want to address is with the reliability of RAID [2]. Requirements for a File Server Now one of my clients has a need for a new fileserver. They need to reliably store terabytes of data (currently 6TB and growing) which is mostly comprised of data files in the 10MB 15MB size range. The data files will almost never be re-written and I anticiapte that the main bottleneck will be the latency of NFS and other network file sharing protocols. I would hope that saturating a GigE network when sending 10MB data files from SATA disks via NFS, AFS, or SMB wouldn t be a technical challenge. It seems that BTRFS is the way of the future. But it s still rather new and the lack of RAID-5 and RAID-6 is a serious issue when you need to store 10TB with today s technology (that would be 8*3TB disks for RAID-10 vs 5*3TB disks for RAID-5). Also the case of two disks entirely failing in a short period of time requires RAID-6 (or RAID-Z2 as the ZFS variant of RAID-6 is known). With BTRFS at it s current stage of development it seems that to recover from two disks failing you need to have BTRFS on another RAID-6 (maybe Linux software RAID-6). But for filesystems based on concepts similar to ZFS and BTRFS you want to have the filesystem run the RAID so that if a block has a filesystem hash mismatch then the correct copy can be reconstructed from parity. ZFS seems to be a lot more complex than BTRFS. While having more features is a good thing (BTRFS seems to be missing some sysadmin friendly features at this stage) complexity means that I need to learn more and test more before going live. But it seems that the built in RAID-5 and RAID-6 is the killer issue. Servers start becoming a lot more expensive if you want more than 8 disks and even going past 6 disks is a significant price point. As 3TB disks are available an 8 disk RAID-6 gives something like 18TB usable space vs 12TB on a RAID-10 and a 6 disk RAID-6 gives about 12TB vs 9TB on a RAID-10. With RAID-10 (IE BTRFS) my client couldn t use a 6 disk server such as the Dell PowerEdge T410 for $1500 as 9TB of usable storage isn t adequate and the Dell PowerEdge T610 which can support 8 disks and costs $2100 would be barely adequate for the near future with only 12TB of usable storage. Dell does sell significantly larger servers such that any of my clients needs could be covered by RAID-10, but in addition to costing more there are issues of power use and noise. When comparing a T610 and a T410 with a full set of disks the price difference is $1000 (assuming $200 per disk) which is probably worth paying to delay any future need for upgrades. Buying Disks The problem with the PowerEdge T610 server is that it uses hot-swap disks and the biggest disks available are 2TB for $586.30! 2TB*8 in RAID-6 gives 12TB of usable space for $4690.40! This compares poorly to the PowerEdge T410 which supports non-hot-swap disks so I can buy 6*3TB disks for something less than $200 each and get 12TB of usable space for $1200. If I could get hot-swap trays for Dell disks at a reasonable price then the T610 would be worth considering. But as 12TB of storage should do for at least the next 18 months it seems that the T410 is clearly the better option. Does anyone know how to get cheap disk trays for Dell servers? Implementation In mailing list discussions some people suggest using Solaris or FreeBSD for a ZFS server. ZFS was designed for and implemented on Solaris, and FreeBSD was the first port. However Solaris and FreeBSD aren t commonly used systems so it s harder to find skilled people to work with them and there is less of a guarantee that the desired software will work. Among other things it s really convenient to be able to run software for embedded Linux i386 systems on the server. The first port of ZFS to Linux was based on FUSE [3]. This allows a clean separation of ZFS code from the Linux kernel code to avoid license issues but does have some performance problems. I don t think that I will have any performance issues on this server as the data files are reasonably large, are received via an ADSL link, and which require quite a bit of CPU time to process them when they are accessed. But ZFS-FUSE doesn t seem to be particularly popular. The ZFS On Linux project provides source for a ZFS kernel module which you can compile and load [4]. As the module isn t distributed with or statically linked to the kernel the license conflict of the CDDL ZFS code and the GPL Linux kernel code is apparently solved. I ve read some positive reports from people who use this so it will be my preferred option. Related posts:
  1. ECC RAM in a Cheap Machine In a comment on my previous post about ECC RAM...
  2. Starting with BTRFS Based on my investigation of RAID reliability [1] I have...
  3. Dell PowerEdge T105 Today I received a Dell PowerEDGE T105 for use by...

15 April 2012

Richard Hartmann: Choo choo two

Now that our departure is looming ahead of us, it's time to give a short update. (Turns out it's not so short, after all...) Preparation Visa Mongolian and Russian visas are done; we will apply for the Chinese ones next Monday. Travel All train tickets, hotels, hostels, and tents have been booked and both a horse-back tour through Mongolia and scuba diving in Lake Baikal have been pre-arranged. The Russian railway seems to do maintenance and development on the live website all the time which meant we were forced to book through a local travel agent. We paid a hefty extra charge for that privilege but still, that was less than half of what we would have paid if we had booked through a German agent. With the help of #china-lug on irc.freenode.net, we found accommodation in a traditional Siheyuan in an old Hutong. Hopefully, this will be a interesting as it sounds. Locals By the power of attending conferences and this blog, I now have local contacts in Moscow, Beijing, and Shanghai. Everyone who contacted me was extremely nice and helpful. I have been getting hints and tips galore; I doubt our preparations would be as neat as they are without this invaluable help. Thanks; you know who you are :) Packing list Apparently there's an interest in what we will be bringing; also, this will be a nice reminder for the future. Information and note-keeping Luggage Clothing Personal and computer security Fastening, storage, and tools Health Photography Minimal scuba gear Food Convenience Leisure That's it. Re-reading this list, I wonder how we managed to fit all of this into an 80-liter travel bag and a 15-liter backpack, but we did and with room to spare.

Richard Hartmann: All aboard the Choo Choo train!

All aboard the Choo Choo train! As some of you know, we are planning our trip on the Trans-Siberian (well, Trans-Mongolian to be exact) railway, at the moment. Given the wall of text my post about Svalbard turned out to be, I am trying to split early and often. The plan Our three-week itinerary is pretty much fixed, by now: Visa issues Not wanting to just book any random package with any random travel agent, we are picking and mixing our own, as usual. Given visa requirements, language barriers and things that are plain weird, it's been an interesting ride before we even pack our bags a month or two from now. Being European, visas are something that do not concern us a lot, normally. I am starting to realize how incredibly lucky we are... barring India, we never needed to do much to be allowed to enter any foreign country. Not so on this trip... While Mongolia has rather lax, or let's just say reasonable, terms, China and Russia are different. To get a visa for Russia, you need: For China, you need to provide a night-to-night itinerary of hotel stays. This may be a remnant of Mao's philosophy of restricting free movement of the population within China, I don't know. On the plus side, they don't require an invitation, health insurance or means of industrial espionage. With processing times ranging from one to four weeks per visa, you will wait one to two months until your passports are back with you and properly visa'ed by everyone. Tickets, please Purchasing tickets in advance, but not through a travel agent, is turning out to be rather complicated. As you can not book train tickets with RZD, the Russian rail company, more than 45 days in advance, you end up with nice circular dependencies. We could pay a premium for faster processing of our visas and buy via travel agencies which sell train tickets from their special contingents, but that's kinda boring, innit? Also, rzd.ru offers Russian order forms, only. With the help of Google translate, ##russian on irc.freenode.net, and a lot of guess-work, we were able to hammer out the connections from Moscow to Novosibirsk and from there on to Irkutsk. Yet, it's quite a different experience not to not be able to fall back to English. No matter where we went in the past, English was the lingua franca, at least to some extent. That's not the case in any of our three intended destinations so those phrase books will get used a lot, I suspect... We are still not sure if the 4-people cabins in second class have a door, let alone one that can be locked, but we decided that this is not too much of an issue. Generally speaking, people are nice and there are train attendants so crime should not be an issue. And even if things do disappear, as long as we stay healthy and I still have one of my several copies of all photos I will take, I will be reasonably happy. As to getting from Irkutsk to Ulan Bator... even a Russian travel agent we contacted claimed you can only purchase tickets on site, not online. From Ulan Bator to Beijing, it's a different booking system. And from Beijing to Shanghai, it's yet another one. This is the most fragmented booking process we ever went through. Other than a few moments of stress and frustration, the hunting for information, cross-referencing and having the occasional eureka moment is tons of fun, though; no complaints here. When in Rome... According to Lonely Planet, the Trans-Siberian is more or less one large picnic where everyone shares whatever they have with them with everyone else. I poked a few Russians about what typically German food they like and they came up with Frankfurters, mustard, and beer. Guess that's what we will be stuffing our backpack with once everything else is packed. And snuff; apparently it's common courtesy to offer your snuff to other men you meet in Mongolia and then snuff from their bottle. When in Rome... German snuff tends to be mixed with menthol, I suspect that will raise some eyebrows ;) No idea if there is any non-obvious social grease to bring to China; still working on that. Initially, we wanted to tour Mongolia's main attractions, but we would either have to sit in a car for days on end or go by plane. Driving for days in between a solid week(!) spent on trains is not very appealing and neither is breaking our personal ground travel record by cheating flying. Thus, we decided to take the eco-tourism route: After a two-hour orientation course in Mongolian behaviour and manners, we will travel a few hundred kilometers by public bus, be picked up by a local, driven out into the outback and ride from ger to ger by camel and horse before returning to Ulan Bator by bus. The stays at the gers are immersive, if short; we will live with the families, help them do their daily chores, and visit local places of interest in between. Oddly enough, I am looking to forward to the bus ride even more than to the gers. At the gers, at least one person is able to speak basic English. Not so with the local bus. People on the various trains will be used to strangers, as are the families at the gers. People on the bus are, most likely, not. Communicating with hand, foot, phrasebook, and a smile will, hopefully, be a lasting experience. If time permits, we will try to visit kiva.org borrowers while in Ulan Bator; this is something we never had a chance to do before and it's been on the bucket list for some time, now. As an aside, the local agent in Mongolia asked us if our sleeping bags are rated down to -26 degrees Celsius. I do hope he was joking... On the privileges of living in a first-world country... Travelling by horse means travelling light. Not a problem for us as we pack light by default and still manage to bring everything from Swiss Tool to water bottles, zip ties, medkit, and everything in between. Still, there's one major thing I have been taking for granted all my life: electricity. Being without any source of power for four to five days is... challenging. Even more so as every single gadget I rely on uses a different type of battery. Flashlight: CR123A and AA; GPS: Ni-MH AA rechargeables and Lithium AAA primaries respectively; Laptop, cameras, cell phone, ebook reader, Nintendo DS and MP3 player: proprietary. I am starting to truly understand why NATO has a hard rule of allowing AA-powered devices, only; planning spares is a pain. Obviously, I am focusing on flashlights, GPS, and, above all, cameras. Battery grips with AA adapters to the rescue! Another even more unsettling realization came when I asked if it would be possible to have boodog (vegetarians/vegans: don't click). Mongolians do not usually prepare boodog in spring as the animals which survived the harsh winter need to fatten and breed before being killed for food. All my life, I have never even once considered the remote possibility of not being able to slaughter domesticated animals due to outside constraints. Mongolian nomads need their animals. This is not about wanting to do things one way or another; this is about survival. Quite fascinating, and humbling, to think about. I am incredibly privileged and, as you are reading this, so are you. An exercise for the reader If you have any information regarding: please let me know. As usual, but just a little bit more than usual, comments are appreciated.

14 April 2012

NeuroDebian: NeuroDebian nd* tools

NeuroDebian nd* tools One of the goals of NeuroDebian is to provide recent versions of scientific software on stable Debian (and Ubuntu) deployments. That is why we build (whenever possible) every new package not only for the Debian unstable (the entry point of packages into Debian) but also for Debian testing and stable, and Ubuntu releases. To automate such procedure we prepared few rudimentary wrappers around cowbuilder allowing to build packages in isolated environment. Also we provide a backport-dsc script to ease backporting with optional application of per-release patchsets. In this blog post we would like to introduce you to these tools. They will be of use for anyone working on a package intended to be uploaded to NeuroDebian repository or anyone interested to verify if package could be easily backported. With a single command you will be able to build a given Debian source package across distributions. As a result you will verify that there are no outstanding backport-ability issues or compatibility problems with core components (e.g. supported versions of Python) if your source package excercises test suites at build time.
Procedure

  • [1-20 min] If you are not running Debian-based distribution, Install NeuroDebian VM; otherwise just add apt sources for NeuroDebian repository.

  • [<1 min] Install the neurodebian-dev package providing nd* tools: 

    sudo apt-get install neurodebian-dev

  • [1-5 min] Adjust default configuration (sudo vim /etc/neurodebian/cmdsettings.sh) used by nd commands to

    • point cowbuilderroot variable to some directory under brain account, e.g. ~brain/debs (should be created by you)
    • remove undesired releases (e.g. deprecated karmic) from allnddist and alldist
    • adjust mirror entries to use the Debian mirror and Ubuntu mirror of your choice or may be even point to your approx apt-caching server

  • [10-60 min] Create the COWs for all releases you left in the configuration file: 

    sudo nd_adddistall
Building At this point you should be all set to build packages for all distributions with a single command. E.g.: 
sudo nd_build4all blah_1-1.dsc
should take the .dsc file you provide, and build it for main Debian sid and all ND-supported releases of Debian and Ubuntu. nd_build4allnd would build only for the later omitting the vanilla Debian sid. The highlevel summary either builds succeed or failed get reported in summary.log in the same directory, pointing to .build log files for the corresponding architecture/release.
Troubleshooting Failing Build Provide --hookdir cmdline pbuilder argument to point to a hook which would get kicked in by pbuilder upon failure, e.g.: 
sudo apt-get install git
git clone https://github.com/neurodebian/neurodebian
sudo nd_build4debianmain *.dsc -- --hookdir $PWD/neurodebian/tools/hooks
If you have any comments (typos, improvements, etc) feel welcome to leave a comment below, or contact us .

Axel Beckert: Automatically hardlinking duplicate files under /usr/share/doc with APT

On my everyday netbook (a very reliable first generation ASUS EeePC 701 4G) the disk (4 GB as the product name suggests :-) is nearly always close to full. TL;DWTR? Jump directly to the HowTo. :-) So I came up with a few techniques to save some more disk space. Installing localepurge was one of the earliest. Another one was to implement aptitude filters to do interactively what deborphan does non-interactively. Yet another one is to use du and friends a lot ncdu is definitely my favourite du-like tool in the meanwhile. Using du and friends I often noticed how much disk space /usr/share/doc takes up. But since I value the contents of /usr/share/doc a lot, I condemn how Nokia solved that on the N900: They let APT delete all files and directories under /usr/share/doc (including the copyright files!) via some package named docpurge. I also dislike Ubuntu s solution to truncate the shipped changelog files (you can still get the remainder of the files on the web somewhere) as they re an important source of information for me. So when aptitude showed me that some package suddenly wanted to use up quite some more disk space, I noticed that the new package version included the upstream changelog twice. So I started searching for duplicate files under /usr/share/doc. There are quite some tools to find duplicate files in Debian. hardlink seemed most appropriate for this case. First I just looked for duplicate files per package, which even on that less than four gigabytes installation on my EeePC found nine packages which shipped at least one file twice. As recommended I rather opted for an according Lintian check (see bugs. Niels Thykier kindly implemented such a check in Lintian and its findings are as reported as tags duplicate-changelog-files (Severity: normal, from Lintian 2.5.2 on) and duplicate-files (Severity: minor, experimental, from Lintian 2.5.0 on). Nevertheless, some source packages generate several binary packages and all of them (of course) ship the same, in some cases quite large (Debian) changelog file. So I found myself running hardlink /usr/share/doc now and then to gain some more free disk space. But as I run Sid and package upgrades happen more than daily, I came to the conclusion that I should run this command more or less after each aptitude run, i.e. automatically. Having taken localepurge s APT hook as example, I added the following content as /etc/apt/apt.conf.d/98-hardlink-doc to my system: 
// Hardlink identical docs, changelogs, copyrights, examples, etc
DPkg
 
Post-Invoke  "if [ -x /usr/bin/hardlink ]; then /usr/bin/hardlink -t /usr/share/doc; else exit 0; fi"; ;
 ;
So now installing a package which contains duplicate files looks like this: 
~ # aptitude install perl-tk
The following NEW packages will be installed:
  perl-tk 
0 packages upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 2,522 kB of archives. After unpacking 6,783 kB will be used.
Get: 1 http://ftp.ch.debian.org/debian/ sid/main perl-tk i386 1:804.029-1.2 [2,522 kB]
Fetched 2,522 kB in 1s (1,287 kB/s)  
Selecting previously unselected package perl-tk.
(Reading database ... 121849 files and directories currently installed.)
Unpacking perl-tk (from .../perl-tk_1%3a804.029-1.2_i386.deb) ...
Processing triggers for man-db ...
Setting up perl-tk (1:804.029-1.2) ...
Mode:     real
Files:    15423
Linked:   3 files
Compared: 14724 files
Saved:    7.29 KiB
Duration: 4.03 seconds
localepurge: Disk space freed in /usr/share/locale: 0 KiB
localepurge: Disk space freed in /usr/share/man: 0 KiB
localepurge: Disk space freed in /usr/share/gnome/help: 0 KiB
localepurge: Disk space freed in /usr/share/omf: 0 KiB
Total disk space freed by localepurge: 0 KiB
Sure, that wasn t the most space saving example, but on some installations I saved around 100 MB of disk space that way and I still haven t found a case where this caused unwanted damage. (Use of this advice on your own risk, though. Pointers to potential problems welcome. :-)

Sandro Tosi: Mercurial: what's the tag containing this changeset?

In software development, you often need to know: when was this feature released? in which releases was this changeset included?

Typical example is: you're debugging a problem, you find out what's the changeset introducing it, now you want to know when that changeset was firstly released (i.e. included in a tag) - how to do that?

hg log -r "cset:: and tag()"

where cset is the changeset you're looking for.

So know I know that Sphinx new way to load searchindex.js was introduced in the this changeset, and released for the first time in 1.0b2:

$ hg log -r "423faa03c908:: and tag()"
changeset: 2261:b494009dccf1
tag: 1.0b2
user: Georg Brandl <georg@python.org>
date: Sun May 30 19:52:08 2010 +0200
summary: Release preparation for 1.0b2.
...

awesome!

PS: why did I need to know that? because Matplotlib is still using the old way, but Debian needs the new one.

Craig Small: PHP floats and locales

I recently had a bug report in jffnms that the SLA checks were failing with bizarre calculations. Things like 300% disk drive utilization and the like. Briefly, JFFNMS is written in PHP and checks values that come out of rrdtool and makes various comparisons like have you used more than 80% of your disk or have there been too many errors. The logs showed strange input variables coming in, all were integers below 10. I don t know of many 1 or 3 kB sized disk drives. What was going on? I ran a rrdtool fetch command on the relevant file and got output of something like 1,780000e+07 which for an 18GB drive seemed ok. Notice the comma, in this locale that s a decimal point hmm. In lib/api.rrdtool.inc.php there is this line around the rrdtool_fetch area: 
$value[] = ($line1[$i]=="nan")?0:(float)$line1[$i];
A quick check and I was finding that my 1,7 e+07 was coming back as 1. We had a float conversion problem. Or more specifically, php has a float conversion problem. I built a small check script like the following: 
setlocale(LC_NUMERIC,'pl_PL.UTF-8');
$linfo = localeconv();
$pi='3,14';
print "Decimal is \"$linfo[decimal_point]\". Pi is $pi and ".(float)($pi)."\n";
print "Half is ".(1/2)."\n";
Which gave the output of:
Decimal is , . Pi is 3,14 and 3 Half is 0,5
So PHP is saying that decimal point is a comma and it uses it BUT if a string comes in with a comma, its not a decimal point. Really?? Are they serious here? I tried various combinations and could not make it parse correctly. The fix was made easier for me because I know rrdtool fetch only outputs values in scientific notation. That means if there is a string with a comma, then it must be a decimal point as it could never be used for a thousands mark. By using str_replace to replace any comma with a period the code worked again and didn t even need the locale to be set correctly, or that the locale for the shell where rrdtool is run is the same as the locale in php.
Related articles

13 April 2012

Lucas Nussbaum: Update of Debian Packaging Tutorial

I ve just updated the Debian Packaging Tutorial. This new version addresses a few comments and questions I received over the past months. Note that there are also french and spanish versions of the tutorial, and I m of course open to adding other translations. The tutorial can be found in the packaging-tutorial package (PDF files are in /usr/share/doc/packaging-tutorial/), or on www.debian.org (see links above).

Martin F. Krafft: Mouse on Mars

The atmosphere in Munich s Backstage Werk just before the opening act to the Mouse on Mars was very chilled. People sat on the stairs or scattered themselves over the dance floor while low-fi ambient tunes came from the speakers. It wasn t loud, you had to try hard to hear the people mumble. I have no idea who the opening act was, and their first tune was very nice and groovy. Then ensued a noise explosion, one could only pity the electronic equipment that was being asked to perform in ways that may be described as everything else than you expect , and of course, the base beat shook the building; I am quite sure they didn t use treble at all, but I may also simply have been unable to hear it. Plus, it seemed to us that the musicians catered for what may be a widespread decrease of attention span: it was noticable how they jumped from one thing to the next, not leaving them (or their listeners) any time to get in the groove. My brother and I went outside for a bit and talked about today s music and its simplicity. We postulated the repetitiveness as the basis of a mass movement, considered scene clubs that played heavy techno to an audience that is so entirely different to who historically frequented such musical performances, and in general tried to avoid assuming a position between simplifying society and accepting that individual freedom is as eclectic as can be. When MoM opened, they continued pretty much in line with their openers and half way through the first tune, I started to wonder how long I would last, or when it would be reasonable to step outside again. I had been a little afraid this would happen, having bought and listened to their latest album Parastrophics in preparation of the concert and not being able to get into it. However, what then followed blew us away. Still heavy, still all over the place, but now they were developing sound scenes, ripping them apart, having fun playing with and teasing the audience, while putting on a groove that inevitably made your muscles twitch with the beat. David Bowie called MoM the next big thing and I have to give it to them: MoM have always had a certain aura of that s what your music is like? we can improve on that! to them, and yesterday, they continued along those lines with astounding consistency, and it felt fresh. It also felt real. They weren t just pushing buttons and computers making music, they were making music and the computers were their instruments. Between the two founding members of MoM sat Dodo Nkishi, drummer and microphone artist, and if you don t believe, fast, big breakbeat can be performed live, well, you re wrong. Most everyone in the room was dancing. And while I was more swaying in awe, watching and wondering how the heck they are doing what they are doing, I couldn t contain the bouncing any longer. They came back for an encore and there was no more stopping the crowd, Thomas or me. Three tracks later, they waved goodbye and left, but a bunch of us simply continued to dance. Thomas questioned who would last longer and I started yelling loudly for another encore. The lights turned on, I considered it a slap in the face, but I did not stop yelling. Others tuned in. And then the lights went off and the band came back. Following their 2.5 hour show, gosh was I exhausted. It was a magnificent show. If you aren t afraid of big beat electronica and you take pleasure in nonstandard art, I heartily recommend you ensure that MoM aren t soon playing near you without you there. PS: MoM will play at the (D sseldorf Open-Source Festival)[http://www.open-source-festival.de/en/] on 30 June 2012! PPS: Now I listen to Parastrophics and I am really enjoying it. NP: Mouse on Mars: Parastrophics

12 April 2012

Thomas Koch: No more self-hosting

A few years ago I started to run my own server. This was an interesting challenge back then but today I only consider it a waste of time and money. Therefore I'm searching for (a) service provider(s) to substitute my self hosted applications:

However I'm a bit insane about privacy, security and reliability. I simply want to be able to TRUST. This rules out IMHO: Google, Github, Yahoo. On the other hand I'd probably be prepared to pay an insane monthly fee for a provider that just keeps me happy.

I'd love to here your recommendations! I'm especially interested in providers that are organized as cooperatives and care about environmental issues.

Do I need to mention free software? An extra plus would be a healthy distaste for PHP.

some cooperative-style provider initiatives that I still need to evaluate:

Next.

Previous.