Search Results: "ressu"

10 December 2008

Russell Coker: New version of Bonnie++ and Violin Memory

I have just released version 1.03e of my Bonnie++ benchmark [1]. The only change is support for direct IO in Bonnie++ (via the -D command-line parameter). The patch for this was written by Dave Murch of Violin Memory [2]. Violin specialise in 2RU storage servers based on DRAM and/or Flash storage. One of their products is designed to handle a sustained load of 100,000 write IOPS (in 4K blocks) and 200,000 read IOPS per second for it’s 10 year life (but it’s not clear whether you could do 100,000 writes AND 200,000 reads in a second). The only pricing information that they have online is a claim that flash costs less than $50 per gig, while that would be quite affordable for dozens of gigs and not really expensive for hundreds of gigs, as they are discussing a device with 4TB capacity it sounds rather expensive - but of course it would be a lot cheaper than using hard disks if you need that combination of capacity and performance. I wonder how much benefit you would get from using a Violin device to manage the journals for 100 servers in a data center. It seems that 1000 writes per second is near the upper end of the capacity of a 2RU server for many common work-loads, this is of course just a rough estimation based on observations of some servers that I run. If the main storage was on a SAN then using data journaling and putting the journals on a Violin device seems likely to improve latency (data is committed faster and the application can report success to the client sooner) while also reducing the load on the SAN disks (which are really expensive). Now given that their price point is less than $50 per gig, it seems that a virtual hosting provider could provide really fast storage to their customers for a quite affordable price. $5 per month per gig for flash storage in a virtual hosting environment would be an attractive option for many people. Currently if you have a small service that you want hosted a virtual server is the best way to do it, and as most providers offer little information on the disk IO capacity of their services it seems quite unlikely that anyone has taken any serious steps to prevent high load from one customer from degrading the performance of the rest. With flash storage you not only get a much higher number of writes per second, but one customer writing data won’t seriously impact read speed for other customers (with hard drive one process that does a lot of writes can cripple the performance of processes that do reads). The experimental versions of Bonnie++ have better support for testing some of these usage scenarios. One new feature is measuring the worst-case latency of all operations in each section of the test run. I will soon release Bonnie++ version 1.99 which includes direct IO support, it should show some significant benefits for all usage cases involving Violin devices, ZFS (when configured with multiple types of storage hardware), NetApp Filers, and other advanced storage options. For a while I have been dithering about the exact feature list of Bonnie++ 2.x. After some pressure from a contributor to the OpenSolaris project I have decided to freeze the feature list at the current 1.94 level plus direct IO support. This doesn’t mean that I will stop adding new features in the 2.0x branch, but I will avoid doing anything that can change the results. So in future benchmark results made from Bonnie++ version 1.94 can be directly compared to results that will be made from version 2.0 and above. There is one minor issue, new versions of GCC have in the past made differences to some of the benchmark results (the per-character IO test was the main one) - but that’s not my problem. As far as I am concerned Bonnie++ benchmarks everything from the compiler to the mass storage device in terms of disk IO performance. If you compare two systems with different kernels, different versions of GCC, or other differences then it’s up to you to make appropriate notes of what was changed. This means that the OpenSolaris people can now cease using the 1.0x branch of Bonnie++, and other distributions can do the same if they wish. I have just uploaded version 1.03e to Debian and will request that it goes in Lenny - I believe that it is way too late to put 1.9x in Lenny. But once Lenny is released I will upload version 2.00 to Debian/Unstable and that will be the only version supported in Debian after that time.

27 November 2008

Russell Coker: Links November 2008

Netatia has an interesting series of articles about running a computer for two people [1]. It is a bit of a kludge, they have a single X server that covers both displays and then use Xephyr to divide it into two virtual screens. The positive aspecct of this is that it shuld allow a single wide monitor to be used by two sessions as displays are getting wider regardless of the wishes of manufacturers and consumers [2] this should be useful. It’s a pity that no-one has solved the problem of having multiple video cards, sound cards, and input devices to allow a single desktop system to be used for 6 or more people. It seems that the problems that need to be solved are only the support for multiple video cards, mouse-wheel support, and sound support. Paul Ewald gave an interesting TED talk about changing the conditions for diseases so that they evolve to be benign [3]. The first example is Cholera which if spread by water will benefit from being as toxic as possible (to cause the greatest amount of diorrhea - killing the host not being a problem), but if spread by human contact benefits from leaving it’s host well enough to walk around and meet people. This and the other examples he cites seem like strong reasons for universal health-care provided by the government. If clean water is provided to all the poor people then cholera will evolve to be less harmful, and if a rich person (such as myself) is unlucky enough to catch it then the results won’t be so bad. He also notes that less harmful bacteria will often result in the victim not seeking anti-biotics and therefore less pressure for the disease to evolve resistance to anti-biotics. Therefore the people who really need them (the elderly, the very young, and people who are already sick) will find them to be more effective. Paul Stamets gave a great TED talk about fungus [4]. One of his discoveries was that fungi can be used for breaking down petro-chemicals (they can eat oil). It would be interesting to see this tested on a large scale with one of the oil spils or with the polluted land around an ooil refinery. Also he has patented a method for using fungus to kill wood-eating ants (such as the ones that briefly infested his home). Robert Full gave an interesting TED talk on robot feet [5]. I found the bit about leg spikes particularly interesting (I had always wondered why insects have spikey legs). Alan Kay gave a very interesting presentation on using computers to teach young children about science [6]. An OLPC is referenced. It makes me want to buy an OLPC for everyone I know who has young children. The start of the talk is a little slow. Dan Barber gave a very interesting TED talk about organic and humane production of foie gras in Extramuda [7]. Apparently it tastes a lot better too. Incidentally I don’t list all the TED talks I watch, only the better ones. Less than half the TED talks that I see announced seem interesting enough to download, and of those less than half are good enough that I will recommend them. The ones that I don’t recommend don’t suck in any way, it’s just that I can’t write a paragraph about every talk. Of recent times my video watching has been divided about equally between “The Bill” and TED talks. Here’s an interesting article about Sarah Palin and “anti-elitism”: The prospects of a Palin administration are far more frightening, in fact, than those of a Palin Institute for Pediatric Neurosurgery. Ask yourself: how has “elitism” become a bad word in American politics? There is simply no other walk of life in which extraordinary talent and rigorous training are denigrated. We want elite pilots to fly our planes, elite troops to undertake our most critical missions, elite athletes to represent us in competition and elite scientists to devote the most productive years of their lives to curing our diseases. And yet, when it comes time to vest people with even greater responsibilities, we consider it a virtue to shun any and all standards of excellence. When it comes to choosing the people whose thoughts and actions will decide the fates of millions, then we suddenly want someone just like us, someone fit to have a beer with, someone down-to-earth in fact, almost anyone, provided that he or she doesn’t seem too intelligent or well educated.[8] Sarah will be representing the Republican party in 2012, the desire for leaders of average intelligence (or less) will still be around then. It will be interesting to see how many votes she gets and amusing to see her interviewed. The proceedings of the “Old Bailey” - London’s Central Criminal Court have been published [9]. It’s interesting to read some of the historical information about the legal system at the time. It made me appreciate how civilised the UK (and other countries that I have visited) are now. Bruce Schneier writes about the feture of ephemeral communication [10]. He concludes with the point “until we have a Presidential election where both candidates have a complete history on social networking sites from before they were teenagers we aren’t fully an information age society“. Of course as he notes the rules are written by the older people, currently I don’t think that any candidate for high office (cabinet minister or above) anywhere in the world can have a good history on the Internet. During the course of a decade or more on the net it’s impossible not to write something that can be used against you and no reasonable person could avoid changing their views on some issues in such a time period. That’s enough to lose an election with the way things currently work.

24 November 2008

Gunnar Wolf: It's just a different mindset. Not necessarily a _sane_ one, though...

Wouter insists that Ruby Gems are enough of an argument to keep Rails at a distance. Even though I agree with the basic claim and think that Gems are basically insane and sick, this should be taken a bit more under perspective.
We are blessed. We are blessed to have Debian, such a rich OS with such a great package management system, and with superb integration between so many packages. Blessed are also the users of most Free Software based distributions, as they share the advantages of systems growing with full consciousness of their interaction's benefits. However, integration with the rest of the world is not seamless.
Most scripting languages have their own infrastructure for managing the modules/libraries/pacakges/whatevers dependencies. Perl has CPAN, PHP has PEAR, Ruby has Gems... I do see Gems as the most obnoxious of them all, but the basics are the same.
The Rails crowd started being Unix-centric, but the Windows (and MacOS - they are no better, believe me, at least in this regard) world has exerted its pressure. Gems caters very well to their needs, but we do suffer the integration at the distro side.
The only sane way the propietary-minded people have managed to stay clear of the well known "DLL hell" is to ship everything a given program requires bundled together - that's the main reason for the bloat of lots of applications, and for the sloppiness of security support. Every application packager is responsible for shipping updated versions to any library it bundles in, except for the very basic core that the OS itself provides. That seems so annoyingly backwards to us that... it is unbelievable.
So, yes, Rails application trees often include Rails itself. For $DEITY sake, even I have grown used to working that way, as things tend to break under your nose otherwise. My proposal (which we talked over at DebConf, but have not pushed so far) is to support simultaneous versions of Rails installed in a Debian system (of course, via different packages), more or less in the way that simultaneous versions of Ruby, PHP or Python (and, in some limited fashion, Perl - Although Perl does not suffer from this incompatible bumps. Yay for Perl!) can be installed.
...And, yes, together with the pkg-ruby-extras team, I have been trying to -slowly, yes- package whatever modules we often use so they don't have to be included in Rails applications.
So far, the best way (although by far not optimal) I have found to limit this explosion of trees is to include most libraries in my Rails application trees as git submodule trees - If they are not explicitly downloaded, the systemwide libraries will be used.
Yes, the "ship the whole thing as a bundle" approach is quite annoying. However, at least I must acknowledge that it works better than the approach I took with previous (mod_perl-based) webapps I wrote... As relatively few people grok mod_perl, I ended up with quite some apps which were not installed by anybody but me. Rails is obnoxious... but seems to be parsable by the humanity at large.

David Welton: Slicehost vs Linode

Since hosting is the major expense for DedaSys LLC, and obviously a critical part of much of what I do, getting the right one is very important. Naturally, "the right one" for me may not be the right one for everyone. I am a fair sysadmin for small numbers of machines, so I don't mind doing that myself - I don't need, nor want to pay for hand holding. I recently moved everything over to Slicehost on the recommendations of friends, and am so far fairly happy with the experience. It's cheaper, simpler, and more flexible than Layered Tech was. To boot, I also like the fact that Slicehost is run out of St. Louis, Missouri, rather than some really expensive tech center like San Francisco or Boston. Hosting is basically a commodity, and much like you wouldn't want to put a factory in San Francisco, you don't want your hosting their either. However, I discovered something annoying about Slicehost: they use x86_64 servers, which, per se, doesn't really matter to me - I use open source code that can run on any number of architectures. The problem is that this particular architecture uses more memory than plain old x86. Significantly more. My Layered Tech server with 1 gig of memory was hitting the swap space a bit, but the same code on a slice was swapping quite heavily, despite the fact that I'd moved PostgreSQL off to its own slice. Since I pretty much exclusively run Rails, I decided to look into Phusion's "Ruby Enterprise Edition", which is basically just some nice hacking on Ruby's garbage collection mechanism. What they've done is nice, and I may end up using it, but ultimately it's buying me space that I would also gain from simply moving back to x86. With that in mind, I decided to take another look at what appears to be emerging as one of Slicehost's main competitors, Linode, who *do* use x86 servers. Here are my results, which are admittedly not all that scientific, but what the heck - you're getting them for free, and they were pretty quick to whip up. Service, Support, Setup But before we begin with the numbers, I want to add a few words of caution. One of the big things about hosting, to me at least, is how they deal with unexpected random negative events. Connections going down, disk breakage, DoS attacks, and so on. It's really hard to get an idea of just what sort of people you're dealing with until you've "ridden the river together". I don't really know what sort of response times, uptime, and all that either Slicehost or Linode have, so there are potentially some big intangibles there that are not quite as easy to draw pretty charts for. In terms of the console/setup/management tools, I liked Slicehost's simplicity more - Linode gives you more options, but they're a bit fiddly and cluttered feeling at times. For instance, Linode lets you pick how much swap space to give your disk image, but doesn't include a bit of javascript to balance out the swap and regular partitions, so that if you type in a larger swap number, you have to do the math and subtract that from the other number. Annoying, but not a big deal. Linode lets you pick various data centers in the US (no Europe, yet). Slicehost gives you the option to do backups of your disk images, which is nice, and something that Linode lacks. Comparing plans Since I'm interested in comparing hardware and machines I actually have access to, here are the plans I am currently signed up with, and the salient numbers:
Slicehost 1024 Slicehost 512 Linode 720
Memory 1024 512 720
Bandwidth 400 200 400
Disk 40 20 24
Cost 70 38 39.95
Memory / Dollar: Bandwidth / Dollar: Disk / Dollar: Linode mostly comes out ahead, but not by that much, except for bandwidth. However, let's take a crude look at x86_64 vs x86 memory usage. First, ./script/console (a Rails console) for the same code base on two different machines, showing the virtual memory size (VSZ) and resident set size (RSS): And just to look at something else that's a bit smaller, I ran the following script: 
puts File.open("/proc/self/status", "r").read
And got the following data: Being charitable with those numbers, we get x86 taking up 77% the memory that x86_64 does. So let's add that to our comparison of how much memory you're getting per dollar on Slicehost and Linode: Wow! That's a fairly significant difference. Performance That's most of what we can glean from published numbers, and running a few simple experiments. However, there's another factor that's important: performance. "Virtual Private Server" systems, or simply VPS's, got a bad name in the past because they were "overbooked" - too many virtual servers competing for the same CPU resources on one machine. Slicehost and Linode both look like they want to avoid that kind of bad reputation, and so far all the systems I have used have felt snappy and responsive. Now here's where we get really unscientific... I decided to try and do something, even if it wasn't much, to get some kind of objective measurement of what kind of CPU I was getting on the machines. This is really unscientific, because the machines do have other duties (I don't have the cash, time or inclination to set up random servers just for testing), and of course, who knows what else is sharing the computer with my systems: if you get a relatively unused computer, you can apparently pick up extra cycles for yourself, over and above your guaranteed minimum. But c'est la vie, and so I decided to do some numbers anyway. I picked a C implementation of mandelbrot from the language shootout more or less "for the hell of it" (I'm going to keep repeating "unscientific" and invite someone to do better tests than I have). I ran this code every 20 minutes for a couple days, and then averaged out the run times: As expected, the higher power Slicehost 1024 machine wins out in terms of raw speed (less time is better), but not by that much over the Linode system. Indeed, when we factor in the relative prices, Linode comes out ahead, again: So it looks like we're not paying a CPU penalty for having more memory, bandwidth, disk, etc... Notes
  • Nothing in these statistics is indicative of future directions that both Linode and Slicehost might take. Slicehost was recently acquired by Rackspace, and that could have effects on the service they provide. Maybe one or both will come under financial pressure and start "overbooking" their servers.
  • The spreadsheet I used is available via Google docs, for the curious. I did most of the work in OpenOffice, and then uploaded it though, because working with the web spreadsheet was kind of painful. Unfortunately, you can't select non-contiguous data ranges in the Google spreadsheet, so the labels are interspersed with the data. Yuck. Also, Google docs doesn't handle 'time' values nicely, which is what the performance data was originally. The spreadsheet is here: http://spreadsheets.google.com/pub?key=pVDeZF6ire2YwATBYb4mBKQ
  • The standard deviation on the processing times is higher for both slices than the linode system. Not quite sure what that means in terms of what's going on under the hood, but I thought it was interesting.

21 November 2008

MJ Ray: Libraries, Cooperatives, OCLC and TTLLP

Thanks to Koha, TTLLP is partly a library services worker cooperative and just now there’s a massive flame-storm about the largest library services consumer cooperative - OCLC, the Online Computer Library Center - because it just updated the terms for sharing its book data collection. OCLC seems to be treating it as a private asset that it can exploit, not the common resource that many librarians thought it was. I’ve been watching this fire with interest - because, at best, I don’t think it’s helping other cooperatives that sell to libraries - and here is my summary. OCLC and the Great Library Scandal is a good introduction and there’s a summary of the Talis Podcast about OCLC WorldCat Record Use Policy with Karen Clahoun and Roy Tennant which reveals some of the OCLC thinking. Just like CC and Open Source? Still No is a great illustration of how some librarians think OCLC is reaching the wrong conclusion, which then leads to asking Is OCLC truly cooperative? and What would it look like if OCLC was broken up? A commenter on Tom Watson: Library data asks: “Why would libraries play this game?” Well, the road to hell is paved with good intentions and this road has taken 40 years. First off, OCLC is essentially a good idea - libraries cooperating to act as a counter-weight to the large library service companies. However, I believe OCLC predates many recent company rule innovations (stronger common asset locks) and data licensing innovations (Share-Alike, Fiduciary Licensing and the general free and open source software movements), so there’s nothing to stop them trying to privatise the assets (library data) that the libraries have given to OCLC and it was only a matter of time before someone tried. I’m having a similar discussion with another organisation that has been given data by its members under vague terms and is now apparently about to exploit that data for the organisation’s benefit, to the detriment of some members. TTLLP is a member but has not contributed much data because I was suspicious of the lack of terms. In time, hopefully, asset locks and more awareness of data licensing will eliminate these problems, but there’s going to be a lot more people getting burnt first. Finally, a quick shout-out to the Open Library Environment as a possible emerging alternative to OCLC for practical services. Raw Thought: Stealing Your Library: The OCLC Powergrab covers some of its motivation (and includes a link to a petition to OCLC, if you want something to sign), but I share Stefano’s (now there’s someone I’d not read in 7 or so years) analysis of some of the challenges:
“any grass-root approach that will get big enough to take on OCLC on the metadata collection and redistribution service that libraries need will have to incorporate under the pressure of its users (if only for legal liability protection) and will have to find an answer to the same set of problems (policy, governance, financial sustainability) that OCLC has.”

14 November 2008

John Goerzen: Education

One of the speakers at OSCon this year — I forget which one — made a point that ran something like this, heavily paraphrased:
Education used to be an end in itself, not a means. It wasn’t about having a high-paying career. It was about knowing the world, about having knowledge and wisdom for its own sake. It was, quite bluntly, the accumulation of useless knowledge by the elite — those that could afford to spend time on such things, knowing that useless knowledge has a way of becoming useful in the most unexpected of ways. How fortunate we are to live in an age where the accumulation of useless knowledge is available to so many, and how sad it is that so few take advantage of it.
What a powerful statement, and it rings true to me. I remember in high school, when people from the local liberal arts college would come and talk. They’d talk about the value throughout a lifetime of knowledge in a broad range of disciplines: English, history, political science, religion, science, and the arts. They’d talk about how their graduates went on to lead distinguished lives, how this broad core of knowledge serves a person well through life. I guess I didn’t believe them, because due to their lack of a computer science major, I went elsewhere. That local school may not have been the best choice for me for other reasons, but as I look back on it, I think they had a much stronger message than I realized back then. Here I am, just two math classes, one computer science class, and one biology class away from a degree. Yet I have had not one class covering the history of east Asia, not one class on different world cultures or religions, and only a very basic understanding of one foreign language (German). This hits me in the face almost every day. Yesterday I was wondering about the history of slavery and racism in Europe. Today I’m curious about China’s history as an economic powerhouse. Last week I was curious about Roman law and daily life. The fact is, everything from philosophy to calculus is screamingly relevant to daily, modern life. We hear talk of “an American revolution” in Washington, of a shift of power in the Senate. It seems we forget that the notion of a Senate is considerably older than the United States is — and that we have such a thing because our founders were aware of this. Macroeconomic theory is thrust in our faces on an almost daily basis these days, yet I’ve never had a class on economics at all. We might feel fear of terrorist attacks, or see our fellow citizens lash out at “the Arabs.” Our own short memories fail to remind us of the light in which we are seen, fail to put the really quite minor terrorist threat in context of what London or Dresden endured in World War II. We demand our government to make us safer, and our government responds by making us less safe but making us *feel* safer at airports. In my own field, I see some universities buckling to pressure from Business to turn out large numbers of mediocre programmers that know the Java or .NET standard library well, but have no sense of the theory behind computer science, and would be utterly lost if asked to, say, write a recursive QuickSort. I find myself almost completely baffled that some companies that want to hire the world’s best programmers are only looking for people that are already fluent in $LANGUAGE — not ones that are good programmers, and so well-versed in computer science that they can easily pick up any language. I think there is a lot to the argument that a good, broad, classical education can serve a person well in any career. I wish I had realized that a little earlier.

29 October 2008

Sami Haahtinen: OpenSSL is way too complicated

It's no wonder people hate dealing with certificates. I'm one of those people who really hate handling certificates. For some reason Linux lacks simple tools to manage certificates. Debian has one set of tools and I bet quite a few other distributions have their own set of tools. So nothing generic. In my opinion the problem lies in OpenSSL, which is an overly complicated piece of software. Sure it's able to encrypt your fries at the local fast food place, but most people never use anything more than the x509 module. And even that is complicated. I'm not saying that it should be point and click operation, but what I want is a tool that allows me to create, renew and verify SSL certificates. Which is pretty much the most common thing you do with OpenSSL. Now, lets assume you have a certificate and a key that you want to check if they match. Can you say from the top of your head how to do that? If so, you are either dealing with this stuff daily or you looked it up from a manual. The correct "magical mumbo jumbo" is: 
server:/tmp# openssl rsa -noout -modulus -in /etc/ssl/private/some.key   openssl md5
94c5fa8b0fe1baa6e8dfcdec75444620
server:/tmp# openssl x509 -noout -modulus -in signed.crt   openssl md5
94c5fa8b0fe1baa6e8dfcdec75444620
Now, how easy was that. It only took me 10 minutes to construct that line. Most of the time went to searching Zimbra scripts for the correct magical line. The reason why I went through the scripts instead of the manual is that I had already seen the scripts do a appropriate check. And there is no way I could have constructed that in that kind of a time frame just by using the manual. Usually I think that people complain too much when they can't figure out how to make ls or something work like they want. A certain degree of manual reading is good for you, but in this case it's too much. In any case, I'm posting this rant as a reminder for myself so that the next time I can just look it up from here.

3 October 2008

Adrian von Bidder: Desktop integration

Sune Vuorela raises a few interesting points about integration of various (meta)data storage frameworks on your KDE desktop (read the comments, too, as several of these issues have been or are being addressed.) A huge, somewhat related, itch to me is that the end user of a Linux desktop still has to care if an application uses KDE, GNOME, XFCE, GNUstep, old style X11 or whatever. As an end user, I don't care that GIMP is a GTK application. Somebody told me that I can use fish:/ URLs to open remote files and I've duly noted down the syntax because I have no idea what this does, but it does open this file the webdude has told me I should edit with kolourpaint, and now I want to use GIMP because it's much nicer to use ... Similar issues with settings (I did set the web proxy in the system settings, now why do half of the application not respect this?) and all kinds of other stuff. I believe it's issues like this that will hamper the Linux on the Desktop the most in future. Obviously, in a controlled (corporate) environment, this is not a big problem because it's a problem for the IT staff, but in the SOHO and home computer market, these are real, difficult problems, and since they don't know any of the technical problems behind it it's also very hard to explain why it's not easy for me to set up their system so it works like they feel it should (I already have problems explaining why there should be different Linux distributions at all ... ) Update: It occurs to me that this is the kind of stuff we (distribution developers) should be concentrating on, by putting pressure on upstream and doing some of the work. I guess we've concentrated too much on just packaging the stuff.

2 October 2008

Neil Williams: Time pressures

It's been really strange coming back from DebConf then Extremadura and having to fit my code time around paid work and housework and general stuff again. I keep wondering why I'm not getting things done and then realising that I've only got so many hours a week nowadays compared to the entire week during the worksessions. What makes it worse is the short gap between the conferences. I misjudged the length of time without income and didn't make any days available for paid work between DebConf and Extremadura. OK, it was only about a week but it was a further week without income followed by the week getting to and from Spain.
I'm now doing extra days to recover from the extra time without income and it makes the contrast even more noticeable. The tiredness from the extra work then makes it hard to get started on what I thought I'd get done. I've done a little bit on emdebian-tools and apt-cross, I've done a little bit on a few RC bugs and I've not done a load of stuff I thought I would have done by now.
If you're wondering why something I said I'd do has not been done, file a bug and I'll get to it. (The bug lists for buildd.emdebian.org or emdebian-tools are my main ToDo lists, if it appears in that list of bugs, I'll not forget. If it appears in an email, I probably will forget.)

31 August 2008

Biella Coleman: The making of

One of my favorite things to watch are the “making of [movie, documentary etc] segments that are now routinely included in any DVD. It is nice to look behind the curtain and see exactly what choices are made, what is excluded, and why they are made. I wish that more books, especially academic ones, had a “making of” section, giving a window into these choices. We are not exactly encouraged to include this commentary in the book itself but one can find this type of insight in author interview. I recently read one such interview with the author of Two Bits: The Cultural Significance of Free Software. The interview is on his site and the original is here. There are a couple of things that I found particularly interesting, such as how to treat iconic figures such as RMS and ESR [not as researchers, for example :-) ] and the importance of letting your research and thinking brew over a fairly long period of time, despite the pressure of publishing fast, quick and dirty, which is especially strong with anything “digital.” The book is hefty and long, but as is emphasized in the interview, it is also pragmatic and readable. If folks on Planet have not yet checked it out and are not ready to commit to a long book, I would checkout Chapter 6 on the creation of the copyleft. It is sure to please academics and geeks alike.

29 August 2008

Aigars Mahinovs: 24th September as a day against software patents - Debian support?

According to the Slashdot article and the StopSoftwarePatents.org website itself and on Digg, the anti-software-patent activists are attempting a world-wide event on the 24th of September as a world-wide day against software patents. USA has them via a weird court ruling, Japan has them as well (not sure why), there have been efforts to force software patents on EU, India, Australia and many other countries either by Microsoft lobbies or even via US trade treaty pressure. While lobbying by local Microsoft branches and their pet companies can be countered locally, like I and many others did in EU a couple years ago (mostly thanks to FFII), trade negotiations are very secretive affairs and it is very hard to lobby there directly. The public needs to be aware of the issue, otherwise the politicians will not be aware of its importance to their voters. Software patents are a threat to free software as it circumvents the power of GPL and other copyright licenses if enough money is thrown at the lawyers. If a software patent is generic enough, it can easily stop development of a whole class of free software applications on a whim of the patent holder. And there are plenty of granted software patents with a very broad scope (progress bar, anyone?). I am aware that to get Debian support for such an initiative, a GR is needed, but how about a personal word of support from the DPL or support from the SPI? It is in direct area of interest for SPI - software patents create an ever present threat of legal action against any and all software in Debian (in USA and Japan, at least) and, as the legal umbrella of Debian in the USA, SPI would be a prime target. What do others think about this?

20 August 2008

Sami Haahtinen: Setting up an OpenVPN tunnel

Recently I fixed my OpenVPN tunnel and figured out what was wrong with my bridge setup. Here is the complete setup that I have. Server I wanted to have as much of the configuration on the server as possible so that I could easily add more clients and wouldn't have a need to update client configuration when ever the server preferences change. Here is my OpenVPN server configuration: 
mode server
tls-server
dev tap0
server-bridge 10.5.6.1 255.255.255.0 10.5.6.200 10.5.6.209
ifconfig-pool-persist /var/run/openvpn-ip.txt
keepalive 10 60
ping-timer-rem
persist-tun
push "route 10.5.6.0 255.255.255.0"
push "dhcp-option DNS 10.5.6.1"
ca /root/easy-rsa-2.0/keys/ca.crt
key /root/easy-rsa-2.0/keys/ressukka.net.key
cert /root/easy-rsa-2.0/keys/ressukka.net.crt
dh /root/easy-rsa-2.0/keys/dh1024.pem
client-to-client
up /etc/openvpn/ovpn-ressukka.sh
and the up script: 
#!/bin/sh
# Bind the tunnel interface to the bridge
BRIDGE=br0
ifup $1
brctl addif $BRIDGE $1
There is nothing really special about that configuration. The server is in TLS mode configures a bridge. The keys are generated with easy-rsa by following a openvpn howto entry. The up script just binds the tap0 device to the network bridge after bringing up the device. Next I created the interface configuration by adding the following to /etc/network/interfaces: 
iface br0 inet static  
 address 10.5.6.17  
 gateway 10.5.6.1  
 netmask 255.255.255.0  
 bridge-ports eth0
The trick here is to create a single bridge with just the eth0 device. We use the up script for openvpn to add the tunnel device to the bridge. Otherwise the bridge would never contain the proper devices. Client As for the client, you simply set the client to use the CA-certificate and Host key created with easy-rsa, set the hostname and tunnel type. Tunnel type is assumed to be a tun-device instead of tap, so in my case I needed to change it too. There is no need to tell the client anything else. Everything else will be negotiated through the tunnel. And since I use NetworkManager to set up my tunnels I didn't have a need to drop in to a shell even once at the client.

Sami Haahtinen: Dynamic network configuration

I've been meaning to write up on this for quite some time now, but always have something seemingly more important things to do. Most modern Linux distributions offer ways to manage network interfaces via some kind of abstraction. Usually this abstraction allows one to dynamically rename and add network interfaces. For Debian family management is done with ifupdown, while with RedHat family it's sysconfig. In addition there is NetworkManager which is a cross platform solution for dynamic network configuration (which eliminates the need to rename interfaces). Case Xenserver Usually this comes in rather handy, but on other occasions it can be a pain. A while I was helping out with an installation of a XenServer instance. This server had, for some reason, ethernet interfaces reversed in comparison to the other XenServers on the site. Luckily the service console has the network configuration in ifcfg scripts. We were easily able to reverse the interfaces by binding the interfaces to certain hardware addresses. The only problem is that the interface is renamed only when the interface is brought up. What's worse the interfaces were enumerated before the server would bring up all of the interfaces. Only eth0 was brought up for management purposes before enumerating. This means that the original eth0 was renamed to _tmp_xxxxxx. The (oh so elegant) solution was to create a script that does ifup eth1; ifdown eth1. Problem solved. Case tap0 I also experienced similar problems when I was setting up my OpenVPN tunnel. I wanted to use a bridged connection to my network, but for that I needed to create a br0 interface with tap0 as a member. It's easy enough to create ifupdown configuration to set up tap0. Hook that up to br0 and you are all set. The same problem will bite you here. The interface tap0 is actually created only when the tap0 is brought up and br0 members are added only if they are present. The solution? In my case, create a custom script that adds the device manually after it's been created properly. I was unable to find anything that works without using a custom script =( In retrospect Looking back at these cases, I should have known better. The problem was obvious and it took me way too long to figure out the cause for my problems. Then again, the utilities should be able to create a reasonable abstraction for themselves that they use to determine the actual status of the whole system. This way the trickery needed for setting up simple interface would be obsolete. You can't have it all, but you can always hope.

12 August 2008

Lucas Nussbaum: Debian s Freeze

Debian’s freeze sounds like a technical hack to address a social problem, and that disturbs me a bit. The social problem is: At some point, we need everybody in Debian to make only non-disruptive changes, so everything can converge very fast into a releasable state. The “solution” we are using is that we are blocking all packages from migrating to testing, and requiring manual review from someone on the release team. Consequences are:
- many people feel that you need to be very convincing to fix a small, not RC bug, even if fixing that bug definitely increases your package’s quality.
- the release team is completely overwhelmed by unblock requests during the freeze
- many people just stop trying to fix things during the freeze (which definitely doesn’t improve Debian’s quality), both because they think it’s hard to get a fix in, and because they don’t want to bother the release team I wonder if we really need such a strict policy. Are there other Free Software projects that use such a technical measure to prevent software from disrupting stable releases? I am the impression that most other projects rely on social pressure instead of technical measures for that, except maybe during the last few hours before the release. Couldn’t we act on the social level? We could default to allow everyone’s package to migrate to testing, and, when someone fucks up and uploads something that should not have been uploaded, block all his packages (switching to manual review mode) until the release. Of course, that require the release team to make decisions about _people_, which is harder than making decisions about _packages_. But if the rules are clearly stated, couldn’t this work?

5 August 2008

Adam Rosi-Kessel: Obama and Tire Pressure

Via Steve — this is nearly perfect. It is immensely satisfying to see a Democratic presidential candidate gracefully obliterate frivolous attacks rather than cower under them. Technorati Tags: , ,

Share This

Sami Haahtinen: Locales, recap

After my post yesterday about locales and my problems, I got some comments. Bryan, Simon and Wouter all pointed out what eventually figured out too but didn't mention in the post (I actually had to re-read the post to see what I wrote) that LC_ALL overrides all of the settings. Even though I figured it out eventually, with these comments it finally me that the purpose is actually to temporarily override the locale. Another thing that didn't occur to me while figuring out the correct locale for my system was that I'm thinking about it all wrong. Since I've already gotten used to broken locales I kept thinking that I want en_US locale with some Finnish settings, but in reality I wanted fi_FI locale with English language. I have to admit that I was pretty sceptical about the solution but decided to try it out. And to my surprise, it actually worked! So the final solution is this: LANG="fi_FI.UTF-8"
LANGUAGE="en"
LC_MESSAGES="en_US.UTF-8"
LC_TIME="en_US.UTF-8" It is a pretty clean solution and I'm happy to live with it. And as Simon pointed out, administrator changes should be preserved through upgrades. If not, one should file a bug for it. There we go, another problem solved. Thanks for the comments and suggestions.

Wouter Verhelst: Locales

Sami, it's really not that bad. There are, as you say, many variables to modify the behaviour of several programs. What you seem to have missed, though, is that there are really only four types of variables:
LANGUAGE
This is a GNU-specific extension which allows you to set fallbacks on different languages than "C" (which is locale-speak for "disable translation completely"). It somewhat overrides LC_ALL, but then at the same time it doesn't. Pretty strange thing, this variable. See "info gettext" and search for LANGUAGE if you want the ugly details.
LC_ALL
This variable exists to (temporarily) override any locale-related variable; for instance, when you want to file a bugreport in grep and you want to make sure you don't get any output that might be unintelligible for the developer, you could run LC_ALL=C grep --broken-parameter to temporarily disable locales. Using LC_ALL completely and utterly overrides anything else, so you probably don't want to set it in the general case.
LC_NUMERIC, LC_PAPER, LC_MESSAGES,...
These variable are meant to be used to divert from the defaults. For instance, as in your case, a user may want to have messages in en_US but have dates and papersizes etc be as is common in fi_FI. In that case, you'd set the default to fi_FI but set LC_MESSAGES (which specifies the language in which messages are set) to en_US. Or you might be a Dutch-speaking Belgian guy living in the US. You'd set the default to nl_BE.utf-8; but since A4 paper isn't easy to come by where you live, you'd set LC_PAPER to en_US.
LANG
This variable is used for "the default". If you haven't specified LC_ALL, nor the other relevant LC_* variable, then the value of LANG is used for whatever you're trying to do.
So what you really want to do is to set LANG to be fi_FI.utf-8, but have LC_MESSAGES be either en_US or, perhaps, C. This should give you english messages, but finnish conventions for date and number formats. Do note, though, that this is only the theory. It may be the case that some software is broken and doesn't look at the correct LC_* variable. If that happens to be true for some software you've seen, just file a bug...

4 August 2008

Sami Haahtinen: The pain of locales

Linux has this great thing called locales. You can basically control everything through a few system variables. The system is so flexible that you could in fact have English language with Swedish dates and Chinese error messages. The problem with the current system is that it's controlled mostly with one variable. Just about everyone I know (including me) set the system locale to en_US because they don't want to use their local language as the system language. This in turn causes new problems, mostly due to the metric system used in the sane countries. I finally got around to debug how to get the system to speak my language. The key here is the file /etc/default/locale which on Debian based systems contain the locale settings. Usually there is just one line in that file. LANG=en_US.UTF-8 Since I live in Finland and want my computer to speak English I can add the following lines: LC_NUMERIC="fi_FI.UTF-8"
LC_PAPER="fi_FI.UTF-8"
LC_NAME="fi_FI.UTF-8"
LC_ADDRESS="fi_FI.UTF-8"
LC_TELEPHONE="fi_FI.UTF-8"
LC_MEASUREMENT="fi_FI.UTF-8" This replaces the settings for numerals (decimal separator and such), paper (yes, the rest of the world uses A4), name, address, telephone and measurement (YAY, metric system!). This way I have a nice English speaking system with Finnish settings. I tried setting LC_ALL to fi_FI.UTF-8 but that causes gnome to speak Finnish to me, even though the LANGUAGE and LANG settings are set the English. LC_TIME is something I'd like to use, but I find the Finnish abbreviations for weekdays and months to be confusing. LC_MESSAGES causes gnome to talk partially Finnish, the general locale is English as it's supposed to be, but for example gnome-panel changes the menu entries to Finnish. I wish that it was easier to set these settings. I also doubt that various tools know how to respect these settings and will overwrite that file with the default setting. That is why I'm writing this entry so that I remember how to fix it when that happens. In the end the system is flexible but it's built in a funny way. The actual variable built by combining 2 or 3 values: language, country and possibly encoding. So to make thing easy for me I would have to declare en_FI.UTF-8 locale and start translating applications to that locale. I don't want to, so I'm sticking to this "temporary" solution.

3 August 2008

Sami Haahtinen: Lazyweb, I asked about dreambox

A while back wanted to know about a replacement for my Dreambox. I was also asked to write up a summary of the replies. Initially I assumed that my Dreambox 7025C was failing, it showed a classic symptom, progressively increasing error rate which wasn't showing up on my neighbors. Since we share the cable they should see the problems too, so I ordered new tuners for the box and the problem persisted. In the end it turns out that the problems were initially showing on such channels that my neighbors weren't watching or in such amount that they didn't pay attention to it. In the end it was a problem in the feed and my box is just fine. I now have 2 brand new DVB-C tuners for the 7025 (anyone want to buy them?) and a perfectly working setup. (YAY!). In a way I feel lucky that the box wasn't failing on me. Nobody suggested any complete solutions, but I was able to spot ReelBox Avantgarde as a possible alternative. It runs Linux like I want it to, but it's way too expensive. Another alternative could have been Maximum 8000, which appears to be a re-branded Marusys C-8000. It has Linux as the OS but the community side appears to be rather lacking. Then again, it's a pretty new device... I was suggested VDR and MythTV as the DIY solution. I dismissed MythTV without even looking in to it. Although I must admit that it's been quite a few years since I tried MythTV, but it was way too unstable for my taste. Also I wasn't too happy about the architecture of it. VDR is something I could have tried. The downside appears to be that it lacks in hardware support. The last time I was building my own PVR I ran in to the common problem of getting a decent output to the TV. VDR folks have traditionally solved this by using a HW decoder card to output the stream directly to the TV. Using VDR with a budget DVB card is possible but not recommended because of the CPU power needed to decode the stream. I've always considered VDR to be the more sophisticated solution of the two, but it is starting to sound a lot more hassle than it's supposed to. Also, the DIY solutions are hard to get in to a decent form. Usually the cases are bulky and custom modifications are required. I'm not one of those guys that install neon lights to their computers to make it look cool, but if I'm having it in my living room I expect it to look decent. Take a look at the VDR boxes section on the website for examples. While I admit that it's up to me to build a decent box, I'm not willing to spend too much time hand picking every single component so that it fits a casing and works with the given application. Finally, I'd like to thank all the people that sent me comments. Even though I ended up sticking with the current setup, it was a valuable look in to what is out there. The Dreambox I have, doesn't have a HDMI output and lacks in few areas. Keeping that in mind I'm eventually going to take this same look in to the alternatives while picking a replacement for the current box. Lets hope VDR, MythTV and Elisa (to name a few) continue to evolve and the video output properties of Linux become better and eventually I'll have the possibility of replacing my current setup with something that is even more flexible.

28 July 2008

Sami Haahtinen: Dear Lazyweb, about dreambox

For almost 2 years I've been a happy Dreambox user. While Dreambox has a lot of good features, the best being that it's based on Linux and people are actually encouraged to create their own firmwares and modify it, it lacks in hardware. All products suffer for some kind of disasters, for the 7025 series it was the power supply which will fail (it's only a matter of when). And now it looks like the rest of the hardware is failing too. So, "Dear Lazyweb" I'm asking if there is something similar to Dreambox in terms of flexibility and features. I'm mostly looking for an active community and solid device. HD capability is a plus and it must have DVB-C receivers (2). I'm not looking to build my own box, but if there is a reasonable framework for that I will look in to it too. In general I'm looking for a box that is flexible and has solid hardware from the "(sane + a bit) < insane" price range. If you happen to know of such a device or framework, drop me a note at ressu@ressukka.net (works with e-mail, jabber and msn, first 2 preferred =)

Next.

Previous.