Search Results: "will"

26 November 2008

Biella Coleman: Hacking Spaces, the Spaces of Hacking

Space and place has long been important to hackers. Whether it was/is the university lab, the workplace, the hacker con, or the particularly high-tech city, hackers congregate and meet face to face, often and everyday. One semi-new development has been the explosion and proliferation of hacker labs and spaces, such as Noisebridge located in the Mission district of San Francisco Foulab in Montreal. I recently got back from San Francisco and was able to spend a few nights at Noisebridge and was jaw droppingly impressed. The space is, well, spacious and nice (and located right by the Bart, a real +++++), but more important is that it is a thriving collective with all sorts of geeky participants and they have just souped up the space with all sorts of equipment, from the usual suspects (lots of computers) to lots of electronic gear such as oscilloscopes. What I also was impressed by was not only the blizzard of events but the open and accessible nature of the organization, which seemed to sit in some contrast to NYC’s hacker collective, NYC Resistor. Like so many organizations in this metropolis, they apparently are lacking in space to grow and the word on the streets (which I cannot confirm or deny as I have had very little contact with them but have heard this repeatedly) is that the organization has had a tough time letting in new members. Some folks are understanding of this given their space limitation, others have been less kind, and have referred to the group as a clique. There are already a few other initiatives under way to find a larger space so as to accommodate a more open, participatory atmosphere for a hacker space but I imagine that over time the culture and developments of NYC Resistor will also change as new spaces develops and do hope that this creates the conditions for more access rather than less (and again, I know next to nothing of the situation though I suspect space plays a real limiting role as it does with nearly everything else in NYC). And thankfully Rose White, a NYC-based sociology graduate student at CUNY, is paying close attention to the rise and development of these hacker spaces. She is well underway doing her dissertation dissertation on these hack spaces and I really look forward to her work.

Steve Kemp: If there isn't a movie about it, it's not worth knowing, is it?

So, I've just got a portable machine. I've configured it to be a pretty minimal installation of Debian Lenny, but one thing that makes me unhappy is mail handling. By default it came with several exim4- packages. Now in general Exim4 rocks. But it is a deamon running all the time, and overhead I could live without. I looked around to find a mail transport agent that would be more suited to the machine and was suprised to find nothing suitable. Basically I figure the machine will never generate "real" emails. Instead it will only receive mails from cron, etc. The machine will never have a real fixed IP, and so relaying mail externally is a waste of time. The mail should just go somewhere predictable and local. There are a couple of lightweight agents which will forward to another system, but nothing seems to exist which will queue mail locally only. So I've hacked a simple script which will do the job. Given the spool director /var/spool/skxmail the following command: 
skxmail root < /etc/passwd
Produces this: 
/var/spool/skxmail/
 -- root
     -- cur
     -- new
         -- 1227702470.P8218M243303Q22.vain.my.flat
     -- tmp
4 directories, 1 file
That seems to be sufficient for my needs. (I support the flag which says "read the receipient from the body). Of course to do this properly I'd be setgid(mailgroup). Instead I assume that local means everybody can see it and /var/spool/skxmail is mode 777. Ooops. Still happy to share if it sounds interesting. ObFilm: Dogma

David Pashley: Vim Syntax Highlighting for Puppet

I've just set up syntax highlighting for Puppet manifest files, and thought I'd share the simple steps. The first thing to do is download the syntax file from http://www.reductivelabs.com/downloads/puppet/puppet.vim and save this to ~/.vim/syntax/puppet.vim. Now when the filetype is set to "puppet", vim will use this syntax file. That's useful, it it would be even nicer if we could make vim know that files ending in .pp were puppet files. Turns out this is very easy to do. You need to create a file to detect the correct filetype when you open a file. You need to put the following lines in ~/.vim/ftdetect/puppet.vim: 
au BufRead,BufNewFile *.pp   setfiletype puppet
Now when you load a file ending in .pp, you should get nice syntax highlighting. You can also make vim use special settings for the puppet filetype by creating a vim script file in one of ~/.vim/ftplugin/puppet.vim, ~/.vim/ftplugin/puppet_*.vim and/or ~/.vim/ftplugin/puppet/*.vim. Vim has a lot of flexible hooks to enable file type specific configuration; hopefully it should be fairly easy to modify these examples for other file formats.
Read Comments (0)

25 November 2008

John Goerzen: Why Do Web Applications Stink So Badly?

So today, I happen to be looking at wikis for two small to mid-sized public proojects (MoinMoin and DokuWiki look like frontrunners right now — any suggestions?) Recently, I’ve also looked at blog and CMS software, and a host of other web apps. It’s as if these people have learned nothing about good software practices over the last 20 years. Warning: Rant Ahead So how many of you have been here before? You download WebApp X. It tells you to cd to your DocumentRoot and unzip/untar it there. At this point, most of them will tell you to chmod -R 777 the install directory. Some of the better ones, such as WordPress, will tell you to chmod it 777, or if that makes you nervous, to instead chown it to the user that your webserver runs as. It is at this point that you realize that the Java-based programs ship with their own webserver that takes 2 minutes to load and uses 2GB of RAM, while the PHP-based programs want you to give them 32MB RAM per process, and probably modify your global PHP settings in a way that breaks some other PHP web app you’re already using. As if that isn’t enough to scare you off, generally speaking, config files — including passwords to databases — are stored in the same directory, along with .htaccess files. Many of these programs are also downloading and updating plugins over the Internet, usually without any kind of cryptographic authentication, and overwriting their own program files in the process. Oh, and this is a class of app that is notorious for security problems to start with, and makes your server known to billions of people via search engines. Absolutely no opportunity for trouble here, of course! That sentence was dripping with sarcasm, in case you didn’t get it. It also makes it almost impossible for people such as Debian maintainers to package up some webapps (such as just about every single one that uses Ruby on Rails) because there is just no sane way to make it behave with respect to the Filesystem Hierarchy Standard. I’d love to see web app developers do a few simple things:
  1. Separate code from data
  2. Separate code from configuration
  3. Separate all of the above from the DocumentRoot to the greatest extent possible
I realize that some of this is purportedly to make things easier to install when you have FTP access only. But to me it seems just really poor design. I’ve written webapps, and it’s not that hard to do this part right. Plus, doing the above right means that I no longer have to do something like use git on my WordPress installations because it’s too much of a hassle to apply security and plugin updates on all three separate ones otherwise.

Joey Hess: merge conflict

I'm writing a piece of autobiography/alternate world fiction, using git. Whether it will get finished or be any good, or be too personal to share I don't know. The idea though is sorta interesting -- a series of descriptions of inflection points in a life, each committed into git at the time it describes. As the life paths diverge, branches form, but never quite merge. Reading this would not be quite like reading one of those choose your own adventure books. Rather you'd start at the end of a path and read back through the choices and events that led there. Or browse around for interesting nuggets in gitk. Or perhaps the point isn't that it be read at all, but is instead in the writing, and the committing.

Ian Wienand: Position Independent Code and x86-64 libraries

If you've ever tried to link non-position independent code into a shared library on x86-64, you should have seen a fairly cryptic error about invalid relocations and missing symbols. Hopefully this will clear it up a little! Let's start with a small program to illustrate. 
$ cat function.c
int global = 100;
int function(int i)  
	return i + global;
 
$ gcc -c function.c
Firstly, inspect the disassembley of this function: 
0000000000000000 <function>:
   0:	55                   	push   %rbp
   1:	48 89 e5             	mov    %rsp,%rbp
   4:	89 7d fc             	mov    %edi,-0x4(%rbp)
   7:	8b 05 00 00 00 00    	mov    0x0(%rip),%eax        # d <function+0xd>
   d:	03 45 fc             	add    -0x4(%rbp),%eax
  10:	c9                   	leaveq
  11:	c3                   	retq
Lets just go through that for clarity: The IP relative move is really the trick here. We know from the code that it has to move the value of the global variable here. The zero value is simply a place holder - the compiler currently does not determine the address (i.e. how far away from the instruction pointer the memory holding the global variable is. It leaves behind a relocation -- a note that says to the linker "you should determine the correct address of foo, and then patch this address of the code to point to that addresss (i.e. foo)." Relocations with addend The image above gives some idea of how it works. We can examine relocations with the readelf tool. 
$ readelf --relocs ./function.o
Relocation section '.rela.text' at offset 0x518 contains 1 entries:
  Offset          Info           Type           Sym. Value    Sym. Name + Addend
000000000009  000800000002 R_X86_64_PC32     0000000000000000 global + fffffffffffffffc
If you try and build a shared object (dynamic library) with this function, you should get something like 
$ gcc -shared function.c
/usr/bin/ld: /tmp/ccQ2ttcT.o: relocation R_X86_64_32 against  a local symbol' can not be used when making a shared object; recompile with -fPIC
/tmp/ccQ2ttcT.o: could not read symbols: Bad value
collect2: ld returned 1 exit status
Position Independent Code (PIC, enabled with -fPIC) just means that the output binary does not expect to be loaded at a particular base address, but is happy being put anywhere in memory (compare the output of readelf --segments on a binary such as /bin/ls to that of any shared library). This is obviously critical for implementing shared-libraries, where you may have many, many libraries loaded in essentially any order, and trying to pre-allocate where in memory they would all live just does not work. However, PIC has slightly wider implications than simply movable base addresses. So, back to relocations. The exact rules for different relocation types are described in the ABI for the architecture. The R_X86_64_PC32 relocation is defined as "the base of the section the symbol is within, plus the symbol value, plus the addend". The addend makes it look more tricky than it is; remember that when an instruction is executing the instruction pointer points to the next instruction to be executed. Therefore, to correctly find the data relative to the instruction pointer, we need to subtract the extra. This can be seen more clearly when layed out in a linear fashion (as in the bottom of the above diagram). What's the problem with this in a shared library? In a shared library situation, we can not depend on the local value of global actually being the one we want. Consider the following example, where we override the value of global with a LD_PRELOAD library. 
$ cat function.c
int global = 100;
int function(int i)  
	return i + global;
 
$ gcc -fPIC -shared -o libfunction.so function.c
$ cat preload.c
int global = 200;
$ gcc -shared preload.c -o libpreload.so
$ cat program.c
#include <stdio.h>
int function(int i);
int main(void)  
   printf("%d\n", function(10));
 
$ gcc -L. -lfunction program.c -o program
$ LD_LIBRARY_PATH=. ./program
110
$ LD_PRELOAD=libpreload.so LD_LIBRARY_PATH=. ./program
210
If the code in libfunction.so has a fixed offset into its own data section, it will not be able to see the overridden value provided by libpreload.so. This is not the case when building a stand-alone executable, where references are satisfied internally. Of course, any problem in computer science can be solved with a layer of abstraction, and that is what is done when compiling with -fPIC. To examine this case, let's see what happens with PIC turned on. 
$ gcc -fPIC -shared -c  function.c
$ objdump --disassemble ./function.o
./function.o:     file format elf64-x86-64
Disassembly of section .text:
0000000000000000 <function>:
   0:	55                   	push   %rbp
   1:	48 89 e5             	mov    %rsp,%rbp
   4:	89 7d fc             	mov    %edi,-0x4(%rbp)
   7:	48 8b 05 00 00 00 00 	mov    0x0(%rip),%rax        # e <function+0xe>
   e:	8b 00                	mov    (%rax),%eax
  10:	03 45 fc             	add    -0x4(%rbp),%eax
  13:	c9                   	leaveq
  14:	c3                   	retq
It's almost the same! We setup the frame pointer with the first two instructions as before. We push the first argument into memory in the pre-allocated "red-zone" as before. Then, however, we do an IP relative load of an address into rax. Next we de-reference this into eax (e.g. eax = *rax in C) before adding the incoming argument to it and returning. 
$ readelf --relocs ./function.o
Relocation section '.rela.text' at offset 0x550 contains 1 entries:
  Offset          Info           Type           Sym. Value    Sym. Name + Addend
00000000000a  000800000009 R_X86_64_GOTPCREL 0000000000000000 global + fffffffffffffffc
The magic here is again in the relocations. Notice this time we have a P_X86_64_GOTPCREL relocation. This says "replace the data at offset 0xa with the global offset table (GOT) entry of global. Global Offset Table operation with data variables As shown above, the GOT ensures the abstraction required so symbols can be diverted as expected. Each entry is essentially a pointer to the real data (hence the extra dereference in the code above). Since the GOT is at a fixed offset from the program code, it can use an IP relative address to gain access to the table entries. This extra reference is obviously slower; however for the most part I imagine the overhead would be essentially immeasurable and is required for "generic" operation. If you have figured the cost of indirection through the GOT is the major bottleneck of your program, I imagine you wouldn't be reading this and would already be considering strategies to remove it! The next question is why this works on plain old x86-32. Inspecting the code reveals why: 
$ objdump --disassemble ./function.o
00000000 <function>:
   0:	55                   	push   %ebp
   1:	89 e5                	mov    %esp,%ebp
   3:	a1 00 00 00 00       	mov    0x0,%eax
   8:	03 45 08             	add    0x8(%ebp),%eax
   b:	5d                   	pop    %ebp
   c:	c3                   	ret
$ readelf --relocs ./function.o
Relocation section '.rel.text' at offset 0x2ec contains 1 entries:
 Offset     Info    Type            Sym.Value  Sym. Name
00000004  00000701 R_386_32          00000000   global
We start out the same, with the first two instructions setting up the frame pointer. However, next we load a memory value into eax -- as we can see from the relocation information, the address of global. Next we add the incoming argument from the stack (0x8(%ebp)) to the value in this memory location; implicitly dereferencing it. This provides the abstraction we need -- if the relocation makes the patched address at 0x4 the address of the GOT entry, it will be correctly dereferenced. It is the inability of the x86-32 architecture to try and optimise by doing instruction-pointer relative offseting which means it always needs to do slower memory references, which turns out to be just what you want when you're making a shared library! So, the executive summary: the ability of x86-64 to use instruction-pointer relative offsetting to data addresses is a nice optimisation, but in a shared-library situation assumptions about the relative location of data are invalid and can not be used. In this case, access to global data (i.e. anything that might be changed around on you) must go through a layer of abstraction, namely the global offset table.

Holger Levsen: awesome bts and snow

Right now, I'm a happy hacker: I just added a keybinding and shell oneliner consisting of four piped commands finally invoking the bts command to my awesome configuration. So now I can press Mod+F4 (anywhere, anytime) and then either enter a bug number or get a tab completion list of bugs I already looked at. Totally awesome. And I guess there is room for improvements once I have used /usr/bin/bts more :)

In related news I'm currently flying from Z rich (which made me think about snow...) to Madrid, in anticipation of the Debian QA/i18n/ftpteam meeting, which will start tomorrow evening with a bus drive to Caceres, the town where DebConf9 will be held in 2009. And somehow I didnt feel like going through my mails, I rather wanted to do some interesting new stuff. So I started to read the bts manpage...

To bring this post to and end, I share Gunnars feeling about all these snow blog posts, you insensitive colds! Last friday morning, Hamburg was among the first cities in Germany getting snow (the cold was coming from Greenland, thus from north west and Hamburg is pretty much north-north-west in Germany). Unfortunatly, the snow was almost gone in the city by friday noon, so when all these "got lots of snow" blog posts popped up on planet, I felt a bit like Gunnar...

Neil Williams: Unexpected functionality

Following from unexpected results, I can't make up my mind why Gtk+ supports rotated labels.
Try it, load Glade3, create any normal Gtk window and add or select a standard GtkLabel, add some text. In the properties of the label, there is a spin button for Angle:. Why is that? Which applications have need for this? I can't think of any examples that I've seen in Debian. Does anybody use rotated labels?
I could possibly understand rotations in increments of 90 degrees but this supports rotation at an precision of two decimal places! I mean, why support a rotation of 89.27 degrees?
Even if this was a version of a progress bar, why is it supported for every GtkLabel anywhere?
Weird.
P.S. Off to Extremadura tomorrow, hopefully I'll have a chance to get a better hackergochi too.

MJ Ray: Please Vote for Weston Woods Path tel:+448716268156 (10p)

I spent the second half of this morning delivering leaflets asking people to vote for the Weston Woods Path Project in today’s itv People’s Millions Big Lottery Fund competition. Just 50,000 gets a smooth, fairly flat, surfaced and edged path and takes some pedestrians off the bendy cliff-top road. Any surplus money from the call charges will be donated to charity. If you vote for it, you can comment on my blog to let me know if you want and you might like to watch WsM Forum for the reaction/news. It’s a beautiful day and North Somerset is a beautiful place but by hell is my jaw cold now! At least there’s no snow here.

Ingo Juergensmann: Buildd.Net updated

After the "Bits from the buildd.debian.org world" mail I made the necessary changes to the database on Buildd.Net to reflect what has been announced in the "Bits..." mail.
That's mainly the changes of buildd admins. Some more changes will be made in the next days. So please visit Buildd.Net to see what have changed and if all changes are correct.

Of course I would appreciate if the new buildd admin would contribute to Buildd.Net and run the update-buildd.net script or use their own updating script as described in the FAQ.

Russell Coker: SE Linux and Decrypted Data

There is currently a discussion on the Debian-security mailing list about how to protect data which came from an encrypted file. I was going to skip that one until someone summoned me by mentioning SE Linux. The issue which was raised is that data from an encrypted file can be read from /dev/mem (for all memory of the machine) or /proc/<pid>/mem (for the memory of the process). It was suggested that SE Linux can prevent such attacks, however it’s not that simple. In most SE Linux configurations there will be a domain with ultimate privileges, examples are unconfined_t for a default configuration (also known as targeted) and sysadm_t for a “strict” configuration. SE Linux differs from many security models (such as the Unix permissions model that we are familiar with) in that there is no inherent requirement for a domain with ultimate privileges. I have in the past written policy for machines in which no domain could read /dev/mem or modify the running policy. This meant that booting from installation media might be required to modify the configuration of the system - some people desire systems like this and have good reasons for doing so! As the vast majority of SE Linux users will have a “targeted” configuration, and the majority of the rest will have a “strict” configuration with minimal changes (IE sysadm_t will work as expected) there will always be a domain that can access /dev/mem (I’m not certain that sysadm_t can directly access /dev/mem on all variants of the policy - but given that it can put SE Linux in permissive mode it has ultimate access to work around that). This however doesn’t mean that SE Linux provides no benefit. A typical Linux system will have many daemons running as root, while a typical SE Linux system will have few processes running as root while also having a SE Linux context that grants ultimate privileges. The root owned processes that SE Linux constrains are generally the network facing daemons and other processes which provide a risk, while the root owned processes that are not constrained by SE Linux policy are generally processes related to early stages of the system boot and programs run as part of the system administrator’s login session - processes that are inherently trusted. Now regarding the memory of the process (and the ptrace API), in most cases a process that is likely to be accessing such a file will run in a domain that permits such access to itself. If the process in question is run from a login session (ssh login etc) then other processes in the same session will have access to the data. It is quite possible to write policy to create a new domain for the program which accesses the decrypted data and to deny the domain the ability to ptrace itself, but it will require at least 10 minutes of work (it’s not a default configuration). So while SE Linux will generally help you to improve your security no matter what your exact requirements may be, it’s not something that can be used to make such corner cases entirely disappear. In a more general sense the idea that only a single program can access certain data is almost unattainable in a typical system. Among other things to consider is the fact that the Linux kernel is one of the larger programs on a typical Linux system…

24 November 2008

Gunnar Wolf: It's just a different mindset. Not necessarily a _sane_ one, though...

Wouter insists that Ruby Gems are enough of an argument to keep Rails at a distance. Even though I agree with the basic claim and think that Gems are basically insane and sick, this should be taken a bit more under perspective.
We are blessed. We are blessed to have Debian, such a rich OS with such a great package management system, and with superb integration between so many packages. Blessed are also the users of most Free Software based distributions, as they share the advantages of systems growing with full consciousness of their interaction's benefits. However, integration with the rest of the world is not seamless.
Most scripting languages have their own infrastructure for managing the modules/libraries/pacakges/whatevers dependencies. Perl has CPAN, PHP has PEAR, Ruby has Gems... I do see Gems as the most obnoxious of them all, but the basics are the same.
The Rails crowd started being Unix-centric, but the Windows (and MacOS - they are no better, believe me, at least in this regard) world has exerted its pressure. Gems caters very well to their needs, but we do suffer the integration at the distro side.
The only sane way the propietary-minded people have managed to stay clear of the well known "DLL hell" is to ship everything a given program requires bundled together - that's the main reason for the bloat of lots of applications, and for the sloppiness of security support. Every application packager is responsible for shipping updated versions to any library it bundles in, except for the very basic core that the OS itself provides. That seems so annoyingly backwards to us that... it is unbelievable.
So, yes, Rails application trees often include Rails itself. For $DEITY sake, even I have grown used to working that way, as things tend to break under your nose otherwise. My proposal (which we talked over at DebConf, but have not pushed so far) is to support simultaneous versions of Rails installed in a Debian system (of course, via different packages), more or less in the way that simultaneous versions of Ruby, PHP or Python (and, in some limited fashion, Perl - Although Perl does not suffer from this incompatible bumps. Yay for Perl!) can be installed.
...And, yes, together with the pkg-ruby-extras team, I have been trying to -slowly, yes- package whatever modules we often use so they don't have to be included in Rails applications.
So far, the best way (although by far not optimal) I have found to limit this explosion of trees is to include most libraries in my Rails application trees as git submodule trees - If they are not explicitly downloaded, the systemwide libraries will be used.
Yes, the "ship the whole thing as a bundle" approach is quite annoying. However, at least I must acknowledge that it works better than the approach I took with previous (mod_perl-based) webapps I wrote... As relatively few people grok mod_perl, I ended up with quite some apps which were not installed by anybody but me. Rails is obnoxious... but seems to be parsable by the humanity at large.

Matt Brubeck: 24 Nov 2008

The Timeless Way of Building: Diagnosis and Repair Architect Christopher Alexander's A Pattern Language is well known among programmers as the inspiration for the Design Patterns school of thought, but for some reason his other books are less widely read. My mother and father are both architects—"real" ones, not "software architects." :) My own amateur interest in the subject led me to read several of Alexander's books over the past couple of years. The Timeless Way of Building is the first volume of an open-ended series. (A Pattern Language is the second.) It is in this book that Alexander first lays out the idea of a pattern language. More importantly, he explains an entire theory and practice of design and construction, of which the pattern language is only one piece. Another piece of the Timeless Way is the practice of diagnosis and repair. Rather than a detailed plan of future projects, Alexander prefers to maintain a diagnosis of current problems. Each new project must be targeted at fixing these existing problem in the system: "At every moment we use the defects of the present state as the starting point for the definition of the new state" (p. 485). Alexander's goal is to create a type of spontaneous growth and order that is possible when change is incremental and decision-making authority is decentralized. Alexander was hired to produce a new type of master plan for the University of Oregon campus. The result was published as the third volume in the "Timeless Way" series: The Oregon Experiment. In it, he writes again about diagnosis and repair:
It is still not clear where global order will come from, without a master plan.… How is the problem solved in an organism? Essentially, the problem is solved by a process of diagnosis and local repair.… The master plan tells us what is right for the future. The diagnosis tells us what is wrong for the present. (p. 146–148)
In software, the same problems and conclusions have led to "agile" processes, which emphasize incremental change and flexible planning. But I haven't seen any software companies adopt a consistent practice of "diagnosis"—keeping track of what's wrong or unsatisfactory with the current system. This extends beyond defect tracking; it should include a constantly updated list of code that, although it works, is hard to understand, difficult to change, or otherwise likely to cause problems. (This should be easy to generate—every programmer knows certain parts of their code base that they dread working on for one reason or another.) This knowledge can then guide new development to include "repair" and address such problems before they become defects. I've applied this on a small scale to my own work, and now I wish I had used it at Amazon, where my team spent much more time maintaining and improving old code than writing new code.

David Welton: Slicehost vs Linode

Since hosting is the major expense for DedaSys LLC, and obviously a critical part of much of what I do, getting the right one is very important. Naturally, "the right one" for me may not be the right one for everyone. I am a fair sysadmin for small numbers of machines, so I don't mind doing that myself - I don't need, nor want to pay for hand holding. I recently moved everything over to Slicehost on the recommendations of friends, and am so far fairly happy with the experience. It's cheaper, simpler, and more flexible than Layered Tech was. To boot, I also like the fact that Slicehost is run out of St. Louis, Missouri, rather than some really expensive tech center like San Francisco or Boston. Hosting is basically a commodity, and much like you wouldn't want to put a factory in San Francisco, you don't want your hosting their either. However, I discovered something annoying about Slicehost: they use x86_64 servers, which, per se, doesn't really matter to me - I use open source code that can run on any number of architectures. The problem is that this particular architecture uses more memory than plain old x86. Significantly more. My Layered Tech server with 1 gig of memory was hitting the swap space a bit, but the same code on a slice was swapping quite heavily, despite the fact that I'd moved PostgreSQL off to its own slice. Since I pretty much exclusively run Rails, I decided to look into Phusion's "Ruby Enterprise Edition", which is basically just some nice hacking on Ruby's garbage collection mechanism. What they've done is nice, and I may end up using it, but ultimately it's buying me space that I would also gain from simply moving back to x86. With that in mind, I decided to take another look at what appears to be emerging as one of Slicehost's main competitors, Linode, who *do* use x86 servers. Here are my results, which are admittedly not all that scientific, but what the heck - you're getting them for free, and they were pretty quick to whip up. Service, Support, Setup But before we begin with the numbers, I want to add a few words of caution. One of the big things about hosting, to me at least, is how they deal with unexpected random negative events. Connections going down, disk breakage, DoS attacks, and so on. It's really hard to get an idea of just what sort of people you're dealing with until you've "ridden the river together". I don't really know what sort of response times, uptime, and all that either Slicehost or Linode have, so there are potentially some big intangibles there that are not quite as easy to draw pretty charts for. In terms of the console/setup/management tools, I liked Slicehost's simplicity more - Linode gives you more options, but they're a bit fiddly and cluttered feeling at times. For instance, Linode lets you pick how much swap space to give your disk image, but doesn't include a bit of javascript to balance out the swap and regular partitions, so that if you type in a larger swap number, you have to do the math and subtract that from the other number. Annoying, but not a big deal. Linode lets you pick various data centers in the US (no Europe, yet). Slicehost gives you the option to do backups of your disk images, which is nice, and something that Linode lacks. Comparing plans Since I'm interested in comparing hardware and machines I actually have access to, here are the plans I am currently signed up with, and the salient numbers:
Slicehost 1024 Slicehost 512 Linode 720
Memory 1024 512 720
Bandwidth 400 200 400
Disk 40 20 24
Cost 70 38 39.95
Memory / Dollar: Bandwidth / Dollar: Disk / Dollar: Linode mostly comes out ahead, but not by that much, except for bandwidth. However, let's take a crude look at x86_64 vs x86 memory usage. First, ./script/console (a Rails console) for the same code base on two different machines, showing the virtual memory size (VSZ) and resident set size (RSS): And just to look at something else that's a bit smaller, I ran the following script: 
puts File.open("/proc/self/status", "r").read
And got the following data: Being charitable with those numbers, we get x86 taking up 77% the memory that x86_64 does. So let's add that to our comparison of how much memory you're getting per dollar on Slicehost and Linode: Wow! That's a fairly significant difference. Performance That's most of what we can glean from published numbers, and running a few simple experiments. However, there's another factor that's important: performance. "Virtual Private Server" systems, or simply VPS's, got a bad name in the past because they were "overbooked" - too many virtual servers competing for the same CPU resources on one machine. Slicehost and Linode both look like they want to avoid that kind of bad reputation, and so far all the systems I have used have felt snappy and responsive. Now here's where we get really unscientific... I decided to try and do something, even if it wasn't much, to get some kind of objective measurement of what kind of CPU I was getting on the machines. This is really unscientific, because the machines do have other duties (I don't have the cash, time or inclination to set up random servers just for testing), and of course, who knows what else is sharing the computer with my systems: if you get a relatively unused computer, you can apparently pick up extra cycles for yourself, over and above your guaranteed minimum. But c'est la vie, and so I decided to do some numbers anyway. I picked a C implementation of mandelbrot from the language shootout more or less "for the hell of it" (I'm going to keep repeating "unscientific" and invite someone to do better tests than I have). I ran this code every 20 minutes for a couple days, and then averaged out the run times: As expected, the higher power Slicehost 1024 machine wins out in terms of raw speed (less time is better), but not by that much over the Linode system. Indeed, when we factor in the relative prices, Linode comes out ahead, again: So it looks like we're not paying a CPU penalty for having more memory, bandwidth, disk, etc... Notes
  • Nothing in these statistics is indicative of future directions that both Linode and Slicehost might take. Slicehost was recently acquired by Rackspace, and that could have effects on the service they provide. Maybe one or both will come under financial pressure and start "overbooking" their servers.
  • The spreadsheet I used is available via Google docs, for the curious. I did most of the work in OpenOffice, and then uploaded it though, because working with the web spreadsheet was kind of painful. Unfortunately, you can't select non-contiguous data ranges in the Google spreadsheet, so the labels are interspersed with the data. Yuck. Also, Google docs doesn't handle 'time' values nicely, which is what the performance data was originally. The spreadsheet is here: http://spreadsheets.google.com/pub?key=pVDeZF6ire2YwATBYb4mBKQ
  • The standard deviation on the processing times is higher for both slices than the linode system. Not quite sure what that means in terms of what's going on under the hood, but I thought it was interesting.

Martin Michlmayr: Debian installer coming to the D-Link DNS-323

My posting about receiving a Conceptronic CH3SNAS a month ago prompted Matt Palmer to work on the installer. Matt did a fantastic job and had patches for everything within a few days: he added support for the revision B of the DNS-323 to the kernel, added code to the kernel so the MAC address is parsed from flash, and added DNS-323 support to various components of the installer. Finally, he wrote a really good porting guide that will hopefully inspire other people to port the installer to more NAS devices. Porting to another Orion based device usually doesn't take much now that we have the infrastructure in place. While Matt's patches came too late for inclusion into the initial release of lenny, the release team seems agreeable to the idea of adding support for the D-Link DNS-323 and Conceptronic CH3SNAS to a stable update of lenny. To this end, the dns323-firmware-tools package which is needed to create proper firmware images for the DNS-323 was accepted for lenny a few days ago. Alan Fletcher also kindly agreed to make a serial cable for me, so hopefully I'll get a chance to test Matt's patches soon and integrate them into the installer. In the meantime, I created a few simple pages about Debian on the D-Link DNS-323 which I will extend once support has been integrated into the installer.

Simon McVittie: Why you shouldn't block on D-Bus calls

I've found myself writing this several times recently in the context of Telepathy, so I thought I'd make a blog post that I can refer people to. D-Bus is defined (by the wire protocol spec) to be an asynchronous message-passing system, and libdbus behaves accordingly; there are no blocking calls at the wire protocol level. However, libdbus provides a "blocking" API (dbus_do_something_and_block), via a mechanism I'll refer to as "pseudo-blocking" for the purposes of this post. Most D-Bus bindings (at least dbus-glib, dbus-python and QtDBus) expose this pseudo-blocking behaviour to library users. These pseudo-blocking calls work like this: The messages received between M and R are delivered when the main loop is next entered. This can cause a number of problems: As a result, telepathy-glib's code generation mechanism does not generate any pseudo-blocking code. There are several alternative modes of operation that we do support: We make a couple of narrowly targeted exceptions to the "no pseudo-blocking" policy in internal code, by allowing telepathy-glib internals to make a small number of pseudo-blocking method calls to the dbus-daemon (which is the one component we can definitely trust to return results promptly). Here are some examples of the same strategies in other libraries:

Gunnar Wolf: You insensitive clods. s/lo/ol/

It's sad that today in Planet Debian we have hit an Eurocentric geographically discriminating meme. Particularly, one I'd love to take part of. Well, at least I can assure you we have reached the usual low temperature of 2 Celsius in Mexico City... As always, people say it's so cold that this year we _will_ get snow. And as always, I'm sure it's just wishful thinking ;-)
So, even with Marcelo's frozen Z calo live again for this winter, I can only reinforce our tropical paradise stereotypes by reminding you that this is less than 500Km away from home - All year 'round:

Aigars Mahinovs: Slowfs

I have just encountered a bug in Gnome that is much more visible when the hard drive is slow or overloaded and responds slowly. This gave me an idea - how about a simple transparent FUSE filesystem that does nothing else than delay, slow down and possibly reorder filesystem requests? Such a filesystem would be very useful for debugging. We developers tend to have high-performance systems and that hides many bugs, but if we could have a slower system on-demand, it will give us the ability to debug our applications better. Even better would be a virtual machine where one could arbitrarily slow down any specific aspects of the system - slower CPU, slower hard drive, slower RAM, slower network, … I know that engineers of network hardware have special devices that emulate long distance and noise - you take two wireless network cards and plug cables into their antennae sockets and then plug those cables into the device. The device then makes a connection between those cables, but with emulation of a variable distance, variable antennae and variable environmental noise, so the wireless cards can be tested for distance and noise resistance right in the lab. We could do the same in software for HDD, CPU, RAM and networking - imagine having two Xen instances and putting an emulated network between them with 10 hops, 1 second latency, 20% average packet drop traffic shaping and limited top bandwidth of 100 kbit/s. Now that is a real test platform for any kinds of applications that have anything to do with networking (or HDD, CPU or RAM). Anyone up to making that? Could someone of you use a system like that?

Josselin Mouette: The great return of the broomsticks

While a few developers have the chance to be paid for their job on Debian, most of us are doing it on our spare time, for fun. Yes, we are making the best operating system out there, for fun. Yet a large number of developers are taking this project way too seriously. To say it in other words, they have a carrot up the ass. I happen to prefer the French expression though, since it conveys better the rigidity of their attitude: they have a broomstick up the ass. Recently, when I launched the idea of the Bug Sprint, someone wrote to me to propose, instead of rewarding people with cookies, to send them money. This is the best example of broomstick behavior: taking the fun out of people by bringing back rampantly the very idea that almost split the project in two the last time it was introduced. The very same person, when faced with a sarcastic email sent to an announcement mailing list, immediately rushed in to send a ban request to list masters (a totally useless one, since I don t have the habit to make the same joke twice). That person happens to be the project leader. So it seems the only thing the DPL has to do, at a time when we d rather need people to fix RC bugs, is to protect the project from seditious members who don t share their love for broomsticks. When someone makes a bad-taste joke, you are not forced to laugh. But you are not forced to impose your rigid and moralist views either. And on top of that, throwing random accusations of sexism without any kind of justification does not improve the mood. This is even very close to defamation, but since I don t have a broomstick in the ass myself, I prefer to just laugh at them with the help of some fellow developers. I will go on shocking these people. And they won t like it more than in the past. Every time one of them climbs on her high horse to take a pathetic and patronizing stance while totally missing the point, it will bring a bit of fun. Bonus #1: as for the promised statistics, we are, at the time of writing, at 493 599 unique IPs who clicked on the trap. Quite a good investment for someone who d want to make money from smelly geeks. Bonus #2: here is a sexist picture.

23 November 2008

Matthew Garrett: Nngh.

Sebastian K gler dissects my dissection, except in fact he appears to be responding to a pile of stuff I didn't write.

First of all, Matthew assumes that the setting "powersave" will actually use the powersave cpufreq governor - citation needed.

the "conservative" governor, which indeed is an "ondemand" governor, meaning it gives you the extra needed CPU power when you ask for it - more slowly than ondemand does, resulting in extra power draw. Never run conservative unless your P-state latency is so high that ondemand won't work.

Powerdevil doesn't offer such option, so Matthew will be happy about this item - I didn't say it did. However, it is a common myth that thermal management is part of the job a power management policy should play. CPU frequency changing is a mechanism that can be driven by two policies - power management and thermal management. The fact that they share a mechanism does not mean that these policies should be conflated. Really. I wasn't just talking about Powerdevil here.

"3d takes more energy than 2d" is something I didn't actually say. I brought this up because various people have advocated disabling compiz to save power and because Powerdevil appears to have a "Disable Kwin compositing" button.

Lesson learned: Don't have an immediate emotional response to perceived criticism and fail to actually read what you're responding to. I would have commented on Sebastian's entry directly, but he forbids comments, so fuck that.

Next.

Previous.