Search Results: "rul"

13 December 2008

David Welton: Startups and Work: Europe vs the US

Michael Arrington of Techcrunch comments on the US vs Europe in terms of startups: Joie De Vivre: The Europeans Are Out To Lunch It's a pretty rough and coarse-grained view of things, but there's a grain of truth there. I've written about this a little bit before. It's something that I feel qualified to talk about, having lived and worked in both the US and Europe (Italy and Austria to be precise), and in that time, generally gravitated towards startups. Here's my quick take on a list of what's good and bad about each. Please note that these are of course not true for everyone, that things are changing, but are still things I think are generally true, even when I could think of several counterexamples for some of them myself. Also, it's very important to keep in mind how diverse "Europe" is, so most of what I write is really about Italy, and to a much smaller degree, Austria. I'd be very curious to hear your own experiences in the comments. Europe Bad Good US Bad Good Conclusions An interesting example of all this from my own experience was Linuxcare. The company was, culturally, a Bay Area startup, with the headquarters in San Francisco, founded by very startup oriented guys (one of whom, Dave Sifry, went on to do technorati, and is currently working on yet another venture). However, most of the actual open source talent was in our satellite offices, in Australia, Italy and Canada, places where, perhaps, people were not so distracted by the prospect of "make money fast!!! hurry!!!" that they had time to work, learn, and create some really great open source code. To be honest, I find myself torn. Business-wise, I prefer the US. However, outside of that, there's a lot to be said for Europe. I also think that some of what's good about business in the US is coming to Europe, albeit slowly in some cases, in Europe. People my age here can see what's going on elsewhere, and try and copy what they like. A lot of what's good about Europe, though, might be more difficult to import into the US, especially the livability of the cities. In any case, I can conclude that it's a complex, difficult topic best discussed over a glass of wine.

12 December 2008

Chris Lawrence: Corollary of the day

King Politics thinks the GOP anti-bailout faction has the politics of the day wrong:
The time for pragmatism is now, but Senate Republicans don t recognize that. Senate Republicans have yet to realize that the GOP lost so many seats in 2006 and 2008 because the American public has a greater desire for pragmatism than ideology.
I think that s true to a point but I suspect it has more to do with post-Katrina George W. Bush than any coherent definition of ideology. As much as the Democrats would like to pretend otherwise, ideological is not a term I d readily apply to the bumbling nature of Bush s second term. Nor am I really convinced that the average voter is doing much more than engaging in post-hoc rationalized-as-something-else economic voting, which doubtless makes me no fun at parties when I play a public opinion scholar. ( Yes, all this crap matters at the margins, and occasionally elections are won at the margins, but most of the time it doesn t matter. ) But I digress. There is a broader lesson, though, in that to the extent the Democrats believe that their recent success is due to their ideology it is at their long-term peril, particularly if they bypass pragmatism in favor of catering to the cobbled-together collection of rent-seekers that passes for the Democratic coalition. To the extent bailing out Detroit is seen as a Democatic handout to its paymasters particularly with the emerging frame of the greedy unions are standing in the way trumping any sort of concept that any deal that tells the UAW to can their contract is essentially an impairment of the obligation of contract (which, believe it or not, is unconstitutional) the auto bailout won t go over well in the 48 or so states that don t host significant Big Three production. On a related note, Steven Taylor notes the defining-down of the filibuster by the media to mean failure to win a cloture vote. While I accept the point graciously, I think this may be more a failure of us as a profession than the media per se; modern congressional procedures (not just filibusters and holds, but also esoterica such as the Rules Committee and UCAs), even superficially treated, aren t a strong point of most American government textbooks, and more often than not that s the only real government orientation budding journalists will get. I make a point of assigning Barbara Sinclair s Unorthodox Lawmaking in my Congress classes, but I doubt the average journalist gets that in-depth in their undergrad days. So here at least I think the blame falls somewhat closer to home than we might want to admit.

Chris Lawrence: Corollary of the day

King Politics thinks the GOP anti-bailout faction has the politics of the day wrong:
The time for pragmatism is now, but Senate Republicans don t recognize that. Senate Republicans have yet to realize that the GOP lost so many seats in 2006 and 2008 because the American public has a greater desire for pragmatism than ideology.
I think that s true to a point but I suspect it has more to do with post-Katrina George W. Bush than any coherent definition of ideology. As much as the Democrats would like to pretend otherwise, ideological is not a term I d readily apply to the bumbling nature of Bush s second term. Nor am I really convinced that the average voter is doing much more than engaging in post-hoc rationalized-as-something-else economic voting, which doubtless makes me no fun at parties when I play a public opinion scholar. ( Yes, all this crap matters at the margins, and occasionally elections are won at the margins, but most of the time it doesn t matter. ) But I digress. There is a broader lesson, though, in that to the extent the Democrats believe that their recent success is due to their ideology it is at their long-term peril, particularly if they bypass pragmatism in favor of catering to the cobbled-together collection of rent-seekers that passes for the Democratic coalition. To the extent bailing out Detroit is seen as a Democatic handout to its paymasters particularly with the emerging frame of the greedy unions are standing in the way trumping any sort of concept that any deal that tells the UAW to can their contract is essentially an impairment of the obligation of contract (which, believe it or not, is unconstitutional) the auto bailout won t go over well in the 48 or so states that don t host significant Big Three production. On a related note, Steven Taylor notes the defining-down of the filibuster by the media to mean failure to win a cloture vote. While I accept the point graciously, I think this may be more a failure of us as a profession than the media per se; modern congressional procedures (not just filibusters and holds, but also esoterica such as the Rules Committee and UCAs), even superficially treated, aren t a strong point of most American government textbooks, and more often than not that s the only real government orientation budding journalists will get. I make a point of assigning Barbara Sinclair s Unorthodox Lawmaking in my Congress classes, but I doubt the average journalist gets that in-depth in their undergrad days. So here at least I think the blame falls somewhat closer to home than we might want to admit.

Adrian von Bidder: Adult content?

The best way to get me to read a blog entry you've made is to apologize about it in the next one ... So I've gone and clicked on the link to Amaya's previous entry. Getting a warning that I'm about to be presented with a page that may contain material only suitable for adults. What the hell, Amaya is an adult and so am I, so let's just confirm this. I was truly shocked to see her entry that indeed did contain the extremely dangerous word “love” in it (can't imagine what else may have triggered the filter.) Together with Youtube's apparent tightening of their filtering (I'd not have noticed except that Illiad commented on it) it seems the Intarwebs are now being converted to a happy teletubby-land. So I'm forced append this disclaimer now: Attention please, the text you've just read may have contained strong words, opinions, sentences, punctuation characters and all kinds of dangerous information only suitable for inhabitants of a small planet somewhere in the vicinity of Betelgeuse.

11 December 2008

Russell Coker: Please Turn off Your Spam Protection

Hi, I’d like to send an email from a small domain that you’ve never heard of or from a big ISP that’s known for being slack about spam (*), I can’t send the mail to you because of your anti-spam measures. I think that this is unfair, it’s discrimination, and you are cutting off your nose to spite your face in rejecting my mail. So please reconfigure your mail server now and accept more spam in your inbox, my message is important enough to justify the extra time you will spend manually deleting mail and the risk of accidentally deleting legitimate mail while deleting heaps of spam (I am in fact more important than you). By not accepting my mail you are being an asshole. I only receive a dozen spam messages a day and I don’t mind it, without even knowing you or having bothered to do a web search to see how well your email address is known I’m sure that you don’t receive any more spam than me and therefore you too can turn off most anti-spam measures and manually sort through the spam. You don’t really have a problem with spam, you are just paranoid, I’m sure that you installed your anti-spam measures before receiving any spam and then never bothered to check the hit rates. My sys-admin knows that one of the DNSBLs has an old entry from when his server was broken, but he won’t request that it be removed - so you can change your server because my sys-admin doesn’t want to click on a URL that you sent him. The RFC-ignorant.org service is used and run by ignorant people - I know this without even reading their web site to discover how it works. The above is a summary of a number of complaints that I have received about my anti-spam measures. I’ve para-phrased them so that they make sense, I have not actually had someone directly say “I’m more important than you so you should just accept more spam”, but the implication is quite clear. Now there are some legitimate reasons for requesting that anti-spam measures be reduced. In the distant past almost everyone had working reverse DNS entries which matched the forward entries and it was common to reject mail from systems that didn’t have valid DNS. Nowadays there are many big ISPs that delegate IP address space without permitting reverse DNS entries, and there are companies that have one department in charge of IP addresses (who don’t have a clue about reverse DNS) and another department running mail servers (who are cluefull). So the environment has changed to make reverse DNS checks a non-viable anti-spam measure. Requesting that people remove such checks is reasonable. Anti-spam measures that attack innocent third parties are bad. Sending “warnings” about viruses has made no sense for many years as all modern viruses fake the sender address, an employee of one company once admitted to me that they were sending out anti-virus “warning” messages as a way of sending unsolicited advertising to random people (I reported them as a spam source). Some time ago on a routine upgrade of ClamAV I accidentally copied in a default configuration file that made it send such warnings - I was grateful when someone informed me of my mistake so that I could fix it. Challenge-response is another technology that causes harm to others. I think it makes a lot of sense for mailing lists (every modern list server will confirm subscription requests), while it does result in sending unwanted mail to innocent third parties (every time a new virus becomes prevalent I receive a number of confirmation messages from list servers), but it’s not something that I will use on a regular email account (but I am prepared to do paid work implementing CR for other people). Requesting that manually implemented blocks be removed is quite reasonable. Occasionally I find that mail from one of my servers is blocked because a previous owner of the IP address space did bad things. In such a situation it is quite reasonable to provide an assurance that the new owner takes abuse issues seriously and to request that the block be removed. Requesting that I make any change to my system without making a minimal effort to get your broken mail server fixed is totally unreasonable. If the system administrator is not prepared to click on a URL to get their system removed from a black-list or if the user is unwilling to report the problem to the sysadmin then I will probably be unwilling to make any change to my system. The only exceptions to this rule are for clients, colleagues, and for people who use mail services that are large and unresponsive to users (IE the users don’t directly pay). I recently made a white-list entry for a large European ISP that is used by a Debian Developer for their work, as the ISP is known to be unresponsive to requests and mail related to Debian work is important to me I added a white-list entry. One thing I am planning to do is to document my anti-spam measures and then allow people the opportunity of suggesting new anti-spam measures that I haven’t tried yet if they want me to turn off any of my current protections. (*) I’m not aware of any big ISP that takes strong measures against spamming customers and customers whose computers are trojaned. I am aware of one having done it in the past, but I suspect that they may have ceased doing so after their brush with bankruptcy. I suspect that many ISPs simply rate-limit their customers connections to the outbound mail relays and hope that they don’t get enough infected customers at any time to get themselves listed as a spam source.

10 December 2008

Enrico Zini: Bash variable indirection

Bash variable indirection This is not how to get, in bash, the value of a variable whose name is in another variable:
$ A=1
$ B=A
$ echo $ $B 
bash: $ $B : bad substitution

This is how:
$ echo $ !B 
1

It however only works in bash. For more portable scripts, eval is needed:
$ eval echo \$$B
1

Thanks to Martin Krafft and Cyril Brulebois.

9 December 2008

Wouter Verhelst: Rails: followup

I received quite some comments on my rant about Ruby on Rails a few days ago; I guess I must have received some inbound link on a ruby-fanboy-website or some such. Most of it is just bashing me since, I'll admit, my post wasn't very constructive (and which according to my policy of only blocking spammers I've not moderated away), some of it just posits another opinion about packaging (which is valid), and one Eric Anderson gave a good and constructive set of arguments that I just have to reply to. First of all, yes, I hate RubyGems more than I hate Ruby on Rails. However, Ruby on Rails is the major success story in the Ruby community, and the main reason why people use that language; it's not unfair to say that I won't use RoR because of RubyGems. But perhaps I should've been clearer. Anyway. Eric mentions that RubyGems "solves two problems that other packaging systems are not addressing": the ability to install two versions of the same library, and the ability to install a single package on multiple operating systems. Well, first of all, the latter one simply is not true. There's CPAN.pm, which does much the same thing as RubyGems; but much of CPAN can be easily turned into a Debian (or RPM, or whatnot) package. The few cases where it can't, you're usually dealing with a broken CPAN package, anyway. There are similar things for Python (egg), and, heck, even TeX (CTAN). The fact that other scripting languages have proper and working packaging systems only outlines that there really isn't an excuse for the horrors of RubyGems. Second, the fact that you even need to be able to install multiple versions of a library really shows that there's a problem with your development ecosystem. C, Perl, and Python programmers are used to stable APIs, and prefer not to change their APIs all that much. Most packaging systems reflect that preference, and deal with incompatible API changes the way they should: as the exception, rather than the rule. The people in the Ruby ecosystem clearly have a different opinion on that, which probably results in the difference in packaging system requirements. Personally, I prefer the non-Ruby way—I'll leave the discussion about "what happens when you need to do an urgent security update of one of your libraries but the API has changed three times since you last looked at it and now the library update breaks your application" as an excercise to the reader. In closing, I'd like to reply to this one sentence that Eric put in his comment:
And of course there is no reason software cannot start as a gem (for people close to the ruby community) but as it's popularity increases packages can be created that are more tightly integrated into the native package manager.
Well, in an ideal world, he'd be right: there would be no reason. Unfortunately, in the RubyGems world—which is far, far away from an ideal world—the fact is that packaging RubyGems in a sane way so that they'll integrate with the rest of a Debian system is simply impossible. You simply cannot currently go from a gem and create a package out of that, because of how gems are done. That, more than anything, was what I was objecting to. I can understand that many Ruby developers do not care about this much; that they are used to RubyGems, and that they can geet anything they need using that system. However, they should understand that different people have different preferences, and that packages for systems other than RubyGems is going to be necessary if their software is to become popular beyond the select set of people who care enough about Ruby. I personally know of plenty of people (myself included) who prefer things to be installed as operating-system level packages on the Linux systems they install; and even if it's not your personal preference to do things that way, you should at the very least not stand in the way of those who do.

7 December 2008

Chris Lamb: Taking Liberties

On Saturday I went to the British Library's Taking Liberties exhibition on "the 900-year struggle for Britain's freedoms and rights". The majority of the items on display were truly iconic, including a copy of the Magna Carta, Ms Davidson's return ticket from Epsom as well as Charles I's death warrant - with Cromwell's signature easily identifiable in third position. In contrast, the video interviews that were scattered around were rather distracting at best (isn't it a library?) and embarrasingly biased at worst. I wonder how many visitors need convicing that detaining suspects without charge for forty-two days isn't a good idea. Near the exit was a wonderfully lo-fi anonymous commenting system based on a wall of bulldog clips and postcard-sized bits of paper. Fascinating in itself.

Kapil Paranjape: A double blow

On the 26th of November 2008, there was a double blow. It had been raining almost all night in Chennai and we put on the TV to figure out what the weather was going to be like only to find out that there had been attacks with guns and bombs in multiple areas in Mumbai. The attacks in Mumbai and the rain in Tamil Nadu continued over the next 36 hours or so --- with disastrous effects all around. All this was more than a week ago and so it feels as if one can write about it with a modicum of objectivity. The cyclone that hit Tamil Nadu was not man-made, but many of its consequences were. The attacks in Mumbai were carried out by people who were human beings once upon a time, but they had their humanity stripped by those who run the camps which trained them. So to some extent it feels as if we have suffered from a natural disaster --- but one which could have been mitigated by humans. The Indian state and people seem to be returning back to "normal" and this is said be a sign of our resilience. On the other hand that may only be because we suffer a thousand cuts every day --- self-inflicted. Indeed, each one of us is a minor terrorist carrying out little acts of rebellion against the state; acts that also threaten the well-being of our fellow citizens. Some examples follow: The list could go on and on ... All of these acts weaken our state and our people to the point where we cannot act swiftly in an emergency: This is what makes us a "soft target for terror". If the Indian government were to indeed go after all those who are weakening it and terrorizing its people, then a good percentage of our citizenry and an even larger percentage of our well-off citizenry would find itself in the cross-hairs. To paraphrase what an assistant commissioner of police once said: If 100% of our citizens are 2% terrorists then it is very hard to catch the 2% of the citizens who are 100% terrorists.

6 December 2008

Cyril Brulebois: No BBC outside the UK

Jon: No, we can t (LWN coverage).

Erich Schubert: On Facebook

I've become a Facebook user. Caused by most of my dancing friends being there: Facebook is a convenient and easy way to keep some contact with all those people that aren't like your core friends, but still friends, and that are all around the world.I've also toyed around with the Facebook API, and actually written two small applications for it. From a technical point of view, I'm actually quite impressed by Facebook.Yes, Facebook seems to have quite some load trouble these days. Sometimes it's very slow. Sometimes it just malfunctions (with being kicked out and having to re-login being just the least). These days, messages were distorted every now and then, some app messages were always missing the last few words.What seriously does impress me is the architecture they use with third party applications. It's designed around cachability (so profile pages aren't slowed down by slow or broken third party applications) and they do jump some hoops to allow application writers do a lot of things while preventing them to disturb other applications or the core functionality.For all I know, Facebook is the first major thing to do CSS and JavaScript rewriting. Data produced by third party applications is fed through a very smart parser and rewriter that allows an impressively large subset of CSS and JavaScript to be used without the developers having to pay attention to not producing conflicts. In CSS, rules are prefixed with a selector to restrict them to their applications scope. In JavaScript, object references are uniquified, and the convenience functions you have for interacting and accessing nodes (including functions to do common things such as modify CSS class assignments) take care of all that. Access to the raw JavaScript methods is filtered, so you can't e.g. use parentNode to get access to objects outside of your scope. At least in theory.Much of this is for the benefit of users: applications are not allowed to do annoying animations unless the user has just interacted with them; apps also can't modify or disturb others, or read data from other applications via DOM.Well, of course there might be one or another security issue still there; some of these things might also be related to the performance issues of Facebook recently. And of course there are bugs. Lots. A couple of things still need to be thought through properly (e.g. aggregation of feed messages with multiple targets, localization functionality for applications, finer grained control of data access). But their CSS and JavaScript rewriting is really cool.

5 December 2008

Martin F. Krafft: forcedeth: nVidia network chips are broken

Yesterday, I removed my new server from the rack and brought it back home, after the problems with the nVidia network chip (forcedeth) took down the NIC to the point that the IPMI chip, which is routed through the primary interface, wasn t reachable anymore. Even though a soft reboot fixed the problem, a bit of large-packet traffic, like downloading via IPv6, broke the card again. Since IPMI is also affected, I cannot remotely manage the machine and thus can t leave it in the rack. In fact, I am strongly considering to make use of the try&buy contract and return the thing. I cannot rule out a software problem, but given that the NIC goes into a state in which it gets unusable even to the IPMI system, which is completely independent of Linux, I somewhat doubt it. Instead, I suspect a hardware error, beyond the known problems with nVidia network chips and segmentation/checksum offloading. On the other hand, it s not news that drivers can break hardware, and the fact that I am using Linux is reason for hope. One alternative, which would allow me to potentially help in fixing this bug, is to use a riser card to stuff a different network card into the server. This would mean I didn t invest all the time into the server for nothing. Unfortunately, this workaround comes with extra costs, and would require Init Seven to allocate another switch port for the IPMI card, which I am not sure they ll be keen about. The bottom line of the story is that I will avoid nVidia even more in the future, and you might want to do so as well. Companies that produce crap hardware and do not cooperate with people writing free drivers for them do not deserve the money. Unfortunately, almost everyone out there uses the MCP55 chipset these days, and deny any problems with it. I guess I will take a look at HP and IBM, although I d really prefer not to pay for their brands, and not to enslave myself to their customer support standards. NP: Neil Young: Dead Man

2 December 2008

David Moreno Garza: DirectoryIndex and the perl-script handler

Wouter Verhelst blogs about combining ScriptAlias (or a way to run CGIs) and DirectoryIndex in Apache. Recently, developing a TinyURL clone I had a similar (not identical, though) issue. Basically, I set SetHandler to perl-script on the “/” location, which basically means, everything accessed on a given virtual host, is affected. Because you are changing the handler for the entire location, DirectoryIndex will have no effect on it because mod_dir is the one dealing with the DirectoryIndex function, that is, using the DIR_MAGIC_TYPE handler. To fix this you can use a mod_perl (2, I never really used 1) Fixup phase handler:
<VirtualHost *:80>
	...
	# some stuff
	...
	# PerlSections rule.
	<Perl>
	$Location "/"  =  
		SetHandler => 'perl-script',
		# some stuff
		PerlFixupHandler => 'Axiombox::Awbox::Fixup',
		# some other stuff
		DirectoryIndex => 'whatever.html',
	 ;
	</Perl>
</VirtualHost>
If it looks incomplete is because some other information, out of the scope of this sample, like DocumentRoot or other mod_perl directives, are hidden as the “other stuff”. My Fixup.pm handler looks like this:
package Axiombox::Awbox::Fixup;
use strict;
use warnings FATAL => qw(all);
use Apache2::Const -compile => qw(DIR_MAGIC_TYPE OK DECLINED);
use Apache2::RequestRec;
sub handler  
	my $r = shift;
	if ($r->handler eq 'perl-script' &&
		-d $r->filename              &&
		$r->is_initial_req)
	 
		$r->handler(Apache2::Const::DIR_MAGIC_TYPE);
		return Apache2::Const::OK;
	 
	return Apache2::Const::DECLINED;
 
1;
Which is very straight-forward: If the request is set to perl-script, the requested file is a directory and if the current request is the main one, then change the handler and return to the normal flow of phases. Otherwise, decline the Fixup phase handler.

Wouter Verhelst: Combining ScriptAlias and DirectoryIndex

Apparently that's not possible. When you say "ScriptAlias /cgi-bin /usr/lib/cgi-bin", everything under /usr/lib/cgi-bin is assumed to be a script. That counts for directories, too; so trying to access 'http://localhost/cgi-bin/myapp' isn't going to work, even if you say 'DirectoryIndex index.cgi' and create a file /usr/lib/cgi-bin/myapp/index.cgi. Workaround: use mod_rewrite:
<Directory /usr/lib/cgi-bin/myapp>
	RewriteEngine on
	RewriteBase /cgi-bin/myapp
	RewriteRule ^$ index.cgi
</Directory>
Sucky, but it works.

1 December 2008

Russell Coker: Gmail and Anti-Spam

I have just received an email with a question about SE Linux that was re-sent due to the first attempt being blocked by my anti-spam measures. I use the rfc-ignorant.org DNSBL services to stop some of the spam that is sent to me. The purpose of rfc-ignorant.org is to list systems that are run by people who don’t know how to set up mail servers correctly. But the majority of mail that is blocked when using them comes from large servers owned by companies large enough that they almost certainly employ people who know the RFCs (and who could for a trivial fraction of their budget hire such people). So it seems more about deliberately violating the standards than ignorance. The person who sent me the email in question said “hopefully, Google knows how to make their MTA compliant with RFC 2142“, such hope is misplaced as a search for gmail.com in the rfc-ignorant.org database shows that it is listed for not having a valid postmaster address [1]. A quick test revealed that two of the Gmail SMTP servers support the postmaster account (or at least it doesn’t give an error response to the RCPT TO command that is referenced in the complaint). However Gmail administrators have not responded to the auto-removal requests, which suggests that postmaster@gmail.com is a /dev/null address. However that is not a reason to avoid using Gmail. Some time ago Gmail took over the role of “mail server of last resort” from Hotmail. If you have trouble sending email to someone then using a free Gmail account seems to be the standard second option. Because so many people use Gmail and such a quantity of important mail is sent through that service (in my case mail from clients and prospective clients) it is not feasible to block Gmail. I have whitelisted Gmail for the rfc-ignorant.org tests and if Gmail starts failing other tests then I will consider additional white-lists for them. Gmail essentially has a monopoly of a segment of the market (that of free webmail systems). They don’t have 100%, but they have enough market share that it’s possible to ignore their competitors (in my experience). When configuring mail servers for clients I make sure that whatever anti-spam measures they request don’t block Gmail. As a rule of thumb, when running a corporate mail server you have to set up anti-spam measures to not block the main ISPs in the country (this means not blocking Optus or Telstra BigPond for Australian companies) and not block Gmail. Not blocking Yahoo (for “Yahoo Groups”) is also a good thing, but I have had a client specifically request that I block Yahoo Groups in the past - so obviously there is a range of opinions about the value of Yahoo. Someone contacted Biella regarding an email that they couldn’t send to me [2]. I have sent an email to Biella’s Gmail account from my Gmail account - that should avoid all possibility of blocking. If the person who contacted Biella also has a Gmail account then they can use that to send me email to my Gmail account (in the event that my own mail server rejects it - I have not whitelisted Gmail for all my anti-spam measures and it is quite possible for SpamAssassin to block mail from Gmail). It turns out that the person in question used an account on Verizon’s server, according to rfc-ignorant.org Verizon have an unusually broken mail server [3]. If your ISP is Optus, BigPond, Verizon, or something similarly broken and you want to send mail to people in other countries (where your ISP is just another annoyance on the net and not a significant entity that gets special treatment) then I suggest that you consider using Gmail. If nothing else then your Gmail account will still work even after your sub-standard ISP “teaches you a lesson” [4].

28 November 2008

Theodore Ts'o: An ethical question involving ebooks

I recently purchased a short story from Fictionwise, which was not DRM’ed, so I could easily get it into a form where I could read it on my Sony eReader. Thanks to that short story, I was introduced to an author, and a character, which I found very engaging. When I decided to find out more about the character, I found that the author had written two additional short stories, and three additional novels many years ago, but has since stopped writing any more books involving that character. Furthermore, the novels have gone out of print, and are only available from amazon.com as used books. Unfortunately, I travel a lot. So much so, that one of the few times that I have time to read is when I’m traveling. And I really dislike having to haul dead-tree versions of my favorite novels around; they take up far too much weight and space in my carry-on luggage. Unfortunately, these out-of-print novels were published by a Neanderthal Publishing company who hasn’t made any of the books available in ebook format, DRM’ed or no. Grumpy, I searched on Internet, and found all three novels were easily available for free download — in a pirated form, of course. Should I download them and convert them into a form which would allow me to read them on my Sony eReader? Well, according to Russell Davis, former chair (and now president of the Science Fiction Writer’s Association) of the SFWA’s Copyright Committee, “electronic infringement is theft”. From a legal perspective, I suppose that is true. And given that as an Open Source programmer, I depend on Copyright Law to assure that my wishes as an author are upheld, it would be hypocritical for me assume that I should be able to ignore Copyright Law just because it is inconvenient. And yet… from a moral perspective, who has really lost anything? The argument made by Russell Davis is that infringement is bad because it is “harming authors and author estates”. Jerry Pournelle has indignantly proclaimed that e-piracy goes against a “specific (and very stern) Biblical injunction against stealing from widows and orphans”. Of course, in this case, the author is still alive (and is female, although I suppose stealing from widowers would be just as bad). Also, given that the author has publically stated she doesn’t plan to write any more books involving this character (since some of her more psychotic readers sent her death threats as a result of reading said books), the publisher is highly unlikely to re-release said novels — and if I buy used dead-tree versions of said novels, the author doesn’t receive any additional royalties. So, then, where is the moral bright line? Somewhere along this continuum, we’ve crossed over from the light-side to the dark-side. Setting aside the observation that the Neanderthal attitudes and business practices of the publisher involved has made it impossible for me to legitimiately follow the law, enjoy the novels, and direct money to the author via royalty payments — what do you think is the morally correct course of action? And why? And if you don’t mind saying so publically, roughly what generation are you from (i.e., Baby Boomer, Gen X, Gen Y, etc.)? I’m curious how attitudes are changing based on age, and whether folks who are currently in college might differ from those who can remember a time when the Internet didn’t exist… Update: I’ve posted a follow-up to this post here.

27 November 2008

Russell Coker: Links November 2008

Netatia has an interesting series of articles about running a computer for two people [1]. It is a bit of a kludge, they have a single X server that covers both displays and then use Xephyr to divide it into two virtual screens. The positive aspecct of this is that it shuld allow a single wide monitor to be used by two sessions as displays are getting wider regardless of the wishes of manufacturers and consumers [2] this should be useful. It’s a pity that no-one has solved the problem of having multiple video cards, sound cards, and input devices to allow a single desktop system to be used for 6 or more people. It seems that the problems that need to be solved are only the support for multiple video cards, mouse-wheel support, and sound support. Paul Ewald gave an interesting TED talk about changing the conditions for diseases so that they evolve to be benign [3]. The first example is Cholera which if spread by water will benefit from being as toxic as possible (to cause the greatest amount of diorrhea - killing the host not being a problem), but if spread by human contact benefits from leaving it’s host well enough to walk around and meet people. This and the other examples he cites seem like strong reasons for universal health-care provided by the government. If clean water is provided to all the poor people then cholera will evolve to be less harmful, and if a rich person (such as myself) is unlucky enough to catch it then the results won’t be so bad. He also notes that less harmful bacteria will often result in the victim not seeking anti-biotics and therefore less pressure for the disease to evolve resistance to anti-biotics. Therefore the people who really need them (the elderly, the very young, and people who are already sick) will find them to be more effective. Paul Stamets gave a great TED talk about fungus [4]. One of his discoveries was that fungi can be used for breaking down petro-chemicals (they can eat oil). It would be interesting to see this tested on a large scale with one of the oil spils or with the polluted land around an ooil refinery. Also he has patented a method for using fungus to kill wood-eating ants (such as the ones that briefly infested his home). Robert Full gave an interesting TED talk on robot feet [5]. I found the bit about leg spikes particularly interesting (I had always wondered why insects have spikey legs). Alan Kay gave a very interesting presentation on using computers to teach young children about science [6]. An OLPC is referenced. It makes me want to buy an OLPC for everyone I know who has young children. The start of the talk is a little slow. Dan Barber gave a very interesting TED talk about organic and humane production of foie gras in Extramuda [7]. Apparently it tastes a lot better too. Incidentally I don’t list all the TED talks I watch, only the better ones. Less than half the TED talks that I see announced seem interesting enough to download, and of those less than half are good enough that I will recommend them. The ones that I don’t recommend don’t suck in any way, it’s just that I can’t write a paragraph about every talk. Of recent times my video watching has been divided about equally between “The Bill” and TED talks. Here’s an interesting article about Sarah Palin and “anti-elitism”: The prospects of a Palin administration are far more frightening, in fact, than those of a Palin Institute for Pediatric Neurosurgery. Ask yourself: how has “elitism” become a bad word in American politics? There is simply no other walk of life in which extraordinary talent and rigorous training are denigrated. We want elite pilots to fly our planes, elite troops to undertake our most critical missions, elite athletes to represent us in competition and elite scientists to devote the most productive years of their lives to curing our diseases. And yet, when it comes time to vest people with even greater responsibilities, we consider it a virtue to shun any and all standards of excellence. When it comes to choosing the people whose thoughts and actions will decide the fates of millions, then we suddenly want someone just like us, someone fit to have a beer with, someone down-to-earth in fact, almost anyone, provided that he or she doesn’t seem too intelligent or well educated.[8] Sarah will be representing the Republican party in 2012, the desire for leaders of average intelligence (or less) will still be around then. It will be interesting to see how many votes she gets and amusing to see her interviewed. The proceedings of the “Old Bailey” - London’s Central Criminal Court have been published [9]. It’s interesting to read some of the historical information about the legal system at the time. It made me appreciate how civilised the UK (and other countries that I have visited) are now. Bruce Schneier writes about the feture of ephemeral communication [10]. He concludes with the point “until we have a Presidential election where both candidates have a complete history on social networking sites from before they were teenagers we aren’t fully an information age society“. Of course as he notes the rules are written by the older people, currently I don’t think that any candidate for high office (cabinet minister or above) anywhere in the world can have a good history on the Internet. During the course of a decade or more on the net it’s impossible not to write something that can be used against you and no reasonable person could avoid changing their views on some issues in such a time period. That’s enough to lose an election with the way things currently work.

Russell Coker: Slicehost vs Linode

Six months ago I investigated the options for Xen virtual servers [1]. I ended up receiving an offer of free hosting and not needing that, but the research was useful. There is a good range of options for Xen servers with different amounts of CPU power, RAM, bandwidth, and disk space. There are a couple of things that seem to be missing, options to upgrade from virtual servers to physical servers, and information on dedicated disk and database performance - but I’ll explain that later after some history. About a week ago a client needed a Xen virtual server in a hurry, their main server (a Xen system that I run on hardware that they rent) was getting a bit overloaded and needed to have one of the largest DomUs moved off. I ended up recommending Linode [2] based on my research and comments I received. The Linode server is working quite well and the client is happy, one nice feature of Linode is the choice of server rooms that they offer. I was able to choose a room in the same region as the other servers that the client owns and thus get ping times that are sometimes less than 2ms! Due to a missing feature in a program that I’m maintaining for the client a large number of MySQL queries are being made. Due to a problem I’m having with MySQL it won’t let me create a slave database server so all the queries go over the VPN and use a large amount of data. This combined with the other traffic that should be going over that link means that about 600G per month is being used, fortunately that is rather cheap. Linode staff handled this very well, after the server had exceeded it’s quota by 120G they asked my client to confirm that the traffic was legitimate and then suggested an upgrade to a plan that could handle the traffic (which went smoothly). Now I have another week to add the feature in question before I meet the quota again. Shortly after getting the new virtual server running at full capacity David Welton wrote a detailed review of Linode and Slicehost for the issues that matter to his use [3]. His conclusion seems strongly in favor of Linode. But now I am looking at getting a Slicehost [4] virtual server for the same client (for a different project) because Slicehost is owned by Rackspace [5], and the new project if successful will need a set of powerful servers and Rackspace seems like a reasonable company to host that. The problem with Rackspace is that they (and every other ISP I’ve researched so far) seems to offer little in regard to customers who need serious disk IO. I am planning some servers that will have a write bottleneck on a MySQL database (or maybe multiple shards), so serious disk capacity is needed. At least I would like to be able to get disk storage by the tray (12-14 disks) with the controllers having RAID-6 support. Rackspace only offers RAID-5 (according to the “livechat” person), and we didn’t get as far as discussing how to add more trays. What would be ideal is if there was an ISP that had both virtual servers and physical servers (so I could start with a virtual server and move to a physical server when things are working well), and also serious storage options. They would offer internal disks, external RAID arrays, and NetApp Filers [6] (or some equivalent device). It would be really nice if I could just instruct the ISP to add another NetApp Filer to my back-end network and have it done for me (I’m certainly not going to visit the US to install new hardware). It’s been over a day since I submitted a sales request to NetApp asking whether they partner with any ISPs and I haven’t received a response. OpenSolaris with ZFS also sounds good for disk IO performance (they have similar features to NetApp). Unfortunately the support for OpenSolaris among ISPs is not that great (while everyone offers Linux and Windows), and I haven’t used any recent version of Solaris. So using OpenSolaris would require finding someone with the skills to manage it who can work for my client - as opposed to a NetApp device that would be just like any other NFS server, SAN, or iSCSI server. But I’m not ruling OpenSolaris out, if someone knows of a good ISP that hosts OpenSolaris machines and supports adding dozens of disks and decent amounts of NVRAM for ZFS then I would be interested to investigate it. Joyent has some interesting OpenSolaris virtual server plans [7], they are a little pricey but offer large amounts of data transfer. They don’t provide any information on disk IO capacity (other than saying that they use ZFS for good performance). I’ve just downloaded Nexenta (Debian with the OpenSolaris kernel) [8] and will test it out over the next few days. One of the reasons I’m tending towards Rackspace at the moment (with Slicehost as the entry point) is that they seem cooperative to customer requests. My discussions with them (on a web based “livechat” and on the phone) have indicated that they may be able to do something special for me.

26 November 2008

Russell Coker: Bill Joy

Some time ago Bill Joy (who is famous among other things for being a co-founder of Sun) [1] wrote an article for Wired magazine titled “Why the future doesn’t need us” [2]. He wrote many sensible things but unfortunately focussed on the negative issues and didn’t receive a good response. On reading it today I thought more highly of the article than I did in 2000 when it was printed, largely due to having done some background research on the topic. I’ve recently been reading Accelerating Future [3] which has a more positive approach. Now a talk by Bill Joy from 2006 has been published on the TED.com web site [4]. He starts by talking about “super empowered individuals” re-creating the 1918 flu. He also claims that “more technology super-empowers people more“. Such claims seem to be excessively hyped, I would be interested to see comments from someone who has a good knowledge of current bio-technology as to the merits of those claims. He talks briefly about politics and has some good points such as “the bargain that gives us civilisation is the bargain to not use power” and “we can’t give up the rule of law to fight an asymmetric threat - which is what we seem to be doing“. He mentions Moore’s law and suggests that a computer costing $1000 then (2006) might cost $10 in 2020. He seems to be forgetting the cost of the keyboard and other mechanical parts. I can imagine a high-end CPU which cost about $800 a couple of years ago being replaced by a $2 CPU in 2020, but I don’t expect a decent keyboard to be that cheap any time before we get fully automated nano-factories (which is an entirely separate issue). Even the PSU (which has a significant amount of government regulations for safety reasons) will have a floor cost that is a good portion of $10. Incidentally the keyboard on my EeePC 701 sucks badly, I guess I’m spoiled by the series of Thinkpad keyboards that I keep getting under warranty (which would cost me a moderate amount of money if I had to pay every time I wore one out). I will make a specific prediction, that by 2015 one of the better keyboards will comprise a significant portion of the entire cost of a computer system (more than a low end computer unit) - such that in some reasonable configurations the keyboard will be the most expensive part. It would be good if PCs could be designed to use external PSUs (such as the Compaq Evo models that took laptop PSUs). Then the PSU, keyboard, and monitor would be optional extras thus giving a small base price. Given that well built PSUs and keyboards tend not to wear out as fast as people want to replace computers, it seems that financial savings could be provided to most customers by allowing them to purchase the computer unit without the extra parts. People like me who type enough to regularly wear out keyboards and who keep using computers for more than 5 years because they still work are in a small minority and would of course be able to buy the same bundle of computer, PSU, and keyboard that new users would get. In 2020 a device the size of an iPaQ H39xx with USB (for keyboard and mouse) and the new DisplayPort [5] digital display interface (that is used in recent Lenovo laptops [6]) would make a great PDA/desktop. You could dock a PDA the way some people dock laptops now and carry all your data around with you. Bill cites an example of running the Mac interface on an Apple ][ (does anyone know of a reference for this) as an example of older hardware being more effective with newer software. It's a pity that often new software needs such powerful hardware. EG it's only recently that hardware developments have overtaken the developments of OpenOffice to make it deliver decent performance. He has an interesting idea of using insurance companies to replace government regulation of food and drugs. The general concept is that if you can convince an insurance company that your new drug is not a great risk to them so that they charge a premium you can afford then you could sell it without any further testing. Normally I'm in favor of government regulation of such things, but given the abject failures of the US government this idea has some merit. Of course there's nothing stopping insurance companies from just taking a chance and running up debts that they could never hope to repay (in a similar manner to so many banks). Finally I think it's interesting to note the camera work that the TED people used. My experience of being in the audience for many lectures (including one of Bill Joy's lectures in the late 90's) is that a speaker who consults their notes as often as Bill does gives a negative impression to the audience. Note that I'm not criticising Bill in this regard, often a great talk requires some significant notes - very few people can deliver a talk in a convincing manner entirely from memory. It seems to me that the choices of camera angle are designed to give a better impression than someone who was seated in the audience might receive - there's no reason why a video of a talk should be spoiled by seeing the top of the speaker's head while they consult their notes!

25 November 2008

Ian Wienand: Position Independent Code and x86-64 libraries

If you've ever tried to link non-position independent code into a shared library on x86-64, you should have seen a fairly cryptic error about invalid relocations and missing symbols. Hopefully this will clear it up a little! Let's start with a small program to illustrate.
$ cat function.c
int global = 100;
int function(int i)  
	return i + global;
 
$ gcc -c function.c
Firstly, inspect the disassembley of this function:
0000000000000000 <function>:
   0:	55                   	push   %rbp
   1:	48 89 e5             	mov    %rsp,%rbp
   4:	89 7d fc             	mov    %edi,-0x4(%rbp)
   7:	8b 05 00 00 00 00    	mov    0x0(%rip),%eax        # d <function+0xd>
   d:	03 45 fc             	add    -0x4(%rbp),%eax
  10:	c9                   	leaveq
  11:	c3                   	retq
Lets just go through that for clarity: The IP relative move is really the trick here. We know from the code that it has to move the value of the global variable here. The zero value is simply a place holder - the compiler currently does not determine the address (i.e. how far away from the instruction pointer the memory holding the global variable is. It leaves behind a relocation -- a note that says to the linker "you should determine the correct address of foo, and then patch this address of the code to point to that addresss (i.e. foo)." Relocations with addend The image above gives some idea of how it works. We can examine relocations with the readelf tool.
$ readelf --relocs ./function.o
Relocation section '.rela.text' at offset 0x518 contains 1 entries:
  Offset          Info           Type           Sym. Value    Sym. Name + Addend
000000000009  000800000002 R_X86_64_PC32     0000000000000000 global + fffffffffffffffc
If you try and build a shared object (dynamic library) with this function, you should get something like
$ gcc -shared function.c
/usr/bin/ld: /tmp/ccQ2ttcT.o: relocation R_X86_64_32 against  a local symbol' can not be used when making a shared object; recompile with -fPIC
/tmp/ccQ2ttcT.o: could not read symbols: Bad value
collect2: ld returned 1 exit status
Position Independent Code (PIC, enabled with -fPIC) just means that the output binary does not expect to be loaded at a particular base address, but is happy being put anywhere in memory (compare the output of readelf --segments on a binary such as /bin/ls to that of any shared library). This is obviously critical for implementing shared-libraries, where you may have many, many libraries loaded in essentially any order, and trying to pre-allocate where in memory they would all live just does not work. However, PIC has slightly wider implications than simply movable base addresses. So, back to relocations. The exact rules for different relocation types are described in the ABI for the architecture. The R_X86_64_PC32 relocation is defined as "the base of the section the symbol is within, plus the symbol value, plus the addend". The addend makes it look more tricky than it is; remember that when an instruction is executing the instruction pointer points to the next instruction to be executed. Therefore, to correctly find the data relative to the instruction pointer, we need to subtract the extra. This can be seen more clearly when layed out in a linear fashion (as in the bottom of the above diagram). What's the problem with this in a shared library? In a shared library situation, we can not depend on the local value of global actually being the one we want. Consider the following example, where we override the value of global with a LD_PRELOAD library.
$ cat function.c
int global = 100;
int function(int i)  
	return i + global;
 
$ gcc -fPIC -shared -o libfunction.so function.c
$ cat preload.c
int global = 200;
$ gcc -shared preload.c -o libpreload.so
$ cat program.c
#include <stdio.h>
int function(int i);
int main(void)  
   printf("%d\n", function(10));
 
$ gcc -L. -lfunction program.c -o program
$ LD_LIBRARY_PATH=. ./program
110
$ LD_PRELOAD=libpreload.so LD_LIBRARY_PATH=. ./program
210
If the code in libfunction.so has a fixed offset into its own data section, it will not be able to see the overridden value provided by libpreload.so. This is not the case when building a stand-alone executable, where references are satisfied internally. Of course, any problem in computer science can be solved with a layer of abstraction, and that is what is done when compiling with -fPIC. To examine this case, let's see what happens with PIC turned on.
$ gcc -fPIC -shared -c  function.c
$ objdump --disassemble ./function.o
./function.o:     file format elf64-x86-64
Disassembly of section .text:
0000000000000000 <function>:
   0:	55                   	push   %rbp
   1:	48 89 e5             	mov    %rsp,%rbp
   4:	89 7d fc             	mov    %edi,-0x4(%rbp)
   7:	48 8b 05 00 00 00 00 	mov    0x0(%rip),%rax        # e <function+0xe>
   e:	8b 00                	mov    (%rax),%eax
  10:	03 45 fc             	add    -0x4(%rbp),%eax
  13:	c9                   	leaveq
  14:	c3                   	retq
It's almost the same! We setup the frame pointer with the first two instructions as before. We push the first argument into memory in the pre-allocated "red-zone" as before. Then, however, we do an IP relative load of an address into rax. Next we de-reference this into eax (e.g. eax = *rax in C) before adding the incoming argument to it and returning.
$ readelf --relocs ./function.o
Relocation section '.rela.text' at offset 0x550 contains 1 entries:
  Offset          Info           Type           Sym. Value    Sym. Name + Addend
00000000000a  000800000009 R_X86_64_GOTPCREL 0000000000000000 global + fffffffffffffffc
The magic here is again in the relocations. Notice this time we have a P_X86_64_GOTPCREL relocation. This says "replace the data at offset 0xa with the global offset table (GOT) entry of global. Global Offset Table operation with data variables As shown above, the GOT ensures the abstraction required so symbols can be diverted as expected. Each entry is essentially a pointer to the real data (hence the extra dereference in the code above). Since the GOT is at a fixed offset from the program code, it can use an IP relative address to gain access to the table entries. This extra reference is obviously slower; however for the most part I imagine the overhead would be essentially immeasurable and is required for "generic" operation. If you have figured the cost of indirection through the GOT is the major bottleneck of your program, I imagine you wouldn't be reading this and would already be considering strategies to remove it! The next question is why this works on plain old x86-32. Inspecting the code reveals why:
$ objdump --disassemble ./function.o
00000000 <function>:
   0:	55                   	push   %ebp
   1:	89 e5                	mov    %esp,%ebp
   3:	a1 00 00 00 00       	mov    0x0,%eax
   8:	03 45 08             	add    0x8(%ebp),%eax
   b:	5d                   	pop    %ebp
   c:	c3                   	ret
$ readelf --relocs ./function.o
Relocation section '.rel.text' at offset 0x2ec contains 1 entries:
 Offset     Info    Type            Sym.Value  Sym. Name
00000004  00000701 R_386_32          00000000   global
We start out the same, with the first two instructions setting up the frame pointer. However, next we load a memory value into eax -- as we can see from the relocation information, the address of global. Next we add the incoming argument from the stack (0x8(%ebp)) to the value in this memory location; implicitly dereferencing it. This provides the abstraction we need -- if the relocation makes the patched address at 0x4 the address of the GOT entry, it will be correctly dereferenced. It is the inability of the x86-32 architecture to try and optimise by doing instruction-pointer relative offseting which means it always needs to do slower memory references, which turns out to be just what you want when you're making a shared library! So, the executive summary: the ability of x86-64 to use instruction-pointer relative offsetting to data addresses is a nice optimisation, but in a shared-library situation assumptions about the relative location of data are invalid and can not be used. In this case, access to global data (i.e. anything that might be changed around on you) must go through a layer of abstraction, namely the global offset table.

Next.

Previous.