Search Results: "cas"

28 March 2009

Sean Finney: quoting and escaping positional parameters, redux

after all, what could be more exciting? okay, so where we last left off, we had something like:
    # escape any single quotes in an argument
    quote() 
      echo "$1"   sed -e "s,','\\\\'',g"
     
    # save up a properly quoted/escaped version of "$@"
    for arg in "$@"; do
      saved="$ saved:+$saved  '$(quote "$arg")'"
    done

i got some really interesting feedback from a few people; Lo c Minier who has some similar code that he cuts and pastes on an as-needed basis, and Ralf Wildenhues who has overseen some similar code that exists in the autoconf system. a notable feature in Lo c's solution was to quote the string entirely within the escaping function, whereas in mine the result was a "quotable" string (i.e. the quotes were put around the output afterwards).
> escape()  
>     echo "$*"   sed "s/'/'\"'\"'/g; s/.*/'&'/"
>  

Ralf pointed out one potential problem with both of these approaches. on the topic of portability, there are some implementations of echo that will directly interpret control characters ('\n', '\t', etc) and others not. he suggested that "printf" be used instead:
- echo "$1"   sed -e "s,','\\\\'',g"
+ printf '%s\n' "$1"   sed -e "s,','\\\\'',g"

though there's no reason i can think of not to take that a bit further and use a herestring, which could save an extra fork if printf isn't a builtin:
    quote() 
      sed -e "s,','\\\\'',g" << EOF
    $1
    EOF
     

also, Ralf suggested in the interest of efficiency that some kind of case statement be used to avoid unnecessary forks when an argument doesn't contain any "'" characters. this is apparently a trick used in autoconf-generated scripts:
    case "$arg" in
    *\'*)
      saved="$ saved:+$saved  '$(quote "$arg")'"
      ;;
    *)
      saved="$ saved:+$saved  '$arg'"
      ;;
    esac

finally, a lingering problem which hasn't been solved on the autoconf side of things (nor in my previous solution) was that in the case that an argument contained trailing newlines, such whitespace would be silently discarded in the quoted version (this is due to how shell command substitution works in general). but really, this is quite the corner case... how often does one run a command line with both an escaped quote and trailing newlines in an argument? an example:
rangda[/home/sean] ./good.sh "isn't very common" "to give args" "with trailing \\n's:
"
original command line, 3 arguments: isn't very common to give args with trailing \n's:
arg: isn't very common
arg: to give args
arg: with trailing \n's:
mangled command line, 0 arguments:
restored command line, 3 arguments: isn't very common to give args with trailing \n's:
arg: isn't very common
arg: to give args
arg: with trailing \n's:
rangda[/home/sean]                                                           :)

however, i noticed that a slightly modified version of Lo c's approach might not suffer from this problem because the string would be quoted before being returned (thus any newlines would be enclosed in quotes that would preserve them). I mentioned this to Ralf who will be bringing it up on the autoconf-patches mailing list... sounds promising anyway. so i give you the combined improved version:
#!/bin/sh -e
# escape any single quotes in an argument and then quote it
quote() 
  sed -e "s,','\\\\'',g; 1s,^,',; \$s,\$,',;" << EOF
$1
EOF
 
# save up a properly quoted/escaped version of "$@"
for arg in "$@"; do
  case "$arg" in
  # when a string contains a "'" we have to escape it
  *\'*)
    saved="$ saved:+$saved  $(quote "$arg")"
    ;;
  # otherwise just quote the variable
  *)
    saved="$ saved:+$saved  '$arg'"
    ;;
  esac
done
echo original command line, $# arguments: "$@"
while test -n "$*"; do echo arg: "$1"; shift; done
echo mangled command line, $# arguments: "$@"
# restore the cmdline
eval set -- "$saved"
echo restored command line, $# arguments: "$@"
while test -n "$*"; do echo arg: "$1"; shift; done

output from the same example cmdline:
rangda[/home/sean] ./better.sh "isn't very common" "to give args" "with trailing \\n's:
"
original command line, 3 arguments: isn't very common to give args with trailing \n's:
arg: isn't very common
arg: to give args
arg: with trailing \n's:
mangled command line, 0 arguments:
restored command line, 3 arguments: isn't very common to give args with trailing \n's:
arg: isn't very common
arg: to give args
arg: with trailing \n's:
rangda[/home/sean]                                                           :)

Adeodato Sim : The F hrer in IMDB

Following-up to the story of having all the movies I ve watched in a PostgreSQL database, I now have a tendency to run the following query after wathing one, to take a look at what other movies I ve watched in which some of the actors also participated:
  movies=> SELECT s.name, s.title
             FROM seen_cast s JOIN seen_cast s2 USING (person_id)
             WHERE s.movie_id != s2.movie_id AND s2.title = 'Sideways' 
             ORDER BY name ASC;
(Replacing Sideways with the movie I just watched, of course. Also, seen_cast is a view, that s why it looks as if the database is not normalized.) So, apart from informing me that I have seen Paul Giamatti in other five films in addition to Sideways, the above query returned the following rows:
       name             title        
  --------------+--------------------
   Adolf Hitler   Germania anno zero
   Adolf Hitler   The Good Shepherd
Apparently, Adolf Hitler has a page in IMDB, and is credited (generally by IMDB only, not by the film credits themselves) in every film that shows any archive footage of him. Which, as of today, are more than 400.

27 March 2009

David Welton: CodingHorror and Open Source Economics

Jeff Atwood writes about the economics of open source software: http://www.codinghorror.com/blog/archives/001247.html Well, he actually doesn't say as much, but that's what it's about. Someone really ought to do a book about this subject. I have some ideas - any takers? It's always tough to write about because there's so much material. It's a topic that is near and dear to my heart - almost 10 years ago, I joined the Free Software Business mailing list, as I got curious about the intersection of free software and business, and how the whole thing worked. I've learned a lot in the meantime, including a stint at Linuxcare, which was one of the Linux companies during the dot com boom, and am still fascinated by the subject. Before we get started, let's reiterate something:
  • It's clear that open source software works. There's just no doubt about it - it's been through ups and downs, the dot com boom and bust, and it will trundle happily (well, as happy as anyone is with this economy) through the recession too.
That point bears repeating. Everything else needs to take it into consideration. Could it work better? Differently? Perhaps, but it does work, so you need to understand it very well before you fiddle with it. Jeff writes:
for the average user? Heck, how realistic is this for the average programmer? Even if you're the type of macho, gung-ho programmer who can master an alien code base just to get some small pet feature or bugfix in -- do you honestly have the kind of time it would take to do that? Sometimes, when people say this:
The source code is freely available. If you feel so strongly about this bugfix/tweak/feature/plugin, why don't you code it yourself?
They're really saying this:
F**k you.
Which is of course a terrible analysis of open source, and its pros and cons. I wrote the following comment on Hacker News:
Heck, how realistic is this for the average programmer? ...
"Actually, I have submitted lots of (mostly small and inconsequential) bugs and minor features over the years, to projects such as the Linux Kernel, Tcl, Erlang, Ruby, Rails, Apache, and probably lots of others I'm forgetting right now. I actually enjoy it quite a bit, come to think of it - there's something cool about diving in and studying something new and different, and a thrill that comes with nailing the bug. Sometimes, the knowledge required to actually fix it is too much, and you have to settle for sending in a good bug report, but in other cases it's very much possible to hack up quick, useful things despite not knowing much about the system in question. And, no, I'm not really that great a programmer... I'd consider myself fairly average." And I know I'm not alone, as I see similar sorts of people in Debian and the Apache Software Foundation. Debian, infact, is all about those kinds of tweaks to improve the operating system as a whole. And this sort of exchange goes on all the time, as evidenced by the huge success of a large number of open source projects. Now, granted, most projects are really driven by one or a few really committed hackers, who do the lion's share of the work. However, that large network of people who contribute, test, document, and suggest is certainly important to any open source project. As long as we're looking at my anecdote, let's look at it in a bit more depth - most of those projects are developer tools or closely related. A cynic might note that Jeff comes from the world of Microsoft, where the core developer tools are provided by a company and 'giving back' is limited to payments and bug reports. But leaving that be, I think the enormous success of stuff like Ruby, Perl, PHP, Python, Tcl, Apache, Linux, GCC, and on and on and on indicates that, at least in terms of stuff that developers share between one another, open source works. However, there are problems, one of which is that he (or, in unfortunately rather rarer cases, she) who writes the code does not necessarily get the money. And without money, he or she who writes the code must dedicate themselves to some means to actually accumulate a bit of money in order to pay for living expenses. Most likely, this is time taken away from the open source project, which is a loss for all the people who benefit from it. Proprietary software solves this feedback loop in a neater way (generally) - companies that create good products get money for their efforts, which can then be reinvested in development. Jeff suggests that "sponsorship seems like a constructive way for people who are unable or unwilling to write code to affect the direction of a project.", which is ok as far as it goes, and more or less works here and there, but hopefully he's not thinking about trying it on a larger scale. This sort of thing has been tried before, and never really seems to take off. Part of the problem is that it can work on a person-to-person basis, but is almost impossible to standardize in a way that it makes sense to have some kind of clearinghouse that operates as a business. There are simply too many factors - what kind of open source project it is, what motivates the people behind it, how the transaction costs of defining a specification, monitoring the results, sending money and so on compare to the actual value of a fix... the list goes on. Ultimately, it's a nice idea, but it doesn't seem to work out too well in practice. Anyway, while it's a good thought on Jeff's part, at the end of the day, it is fairly evident that he doesn't have a lot of first hand knowledge of open source business and economics. That said, to some degree, open source took everyone by surprise, and, lacking the nice, simple feedback loop that proprietary software has, can be a lot more difficult to get a handle on. I find it quite pleasant that way - it attracts both hard-headed business types and social-experiment idealist visionaries; both huge corporations (IBM!) and a huge number of individuals; both people who have gotten quite wealthy from open source as well as most of the rest of us who manage to make ends meet but still love it. It's not something that's easy to pigeonhole, and as such it's hard enough to figure out the many ways in which it does work, let alone figure out how to improve on it. Stay tuned for further installments: Software Economics - Public Goods

David Welton: CodingHorror and Open Source Economics

Jeff Atwood writes about the economics of open source software: http://www.codinghorror.com/blog/archives/001247.html Well, he actually doesn't say as much, but that's what it's about. Someone really ought to do a book about this subject. I have some ideas - any takers? It's always tough to write about because there's so much material. It's a topic that is near and dear to my heart - almost 10 years ago, I joined the Free Software Business mailing list, as I got curious about the intersection of free software and business, and how the whole thing worked. I've learned a lot in the meantime, including a stint at Linuxcare, which was one of the Linux companies during the dot com boom, and am still fascinated by the subject. Before we get started, let's reiterate something:
  • It's clear that open source software works. There's just no doubt about it - it's been through ups and downs, the dot com boom and bust, and it will trundle happily (well, as happy as anyone is with this economy) through the recession too.
That point bears repeating. Everything else needs to take it into consideration. Could it work better? Differently? Perhaps, but it does work, so you need to understand it very well before you fiddle with it. Jeff writes:
for the average user? Heck, how realistic is this for the average programmer? Even if you're the type of macho, gung-ho programmer who can master an alien code base just to get some small pet feature or bugfix in -- do you honestly have the kind of time it would take to do that? Sometimes, when people say this:
The source code is freely available. If you feel so strongly about this bugfix/tweak/feature/plugin, why don't you code it yourself?
They're really saying this:
F**k you.
Which is of course a terrible analysis of open source, and its pros and cons. I wrote the following comment on Hacker News:
Heck, how realistic is this for the average programmer? ...
"Actually, I have submitted lots of (mostly small and inconsequential) bugs and minor features over the years, to projects such as the Linux Kernel, Tcl, Erlang, Ruby, Rails, Apache, and probably lots of others I'm forgetting right now. I actually enjoy it quite a bit, come to think of it - there's something cool about diving in and studying something new and different, and a thrill that comes with nailing the bug. Sometimes, the knowledge required to actually fix it is too much, and you have to settle for sending in a good bug report, but in other cases it's very much possible to hack up quick, useful things despite not knowing much about the system in question. And, no, I'm not really that great a programmer... I'd consider myself fairly average." And I know I'm not alone, as I see similar sorts of people in Debian and the Apache Software Foundation. Debian, infact, is all about those kinds of tweaks to improve the operating system as a whole. And this sort of exchange goes on all the time, as evidenced by the huge success of a large number of open source projects. Now, granted, most projects are really driven by one or a few really committed hackers, who do the lion's share of the work. However, that large network of people who contribute, test, document, and suggest is certainly important to any open source project. As long as we're looking at my anecdote, let's look at it in a bit more depth - most of those projects are developer tools or closely related. A cynic might note that Jeff comes from the world of Microsoft, where the core developer tools are provided by a company and 'giving back' is limited to payments and bug reports. But leaving that be, I think the enormous success of stuff like Ruby, Perl, PHP, Python, Tcl, Apache, Linux, GCC, and on and on and on indicates that, at least in terms of stuff that developers share between one another, open source works. However, there are problems, one of which is that he (or, in unfortunately rather rarer cases, she) who writes the code does not necessarily get the money. And without money, he or she who writes the code must dedicate themselves to some means to actually accumulate a bit of money in order to pay for living expenses. Most likely, this is time taken away from the open source project, which is a loss for all the people who benefit from it. Proprietary software solves this feedback loop in a neater way (generally) - companies that create good products get money for their efforts, which can then be reinvested in development. Jeff suggests that "sponsorship seems like a constructive way for people who are unable or unwilling to write code to affect the direction of a project.", which is ok as far as it goes, and more or less works here and there, but hopefully he's not thinking about trying it on a larger scale. This sort of thing has been tried before, and never really seems to take off. Part of the problem is that it can work on a person-to-person basis, but is almost impossible to standardize in a way that it makes sense to have some kind of clearinghouse that operates as a business. There are simply too many factors - what kind of open source project it is, what motivates the people behind it, how the transaction costs of defining a specification, monitoring the results, sending money and so on compare to the actual value of a fix... the list goes on. Ultimately, it's a nice idea, but it doesn't seem to work out too well in practice. Anyway, while it's a good thought on Jeff's part, at the end of the day, it is fairly evident that he doesn't have a lot of first hand knowledge of open source business and economics. That said, to some degree, open source took everyone by surprise, and, lacking the nice, simple feedback loop that proprietary software has, can be a lot more difficult to get a handle on. I find it quite pleasant that way - it attracts both hard-headed business types and social-experiment idealist visionaries; both huge corporations (IBM!) and a huge number of individuals; both people who have gotten quite wealthy from open source as well as most of the rest of us who manage to make ends meet but still love it. It's not something that's easy to pigeonhole, and as such it's hard enough to figure out the many ways in which it does work, let alone figure out how to improve on it. Stay tuned for further installments: Software Economics - Public Goods

David Welton: Software Economics - Public Goods

To look at the economics of open source software, you have to start with the economics of software as a category, both proprietary and open source. Note: it should be perfectly obvious to everyone that open source software works, and works quite well for some things. This article is more about understanding the how and they why. Just to avoid any confusion... If you look at software as a product, it has a couple of defining characteristics: What do you get when you combine something that's "non excludable" and "non rivalrous"? In economics, these are known as "public goods": http://en.wikipedia.org/wiki/Public_good - read the whole thing if you have a few minutes, it's what I'll be talking about. As per the article, "The Problem" as it were, is the free rider problem. Let's stop and look at a contrived example: Bob is an ex-dentist and really knows what it would take to write a great program for managing a dentist's office (or pick your own example... it's contrived, don't quibble about the details). He's also a good programmer, with the skills necessary to implement the program. He talks to a few of his dentist friends, and they're enthusiastic about the whole thing. Let's look at a few scenarios of how things might play out: Bob values his time at 10 dollars an hour, and it will take him 100 hours to complete the project, meaning the whole project would cost about 1000 dollars. One way things could work is that he could find a dentist friend who needs this program, and bill him 1000$ for his time and services, bang out the code, and release it as open source software. Bob gets his money, the dentist gets his code... and all the other dentists in town get it for free, which is bound to make the dentist who shelled out the money a bit irritated, given that he's a human, not a homo-economus, and doesn't like paying and see others get the same thing for free. Let's introduce another variable: the software is only really worth 200$ to the dentist - any more than that and yeah, it's helpful, but not worth that much money (And please stay with me, I know it's contrived, but keep in mind that the situation *is* possible, even if the numbers might be different). That means that if Bob wants 1000$ for his time, but the dentist will only pay 200$, then no transaction will occur! The software won't get written, Bob won't get any money, and the dentist will be worse off because he won't have the nice program, and neither will his dentist colleagues. One way things might still work is if the dentist rounds up some friends and they all contribute a bit, however, that's kind of a pain in the neck for everyone concerned - the dentist doesn't want to get into the software business and spend a bunch of time trying to chase his friends for money, and even if 10 dentists contribute, the ideal position to be in his the 11th guy, who pays nothing, and still gets the software for free. So it's in everyone's interests to not return the dentist's calls in the hope of being the 11th guy. Also, none of them gets the software until they're all on board. So, how to resolve this problem? The wikipedia page talks about introducing an exclusion mechanism, which in this case would be a copyright system so that Bob can sell a copy of his software for 100$ and be sure that it won't be open to everyone to use. This way, the dentists get the software for 100$ less than they value it at (200$), and as soon as Bob signs up an 11th dentist, he's making more than his time/materials costs, and is therefore coming out ahead! The dentists all know that they're paying the same price, so have no reason to grumble about free riders. In short, everyone is pretty happy. Of course, in the real world, things don't play out like that all the time. There are problems with the simplified example above, some of which are very evident: the cost of enforcing copyright laws, in an age when copying is extremely easy, is very high and potentially very intrusive. However, "intellectual property" is, by and large, a compromise solution that has had positive aspects, especially when you ignore the outliers like Microsoft, and look at all kinds of niche software vendors, who make a nice living, create good products, and have happy, satisfied customers. If you look at the other solutions to the free rider problem listed on the wikipedia page, we can see how they might be applied to the world of software. None of these solutions is perfect, and all, including exclusion in the form of intellectual property laws, or simply doing nothing at all, have problems of their own. Due to the very nature of software, this is somewhat inevitable, and the best we can do is come up with good compromises, and continue to reevaluate them as technology and society change (this is something that needs to happen with IP laws, for instance).

Matthew Garrett: Reducing disk use

UNIX filesystems generally store three pieces of timing information about files - ctime (when the file was changed in any way), mtime (when the file contents, as opposed to its metadata, was last changed) and atime (when the file was last accessed by any process). This is a usefully flexible system, but the semantics of atime can be troublesome. atime must be updated every time a file is read, causing a read operation to instead become a read/write operation. This results in a surprising amount of io being generated in normal filesystem use, slowing the more relevant io and causing disks to spin up due to atime updates being required even if the file was read out of cache. It also results in a lot of unnecessary activity on flash media which may reduce their lifetime.

One option is to disable atime updates entirely. The problem with this approach is that certain applications depend on atime. This is especially common in mail clients which compare atime to mtime in order to determine whether a mailbox has been read since it was last modified. So, unfortunately, disabling atime entirely is impractical as a default. Back in 2006, Valerie Aurora submitted a patch that worked around this issue. The new relatime option meant that atime would only be updated if it would otherwise be older than ctime or mtime. Mail clients became happy and the world rejoiced.

Unfortunately, it turned out that there was one other common case of atime being used. Applications like tmpwatch monitor files in /tmp and delete them if they appear unused. In this case, "unused" means "has an atime older than a certain date". Since merely reading files doesn't update the ctime or mtime, relatime wouldn't cause the atime on these files to be updated and tmpwatch would happily delete them - even if users were reading them on a daily basis.

Ingo Molnar submitted a patch to add a further heuristic to the relatime behaviour. With it, the atime of a file will be updated if it's older than mtime, older than ctime or (and this is the important one) more than 24 hours in the past. This deals with the tmpwatch case nicely, while still providing a significant reduction in the quantity of atime updates.

Fedora shipped this patch for several releases, and Ubuntu have used it by default since 8.04. Unfortunately there were some concerns over certain aspects of its behaviour (in respect to its interface as opposed to the relatime functionality itself) and it never got merged. I pushed a trimmed down version that purely implements the change to the relatime behaviour, and earlier today Linus merged it and a further patch that makes relatime the default behaviour on Linux.

Most users won't notice this change in behaviour at all, other than as a small improvement in io performance and a reduction in the number of drive spinups. For users that do have issues, a new strictatime mount option has been added - using this will require an updated mount command, but it's a trivial patch. I'd be surprised if there are any real world use cases that are negatively affected by this, especially since it's been default behaviour in several distributions for a while, but there's always the potential that someone will be tripped up by it. We'll see.

26 March 2009

Holger Levsen: Ada Lovelace naming scheme

Two days ago I read about the Ada Lovelace day, which was an event were men and women were asked to blog about the women in technology which made an impact on them, in an effort to present girls (and women) all over the world with female role models in technology.

In similar though more personal motivation I choose my machine naming scheme at home two years ago: my (virtual) machines at home are named after women who made a difference, mostly in IT, but also some in more general science. Even though this list is not 100% in the spirit of pointing out the one woman who made a personal difference to one self, I thought I share my list, as I found it very interesting to learn about these great women myself.

goldberg is named after Adele Goldberg, who pioneered the smalltalk programming language in the 70s. According to a sympathetic annecdote she refused to show the system to Steve Jobs and only did so after her superiors ordered her to do so. Thats were Apple got many ideas which made them famous for being innovative...

goldwasser is named after Shafi Goldwasser, who is a RSA professor at MIT, doing research in complexity theory, cryptography and computational number theory. She is the co-inventor of zero-knowledge proofs.

lovelace is named after Ada Lovelace, and is widely recognized as the first programmer, who wrote a program (something that manipulated symbols according to rules) for a machine which has not even been build at that time. Of course, some men on this planet object to her being the first programmer, as work by women is always insignificant... </irony or maybe sarcasm>

bunten is named afer Danielle Bunten Berry, who was a game designer and programmer best known for the games M.U.L.E. and Seven Cities of Gold.

holberton is named after Betty Holberton, who was one of the six original programmers of ENIAC, the first general-purpose electronic digital computer, during and after World War II.

hopper is named after Grace Hopper who wrote the first compiler ever, the A compiler. She was also heavily involved in creating COBOL and standardizing COBOL and FORTRAN.

koller is named after Daphne Koller who does AI research at MIT and who seems to behind the idea of bayesian machine learning, which you might know from your spam filters... (I can only state her involvement in bayesian learning so vaguely as I'm basically clueless about the topic :)

leveson is named after Nancy Leveson who is also an MIT professior and is an expert in systems and software safety.

Last but not least I like to remark that none of the women above is even nearly covered with what I wrote here or what is written in wikipedia, which I used as my main source for this blog post. It's simple impossible to sum up one life that brief and be close to accurate.

And then there is another group of women whose achievements are noteworthy as well and who choose to present themselves on that webpage to show others that Debian is not a mens club. Thank you (a lot) for that and for your other contributions to Debian! If you want to learn more about these giants, I recommend this talk. Guess who's achievements I admire most in this group ;-)

Somewhat unrelated I wish to shout cheers to Yvonne, who tought me many things and who died six years ago today. You will also not be forgotten, though the pain is mostly gone now (well, it's gone, except for days like today). Thank you, among other stuff, for teaching me to celebrate and cheer every day, even if not everything is cheerful. Wenn die Nacht am tiefsten, ist der Tag am n chsten! :-)

Stefano Zacchiroli: enabling comments with style

Enabling ikiwiki comment with a bit of CSS I've finally spent some time to understand why the amazing comments ikiwiki plugin was not working for me. Obviously the reason were page templates which I had customized and hence were missing the needed template variables that expand to comments links and content. That made me wonder about keeping my ~/.ikiwiki/templates directory as a Git repository (which actually already is, even though for different reasons) which tracks Joey's ikiwiki repo for better merging future changes to upstream templates. Looks like a good idea, but I didn't have the time to testdrive it yet. Still, I've updated my ikiwiki CSS (also available from ikiwiki's CSS market) for fancier rendering of comments and I'm quite happy about the result. So, to the nice guy who once wrote the following in the discussion page of an old blog post of mine: p.s. this is the dumbest comment method I've ever used You were totally right!, the issue is fixed now, big up to Simon McVittie for the plugin. /me now only dreams of an automatic conversions of old discussion pages to comment series, knowing that it is impossible in the general case due to the lack of structure :-( ...

Sven Mueller: Apache SSL oddity

If you ever have the problem described below, you have to realize that Apache HTTPD is selecting the SSL certificate (and/or key) not based on you VirtualHost definition, but based on the ServerName in the current context. This means that you have two virtual hosts with the same ServerName, but on two different IPs, and are wondering why the wrong key (and/or certificate) is used, it might be because of this. Assume we have the following (simplified) configuration:
<VirtualHost a.b.c.d:443>
ServerName a.b.c.d
SSLCertificateFile /etc/ssl/certs/a.b.c.d_443.pem
SSLCertificateKeyFile /etc/ssl/certs/a.b.c.d_443-key.pem
</VirtualHost>
<VirtualHost a.b.c.d:444>
ServerName a.b.c.d
SSLCertificateFile /etc/ssl/certs/a.b.c.d_444.pem
SSLCertificateKeyFile /etc/ssl/certs/a.b.c.d_444-key.pem
</VirtualHost>

(note: This happened to me when wanting to open a SSL server for subversion on a host that also had Apache run as a reverse/application proxy for some specific application that managed its own CA) Both VirtualHosts will use the very same certificate and key (as far as I was able to tell, but at least the same key), in my case that of the specific application (which was(is?), in its way, broken regarding SSL). This might as well be a bug or a design decision (AKA feature), I don t know. However the workaround that worked for us was to change one of the server names to something else (since it seems it is unused anyway, unless the apache decides to generate a self-refering URL, which it certainly won t do for the reverse/application proxy part). Update: Fixed formatting on configuration sample to make it clear this is not about namebased virtual hosts.

25 March 2009

Marco d'Itri: Running kvm without root privileges

This is my recipe for easily running kvm as an unprivileged user without using sudo and custom network configuration scripts. /etc/network/interfaces:
iface kvmnet inet static
        address         10.5.0.1
        netmask         255.255.255.0
        broadcast       10.5.0.255
        pre-up          vde_tunctl -u md -t $IFACE
        post-down       vde_tunctl -d $IFACE
        up              echo 1 > /proc/sys/net/ipv4/ip_forward
        up              iptables --table nat --append POSTROUTING --jump MASQUERADE --source $IF_ADDRESS/$IF_NETMASK
        down            iptables --table nat --delete POSTROUTING --jump MASQUERADE --source $IF_ADDRESS/$IF_NETMASK
~/bin/kvm:
#!/bin/sh -e
iface='kvmnet'
macaddr='DE:AD:42:00:00:01'
model='virtio'
exec kvm \
    -net nic,vlan=0,macaddr=$macaddr,model=$model \
    -net tap,vlan=0,ifname=$iface,script=no,downscript=no \
    "$@"
Configuring a PV network interface is optional, but it is more efficient and if the guest uses udev and a modern kernel it will just work. Bonus tip: use "-vnc :0 -usbdevice tablet -serial telnet:127.0.0.1:4444,server,nowait" for a headless guest which if needed will still have proper consoles.

Neil Williams: cross-building is still hard

OK, it's gone 2am and I'm still battling with what was meant to be a simple task - check the latest (unreleased) changes in tslib on my touchscreen device.

First problem was that I forgot that the balloon is running Lenny, so I suddenly needed the armel libc6 from Sid. OK, that's not bad, a bit of a pain to have to install it but less pain than creating a Lenny chroot to cross-build tslib - marginally.

Second problem was more awkward - turns out that emdebian-tools 1.6.1 has broken the more fragile cross-build support which carefully wraps broken libtool installations. I was intending to work out how to fix each one of those before Squeeze anyway. The fix for the workaround was relatively straightforward and will be in 1.6.2, due for release once I've had some sleep.

Third problem was something much older - turns out that this broken libtool wrapper had only been able to work for the default cross-building architecture since about emdebian-tools 1.4 or 1.5 and if asked to build for a different arch, it got the wrong symbols. Hmm.

The diffs for each of these changes appear simple but, as ever, identifying the cause of the bug was harder. (In this case, sometimes the error showed up as "C++ compiler unable to create executables" - thanks autotools, that's really handy to know; other times it showed up as ERROR, symbols from ./.libs/libfoo are ABI 4 but symbols from libc are ABI version 0 - still cryptic but at least it is entirely accurate.)

Still, I've now been able to test tslib 1.0-7 and it works just fine and I've got an extra upload of emdebian-tools 1.6.2 to make tomorrow. Ah well.

24 March 2009

John Goerzen: Review: David Copperfield

I finished reading David Copperfield on the Kindle a few days ago. This is a review of the novel, not the Kindle. I m not an English major, and so I m not going to pretend to be one. I m not going to discuss what themes the book touches on, what category it fits in, or generally dissect it to the point where it s more monotonous than fun. I read the book because I wanted to, not because I had to write a paper about it. I must say, first of all, that this has got to be one of the best books I ve ever read. The vivid descriptions of the characters were just fun to read. One particularly meek man was described like this: He was so extremely conciliatory in his manner that he seemed to apologize to the very newspaper for taking the liberty of reading it. Some of the scenes in the novel are amazingly vivid and memorable. The hilarious and tense scene towards the end where one of the main villains is taken down was one, and of course just about every scene involving David s aunt is too. Dickens is a master of suspense. He does it through subtle premonitions in the book. You might not even really notice them as you re reading. But it sure had an effect on me: I had trouble putting the book down, and stayed up later than I should have on more than one night to keep reading another chapter or three. Like any good book, this one left me to think even after I was done reading it, and left me wanting to read it again. Right now. There are some practical downsides to it, though. It was written in the 1850s, and some of the vocabulary and British legal, business, and monetary discussions are strange to a modern American audience. Nevertheless, with the exception of the particularly verbose Mr. Micawber, you can probably make it through without a dictionary, though one will be handy. I read it on the Kindle, which integrates a dictionary and makes it very easy to look up words. I learned that a nosegay is a bouquet of showy flowers. And that Mr. Micawber was fond of using words obsolete since the 17th century, according to the Kindle. If you remember that pecuniary emoluments refers to a salary, you ll be doing OK. The other thing that occasionally bugged me was that the narrator (David) would comment on some sort of gesture, or comment that wasn t very direct, and then say something like, But she didn t need to be more explicit, because I understood the meaning perfectly. Well, sometimes I didn t. Though I usually figured it out after a bit. I was never quite sure if Dickens was being intentionally needling to the reader, or if an 1850s British reader would have figured out the meaning perfectly well. But that was part of the fun of it, I think.

Jordi Mallach: Stepping down as the GNOME Catalan translations coordinator

As of this morning, Damned Lies finally reflects what has been the de-facto reality for at least four major GNOME releases. I started to invest a lot of time on translating GNOME to Catalan in the middle of the long 1.5 journey towards GNOME 2.0. That was a long time ago, and somehow was the way I ended being abduced by Softcatal to eventually work with them on the localisation of some other projects. However, I've been watching how my free time and motivation has been slowly shrinking, until the point I was no longer doing some of the stuff I was expected to do, or was doing it badly and late. Luckily, Softcatal 's GNOME team, a model for our organisation, has been able to smoothly replace heavily contributing members with a constant stream of new blood. In my case, I first stopped having that many modules assigned, then focused on coordination and finally stopped doing even that. Gil Forcada has filled the gap perfectly and has been the main lead of effort for a pair of years. Passing the baton was long overdue; I think GNOME is lucky to count on Gil's amazing drive and motivation. Gil, congrats on earning yet another marronet! ;)

23 March 2009

Sami Haahtinen: Configuring split DNS

Since I enabled comments in this blog, I finally needed to configure a split DNS for my network. There are various reasons why one needs a split DNS and as it's usually pointed out, the reasons are usually non-technical. In my case the reasons are technical: I have a NAT in my local network that allows me to host this website locally. What causes problems is that the domain name ressukka.net points to the external IP address and that doesn't work from the inside. So split DNS it is. There are various ways of building a split DNS, one can use the views feature in bind9 or you can set up 2 separate DNS servers that provide different information (and redirect your local resolver to use the internal server). The latter is more secure if the internal zone is sensitive. I decided to use a hybrid solution. I already knew that PowerDNS Recursor was capable of serving authoritative zones (think pre-cached) so I decided to leverage on that. Setting this up turned out to be simpler than I expected. First I made a copy of the existing zone and edited it to fit my needs. I changed the IP address of ressukka.net to point to the IP address on the local network. I also adjusted some other entries that pointed to the local network. Next I modified bind to listen on the external IP address. This can be accomplished by adding a listen-on 1.2.3.4; ; to the options in the configuration. I also disabled the resolver by adding recursion no;, this forces the bind to work as authoritative only. Then I installed the PowerDNS Recursor (pdns-recursor package in debian) and configured it to listen on the internal address only (local-address=10.0.0.1) and added the pre-cached zone to the configuration with auth-zones=ressukka.net=/path/to/internal-zone Now, after restarting both daemons, I had a working split DNS with minimal configuration. I was also able to change the external DNS to authoritative only mode, which is a good idea in any case.

Daniel Burrows: Sometimes, tools are useful.

I generally try not to get caught up in writing auxiliary tools for my free software projects. My experience in the past has been that given how little spare time I have each week, a tool that takes more than an hour or two to write quickly ends up becoming a time sink that prevents me from making progress on the main project. But occasionally it's worth the effort. screenshot I spent the last couple weeks putting together a tool to extract information about a dependency search from a log trace of aptitude's resolver. The search is displayed in a log interface that shows both the steps of the search in order, and the tree structure of the search. It took a long time to write, but after working with it for just a few hours, I was able to track down the bugs in the resolver modifications I was trying to write; without a tool like this, it would have taken a lot of painful grepping through log files to get all the information I needed. In other news, the resolver now stores the choices it has made as a single set of choices, instead of using two different sets for the two types of choices it can make. There's no user-visible change (although I did fix another performance problem that affected safe-upgrade), but this will pave the way for adding more types of choices in a cleaner way: in particular, the resolver should get explicit support for package replacements in the near future. The code for this is available in the tools directory of aptitude's experimental Mercurial repository, http://hg.debian.org/hg/aptitude/post-lenny. When I get around to merging it this with the main repository, it will also be available at http://hg.debian.org/hg/aptitude/head.
A brief digression regarding the implementation language The tool is implemented in Haskell, a decision that I'm not sure was the right one. Haskell's data model was perfect for this task: having algebraic datatypes allows me to precisely represent cases where parts of the structure can't be inferred from the log file, or can only be inferred incompletely. And of course there are the other well-known benefits of Haskell (pure, a strong type system, good support for higher-order programming, etc). But all those benefits were wiped out by the time I spent fighting the runtime. The thing is, Haskell is a lazy language. In theory, this means that it never evaluates anything you don't need and that you can work with arbitrarily large (even infinite) values and only allocate the parts you actually use. In practice, this means that you have to pay excruciatingly close attention to every single expression you write, or as soon as you run your program on non-trivial input, it will either run very slowly, eat all your memory, die with a stack overflow, or all of the above. Oh, and did I mention that laziness means that you can't have a runtime debugger, that trace statements are mostly-useless, because your program doesn't run in any predictable order, and that it's impossible to get backtraces when there's a fatal error? I think if I were doing this over again, I'd use O'Caml. I'm less familiar with it and its syntax is clunkier, but I have used it productively in the past and it provides the core stuff I wanted from Haskell (functional programming with algebraic datatypes). If someone could make a language that was pure, had Haskell's nice syntax and type features, but was also strict, and that had decent tool support and development resources, I think I'd be in nerd heaven.

Julian Andres Klode: Change has come to python-apt - aka 0.7.9 released


With version 0.7.9 change has come to python-apt, and I will tell you now what the new stuff is about: 1. Introduction of the documentation As you may know, python-apt 0.7.8 s only documentation was available from the source code and to some parts from the docstrings. This changed in python-apt 0.7.9~exp2, when I introduced a complete documentation written in reStructuredText, and generated using Sphinx. This documentation has even improved in the final 0.7.9 release, and I will push it to the online mirror at http://apt.alioth.debian.org/python-apt-doc/ soon (it currently still has 0.7.9~exp2). 2. Graphical Progress The module apt.progress.gtk2 provides widgets for Python GTK+ coders, allowing them to easily build graphical applications interacting directly with apt. I am still looking for someone to write a apt.progress.qt4 module, if no one wants to do this, I will do it myself. 3. Enhanced apt.debfile module A lot of code has been merged from gdebi, now allowing to do a large set of operations on local .deb files. For example, you can now install these packages. This module is also the first module to follow PEP8 naming conventions (lowercase_with_underscores), the others will be adapted at a later point. 4. Complete code cleanup I have worked on cleaning up the whole code, fixing white space problems, and more. Ben Finney was very helpful, I merged some of his patches for whitespace. I also fixed several other problems related to PEP8 conformance. Previously the PEP8 checking tool (see documentation) found about possible 964 issues, this has been reduced to 7 (-957), but these 7 issues are simply the usage of has_key() at locations where it can t be avoided [interfacing with TagSection objects], i.e. they are false positives. 5. Introduction of apt.package.Version In python-apt 0.7.9 I am introducing a new class apt.package.Version. This class provides access to various attributes of a version, like the record, the version number, the origin and even allows you to fetch the corresponding source code. But one of the best things about it is that it is sortable, allowing you to write:
print 'Highest bash version:', max(apt.Cache()['bash'].versions)
This makes it very clear that you can build really cool applications with it. The various candidate*() and installed*() methods of apt.Package objects haven been deprecated, and you should use the Version() objects available via Package.installed or Package.candidate properties. Summary All in all, the python-apt 0.7.9 is a big step forward. It is one of the largest releases in the history of python-apt and provides various new features to make developing easier. With all the code cleanups, and the documentation, you should be able to find everything you need; if not please report a bug. When we are asked Can we do this with python-apt? , we can finally say: Yes we can! Posted in Debian

Francois Marier: Using Wikia Search in Firefox and vimperator

Wikia search is a new project from Jimmy Wales, founder of Wikipedia. As discussed on the second autonomo.us podcast, his objective is to create a Free Network Service for Web search: an alternative search engine compatible with the Franklin Street Statement.

To add Wikia Search to your Firefox search bar:
If you use vimperator to reduce the amount of mousing involved in your daily web browsing, put this in your ~/.vimperatorrc to make Wikia your default search plugin:
:set defsearch=wikia
Please let me know if this stops working, I'll update the search extension.

22 March 2009

Neil Williams: multistrap - debootstrap for multiple repositories

debootstrap has only ever been designed to work with a single repository and the current code includes a comment:

# XXX: I can't think how to deal well with dependency resolution and
# lots of Packages files. -- aj 2005/06/12

The main reason for that, AFAICT, is that debootstrap has a perfectly valid goal of trying to work without needing dpkg or apt themselves, doing everything in the chroot. Theoretically, anything that can run POSIX shell should be able to run debootstrap.

Emdebian has a need for a different script that can assume that both dpkg and apt are fully functional already and behave as normal Debian packages which opens the door to supporting as many repositories as apt itself can handle and, until now, that support was --foreign only. i.e. it was explicitly for cross-debootstrap (armel, etc.) for use with Emdebian Grip and actually used debootstrap for some of the work. That code has been simplified and there is no need for debootstrap now in multistrap. (multistrap itself is written in perl.)

I recognise that native support could be useful so I've been working on a fix for the native support - partially achieved by dropping the need to use debootstrap at all and porting the existing fix for #518719 from the emdebian-rootfs package (shell) into perl.

In effect, multistrap allows you to collapse the following stages into one:

  1. $ sudo debootstrap $suite $dir $mirror

  2. $ sudo chroot $dir

  3. # nano /etc/apt/sources.list

  4. # apt-get update

  5. # apt-get install $package_list

  6. # apt-get clean

  7. $ sudo $my_config_script


That much is architecture-neutral (due to the support for working around #518719 which allows packages to be unpacked without running the configure scripts) - what I've added for native support is that sudo chroot $dir dpkg --configure -a can be automatically run before the config_script. In the typical Emdebian case, the configure stage would happen immediately after unpacking the multistrap filesystem onto the device (certainly before first boot).

That gives you a single method for creating a complete root filesystem with everything you need installed, everything configured and the config_script hook allows you to put in things like kernels and kernel modules etc. too - all ready for tar -czf and unpacking onto your device. Instead of a plain debootstrap, multistrap can give you a complete filesystem with all the packages you'd only install as soon as debootstrap had finished.

In addition, multistrap allows apt to decide which versions to install from the collection of Packages files so you can implement fallbacks and other permutations of mirrors and packages. Emdebian is using this functionality to mix private packages with official packages from Emdebian Grip 1.0 and ensure that not only are all dependencies met but that each dependency has the chance to be the latest version without needing any configuration or manual parsing. Just specify the repositories and which packages you need at the top level from each one. apt does the rest.

So, the question:

Does this deserve to be a separate source package ? For now, it's heading into Emdebian as part of the emdebian-rootfs package, version 1.6.1.

The advantages will be that multistrap itself doesn't need to be tied to the still rapidly developing emdebian-tools source package and doesn't need to carry the quite large changelog from emdebian-tools (which is just going to keep on growing).
 3014 2009-03-17 20:54 ./usr/share/doc/multistrap/README
2265 2009-02-18 18:32 ./usr/share/doc/multistrap/TODO
1536 2008-08-20 14:25 ./usr/share/doc/multistrap/copyright
21067 2009-03-17 21:58 ./usr/share/doc/multistrap/changelog.gz
1233 2008-10-28 13:13 ./usr/share/doc/multistrap/NEWS.Debian.gz


Disadvantages, it's very small (a lot of the current content is actually POD documentation) and probably doesn't warrant a source package all alone.

17774 2009-03-22 13:41 ./usr/sbin/multistrap


Details:
 Package: multistrap
Source: emdebian-tools
Version: 1.6.1
Architecture: all
Maintainer: Neil Williams
Installed-Size: 120
Depends: perl, apt, debhelper (>= 5),
libconfig-auto-perl, libparse-debian-packages-perl
Replaces: emdebian-rootfs (<= 1.6.0)
Section: admin
Priority: extra
Homepage: http://www.emdebian.org/
Description: multiple repository bootstrap based on apt
debootstrap replacement with multiple repository support
using apt to handle all dependency issues and conflicts.
.
Includes support for foreign architecture multistraps that need
only minimal configuration on the final device. Supports cleaning
up the generated bootstrap filesystem to remove downloaded packages
and hooks to modify the files in the bootstrap filesystem after
the packages have been unpacked but before being configured.


Is there a better home? devscripts? Ideas? (It would need to be a package I can join as Uploader: and I'd prefer if that package used subversion.)

21 March 2009

Mike Hommey: Orphaning packages

Since I am expected to spend more than half my packaging time updating the debian/copyright file, I am hereby orphaning nspr, nss, iceape and xulrunner. I am also stopping work on webkit and iceweasel, but they don t end up in the orphan state since they are comaintained. Good luck to my fellow developpers. (And sorry, sincerely) Update: As I do realize writing while being pissed doesn t help making the motives right, and apparently, some people have seen this as an extortion, let me make things clearer: I was starting to work on xulrunner 1.9.1 when the discussion about the copyright files came up. It will require a significant amount of time, and while Noah Slater s opinion alone wouldn t really have carried me that far, despite me saying so because I got pissed by his words, 2 messages from J rg Jaspert (the only ones he posted so far in the thread, by the way) did make it clear that my work on xulrunner 1.9.1 was going to be a waste of time, which I already lack to properly handle the bugs in my maintained packages, let alone keeping the copyright file up-to-date. As I am obviously unable to handle the amount of work required to maintain big packages, as drawing new blood in the mozilla team has always failed so far, I just prefer to stop than to over-overflow. Call it extortion to get people in the mozilla team if you want, I m fine with the notion. I ve been thinking to stop working on big packages for nearly a year already, but never mentionned it but to a few people in a few occasions. I couldn t resolve myself to do it, though I did reduce the amount of time I spend on these packages (I was overflown, a year ago). I just found an excuse to actually do it. I must say I feel awkward now, and I still don t know if I will be able to keep this resolution. As for the new copyright file format, with full licensing information and copyright holders list, I *did* try, on a significantly smaller piece of software than the mozilla packages, namely on Webkit, which is not really small either but still 6 times less files than xulrunner. I must say I hate to have to list copyright holders and file names with a passion, and the amount of time it takes. It is the main reason why there wasn t more uploads of WebKit svn snapshots in the archive Last but not least, thanks for the nice comments.

Sean Finney: saving, escaping, and restoring positional parameters in a POSIX shell

and now for something completely different... sometimes in a shell script you need to do stuff that modifies the positional parameters ($@, $1, $2, etc) from the cmdline, usually via the builtin shift, maybe from using getopt(1) or similar. for example, it's quite common to see the following in a shell script that parses cmdline flags with getopt
# parse cmdline options with getopt
TEMP= getopt -o h123 -n $0 -- "$@" 
if [ $? != 0 ] ; then usage >&2 ; exit 1 ; fi
# overwrites $@ with getopt-generated/sanitized arguments
eval set -- "$TEMP"
# destructively iterates across $@
while true; do
  case "$1" in
        -h)
          # handle option h
          shift
        ;;
        ...
        --)
          shift
          break
        ;;
      esac
...
done

but sometimes you want to be able to restore the original positional parameters later, or otherwise quote them in a manner where they can be passed along to another script/function, log them, embed them in a string/file, etc. you can't always just stash them in another variable to use that later, because "$@" has a special meaning to the shell that can't otherwise be reproduced with another variable. one way you could try to store/save them would be to save the original copy of $@ and use eval set -- later on to change it back to the saved variable. using a slightly simplified version of the above:
#!/bin/sh
# save the cmdline
saved="$@"
echo original command line, $# arguments: "$@"
while test -n "$*"; do echo arg: "$1"; shift; done
echo mangled command line, $# arguments: "$@"
# restore the cmdline
eval set -- "$saved"
echo restored command line, $# arguments: "$@"
while test -n "$*"; do echo arg: "$1"; shift; done

and at first glance, it seems to work:
rangda[/home/sean] ./bad.sh one two three
original command line, 3 arguments: one two three
arg: one
arg: two
arg: three
mangled command line, 0 arguments:
restored command line, 3 arguments: one two three
arg: one
arg: two
arg: three

but you may run into problems when some parameters have spaces, or quotes:
rangda[/home/sean] ./bad.sh "1: one" "2: two"
original command line, 2 arguments: 1: one 2: two
arg: 1: one
arg: 2: two
mangled command line, 0 arguments:
restored command line, 4 arguments: 1: one 2: two
arg: 1:
arg: one
arg: 2:
arg: two
rangda[/home/sean] ./bad.sh "one's one"
original command line, 1 arguments: one's one
arg: one's one
mangled command line, 0 arguments:
./bad.sh: eval: line 12: unexpected EOF while looking for matching  ''

or worse, shell meta characters:
rangda[/home/sean] ./bad.sh 'i wonder what  pwd  would do'
original command line, 1 arguments: i wonder what  pwd  would do
arg: i wonder what  pwd  would do
mangled command line, 0 arguments:
restored command line, 6 arguments: i wonder what /home/sean would do
arg: i
arg: wonder
arg: what
arg: /home/sean
arg: would
arg: do

the solution is to build up the saved value a bit more manually, single-quoting each parameter (which should prevent any unintended side effects of space/quotes/metacharacters), and escaping existing single quotes inside of parameters.
# escape any single quotes in an argument
quote() 
  echo "$1"   sed -e "s,','\\\\'',g"
 
# save up a properly quoted/escaped version of "$@"
for arg in "$@"; do
  saved="$ saved:+$saved  '$(quote "$arg")'"
done
echo original command line, $# arguments: "$@"
while test -n "$*"; do echo arg: "$1"; shift; done
echo mangled command line, $# arguments: "$@"
# restore the cmdline
eval set -- "$saved"
echo restored command line, $# arguments: "$@"
while test -n "$*"; do echo arg: "$1"; shift; done

let's see how it behaves:
rangda[/home/sean] ./good.sh "1 one" "2: two"
original command line, 2 arguments: 1 one 2: two
arg: 1 one
arg: 2: two
mangled command line, 0 arguments:
restored command line, 2 arguments: 1 one 2: two
arg: 1 one
arg: 2: two
rangda[/home/sean] ./good.sh "one's one"
original command line, 1 arguments: one's one
arg: one's one
mangled command line, 0 arguments:
restored command line, 1 arguments: one's one
arg: one's one
rangda[/home/sean] ./good.sh 'i wonder what  pwd  would do'
original command line, 1 arguments: i wonder what  pwd  would do
arg: i wonder what  pwd  would do
mangled command line, 0 arguments:
restored command line, 1 arguments: i wonder what  pwd  would do
arg: i wonder what  pwd  would do

while not entirely pretty, it does to do the trick. i'd be interested to know if anyone has a better way of doing this (or if there are hidden problems in what i've posted above).

Next.

Previous.