Search Results: "tar"

9 January 2009

Adrian von Bidder: Software updates....

Upgrading a laptop that has not been updated for ca. 2 months. Now it requests me to restart the system after every 5 minutes of downloading and installing packages. Repeatedly: I had to restart the upgrade process about 10 times already. In addition, I'm sure I've seen it updating some of the packages at least three times. But I'm sure by the end of today I'll be there. OpenSUSE. For when Windows was not annoying enough.

8 January 2009

David Welton: Angry Perl Users

I work on LangPop.com for fun. I make a tiny bit of money from the adsense there, but nothing much to speak of. Unlike the TIOBE folks, I don't sell my data to anyone. So it can be a bit frustrating when people get all bent out of shape about the results. A case in point was my recent article here, entitled Python "Surpasses" Perl?. In case the title wasn't a dead giveaway, I'll spell things out:
  • I was poking a bit of fun at the idea that any statistics of this nature could pinpoint, to a specific month, the moment when one language "passes" another. They're way too fuzzy for that!
  • I point out that Perl is still quite popular. According to the Freshmeat metric I was reporting, Perl actually still has more code out there than Python.
And just to be clear: I don't use Perl or Python or really care which one is more popular. I mostly use Ruby these days, with Tcl, Java, and bits of C thrown in for fun. In other words: langpop.com is not an evil conspiracy to discredit Perl. However, I also had the temerity to suggest that Python might be a little bit more popular in terms of new things happening. Shock! Horror! Well, that certainly irritated some Perl users. They called my statistics lies, they misspelled my name (as well as the word "you", which is a lot simpler than my name by any standard), and some of them went off on rants that were pretty badly off target (SourceForge is never mentioned in Langpop or this online journal!). However, not once did they really get around to answering my points, or proposing any sort of valid alternative metrics. One of my favorite Linux/Free Software news sources, lwn.net published their 2009 predictions, and had this to say about Perl:
It will be a make-or-break year for Perl. If the Perl developers cannot either bring new life to Perl 5 or turn Perl 6 into something real, this language will, by the end of the year, have moved well down the road to "legacy" status.
As can be imagined, that set them off all over again. The mental image I get is a big angry bull, frothing at the mouth, and ready to stomp that damn red cape. Here's the problem, though, guys: the cape isn't what's holding you back. Trust me, I know. Once upon a time, when people spoke of scripting languages, they talked about "Perl, Python and Tcl". No Ruby, no PHP, although both existed. I happen to like Tcl a lot, and it measures much worse than Perl on all the popularity charts... worse than plenty of other languages, these days. Lately, I don't use it so much, but once upon a time I dedicated a lot of mental energy to wondering why things weren't going better. And in that community too, there were people who consistently held their fingers in their ears and yelled "LALALALA there is no problem!". In the long term, it's not an effective strategy: it makes everyone but the other True Believers think you're going off the rails, because the evidence is plain to see that things weren't quite like they used to be, that there are problems. And it doesn't really address the problem in a constructive way, either. Long term, what works is to produce good code, and get it in people's hands. The most effective way of dealing with problems of this nature includes these points:
  1. Not froth at the mouth any time you perceive a slight to your technology of choice.
  2. Talk up the cool things you're working on. Be positive, explain what's new and exciting.
  3. Acknowledge reality. For instance with Perl (or Tcl), it's plain to anyone that they aren't as popular as they once were. Don't deny it, but focus on the fact that they're still widely used, and lots of new code *is* still written in them, and that they're still the focus of a lot of work to improve them.
  4. Actually follow through. Jon Corbet's (Linux Weekly News) point is not a bad one - you can have all the internal releases and milestones and general fun hackery that you want, but at a certain point, you need to get working code out in front of people. This is something Ruby seems to be suffering from these days too (as observed from afar) - lots of next generation fiddling, and tons of good ideas and energy, but to me there is a lack of technology actually flowing my way in a timely manner. With the possible exception of JRuby, which is making lots of progress and putting good code out there for people to use.
To conclude things, since the Perl guys were so unhappy with Freshmeat, here are some statistics from Amazon and Craigslist, comparing about a year ago with the most recent numbers: As can be seen, both Perl and Python are dropping in Craigslist postings, but Perl is falling farther. As a final footnote, I am sorry about this journal not accepting comments past a certain point (10 days is the cutoff) - otherwise I get 100's of spam postings a day. Unfortunately, the software I'm using didn't deal well with alerting people to the fact that comments were closed. I filed a bug, and they're working on it.

David Welton: Angry Perl Users

I work on LangPop.com for fun. I make a tiny bit of money from the adsense there, but nothing much to speak of. Unlike the TIOBE folks, I don't sell my data to anyone. So it can be a bit frustrating when people get all bent out of shape about the results. A case in point was my recent article here, entitled Python "Surpasses" Perl?. In case the title wasn't a dead giveaway, I'll spell things out:
  • I was poking a bit of fun at the idea that any statistics of this nature could pinpoint, to a specific month, the moment when one language "passes" another. They're way too fuzzy for that!
  • I point out that Perl is still quite popular. According to the Freshmeat metric I was reporting, Perl actually still has more code out there than Python.
And just to be clear: I don't use Perl or Python or really care which one is more popular. I mostly use Ruby these days, with Tcl, Java, and bits of C thrown in for fun. In other words: langpop.com is not an evil conspiracy to discredit Perl. However, I also had the temerity to suggest that Python might be a little bit more popular in terms of new things happening. Shock! Horror! Well, that certainly irritated some Perl users. They called my statistics lies, they misspelled my name (as well as the word "you", which is a lot simpler than my name by any standard), and some of them went off on rants that were pretty badly off target (SourceForge is never mentioned in Langpop or this online journal!). However, not once did they really get around to answering my points, or proposing any sort of valid alternative metrics. One of my favorite Linux/Free Software news sources, lwn.net published their 2009 predictions, and had this to say about Perl:
It will be a make-or-break year for Perl. If the Perl developers cannot either bring new life to Perl 5 or turn Perl 6 into something real, this language will, by the end of the year, have moved well down the road to "legacy" status.
As can be imagined, that set them off all over again. The mental image I get is a big angry bull, frothing at the mouth, and ready to stomp that damn red cape. Here's the problem, though, guys: the cape isn't what's holding you back. Trust me, I know. Once upon a time, when people spoke of scripting languages, they talked about "Perl, Python and Tcl". No Ruby, no PHP, although both existed. I happen to like Tcl a lot, and it measures much worse than Perl on all the popularity charts... worse than plenty of other languages, these days. Lately, I don't use it so much, but once upon a time I dedicated a lot of mental energy to wondering why things weren't going better. And in that community too, there were people who consistently held their fingers in their ears and yelled "LALALALA there is no problem!". In the long term, it's not an effective strategy: it makes everyone but the other True Believers think you're going off the rails, because the evidence is plain to see that things weren't quite like they used to be, that there are problems. And it doesn't really address the problem in a constructive way, either. Long term, what works is to produce good code, and get it in people's hands. The most effective way of dealing with problems of this nature includes these points:
  1. Not froth at the mouth any time you perceive a slight to your technology of choice.
  2. Talk up the cool things you're working on. Be positive, explain what's new and exciting.
  3. Acknowledge reality. For instance with Perl (or Tcl), it's plain to anyone that they aren't as popular as they once were. Don't deny it, but focus on the fact that they're still widely used, and lots of new code *is* still written in them, and that they're still the focus of a lot of work to improve them.
  4. Actually follow through. Jon Corbet's (Linux Weekly News) point is not a bad one - you can have all the internal releases and milestones and general fun hackery that you want, but at a certain point, you need to get working code out in front of people. This is something Ruby seems to be suffering from these days too (as observed from afar) - lots of next generation fiddling, and tons of good ideas and energy, but to me there is a lack of technology actually flowing my way in a timely manner. With the possible exception of JRuby, which is making lots of progress and putting good code out there for people to use.
To conclude things, since the Perl guys were so unhappy with Freshmeat, here are some statistics from Amazon and Craigslist, comparing about a year ago with the most recent numbers: As can be seen, both Perl and Python are dropping in Craigslist postings, but Perl is falling farther. As a final footnote, I am sorry about this journal not accepting comments past a certain point (10 days is the cutoff) - otherwise I get 100's of spam postings a day. Unfortunately, the software I'm using didn't deal well with alerting people to the fact that comments were closed. I filed a bug, and they're working on it.

Simon McVittie: Converting Debian packaging from bzr to git

I recently converted most of the Debian packaging for Telepathy and related projects from bzr to git, while changing the repository contents from just the debian/ directory to the whole source tree. Here's the recipe, using the latest one to be converted (pymsn) as an example.

Create a repository
mkdir pymsn
cd pymsn
git init

Mass tarball import
  • download all the tarballs into ../tarballs
  • rename all the tarballs to *.orig.tar.gz:
    rename 's/^pymsn-/pymsn_/' *.tar.gz
    rename 's/\.tar\.gz$/.orig.tar.gz/' *.tar.gz
    
  • use git-import-orig to import them:
    cd ../pymsn
    ls ../tarballs/pymsn_*.orig.tar.gz
    for v in 0.2 0.2.1 0.2.2 0.3.0 0.3.1 0.3.2 0.3.3; do \
    git-import-orig --no-merge --no-dch --pristine-tar \
        -u $v ../tarballs/pymsn_$v.orig.tar.gz; done
    
  • throw away the resulting master branch in favour of using one we convert from bzr later:
    git checkout upstream
    git branch -D master
    

Converting the Debian packaging This has the slight complication that in some (but not all) of the bzr commits, the repository contained only the contents of the debian directory, in the root of the repository. So, I needed to rewrite history, with this script:
#!/bin/sh
# index-filter.sh
if git ls-files -s   grep debian; then
    :
else
    git ls-files -s  
        sed -e "s-\t-&debian/-"  
        GIT_INDEX_FILE=$GIT_INDEX_FILE.new git update-index --index-info &&
    mv $GIT_INDEX_FILE.new $GIT_INDEX_FILE
fi
  • Download unstable and experimental packaging into ../pymsn-experimental and ../pymsn-unstable
  • Use bzr-fast-export and git-fast-import to import the two branches:
    ~/.bazaar/plugins/fastimport/exporters/bzr-fast-export.py \
        ../pymsn-unstable   git fast-import
    git branch -m master debian-lenny
    ~/.bazaar/plugins/fastimport/exporters/bzr-fast-export.py \
        ../pymsn-experimental   git fast-import
    git branch -m master debian
    git branch -D tmp
    
  • Rewrite history so changelog, control etc. are consistently inside a debian directory, with the script shown above:
    git filter-branch --index-filter $(pwd)/../index-filter.sh debian-lenny
    rm -r .git/refs/original
    git filter-branch --index-filter $(pwd)/../index-filter.sh debian
    rm -r .git/refs/original
    
  • Merge the appropriate upstream versions into each branch:
    git checkout debian
    git merge upstream/0.3.3
    vim debian/control        # and change Vcs-Bzr to Vcs-Git
    dch "Move packaging to git"
    debcommit -a
    git checkout debian-lenny
    git merge upstream/0.3.1
    vim debian/control        # and change Vcs-Bzr to Vcs-Git
    dch "Move packaging to git"
    debcommit -a
    
  • Rescue Debian patches and put them on a debian-patches branch, which is rebased with each upstream release. For pymsn, there aren't any patches in the debian branch at the moment:
    git branch debian-patches upstream/0.3.3
    
    but there is one in the debian-lenny branch:
    git branch debian-lenny-patches upstream/0.3.1
    git checkout debian-lenny
    cp debian/patches/00-fix-dns-resolution.diff ..
    git checkout debian-lenny-patches
    patch -p1 < ../00-fix-dns-resolution.diff
    git commit -a
    
  • Make a repository on alioth:
    cd /git/pkg-telepathy
    GIT_DIR=pymsn.git git init --bare --shared
    GIT_DIR=pymsn.git git config hooks.mailinglist ...
    vim pymsn.git/description
    
  • Push to the repository:
    git gc --aggressive
    git remote add origin git+ssh://git.debian.org/git/pkg-telepathy/pymsn.git
    git push --all
    
  • On alioth, do some more setup after the initial push:
    cat > pymsn.git/hooks/post-receive <<EOF
    #!/bin/sh
    exec /usr/local/bin/git-commit-notice
    EOF
    chmod +x pymsn.git/hooks/post-receive
    GIT_DIR=pymsn.git git symbolic-ref HEAD refs/heads/debian
    
  • Mark the old repositories as no longer used:
    cd ../pymsn-unstable
    dch $'This repository is no longer used, instead please use git:\n'\
    "git://git.debian.org/git/pkg-telepathy/pymsn.git" -c changelog
    debcommit -c changelog
    bzr push
    cd ../pymsn-experimental
    dch $'This repository is no longer used, instead please use git:\n'\
    "git://git.debian.org/git/pkg-telepathy/pymsn.git" -c changelog
    debcommit -c changelog
    bzr push
    cd ../pymsn
    

Sven Mueller: About Usability

Sami Haahtinen wrote a nice post about usability. I do mostly agree with him, except for one little thing: A few years back, I noticed that I often started as a kind of Joe User with new applications, and often even stayed that way, often wishing that the UI of that application was simpler or at least made it more obvious which options are important and which I could most likely keep at their defaults.
Yet I am a system administrator and programmer/developer, so most people would probably consider me as an advanced user by default (wether that is correct in any given context or not, for example I still don t really grok GIMP).
Also I did have quite a lot of contact with users that have very little experience, knowledge and interest in computers and software and only use them to achive certain goals. Anyway, I m quite sure most readers of this blog are aware that Linus Torvalds once complained about the Gnome printing dialog being simplified way too much. And at that time, I had to agree (note that I didn t check out Gnome again since those days, at least 2 years ago). What I really wonder (especially since I realized this in some of my own applications) is why it is so hard for developers to add some additional checkbox or button that enables/disables advanced options. For example, let s take an applications printing dialog. By default, it would only allow the selection of a printer, orientation (if sensible) and (if the selected printer supports more than one) the paper size. Now advanced options might include duplex printing (automatic or manual), printing mutliple pages on a single sheet of paper, printing in grey on color printers, These options might be of use to anyone, but would most likely confuse beginners. So what I did in most of my little application was to just show the basic options and add an advanced options button which provided access to additional functionality which was not normally needed. As for the issue with firefox updates mentioned by Sami: I don t think it is a problem that firefox asks what to do with extensions. For one, it only lists extensions which are installed but claim not to support the new version of firefox. Second, practically everyone I know of who uses any extension can be considered to be an advanced user. And third: The dialog is pretty straightforward about what it is asking and when I once hit it while a ( dumb user ) friend was sitting with me, he understood what it was about quite easily (I didn t explain the dialog, just what extensions were). So my idea of a good UI is:
As simple as possible for beginning users, but allow advanced users to get to the details ( Details or Advanced options buttons/checkboxes/dialogs).

7 January 2009

Felipe Augusto van de Wiel: 8 Jan 2009

Debian

Re: Blog rewrite upcoming
Erich, it seems that you got TONS of recommendations for different blog software, so you probably already received a recommendation for Chronicle: The Blog Compiler, it is written by Steve Kemp inspired by Joey's ikiwiki. I've been also wondering about using ikiwiki and/or chronicle. :-)


From the privacy-issues-department...
This is in my draft for a long time now...

Nico, I never understood what was so private about your debian.net page (Thanks WayBackMachine!).

PS: Happy 2009!

6 January 2009

Joey Hess: proposing rel-vcs

I'm working on designing a microformat that can be used to indicate the location of VCS (git, svn, etc) repositories related to a web page. I'd appreciate some web standards-savvy eyes on my rel-vcs microformat rfc. If it looks good, next steps will be making things like gitweb, viewvc, ikiwiki, etc, support it. I've already written a preliminary webcheckout tool that will download an url, parse the microformat, and run the appropriate VCS program(s). (Followed by, with any luck, github, ohloh, etc using the microformat in both the pages they publish, and perhaps, in their data importers.) Why? Well,
  1. A similar approach worked great for Debian source packages with the XS-VCS-* fields.
  2. Pasting git urls from download pages of software projects gets old.
  3. I'm tired of having to do serious digging to find where to clone the source to websites like Keith Packard's blog, or cariographics.org, or St Hugh of Lincoln Primary School. Sites that I know live in a git repo, somewhere.
  4. With the downturn, hosting sites are going down left and right, and users who trusted their data to these sites are losing it. Examples include AOL Hometown and Ficlets, Google lively, Journalspace, podango, etc etc. Even livejournal's future is looking shakey. Various people are trying to archive some of this data before it vanishes for good. I'm more interested in establishing best practices that make it easy and attractive to let all the data on your website be cloned/forked/preserved. Things that people bitten by these closures just might demand in the future. This will be one small step in that direction.

Matt Brubeck: 6 Jan 2009

MapReduce with Parallel Make Ted Gould s post about transcoding music files using Make got me thinking about combining Make and RecordStream into a barebones MapReduce framework. As a demonstration, I wrote a partial solution to Tim Bray s Wide Finder challenge, to calculate hit counts from web server logs. More details (and code).

David Welton: Git is a Pain in the Ass

Everyone's been talking about how great and wonderful git is, and I've tried using it for a few projects of my own, on a local basis, sort of like an advanced form of 'rcs'. So I thought I'd try it out in a little bit more complicated setup, for a site I host on my web server. I want to have a main repository, that I "check out" (or 'clone', in git terms) on my laptop. My laptop can't be the main repository, because I also want to be able to commit from the web server (once in a while a quick live update is called for), so I need a stable address. Ok, I'll put it in my home directory on the web server, just to see how things work.
mkdir foobar
cd foobar
git init
Ok, so far so good. Now I try and clone it on my laptop, where I will then add files, commit, then push them. Oops, that doesn't work. You can't clone an empty repository. Ok, I add something on the server where I init'ed it, an empty file, just to give it some content. Now cloning it works on the laptop. Great, we're in business! I add a bunch of files from the project to the laptop repository, commit, then push them. Seems to be ok so far... I do a checkout on the remote/server machine, and I see my files. Good. Ok, let's try making a change on the laptop. I remove a file, commit it, push it (which is already an extra step compared to subversion... hrmph). I do a checkout on the server, but it won't erase the file I removed on the laptop. Weird. I google around a bit, and find that this is supposed to be for my own good, so I won't wreck things on the server. I need to do git reset --hard (which isn't a very reassuringly named command) there, and then a checkout, and now things work. That, however, is a lot of work just to commit and update stuff! I ask for some help on #git, where they mention the --bare option to init. Since I'm just messing around, I go back and redo the init step, wiping the old repository. Now I try and clone that to start over again. Oops, I forgot, I can't clone it because it's empty. Grrrrrr.... this is getting annoying. So, the kind folks on #git tell me I should push to the new server repository from one that's already populated. Ok, let's try setting up the laptop:
git init
... add/commit some files ...
git remote add origin ...myurl...
Now let's try pushing. Nope, still doesn't do it, I'm missing something. Frustrating. A bit more fiddling and googling, and i find:
get push origin master
Aha! It worked! Weird. And now a simple git push works too. The whole thing seems kind of shaky in the sense that it's not very confidence inspiring: I feel like it wouldn't take much to make a wrong turn and find all my files gone forever. I can see some of the advantages, and will likely stick with it - git is quite convenient for local files that I might not have bothered putting under version control in the past, but it's also a bit more "bureaucratic" in that you have more steps to do, and you have to fill in the forms just so... or else!

David Welton: Git is a Pain in the Ass

Everyone's been talking about how great and wonderful git is, and I've tried using it for a few projects of my own, on a local basis, sort of like an advanced form of 'rcs'. So I thought I'd try it out in a little bit more complicated setup, for a site I host on my web server. I want to have a main repository, that I "check out" (or 'clone', in git terms) on my laptop. My laptop can't be the main repository, because I also want to be able to commit from the web server (once in a while a quick live update is called for), so I need a stable address. Ok, I'll put it in my home directory on the web server, just to see how things work.
mkdir foobar
cd foobar
git init
Ok, so far so good. Now I try and clone it on my laptop, where I will then add files, commit, then push them. Oops, that doesn't work. You can't clone an empty repository. Ok, I add something on the server where I init'ed it, an empty file, just to give it some content. Now cloning it works on the laptop. Great, we're in business! I add a bunch of files from the project to the laptop repository, commit, then push them. Seems to be ok so far... I do a checkout on the remote/server machine, and I see my files. Good. Ok, let's try making a change on the laptop. I remove a file, commit it, push it (which is already an extra step compared to subversion... hrmph). I do a checkout on the server, but it won't erase the file I removed on the laptop. Weird. I google around a bit, and find that this is supposed to be for my own good, so I won't wreck things on the server. I need to do git reset --hard (which isn't a very reassuringly named command) there, and then a checkout, and now things work. That, however, is a lot of work just to commit and update stuff! I ask for some help on #git, where they mention the --bare option to init. Since I'm just messing around, I go back and redo the init step, wiping the old repository. Now I try and clone that to start over again. Oops, I forgot, I can't clone it because it's empty. Grrrrrr.... this is getting annoying. So, the kind folks on #git tell me I should push to the new server repository from one that's already populated. Ok, let's try setting up the laptop:
git init
... add/commit some files ...
git remote add origin ...myurl...
Now let's try pushing. Nope, still doesn't do it, I'm missing something. Frustrating. A bit more fiddling and googling, and i find:
get push origin master
Aha! It worked! Weird. And now a simple git push works too. The whole thing seems kind of shaky in the sense that it's not very confidence inspiring: I feel like it wouldn't take much to make a wrong turn and find all my files gone forever. I can see some of the advantages, and will likely stick with it - git is quite convenient for local files that I might not have bothered putting under version control in the past, but it's also a bit more "bureaucratic" in that you have more steps to do, and you have to fill in the forms just so... or else!

Uwe Hermann: Running homebrew and open-source software on your Wii, using the Twilight Hack and the Homebrew Channel

Wii I've been owning a Nintendo Wii for quite a while now, but so far only played an occasional game of tennis or the like. Over the holidays I got The Legend of Zelda: Twilight Princess, which is a really nice game in itself. What is even more interesting though, is that this game allows you to run homebrew software on your Wii. What you usually want to do, is to use the so-called Twilight Hack to install the Homebrew Channel on your Wii (you'll need an SD card and the Twilight Princess game for this). Once the process has finished you won't need the game anymore though. So, here's a quick tutorial on how to install the Homebrew Channel. I basically followed the Setting up your Wii for Homebrew HOWTO from wiibrew.org. Requirements An SD card max. 2GB in size (not an SDHC card!), an SD card reader in your PC or laptop, and the Twilight Princess game (which you must play at least once). Backup First, you'll want to backup your Twilight Princess save games (if any). For that, we format the SD card (it needs a FAT16/FAT32) filesystem.
 $ mkfs.vfat /dev/mmcblk0p1 (you may need to change "mmcblk0p1" to whatever fits your setup)
Now insert the SD card into the Wii, start the Wii, go to Wii Options --> Data Management --> Save Data --> Wii and copy your Twilight Princess save games onto the SD card. Then, poweroff, remove the SD card, insert it into your laptop/PC and copy the save games to a safe place.
 $ mount -t vfat /dev/mmcblk0p1 /mnt
 $ mv /mnt/private ~/zelda_savegames
Twilight Hack and Homebrew Channel The Legend of Zelda: Twilight Princess Next up: download the Twilight Hack and Homebrew Channel, and put the files on the SD card:
 $ wget http://hbc.hackmii.com/dist/twilight-hack-v0.1-beta1.zip
 $ unzip twilight-hack-v0.1-beta1.zip
 $ cp -r private /mnt
 $ wget http://hbc.hackmii.com/dist/the_homebrew_channel-1.0.1.tar.gz
 $ tar xfvz the_homebrew_channel-1.0.1.tar.gz
 $ cp the_homebrew_channel-1.0.1/boot.dol /mnt
 $ umount /mnt
Now remove the SD card, insert it into the Wii and power-up the Wii. Go to Wii Options --> Data Management --> Save Data --> Wii, delete the Wii Twilight Princess save games, then copy the "Twilight Hack" save game from the SD card onto the Wii (you need to select the correct one for your region). Quit the menu, start Twilight Princess, load the "Twilight Hack" savegame and finally talk to the person in front of you (do not perform any other actions, or the Wii may crash!). Follow the remaining instructions on the screen and you're done. You now have an additional channel named Homebrew Channel where all your homebrew programs/games (and your own Free Software programs you may write/port) will appear. Installing homebrew software So far, there are no real applications in the Homebrew Channel, you need to put all the homebrew software you want on the SD card. There's a huuuge list of applications and games to choose from, many of them Free Software with source code, some however are binary-only. Basically, you put each application in a sub-directory of apps/ on the SD card, the most important file in every sub-directory is always boot.dol. If you have a boot.elf file instead, you'll probably need to convert it using the ELF to DOL converter. Restoring Twilight Princess save games You may now want to restore your Twilight Princess save games onto the Wii, you no longer need the "Twilight Hack" save game. Put your ~/zelda_savegames directory on the SD card and name it private/ again. Insert the SD card into the Wii and copy the save games on the Wii (similar procedure as above). Have fun with homebrew software on your Wii, or even better write your own software or port existing Linux applications/games!

MJ Ray: Copyrighting the Future consultation ends this month

The Copyrighting the Future: Keeping ahead of the game consultation finishes in a month s time (6 February). They actually called it that - copyrighting the future - can you believe it? In case you haven t seen it yet, this is the consultation that ignores the Gowers review findings and tries to start it all over again. Please respond and try to avoid term extension, DRM/TPM and other similar landmines. Any comments you can leave me to help us all would be very much appreciated!

5 January 2009

Ben Hutchings: Searching for invalid object code

J r my Bobbio suggested that I should explain how I looked for packages affected by a compiler bug (Debian bug 506713; gcc bug 38287). I don't claim that this is a particularly good way to do it, but here it is: First, I identified a pattern to search for. Unfortunately I don't really understand the cause or fix for the bug, but I did have the example which led to this bug report: Debian bug 490999. The bad code was a stack-pointer-relative load immediately after a stack allocation (SPARC save instruction) where the offset was not adjusted for the stack allocation:
save  %sp, -112, %sp
ld  [ %sp + 0x40 ], %i5
I generalised this to:
save  %sp, offset1, sp
...
ld  [ %sp + offset2 ], register
where offset1 + offset2 < 0. Of course, this may be valid if the intervening instructions include a restore, branch or store to the effective stack location that the last instruction loads from. I ended up allowing up to 10 intervening instructions and examining a disassembly to work out which cases were valid. I looked up the instruction encoding for these two instructions. Thankfully SPARC is RISC so they are simple and regular: I took the dumb but effective approach of scanning entire files for this pattern rather than only scanning their code sections. This seemed to work - I got no false hits for non-code - but might not work for other patterns that could match ASCII text. I wrote the scanning program in Python, which is my default choice of language unless I know it's going to be too slow. I was hoping to be able to read the code files into arrays, but unfortunately the Python array type only supports the native byte-order (SPARC is big-endian and I was intending to use an x86 which is little-endian). I tried reading into a tuple using struct.unpack, which does support explicit byte-ordering, but this used so much memory for larger files that the program swapped to a crawl. So finally I resorted to reading the file into a string, doing a string search for '\x9d\xe3', rejecting matches that weren't appropriately aligned, then unpacking and comparing the code words from the point of the string match. (In Python 3.0 I would have to use the bytes type for this, as str is a Unicode string type.) So that's how I scanned single files. The next step was to find, unpack and scan all the SPARC shared libraries in the archive. (This particular code generation bug is understood to affect only PIC code, and that is normally only used in shared libraries.) I wrote functions to search Contents-sparc for shared library files - assumed to match the pattern ([^\s]*/lib[^/\s]+\.so(?:\.[^/\s]*)?) - and to parse Packages to find the filenames for the packages containing those files. The latter uses the debian_bundle.deb822 module from python-debian. The last key function downloads and unpacks a package using wget and dpkg-deb. I could have used the httplib module for downloading but I correctly anticipated that I'd need to restart the script several times so I wanted to cache the packages which was easier to do using wget. So, that's the explanation. If you really want to see it, here's the code.

4 January 2009

Andrew Pollock: [tech] Feed breakage, Blosxom's shiteful RSS 0.91 implementation

(or echo "blosxom hold" sudo dpkg --set-selections) A few people had pointed out to me that my blog was appearing weirdly in Google Reader. I myself read my blog via Planet Debian and Planet Linux Australia, where it appears fine, so I hadn't noticed the problem first hand. I did some digging tonight with the Feed Validator, and I think I've figured it out. Funnily enough, it all started with a previous session with the Feed Validator... I wanted my blog to be UTF-8, with it being the new hotness and all. Blosxom was spitting it out as ISO-8859-1. I've just learned from this evening's debugging, that this is the behaviour of CGI.pm's header() function if not provided with the -charset parameter. I'd previously worked around this by defining my own content_type for the RSS and HTML flavours. In this file, I just put text/xml; charset=utf-8and text/html; charset=utf-8, respectively. Unfortunately, this then breaks Blosxom's internal HTML-escaping-for-RSS-feed code, which is conditional on the content type being XML:
if ($content_type =~ m \Wxml$ )  
        # Escape <, >, and &amp;, and to produce valid RSS
        my %escape = ('<'=>'&lt;', '>'=>'&gt;', '&'=&gt;'&', '"'=>'&quot;');  
        my $escape_re  = join ' ' => keys %escape;
        $title =~ s/($escape_re)/$escape $1 /g;
        $body =~ s/($escape_re)/$escape $1 /g;
       
This block of code was being ignored because the regex was anchored. "text/xml; charset=utf-8" didn't match "\Wxml$". Whenever I last had an attempt to make my blog's feed pass the Feed Validator, I must have edited the Blosxom source code to make this regex not be anchored. Then on November 11, 2008, I updated my Blosxom package to 2.0-14+etch1, to address CVE-2008-2236 (incidentally, there doesn't appear to be a Debian Security Advisory for this?), and my local modification got clobbered. That'll teach me to make local modifications and not put the package on hold. Blah. It seems I'm screwed, there's no way to have a well-formed UTF-8 RSS feed without requiring a source-code modification to Blosxom. I've either got to change the way the header() function is being called, or I've got to change the regex to allow me to specify a charset in the flavour file, and not disable HTML escaping. Speaking of screwed, Blosxom seems incapable of making a well-formed RSS 0.91 feed anyway. It wants to put the <pubDate> tag inside an <item>, which is a violation of the DTD. There's supposed to be one for the whole <channel>, not one per item (or "story" in Blosxom-flavour-speak). You can't just move that from the story flavour file to the head flavour file, because a pile of variables that you'd otherwise use seem to only get set on a per-story basis. Double blah. I'm starting to feel like Blosxom is outliving its usefulness, but I really do not feel like embarking on a wholesale move to a new blogging platform, I'm fairly entrenched.

Jeff Bailey: ... 2008

I try to practice Inbox Zero, and this past year has been brutal far staying on top of my Inbox. When I started pruning the home email a week or so ago (having just gotten my work email box down to zero). It's giving me a pretty good review of 2008. And, umm.. wow:

Spinal tumor, surgery, hating Mountain View (the place, not the job there. My coworkers are pretty uniformly awesome), having a toddler, not being able to lift my toddler, crazy mood altering drugs, visa renewal, a funeral, unexpected pregnancy, weddings, buying a house, mid-term miscarriage, Angie's hospitalisation from that, pneumonia, a move home, having to withdraw from my Master's courses 3 times over the course of the year for medical reasons - eventually causing me to withdraw from the program. Extend it by another 3 months into 2007 (because while I'd love to believe that stress reset at the start of a calendar year...), and we have another wedding, a move to the US and a new job.

I've just gotten to an email where I took notes on what the risks were for the angiogram before I'd told many friends. There's a small pile of emails and such from people wishing me well that I never managed to reply to. I also find myself thinking back to the people I didn't hear from, and the reevaluation of my communities and friendships that came along with that.

The lyrics from Counting Crows come to mind: "It's been a long December and there's reason to believe that maybe this year will be better than the last."

Here's to 2009. =)

Andrew Pollock: [tech] Mixing electricity and water: monitoring the cat water bowl with Nagios

(this is something I've had "in production" for many months now, I just haven't had the time or energy to do a proper write up about what I did) We have a cat water bowl, it looks like this: The cat water bowl Under "normal" circumstances, it usually lasts about seven days. So when our weekly routine is happening, we'll refill it on a Saturday whilst doing house chores, and it'll last until the following Saturday when it gets refilled. Unfortunately, sometimes our routine gets disrupted, and we forget. Sometimes, we travel and have a house-sitter, who may not pay as close attention to such things as ourselves. Once, one of our cats was licking the condensation off a chilled bottle of soft drink that was on the kitchen counter one evening, before we realised the water bowl needed refilling. Naturally, we felt like terrible pet owners. So I think it was some time around the 2008 Maker Faire, that I hatched the idea of having some sort of water sensor on the cat bowl, which would communicate to one of the various computers in house. At the Maker Faire, I bought a copy of Making things talk, and an Arduino starter kit, which consisted of a Diecimila board and a make-it-yourself proto shield. I also bought a little electronics starter kit, which consisted of a breadboard and various components, and a USB-TTL cable. I decided to use Bluetooth to communicate with the board, as I already had my MythTV setup using a Bluetooth keyboard and mouse, and it was within range of the water bowl. I decided against using Zigbee, because I didn't know anything about it, and I didn't want to add (or learn about) yet another wireless infrastructure just for this project. I should point out that I know very little about electronics. I'd never owned (or really used) a soldering iron until I embarked on this project. I took a basic soldering class at the Tech Shop, but I'd already assembled the proto shield by the time I took the class, so I'd pretty much figured it out. I had a very naive vision that I could just basically shove two wires in some water and it'd close a circuit and that would be my water sensor. Of course this didn't work, so I started hunting around on the Internet for a circuit that would do this. I happened upon a circuit (I don't seem to have retained the URL, so I can't link to it), which just consisted of a couple of transistors and some resistors. So I headed off to Fry's to try and buy the transistors I needed. I quickly discovered that I didn't have sufficient information to identify the transistors that I wanted, but I did happen to stumble upon a cheap assemble-it-yourself water alarm. It consisted of a PCB, and a transistor and some resistors and a buzzer. I bought a couple of these instead. Between studying the PCB and the circuit diagram that came with the alarm, I was able to reproduce it on my breadboard instead of on the PCB. Sure enough, placing the two probes in water closed the circuit. I replaced the buzzer with an LED so I could see what was going on. I attached the circuit to the Arduino proto shield, and had it feed into one of the digital I/O ports. I wrote some quick and dirty Wiring code so that when water not present (i.e. the circuit was open and no current was detected on that I/O port) the LED was switched on. Really at this point, I didn't need a microcontroller, I could have presumably achieved the same thing with a NOT gate. At this point, I wanted to make the sensor remotely queryable. I bought a BlueSMiRF Silver Bluetooth modem, which I attached to the TX and RX lines of the board (I first configured it by attaching the USB-TTL cable to it and using Minicom on my laptop). I extended the Wiring code to provide a rudimentary prompt, and accept a command to check if water was present. I think around this time it also dawned on me that I could use the digital I/O pins as a switch. When they're "high" they provide power. So rather than constantly running a current through the water, I only needed to briefly power up the water detection circuit, see if the circuit closed or not, and then report if water was present if it did. I much preferred this, as at the time, I was endeavouring to power the whole sensor off a 9 volt battery. I figured I'd get much better battery life if I wasn't running a current through the water the entire time. I should point out that I did some "tongue tests" in a glass of water while the circuit was powered up, and couldn't detect a difference between when the circuit was on or off. The last thing I wanted to be doing was zapping the cats! At this point, the Wiring and Arduino work was pretty much complete. I setup ser2net on the MythTV server, so that I could just connect to port 4000, and be connected to the water sensor.
apollock@icarus:~$ telnet teevee 4000
Trying 172.16.0.9...
Connected to teevee.andrew.net.au.
Escape character is '^]'.
ser2net port 4000 device /dev/rfcomm0 [9600 N81] (Debian GNU/Linux)
waterbowl> s
Water is present
waterbowl>
telnet> q
Connection closed.
apollock@icarus:~$
The Wiring code running on the Arduino board is checked in here. Next, I wanted to monitor this with Nagios. One thing I found with the Bluetooth connection was that it wasn't all that reliable. Not every connection to port 4000 resulted in a connection with the water sensor. I elected to write some standalone code that submitted results to Nagios by way of a passive check, rather than having Nagios try to actively monitor it. Again, trying to conserve energy, I decided to only check the sensor once every 8 hours. So I wrote a Python daemon, which tried to be as robust as possible. If at first it doesn't make a connection on its 8 hourly schedule, it keeps trying until it does. Nagios itself has some freshness detection for this monitor, so if no passive result is submitted within 8 hours and one minute, Nagios alerts that something is possibly wrong with the sensor itself. (The original intent was to deal with the situation where the battery went flat) This is the Nagios service definition I've got. Some of it may be unnecessary or redundant:
define service  
        host_name                       teevee
        service_description             Cat water bowl
        check_command                   check_stale!2!"Check water bowl monitor is on and reachable"
        normal_check_interval           480
        notification_interval           240
        active_checks_enabled           0
        check_freshness                 1
        freshness_threshold             28860
        max_check_attempts              1
        check_period                    24x7
        use                             generic-service
        stalking_options                o,w,c
        contact_groups                  everyone
 
The code for the Python daemon is checked in here. So everything was going swimmingly, apart from I couldn't get even 24 hours of monitoring on my conservative 3 times a day schedule, and the Bluetooth modem configured for all of its power saving options, and the water sensor itself only being powered on when it was performing a check. There wasn't much more I could do to try and reduce power consumption, at least with my limited electronics knowledge. Perhaps using Zigbee instead of Bluetooth would have helped, but I still think I'd have been going through 9 volt batteries more quickly than I'd have liked. I also had a brief foray into solar powering the board. I thought maybe the lighting in the house would be sufficient to power the board. Of course this didn't work out. I'd also have needed to make the code running on the board more self-sufficient, and have it just provide a water status indication whenever it had enough power to do so, instead of being a fairly dumb device like it currently is. This all felt a whole level more complicated, and out of my league, and I wasn't interested in attempting this sort of remote-sensing exercise at this time. The Arduino board can also be powered by USB, so as I already had some long USB type A to type B cables (that had funky lights in them to boot), I decided to just get a wall wart that had a USB type A receptacle, and powered the Arduino board that way. So much for being completely "wireless". (I could have also gotten a general purpose DC power supply that was capable of spitting out 9 volts, but I doubted I'd get one with a sufficiently long cable, which is why I went for the USB-powered option, as I already had a cable long enough) Speaking of wires, the most challenging part was the probes for the water sensor. As they were going to be permanently in the cats' drinking water, I didn't want to contaminate the water with them. I figured plain untinned copper wire would be okay, since water pipes are copper. Finding untinned, unstranded copper wire was a real challenge. I started out using some FM antenna cable, but that was stranded, and the shielding was a nightmare to strip. It was also reasonably difficult to make go where I wanted it. What I really wanted was something more solid, that would flex and stay where I put it. I cannibalised a spare IEC power cable, but it was also stranded copper wire. I finally managed to obtain some solid-core CAT-5 cable from a hardware store, and this has worked exactly as I wanted. I haven't done any further work on the setup since getting the CAT-5 cable for the probe. Further improvements that I'd like to do at some point: The finished product I had a lot of fun with this project. I had a real sense of achievement, being able to go from concept to completion, and learn a few things about electronics along the way. I'm normally not a fan of messing around with hardware. Some photos of the project are here.

Wouter Verhelst: Quake I

Years ago I think some two years after it was released I bought me a copy of Quake. I found the original box in a shop that had just one left, and was selling it at a serious discount; I still have the box, with the original artwork. I didn't have Internet at the time, though, so I never played it online. I did finish the game at one point, though I don't remember whether it was with or without cheatcodes. I'm certain I got very far, though I had a tendency to start using cheatcodes when the monsters I had to pass got too hard for me to actually be able to pass them. When I did get an Internet connection, Quake III Arena had just been released, so not many people still played Quake I. However, my very own ISP then did have a QuakeWorld server still running, and I fondly remember playing a lot on that server. Since I hadn't heard of +mlook, which enables modern-style mouse-based movement in quake, I was still playing with doom-style controls, that is, keyboard only. At first I got fragged all the time, but it didn't take me long to master this method. I got so good at it that I've won some deathmatch games, with keyboard only. Seriously. Eventually, however, I migrated to a state where I did not have any Microsoft operating systems installed on my system anymore; and while Debian had a 'quakeforge' package in potato, it was rather buggy; I remember it would reproducibly crash when you would load a particular level from the third episode. I did think about fixing the code, but then I didn't really have the time or expertise to do so, so that never got anywhere. Eventually, it was pulled from the archives; and after a while around the time when my ISP shut down their quakeworld server I stopped playing Quake; and though I was sad about it not being installed anymore, I just forgot about it. Recently, however, I somehow stumbled upon this 'Linux Quake HOWTO' over at tldp, and noticed that apart from Quakeforge, there really are other engines, and there really still is a Quake I community around. With quakeworld servers and everything. Oh boy. So I installed this 'tyrquake' a few days ago, and I've now already reached episode 3. Some things are coming back, but others aren't really... Of course, when I connect to one of those quakeworld servers, I get fragged all over. I'm lucky when I end up with a positive frag count. For reference: in Quake I, you only lose frag points when you kill yourself... Hah. To be continued, I guess...

Julien Blache: pommed v1.25: wireless keyboard, DBus configuration

Two fixes in this release, let s start with the more important one :) The DBus configuration file shipped with pommed now explicitly allows method calls to org.pommed, which is needed with the newer versions of DBus where the default configuration on the system bus has been made more strict to close a security hole. If the pommed clients stopped working for you after a DBus update, replace /etc/dbus-1/system.d/pommed.conf by the dbus-policy.conf file shipped with pommed. Second fix is related to the Apple wireless (Bluetooth) keyboard, which previous versions of pommed would recognize just fine but reject because it exposes a type of events usually indicative of a mouse (and pommed doesn t listen to mouse devices). This is now fixed for this particular keyboard, so pommed will react to the hotkeys on this keyboard too. For the complete list of changes, see the ChangeLog file in the source distribution. Comments at the usual address :)

Sami Haahtinen: About Usability

Some people consider the gnome usability guidelines a nuisance and some consider certain applications way too simplistic. While it is really hard to get the usability right, it's well worth it. We need to keep in mind that as computer oriented people we tend to see things differently. Things that are simple to us aren't really that simple to the "normal people". But one of the simple things we can do the insure that the software we write serve the people it's designed for is to remove all not needed pop-ups and questions. A good way to detect these would be to ask yourself why would a user choose anything else but the most logical option. It's kind of hard to explain, so lets pick an example: Firefox is updated, the first question it usually asks after upgrade is about incompatible extensions. The user is presented with choices, check for updates or cancel. Now, we are dealing with internet browser so the user should be connected to the internet so there is no problem with checking for the updates, we can rule out that scenario. The other scenario that I can come up with is that a developer doesn't want to update some certain extension. So at least I can't come up with any reasonable scenario why someone would want to select anything else but to upgrade the main option of upgrading the extensions. Why not just leave out the option and instead do the upgrade automatically. If you wish to be transparent, you can show the user that you are doing the upgrade. Or you can do like some applications do and just do it without bothering the user with the options. I know I sound like a Google fanboy, but Google generally gets this right. Their applications skip out all upgrade related notices and just do the upgrade. Regular user doesn't want to upgrade because the user has been scared with incompatibility notices and upgrade checklists for so long. Just going ahead with the upgrade in complete silence keeps their software up to date as well. Another example would be from few years back: The Ubuntu installation. Back in the day Debian was working on Debian Installer, which is also used as the main installer in Ubuntu alternative installation media. Debian Installer is capable of doing most things silently, but with Debian it by default asks a lot of questions. It doesn't matter, since most people who install Debian can be categorized as developers. But in my opinion, the thing that made Ubuntu a success was that it doesn't ask the questions that can be answered without asking the user. So, back to usability. There are basically 2 camps, the "normal" users and the developers. Developers want and need to see a lot of the backend behaviour, just to debug problems. Currently a lot of the open source software is focused towards developers while they are gaining grounds on the "normal" population as well. We should start focusing on the users for a change.

3 January 2009

David Moreno: Ruby goodies: Modules and methods for my everyday Ruby

I make a lot of Web script processing, whether scraping, webservices, systems administration, etc. Because I sometimes happen to repeat small and useful chunks of code for different projects, I thought that, given making new modules and methods is usually hassleless in Ruby, I should make my own set of methods and goodies I constantly use. Example 1, I sometimes miss Perl s LWP::Simple simplicity where I just pass a URL to a subroutine and get the content on a variable, quick, one-liner. Example 2, extract all links on a given URL on an array that I can then iterate and maybe fetch given the first example. Getting all A links is very easy to do, say with regex or with Hpricot (which should the best way to parse HTML), but most of the time I (and people, I d bet) need absolute URLs which is fairly more complex (relative, absolute URLs, BASE href declarations, etc, the same case as in Feedbag). Well, for different cases like that one, I ve started my own set of Ruby goodies. If you don t find them useful, I understand, they are mostly for my needs, I just want to make it public, because maybe, some other people might indeed find them useful. Simple installation:
sudo gem install damog-goodies -s http://gems.github.com/
As time and needs pass, I will be adding stuff into it. For the time being, here s both above examples in action:

>> require "goodies"
requiring /var/lib/gems/1.8/gems/damog-goodies-0.1/lib/goodies/array.rb
requiring /var/lib/gems/1.8/gems/damog-goodies-0.1/lib/goodies/lwr.rb
requiring /var/lib/gems/1.8/gems/damog-goodies-0.1/lib/goodies/html.rb
=> true
>> pp HTML::Links.find "damog.net"
["mailto:david-YOUKNOWTHEDEAL-axiombox.com",
 "http://log.damog.net/",
 "http://historiasdenuevayork.com",
 "http://axiombox.com/",
 "http://flickr.com/photos/raquelydavid",
 "http://last.fm/user/damog",
 "http://maggit.com.mx",
 "http://twitter.com/damog",
 "http://www.facebook.com/profile.php?id=670490388",
 "http://www.chess.com?ref_id=1380378",
 "http://www.chess.com",
 "http://www.chess.com/members/view/damog?ref_id=1380378",
 "http://www.chess.com/echess/create_game.html?uid=1380378&ref_id=1380378",
 "http://www.chess.com/home/game_archive.html?member=damog&ref_id=1380378"]
=> nil
>> pp HTML::Links.find("http://google.com").first(10)
["http://images.google.com/imghp?hl=en&tab=wi",
 "http://maps.google.com/maps?hl=en&tab=wl",
 "http://news.google.com/nwshp?hl=en&tab=wn",
 "http://www.google.com/prdhp?hl=en&tab=wf",
 "http://mail.google.com/mail/?hl=en&tab=wm",
 "http://www.google.com/intl/en/options/",
 "http://video.google.com/?hl=en&tab=wv",
 "http://groups.google.com/grphp?hl=en&tab=wg",
 "http://books.google.com/bkshp?hl=en&tab=wp",
 "http://scholar.google.com/schhp?hl=en&tab=ws"]
=> nil
>> get "debian.org"
[snip]
>> get("planeta.debian.net")[0, 100]
=> "<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd"
>>
Repository is, as usual, kindly hosted at GitHub on damog/goodies.

Next.

Previous.