Search Results: "spa"

9 January 2009

Raphael Geissert: PHPIDS bypass of the day


What to do when bored and reading the latest blog posts from The Spanner?
Once again I read about a PHPIDS bypass so I tried to find a way to bypass it too, so here it is:


you=this
,the=('a\
ler\
t'),you=you[the]
you('hello world \
by Raphael Geissert \
based on example by\
Gareth Heyes\
')
,/abc abc\
abc abc abc\
abc\
/,/abc abc\
abc abc abc\
abc\
/

8 January 2009

David Welton: Angry Perl Users

I work on LangPop.com for fun. I make a tiny bit of money from the adsense there, but nothing much to speak of. Unlike the TIOBE folks, I don't sell my data to anyone. So it can be a bit frustrating when people get all bent out of shape about the results. A case in point was my recent article here, entitled Python "Surpasses" Perl?. In case the title wasn't a dead giveaway, I'll spell things out:
  • I was poking a bit of fun at the idea that any statistics of this nature could pinpoint, to a specific month, the moment when one language "passes" another. They're way too fuzzy for that!
  • I point out that Perl is still quite popular. According to the Freshmeat metric I was reporting, Perl actually still has more code out there than Python.
And just to be clear: I don't use Perl or Python or really care which one is more popular. I mostly use Ruby these days, with Tcl, Java, and bits of C thrown in for fun. In other words: langpop.com is not an evil conspiracy to discredit Perl. However, I also had the temerity to suggest that Python might be a little bit more popular in terms of new things happening. Shock! Horror! Well, that certainly irritated some Perl users. They called my statistics lies, they misspelled my name (as well as the word "you", which is a lot simpler than my name by any standard), and some of them went off on rants that were pretty badly off target (SourceForge is never mentioned in Langpop or this online journal!). However, not once did they really get around to answering my points, or proposing any sort of valid alternative metrics. One of my favorite Linux/Free Software news sources, lwn.net published their 2009 predictions, and had this to say about Perl:
It will be a make-or-break year for Perl. If the Perl developers cannot either bring new life to Perl 5 or turn Perl 6 into something real, this language will, by the end of the year, have moved well down the road to "legacy" status.
As can be imagined, that set them off all over again. The mental image I get is a big angry bull, frothing at the mouth, and ready to stomp that damn red cape. Here's the problem, though, guys: the cape isn't what's holding you back. Trust me, I know. Once upon a time, when people spoke of scripting languages, they talked about "Perl, Python and Tcl". No Ruby, no PHP, although both existed. I happen to like Tcl a lot, and it measures much worse than Perl on all the popularity charts... worse than plenty of other languages, these days. Lately, I don't use it so much, but once upon a time I dedicated a lot of mental energy to wondering why things weren't going better. And in that community too, there were people who consistently held their fingers in their ears and yelled "LALALALA there is no problem!". In the long term, it's not an effective strategy: it makes everyone but the other True Believers think you're going off the rails, because the evidence is plain to see that things weren't quite like they used to be, that there are problems. And it doesn't really address the problem in a constructive way, either. Long term, what works is to produce good code, and get it in people's hands. The most effective way of dealing with problems of this nature includes these points:
  1. Not froth at the mouth any time you perceive a slight to your technology of choice.
  2. Talk up the cool things you're working on. Be positive, explain what's new and exciting.
  3. Acknowledge reality. For instance with Perl (or Tcl), it's plain to anyone that they aren't as popular as they once were. Don't deny it, but focus on the fact that they're still widely used, and lots of new code *is* still written in them, and that they're still the focus of a lot of work to improve them.
  4. Actually follow through. Jon Corbet's (Linux Weekly News) point is not a bad one - you can have all the internal releases and milestones and general fun hackery that you want, but at a certain point, you need to get working code out in front of people. This is something Ruby seems to be suffering from these days too (as observed from afar) - lots of next generation fiddling, and tons of good ideas and energy, but to me there is a lack of technology actually flowing my way in a timely manner. With the possible exception of JRuby, which is making lots of progress and putting good code out there for people to use.
To conclude things, since the Perl guys were so unhappy with Freshmeat, here are some statistics from Amazon and Craigslist, comparing about a year ago with the most recent numbers: As can be seen, both Perl and Python are dropping in Craigslist postings, but Perl is falling farther. As a final footnote, I am sorry about this journal not accepting comments past a certain point (10 days is the cutoff) - otherwise I get 100's of spam postings a day. Unfortunately, the software I'm using didn't deal well with alerting people to the fact that comments were closed. I filed a bug, and they're working on it.

David Welton: Angry Perl Users

I work on LangPop.com for fun. I make a tiny bit of money from the adsense there, but nothing much to speak of. Unlike the TIOBE folks, I don't sell my data to anyone. So it can be a bit frustrating when people get all bent out of shape about the results. A case in point was my recent article here, entitled Python "Surpasses" Perl?. In case the title wasn't a dead giveaway, I'll spell things out:
  • I was poking a bit of fun at the idea that any statistics of this nature could pinpoint, to a specific month, the moment when one language "passes" another. They're way too fuzzy for that!
  • I point out that Perl is still quite popular. According to the Freshmeat metric I was reporting, Perl actually still has more code out there than Python.
And just to be clear: I don't use Perl or Python or really care which one is more popular. I mostly use Ruby these days, with Tcl, Java, and bits of C thrown in for fun. In other words: langpop.com is not an evil conspiracy to discredit Perl. However, I also had the temerity to suggest that Python might be a little bit more popular in terms of new things happening. Shock! Horror! Well, that certainly irritated some Perl users. They called my statistics lies, they misspelled my name (as well as the word "you", which is a lot simpler than my name by any standard), and some of them went off on rants that were pretty badly off target (SourceForge is never mentioned in Langpop or this online journal!). However, not once did they really get around to answering my points, or proposing any sort of valid alternative metrics. One of my favorite Linux/Free Software news sources, lwn.net published their 2009 predictions, and had this to say about Perl:
It will be a make-or-break year for Perl. If the Perl developers cannot either bring new life to Perl 5 or turn Perl 6 into something real, this language will, by the end of the year, have moved well down the road to "legacy" status.
As can be imagined, that set them off all over again. The mental image I get is a big angry bull, frothing at the mouth, and ready to stomp that damn red cape. Here's the problem, though, guys: the cape isn't what's holding you back. Trust me, I know. Once upon a time, when people spoke of scripting languages, they talked about "Perl, Python and Tcl". No Ruby, no PHP, although both existed. I happen to like Tcl a lot, and it measures much worse than Perl on all the popularity charts... worse than plenty of other languages, these days. Lately, I don't use it so much, but once upon a time I dedicated a lot of mental energy to wondering why things weren't going better. And in that community too, there were people who consistently held their fingers in their ears and yelled "LALALALA there is no problem!". In the long term, it's not an effective strategy: it makes everyone but the other True Believers think you're going off the rails, because the evidence is plain to see that things weren't quite like they used to be, that there are problems. And it doesn't really address the problem in a constructive way, either. Long term, what works is to produce good code, and get it in people's hands. The most effective way of dealing with problems of this nature includes these points:
  1. Not froth at the mouth any time you perceive a slight to your technology of choice.
  2. Talk up the cool things you're working on. Be positive, explain what's new and exciting.
  3. Acknowledge reality. For instance with Perl (or Tcl), it's plain to anyone that they aren't as popular as they once were. Don't deny it, but focus on the fact that they're still widely used, and lots of new code *is* still written in them, and that they're still the focus of a lot of work to improve them.
  4. Actually follow through. Jon Corbet's (Linux Weekly News) point is not a bad one - you can have all the internal releases and milestones and general fun hackery that you want, but at a certain point, you need to get working code out in front of people. This is something Ruby seems to be suffering from these days too (as observed from afar) - lots of next generation fiddling, and tons of good ideas and energy, but to me there is a lack of technology actually flowing my way in a timely manner. With the possible exception of JRuby, which is making lots of progress and putting good code out there for people to use.
To conclude things, since the Perl guys were so unhappy with Freshmeat, here are some statistics from Amazon and Craigslist, comparing about a year ago with the most recent numbers: As can be seen, both Perl and Python are dropping in Craigslist postings, but Perl is falling farther. As a final footnote, I am sorry about this journal not accepting comments past a certain point (10 days is the cutoff) - otherwise I get 100's of spam postings a day. Unfortunately, the software I'm using didn't deal well with alerting people to the fact that comments were closed. I filed a bug, and they're working on it.

7 January 2009

MJ Ray: For Debian Developers who watch discussions about possible GRs

I believe that most debian developers ignore discussions of possible general resolutions like the current one, until/unless they look like reaching the required number of seconds to trigger a vote. It s hard to prove that a group is ignoring something, but disproof is simple: please could all DDs who watch pre-proposal discussions of possible GRs please email mjr-possiblegr at debian.org. I ll count with from -f possiblegr.mbox wc -l in a week or so, after filtering out any emails from non-DDs. Feel free to ask other debian developers to email if they watch reasonably likely places (planet, non-technical lists, ) for pre-proposal discussions of possible GRs. Please don t spam debian-devel or announce lists about it, unless you re expecting future possible GRs to be sent there. I d welcome a quick comment if you post it anywhere. If you can see a way to prove that a group is ignoring something, let me know. I realise that failing to disprove this isn t the same as proving the opposite, but I think it would still be useful.

Kartik Mistry: Aye Raju!


* After all things Raju did to make financial situation worst, I am dedicating this song to him.

6 January 2009

Andrew Pollock: [tech] On trying to use a Sansa e260 on Linux

I received a SanDisk Sansa e260 (v2) for my birthday. It was one of those presents that acquired for myself on behalf of someone else because it worked out easier that way. I bought it for cheap from Woot! I thought I'd use it to play podcasts while I'm at the gym. We've been listening to NPR for a while now, and there's lots of excellent programming, and it's all available in download for as well. Since I don't spend all that much time in the car, I'm not listening to much of it. The first mission was to upgrade the firmware. Getting the player into MSC (instead of MTP) mode with the original firmware was a little tricky, but I got there in the end. Then I had to locate the firmware in a Linux-installable format. I think I ended up resorted to using Windows, but I later found the raw firmware image without needing an installer. I'm using Rhythmbox at the moment to download the podcasts. I thought I'd try out its MTP support, just because it actually had it, but either Linux 2.6.26 has a problem with MTP, or the player is particularly flaky doing it with Linux, because I just seemed to get a pile of USB errors when connecting the device in this mode, so I'm sticking with MSC for now. Oh, and I had to put a .is_audio_player file in the root of the device to make it show up in Rhythmbox. I later discovered that it worked most optimally if I put
audio_folders=MUSIC/
folder_depth=2
in the file. I'm still trying to come up with an optimal way of managing the audio I want to listen to. It seems somewhat tedious to drag and drop all of the podcasts from Rhythmbox to the device. I kind of want some sort of plug and play auto-sync method. It seems I can't use a connection-triggered rsync, because some of the podcasts seem to get downloaded with non-FAT-filesystem-friendly filenames. Rhythmbox seems to take care of this by doing some sort of automagical renaming at transfer time. That said, not everything is showing up in the music list with sane names. There's a few things that have an unknown artist I think. That may be a deficiency in the ID3 tags of the MP3s themselves though... What I really want some way of getting home, hooking up the player and just getting what's new, without expending too much effort (at the time of hooking up the player, anyway. I'm happy to expend some effort getting to that point) The next weird thing is that the device wants to talk USB 1.1. I'm not sure what is at fault here, but I can, allegedly at least, make the device reconnect at USB 2.0 by removing and reloading the ehci-hcd module after connecting it. Here we have the device initially attached:
apollock@icarus:~$ lsusb -t
Bus#  5
 -Dev#   1 Vendor 0x1d6b Product 0x0002
Bus#  4
 -Dev#   1 Vendor 0x1d6b Product 0x0001
Bus#  3
 -Dev#   1 Vendor 0x1d6b Product 0x0001
   -Dev#   9 Vendor 0x0781 Product 0x7423
Bus#  2
 -Dev#   1 Vendor 0x1d6b Product 0x0001
   -Dev#   6 Vendor 0x413c Product 0x8103
Bus#  1
 -Dev#   1 Vendor 0x1d6b Product 0x0001
lsusb says that bus #3 is ID 1d6b:0001 Linux Foundation 1.1 root hub Then if I do a spot of presto-changeo...
apollock@icarus:~$ sudo modprobe -r ehci-hcd
apollock@icarus:~$ sudo modprobe ehci-hcd
apollock@icarus:~$ lsusb -t
Bus#  5
 -Dev#   1 Vendor 0x1d6b Product 0x0002
   -Dev#   3 Vendor 0x0781 Product 0x7423
Bus#  4
 -Dev#   1 Vendor 0x1d6b Product 0x0001
Bus#  3
 -Dev#   1 Vendor 0x1d6b Product 0x0001
Bus#  2
 -Dev#   1 Vendor 0x1d6b Product 0x0001
   -Dev#   7 Vendor 0x413c Product 0x8103
Bus#  1
 -Dev#   1 Vendor 0x1d6b Product 0x0001
Bam! It's moved over to bus #5, which lsusb says is 1d6b:0002 Linux Foundation 2.0 root hub Weird. I wonder if I should try running Rockbox on it? Seems not, it's only for v1 players. Oh well.

Joey Hess: proposing rel-vcs

I'm working on designing a microformat that can be used to indicate the location of VCS (git, svn, etc) repositories related to a web page. I'd appreciate some web standards-savvy eyes on my rel-vcs microformat rfc. If it looks good, next steps will be making things like gitweb, viewvc, ikiwiki, etc, support it. I've already written a preliminary webcheckout tool that will download an url, parse the microformat, and run the appropriate VCS program(s). (Followed by, with any luck, github, ohloh, etc using the microformat in both the pages they publish, and perhaps, in their data importers.) Why? Well,
  1. A similar approach worked great for Debian source packages with the XS-VCS-* fields.
  2. Pasting git urls from download pages of software projects gets old.
  3. I'm tired of having to do serious digging to find where to clone the source to websites like Keith Packard's blog, or cariographics.org, or St Hugh of Lincoln Primary School. Sites that I know live in a git repo, somewhere.
  4. With the downturn, hosting sites are going down left and right, and users who trusted their data to these sites are losing it. Examples include AOL Hometown and Ficlets, Google lively, Journalspace, podango, etc etc. Even livejournal's future is looking shakey. Various people are trying to archive some of this data before it vanishes for good. I'm more interested in establishing best practices that make it easy and attractive to let all the data on your website be cloned/forked/preserved. Things that people bitten by these closures just might demand in the future. This will be one small step in that direction.

Gunnar Wolf: Historias de la Historia del c mputo en m xico


Some months ago, I got a phone call from Rafael Fern ndez Flores. He wanted to interview me for a book he was working on regarding the history of computers in Mexico. The first computer in Latin America was installed in 1958 at my University (UNAM), and last year there were several activities conmemorating it. One of said activities is the publication of the book Historias de la Historia del c mputo en m xico, by Rafael Fern ndez and Margarita Ontiveros.
The book was printed in November, and Rafael gave me my copy in early December. It is quite an entertaining read - I mostly enjoyed the archaeological parts of it, referring to the 1950s and 1960s, and with many people that I know first hand (as my father is one of the founding researchers of the Centro de Investigaci n en Matem ticas Aplicadas, Sistemas y Servicios, CIMASS, now IIMAS).
I do believe, anyway, the book is focused too heavily on what happened in the large-scale computer world during a fundamental point in time for me (late 1970s, early 1980s) - It shows that the authors were very involved in the important projects the University set foot on, but they overlook fundamental pieces of the history. Very important developments were made in smaller venues (it was shocking for me to find only one mention, and just as a reference, to Fundaci n Arturo Rosenblueth and its great Centros Galileo, where many hundreds of kids (me included) learned to love computers, to program, and had a thriving socialization place. I also missed mentions of the BBS scene in Mexico, for which there are various exponents. And, just to single out one person, I found it absurd to have me interviewed and not to include La Mancha de la Calabaza que Ladra.
One of the last chapters -there are over 40 chapters, stemming from over 30 individual interviews- publishes the talk I had with Rafael. I must say there are small errata in its transcription (the first example that comes to my mind: I told him that one of the fruits of the OLPC project was the appearance of the now-popular netbooks, partly due to the appearance of lower cost parts, but I must reiterate I didn't say the Asus EEE is a part of said project). You can I am attaching my interview (as scanned, low-res images) to this post, in case you are interested.
Anyway - If the topic interests you, you will find many interesting passages, many passages you will surely laugh with and probably remember. The book is very well laid out. And it is a great joy to be part of it!

Julien Valroff: DSPAM-Community - first RC: call for testing

I was happy to receive Dov Zamir s email announcing the creation of a fork of the DSPAM anti-spam filter, called dspam-community. I have been using DSPAM for a while now, and am very happy with it, but was quite anxious that the project is left out by Sensory Networks (the company took over DSPAM in May 2007). I give all my confidence in the fork as the original author, Jonathan Zdziarski, is part of the new development team! I have prepared EXPERIMENTAL packages of the first RC of DSPAM-Community.

These are only meant for testing and should not be used in a production environment! I haven t tested these packages in depth (only tested installation or upgrade from previous official/unofficial packages).
There might still be problems due to the new names. I have built packages for i386 and amd64 architectures. They are built against current unstable distribution. They might be used on Ubuntu, but I haven t tested this yet. I have setup a quick page for these packages. All the applied patches are available for review and possible merge into git: http://packages.kirya.net/dspam-community/debian-patches/ To use these packages, add these lines to your /etc/apt/sources.list:
deb http://packages.kirya.net/dspam-community/ experimental main
deb-src http://packages.kirya.net/dspam-community/ experimental main
The repository uses secure-apt, you can hence check packages integrity thanks to my dedicated GPG key:
gpg --keyserver wwwkeys.eu.pgp.net --recv-keys FBABB737
gpg --export -a FBABB737   apt-key add -
or
wget -O - http://packages.kirya.net/Kirya.netDebianpackagesVerificationKey.asc   apt-key add -
Fingerprint: 97A0 69D9 A150 687E 599F F168 EEFB 43B2 FBAB B737 Feel free to test the packages and the first DSPAM-Community RC.
Any comment is welcome! To the official DSPAM packaging team: feel free to take these packages as a basis for your work in case you are interested in packaging DSPAM-Community!

Uwe Hermann: Running homebrew and open-source software on your Wii, using the Twilight Hack and the Homebrew Channel

Wii I've been owning a Nintendo Wii for quite a while now, but so far only played an occasional game of tennis or the like. Over the holidays I got The Legend of Zelda: Twilight Princess, which is a really nice game in itself. What is even more interesting though, is that this game allows you to run homebrew software on your Wii. What you usually want to do, is to use the so-called Twilight Hack to install the Homebrew Channel on your Wii (you'll need an SD card and the Twilight Princess game for this). Once the process has finished you won't need the game anymore though. So, here's a quick tutorial on how to install the Homebrew Channel. I basically followed the Setting up your Wii for Homebrew HOWTO from wiibrew.org. Requirements An SD card max. 2GB in size (not an SDHC card!), an SD card reader in your PC or laptop, and the Twilight Princess game (which you must play at least once). Backup First, you'll want to backup your Twilight Princess save games (if any). For that, we format the SD card (it needs a FAT16/FAT32) filesystem.
 $ mkfs.vfat /dev/mmcblk0p1 (you may need to change "mmcblk0p1" to whatever fits your setup)
Now insert the SD card into the Wii, start the Wii, go to Wii Options --> Data Management --> Save Data --> Wii and copy your Twilight Princess save games onto the SD card. Then, poweroff, remove the SD card, insert it into your laptop/PC and copy the save games to a safe place.
 $ mount -t vfat /dev/mmcblk0p1 /mnt
 $ mv /mnt/private ~/zelda_savegames
Twilight Hack and Homebrew Channel The Legend of Zelda: Twilight Princess Next up: download the Twilight Hack and Homebrew Channel, and put the files on the SD card:
 $ wget http://hbc.hackmii.com/dist/twilight-hack-v0.1-beta1.zip
 $ unzip twilight-hack-v0.1-beta1.zip
 $ cp -r private /mnt
 $ wget http://hbc.hackmii.com/dist/the_homebrew_channel-1.0.1.tar.gz
 $ tar xfvz the_homebrew_channel-1.0.1.tar.gz
 $ cp the_homebrew_channel-1.0.1/boot.dol /mnt
 $ umount /mnt
Now remove the SD card, insert it into the Wii and power-up the Wii. Go to Wii Options --> Data Management --> Save Data --> Wii, delete the Wii Twilight Princess save games, then copy the "Twilight Hack" save game from the SD card onto the Wii (you need to select the correct one for your region). Quit the menu, start Twilight Princess, load the "Twilight Hack" savegame and finally talk to the person in front of you (do not perform any other actions, or the Wii may crash!). Follow the remaining instructions on the screen and you're done. You now have an additional channel named Homebrew Channel where all your homebrew programs/games (and your own Free Software programs you may write/port) will appear. Installing homebrew software So far, there are no real applications in the Homebrew Channel, you need to put all the homebrew software you want on the SD card. There's a huuuge list of applications and games to choose from, many of them Free Software with source code, some however are binary-only. Basically, you put each application in a sub-directory of apps/ on the SD card, the most important file in every sub-directory is always boot.dol. If you have a boot.elf file instead, you'll probably need to convert it using the ELF to DOL converter. Restoring Twilight Princess save games You may now want to restore your Twilight Princess save games onto the Wii, you no longer need the "Twilight Hack" save game. Put your ~/zelda_savegames directory on the SD card and name it private/ again. Insert the SD card into the Wii and copy the save games on the Wii (similar procedure as above). Have fun with homebrew software on your Wii, or even better write your own software or port existing Linux applications/games!

5 January 2009

Ben Hutchings: Searching for invalid object code

J r my Bobbio suggested that I should explain how I looked for packages affected by a compiler bug (Debian bug 506713; gcc bug 38287). I don't claim that this is a particularly good way to do it, but here it is: First, I identified a pattern to search for. Unfortunately I don't really understand the cause or fix for the bug, but I did have the example which led to this bug report: Debian bug 490999. The bad code was a stack-pointer-relative load immediately after a stack allocation (SPARC save instruction) where the offset was not adjusted for the stack allocation:
save  %sp, -112, %sp
ld  [ %sp + 0x40 ], %i5
I generalised this to:
save  %sp, offset1, sp
...
ld  [ %sp + offset2 ], register
where offset1 + offset2 < 0. Of course, this may be valid if the intervening instructions include a restore, branch or store to the effective stack location that the last instruction loads from. I ended up allowing up to 10 intervening instructions and examining a disassembly to work out which cases were valid. I looked up the instruction encoding for these two instructions. Thankfully SPARC is RISC so they are simple and regular: I took the dumb but effective approach of scanning entire files for this pattern rather than only scanning their code sections. This seemed to work - I got no false hits for non-code - but might not work for other patterns that could match ASCII text. I wrote the scanning program in Python, which is my default choice of language unless I know it's going to be too slow. I was hoping to be able to read the code files into arrays, but unfortunately the Python array type only supports the native byte-order (SPARC is big-endian and I was intending to use an x86 which is little-endian). I tried reading into a tuple using struct.unpack, which does support explicit byte-ordering, but this used so much memory for larger files that the program swapped to a crawl. So finally I resorted to reading the file into a string, doing a string search for '\x9d\xe3', rejecting matches that weren't appropriately aligned, then unpacking and comparing the code words from the point of the string match. (In Python 3.0 I would have to use the bytes type for this, as str is a Unicode string type.) So that's how I scanned single files. The next step was to find, unpack and scan all the SPARC shared libraries in the archive. (This particular code generation bug is understood to affect only PIC code, and that is normally only used in shared libraries.) I wrote functions to search Contents-sparc for shared library files - assumed to match the pattern ([^\s]*/lib[^/\s]+\.so(?:\.[^/\s]*)?) - and to parse Packages to find the filenames for the packages containing those files. The latter uses the debian_bundle.deb822 module from python-debian. The last key function downloads and unpacks a package using wget and dpkg-deb. I could have used the httplib module for downloading but I correctly anticipated that I'd need to restart the script several times so I wanted to cache the packages which was easier to do using wget. So, that's the explanation. If you really want to see it, here's the code.

4 January 2009

Andrew Pollock: [tech] Mixing electricity and water: monitoring the cat water bowl with Nagios

(this is something I've had "in production" for many months now, I just haven't had the time or energy to do a proper write up about what I did) We have a cat water bowl, it looks like this: The cat water bowl Under "normal" circumstances, it usually lasts about seven days. So when our weekly routine is happening, we'll refill it on a Saturday whilst doing house chores, and it'll last until the following Saturday when it gets refilled. Unfortunately, sometimes our routine gets disrupted, and we forget. Sometimes, we travel and have a house-sitter, who may not pay as close attention to such things as ourselves. Once, one of our cats was licking the condensation off a chilled bottle of soft drink that was on the kitchen counter one evening, before we realised the water bowl needed refilling. Naturally, we felt like terrible pet owners. So I think it was some time around the 2008 Maker Faire, that I hatched the idea of having some sort of water sensor on the cat bowl, which would communicate to one of the various computers in house. At the Maker Faire, I bought a copy of Making things talk, and an Arduino starter kit, which consisted of a Diecimila board and a make-it-yourself proto shield. I also bought a little electronics starter kit, which consisted of a breadboard and various components, and a USB-TTL cable. I decided to use Bluetooth to communicate with the board, as I already had my MythTV setup using a Bluetooth keyboard and mouse, and it was within range of the water bowl. I decided against using Zigbee, because I didn't know anything about it, and I didn't want to add (or learn about) yet another wireless infrastructure just for this project. I should point out that I know very little about electronics. I'd never owned (or really used) a soldering iron until I embarked on this project. I took a basic soldering class at the Tech Shop, but I'd already assembled the proto shield by the time I took the class, so I'd pretty much figured it out. I had a very naive vision that I could just basically shove two wires in some water and it'd close a circuit and that would be my water sensor. Of course this didn't work, so I started hunting around on the Internet for a circuit that would do this. I happened upon a circuit (I don't seem to have retained the URL, so I can't link to it), which just consisted of a couple of transistors and some resistors. So I headed off to Fry's to try and buy the transistors I needed. I quickly discovered that I didn't have sufficient information to identify the transistors that I wanted, but I did happen to stumble upon a cheap assemble-it-yourself water alarm. It consisted of a PCB, and a transistor and some resistors and a buzzer. I bought a couple of these instead. Between studying the PCB and the circuit diagram that came with the alarm, I was able to reproduce it on my breadboard instead of on the PCB. Sure enough, placing the two probes in water closed the circuit. I replaced the buzzer with an LED so I could see what was going on. I attached the circuit to the Arduino proto shield, and had it feed into one of the digital I/O ports. I wrote some quick and dirty Wiring code so that when water not present (i.e. the circuit was open and no current was detected on that I/O port) the LED was switched on. Really at this point, I didn't need a microcontroller, I could have presumably achieved the same thing with a NOT gate. At this point, I wanted to make the sensor remotely queryable. I bought a BlueSMiRF Silver Bluetooth modem, which I attached to the TX and RX lines of the board (I first configured it by attaching the USB-TTL cable to it and using Minicom on my laptop). I extended the Wiring code to provide a rudimentary prompt, and accept a command to check if water was present. I think around this time it also dawned on me that I could use the digital I/O pins as a switch. When they're "high" they provide power. So rather than constantly running a current through the water, I only needed to briefly power up the water detection circuit, see if the circuit closed or not, and then report if water was present if it did. I much preferred this, as at the time, I was endeavouring to power the whole sensor off a 9 volt battery. I figured I'd get much better battery life if I wasn't running a current through the water the entire time. I should point out that I did some "tongue tests" in a glass of water while the circuit was powered up, and couldn't detect a difference between when the circuit was on or off. The last thing I wanted to be doing was zapping the cats! At this point, the Wiring and Arduino work was pretty much complete. I setup ser2net on the MythTV server, so that I could just connect to port 4000, and be connected to the water sensor.
apollock@icarus:~$ telnet teevee 4000
Trying 172.16.0.9...
Connected to teevee.andrew.net.au.
Escape character is '^]'.
ser2net port 4000 device /dev/rfcomm0 [9600 N81] (Debian GNU/Linux)
waterbowl> s
Water is present
waterbowl>
telnet> q
Connection closed.
apollock@icarus:~$
The Wiring code running on the Arduino board is checked in here. Next, I wanted to monitor this with Nagios. One thing I found with the Bluetooth connection was that it wasn't all that reliable. Not every connection to port 4000 resulted in a connection with the water sensor. I elected to write some standalone code that submitted results to Nagios by way of a passive check, rather than having Nagios try to actively monitor it. Again, trying to conserve energy, I decided to only check the sensor once every 8 hours. So I wrote a Python daemon, which tried to be as robust as possible. If at first it doesn't make a connection on its 8 hourly schedule, it keeps trying until it does. Nagios itself has some freshness detection for this monitor, so if no passive result is submitted within 8 hours and one minute, Nagios alerts that something is possibly wrong with the sensor itself. (The original intent was to deal with the situation where the battery went flat) This is the Nagios service definition I've got. Some of it may be unnecessary or redundant:
define service  
        host_name                       teevee
        service_description             Cat water bowl
        check_command                   check_stale!2!"Check water bowl monitor is on and reachable"
        normal_check_interval           480
        notification_interval           240
        active_checks_enabled           0
        check_freshness                 1
        freshness_threshold             28860
        max_check_attempts              1
        check_period                    24x7
        use                             generic-service
        stalking_options                o,w,c
        contact_groups                  everyone
 
The code for the Python daemon is checked in here. So everything was going swimmingly, apart from I couldn't get even 24 hours of monitoring on my conservative 3 times a day schedule, and the Bluetooth modem configured for all of its power saving options, and the water sensor itself only being powered on when it was performing a check. There wasn't much more I could do to try and reduce power consumption, at least with my limited electronics knowledge. Perhaps using Zigbee instead of Bluetooth would have helped, but I still think I'd have been going through 9 volt batteries more quickly than I'd have liked. I also had a brief foray into solar powering the board. I thought maybe the lighting in the house would be sufficient to power the board. Of course this didn't work out. I'd also have needed to make the code running on the board more self-sufficient, and have it just provide a water status indication whenever it had enough power to do so, instead of being a fairly dumb device like it currently is. This all felt a whole level more complicated, and out of my league, and I wasn't interested in attempting this sort of remote-sensing exercise at this time. The Arduino board can also be powered by USB, so as I already had some long USB type A to type B cables (that had funky lights in them to boot), I decided to just get a wall wart that had a USB type A receptacle, and powered the Arduino board that way. So much for being completely "wireless". (I could have also gotten a general purpose DC power supply that was capable of spitting out 9 volts, but I doubted I'd get one with a sufficiently long cable, which is why I went for the USB-powered option, as I already had a cable long enough) Speaking of wires, the most challenging part was the probes for the water sensor. As they were going to be permanently in the cats' drinking water, I didn't want to contaminate the water with them. I figured plain untinned copper wire would be okay, since water pipes are copper. Finding untinned, unstranded copper wire was a real challenge. I started out using some FM antenna cable, but that was stranded, and the shielding was a nightmare to strip. It was also reasonably difficult to make go where I wanted it. What I really wanted was something more solid, that would flex and stay where I put it. I cannibalised a spare IEC power cable, but it was also stranded copper wire. I finally managed to obtain some solid-core CAT-5 cable from a hardware store, and this has worked exactly as I wanted. I haven't done any further work on the setup since getting the CAT-5 cable for the probe. Further improvements that I'd like to do at some point: The finished product I had a lot of fun with this project. I had a real sense of achievement, being able to go from concept to completion, and learn a few things about electronics along the way. I'm normally not a fan of messing around with hardware. Some photos of the project are here.

Sami Haahtinen: About Usability

Some people consider the gnome usability guidelines a nuisance and some consider certain applications way too simplistic. While it is really hard to get the usability right, it's well worth it. We need to keep in mind that as computer oriented people we tend to see things differently. Things that are simple to us aren't really that simple to the "normal people". But one of the simple things we can do the insure that the software we write serve the people it's designed for is to remove all not needed pop-ups and questions. A good way to detect these would be to ask yourself why would a user choose anything else but the most logical option. It's kind of hard to explain, so lets pick an example: Firefox is updated, the first question it usually asks after upgrade is about incompatible extensions. The user is presented with choices, check for updates or cancel. Now, we are dealing with internet browser so the user should be connected to the internet so there is no problem with checking for the updates, we can rule out that scenario. The other scenario that I can come up with is that a developer doesn't want to update some certain extension. So at least I can't come up with any reasonable scenario why someone would want to select anything else but to upgrade the main option of upgrading the extensions. Why not just leave out the option and instead do the upgrade automatically. If you wish to be transparent, you can show the user that you are doing the upgrade. Or you can do like some applications do and just do it without bothering the user with the options. I know I sound like a Google fanboy, but Google generally gets this right. Their applications skip out all upgrade related notices and just do the upgrade. Regular user doesn't want to upgrade because the user has been scared with incompatibility notices and upgrade checklists for so long. Just going ahead with the upgrade in complete silence keeps their software up to date as well. Another example would be from few years back: The Ubuntu installation. Back in the day Debian was working on Debian Installer, which is also used as the main installer in Ubuntu alternative installation media. Debian Installer is capable of doing most things silently, but with Debian it by default asks a lot of questions. It doesn't matter, since most people who install Debian can be categorized as developers. But in my opinion, the thing that made Ubuntu a success was that it doesn't ask the questions that can be answered without asking the user. So, back to usability. There are basically 2 camps, the "normal" users and the developers. Developers want and need to see a lot of the backend behaviour, just to debug problems. Currently a lot of the open source software is focused towards developers while they are gaining grounds on the "normal" population as well. We should start focusing on the users for a change.

3 January 2009

Julian Andres Klode: python-apt: Deprecation of mixedCase names, re-organisation of apt.progress, and more


[as sent to deity@lists.debian.org] Hello, while hacking a bit on python-apt in the last days, I formulated some
proposals which I want to present to you. I have already implemented proposals I and VII, started with the implementation
of proposal II, and have partially implemented proposal III, all in
my local branch. I could implement the rest next week, and this could
result in python-apt 0.7.9~exp2 (at least this is my wish, I do not
have commit access to the python-apt repository, and this mail is not
discussed yet :) ). My local branch is not online yet, but I m working on it. It is based
off debian-sid revision 202 (which is 0.7.9~exp1). I would have also suggest to (at least partially) follow PEP 7 (C code),
but re-indenting the whole extension code is not a good idea. Before I go on, I would like to know your opinions about these proposals,
what you like and what not, your suggestions, etc, etc List of Proposals
 I   Deprecation of mixedCase naming conventions in apt module
 II  Re-organisation of the apt.progress module
 III Language updates
 IV  Unification of testing code
 V   Cleanup of the code
 VI  Using unique variable names throughout the modules
 VII Some other changes
 
I. Deprecation of mixedCase naming conventions in apt module In order to comply with PEP 8, we should start to change our names
from mixedCase to lowercasewithunderscores style. This style is
more common in the Python world, and used eg. in the Python library. I hereby propose that:
  1. a new module apt.deprecate is introduced, which provides functions
    for compatibility and a class named DeprecateMixedCase
  2. Most classes are changed to subclass DeprecateMixedCase instead of
    object.
  3. All methods, attributes and variable names are changed from the
    mixedCase style to the lowercasewithunderscores style.
  4. apt_pkg will be changed to call the new methods on progress objects,
    and fall back to the old names if methods with new names are not
    available.
How backward compatibility is implemented: The DeprecatedMixedName class provides the methods getattr()
and setattr(). These classes will forward all access to attributes
with mixedNames to the ones with lowercasewithunderscores names. In case of methods, the getattr() method returns a helper function
which fixes all keyword arguments with mixedCase names and than calls
the right function. As this only works for methods with mixedCase names, methods which have
mixedCase-named args, get the argument **kwds, and its content will be
checked for mixedCase-named arguments. [On the other side, given that
the API is marked as not stable, we could also drop that level of
compatibility.] Schedule:
  1. Implement all the changes in an experimental version of python-apt
    as soon as possible.
  2. (A) Once all packages have been adjusted to use the new names, remove
    the old ones (about 50 binary packages, and I would write the needed
    patches). This would Breaks: about 50 packages.
    or
    (B) After the release of Debian Squeeze, remove the compatibility
    functions (and the deprecate module from apt). This is the same
    as (A), but does not require Breaks on 50 packages.
Known Issues:
  1. The methods will not be available using their oldName at class
    level, only at object level. This means that some code subclassing
    apt.* classes may fail. Most code will not break, however.
II. Re-organisation of the apt.progress module The apt.progress module is not very consistent in the way classes
are named. For example, there is OpTextProgress but TextFetchProgress. Therefore, I hereby propose that:
  1. For each of the for types of progress ( Cdrom , Fetch , Install , Op ),
    a new class is introduced with the name: Abstract%sProgress % progress
  2. For each type of progress a class Text%sprogress % progress is introduced,
    which subclasses Abstract%sProgress and provides textual output
  3. For further classes, a general naming scheme is introduced, which is
    %(outputtype)s%(progresstype)sProgress [whereas (1) and (2) implement
    this naming scheme]
Schedule: Same as in proposal I. III. Language updates The apt package has been written some time ago, and has not been updated
to make use of the latest language features. Therefore, I propose that:
  1. all code using old language features is adjusted to use language
    features as of Python 2.5.
    1.1 Examples include the use of decorators, and the use
    of str methods instead of functions of the string module.
  2. The with statement will be used where appropriate, and that
    apt_pkg.*Lock() gain support for being used as context managers.
Schedule: Implement as soon as possible. IV. Unification of testing code Most modules ship test code which will be called when the module is
run as a script. Therefore, I propose that:
  1. All testing code in the apt module is removed
  2. All the tests in the tests directory of python-apt
    are converted to a unified testing suite.
  3. Testing code which has been removed as part of (1) is
    reintroduced in the relevant test module, if appropriate.
Schedule: Implement as soon as possible. V. Cleanup of the code In accordance to PEP 8, which is also named as a rule in apt/README.apt,
code will be reformatted where needed. This includes 2 lines between module-level definitions, spaces after
, , comparing to None using is instead of = , and more. Schedule: Implement as soon as possible. VI. Using unique variable names throughout the modules I hereby propose the following names:
 cache    - An apt.cache.Cache() object
 pkg      - An object returned by cache[str()] (in special cases)
 pkg      - An apt.package.Package() object
 cand     - An object returned by depcache.GetCandidateVer()
This would remove some confusion for developers, especially
if you see that apt.Package() objects are named cand in debfile,
but pkg everywhere else. Schedule: Implement as soon as possible. VII. Some other changes Here is a list of other changes I have already made in my branch:
  • Remove apt.package.Record(), it behaves exactly like the
    object it is wrapping (apt_pkg.ParseTagFile() result)
  • Move the Origin class out of the Package() class and at
    the module level.
  • If a package should be marked for upgrade, but is not upgradable
    raise an exception (which will be apt.package.NotUpgradableError)
Schedule: Implement as soon as possible. Posted in Debian

Steve Kemp: Some of us are just better at hiding it, that's all.

Since I managed to get my blogspam plugin listed on wordpress.org I've seen a surge of interest. In the past few hours capture stats are:
Good comments1
SPAM comments100
Almost exactly 1% of submitted comments are OK, and a SPAM rate of 99%. Either forum and blog spam are even more rife than I'd expected or I am over-zealous! Anyway that's enough on this topic for a while, if you want to follow the work you know where to look and if you don't my writing about it again will just drive you mad.. ObFilm: The Breakfast Club

1 January 2009

Sandro Tosi: Compress MySQL Tables

I got this long running task that (fsck!) died almost at the end of the job, and it logs its operation in a MySQL table.

Since I want to keep those "partial" results for further inspection (even after I'll fix and let the tool run again until the end) I don't want to waste disk space (it's ~650mb), so why don't compress that table?

With MySQL (and MyISAM tables like mine), it's pretty easy: go into the directory of the db schema:
# cd /var/lib/mysql/tabs_compare/
# ls -l
total 659500
-rw-rw---- 1 mysql mysql 65 2008-09-21 00:35 db.opt
-rw-rw---- 1 mysql mysql 12840 2008-12-25 00:44 log_1230162297.frm
-rw-rw---- 1 mysql mysql 675299244 2009-01-01 20:34 log_1230162297.MYD
-rw-rw---- 1 mysql mysql 1024 2009-01-01 20:37 log_1230162297.MYI
Pretty big :) Now, check the table:
# myisamchk log_1230162297.MYI
Checking MyISAM file: log_1230162297.MYI
Data records: 4293747 Deleted blocks: 0
- check file-size
- check record delete-chain
- check key delete-chain
- check index reference
- check record links
Hopefully there is no error, so we can compress that table:
# myisampack log_1230162297.MYI
Compressing log_1230162297.MYD: (4293747 records)
- Calculating statistics
- Compressing file
43.05%
And the result it's really nice:
# ls -l
total 375572
-rw-rw---- 1 mysql mysql 65 2008-09-21 00:35 db.opt
-rw-rw---- 1 mysql mysql 12840 2008-12-25 00:44 log_1230162297.frm
-rw-rw---- 1 mysql mysql 384558282 2009-01-01 20:34 log_1230162297.MYD
-rw-rw---- 1 mysql mysql 1024 2009-01-01 20:49 log_1230162297.MYI
The compress command didn't even take a full core on my machine and almost took 20secs to complete.

Now rebuild indexes (if any) and recheck the table to be error free:
# myisamchk -r log_1230162297.MYI
...
# myisamchk log_1230162297.MYI
...
now tell MySQL to reload those table with:
mysql> flush tables <schema>;
Keep always in mind that compressed tables are read-only, so no DML commands, while DDL are allowed (so you can DROP it, for example).

Amaya Rodrigo: Mooch @ MAD


Mooch @ MAD
Originally uploaded by Amayita.

31 December 2008

Andrew Pollock: [tech] On unlocking a Vonage-provided Linksys RTP300

Well that was a bit of process. The best set of instructions I found were here. I think a lot of the other instructions I found assumed you were at firmware version 1.00.62. The CYT unlocker program seems to only work with this firmware version. The CYT unlocker program is a Windows console-mode program. It seems to run under Wine, but I switched to Windows before I realised that possibly my problems running it under Wine were related to the firmware version on the RTP300 and not a problem talking to the network. All of the remaining steps on the above page could have been performed on Linux. In fact, the CYT unlocker program merely makes a HTTP POST to a specific URL (and I think it could just as easily be an HTTP GET) and then starts up a webserver on port 2400 to hand an XML blob of provisioning information when the RTP300 requests it as a result of the HTTP POST. A Linux version would be easy to implement. In all of the futzing around, I noticed this gem in the HTML source of the web interface:
*********************************************************
*   Copyright 2005, CyberTAN  Inc.  All Rights Reserved *
*********************************************************
This is UNPUBLISHED PROPRIETARY SOURCE CODE of CyberTAN Inc.
the contents of this file may not be disclosed to third parties,
copied or duplicated in any form without the prior written
permission of CyberTAN Inc.
This software should be used as a reference only, and it not
intended for production use!
Now that I've unlocked it, I guess I should try and make it talk to my Asterisk box...

Russell Coker: Tidal River

Tim (a member of my local LUG) writes about some observations he has made of a nearby river and speculates on a tidal bore-like phenomenon [1]. One thing that surprised me was how short the list was on the Tidal Bore Wikipedia page [2], and the fact that is it missing an entry for Tidal River at Wilson s Promontory [3] (where my family often spent the Christmas holidays when I was young). Some of the tidal bores are described as having a wave as high as two meters, Tidal River is not so impressive, my observation was that during the 80 s it was about 40cm near the mouth of the river. The area near the river mouth had many bends when I last saw it which absorbed some of the energy of the wave (but I expect that the river changes course constantly so it might be straight from time to time). On one occasion I River Surfed [4] about 500 meters upstream at Tidal River on a surf-mat (an inflatable surf-board). I have searched for research into this issue, the Tidal Bore Research Society [5] seems to just maintain a list of tidal bores and not do any real research. Pierre Lubin, Stephane Glockner, and Hubert Chanson published a paper titled Numerical simulation of turbulence generated by a tidal bore [6]. Hubert Chanson at the University of Queensland has written an interesting paper titled Physical Modelling of the Flow Field in an Undular Tidal Bore [7]. Hubert seems to have published more papers related to tidal bores than anyone else (or at least more papers that are publicly accessible).

Biella Coleman: odd-old-san-juan-dog



odd-old-san-juan-dog, originally uploaded by the biella.
We may not have did up my house like this OSJ one but you did a fine job > thanks (and thanks for laughing with me over this odd dog and his portrait!)

Next.

Previous.