Search Results: "Andrew McMillan"

16 September 2010

Andrew McMillan: Changing an old habit

I'm a keen amateur photographer and have been since I was at school. Of course back then it involved smelly chemicals, strange lights and dark rooms. The expense and hassle of film eventually did it in for me and in 2000 I bought my first digital camera. After a series of increasingly sophisticated devices, back in 2007 I bought a Nikon D80 and soon thereafter I discovered that you can do things with raw camera files that have some analogies to the darkroom processing of my youth. For most of the time since then I've used UFraw for processing my raw images, combined with increasingly sophisticated batch files to harass it into shape. It has increasingly frustrated me that UFraw doesn't do several things that are essential to processing most files, since once the image leaves that programming it usually ends up in an 8-bpp processing path. It seems to me that the saving the processed image into an 8bpp format like jpeg is the last step before leaving the image alone. Of course this can't always be the case since sometimes a judicious use of the 'Resynthesizer' plugin can make some unsightly elements just disappear, or one needs to carefully apply some censorship to the 14 year old in the background so that the 7 year-old's parents can put the photo on the mantelpiece without worrying about what Granny will say about how much better children's manners were in her day. I'm hoping too, that when I have grandchildren I'll be able to tell them that 'in my day we only had 8 bits per pixel, and we were lucky', but The Gimp isn't there yet, no matter how much promise Gegl seems to offer. The thing that has particularly annoyed me is sharpening. UFraw doesn't, and as a result my workflow has either involved applying some blanket sharpening, much like the camera does, or moving the image into a second tool just to apply some sharpening. This week has been particularly demanding, as the school asked me to 'take some photographs of the production'. So of course that's a thousand-odd photos that I want to do a cull down to a hundred or so, and then I want to quickly do justice to the good ones so I can move on to other stuff. Well this time my frustration with UFraw hit that ceiling that got me to try something else. I've done this in the past, and found myself going full circle and ending up right back at UFraw with a few extra lines added to my script, but this time I tried rawstudio and in processing the 'can you do the class shots overnight please' test it produced much nicer results in much less time than I have ever been able to with UFraw. So I've switched, and the more I use it the more convinced I am that UFraw will be getting uninstalled pretty soon. That's not to say that rawstudio is perfect: it isn't. In particular it has some user interface failures that are kind of annoying to me, but they aren't enough to stop me using it. In fact I liked it so much that I immediately had to pull the nightly repository and build that, just to get the new saturation handling. The user-interface niggles that do get to me are: Things that I immediately love, over what UFraw gave me: Things that I immediately miss from UFraw: There are a bunch of things that RawStudio also does that I haven't particularly found a use for yet, but which I suspect I will quite like when I do. The image classification and stuff seems likely to be useful, for sure. In choosing to switch RawStudio I also looked at RawTherapee, which has recently relicensed under the GPL, but I chose RawStudio because it has vastly better performance on my laptop, and because I found the interface much more immediately accessible. RawTherapee also exceeds what UFraw can do in many ways, and looks like it might be useful for the really difficult cases, but learning to drive it with any facility could take some time. Especially if you only have an i7 740QM with 6G RAM, it would seem.

8 September 2010

Andrew McMillan: Making my laptop quiet

A few days ago I talked about how noisy my new Dell Studio 15 was but I can now report back with the beginnings of a solution to that problem, and it doesn't appear to be ACPI related. The first clue I needed was that if I switch to the proprietary fglrx drivers for the Radeon the fan quite quickly drops off to a much more reasonable level. It seems the fglrx drivers have issues, however, in particular I get big black patches on my screen. This video corruption happens especially in Firefox, but sometimes in other applications as well. They also appear to screw up my suspend/resume, which is probably even more annoying to me. The second clue that I needed was that Radeon power management support has only just made it into recent kernels. Thanks to Michael Kirkland for providing me with both of those clues :-) Looking under /sys/class/drm/ I find a whole bunch of stuff, but in particular there are /sys/class/drm/card0/device/power_method and /sys/class/drm/card0/device/power_profile. Looking through the kernel source code I can see that power_profile can be set to low, mid, high, auto and default, while power_method can be set to either dynpm or profile. Trying out all of these values, it seems I get the quietest result with the profile method and either the low or mid profile. The dynpm method is nearly as good, and I would think it should really be the default for a 'Mobility' chipset. From the detailed benchmarking that Phoronix did I wonder if it shouldn't be the default for everyone. For myself, I see some small 'tearing' artifacts occasionally when running with the low profile. These disappear when I run with the mid profile, and since that seems to have pretty much the same temperature (and noise) results I'll go with that one. Though the laptop often does still make more noise than I would prefer it to, it is no longer annoying everyone in the room. Not unexpectedly this seems to have a huge impact on power use, too. It appears that the laptop should now give me around 4.5 hours when I do everything I can think of to lower the power use, whereas before it was more like 2.5 hours. Now I guess I can get back to hacking on DAViCal...

5 September 2010

Andrew McMillan: I guess I should have known better

Really, I should have known better than to buy myself a rather 'bleeding-edge' laptop. I suppose I've been lucky with my last couple of laptops and was thinking that pretty much everything works on Linux. Or maybe I'm being picky, because in fact pretty much everything does work on my new Dell Studio. I bought this laptop mainly because it has a 1920x1080 screen, without being a 17" monster. I had such a monster about four laptops ago (an enormous Sony Vaio weighing in at around 5kg and with a 1hr battery life) and it was an disaster: lots of weird proprietary stuff that never worked, like the external speakers in the docking station. But I loved that screen right up until the day Fraser threw a pair of scissors into it. With the advent of mass-market 1920x1080 screens I've been waiting for them to arrive in laptops at a reasonable price, and the availability of this screen in a few possible laptops seemed to me to be that point. It's a Dell Studio 15, and as well as the nice screen, it comes with a new Intel i7 Quad Core CPU, 6G RAM, 640G hard drive and a Blu-ray drive. While there are a few similar models around this one seemed to have the best mix of features-for-weight-and-price, so I bought my first Dell. I built a USB key with the latest Debian Installer from Squeeze and installed that and everything came up pretty well. I had to use the broadcom-sta driver for the wireless, since the BCM 43224 is not (yet) supported by the free b43 driver (though from looking the mailing list people seem to be working on it, and it looks like it might not be too far away). Everything seemed good. X came up on the (georgeous) screen. I don't care too much about 3D performance, but it was nice to note that the ATI 5000 series is expected to have full support as soon as it's integrated into the right places - it seems the code is written, and public, it just isn't incorporated into the 'radeon' driver quite yet. The fact that ATI is nowadays a free-software friendly company was one of the reasons for choosing this, in preference to an NVidia-based laptop. But it seems that it isn't all roses. The fan in the laptop seems permanently on, and not on, in a quiet-just-barely-audible way, but ON, in an 'IN UR LAPTOPZ KOOLIN UR P0RCESSAZ' kind of a way. Too damn noisy for me to concentrate in a quiet room. I've investigated what's going on, and it seems likely there are some ACPI misunderstandings happening. Some googling and some pecking around in /sys makes me wonder if the laptop expects some kind of configuration choice between favouring Active cooling and Passive cooling, and it's defaulted to the first one. Perhaps you get better benchmarks that way. Linux ACPI also seems to only half understand it, it has two thermal zones, and one of them always appears with 0 temperature. I can see set points for the fans, but have no control over them. In particular this got me out reading the ACPI 4.0 specification, and it explicitly mentions this choice of favouring 'Active' vs 'Passive' as being done by configuring the passive trip point setting to a higher value than the low & high active trip points. In the ACPI 4 spec (page 409) it says:
To implement a preference towards performance or energy conservation,
OSPM can request that the platform change the priority of active cooling
(performance) versus passive cooling (energy conservation/silence) by evaluating
the _SCP (Set Cooling Policy) object for the thermal zone or a corresponding
OS-specific interface to individual devices within a thermal zone.
This laptop is showing 'passive' at 95000 millidegrees celsius, and 'active' at 55000 / 75000 so right in line with the suggestion. However I don't see any way in Linux to change these trip point values, and in any case I'm not convinced the laptop is actually obeying them at all. It should be that if I do nothing on the laptop for a while the temperature would surely drop below 55 degrees celsius. Surely a modern laptop will sit at under 40 degrees when it's quiescent, and the temperature sensor that does show me something sits at around 30 degrees in this too, 26.8, to be overprecise. That one doesn't budge either. Looking in /sys/ I can't for the life of me find a way to set that cooling policy, and when I disassemble the DSDT it looks like a noop... Thinking that perhaps there was a newer BIOS, I looked at the Dell site and found there was one. Great! Investigating even further, it seems Dell hardware has some interesting capability via libsmbios and some utilities which will let me install my BIOS from a Linux system... even better! Linux is a second-class citizen though, and when I look for the Dell BIOS Hdr file I need there is no version of the file for this laptop - let alone the current BIOS version. And when I download the BIOS from Dell's website and unzip the first layer of packaging I find that both of the files inside will only run on Windows. I added a FreeDOS boot stanza to Grub 2 in order to discover this, also discovering in the process that this is not well explained, and I could not find how to do it on the FreeDOS or Grub2 websites, so in passing, here is what I added into /etc/grub/40_custom :
menuentry 'BIOS Flash 1558 A09' --class os  
        set root='(hd0,msdos1)'
        linux16 /boot/memdisk
        initrd16  /boot/1558_A09.img
 
Which certainly does boot FreeDOS from Grub2, into the image built from the 8M FreeDOS base image, and which I called '1558_A09.img' because that seems to be one of Dell's reference numbers for this model / BIOS version. So still no BIOS update for Andrew, it seems, and the laptop is probably going to continue to be noisy for some time. Possibly forever. Suspend/Resume works though. Kernel mode-setting for the ATI Radeon Mobility HD 5450 (or whatever it is) seems to work. Building kernels seems to work. The Blu-ray drive seems to work, at least as far as reading CDs - just because I got a laptop with a "Full HD" screen doesn't mean I watch movies on it: those pixels are for programming on! In any case, after reading the literature of pain about playing Blu-ray disks on Linux I'm convinced that we won't be shelling out to buy one any any time soon. After a bit more frustration I did manage to get the BIOS installed, using the hack I found here of running the Windows BIOS update under Wine and copying the BIOS file out of the temp directory while the error message is displayed. I could then download a random DOS-installable Phoenix BIOS to get copy of phlash16.exe and put those two files on the FreeDOS image I created earlier. Finally I was able to put that learning about how to boot FreeDOS from Grub2 to good use! Unfortunately, having jumped through all of those hoops (including the frustration of trying to build and use Dell's firmware-extract tools on a Debian system), the fan continues to grind away at annoying volume, with no way that I have been able to find to control it, and the temperature sensors also seem wrong, since one doesn't budge from 0 and the other doesn't budge from 26800. During boot I get a bunch of error messages, like:
[    1.049936] \_SB_.PCI0:_OSC invalid UUID
[    1.052202] pci 0000:00:1c.0: PME# supported from D0 D3hot D3cold
[    1.052210] pci 0000:00:1c.0: PME# disabled
[    1.065228] pci 0000:07:00.2: PME# supported from D0 D1 D2 D3hot D3cold
[    1.065237] pci 0000:07:00.2: PME# disabled
[    1.079086] Unable to assume PCIe control: Disabling ASPM
[    1.086412] HEST: Table is not found!
[    1.249906] ACPI: Fan [FAN0] (off)
[    1.249959] ACPI: Fan [FAN1] (off)
[    1.250093] [Firmware Bug]: ACPI: ACPI brightness control misses _BQC function
[    1.259118] acpi device:02: registered as cooling_device2
[    1.262132] thermal LNXTHERM:01: registered as thermal_zone0
[    1.262139] ACPI: Thermal Zone [TZ00] (27 C)
[    1.262561] thermal LNXTHERM:02: registered as thermal_zone1
[    1.262571] ACPI: Thermal Zone [TZ01] (0 C)
[    1.262598] ERST: Table is not found!
[    2.769664] i801_smbus 0000:00:1f.3: PCI INT C -> GSI 18 (level, low) -> IRQ 18
[    2.769670] ACPI: resource 0000:00:1f.3 [io  0x1840-0x185f] conflicts with ACPI
                         region SMBI [bus 1840-184f pref window disabled]
[    2.769672] ACPI: If an ACPI driver is available for this device, you should use it
                         instead of the native driver
I don't yet know what this all means, so I guess I'll be signing up for the linux-acpi mailing list to see if there's anything I can do to get things working better. Any pointers will be gratefully received :-) Right about now I wonder if maybe I should have gone for the HP or the Asus, even if they were a little heavier and a bit more expensive and next time I get a new laptop I must remember not to spend more than $900 (USD$600) on it. [Updated: 2010-09-08... Noise Problems Solved

16 August 2010

Andrew McMillan: Thank you Debian

I do like that some of my friends in Debian have conspired to set up the website http://thank.debian.net/, and I composed this little Haiku in their honour:
Debian is fun
for seventeen years.
Beautiful balloons.
I'd also like to thank everyone involved in Debian over the years for being such a lovable, friendly, enjoyable and welcoming bunch of people to work with. It's tempting to single out people for special mention, because there are people who do exceptional work in the project, but the list would go on for so long, and I would be afraid of missing out someone who would be obvious five minutes later, and then I'd have to come back and edit this post repeatedly with the names of more and more people over the coming days. We all know exactly what a chore that kind of maintenance is. So I'll just give up completely and thank everyone, and if you think you deserve a special thanks, you probably do, and I'm sure you're on that list in my head. When we meet, as I hope we shall, let me shake your hand, lend you my ear and buy you a $BEVERAGE of your choice. Thank you,
Andrew McMillan.

23 June 2010

Andrew McMillan: Using dnsmasq to add SRV records for CardDAV and CalDAV

I've been working on implementing CardDAV support into DAViCal at the moment, and the first problem I encountered when I went to try and use it from iCal, was that the configuration on iCal didn't seem to want to let me enter a URL to my addressbook. After I hauled out Wireshark and watched what was happening on the server it was clear that it was doing an SRV request to try and find the CardDAV server for the domain I had entered in. Unfortunately the DNS management front-end that I normally use doesn't support SRV records, so I was all ready to switch my whole DNS infrastructure around (one of those 'round tuit' tasks that's been on my todo list for somewhat more than a year) until I wondered if the magic that is dnsmasq would be able to help me out. It turns out that it can, and so after a little searching I'd managed to track down the magic syntax to do this:
srv-host=_caldav._tcp.home.mcmillan.net.nz.,davical.home.mcmillan.net.nz.,8008
srv-host=_carddav._tcp.home.mcmillan.net.nz.,davical.home.mcmillan.net.nz.,8008
Of course this is a special-case server for testing at home, and I run it on the funny port because it makes for less pollution when I sniff the traffic coming in. In the real world you'd want to add SRV records for caldavs and carddavs, to get SSL for these services, something like:
srv-host=_caldavs._tcp.home.mcmillan.net.nz.,davical.home.mcmillan.net.nz.,443
srv-host=_carddavs._tcp.home.mcmillan.net.nz.,davical.home.mcmillan.net.nz.,443
Once I'd done that, and put in a few more hours of debugging, DAViCal was working with iCal's CardDAV, at least for viewing / editing / deleting contacts - there's still some work to do yet on searching. I've now also got it working with Evolution's 'WebDAV contacts' plugin as well, after a tip from Milan Crha on the Evolution IRC channel. Heather gets home soon with the Apple phone I gave her when I upgraded to Android, so I'll hopefully be able to test against their new OS release and then get a new DAViCal release out next week. Maybe. Hopefully :-)

20 April 2010

Andrew McMillan: DAViCal 0.9.9 - Tickets, Binds & CalDAV Scheduling

I've now released DAViCal 0.9.9, after five pretty intensive weeks of work. This includes a number of features that were implemented to support dotCal, which has now replaced their old Cosmo backend with DAViCal to break through a performance bottleneck. That turned into a six-week challenge, because Cosmo had wandered some distance from where calendaring standards are currently going, and so there was a lot of work under the covers of dotCal to switch from using various Cosmo-specific features to do the same thing, using open calendaring standards like CalDAV. One standard that Cosmo did support, and which DAViCal now supports also, are WebDAV tickets. Unfortunately the draft RFC for webdav tickets expired, and was never very well thought through, but the functionality is useful, in order to be able to hand people a secret 'ticket' that they can use to grant specific privileges to a single calendar. There's no UI in the DAViCal admin pages for this yet, but expect one to arrive in 0.9.9.1. Something that dotCal lets people do in the web calendar, is to add subscriptions to other dotCal user's calendars, which is a very nice feature, and certainly one we didn't want to lose. In Cosmo this was done by simply duplicating the calendars on a regular basis - a very resource intensive approach! So in order to support this in DAViCal I decided to add support for the WebDAV 'BIND' method, allowing calendars to be linked from one place into another, without requiring all the events to be duplicated. Mixing that up with the WebDAV tickets, it turns out to be very easy to make that stuff work in a read/write manner, and so I did that as well. Again, unfortunately there is no UI to see this at 0.9.9, but something should be in place for 0.9.9.1 because I know that this combination will be particularly useful to people using principal-centric CalDAV clients like the iCal, and especially like the iPhone, which can't see delegations, and where this would be an especially useful feature. The other major feature added in 0.9.9 is the long-awaited work on the Scheduling Extensions for CalDAV. This is currently an expired draft, but I've checked with Bernard Desruisseaux and there will definitely be a new draft out very soon with a few minor changes, and then there will be a push to finalise it as an RFC. Most of the work on this was done by Rob Ostensen, who has been working on a few DAViCal features now, and it was great to meet up with him in person at LCA 2010. Rob has also added support for pulling groups from LDAP into DAViCal 0.9.9 too, which will please a few people. This is the 0.9.9 release for a reason, too: it implements pretty much all of the functionality that a CalDAV server can usefully do at this point. The work that Rob did to get the scheduling extensions done was really the only hole in coverage of what current clients can do, and in the 0.9.9 series I plan to polish up what we have, add some basic UI to expose the tickets and binds (since no clients really do that at all, other than dotCal, which is proprietary :-). I'll also be working on implementing the bones of CardDAV support, and Rob will be moving on to implementing some parts of the iSchedule draft for server-to-server calendar scheduling. Enjoy!

22 March 2010

Andrew McMillan: The cost of crap

For several years now we've been buying our groceries online. It's worked well, and for the last couple of christmases I remember Heather adding a six-pack into the pre-christmas order so she could pull it out and hand it off to the delivery guy. Fair enough too, because he was their front-line man. He was the guy who had to actually meet the customer, and even if only for two minutes face time, the impression he gave with his cheery "seeya mate" on the way out, and his always-happy smile, was that getting the groceries delivered was fun. Well that ongoing encouragement has gone now. His surly replacement who moans about having to carry the boxes up a flight of stairs does nothing to promote a repeat experience, and he brings into sharp focus the disadvantages of online grocery ordering: sometimes the wrong thing is substituted, or sent. Or the fruit doesn't travel quite so well as it does with the personal loving kindness we supply in our own padded wagon. Two visits from the new guy, and we're back to driving to the supermarket ourselves, patrolling the aisles once more in a cereal haze of consumer demand. Distress-purchasing done, and back to the car A.S.A.P. Carrying our own boxes up the stairs. I hate it, which is why I promoted the idea that we try this online grocery thing a few years ago, and it's worked just fine until this sullen bastard came along and destroyed the fun. I wonder how the company feels? Do they realise what this person is doing to their customers? Will they notice, and if they do, will they even be able to tell where the source of the problem is? In these days of increasing online experience, customer service is still important, if not more important than ever before.

26 February 2010

Andrew McMillan: DAViCal 0.9.8.3 - A return to stability

After a whole bunch of drastic changes, lots of bugfixes, rewriting and refactoring, I released DAViCal 0.9.8 on Christmas day last year. I didn't release it very publicly, because I knew it sucked significantly. In an increasingly fast turnaround I've subsequently released 0.9.8.1 to a somewhat wider audience while I was at the recent Calendaring and Scheduling Consortium meeting in Costa Mesa at the beginning of the month. Earlier this week I released 0.9.8.2 to the more interested of the DAViCal userbase. So finally, yesterday, I released DAViCal 0.9.8.3 to fix the few significant problems that were left with that, and this is the release that I'm telling everyone about, because I believe it should be a good stable platform to people who want to upgrade their production DAViCal servers, or who have been hanging back from using DAViCal until the platform was a little more stable.
<!--break-->
This release marks some significant departures from previous DAViCal. I've decided to walk away from some of the more ancient legacy software I tried to work with in the past (PHP4 & PostgreSQL 7.x), to move to support the most ancient-but-not-superseded legacy Linux release I know of, which is CentOS / RHEL, so DAViCal now has a minimum requirement of PHP 5.1 & PostgreSQL 8.1. The biggest changes to DAViCal are under the covers. It is server software, after all, so the largest changes often are not visible on the surface. What has changed is that the architecture for constructing responses as been refactored into a core object which is used to access any DAV resource in the system. This greatly simplifies the general coding and reduces a fair bit of code duplication which had been happening. Another long overdue change is to get rid of the old arcane & obtuse permissions model, replacing it with something which will be radically faster in installations with more than a few hundred users, as well as being more consistent, easier for people to control, and directly in alignment with the DAV ACL specification. Some of the work here was assisted by NZ Registry Services who gave me a Macbook to help with my compatibility testing. Not all of the changes are under the covers though. The new DAViCal also has a thoroughly revised administrative interface, and over the last couple of releases I've gone through the system trying to uncover all the places which were not properly localisable, too, so if you see DAViCal displaying strings in a language you didn't configure it for, it should be a fixable thing. I'd like to particularly thank Masahiro Mikami for turning around some Japanese translations very quickly which enabled me to quickly and visually see what was not properly translatable. There's still more work to do (there's always more work to do) but a fully translated DAViCal should, I hope, be quite usable by non-English speakers now. The new DAViCal also supports some rather significant extra functionality beyond the CalDAV standard, in particular we can thank Rob 'Cave' Ostensen for providing an implementation of pub/sub event change notifications. This version also supports draft 2 of the proposed WebDAV synchronisation standard - that's still under development, but we expect to track pretty close to released versions of that. Of course with all of this work there are some areas that didn't get changed. I didn't manage to finish an implementation of the scheduling extensions draft, much as I would have liked, but that is going to become very much a focus in the coming couple of months, and I'll be calling it "1.0" when things settle down after we get that done. Other things I will be working on include a ticketing system for providing targeted, controlled access to a specific collection or event, and making a start on CardDAV support for addressbook storage. DAViCal seems to be fitting a need for people, and I know there are several thousand installations around the world, even if I'm not sure where they all are. I know some people are running DAViCal on their 8-way server, where I've no doubt it gurgles along pretty happily, and that there are some people running it on their OpenWRT installation, with some tiny amount of resources - just enough for a family calendar. Traffic on the wiki has increased to several hundred thousand hits per month, perhaps three times what it was a year ago, and I hope that I can continue to provide a valuable and useful piece of infrastructure for that little calendaring corner of your organisation. What heartens me the most in this ongoing experience is the nascent community that appears to be forming around my software, with people writing 'tip sheets' and helping to answer questions on the IRC channel and mailing list. These small contributions made by many people - people whom I have never met, for the most part - are enabling me to put more time into feature development and expansion, and that's what I really love.

20 October 2009

Andrew McMillan: Using incron to autocommit changes in a folder

A friend e-mailed me this morning asking for some help with a problem he had where he wanted to make a folder writable by a group of people without making the files deletable. Stepping back from his question, I first pointed out that if the files are editable then they can be effectively deleted by removing the content from them, regardless of whether the directory entries themselves are retained. One solution which occurred to me would be to automatically version the content of the directory, and this reminds me of why versioning of /etc has never worked for me: it only happens when I remember to commit.
<!--break-->
Normally when I edit files in /etc, I am focused on achieving something now, and not on being able to undo it later. To this end I have written myself a script which will commit the contents of a directory (including adding new files and removing deleted ones) into a git repository, as follows:
#!/bin/sh
#
# Allow for debugging.
[ -n "$DEBUG" ] && set -o xtrace
cd $1
MESSAGE="Autocommit"
# Initialise the directory if it's not in Git currently
if [ ! -d .git ]; then
  git init
  MESSAGE="Initial commit"
fi
# Any files that aren't in the directory, but are in the repository can be removed.
for F in  git ls-tree HEAD   cut -f2 ; do
  [ -e "$ F " -o -L "$ F " ]   git rm "$ F "
done
# Any files that are in the directory, but aren't in the repository can be added
git add .
# And commit...
git commit -m "$MESSAGE"
So this script takes one parameter: the name of the directory where changes have (presumably) occurred, and just blindly commits everything there. The magic glue, then, is the awesomeness that is 'incron' :-) My first target is the ~/bin directory of all of those useful little scripts that I tend to randomly edit without nearly enough version control:
incrontab -e
home/andrew/bin IN_MODIFY,IN_DELETE,IN_CLOSE_WRITE,IN_MOVE \
     /home/andrew/bin/commit-directory /home/andrew/bin 2>&1   logger -t andrew-bin
With this, it means that whenever a file changes, is deleted from, moved into, or out of the directory we are monitoring, our commit-directory script is run. Just for testing I delete some ancient files that should have been killed a few computers back. A quick 'git status' in the directory shows everything was committed nicely. I edit a file and make some quick changes. Once again, I go into gitk and can see all the history. I sit back and ratchet my laziness another notch with a nice cup of tea, happy that, for now, at least, changes to my scripts are under revision control at last. Outstanding Issues Well, ok: it's usable as is, but there are a few small things that could be improved: I'll keep monitoring in the real world if these are real annoyances, and whatever other annoyances I find, and maybe I'll tweak that commit script a bit more - add a delay and/or some locking too it. I don't think that adding some push would be too big a deal either, and I'd love to get suggestions from Git geeks as to how to do this more simply/reliably too.

3 October 2009

Andrew McMillan: I still know where it came from...

I publish my photos on the internet. Well, I do when I get around to updating it, anyway. When I do so, I include some information about licensing them. I say to people "I'm very open to people wanting to use these pictures for something else, and will be happy to release photos into the public domain on request." so when I discover one of my photos abused by a travel website I do wonder why I didn't hear about it. Well it seems I also upload a lot of photos to the Burning Well public domain photo repository or to Wikimedia Commons where I also mostly mark them as PD-self. Still, you don't expect a photo of Cuilapan Monastery near Zaachila, Oaxtepec, Mexico to appear on a page about Edinburgh Castle, but really. Am I just being picky? Is this the norm? Are there really people all over the intawebs taking this little care about what photos they use for this sort of thing? I realise there isn't much sunshine in my photos of Scotland, and my photos of Edinburgh Castle, but really. Scotland is renowned for it's preponderance of rain over sunshine and it is considerably glossing the reality of a visit to the real Edinburgh Castle to even suggest the sun might appear. On closer examination I see that the site is actually talking about some weird nutter's home in Jamaica. Nothing to do with anywhere I've ever been, though if I ever visit Jamaica I probably would actually visit the place they're talking about. Now that I know about it, anyway, and especially if I was with Max who claims his favourite castles are Craigmillar & Tolquhon because most of the others aren't at quite the right stage of ruination. Oh well. I guess by putting the photo into the public domain I gave up responsibility for them, and it does seem that it might be hard to find real photos of the ruin in question (let alone free ones). On the other hand I do feel obliged to point and laugh, because that photo isn't Jamaica by a considerable distance, and just because I PD a photo I should still be well within my rights to have a good old giggle when I see a Mexican monastery linked to a Scottish castle. Even one in Jamaica.

19 September 2009

Andrew McMillan: Another way for IPv6 to blow up an IPv4 website

I found another interesting avenue for affecting a web application recently when Heather was trying to renew one of her magazine subscriptions. She mentioned that the site was getting a '500 Server Error' and I recognised the e-mail address it was suggesting, so I banged an e-mail off to advise the problem. Curiously, they weren't able to duplicate the issue while I was still seeing the problem. I did a little fooling around and discovered that I only saw the error when I was making the request through my proxy server. A little more digging and I ascertained that if I connected to the proxy normally via IPv6 I got the '500 Server Error', but if I instead connected to the proxy via IPv4 it all worked just fine. Communicating this back to the site developer, it turns out the site had some blacklist checking in it which was checking not just my source IP (the site was only hosted on IPv4) but also checking the 'Via' header for my IP, which was coming through as an IPv6 address and causing the software to throw it's hands in the air at that point. So chalk up another interesting way in which an IPv6 address on the somewhat remote client can be passed through as input to a web application running on IPv4. I believe I've heard talk of ISPs considering running native IPv6 internally and proxying requests out to IPv4 in exactly this manner, so the sooner web developers realise that addresses could come in more than one shape, the better. And yet another reason to validate your data: just because you received a 'Via' header in the request does not mean there was ever a proxy involved in the request - it is untrusted data, people: like every other piece of data you receive down the wire from the client at the far end.

15 September 2009

Andrew McMillan: DAViCal 0.9.7.2 released

I released a new 0.9.7.2 version of DAViCal yesterday. This reflects quite a lot of stability and small fixes for some subtle problems, and quite a lot of work with the iPhone, adding the possibility of a simpler configuration experience for iPhone users. One long-standing, but less obvious problem (well, less obvious to me) was the handling of usernames with odd characters in them. In the round-trip these needed to be urlencoded, and that wasn't happening correctly in all places, but it hopefully is now. People who want to use e-mail addresses in usernames, or spaces in calendar names, or both, etc., should all work now. One newer problem I've been seeing the effects of a little recently is that PHP 5.3 deprecates call-time pass by reference, which I had used in a few places. That's fixed now, so it should be OK to upgrade to PHP 5.3 without breaking DAViCal or having to explicitly turn those warnings off. The iPhone is a curious beast. Lots of people love them, and it is about the only mobile device that supports CalDAV (though I'm hoping the Nokia N900 also will), so it's important to support it as transparently as possible regardless of my personal loathing for it's horribly locked down state. It was pointed out to me that it is possible for it to be configured by entering three pieces of data (domain name, username, password) so I decided to make DAViCal support that process. In order for it to work for you you will need to have DAViCal running on an SSL server on port 8443, which isn't too big a deal. You can also plain http on 8008 which it will also attempt after prompting when the SSL connection fails. You'll also have to enable mod_rewrite and use some rewriting rules like the example on the wiki. For myself, I use the free SSL certificates from CACert.Org so I had to browse to their root certificate page and load the class 1 and class 3 certs into the iPhone before it would talk happily to my DAViCal server. I'm hoping to continue with reasonably frequent incremental releases of DAViCal for the rest of the year, hopefully with 0.9.7.4 sometime in late October. What will be included there might also depend on how productive my visit to CalConnect XVI is in three weeks too, and I'm looking forward to that.

3 September 2009

Andrew McMillan: Storing Secrets

Something that has been annoying me recently with my bank has been that their website tells me that they will never ask for my password over the phone. And then their call centre asks me for my password. Over the phone. Of course the call centre doesn't mean my website password - they mean the special 'ultra-secure 5ekr1t code phrase', but they don't have a good, universally understood word to use for that. Hopefully they'll work one out, but they appear to have got the message anyway. This got me to thinking about how these phrases are used, and how insecure they are in reality. After all when I store a website password I go to significant lengths to ensure that the same password is not represented by the same string of characters in my database. How vulnerable are our secrets in the databases of organisations we do business with?
<!--break--> Simple Password Storage Surprisingly often people do store passwords in databases in plain text, so that should their website get hacked someone would quite possibly be able to download the whole password database. Please feel free to name and shame these organisations in the comments below. My own pet hate in this regard is the 'Mailman' mailing list software: by default on 'mailman day' - the first day of each month - it sends me my password. In plain text. Of course many developers recognise this flaw, and work around it by using a one-way hash to obscure the password. Usually they choose md5 for their hashing algorithm though, and they often fail to use a 'salt' to randomise the plaintext prior to hashing. This means that even though a password might seem obscure like 'Supercalifragilisticexpialidocious!', and no doubt it will hash to something that seems obscure like 'a7290d426b6a1764af6fd7fba5db214e', but you can often find it straighforwardly by looking it up through one of the friendly reverse hash lookup websites. There's even a Digest::MD5::Reverse perl module on CPAN to interface to a bunch of these in a more automated way! Oh dear. One way to go beyond this is using a 1-way hashing algorithm, with a random salt included into the plaintext before the hashing, so that if (god forbid) two users had 'password' for their password I might see two rows in my database like:
davical=# select username, password from usr;
  username                       password                          
-------------+------------------------------------------------
 user1          SSHA qCctCH5dirYCf29uxJiE68LvmLRDdnBkbldiWlE=
 user2          SSHA y8yOzjoh9fSkVwLaXGoVtWdiIYxmU2FCb2dOZXc=
(2 rows)
When the user wants to log in I apply the same transformation to their incoming password (appending the same salt) and compare against my stored hash. If they match then it must be the same password they used previously. Storing passwords in this way secures them from casual, or even reasonably determined access, although naturally they can still be logged at the beginning and end of the communication - or even in the middle, if we didn't encrypt that bit! The PHP function I use to salt and hash the password is as follows:
/**
* Make a salted SHA1 string, given a string and (possibly) a salt.  PHP5 only (although it
* could be made to work on PHP4 (@see http://www.openldap.org/faq/data/cache/347.html). The
* algorithm used here is compatible with OpenLDAP so passwords generated through this
* function should be able to be migrated to OpenLDAP.
*
* If no salt is supplied we will generate a random one.
*
* @param string $instr The string to be salted and SHA1'd
* @param string $salt Some salt to sprinkle into the string to be SHA1'd so we don't
*                     get the same PW always hashing to the same value.
* @return string  SSHA  followed by a base64 encoded SHA1 of the salted string.
*/
function session_salted_sha1( $instr, $salt = "" )  
  if ( $salt == "" )  
    $salt = substr( base64_encode(sha1(rand(100000,9999999),true))), 2, 9);
   
  return ( sprintf(" SSHA %s", base64_encode(sha1($instr.$salt, true) . $salt)));
 
What about Secret Code Phrases? The problem with these secret code phrases, apart from all of the forgetability and guessability problems that have repeatedly been identified elsewhere, is that they are much less likely to be stored in a one-way hash. Are you going to ask your call-centre staff to type in the customer's secret code phrase? Didn't think so. And if you did it's going to add pronouncability issues to the whole mix. So this means that those organisations who have our secret code phrases in their database will, in all likelihood, have them stored directly as plain text, displaying them to the random call-centre staffer along with all of our other account details, and especially making them vulnerable to accidental disclosure. Disclosure of a sort that doesn't necessarily involve knowing they have been disclosed. Proliferation of Use These things provide the appearance of security - 'Security Theatre' as Bruce Schneier terms it - and because of that they're taken up in a kind of a cargo cult of security: "if the banks do it that way it must be a really good form of security". This makes the problem much worse, because now I have to remember a secret code phrases not only for banks, but for ISPs, phone companies, online auction websites, and so on. How many mother's maiden names, favourite teachers and friend's phone numbers do I have? I'm sure I'm at well up whatever curve it is that measures the number of passwords a person has, because five years ago I had so many I started to store them all in an encrypted database - protected by a yet another password, of course. Now in order to get my story straight I have to store my 'secret code phrases' in there too. If I didn't store my secret code phrases in that database, I'd obviously be re-using them everywhere, from a very small set - perhaps the same one that everyone around me overhears, every time I have to ring my bank to authorise another payment from my account. Because the proliferation of use is not just the breadth of wannabe thespians hoping to climb on the stage of this latest play, but the way they want to use it all the time, too. In fact the only conversation I've had recently with my bank where they didn't want it was when they rang me. Obviously I was the only person who could answer a phone in my house, right? It isn't just security theatre: I think we can see that this analogy belongs much deeper into the sub-genre of 'Security Farce'. Is there a solution? I don't have any easy answers - other than the ones to my security questions, of course - but some improvements are possible. Other banks have quizzed me about stuff like recent expenditure or credit card limits from time to time, but I've usually passed those tests by reading my last bank statement - or failed them by not having it to hand! I don't really think that the answers can lie in that direction because the information is only quite loosely tied to my identity. For some parts of the call-centre handling of secret code phrases there are changes that could make them more secure, but in the fairly short term these organisations need to find a different way to perform these out of band identity checks. For the actual storage of the code phrases it would be a marginal improvement if the database did not contain the actual phrase. Perhaps it could be encrypted with some application-known key, so that it can be unencrypted when it needs to be displayed, but never stored in the clear. Of course there's still the problem with that key... Verification of the secret code phrase could be done by someone not involved in the transaction, so that the call could be temporarily passed into a 'verification stream' where a different person performed the verification step without the context of the account details or enquiry. Though this sort of complexity seems unlikely with call centres seemingly being outsourced to the cheapest supplier. One thing does seem likely to become increasingly true: there is less and less private data in our lives, and every time we share one of these little nuggets with our bank, or our electricity company, or our on-line associates-we-call-friends, is one more chance that it escapes into the hands of the foaming-at-the-mouth-terrorist-cracker-communist-nazi-right-wing-religious-fruitcake hordes. The highest bar for personal verification which any of my banks currently sets for me is a random choice from a set of personally entered questions, with a set of personally entered answers, for which I have to enter two randomly selected characters using my mouse. That's not bad for safe verification, and I'd have to be really impressed with their security if that was stored in the database by a passphrase-protected encryption key. With that bank I don't know what they do over the phone - I assume they've concluded they can trust my logged on persona enough that I can do what I want online, consequently I haven't had to call them and share those secrets with everyone in earshot. Maybe paranoid freaks like me will go back to a chequebook and close down on-line access to their bank accounts entirely when they find themselves having to supply a skin scraping in order to authorise their next $500 payment. "Please insert your finger in the AccuYou(tm) BloodSucker(tm) to proceed with this payment" - well, I guess it might cut down my spending! In any case, biometrics need to be understood before they can be used effectively and appropriately - and remotely over the phone is probably not one of the ways that they can be trusted to work.

11 May 2009

Andrew McMillan: Internet NZ Sponsoring LCA 2010

It's great that in these supposedly straightened times InternetNZ have wasted no time in confirming themselves as key sponsors for LCA in Wellington in 2010. Now if only all of our other sponsors could line up behind them (please) and tell me the extent of their sponsorship, we would have some facts to moderate our plans... But seriously: thanks to InternetNZ for stumping up with the basics to make sure this conference will become a reality. Anyone else looking to sponsor the best linux conference in the world should send me an e-mail. Soon... :-)

11 April 2009

Andrew McMillan: DAViCal 0.9.6.3 released

After far too long (too many holidays away from my keyboard :-) I've released DAViCal 0.9.6.3 and AWL 0.36. This is mostly a stability release, fixing all those little niggles that might only affect a few corner cases, but it frees me up now to concentrate on adding more functionality for a 0.9.7 release in due course. Hopefully this will be the last in the 0.9.6.x series. More information in the DAViCal release notes on the wiki. There are quite a few changes to the AWL libraries which this release depends on, but most of those changes have been driven by my Work Request Management System and Capital APMS projects, rather than from DAViCal.

25 February 2009

Andrew McMillan: Conference Audio: Problems, Procedures & Suggestions for the Future

I was going to reply to this in the mailing list, but after I spent half an hour furiously typing this, I thought it might be better to put it into a blog, for a slightly wider audience, perhaps a more permanent life and partly in an effort to end this thread on a more or less constructive note. The backchat for this is that the LCA 2009 video has started to go up on the web, and some talks have some gaps in the audio stream... Russell Coker wants to make a constructive suggestion, as follows: On Wed, 2009-02-25 at 18:02 +1100, Russell Coker wrote:
>
> I've pasted in the below paragraph (the third time I've included it in this
> thread) to reiterate my point. Note that I am not asking you to re-do any
> videos, I am making a suggestion for future people.
>
> # I suggest that in future when preparing such videos the sections with no
> # sound be omitted and replaced by a short note explaining the omission,
> # something like "sorry, due to technical difficulties four minutes of sound
> # were not available". Having a second ogg file for the lip-readers would be
> # an option although I expect that the demand would be minute.
> As someone who has not been at all involved with the production of the LCA videos, but who has been intimately involved with the process of streaming DebConf 6, 7 & 8, I think this sort of additional effort will only happen if people voluntarily do it themselves. Just the task of *watching* all of the produced videos is a huge job for one individual, but as a community of individuals it is quite likely that at least one individual in the community watches each video. If each person who was offended by the quality of post-production work took it upon themselves to fix up only one video then it is probable that we would see some changed. Otherwise I'm afraid it is unlikely. Further in his e-mail, Russell Coker wrote:
>
> My observation was that the microphones were put down AFTER they ceased
> operating. A non-functional lapel microphone is no better than a non-
> functional hand-held microphone.
>
> Having a large supply of new batteries would be one way of alleviating the
> problem (I believe that some of the instances were due to flat batteries).
> Another possibility is having two microphones on hand so that if one died the
> other could be turned on.
>
> A wired microphone that doesn't rely on battery power would probably be the
> most reliable option. That of course might not fit with OH&S issues. Most speakers are uncomfortable / unfamiliar with wired microphones. Spares are essential, of course, but the problem can usually be avoided by having a checklist for the audio person to confirm several things: Before the start of the talk: At the start of the talk: At the end of the talk: There are probably a couple of things that I've missed, but if you start with a checklist you pretty soon modify it into a *good* checklist, and it *really* helps when people are under fire. In fact lapel mics are not the best microphones for these kinds of presentations. The best mics are the ultra lightweight 'headset' models which place a the mic near to the speakers mouth. These do not suffer when the speaker turns their head hard to the left or right, at which points a lapel mic stops getting their speech. They also work well with that more hirsute minority so over-represented in our particular community (alas, that this set no longer includes Bdale :-) Of course DebConf has a large team of volunteers for streaming the conference, and has developed these kinds of procedures over a number of years. At DebConf there are usually only two main streams, each of which involves: Ideally that really is 7 people (times two streams), and you can get by if you can't find all of the last three, but the other four are increasingly desirable. We mostly manage to do that with volunteers for two streams at DebConf, but for five or more streams at LCA it is inevitably a lot harder, and the quality necessarily has to be cut back to match the resources available. While it is true that sound is critical to this communication, it is unfortunately also true that most people's (even most geek's) eyes glaze over when presented with a mysterious box with a mere 6 sliding potentiometers, let alone when there are 36 of them, each associated with another 8 rotary knobs, and a rats nest of cables worthy of the worst network nightmare they've seen, and where everything appears to be literally held in place with duct tape. So it is not so surprising then that while it is relatively easy to find a gadget mad geek capable of operating a camera, or even experienced with operating a camera, finding people with experience operating a sound mixer is an order of magnitude harder. At LCA the operation appears superficially to often involve a (single) speaker camera operator with a very basic sound mixer which has been configured once by an overworked person who is unobtainable during the actual talk to solve any audio problems. These people are volunteers, and are doing their best, but it simply isn't possible to get a consistently high quality of video and audio in those circumstances. So as one of the organising committee for LCA 2010 in Wellington I will be watching what we do, and although I don't intend to get personally involved in the video production (I've got plenty of other stuff on *my* plate :-) I do intend to provide what advice and assistance I can. I know that we hope to have some professionals involved (as volunteers), but how many volunteers, and with what levels of skill & experience, we are unlikely to know until much closer than the date. I'll also personally try and get a run-down on all of the audio equipment so that in an emergency, if I happen to be available, I can sub as a sound mixer operator. So we will try and do better in Wellington in 2010. Come to the conference, though, because we can make no guarantees, and if you do come to Wellington, and you see a single cameraman, and you know something about running a sound mixer, then perhaps you can come and volunteer to help out in that capacity, for the talks you are attending anyway. We'd really appreciate the help. And finally I must say thanks, in particular, to Holger Levsen for all the learning I have gained since joining the videoteam at DebConf5 in Helsinki in 2005. And too, to all of those past conference teams (LCA, DebConf and otherwise) who have put their best efforts into providing videos of the talks I couldn't see in person. I know it's a bloody hard job, and often a less high profile one, so:

Thank You.

23 January 2009

Andrew McMillan: LCA2010 in Wellington, New Zealand

The news is out now, that Wellington will be hosting LCA 2010. As someone on the core team for that (potential sponsors please contact me sooner rather than later :-) I really hope we can do something as awesome as the current week we've had here in Hobart. Thanks to all of the Hobart team for doing a great job.

Andrew McMillan: Summary of Paul's Perl talk

For the benefit of Debian users, here's a quick summary of Paul Fenwick's talk about cool stuff in recent Perl versions.

apt-get install libmoose-perl
apt-get install libpar-packer-perl
apt-get install libdevel-nytprof-perl
apt-get install libautodie-perl
apt-get install libperl-critic-perl
Now I have to integrate all that into my writing of Perl programs, which doesn't really happen frequently enough for such changes to become easily ingrained... Great fun, and it's tempting to install Windows just so I could run some of that stuff, but perhaps Paul hasn't realised that you can run Minesweeper under Wine?

22 January 2009

Andrew McMillan: Seeing my photo

Today in Anton Hughes talk at LCA 2009 I saw my photo of Eben Moglen up on the big screen. Woohoo! :-) As with most of the photos I have put on Wikimedia Commons it is released under PD-Self, so it will be interesting whether he chooses to credit me or not. He seems to have a pretty good argument, and I hope that this gets visibility and acceptance in the legal community over time. The free software audience is pretty sympathetic, of course.

20 January 2009

Andrew McMillan: LCA 2009 - The first couple of days

LCA is once again proving it's premiere status as one of the top free & open source software conferences in the world. Our hosts in Hobart are well-organised and friendly, and though I've been locked into the Systems Administration miniconf by virtue of being one of the organisers, what I've been hearing from the other delegates with the freedom to sample other streams is that all of the miniconfs have once again achieved the high standards that we've come to expect from the conference. Right now I'm in the keynote by Tom Limoncelli who's giving us all a good kick in the proverbial, to switch us to the mindset of plenty, and to see beyond limited scarcity. Wonderful stuff. Linux == Infinite Love :-)

Next.

Previous.