Search Results: "will"

15 January 2013

Russ Allbery: More on Aaron Swartz

I got some feedback that the analogy I used in my last post was a bit confusing, and indeed I blew the phrasing of the analogy (also now corrected). So let me try this again, since I think there's a subtlety here that may be missed. I should note for the record that my understanding of what Swartz did that started the process is apparently somewhat based on the description from the prosecution, so it may not be the complete or accurate facts. Since there will now be no trial, we may never find out what the defense was, and whether those facts would be challenged. So it may be best to think of this as a hypothetical. We never established, or will establish, in court exactly what happened. Swartz was, generally speaking, charged with two things that I consider quite distinct, at least from an ethical perspective. Most of the focus is on the copyright part: downloading JSTOR articles with an intent (never acted upon) to distribute them to the world. There are a bunch of reasons why this may or may not be justified, which are tied into the origin of those articles (many of them were publicly funded) and the legitimacy of copyright licensing agreements. I think there is significant room to hold a variety of opinions on this, although I don't believe that "crime worth 35 years in prison" is even remotely close to justifiable under any interpretation. However, there is a second part of what he was charged with, and that's primarily what I was commenting on, not the copyright part. He allegedly hooked up a laptop in an unlocked MIT wiring closet without permission and then used MIT's network and JSTOR license to download information. This is, to me, a subtlely but entirely distinct act from the question of whether taking JSTOR's data was ethically or legally wrong. Whether or not one believes that JSTOR's copyrights are not legitimate, it's still not okay to use someone else's network and license or to trespass in their wiring closets without permission. (I work in central IT for a university, so this strikes closer to home.) And this is where the analogy came in, which I flubbed. I had originally said that this was akin to "traipsing into someone's barn or backyard shed without their permission and taking some of their tools because you want to use them." The word "taking" was wrong; it falsely implies that you weren't going to return the tools. I had been thinking "taking and then returning" when I wrote that, and the important second part didn't make it into the post. So let's try this again: Swartz's actions at MIT, as I understand them (and many things could change this, such as a revelation that he had MIT's prior permission), are akin to going onto someone's property and into their barn without permission, borrowing their hedge trimmer for a while because you want to use it, and returning the hedge trimmer without any damage when you're done (and without them noticing it was gone). I'm quite fond of this analogy, since I think it clearly establishes two things:
  1. Most people are going to be unhappy about this happening and will intuitively feel like it should probably be illegal. Not everyone; there are folks who don't believe in personal property, or at least wouldn't extend it to tools in a barn. But most people will feel that someone should ask first before they come borrow your tools. Even if they don't damage them, even if they return them before you notice they're gone, you might have wanted to use the tool at the same time or they might have damaged them without intending to, and they should just ask first. It's common politeness, and depending on the circumstances, someone who doesn't ask and is covert about borrowing tools might be worth calling the police over.
  2. There is absolutely no way in any reasonable moral system that doing this should result in 35 years in prison. Or even 10, or even 1. Yes, most people would consider this a crime, but most people would consider it a minor crime. It's the sort of thing where you might have to impose some consequences just to make sure the message of "knock it off" is delivered firmly, but someone doing this is rude and inconsiderate, not evil.
The JSTOR copyright stuff is more complex to analyze and is more politically divisive, but for me the key points are (a) Swartz never released the data, and (b) JSTOR declined to press charges. To me, that means the deeper copyright questions, which are quite interesting, were never actually reached in this particular case. The crime that did apparently happen was the trespass at MIT, for which I think the above analogy is the right way to think about it. The point I do want people to take away from this is that one should not overlook the trespass at MIT even if one wants to celebrate the undermining of the copyright regime and doesn't believe JSTOR's data should be considered their private property. Social activism and political disobedience are important and often valuable things, but performing your social activism using other people's stuff is just rude. I think it can be a forgivable rudeness; people can get caught up in the moment and not realize what they're doing. But it's still rude, and it's still not the way to go about civil disobedience. For both ethical and tactical reasons, involving bystanders in your act of social activism without their consent is a bad approach. ETA: The problem, of course, with discussing all of this is that while it's relevant and possibly even somewhat important in the broader sense of how our community acts going forward, it also doesn't capture the fact that this was only one incident in a remarkable life. One of the worst problems with the abusive prosecution of Swartz is that it blew this incident completely out of proportion. A moment of arguable judgement should not dominate one's life or cast a shadow over all of one's other accomplishments; the prosecution tried to make it do just that. That's part of what I'm arguing, but ironically that partly feeds into the lack of proportionality in the discussion. I've gotten kind of far afield here, so let me go back and say explicitly: Swartz was a remarkable person who did much to admire and respect, and the world is a worse place without him.

14 January 2013

Russell Coker: Promoting Enthusiasm

Rusty wrote an insightful post titled What Can I Do To Help? about reactions to new ideas [1]. He suggests that people make an effort to have a positive approach when someone talks about a new idea, it s quite common for people to point out reasons why the new idea might not work out which is discouraging for the person who had the idea. I think that is a really good point. I probably haven t done too well in that regard in the past and will try to do better in future. Code Written by Assholes Rusty previously wrote a post titled If you didn t run code written by assholes, your machine wouldn t boot which implies that we should just let assholes be assholes [2]. That doesn t go well with his What Can I Do To Help? post. Note that I m not accusing Rusty of hypocrisy here, giving advice to help people who want to get along well with others is not in contradiction with refraining from giving unsolicited advice and encouragement to difficult people who have expressed no interest in improving their behavior. A comment on the latter post by Doctor Whom says If I had seen this kind of talk when I was a teenager, I would have thought twice about picking up coding , presumably given the number of people who read Rusty s blog there are some teenagers who experienced some discouragement towards a career in computers (or a hobby in FOSS) from Rusty s post. I ve already written a response to the If you didn t run code written by assholes post, among other things I suggested that people who are minor assholes should be assisted to be less difficult and major assholes should be excluded [3]. In that post I was working on the assumption that for every significant task that needs to be completed (such as making a popular OS bootable) someone will do it, if the person working on it disappears then someone else will take over there is a community of programmers who will work on whatever needs to be done. The Importance of Individuals But in terms of new ideas it really comes down to individuals. Most projects which are significant and important now probably started out as one person or a small group who had an idea that seemed unlikely to succeed at the time. So while any big and successful project can have people replaced (which is among other things a requirement of long-term success) there are situations in which individuals with ideas matter. Another important factor is that even ideas which turn out to be impractical are still useful. Someone who has an impractical idea about a technical issue and investigates it fully will learn a lot and may end up working on the less radical ways of solving similar problems this is good for the individual and the community. Another Way of Promoting Enthusiasm In terms of promoting enthusiasm it seems that one thing that can be done by high profile people is to avoid writing posts like If you didn t run code written by assholes, your machine wouldn t boot . When people in positions of power and influence appear to have no interest in promoting good behavior it really discourages people who are vulnerable to the assholes which among other things means most members of minority groups. Obviously Rusty could t stamp out all asshole behavior, but if he announced a plan to try and make things better in that regard then it would help. It s difficult to be enthusiastic when faced with discrimination from a minority and disinterest from the majority. Of course with the way the Internet works I m sure someone will say what about the assholes who have great ideas, shouldn t we nurture their enthusiasm by letting them keep doing asshole things? . I think that for the major assholes this won t be a problem, for example anyone who s racist will be well aware that many people disagree strongly with them and thus won t be particularly discouraged when they meet more people who disagree. For the minor assholes (people who don t want to be assholes) it will be somewhat discouraging to be corrected, but that could be a learning experience for them that s worth more than support in implementing their latest technical idea. Update: Why Rusty is Important In response to a comment by private mail I ve added this section after publication. Firstly I think that the opinions of all members of the community matter as they all affect the social environment which determines what types of behavior are encouraged and discouraged. But Rusty is more important than most people. Firstly Rusty has a Wikipedia page [4], that alone is an objective criteria indicating his importance. But in terms of influencing people in the FOSS community the most important things are that he s a high profile Linux kernel programmer (which alone gives significant status and influence) and that he s the founder of the first Linux conference in Australia (which is now known as Linux.conf.au AKA LCA). When issues such as the anti-harassment policy for LCA are being discussed any opinion that Rusty offered would be taken very seriously. But so far he doesn t seem to be involved in any of the public discussions.

Ben Hutchings: What's in the Linux kernel for Debian 7.0 'wheezy', part 3

The Debian package of the Linux kernel has some additional features not present in mainline Linux 3.2. This continues from part 1 and part 2. Real-time scheduling The 'PREEMPT_RT' aka 'linux-rt' patchset provides true real-time scheduling under Linux. This can allow applications to respond to I/O with very low latency and jitter, although the results depend on application design and on the hardware and firmware used. Real-time scheduling is commonly used in industrial control, live audio/video applications, and high-frequency trading systems; it can also be useful for tightly-coupled cluster computing. However, most applications will run more slowly under these kernels, and may not be any more responsive. The Linux 3.2-rt stable series maintained by Steven Rostedt is used to build alternate kernels for amd64 and i386. You can install these using the metapackage linux-image-rt-amd64 or linux-image-rt-686-pae. Hyper-V guest drivers Like most VM hypervisors, Microsoft Hyper-V supports paravirtualised drivers which can provide much better performance than emulation of old hardware devices. Linux 3.2 had quite early versions of these drivers that were relegated to the 'staging' area, but we've updated them to the much improved versions found in Linux 3.4. I hoped we would also be able to package the userland daemon that supports management of Hyper-V guests, but this also needed many fixes and wasn't ready in time.

Robert Collins: Multi-machine parallel testing of nova with testrepository

I recently added a formal interface to testrepository to enable cross-machine scaling of test runs. As testrepository is still a static scheduler, this isn t perfect, but its quite a minimal interface, which makes it easy to implement. I will likely evolve it in reaction to feedback and experience. In the long term I d love to have a super generic tool that matches that interface, so the project VCS copy of .testr.conf can just call out to it. However I don t yet have that, but I do have a simple by-hand implementation that I use to run nova s tests across my personal laptop, desktop and work laptop. Testr models this by assuming each test running process can be mapped to a single instance id (which could be a chroot, vm, cloud instances, ) and then running one or more commands in the instance, before disposing of it. This by hand implementation consists of 4 things:
  1. A tiny script to rsync my source directory to the relevant places before I run tests. (This takes <2seconds on my home wifi).
  2. A script to allocate instance ids (I just use ints)
  3. A script to discard them
  4. And a script to copy tempfiles onto the target machine and run a given command.
I do my testing in lxc containers, because I like my primary environment to be free of project-specific quirks and workarounds. lxc is not needed though, if you don t want it. So, to set this up for yourself:
  1. on each host, make an lxc container (e.g. following) http://wiki.openstack.org/DependsOnUbuntu
  2. start them all (lxc-start -n nova -d)
  3. Make SSH config entries for the lxc containers, so you can get at them remotely. (make sure your host * rules are at the end of the file otherwise the master overrides won t work [and you might not notice for some time...]):
    Host desktop-nova.lxc
    # lxc addresses may be present on localhost too, so namespace the control
    # path to avoid connecting to the wrong container.
      ControlPath ~/.ssh/master-lxc-%r@%h:%p
      hostname 10.0.3.19
      ProxyCommand ssh 192.168.1.106 nc -q0 %h %p
    Host hplaptop-nova.lxc
    # lxc addresses may be present on localhost too, so namespace the control
    # path to avoid connecting to the wrong container.
      ControlPath ~/.ssh/master-lxc-%r@%h:%p
      hostname 10.0.3.244
      ProxyCommand ssh 192.168.1.116 nc -q0 %h %p
  4. make a script to copy your nova source tree to each test location. I called mine sync
    #!/bin/bash           
    cd $(dirname $0)
    echo syncing in $(pwd) 
    (rsync -a . desktop-nova.lxc:source/openstack/nova --delete-after && echo dell done) &
    (rsync -a . hplaptop-nova.lxc:source/openstack/nova --delete-after && echo hp done)
  5. Make sure you have the base directory on each location
    ssh desktop-nova.lxc mkdir -p source/openstack
    ssh hplaptop-nova.lxc mkdir -p source/openstack
  6. Sync your code over.
    ./sync
  7. And check tests run by running a few.
    ssh hplaptop-nova.lxc "cd source/openstack/nova && ./run_tests.sh compute"
    ssh hplaptop-nova.lxc "cd source/openstack/nova && ./run_tests.sh compute"
    This will check the test environment: we re not going to be running tests on each node via run-tests or even testr (because it gets immediately meta), but if this fails, later attempts won t work. Your test virtualenv is inside the source tree, so it is copied implicitly by the sync.
  8. Decide what concurrency you want. For me, I picked 12: I have a desktop i7 with 4 cores, and two laptops with 2 cores each, and hyperthreads are on on all of them I m going to set a concurrency figure of 12 between the cores (8) and threads (16) counts, and possibly balance it more in future. A higher number assumes less contention between ALU s and other elements of the core pipeline, and I expect quite some contention because most of nova s unittests are CPU bound not I/O. If the test servers are not busy, I can always raise it later.
  9. Create scripts to create / dispose / execute logical worker threads.
  10. Creation. I call this instance-provision and all it does is find the lowest ints not currently allocated and return them.
    #!/usr/bin/env python
    import os.path
    import sys
    if not os.path.isdir('.instances'):
        os.mkdir('.instances')
    running_ids = os.listdir('.instances')
    count = int(sys.argv[1])
    top = count + len(running_ids)
    ids = [str(i) for i in range(top)]
    new = set(ids) - set(running_ids)
    for id in new:
        file('.instances/%s' % id, 'w').close()
    print(' '.join(new))
  11. Disposal is easy: remove the file marking the instance as in-use.
    #!/bin/bash
    echo freeing $@
    cd .instances
    rm $@
  12. Execution is a little trickier. We need to run some commands locally, and other ones by copying in temp files that testr has setup to the machine sshing to the remote machine, cd ing to the right directory, sourcing the virtual env, and finally running the command.
    #!/bin/bash
    instance="$(($1 % 4))"
    case $instance in
    [0]) node=
         local="true"
         ;;
    [1]) node=hplaptop-nova.lxc
         local=""
         ;;
    [2-3]) node=desktop-nova.lxc
         local=""
         ;;
    *)   echo "Unknown instance $instance" >&2
         exit 1
         ;;
    esac
    shift
    files=
    # accumulate files to copy
    while [ "--" != "$1" ]; do 
    files="$files $1"
    shift ; done 
    shift   
    if [ -n "$files" -a -z "$local" ]; then
        echo copying $files to node.
        for f in $files; do
            rsync $f $node:$(dirname $f) ;
        done
    fi  
    if [ -n "$local" ]; then
        eval $@
    else
        echo ssh to $node
        ssh $node "cd source/openstack/nova && . .venv/bin/activate && $@"
    fi
  13. Finally, tell testr how to use this. (Don t commit this change to nova, as it would break other people). Add this to your .testr.conf.
    test_run_concurrency=echo 12
    instance_provision=./instance-provision $INSTANCE_COUNT
    instance_execute=./instance-execute $INSTANCE_ID $FILES -- $COMMAND
    instance_dispose=./instance-dispose $INSTANCE_IDS
Now, when you run testr run parallel, it will run across your machines. Just do a ./sync before running tests to get the code out there. It is possible to wrap all of this up via automation (or to include just-in-time provisioned cloud instances), but I like the results of still rough scripts here it strikes a good balance between effort, reliability and performance.

13 January 2013

Russ Allbery: Slacktivism that may actually help

Normally, I'm not that much of a fan of the slacktivist trend of signing pre-canned petitions, signing things that political organizations send in the mail, and so forth. It takes very little effort and therefore carries very little weight. A lot of those efforts are more exercises in helping the signers feel better about themselves. But there is the occasional exception. Most of you probably already know about the death of Aaron Swartz. For those who aren't familiar, see Larry Lessig's article. Swartz suffered from depression (and, seriously, fuck depression it's an awful, horrible disease), but there's little doubt that the ongoing federal prosecution using the full weight of the US district attorney's office to hound him for what amounts to political trespassing was part of what led to his suicide. As it happens, I personally believe that Swartz committed a crime, and probably should have paid some consequence for it (on the order of a fine or some community service). Hooking your devices up to someone else's network without their permission and messing around in their wiring closets (locked or not) is, for me, akin to traipsing into someone's barn or backyard shed without their permission and using some of their tools because you want to use them. And whether or not one likes the current copyright regime (and I don't like it at all), MIT is still in the awkward position of having to work with it. Abusing their license for your political goals is effectively recruiting them into your activism without their permission, and as I've mentioned before, I have a mild obsession with consent. It was a crime. However, it was a minor crime, and that's where this whole situation went completely off the rails. One aspect of a justice system is fairness: uniform application of the laws to everyone. However, another aspect of a justice system is proportionality: punishments that fit the crime. Without ever having been convicted, Swartz was already punished completely out of proportion to what he actually did, both financially and emotionally, by prosecution by the US attorney that went far, far beyond zealous into actively abusive. This despite the fact that the owner of the academic papers that he was downloading as an act of open access activism stated they did not want the case to proceed and asked the US government to drop the charges. I don't think what Swartz did was right, or legal. However, the correct reaction was "look, involuntarily recruiting MIT as an accomplice to your act of civil disobedience is not okay don't ever do that again." Not "you are evil and should be locked up in prison for 35 years." And I'm completely fed up with the disproportionality of our justice system and the practice of ridiculous over-charging of crimes in an attempt to terrify people into bad plea bargains. Which brings me to the slacktivism. This is exactly the sort of situation where popular opinion matters. US attorneys who have lost the faith and support of the population they serve won't keep their jobs. And no one in government particularly wants this case to be splashed across the front pages, or to have to answer questions about the appropriateness or proportionality of the prosecution while people are mourning a dead young man. If we make it clear enough to the Obama administration that people are watching, that this matters, and that we're angry about it, not only is it quite likely there will be consequences for this prosecutor, but it may serve as a deterrant for other prosecutors in the future. There is a petition on WhiteHouse.Gov to remove the district attorney for prosecutorial overreach, and for once taking five minutes to create an account and clicking on a petition may be both useful and appropriate. Taking disciplinary action here is, unlike with a lot of petitions, something that the Obama administration can actually do, directly, without involving the rest of the dysfunctional US government, and without making new law. If you are a US citizen, please consider going to this site and signing it to say that this matters to you and you believe this prosecution was excessive and inappropriate. Please note: you do not have to believe that Swartz died solely or even primarily because of this prosecution to do this. We'll never really know the complex factors behind his death. But this was a burden that he shouldn't have had to deal with. Please also note that you do not have to think he was justified in his behavior to sign this petition. This is not a question of whether what he did should be legal or was ethical. Rather, even assuming it was illegal, it's a question of appropriate punishment and proportionality of response. Whether or not you agree with his political cause, there simply was not enough damage done, to anyone, to warrant this kind of aggressive prosecution. And the direct victims agreed, which was the point at which the district attorney should have scaled way back on their actions or dropped the matter entirely. Not doing so was an abuse of office and position, and that should have consequences. ETA: Corrected "taking" to "using" in the analogy about tools. I was actually thinking "taking and then returning" when I wrote that, but only the first word made it into the post, and ended up creating a confusing parallel with theft that wasn't intended.

Ben Hutchings: What's in the Linux kernel for Debian 7.0 'wheezy', part 2

The Debian package has some additional features not present in mainline Linux 3.2. Some of these were described in part 1, and this continues the list. CPU devices CPUs are exposed in the device model as normal devices, not 'sysdevs'. For x86 CPUs, model and feature information is included in the uevent for the CPU device. This results in automatic loading of appropriate device drivers with the help of udev, rather than specialised init scripts: Control groups Most control groups (cgroups) features are included, but some are disabled by default at run-time: EFI stub On x86 architectures the kernel is built as an EFI executable which can be directly loaded from an EFI shell, e.g.:
Shell> vmlinuz-3.2.0-4-amd64 root=/dev/sda2 initrd=initrd.img-3.2.0-4-amd64
Debian will usually boot on an EFI system via grub-efi, but this can be useful for some unusual configurations and some of the simple EFI boot loaders that are currently being developed. Linux Security Modules The kernel side of AppArmor has some extensions for backward compatibility with AppArmor userland versions 2.4-2.7, matching wheezy userland. Debian appears to be the only major distribution to provide a choice of LSMs: AppArmor, SELinux or TOMOYO. Use the security= boot parameter to enable any of them. CPU microcode updates for Xen Like all hardware, CPUs have bugs, which can sometimes be fixed in microcode. x86 microcode updates may be applied by either the BIOS or operating system, but it's easier to upgrade packages than to upgrade the BIOS. Xen 4.1 can apply microcode updates but relies on the dom0 kernel to load them, and the mainline Linux kernel does not do this. We include an extension of the microcode loader that does so.

Adnan Hodzic: Enable global menu and HUD support in Eclipse IDE

As I couldn t retain my curiosity for Ubuntu s Raring Ringtail release, I ended up having a dual boot with Ubuntu 13.04 and Debian Sid. Even thought at the moment 13.04 is only 24% complete, it s already a pretty promising release. However, one thing that heavily annoyed me is that global menu and HUD (still) don t work out of box with Eclipse IDE. Being part of Debian Java team (working on Eclipse) I had to do something about it. To make it as simple as possible for you, I made a package which enables mentioned features. Package is made for/tested and working on 32/64 bit architectures on 12.04/12.10/13.04 Ubuntu. enable eclipse global menu and HUD In following days I ll push a Eclipse patch which will enable global menu and HUD support out of box, thus eliminating need to install this package in future.

Thorsten Glaser: PSA: Unicode codepoints, referring to

PSA: Referring to Unicode codepoints. If your Unicode codepoint is, numerically, between 0 and 65533, inclusive, convert it to hexadecimal and zero-pad it to four nibbles. For example, the Euro sign is Unicode codepoint #8364 which is 20AC hex; the Eszett is 223 which is DF hex, padded 00DF.
Then write an uppercase U , a plus sign + , and the four nibbles: U+20AC U+00DF
In mksh, JSON, etc. it s a backslash \ , a lower-case u and four nibbles. Otherwise, your Unicode codepoint will be, numerically, between 65536 and 1114111, inclusive, that is hex 10000 to 10FFFF. (There s nothing on 65534 and 65535, nor above these figures.) In this case, convert it to hex, zero-pad it to eight nibbles and write it as an uppercase U , a hyphen-minus - and the eight nibbles. In C-like escapes for environments supporting the Unicode SMP, that s a backslash \ , an upper-case U and eight nibbles. Do not, in either case, use less (or more) hex digits than specified here. For example, there s a famous Unicode codepoint U-0001F4A9 PILE OF POO . That s not the same as U+1F4A9. The latter reads as U+1F4A GREEK CAPITAL LETTER OMICRON WITH PSILI AND VARIA and a digit 9 ( 9). Be educated.

Neil Williams: Status of klibc on AArch64

With significant assistance from Steve McIntyre and some judicious delving into the ARM Information Centre, I've now got the assembly portions of klibc sorted (but untested) for AArch64 (arm64).

Andy & I started by copying the old ARM support as a new directory and one of the final steps was to remove a whole bunch of legacy code from the days before Thumb and all the #ifdef lines which went with it. Some files disappeared entirely. setjmp.S was the largest amount of work as the load&store multiple support of ARMv7 has gone in ARMv8, so the stmia mnemonic had to be expanded to multiple stp calls but that makes it more explicit, so it's not a bad thing.

Steve & I borrowed from the glibc AArch64 code from upstream by Marcus Shawcroft, simplifying the glibc macros for klibc and also got clues about what extra registers needed handling compared to ARMv7.

I've now got to look at some traditional cross-compilation issues because the Linaro AArch64 toolchain doesn't install to typical Debian cross-building paths and the build now moves past the AArch64-specific assembly and fails later when the C code gets the wrong include path and ends up including /usr/include/i386-linux-gnu/asm/byteorder.h with predictable results.

If someone has an AArch64 toolchain already set up, feel free to clone my modified klibc tree and let me know if there are subsequent build errors. Of course, if you fancy testing a build in the Foundation Model for ARMv8, that would be good too! (Report issues via github.)

Whilst I'm sorting out the toolchain, I've also been updating perl-cross-debian (which has also seen some more upstream testing and improvement).

Russell Coker: Android Multitasking

My new Samsung Galaxy S3 has support for Multi Window Mode , here is a video which shows how to use this, Multi Window Mode starts at about 2:30 [1]. A common complaint about Android is the lack of multitasking, which is partly true and only slightly alleviated by Multi Window Mode. Running Multiple Programs Traditionally Android has multitasked with a similar ability to a Unix shell session, you can have applications running in the background and switch between them but you can t see multiple applications running at the same time (apart from seeing notification messages at the top of the screen). The new Multi Window Mode allows two or more applications to share a screen. But it only applies to a small number of applications which support it, on my phone that is applications shipped by Samsung and Google Chrome. Also I can t have multiple copies of Chrome open at the same time which means I can t do the things that I do on every PC that runs Chromium (the non-Google build of Chrome). I have not yet found a situation where Multi Window Mode has been useful to me, the applications I use for most tasks don t support it. So while Android being based on Linux does multitask really well in the technical computer-science definition it doesn t do so well in the user-centric definition. In practice Android multitasking is mostly about task switching and doing things like checking email in the background. Having multiple programs running at once is particularly difficult due to the Android model of applications sometimes terminating when they aren t visible. A common task is to view a message in a MUA and switch between that and another window (EG a web browser). K9 is my preferred MUA for Android which seems to have no option to switch back and still be viewing the same message as before the task switch so at least three actions need to be taken to get back to the same message after I resume K9. One major feature to make multitasking on Android more usable would be a way of rapidly switching between two applications and being certain that each one would be in the same state when the user switches back to it. Copy and Paste Another problem with Android multitasking is the difficulty in copying and pasting text. While copy/paste is not strictly required for multitasking it is a logical requirement when you have multiple applications running on behalf of a single user. For PCs everyone knows that you select text by holding down the SHIFT key while using the cursor control keys or by holding down the mouse button and swiping the mouse cursor over the text and you then use CTRL-C or the Edit menu to copy text. On Android it s a long press to select text which then gives you markers for the start and end, you drag those markers around and then select that you want to copy the text. This is at best a lot more inconvenient than using a high resolution input device like a mouse to select text. At worst it doesn t seem particularly reliable, K9 Mail for example won t let me copy text from a message for unknown reasons on a desktop OS such problems are vanishingly rare such that I can t think of an example of it happening. IO and Multitasking Multitasking for a user (as opposed to the multitasking needed to host dozens or hundreds of concurrent users) on Unix servers was very limited in the days of VT100 terminals and similar devices. Programs such as GNU Screen [2] allowed a text display to perform windowing functions that are similar to a modern GUI. But generally it seems that the ability for a user to run multiple programs at once is largely limited by their ability to see the output and to rapidly switch between programs or sessions. As a keyboard with ~100 keys and a text display with 80*25 characters is a major limitation it s obviously going to be a comparable (and often greater) limitation to have an on-screen keyboard that takes half the screen space, a single program taking all the screen, and a drop-down status bar that might be useful for multitasking. With Android 4.0 and above you can activate a task switcher by holding down the home button for two seconds [3]. There are also a variety of third party task switching programs on the Google Play store which all seem to start by holding down the home button. One problem with these options is that they require an extended press of the home button where the ideal is something that is as quick as ALT-TAB or a single mouse movement on a desktop system. One possible input action would be to switch between the most recent tasks with a palm swipe on the screen an operation that is quick and easy. Currently a palm swipe is used for a screen capture, but as there are four possible directions for swiping the screen one of them could be used for screen capture and another two for task switching. But this wouldn t do a lot of good without the ability to switch tasks without losing the context that either requires Android application changes or having the OS not tell an application that it was occluded. The iPaQ had interesting capabilities for input, it had a main button on the front that could be pushed in four directions (usually for cursor control) as well as being pushed in, four extra buttons on the front, and a button on the side [4]. I don t recall the methods that Familiar Linux on the iPaQ [5] used for task switching, but it was less of an issue as the iPaQ had less RAM, no Internet access and no telephony functions. I think that adding a bunch of extra buttons to an Android phone would make it a lot more useful. The iPaQ method of cursor control is one that could be considered (it could alleviate the copy/paste problems among other things). As an aside in two years of using Android phones I ve done less serious writing on Android than I did in my first two years of using Familiar on the iPaQ largely due to the lack of keys and a stylus on my Android phones. The screen size on Android phones is also a limit for multitasking. The earlier/cheaper phones that have small screens with resolutions such as 320*480 have very limited ability to display two programs at once. The 720*1280 display in the Galaxy S3 has a lot more potential in this regard and the Galaxy Note 2 and the HTC J Butterfly AKA Droid DNA have even more potential. In the future it seems that screen size limitations on phone multitasking will be a solved problem for everyone who can afford one of the high end phones which incidentally are much cheaper than the iPaQ was 10 years ago. Conclusion Checking email in the background etc is very useful on Android systems. But in terms of the user running two programs at once it seems very limited, and that situation seems likely to remain until the vendors adopt multi-window support. This could be difficult given that Google applied a lot of pressure to CyanogenMod to stop them from doing it [6]. Even when a system with a large display (particularly a tablet) runs a version of Android that supports multi-window mode that still won t entirely solve the problem. No matter how big your display is there are occasions when you need to use it all for a single application while still having the ability to rapidly switch to another application. User interface tweaks to allow rapid task switching without losing application context are necessary. Finally for the past two years that I ve been using Android devices I have been disappointed in the ways that they compare poorly to the iPaQ running Familiar I used 10 years ago. I once wrote a feature article for a magazine on an iPaQ while so far I haven t even written a blog post on an Android device. I think that some of the earlier Android devices might have been better in some ways, the trackball on the HTC Nexus One might have made it more suitable for writing long articles than more recent Android devices.

Russ Allbery: afs-monitor 2.4

I love free software. Two more patches from users of my AFS monitoring scripts (primarily for Nagios) on the heels of the last release, so have another release. This one adds another ignore line to check_afs_bos for the output when there are long-running scheduled jobs, and adds an optional regex filter to check_afs_quotas to select the volumes whose quota one wants to check. Thanks to Georg Sluyterman and Christian Ospelkaus for the patches. You can get the latest version from the afs-monitor distribution page. As before, Debian packages have been uploaded to my personal repository. Also, a quick note about a previous journal entry: Philip Martin and Bernhard R. Link pointed me at the relevant part of the gpg source, which will only check signatures if all signatures use the same class and digest. I updated my previous entry. Thank you!

12 January 2013

Sylvestre Ledru: Some more cool stuff with LLVM/Clang

One of the drawback of C is the hard memory management. It is because of this aspect that Clang provides a way to detect, at runtime, memory errors. Called Address Sanitizer, it allows, while the program is running, to keep track of the memory and detect typical errors (out-of-bound accesses, use of a variable after a free, etc).
# With the following packages (version 3.2-1~exp3)
$ sudo apt-get install clang clang-3.2 compiler-rt -t experimental
Taking the simple following example, even if the errors are obvious, compilers will accept this code.
#include <stdlib.h>
int main()
char *x = (char*)malloc(10 * sizeof(char*));
free(x);
return x[5];
Built and run with:
$ clang -O1 -g -fsanitize=address -fno-omit-frame-pointer foo.c -o foo
$ ./foo &> memoryDebug.log
The previous command will generate a log file. Log which can be post processed with the asan_symbolize command.
$ asan_symbolize memoryDebug.log
which will give:
=================================================================
==21368== ERROR: AddressSanitizer: heap-use-after-free on address 0x7fb22e547f45 at pc 0x408c44 bp 0x7ffff60c10b0 sp 0x7ffff60c10a8
READ of size 1 at 0x7fb22e547f45 thread T0
    #0 0x408c43 in main /tmp/foo.c:5
    #1 0x7fb22d99f6ac in __libc_start_main /home/adconrad/eglibc-2.16/csu/libc-start.c:227
0x7fb22e547f45 is located 5 bytes inside of 80-byte region [0x7fb22e547f40,0x7fb22e547f90)
freed by thread T0 here:
    #0 0x408c90 in __interceptor_free ??:0
    #1 0x408c0a in main /tmp/foo.c:4
    #2 0x7fb22d99f6ac in __libc_start_main /home/adconrad/eglibc-2.16/csu/libc-start.c:227
previously allocated by thread T0 here:
    #0 0x408d50 in __interceptor_malloc ??:0
    #1 0x408bff in main /tmp/foo.c:3
    #2 0x7fb22d99f6ac in __libc_start_main /home/adconrad/eglibc-2.16/csu/libc-start.c:227
Shadow byte and word:
  0x1ff645ca8fe8: fd
  0x1ff645ca8fe8: fd fd fd fd fd fd fd fd
More shadow bytes:
  0x1ff645ca8fc8: fa fa fa fa fa fa fa fa
  0x1ff645ca8fd0: fa fa fa fa fa fa fa fa
  0x1ff645ca8fd8: fa fa fa fa fa fa fa fa
  0x1ff645ca8fe0: fa fa fa fa fa fa fa fa
=>0x1ff645ca8fe8: fd fd fd fd fd fd fd fd
  0x1ff645ca8ff0: fd fd fd fd fd fd fd fd
  0x1ff645ca8ff8: fa fa fa fa fa fa fa fa
  0x1ff645ca9000: fa fa fa fa fa fa fa fa
  0x1ff645ca9008: fa fa fa fa fa fa fa fa
Stats: 0M malloced (0M for red zones) by 1 calls
Stats: 0M realloced by 0 calls
Stats: 0M freed by 1 calls
Stats: 0M really freed by 0 calls
Stats: 0M (128 full pages) mmaped in 1 calls
  mmaps   by size class: 8:2047;
  mallocs by size class: 8:1;
  frees   by size class: 8:1;
  rfrees  by size class:
Stats: malloc large: 0 small slow: 1
==21368== ABORTING
The main advantage compare to valgrind is that asan is supposed to be way more faster. Threads can be also tricking to develop.
With the following example from stolen from upstream:
#include <pthread.h>
int Global;
void *Thread1(void *x)
Global = 42;
return x;
int main()
pthread_t t;
pthread_create(&t, NULL, Thread1, NULL);
Global = 43;
pthread_join(t, NULL);
return Global;
$ clang -fsanitize=thread -g -O1 foo2.c -fPIE -pie -o foo
$ ./foo
==================
WARNING: ThreadSanitizer: data race (pid=21416)
  Write of size 4 at 0x7f2f1a214a50 by thread 1:
    #0 Thread1 /tmp/foo2.c:4 (exe+0x00000000f850)
  Previous write of size 4 at 0x7f2f1a214a50 by main thread:
    #0 main /tmp/foo2.c:10 (exe+0x00000000f8a4)
  Thread 1 (tid=21417, running) created at:
    #0 pthread_create ??:0 (exe+0x00000001267e)
    #1 main /tmp/foo2.c:9 (exe+0x00000000f894)
==================
ThreadSanitizer: reported 1 warnings
Note that clang also provides scan-build, a static analyzer for memory issues. Not as powerful as the Address Sanitizer (it only works on a file), it provides some excellent reports. See the Wouter's blog post on this subject or the automatic report of scan-build on Scilab.

Russ Allbery: Review: Asimov's, April/May 2011

Review: Asimov's Science Fiction, April/May 2011
Editor: Sheila Williams
Issue: Volume 35, No. 4 & 5
ISSN: 1065-2698
Pages: 192
Williams's editorial this issue is about the tendency of SF to take a rose-colored view of the world, which on the surface seems odd given the tendency of recent SF towards dystopia. But she makes a good point that the portrayal of the past is rose-colored, linking that into the current steampunk trend. She doesn't take the argument quite as far as I'd like, but I'm glad to see editorials raising points like this. I'm inclined to think that a lot of the rose-colored frame of the past is because few of us want to read about real historic conditions at any length, even for edification, because the stench and discomfort isn't fun to read about. Silverberg's column is another discussion of programmatic plot generators, which mostly makes the point that plot ideas are the easy part of writing. James Gunn contributes an extended biography of Isaac Asimov that probably won't be new to long-time genre readers but may fill in some details (although it politely sticks to mostly flattering material). Spinrad's book review column is one of his better ones; it looks at two novels by China Mi ville and two by Ian McDonald and explores differences in world-building. Spinrad predictably makes the case in favor of science fiction with rules and against the New Weird, but the discussion along the way was worth reading. "The Day the Wires Came Down" by Alexander Jablokov: Speaking of steampunk, here's an example. There is even an airship, although the primary technological focus is suspended street cars. Jablokov postulates a city-wide transportation network of suspended carriages called telpher cars, along with a city built around the telpher cables: stores on roofs, windows displaying merchandise to passing cars, and even a history of heated competition and dirty tricks between competing telpher networks. The story is set, as the title would indicate, on the last day of the network. It's being shut down for cost, with some hints that progress is destroying something precious. There is a plot here, revolving around some mysteries of the history of the telpher network and the roles of several people in that history. But the story is primarily a celebration of old technology. It's a rail fan's story recast with a steampunk technology, featuring the same mix of fascination with mechanics and a sense that the intricate details are falling out of common knowledge (and perhaps usefulness). As a story, it's a bit slow-moving, but I enjoyed the elegiac tone. (7) "An Empty House with Many Doors" by Michael Swanwick: This is a very short story, more of an emotional profile, involving a man's reaction to the death of his wife. Oh, and parallel universes. It's sort of the inverse of Niven's classic "All the Myriad Ways." Similar to Niven's story, I found the idea vaguely interesting but the conclusion and emotional reaction unbelievable and alien. (5) "The Homecoming" by Mike Resnick: Resnick tends to yank on the heart-strings rather sharply in his stories, so I knew roughly what to expect when a father comes home to find his son is visiting. A son who, rather against his father's wishes, has been significantly altered to be able to live with aliens. Throw in a mother with serious dementia, and you can probably predict what Resnick does with this. Still, most of the story is a two-sided conversation, and I thought he succeeded in doing justice to both sides, even though one of them was destined to lose. (6) "North Shore Friday" by Nick Mamatas: Illegal Greek immigrants, a family-run system for getting them married off before the INS catch them, government psi probes and eavesdropping on thoughts, joint projects between computer and religion departments, secret government experiments, and even ghosts... this story is a complex mess, with numerous thoughts stuck into small boxes and scattered through the surface story. It's one of those stories where figuring out what's going on, and even how to read the story in a sensible way, is much of the fun. If you find that fun, that is; if not, it will probably be frustrating. I wished there was a bit more plot, but there's something delightful about how much stuff Mamatas packs into it. (6) "Clockworks" by William Preston: This is a prequel to Preston's earlier "Helping Them Take the Old Man Down". Like that story, it's primarily a pulp adventure, but layered with another level of analysis and thoughtfulness that tries to embed the pulp adventure in our understanding of human behavior and the nature of the world, although this one stays a bit more pulp than its predecessor. As with Preston's other story, we don't get directly in the head of the Old Man (here, just called the man, but identifiable from clues in both stories as Doc Savage); instead, the protagonist is a former villain named Simon Lukic who the man hopes to have fixed by operating on his brain. The undercurrent that lies beneath a more typical pulp adventure is the question of whether Lukic is actually healed. I think there was a bit too much daring-do and human perfection, but it's a perfectly servicable pulp story with some depth. (6) "The Fnoor Hen" by Rudy Rucker: If you've read any of Rucker's work before, you probably know what to expect: a mind-boggling blizzard of mathematically-inspired technobabble that turns into vaguely coherent surrealism. (You can probably tell that I'm not much of a fan, although the clear good humor in these stories makes it hard to dislike them too much.) There's a mutated chicken and some sort of alternate mathematical space and then something that seems like magic... I'd be lying if I said that I followed this story. If you like Rucker, this seems like the sort of thing that you'd like. (4) "Smoke City" by Christopher Barzak: At the start of this story, I thought it was going to be an emotional parable about immigration. The progatonist lives two lives: one in our world, and one in the Smoke City of industry, a world of hard labor, pollution, and little reward, with families in both. But nearly all of the story is set within Smoke City, and the parable turns out to be a caustic indictment of industry and its exploitation of labor. I kind of wish Barzak hadn't used rape as a metaphor, but when the captains of industry show up, I can't argue with how deeply and accurately the story shoves in the knife. There isn't much subtlety here, but it's still one of the better stories in this issue. (7) "A Response from EST17" by Tom Purdom: I'm very happy to see Purdom's writing appearing regularly. His stories are always quiet and matter-of-fact, and at first seem to miss emotional zest, but they almost always grow on me. He lets the reader fill in their own emotional reactions to events, and there's always a lot going on. This story is a first-contact story, except that the "humans" here are not human at all. They're automated probes sent by two separate human civilizations, with different programming and different governance algorithms, and they quickly start competing negotiations. The aliens they've discovered similarly have factions, who start talking to the different probes in an elaborate dance of gathering information without giving too much away. The twist is that this pattern has replayed itself many times in the past, and information itself can be a weapon. I enjoyed this one from start to finish. (7) "The One That Got Away" by Esther M. Friesner: Friesner is best known, at least to me, for humorous fantasy, and this story is advertised as such from early on. The first-person protagonist is a prostitute in a seaside town. She's bemused to finally be invited over by a sailor who's been eyeing her all evening, but that sailor has something else in mind than normal business. For much of this story, the fantasy element is unclear; when it finally comes, it was an amusing twist. (7) "The Flow and Dream" by Jack Skillingstead: This is a mildly interesting variation on the old SF story of hibernating humans (on a generation ship or elsewhere) waking up to a transformed world. Here, it's not a ship, it's a planet, and the hiberation was to wait for terraforming rather than for transit. The twist comes from an excessively literal computer and the fun of putting together the pieces. Sadly, the story trails off at the end without much new to say. (5) "Becalmed" by Kristine Kathryn Rusch: "Becalmed" takes place immediately before "Becoming One with the Ghosts" and explains the incident that created the situation explored in that story. The first-person protagonist of "Becalmed" is a linguist, an expert in learning alien languages so that the Fleet can understand the civilizations that it runs across. But something went horribly wrong at their last stop, something that she's largely suppressed, and now she's confined to quarters and possibly in deep trouble. As is the ship; they're in foldspace, and they have been for days. "Becalmed" is structed like a mystery, centered around recovering the protagonist's memories. It's also a bit of a legal procedural; the ship is trying to determine what to do with her and to what degree she's responsible. But the heart of the story is a linguistic and cultural puzzle. This is another great SF story from Rusch, whose name on a cover will make me eager to start reading a new magazine. I love both angles on the universe she's built, but I think I like the Fleet even better than the divers. The Fleet captures some of the magic of the original Star Trek, but with much more mature characters, more believable situations, and a more sensible and nuanced version of the Prime Directive. Rusch writes substantial, interesting plots that hold my interest. I'd love to see more like this. (8) Rating: 7 out of 10

11 January 2013

Patrick Matth i: What an ugly (PHP) work..

We still have got some more or less webapplications which are not compatible with PHP higher than version 5.2.x, which is the only blocker for the last Lenny servers to upgrade them to Squeeze.. I do not think that I am alone with this ****** topic :) So the new masterplan is to deploy those applications on seperated Wheezy servers, with PHP 5.2.x running as FastCGI, so that most parts of the system are security supported . First; I didn t documented my steps and I am not 100% done (something like 99%) but I have done the following to have it as clean as possible : PHP 5.2.x is just not able to detect the new multiarch paths, it fails at most dir options. Since patching the whole build system would be *too* much work I decided to hack around this (ln -s /usr/lib/x86 ./foo.so /usr/lib/), then some adjustions to the build dependencies, disablieng some modules, like SSL and libdb (incompatible versions), disabling merged patches and refresh the suhosin hardening patch; I get an working PHP 5.2.17 package on Wheezy.
But this is too easy!
I want packages which I could co-install with the PHP 5.4 packages from Wheezy, 5.2 should only used withing vHosts where I have enabled them.. So I rewrote the whole packaging (*burg* IMHO at all) to use php52 instead of php5 as packaging namespace and also everything is put into /opt as prefix. Much painfull work, but yeah it works.. :)
Some packages, like php52-dev or php-pear are broken, but those were not my goal of this action. If someone is interested in those packages please send me an email.
Since PHP 5.2 is not supported any longer (and that this is at all a big hack) I will not publish the source and binaries at all.

Neil Williams: perl-cross-debian updates

Working with perl upstream, we're getting closer to a fully cross-built upstream perl without needing the external perl installation. The branch (which is also available here with a few of my changes) now builds a host miniperl, cross-builds the rest of perl and almost gets through the rest of the build by using the host miniperl to handle the extensions up as far as XS::Typemap:

make[1]: Leaving directory /home/neil/code/debian/src/perl/cross/git/perl/ext/XS-APItest'
./miniperl -Ilib make_ext.pl lib/auto/XS/Typemap/Typemap.so MAKE=make LIBPERL_A=libperl.a LINKTYPE=dynamic
Making XS::Typemap (all)
make[1]: Entering directory /home/neil/code/debian/src/perl/cross/git/perl/ext/XS-Typemap'
make[1]: Leaving directory /home/neil/code/debian/src/perl/cross/git/perl/ext/XS-Typemap'
Making all in ext/XS-Typemap
make all PERL_CORE=1 LIBPERL_A=libperl.a LINKTYPE=dynamic
make[1]: Entering directory /home/neil/code/debian/src/perl/cross/git/perl/ext/XS-Typemap'
make[1]: Leaving directory /home/neil/code/debian/src/perl/cross/git/perl/ext/XS-Typemap'
./perl -f -Ilib pod/buildtoc -q
Can't load module Digest::MD5, dynamic loading not available in this perl.
(You may need to build a new perl executable which either supports
dynamic loading or has the Digest::MD5 module statically linked into it.)
at Porting/pod_lib.pl line 4.
Compilation failed in require at Porting/pod_lib.pl line 4.
BEGIN failed--compilation aborted at Porting/pod_lib.pl line 4.
Compilation failed in require at pod/buildtoc line 17.
BEGIN failed--compilation aborted at pod/buildtoc line 18.
make: *** [pod/perltoc.pod] Error 2

Note the change from ./miniperl (which is itself a bug as it should be ./host/miniperl) to ./perl which is, naturally, an armel binary. It's also copied into the local directory, replacing the system perl if I copy it in.

So, more to do, but at least it gets this far.

Improved support for the extensions should also make it easier to clean up the current Debian cross-build diff which is the remaining bit of awkwardness / kludge.

Julien Danjou: Overriding cl-json object encoding

CL-JSON provides an encoder for Lisp data structures and objects to JSON format. Unfortunately, in some case, its default encoding mechanism for CLOS objects isn't exactly doing the right thing. I'll show you how Common Lisp makes it easy to change that. Identifying the problem CL-JSON & CLOS CL-JSON mechanism encoding CLOS object is really neat. Let's see how it works for a simple case:
(defclass kitten ()
((tail :initarg :tail)))

(json:encode-json-to-string (make-instance 'kitten :tail 'black))

will produce:
 "tail":"black" 

Still using CL-JSON, we can also decode the JSON object to a CLOS object:
(slot-value
(json:with-decoder-simple-clos-semantics
(json:decode-json-from-string " \"tail\":\"black\" "))
:tail)

That code will return "black". Note that it's also possible to specify which class should be used when decoding objects, but that's beyond the purpose of this article. Postmodern Now, let's introduce Postmodern, a wonderful Common Lisp system providing access to the wonderful PostgreSQL database. It also provides a simple system to map rows in a database to CLOS classes, called DAO for Database access objects. With this, we can easily store our kitten into a table.
(defclass kitten ()
((tail :initarg :tail))
(:metaclass postmodern:dao-class))

If we try to encode this to JSON, it will produce the exact same result seen previously. The problem is what happens when one of our column has a NULL value. Postmodern encodes this using the :null symbol. So this code:
(defclass kitten ()
((tail :initarg :tail :col-type (or s-sql:db-null text)))
(:metaclass postmodern:dao-class))

(postmodern:deftable kitten
(postmodern:!dao-def))

(postmodern:connect-toplevel )

(postmodern:create-table 'kitten)

(json:encode-json-to-string
(postmodern:make-dao 'kitten))

will return:
" "tail":"null" "

Fail! The fact that the column is NULL is represented by the :null symbol. And CL-JSON encodes all symbols as string. This is not at all what we want here! Overriding encode-json CL-JSON provides and uses the encode-json method to encode all kind of object. It is defined as a generic function, and a lot of different methods are implemented to handle the different standard Common Lisp types. The one used for standard-object is defined liked that:
(defmethod encode-json ((o standard-object)
&optional (stream *json-output*))
"Write the JSON representation (Object) of the CLOS object O to
STREAM (or to *JSON-OUTPUT*)."
(with-object (stream)
(map-slots (stream-object-member-encoder stream) o)))

All we need to do here, is to create a new method for our kitten objects, that handles correctly the :null case.
(defclass kitten ()
((tail :initarg :tail :col-type (or s-sql:db-null text)))
(:metaclass postmodern:dao-class))

(export 'kitten)

;; Switch package just to define the new method
(in-package :json)
(defmethod encode-json ((o cl-user:kitten)
&optional (stream json:*json-output*))
"Write the JSON representation (Object) of the postmodern DAO CLOS object
O to STREAM (or to *JSON-OUTPUT*)."
(with-object (stream)
(map-slots (lambda (key value)
(as-object-member (key stream)
(encode-json (if (eq value :null) nil value) stream)))
o)))

;; Go back into our package
(in-package :cl-user)

(postmodern:deftable kitten
(postmodern:!dao-def))

(postmodern:connect-toplevel )

(postmodern:create-table 'kitten)

(json:encode-json-to-string
(postmodern:make-dao 'kitten))

With that new method, as soon as we encounter a :null symbol as a value for an object's slot, we replace it by nil. Now if we try to encode another kitten, we'll get:
 "tail":null 

which is far better for our JavaScript data consumers! In the end, I think that this kind of trick is feasible that easily because of the way CLOS provides its generic method implementation. The fact that methods don't belong to any class makes the extension of every program, library and class so much easier. Doing this in another language like Java would likely by impossible, and in Python it would unlikely be as clean as it is done in Common Lisp. The ability to teach any library about how it should handle your class just by defining a new method is really handy!

Benjamin Mako Hill: Goodbye PyBlosxom, Hello WordPress

Since 2004, I ve used the blogging software PyBlosxom. Over that time, the software has served me well and I have even written a series of patches and plugins. PyBloxsom is blog software designed for hackers. It assumes you already have a text editor you love and relies on features of a POSIX filesystem instead of a relational database. It s designed so you can keep your blog under revision control (since 2004 I ve used GNU Arch, baz, bzr and now git). It is also hackers software in the sense that you should expect to write code to use it (e.g., the configuration is pure Python). I love it. What PyBlosxom does not have is a large community. This summer, the most recent long-time maintainer of the project, Will Kahn-Greene, stepped down. Although the project eventually found a new maintainer, the reality is the project entered maintenance mode years ago and many features available in more popular blogging platforms are unlikely to make it PyBlosxom. The situation with comment spam is particularly dire. I ve written several antispam plugins but time has shown that I don t have the either the expertise or the time to make them as awesome as they need to be to really work in today s web. So, after many months of weighing, waffling, and planning I ve switched to WordPress a great piece of free software with an enormous and established community As you ll know if you ve read my interview on The Setup, I think a lot about the technology I surround myself with. I considered WordPress when I started my blog back in 2004 and rejected it soundly. Eight years ago, I would have laughed at you if you told me I d be using it today; If PyBlosxom is for hackers, WordPress is designed for everyone else. But the way I evaluate software has changed over that period. In the nineties, I used to download every new version of the Linux kernel to compile it it took hours! to try out the latest features. Configurabilty, hackability, and the ability to write my own features was after a point more important than the features the software came with. Today, I m much more aware of the fact that for all the freedom that my software gives me, I simply do not have the time, energy, or inclination to take advantage of that freedom to hack very often. Today, I give much more value to software that is not just free, but that is maintained by a community of people who can do all the work that I would do if I had unlimited time. Although I don t have the time or experience to make WordPress do everything I would like, the collective of all WordPress users does. And they ve done a lot of it already! The flip side matters as well: Today, I give more value to other people using my software. When WordPress doesn t do something and I write a plugin or patch, there are tons of people ready to pick it up and use it and perhaps even to collaborate on it with me. Want to guess how many patches my PyBlosxom plugins have received? None, if my memory serves me. In the past, I ve written about how free software is a victory even when it doesn t build a community. I still believe that. But the large communities at the heart of the most successful free software communities (the promise of open source ) are deeply important in a way that I increasingly value. In that spirit: If you want to make the jump from PyBlosxom to WordPress, I ve shared a Git repository with the scripts I used and wrote for the transition.

Julian Andres Klode: Recursive-descent in Python, using generators

Writing recursive descent parsers is easy, especially in Python (just like everything is easy in Python). But Python defines a low limit on the number of recursive calls that can be made, and recursive descent parsers are prone to exceed this limit. We should thus write our parser without real recursion. Fortunately, Python offers us a way out: Coroutines, introduced in Python 2.5 as per PEP 342. Using coroutines and a simple trampoline function, we can convert every mutually recursive set of functions into a set of coroutines that require constant stack space. Let s say our parser looks like this (tokens being an iterator over characters):
def parse(tokens):
    token = next(tokens)
    if token.isalnum():
        return token
    elif token == '(':
         result = parse(tokens)
         if next(tokens) != ')': raise ...
         return result
    else:
         raise ...
We now apply the following transformations:
return X => yield (X)
parse() => (yield parse()) That is, we yield the value we want to return, and we yield a generator whenever we want to call (using a yield expression). Our rewritten parser reads:
def parse(tokens):
    token = next(tokens)
    if token.isalnum():
        yield token
    elif token == '(':
         result = yield parse(tokens)
         if next(tokens) != ')': raise ...
         return result
    else:
         raise ...
We obviously cannot call that generator like the previous function. What we need to introduce is a trampoline. A trampoline is a function that manages that yielded generators, calls them, and passes their result upwards. It looks like this:
def trampoline(generator):
    """Execute the generator using trampolining. """
    queue = collections.deque()
    def schedule(coroutine, stack=(), value=None):
        def resume():
            if 0:
                global prev
                now = stack_len(stack)
                if now < prev:
                    print("Length", now)
                    prev = -1
                elif prev != -1:
                    prev = now
            result = coroutine.send(value)
            if isinstance(result, types.GeneratorType):     # Normal call
                schedule(result, (coroutine, stack))
            elif isinstance(result, Tail):                  # Tail call (if you want to)
                schedule(result.value, stack)
            elif stack:                                     # Return to parent
                schedule(stack[0], stack[1], result)
            else:                                           # Final Return
                return result
        queue.append(resume)
    schedule(generator)
    result = None
    while queue:
        func = queue.popleft()
        result = func()
    return result
This function is based on the code in PEP 342, the difference being that For a more generic version of that, you might want to re-add exception passing, but the exceptions will then have long tracebacks, so I m not sure how well they will work if you have deep recursion. Now, the advantage of that is that our parser now requires constant stack space, because the actual real stack is stored in the heap using tuples which are used like singly-linked lists in scheme here. So, the only recursion limit is available memory. A disadvantage of that transformation is that the code will run slightly slower for small inputs that could be handled using a normally recursive parser.
Filed under: Python

Russ Allbery: release 1.45 and multiple PGP signatures

release is the tool that I use to manage software releases. This version adds support for signing a tarball with multiple PGP keys at once. For WebAuth, we have an official signing key, but I also wanted to add my personal signature. The minimum supported version of Python was also bumped to 2.5 (see below for why). You can get the latest version from my scripts page. Normally, I wouldn't announce this to everyone, since it's a fairly minor change, except I had a question and a quick Python note. The question: how does one get GnuPG to verify multiple signatures? When I run GnuPG on the resulting detached *.asc file created by this new code, I get the following:
gpg: WARNING: multiple signatures detected.  Only the first will be checked.
gpg: Signature made Thu 10 Jan 2013 03:58:43 PM PST using RSA key ID 5736DE75
gpg: Good signature from "Russ Allbery <rra@stanford.edu>"
gpg:                 aka "Russ Allbery <rra@debian.org>"
gpg:                 aka "Russ Allbery <eagle@windlord.stanford.edu>"
gpg:                 aka "Russ Allbery <eagle@eyrie.org>"
That's nice, I suppose, but it's not as helpful as it could be. Why not verify both signatures and report all of the results? I searched both the gpg man page and the Internet for some option I'm missing, without any luck. (Searching Google for that exact message amusingly turns up lots of verbatim output from Debian repository verification and almost nothing else.) If anyone knows, send me email (rra@stanford.edu or any other convenient address), and I'll update this post with the answer. UPDATE: gpg can currently only verify multiple signatures if the signatures are from keys with the same class and digest. I was attempting this with a 2048-bit RSA key and a 1024-bit DSA key, which gpg does not support. (Yet another reason to replace the WebAuth signing key with a newly-generated key.) Thanks to Philip Martin and Bernhard R. Link for the information! The note is that, way back when I was first learning Python, one of the things that bothered me the most was the lack of safe program execution. The best available was os.spawnv(), which left something to be desired. If you too were bothered by this but don't program in Python enough to keep track of the current state of the art, there's now a subprocess module that has much saner interfaces that work properly, do reasonable things, and avoid the shell. It was added in Python 2.4, with some nicer interfaces added in Python 2.5, so it's fairly safe to use at this point.

10 January 2013

Petter Reinholdtsen: Moved the pymissile Debian packaging to collab-maint

As part of my investigation on how to improve the support in Debian for hardware dongles, I dug up my old Mark and Spencer USB Rocket Launcher and updated the Debian package pymissile to make sure udev will fix the device permissions when it is plugged in. I also added a "Modaliases" header to test it in the Debian archive and hopefully make the package be proposed by jockey in Ubuntu when a user plug in his rocket launcher. In the process I moved the source to a git repository under collab-maint, to make it easier for any DD to contribute. Upstream is not very active, but the software still work for me even after five years of relative silence. The new git repository is not listed in the uploaded package yet, because I want to test the other changes a bit more before I upload the new version. If you want to check out the new version with a .desktop file included, visit the gitweb view or use "git clone git://anonscm.debian.org/collab-maint/pymissile.git".

Next.

Previous.