Search Results: "tar"

16 January 2013

Russ Allbery: afs-admin-tools 2.0

For years, I've distributed the individual AFS administrative tools that we use at Stanford as separate software releases on my web site, but this has clearly been less and less efficient. We've needed to package them for use at Stanford and didn't want to create separate packages for each individual script, we've wanted to get them out of their current CVS repository, and they didn't support any unified configuration system or simple method for local site customization. I started trying to solve those problems over a year ago by converting all the scattered CVS files into a coherent Git repository and turning the collection of scripts into an actual distribution that I can version and release more like regular software packages. But other things kept coming up, and it's one of the yearly goals I had for last year that I didn't finish. Today, I finally took the time to finish this off. The scripts frak, fsr, lsmounts, mvto, partinfo, volcreate, volcreate-logs, and volnuke, which were previously distributed separately, are now collectively distributed as afs-admin-tools and all support a standard configuration file and location. I also rolled up some previously unreleased fixes and removed some remaining Stanford-specific behavior. I have not, with this release, incorporated any of the various changes that people have sent me to the individual scripts. That's next. They're also all still standalone scripts without any common code moved into a Perl module, but I may change that in the future. Note that some of these scripts refer to scripts that will be part of the afs-mountpoints package. Putting that together is my next task, but it's a bit trickier since we moved the mount point database into an actual database, so I need to put some work into the Perl modules that wrap that database. Hopefully I can get that released before too long as well. You can get the latest release from the (new! shiny!) afs-admin-tools distribution page.

15 January 2013

Michal Čihař: New theme for phpMyAdmin

phpMyAdmin just got a new theme. It's available for download on our themes page. New theme reflects look of new Windows and I have to admit it looks quite nice: Metro theme screenshot As you can see, it also offers four color variants, configurable in it's layout.inc.php file. PS: More details on theme can be found on deviantART.

Filed under: English Phpmyadmin 0 comments Flattr this!

Daniel Kahn Gillmor: in memory of Aaron Swartz

I was upset to learn about Aaron Swartz's death last week. I continue to be upset about his loss, and about our loss. He didn't just show promise of great things to come in the future -- he had already done more work for the public good than many of us will ever do. I'd only met him IRL a couple times, but (like many others) i had encountered him on the 'net in many places. He was a good person, someone i didn't need to always agree with to respect. I read Russ Allbery's posts about Aaron and "slacktivism" with much appreciation. I had been ambivalent about signing the whitehouse.gov petition asking for the removal of the prosecutor for overreach, because I generally distrust the effectiveness of online petitions (and offline petitions, for that matter). But Russ's analysis convinced me to go ahead and sign it. The petition is concrete, clear (despite wanting a grammatical proofread), and actionable. For people willing to go beyond petition signing to civil disobedience, The Aaron Swartz Memorial JSTOR Liberator is an option. It makes it straightforward to potentially violate the onerous JSTOR terms of service by re-publishing a public-domain article from JSTOR to archive.org, where it will be accessible to anyone directly. As someone who builds and maintains information/communications infrastructure, i have very mixed feelings about most online civil disobedience, since it often takes the form of a Distributed Denial of Service (DDoS) attack of some sort. DDoS attacks of public services are notoriously difficult to defend against without having huge resources to throw at the problem, so encouraging participation in a DDoS often feels a little bit like handing out cans of gasoline when you know that everyone is living in a house of straw. However, the JSTOR Liberator is not a DDoS at all -- it's simply a facilitation of people breaking the JSTOR Terms of Service (ToS), some of the same terms that Aaron was facing charges for violating. So it is a well-targeted way to demonstrate that the prosecutions were overreaching. I wanted to take issue with one of Russ' statements, though. In his second post about the situation, Russ wrote:
Social activism and political disobedience are important and often valuable things, but performing your social activism using other people's stuff is just rude. I think it can be a forgivable rudeness; people can get caught up in the moment and not realize what they're doing. But it's still rude, and it's still not the way to go about civil disobedience.
While i generally agree with Russ' thoughtful consideration of consent, I have to take issue with this elevation of some sort of hyper-extended property right over the moral agency that drives civil disobedience. To use someone else's property for the sake of a just cause without damaging the property or depriving the owner of its use is not "forgivable rudeness" -- it's forgivable, laudable even, because it is just. And the person using the property doesn't need to be "caught up in the moment and not realize what they're doing" for it to be acceptable. Civil disobedience often involves putting some level of inconvenience or discomfort on other people, including innocent people. It might be the friends and family of the activist who have to deal with the jail time; it might be the drivers stuck in a traffic jam caused by a demonstration; it might be the people forced to shop elsewhere because the store's doors are barricaded by protestors. All of these people could be troubled by the civil disobedience more than MIT's network users and admins were troubled by Aaron's protest, and that doesn't make the protests described worse or "not the way to go about civil disobedience." The trouble highlights a more significant injustice, and in its troubling way does what it can to help right it. Aaron was a troublemaker, and a good one. He will be missed. Tags: aaronsw

Russ Allbery: More on Aaron Swartz

I got some feedback that the analogy I used in my last post was a bit confusing, and indeed I blew the phrasing of the analogy (also now corrected). So let me try this again, since I think there's a subtlety here that may be missed. I should note for the record that my understanding of what Swartz did that started the process is apparently somewhat based on the description from the prosecution, so it may not be the complete or accurate facts. Since there will now be no trial, we may never find out what the defense was, and whether those facts would be challenged. So it may be best to think of this as a hypothetical. We never established, or will establish, in court exactly what happened. Swartz was, generally speaking, charged with two things that I consider quite distinct, at least from an ethical perspective. Most of the focus is on the copyright part: downloading JSTOR articles with an intent (never acted upon) to distribute them to the world. There are a bunch of reasons why this may or may not be justified, which are tied into the origin of those articles (many of them were publicly funded) and the legitimacy of copyright licensing agreements. I think there is significant room to hold a variety of opinions on this, although I don't believe that "crime worth 35 years in prison" is even remotely close to justifiable under any interpretation. However, there is a second part of what he was charged with, and that's primarily what I was commenting on, not the copyright part. He allegedly hooked up a laptop in an unlocked MIT wiring closet without permission and then used MIT's network and JSTOR license to download information. This is, to me, a subtlely but entirely distinct act from the question of whether taking JSTOR's data was ethically or legally wrong. Whether or not one believes that JSTOR's copyrights are not legitimate, it's still not okay to use someone else's network and license or to trespass in their wiring closets without permission. (I work in central IT for a university, so this strikes closer to home.) And this is where the analogy came in, which I flubbed. I had originally said that this was akin to "traipsing into someone's barn or backyard shed without their permission and taking some of their tools because you want to use them." The word "taking" was wrong; it falsely implies that you weren't going to return the tools. I had been thinking "taking and then returning" when I wrote that, and the important second part didn't make it into the post. So let's try this again: Swartz's actions at MIT, as I understand them (and many things could change this, such as a revelation that he had MIT's prior permission), are akin to going onto someone's property and into their barn without permission, borrowing their hedge trimmer for a while because you want to use it, and returning the hedge trimmer without any damage when you're done (and without them noticing it was gone). I'm quite fond of this analogy, since I think it clearly establishes two things:
  1. Most people are going to be unhappy about this happening and will intuitively feel like it should probably be illegal. Not everyone; there are folks who don't believe in personal property, or at least wouldn't extend it to tools in a barn. But most people will feel that someone should ask first before they come borrow your tools. Even if they don't damage them, even if they return them before you notice they're gone, you might have wanted to use the tool at the same time or they might have damaged them without intending to, and they should just ask first. It's common politeness, and depending on the circumstances, someone who doesn't ask and is covert about borrowing tools might be worth calling the police over.
  2. There is absolutely no way in any reasonable moral system that doing this should result in 35 years in prison. Or even 10, or even 1. Yes, most people would consider this a crime, but most people would consider it a minor crime. It's the sort of thing where you might have to impose some consequences just to make sure the message of "knock it off" is delivered firmly, but someone doing this is rude and inconsiderate, not evil.
The JSTOR copyright stuff is more complex to analyze and is more politically divisive, but for me the key points are (a) Swartz never released the data, and (b) JSTOR declined to press charges. To me, that means the deeper copyright questions, which are quite interesting, were never actually reached in this particular case. The crime that did apparently happen was the trespass at MIT, for which I think the above analogy is the right way to think about it. The point I do want people to take away from this is that one should not overlook the trespass at MIT even if one wants to celebrate the undermining of the copyright regime and doesn't believe JSTOR's data should be considered their private property. Social activism and political disobedience are important and often valuable things, but performing your social activism using other people's stuff is just rude. I think it can be a forgivable rudeness; people can get caught up in the moment and not realize what they're doing. But it's still rude, and it's still not the way to go about civil disobedience. For both ethical and tactical reasons, involving bystanders in your act of social activism without their consent is a bad approach. ETA: The problem, of course, with discussing all of this is that while it's relevant and possibly even somewhat important in the broader sense of how our community acts going forward, it also doesn't capture the fact that this was only one incident in a remarkable life. One of the worst problems with the abusive prosecution of Swartz is that it blew this incident completely out of proportion. A moment of arguable judgement should not dominate one's life or cast a shadow over all of one's other accomplishments; the prosecution tried to make it do just that. That's part of what I'm arguing, but ironically that partly feeds into the lack of proportionality in the discussion. I've gotten kind of far afield here, so let me go back and say explicitly: Swartz was a remarkable person who did much to admire and respect, and the world is a worse place without him.

14 January 2013

Russell Coker: Promoting Enthusiasm

Rusty wrote an insightful post titled What Can I Do To Help? about reactions to new ideas [1]. He suggests that people make an effort to have a positive approach when someone talks about a new idea, it s quite common for people to point out reasons why the new idea might not work out which is discouraging for the person who had the idea. I think that is a really good point. I probably haven t done too well in that regard in the past and will try to do better in future. Code Written by Assholes Rusty previously wrote a post titled If you didn t run code written by assholes, your machine wouldn t boot which implies that we should just let assholes be assholes [2]. That doesn t go well with his What Can I Do To Help? post. Note that I m not accusing Rusty of hypocrisy here, giving advice to help people who want to get along well with others is not in contradiction with refraining from giving unsolicited advice and encouragement to difficult people who have expressed no interest in improving their behavior. A comment on the latter post by Doctor Whom says If I had seen this kind of talk when I was a teenager, I would have thought twice about picking up coding , presumably given the number of people who read Rusty s blog there are some teenagers who experienced some discouragement towards a career in computers (or a hobby in FOSS) from Rusty s post. I ve already written a response to the If you didn t run code written by assholes post, among other things I suggested that people who are minor assholes should be assisted to be less difficult and major assholes should be excluded [3]. In that post I was working on the assumption that for every significant task that needs to be completed (such as making a popular OS bootable) someone will do it, if the person working on it disappears then someone else will take over there is a community of programmers who will work on whatever needs to be done. The Importance of Individuals But in terms of new ideas it really comes down to individuals. Most projects which are significant and important now probably started out as one person or a small group who had an idea that seemed unlikely to succeed at the time. So while any big and successful project can have people replaced (which is among other things a requirement of long-term success) there are situations in which individuals with ideas matter. Another important factor is that even ideas which turn out to be impractical are still useful. Someone who has an impractical idea about a technical issue and investigates it fully will learn a lot and may end up working on the less radical ways of solving similar problems this is good for the individual and the community. Another Way of Promoting Enthusiasm In terms of promoting enthusiasm it seems that one thing that can be done by high profile people is to avoid writing posts like If you didn t run code written by assholes, your machine wouldn t boot . When people in positions of power and influence appear to have no interest in promoting good behavior it really discourages people who are vulnerable to the assholes which among other things means most members of minority groups. Obviously Rusty could t stamp out all asshole behavior, but if he announced a plan to try and make things better in that regard then it would help. It s difficult to be enthusiastic when faced with discrimination from a minority and disinterest from the majority. Of course with the way the Internet works I m sure someone will say what about the assholes who have great ideas, shouldn t we nurture their enthusiasm by letting them keep doing asshole things? . I think that for the major assholes this won t be a problem, for example anyone who s racist will be well aware that many people disagree strongly with them and thus won t be particularly discouraged when they meet more people who disagree. For the minor assholes (people who don t want to be assholes) it will be somewhat discouraging to be corrected, but that could be a learning experience for them that s worth more than support in implementing their latest technical idea. Update: Why Rusty is Important In response to a comment by private mail I ve added this section after publication. Firstly I think that the opinions of all members of the community matter as they all affect the social environment which determines what types of behavior are encouraged and discouraged. But Rusty is more important than most people. Firstly Rusty has a Wikipedia page [4], that alone is an objective criteria indicating his importance. But in terms of influencing people in the FOSS community the most important things are that he s a high profile Linux kernel programmer (which alone gives significant status and influence) and that he s the founder of the first Linux conference in Australia (which is now known as Linux.conf.au AKA LCA). When issues such as the anti-harassment policy for LCA are being discussed any opinion that Rusty offered would be taken very seriously. But so far he doesn t seem to be involved in any of the public discussions.

Robert Collins: Multi-machine parallel testing of nova with testrepository

I recently added a formal interface to testrepository to enable cross-machine scaling of test runs. As testrepository is still a static scheduler, this isn t perfect, but its quite a minimal interface, which makes it easy to implement. I will likely evolve it in reaction to feedback and experience. In the long term I d love to have a super generic tool that matches that interface, so the project VCS copy of .testr.conf can just call out to it. However I don t yet have that, but I do have a simple by-hand implementation that I use to run nova s tests across my personal laptop, desktop and work laptop. Testr models this by assuming each test running process can be mapped to a single instance id (which could be a chroot, vm, cloud instances, ) and then running one or more commands in the instance, before disposing of it. This by hand implementation consists of 4 things:
  1. A tiny script to rsync my source directory to the relevant places before I run tests. (This takes <2seconds on my home wifi).
  2. A script to allocate instance ids (I just use ints)
  3. A script to discard them
  4. And a script to copy tempfiles onto the target machine and run a given command.
I do my testing in lxc containers, because I like my primary environment to be free of project-specific quirks and workarounds. lxc is not needed though, if you don t want it. So, to set this up for yourself:
  1. on each host, make an lxc container (e.g. following) http://wiki.openstack.org/DependsOnUbuntu
  2. start them all (lxc-start -n nova -d)
  3. Make SSH config entries for the lxc containers, so you can get at them remotely. (make sure your host * rules are at the end of the file otherwise the master overrides won t work [and you might not notice for some time...]):
    Host desktop-nova.lxc
    # lxc addresses may be present on localhost too, so namespace the control
    # path to avoid connecting to the wrong container.
      ControlPath ~/.ssh/master-lxc-%r@%h:%p
      hostname 10.0.3.19
      ProxyCommand ssh 192.168.1.106 nc -q0 %h %p
    Host hplaptop-nova.lxc
    # lxc addresses may be present on localhost too, so namespace the control
    # path to avoid connecting to the wrong container.
      ControlPath ~/.ssh/master-lxc-%r@%h:%p
      hostname 10.0.3.244
      ProxyCommand ssh 192.168.1.116 nc -q0 %h %p
  4. make a script to copy your nova source tree to each test location. I called mine sync
    #!/bin/bash           
    cd $(dirname $0)
    echo syncing in $(pwd) 
    (rsync -a . desktop-nova.lxc:source/openstack/nova --delete-after && echo dell done) &
    (rsync -a . hplaptop-nova.lxc:source/openstack/nova --delete-after && echo hp done)
  5. Make sure you have the base directory on each location
    ssh desktop-nova.lxc mkdir -p source/openstack
    ssh hplaptop-nova.lxc mkdir -p source/openstack
  6. Sync your code over.
    ./sync
  7. And check tests run by running a few.
    ssh hplaptop-nova.lxc "cd source/openstack/nova && ./run_tests.sh compute"
    ssh hplaptop-nova.lxc "cd source/openstack/nova && ./run_tests.sh compute"
    This will check the test environment: we re not going to be running tests on each node via run-tests or even testr (because it gets immediately meta), but if this fails, later attempts won t work. Your test virtualenv is inside the source tree, so it is copied implicitly by the sync.
  8. Decide what concurrency you want. For me, I picked 12: I have a desktop i7 with 4 cores, and two laptops with 2 cores each, and hyperthreads are on on all of them I m going to set a concurrency figure of 12 between the cores (8) and threads (16) counts, and possibly balance it more in future. A higher number assumes less contention between ALU s and other elements of the core pipeline, and I expect quite some contention because most of nova s unittests are CPU bound not I/O. If the test servers are not busy, I can always raise it later.
  9. Create scripts to create / dispose / execute logical worker threads.
  10. Creation. I call this instance-provision and all it does is find the lowest ints not currently allocated and return them.
    #!/usr/bin/env python
    import os.path
    import sys
    if not os.path.isdir('.instances'):
        os.mkdir('.instances')
    running_ids = os.listdir('.instances')
    count = int(sys.argv[1])
    top = count + len(running_ids)
    ids = [str(i) for i in range(top)]
    new = set(ids) - set(running_ids)
    for id in new:
        file('.instances/%s' % id, 'w').close()
    print(' '.join(new))
  11. Disposal is easy: remove the file marking the instance as in-use.
    #!/bin/bash
    echo freeing $@
    cd .instances
    rm $@
  12. Execution is a little trickier. We need to run some commands locally, and other ones by copying in temp files that testr has setup to the machine sshing to the remote machine, cd ing to the right directory, sourcing the virtual env, and finally running the command.
    #!/bin/bash
    instance="$(($1 % 4))"
    case $instance in
    [0]) node=
         local="true"
         ;;
    [1]) node=hplaptop-nova.lxc
         local=""
         ;;
    [2-3]) node=desktop-nova.lxc
         local=""
         ;;
    *)   echo "Unknown instance $instance" >&2
         exit 1
         ;;
    esac
    shift
    files=
    # accumulate files to copy
    while [ "--" != "$1" ]; do 
    files="$files $1"
    shift ; done 
    shift   
    if [ -n "$files" -a -z "$local" ]; then
        echo copying $files to node.
        for f in $files; do
            rsync $f $node:$(dirname $f) ;
        done
    fi  
    if [ -n "$local" ]; then
        eval $@
    else
        echo ssh to $node
        ssh $node "cd source/openstack/nova && . .venv/bin/activate && $@"
    fi
  13. Finally, tell testr how to use this. (Don t commit this change to nova, as it would break other people). Add this to your .testr.conf.
    test_run_concurrency=echo 12
    instance_provision=./instance-provision $INSTANCE_COUNT
    instance_execute=./instance-execute $INSTANCE_ID $FILES -- $COMMAND
    instance_dispose=./instance-dispose $INSTANCE_IDS
Now, when you run testr run parallel, it will run across your machines. Just do a ./sync before running tests to get the code out there. It is possible to wrap all of this up via automation (or to include just-in-time provisioned cloud instances), but I like the results of still rough scripts here it strikes a good balance between effort, reliability and performance.

13 January 2013

Russ Allbery: Slacktivism that may actually help

Normally, I'm not that much of a fan of the slacktivist trend of signing pre-canned petitions, signing things that political organizations send in the mail, and so forth. It takes very little effort and therefore carries very little weight. A lot of those efforts are more exercises in helping the signers feel better about themselves. But there is the occasional exception. Most of you probably already know about the death of Aaron Swartz. For those who aren't familiar, see Larry Lessig's article. Swartz suffered from depression (and, seriously, fuck depression it's an awful, horrible disease), but there's little doubt that the ongoing federal prosecution using the full weight of the US district attorney's office to hound him for what amounts to political trespassing was part of what led to his suicide. As it happens, I personally believe that Swartz committed a crime, and probably should have paid some consequence for it (on the order of a fine or some community service). Hooking your devices up to someone else's network without their permission and messing around in their wiring closets (locked or not) is, for me, akin to traipsing into someone's barn or backyard shed without their permission and using some of their tools because you want to use them. And whether or not one likes the current copyright regime (and I don't like it at all), MIT is still in the awkward position of having to work with it. Abusing their license for your political goals is effectively recruiting them into your activism without their permission, and as I've mentioned before, I have a mild obsession with consent. It was a crime. However, it was a minor crime, and that's where this whole situation went completely off the rails. One aspect of a justice system is fairness: uniform application of the laws to everyone. However, another aspect of a justice system is proportionality: punishments that fit the crime. Without ever having been convicted, Swartz was already punished completely out of proportion to what he actually did, both financially and emotionally, by prosecution by the US attorney that went far, far beyond zealous into actively abusive. This despite the fact that the owner of the academic papers that he was downloading as an act of open access activism stated they did not want the case to proceed and asked the US government to drop the charges. I don't think what Swartz did was right, or legal. However, the correct reaction was "look, involuntarily recruiting MIT as an accomplice to your act of civil disobedience is not okay don't ever do that again." Not "you are evil and should be locked up in prison for 35 years." And I'm completely fed up with the disproportionality of our justice system and the practice of ridiculous over-charging of crimes in an attempt to terrify people into bad plea bargains. Which brings me to the slacktivism. This is exactly the sort of situation where popular opinion matters. US attorneys who have lost the faith and support of the population they serve won't keep their jobs. And no one in government particularly wants this case to be splashed across the front pages, or to have to answer questions about the appropriateness or proportionality of the prosecution while people are mourning a dead young man. If we make it clear enough to the Obama administration that people are watching, that this matters, and that we're angry about it, not only is it quite likely there will be consequences for this prosecutor, but it may serve as a deterrant for other prosecutors in the future. There is a petition on WhiteHouse.Gov to remove the district attorney for prosecutorial overreach, and for once taking five minutes to create an account and clicking on a petition may be both useful and appropriate. Taking disciplinary action here is, unlike with a lot of petitions, something that the Obama administration can actually do, directly, without involving the rest of the dysfunctional US government, and without making new law. If you are a US citizen, please consider going to this site and signing it to say that this matters to you and you believe this prosecution was excessive and inappropriate. Please note: you do not have to believe that Swartz died solely or even primarily because of this prosecution to do this. We'll never really know the complex factors behind his death. But this was a burden that he shouldn't have had to deal with. Please also note that you do not have to think he was justified in his behavior to sign this petition. This is not a question of whether what he did should be legal or was ethical. Rather, even assuming it was illegal, it's a question of appropriate punishment and proportionality of response. Whether or not you agree with his political cause, there simply was not enough damage done, to anyone, to warrant this kind of aggressive prosecution. And the direct victims agreed, which was the point at which the district attorney should have scaled way back on their actions or dropped the matter entirely. Not doing so was an abuse of office and position, and that should have consequences. ETA: Corrected "taking" to "using" in the analogy about tools. I was actually thinking "taking and then returning" when I wrote that, but only the first word made it into the post, and ended up creating a confusing parallel with theft that wasn't intended.

Adnan Hodzic: Enable global menu and HUD support in Eclipse IDE

As I couldn t retain my curiosity for Ubuntu s Raring Ringtail release, I ended up having a dual boot with Ubuntu 13.04 and Debian Sid. Even thought at the moment 13.04 is only 24% complete, it s already a pretty promising release. However, one thing that heavily annoyed me is that global menu and HUD (still) don t work out of box with Eclipse IDE. Being part of Debian Java team (working on Eclipse) I had to do something about it. To make it as simple as possible for you, I made a package which enables mentioned features. Package is made for/tested and working on 32/64 bit architectures on 12.04/12.10/13.04 Ubuntu. enable eclipse global menu and HUD In following days I ll push a Eclipse patch which will enable global menu and HUD support out of box, thus eliminating need to install this package in future.

Bernhard R. Link: some signature basics

While almost everyone has already worked with cryptographic signatures, they are usually only used as black boxes, without taking a closer look. This article intends to shed some lights behind the scenes. Let's take a look at some signature. In ascii-armoured form or behind a clearsigned message one often does only see something like this:
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)
iQIcBAABAgAGBQJQ8qxQAAoJEH8RcgMj+wLc1QwP+gLQFEvNSwVonSwSCq/Dn2Zy
fHofviINC1z2d/voYea3YFENNqFE+Vw/KMEBw+l4kIdJ7rii1DqRegsWQ2ftpno4
BFhXo74vzkFkTVjo1s05Hmj+kGy+v9aofnX7CA9D/x4RRImzkYzqWKQPLrAEUxpa
xWIije/XlD/INuhmx71xdj954MHjDSCI+9yqfl64xK00+8NFUqEh5oYmOC24NjO1
qqyMXvUO1Thkt6pLKYUtDrnA2GurttK2maodWpNBUHfx9MIMGwOa66U7CbMHReY8
nkLa/1SMp0fHCjpzjvOs95LJv2nlS3xhgw+40LtxJBW6xI3JvMbrNYlVrMhC/p6U
AL+ZcJprcUlVi/LCVWuSYLvUdNQOhv/Z+ZYLDGNROmuciKnvqHb7n/Jai9D89HM7
NUXu4CLdpEEwpzclMG1qwHuywLpDLAgfAGp6+0OJS5hUYCAZiE0Gst0sEvg2OyL5
dq/ggUS6GDxI0qUJisBpR2Wct64r7fyvEoT2Asb8zQ+0gQvOvikBxPej2WhwWxqC
FBYLuz+ToVxdVBgCvIfMi/2JEE3x8MaGzqnBicxNPycTZqIXjiPAGkODkiQ6lMbK
bXnR+mPGInAAbelQKmfsNQQN5DZ5fLu+kQRd1HJ7zNyUmzutpjqJ7nynHr7OAeqa
ybdIb5QeGDP+CTyNbsPa
=kHtn
-----END PGP SIGNATURE-----
This is actually only a form of base64 encoded data stream. It can be translated to the actual byte stream using gpg's --enarmor and --dearmour commands (Can be quite useful if some tool only expects one BEGIN SIGNATURE/END SIGNATURE block but you want to include multiple signatures but cannot generate them with a single gpg invocation because the keys are stored too securely in different places). Reading byte streams manually is not much fun, so I wrote gpg2txt some years ago, which can give you some more information. Above signature looks like the following:
89 02 1C -- packet type 2 (signature) length 540
        04 00 -- version 4 sigclass 0
        01 -- pubkey 1 (RSA)
        02 -- digest 2 (SHA1)
        00 06 -- hashed data of 6 bytes
                05 02 -- subpacket type 2 (signature creation time) length 4
                        50 F2 AC 50 -- created 1358081104 (2013-01-13 12:45:04)
        00 0A -- unhashed data of 10 bytes
                09 10 -- subpacket type 16 (issuer key ID) length 8
                        7F 11 72 03 23 FB 02 DC -- issuer 7F11720323FB02DC
        D5 0C -- digeststart 213,12
        0F FA -- integer with 4090 bits
                02 D0 [....]
Now, what does this mean. First all gpg data (signatures, keyrings, ...) is stored as a series of blocks (which makes it trivial to concatenate public keys, keyrings or signatures). Each block has a type and a length. A single signature is a single block. If you create multiple signatures at once (by giving multiple -u to gpg) there are simple multiple blocks one after the other. Then there is a version and a signature class. Version 4 is the current format, some really old stuff (or things wanting to be compatible with very old stuff) sometimes still have version 3. The signature class means what kind of signature it is. There are roughly two signature classes: A verbatim signature (like this one), or a signature of a clearsigned signature. With a clearsigned signature not the file itself is hashed, but instead a normalized form that is supposed to be invariant under usual modifications by mailers. (This is done so people can still read the text of a mail but the recipient can still verify it even if there were some slight distortions on the way.) Then the type of the key used and the digest algorithm used for creating this signature. The digest algorithm (together with the signclass, see above) describes which hashing algorithm is used. (You never sign a message, you only sign a hashsum. (Otherwise your signature would be as big as your message and it would take ages to create a signature, as asymetric keys are necessarily very slow)). This example uses SHA1, which is no longer recommended: As SHA1 has shown some weaknesses, it may get broken in the not too distant future. And then it might be possible to take this signature and claim it is the signature of something else. (If your signatures are still using SHA1, you might want to edit your key preferences and/or set a digest algorithm to use in your ~/.gnupg/gpg.conf. Then there are some more information about this signature: the time it was generated on and the key it was generated with. Then, after the first 2 bytes of the message digest (I suppose it was added in cleartext to allow checking if the message is OK before starting with expensive cryptograhic stuff, but it might not checked anywhere at all), there is the actual signature. Format-wise the signature itself is the most boring stuff. It's simply one big number for RSA or two smaller numbers for DSA. Some little detail is still missing: What is this "hashed data" and "unhashed data" about? If the signed digest would only be a digest of the message text, then having a timestamp in the signature would not make much sense, as anyone could edit it without making the signature invalid. That's why the digest is not only signed message, but also parts of the information about the signature (those are the hashed parts) but not everything (not the unhashed parts).

Neil Williams: Status of klibc on AArch64

With significant assistance from Steve McIntyre and some judicious delving into the ARM Information Centre, I've now got the assembly portions of klibc sorted (but untested) for AArch64 (arm64).

Andy & I started by copying the old ARM support as a new directory and one of the final steps was to remove a whole bunch of legacy code from the days before Thumb and all the #ifdef lines which went with it. Some files disappeared entirely. setjmp.S was the largest amount of work as the load&store multiple support of ARMv7 has gone in ARMv8, so the stmia mnemonic had to be expanded to multiple stp calls but that makes it more explicit, so it's not a bad thing.

Steve & I borrowed from the glibc AArch64 code from upstream by Marcus Shawcroft, simplifying the glibc macros for klibc and also got clues about what extra registers needed handling compared to ARMv7.

I've now got to look at some traditional cross-compilation issues because the Linaro AArch64 toolchain doesn't install to typical Debian cross-building paths and the build now moves past the AArch64-specific assembly and fails later when the C code gets the wrong include path and ends up including /usr/include/i386-linux-gnu/asm/byteorder.h with predictable results.

If someone has an AArch64 toolchain already set up, feel free to clone my modified klibc tree and let me know if there are subsequent build errors. Of course, if you fancy testing a build in the Foundation Model for ARMv8, that would be good too! (Report issues via github.)

Whilst I'm sorting out the toolchain, I've also been updating perl-cross-debian (which has also seen some more upstream testing and improvement).

Russell Coker: Android Multitasking

My new Samsung Galaxy S3 has support for Multi Window Mode , here is a video which shows how to use this, Multi Window Mode starts at about 2:30 [1]. A common complaint about Android is the lack of multitasking, which is partly true and only slightly alleviated by Multi Window Mode. Running Multiple Programs Traditionally Android has multitasked with a similar ability to a Unix shell session, you can have applications running in the background and switch between them but you can t see multiple applications running at the same time (apart from seeing notification messages at the top of the screen). The new Multi Window Mode allows two or more applications to share a screen. But it only applies to a small number of applications which support it, on my phone that is applications shipped by Samsung and Google Chrome. Also I can t have multiple copies of Chrome open at the same time which means I can t do the things that I do on every PC that runs Chromium (the non-Google build of Chrome). I have not yet found a situation where Multi Window Mode has been useful to me, the applications I use for most tasks don t support it. So while Android being based on Linux does multitask really well in the technical computer-science definition it doesn t do so well in the user-centric definition. In practice Android multitasking is mostly about task switching and doing things like checking email in the background. Having multiple programs running at once is particularly difficult due to the Android model of applications sometimes terminating when they aren t visible. A common task is to view a message in a MUA and switch between that and another window (EG a web browser). K9 is my preferred MUA for Android which seems to have no option to switch back and still be viewing the same message as before the task switch so at least three actions need to be taken to get back to the same message after I resume K9. One major feature to make multitasking on Android more usable would be a way of rapidly switching between two applications and being certain that each one would be in the same state when the user switches back to it. Copy and Paste Another problem with Android multitasking is the difficulty in copying and pasting text. While copy/paste is not strictly required for multitasking it is a logical requirement when you have multiple applications running on behalf of a single user. For PCs everyone knows that you select text by holding down the SHIFT key while using the cursor control keys or by holding down the mouse button and swiping the mouse cursor over the text and you then use CTRL-C or the Edit menu to copy text. On Android it s a long press to select text which then gives you markers for the start and end, you drag those markers around and then select that you want to copy the text. This is at best a lot more inconvenient than using a high resolution input device like a mouse to select text. At worst it doesn t seem particularly reliable, K9 Mail for example won t let me copy text from a message for unknown reasons on a desktop OS such problems are vanishingly rare such that I can t think of an example of it happening. IO and Multitasking Multitasking for a user (as opposed to the multitasking needed to host dozens or hundreds of concurrent users) on Unix servers was very limited in the days of VT100 terminals and similar devices. Programs such as GNU Screen [2] allowed a text display to perform windowing functions that are similar to a modern GUI. But generally it seems that the ability for a user to run multiple programs at once is largely limited by their ability to see the output and to rapidly switch between programs or sessions. As a keyboard with ~100 keys and a text display with 80*25 characters is a major limitation it s obviously going to be a comparable (and often greater) limitation to have an on-screen keyboard that takes half the screen space, a single program taking all the screen, and a drop-down status bar that might be useful for multitasking. With Android 4.0 and above you can activate a task switcher by holding down the home button for two seconds [3]. There are also a variety of third party task switching programs on the Google Play store which all seem to start by holding down the home button. One problem with these options is that they require an extended press of the home button where the ideal is something that is as quick as ALT-TAB or a single mouse movement on a desktop system. One possible input action would be to switch between the most recent tasks with a palm swipe on the screen an operation that is quick and easy. Currently a palm swipe is used for a screen capture, but as there are four possible directions for swiping the screen one of them could be used for screen capture and another two for task switching. But this wouldn t do a lot of good without the ability to switch tasks without losing the context that either requires Android application changes or having the OS not tell an application that it was occluded. The iPaQ had interesting capabilities for input, it had a main button on the front that could be pushed in four directions (usually for cursor control) as well as being pushed in, four extra buttons on the front, and a button on the side [4]. I don t recall the methods that Familiar Linux on the iPaQ [5] used for task switching, but it was less of an issue as the iPaQ had less RAM, no Internet access and no telephony functions. I think that adding a bunch of extra buttons to an Android phone would make it a lot more useful. The iPaQ method of cursor control is one that could be considered (it could alleviate the copy/paste problems among other things). As an aside in two years of using Android phones I ve done less serious writing on Android than I did in my first two years of using Familiar on the iPaQ largely due to the lack of keys and a stylus on my Android phones. The screen size on Android phones is also a limit for multitasking. The earlier/cheaper phones that have small screens with resolutions such as 320*480 have very limited ability to display two programs at once. The 720*1280 display in the Galaxy S3 has a lot more potential in this regard and the Galaxy Note 2 and the HTC J Butterfly AKA Droid DNA have even more potential. In the future it seems that screen size limitations on phone multitasking will be a solved problem for everyone who can afford one of the high end phones which incidentally are much cheaper than the iPaQ was 10 years ago. Conclusion Checking email in the background etc is very useful on Android systems. But in terms of the user running two programs at once it seems very limited, and that situation seems likely to remain until the vendors adopt multi-window support. This could be difficult given that Google applied a lot of pressure to CyanogenMod to stop them from doing it [6]. Even when a system with a large display (particularly a tablet) runs a version of Android that supports multi-window mode that still won t entirely solve the problem. No matter how big your display is there are occasions when you need to use it all for a single application while still having the ability to rapidly switch to another application. User interface tweaks to allow rapid task switching without losing application context are necessary. Finally for the past two years that I ve been using Android devices I have been disappointed in the ways that they compare poorly to the iPaQ running Familiar I used 10 years ago. I once wrote a feature article for a magazine on an iPaQ while so far I haven t even written a blog post on an Android device. I think that some of the earlier Android devices might have been better in some ways, the trackball on the HTC Nexus One might have made it more suitable for writing long articles than more recent Android devices.

12 January 2013

John Sullivan: Aaron Swartz

Aaron was an inspiration to me personally, politically, and professionally ever since we met (ice cream and word games with a small group in a bank vault at Herrell's in Harvard Square) several years ago. I don't understand how things got to this point, but I know I'm angry along with Lessig. I'm so sorry for all of his family and friends; all the rest of us can do is try to make even a tiny sliver of the difference he did.

Sylvestre Ledru: Some more cool stuff with LLVM/Clang

One of the drawback of C is the hard memory management. It is because of this aspect that Clang provides a way to detect, at runtime, memory errors. Called Address Sanitizer, it allows, while the program is running, to keep track of the memory and detect typical errors (out-of-bound accesses, use of a variable after a free, etc).
# With the following packages (version 3.2-1~exp3)
$ sudo apt-get install clang clang-3.2 compiler-rt -t experimental
Taking the simple following example, even if the errors are obvious, compilers will accept this code.
#include <stdlib.h>
int main()
char *x = (char*)malloc(10 * sizeof(char*));
free(x);
return x[5];
Built and run with:
$ clang -O1 -g -fsanitize=address -fno-omit-frame-pointer foo.c -o foo
$ ./foo &> memoryDebug.log
The previous command will generate a log file. Log which can be post processed with the asan_symbolize command.
$ asan_symbolize memoryDebug.log
which will give:
=================================================================
==21368== ERROR: AddressSanitizer: heap-use-after-free on address 0x7fb22e547f45 at pc 0x408c44 bp 0x7ffff60c10b0 sp 0x7ffff60c10a8
READ of size 1 at 0x7fb22e547f45 thread T0
    #0 0x408c43 in main /tmp/foo.c:5
    #1 0x7fb22d99f6ac in __libc_start_main /home/adconrad/eglibc-2.16/csu/libc-start.c:227
0x7fb22e547f45 is located 5 bytes inside of 80-byte region [0x7fb22e547f40,0x7fb22e547f90)
freed by thread T0 here:
    #0 0x408c90 in __interceptor_free ??:0
    #1 0x408c0a in main /tmp/foo.c:4
    #2 0x7fb22d99f6ac in __libc_start_main /home/adconrad/eglibc-2.16/csu/libc-start.c:227
previously allocated by thread T0 here:
    #0 0x408d50 in __interceptor_malloc ??:0
    #1 0x408bff in main /tmp/foo.c:3
    #2 0x7fb22d99f6ac in __libc_start_main /home/adconrad/eglibc-2.16/csu/libc-start.c:227
Shadow byte and word:
  0x1ff645ca8fe8: fd
  0x1ff645ca8fe8: fd fd fd fd fd fd fd fd
More shadow bytes:
  0x1ff645ca8fc8: fa fa fa fa fa fa fa fa
  0x1ff645ca8fd0: fa fa fa fa fa fa fa fa
  0x1ff645ca8fd8: fa fa fa fa fa fa fa fa
  0x1ff645ca8fe0: fa fa fa fa fa fa fa fa
=>0x1ff645ca8fe8: fd fd fd fd fd fd fd fd
  0x1ff645ca8ff0: fd fd fd fd fd fd fd fd
  0x1ff645ca8ff8: fa fa fa fa fa fa fa fa
  0x1ff645ca9000: fa fa fa fa fa fa fa fa
  0x1ff645ca9008: fa fa fa fa fa fa fa fa
Stats: 0M malloced (0M for red zones) by 1 calls
Stats: 0M realloced by 0 calls
Stats: 0M freed by 1 calls
Stats: 0M really freed by 0 calls
Stats: 0M (128 full pages) mmaped in 1 calls
  mmaps   by size class: 8:2047;
  mallocs by size class: 8:1;
  frees   by size class: 8:1;
  rfrees  by size class:
Stats: malloc large: 0 small slow: 1
==21368== ABORTING
The main advantage compare to valgrind is that asan is supposed to be way more faster. Threads can be also tricking to develop.
With the following example from stolen from upstream:
#include <pthread.h>
int Global;
void *Thread1(void *x)
Global = 42;
return x;
int main()
pthread_t t;
pthread_create(&t, NULL, Thread1, NULL);
Global = 43;
pthread_join(t, NULL);
return Global;
$ clang -fsanitize=thread -g -O1 foo2.c -fPIE -pie -o foo
$ ./foo
==================
WARNING: ThreadSanitizer: data race (pid=21416)
  Write of size 4 at 0x7f2f1a214a50 by thread 1:
    #0 Thread1 /tmp/foo2.c:4 (exe+0x00000000f850)
  Previous write of size 4 at 0x7f2f1a214a50 by main thread:
    #0 main /tmp/foo2.c:10 (exe+0x00000000f8a4)
  Thread 1 (tid=21417, running) created at:
    #0 pthread_create ??:0 (exe+0x00000001267e)
    #1 main /tmp/foo2.c:9 (exe+0x00000000f894)
==================
ThreadSanitizer: reported 1 warnings
Note that clang also provides scan-build, a static analyzer for memory issues. Not as powerful as the Address Sanitizer (it only works on a file), it provides some excellent reports. See the Wouter's blog post on this subject or the automatic report of scan-build on Scilab.

Russ Allbery: Review: Asimov's, April/May 2011

Review: Asimov's Science Fiction, April/May 2011
Editor: Sheila Williams
Issue: Volume 35, No. 4 & 5
ISSN: 1065-2698
Pages: 192
Williams's editorial this issue is about the tendency of SF to take a rose-colored view of the world, which on the surface seems odd given the tendency of recent SF towards dystopia. But she makes a good point that the portrayal of the past is rose-colored, linking that into the current steampunk trend. She doesn't take the argument quite as far as I'd like, but I'm glad to see editorials raising points like this. I'm inclined to think that a lot of the rose-colored frame of the past is because few of us want to read about real historic conditions at any length, even for edification, because the stench and discomfort isn't fun to read about. Silverberg's column is another discussion of programmatic plot generators, which mostly makes the point that plot ideas are the easy part of writing. James Gunn contributes an extended biography of Isaac Asimov that probably won't be new to long-time genre readers but may fill in some details (although it politely sticks to mostly flattering material). Spinrad's book review column is one of his better ones; it looks at two novels by China Mi ville and two by Ian McDonald and explores differences in world-building. Spinrad predictably makes the case in favor of science fiction with rules and against the New Weird, but the discussion along the way was worth reading. "The Day the Wires Came Down" by Alexander Jablokov: Speaking of steampunk, here's an example. There is even an airship, although the primary technological focus is suspended street cars. Jablokov postulates a city-wide transportation network of suspended carriages called telpher cars, along with a city built around the telpher cables: stores on roofs, windows displaying merchandise to passing cars, and even a history of heated competition and dirty tricks between competing telpher networks. The story is set, as the title would indicate, on the last day of the network. It's being shut down for cost, with some hints that progress is destroying something precious. There is a plot here, revolving around some mysteries of the history of the telpher network and the roles of several people in that history. But the story is primarily a celebration of old technology. It's a rail fan's story recast with a steampunk technology, featuring the same mix of fascination with mechanics and a sense that the intricate details are falling out of common knowledge (and perhaps usefulness). As a story, it's a bit slow-moving, but I enjoyed the elegiac tone. (7) "An Empty House with Many Doors" by Michael Swanwick: This is a very short story, more of an emotional profile, involving a man's reaction to the death of his wife. Oh, and parallel universes. It's sort of the inverse of Niven's classic "All the Myriad Ways." Similar to Niven's story, I found the idea vaguely interesting but the conclusion and emotional reaction unbelievable and alien. (5) "The Homecoming" by Mike Resnick: Resnick tends to yank on the heart-strings rather sharply in his stories, so I knew roughly what to expect when a father comes home to find his son is visiting. A son who, rather against his father's wishes, has been significantly altered to be able to live with aliens. Throw in a mother with serious dementia, and you can probably predict what Resnick does with this. Still, most of the story is a two-sided conversation, and I thought he succeeded in doing justice to both sides, even though one of them was destined to lose. (6) "North Shore Friday" by Nick Mamatas: Illegal Greek immigrants, a family-run system for getting them married off before the INS catch them, government psi probes and eavesdropping on thoughts, joint projects between computer and religion departments, secret government experiments, and even ghosts... this story is a complex mess, with numerous thoughts stuck into small boxes and scattered through the surface story. It's one of those stories where figuring out what's going on, and even how to read the story in a sensible way, is much of the fun. If you find that fun, that is; if not, it will probably be frustrating. I wished there was a bit more plot, but there's something delightful about how much stuff Mamatas packs into it. (6) "Clockworks" by William Preston: This is a prequel to Preston's earlier "Helping Them Take the Old Man Down". Like that story, it's primarily a pulp adventure, but layered with another level of analysis and thoughtfulness that tries to embed the pulp adventure in our understanding of human behavior and the nature of the world, although this one stays a bit more pulp than its predecessor. As with Preston's other story, we don't get directly in the head of the Old Man (here, just called the man, but identifiable from clues in both stories as Doc Savage); instead, the protagonist is a former villain named Simon Lukic who the man hopes to have fixed by operating on his brain. The undercurrent that lies beneath a more typical pulp adventure is the question of whether Lukic is actually healed. I think there was a bit too much daring-do and human perfection, but it's a perfectly servicable pulp story with some depth. (6) "The Fnoor Hen" by Rudy Rucker: If you've read any of Rucker's work before, you probably know what to expect: a mind-boggling blizzard of mathematically-inspired technobabble that turns into vaguely coherent surrealism. (You can probably tell that I'm not much of a fan, although the clear good humor in these stories makes it hard to dislike them too much.) There's a mutated chicken and some sort of alternate mathematical space and then something that seems like magic... I'd be lying if I said that I followed this story. If you like Rucker, this seems like the sort of thing that you'd like. (4) "Smoke City" by Christopher Barzak: At the start of this story, I thought it was going to be an emotional parable about immigration. The progatonist lives two lives: one in our world, and one in the Smoke City of industry, a world of hard labor, pollution, and little reward, with families in both. But nearly all of the story is set within Smoke City, and the parable turns out to be a caustic indictment of industry and its exploitation of labor. I kind of wish Barzak hadn't used rape as a metaphor, but when the captains of industry show up, I can't argue with how deeply and accurately the story shoves in the knife. There isn't much subtlety here, but it's still one of the better stories in this issue. (7) "A Response from EST17" by Tom Purdom: I'm very happy to see Purdom's writing appearing regularly. His stories are always quiet and matter-of-fact, and at first seem to miss emotional zest, but they almost always grow on me. He lets the reader fill in their own emotional reactions to events, and there's always a lot going on. This story is a first-contact story, except that the "humans" here are not human at all. They're automated probes sent by two separate human civilizations, with different programming and different governance algorithms, and they quickly start competing negotiations. The aliens they've discovered similarly have factions, who start talking to the different probes in an elaborate dance of gathering information without giving too much away. The twist is that this pattern has replayed itself many times in the past, and information itself can be a weapon. I enjoyed this one from start to finish. (7) "The One That Got Away" by Esther M. Friesner: Friesner is best known, at least to me, for humorous fantasy, and this story is advertised as such from early on. The first-person protagonist is a prostitute in a seaside town. She's bemused to finally be invited over by a sailor who's been eyeing her all evening, but that sailor has something else in mind than normal business. For much of this story, the fantasy element is unclear; when it finally comes, it was an amusing twist. (7) "The Flow and Dream" by Jack Skillingstead: This is a mildly interesting variation on the old SF story of hibernating humans (on a generation ship or elsewhere) waking up to a transformed world. Here, it's not a ship, it's a planet, and the hiberation was to wait for terraforming rather than for transit. The twist comes from an excessively literal computer and the fun of putting together the pieces. Sadly, the story trails off at the end without much new to say. (5) "Becalmed" by Kristine Kathryn Rusch: "Becalmed" takes place immediately before "Becoming One with the Ghosts" and explains the incident that created the situation explored in that story. The first-person protagonist of "Becalmed" is a linguist, an expert in learning alien languages so that the Fleet can understand the civilizations that it runs across. But something went horribly wrong at their last stop, something that she's largely suppressed, and now she's confined to quarters and possibly in deep trouble. As is the ship; they're in foldspace, and they have been for days. "Becalmed" is structed like a mystery, centered around recovering the protagonist's memories. It's also a bit of a legal procedural; the ship is trying to determine what to do with her and to what degree she's responsible. But the heart of the story is a linguistic and cultural puzzle. This is another great SF story from Rusch, whose name on a cover will make me eager to start reading a new magazine. I love both angles on the universe she's built, but I think I like the Fleet even better than the divers. The Fleet captures some of the magic of the original Star Trek, but with much more mature characters, more believable situations, and a more sensible and nuanced version of the Prime Directive. Rusch writes substantial, interesting plots that hold my interest. I'd love to see more like this. (8) Rating: 7 out of 10

11 January 2013

Julien Danjou: Overriding cl-json object encoding

CL-JSON provides an encoder for Lisp data structures and objects to JSON format. Unfortunately, in some case, its default encoding mechanism for CLOS objects isn't exactly doing the right thing. I'll show you how Common Lisp makes it easy to change that. Identifying the problem CL-JSON & CLOS CL-JSON mechanism encoding CLOS object is really neat. Let's see how it works for a simple case:
(defclass kitten ()
((tail :initarg :tail)))

(json:encode-json-to-string (make-instance 'kitten :tail 'black))

will produce:
 "tail":"black" 

Still using CL-JSON, we can also decode the JSON object to a CLOS object:
(slot-value
(json:with-decoder-simple-clos-semantics
(json:decode-json-from-string " \"tail\":\"black\" "))
:tail)

That code will return "black". Note that it's also possible to specify which class should be used when decoding objects, but that's beyond the purpose of this article. Postmodern Now, let's introduce Postmodern, a wonderful Common Lisp system providing access to the wonderful PostgreSQL database. It also provides a simple system to map rows in a database to CLOS classes, called DAO for Database access objects. With this, we can easily store our kitten into a table.
(defclass kitten ()
((tail :initarg :tail))
(:metaclass postmodern:dao-class))

If we try to encode this to JSON, it will produce the exact same result seen previously. The problem is what happens when one of our column has a NULL value. Postmodern encodes this using the :null symbol. So this code:
(defclass kitten ()
((tail :initarg :tail :col-type (or s-sql:db-null text)))
(:metaclass postmodern:dao-class))

(postmodern:deftable kitten
(postmodern:!dao-def))

(postmodern:connect-toplevel )

(postmodern:create-table 'kitten)

(json:encode-json-to-string
(postmodern:make-dao 'kitten))

will return:
" "tail":"null" "

Fail! The fact that the column is NULL is represented by the :null symbol. And CL-JSON encodes all symbols as string. This is not at all what we want here! Overriding encode-json CL-JSON provides and uses the encode-json method to encode all kind of object. It is defined as a generic function, and a lot of different methods are implemented to handle the different standard Common Lisp types. The one used for standard-object is defined liked that:
(defmethod encode-json ((o standard-object)
&optional (stream *json-output*))
"Write the JSON representation (Object) of the CLOS object O to
STREAM (or to *JSON-OUTPUT*)."
(with-object (stream)
(map-slots (stream-object-member-encoder stream) o)))

All we need to do here, is to create a new method for our kitten objects, that handles correctly the :null case.
(defclass kitten ()
((tail :initarg :tail :col-type (or s-sql:db-null text)))
(:metaclass postmodern:dao-class))

(export 'kitten)

;; Switch package just to define the new method
(in-package :json)
(defmethod encode-json ((o cl-user:kitten)
&optional (stream json:*json-output*))
"Write the JSON representation (Object) of the postmodern DAO CLOS object
O to STREAM (or to *JSON-OUTPUT*)."
(with-object (stream)
(map-slots (lambda (key value)
(as-object-member (key stream)
(encode-json (if (eq value :null) nil value) stream)))
o)))

;; Go back into our package
(in-package :cl-user)

(postmodern:deftable kitten
(postmodern:!dao-def))

(postmodern:connect-toplevel )

(postmodern:create-table 'kitten)

(json:encode-json-to-string
(postmodern:make-dao 'kitten))

With that new method, as soon as we encounter a :null symbol as a value for an object's slot, we replace it by nil. Now if we try to encode another kitten, we'll get:
 "tail":null 

which is far better for our JavaScript data consumers! In the end, I think that this kind of trick is feasible that easily because of the way CLOS provides its generic method implementation. The fact that methods don't belong to any class makes the extension of every program, library and class so much easier. Doing this in another language like Java would likely by impossible, and in Python it would unlikely be as clean as it is done in Common Lisp. The ability to teach any library about how it should handle your class just by defining a new method is really handy!

Benjamin Mako Hill: Goodbye PyBlosxom, Hello WordPress

Since 2004, I ve used the blogging software PyBlosxom. Over that time, the software has served me well and I have even written a series of patches and plugins. PyBloxsom is blog software designed for hackers. It assumes you already have a text editor you love and relies on features of a POSIX filesystem instead of a relational database. It s designed so you can keep your blog under revision control (since 2004 I ve used GNU Arch, baz, bzr and now git). It is also hackers software in the sense that you should expect to write code to use it (e.g., the configuration is pure Python). I love it. What PyBlosxom does not have is a large community. This summer, the most recent long-time maintainer of the project, Will Kahn-Greene, stepped down. Although the project eventually found a new maintainer, the reality is the project entered maintenance mode years ago and many features available in more popular blogging platforms are unlikely to make it PyBlosxom. The situation with comment spam is particularly dire. I ve written several antispam plugins but time has shown that I don t have the either the expertise or the time to make them as awesome as they need to be to really work in today s web. So, after many months of weighing, waffling, and planning I ve switched to WordPress a great piece of free software with an enormous and established community As you ll know if you ve read my interview on The Setup, I think a lot about the technology I surround myself with. I considered WordPress when I started my blog back in 2004 and rejected it soundly. Eight years ago, I would have laughed at you if you told me I d be using it today; If PyBlosxom is for hackers, WordPress is designed for everyone else. But the way I evaluate software has changed over that period. In the nineties, I used to download every new version of the Linux kernel to compile it it took hours! to try out the latest features. Configurabilty, hackability, and the ability to write my own features was after a point more important than the features the software came with. Today, I m much more aware of the fact that for all the freedom that my software gives me, I simply do not have the time, energy, or inclination to take advantage of that freedom to hack very often. Today, I give much more value to software that is not just free, but that is maintained by a community of people who can do all the work that I would do if I had unlimited time. Although I don t have the time or experience to make WordPress do everything I would like, the collective of all WordPress users does. And they ve done a lot of it already! The flip side matters as well: Today, I give more value to other people using my software. When WordPress doesn t do something and I write a plugin or patch, there are tons of people ready to pick it up and use it and perhaps even to collaborate on it with me. Want to guess how many patches my PyBlosxom plugins have received? None, if my memory serves me. In the past, I ve written about how free software is a victory even when it doesn t build a community. I still believe that. But the large communities at the heart of the most successful free software communities (the promise of open source ) are deeply important in a way that I increasingly value. In that spirit: If you want to make the jump from PyBlosxom to WordPress, I ve shared a Git repository with the scripts I used and wrote for the transition.

Russ Allbery: release 1.45 and multiple PGP signatures

release is the tool that I use to manage software releases. This version adds support for signing a tarball with multiple PGP keys at once. For WebAuth, we have an official signing key, but I also wanted to add my personal signature. The minimum supported version of Python was also bumped to 2.5 (see below for why). You can get the latest version from my scripts page. Normally, I wouldn't announce this to everyone, since it's a fairly minor change, except I had a question and a quick Python note. The question: how does one get GnuPG to verify multiple signatures? When I run GnuPG on the resulting detached *.asc file created by this new code, I get the following:
gpg: WARNING: multiple signatures detected.  Only the first will be checked.
gpg: Signature made Thu 10 Jan 2013 03:58:43 PM PST using RSA key ID 5736DE75
gpg: Good signature from "Russ Allbery <rra@stanford.edu>"
gpg:                 aka "Russ Allbery <rra@debian.org>"
gpg:                 aka "Russ Allbery <eagle@windlord.stanford.edu>"
gpg:                 aka "Russ Allbery <eagle@eyrie.org>"
That's nice, I suppose, but it's not as helpful as it could be. Why not verify both signatures and report all of the results? I searched both the gpg man page and the Internet for some option I'm missing, without any luck. (Searching Google for that exact message amusingly turns up lots of verbatim output from Debian repository verification and almost nothing else.) If anyone knows, send me email (rra@stanford.edu or any other convenient address), and I'll update this post with the answer. UPDATE: gpg can currently only verify multiple signatures if the signatures are from keys with the same class and digest. I was attempting this with a 2048-bit RSA key and a 1024-bit DSA key, which gpg does not support. (Yet another reason to replace the WebAuth signing key with a newly-generated key.) Thanks to Philip Martin and Bernhard R. Link for the information! The note is that, way back when I was first learning Python, one of the things that bothered me the most was the lack of safe program execution. The best available was os.spawnv(), which left something to be desired. If you too were bothered by this but don't program in Python enough to keep track of the current state of the art, there's now a subprocess module that has much saner interfaces that work properly, do reasonable things, and avoid the shell. It was added in Python 2.4, with some nicer interfaces added in Python 2.5, so it's fairly safe to use at this point.

10 January 2013

Vincent Fourmond: Introducing dvdcopy, an error-resistant command-line DVD copier

My wife likes watching DVDs a lot, which is why she borrows many from the public library. The problem is that many of those were handled by people who have no clue whatsoever about the proper handling of DVDs, which means that they are in a state ranging from moderately scratched in the best cases to properly scarred in the worst ones... xine or VLC fail miserably to read such DVDs. What's worse is that failures happen right in the middle of DVDs, which is quite frustrating !That's why I've started quite some time ago to write a small C wrapper around libdvdread to first copy the DVD files onto the disk and then read them from there. It was also an easy solution to copy files to a driveless netbook to watch the DVD there. With time, this project evolved into a neat C++ command-line utility that's able to keep track of badly read sectors and come back to them later.Today is the 0.0 release of dvdcopy. You can get the tarball there. It follows the standard autoconf dance:
~ ./configure
~ make
~ make install
Using it is as simple as running:
~ dvdcopy /dev/dvd MyMovieDirectory
If everything went fine, you're set, and you can open the directory created with vlc or xine:
~ xine dvd:// pwd /MyMovieDirectory
(yes, xine needs the full path, I haven't checked with VLC, but from what I remember, you can open DVD directories directly from a menu).If you're not that lucky and dvdcopy found some bad sectors, try reading again using
~ dvdcopy --second-pass /dev/dvd MyMovieDirectory
I found that in many cases, one or two second passes get all the sectors back. While that wasn't really the intended purpose, the fact that most copy protection schemes (in addition to CSS) are implemented using bad sectors make dvdcopy quite resistant to them.I have no plan of making a Debian package for it at the time being, but if there is demand for that, I may reconsider. Enjoy !

Hideki Yamane: 256 -> Debian

I had a cold, so couln't write anything (but back to be normal, almost).

I'm also Ubuntu LoCo team member as Ubuntu Japanese team. Some of team members write articles on gihyo.jp website during the years, and now our team leader Jun Kobayashi(*) celebrate 256 times(!) as Android Ubuntu Debian (Run Ubuntu Debian on Android)... Yes, Debian, you know :)

*) He also wrote very funny article ago, see this...

Debian Med: Bits from Debian Med team (Posted by Andreas Tille)

This are the new years bits from Debian Med team. We try to make the wishes for a healthy new year becoming true with Free Software. :-)
Debian Med Bug Squashing Advent Calendar 2012 As last year the Debian Med team did some advent calendar bug squashing. The summary states 15 bugs from Debian Med scope which is less than last year but Debian Med team members also did some general bug squashing to help the Wheezy release. Finally this makes a lot of sense also for Debian Med if we can speed up the release. When counting the additional 12 RC bugs fixed in the advent calendar effort we again managed to fix more than one bug per day. Please keep on squashing RC bugs for Wheezy!
Mentoring of Month Last year Andreas Tille started the Mentoring of Month effort to lower the entrance barrier for newcomers. There was no student for every month and not each package was uploaded at end of month but expecting such a high number of newcomers would be unrealistic anyway. So I'd like to draw some positive conclusion that we now have some newcomers friendly invited to the project and some packages uploaded or at least prepared which would not be there without the MoM project. Finally all MoM students liked the project and we will continue it in the future and would like to recommend other teams to do something similar.
Planed Debian Med sprint in Kiel The next Debian Med sprint for real life meeting of Debian Med developers and users will happen in Kiel 23rd/24th of February. All interested people are invited to join to continue the success of the past two sprints. We are currently in the planing phase but there are just some hot topics which are workflows, licensing (specifically bug #694908), cooperation with Ubuntu and BioLinux and other things.
Metapackages targeting at Wheezy Currently Debian Med metapackages version 1.13.2 are in unstable and once the unblock request is accepted by Debian release team this version will target at Wheezy. This means your very last chance to influence the metapackage content regarding some missing dependencies is defacto over now. Please closely observe our web sentinel in case you are interested in the work of Debian Med and tell us if packages are missing from our focus.

Kind regards and see you at Debian Med sprint
Andreas.

Next.

Previous.