Search Results: "nate"

15 November 2009

Jonathan McDowell: Playing with an e-reader

I've always preferred dead tree to reading things on screen; I just find it easier. I've tried reading fiction in the past off a laptop and just didn't find it as enjoyable an experience; whether it was the form factor (hard to curl up with a laptop, even a netbook), the quality of the screen or the interface I don't know. Equally with technical documentation if it's something I'm using a lot I prefer a printed copy to flick through. However e-ink based e-readers are becoming much more common and affordable and I figured I should give the whole ebook experience another try.

I spent 3 weeks in October in the Bay Area on work, so I borrowed my Dad's Sony PR505 for the trip. First problem is that it's picky about USB charging - works fine if attached to a computer, but using a Blackberry charger looks like it's fine but then results in what looks like a hanging reader. However once charged it lasts for ages - I charged it before I went and only hooked it up to my laptop once during the trip to transfer some new content onto it. The form factor is also quite good; a bit heavier than a book, but not excessively so. Holdable in one hand, big enough screen that I wasn't squinting at it. Slightly too big physically to easily go in a coat pocket the way a paperback would unfortunately, but perfectly fine for taking to read at breakfast every morning in the hotel.

The screen was also much better than I'd expected. I knew the technology was different and supposed to be easier to read than traditional LCDs, but I was sceptical. Overall I had no problems with it. I was even quite impressed with the Sony's zoom function - one of the PDFs I was reading was too small when view full screen, so I zoomed in so it was readable. The nice feature was that then the "next page" button gave me the next bit of text to read, rather than the actual next page, so it remained very easy to navigate through as I read. A small touch, but very welcome.

This isn't intended to be a review of the Sony reader; that just happened to be the model I was easily able to borrow. I happened to see a Kindle in use on the plane and I was struck by how much bulkier it seemed, though having looked at the relative specs it seems this is entirely due to the keyboard rather than any difference in screen size. The Nook was also announced during my trip and I'd quite like a play with that as it looks quite nice.

Despite my positive experience with the Sony I'm not planning to go out and get one just yet. And that's the lack of sensibly priced content. When I bought my first mp3 player I could take my exising CD collection, rip it and be able to play it on my new device. This meant I got an immediate benefit of having my entire music collection with me all the time, just by buying the player. If I buy an e-reader then in order to get all my existing books on it I have to go and buy them again. What's worse is they'll cost me the same or more than I paid for the paperbacks. I can't go and exchange my Pratchett collection for the electronic versions for a nominal fee. I can't easily scan them in myself and produce some decent ebooks. I can't even go and buy the entire set for 20; I'd have to spend something like ten times that. With mp3s I can continue to buy the real item and also have it on my portable player. Or, while albums still don't seem to be much cheaper electronically than on CD, there is at least the ability to buy a single track if that's all I want. Books don't have a comparison. I'm not going to want to buy a single chapter, am I?

There are some sites out there that can provide cheaper ebooks - Rachel Willmer runs ebookprice.info which lets you compare pricing from different vendors. There's also Project Gutenberg if you're looking for out of copyright books. Finally Peter Corlett pointed me at the Baen Free Library, which I haven't downloaded anything from but will definitely investigate at some point. I still maintain that none of these are enough and that content provision will continue to be a hinderance for e-reader mass adoption until there is some fundemental change in the way its provision is handled.

13 November 2009

Matthew Garrett: Legacy PC design misery

I've spent chunks of the last couple of days fighting a problem that's existed for about 25 years. The 8086 was a 16-bit processor with a 20-bit address space, limiting the maximum physical address that could be accessed to 1MB. However, quirks of the segmented memory system meant that addresses greater than 1MB could be constructed - these would wrap around to the bottom of memory. Because loading the segment registers was a time consuming operation, some programmers used this behaviour as a performance optimisation.

The 80286 introduced 24 bit address space. Unfortunately, this meant that the addresses that previously wrapped to the bottom of memory now pointed at real addresses - not ideal if you were expecting the old behaviour. IBM fixed this by tying the 21st address line (A20 - they're zero indexed) through an and gate, with the default behaviour being to keep it tied at 0 and thus maintaining the old wraparound behaviour. Applications that wanted to access the full address space needed to enable the A20 logic gate. IBM didn't want to add any extra hardware to their system if they could avoid it, so tied the other side of the and gate to a spare pin on the keyboard controller. By writing a couple of bytes to the keyboard controller, your PC-AT stopped pretending to be an XT and gave you access to all of the insanely expensive RAM it had stuffed in it. Hurray!

PCs have been emulating this behaviour since the AT was first cloned. Of course, this being the PC industry, many have got it wrong. There's a set of approaches for controlling the A20 gate that may work, varying in terms of performance and desirability. Most hardware will give the desired result (ie, I have no desire to run DOS executables from 1982, make my A20 work damnit) using any of the various methods of A20 enabling. Some hardware doesn't. The most common method used in bootloaders (where we still have access to system BIOS services) is to call int 15h with an ax of 0x2401, which asks the BIOS to enable A20 for us. This isn't implemented on all hardware, but we should get a failure back that lets us go and bang on the keyboard controller in an attempt to get it to pay attention[1].

Enter the Kohjinsha SC3.

I picked this up second hand in Japan. It's a ridiculously cute little tablet, only slightly larger than hardware that's comfortably in the MID range. It booted a Fedora liveCD perfectly, though having GMA500 graphics meant that what appeared wasn't terribly attractive. Installation proceeded happily enough, followed by a reboot and... nothing. Grub loaded the kernel and initrd, jumped to the kernel and everything hung.

So, for the past couple of days, I was stepping through the kernel setup code, trying to work out where and why it was hanging. I'd got it narrowed down to the region where the kernel tried to free the memory used by the initramfs, but the failure hopped around depending on my kernel build. Something was clearly very wrong. The strangest thing about this was that if I booted the liveCD boot menu and selected "Boot from local drive", everything worked perfectly. isolinux was clearly doing something that grub wasn't, but there's rather a lot of code to step through there.

Things became a lot easier once I found that the OpenSuse version of grub worked. Their grub has a rather smaller set of patches than ours, and only a few looked even plausibly relevant. It only took ten minutes or so to figure out that it was one that altered the A20 code. Things became much clearer then.

The main functional difference between the Suse A20 implementation and the upstream one[2] is that the Suse one explicitly tests whether the A20 enabling worked by putting values at two different addresses that would be the same if A20 is disabled. By comparing them, we know whether A20 is working properly or not. If not it can then fall back to other mechanisms. The Fedora code trusted the BIOS's claim that the int 15 call had worked. The Kohjinsha's BIOS lied, A20 remained disabled, grub copied the kernel and initramfs to chunks of address space that contained lies rather than RAM and everything fell over horribly.

Thankfully, not a difficult fix once the problem was identified. But seriously, people. How hard can it be not to screw this up?

(For an excrutiatingly detailed analysis of how hard it can be not to screw this up, see here)

[1] the Intel Macs don't implement the int 15 approach, but return a failure. They also don't have a legacy keyboard controller, so attempting to hit that resulted in grub falling over. The magic IO port approach works. Another example of how the Intel Macs aren't really PCs...

[2] grub2 implements the more paranoid check

Russell Coker: What to do When You Break a Server

Everyone who does any significant amount of sysadmin work will break a server. Most people who have any significant experience will have broken several. Anyone who has never broken one should be treated with suspicion by other members of the sysadmin team, they probably haven t learned the caution that most of us learn from stuffing something up badly. When you break a server there are some things that you can do to greatly mitigate the scale of the disaster. Firstly carefully watch what happens, for example don t type rm -rf * and then walk away, watch the command if it takes unusually long then press ^C and double-check that you are removing the right files. Fixing a half broken server is often easier than fixing one that has been broken properly and completely! If you don t know what to do then do nothing! Doing the wrong thing can make things worse. Seek advice from the most experienced sysadmin who is available. If backups are inadequate then leave the server in a broken state while seeking advice, particularly if you were working at the end of the day and it doesn t need to be up for a while. There is often time to seek advice by email before doing something. Do not reboot the system! Even if your backups are perfect there is probably some important data that has been recently modified and can be salvaged. Certain types of corruption (such as filesystem metadata corruption) will leave good data in memory where it can be recovered. Do not logout! If you do then you may not be able to login again and this may destroy your chances of fixing the system or recovering data. Do not terminate any programs that you have running. There have been more than a few instances where the first step towards recovering from disaster involved using an open editor session (such as vi or emacs). If the damage prevents an editor from being used (EG by removing the editor s program or a shared object it relies on) then having one running is very important. These procedures are particularly important if you are unable to visit the server. For example when using a hosted server at an ISP the more cost effective plans give you no option to ever gain physical access. So plugging the disks into another machine for recovery is not an option. Does anyone have any other suggestions as to what to do in such a catastrophy? PS This post is related to the fact that I had to recover the last couple of weeks of blog comments and posts from Google s cache Update: I got my data back, not I have to copy one day of blog stuff from one database to another.

10 November 2009

Gunnar Wolf: PieSpy: Modelling IRC interactions

As some of you know, yesterday we started the two weeks of activities for the Fifth On-line Encounter of Free Education, Culture and Software. This year, the main topic for the conference is social networks, but not in the twitter-facebook sense Social networks as ways of understanding and modelling human-to-human interactions. Of course, there is interest from many attendees in using and taking advantage of said social network systems (and others), but the topic is far broader. One of the core tools of EDUSOL, and the only tool that is constantly active through our two weeks of activity, is IRC Which is somewhat of a challenge, as we receive some not-tech-savvy users from all over the world. The first year or two we asked everybody to connect via an IRC client (and we had even decided that it would be a technological filter We thought we would not cater for people without the technical skills to install and use IRC). Life proved us wrong, and we ended up using two web-to-IRC gateways: One of them, via an instance of CGI:IRC, and the other one via a third-party service, Mibbit. Now, the core mind behind EDUSOL is Alejandro Miranda, Pooka; I am not really into education (it is a very interesting topic but far from what I am good at). Pooka has been invovled in on-line communities analysis for several years already, since we worked together at FES Iztacala. For a long time he wanted to graph the interactions at EDUSOL, which was obviously possible given it was all centered on three tools (IRC for live chat, Drupal for generic information and Moodle for the talks and workshops' material This has changed this year, as we are giving much more weight to Drupal and taking it off Moodle), and so last year he was finally able to generate a EDUSOL 2008 conference photo (warning: 2MB image, 4311x3352px), where each person's avatar appears where most of his interactions were. This photo was (unfortunately) achieved using non-free software, but is very much looking at! Now, this year Pooka stumbled upon a great tool, PieSpy (available in Debian package piespy). Piespy generates very interesting interaction graph for IRC channels, as you can see here: Piespy is a bot that sits in the IRC channel logging everything that any given user "says" directly to another one, and graphs it. Not much to say here, only that it generates a very good (and practically real-time) view of what happens on the channel. For this particular channel, and during EDUSOL, you can see here the latest IRC snapshots. Very fun to see - and somewhat addictive how it morphs across time :)
AttachmentSize
EDUSOL 2008 - Conference photo1.97 MB
Current (Monday+Tuesday) interaction graph for our IRC channel39.89 KB

Jon Dowland: Sweet and Sour

Earlier in the year I bought some stuff from UK clothes retailer USC. USC kindly included some gift vouchers in my delivery. This is a good example of how to sweeten a deal and make your customers happy. The vouchers did not include any expiry date on them. I banked them until the autumn, so I could maximize their potential by using them in conjunction with a heavy winter coat. I recently spotted a coat I liked a lot. It's a bit pricey at nearly 200, so I decided to cash in my vouchers. Unfortunately, the vouchers didn't work Frustrating! I emailed the company to ask them why, and it turns out that -- despite not including a printed expiry notice -- the vouchers had expired. I asked the company if they would be willing to issue me with a replacement voucher, and they stopped replying to my emails. This is a good example of how to really sour a customer's impression of you and make them look elsewhere. I did so, and found another retailer selling the coat 20 cheaper.

Matt Zimmerman: Amplify Your Effectiveness (AYE) Conference: Day 1


Morning Session: Project Patterns I chose to attend a session entitled: Is this the Way We Really Want to do Things? Seeing Project Patterns and Changing What You Don t Like (Johanna Rothman). My goal was to explore the causes of the troublesome patterns I see in projects at work. In particular, I see:
  • Too many projects starting up at once
  • Projects being instantiated without enough consideration for the probability of success ( is this a good idea? rather than can we realistically achieve this? )
  • Key people finding out too late about projects which affect them
All of these patterns lead to increased project risk, communication bottlenecks, low motivation, and high stress. In the session, we conducted a simulation of a team, with engineers, a project manager, a senior manager and a customer. I took on the role of the senior manager. In the course of the simulation, we received requirements from the customer, implemented them, and delivered products. While the team was working on implementation, I talked with the customer about what was coming next: what would happen when we delivered, what the next project would be, and so on. Part of the simulation was that I had to be separated from the group while they were working. When we delivered the first batch of products, and the customer was happy with them, it was time to decide what to do next. We gave the customer a choice of two projects we had discussed, one of which was similar to the previous one (but larger scale and more involved), while the other was different. Despite repeated attempts, we could not persuade the customer to prioritize one over the other. So, I decided that we should change gears and start work on the different project. It seemed to be of greater economic value to the customer, and simpler to execute. One of the engineers disagreed with this decision, but didn t explain why. The project manager seemed to agree, and I left the team to work. They produced a prototype, which the customer liked, and with a few small changes it was accepted as a finished product. To my surprise, though, I found out later that the team was in fact working on both projects at once, delivering two different types of products. The decision hadn t actually been made. These unexpected products were delivered to the customer, but didn t meet the expanded requirements, and that work was wasted. The debrief which followed was unfortunately too short, and I didn t feel that we were able to fully explore what the simulation had revealed. The project manager indicated that he hadn t understood the decision to have been made, pointing to a communication failure. This reminded me that while we often think of a decision as an event which happens at a point in time, it is more commonly a process of change, which takes time and must be revisited in order to check progress and evaluate. A decision is really just an idea for a change, and there is more work to be done in order to implement that idea. This can be true even when there is a very explicit decision point : it still takes time for that message to be received, interpreted and accepted. One of the tangents we followed during the debrief had to do with how humans think about numbers. Jerry asked each member of the group to write down a random number, and then we wrote them all on a flipchart. They were: 8, 75, 47, 72, 45, 44, 32, 6, 13 and 47. This reminded me of the analyses of election results which indicate fraud. Afternoon Session: Saying No After lunch, I decided to attend Jerry Weinberg s session, Saying No That Really Means No. This was much larger than the morning session, with over 40 people sitting in a large circle. The subject of discussion was the variety of difficulties that people face in saying no to things which don t seem right for them. For example, saying yes to a project which is doomed to failure. This seemed like a good follow-on to the morning s exercise. Jerry began by asking the audience to name some of their difficulties, and tell stories of times when they had trouble saying no . One of these stories was role-played and analyzed as an example. Most of the time, though, was filled with storytelling and discussion. This is a deeply complex topic, because this problem is rooted in self-image, social norms, egocentrism, misperception, and other cognitive phenomena. There was no key insight for me, just a reinforcement of the necessity of self-awareness. The only way to avoid patterns like this is to notice when they are happening, and that can be challenging, especially in a stressful situation. Once you realize what s happening, there are all sorts of tools which can be applied to the problem: negotiation, requests for help, problem-solving, even simple inaction.

9 November 2009

Bradley Smith: The AVR32 Debian port

So, Debian have kindly funded some new hardware for the AVR32 port, it is the 300 ICnova board, probably one of the best AVR32 boards you can buy to date. The first thing I noticed after finally taking delivery of this a few days ago, it how small it is. I was expecting it to be pretty large, especially seeing as how much stuff it has on it. In fact, it's just smaller than the current NGW100 board I have. So why then new board you ask? Well firstly, I don't own nor have physical access to a board with a display, so this makes testing the all important X server rather difficult [1]. Secondly, the board I currently have is pretty slow due to its lack of RAM, and the fact that it's only 16bit.. This makes using it for porting things rather painful. Finally, the port is getting to the stage where it really needs a porter machine, so that's what this machine will become eventually. Whilst I'm here, I'm going to take this opportunity to prod people about the AVR32 port. ;) The port is currently at a complete standstill since I've run out of buildable packages. This is solely due to the GCC port being buggy (I'm currently fighting with a GCC internal compiler error whilst building Qt4, did I mention that delta is a godsend?) and lots and lots of packages with outdated config. guess,sub files. I currently have around 120 bugs filed for this, so if one of your packages is one of them please fix it. Also, if one of your packages has an Architecture field which avr32 should be in (it should be in most since it's just another linux architecture) please add it. I'd also like to point people at AVR32 dependency status. This is a summary of unbuildable packages and what other packages are stopping them from building (thanks to KiBi for writing this). If you see one of your packages at the bottom of one of the graphs, that means your package is blocking a load of packages from building, and that's bad, m'kay :) I'd very much appreciate help from other people with this, especially for the silly build failures that are trivial to fix. For those who are interested, the port currently stands like this: Now for some useful and interesting AVR32 port related links.. Finally, a picture of the new board, and a big thanks to Debian for funding this! [1] Unfortunately X doesn't seem to work at all yet, it just segfaults. :(

Matt Zimmerman: Ubuntu Server Edition 9.10: No hardware required


In the 9.10 release of Ubuntu Server Edition, we introduced something new for people who are exploring cloud computing using Amazon s Elastic Compute Cloud (EC2). If you haven t tried it yet, EC2 is essentially an API for managing virtual servers. Using a command line tool or script, you request a new VM, and moments later, it is ready for you to ssh in, preconfigured with your ssh public key. When you re finished, you shut it down, and receive a bill for only the time and Internet bandwidth you used (about $0.10 per hour and $0.10-$0.17 per gigabyte). There is no downloading, no installing, and no hardware required for you to set up a server. The first step is to boot it up. Starting with release 9.10, every release of Ubuntu Server Edition is simultaneously available on EC2. This means you can have a new Ubuntu server up and running using your EC2 account with a single command. Ready-to-run Ubuntu machine images are published on EC2 whenever a new Ubuntu release or milestone is available. All you need to know is the AMI, a short string which uniquely identifies the image you want. The AMIs for the 9.10 release are on the download page, in 32- and 64-bit versions for US and Europe zones. Similarly, all Server Edition development builds are available on EC2 as well. When the first builds of Lucid (Ubuntu 10.04) are created, there will be AMIs for those as well. If you want to test drive a new feature, or check compatibility with your application, just fire up a new instance on EC2, do your work, and then terminate it. The whole process can be completed in less than a minute. If you find a problem in our development builds, just run ubuntu-bug on the virtual machine as you normally would, and apport will automatically attach the relevant EC2 details to your bug report. As I mentioned above, EC2 does charge for Internet bandwidth. It does not charge for local bandwidth within your EC2 zone. For this reason, Canonical has set up Ubuntu archive mirrors within EC2, so that you can download all Ubuntu packages and updates for free. Ubuntu virtual machines inside EC2 are automatically configured to use the appropriate mirror, so you don t need to think about it. This is an exciting step forward in making Ubuntu more convenient and powerful to use on EC2, and I encourage you to give it a try. If you ve never used EC2 before, just follow our Starter s Guide to get set up.

7 November 2009

Stefano Zacchiroli: RC bugs of the week - week 9

RCBW - week #9 Here is this week squashes, by yours truly: This week NMUs have been mostly dominated by Haskell's "ambiguous type variable e" squashing.

5 November 2009

Eddy Petrișor: Know your rights!

It is only a few days since my last post about the referendum which is due to happen on the same day as the Presidential elections, but I am forced to make another post because there are lots of incorrect views on the subject.

So let's try take the issues one by one and see what is the problem with every one of them, using reason and the information we have available from more mature democracies than Romania's.

What's the referendum about

The referendum, in this case, is about two issues:

Information on the subject

For an election such as the referendum (basically asking the population on a subject of national importance) every citizen has the right to be correctly informed about the options of the referendum, what each implies, what are the pros and cons for every option, what problems does the referendum tries to resolve.

There is no information on the subject, no videos, no spots, no fliers, nothing, just biased posters (I'll explain later).

Debate on the subject

In order to have a correct information, a clear public debate which should happen, without the pressure of a presidential campaign at the same time as the debate.


Well, we have no debate, so I guess there is no problem of overlapping (sad, but true), but on the other hand, there is no information on the subject.

Analysis of the options

Now let's analyse the two proposed measures in opposition to the current situation.

  1. reducing the number of representatives
    • pros
      • reducing costs
      • minorities have a bigger influence (they can defend their rights - see comments for explanations)
    • cons
      • easier to corrupt (obtaining a corrupt 5% in a 300 seats parliament is easier than in a 500 seats parliament)
      • minorities have a bigger influence (they are disproportionally powerful, in regard to the population they represent - see comments)
      • absence of a representative with a big vote power could allow passing an abusive law (to pass a regular law there is a need of a majority relative to the number of present representatives, not the total number), but, in a numerous parliament, the absence of a person isn't that risky (other colleagues would be present to stop the abusive initiative)
  2. one chamber instead of two
    • pros
      • faster processing of decisions
      • less bureaucratic
    • cons
      • in the current system there can be a passive vote (law passes silently if not discussed) which has just one occasion to be caught, while in the current system silent passing is allowed only in one of the chambers, but needs to be discussed in the other
      • there is a chance that one chamber might amend/catch abusive initiatives with "two chambers", but with "one chamber" there is one chance only
      • more prone to power abuse

So, there are pros and cons for all solutions, but now we can make an informed decision and know the risks for each option.

This shouldn't have been taken in haste without any public debate.


I don't want to influence anyone's vote, so I'll abstain from telling my options.

The president has the right to associate his image with one of the options.

OK, let's analyse that.

The law doesn't explicitly forbids this, but there are some recommendations from Venice on fair elections (sorry, I couldn't find the link) which say that in a vote such as the presidential elections there shouldn't be an unfair advantage for one of the candidates. Why is that relevant? Because the current president actually uses the referendum as his campaign and he's speculating people's innate hatred for political figures.

His election material is only comprised of virulent attacks at the representatives or calls to the population to vote in favour of both questions.


Why is that unfair? Because the current president is using his current position (president) as launch ramp for his campaign as a candidate.

OTOH, if the seal of the presidency was used instead his face, the problem wouldn't have been that bad.


Organisational issues

There were various rumours about how the voting will take place. Here is a list of things I heard:
  1. there will be a single list for both presidential elections and referendum and there will be a single column for signature to confirm your presence at vote
  2. there will be one paper, but two columns for signatures
  3. no matter what the list looks like, you'll be given both sets of ballots, but you can annul the one you don't want to vote on
All the variants above are incorrect from a democratic pov, but to different degrees:

  1. a single signature covering for either of the vaiants "presidential vote only", "refrendum only", "both" asks for trouble because unused ballots in "only" variants might be added later to the valid votes as if they were expressed by voters; since there is no distinct count for each of the ballots to confront that with signatures, before the counting period one could use unused ballots and vote for you on the issue you didn't want to express an opinion on
  2. informing on what to do in some cases such as "only" vote is essential to prevent fraud; e.g.: strike through the cell corresponding to the the election you don't want to express a vote on, instead of signing
  3. giving the ballot unconditionally to everybody, leaving you to decide to annul it or not is abusive, because vote is not compulsory in Romania; more than that, this artificial presence burst (even with void votes) could end up validating the referendum since a 50%+1 presence is needed to pass Constitution modification, as we have here
So, recap:
  1. favours fraud by allowing inside people to obtain untraceable/unaccounted ballots
  2. preventing fraud isn't of any concern, it seems (and could be achieved)
  3. forced voting might lead to validation of the referendum because the voting presence is sufficient, even if there are many annuled ballots
Vote validation and artificial voter presence

Since validation of a referendum on the changes on the constitution depends on sufficient voter presence (there is some mandatory threshold, but I can't find references now), it is in the interest of people wanting to promote the changes to mobilize people to vote.

By different measures the vote presence at the referendum is artificially increased raising the risk of validation, when people, in fact, wouldn't want to validate it and would, to express neutrality, will be be forced to annul the ballot.

Also, a high presence in the referendum might increase the number of voters for presidency, and the people mobilized this way may have a high chance to be biased in favour of the current president because he associates himself with the violent message against the representatives.

What does it take to implement, if referendum passes

If the referendum passes, the changes aren't immediate. The proposed changes are discussed in the Parliament and a final proposal for the constitution changes is made; after approval of the proposals by the Parliament, the ball goes back to population for final approval.

This means it will take a while to see a changed parliament.

Conclusions

Romania is not a true democracy yet and there are areas where individuals should know what their rights are and not let authorities fool them into buying any argument uncritically.

Romanians must learn their rights so they can't be manipulated whenever somebody wants to trick them into some scheme by knowingly omitting issues and creating insane situations.


Romanians, wake up, learn you have rights, form your own oppinion, as distinct individuals, then we might see a change.

4 November 2009

Enrico Zini: Custom function decorators with TurboGears 2

Custom function decorators with TurboGears 2 I am exposing some library functions using a TurboGears2 controller (see web-api-with-turbogears2). It turns out that some functions return a dict, some a list, some a string, and TurboGears 2 only allows JSON serialisation for dicts. A simple work-around for this is to wrap the function result into a dict, something like this: 
@expose("json")
@validate(validator_dispatcher, error_handler=api_validation_error)
def list_colours(self, filter=None, productID=None, maxResults=100, **kw):
    # Call API
    res = self.engine.list_colours(filter, productID, maxResults)
    # Return result
    return dict(r=res)
It would be nice, however, to have an @webapi() decorator that automatically wraps the function result with the dict: 
def webapi(func):
    def dict_wrap(*args, **kw):
        return dict(r=func(*args, **kw))
    return dict_wrap
# ...in the controller...
    @expose("json")
    @validate(validator_dispatcher, error_handler=api_validation_error)
    @webapi
    def list_colours(self, filter=None, productID=None, maxResults=100, **kw):
        # Call API
        res = self.engine.list_colours(filter, productID, maxResults)
        # Return result
        return res
This works, as long as @webapi appears last in the list of decorators. This is because if it appears last it will be the first to wrap the function, and so it will not interfere with the tg.decorators machinery. Would it be possible to create a decorator that can be put anywhere among the decorator list? Yes, it is possible but tricky, and it gives me the feeling that it may break in any future version of TurboGears: 
class webapi(object):
    def __call__(self, func):
        def dict_wrap(*args, **kw):
            return dict(r=func(*args, **kw))
        # Migrate the decoration attribute to our new function
        if hasattr(func, 'decoration'):
            dict_wrap.decoration = func.decoration
            dict_wrap.decoration.controller = dict_wrap
            delattr(func, 'decoration')
        return dict_wrap
# ...in the controller...
    @expose("json")
    @validate(validator_dispatcher, error_handler=api_validation_error)
    @webapi
    def list_colours(self, filter=None, productID=None, maxResults=100, **kw):
        # Call API
        res = self.engine.list_colours(filter, productID, maxResults)
        # Return result
        return res
As a convenience, TurboGears 2 offers, in the decorators module, a way to build decorator "hooks": 
class before_validate(_hook_decorator):
    '''A list of callables to be run before validation is performed'''
    hook_name = 'before_validate'
class before_call(_hook_decorator):
    '''A list of callables to be run before the controller method is called'''
    hook_name = 'before_call'
class before_render(_hook_decorator):
    '''A list of callables to be run before the template is rendered'''
    hook_name = 'before_render'
class after_render(_hook_decorator):
    '''A list of callables to be run after the template is rendered.

    Will be run before it is returned returned up the WSGI stack'''
    hook_name = 'after_render'
The way these are invoked can be found in the _perform_call function in tg/controllers.py. To show an example use of those hooks, let's add a some polygen wisdom to every data structure we return: 
class wisdom(decorators.before_render):
    def __init__(self, grammar):
        super(wisdom, self).__init__(self.add_wisdom)
        self.grammar = grammar
    def add_wisdom(self, remainder, params, output):
        from subprocess import Popen, PIPE
        output["wisdom"] = Popen(["polyrun", self.grammar], stdout=PIPE).communicate()[0]
# ...in the controller...
    @wisdom("genius")
    @expose("json")
    @validate(validator_dispatcher, error_handler=api_validation_error)
    def list_colours(self, filter=None, productID=None, maxResults=100, **kw):
        # Call API
        res = self.engine.list_colours(filter, productID, maxResults)
        # Return result
        return res
These hooks cannot however be used for what I need, that is, to wrap the result inside a dict. The reason is because they are called in this way: 
        controller.decoration.run_hooks(
                'before_render', remainder, params, output)
and not in this way: 
        output = controller.decoration.run_hooks(
                'before_render', remainder, params, output)
So it is possible to modify the output (if it is a mutable structure) but not to exchange it with something else. Can we do even better? Sure we can. We can assimilate @expose and @validate inside @webapi to avoid repeating those same many decorator lines over and over again: 
class webapi(object):
    def __init__(self, error_handler = None):
        self.error_handler = error_handler
    def __call__(self, func):
        def dict_wrap(*args, **kw):
            return dict(r=func(*args, **kw))
        res = expose("json")(dict_wrap)
        res = validate(validator_dispatcher, error_handler=self.error_handler)(res)
        return res
# ...in the controller...
    @expose("json")
    def api_validation_error(self, **kw):
        pylons.response.status = "400 Error"
        return dict(e="validation error on input fields", form_errors=pylons.c.form_errors)
    @webapi(error_handler=api_validation_error)
    def list_colours(self, filter=None, productID=None, maxResults=100, **kw):
        # Call API
        res = self.engine.list_colours(filter, productID, maxResults)
        # Return result
        return res
This got rid of @expose and @validate, and provides almost all the default values that I need. Unfortunately I could not find out how to access api_validation_error from the decorator so that I can pass it to the validator, therefore I remain with the inconvenience of having to explicitly pass it every time.

3 November 2009

Daniel Leidert: Toshiba Tecra A10 (PTSB5E) - Part I

I recently got a Toshiba Tecra A10-1HU laptop. This is a series describing my experiences with this hardware using Debian Sid. This one comes without any operating system. So I got a recent testing-netinstall CD-image and ran the installer procedure (in expert mode). Everything went fine and I ended with a shiny Squeeze and kernel 2.6.30. Because I use Sid as my usual system, my first action was a dist-upgrade to Sid. This is what lspci -nn tells: 
00:00.0 Host bridge [0600]: Intel Corporation Mobile 4 Series Chipset Memory Controller Hub [8086:2a40] (rev 07)
00:02.0 VGA compatible controller [0300]: Intel Corporation Mobile 4 Series Chipset Integrated Graphics Controller [8086:2a42] (rev 07)
00:02.1 Display controller [0380]: Intel Corporation Mobile 4 Series Chipset Integrated Graphics Controller [8086:2a43] (rev 07)
00:03.0 Communication controller [0780]: Intel Corporation Mobile 4 Series Chipset MEI Controller [8086:2a44] (rev 07)
00:19.0 Ethernet controller [0200]: Intel Corporation 82567LM Gigabit Network Connection [8086:10f5] (rev 03)
00:1a.0 USB Controller [0c03]: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #4 [8086:2937] (rev 03)
00:1a.1 USB Controller [0c03]: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #5 [8086:2938] (rev 03)
00:1a.2 USB Controller [0c03]: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #6 [8086:2939] (rev 03)
00:1a.7 USB Controller [0c03]: Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #2 [8086:293c] (rev 03)
00:1b.0 Audio device [0403]: Intel Corporation 82801I (ICH9 Family) HD Audio Controller [8086:293e] (rev 03)
00:1c.0 PCI bridge [0604]: Intel Corporation 82801I (ICH9 Family) PCI Express Port 1 [8086:2940] (rev 03)
00:1c.1 PCI bridge [0604]: Intel Corporation 82801I (ICH9 Family) PCI Express Port 2 [8086:2942] (rev 03)
00:1c.2 PCI bridge [0604]: Intel Corporation 82801I (ICH9 Family) PCI Express Port 3 [8086:2944] (rev 03)
00:1d.0 USB Controller [0c03]: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #1 [8086:2934] (rev 03)
00:1d.1 USB Controller [0c03]: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #2 [8086:2935] (rev 03)
00:1d.2 USB Controller [0c03]: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #3 [8086:2936] (rev 03)
00:1d.7 USB Controller [0c03]: Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #1 [8086:293a] (rev 03)
00:1e.0 PCI bridge [0604]: Intel Corporation 82801 Mobile PCI Bridge [8086:2448] (rev 93)
00:1f.0 ISA bridge [0601]: Intel Corporation ICH9M-E LPC Interface Controller [8086:2917] (rev 03)
00:1f.2 SATA controller [0106]: Intel Corporation ICH9M/M-E SATA AHCI Controller [8086:2929] (rev 03)
01:00.0 Network controller [0280]: Intel Corporation Wireless WiFi Link 5100 [8086:4232]
05:0b.0 CardBus bridge [0607]: Ricoh Co Ltd RL5c476 II [1180:0476] (rev ba)
05:0b.1 FireWire (IEEE 1394) [0c00]: Ricoh Co Ltd R5C832 IEEE 1394 Controller [1180:0832] (rev 04)
05:0b.2 SD Host controller [0805]: Ricoh Co Ltd R5C822 SD/SDIO/MMC/MS/MSPro Host Adapter [1180:0822] (rev 21)
05:0b.3 System peripheral [0880]: Ricoh Co Ltd R5C843 MMC Host Controller [1180:0843] (rev ff)
05:0b.4 System peripheral [0880]: Ricoh Co Ltd R5C592 Memory Stick Bus Host Adapter [1180:0592] (rev 11)
05:0b.5 System peripheral [0880]: Ricoh Co Ltd xD-Picture Card Controller [1180:0852] (rev 11)
Things that worked out-of-the-box: Display/Graphics (Intel GMA 4500M HD) The integrated graphics chipset, an Intel GMA 4500M HD, works with the xserver-xorg-video-intel package and the default X.org configuration. No custom /etc/X11/xorg.conf is necessary. The (IMHO non-reflecting?) display runs at 1280 800 at 60 Hz. Cable network device The integrated Intel 82567LM Gigabit Network device works with the e1000e kernel module. No customization was necessary. Touchpad/Trackpoint The notebook comes with the so called Toshiba Dual Pointing Device a touchpad (+2 buttons) and a trackpoint (+2 buttons). Both worked without customization. Energy savings/Suspend/Resume apmd and acpid handle this. No problems yet. Suspend/Resume works. I did not yet test (and I m not sure if I should) hibernate and uswsusp. Webcam The webcam is a CNA7157 model or at least detected as such. The video4linux modules handle it and the cheese application produces pictures. Volume-control wheel It works. Module and package information will follow as soon as I figure them out. Things that needed customization WLAN (Intel WiFi Link 5100) The notebook comes with an Intel WiFi Link 5100 device. It does not work out-of-the-box. Following this article, I ve installed the firmware-iwlwifi package and loaded the iwlagn module. Sound (Intel) For the sound device to work the snd_hda_intel module is necessary. Further the following lines must be added to /etc/modprobe.d/alsa-base.conf 
# not sure about the first line, start adding the second only
options snd-cmipci mpu_port=0x330 fm_port=0x388
options snd-hda-intel index=0 model=toshiba position_fix=1
Unsure/Not yet working Temperature sensor After installing the sensors-applet I got two identical temperature displays. I m not sure what they show. Anybody an idea how to examine this? Modem The notebook comes with an internal modem device: 
00:03.0 Communication controller [0780]: Intel Corporation Mobile 4 Series Chipset MEI Controller [8086:2a44] (rev 07)
From what I read I need hsfmodem or the sl-modem packages. Unfortunately the latest version is not available for amd64 from the archive, although it seems to have been built. Untested I did not yet test Summit This what lspci -k -nn tells about the used kernel modules: 
00:00.0 Host bridge [0600]: Intel Corporation Mobile 4 Series Chipset Memory Controller Hub [8086:2a40] (rev 07)
	Subsystem: Toshiba America Info Systems Device [1179:0001]
	Kernel driver in use: agpgart-intel
00:02.0 VGA compatible controller [0300]: Intel Corporation Mobile 4 Series Chipset Integrated Graphics Controller [8086:2a42] (rev 07)
	Subsystem: Toshiba America Info Systems Device [1179:0004]
00:02.1 Display controller [0380]: Intel Corporation Mobile 4 Series Chipset Integrated Graphics Controller [8086:2a43] (rev 07)
	Subsystem: Toshiba America Info Systems Device [1179:0004]
00:03.0 Communication controller [0780]: Intel Corporation Mobile 4 Series Chipset MEI Controller [8086:2a44] (rev 07)
	Subsystem: Toshiba America Info Systems Device [1179:0001]
00:19.0 Ethernet controller [0200]: Intel Corporation 82567LM Gigabit Network Connection [8086:10f5] (rev 03)
	Subsystem: Toshiba America Info Systems Device [1179:0001]
	Kernel driver in use: e1000e
00:1a.0 USB Controller [0c03]: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #4 [8086:2937] (rev 03)
	Subsystem: Toshiba America Info Systems Device [1179:0001]
	Kernel driver in use: uhci_hcd
00:1a.1 USB Controller [0c03]: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #5 [8086:2938] (rev 03)
	Subsystem: Toshiba America Info Systems Device [1179:0001]
	Kernel driver in use: uhci_hcd
00:1a.2 USB Controller [0c03]: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #6 [8086:2939] (rev 03)
	Subsystem: Toshiba America Info Systems Device [1179:0001]
	Kernel driver in use: uhci_hcd
00:1a.7 USB Controller [0c03]: Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #2 [8086:293c] (rev 03)
	Subsystem: Toshiba America Info Systems Device [1179:0001]
	Kernel driver in use: ehci_hcd
00:1b.0 Audio device [0403]: Intel Corporation 82801I (ICH9 Family) HD Audio Controller [8086:293e] (rev 03)
	Subsystem: Toshiba America Info Systems Device [1179:0001]
	Kernel driver in use: HDA Intel
00:1c.0 PCI bridge [0604]: Intel Corporation 82801I (ICH9 Family) PCI Express Port 1 [8086:2940] (rev 03)
	Kernel driver in use: pcieport-driver
00:1c.1 PCI bridge [0604]: Intel Corporation 82801I (ICH9 Family) PCI Express Port 2 [8086:2942] (rev 03)
	Kernel driver in use: pcieport-driver
00:1c.2 PCI bridge [0604]: Intel Corporation 82801I (ICH9 Family) PCI Express Port 3 [8086:2944] (rev 03)
	Kernel driver in use: pcieport-driver
00:1d.0 USB Controller [0c03]: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #1 [8086:2934] (rev 03)
	Subsystem: Toshiba America Info Systems Device [1179:0001]
	Kernel driver in use: uhci_hcd
00:1d.1 USB Controller [0c03]: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #2 [8086:2935] (rev 03)
	Subsystem: Toshiba America Info Systems Device [1179:0001]
	Kernel driver in use: uhci_hcd
00:1d.2 USB Controller [0c03]: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #3 [8086:2936] (rev 03)
	Subsystem: Toshiba America Info Systems Device [1179:0001]
	Kernel driver in use: uhci_hcd
00:1d.7 USB Controller [0c03]: Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #1 [8086:293a] (rev 03)
	Subsystem: Toshiba America Info Systems Device [1179:0001]
	Kernel driver in use: ehci_hcd
00:1e.0 PCI bridge [0604]: Intel Corporation 82801 Mobile PCI Bridge [8086:2448] (rev 93)
00:1f.0 ISA bridge [0601]: Intel Corporation ICH9M-E LPC Interface Controller [8086:2917] (rev 03)
	Subsystem: Toshiba America Info Systems Device [1179:0001]
00:1f.2 SATA controller [0106]: Intel Corporation ICH9M/M-E SATA AHCI Controller [8086:2929] (rev 03)
	Subsystem: Toshiba America Info Systems Device [1179:0001]
	Kernel driver in use: ahci
01:00.0 Network controller [0280]: Intel Corporation Wireless WiFi Link 5100 [8086:4232]
	Subsystem: Intel Corporation Device [8086:1201]
	Kernel driver in use: iwlagn
05:0b.0 CardBus bridge [0607]: Ricoh Co Ltd RL5c476 II [1180:0476] (rev ba)
	Subsystem: Toshiba America Info Systems Device [1179:0001]
	Kernel driver in use: yenta_cardbus
05:0b.1 FireWire (IEEE 1394) [0c00]: Ricoh Co Ltd R5C832 IEEE 1394 Controller [1180:0832] (rev 04)
	Subsystem: Toshiba America Info Systems Device [1179:0001]
	Kernel driver in use: firewire_ohci
05:0b.2 SD Host controller [0805]: Ricoh Co Ltd R5C822 SD/SDIO/MMC/MS/MSPro Host Adapter [1180:0822] (rev 21)
	Subsystem: Toshiba America Info Systems Device [1179:0001]
	Kernel driver in use: sdhci-pci
05:0b.3 System peripheral [0880]: Ricoh Co Ltd R5C843 MMC Host Controller [1180:0843] (rev ff)
	Kernel driver in use: ricoh-mmc
05:0b.4 System peripheral [0880]: Ricoh Co Ltd R5C592 Memory Stick Bus Host Adapter [1180:0592] (rev 11)
	Subsystem: Toshiba America Info Systems Device [1179:0001]
05:0b.5 System peripheral [0880]: Ricoh Co Ltd xD-Picture Card Controller [1180:0852] (rev 11)
	Subsystem: Toshiba America Info Systems Device [1179:0001]

1 November 2009

Axel Beckert: /var/cache/apt/ on tmpfs

My ASUS EeePC 701 (4G) nemo running Debian Sid has a 4 GB SSD as main disk, which is on the one hand quite full (mostly with software I use, but also local working copies of software I work on) and on the other hand an SSD, so I always try to reduce the amount of write to disk without losing convenience. Similar issues have systems which run off a CF or SD card or maybe even an USB stick. Since I ordered a 2 GB RAM bar together with the EeePC, I not bound to the 512 MB which it had originally. But on the other hand I seldom needed more than 1 GB of RAM. Usually I needed between 400 MB and 1 GB of RAM. So it s quite obvious to use tmpfs on as many places as possible. Making /tmp, /var/run and /var/lock tmpfs were the most obvious directories to mount as tmpfs. Especially /var/run on tmpfs brought up a few bugs a while ago (mostly init.d scripts relying on /var/run/$PACKAGENAME/ s existence), but it s no hassles to use nowadays. Even in Debian Stable such bugs got fixed. Next target to explore for was /var/cache. According to the FHS, /var/cache is intended for cached data from applications. [ ] The application must be able to regenerate or restore the data. So it should be safe to put anything under /var/cache on tmpfs. One directory in there which gets written quite often and with a lot of data on Debian Unstable is /var/cache/apt and its subdirectories, especially /var/cache/apt/archives. If you update your Sid installation daily, all new or updated .debs will be downloaded to /var/cache/apt first. So I put /var/cache/apt on tmpfs by putting the following line into /etc/fstab: 
tmpfs /var/cache/apt tmpfs defaults,noexec,nosuid,nodev,mode=0755 0 0
But despite FHS stating that anything under /var/cache must be reproducible by the application, apt is puking and refusing to work: 
!447 Z31 ?0 L1 root@nemo:pts/0 (-zsh) 16:13:10 [~] # apt-get update
E: Archive directory /var/cache/apt/archives/partial is missing.
!448 Z32 ?100 L1 root@nemo:pts/0 (-zsh) 16:13:17 [~] #
If you create /var/cache/apt/archives/partial, it will also argue about /var/cache/apt/partial. Of course the workaround is simple: Just put mkdir -p /var/cache/apt/partial /var/cache/apt/archives/partial in /etc/rc.local. But nevertheless, this is a bug in apt which already has been reported by madduck earlier this year (#523920). Unfortunately the APT maintainers have not yet even commented on this FHS violation and therefore also a Debian Policy (Section 9.1.1) violation. One more thought about /var/cache/apt vs only /var/cache/apt/archives: apt-file also caches its data under /var/cache/apt. So if you want to use apt-file after a reboot and have /var/cache/apt mounted as tmpfs, you have to run apt-file update first and it will download all Contents files (can be dozens of megabytes) and not only the differences to previously downloaded Contents files. So if you use apt-file a lot, you probably go better with making only /var/cache/apt/archives tmpfs and not whole /var/cache/apt.

Jaldhar Vyas: Unfortunately They Don't Make Squeeze Costumes

woody Jersey City needs a candy stimulus.

Dirk Eddelbuettel: Adventures with Comcast: Part ohnoesnotanotherone in an ongoing series

Regular readers of this blog (yes, both of you!) may remember the computer/broadband/ directory that this post appears in as the collection of my Comcastic (yeah right) experiences with my ISP. But I think this week may top everything. I'll just try to jot down some notes before I forget all the gory details:

30 October 2009

Martin F. Krafft: Webcast of my Ph.D. defence

I am sitting in Lero, waiting for the next 30 minutes to pass before I shall defend my Ph.D. thesis in front of a board of three examiners. It will start at 9:30 GMT. I cannot decide whether I am nervous, or whether I had too much coffee. Someone just asked me if I am well-prepared Yes, I worked four years for this. Excited and prepared. If you want to tune in, my initial, public presentation will be webcast (using proprietary media, unfortunately (Flash required)). I hope to have a recording available later. The following interrogation by the examiners is not public (even though I wanted it to be). I forgot my Lederhosen at home, or I would have worn them today.

Matt Brubeck: Compleat: Programmable Completion for Everyone

Compleat is an easy, declarative way to add smart tab-completion for any command-line program. For a quick description, see the README. For more explanation and a brief tutorial, keep reading... Background I'm one of those programmers who loves to carefully tailor my development environment. I do nearly all of my work at the shell or in a text editor, and I've spent a dozen years learning and customizing them to work more quickly and easily. Most experienced shell users know about programmable completion, which provides smart tab-completion for for supported programs like ssh and git. (If you are not familiar it, you really should install and enable bash-completion, or the equivalent package for your chosen shell.) You can also add your own completions for programs that aren't supported but in my experience, most users never bother. When I worked at Amazon, everyone used Zsh (which has a very powerful but especially baroque completion system) and shared the completion scripts they wrote for our myriad internal tools. Now that I'm in a startup with few other command line die-hards, I'm on my own when it comes to extending my shell. So I read the fine manual and started writing my own completions. Over on GitHub you can see the script I made for three commands from the Google Android SDK. It's 200 lines of shell code, fairly straightforward if you happen to be familiar with the Bash completion API. But as I cranked out more and more case statements, I felt there must be a better way... The Idea It's not hard to describe the usage of a typical command-line program. There's even a semi-standard format for it, used in man pages and generated by libraries like GNU AutoOpt. Here's one for android, one of the SDK commands supported by my script: 
 android [--silent   --verbose]
   ( list [avd target]
     create avd ( --target <target>   --name <name>   --skin <name>
                   --path <file>   --sdcard <file>   --force ) ...
     move avd (--name <avd>   --rename <new>   --path <file>) ...
     (delete update) avd --name <avd>
     create project ( (--package --name --activity --path) <val>
                       --target <target> ) ...
     update project ((--name --path) <val>   --target <target>) ...
     update adb )
My idea: What if you could teach the shell to complete a program's arguments just by writing a usage description like this one? The Solution With Compleat, you can add completion for any command just by writing a usage description and saving it in a configuration folder. The ten-line description of the android command above generates the same results as my 76-line bash function, and it's so much easier to write and understand! The syntax should be familiar to long-time Unix users. Optional arguments are enclosed in square brackets; alternate choices are separated by vertical pipes. An ellipsis following an item means it may be repeated, and parentheses group several items into one. Words in angle brackets are parameters for the user to fill in. Let's look at some more features of the usage format. For programs with complicated arguments, it can be useful to break them down further. You can place alternate usages on their own lines separated by semicolons, like this: 
android <opts> list [avd target];
android <opts> move avd (--name <avd> --rename <new> --path <file>)...;
android <opts> (delete update) avd --name <avd>;
...and so on. Rather than repeat the common options on every line, I used a parameter <opts>. I can define that parameter using the same usage syntax. 
opts = [ --silent   --verbose ];
For parameters whose values are not fixed but can be computed by another program, we use a ! symbol followed by a shell command to generate completions, like this: 
avd = ! android list avd   grep 'Name:'   cut -f2 -d: ;
target = ! android list target   grep '^id:'  cut -f2 -d' ' ;
And any parameter without a definition will just use the shell's built-in completion rules, which suggest matching filenames by default. The README file has more details of the usage syntax, and instructions for installing the software. Give it a try, and please send in any usage files that you want to share! (Questions, bug reports, or patches are also welcome.) Future Work For the next release of Compleat, I would like to make installation easier by providing better packaging and pre-compiled binaries; support zsh and other non-bash shells; and write better documentation. In the long term, I'm thinking about replacing the usage file interpreter with a compiler. The compiler would translate the usage file into shell code, or perhaps another language like C or Haskell. This would potentially improve performance (although speed isn't an issue right now on my development box), and would also make it easy for usage files to include logic written in the target language. Final Thoughts Recently I realized that parts of my work are so specialized that my parents and non-programmer friends will probably never really get them. For example, Compleat is a program to generate programs to help you... run programs? Sigh. Well, maybe someone out there will appreciate it. Compleat was my weekends/evenings/bus-rides project for the last few weeks (as you can see in the GitHub punch card), and my most fun side-project in quite a while. It's the first "real" program I've written in Haskell, though I've been experimenting with the language for a while. Now that I'm comfortable with it, I find that Haskell's particular combination of features works just right to enable quick exploratory programming, while giving a high level of confidence in the behavior of the resulting program. Compleat 1.0 is only 160 lines of Haskell, excluding comments and imports. Every module was completely rewritten at least once as I tried and compared different approaches. This is much less daunting when the code in question is only a couple dozen lines. I don't think this particular program would have been quite as easy to write at least for me in any of the other platforms I know (including Ruby, Python, Scheme, and C). I had the idea for Compleat more than a year ago, but at the time I did not know how to implement it easily. I quickly realized that what I wanted to write was a specialized parser generator, and a domain-specific language to go with it. Unfortunately I never took a compiler-design class in school, and had forgotten most of what I learned in my programming languages course. So I began studying parsing algorithms and language implementation, with Compleat as my ultimate goal. My good friend Josh and his Gazelle parser generator helped inspire me and point me toward other existing work. Compleat actually contains three parsers. The usage file parser and the input line tokenizer are built on the excellent Parsec library. The usage file is then translated into a parser that's built with my own simple set of parser combinators, which were inspired both by Parsec and by the original Monadic Parser Combinators paper by Graham Hutton and Erik Meijer. The simple evaluator for the usage DSL applies what I learned from Jonathan Tang's Write Yourself a Scheme in 48 Hours. And of course Real World Haskell was an essential resource for both the nuts and bolts and the design philosophy of Haskell. So besides producing a tool that will be useful to me and hopefully others, I also filled in a gap in my CS education, learned some great new languages and tools, and kindled an interest in several new (to me) research areas. It has also renewed my belief in the importance of "academic" knowledge to real engineering problems. (I've already come across at least one problem in my day job that I was able to solve faster by implementing a simple parser than I would have a year ago by fumbling with regexes.) And I'll be even happier if this inspires some friends or strangers to take a closer look at Haskell, Parsec, or any problem they've thought about and didn't know enough to solve. Yet.

29 October 2009

John Goerzen: Apache Update

Thanks everyone for the helpful comments yesterday on Apache vs. lighttpd. I ve taken the first few steps towards improving things. I ve eliminated mod_php5, switched all PHP to FastCGI, and switched from the prefork to the event MPM. The event MPM docs say that it s incompatible with mod_ssl, but it worked, and googling for that turned up mailing list posts from 2006 saying it was fixed. The only real glitch was that egroupware inexplicably depends on libapache2-mod-php5. It works fine without it, but I had to create a fakelibapache2-mod-php5 that provides libapache2-mod-php5 to convince the system not to remove egroupware when I switched to the event MPM. I went from 28 Apache processes to 4 Apache processes plus an average of around 2 php5-cgi processes which, despite the name, actually do grok FastCGI. Three of my Apache processes now have an RSS of about 20M, and the other of about 30M. The php5-cgi processes are at around 40M. My old Apache processes ranged from 18M to over 40M. So there is some memory savings here, though not drastic. My Redmine processes, two of them, each use over 50M. Ruby on Rails still is leaving me with a bad taste. At least it s not a Java app; it seems a lot of those allocate 2GB before they ever start accepting connections. I ll see how this goes for awhile, but for now I doubt that moving to lighttpd (or similar) will yield enough benefit to make it worth the effort. But there may be some benefit in inserting a caching proxy in front of Apache, so I may yet do that.

27 October 2009

Anand Kumria: Falsum in uno, falsum in omnibus.

Note: via Blog post (obviously different formatting) and sent by email roughly: Lying in one, lying in all. That is from Steve Waddington. Please see the attached screenshot I have obliterated service identifying information. View of my inbox for the last 18 months show more than one email. I find it telling that what I said in my original post, of all the things there the one you decided to fact-check was the number of emails between us. And you got that wrong. What was is you said again? Falsum in uno, falsum in omnibus? Do you still stand by that statement? If you have looked at the original complaint, no one asked for anything but information on 'what went wrong'. It simply noted that fact that payment occurred, and we were disconnected and went on to say "We find the above experience of diconnecting [sic] our service by EXETEL a very poor customer relationship experience." The response to that could have been: "Yes, that is a poor customer experience. We will look into it and see if we can improve." Instead we had an accusatory email back commenting, firstly, of the amount of time of the disconnection. As if someone paying for something and not receiving it for only 1 minute is OK. The point being missed we paid for something and we did not receive it. The point being missed -- this is a poor customer experience The second email then, falsely, accused us of not actually paying in the first place! Later the individual concerned, after the TIO complaint, clarifies that 'yes', we did indeed pay. Very telling indeed, then, when you were pointed to my blog entry and rather than pick up on the stratgeic ("we are getting feedback, for free, on how we can improve our processes") you looked at the tactical ("he is lying! How I prove it and thus be happy in ignoring what was said"). As I said, It is easier to hear when your fingers are not in your ears. In light of your recent blog post, and the recently released TIO report, I find Exetels' behaviour all the more perplexing. Complain to Exetel, get your service contract summarily terminated. Complain about Exetel, and get libelled publicly. But I guess when you have 1/6th (163 out of 620) of your complaints related to Customer Service (from the TIO report) and 3% of your own existing customers (from your blog post) complaining, you get the reputation you earn.

26 October 2009

Patrick Schoenfeld: Building a 15W Debian GNU/Linux system

When the Intel Atom was revealed to the public I didn't came around to say: "Wow!", because that piece of hardware promises to be a generic-x86 1.6 GHZ CPU with a total power consumption of 2 Watt, which is amazing considering that x86 hardware generally wasn't an option if you wanted to build a low-power system. But then the first chipsets were presented to the public and the Atom became a farce, because you don't want to have a chipset that eats over 25W for a CPU which consumes 2W. That was basically laughable.

Recently I managed to find out that there is a new chipset out there, the Intel i945GSE, which runs at about 11W TDP, including the soldered-on-board N270 atom cpu. And I convinced myself that this could get my new homeserver. So together with a 2.5" drive I could get a system which runs with about 15W maximum power consumption, which is amazing, given that the Arcor Easybox my provider gave to me seems to have similar maximum power consumptions. And it isn't able to provide me with the great flexibility, the new Atom system is.

So I bought the following components:

It took a while to get those components together, especially because I previously decided for an Antec case which I ordered from K&M Elektronik, but as they didn't keep their delivery promise I came to the M350. What a luck.

Running Debian on this machine is the easy part, you would think. This is true, for some exceptions. First: Lenny runs fine. I've installed the notebook hd in my desktop and then put it in the atom, when I got the first hardware and it worked right away. Except of a grub message, which is disturbing and which I didn't manage to fix right now (grub says "Error: No such disk" just to get the menu a seconds later anyway and boot the system flawless).

What didn' t work exactly reliable was the included network chip. Its quiet a shame to say that, but if you buy an Intel board, wouldn't you expect that it would run Intel components? Unfortunately this is not true for the atom board. It has a Realtek RTL8111 network chip, which isn't properly supported by the 2.6.26 kernel (that means the kernel think it is and loads a rtl8169 module, which isn't able to properly detect a link).
The workaround for this is to use a 8168 module from Realtek and compile it for your kernel, but as I equipped this system with an Atheros 2424 PCIe chipset for playing WLAN AP, too, I had to upgrade to 2.6.31 anyway and there the chip is fully supported by 8169.

Making the system an access point has been surprisinly easy as well. The greatest pain was to find a Mini PCIe WLAN card, because after all this isn't very common. However I found one based on an Atheros 2424 chipset and bought it. I additionaly bought an SMA-antenna connector that I could mount into the case (the M350 has a preparation hole for it) and an SMA antenna.
Setting this up, has been fairly easy. You need to know, that running master mode with newer mac-subsystem-drivers in Linux doesn't allow setting master mode directly. Instead you need to use an application to manage everything, which is capable of running cards over netlink. Thats hostapd. The unfortune is, that the lenny version is too old and so I built myself a (hacky) backport of the sid version, which isn't that hard anyway, because rebuilding against Lenny is enough. Additional you need a kernel 2.6.30 with compat-wireless extensions, or an 2.6.31, because previously the ath5k driver didn't support the master mode. After that getting hostapd up is a matter of a 4 - 15 lines configuration file. For me its now running in 802.11g with WPA and a short rekeying interval with 14 lines of configuration.

After all I'm satisfied with the system. Without any fan the CPU constantly runs at 55 C, which is okay, given that it must operate within 0 and 90 C according to the tech specs. The system and the disk are somewhat lower (47 and 39 C). The power of this system is more then enough. Its booting quick and working with it works without latencies, even when the system is doing something. What I haven't yet tested is weither the power consumption actually fulfills the expactations. I will do so, once I got a wattmeter.

Next.

Previous.