Search Results: "tar"

13 April 2008

Joey Hess: when we vote

votes-percent.png votes.png These graphs can help give some idea of how the shortened DPL voting period this year changed voting patterns. Each vote starts on a Sunday (except 2005 which was a day late). Calls for votes are posted again each weekend during the vote, and a final call two days before the end of the vote. These are responsible for most of the spikes on the graph, and the trail off after each CFV is probably due to some people seeing it the next day. Shortening the voting period seems to have made more people respond to the second and third CFV (both in percentage and absolute number of voters). It's also interesting that a smaller percent than usual of votes were recieved on the last day this year. There seems to not have been a rush to vote on the last day despite the shortened period. I wonder if posting CFVs more frequently would result in more voters, or if it would just level off the graph? (Graph produced using this tool and final vote timing logs provided by Manoj.)

Julien Blache: Getting things done on SANE again

There have been a number of discussions on sane-devel in the past months revolving around the SANE2 standard (still in development) and/or extensions to the current SANE1 standard. SANE2 is an effort that has been “ongoing” (on and off, mostly off, unfortunately) for a number of years already. This led to a number of questions being raised, most notably this one: do we need SANE2, or can we get away only with extensions to SANE1? Or should we just redesign the whole stack from the ground up?
This matter is undecided as of yet, but extensions to SANE1 are planned, being discussed or in the making for some of them. During the discussions, a fork of SANE has been announced, dubbed “SANE Evolution”. No idea where that is going, but it’ll probably end up being merged back into SANE in the end. So, in a nutshell, things are moving again (albeit slowly), which is good news considering the project was mostly asleep in recent times. To celebrate that (and because I now have total control over my free time again), I’ve been working on the network side of SANE again. In 2 weeks (3 week-ends, roughly): Turning saned into a regular daemon made it possible to clean up a good chunk of the code duplication that was introduced by my previous AF-indep/IPv6 work; the startup code is readable again. The Avahi support is (I hope) a very nice thing as, when enabled, configuration of the net backend on the client side is no longer needed. This is an experimental feature still and as such it’s disabled by default at build time. I have a couple of ideas to improve that feature, but this means saned will have to evolve even more so it’ll take some time. The WireShark dissector is a tool for working on enhancing the network protocol. It proved useful already with the information leak I spotted with it (though it’s minor). It’ll be a long work to extend the network protocol and implement that into saned and the net backend in a backward-compatible manner.

Russell Coker: Car Sharing in Melbourne

Recently I noticed that some parking spots in the city area are reserved for car sharing. There seems to be two car sharing companies operating. Flexicar [1] costs $50 to join, has a $10 per month membership fee (which includes one hour of driving at a value of $12), and costs $12 per hour or $80 per day to drive a car plus $0.15 per Km if you drive more than 100Km in a day. They also have pre-paid plans which bring all the guesswork and complexity of mobile phone bills to car rental. Also they give discounts to members of the City Rewards [2] program which are greater than the membership cost, so anyone who plans to join them should join City Rewards first. To use a car you phone up or use their web site to make a reservation. Then you find the car you reserved at it’s designated location and swipe your card across the windscreen to unlock the doors (presumably it’s an RFID card). The glovebox has the key for the ignition as well as fuel cards for any fuel you use a BP, Shell, and Caltex petrol stations (presumably if you run low on petrol when not near those ones you end up paying). All you pay is the rental rate and any tolls for toll roads you use. When someone doesn’t return the vehicle to it’s designated spot on time there are penalty rates, which may include the cost of a taxi fare for the next person who had booked it. It seems like a really good idea that can save significant amounts of money for people who live in the central areas (the costs of maintaining and insuring a car are significant, as is the depreciation on a new car as an asset). The competitor is Charterdrive [3] which costs $25 to join (less than Flexicar but the same once you consider the discount), the same $10 per hour for 8:30AM to 5:30PM hours, and then cheaper rates for evenings and weekends. $25 for a night (5:30PM to 8:30AM the next morning) is good value if you want to drive home from work and then return the next day (if you work late then you might expect to pay $30 or more for a single taxi ride to get home). The weekend rate of $90 for 5:30 Friday to 8:30 Monday is also quite competitive, I expect that there are many people who only use a car on the weekend who could benefit a lot from this. $90 per weekend for 40 weekends a year (some weekends you would stay home or just use public transport) is $3600, insurance, registration, and basic maintenance of a car that you would want to own would cover most of that. Charterdrive does charge $0.20 per Km though for all journeys (with a discount rate of $0.15 per Km for long journeys on some plans), as opposed to Flexicar only charging $0.15 per Km for distances in excess of 100Km per day. Charterdrive seems to be a newer company and has a far smaller presence. But it’s business model seems a little different and the focus on renting cars for people to drive home means that some people might benefit from being members of both companies. The $0.20 per Km makes Charterdrive more expensive more expensive for most city use, and the discounts offered for Flexicar seem to make it cheaper for use during business hours. Charterdrive states that they have a deal with Red Spot Car Rentals [4]. It is not stated on their web site if you want to use one of their cars and they are all in use, I wonder whether a Red Spot car would be provided for the same price. Flexicar however claims that they aim to have a ratio of cars to members sufficient to make such things unlikely. Maybe it would be prudent to join both organisations so that if one had no vehicles available then you could use the other? In one city car park that I often pass the Flexicar and Charterdrive parking spots are adjacent so there would be no difference in convenience in terms of which one you use. I wonder whether they will continue getting adjacent spots. If many people join both organisations then it would be more effective if they don’t get adjacent spots to get better aggregate coverage. I think that at the moment the main challenge for both companies is to grow the popularity of the car-sharing business. I expect that the real competition for who gets the biggest slice of that business will happen in a few years time. Update: There is another Australian car-share company in operation named GoGet [5], interestingly I discovered their existence when I reviewed the Google advertising on this post… GoGet has a significant presence in Sydney, a small presence in Melbourne, and is only making a start in Queensland. Their hourly rates are significantly lower than the others (as little as $4.40 per hour) but distance rates are as high as $0.35 per Km. One significant benefit is that they have plans for two or three drivers which could allow an entire family to sign up on one account. If you drive less than 20Km in an hour (which would not be uncommon in city driving) then GoGet would be cheapest.

Jeff Bailey: Update of the moment

So here's the latest update on how I'm doing:

* Got the stomach flu this week. After 3 days of being ill, went to the doctors and got anti-nausea, anti-diarrhea medication, and 400ccs of Saline by IV for the dehydration. Joy. I'm going to try to go off the anti-nausea drugs tomorrow to see if I'm keeping things down now.

* Leif swam for the first time. I didn't get video, but opted for some nice stills. I *will* actually upload the last 5 months of pictures sometime soon. Really.

* Found an accountant who knows what to do for Canadians who just moved here. Yay! US Taxes will get done!

* The incision has no redness around it at all anymore, which is way cool. On Tuesday, I go in for my 6-week checkup. I asked at the 4-week mark whether I could go in the hottub, and was told that they'd probably OK it at the 6-week mark. I start 6 weeks of physiotherapy the day after that.

* My stamina for walking seems largely back. I can walk to Safeway at a decent pace (I still have to rest when I'm there). I can get to the mailbox and back without a break now. The part that causes me grief still is the muscles on my left shoulders. I'll ask Angie which one it is. It's been more sore than the others since the beginning, but hopefully it's something the physio can help me sort out. If it weren't for that, I'd be cutting back the pain killers already. I still might do that when I know I have a 6-hour stretch of low activity.

* For lack of anything better to do, I've been playing with Eclipse. I rediscovered eclipse a year ago when I was working on the ConnectUU site. The PHP mode was surprisingly good and helpful, and I was quite productive in it. I tried shortly after that to use PyDev, and it did horrible things to eclipse. I've tried it again, and I'm enjoying doing a bit of Java coding (homework), and Python coding in it now. I may finally have found an environment in which I can be more productive than I was in vi.

* Ernie (Angie's dad) and Janet (his wife / one of Leif's Grandmothers) have been here for the weekend helping us clean up. Janet is an absolute whiz at organising things, and the kitchen is way more usable then it has been since we got here in November. I need to make sure that I send out thank-you cards to everyone who's come and helped us. Kit was here a couple of weekends ago. Heather was here last weekend. Leslie, Grant, Jim/Sarah, and Andrew/Sarah have each brought us food. Jamu and Sandy each came for longer periods right after the surgery. Having people come and help has made a huge difference to us keeping things together. Without outside help, I don't know how we would've managed.

I'm a day and a bit off of 6-weeks post-surgery. I'm feeling good.

12 April 2008

Gerfried Fuchs: Pregnant Husband

It's a bit strange. Still. Writing this personal part of my blog, opening some of my most inner thoughts to the wild public. Though, so wild it doesn't seem to be. To be honest I can't remember having received any bad feedback on my personal stuff, only positive, supportive ones. (Or, there was one. Though, it wasn't related to the core of the personal section but someone thinking it would be "cool" to misinterpret some statement therein and try to hurt me with it. It only made me laugh at them, trying to "use" it as an argument.)
No bad feedback might be related to that I don't have comments enabled because I don't want to have a moderation system that would make it look like I filter out bad comments but I also don't want to open it up to SPAM. And people who propably usual leave scathing comments don't consider it convenient to address me directly, via any IM system, including emails. Anyway, opening in that way has quite some benefits: For a start, it helps me myself to keep track of things that happened. Secondly, it hopefully helps others that are in similar situations to see that they aren't alone out there and that one can survive with not hiding it. But last but not least some people address me and provide me with interesting links on the topic. I think they shouldn't be just hidden in my personal mailbox so I am going to offer them to a broader audience here. I won't show the names of who sent them along, I'm not sure if they would like being connected to the topic. But they can be assured of my blessing for offering them to me. First link I like to hand out is an article from advocate.com about a pregnant husband. Yes, this was no typo and the reason why I haven't posted it right the next day because I received that link on March 31st. I can just wish all the best to Thomas, Nancy and their yet unborn girl. Looking forward to see baby photos. :) The second article I got sent lately is When Girls Will Be Boys. It is about the transition story of Ray and acceptance problems. Pretty long but definitely worth reading. Again, thanks to the people who offered me the links to the articles, I truly appreciate them—and I hope some of the people reading my blog will too, or at least that it might change their perception and opinion on "such people".

Evan Prodromou: 24 Germinal CCXVI

I'm pretty impressed with the new Geotag Icon Project. The well-known feed icon has become ubiquitous, and it makes sense to have a free-to-use icon for geographically-tagged information. I can see it being very useful for hcard and geo microformats, for example. tags:

Relayed for your review without comment For all those who think Open Content guidebooks like Wikitravel are by definition less reliable than proprietary ones: "Travel writer tells newspaper he plagiarized, dealt drugs". "A Lonely Planet author says he plagiarized or made up portions of the popular travel guidebooks and dealt drugs to supplement poor pay. [...] He didn't travel to Colombia to write the guidebook on the country because 'they didn't pay me enough. I wrote the book in San Francisco. I got the information from a chick I was dating.'" Cough, cough. tags:

Steve Kemp: Adopt a less marital tone.

If you upload a new package to the Debian archive which contains a setuid or setgid binary please please ask for a security audit, or carry out one yourself. I certainly accept that the security audit project webpages are not terribly current, and the mailing list is essentially dead, but there are people, such as myself, who would gladly look at your package. All you have to do is ask. When I see two packages in testing with trivialy obvious security bugs it just makes me wonder why we bother. I'm going to take this chance to restate my hardline position on package maintainence - even though it might not be directly applicable - If you cannot program/debug/handle the language a package is developed in you shouldn't maintain it. Too often I've seen signs of this; somebody maintaining a C-based program but unable to program in C. Why? I wonder if we could have a policy / guideline that any new setuid/setgid application must have at least two maintainers, or a documented audit prior to acceptance? Hard to manage but I think it would be useful even if it didn't catch everything. Some bugs such as #475747 (lovely number!) are trivial to discover. ObQuote: Dangerous Liaisons

Lior Kaplan: A begining of an Israeli openoffice.org community ?


While issue #86811 is going to the right direction, I decided to try and use the momentum around it to unite people into an Israeli openoffice.org community. The first thing was to start and do a little triage of RTL/CTL related bugs. A few were confirmed as existing in 2.4.0 and should be moved from UNCONFIRMED to NEW, and get re-target. I also published a list of other bugs that need confirmation and links to the complete RTL/CTL lists so other people could continue my work. Maybe creating a meta issue to and get blocked by all other bugs will simplify the process ? I don’t know if that’s acceptable in the openoffice.org issuezilla. Although I think that is the case in Mozilla’s bugzilla. The confirmed bugs already got some more votes, which is a good beginning, but I’d like people to do more. Next items on my todo list are documentation, and translation of version 3.0. The documentation is mostly thickening the existing Hebrew wikibook about openoffice. I’m not quite sure were to start with translation of version 3.0, although I saw quite a few translations for other languages.

Russell Coker: A Bio-fuel Petrol Station

Today I happened to see a bio-fuel petrol station! I decided to check it out (even though I was traveling by tram in a part of town that I don’t normally visit so there was no possibility of any real purchase). The station is Conservo [1]. Their main products are E10 petrol (10% Ethanol and 90% Petrol), B20 bio-Diesel (20% bio-Diesel and 80% Petroleum based fuel oil), and B100 (100% bio-Diesel). All the fuel that they sell has a biological based component. The prices for the fuel seemed a little lower than is charged by other petrol stations, but it’s difficult to tell as fuel prices can change rapidly. I spoke to one customer who had just filled up his 4WD with bio-Diesel about his experiences. He said that he sometimes used B20 and sometimes B100. He had found no down-side to using such fuels but had noticed that when under hard acceleration the bio-Diesel fuel seemed to cause less dark smoke (IE less soot). All the fuels that they sell are produced in Australia. There are issues with imported bio-fuels which are sometimes produced with slash and burn agriculture and often increase the prices for essential food items (such as corn in South America). As the fuel is produced in Australia such issues should not apply. According to a brochure they have facilities to allow people to deposit used vegetable oil which can then be converted to bio-Diesel. Inside the store they sell a variety of organic foods and drinks, I bought a bottle of carbonated organic apple juice which was quite nice and at $3 was not outside the price range that I expect from a petrol station (which do tend to charge high rates for refreshments). It was not an unreasonably high price for an organic drink. In the store they sell and promote a range of producthttp://en.wikipedia.org/wiki/Flash_points that are positive for the environment. They have a display about using algae to produce bio-Diesel fuel which has some interesting information but unfortunately didn’t list the productivity of algae fields in terms of the number of tons per hectare per year (or month or other time period). One really interesting point I read on their web site concerns the flash point [2] of fuel. The flash-point is the temperature which allows enough of the flammable substance to evaporate to produce an ignitable mixture. Petrol is listed as less than -40C, fossil-Diesel fuel is greater than 62C according to Wikipedia (greater than 55C according to Conservo) while Conservo list the flash point of bio-Diesel as greater than 110C. Wikipedia lists the flash point of canola (rape seed) oil as 327C. This is described as being a benefit of bio-Diesel. While it’s obvious that this is a disadvantage for Petrol, I find it difficult to imagine a situation where a fuel tank could reach a temperature greater than 55C but less than 110C. In their Good for the Environment [3] page they claim that the exhaust from burning bio-Diesel is less harmful to human health than that from burning fossil fuels. My previous post about Vegie Cars [4] is getting some comments suggesting otherwise. So far I haven’t found good references either way, but the discussion has raised some really good issues. Update: Petrol’s flash-point is less than -40C not +40C.

11 April 2008

MJ Ray: Debian Project Leader Turnouts

After starting to write a comment on Buxy r ve tout haut: DPL election: low participation speculating about whether the slow turnout in the DPL voting is caused by the lack of contraversial candidates or the stupid retiming that will make DPL elections clash with Easter holidays more often, I did a quick grep of the past votes. 2008: no contraversial candidates, Easter during voting, turnout high 30s%? 2007: contraversial(s), Easter after voting, turnout 47% 2006: contraversial(s), Easter after voting, turnout 43% 2005: contraversial(s), Easter during voting, turnout 52% 2004: contraversial(s), Easter during voting, turnout 53% 2003: contraversial(s), Easter after voting, turnout 59% 2002: contraversial(s), Easter during voting, turnout 51% 2001: contraversial(s), Easter after voting, turnout unsure (between 33% and 90%!) 2000: dunno about contraversials, Easter after voting, turnout 62% 1999: dunno about contraversials, Easter after voting, turnout 59% Not sure what to make of that. What did I miss?

10 April 2008

Steve Kemp: I burn, I pine, I perish.

So here's the hard part, if you start writing about people and relationships you have to continue, otherwise people who do follow will make assumptions, or not be aware of any potential gaffes they are about to make. I've written in the past about some stuff which I now wish I hadn't because this post is suprisngly hard to compose.
I'm now single, and living alone.
No fiance, no partner, and no kitten. The next time I write about friends I'll make sure it is for local people only. Unless I get carried away, in which case I should be stopped. ObQuote: The Taming of the shrew; via 10 Things I hate about you.

Bdale Garbee: Paperless no more!

Quite some time back, I wrote about my disappointment that Ampad had ceased manufacturing their Quad Ruled Steno Books, which I use to make notes to myself on just about everything. Several of my friends in Europe responded that similar pads in A5 size were commonly available in retail stores, and I quite happily bought a few on a trip to Germany. Then, one of my colleagues at work had some nice notebooks made up for a customer event with grid paper and suitable logos on the cover that I grabbed a few of and have used quite happily. Sometime in the intervening years, Ampad started producing these pads again, but with a different product number, 42036. I stumbled over some at a local office supply store and bought a big pile... others have noticed too, and sent me email about it, which is why I'm taking the time to write this... Yes, I'm much happier now!

Francois Marier: Two-tier encryption strategy: Archiving your files inside an encrypted loopback partition

Even with a fully encrypted system (root and swap partitions), your data is still vulnerable while your computer is on. That's why Bruce Schneier recommends a two-tier encryption strategy.

The idea is that infrequently used files are moved to a separate partition, encrypted with a different key. That way, the bulk of your data files is protected even if your laptop is hijacked or if an intruder manages to steal some files while your main partition is decrypted.

On Debian and Ubuntu, a secure archive area can be created easily using an encrypted loopback partition and the cryptmount package.

Add this to /etc/cryptmount/cmtab:
archives
dev=/home/username/.archives
dir=/home/username/archives
fstype=ext3 fsoptions=defaults cipher=aes

keyfile=/home/username/.archives.key
keyhash=sha1 keycipher=des3
Create the key and the 512 MB loopback partition:
sudo cryptmount --generate-key 32 archives
dd if=/dev/zero of=.archives bs=1M count=512
mkdir archives
sudo cryptmount --prepare archives
sudo mkfs.ext3 -m 0 /dev/mapper/archives
sudo cryptmount --release archives
Fix the permissions so that you can write to this partition with your normal user account:
cryptmount archives
cd archives
sudo chown username:username .
cryptmount -u archives
Then you can mount and umount that partition using:
cryptmount archives
and
cryptmount -u archives

Ian Wienand: Tips and tricks for your E3 Visa

Having recently been through the process, I've managed to come out the other-side with some some advice for anyone taking this on.
  • If anything really stuffs up you'll have to enter the US immigration services, which is no fun. If this happens to you in the Bay Area go straight to the San Francisco office; the San Jose office doesn't have deferred inspections so you'll probably just have to go there anyway.
  • Your dependent on the E3D Visa will need to fill out an I-765 work authorisation form. This will take up to 3 months to process; you can find out the approximate queue time here. The form doesn't appear to have been updated for E3D; I would strongly recommend getting a lawyer to file it for you so you don't stuff it up. Budget about $400. After this has been processed, the dependent will then need to file for a SSN before they can be paid.
  • Tax is a black-art. In California expect to pay something like 20-25% federal, 10% state and up to 7% more for Medicare and social security. Expect another $80/month for health insurance.
  • You can get a bank account without a SSN easily. A good way to transfer money is OzForex, but watch out for fees on the US side (you may have to do wire-transfers which cost about $10 a go, though some banks have bill-pay). The country still runs on cheques for some reason.
  • You can rent without a SSN, but it will cost you. Budget for 2 months extra rent as deposits and paying ahead.
  • A car lease is probably a good way to go. Finance will usually look at your visa and only give you a two-year loan, and with no credit that will be at a very high interest rate. In California you can get a 2 year lease on a base model new car (sticker price ~$17,000) drive-away for a down-payment of about $2500 and then $250/month plus taxes. Insurance will be about $120/month on-top of that. The beauty is you just give it back when you go home. For long holiday trips SUV rental may contribute to global warming, but is a very comfortable way to see the country.
  • In California you'll need to book in for a driving test to get a license; you can't do this in a rental car but some local driving schools offer a short introduction course and then let you use their car for the test.
  • If you're living in temporary accommodation without great facilities Dream Dinners is a good way to easily get some meals together.
  • In the Bay Area, Yelp is an awesome resource.
  • Comcast make Telstra look like a well-run, efficient and customer focused organisation!
  • Feel free to drop me a line if you're thinking of taking an opportunity and have any questions.

    9 April 2008

    Christian Perrier: News from the lenny l10n NMU campaign

    The lenny l10n NMU campaign is now running for about two months now. The objectives of this campaign are tackling pending po-debconf l10n bugs. The process is the following: Of course, when the maintainer wants to upload him herself, I immediately stop any action....but I still watch what happens and I nag her him in case nothing happens. In short, you can escape from my trap once I got you! Status as of now: A few utterly outdated and obsolete packages have also been removed fro mthe archive. All this helped the statistics for many languages to raise up. However, thanks to the effort of many maintainers who invest a lot of energy in emptying down the bottle we're filling up, the top languages statistics don't really move (new debconf templates, modified ones, etc.) With the lenny release becoming closer, I expect package maintainers to calm down now and leave us the opportunity to completee our l10n work without moving the target away when we reach it. The closer the release, the more aggressive l10n NMUs will be. Even for recent bug reports. Those translation issues are the easiest ones to fix, so please fix them. Please also note that I'll very happily sponsor any non DD upload which just fixes l10n issues.

    Christian Perrier: Screwing samba packages

    A few days ago, I just screwed the samba packages maintenance. It's been quite some time since we were preparing Samba 3.2.0 packages. Upstream has not published a release version, only pre-releases. The release was planned for next week but will not happen as some last minutes issues have been found in pre-releases. Anyway, we had ready packages for 3.2.0 pre-release 2....which were planed to go to experimental. However, these packages were under preparation for quite a while, so when I finally agreed with Steve Langasek that it's OK to upload, I built and uploaded the packages. Then got a good laugh when discovering that I forgot "UNRELEASED" as target in debian.changelog. It was late, I then "s/UNRELEASED/unstable" as I often do in such case....then uploaded...then forgot about the issue and went to something else. The package went through NEW (it has a new binary package) and 1 or 2 days later quietly entered unstable. ...which I discovered only the next day, when it was way too late to remove from unstable something that was meant for experimental. All this explains why samba packages now use an epoch...:-) Things are back to normal now. Unstable has 1:3.0.28a-2 and experimental will hopefully have 3.2.0~pre2 soon. Next week I go to the Samba annual conference. I'll probably spend some time hunting down some bugs in the package and discuss with upstream about some of the issues we have (old bugs, FHS patch, upstream's .deb packages). And maybe also get better ideas about Samba 3.2.x schedule, to be able to decide with Steve and Noel which release is the target for lenny: 3.0.28* or 3.2.x.

    Neil Williams: emdebian-tools 0.9.4 uploaded to Emdebian

    Where, oh, where do I start with this one??
    $ parsechangelog   wc
    97 644 4793

    97 lines, 644 words just for the changes in this version and some quite important and far-reaching. I have to try and summarise somewhere, so:
    1. dpkg-dev (>1.14.17) support

    2. New debconf question to identify the targetsuite

    3. Drop-in lintian support for cross-built Emdebian packages

    4. emchain support for a uClibc toolchain built on a glibc/gcc toolchain

    5. Require a genuine Debian environment for building

    Some detail from the changelog
    emdebian-tools (0.9.4) unstable; urgency=low

    For the full list, see the changelog and emdebian manpages.

    Anthony Towns: Select and Python Generators

    One of the loveliest things about Unix is the select() function (or its replacement, poll()), and the way it lets a single thread handle a host of concurrent tasks efficiently by just using file descriptors as work queues. Unfortunately, it can be a nuisance to use – you end up having to structure your program as a state machine around the select() invocation, rather than the actual procedure you want to have happen. You can avoid that by not using select() and instead just having a separate thread/process for every task you want to do – but that creates a bunch of tedious overhead for the OS (and admin) to worry about. But magically making state machines is what Python’s generators are all about; so for my little pet project that involves forking a bunch of subprocesses to do the interesting computational work my python program wants done, I thought I’d see if I could use that to make my code more obvious. What I want to achieve is to have a bunch of subprocesses accepting some setup data, then a bunch of two byte ids, terminated by two bytes of 0xFF, and for each of the two byte inputs to output a line of text giving the calculation result. For the time being at least, I want the IO to be asynchronous: so I’ll give it as many inputs as I can, rather than waiting for the result before sending the next input. So basically, I want to write something like:
    
    def send_inputs(f, s, n):
    	f.write(s) # write setup data
    	for i in xrange(n):
    		f.write(struct.pack("!H", i))
    	f.write(struct.pack("!H", 0xFFFF))
    def read_output(f):
    	for line in f:
    		if is_interesting(line):
    			print line
    
    Except of course, that doesn’t work directly because writing some data or reading a line can block, and when it does, I want it to be doing something else (reading instead of writing or vice-versa, or paying attention to another process). Generators are the way to do that in Python, with the “yield” keyword passing control flow and some information back somewhere else, so adopting the theory that: (a) I’ll only resume from a “yield” when it’s okay to write some more data, (b) if I “yield None” there’s probably no point coming back to me unless you’ve got some more data for me to read, and (c) I’ll provide a single parameter which is an iterator that will give me input when it’s available and None when it’s not, I can code the above as:
    
    def send_inputs(_):
    	# s, n declared in enclosing scope
    	yield s
    	for i in xrange(n):
    		yield struct.pack("!H", i))
    	yield struct.pack("!H", 0xFFFF)
    def read_output(f):
    	for line in f:
    		if line is None: yield None; continue
    		if is_interesting(line):
    			print line
    
    There’s a few complications there. For one, I could be yielding more data than can actually be written, so I might want to buffer there to avoid blocking. (I haven’t bothered; just as I haven’t worried about “print” possibly blocking) Likewise, I might only receive part of a line, or I might receive more than one line at once, and afaics a buffer there is unavoidable. If I were doing fixed size reads (instead of line at a time), that might be different. So far, the above seems pretty pleasant to me – those functions describe what I want to have happen in a nice procedural manner (almost as if they had a thread all to themselves) with the only extra bit the “None, None, continue” line, which I’m willing to accept in order not to use threads. Making that actually function does need a little grunging around, but happily we can hide that away in a module – so my API looks like:
    
    p = subprocess.Popen(["./helper"], stdin=PIPE, stdout=PIPE, close_fds=True)
    comm = communicate.Communication()
    comm.add(send_inputs, p.stdin, None)
    comm.add(read_output, None, p.stdout, communicate.ByLine())
    comm.communicate()
    
    The comm.add() function takes a generator function, an output fd (ie, the subprocess’s stdin), an input fd (the subprocess’s output), and an (optional) iterator. The generator gets created when communication starts, with the iterator passed as the argument. The iterator needs to have an “add” function (which gets given the bytes received), a “waiting” function, which returns True or False depending on whether it can provide any more input for the generator, and a “finish” function that gets called once EOF is hit on the input. (Actually, it doesn’t strictly need to be an iterator, though it’s convenient for the generator if it is) The generator functions once “executed” return an object with a next() method that’ll run the function you defined until the next “yield” (in which case next() will return the value yielded), or a “return” is hit (in which case the StopIteration exception is raised). So what we then want to do to have this all work then, is this: (a) do a select() on all the files we’ve been given; (b) for the ones we can read from, read them and add() to the corresponding iterators; (c) for the generators that don’t have an output file, or whose output file we can write to, invoke next() until either: they raise StopIteration, they yield a value for us to output, or they yield None and their iterator reports that it’s waiting. Add in some code to ensure that reads from the file descriptors don’t block, and you get:
    
    def communicate(self):
        readable, writable = [], []
        for g,o,i,iter in self.coroutines:
            if i is not None:
                fcntl.fcntl(i, fcntl.F_SETFL, 
                            fcntl.fcntl(i, fcntl.F_GETFL)   os.O_NONBLOCK)
                readable.append(i)
            if o is not None:
                writable.append(o)
        
        while readable != [] or writable != []:
            read, write, exc = select.select(readable, writable, [])
            for g,o,i,iter in self.coroutines:
                if i in read:
                    x = i.read()
                    if x == "": # eof
                        iter.finish()
                        readable.remove(i)
                    else:
                        iter.add(x)
                if o is None or o in write:
                    x = None
                    try:
                        while x is None and not iter.waiting():
                            x = g.next()
                        if x is not None:
                            o.write(x)
                    except StopIteration:
                        if o is not None:
                            writable.remove(o)
        return
    
    You can break it by: (a) yielding more than you can write without blocking (it’ll block rather than buffer, and you might get a deadlock), (b) yielding a value from a generator that doesn’t have a file associated with it (None.write(x) won’t work), (c) having generators that don’t actually yield, and (d) probably some other ways. And it would’ve been nice if I could have somehow moved the “yield None” into the iterator so that it was implicit in the “for line in f”, rather than explicit. But even so, I quite like it.

    Dann Frazier: etchnhalf kernels available for testing

    2.6.24 kernels targeted for etch are available for testing in etch-proposed-updates. Some architectures are missing for this first upload, but fixes are pending for the next upload.

    8 April 2008

    Gunnar Wolf: Some Rails plugins: Real FK

    I have mostly shifted my main development language to Ruby, as most of the work I do tends to be pretty well covered with Rails. That's not new, however - What is new is that I feel I have got to the point where I actually have something to share. And it feels very good.
    So my next couple of posts (three, actually) will be covering some plugins I've written. They are, of course, just sketched - but they work reliably enough to be deemed as useful by their author ;-)
    Today I will start by blogging about RealFK. Of course, I will be copying from its documentation/description - I wrote it as well, so I'm not really plagiarizing my own site. I'm not suing self. You can of course also browse the RealFK SVN tree. References Rails is built upon a "dumb database" concept. Rails assumes the database will not force restrictions upon the user - i.e. that the model validators are all there is. But I disagree ;-) I was raised with PostgreSQL, not with MySQL, and I do believe the database can and should impose sane values - It should impose trust on the data..
    So, if you want to create a relation field, properly acting as a foreign key and refering to another table, instead of doing it via the ActiveRecord::ConnectionAdapters::TableDefinition#column (or even ActiveRecord::ConnectionAdapters::TableDefinition#references) methods, declare them with add_refrence:
      def self.up
        create_table :proposals do  t 
          (...)
        end
        create_table :proposal_types do  t 
          (...)
        end
        create_table :proposal_statuses do  t 
          (...)
        end
        add_reference :proposals, :proposal_types
        add_reference :proposals, :proposal_statuses, :null => false, :default => 1
      end
    
    The corresponding fields (proposal_type_id and proposal_status_id, respectively) will be created in proposals, and the RDBMS will impose a Foreign Key constraint. The references can be removed via remove_reference, i.e., for inclusion in down migrations:
      def self.down
        remove_reference :proposals, :proposal_types
        remove_reference :proposals, :proposal_statuses
        drop_table :proposal_statuses
        drop_table :proposal_types
        drop_table :proposals
      end
    
    Of course, in this case the remove_reference call is not really needed - But if you are dropping a table that is referred to from a table that will remain existing, you will have to remove the foreign key constraints (that is, the referring columns). Join (HABTM) tables When you define a simple has_and_belongs_to_many (HABTM) relation in your model, you are expected to create an extra table representing this relation. This is a very simple table (i.e. it has only a row for each of the related table‘s IDs), and it carries foreign key constraints.
    Please note that join tables are not supposed to carry any extra columns - If you need to add information to join tables, think twice and better use a has_many :elements, :through => :othermodel declaration, and create the othermodel table manually. To create a HABTM table representing that each person can attend many conferences and each conference can be attended by many people:
      def self.up
        create_table :people do  t 
          (...)
        end
        create_table :conferences do  t 
          (...)
        end
        create_habtm :people, :conferences
      end
      def self.down
        drop_habtm :people, :conferences
        drop_table :people
        drop_table :conferences
      end
    
    The last call will create a conferences_people table (according to Rails' conventions, table names are sorted alphabetically to get the join table‘s name).
    Note that in this case, in the down migration, the drop_habtm call must appear before the drop_table calls, as it carries foreign key constraints.

    Next.

    Previous.