A new version of
PerlConsole is available. As you can see in the
changelog, lots of goodness with this release comes to you (~/.perlconsolerc rules, thanks to S. Zachariah Sprackett for his patch).
After reading the source code of Devel::REPL and learning from it (this is really a great library, congrats to
mst) I decided not to orphan Perl Console. Yes, that was a possibility as Devel::REPL provides anything you need to build a console (and comes with “re.pl” which actually is a console itself). I decided not to give it up for one reason: Devel::REPL is great, and better written than Perl Console (it supports plugins and profiles and is Moose-based) but, it’s slow.
When you launch it, you have to wait a couple of seconds before the console prompts you. This is because of Moose I suppose. I know that’s not blocker as a console is a development tool, but I feel like some of PerlConsole’s users would like to have the choice between a full featured, extensible and Moose-base console that’s slow, and a light one that is basic in design and implementation.
I backported one of the most interesting plugin of Devel::REPL to PerlConsole, being the console’s lexical environement.
Devel::REPL did a great thing there, using Lexical::Persistence to simulate a namespace in the console.
Feel free to give your feedback about this, I wonder if there’s still an audience for PerlConsole when Devel::REPL is in the town. I think so, maybe I’m wrong, then tell me.
After (again) searching for some good photo album generator, I'm more
and more inclined to writing yet another tool to do this. Maybe I just
want too much from this, but I haven't found solution that would fit my
needs:
- Generate static HTML (or XHTML).
- Design should be done using CSS.
- Ability to process whole tree of photos into single album with
directory based sub albums.
- Generate data only for changed/new photos (timestamp checking).
- Support for including EXIF information in results.
- Clean and nicely looking default template.
- Support for image comments from EXIF, JPEG comment field and external
file.
- Automatic generating of images suitable for web browsing as well as
including links to full sized images from camera.
- Somehow active development. I fear of using code which has not been
updated for 3 years.
- It should be fast. This excludes anything written in bash as spawning
tons of external programs can not be fast.
- Written in some language I know. I'm pretty sure I will hit some bug
and I would like to be able to fix it. In fact only suitable language
for me seems to be Python.
- Command line interface is a must.
So far I'm using a bit customised
Matew which really does not
fulfil everything, but at least works reasonably good (but extremely
slow). However the more and more pictures are in my album, the slower
this beast seems to be, so it's probably time to switch to something
new. So far I haven't found anything that would actually fit, but
PyAC seems to be usable as starting point for my new project for
generating albums. Or is there a working solution?
A few days ago,
Joe Reagle was telling me about the rise of (sometimes very arcane) policies and bureaucratic imperatives that now characterize Wikipedia. A few days later I stumbled across a few interesting
posts on the topic, posts (especially Nick Carr’s) seeping with almost celebratory gloom and doom:
‘But, given human nature, is it really so “incredible” that Wikipedia has evolved as it has? Although writers like Yochai Benkler have presented Wikipedia as an example of how widescale, volunteer-based “social production” on the Internet can exist outside hierarchical management structures, the reality is very different. As Wikipedia has grown, it has developed a bureaucracy that is remarkable not only for the intricacies of its hierarchy but for the breadth and complexity of its rules. The reason Deletionism has triumphed so decisively over Inclusionism is pretty simple: It’s because Deletionism provides a path toward ever more elaborate schemes of rule-making - with no end - and that’s the path that people prefer, at least when they become members of a large group. The development of Wikipedia’s organization provides a benign case study in the political malignancy of crowds.”
It is without question that a problem has arisen in Wikipedia, a problem composed of a thick web and net of rules that can be helpful as guidelines but often are often confusing and clearly work to ensnarl new users.
But can we so quickly put blame on so-called human nature? Or is it not a problem of human organization, which as the anthropological and sociological record shows, can take many forms? And is it not just that: a problem begging for a solution instead of an opportunity to declare the fundamental nature of wikipedia (and that of human nature?). Not only may history prove him wrong, other large scale collaborative projects at least prove that solutions can be found to deal with problems of growth and scale.
It is as if Carr just wants to see a project like Wikipedia fail, which somehow, this morning hit a raw nerve. What I find exciting about large scale projects of (at times unwieldy) collaboration are not just the explicit outputs of the projects (an encyclopedia or operating system) but the social worlds they create. And there is no inevitable path they *must* follow. These groups have a choice to react to and respond to these sort of problems and enact solutions that will hopefully solve them and allow these projects to change.
Debian, a slightly older project than Wikipedia, has gone through many growing pains and there was even a period when the the whole process of integrating new Debian developers was shut down and if my memory serves me correctly, for 2 years! At the time, it could have been possible to say: “This signals the end of Debian” but eventually a solution was found, the New Maintainer process, which while not perfect (what is?) allowed the project to grow and produce a great operating system for years to come.
Debian today faces new problems and is working to find solutions. I hope that Wikipedia can and will do the same. And instead of declaring its death, why not wait and see, and offer something a little more constructive and illuminating, than destructive?
When I was about 11 years old I decided that I wanted a career related to computers. My first computer was the
TEC-1 single-board Z80 based kit computer from Talking Electronics magazine (see the photo below). I think that I built this when I was 10.
The computer is 16cm high and 25cm wide. The six
seven segment displays are the only built-in output device (there were optional kits for other output devices). The keypad has the hexadecimal number keys, an “ad” button for entering addresses, a “go” button for executing programs, and “+” and “-” keys for incrementing and decrementing the address. Below the reset button (labelled “R“) you will see the optional function key (of which I can’t remember the purpose). Programming this computer required entering the hexadecimal code on the keypad with the “+” and “-” keys being the main method of editing (the “ad” key was used to jump to a different section of RAM). In editing mode the first four seven-segment displays showed the address (the Z80 could only address 64K of RAM) and the other two showed the memory contents (the word size was one byte). In terms of user-friendlyness it was probably about equal to punched cards - apart from the lack of non-volatile storage (unless you built the optional NVRAM kit).
My TEC-1 has 2K of RAM (the 83251R chip is equivalent to an Intel 16kilo-bit 6116 static RAM chip) and 2K of ROM (the chip with the orange sticker labeled Mon1 is a 2716
EPROM - 16kilo-bit).
Not long after that my parents bought the first serious computer for the family, a
Microbee Z80 based system with a tape drive that used a monochrome monitor of resolution approximately equal to
CGA and which had either 16K or 32K of RAM (I can’t recall). The next family computer was a Microbee Premium series 128K which is probably the same model as the one depicted on the
Microbee Wikipedia page (a serious omission of the Wikipedia page is that it has no picture of the box containing the PSU and the floppy drives for the Premium Series). My first published article in a computer magazine was when I was about 15 years old and I wrote a long email on a
Fidonet echo (mailing list) reviewing a 3rd party update to the
CP/M system for the Premium Series Microbee and was surprised by having it published in the Microbee club magazine (in those days we didn’t bother much about copyright so no-one asked for my permission before publishing).
I wonder if starting with computers at such an age is typical for people who now contribute to free software development. I think it would be interesting to see some blog posts from other people in the community about how old they were when they started with computers and what type of computer they started with.
I also wonder about the correlation between the age of starting with computers and career success in the computer industry. One significant benefit of starting early was that I could learn things that would be useful for my career in later decades while other children were wasting time studying what teachers told them to study. It also meant that in later years of high-school I could relax knowing that I could get straight B’s without effort which was more than was required to enter a CS degree program at that time. Until half-way through year 12 I tried to avoid ever doing home-work at home - home-time was computer time! Do you think that the age at which you chose your career significantly affected your success? If so in what way?
If you were asked for advice by parents as to when their child should be given it’s first computer what age would you suggest? Unfortunately I usually get asked for advice about such things by people who have children aged 16+ (which is way too late IMHO).
Share This
No, I am not confusing version numbers. Yes, xserver-xorg-video-ati 6.7.191 follows 6.6.193. Upstream thought it was useless to try fixing the dead-end 6.7 branch (where 6.6.193 came from) since they were already mostly working on the randr-1.2 branch. So 6.7 will probably never happen. The randr-1.2 support has been merged in master and will be released as driver 6.8. 6.7.191 is its first release candidate.
With RandR-1.2 support, you get mode auto-detection and so, as in the Intel driver 2.x. So you may clean your xorg.conf as described
here. If you don't get your desired resolution at startup, see
here. However, the old dual-head support code (zaphod) is dead, so some people might have to get used to xrandr instead.
Several people will also be happy that the GATOS TV-output support code has been merged and improved to support more cards. You will see a S-video output and you can enable it as usual with xrandr. Not sure how well tested it has been, but at least it seems to work fine on my rv370 (Mobility X300).
Don't forget that this is a release candidate. I personally consider the code as highly experimental since I seem to get more GPU lockups when playing xmoto than in 6.6.193. The code is still under heavy development. Multiple ATI boards have strange behaviors that may not be supported yet (but it shouldn't be worse than in 6.6.193). Fortunately, upstream people are very responsive and doing a great job at fixing bugs, so feel free to ping them on IRC or report a bug directly on http://bugzilla.freedesktop.org.
The package is in now in debian experimental. But the Xserver release candidate is there too. So you will need to upgrade xserver-xorg-core too. If you don't want to do so, just rebuild the driver against unstable, you should just need to downgrade xserver-xorg-dev build-depend to 1.3.0 or so.
A good read eval print loop for perl has been on my Wanted list for a long time. Matt S Trout’s
Devel::REPL has been looking promissing but I didn’t want to fiddle with installing the dependencies. Thanks to Alexis Sukrieh and the Debian Perl Group
Devel::REPL will enter unstable. Yeahhh…
Another thing on my wanted list was Debian packages of Perl::Critic. Couldn’t find a ITP, but looking around at the Debian Perl Group’s website to find out how to package it I discovered that Joey Hess allready did the work.
I’ve been using the morning installing the above from the subversion repository on pkg-perl.alioth.debian.org. Only problem was that I needed a newer package of libppi-perl which wasn’t maintained in subversion.
What’s next? Looking at my ~/.perl Perl::Tidy seems to be a candidate.
It’s interesting to see that
Audi is releasing a car with LEDs for all lights including the headlights. This is being promoted as an environmental benefit, however a quick google search revealed that my Volkswagen
Passat apparently takes 55W headlights (giving a total of 110W of electricity used). Even allowing for some inefficiency in the alternator this would make a very small impact on the fuel use of a engine rated at 140KW. The Audi in question is the
R8 (wikipedia link because the Audi web site is badly broken) and has a 300KW engine…
A simple implementation of LED headlights will do some good for plug-in hybrid cars and all-electric vehicles where saving power is more important - when the technology filters down to cheaper vehicles. Also one possible use for the technology is to dim the headlights by turning off some of the LEDs in the bank (according to the
LED Wikipedia page it is currently impossible to create a single LED that takes more than 1W of power, so a bank of LEDs would be used). Currently you have a choice of using “parking lights” or “head-lights” when driving, and when driving just before sun-set or at night in the city (where the street lights are bright) you need head-lights to allow other drivers to clearly see you but don’t need them as bright as they have to be when driving at night in the country. So a range of levels of luminosity could be effectively used in headlights to increase efficiency in some situations and increase light levels in others.
According to the
Luminous efficiency Wikipedia page current LEDs are up to three times as efficient as quartz halogen incandescent globes and future developments are likely to increase that to six times the efficiency. Combine that with more effective use of headlights to provide the light at the location and level that’s needed and the result could be using at little as 10% of the electricity for headlights on average!
Another thing that I would like to see is the Adaptive Headlights feature of the better BMWs (which I referenced in a previous post about the
BM 5 and 7 series) implemented in a cheaper and more reliable manner. The feature in question is that the headlights will turn when driving around a corner to show the road ahead instead of just shining off the edge of the corner. Implementing such a feature with incandescent lights is difficult because they have to be physically turned and moving parts tend to break (which increases maintenance costs and decreases the overall reliability of the vehicle). An obvious alternate design is to have a set of LEDs pointing in different directions and which LEDs get power would determine where the light goes (this would also react faster than physically moving a light). Once LED headlights become common the Adaptive Headlights feature could be implemented in the cheapest cars on the road with minimal extra cost - currently it’s a feature that would be expensive to implement and would increase the sale price of a small car and probably the service price too.
Share This
It's been a while since I've posted here; I've had a busy two weeks, and posting on my blog hasn't been a priority. Something I should do more often. Of course, as time backs up it seems harder and harder to actually get around to posting.
I wanted to bring the attention of my friends and colleagues, though, to a notice that
Maj put on the
Wikitravel shared logbook. As of 15 August 2007,
Maj is no longer working on Wikitravel. In addition, my time has been curtailed quite a bit, and I'm concentrating only on community and content issues -- no more programming or sysadmin work.
This is all coming about for several reasons. The main one is that
Internet Brands, the company that now owns wikitravel.org, wanted to have their own staff do the programming and sysadmin work. The end of our service contract seemed like an ideal time to effect this change.
On our side, it's been challenging for us to work with a large organization not specialized in
Open Content and wiki sites. We've gotten along really well with the IB people, but we haven't always seen eye-to-eye on the best way to grow and develop the site. As the decision was made to remove us from our technical roles, Maj decided it was a good time to opt out entirely.
I'm going to miss her tremendously. As most people reading this already know, we started the Wikitravel project together, and over the last 4 years it's been a constant important part of our family life. Maj's information-architecture skills and dedication to travel have been crucial to the development of the site. She's been a strategizer, a spokesperson, a content organizer and a content builder. Probably most important for the future of the project, she developed almost single-handedly the
Wikitravel Extra personal experience component of the site.
We've got a lot to keep us busy, though. We recently launched
Keiki, a project to create an Open Content parenting guide. For myself, I've also started
Vinismo, an Open Content wine guide (sensing a pattern here...?). And we'll be working with our partner Jani P. on
Wikitravel Press, an independent project to publish printed travel guidebooks based on Wikitravel articles.
Despite the shakeups, I anticipate that the future of Wikitravel will be good. We have an incredibly strong and smart community, dedicated to an important and noble purpose. I'll remain involved on a part-time basis. We've got a lot of traction in public opinion and great Web traffic. I don't think that IB is going to make any bad decisions about the site, but if they do there's a local and global community of people that take Wikitravel and Open Content travel guides extremely seriously. I think things are probably going to be OK.
tags:
wiki wikitravel opencontent maj work ib
On the subject of wiki
The South by Southwest 2008 Panel Picker app is up for public review right now. SXSW Interactive, in Austin, is one of the best tech conferences of the year. I've got a panel proposal up, on Wiki 2.0 -- I really want to talk about some of the interesting new wiki apps, new wikis, and new wiki business models coming up.
There are a couple of other wiki-related panels proposed -- including one on commercial wikis, which I presented about last year (Talks/SXSW07). Folks who are interested in wikis, or in SXSW, should take some time to give their feedback on the proposed panels and improved the level of discussion for 2008.
tags: sxsw wiki vote panel sxsw08
A question that is often asked is whether to use SE Linux or a chroot to restrict a program.
In Unix chroot is a way of running a program with a restricted set of directories available (it used to be merely a sub-tree but with bind mounts it can be any arbitrary set of directory trees). A chroot can be implemented in a daemon (it can call the chroot(2) system call before it drops it’s privileges) or by a shell script (through the chroot(8) utility). The disadvantages of a chroot are that root can escape from it, a chroot process can see the existence of non-chroot processes (ps and similar programs work in the same way in all chroot environments), and inter-process communication is not prevented. One solution to this is to have an enhanced chroot environment (which typically requires a kernel patch) where the chrooted processes can not run ps without restriction and have other limits applied to what they are permitted to do (there are several kernel patches that implement such restrictions). In the early days of SE Linux development I implemented similar functionality in SE Linux policy (
here is the paper I presented at Linux Kongress 2002).
Configuring a chroot environment is inconvenient. If it is configured in the traditional manner (copying files to the chroot instead of bind mounting the directories) then old versions may exist in the chroot after new versions with security fixes have been installed in the main environment.
SE Linux provides better security than a typical chroot environment by controlling all interaction between processes. It provides more flexibility than an enhanced chroot environment by being configured entirely by policy and not requiring a kernel recompile to change the way it works.
I believe that the correct thing to do is to cease using chroot entirely and use SE Linux instead.
Share This
I just had a hard time finding
this excellent procmail resource on the web. I am thus blogging
it for posterity, in case anyone is looking for procmail documentation, tips,
tricks, a how-to, or anything else related to procmail.
And if you procmail and have not read the document, I suggest you do. It's
truly outstanding.
NP:
A Silver Mt. Zion:
He Has Left us Alone, but Shafts of Light Sometimes Grace the Corner of our Rooms
I looked at the Foleo and played with one while I was at Linux World a few weeks ago, but it is so restrictive in what it can do that I was completely unimpressed.
First of all, according to one of the people at the booth, it will only work with a select set of Treo's; mostly the newer ones. A colleague I was with had just gotten a Treo 650, and the person at the booth said that it wouldn't work with that model of Treo. (WTF?) Furthermore, it will only do e-mail (POP or IMAP) by linking with the Treo over bluetooth connection and letting the Treo pull down the e-mail and store it on the Treo. Given that it only has 128 megs of RAM and 256 megs of Flash, it just doesn't have enough storage apparently to run a stand-alone e-mail application, which is a little bit scary. The limited amount of memory is probably why it is using Opera as a web browser, which previous experience on the N800 has largely unimpressed me in terms of compatibility with Web 2.0 sites that aggressively use AJAX or flash.
So OK, it's not supposed to be a laptop. But the problem is, for 0.2 pounds more, I can get a laptop. Let's review the critical statistics, shall we? The Foleo costs $499, weighs 2.5 pounds, and has a size of 10.5" by 6.7" by 0.9". As stated before it has 128 megs of RAM and 256 megs of flash, with a SD slot for expansion purposes, and it has a claimed 5-6 hours of battery life. But let's compare that with my IBM X41 which I recently purchased off of eBay for $800. It has 1.5 gigs of memory, and 60 gigs of hard disk. It has two batteries; with the 4 cell battery it weighs 2.7 pounds and delivers 2 hours of battery life, and with the 8 cell battery it weighs 3.2 pounds and delivers 4 hours of battery life. So true, even with the 8 cell battery the X41 is 1.7 pounds heavier and still has slightly less battery life. But you can do a lot more with the X41! Furthermore, the Foleo's weight advantage is somewhat nullified by the fact that you have to bring the Treo around to do certain activities, and the Treo has to be powered on since the Foleo is mostly designed around being a remote large screen for the Treo.
At the end of the day it's all about tradeoffs. Perhaps if enough companies created enough killer apps that could fit in 128 meg of ram for the Foleo, it might be useful enough to justifying buying it. I hear for example, that even though the Foleo doesn't have any kind of PIM functionality, a 3rd party ISV is planning on making a product available that will provide calendar and contact functionality that can sync Palm PDA's. No word on how much it will cost or how usable it will be, but with enough applications, maybe Foleo could be useful enough to justify its size/weight. I imagine that these apps will probably be commercial ones, since open source apps like Evolution will probably have difficulty fitting in the Foleo's constrained environment. :-)
And, of course, it's a lot cheaper than my used X41 laptop, never mind a brand-new Lenovo X61s, which could run 2 or 3 kilobucks fully outfitted with the 4 gigs of memory and 160 gig/7200 rpm drive. However, as a road warrior, my priorities are not just weight, but functionality. A 2.5 pound solid-state laptop with only 128 megs of memory which massively restricts what I can do is not a good use of the space in my laptop bag. What I'm waiting for is the next generation of the Thinkpad X series which has a solid state disk --- which shouldn't be that far off --- and the elimination of the spinning magnetic media would mean that we should have something with Foleo's battery life without the Foleo's limited usefulness. Sure, maybe I will need to wait another year or two for my perfect laptop (12", 1024x768 LED display, at least 60 gigs solid state disk, at least 2-4 gigs memory, Intel core 2 or follow-on processor, < 3 pounds, > 5 hours useful lifetime) to become available, but the technology to do this exists today; it's just a matter of making it affordable. But given that kind of long-term future, I'm willing to settle for now with either 2 or 4 hour battery life, and a slightly heavier laptop, than to use something today with a desperately slow ARM processor and only 128 megs of memory. The weight/size savings and the increased battery life of the Foleo isn't a fair tradeoff given its very limited capabilities.
This is all really too bad, because if Palm is counting on the Foleo to allow it to succeed, I think the concept has some massive shortcomings, much like their claimed LifeDrive product, which didn't last very long. And I really like the Palm company; I still haven't found better PDA functionality on a decade-old Palm design compared to what is available on all of the Nokia phones I've looked at, the WinCE phones/PDA's, the N800, etc. So I want Palm as a company to stick around. But with Treo getting eclipsed by newer smart phones, and the Foleo not getting particularly good buzz by folks reviewing it --- and after I played with it, I have to agree with the majority of the reviewers that this is not the Next Big Thing --- I'm not sure how much longer Palm is going to be around.
The
next release of
DebTorrent is now available. This
release includes new functionality for communicating with APT using
a new transport method specifically designed for debtorrent, and
many bug fixes.
The major changes in this release are all in the communication
between APT and DebTorrent. The HTTP server in DebTorrent that
listens for APT requests has been upgraded to support HTTP/1.1
persistent connections and pipelining, which allows APT to have
multiple outstanding requests for files. This is useful as
DebTorrent suffers from the typical bittorrent slow start, so
requesting multiple files at a time helps to speed up the download
considerably.
Though better, HTTP/1.1 is not ideal for DebTorrent however, as a
maximum of 10 outstanding requests is maintained by APT's http
method, and files must still be returned in the order they were
requested (which is not ideal for bittorrent-type downloading since
downloads occur randomly).
To further improve the APT communication I have modified APT's http
method to create a debtorrent method. This new debtorrent transport
for APT is packaged separately as apt-transport-debtorrent, and once
installed APT can be told to use it by replacing "http://" with
"debtorrent://" in your sources.list file. This method sends all
requests it receives immediately to DebTorrent, and will receive
responses from DebTorrent in any order. You can find this new method
on the
Alioth project, or in my
personal repository
(only amd64 and i386 versions are currently available).
Unfortunately, the story doesn't end here. The APT code responsible
for sending requests to the method also limits the maximum number of
outstanding requests that it will send to the method to 10, which is
not really necessary since all existing methods limit the requests
they send out themselves. I have therefore patched the current APT
code to increase this limit to 1000 (a one line change), and
released this patched version as 0.7.6-0.1. You can find this
patched version in my
personal repository (again, only for
i386 and amd64). I have tested it with the other methods available
and it causes no problems, and I hope to get the change included in
the regular APT code soon.
To sum up:
- new DebTorrent over HTTP = fast
- new DebTorrent with new apt-transport-debtorrent = faster
- new DebTorrent with new apt-transport-debtorrent and a patched
APT = fastest
The last DebTorrent version (0.1.3.1) is currently in the
NEW
queue, and judging by the length of it, will be there for about
another week. After DebTorrent is added to the archive, I will be
upgrading it to this new version. I also hope to get the new
apt-transport-debtorrent package into the NEW queue soon.
This brings to an end the Google
Summer of Code (which this
project was
created as a part of), but development of
DebTorrent will of course continue (probably a little slower). The
next major change will be the addition of
unique piece
numbers, which is almost complete but needs to be extensively
tested. I'd like to thank Anthony Towns, Steve McIntyre, and Michael
Vogt for their help over the last 4 months, and also the many others
who sent me encouraging emails or engaged in interesting discussions
about this project. It's the people who make a project like this a
fun and memorable thing to do.
Here's the changelog for the new DebTorrent release:
- APT communication supports HTTP/1.1 connections, including
persistent connections and pipelining
- Add support for the new debtorrent APT transport method
(see the new apt-transport-debtorrent package)
- Make the Packages decompression and torrent creation threaded
- Improve the startup initialization of files
- Add init and configuration files for the tracker
- bug fixes:
- restarts would fail when downloaded files have been modified
- deleting old cached data would fail
- small tracker bug causing exceptions
- prevent enabling files before the initialization is complete
- only connect to unique peers from the tracker that are not
already connected
- tracker would return all torrents' peers for every request
The
next release of
DebTorrent is now available. This
release includes new functionality for communicating with APT using
a new transport method specifically designed for debtorrent, and
many bug fixes.
The major changes in this release are all in the communication
between APT and DebTorrent. The HTTP server in DebTorrent that
listens for APT requests has been upgraded to support HTTP/1.1
persistent connections and pipelining, which allows APT to have
multiple outstanding requests for files. This is useful as
DebTorrent suffers from the typical bittorrent slow start, so
requesting multiple files at a time helps to speed up the download
considerably.
Though better, HTTP/1.1 is not ideal for DebTorrent however, as a
maximum of 10 outstanding requests is maintained by APT's http
method, and files must still be returned in the order they were
requested (which is not ideal for bittorrent-type downloading since
downloads occur randomly).
To further improve the APT communication I have modified APT's http
method to create a debtorrent method. This new debtorrent transport
for APT is packaged separately as apt-transport-debtorrent, and once
installed APT can be told to use it by replacing "http://" with
"debtorrent://" in your sources.list file. This method sends all
requests it receives immediately to DebTorrent, and will receive
responses from DebTorrent in any order. You can find this new method
on the
Alioth project, or in my
personal repository
(only amd64 and i386 versions are currently available).
Unfortunately, the story doesn't end here. The APT code responsible
for sending requests to the method also limits the maximum number of
outstanding requests that it will send to the method to 10, which is
not really necessary since all existing methods limit the requests
they send out themselves. I have therefore patched the current APT
code to increase this limit to 1000 (a one line change), and
released this patched version as 0.7.6-0.1. You can find this
patched version in my
personal repository (again, only for
i386 and amd64). I have tested it with the other methods available
and it causes no problems, and I hope to get the change included in
the regular APT code soon.
To sum up:
- new DebTorrent over HTTP = fast
- new DebTorrent with new apt-transport-debtorrent = faster
- new DebTorrent with new apt-transport-debtorrent and a patched
APT = fastest
The last DebTorrent version (0.1.3.1) is currently in the
NEW
queue, and judging by the length of it, will be there for about
another week. After DebTorrent is added to the archive, I will be
upgrading it to this new version. I also hope to get the new
apt-transport-debtorrent package into the NEW queue soon.
This brings to an end the Google
Summer of Code (which this
project was
created as a part of), but development of
DebTorrent will of course continue (probably a little slower). The
next major change will be the addition of
unique piece
numbers, which is almost complete but needs to be extensively
tested. I'd like to thank Anthony Towns, Steve McIntyre, and Michael
Vogt for their help over the last 4 months, and also the many others
who sent me encouraging emails or engaged in interesting discussions
about this project. It's the people who make a project like this a
fun and memorable thing to do.
Here's the changelog for the new DebTorrent release:
- APT communication supports HTTP/1.1 connections, including
persistent connections and pipelining
- Add support for the new debtorrent APT transport method
(see the new apt-transport-debtorrent package)
- Make the Packages decompression and torrent creation threaded
- Improve the startup initialization of files
- Add init and configuration files for the tracker
- bug fixes:
- restarts would fail when downloaded files have been modified
- deleting old cached data would fail
- small tracker bug causing exceptions
- prevent enabling files before the initialization is complete
- only connect to unique peers from the tracker that are not
already connected
- tracker would return all torrents' peers for every request
One situation that you will occasionally encounter when running a
Heartbeat cluster is a need to prevent a STONITH of a node. As documented in
my previous post about testing STONITH the ability to STONITH nodes is very important in an operating cluster. However when the sys-admin is performing maintenance on the system or programmers are working on a development or test system it can be rather annoying.
One example of where STONITH is undesired is when upgrading packages of software related to the cluster services. If during a package upgrade the data files and programs related to the
OCF script are not synchronised (EG you have two programs that interact and upgrading one requires upgrading the other) at the moment that the status operation is run then an error may occur which may trigger a STONITH. Another possibility is that if using small systems for testing or development (EG running a cluster under Xen with minimal RAM assigned to each node) then a package upgrade may cause the system to thrash which might then cause a timeout of the status scripts (a problem I encounter when upgrading my Xen test instances that have 64M of RAM).
If a STONITH occurs during the process of a package upgrade then you are likely to have consistency problems with the OS due to
RPM and DPKG not correctly calling fsync(), this can cause the OCF scripts to always fail to run the status command which can cause an infinite loop of the cluster nodes in question being STONITHed. Incidentally the best way to test for this (given the problems of a STONITH sometimes losing log data) is to boot the node in question without Heartbeat running and then run the OCF status commands manually (
I previously documented three ways of doing this).
Of course the ideal (and recommended) way of solving this problem is to migrate all services from a node using the crm_resource program. But in a test or development situation you may forget to migrate all services or simply forget to run the migration before the package upgrade starts. In that case the best thing to do is to be able to remove the ability to call STONITH . For my testing I use Xen and have the nodes
ssh to the Dom0 to call STONITH, so all I have to do to remove the STONITH ability is to stop the ssh daemon on the Dom0. For a more serious test network (EG using IPMI or an equivalent technology to perform a hardware STONITH as well as ssh for OS level STONITH on a private network) a viable option might be to shut down the switch port used for such operations - shutting down switch ports is not a nice thing to do, but to allow you to continue work on a development environment without hassle it’s a reasonable hack.
When choosing your method of STONITH it’s probably worth considering what the possibilities are for temporarily disabling it - preferably without having to walk to the server room.
Share This
One situation that you will occasionally encounter when running a
Heartbeat cluster is a need to prevent a STONITH of a node. As documented in
my previous post about testing STONITH the ability to STONITH nodes is very important in an operating cluster. However when the sys-admin is performing maintenance on the system or programmers are working on a development or test system it can be rather annoying.
One example of where STONITH is undesired is when upgrading packages of software related to the cluster services. If during a package upgrade the data files and programs related to the
OCF script are not synchronised (EG you have two programs that interact and upgrading one requires upgrading the other) at the moment that the status operation is run then an error may occur which may trigger a STONITH. Another possibility is that if using small systems for testing or development (EG running a cluster under Xen with minimal RAM assigned to each node) then a package upgrade may cause the system to thrash which might then cause a timeout of the status scripts (a problem I encounter when upgrading my Xen test instances that have 64M of RAM).
If a STONITH occurs during the process of a package upgrade then you are likely to have consistency problems with the OS due to
RPM and DPKG not correctly calling fsync(), this can cause the OCF scripts to always fail to run the status command which can cause an infinite loop of the cluster nodes in question being STONITHed. Incidentally the best way to test for this (given the problems of a STONITH sometimes losing log data) is to boot the node in question without Heartbeat running and then run the OCF status commands manually (
I previously documented three ways of doing this).
Of course the ideal (and recommended) way of solving this problem is to migrate all services from a node using the crm_resource program. But in a test or development situation you may forget to migrate all services or simply forget to run the migration before the package upgrade starts. In that case the best thing to do is to be able to remove the ability to call STONITH . For my testing I use Xen and have the nodes
ssh to the Dom0 to call STONITH, so all I have to do to remove the STONITH ability is to stop the ssh daemon on the Dom0. For a more serious test network (EG using IPMI or an equivalent technology to perform a hardware STONITH as well as ssh for OS level STONITH on a private network) a viable option might be to shut down the switch port used for such operations - shutting down switch ports is not a nice thing to do, but to allow you to continue work on a development environment without hassle it’s a reasonable hack.
When choosing your method of STONITH it’s probably worth considering what the possibilities are for temporarily disabling it - preferably without having to walk to the server room.
Share This
debcheckout: some new bits
Some new bits about
debcheckout (talk is cheap,
code here):
-
authenticated mode. Consider svn (similar
arguments stand for other VCS). When checking out alioth
repositories using the svn:// prefix, the resulting local copy
can't be committed to, since it would require (assuming you have an
alioth account and the needed permissions) a svn+ssh:// access.
"authenticated mode" is precisely for that: when checking out
well-known repositories (only alioth's ATM) you can specify an
extra "-a" argument, with an optional "-u" to specify your user
name, and debcheckout will rewrite the repository URL so that the
resulting local copy can be committed to. ATM authenticated mode
works for svn, hg, bzr, git.
-
destination dir. It is now possible to specify
where do you want to check out a package repository (so that we
avoid ending up with tons of anonymous "trunk" directories). The
syntax is the common "debcheckout PKG DESTDIR" idiom and DESTDIR,
if not provided, defaults to the package name. (Thanks to JoeyH for the idea and the
initial patch.)
-
sorry about arch, but ATM it's almost
non-functioning, and I'm not willing to lose time on it, since
among all the VCSs supported by debcheckout it's the only one I've
never used. If you want support for it, please provide code!
(Perl) Tip of the day: Switch.pm
With
"use
Switch;" you will win a switch statement for your Perl
programs, which can be used as follows:
switch ($repo_type)
case "cvs" my $module = pop @cmd; push @cmd, ("-d", $destdir, $module);
case /^(bzr darcs git hg svn)$/
push @cmd, $destdir;
else die "sorry, don't know how to set the destination directory for $repo_type repositories (patches welcome!)\n";
This is far better than a chain of if/elsif statements and has
even a sane semantics (e.g. no need of explicit breaks, possibility
to have higher-case branches, ...). Unfortunately, it is not
possible (using a simple syntax) to match a scalar value against an
array case branch. Therefore the only way to factorize branches is
(when possible) to rely on regexp alternative branches, as it is
done in the code snippet above.
I'm pleased to learn from Romain Francoise that
only
3.1% of packages aren't shipping md5sums of their contents. I was
pleasantly surprised recently to discover that the level of participation
seemed to be a lot higher than I'd previously thought.
So given that Debian tends to make things policy after they've got broad
adoption, I wonder if we should make providing md5sums standard practice
now?
The RPM proponents have always touted
rpm -V as one of the things
that it did that Debian didn't.
The first version of the debian package of Perl Console has been uploaded to the NEW queue. For those who are waiting for it, I’ve also uploaded the package
here.
Thanks to the patch sent by Antonio Terceiro, the version 0.3 will be properly packaged
ala Perl (namely with the famous Makefile.PL, MANIFEST and friends).
I plan to adress the multi-line issue for 0.3 (mainly handling code with loops or conditional structures), as
Florian Ragwitz underlined, it could be worth using
Devel::REPL instead of rewriting the wheel.
Michael Hirschorn in
this month’s Atlantic reduces my generation’s entire cultural zeitgeist to a single word:
quirk.
Quirk, loosed from its moorings, quickly becomes exhausting. It s easy for David Cross s character on Arrested Development to cover himself in paint for a Blue Man Group audition, or for the New Zealand duo on Flight of the Conchords to make a spectacularly cheesy sci-fi video about the future while wearing low-rent robot costumes. But the pleasures are passing. Like the proliferation of meta-humor that followed David Letterman and Jerry Seinfeld in the 90s, quirk is everywhere because quirkiness is so easy to achieve: Just be odd but endearing. It becomes a kind of psychographic marker, like wearing laceless Chuck Taylors or ironic facial hair a self-satisfied pose that stands for nothing and doesn t require you to take creative responsibility. Just because you can doesn t mean you should.
Hirschorn makes a fair point which, I think, can be restated that much of the content I enjoy is really just candy.
The Atlantic seems to have recently figured out its readership (or at least figured out
me).
Hard to Swallow (by
B.R. Myers, who more typically writes about Korean issues) is a pointed moral critique of modern food lovers (
chowhounds?) and food writing (including Michael Pollan’s
The Omnivore’s Dilemma: A Natural History of Four Meals
, much read by my contemporaries). If I weren’t already a vegetarian, I might take umbrage.
“Insider baseball” pieces on
Karl Rove and
Michael Gerson are also excellent, and in the case of the former quite timely.
To wrap up this encomium, props to the magazine for its
clean new website design, which I believe premiered today, and for including an embedded Youtube video in the online version of the
quirk article.
Share This
