Search Results: "tar"

10 January 2011

Christian Perrier: [life] How to discourage the use of public transport

As some of my readers may now, I'm a regular user of public transport. Even though I'm living in an area (far suburbs of Paris) where the use of cars for nearly everything is common, I managed up to now to not use my car for my daily commute. I was using the public bus line of our town, that starts close from our house and ends up at the local train station where I can commute to U line of Paris transport system (then C line, then another bus ride). You can compare this complicated 1h30 travel to the 35 minutes drive I would have if I was using my car. I *was*. I'll stop from tomorrow. The reason is simple: my U train start hour is 6h57. The bus formerly started at 6h36, then reached the train station at 6h51, which was a quite good commute. That made me leave our house at about 6h25, considering a 10min walk from the house to the bus stop. That's 32 minutes for....3 kilometers. Already needs some motivation, right? This changed today (with minimal notice). To catch the same train, I should have a bus at 6h31, thus leave 5 minutes earlier, so *nearly 40 minutes before the train schedule*. Considering that driving from our place to the train station is about 12 minutes, the deal is now made. I'll stop being the only non captive person using the bus in our neighbourhood (all others are people who can't use a car). So much for environment, etc. This is an 'interesting' way to understand why it is so easy to discourage proper use of public transport and minimize the impact on environment. A small change can lead to such effects. Of course, as you may expect, I won't stop there. Letters to the (public-funded) bus company and people in charge of public transport in our city (the bus line deficit is funded by the city), will follow very soon. Tehre are indeed many other reasons why people don't use our local buses that much. One being that, even though there are 2 trains every 15 minutes leaving the station (U and N lines), both alternated with a 7.5 minutes interval....there is only one bus every 15 minutes....on each bus line. Sometimes, even, you don't know why, there's a 23 minutes gap. Imagine waiting for your bus for more than 20 minutes at night in winter...when you're about 2-3 kilometers away from your home... Sure, that would need a lot of funding to have a really convenient system. But how about spending the billions of euros spent in super-highways, toll tunnels, etc., in public transport infrastructure? Funding my local bus line to have 2 buses every 15 minutes at peak hours would cost about the price of 100m of a 3-lane highway, for one year. Make the deal. (All that will indeed reinforce my backup system : run from my house to the train station, for 3km and 15 minutes...which involves another run at the end of my train ride, for 4km and 20 minutes. That, however, involves some "logistics" organization to be sure that I don't spend the day working in my stinking running suit. That also means up to 56km run every week, which I can't always do, particularly if I plan a race for the next week-end!) Another side effect: I will have less time for Debian as my bus commutes were used for hacking, while it will be much more difficult when driving my car to the train station, isn't it?

9 January 2011

Nikita Youshchenko: checkrestart

After aptitude or similar upgrade tool completes its job, new versions of files are installed into filesystem. However, some running processes may be still using old files (and actually Linux file systems continue to store invisible copies of old files while those are mapped by running processes). In case of a security update, having old versions running keeps system vulnerable even after update completes, which is not good. A reboot is possible, but that s an ugly solution. Fortunately tools exist to find processes that have deleted files mapped. One of these tools is checkrestart (from debian-goodies package). Just run checkrestart -p after upgrade, and you will see what to restart to be safe and even what init.d scripts to run with restart argument. P.S.
On a server with hundreds of running processes, checkrestart -p is terribly slow, likely because it makes separate queries to dpkg about files mapped by each process. Perhaps it could be seriously accelerated if first all names to query are gathered, and then each name is queried only once. Any volunteers to do that? :)

8 January 2011

Nikita Youshchenko: Psi / Psi+: want to avoid auto-online after auto-offline

Older versions of Psi jabber client required manual changing status to online to leave auto-offline state. It was very handy for me: I connect to my jabber account from two locations (from home and from office), so after I left one of these locations, auto-offline soon happened, and Psi at that location did not return online until I returned and manually instructed it to re-login. Unfortunately newer versions of Psi (I guess, starting from commit 4a54500c on git://git.psi-im.org/psi.git, which was between 0.11 and 0.12 releases), and also Psi+, behave differently: if user activity is detected on the desktop, auto-offline state is left, and status becomes online again similar to leaving auto-away and auto-xa states. This change is not good for me, at least in two situations: In both these situations, Psi client not at my current location becomes online, which can cause routing incoming messages to wrong location. I don t state that new behaviour is somehow worse than old behaviour. But for my use cases, I want to get old behaviour back. I ve done some code reading. I think that point to fix is near the bottom of PsiAccount::secondsIdle() method, where any auto status is disabled if no timeout condition currently exists. This should be changed so that auto-offline state won t be left if some config key is set. I will try to implement this change soon (or at least I hope I will try ) P.S.
To make things even better, a check of some external condition could be performed to decide if client should return online from auto-offline or not. Then fancy things could be set up, e.g. return to online at office only if my mobile phone is currently registered in the office wifi network not really required but cool.

David Bremner: Beamer overlays in highlighted source code

I use a lot of code in my lectures, in many different programming languages. I use highlight to generate HTML (via ikiwiki) for web pages. For class presentations, I mostly use the beamer LaTeX class. In order to simplify generating overlays, I wrote a perl script hl-beamer.pl to preprocess source code. An htmlification of the documention/man-page follows.
NAME hl-beamer - Preprocessor for hightlight to generate beamer overlays. SYNOPSIS hl-beamer -c // InstructiveExample.java highlight -S java -O latex > figure1.tex DESCRIPTION hl-beamer looks for single line comments (with syntax specified by -c) These comments can start with @ followed by some codes to specify beamer overlays or sections (just chunks of text which can be selectively included). OPTIONS CODES EXAMPLE Example input follows. I would probably process this with 
hl-beamer -s 4 -k encodeInner
Sample Input 
 // @( omit
 import java.io.BufferedReader;
 import java.io.FileReader;
 import java.io.IOException;
 import java.io.Serializable;
 import java.util.Scanner;
 // @)
     // @( encoderInner
     private int findRun(int inRow, int startCol) 
         // @<
         int value=bits[inRow][startCol];
         int cursor=startCol;
         // @>
         // @<
         while(cursor<columns && 
               bits[inRow][cursor] == value) 
             //@<
             cursor++;
             //@>
         // @>
         // @<
         return cursor-1;
         // @>
      
     // @)
BUGS AND LIMITATIONS Currently overlaytype and section must consist of upper and lower case letters and or underscores. This is basically pure sloth on the part of the author. Tabs are always expanded to spaces.

Nikita Youshchenko: Psi+

Just discovered Psi+ project. It is a fork of well-known Psi jabber client, with many improvements and new features. Among others things, Psi+ implements plugins feature, and comes with several interesting and useful plugins. One of those provides anti-spam protection, and other is designed specifically for ICQ users. Psi+ is a drop-in replacement for Psi, it uses same per-user configuration. Package is available for Debian Squeze, so if you are already running that version, just run 
aptitude install psi-plus psi-plus-plugins
Then restart Psi and enjoy :) .

Joerg Jaspert: Keyboard layout - 10 finger typing

For some reason, I don t really know why, I decided to switch my keyboard layout over to something weird. No, not the french brainfuck :) that is actually possible to type on in a halfway acceptable speed (shutup Bubulle), but the Neo thing for german keyboards. Hell. I didn t know how slow one can be typing. About a letter in 3 seconds, or something like it. I think, we should add something to our mandantory set of rules for anyone on any Debian Mailinglist: Whenever a flamewar starts, everyone who participates has to switch their keyboard layout to a random new layout. For every paragraph of every mail they write. Would stop the thread before it really happens. Ohwell, lets see where I stand in a week/month from now. Maybe I can actually use it then. And maybe I really type with 10 fingers then, and not something like the 3 til 8 i currenly use (depends on moon phase how many, i think). I m not slow with my current system, far from it, but I do sometimes have pain in my hands. Maybe that goes away too if I adopt something better, with less weird hand/finger movings. Though the way over will be painful in itself.

7 January 2011

Evgeni Golov: Is Delicious loosing their data?

I use Delicious from time to time to save bookmarks and search for stuff I can t remember where I found it. It was sad to read rumors that Yahoo! will shut it down, but that s not the point here now.In the last months I noticed some strange behavior with their site: when surfing a page of a bookmark, you see the total number of people who have bookmarked that particular link and that number started to jump from view to view. First loaded you see a total of 183 bookmarks, 10 minutes later it s 265 bookmarks, and then 183 again. What s going on there? I guess that is some inconsistency between their datacentres or something, but I m not sure (and I am sure that this should not happen!).Today I noticed another funny thing: bookmarks without an owner:

Again, not sure what is happening here. Did the user delete his account? Delicious says all bookmarks will be gone when deleting, so this should be something different, but again, I have no clue.I fear I will have to search a new home for my bookmarks. Any (open-source) social bookmarking service out there worth a try?

Steve McIntyre: Isohybrid CDs being produced

New feature for Debian CDs We've had a wishlist bug (#551951) open against debian-cd for some time, asking for the creation of "isohybrid" CD images for i386 and amd64. These are special in that as well as the normal CD-based ISO9660 filesystem they also contain a valid-looking DOS-style partition table. Thus, if you simply copy one of these images raw to a USB stick a normal PC BIOS will boot the image directly. This would be a neat feature, making it much easier for people to use standard Debian installer images on their USB sticks without having to follow a lot more instructions. Support for this feature was added in syslinux/isolinux quite a while back (thanks to H Peter Anvin and the other syslinux contributors), but this depended on post-processing images with the isohybrid program. Due to the special way that we make our CD and DVD images to produce jigdo images as well, that wouldn't work for us: the jigdo files would no longer match the ISO files. Bugger...! Recently, along came more help in the form of Thomas Schmitt (author of xorriso) and George Danchev, its Debian maintainer. Recent versions of xorriso support creating isohybrid images directly, which is very handy. We've worked together in the last few months on porting my old JTE code (which creates our jigdo images) from cdrkit to xorriso, and after a lot of testing and debugging we now have things working fully. I've added a small amount of code in debian-cd to use the new xorriso features, and for the last couple of weeks all of the i386, amd64 and i386-amd64 multi-arch CDs and DVDs have been built as hybrids. It's been a delight working with Thomas and George - they're enthusiastic, helpful and friendly. Thanks, guys! What does this mean for end users? Instead of having to specially prepare USB sticks for the installer, you can now simply use dd to write the image straight to the raw stick, e.g.: 
# dd if=debian-testing-i386-netinst.iso of=/dev/sdX
The USB stick should now boot directly, showing exactly the same bootup screen as if you'd written the image directly to a CD. Yay! I'm expecting to be using this code from now on for new releases, i.e. d-i beta and RC releases and for Squeeze itself when we get there. I'm not planning on backporting this code into the Lenny branch of debian-cd as it's a lot of work and I don't want to destabilise things there. What's next? We need to clear up some of the maintenance around the code as it stands; I'm planning on moving libjte inside my existing jigit package, as that's a sensible place to put it. That will come after the Squeeze release, though. We need to finalise boot support for other architectures in xorriso; I added support for them into mkisofs/genisoimage a long time ago and we've already moved some of that into xorriso. Thomas and George have also been working hard to give me faster/better size estimation code, useful for making debian-cd run faster. Finally: I'd like to add support into xorriso for creating the nasty HFS hybrid images that are needed for booting Macs. The code that does this in cdrkit is probably some of the worst that I've ever worked with, and I'd like to get away from it. If only Apple hadn't stupidly built their proprietary platform around this shit and had used open standards instead. :-( Once we have these bits in xorriso, I'll be able to move production of all of the CDs across from genisoimage to xorriso. Very nearly there for being able to drop the last remnants of Schily-ware from the Debian archive...

Rapha&#235;l Hertzog: How to create custom RSS feeds with WordPress

WordPress has many alternate built-in feeds: per category, per tag, per author, per search-keyword. But in some cases, you want feeds built with some more advanced logic. Let s look at the available options. WordPress advanced built-in feeds You can create feeds for unions or intersections of tags, you just have to use a URL like /tag/foo,bar/feed/ (all articles tagged with foo or bar) or /tag/foo+bar/feed/ (all articles tagged with foo and bar). You can also have feeds excluding a category, although that requires you to know the category identifier (and hardcode it in the URL like this: /?feed=rss2&cat=-123 where 123 is the category id that you want to exclude). But there s no simple way to have a feed that excludes articles with a given tag. The best solution found involves creating a custom feed. I ll show you a variation of this below. Creating a custom feed
  1. First of, install the Feed Wrangler plugin, it will take care of registering our custom feeds with wordpress.
  2. Go to Settings > Feed Wrangler in your WordPress administrative interface and create a new feed, let s call it myfeed .
  3. You should now create a feed-myfeed.php file and put it in your current theme s directory. The initial content of that file should be this: 
    <?php
include('wp-includes/feed-rss2.php');
?>
  4. At this point, you already have a new feed that you can access at /feed/myfeed/ (or /?feed=myfeed). It s a complete feed like the main one.
Now, we re going to look at ways to customize this feed. We re going to do this by changing/overriding the default query that feed-rss2.php s loop will use. A feed excluding articles with a tag If you want to create a feed that excludes the tag foo , you could use this: 
<?php
global $wp_query;
$tag = get_term_by("slug", "foo", "post_tag");
$args = array_merge(
        $wp_query->query,
        array('tag__not_in' => array($tag->term_id))
);
query_posts($args);
include('wp-includes/feed-rss2.php');
?>
That was relatively easy, thanks to the tag__not_in query parameter. Now you can further customize the feed by adding supplementary query parameters to the $args array. The documentation of query_posts details the various parameters that you can use. A feed excluding articles with a custom field (meta-data) I went further because I did not want to use a tag to exclude some posts: that tag would have been public even if it was only meaningful to me. So I decided to use a custom field to mark the posts to exclude from my new feed. I named the field no_syndication and I always give it the value 1 . This time it s not so easy because we have no query parameter that can be used to exclude posts based on custom fields. We re going to use the post__not_in parameter that can be used to exclude a list of posts. But we must first generate the list of posts that we want to exclude. Here we go: 
<?php
global $wp_query;
$excluded = array();
$args_excluded = array(
    'numberposts'     => -1,
    'meta_key'        => 'no_syndication',
    'meta_value'      => 1,
    'post_type'       => 'post',
    'post_status'     => 'published'
);
foreach (get_posts($args_excluded) as $item)  
        $excluded[] = $item->ID;
 
$args = array_merge(
        $wp_query->query,
        array('post__not_in' => $excluded)
);
query_posts($args);
include('wp-includes/feed-rss2.php');
?>
A feed with modified content You might want to add a footer to the articles that are syndicated. I use the Ozh Better Feed plugin for this but it applies to all your feeds. You could do that sort of transformation only in your customized feed by using the WordPress filter named the_content_feed. Here s a simple example: 
<?php
function myfeed_add_footer($content)  
        return $content . "<hr/>My footer here";
 
add_filter('the_content_feed', 'myfeed_add_footer');
include('wp-includes/feed-rss2.php');
?>
I ll stop here but obviously you have lots of options and many ways to tweak all the snippets above. They have been tested with WordPress 3.0.4. Note that in all those examples, I took care to not duplicate the code from feed-rss2.php, instead I used include() to execute it. That way my custom feeds will automatically benefit from all the future enhancements and fixes made by the WordPress developers. But if you have to modify the XML structure of your custom feeds, you can paste the content of feed-rss2.php in your file and change it like you want

One comment Liked this article? Click here. My blog is Flattr-enabled.

Axel Beckert: "peer holds all free leases" on both DHCP servers

At work we run a pair of ISC DHCP servers running Debian Lenny in a classical ISC DHCP failover setup which provide DHCP service to several subnets, some only with static IPs (e.g. for printers) and some with half static and half dynamic IPs. Today I got a call from a user that her laptop doesn t get an IP despite it s correctly registered in our MAC address database from which we generate the group sections of the dhcpd.conf. Everything looked fine, but every DHCPDISCOVER package got logged in the syslog on both servers like this: 
Jan  7 14:34:39 dhcp1 dhcpd: DHCPDISCOVER from 01:23:45:67:89:ab via eth2: peer holds all free leases
Jan  7 14:34:39 dhcp2 dhcpd: DHCPDISCOVER from 01:23:45:67:89:ab via eth2: peer holds all free leases
Searching the web for this error message mostly results in mails which say If have this on one server but not the other, you soon run out of IP addresses , but none which mentions what happens if you got them on both sides. Following a coworker s idea of adding both servers to the search term, I found Debian bug #563449 (dhcp3-server: Incorrect peer holds all free leases log entries) which turned out as configuration error or at least unexpected configuration (machine was blocked from getting an IP on purpose) and misleading error messages. So I checked under which circumstances this computer would not get an IP despite it had a static IP configured: 
  host somehost  
    hardware ethernet 01:23:45:67:89:ab;
    fixed-address 192.0.2.123;
That computer would not get an IP address in any subnet which has different IP range and no dynamic IP addresses. And even if I comment out the fixed-address setting, it wouldn t get an IP in any static-IPs-only subnet either. And *bingo*, that computer was plugged into the printer subnet which has only static IPs, e.g. in the 198.51.100.x range. So if you get the peer holds all free leases error message from both your DHCP servers, chances are very high that the mentioned MAC address should really not get an IP address on this network (as it does :-). The error messages are just somewhat misleading. Hope, this saves someone some time. :-)

Paul Wise: Another year, another log entry

It has been almost a full year since my last log entry. It has been a busy work year, I attended some nice conferences and did minimal FLOSS stuff. On the work side of things I was a third of an Australian VoIP startup that came and went. I setup Debian servers, installed OpenSIPS and associated software, wrote OpenSIPS scripts, wrote peripheral software and did customer support. We had a good thing going there for a while, some fans on the Whirlpool forums but in the end there wasn't enough money for the requisite marketing and local market circumstances were squeezing Australian VoIP providers anyway. On the conference side of things I went to LCA 2010, the Thai Mini-DebCamp 2010, DebConf10 and FOSSASIA 2010. Had a great time at all of them. At LCA 2010 in windy Wellington, New Zealand the distributions summit organised by Martin Krafft was one of the highlights. It was dominated by Debian/Ubuntu talks but there were some other interesting ones, especially the one on GoboLinux's integration of domain-specific package managers. Also excellent were the keynotes given by Gabriella Coleman (Best & worst of times), Mako Hill (Antifeatures) and others, which I felt gave LCA an improved and very welcome focus on software freedom. There were quite a few Debian folks at LCA, it was great to hang out with them during the week and afterwards. Monopedal sumo with mako and others was hilarious fun. At the Thailand Mini-DebCamp 2010 in Khon Kaen, I was glad to see Andrew Lee (Taiwan) and Christian Perrier (France) again and meet Yukiharu YABUKI (Japan) and Daiki Ueno (Japan). In addition to the five international folks, there were quite a few locals, including Thailand's currently sole Debian member, Theppitak Karoonboonyanan. The event was hosted at Khon Kaen University and opened with my talk about the Debian Social Contract and the Debian Free Software Guidelines. This was followed by a number of talks about Debian package building, a 3-day BSP where we touched 57 bugs, a great day of sightseeing and talks about i18n, derivative distros, keysigning, mirrors, contribution and a discussion about DebConf. During the week there was also the usual beersigning, combined with eating of unfamiliar and "interesting" Thai snacks. After the conference Andrew and I roamed some markets in Bangkok and got Thai massages. Beforehand I also visited a friend from my travels on the RV Heraclitus in Chiang Mai, once again experiencing the awesomeness of trains in Asia, unfortunately during the dry season this time. I took a lot of photos during my time in Thailand and ate a lot of great and spicy food. As a vegetarian I especially appreciated the organiser's efforts to accommodate this during the conference. At DebConf10 in New York City, by far the highlight was Eben Moglen's vision of the FreedomBox. Negotiating the hot rickety subways was fun, the party at the NYC Resistor space was most excellent, Coney Island was hot and the water a bit yuck, zack threw a ball, the food and campus was really nice. Really enjoyed the lintian BoF, ARM discussions, shy folks, GPLv3 question time, paulproteus' comments & insights, wiki BoF, puppet BoF, derivatives BoF, Sita, astronomy rooftop, cheese, virt BoF, Libravatar, DebConf11, Brave new Multimedia World, bagels for breakfast, CUT, OpenStreetMap & lightning talks. Having my power supply die was not fun at all. Afterwards I hung out with a couple of the exhausted organisers, ate awesome vegan food and fell asleep watching a movie about dreams. One weird thing about DebConf10 was that relatively few folks used the DebConf gallery to host their photos, months later only myself and Aigars Mahinovs posted any photos there. At FOSSASIA 2010 in H Ch Minh City (HCMC) was a mini-DebConf. I arrived at the HCMC airport and was greeted by Huyen (thanks!!), one of FOSSASIA's numerous volunteers, who bundled me into a taxi bound for the speakers accommodation and pre-event meetup at The Spotted Cow Bar. The next day the conference opened at the Raffles International College and after looking at the schedule I noticed that I was to give a talk about Debian that day. Since I didn't volunteer for such a talk and had nothing prepared, the schedule took me by surprise. So shortly after an awesome lunch of Vietnamese pancakes we gathered some Debian folks and a random Fedora dude and prepared a short intro to Debian. The rest of the day the highlights were the intro, video greetings and the fonts, YaCy and HTML5 talks. The next day the Debian MiniConf began with Arne Goetje and everyone trying to get Debian Live LXDE USB keys booted on as many machines in the classroom as possible (many didn't boot). Once people started showing up we kicked off with Thomas Goirand's introduction to the breadth of Debian. Others talked about Debian pure blends, Gnuk and building community and packages. The second last session was about showing the Vietnamese folks in the room how to do l10n and translation since Debian had only one Vietnamese translator (Clytie Siddall). After manually switching keyboard layouts (seems LXDE doesn't have a GUI for that) on the English LXDE installs, the two Cambodian folks were able to do some Khmer translation too. This was a great session and it resulted in two extra Vietnamese translators joining Debian. It went over time so I didn't end up doing my presentation about package reviewing. We rushed off to a university where the random Fedora ch^Wambassador was hosting a Fedora 14 release party in a huge packed classroom. There were a lot of excited faces, interesting and advanced questions and it was in general a success. Afterwards we had some food, joined up with some other speakers and ended up in a bar in the gross tourist zone. On the final day we hung around in the Debian room, went downstairs for the group photo and final goodbyes. Later we found a place with baked goods, coffee and juices and navigated the crazy traffic to a nice local restaurant. The next morning Arne & I went to the airport, others went on a Mekong Delta tour and Jonas hung out with the organisers. I took less photos than at other events but got a few interesting ones. I avoided doing a lot of FLOSS stuff over the last year, I hope to work on some things in the coming months; I'm also planning some interesting travel and acquiring some new technological goods, more on those in some later posts.

6 January 2011

Michael Prokop: Booting ISO images from within GRUB2

You might be aware of GRUB s loopback option for booting an ISO, I wrote about it in Boot an ISO via Grub2 more than a year ago. A few months ago Goswin von Brederlow came up with this idea:
grml functions great as rescue system. So it would be nice to have a boot entry for it in grub instead of having to go look for the CD when needed. To make installing and updating simple it would be great if one could install this as a normal debian package which would register itself in grub (and maybe lilo too).
What a lovely idea: no need for a CD or USB stick as long as the bootloader and the harddisk are still working. Minimal manual intervention needed to keep it up2date and working sounds like the perfect rescue system. 8-) Now since end of 2010 a new stable version of sysadmin s favorite live system (AKA Grml 2010.12) is available and the great news is that we came up with two independent solutions known as grml-rescueboot and grub-imageboot. Having shipped them to one of my customers already I d like to write some words about it and why you should also consider using it on all your systems where GRUB2 is available. grml-rescueboot uses GRUB2 and its loopback feature for booting. Grml as well as Ubuntu provide all what s needed out-of-the-box. If you re interested in the details check out Jordan Uggla s excellent Loopback.cfg webpage in the supergrubdisk wiki. The best about it: you can provide custom bootoptions automatically when booting Grml. For example just set CUSTOM_BOOTOPTIONS= ssh=grml2011 in /etc/default/grml-rescueboot and the Grml rescue system will automatically start the OpenSSH server with the specified argument as password for user grml. Of course you can use all the other nifty bootoptions like scripts/netscript/ to further customize your Grml rescue system. [A note to other distributions like Ubuntu: I'd be interested to establish a mechanism to pass kernel options for loopback boot in a standardized way, drop me a note if you're interested in this.] grub-imageboot uses GRUB2 and syslinux memdisk to boot ISOs and floppy images. It doesn t rely on the loopback feature but instead maps the ISO into the memory directly. This is great for Linux ISOs that can t and won t support the the loopback feature (and have everything inside their initrd so the ISO doesn t have to find itself during booting), like BIOS or firmware updates. Also for example FreeDOS and Alpharev are known to work just fine. But sadly memdisk ISO emulation doesn t work with all Linux systems, as documented in the syslinux wiki. The good news is that Grml supports the memdiskfind/phram/mtdblock approach out-of-the-box. As far as I know therefore Grml is the first Debian based live system supporting memdisk ISO boot, though my patches already went to the Debian-Live project so if you re using live-boot >=2.0.14-1 or >=2.0.12-1+grml.04 your live system should be able to boot via memdisk ISO emulation as well. Summary: If you want to boot a Linux system which supports loopback.cfg use grml-rescueboot as its a more powerful tool to boot Linux live systems like Grml and Ubuntu. If you want to boot non-Linux systems (BIOS-/Firmware-Updates/FreeDOS/ .) use grub-imageboot. Alright how do you deploy this solution? Just grab and install grml-rescueboot and/or grub-imageboot. To deploy grml-rescueboot (adjust $VERSION if you want to use another flavour): 
# choose Grml version:
VERSION=grml64-medium_2010.12
# create directory
mkdir -p /boot/grml
# download and verify ISO
cd /boot/grml
wget download.grml.org/$ VERSION .iso ,.md5 
md5sum -c $ VERSION .iso.md5
To deploy grub-imageboot: 
# choose Grml version:
VERSION=grml64-medium_2010.12
# create directory
mkdir -p /boot/images
# download and verify ISO
cd /boot/images
wget download.grml.org/$ VERSION .iso ,.md5 
md5sum -c $ VERSION .iso.md5
That s it! Now when running update-grub you should get something like: 
# update-grub
Generating grub.cfg ...
Found linux image: /boot/vmlinuz-2.6.36-grml64
Found initrd image: /boot/initrd.img-2.6.36-grml64
Found memtest86+ image: /boot/memtest86+.bin
Found memtest86+ multiboot image: /boot/memtest86+_multiboot.bin
  No volume groups found
Found Grml ISO image: /boot/grml/grml64-medium_2010.12.iso
Found memdisk: /boot/memdisk
Found iso image: /boot/images/grml64-medium_2010.12.iso
done
Voil , when rebooting your system you should see something like: Screenshot: grml-rescueboot The Grml Rescue System entry is what grml-rescueboot provides and Bootable ISO Image is what s provided by grub-imageboot. Just select the entry you d like to use, press enter and and you should get the bootsplash of the according ISO. BTW: I ve tested this with Ubuntu 10.10 too, grml-rescueboot works out-of-the-box with the Ubuntu ISO as well, it just doesn t support the memdisk ISO boot by grub-imageboot (yet). Tip: if you want to use grml-rescueboot and grub-imageboot with the same ISOs without having them twice on the disk just point the configuration option IMAGES in /etc/default/grub-imageboot to the according directory (like /boot/grml). Note: if you re using the ext 2,3,4 filesystem on the partition that s being used for /boot please be aware that you need GRUB >=1.98+20100804-12 (newer version is already available in Debian/unstable and hopefully migrates to squeeze in time) or GRUB >=1.99~20101122-1 because of the INDIRECT_BLOCKS support, see #543924 for the details. PS: We think about providing grml-rescueboot and grub-imageboot within official Debian. If you re interested in seeing integration support for other distributions as well please help and drop us a note.

Michael Prokop: Simple DNS in chroots

Update: Ulrich mru Dangel suggested pdnsd as nice alternative to dnsmasq and Cyril KiBi Brulebois pointed out, that it s not necessary to invoke dnsmasq after fresh installation as it s running by default then I adjusted the text accordingly, thanks for the pointers! If /etc/resolv.conf doesn t provide any nameserver entries glibc[1] will automatically initialize the nameserver to the loopback address. This is nice for dealing with chroots without having to manually edit resolv.conf to get working DNS. Just install a basic DNS forwarder like dnsmasq ( apt-get install dnsmasq ) or if it s already on your box just start it ( /etc/init.d/dnsmasq start ). That s it. Now when chrooting into a system without an existing resolv.conf configuration it will give you a working DNS setup without any further work. PS: Grml ships dnsmasq by default and /etc/init.d/dnsmasq start or Start dnsmasq will work out-of-the box. In Ubuntu the dnsmasq package is available only through the universe repository but the dnsmasq-base package (providing e.g. the init script) is shipped by default, so replace the /etc/init.d/dnsmasq command with a simple dnsmasq there. [1] At least the [e]glibc versions provided on Debian and Ubuntu are known to provide this glibc extension. Other libc implementations like dietlibc and uclibc don t seem to provide it, so don t strictly rely on this feature but use it as the icing on the cake.

Christian Perrier: [life] [running] 2010 summary and early 2011 plans

So, 2010 ended and it's now time for me to make a small summary for my running activities. At first, it's more and more obvious that running became an increasing part of my life. Sometimes eating the free time I was using for Debian work....and sometimes just filling holes in my daily activities, where I was just doing nothing..:-) According to my running watch(es) records, I ran for 2172km this year in about 196 hours (8.25 days) at an average speed of 11.1km/h. Indeed, the speed ranges from 10km/h in the hardest trail runs up to 13.5km/h in the fastest (and shortest) asphalt runs. I actually never run at "slow" paces, which is probably not the best for training as all official training plans do include slow runs. But, eh, this is how it works for me..:) I ran 12 official or semi official races, ranging from 5km cross-country races up to marathons. More proeminently speaking, I ran 3 half-marathons, 2 marathons and one long distance trail race (33km) that is worth being counted as a marathon..:). I also broke my personal best times for 10km (48'05", ran alone), half-marathon (1h37') and marathon (3h40'). Quite a good year, indeed..:-) I'm currently busy building my plans for 2011. Some are already decided: In general, I'll probably put more focus on trail and ultra (above martahon) runs. 2012 could then be the year of my first 100km race..:-) What will be fun will be reading this in January 2012 and see what I have completed among all these goals..:-)

Michal &#268;iha&#345;: Finding a GPL violation

As the communication in this case seems to be stuck and the issue is almost one year old, it's time to make bits of it public. The first post will just describe how I've discovered it, later will bring more details. All started innocently with bug report on Gammu that it fails to work with some Sonim phones. We started to dig some information about what extensions does Sonim use and one of obvious ways was to try their software. I picked up software for Sonim XP3.20 Quest and started to look at it (if you are not going to install them, but want to look, you need unrar and unshield to unpack it). At first look the names of DLLs looked familiar to me and running strings on the DLLs just confirmed my suspicion - the phone which is not supported by Gammu bases it's official software on it. I was able to recognize at least seven DLLs derived from (GPL licensed) Gammu (they even use same names as Gammu modules had in that time): To make the thing more interesting they use several other components released under free software licenses, where they also should provide sources for them: There might be more, but I did not want to spend more time on deep analysis, because I already had enough information.

Filed under: English Gammu Gpl 6 comments Flattr this!

Gunnar Wolf: Back with Ruby on Debian polemics

Once again, a polemic regarding how to properly integrate the Ruby language and libraries with the Debian distribution has been ignited. Similar arguments were presented in November and December 2008 and September 2010 And excuse me for just refering to my own blog, but there are links to some other posts from there... and there will surely be many others I just missed. There were even nice collaboration attempts (such as DebGem, announced in January 2009... But which apparently never stuck on, as it is still listed as in its public Beta period and lists only support for Debian 4.0 and Ubuntu 8.04 and 8.10). Some days ago, while I was on vacation, I received a mail from Lucas Nussbaum expressing the burnout he had been suffering from this situation. Some days after that, he posted a corresponding blog post - Giving up on Ruby packaging. The comments following his post are most interesting The one comment I'd like to highlight (noting I skimmed a whole deal of them) is by Paul Brannan, one of the original RubyGems authors. Yes, possibly the design criteria for Gems included some mutually-conflicting goals, and they cover some of Debian's goals. Unfortunately, this conflicting criteria... Was resolved to the opposite way we would have wanted (specifically, that a sysadmin should be able to install software with the same tools he was already familiar with). And of course, there are deeper disagreements, which are rightly stemmed from different priorities: Debian (+derivatives) is meant to make user's and sysadmin's lifes simpler. Gems are meant to make developers' lifes simpler. And while all developers are some sort of users, and sysadminsthe reverse is (obviously) not so. A developer maintaining several different codebases will surely suffer (well, this particular developer has surely suffered) if he has to maintain them all using coherent versions of system libraries. When a system is programmed, its developers want to use the greatest, latest tools to offer the best experience and functionality. And it's natural for libraries to evolve over time. However, any sysadmin will grind their teeth at the prospect of having many different versions of libraries, as the Gem model proposes. Why? Well, I was bitten by a minor example of it Bug fixes do not get backported. Of course, if simple bugs are not backported, security and stability of the system as a whole suffers. Often, new software versions will not only add functionality, but improve on how things are done. Not only bugs get corrected, but we get better response times, better memory handling, etc. It might sound harsh to say this, and even more, as a developer it feels I am talking against myself But while development time should be minimized during the system design/implementation, once systems are in production, it should be used to allow for friendlier sysadmin relations. I have to wear both hats at my real-life job. So, what is my take on this? Of course, blaming myself for choosing the wrong version is a no-go. When evaluating a library before using it in my projects, I try to look at its history and API-stability. APIs that change often speak about immature libraries which are still trying to get the right way to implement their functionality; just-started projects might be great and revolutionary, but they do not yet show any kind of long-term committment... And can lead (and have often led me) to painful rewrites when the Next Big Thing is reached. As for the users: Even if your favorite language has the best and friendliest distribution method, users just do not care about what language was a particular piece of software implemented in. Users want to be able to install and uninstall with their system's usual tools. Directly using Gems, CPAN, PEAR and whatnot is just unnatural for users, however convenient they are for developers. Distributions offer a non-technical advantage in this regard as well: A human filter. As an example, as I write this there are 19392 Ruby Gems and 19092 Perl modules in CPAN (note that CPAN stores some older versions but discourages authors from keeping too many - So no, they are still very different in size). Debian has around 30,000 packages. Why is that? Because all Debian packages _must_ be human-generated, human-reviewed, human-submitted. This means, a person must think each packaged piece of code is worth packaging, is stable enough and provides value to users as it is, and is fit for being part of a stable release. I am not saying with this that 90% of CPAN and Gems are crap I am saying that they are probably early implementations, to be installed, tested and improved by developers, and still not apt for general public use. Or maybe not interesting enough to be packaged as a service for the non-techie public at large. Oh, and one last point: Ruby is not a new language anymore. It is a mature, powerful language with different implementations, every time more stable... But it is a language deeply affected by the (not so new anymore either) the appearance of Rails. Why do I say this? Because although the language is in no way tied to Web development, many of its strongest uses are Web-oriented. How does this affect the current discussion? Well, because many people argue that users are no longer needed to install the software. Web systems are installed by sysadmins and used via a Web browser, and sysadmins are expected be more skilled than casual users. Still, in Debian (and in other distributions, surely) we try to make sysadmin's lives simpler I have (again, talking out of personal experience) installed several webapps (and system tools, and whatnot) for which I never followed any instructions besides aptitude install foo Using different languages, frameworks, and so on. Can I troubleshoot their installs? Probably, as there is a common logic for how the distribution I have chosen and specialized in works. Can I find causes for bugs in them? Possibly, although there are some languages and frameworks I dare not touch. Can I get help on getting them out of a tight spot? Surely, as there is a central bug tracking system for my distribution And one of the maintainer's tasks is to separate the problems related to the distribution (packaging, installing, simple user questions and misconceptions) to those derived from real bugs upstream. Anyway I am not saying that our way is the best way. No, by a long shot. Again, developers should have an easy, convenient way for installing whatever they want to play with. And to publish it without jumping through hoops. With this post, I'm only trying to express again why Debian works the way it does. And hope for better cooperation in the future. And as for several comments of what I read in Lucas' post, I think that there is interest for this synergy among some of the most committed Ruby people.

Michal &#268;iha&#345;: Impressed by xz compression

I knew that xz (or lzma) provides better compression ratio than bz2 or others, but I never thought the difference might be so huge. Simply I was impressed after I've enabled xz compressed snapshots for Gammu - the bz2 compressed tarball has 5.4M while xz compressed on only 1.6M. Wow.

Filed under: English Gammu 3 comments Flattr this!

Enrico Zini: Streaming JSON objects with python

Streaming JSON objects with python Dear Lazyweb, Suppose I'd like to create an HTTP service that gives you a looong list of JSON objects. For example, one JSON object for every package in Debian. One way I could do that is to just transfer a big, dozens of megabytes JSON array with one element per package. With the standard json module in python, this requires the client to buffer all the data in memory for decoding, as the JSON result would be something like this: 
[
    info for package 1  ,
    info for package 2  ,
  ...dozens of megabytes...
    info for package 30000  ,
  ...
]
However, it would be reasonable to engineer things so that the client can process packages one at a time, as they are produced. This can dramatically reduce client memory usage, as well as remove a large latency between the start of the request and the start of processing. One trivial idea would be to just send JSON objects one after the other, and sort of call load multiple times. It doesn't work: 
import json
from cStringIO import StringIO
a = StringIO()
json.dump( 1:3 , a)
json.dump( 2:4 , a)
a.reset()
print a.getvalue()
print json.load(a)
gives: 
$ python z1.py 
 "1": 3 "2": 4 
Traceback (most recent call last):
  File "z1.py", line 8, in <module>
    print json.load(a)
  File "/usr/lib/python2.6/json/__init__.py", line 267, in load
    parse_constant=parse_constant, **kw)
  File "/usr/lib/python2.6/json/__init__.py", line 307, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python2.6/json/decoder.py", line 322, in decode
    raise ValueError(errmsg("Extra data", s, end, len(s)))
ValueError: Extra data: line 1 column 8 - line 1 column 16 (char 8 - 16)
And the json module documentation does not provide any way of telling the "stream" decoder to just stop parsing after the first valid object. An alternative can be to use chunked transfer encoding and transfer each JSON object in a separate HTTP chunk. It would add the overhead of a little HTTP header per JSON object, which could be significant when generating lots of small JSON objects, but it would be an implementation that strictly follows HTTP and JSON standards to the letter in a clean way. Except:
  1. How do you stream HTTP chunks in WSGI? The WSGI specification's only mention of chunks is that if you produce a result as a generator of "chunks", they are just sent one after the other (effectively, concatenated) in the response stream.
  2. How do you read and process one chunk at a time with urllib/urllib2? I searched on Google but I only found people in tears.
Another alternative is to wrap the JSON objects in some custom container protocol, but then we are not talking standards anymore. Is this all you can do with the standard python library? Can this only be done by adding a dependency on a C library with two obscure separately maintained bindings, or by writing my own JSON parser? Ideas? Please mail enrico@enricozini.org and I'll add them here. Updates Wouter van Heyst mentions that he heard of ijson, which indeed has my exact use case in the synopsis, but after some research looks like a third binding for yajl. Aigars Mahinovs suggests using YAML instead of JSON. Indeed YAML supports streaming very well (it's even already supported in DDE, but last time I tried to do anything serious with it, PyYaml was slooow. Of course, this being the Python ecosystem, There Are Lots Other Ways To Do It And You Shall Be Patronised For Not Chosing The Proper One, but Debian only has PyYaml. Another itch with YAML is that there is no YAML parser in the standard python library, which requires clients to add an extra dependency. Jyrki Pulliainen has so far mailed my favourite solution: disable formatting in the encoder and separate JSON objects with newlines: 
we built a streaming JSON app as a part of our platform. However, we
didn't stream a single JSON object as that was a bit cumbersome. When
having syntax like:
[
  <json-obj1>,
  <json-obj2>,
  ....
  <json-objN>
]
The reader has some problems. Should it read the whole stream to find
the closing bracket ] even if the stream would me tens of megabytes?
Should it try to split on , and parse single objects? What if the
objects contain , so should there be some regexp magic?
We ended up streaming separate JSON objects separated by newlines
(\n). This way the reader can read until hitting \n or end of stream
and then split the result on newlines. The new stream looked like
this:
<json-obj1>\n
<json-obj2>\n
...
<json-objN>\n
This way we could efficiently stream hundreds of thousands of JSON
objects from server to client without reserving a load of memory. For
the WSGI part how to do this, the best would probably to do a
generator that would yield a single JSON object with the newline (we
didn't use WSGI back then). Reading can be done using any reader
capable of handling streams. We used pycurl for reading, but I can't
really recommend it as it is way too complicated for a simple task
like this :p
Hopefully this helps!
This means that if the client has no access to a JSON reader that supports a stream of concatenated toplevel objects, they can still trivially code it on top of any JSON library. This means that working clients can be written based just on Squeeze's python standard library. That's what I'm going to do; I'll probably implement it as a separate 'jsons' output format for DDE, because producing a JSON response with multiple toplevel obejcts somehow breaks the JSON standard. Thanks to all who provided this excellent information!

Matthew Palmer: Tethering to an Android phone

Whilst my philosophy with mobile phones has always been "if it can make and receive calls and SMSes, it's good enough for me", the geek in me occasionally lusts for a smartphone. So I now have an Android phone. Once you have a smartphone, you need a plan that includes data. Once you've got that, you realise that your USB mobile internet dongle is somewhat redundant, as long as you can get your laptop talk through the phone. It turns out, with Android, it's pretty easy. First, root your phone. There are a couple of tethering apps that don't require rooting, but the tradeoff is you end up having to install proprietary drivers in your laptop, and if I'm going to have a security hole somewhere, I prefer it to be in my phone, not my laptop. Rooting an Android phone is trivially easy with Universal Androot. Despite warnings I've seen to the contrary, it didn't even wipe my phone's apps and settings, although that might be a lucky coincidence. Backups, people! Then we need the tethering app -- I use Barnacle Wifi Tether, because it's open source, easy to use, and fully featured. Trivial to install from the market, you just start it up and it's ready to go. It acts as an ad-hoc wireless router, with a DHCP server and NAT to make life easy. In theory it supports WPA and WEP, but that doesn't appear to work for me. If anyone on the train is desperate enough to leech off my wireless, they're welcome to it. The hardest part, ironically, was getting the laptop to connect. Having never played with ad-hoc mode before, it took me a little while to sort out exactly what was supposed to happen, but within 5 minutes I'd worked it out and was pinging The Internet. Win!

5 January 2011

Christian Perrier: Becoming Libre

After reading about noise made around Ubuntu switching to LibreOffice, I just decided to look at what we have in Debian. And, of course, with no surprise, I found the work of our tireless Ren Engelhard, who built Debian packages for LibreOffice, in experimental (as, currently, unstable is targeted to the squeese release). I hope that Debian and Ubuntu efforts to get LibreOffice packages are synced...though the announcement pointed by Simon Phipps leaves me with huge doubts about this really happening. Hopefully, Ubuntu zealots or Canonical employees will prove me wrong. And, in the meantime...no more OpenOffice.org packages on my system. LibreOffice packages are here and meant to stay here.

Next.

Previous.