Search Results: "ag"

7 January 2011

Raphaël Hertzog: How to create custom RSS feeds with WordPress

WordPress has many alternate built-in feeds: per category, per tag, per author, per search-keyword. But in some cases, you want feeds built with some more advanced logic. Let s look at the available options. WordPress advanced built-in feeds You can create feeds for unions or intersections of tags, you just have to use a URL like /tag/foo,bar/feed/ (all articles tagged with foo or bar) or /tag/foo+bar/feed/ (all articles tagged with foo and bar). You can also have feeds excluding a category, although that requires you to know the category identifier (and hardcode it in the URL like this: /?feed=rss2&cat=-123 where 123 is the category id that you want to exclude). But there s no simple way to have a feed that excludes articles with a given tag. The best solution found involves creating a custom feed. I ll show you a variation of this below. Creating a custom feed
  1. First of, install the Feed Wrangler plugin, it will take care of registering our custom feeds with wordpress.
  2. Go to Settings > Feed Wrangler in your WordPress administrative interface and create a new feed, let s call it myfeed .
  3. You should now create a feed-myfeed.php file and put it in your current theme s directory. The initial content of that file should be this:
    <?php
    include('wp-includes/feed-rss2.php');
    ?>
  4. At this point, you already have a new feed that you can access at /feed/myfeed/ (or /?feed=myfeed). It s a complete feed like the main one.
Now, we re going to look at ways to customize this feed. We re going to do this by changing/overriding the default query that feed-rss2.php s loop will use. A feed excluding articles with a tag If you want to create a feed that excludes the tag foo , you could use this:
<?php
global $wp_query;
$tag = get_term_by("slug", "foo", "post_tag");
$args = array_merge(
        $wp_query->query,
        array('tag__not_in' => array($tag->term_id))
);
query_posts($args);
include('wp-includes/feed-rss2.php');
?>
That was relatively easy, thanks to the tag__not_in query parameter. Now you can further customize the feed by adding supplementary query parameters to the $args array. The documentation of query_posts details the various parameters that you can use. A feed excluding articles with a custom field (meta-data) I went further because I did not want to use a tag to exclude some posts: that tag would have been public even if it was only meaningful to me. So I decided to use a custom field to mark the posts to exclude from my new feed. I named the field no_syndication and I always give it the value 1 . This time it s not so easy because we have no query parameter that can be used to exclude posts based on custom fields. We re going to use the post__not_in parameter that can be used to exclude a list of posts. But we must first generate the list of posts that we want to exclude. Here we go:
<?php
global $wp_query;
$excluded = array();
$args_excluded = array(
    'numberposts'     => -1,
    'meta_key'        => 'no_syndication',
    'meta_value'      => 1,
    'post_type'       => 'post',
    'post_status'     => 'published'
);
foreach (get_posts($args_excluded) as $item)  
        $excluded[] = $item->ID;
 
$args = array_merge(
        $wp_query->query,
        array('post__not_in' => $excluded)
);
query_posts($args);
include('wp-includes/feed-rss2.php');
?>
A feed with modified content You might want to add a footer to the articles that are syndicated. I use the Ozh Better Feed plugin for this but it applies to all your feeds. You could do that sort of transformation only in your customized feed by using the WordPress filter named the_content_feed. Here s a simple example:
<?php
function myfeed_add_footer($content)  
        return $content . "<hr/>My footer here";
 
add_filter('the_content_feed', 'myfeed_add_footer');
include('wp-includes/feed-rss2.php');
?>
I ll stop here but obviously you have lots of options and many ways to tweak all the snippets above. They have been tested with WordPress 3.0.4. Note that in all those examples, I took care to not duplicate the code from feed-rss2.php, instead I used include() to execute it. That way my custom feeds will automatically benefit from all the future enhancements and fixes made by the WordPress developers. But if you have to modify the XML structure of your custom feeds, you can paste the content of feed-rss2.php in your file and change it like you want

One comment Liked this article? Click here. My blog is Flattr-enabled.

Axel Beckert: "peer holds all free leases" on both DHCP servers

At work we run a pair of ISC DHCP servers running Debian Lenny in a classical ISC DHCP failover setup which provide DHCP service to several subnets, some only with static IPs (e.g. for printers) and some with half static and half dynamic IPs. Today I got a call from a user that her laptop doesn t get an IP despite it s correctly registered in our MAC address database from which we generate the group sections of the dhcpd.conf. Everything looked fine, but every DHCPDISCOVER package got logged in the syslog on both servers like this:
Jan  7 14:34:39 dhcp1 dhcpd: DHCPDISCOVER from 01:23:45:67:89:ab via eth2: peer holds all free leases
Jan  7 14:34:39 dhcp2 dhcpd: DHCPDISCOVER from 01:23:45:67:89:ab via eth2: peer holds all free leases
Searching the web for this error message mostly results in mails which say If have this on one server but not the other, you soon run out of IP addresses , but none which mentions what happens if you got them on both sides. Following a coworker s idea of adding both servers to the search term, I found Debian bug #563449 (dhcp3-server: Incorrect peer holds all free leases log entries) which turned out as configuration error or at least unexpected configuration (machine was blocked from getting an IP on purpose) and misleading error messages. So I checked under which circumstances this computer would not get an IP despite it had a static IP configured:
  host somehost  
    hardware ethernet 01:23:45:67:89:ab;
    fixed-address 192.0.2.123;
   
That computer would not get an IP address in any subnet which has different IP range and no dynamic IP addresses. And even if I comment out the fixed-address setting, it wouldn t get an IP in any static-IPs-only subnet either. And *bingo*, that computer was plugged into the printer subnet which has only static IPs, e.g. in the 198.51.100.x range. So if you get the peer holds all free leases error message from both your DHCP servers, chances are very high that the mentioned MAC address should really not get an IP address on this network (as it does :-). The error messages are just somewhat misleading. Hope, this saves someone some time. :-)

Simon Josefsson: On Password Hashing and RFC 6070

The RFC Editor has announced a new document, RFC 6070, with test vectors for PKCS5 PBKDF2. The document grow out of my implementation of SCRAM for GNU SASL. During interop testing, more than one other implementation turned out to have mistakes in the PBKDF2 implementation. It didn t help that there weren t any stable test vectors for PBKDF2, so that we could do black-box testing of our PBKDF2 implementations against well-known and stable test vectors. Debugging this was time consuming. The document addresses this problem. So what is PBKDF2?

Briefly, the PBKDF2 is the state of the art way to convert a password string into a cryptographic binary key. There are many ways to design such an algorithm. To understand why PBKDF2 is designed the way it is we need some basic understanding. The typical attack you would be concerned with is if someone compromises your authentication servers and steals the credential database. This is what happened with the recent Gawker attack, where passwords were apparently stored in clear text. The first security improvement compared to storing passwords in clear text is to only store a hashed or encrypted form of the password. Hashing is preferable over encryption in this context, because with encryption it is possible to go back to the unencrypted form (if you know the key) whereas with a hash it is computationally infeasible to do so. The downside of hashing is that it is impossible to be compatible with all kind of authentication systems: the hash of the password in the database needs to be computed using the same algorithm as the system will be using. Unfortunately there are many different schemes out there to do password hashing, including the old Unix crypt(3) algorithm and the Windows LM-hash. You normally never know in advance what kind of systems your hashed credential database needs to work for, so you have to pick some set of algorithms and hope for the best. The next security improvements are iterations and salting. Iterations means to run the hash function not once, but hundreds or even thousands of times, on the input data (e.g., password). The extra time is hardly noticeable for the user, but for an attacker who is testing an entire dictionary of common passwords against your hash, making the work cost thousand times higher quickly makes the attack vector computationally infeasible. Salting means to include something other than the password in the input to the hash function, typically a random string. Without salting, a pre-computed rainbow table would provide a dictionary attack against all hashed password databases (one rainbow table per hashing algorithm). By using a sufficiently long and random salt, a rainbow table for one system is not applicable to another system. Since generating a rainbow table is time consuming, this adds another barrier to the attacker. PBKDF2 combines these three concepts hashing, iteration and salting into one construct, and it is designed flexible enough so that it can work on any pseudo-random function. In practice, I ve only ever seen HMAC-SHA1 be used. Some final words on the publication behind RFC 6070: Out of my earlier publications, this was the document that (by a large margin) spent the least time in the IETF process. You see various milestones of the document in the RFC 6070 datatracker history page. The time in the publication process was from 2010-08-09 to 2011-01-06. One month was in the AUTH48 state, of which the initial 20 days were waiting for me to respond. Thank you to my sponsoring AD, Sean Turner, for making this possible!

Daniel Baumann: Booting images from within EXTLINUX

Similar as people are doing with a seperate package for grub, Debians extlinux package as of version 2:4.03+dfsg-5 (and if you want a graphical boot menu, you will need syslinux-themes-debian version 4-1 or newer; both uploaded to experimental) handles booting images through memdisk out of the box now. After you have upgraded your packages, just do: Booting images from within EXTLINUX Update: After some more testing, EXTLINUX_MEMDISK defaults to true now.

Alexander Reichle-Schmehl: Release Critical Bug report for Week 01

The bug webinterface of the Ultimate Debian Database currently knows about the following release critical bugs:
In Total:613
Affecting Squeeze:93
Squeeze only:33
Remaining to be fixed in Squeeze:60
Of these 60 bugs, the following tags are set:
Pending in Squeeze:3
Patched in Squeeze:13
Duplicates in Squeeze:13
Can be fixed in a security Update:1
Contrib or non-free in Squeeze:2
Claimed in Squeeze:0
Delayed in Squeeze:0
Otherwise fixed in Squeeze:12
Ignoring all the above (multiple tags possible) 25 bugs need to be fixed by Debian Contributors to get Debian 6.0 Squeeze released. However, with the view of the Release Managers, 68 need to be dealt with for the release to happen. Please see Interpreting the release critical bug statistics for an explanation of the different numbers.

Paul Wise: Another year, another log entry

It has been almost a full year since my last log entry. It has been a busy work year, I attended some nice conferences and did minimal FLOSS stuff. On the work side of things I was a third of an Australian VoIP startup that came and went. I setup Debian servers, installed OpenSIPS and associated software, wrote OpenSIPS scripts, wrote peripheral software and did customer support. We had a good thing going there for a while, some fans on the Whirlpool forums but in the end there wasn't enough money for the requisite marketing and local market circumstances were squeezing Australian VoIP providers anyway. On the conference side of things I went to LCA 2010, the Thai Mini-DebCamp 2010, DebConf10 and FOSSASIA 2010. Had a great time at all of them. At LCA 2010 in windy Wellington, New Zealand the distributions summit organised by Martin Krafft was one of the highlights. It was dominated by Debian/Ubuntu talks but there were some other interesting ones, especially the one on GoboLinux's integration of domain-specific package managers. Also excellent were the keynotes given by Gabriella Coleman (Best & worst of times), Mako Hill (Antifeatures) and others, which I felt gave LCA an improved and very welcome focus on software freedom. There were quite a few Debian folks at LCA, it was great to hang out with them during the week and afterwards. Monopedal sumo with mako and others was hilarious fun. At the Thailand Mini-DebCamp 2010 in Khon Kaen, I was glad to see Andrew Lee (Taiwan) and Christian Perrier (France) again and meet Yukiharu YABUKI (Japan) and Daiki Ueno (Japan). In addition to the five international folks, there were quite a few locals, including Thailand's currently sole Debian member, Theppitak Karoonboonyanan. The event was hosted at Khon Kaen University and opened with my talk about the Debian Social Contract and the Debian Free Software Guidelines. This was followed by a number of talks about Debian package building, a 3-day BSP where we touched 57 bugs, a great day of sightseeing and talks about i18n, derivative distros, keysigning, mirrors, contribution and a discussion about DebConf. During the week there was also the usual beersigning, combined with eating of unfamiliar and "interesting" Thai snacks. After the conference Andrew and I roamed some markets in Bangkok and got Thai massages. Beforehand I also visited a friend from my travels on the RV Heraclitus in Chiang Mai, once again experiencing the awesomeness of trains in Asia, unfortunately during the dry season this time. I took a lot of photos during my time in Thailand and ate a lot of great and spicy food. As a vegetarian I especially appreciated the organiser's efforts to accommodate this during the conference. At DebConf10 in New York City, by far the highlight was Eben Moglen's vision of the FreedomBox. Negotiating the hot rickety subways was fun, the party at the NYC Resistor space was most excellent, Coney Island was hot and the water a bit yuck, zack threw a ball, the food and campus was really nice. Really enjoyed the lintian BoF, ARM discussions, shy folks, GPLv3 question time, paulproteus' comments & insights, wiki BoF, puppet BoF, derivatives BoF, Sita, astronomy rooftop, cheese, virt BoF, Libravatar, DebConf11, Brave new Multimedia World, bagels for breakfast, CUT, OpenStreetMap & lightning talks. Having my power supply die was not fun at all. Afterwards I hung out with a couple of the exhausted organisers, ate awesome vegan food and fell asleep watching a movie about dreams. One weird thing about DebConf10 was that relatively few folks used the DebConf gallery to host their photos, months later only myself and Aigars Mahinovs posted any photos there. At FOSSASIA 2010 in H Ch Minh City (HCMC) was a mini-DebConf. I arrived at the HCMC airport and was greeted by Huyen (thanks!!), one of FOSSASIA's numerous volunteers, who bundled me into a taxi bound for the speakers accommodation and pre-event meetup at The Spotted Cow Bar. The next day the conference opened at the Raffles International College and after looking at the schedule I noticed that I was to give a talk about Debian that day. Since I didn't volunteer for such a talk and had nothing prepared, the schedule took me by surprise. So shortly after an awesome lunch of Vietnamese pancakes we gathered some Debian folks and a random Fedora dude and prepared a short intro to Debian. The rest of the day the highlights were the intro, video greetings and the fonts, YaCy and HTML5 talks. The next day the Debian MiniConf began with Arne Goetje and everyone trying to get Debian Live LXDE USB keys booted on as many machines in the classroom as possible (many didn't boot). Once people started showing up we kicked off with Thomas Goirand's introduction to the breadth of Debian. Others talked about Debian pure blends, Gnuk and building community and packages. The second last session was about showing the Vietnamese folks in the room how to do l10n and translation since Debian had only one Vietnamese translator (Clytie Siddall). After manually switching keyboard layouts (seems LXDE doesn't have a GUI for that) on the English LXDE installs, the two Cambodian folks were able to do some Khmer translation too. This was a great session and it resulted in two extra Vietnamese translators joining Debian. It went over time so I didn't end up doing my presentation about package reviewing. We rushed off to a university where the random Fedora ch^Wambassador was hosting a Fedora 14 release party in a huge packed classroom. There were a lot of excited faces, interesting and advanced questions and it was in general a success. Afterwards we had some food, joined up with some other speakers and ended up in a bar in the gross tourist zone. On the final day we hung around in the Debian room, went downstairs for the group photo and final goodbyes. Later we found a place with baked goods, coffee and juices and navigated the crazy traffic to a nice local restaurant. The next morning Arne & I went to the airport, others went on a Mekong Delta tour and Jonas hung out with the organisers. I took less photos than at other events but got a few interesting ones. I avoided doing a lot of FLOSS stuff over the last year, I hope to work on some things in the coming months; I'm also planning some interesting travel and acquiring some new technological goods, more on those in some later posts.

6 January 2011

Michael Prokop: Booting ISO images from within GRUB2

You might be aware of GRUB s loopback option for booting an ISO, I wrote about it in Boot an ISO via Grub2 more than a year ago. A few months ago Goswin von Brederlow came up with this idea:
grml functions great as rescue system. So it would be nice to have a boot entry for it in grub instead of having to go look for the CD when needed. To make installing and updating simple it would be great if one could install this as a normal debian package which would register itself in grub (and maybe lilo too).
What a lovely idea: no need for a CD or USB stick as long as the bootloader and the harddisk are still working. Minimal manual intervention needed to keep it up2date and working sounds like the perfect rescue system. 8-) Now since end of 2010 a new stable version of sysadmin s favorite live system (AKA Grml 2010.12) is available and the great news is that we came up with two independent solutions known as grml-rescueboot and grub-imageboot. Having shipped them to one of my customers already I d like to write some words about it and why you should also consider using it on all your systems where GRUB2 is available. grml-rescueboot uses GRUB2 and its loopback feature for booting. Grml as well as Ubuntu provide all what s needed out-of-the-box. If you re interested in the details check out Jordan Uggla s excellent Loopback.cfg webpage in the supergrubdisk wiki. The best about it: you can provide custom bootoptions automatically when booting Grml. For example just set CUSTOM_BOOTOPTIONS= ssh=grml2011 in /etc/default/grml-rescueboot and the Grml rescue system will automatically start the OpenSSH server with the specified argument as password for user grml. Of course you can use all the other nifty bootoptions like scripts/netscript/ to further customize your Grml rescue system. [A note to other distributions like Ubuntu: I'd be interested to establish a mechanism to pass kernel options for loopback boot in a standardized way, drop me a note if you're interested in this.] grub-imageboot uses GRUB2 and syslinux memdisk to boot ISOs and floppy images. It doesn t rely on the loopback feature but instead maps the ISO into the memory directly. This is great for Linux ISOs that can t and won t support the the loopback feature (and have everything inside their initrd so the ISO doesn t have to find itself during booting), like BIOS or firmware updates. Also for example FreeDOS and Alpharev are known to work just fine. But sadly memdisk ISO emulation doesn t work with all Linux systems, as documented in the syslinux wiki. The good news is that Grml supports the memdiskfind/phram/mtdblock approach out-of-the-box. As far as I know therefore Grml is the first Debian based live system supporting memdisk ISO boot, though my patches already went to the Debian-Live project so if you re using live-boot >=2.0.14-1 or >=2.0.12-1+grml.04 your live system should be able to boot via memdisk ISO emulation as well. Summary: If you want to boot a Linux system which supports loopback.cfg use grml-rescueboot as its a more powerful tool to boot Linux live systems like Grml and Ubuntu. If you want to boot non-Linux systems (BIOS-/Firmware-Updates/FreeDOS/ .) use grub-imageboot. Alright how do you deploy this solution? Just grab and install grml-rescueboot and/or grub-imageboot. To deploy grml-rescueboot (adjust $VERSION if you want to use another flavour):
# choose Grml version:
VERSION=grml64-medium_2010.12
# create directory
mkdir -p /boot/grml
# download and verify ISO
cd /boot/grml
wget download.grml.org/$ VERSION .iso ,.md5 
md5sum -c $ VERSION .iso.md5
To deploy grub-imageboot:
# choose Grml version:
VERSION=grml64-medium_2010.12
# create directory
mkdir -p /boot/images
# download and verify ISO
cd /boot/images
wget download.grml.org/$ VERSION .iso ,.md5 
md5sum -c $ VERSION .iso.md5
That s it! Now when running update-grub you should get something like:
# update-grub
Generating grub.cfg ...
Found linux image: /boot/vmlinuz-2.6.36-grml64
Found initrd image: /boot/initrd.img-2.6.36-grml64
Found memtest86+ image: /boot/memtest86+.bin
Found memtest86+ multiboot image: /boot/memtest86+_multiboot.bin
  No volume groups found
Found Grml ISO image: /boot/grml/grml64-medium_2010.12.iso
Found memdisk: /boot/memdisk
Found iso image: /boot/images/grml64-medium_2010.12.iso
done
Voil , when rebooting your system you should see something like: Screenshot: grml-rescueboot The Grml Rescue System entry is what grml-rescueboot provides and Bootable ISO Image is what s provided by grub-imageboot. Just select the entry you d like to use, press enter and and you should get the bootsplash of the according ISO. BTW: I ve tested this with Ubuntu 10.10 too, grml-rescueboot works out-of-the-box with the Ubuntu ISO as well, it just doesn t support the memdisk ISO boot by grub-imageboot (yet). Tip: if you want to use grml-rescueboot and grub-imageboot with the same ISOs without having them twice on the disk just point the configuration option IMAGES in /etc/default/grub-imageboot to the according directory (like /boot/grml). Note: if you re using the ext 2,3,4 filesystem on the partition that s being used for /boot please be aware that you need GRUB >=1.98+20100804-12 (newer version is already available in Debian/unstable and hopefully migrates to squeeze in time) or GRUB >=1.99~20101122-1 because of the INDIRECT_BLOCKS support, see #543924 for the details. PS: We think about providing grml-rescueboot and grub-imageboot within official Debian. If you re interested in seeing integration support for other distributions as well please help and drop us a note.

Michael Prokop: Simple DNS in chroots

Update: Ulrich mru Dangel suggested pdnsd as nice alternative to dnsmasq and Cyril KiBi Brulebois pointed out, that it s not necessary to invoke dnsmasq after fresh installation as it s running by default then I adjusted the text accordingly, thanks for the pointers! If /etc/resolv.conf doesn t provide any nameserver entries glibc[1] will automatically initialize the nameserver to the loopback address. This is nice for dealing with chroots without having to manually edit resolv.conf to get working DNS. Just install a basic DNS forwarder like dnsmasq ( apt-get install dnsmasq ) or if it s already on your box just start it ( /etc/init.d/dnsmasq start ). That s it. Now when chrooting into a system without an existing resolv.conf configuration it will give you a working DNS setup without any further work. PS: Grml ships dnsmasq by default and /etc/init.d/dnsmasq start or Start dnsmasq will work out-of-the box. In Ubuntu the dnsmasq package is available only through the universe repository but the dnsmasq-base package (providing e.g. the init script) is shipped by default, so replace the /etc/init.d/dnsmasq command with a simple dnsmasq there. [1] At least the [e]glibc versions provided on Debian and Ubuntu are known to provide this glibc extension. Other libc implementations like dietlibc and uclibc don t seem to provide it, so don t strictly rely on this feature but use it as the icing on the cake.

Christian Perrier: [life] [running] 2010 summary and early 2011 plans

So, 2010 ended and it's now time for me to make a small summary for my running activities. At first, it's more and more obvious that running became an increasing part of my life. Sometimes eating the free time I was using for Debian work....and sometimes just filling holes in my daily activities, where I was just doing nothing..:-) According to my running watch(es) records, I ran for 2172km this year in about 196 hours (8.25 days) at an average speed of 11.1km/h. Indeed, the speed ranges from 10km/h in the hardest trail runs up to 13.5km/h in the fastest (and shortest) asphalt runs. I actually never run at "slow" paces, which is probably not the best for training as all official training plans do include slow runs. But, eh, this is how it works for me..:) I ran 12 official or semi official races, ranging from 5km cross-country races up to marathons. More proeminently speaking, I ran 3 half-marathons, 2 marathons and one long distance trail race (33km) that is worth being counted as a marathon..:). I also broke my personal best times for 10km (48'05", ran alone), half-marathon (1h37') and marathon (3h40'). Quite a good year, indeed..:-) I'm currently busy building my plans for 2011. Some are already decided: In general, I'll probably put more focus on trail and ultra (above martahon) runs. 2012 could then be the year of my first 100km race..:-) What will be fun will be reading this in January 2012 and see what I have completed among all these goals..:-)

John Sullivan: Life imitates Seinfeld

CLEMSON, S.C. (CBS/WSPA/AP) A man was struck by an SUV in Clemson Monday evening after police say he was playing a real-life version of the video game "Frogger" on a highway.

Game Over: Man Playing Real-Life Frogger Struck by SUV, Say S.C. Cops - Crimesider - CBS News ...as it always does.

Michal &#268;iha&#345;: Imported old Czech posts

Just for simple reason having all my blog posts in single place, I've decided to import all Czech blog posts from abclinuxu.cz into this blog. You can now find them in archives, all of them being tagged with Czech. There is no intention to add new blog posts in Czech for now. In case I would might change my mind sometimes in future, all English posts are tagged with English.

Filed under: English Website 1 comments Flattr this!

Gunnar Wolf: Back with Ruby on Debian polemics

Once again, a polemic regarding how to properly integrate the Ruby language and libraries with the Debian distribution has been ignited. Similar arguments were presented in November and December 2008 and September 2010 And excuse me for just refering to my own blog, but there are links to some other posts from there... and there will surely be many others I just missed. There were even nice collaboration attempts (such as DebGem, announced in January 2009... But which apparently never stuck on, as it is still listed as in its public Beta period and lists only support for Debian 4.0 and Ubuntu 8.04 and 8.10). Some days ago, while I was on vacation, I received a mail from Lucas Nussbaum expressing the burnout he had been suffering from this situation. Some days after that, he posted a corresponding blog post - Giving up on Ruby packaging. The comments following his post are most interesting The one comment I'd like to highlight (noting I skimmed a whole deal of them) is by Paul Brannan, one of the original RubyGems authors. Yes, possibly the design criteria for Gems included some mutually-conflicting goals, and they cover some of Debian's goals. Unfortunately, this conflicting criteria... Was resolved to the opposite way we would have wanted (specifically, that a sysadmin should be able to install software with the same tools he was already familiar with). And of course, there are deeper disagreements, which are rightly stemmed from different priorities: Debian (+derivatives) is meant to make user's and sysadmin's lifes simpler. Gems are meant to make developers' lifes simpler. And while all developers are some sort of users, and sysadminsthe reverse is (obviously) not so. A developer maintaining several different codebases will surely suffer (well, this particular developer has surely suffered) if he has to maintain them all using coherent versions of system libraries. When a system is programmed, its developers want to use the greatest, latest tools to offer the best experience and functionality. And it's natural for libraries to evolve over time. However, any sysadmin will grind their teeth at the prospect of having many different versions of libraries, as the Gem model proposes. Why? Well, I was bitten by a minor example of it Bug fixes do not get backported. Of course, if simple bugs are not backported, security and stability of the system as a whole suffers. Often, new software versions will not only add functionality, but improve on how things are done. Not only bugs get corrected, but we get better response times, better memory handling, etc. It might sound harsh to say this, and even more, as a developer it feels I am talking against myself But while development time should be minimized during the system design/implementation, once systems are in production, it should be used to allow for friendlier sysadmin relations. I have to wear both hats at my real-life job. So, what is my take on this? Of course, blaming myself for choosing the wrong version is a no-go. When evaluating a library before using it in my projects, I try to look at its history and API-stability. APIs that change often speak about immature libraries which are still trying to get the right way to implement their functionality; just-started projects might be great and revolutionary, but they do not yet show any kind of long-term committment... And can lead (and have often led me) to painful rewrites when the Next Big Thing is reached. As for the users: Even if your favorite language has the best and friendliest distribution method, users just do not care about what language was a particular piece of software implemented in. Users want to be able to install and uninstall with their system's usual tools. Directly using Gems, CPAN, PEAR and whatnot is just unnatural for users, however convenient they are for developers. Distributions offer a non-technical advantage in this regard as well: A human filter. As an example, as I write this there are 19392 Ruby Gems and 19092 Perl modules in CPAN (note that CPAN stores some older versions but discourages authors from keeping too many - So no, they are still very different in size). Debian has around 30,000 packages. Why is that? Because all Debian packages _must_ be human-generated, human-reviewed, human-submitted. This means, a person must think each packaged piece of code is worth packaging, is stable enough and provides value to users as it is, and is fit for being part of a stable release. I am not saying with this that 90% of CPAN and Gems are crap I am saying that they are probably early implementations, to be installed, tested and improved by developers, and still not apt for general public use. Or maybe not interesting enough to be packaged as a service for the non-techie public at large. Oh, and one last point: Ruby is not a new language anymore. It is a mature, powerful language with different implementations, every time more stable... But it is a language deeply affected by the (not so new anymore either) the appearance of Rails. Why do I say this? Because although the language is in no way tied to Web development, many of its strongest uses are Web-oriented. How does this affect the current discussion? Well, because many people argue that users are no longer needed to install the software. Web systems are installed by sysadmins and used via a Web browser, and sysadmins are expected be more skilled than casual users. Still, in Debian (and in other distributions, surely) we try to make sysadmin's lives simpler I have (again, talking out of personal experience) installed several webapps (and system tools, and whatnot) for which I never followed any instructions besides aptitude install foo Using different languages, frameworks, and so on. Can I troubleshoot their installs? Probably, as there is a common logic for how the distribution I have chosen and specialized in works. Can I find causes for bugs in them? Possibly, although there are some languages and frameworks I dare not touch. Can I get help on getting them out of a tight spot? Surely, as there is a central bug tracking system for my distribution And one of the maintainer's tasks is to separate the problems related to the distribution (packaging, installing, simple user questions and misconceptions) to those derived from real bugs upstream. Anyway I am not saying that our way is the best way. No, by a long shot. Again, developers should have an easy, convenient way for installing whatever they want to play with. And to publish it without jumping through hoops. With this post, I'm only trying to express again why Debian works the way it does. And hope for better cooperation in the future. And as for several comments of what I read in Lucas' post, I think that there is interest for this synergy among some of the most committed Ruby people.

Eugene V. Lyubimkin: cupt: 2.0.0~alpha3

Notable changes since 2.0.0~alpha2:

Full changelog is available in debian/changelog, as usual.

binary packages for i386, source package and README

Daniel Kahn Gillmor: Liberating Knowledge: A Librarian's Manifesto

A friend just pointed me to Liberating Knowledge: A Librarian's Manifesto for Change by Barbara Fister, an academic librarian who also happens to write mystery novels. She has an excellent perspective on the meaning of libraries, and the tradeoffs involved with the current societal trend toward privatizing knowledge through so-called "intellectual property" regulations. In a great critique of the passivity of academia and libraries in the face of attempts at intellectual enclosure by private corporations, she writes:
This uninformed indifference is laying the groundwork for a new tragedy of the commons: a world in which knowledge is turned into intellectual property, monetizied, and made artificially scarce.
She closes with a six-point manifesto that begins:
Liberation bibliography arises out of outrage at the injustice of the current system. It s not about saving money, it s about the empowering nature of knowledge and the belief that it shouldn t be a luxury good for the few.
The article abounds in examples of heinous arrangements in the current system that seem to be accepted as standard procedure, and clear thinking about what the actual tradeoffs are (and how we, as a society, are making them poorly). If i had one objection, it would be that she neglects to mention increased surveillance as one of the problems that come with privatization of knowledge. Our abilities to read privately and anonymously, and to correspond confidentially are at risk because of these systems of control. Anyway, I'd love to see more open allegiances between librarians and free software folks; the ideals and struggles are very much in parallel. Go talk to your librarian friends about this stuff today!Tags: academia, freedom, intellectual property, libraries

Enrico Zini: Streaming JSON objects with python

Streaming JSON objects with python Dear Lazyweb, Suppose I'd like to create an HTTP service that gives you a looong list of JSON objects. For example, one JSON object for every package in Debian. One way I could do that is to just transfer a big, dozens of megabytes JSON array with one element per package. With the standard json module in python, this requires the client to buffer all the data in memory for decoding, as the JSON result would be something like this:
[
    info for package 1  ,
    info for package 2  ,
  ...dozens of megabytes...
    info for package 30000  ,
  ...
]
However, it would be reasonable to engineer things so that the client can process packages one at a time, as they are produced. This can dramatically reduce client memory usage, as well as remove a large latency between the start of the request and the start of processing. One trivial idea would be to just send JSON objects one after the other, and sort of call load multiple times. It doesn't work:
import json
from cStringIO import StringIO
a = StringIO()
json.dump( 1:3 , a)
json.dump( 2:4 , a)
a.reset()
print a.getvalue()
print json.load(a)
gives:
$ python z1.py 
 "1": 3 "2": 4 
Traceback (most recent call last):
  File "z1.py", line 8, in <module>
    print json.load(a)
  File "/usr/lib/python2.6/json/__init__.py", line 267, in load
    parse_constant=parse_constant, **kw)
  File "/usr/lib/python2.6/json/__init__.py", line 307, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python2.6/json/decoder.py", line 322, in decode
    raise ValueError(errmsg("Extra data", s, end, len(s)))
ValueError: Extra data: line 1 column 8 - line 1 column 16 (char 8 - 16)
And the json module documentation does not provide any way of telling the "stream" decoder to just stop parsing after the first valid object. An alternative can be to use chunked transfer encoding and transfer each JSON object in a separate HTTP chunk. It would add the overhead of a little HTTP header per JSON object, which could be significant when generating lots of small JSON objects, but it would be an implementation that strictly follows HTTP and JSON standards to the letter in a clean way. Except:
  1. How do you stream HTTP chunks in WSGI? The WSGI specification's only mention of chunks is that if you produce a result as a generator of "chunks", they are just sent one after the other (effectively, concatenated) in the response stream.
  2. How do you read and process one chunk at a time with urllib/urllib2? I searched on Google but I only found people in tears.
Another alternative is to wrap the JSON objects in some custom container protocol, but then we are not talking standards anymore. Is this all you can do with the standard python library? Can this only be done by adding a dependency on a C library with two obscure separately maintained bindings, or by writing my own JSON parser? Ideas? Please mail enrico@enricozini.org and I'll add them here. Updates Wouter van Heyst mentions that he heard of ijson, which indeed has my exact use case in the synopsis, but after some research looks like a third binding for yajl. Aigars Mahinovs suggests using YAML instead of JSON. Indeed YAML supports streaming very well (it's even already supported in DDE, but last time I tried to do anything serious with it, PyYaml was slooow. Of course, this being the Python ecosystem, There Are Lots Other Ways To Do It And You Shall Be Patronised For Not Chosing The Proper One, but Debian only has PyYaml. Another itch with YAML is that there is no YAML parser in the standard python library, which requires clients to add an extra dependency. Jyrki Pulliainen has so far mailed my favourite solution: disable formatting in the encoder and separate JSON objects with newlines:
we built a streaming JSON app as a part of our platform. However, we
didn't stream a single JSON object as that was a bit cumbersome. When
having syntax like:
[
  <json-obj1>,
  <json-obj2>,
  ....
  <json-objN>
]
The reader has some problems. Should it read the whole stream to find
the closing bracket ] even if the stream would me tens of megabytes?
Should it try to split on , and parse single objects? What if the
objects contain , so should there be some regexp magic?
We ended up streaming separate JSON objects separated by newlines
(\n). This way the reader can read until hitting \n or end of stream
and then split the result on newlines. The new stream looked like
this:
<json-obj1>\n
<json-obj2>\n
...
<json-objN>\n
This way we could efficiently stream hundreds of thousands of JSON
objects from server to client without reserving a load of memory. For
the WSGI part how to do this, the best would probably to do a
generator that would yield a single JSON object with the newline (we
didn't use WSGI back then). Reading can be done using any reader
capable of handling streams. We used pycurl for reading, but I can't
really recommend it as it is way too complicated for a simple task
like this :p
Hopefully this helps!
This means that if the client has no access to a JSON reader that supports a stream of concatenated toplevel objects, they can still trivially code it on top of any JSON library. This means that working clients can be written based just on Squeeze's python standard library. That's what I'm going to do; I'll probably implement it as a separate 'jsons' output format for DDE, because producing a JSON response with multiple toplevel obejcts somehow breaks the JSON standard. Thanks to all who provided this excellent information!

5 January 2011

Wouter Verhelst: Substituting gdb debug paths

Gdb is a debugger with many, many, many(1) options. Some of them are rather obscure, others... less so. While it's always had the ability to read source files and do useful stuff with them, recent versions of gdb have gained another interface (which you get by calling "gdb -tui") that makes it a bit easier to watch the source as you step through the code. Additionally (and I don't know how long that feature's been there), you can do some interesting things with source paths. Such as substituting a source path prefix by another prefix -- useful if you've moved sources around. This is accomplished by way of the 'set substitute-path' command:
(gdb) set substitute-path /home/buildd /home/wouter
Now whenever gdb is told by the debugging info to search for a file /home/buildd/foo/bar/baz.c, it will instead search for /home/wouter/foo/bar/baz.c. Very useful. One problem with the above approach, however, is that you need to know what the original path is the /home/buildd in our example. If you've built the binary yourself, you just need to remember where you built it, and you're all set. But what if you haven't built the binary yourself? I used to use strings for that, and look for any likely candidates, but that isn't very reliable and requires a bit of hunting through many many lines that just happen to look like strings to what really is a fairly naive program. Additionally, it doesn't work when the debugging symbols are in a separate file (as implemented by dh_strip), amongst others. But no worries! There's a simple solution. Rather than using strings, use readelf -w instead, and grep for DW_AT_comp_dir, which will give you the correct source directory to substitute. I'm happily single-stepping through a library now. If that isn't nice. 1 no really, many

Iustin Pop: More Haskell

It's no secret to the people I around me that over the past two years I've become a Haskell fan, to put it mildly. I mean, what's not to like? Just in the past month: The downside is that whenever I feel Ok, now I kind of know my way around Haskell , I learn of another thing that completely resets how I feel about my Haskell understanding. The most recent example is Template Haskell. I mean, there are people who already blame Haskell for its arcane syntax, which I like. But Template Haskell goes a step forward and allows you to use a Haskell library to write Haskell code, with an even more contorted syntax or at least that how it looks for now to my eyes. Gladly this is not easily doable in a recursive way (meta-TH code writing TH code that will write Haskell code), otherwise heads would explode (at least mine). But even though it seems very complicated, I feel that the power that it enables (a macro system that generates code that will be statically checked) is removing one of the last big advantages that (for me) Python had over Haskell: it's easy to do class modification in Python and introspection (e.g. via metaclasses) to automate boiler plate away, but it's of course not type safe (so unittests galore). With TH, you can do the same customization of data types (and not only) and compile-time introspection (called reification) to alter (some) of the properties of the respective type, for example automatic instance derivation, so one less reason to say this particular problem is harder to do in Haskell than in other languages . I haven't yet used TH for my own code and until I actually use it extensively I won't be sure it's actually a good replacement for throw a __metaclass__ in there and automate the boring parts away . But in any case, it looks to be yet another awesome tool in the Haskell programmer's toolbox, which I can't wait to use.

Christian Perrier: Becoming Libre

After reading about noise made around Ubuntu switching to LibreOffice, I just decided to look at what we have in Debian. And, of course, with no surprise, I found the work of our tireless Ren Engelhard, who built Debian packages for LibreOffice, in experimental (as, currently, unstable is targeted to the squeese release). I hope that Debian and Ubuntu efforts to get LibreOffice packages are synced...though the announcement pointed by Simon Phipps leaves me with huge doubts about this really happening. Hopefully, Ubuntu zealots or Canonical employees will prove me wrong. And, in the meantime...no more OpenOffice.org packages on my system. LibreOffice packages are here and meant to stay here.

Dirk Eddelbuettel: The kindness of strangers

A couple of days ago, an email arrived in which I am kindly asked to provide my PayPal coordinates. And before you dear reader start to laugh, it was actually perfectly genuine and came from a very happy user of the Finance::YahooQuote Perl module. DJ Padzensky wrote this well over a decade ago, and I first introduced and then maintained it as a package in Debian for what is now almost twelve years. I have also been acting as upstream maintainer for the last eight or so years providing the occassional patch to keep up with changes at the Yahoo! server side. After a short reflection, I decided to keep it simple. I undusted a wishlist I had once created over at Amazon, yet never publicised. And amazingly, within minutes of minutes of passing the URL on, I received a cheerful email stating that three books are their merry way. Appropriately enough, two were from Springer's useR! series and another one was a brand new GPU programming title. And they all arrived yesterday. I am still stunned. So with that, my time to say thanks! This was a genuinely nice gesture, and I really appreciate it. I have been doing open source work for over fifteen years now, and I would not still be doing it if I did not enjoy it myself. I continue to learn a lot in the process, and have had a chance to meet lot interesting folks. Yet it is also nice to get a little Thanks every now and them.

4 January 2011

Yves-Alexis Perez: gvim remote server and workspace-specific editor window

These days, I've been used to a very specific $EDITOR usage. I use my (Xfce) workspaces as context switch. Basically, each time I need to start a new activity I open a new workspace, open some terminals, maybe a browser, an editor (gvim) window etc. Then when I'm done, I delete that workspace. I can have multiple activities at once, and switch from one to another depending on various stuff. On those workspaces, I'd like to be able to have only one editor window, even when I open new files using gvim foo.c . But I'd like to have one editor window per workspace, not one global editor, since I'd like the editor to be context-specific too. I recently heard about the gvim remote stuff, and I've came with the following gvim function (zsh syntax):

function gvim()
desktop=$(xprop -root -notype -f _NET_CURRENT_DESKTOP 0c '$0+\n' _NET_CURRENT_DESKTOP)
if [ "$#" -ge "1" ];
then
gvimargs="--remote-tab-silent"
fi
=gvim --servername $desktop $gvimargs $*
The xprop trick is a hackish way to stick one gvim window per workspace, it might need to be adjusted if one has a two monitors setup and want to be able to have one editor per monitor per workspace. Basically, that snippet checks if a gvim server already exists with the current workspace name. If yes, it opens the file(s) in it, if not it run a new gvim server and opens the file there. It works more or less fine, but it has one problem: it fails badly when one needs to give args to the gvim call. Those args are usually placed *before* filenames, so in my case it won't work. One way to fix this would be to loop on all the args, check if the arg is an existing file, in case add it to $files. If not, add it to $args, then call gvim with the correct order. Works, except when you want to edit a new file, where you're back to point one. All in all, it'd work, but it's not really a nice way to do it, imho. So I'm calling for help, in case anyone has an idea about that. Basically, is there a way to say to gvim always open files in a remote server, run it if it's not already running using the config file and not an argument (so I don't mess with the command line). I do need the servername to be desktop specific though, so it's still a difficulty. NOTE: I know that emacs has some kind of remote server habilities too, but I'm not sure how it works and if it'd be possible to do that in emacs, and I'm not really an emacs user and don't really intend to switch. If you have any idea, feel free to comment (by mail), I might do a later post when I have an enhanced solution.

Next.

Previous.