Lenny general freeze is approaching, so I'm walking thru my packages bugs and I found this one reported against fdupes.

I got 4Gb of memory, so how to reproduce an OoM? Let's start with a rather large file base, but I need a script to generate it. With 4 cores and a fast Raptor hd, I want something to stess them while learning something: Python + ParallelPython was the answer!

I wrote a small script that takes 3 parameters: the destination directory, the number of dirs to create and the number of files (with random contents) to created inside each of the created dirs. I ran it with 1000 dirs and 1000 files, so it generated 1M files set to work on.

But even with so many files, fdupes ran fine... At that point I realized that ulimit could be my friend, and the "-v" option would have saved me some time, but without the fun to write the script :)

If you don't know pp, here is a little explanation of the main part of the script (I forgot to comment the script well enough):

• job_server = pp.Server() is the pp job server: with no parameter, it allocates execution slots for the number of cores/CPUs on your system (you can force more/less)
  
• jobs.append(job_server.submit(gen_dir_set, (dirname+'/'+str(i),fileno), (), ('random',))) appends to jobs list the function returned by job_server.submit: this command takes the function to execute, the parameters tuple, the functions called by the first one (here empty) and the tuple of needed modules for the called function (here random).
• for job in jobs: job() execute each item in jobs: they are functions that return the results for every jobs (here they are discarded, but you could aggregate them to obtain the final result)
• job_server.print_stats() prints some stats about jobs executions (I love stats :D )

ParallelPython is more powerful than this: for examples, it allows to send your jobs to job nodes, machine dedicated to execute tasks (while you keep your own machine as the administration one), and more! Give it a try ;)

I was on trains these days, so I got some time, and what's best then take xchat log of #d-d-c and print some graphs? well, there are tons best, but that's what I chose to do...

The script to generate the graph is here (I know, it's not beautiful, you can finds many points to improve it, etc etc) and the graph it generates is (run a couple of minutes ago):


Guess what's NEW and what's RC bugs :)

It lakes dates, legends or so; if someone is interested I can cron its generation (or lamby can take the script, it's Python+matplotlib, and use it somewhere).

I'm just bouncing the news from Jesse Noller blog about Guido van Rossum accepting the PEP for pyprocessing.
There were a lot of discussions about pyprocessing inclusion in Python (here a couple of blog posts and the mail to python-dev), and now it finally made it! I'm really happy since Richard is a very prepared and responsive guy.

Well, it's just by chance I co-maintain python-processing in Debian :)

On a slow line (like mine, @56k) even updating pbuilder could be a problem (or a really boring task):

[host]$ sudo pbuilder update
[...]                                                      
Get:4 http://debian.fastweb.it unstable/main Packages [5302kB]

5Mb are enough to do this trick (NOTE: it will work only if you machine and the pbuilder use the same mirror). First, update your box apt packages list with

[host]# apt-get update

once done, you'll find in /var/lib/apt/lists/ the packages list we need. Let's copy those files in a new location:

[host]# cp -arp /var/lib/apt/lists/ /tmp/

then, login into the pbulder chroot with:

[host]$ sudo pbuilder --login --bindmounts /tmp/lists/ --save-after-login

(/tmp/lists/ will be mounted inside the chroot with the files we've just copied, and --save-after-login will save our changes once done). Now

[chroot]# cp /tmp/lists/*astwe* /var/lib/apt/lists/

to copy the lists for Fastweb mirror only (the one I'm using), now

[chroot]# apt-get update
Get:1 http://debian.fastweb.it unstable Release.gpg [189B]
Get:2 http://debian.fastweb.it unstable Release [86.4kB]
Get:3 http://debian.fastweb.it unstable/main Packages/DiffIndex [2038B]                                                       
Fetched 88.7kB in 12s (6962B/s)                                                                                               
Reading package lists... Done
[chroot]#

now only few files are downloaded.

I used this trick earlier, in cases where creating or updating the apt packages list for pbuilder was a problem, since the files always resulted corrupted.

Hi all!

This is my first post on Planet!

I'm an italian guy working in Debian since a couple of years (but I'm a user since 5 years).

I maintain some packages (and I'm a DM), and I'm particular active in Python stuff (you can find me often on #debian-python@OFTC).

I'm in the NM queue (P&P1 passed, waiting for a reply on P&P2 :D ).

You'll hear back from me soon!

I just added two new developers to the reportbug project on Alioth, Sandro Tosi and Y Giridhar Appaji Nag, who seem to be enthusiastic about working through the big backlog of bug reports and wishlist requests associated with the package. Since my Debian-related interests are largely elsewhere these days, mostly focused on R stuff that has a more tangible relationship with my research (and by extension my future job mobility and/or tenurability) and a few other infrastructural things in Debian (primarily LSB support and printing), I think this is a good development overall. But still, when you’ve been hacking away at something for almost nine years it’s hard not to develop a bit of a sentimental attachment to it. I still plan to be doing some hacking away at reportbug, but hopefully the new blood can take the lead in terms of day-to-day maintenance while I work on some of the desperately-needed code refactoring issues with the software.

Paid work burnt all my free time recenlty, that's why there wasn't a rc-bug-a-day for two days. I'm back with two removals, a couple of gcc-4.3 FTBFSes fixed. And 5 uploads for RG bugs from Sandro Tosi that were rotting in my mail client. The bad news for tonight is that gcc-4.3 FTBFSes are now RC, since it's the default compiler on several architectures. That's almost 40 new RC we now have to deal with. The fun part will be that those RC will probably hinder some transitions to testing. Maintainers, please, fix your packages. There are still too many packages with RG bugs on them, open for many days, from maintainer that are otherwise quite active on IRC or the lists

Today's RC bug was 470462, removal of a package that as a patch for a FTBFS and absolutely no maintainer answers (#417047). I'm also quite excited because I have a mail with 9 nmudiff's for RG/RC bugs waiting to be processed from Sandro Tosi. I want MOAR !

happy newS year So this was Christmas, new year, et al. Too bad Murphy struck again on the first day of my (theoretical) vacation, delivering me a nice flu (not yet completely absorbed). Nevertheless I managed to do something funny in the last few days:

• I remembered that I do have, as all other DDs, a personal lintian page and I have thus spent some time fixing warning ignored since ever (almost all uploads are still in delayed queues however); while I was at it I also decided to "destroy" my lintian page moving my name to Uploaders for packages that still weren't co-maintained. (/me eager to see this implemented, go Russ, go!)
• I have fixed the vcs field stats page to consider as equal Vcs-Svn fields pointing to URLs like svn://svn.debian.org/foo and svn://svn.debian.org/svn/foo which are indeed the same final path on alioth. That's why in the graph you now see an inversion in the relative positions of field with wrong value and correct field, in case you were wondering. Thanks to Sandro Tosi for the hint.
• I have finally managed to leave the 18th century my mail set-up was living in: goodbye pop + fetchmail + mutt, welcome imap4 + offlineimap + mutt. In the former setup I was unable to read my old mails while away from my laptop, now I have decent replicas of my full mailboxes in various places. I'm still a bit unhappy about how can I force a send/receive email cycle: with fetchmail I used to have a demonized fetchmail running in the background and just executing fetchmail I was able to force that. Now, with offineimap, I've an autorefresh-ed offlineimap running inside screen and I need to bring up the screen and then hit 1 to trigger a sync cycle.

... now back to that paper introduction due for December 26th