Search Results: "lucas"

3 July 2007

Dirk Eddelbuettel: More on 'nicer charts'

Via the Planet Debian aggregator and his blog, Sven followed up on my post regarding Lucas' plot of the package age distribution. As some of my points didn't seem to make it across, I will reiterate them more plainly: Sven also addresses the fact that what we really want is to see the quantiles of the data set. Quite right, and taking logs makes that easier. Consider the two charts below which plot the 'package age in days' as an empirical cumulative distribution function using built-in R functions ecdf and plot.stepfun (rather than redoing it ad-hoc as I had done), and also add explicitly quantiles. The two charts use the exact same instructions; however the second chart transforms the x-axis to a logarithmic scale. Debian Package Age-since-recompile Distributions charted two ways While it is close to impossible to find the 25 or 50 percentile on the first chart, it becomes a lot easier on the second chart because the x-axis is 'stretched' using the log transform. About one quarters of the distribution appears to be rebuild within 1.5 months old, and about half is younger than four months (as a quick call to summary(pkgAge) confirms). Reading these proprtions off the original chart, or the non-log chart, is much more difficult.

2 July 2007

Sven Mueller: Improving graphs

In Improving simple charts, Dirk Eddelbuettel wrote a nice little summary how to produce “nicer” charts using R. However, in my opinion, the charts he provided as alternate options to the original, purely linear graph are less readable for the average person. If anything, using a logarithmic scale on the original Y axis (number of packages not built after a specific date) would be of use, but the other graphs only hide the original point of the post Dirk replied to (How old are our (Debian’s) packages?). The point was that 94% of all packages had been rebuilt since the release of Sarge or, if you want to put it the other way around, 1264 (6%) were built before Sarge was released. This relationship is hidden from immediate recognition by the reader of the graphs. I admit though, that Dirk’s second graph (Number of packages against age in days with the age being on y logarithmic scale) is also insightful. The other two graphs he presents might make sense in a scientific publication if some point in the “more detailed” areas of the graph needs proving, but for this specific problem, they really don’t make sense to me. Actually, I would question their use even in scientific publications, since they distort the original data a lot. Scientists are pretty much used to logarithmic scales these days, but I couldn’t work out how the scales work on those two graphs. And another note on the second graph (logarithmic age): The way the X axis is labeled (10, 20, 50, 100,..) is also pretty non-intuitive to me. I’m much more used to something like 5, 10, 20, 40, 80, 160 and so on, which gives the same distance between each label and seems a lot more usable to me.. (edit: Fix spelling of Dirk’s surname)

30 June 2007

Dirk Eddelbuettel: Improving simple charts

Earlier today and via Planet Debian, Lucas blogged about the 'age distribution' of Debian packages, defined as the time since the last (re-)compilation. He illustrated his findings with an, umm, rather ugly chart. Having climbed onto the soap box once before, I would like to point out how easy it can be to create simple, informative, and, at to least to me, prettier charts using R. Lucas included a URL to the data. The first nice thing to note that we can read the data directly from the URL -- no need to copy the file:
pkgAge <- read.table(file="http://people.debian.org/~lucas/arch-age/arch-age.log", col.names=c("pkg","yyyymmdd"))
read the data into a data.frame which we have given two column names.
pkgAge[,"date"] <- as.Date(as.character(pkgAge[,"yyyymmdd"]), "%Y%m%d")
pkgAge[,"age"] <- as.numeric(difftime(Sys.Date(), pkgAge[,"date"], units="day"))
pkgAge[,"prop"] <- (1:nrow(pkgAge)) / nrow(pkgAge) * 100
We then create three new columns. First is a date, by parsing the (integer) dates (after first casting them into characters) by supplying the format in standard C notation: "%Y%m%d" for year, date and month without any separators or formatters. Now, having the date as an actual date object inside a real data analysis language we can do things as e.g. computing date differences. The difftime function does just that, using the current date as other point. We ask for the return to be in days, and cast this down to a purely numeric vector (instead of datediff object). Lastly, we quickly compute the date proportion in percentages. We can then view the date. Before we plot,
png("packageAges.png", quality=100, width=640, height=480, pointsize=10)
oldpar <- par(mfrow=c(2,2), mar=c(2.5,2.5,3,1))
we direct the charts to a png file of given dimensions, and ask for all plots in one figure (using mfrow with two rows by two) with somewhat smaller figure margins using the mar argument to par. The first chart shows again proportion over date:
with(pkgAge, plot(date, prop, type='l', main="Standard Plot"))
(The with() function simply allows us to refer to the columns by their names without explicit subsetting. plot(pkgAge[,"date",], pkgAge[,"prop"]) is equivalent, but more cumbersome.) As it clear that the data has a fairly long tail in the older dates, we can also try to plot the plot over logarithmic time differences. This doesn't work for dates, but it works for our (positive-valued) age variable:
with(pkgAge, plot(age, prop, type='l', log="x", main="More linear as log(age in days)"))
The very far left tail below 0.5 percent is interesting as the one very old package is clearly an outlier within an outlier region. We use the subset function to take just one portion of the data, use logs, and explicit plotting symbols '+' in a points-and-lines plot:
with(subset(pkgAge, prop0.5), plot(date, prop, type='b', log="y", pch="+", main="Detail in left tail, up to 0.5%"))
Lastly, the upper quartile is fairly linear.
with(subset(pkgAge, prop>75), plot(date, prop, type='l', pch=".", main="Yet fairly linear in top 25%"))
At the end
oldpar <- par(mfrow=c(2,3))
dev.off()
we restore the graphics paramters and close the device (here the file). All this then yields the following chart: Debian Package Age-since-recompile Distributions Updated to correctly display the assignment operator <-

29 June 2007

Lucas Nussbaum: How old are our packages?

Debian’s binary packages are only built when they are uploaded (or when a binNMU is requested, but that doesn’t happen frequently). They are never rebuilt later. This means that some binary packages in etch weren’t build in an up-to-date etch environment, but possibly in a much older environment. Is this a problem? It depends. Old packages don’t benefit from the improvements introduced in Debian after they were built. Like new compiler optimizations, or other changes in the toolchain that could induce changes in binary packages. For example, some files used to be at some place, but are now put at another place. Also, some parts of maintainers scripts are automatically generated, and would be different if the package was rebuilt today. But is it really a problem? Are our packages _that_ old? Also, when a big change is made in the way we generate our binary packages (like Raphael Hertzog’s new dpkg-shlibdeps), when can we expect that the change will be effective in all packages? I went through all binary packages in unstable (as on 24/06/2007) (both main, contrib and non-free packages) on i386. Using dpkg-deb –contents, I extracted the most recent date of each package (which can reasonably be taken as the date of the package’s creation). And here are the results. Most packages are actually quite recent. 9008 packages (43%) were built after the release of etch. And 19857 packages (94%) were built after the release of sarge. But that still leaves us with 1265 packages that were built before sarge was released, and even one package (exim-doc-html) that was built before the release of woody! (the removal of this package has been requested, so we will soon be woody-clean :-) Now, what could we do with this data? We could review the older packages, to determine if: Here is the full sorted list of packages.

25 June 2007

Lucas Nussbaum: debconf7

DebConf7 finished today. This was my first DebConf, and probably not my last one :-) I had a really interesting time in Edinburgh, despite the weather (but at least we are prepared for DebConf8 now!). The organization team really did a fantastic work, the venue was gorgeous, etc, etc, etc. Again, I was positively surprised by how nice everybody was at DebConf. It’s weird to think that the same people are part of the regular flames on the mailing lists. Which proves that most flames are just caused by people being passionate about Debian, and looking for the best for the project, and not by people trying to be annoying, like one could think :) I gave a talk about the usual QA stuff I’m doing: archive-wide rebuilds, piuparts runs, and how to make all this more efficient by working inside the collab-qa project. So far, it’s not like a lot of people have been joining the project, which is kind of a failure. But at least everybody seem to think that the idea is good. I also gave a lightning talk about Debian Package of the Day, which I didn’t plan to give originally (I think I never prepared slides in such a short time). I think it was well received, and we even got a new submission half an hour after the talk. I even managed to clear some tasks on my TODO list during Debconf, the most visible one being the “automated monthly mails to maintainers of packages with serious problems”. I was quite afraid of being flamed, but I sent 250 mails, and only one person replied aggressively. I also got a lot of positive feedback, so the next batch will probably use slightly less strict criterias. And of course, the most important result of debconf was the numerous discussions, that resulted in a lot of good and interesting ideas.

13 June 2007

Lucas Nussbaum: maintainer wanted for next generation tetrinet server!

You probably all know tetrinet - if you don’t, you should really try that game. There are several servers for that game out there, and one of them is tetrinetx. It is packaged in Debian, but it has several problems: A few years ago, I worked with a few friends on improving tetrinetx. This resulted in: This is hosted on sourceforge under the name tetrinetx-ng, but it hasn’t seen any activity for the last two years. It would really be great if a tetrinet fan could take over the project and start making it alive again. Then tetrinet.debian.net could support tetrifast ;) Also, I could try to setup a server with tetrinetx-ng. Someone wants to host tetrifast.debian.net for me ? Needed: MySQL DB + Apache (for the stats), + not being afraid of insecure code (doesn’t need to run as root)

5 June 2007

Ross Burton: Postr 0.6

Postr 0.6 is here! What is new I hear you ask. Well: The tarball is in the usual place, and I'll make Debian packages shortly. NP: Konfusion, Skalpel

4 June 2007

Lucas Nussbaum: Distributed SCM and branching a sub-directory ?

I am considering switching from SVN to a distributed SCM for my personal stuff. I had a look at git and mercurial, but neither really support branching a sub-directory: Often, I am working on a big private project, and, while working on a sub-project (stored inside the project’s repository), I’d like to share that sub-project with others. So there are actually two problems: SVN only partially meets my needs with that (it’s possible to checkout a sub-directory directly, for example with svn co svn://svn.debian.org/svn/pkg-ruby-extras/tools/ruby-pkg-tools). I think that it’s possible to do fine-grained access control
using libapache2-svn, but I haven’t tried yet. It seems that mercurial can do that, using the forest extension. But you have to convert the specific directory into a repository, with a complex step to keep the history. Amongst the distributed SCM, is there one that supports that ? (at least the sub-directory branching part)

18 April 2007

Lucas Nussbaum: Bash is weird.

Consider the following command:
echo (cat /etc/ motd,passwd ) (you can replace “echo” with any command that takes one file as argument and cannot take it as stdin) It’s obvious that the goal is to expand this to:
echo (cat /etc/motd /etc/passwd) Then to:
echo /dev/fd/63 However, it doesn’t work like this:
$ echo (cat /etc/ motd,passwd )
++ cat /etc/motd
++ cat /etc/passwd
+ echo /dev/fd/63 /dev/fd/62
/dev/fd/63 /dev/fd/62
But bash doesn’t work like this. Brace expansion is done first, but inside parameters of the command (that is, (cat /etc/ motd,passwd )) not words. So when (cat /etc/ motd,passwd ) is expanded, the prefix is (cat /etc/, the suffix is “)”, so it’s expanded to (cat /etc/motd) (cat /etc/passwd). After reporting a bug about that, Chet Ramey gave me the correct way to reach the initial goal:
cat (eval cat /etc/\ passwd,motd )

19 March 2007

Lucas Nussbaum: Condorcet method and the french presidential election

The french presidential election (in april/may 2007) will use plurality voting with two rounds (voters will choose 1 amongst 11 or 12 candidates during the first round. The two candidates with the most votes will be qualified for the second round). The latest polls clearly show how this election method fails to satisfy the Condorcet criterion. Using the second round results from this poll (only considering the 3 main candidates, since the other don’t have any chance to win the presidential election), we get this: 2007 elect results So Francois Bayrou is the Condorcet winner, and should win the election. However, the first round results, according to the same poll, could be:
  1. Sarkozy 31%
  2. Royal 24%
  3. Bayrou 22%
  4. Le Pen 12%
Since only the two first candidates are qualified to second round, Bayrou wouldn’t be elected. It will be interesting to see how those scores evolves until the first round (April 22nd).

16 March 2007

Lucas Nussbaum: Most broken spam protection ever

Just received that, in reply to a mail I sent:
This email is from X. My email address (X@Y.com) is protected against spam and viruses by MailInBlack. Please click on the following link in order to identify yourself to me and to allow your message to reach me.
http://192.168.0.252/v/?C8BEF10E72C&tmstp=20070316084314&tk=message_confirm&tkid=7951&lang=2 This needs to be done only once, for this email and all future email correspondence. Thank you for your understanding. X
MailInBlack seems to be a french company. No wonder why they guarantee that 100% of spams are stopped. (To be fair, I am not sure yet of who fucked up, it might be the admin)

14 March 2007

Lucas Nussbaum: Jabber clients and OS usage stats (3)

Using XMPP4R, I did some stats about Jabber clients usage on the Apinc Jabber server, which hosts im.apinc.org, jabber.fr, and many more using virtual hosting. I already did similar stats in March 2006 and September 2005. The poll was done by sending jabber:iq:version to online users, around 1:00 PM (french local time, most of the users are french). 1343 clients were pinged, and 1315 answered, which is better than last year (1145 answers). Of the 1145 jids that answered last year, 368 were also part of the poll this year (I don’t know if this is good, or not enough). Systems: Clients: (all clients with at least one reply are listed here) I also did some stats on the Linux distros. With 566 Linux users, one can consider that statistically signifiant. (Other distros are below 5 replies) The Debian/Ubuntu situation is interesting. Debian is not dying, even on the desktop ! The server is hosting jabber.ubuntu-fr.org, so the results are biased towards Ubuntu. Update: it seems that Psi reports “Debian GNU/Linux (testing/unstable)” even when running on Ubuntu. This is the case for ~5% of linux users. To server admins: if you are running a large jabber server with jabberd 1.6, it’s easy to give me the right to get the list of online users, so contact me if your are interested in me doing your users stats. It would be interesting to see if users from different servers/countries have different behaviours. I’m not sure if it’s possible with ejabberd and other servers.

26 February 2007

Lucas Nussbaum: Slides for my FOSDEM talks about Debian QA

As promised, the slides from my FOSDEM talks about Automated Testing of Debian Packages and Use of Grid Computing for Debian Quality Assurance are available. Don’t hesite to ask questions or post comments. The videos for both talks should be available so you can laugh at my frenglish, but I haven’t heard of an ETA yet.

Lucas Nussbaum: Efficient key signing

I just finished signing all the keys from the FOSDEM KSP. Importing keys I received:
Mutt is nice. You can tag all messages with ‘t’, then use ‘;’ (tag-prefix) and ^K (Ctrl + K) to import all signatures. (Thanks Myon + sam for the tip) Signing keys:
caff (featured on debaday last week) has some useful info in /usr/share/doc/signing-party/, especially the README.many-keys and README.gpg-agent files.

23 February 2007

MJ Ray: Debian: Links for 2007-02-23

While we're fixing some network problems, here is a musical interlude^W^Wset of links...
Explaining Debian at parties
'Do not try to convince people to try it out or you will become a support hotline. Instead, tell them about LUGs and mailing lists (and how cool it can be) and put the challenge on them: "if you try it, those are your support channels. Are you up for it?"'
Index of /~joerg/templates
Explanations of some common reasons for getting REJECT from ftp-master.
gravityboy: Politics
"If we're going to make interactions between developers more civil and friendly and less personally hostile then we need a different kind of stick. Maybe a carrot of sorts is in order too?"
Consensus and community review in open source and open standards Decentralized Information Group (DIG) Breadcrumbs
'this is quite a stretch with respect to the normal dictionary meaning of "consensus"'
Social Drag
How many Debian Developers are dressed in Social Drag?
debian Gender Research
More research to add to the surveys page
#354950 - ITP: gauche-readline -- A readline-like library for the Gauche Scheme implementation - Debian Bug report logs
Contemplating sponsoring again.
The joys of English cuisine are abomination in God's sight Ted Walther's Private Diary
Blast from debian past: Ted Walther's gullibility for stereotyping is an abomination in God's sight and those foods are hardly typical English. For instance, the only one on his PS list that I've eaten recently is flapjack (not flapjacks, which is something else and not English AFAIK) and I can only remember eating five of the list in the last year. Also, the linked site seems to concentrate on processed pap from multinationals and supermarkets, rather than real trad English food. Example: "Yorkshire puddings are basically a bread type product. You can use them in place of dinner rolls." Why not make 'em a bit harder, cook 'em a bit longer and you can use them for cobbling the street instead? Yorkshire pudding should be a baked batter with a soft middle and a crisp puffy edge. Nothing like bread. And don't get me started about the pasties. Honestly, if you learn about food from such sources, no wonder you'd think it an abomination.
Re: Position Statement to the Dunc-Tanc "experiment"
"had somebody wanted to kill (or inflict maximum damage) to the project, he couldn't have done any better than the current DPL"
Releas-o-meter
Interesting idea - why no cron-run copy on the web?
EUPL v 1.0 revokable ?
One that may appear in a future 'bits from debian-legal' - it seems the EU are popularising a type of termination clause. Yay.
Future of Debian Weekly News
As I hoped, Joey is explicitly calling for sponsorship for DWN production.
DPL 2007, Current Candidates
still don't have a title is keeping a list of DPL candidates, so I won't put one up this year.
#411041 - request for debian-l10n-persian mailing list - Debian Bug report logs
Tell all your persian-speaking friends to show their support!
Interprete I cite, you cite, I rant
Lack of references is a problem on lists like debian-legal where explaining everything again can drive you insane and still not convince some people.
Lucas Nussbaum's Blog State of software for Suspend to RAM/Disk ?
More tips on the "Will I Work Or Not?" suspend problems.
only on debian-devel - she.geek.nz
> Am I mistaken or this is a flame on how to flame ?

21 February 2007

Lucas Nussbaum: State of software for Suspend to RAM/Disk ?

Dear readers, I am the mostly happy owner of a Dell Latitude D610 (the one with the ATI video card), running Debian Etch. I installed the hibernate package to try to get Suspend to RAM and to Disk to work, and Suspend to Disk worked out of the box. But Suspend to RAM didn’t work (garbled screen at resume). I never bothered to investigate this until today, when I discovered that acpi-support provides the same functionality, and works both for Suspend to RAM and to Disk ! I’m happier now, but I would like to understand, and the world of ACPI and Suspend to RAM/Disk is very difficult to grasp for newcomers. What is the story behind hibernate and acpi-support ? Why was this duplication of effort needed ? Also, it seems that gnome-power-manager and kpowersave provide the same functionality. Why can’t they just call the relevant acpi-support or hibernate scripts ? How do those different implementations compare (what are their main differences ?) ? Looking at the packages content, it seems that hibernate tries to be generic, while acpi-support gave up, and handles each and every laptop model differently. Correct ? Also, hibernate seems to be handling a few things better, like getting my network interface back up after hibernation. Finally, the last question: how do one debug Suspend issues, when it fails to work ? Is there a document somewhere explaining that ?

8 February 2007

Lucas Nussbaum: Working evince and poppler in Debian Etch

Evince and poppler (the library used to render PDFs) are quite outdated in etch, and fail to display correctly a number of PDF files (including PDFs generated by latex-beamer). There’s a new poppler version (0.5.x, vs 0.4.5 in etch), but it bumps the soname, so it’s a no-go. Since I was particularly annoyed by this (I’m an evince fan), I looked at the issue, and found that entry in poppler’s changelog for version 0.5.1:
2006-02-16  Albert Astals Cid
        * qt4/src/Makefile.am:
        * qt/Makefile.am:
        * poppler/Makefile.am:
        * glib/Makefile.am: Update soname as we are not really compatible
        anymore with previous releases that had soname 0.0.0
And I thought that maybe, the soname update was not totally justified. I modified the package in experimental (v.0.5.4) to remove the soname bump, and renamed the binary packages so it becomes a drop-in replace for the version in etch. It works fine for me with the version of evince in etch, and seems to fix at least #369164, #409758, and #410085. Packages are available here. Of course, this is totally unsupported, and I have not done any careful testing. But it works for me until now, and I’l update this entry if I discover some problems. Also, I don’t plan to maintain this on the long term, so beware of security bugs. Update : This breaks tetex (see #356079). Arg. I’ll have to use the experimental version…

Lucas Nussbaum: Running Debian on your Linksys WRT54G* sort of

I’m the happy owner of a Linksys WRT54GL. OpenWRT is nice, but … well. Debian is just nicer. And I couldn’t resist the idea of running Debian on this little MIPS system. Since there’s clearly not enough space available on the Linksys, I decided to install etch in a chroot, that I would mount using NFS. Joey tried that already, the Debian wiki provides some information, but I use another technique. First, on another system (an i386, I debootstrap’ed a mipsel etch, using –foreign. This tells debootstrap not to run the second stage:
debootstrap –arch mipsel –foreign etch /space/debian-mipsel http://ftp.fr.debian.org/debian Then, I modified /etc/exports to allow the router to mount that chroot:
/space/debian-mipsel 192.168.1.1(rw,sync,no_root_squash) I mounted it on the router:
ipkg install kmod-nfs
insmod sunrpc
insmod lockd
insmod nfs
mount -t nfs star:/space/debian-mipsel /debian -o nolock
mount -t proc /dev/null /debian/proc
I set up some swap space on the NFS mount (mandatory, or debootstrap’s second stage will fail):
ipkg install losetup
ipkg install kmod-loop
ipkg install swap-utils
dd if=/dev/zero of=/debian/swapfile bs=1M count=100
losetup /dev/loop/0 /debian/swapfile
mkswap /dev/loop/0
swapon /dev/loop/0
I chrooted inside /debian, and ran debootstrap’s second stage:
chroot /debian /bin/bash
debootstrap/debootstrap –second-stage
When you are done playing, you can disable the swap space and umount everything:
swapoff /dev/loop/0
losetup -d /dev/loop/0
umount /debian/proc
sleep 1 # or umount /debian will fail
umount /debian
If you want to re-mount everything, all you need to do is:
insmod sunrpc
insmod lockd
insmod nfs
mount -t nfs star:/space/debian-mipsel /debian -o nolock
mount -t proc /dev/null /debian/proc
losetup /dev/loop/0 /debian/swapfile
swapon /dev/loop/0
chroot /debian /bin/bash
That first stage/second stage split in debootstrap is really cool: it’s an easy way to run Debian anywhere, only requiring to be able to chroot at some point.

15 January 2007

Lucas Nussbaum: ruby & native threading

Stefano Zacchiroli said:
To balance this, according to my first read of Ruby’s threading capability, it’s my impression that not only at most one thread can execute Ruby code at a time (limitation shared by OCaml, due to the non-distributed nature of mark and sweep garbage collectors), but also a thread blocked on a syscall will block all other threads to run. Dumb Ruby threads (but I still hope I’m wrong …)
This is not totally true:

4 January 2007

Dann Frazier: IRC proxies

Lucas, try bip. It's not nearly as buggy as ctrlproxy, and allows multiple client connections, unlike dircproxy.

Next.

Previous.