Search Results: "Lucas Nussbaum"

18 September 2008

Lucas Nussbaum: tool to mirror a website locally?

Dear lazyweb, I need a tool to mirror a website locally (so I can browse it offline). Requirements:
- not GUI-based (I want to run it in a script)
- support recursive retrieval and include/exclude lists (like wget)
- no output when everything is fine, but still output errors (not possible with wget, which still output “basic information” when running with –no-verbose, and doesn’t output errors when running with –quiet)
- understands timestamps, and retransfers files if timestamps or sizes don’t match
- not too over-engineered, not too badly maintained, etc… Thank you.

16 September 2008

Lucas Nussbaum: New Debian Developers!

We got a lot of (>= 10) new Debian developers recently. I’m really happy to see that the bottlenecks in the New Maintainer process were (at least partially) solved. My first NM (actually my second, my first one is on hold) also became a DD today. So, how long does it take to become a DD ? Let’s take 2 examples. Both are very active and skilled new contributors, that probably were quite close from being the faster you can be through NM:
Name Applied AM assigned Approved by AM Account created
Chris Lamb 2008-05-01 2008-06-12 2008-07-22 2008-09-16
Sandro Tosi 2008-03-24 2008-05-06 2008-06-22 2008-09-16
We have the proof: provided you have all the required skills, you can become a DD in less than 6 months! Of course, some things are not perfect yet:

Lucas Nussbaum: Is Mozilla the new XFree86? Could Ubuntu actually help?

All the recent moves of Mozilla make me feel that they are really taking the XFree86 path. Reading the Launchpad bug log about the EULA shows that most of the posters agree on who is on the wrong side, and favor switching to IceWeasel or Epiphany+Webkit. Even if Mozilla is apparently going to back off on the EULA story, it looks like the harm was done. If they want to fix that, they will have to start listening to other players in the Free Software community. Or just watch Webkit eat their market share. Since Ubuntu leaders are apparently talking to Mozilla about that, I really hope that they are aiming for a solution that will help the Free Software community as a whole, and are not looking for a work-around that will “fix” the problem for Ubuntu. There has been a lot of noise about the lack of “giving back” to the community by Ubuntu. Using Ubuntu user base to weight in and solve such issues in a way that benefit the whole community would probably be seen as a much more valuable contribution than another bunch of patches.

10 September 2008

Lucas Nussbaum: UDD and Buildstat

Ultimate Debian Database (UDD) is a GSoC project (now finished) which aimed at importing all different data sources that we have in Debian in a single SQL database, to make it easy to combine that data. Currently, we import information about source and binary packages, all bugs (both archived and unarchived), lintian, carnivore, popcon, history of uploads, history of migrations to testing, and orphaned packages. The goal is really to have that data, without really thinking of a specific use cases: there will be lots of use cases. Buildstat is a project by Gon ri Le Bouder, that provides a framework for running QA tests (rebuilds and lintian currently, but buildstat is built in a very extensible way) on packages, using both packages in the archive, and packages in the VCS repositories of teams. This is pretty cool: it allows teams to get an overview of the status of their packages, not using the archive as reference, but using their VCS. Buildstat schedules and runs the tests, store the data in an SQL database, and allows to browse the data using a web interface. Buildstat also import some data from other sources (only the BTS currently, using the LDAP dump) to display it on the web interface. Since both projects are using an SQL DB, people have been asking why we don’t simply merge them. The big advantage would be that the data is synchronized: buildstat would display more up-to-date info about the data sources it doesn’t generate locally (like bugs), and UDD would get fresh buildstat data. We have been talking a lot with Gon ri, considering the different possibilities. But I don’t think it’s a good idea. I think that both projects should try to do one thing, but do it very well, instead of trying to fix the world. UDD focuses on importing data that exists elsewhere. Sometimes it means doing some complex processing. But data should be be generated by UDD. Merging the projects would mean having a very big piece of software that does everything (or tries to do everything). Both databases were designed differently, with different goals. UDD tries to stay close to the data it imports. There might be some incoherences in the data sources, but that’s fine: one of the goal of UDD is to make it easy to find them (and fix them), so we need them in the DB. In buildstat, since the goal of importing data is to display it on the web interface, with a strict use case, you can freely “simplify” data if it helps. Another big difference is that UDD is designed to be easy to use (ie write and run queries) by a human user: UDD uses multi-column primary keys, while buildstat uses surrogate keys (integer “id” keys), that ORM tools usually require. There are also more technical concerns: currently UDD makes a compromise, for each data source, on what it imports: it tries to import the data that is useful, not all the data available. Merging buildstat and UDD would mean increasing the DB size significantly, by adding all the “private” data that buildstat needs. Another problem is the stable API problem: if buildstat and UDD are merged, it means that buildstat cannot change its DB schema without making sure that it wouldn’t break what UDD users are doing. So, what should we do, from my POV, instead of merging? - Continue to talk, and get Gon ri into the UDD “team”. He gathered a lot of experience working on buildstat, and he probably would be able to help a lot. - Data that is not generated by buildstat (bugs data) should be imported from UDD. Doing SQL->SQL will probably make things easier there. - A summary of buildstat’s infos should be imported into UDD. There’s also the issue of providing the data through a web interface, to the DDs, which buildstat tries to address partially. The Debian Developer Packages Overview (DDPO)’s main limitations are: - lack of knowledge about VCS (what buildstat solves) - lack of knowledge about complex organizations (you can’t get any list of packages, or list of packages maintained by teams not using a consistant Maintainer/Uploaders scheme, or list of packages from tasks, etc) - poor handling of large amount of packages. In teams with lots of packages, it’s useful to have restricted views, such as “packages outdated compared to upstream”, “packages which have bugs/RC bugs”, “packages which are newer in the VCS than in the archive”. The perl team’s work on PET clearly shows the kind of things that are needed. At this point, I think that DDPO would benefit from a full rewrite (using the existing code as a source of inspiration, and making sure that there are no regressions, of course). Using UDD as the data source, it should be easy to get something done (even if UDD still lacks some of the data DDPO has currently). But it still requires web developer skills, which I don’t have. If you are interested, contact me!

19 August 2008

Lucas Nussbaum: tiling terminals manager

I tried terminator (thanks go to Nicolas Valcarcel for asking me to sponsor a Debian upload, thus forcing me to try it, and Asheesh Laroia for doing a lightning talk at debconf about it), but I’m not convinced.
- More keybindings are clearly missing. You can only switch terminals using Previous/Next keybindings.
- More features would be great, like the ability to switch the position of two terminals (so you could reorganize them).
- It has some small usability problems, like the fact that the config is text-based, not using gconf, that it’s not possible to change the config without restarting it, that the title bar doesn’t display anything useful most of the time, since it prefixes the current terminal’s title with “Terminator: “, etc. So, is there any other tiling terminals manager I should try, before filing tons of feature requests on terminator? My other requirement is that it mustn’t reinvent the wheel, but use the gnome-terminal widget. Thank you.

18 August 2008

Jurij Smakov: DebConf8 impressions

What I liked What I did not like What I did

12 August 2008

Lucas Nussbaum: Debian s Freeze

Debian’s freeze sounds like a technical hack to address a social problem, and that disturbs me a bit. The social problem is: At some point, we need everybody in Debian to make only non-disruptive changes, so everything can converge very fast into a releasable state. The “solution” we are using is that we are blocking all packages from migrating to testing, and requiring manual review from someone on the release team. Consequences are:
- many people feel that you need to be very convincing to fix a small, not RC bug, even if fixing that bug definitely increases your package’s quality.
- the release team is completely overwhelmed by unblock requests during the freeze
- many people just stop trying to fix things during the freeze (which definitely doesn’t improve Debian’s quality), both because they think it’s hard to get a fix in, and because they don’t want to bother the release team I wonder if we really need such a strict policy. Are there other Free Software projects that use such a technical measure to prevent software from disrupting stable releases? I am the impression that most other projects rely on social pressure instead of technical measures for that, except maybe during the last few hours before the release. Couldn’t we act on the social level? We could default to allow everyone’s package to migrate to testing, and, when someone fucks up and uploads something that should not have been uploaded, block all his packages (switching to manual review mode) until the release. Of course, that require the release team to make decisions about _people_, which is harder than making decisions about _packages_. But if the rules are clearly stated, couldn’t this work?

9 August 2008

Lucas Nussbaum: Code for Debian versions comparison?

Do you know code that compare versions of debian packages (for example, that knows if 2:23.2.3~rc1-1 is lower than 2:23.2.3-2?), besides dpkg –compare-versions? If yes, please write a comment to this blog post, preferably with a link to the code. Also, did someone already write a test suite for that? Who would be interested in such a test suite? I’m considering writing a function in PL/SQL to compare debian versions (for the Ultimate Debian Database project). If someone already wrote that, I’m interested as well.

8 August 2008

Lucas Nussbaum: Of popular packages removed from testing, and the Ultimate Debian Database GSOC project

Some time ago, there was some flamewars^H^Hdebate about the Release Team’s removals of RC-buggy packages from testing. Basically, some people claimed that popular packages shouldn’t be removed, even if RC-buggy. But, do we really miss popular packages in testing? It’s difficult to know. You could get the popcon data, and compare it with the Packages files for testing and unstable. Or work with source packages (which removes a lot of noise), but then, you have to convert the popcon data (which uses binary packages names) to source packages. Not completely trivial. That’s where the Ultimate Debian Database GSOC project comes to the rescue. The goal of Christian von Essen’s project is to gather data from various sources in Debian into a single SQL DB, so queries that combine all those data sources can easily be written. For example, here is the query that lists the source packages that are in unstable, but not in testing, sorted by their popcon (using the number of insts of the most popular binary package of the source package as value for the source package):
SELECT DISTINCT unstable.package, insts
FROM (SELECT DISTINCT package FROM sources
WHERE distribution = 'debian' and release = 'sid') AS unstable, popcon_src
WHERE unstable.package NOT IN (
   SELECT package FROM sources
   WHERE distribution = 'debian' AND release = 'lenny')
AND popcon_src.source = unstable.package ORDER BY insts DESC;

And the results are available on the web! Top packages (> 1000 insts):
lzo	64962
gnome-cups-manager	32346
db4.6	20708
ffmpeg-debian	12908
freetype1	10569
flashplugin-nonfree	7116
perlftlib	6769
nvidia-graphics-drivers	3864
wxwindows2.4	3640
dvi2tty	2239
kdebase-runtime	1725
easytag	1717
g-wrap	1582
yaird	1507
slocate	1499
youtube-dl	1390
hugin	1275
w3c-libwww	1058
Interested in UDD? Join #debian-qa or debian-qa@lists.d.o (or talk to me @DebConf!)

7 August 2008

Lucas Nussbaum: Exporting logs from Suunto X6HR watches on Linux

I’m the happy owner of a nice geeky toy: a Suunto X6HR watch, that includes an altimeter and an heart rate monitor, which I use mainly for moutain biking and hiking. During outings, the watch can log the altitude and heart rate every 2, 10 or 60 seconds, and the data can be transfered to a PC using a serial interface. The problem is that Suunto only provides software for Windows. I got tired of using virtualbox to connect to the watch (qemu doesn’t work, Suunto Activity Manager apparently does strange things with the serial port), so I reverse-engineered the protocol (using skimanager and J rome Kieffer’s work as a basis) and implemented a script to fetch the logs, and export them in a format suitable for gnuplot. Of course, Suuntux is publicly available. I’d be happy to hear from you if it works for you too. Also, if you own a Suunto X6 (similar watch, without HRM), I’d be interested in supporting it too (if it’s not supported already). Below is a example graph, from a short mountain bike ride just before leaving for Debconf. example suuntux output

11 July 2008

Lucas Nussbaum: 3G Internet access using mobile phone + laptop in France

In France, we only have 3 mobile network operators: Orange, SFR, and Bouygues Telecom. They usually discuss their rates together to make sure that one of them doesn’t break the market by creating too interesting rates (that’s why it would be so cool if Free.fr could get a 3G license, and why the governement can tell them “do what we want, or we will give Free a 3G license” - source). I’ve long been interested in using my mobile phone with my laptop to access the Internet. With the 3 operators, you can get a contract for a 3G USB key, but that’s not really interesting: it’s expensive, and there’s a limit on volume of data, so you have to monitor your bandwidth. Also, I only need to use my mobile phone as modem a few times a month. So I’m not really interested in paying an expensive monthly fee for that. What do french MNOs offer? Most of them have rather interesting offers when you want to get unlimited internet access from your mobile, but that doesn’t cover the case of using your phone as a modem, to access the internet on another device (e.g laptop). I’m not sure how they can see the difference, but apparently they do. (It seems that there are hacks making use of those unlimited rates, using your phone as a router between the Wifi network (but your phone has to support Wifi) and the 3G network, using apps like WMWiFiRouter. Another hack is to use the same HTTP proxy on the laptop than on the phone, which seems to give unlimited HTTP access.) But offers for using your phone as a modem are clearly less interesting (probably because they want to push their USB 3G cards offers): Bouygues:
Apparently, you are billed only based on the volume (see page 46 of this document), with a progressive rate. 5 MB/month costs 9 EUR, 100 MB costs 20 EUR, and over 100 MB, you pay 1 EUR/MB (!). SFR:
See page 40-41 of this document. You pay 0.50 EUR for a 30 minutes session, with 2 MB of data included. After those 2 MB of data, you pay 1 EUR/MB. Those rates are really crazy. Using a mobile phone on an HSDPA network (called “3G+” in France), you can easily reach 1 Mbps. (and I did see this rate while fetching files using rsync, so I was not particularly aiming at performance!). 1 Mbps translates to 7.5 EUR/min using the 1 EUR/MB rate. Or 12.5 cts/sec!!! Surprisingly, Orange (ex-France Telecom), which is usually not the most innovative MNO and ISP in France, saves us. With Orange, you pay 0.5 EUR for a 20 mins session, with unlimited volume (sessions called “session multim dia” by Orange). I was so surprised that I called the assistance to check. And after trying it yesterday, I was a bit anxious when checking my online bill this morning. But it works! Important note before you try (from Orange website):
La session multim dia est une tarification valable pour les clients forfaits mobile Orange (hors forfaits Orange pour iPhone, Mobicarte et cartes pr pay es). Pour les clients Classique, Intense, Pro, Click le forfait et Initial dont la souscription ou le r engagment est ant rieur au 14 juin 2007, pour les clients forfaits bloqu s dont la souscription ou le r engagement est ant rieur au 16 ao t 2007 et pour les clients b n ficiant d une offre blackberry, l accord expr s de l abonn b n ficier de la session multim dia est n cessaire. Les autres clients b n ficient de la tarification au volume.

24 June 2008

Lucas Nussbaum: question to Git (power)users

I’m slowly switching to Git for my personal projects, and I like it more and more. And it seems that I can only find nice things to read about it. So, I’m wondering: what are the things that you dislike with Git? That make you regret/prefer another VCS? If Git was redesigned from scratch, what should be changed?

19 June 2008

Raphael Geissert: Re: Ubuntu divergence

Lucas Nussbaum recently posted some graphics showing the number of packages that are grabbed from Debian and kept untouched, modified, out of sync, etc, etc.

But, "statistics are tricky":


$ wget http://patches.ubuntu.com/p/php5/php5_5.2.6-1ubuntu3.patch
--2008-06-19 22:25:03-- http://patches.ubuntu.com/p/php5/php5_5.2.6-1ubuntu3.patch
Resolving patches.ubuntu.com... 91.189.90.149
Connecting to patches.ubuntu.com 91.189.90.149 :80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 441869 (432K) [text/plain]
Saving to: php5_5.2.6-1ubuntu3.patch'

100%[======================================================================>] 441,869 62.3K/s in 7.3s

2008-06-19 22:25:11 (59.4 KB/s) - php5_5.2.6-1ubuntu3.patch' saved [441869/441869]



$ lsdiff -i '*out' php5_5.2.6-1ubuntu3.patch wc -l
44


I believe people familiar with test suites know what that means...

Update: it means that the test suite files (outputs, expected results, and so on) are on the .diff.gz, thus contaminating it

Oh, and:


$ out=$(mktemp); filterdiff -i '*tests*' php5_5.2.6-1ubuntu3.patch > $out
$ du -h php5_5.2.6-1ubuntu3.patch $out
436K php5_5.2.6-1ubuntu3.patch
372K /tmp/tmp.eAxRv20298

...what a waste of space

Lucas Nussbaum: Ubuntu divergence

in main:
divergence in Ubuntu main
in universe:
Divergence in Ubuntu universe Those graphs (found on http://merges.ubuntu.com/) show the number of unmodified vs modified packages in Ubuntu main and universe. Interesting.

15 June 2008

Lucas Nussbaum: Ubuntu information on the Debian Package Tracking System and the Developer Packages Overview

Users of Debian derivatives sometimes report bugs that are not reported in the
Debian BTS, but that also affect Debian. It already happened a few times that
looking at the Ubuntu bugs for my packages allowed me to fix an unreported bug
in my Debian packages. But it’s difficult to keep track of the status of our packages in Ubuntu, since
Launchpad doesn’t provide a per-Debian-maintainer summary <insert here a rant
about closed-source software and how it restricts users>. Since it’s always fun to abuse proprietary software, I fetched all the bug data from Launchpad and inserted it in an SQLite DB (takes about 30 mins at 1200 HTTP requests/minute — it would be so much easier if the Launchpad devs added a text export of all bugs). The result is that there’s now an “Ubuntu” box on the Packages Tracking System, giving the current version in Ubuntu, a link to the Ubuntu patch (if any), and the number of open bugs. An Ubuntu column has also been added to the Debian Developer Packages Overview by Christoph Berg, with the current version in Ubuntu and the number of open bugs. It’s hidden by default: click on Display Configuration to enable it (then it’s stored in a cookie). I hope that this will help Debian maintainers to track what has been reported/fixed in Ubuntu. Also, if other Debian derivatives want to export the same kind of information, don’t hesitate to contact us. See for example: PS: the data might be slightly outdated, as it is processed on merkel.d.o, which was offline until recently. Expect it to be up-to-date in the next 24 hours.

9 June 2008

Lucas Nussbaum: Text normalizer, anyone?

When I write text documents (using LaTeX or Docbook), I like to wrap lines, as it makes them easier to edit (less things moving on the screen), and allow to have easy-to-read diffs. However, I always hesitate before rewrapping paragraphs (using vim’s gqap): this mean that I will add noise to my git history. So I only do that from time to time, making “rewrapping-only commits”. But that sucks, since in the meantime, I sometimes make a lot of changes, and my lines grow long again. Of course, I could rewrap my paragraphs before each commit, but if I simply add a word to a paragraph, it might cause all the lines to be rewrapped. So I what I would need is some kind of “text normalizer” that will: I’ve looked at plasTeX: I could use it to parse a LaTeX document, and export it as LaTeX. But then it would be a LaTeX-only solution. Does anyone have a better solution?

7 June 2008

Lucas Nussbaum: Datamining Launchpad bugs

One think that really is annoying with Launchpad is its lack of interfaces with the outside world. No SOAP interface (well, I think that work is being done on this), no easy way to export all bugs. The only way to get all the bug data in a machine-parseable is to first fetch this URL, and then, for each bug number listed there, to make another request for https://launchpad.net/bugs/$bug/+text. I filed a bug a few weeks ago, asking for a simpler way to get all the data. A Launchpad dev suggested to do what I just described (fetch all the number, then fetch the data for each bug). I originally dismissed the idea because it just sounded too dirty/aggressive/whatever, but since I needed to practice python, I gave it a try. And actually, it works: I was able to get all the data in less than an hour (but that probably put some load on Launchpad ;-)). That allows to write cool SQL queries. Bugs with the most subscribers:
select bugs.bug, title, count(*) as subscribers
from bugs, subscribers
where bugs.bug = subscribers.bug
group by bugs.bug, title
order by subscribers desc
limit 10;
bug firefox subscribers
188540 firefox-3.0 crashed with SIGSEGV in g_slice_alloc() 291
154697 package update-manager 1:0.81 failed to install/upgrade: ErrorMessage: SystemError in cache.commit(): E:Sub-process /tmp/tmpjP6Bsx/backports/usr/bin/dpkg returned an error code (1), E:Sub-process /tmp/tmpjP6Bsx/backports/usr/bin/dpkg returned an error code (1), E:Sub-process /tmp/tmpjP6Bsx/backports/usr/bin/dpkg returned an error code (1), E:Sub-process /tmp/tmpjP6Bsx/backports/usr/bin/dpkg returned an error code (1) 278
141613 npviewer.bin crashed with SIGSEGV 262
59695 High frequency of load/unload cycles on some hard disks may shorten lifetime 182
215005 jockey-gtk crashed with AttributeError in enables_composite() 171
216043 compiz.real crashed with SIGSEGV 168
121653 [gutsy] fglrx breaks over suspend/resume 144
1 Microsoft has a majority market share 142
145360 compiz.real crashed with SIGSEGV 134
23369 firefox(-gnome-support) should get proxy from gconf 126
Bugs where someone is subscribed twice:
select bug, subscriber_login as cnt
from subscribers
group by bug, subscriber_login
having count(*) > 1;
bug subscriber
33065 mvo
48262 mvo
144628 skyguy
158126 benekal
213741 sandro-grundmann
216043 jotacayul
221630 kami911
(Yes, that forced me to change a primary key) Packages with the most bugs:
select package, count(distinct bug) as cnt
from tasks
group by package
order by cnt desc
limit 10;
package number
ubuntu 5392
linux 1464
linux-source-2.6.20 1034
update-manager 826
linux-source-2.6.22 724
firefox 684
kdebase 673
firefox-3.0 668
ubiquity 590
openoffice.org 566
Bugs with the shortest titles:
select bug, title, length(title) as len
from bugs
order by len asc
limit 5;
bug title length
190560 - 1
160381 uh 2
224350 css 3
133621 gnus 4
138052 pbe5 4
If you want to play too, you can fetch the SQLite3 DB (5.8M, lzma-compressed), the DB creation script, and the script that fetches the bugs and import them into the DB. Comments about my code would be very appreciated (stuff like “oh, there’s a better way to do that in python!”), as I’m not very confident about my pythonic skills. :-) Update: apparently, I’m not really fetching all the bugs. I’m getting the same results as when you just press “Search” on https://launchpad.net/ubuntu/+bugs. But if you click on “Advanced search”, then select all the bug statuses, and click search, you get a lot more bugs (154066 vs 49031). If someone know which bugs are excluded with the default search, I’m interested! Update 2: Got it. Apparently the default search doesn’t list bugs that have all their “tasks” marked “Won’t fix”, “Fix Released”, or “Invalid”.

27 May 2008

Lucas Nussbaum: Searching for cheap flights?

Dear readers, What are currently the best websites to look for cheap flights? liligo.fr is OK, but doesn’t allow to specify “this date is flexible”, and the listing of the results doesn’t provide a lot of info.

23 May 2008

Lucas Nussbaum: FOSSCAMP

Last week-end, I was in Prague for FOSSCAMP, a Canonical-sponsor event aiming at bringing together people from various Free Software projects. Such an event is a very good idea, especially after the openssl debacle, where we saw how difficult it is to build good relationships with other Free Software projects. The event was organized in a rather interesting way: the attendees make up the schedule as the event happens, by adding sessions to the timetable using marker pens on a whiteboard (see picture, hi Jorge!). If we have a spare BOF room at Debconf, it would be great to do the same thing: I always find it frustrating that many important discussions at Debconf happen outside BOFs/lectures, and are so easy to miss, simply because you submit talk proposals months before Debconf, so you can’t know yet what will be the hot stuff when Debconf finally arrives :-) It was also interesting to see all the different ways people organized BOFs. It might be interesting to write a list of DOes and DON’T about BOFs (not that the sessions weren’t of good quality!). Does anyone know if such a list already exist? And finally, as usual, it was great to meet all those people from this nice community again. If you still believe that some projects are fighting with each other, you need to attend such an event. And special thanks go to Ond ej ert k for guiding me through a visit of Prague!

13 May 2008

Lucas Nussbaum: No, we don t need a new openssl maintainer

DSA-1571 is totally embarrassing. But I disagree with Julien BLACHE: we don’t need a new openssl maintainer. Mistakes sometimes happen to people who do things. That sucks, but it’s unavoidable, no matter how many levels of checking you add. Kurt has done a lot of excellent work on Debian in the past years, and I’m sure he will continue. Of course, it’s a lot easier and less risky to write tons of stupid blog posts. By the way, Julien, there’s an RFH bug for openssl open for more than 2 years. What about getting involved and helping Kurt yourself? On a more constructive side: it’s not the first time that a Debian-specific change broke something (I can think of a recent grep change …). It might be useful to provide a simpler way to review those changes (reading the diff.gz files is not really user-friendly). We could have something like http://patches.ubuntu.com/ (or even something nicer :-).

Next.

Previous.