Search Results: "ag"

15 May 2009

Eddy Petrișor: svn-buildpackage is now orphaned

I have decided that is high time to declare svn-buildpackage orphaned since many people are replying on it to be useful, but nobody cared enough to answer to the RFH Eduard Bloch made some 3 years ago.

I know it would have been better if I would have asked a request for adoption, but, honestly, the package didn't had a maintainer in the last year and that would have been lying to everybody.

The package is quite important and I would hate to see it leave the official archive since many people do need it and use it regularly (I know for sure the GNOME, Perl and the Debian Games Team use it) and I have enjoyed working on the package while I was motivated.

The code is maintained in SVN and the trunk can be viewed at:

http://svn.debian.org/wsvn/collab-maint/deb-maint/svn-buildpackage/trunk/

and can be checked out with any of these commands:

# read-only copy
svn co svn://svn.debian.org/svn/collab-maint/deb-maint/svn-buildpackage/trunk svn-buildpackage

# read write, if you have an account on alioth.debian.org:

svn co svn+ssh://svn.debian.org/svn/collab-maint/deb-maint/svn-buildpackage/trunk


Also, note that the package has several features which the new maintainer(s) should be aware of and must make sure they all work when changing the code. Also, there are some things to remember:

These being said, I am really hoping svn-buildpackage will find a new maintainer since it deserves a lot more attention than I am providing. Don't forget, 0.6.24 is wating for an upload already.



Update: Neil Williams answered and agreed to take over maintenance, but he doesn't usually use svn-inject and svn-upgrade, so somebody taking care of those would be needed. Ironically, those two scripts are the ones in the worst shape of the three main scripts.

Jan Hauke Rahm: Bug#528733: O: svn-buildpackage

Oh dear, it happened Hundreds of developers seem to have switched to git (man, I want to be as famous as Linus) and finally the packaging helper for subversion is orphaned. Yeah, I know, this is just partly related if at all but at least the low interest in its further development seems to prove my point. Anyways, I m not going to fully give up on it. Eddy just copied the bug message to d-devel to arrest some attention and actually I m trying the same here. svn-buildpackage is still needed, let alone the entire texlive stuff which is maintained in subversion repositories and many packages in collab-maint. So, let s see if we can t get it somewhere. 0.6.24 - release soon As Eddy mentioned in the bug report there is a version 0.6.24 prepared in trunk. Looking at my working copies I seem to have fixes for #363003, #414564, #467614, #480684, and #506876 lying around which I didn t commit yet. They should be sent to the bug reports, though. If you consider working on a release of svn-bp please contact me (or just write to the bug reports, I m subscribed). 0.7 - do it right ;-) This is far away but maybe a worthy goal. Two months ago I started working on a rewrite of svn-bp in order to fix some long outstanding bugs which would need gross hacks in the current version or aren t even fixable by design. Find my first experiments in branches/0.7. But then let me be honest, I didn t do anything about it lately - partly because of the low interest, but majorly because of vcs-pkg. I hope to figure out during debconf9 (I hope I can attend) if it is a possible replacement of svn-bp. But that s still up in the air. Well, that s my two cents about the situation.

14 May 2009

Russell Coker: The Main Security Problem

All security problems are to some degree people problems. Code may be buggy, but it was written by people who could have been better trained, had more time to spend on code review, etc. When there are multiple programs, OSs, libraries, etc to choose from then choosing a suitable combination of software is a matter of the skill and background knowledge of the people involved. There are issues of software choice where there is no provable benefit of making one particular choice, EG choosing between a popular product that is OK and for which it is easy to hire skilled people to use it and a less popular product that has better security features but less public knowledge. But this is minor compared to other security problem. I believe that the greatest security problem is stupid people. Stupid people in technical positions write buggy code and configure servers to be insecure. In consulting and analysis roles they develop bad procedures. In management they hire bad people to do technical work. The vast majority of security problems can be fairly directly and immediately traced back to stupidity. In the corporate environment that is stupid programmers, stupid managers who hire people who are obviously stupid, and often stupid executives for mandating that software that everyone knows to be insecure should be used across the entire enterprise. In both the home and corporate environments there are a huge number of people who run machines that they know to be compromised. Apparently using a computer that is known to be under the control of an unknown hostile person is something that they don t consider to be a problem - in spite of the obvious risks of fraud, data destruction, and risk of being implicated in crimes such as the distribution of child porn.

Sean Finney: automatic commit log messages with git

continuing on the trend of finding novel ways to make life more interesting with git, here's the description from a new hook:
# prepare-commit-msg hook for debian package git repositories
#
# this script scans the diff that is going into a commit, and automatically
# injects some "proposed" comments based on what it finds in the diff.  this
# can be used to avoid a few extra keystrokes when performing some of the
# more standard/boring tasks.

(see below for how to fetch the script) this one definitely falls in the "carrot" category, as it encourages properly isolated (and thus automatically identifiable) changes. the proposed comments are then given to the standard editor, so one can easily amend them, append "Closes:" lines, etc. some sample use-cases currently implemented: as always, comments/feedback/suggestions/etc welcome :) using this new hook note this is the same repo as the previous hooks i've blogged about, so if you already have that set up you can skip the clone and instead just pull in the changes. also note that this is in your local repo, not the remote one. to set it up:
REPO_PATH=/path/to/your/repo.git
HOOK_REPO_PATH=/somewhere/you/want/to/put/it
git clone git://git.debian.org/users/seanius/vcs-hooks/git-hooks.git $HOOK_REPO_PATH
ln -sf $HOOK_REPO_PATH/debian/git-hooks/prepare-commit-msg-guess-message.py $REPO_PATH/.git/hooks/prepare-commit-msg

there aren't currently any configurable options in this hook.

Jeff Bailey: Stupid Locksmith

Our place has a fancy magical lock on the front door that has been sticking lately. We phoned the locksmith who put it in to discover that he apparently can't talk to us because we're not the registered owners of the lock.

Right - when we bought the place, we apparently don't automatically get the title to the locks on our front door. The former owners could phone at anytime and get into our place. If we were locked out in the middle of the night, he couldn't let us in.

Fuck that. Will replace the lock myself with a crowbar if I have to.

Julian Andres Klode: Ubuntu One


Today, I was testing Canonical s new Ubuntu One service. Ubuntu One is a service for syncing and sharing files online, with 2GB storage for free. I installed the Ubuntu One client on Ubuntu 9.04 and it s cool. Ubuntu One creates a directory named Ubuntu One in your home directory. Within this directory, there are two subdirectories. The first one is My Files and the second one is named Shared With Me . When you place files in the My Files directory, the Ubuntu One client gets notified (using inotify) about the change and begins uploading the file to the Ubuntu one server. When you access the web interface, which should work in every modern browser, and upload a file there, the next time your local client connects the files are fetched to your local hard disk. This also works when you have two different computers and create a file on the one computer, it will be visible on the second one as soon as it has fetched the new file. You could also copy your .mozilla directory into the synced directory, and create a symlink from your home directory to it. I have not tried it myself, but in theory this would allow you to have your profile synced on all your computers. And the best thing about Ubuntu One is that the client is completely free software and written in Python. This makes it possible to package the client for other distributions, like Debian. Packaging it for RPM-based distribution such as Fedora should also be doable, but may require some more time. There seems to have been some criticism that the server side is not free software. While that may not be the good, it s certainly better than other services where even the client is proprietary. And there still is the possibility to write your own server as the protocol is available. Ubuntu One is currently in private beta, if you want to try it out, you need an invitation (visit ubuntuone.com for further information). Posted in Ubuntu

Antti-Juhani Kaijanaho: Asynchronous transput and gnutls

CC0
To the extent possible under law,
Antti-Juhani Kaijanaho has waived all copyright and related or neighboring rights to
Asynchronous transput and gnutls. This work is published from Finland.
GnuTLS is a wonderful thing. It even has a thick manual but nevertheless its documentation is severely lacking from the programmer s point of view (and there doesn t even seem to be independent howtos floating on the net). My hope is to remedy with this post, in small part, that problem. I spent the weekend adding STARTTLS support to the NNTP (reading) server component of Alue. Since Alue is written in C++ and uses the Boost ASIO library as its primary concurrency framework, it seemed prudent to use ASIO s SSL sublibrary. However, the result wasn t stable and debugging it looked unappetizing. So, I wrote my own TLS layer on top of ASIO, based on gnutls. Now, the gnutls API looks like it works only with synchronous transput: all TLS network operations are of the form do this and return when done ; for example gnutls_handshake returns once the handshake is finished. So how does one adapt this to asynchronous transput? Fortunately, there are (badly documented) hooks for this purpose. An application can tell gnutls to call application-supplied functions instead of the read(2) and write(2) system calls. Thus, when setting up a TLS session but before the handshake, I do the following:
                gnutls_transport_set_ptr(gs, this);
                gnutls_transport_set_push_function(gs, push_static);
                gnutls_transport_set_pull_function(gs, pull_static);
                gnutls_transport_set_lowat(gs, 0);
Here, gs is my private copy of the gnutls session structure, and the push_static and pull_static are static member functions in my sesssion wrapper class. The first line tells gnutls to give the current this pointer (a pointer to the current session wrapper) as the first argument to them. The last line tells gnutls not to try treating the this pointer as a Berkeley socket. The pull_static static member function just passes control on to a non-static member, for convenience:
ssize_t session::pull_static(void * th, void *b, size_t n)
 
        return static_cast<session *>(th)->pull(b, n);
 
The basic idea of the pull function is to try to return immediately with data from a buffer, and if the buffer is empty, to fail with an error code signalling the absence of data with the possibility that data may become available later (the POSIX EAGAIN code):
class session
 
        [...]
        std::vector<unsigned char> ins;
        size_t ins_low, ins_high;
        [...]
 ;
ssize_t session::pull(void *b, size_t n_wanted)
 
        unsigned char *cs = static_cast<unsigned char *>(b);
        if (ins_high - ins_low > 0)
         
                errno = EAGAIN;
                return -1;
         
        size_t n = ins_high - ins_low < n_wanted
                ?  ins_high - ins_low
                :  n_wanted;
        for (size_t i = 0; i < n; i++)
         
                cs[i] = ins[ins_low+i];
         
        ins_low += n;
        return n;
 
Here, ins_low is an index to the ins vector specifying the first byte which has not already been passed on to gnutls, while ins_high is an index to the ins vector specifying the first byte that does not contain data read from the network. The assertions 0 <= ins_low, ins_low <= ins_high and ins_high <= ins.size() are obvious invariants in this buffering scheme. The push case is simpler: all one needs to do is buffer the data that gnutls wants to send, for later transmission:
class session
 
        [...]
        std::vector<unsigned char> outs;
        size_t outs_low;
        [...]
 ;
ssize_t session::push(const void *b, size_t n)
 
        const unsigned char *cs = static_cast<const unsigned char *>(b);
        for (size_t i = 0; i < n; i++)
         
                outs.push_back(cs[i]);
         
        return n;
 
The low water mark outs_low (indicating the first byte that has not yet been sent to the network) is not needed in the push function. It would be possible for the push callback to signal EAGAIN, but it is not necessary in this scheme (assuming that one does not need to establish hard buffer limits). Once gnutls receives an EAGAIN condition from the pull callback, it suspends the current operation and returns to its caller with the gnutls condition GNUTLS_E_AGAIN. The caller must arrange for more data to become available to the pull callback (in this case by scheduling an asynchronous write of the data in the outs buffer scheme and scheduling an asynchronous read to the ins buffer scheme) and then call the operation again, allowing the operation to resume. The code so far does not actually perform any network transput. For this, I have written two auxiliary methods:
class session
 
        [...]
        bool read_active, write_active;
        [...]
 ;
void session::post_write()
 
        if (write_active) return;
        if (outs_low > 0 && outs_low == outs.size())
         
                outs.clear();
                outs_low = 0;
         
        else if (outs_low > 4096)
         
                outs.erase(outs.begin(), outs.begin() + outs_low);
                outs_low = 0;
         
        if (outs_low < outs.size())
         
                stream.async_write_some
                        (boost::asio::buffer(outs.data()+outs_low,
                                             outs.size()-outs_low),
                         boost::bind(&session::sent_some,
                                     this, _1, _2));
                write_active = true;
         
 
void session::post_read()
 
        if (read_active) return;
        if (ins_low > 0 && ins_low == ins.size())
         
                ins.clear();
                ins_low = 0;
                ins_high = 0;
         
        else if (ins_low > 4096)
         
                ins.erase(ins.begin(), ins.begin() + ins_low);
                ins_high -= ins_low;
                ins_low = 0;
         
        if (ins_high + 4096 >= ins.size()) ins.resize(ins_high + 4096);
        stream.async_read_some(boost::asio::buffer(ins.data()+ins_high,
                                                   ins.size()-ins_high),
                               boost::bind(&session::received_some,
                                           this, _1, _2));
        read_active = true;
 
Both helpers prune the buffers when necessary. (I should really remove those magic 4096s and make them a symbolic constant.) The data members read_active and write_active ensure that at most one asynchronous read and at most one asynchronous write is pending at any given time. My first version did not have this safeguard (instead trying to rely on the ASIO stream reset method to cancel any outstanding asynchronous transput at need), and the code sent some TLS records twice which is not good: sending the ServerHello twice is guaranteed to confuse the client. Once ASIO completes an asynchronous transput request, it calls the corresponding handler:
void session::received_some(boost::system::error_code ec, size_t n)
 
        read_active = false;
        if (ec)   pending_error = ec; return;  
        ins_high += n;
        post_pending_actions();
 
void session::sent_some(boost::system::error_code ec, size_t n)
 
        write_active = false;
        if (ec)   pending_error = ec; return;  
        outs_low += n;
        post_pending_actions();
 
Their job is to update the bookkeeping and to trigger the resumption of suspended gnutls operations (which is done by post_pending_actions). Now we have all the main pieces of the puzzle. The remaining pieces are obvious but rather messy, and I d rather not repeat them here (not even in a cleaned-up form). But their essential idea goes as follows: When called by the application code or when resumed by post_pending_actions, an asynchronous wrapper of a gnutls operation first examines the session state for a saved error code. If one is found, it is propagated to the application using the usual ASIO techniques, and the operation is cancelled. Otherwise, the wrapper calls the actual gnutls operation. When it returns, the wrapper examines the return value. If successful completion is indicated, the handler given by the application is posted in the ASIO io_service for later execution. If GNUTLS_E_AGAIN is indicated, post_read and post_write are called to schedule actual network transput, and the wrapper is suspended (by pushing it into a queue of pending actions). If any other kind of failure is indicated, it is propagated to the application using the usual ASIO techniques. The post_pending_actions merely empties the queue of pending actions and schedules the actions that it found in the queue for resumption. The code snippets above are not my actual working code. I have mainly removed from them some irrelevant details (mostly certain template parameters, debug logging and mutex handling). I don t expect the snippets to compile. I expect I will be able to post my actual git repository to the web in a couple of days. Please note that my (actual) code has received only rudimentary testing. I believe it is correct, but I won t be surprised to find it contains bugs in the edge cases. I hope this is, still, of some use to somebody :)

Julian Andres Klode: PHP, MySQL, PostgreSQL, etc.


Well, I m currently writing a PHP and SQL application at school, so here are some of my thoughts on the problems I encounter. PHP is crap Many PHP applications are low quality applications. The problem is that PHP makes it hard for developers to write good applications. PHP developers tend to use many global variables, global code, and not many classes or functions. And when writing applications for the web, PHP makes it easy to mix PHP and HTML, in my opinion even too easy. Well-written applications follow some basic rules. First, separate your code into functions and classes, and (2) split your code over multiple files. And there is one rule which is especially important for non-commandline applications: (3) seperate logic and presentation. This means if you write a web application, use a template engine. PHP error and exception handling is also an interesting topic. The problem here lies in the distinction between the both. In my opinion, there should only be one way to handle such stuff: Using exceptions. Exceptions are a very widespread way to program today, and are used in languages like Java and Python for cases where PHP uses errors. The concept of PHP s errors results in code like @do_something or die("blah"); being written in libraries which is a problem as it clearly violates rule (3) I wrote above, because there is no way to catch this die() error. Furthemore, PHP has no clear concept of namespaces in the current stable versions (< 5.3). PHP 5.3 finally gets namespaces, but using a backslash character, which looks really bad and makes everyone think of eithers Windows systems or escape characters. There are even more problems with PHP. From time to time, PHP just crashes with a segmentation fault. MySQL vs. PostgreSQL MySQL seems to have many problems with checking data, which can confuse me. It got better in MySQL 5 with the introduction of the new sql_modes like STRICT_ALL_TABLES and TRADITIONAL. And using storage engines like InnoDB also allows you to take advantage of stuff like foreign keys and transactions. But MySQL is still problematic. For example, in an application I wrote I had multiple entries identified by a numeric id (an INTEGER). One day, I wanted to check whether my error handling was OK and requested an entry with the invalid ID 1X , which resulted in SELECT * from A where id='1X';. Now I expected to get an error or at least no result, but MySQL happily proceeded with the query and returned the entry with the ID 1. I tried the same thing on PostgreSQL and got an error message that 1X is not integer, as I expected it. As another example, when I run the same PHP application using PDO on a MySQL database and a PostgreSQL database, I always get strings from the MySQL database whereas I get integers from PostgreSQL if the columns are integers. Another thing which PostgreSQL does better than MySQL is password encryption. MySQL only provides access to the standard crypt() function and by default only generates DES passwords. It allows other methods to be used (although not all are available everywhere and therefore not documented), but it provides no way to create a salt for them, which makes them not very useful. PostgreSQL in contrast provides less encryption methods but it provides the gen_salt() function which can be used to generate salts for blowfish encryption (gen_salt('bf')). Another problem is that MySQL simply ignores references when they are added to the column definition instead of a separate FOREIGN KEY . Because MySQL gives no errors when a feature like REFERENCES is not supported, developers may think that the feature is supported and write code using this feature just to realize later that what they wrote does not what they want. And you can t write AGPL applications using MySQL client libraries, because MySQL is GPL-2 which is incompatible with the AGPL. You might think that the FOSS License Exception helps here, but this only lists (among others) the GPL-3. Or am I wrong? But PostgreSQL has other problems, at least when it comes to the performance of the information_schema.columns view. This may be related to the fact that PostgreSQL has > 1000 rows whereas MySQL only has 250. And PostgreSQL may be a bit harder to configure. Final words All in all, PHP and MySQL are not the best choices for writing applications. They are to tolerant, and do not treat errors in an acceptable way by aborting the current action. I can get around some problems in PHP by using error handlers which raise exceptions and setting error_reporting to E_ALL, but in fact this should be the default. MySQL problems can be solved to a large extend by setting sql_mode to TRADTIONAL and using InnoDB databases, which should in my opinion be the default and not an option. My recommendation is to use PostgreSQL servers and for the applications: use frameworks (if you are allowed to, which may not be always the case [e.g. when you are learning SQL, using a ORM is clearly the wrong way]), document your code, separate your code into classes and functions, and make sure you are not using to many global variables. And use a SCM to track your changes. Posted in General

Riku Voipio: I'm Going to DebConf 9



Turns out TAP is starting to fly Lisbon - Helsinki this summer, and thus offers cheapest flights from Finland to near debconf (at least with my ticket searching magic skill level). And a (night) train from Lisbon to Madrid goes via C ceres. It appears one can't order the train tickets yet (from renfe.es), the reserving opens 60 days beforehand.

13 May 2009

Jonathan McDowell: Breaking the Web of Trust

With all the discussion about SHA-1 weaknesses and generation of new OpenPGP keys going on there's some concern about how the web of trust will be affected. I'm particularly interested in the impact on Debian; while it's possible to add new keys and keep the old ones around that hasn't worked so well for us with the migration away from PGPv3 keys. We still have 125 v3 keys left, many of them for users who also have a v4 key but haven't asked for the v3 key to be removed or responded to my email prodding them about it. I don't want to repeat that.

So if we're looking at key replacement we need to have some idea about where our Web of Trust currently stands, and what effect various changes might have on it. I managed to find the keyrings Debian shipped all the way back to slink and ran the keyanalyze and cwot stats against them. I then took the current keyring, pull in all the updates for the keys in it (so that any signatures from newly generated keys would be included) and ran the stats again. Finally I took details of 12 key migrations (mostly from Debian Planet but also a couple of others I knew about) and calculated what the effect of removing each key would be. These stats are cumulative and I replaced the most well connected (by centrality) keys first.

The results are below.

TotalSCSReachableMSDCentrality
1999-02-06 (slink)22836(15.78%)50 (21.92%)2.9022
2000-01-03 (potato)375104 (27.73%)180 (48.00%)4.3382
2001-09-22 (woody)948538 (56.75%)704 (74.26%)4.73202008.6249
2005-05-28 (sarge/etch) 1106883 (79.83%)969 (87.61%)3.34852074.6604
2007-12-0411911001 (84.04%)1062 (89.16%)3.11032113.3747
2009-01-18 (lenny)1126947 (84.10%)1010 (89.69%)3.04891941.2594
2009-04-04 (squeeze/sid)1121946 (84.38%)1008 (89.91%)3.04661936.9761
2009-05-06 (current)1067894 (83.78%)958 (89.78%)2.96701759.4363
TotalSCSReachableMSDCentrality
base1067904 (84.72%)959 (89.87%)2.96401776.4389
update-93sam1067902 (84.53%) 958 (89.78%)2.97341780.9874
update-joerg1067900 (84.34%) 958 (89.78%)2.97761780.7578
update-aurel321067898 (84.16%) 957 (89.69%)2.98031779.2497
update-noodles1067896 (83.97%) 956 (89.59%)2.98311777.8326
update-jaldhar1067896 (83.97%) 955 (89.50%)2.98551779.9193
update-srivasta1067896 (83.97%) 955 (89.50%)2.99041784.3382
update-ana1067895 (83.88%) 954 (89.40%)2.99261784.3102
update-nobse1067893 (83.69%) 953 (89.31%)2.99471782.2392
update-neilm1067892 (83.59%) 951 (89.12%)2.99741782.6098
update-reg1067891 (83.50%) 950 (89.03%)2.99771780.8515
update-rmayorga1067890 (83.41%) 949 (88.94%)2.99841779.4910
update-evgeni1067889 (83.31%) 948 (88.84%)2.99741776.6445

This is actually more hopeful than I thought. There's an obvious weakening as a result of the migrations, but the MSD stays under 3 and the centrality stays fairly constant too. The reachable/SCS counts do decrease, but at this point it looks fairly linear rather than an instant partition. Of course the more keys that are removed the more likely this is to drop off suddenly. Counteracting that DebConf9 is coming up which will provide a good opportunity for normally geographically disperse groups to cross sign, reinforcing the WoT for these new keys.

Either way I at least have a better handle on the current state of play, which gives me something to work with when thinking about how to proceed. For now, bed.

Jaldhar Vyas: I Am Special, I Am Special, I Am Me.

Debian Times asked me a few questions I received my first invitation to AARP membership. Aren't you supposed to be atleast 50? Speaking of old age, the other day I overheard Shailaja tell her little brother while a Seinfeld rerun was on TV, "That's a classic. That's what they [indicating me and my wife] used to watch in the olden times." Look what came in the mail today: Ph33r! I am me. The best me I can be.

Jon Dowland: The Revolution Business

My copy of Charles Stross's The Revolution Business, (book 5/6 in the "Merchant Princes" saga) arrived today. I'm currently half way through Iron Sunrise. I will have to put one of them aside for a while, but which one...

Lucas Nussbaum: UDD and packages metrics

Peter Eisentraut played with Ultimate Debian Database, and wanted to create a maintenance effort metric by multiplying each package s installed size by its popcon. His query is:
SELECT rank() OVER (ORDER BY score DESC), source,
sum(installed_size::numeric * insts) AS score
FROM packages JOIN popcon USING (package)
WHERE distribution = 'debian' AND release = 'sid'
AND component = 'main' AND architecture IN ('all', 'i386')
GROUP BY source, version ORDER BY score DESC LIMIT 30;
Besides all the interesting things that I learnt by looking at his query (rank(), and a bug in UDD because installed_size should really be numeric to avoid the conversion), Peter had a problem with his query: linux-2.6 is missing from the results, while it should obviously have a large popcon and a large install size. The problem is that the binary packages for linux-2.6 often change, so they don t get very high in popcon. The unstable kernel package gets a ridiculous popcon score: select package, insts from popcon
where package in (select package from packages where source ='linux-2.6' and release='sid')
order by insts desc limit 30;
package                             insts
----------------------------------+-------
linux-libc-dev                      38703
linux-source-2.6.29                   614
linux-headers-2.6.29-2-common         256
linux-image-2.6.29-2-amd64            239
A solution could be to change the metric to be: MAX(insts over all binary packages from this source package) * SUM(installed_size)
The good thing is that UDD already offers a popcon_src view, that gives the popcon score for a source package. So the query becomes:
SELECT rank() OVER (ORDER BY score DESC), source,
sum(installed_size::numeric * insts) AS score
FROM packages JOIN popcon_src USING (source)
WHERE distribution = 'debian' AND release = 'sid'
AND component = 'main' AND architecture IN ('all', 'i386')
GROUP BY source ORDER BY score DESC LIMIT 30;
 rank       source         score
------+---------------+-------------
1   openoffice.org      92177633504
2   qt4-x11             18503941620
3   linux-2.6           16036201020
4   gcc-4.3             14369300376
5   mesa                12962475968
6   eglibc              12581290240
7   gcc-4.4             11411296672
8   samba               10021083072
9   xulrunner            9037295424
10   mysql-dfsg-5.0      8348333532
This time, linux-2.6 shows up near the top of the list.

Peter Eisentraut: The Big Shots

As the occasional thinker about open-source development practices, communities, and issues, I have been wondering for a while: What are the largest open-source projects? What projects have the most code, the most users, and the most issues to deal with, and how do they cope?

The Debian archive should provide some insights into the first one or two questions, as it contains a very large portion of all available and relevant open-source software and exposes them in a fairly standard form. In the old days one might have gotten out grep-dctrl to create some puzzling statistics, but nowadays this information is actually available in an SQL database: the Ultimate Debian Database (UDD). (And it's in PostgreSQL. And it comes with a postgresql_autodoc-generated schema documentation. Excellent.)

So here is a first question. Well, the zeroth question would have been, which source packages have the largest unpacked orig tarball, but that information doesn't seem to be available, either via UDD or via apt. So the first question anyway is, which source packages produce the largest installation size across all their binary packages:
udd=> SELECT source, sum(installed_size)/1024 AS mib FROM packages WHERE distribution = 'debian' AND release = 'sid' AND component = 'main' AND architecture IN ('all', 'i386') AND section 
 'debug' GROUP BY source, version ORDER BY mib DESC LIMIT 30;                                                                                   
source mib
------------------+------
openoffice.org 1797
kde-l10n 648
gcj-4.4 544
vtk 465
linux-2.6 404
openclipart 353
vegastrike-data 311
ghc6 308
gclcvs 303
wesnoth 300
fpc 269
axiom 256
webkit 255
gcc-snapshot 255
lazarus 241
kdebase-workspace 226
plt-scheme 221
torcs-data-tracks 219
scilab 213
openscenegraph 211
eclipse 210
sagemath 201
insighttoolkit 198
acl2 195
kdebindings 181
atlas 165
gcl 163
trilinos 153
paraview 153
asterisk 144
(30 rows)
This produces a few well-known packages, but also a number of obscure ones. If you look closer, many of them appear to be themed around scientific, numerical, visualization, Scheme, Lisp, that sort of thing. Hmm.

Here is another idea. Take a package's installation footprint and multiply it by its popularity contest installation count. So you get some kind of maintenance effort score, either because the package is large or because you have a lot of users or both.
SELECT rank() OVER (ORDER BY score DESC), source, sum(installed_size::numeric * insts) AS score FROM packages JOIN popcon USING (package) WHERE distribution = 'debian' AND release = 'sid' AND component = 'main' AND architecture IN ('all', 'i386') GROUP BY source, version ORDER BY score DESC LIMIT 30;
rank source score
-----+-----------------------------+-------------
1 openoffice.org 12638492332
2 mysql-dfsg-5.0 3411344560
3 eglibc 3371485240
4 perl 3019183024
5 evolution 2669948000
6 samba 2308923872
7 mesa 1853902860
8 texlive-base 1684245516
9 gcj-4.3 1610495484
10 foomatic-db-engine 1608178104
11 foomatic-db 1423947704
12 inkscape 1413910080
13 qt4-x11 1258220636
14 gcc-4.3 1248741312
15 kdelibs 1021058256
16 gnome-applets 998434136
17 xulrunner 958232688
18 coreutils 954766896
19 openssl 877067672
20 ncurses 827679424
21 python2.5 815826384
22 aptitude 808161380
23 gimp 786015124
24 gnome-utils 781756328
25 nautilus 774319690
26 openoffice.org-dictionaries 761075576
27 eclipse 756072380
28 dpkg 736626200
29 openclipart 731244240
30 wine 707967500
(30 rows)
(Yeah, they run this thing on PostgreSQL 8.4 beta 1.)

I noticed linux-2.6 is suspiciously absent because of a low popcon score (?!?).

I don't want to dump the entire database into this blog post, but if you try this yourself you can look at about the first 200 to 300 places to find reasonably large and complex projects before it gets a bit more obscure. A few highlights:
  51   gnupg                           455660464
59 php5 386417572
60 mutt 381148176
83 icu 258602756
84 xorg-server 255186332
101 exim4 224857700
107 openssh 215792828
113 tar 201520400
114 postgresql-8.3 196844584
115 libx11 195856564
116 ruby1.8 194681656
272 emacs22 62047476
This is obviously still biased in a lot of ways, but it does show the major projects.

The UDD is also an interesting use case that shows how you can deploy a PostgreSQL database as a semi-public service with direct access. A great tool, and a great tool to build other great tools on top of.

Lior Kaplan: Will the Knesset install Firefox on its computers?


In a letter to the chairman of the Knesset (the Israeli parliament), Ilan Gilon asks why the Knesset s computing unit refuses to install free software (mainly Firefox). I m quite interested to see the answer (: The question already started to be published in the media by Calcalist (in Hebrew). The letter was also published (in Hebrew) by a blogger at room404. Posted in Free software applications, Mozilla Tagged: Firefox, free software

Martin Michlmayr: Update on QNAP TS-119/TS-219 kernel support

I mentioned a few weeks ago that initial kernel support for the QNAP TS-119 and TS-219 Turbo NAS got accepted for 2.6.30 but that there were some issues left. Here's an update: The SATA and flash fixes are already in our 2.6.29 kernel.

Jordi Mallach: Spanish Cup final in Val ncia

Val ncia is again taken over by football fans, who have come from all over the Basque Country and Catalunya to watch the Copa de Espa a final in Mestalla stadium. The city is literally tinted in red, white, blue and maroon and thousands of supporters (more than 60.000) have flooded the streets. If you can't beat them, join them, so for a change, I'm going to join the crazyness and will go to the Athletic Hiria to watch the game in the middle of the leonera. I can't wait to see how the Basque and Catalan supporters who have tickets for the game will react when the King of Spain enters the VIP area in Mestalla. Apparently, more powerful loudspeakers have been installed in the stadium in an attempt to mitigate what I expect to be the biggest catcall in a Spanish football stadium since we have a King... Gora Bar a, Visca l'Athletic!

John Goerzen: A Few Snippets of Jacob

It s been awhile since I posted some Jacob updates. He s 2.5 years old now. So here we go! Here s what I think A recent conversation after Jacob had been misbehaving. John: I think we should have some consequences for that.
Terah: Do you think he should have time out?
Jacob, eavesdropping: I THINK NOT! He managed to evade time out that time because once we got over our shocked silence, we were both laughing too hard. The Post Office Every time Jacob and I drive past a Sonic (a drive-in restaurant), he says, See post office over there. Have cherry limeade! Yes, he was there with Terah to get a cherry limeade once. The post office thing was a mystery to me. Near as we can figure, he had gone to Sonic with Terah right after the post office one day, and got them confused. Then I took him to an actual post office last week. I told him we were going to a post office. I started getting him out of the car. He looked confused (after all, post offices don t look like this, right?) But then he said, This post office. Have cherry limeade inside! I had to tell him it is a different kind of post office, with mailboxes inside. Hmm. I gave him a piece of junk mail to carry to the car, which I think may have been more exciting for him than even a cherry limeade. Terah has tried to explain that a Sonic isn t a post office, but he doesn t believe her. Trains and Chicago We took the train to Chicago for a quick weekend last week. Jacob loved it. He jabbers on and on about going to the train station to get on the night train (we did get on the train at 4AM), eating in the train diner car , and seeing all the conductors on the train. He also loves to talk about the elevators in our hotel, riding on a city bus, and seeing ambulances on the streets. Didn t Notice It We got a new Roomba off Woot the other day. Its box has been sitting in our kitchen for a few days now, since we didn t have time to unbox it yet (already have another Roomba). Finally Terah started unboxing it. Jacob said, Ooo, a SPECIAL Roomba! I didn t notice it there. Terah: What makes it special, Jacob? Jacob: Comes from Walmart. Can t win em all, I guess. Colors Terah read in her What to Expect book about the toddler years that, at Jacob s age, there s a small chance that children can identify one color. Jacob can identify more than a dozen, maybe even two. Navigation This really shocks me. As we re driving down the road, Jacob will correctly point out directions to me: This way to day care! That way to grandma s house! That way to Newton! What really shocked me, though, was when he pointed to an entrance ramp to an Interstate one which we never take with him and correctly told me where it went. And when he pointed down a country road that we have never even talked about, let alone driven on, and told me where it must wind up (in Newton, yep). I have no idea how he managed to figure that out. Scary Memory Last week, Jacob got a balloon. It reminded him of a balloon he got last Halloween six whole months ago. He was at a Halloween event for kids, and they gave him a black balloon. It was filled with helium, and it rose up high enough that it touched a sharp edge of a fire sprinkler and popped. Jacob told us about all of that, right down to it being a black balloon that popped on a sprinkler. They gave him another balloon to replace it orange. Jacob told us, Orange balloon not pop. Six months ago. I think it s time to get him signed up for the debate team Days of the week Jacob doesn t know what day it is, or in what order they come, but he does know that we go to church on Sunday, that he goes to grandma and grandpa s house on Tuesday, and to the clinic and Walmart on Wednesdays.

12 May 2009

Julien Valroff: HADOPI law finally voted by my local representative

As expected, the French HADOPI law was voted earlier today by the Assembl e Nationale. As expected, my local representative voted in favour of this law. I had however kept a secret hope that he would abstain for this vote, following to the email I have received from his assistant following to our meeting. Mr Vannson planned to take the floor during the finale discussions. He had sent me for review the text they wrote, which perfectly matched my arguments, and even referred to the law as a Rube Goldberg machine ( Ma conviction est que nous avons mont une usine gaz in the original version). This text also stated that he was netiher convinced nor satisfied by the law.

His speech planned for the final discussions of the law never happened, and was replaced by a single line question to the Minister, during a meeting of the Law Commission.
This question was even technically incorrect as he refers to firewall, whereas I had (with a great help from other members of APRIL) amended this part of the text to refer to usage control software :
M. Fran ois Vannson. Les logiciels pare-feu ne risquent-ils pas, madame la ministre, de nuire aux logiciels libres ?
Mme la ministre. Il n y a aucune menace sur les logiciels libres, qui peuvent tout fait accepter des logiciels de s curisation.
From: http://www.assemblee-nationale.fr/13/rapports/r1626.asp Which could be approximately translated as follows:
M. Fran ois Vannson. Mrs Minister, aren t firewall a threat on Free Software?
Mrs Minister. There is no threat on Free Software who can accept well accept securisation software.
I must hence express my real disappointment.
Now, let s hope this law won t be applicable, following the idea of most of its opponents. I won t write to my local representative, as I think it is totally hopeless. I guess he will get the pingbacks from this blog post, as his own political weblog runs on Free Software!

Michal &#268;iha&#345;: Gammu test version 1.24.90

I finally find time to make another release of Gammu. This time it is really just bug fixing and I will stay in this mode for some time, because I have simply too much things to do in real life. Full list of changes follows: You can download from usual place: http://cihar.com/gammu/, Debian users will find packages in experimental soon, SUSE, Fedora and Mandriva users can check experimental repository in OpenSuse Build Service.

Next.

Previous.