Gunnar Wolf: Main character for The Gunnar Wolf chronicles
Google Checkout recently gained support for taking recurring payments, a frequently requested enhancement for my Xen-based server.
However, to use any of the non-trivial features we must provide Google with generate a signed XML file for each transaction. This would normally preclude the use of some server-side scripting, but if the details of transaction are static there is no reason we shouldn't be able to generate the relevant HTML ahead-of-time.
The resulting HTML file can be hosted anywhere, is almost completely independent of the server configuration and greatly reduces the attack surface by being a simple static file. The main disadvantage of this approach (and indeed of this submission method in general) is that allows trivial replay attacks of previous transactions.
Anyway, first we create our request XML:
$ head -n6 order.xml
<?xml version="1.0" encoding="UTF-8"?>
<checkout-shopping-cart xmlns="http://checkout.google.com/schema/2">
<shopping-cart>
<items>
<item>
<item-name>Lunar object affixed to slender carrying attachment</item-name>
$ cat order.html.in
<h1>Order stuff</h1>
<form method="POST"
action="https://checkout.google.com/api/checkout/v2/checkout/Merchant/__MERCHANT_ID__">
<input type="hidden" name="cart" value="__CART__">
<input type="hidden" name="signature" value="__SIGNATURE__">
<input type="image" name="Google Checkout" alt="Fast checkout through Google"
src="http://checkout.google.com/buttons/checkout.gif?merchant_id=__MERCHANT_ID__&w=180&h=46&style=white&variant=text&loc=en_US"
height="46" width="180">
</form>
MERCHANT_ID = # Enter your merchant ID here
MERCHANT_KEY = # Enter your merchant key here
FILES = order.html
all: $(FILES)
%.html: %.html.in %.xml
sed -e 's __MERCHANT_ID__ $(MERCHANT_ID) g' \
-e 's __CART__ '$$(base64 -w0 $*.xml)' g' \
-e 's __SIGNATURE__ '$$( \
openssl dgst -sha1 -hmac $(MERCHANT_KEY) -binary <$*.xml base64 -w0)' ' \
<$< >$@
clean:
rm -f $(FILES)
.PHONY: clean
$ make
[...]
$ head -n4 index.html
<h1>Order stuff</h1>
<form method="POST"
action="https://checkout.google.com/api/checkout/v2/checkout/Merchant/007">
<input type="hidden" name="cart" value="iAgICAgICAgICAgIDxpdGVtLWRlc2NyaXB0aW9 [..]
Just this week I went to the movies to check the latest incarnation of Cap. Kirk, Mr. Spock et al. It really was a great movie I was expecting less, I have to admit. But the movie was not bad at all
With all that space and beyond thing in my mind, yesterday I was surprised with a greatly inspiring picture taken from 
How inspiring can that be?
You can find the original here, along with the explanation of how and when it was taken.
After i had to cancel my last attempt to go to Spain due to suddenly catching a heavy flu, I'm now having some holidays in the sun! Not quite like in those lyrics, rather some unexpected but long needed break from my normal routines. Part of them I'll be keeping here, ie I'll be going to C ceres in the next days for real... at least that's my plan now. 
This document explains how to use deboostrap --foreign to build from scratch and install a minimal Debian system on the Sheevaplug (but it is equally applicable to many other embedded systems).
The only piece missing to be able to install from bare metal is a working kernel and initramfs pair, so this document assumes that you already have a kernel which can mount the root file system from a USB key. For the Sheevaplug, the factory kernel in the internal NAND flash will work fine.
The Debian kernel package puts us in a chicken and egg situation: it cannot mount the root without an initramfs (because all drivers are modular), but we cannot build one until we can boot the system. I suppose that, as an exercise of style, somebody could create a minimal initramfs with busybox and the kernel modules needed to mount the root and then mount it manually, but if you really lack a working kernel then it's probably faster to boot the file system image for the first time using qemu and the "versatile" debian-installer target.
The first step is to run the first stage of debootstrap on another system (which does not need to be ARM), to build a temporary root file system. I use the --variant and --include options to build the smallest possible system with network support, but they can be omitted.
debootstrap \
--foreign --arch=armel \
--variant=minbase \
--include=linux-image-2.6-kirkwood,module-init-tools,udev,netbase,
net-tools,ifupdown,iproute,whiptail,vim-tiny \
sid ./target/ http://ftp.it.debian.org/debian/
As usual when using debootstrap, some manual configuration is needed:
cd target/ echo bokassa.mi.bofh.it > etc/hostname vi etc/network/interfaces vi etc/resolv.conf echo LANG=C > etc/default/locale echo '127.0.0.1 localhost' > etc/hosts cat <<END > etc/fstab /dev/root / ext3 noatime 0 1 tmpfs /tmp tmpfs defaults 0 0 #/tmp /var/tmp none bind 0 0 ENDI took some hints from http://wiki.debian.org/ReadonlyRoot to reduce writes to the flash file system:
# the blkid cache is evil anyway ln -s /dev/null etc/blkid.tab # this link will be created anyway by the package ln -s /var/lib/initscripts/nologin etc/nologin # it's better to not use a static mtab with modern kernels ln -s /proc/mounts etc/mtab # unless you plan to use multiple removable network interfaces, persistent # interface names are not needed : > etc/udev/rules.d/75-persistent-net-generator.rulesTwo small workarounds are needed:
# if this is not set then the preinst of linux-image-* will die trying # to ask a debconf question in the debootstrap second stage echo 'do_initrd = yes' > etc/kernel-img.conf # a workaround for bug #520742 : > etc/udev/disabledNow you can format a USB pen drive and copy the root file system on it:
mke2fs /dev/sdb1 tune2fs -i 0 -c 0 /dev/sdb1 mount /dev/sdb1 /mnt/ rsync -aH target/ /mnt/ umount /mnt/After powering the plug press a key to get to the U-Boot prompt and use the factory kernel to boot from USB:
setenv bootargs console=ttyS0,115200 root=/dev/sda1 rootdelay=10 panic=10 init=/bin/bash run bootcmdThen you can run the second stage of debootstrap, finish the bare minimal configuration and start init:
./debootstrap/debootstrap --second-stage # disable the gettys for /dev/tty[1-6], which do not exist vi /etc/inittab # and add one for the serial console echo 'T0:2345:respawn:/sbin/getty -L ttyS0 115200 linux' >> /etc/inittab # do not forget to set the root password or you will not be able to login... vi /etc/shadow rm /etc/udev/disabled exec /bin/bash mount / -o ro,remount exec /sbin/initCongratulations, now you can login on the system and start to install the packages you like:
echo 'deb http://ftp.it.debian.org/debian/ unstable main' > /etc/apt/sources.list echo 'APT Install-Recommends "false"; ;' > /etc/apt/apt.conf.d/no-recommends apt-get update # dash is much faster apt-get install dash dpkg-divert --add /bin/sh ln -sf dash /bin/sh # I do not like these, YMMV rm /vmlinuz /initrd.img rmdir /selinux /srv # the kernel is able to read the time from the RTC by itself, and this # will allow using a read only root echo HWCLOCKACCESS=no >> /etc/default/rcS # faster boot echo CONCURRENCY=shell >> /etc/default/rcS # useless on this platform mv /etc/rcS.d/S01glibc.sh /etc/rcS.d/K01glibc.sh dpkg-reconfigure tzdata apt-get install openssh-server screen iptables wget lessU-Boot requires that the kernel and initramfs have a special header:
apt-get uboot-mkimage cd /boot/ mkimage -A arm -O linux -T kernel -C none -a 0x00008000 -e 0x00008000 \ -n Linux-2.6.29-2 -d vmlinuz-2.6.29-2-kirkwood uImage mkimage -A arm -O linux -T ramdisk -C gzip -a 0x00000000 -e 0x00000000 \ -n initramfs -d initrd.img-2.6.29-2-kirkwood uInitrdNow you can reboot and configure u-boot to boot from the USB device. This is the variables scheme I like and it is not mandatory in any way, so I recommend that you read and understand the existing configuration before modifying it (some lines have been broken for readability):
setenv bootargs_nand $(bootargs) setenv bootcmd_nand $(bootcmd) setenv bootargs_root root=/dev/sda1 setenv bootargs_misc 'ro panic=10 mtdparts=nand_mtd:0x400000@0x100000(uImage),0x1fb00000@0x500000(rootfs)' setenv bootcmd_usb 'usb start; ext2load usb 0:1 0x00200000 /boot/uImage; ext2load usb 0:1 0x01100000 /boot/uInitrd' setenv bootcmd 'setenv bootargs $(console) $(bootargs_root) $(bootargs_misc) $(bootargs_more); run bootcmd_usb; bootm 0x00200000 0x01100000'Since now you are not using anymore the Marvell kernel tree it is mandatory to set these two variables and reboot:
setenv mainlineLinux yes setenv arcNumber 2097 saveenv resetInstallation to SD is identical, but requires an updated version of U-Boot with SD support. Due to a kernel bug which prevents autoloading the driver you will need to add mvsdio and mmc_block to /etc/initramfs-tools/modules and rebuild the initramfs. (Yes, the naming theme I use at home is "african dictators"...)
Gobby 0.5 (or to be more precise 0.4.92) just hit the Karmic Koala. Below you
can see a screenshot of the new version (just click on it to get a large version).
It is sadly not protocol compatible to the old version but it features local
undo and redo! Furthermore it's not yet stable on-the-wire wise. The software
itself is fairly stable but it could be a bit bumpy if the server protocol
version goes out of date.
If you want to try it out on earlier versions, I offer backports in the
Infinote PPA for Intrepid and up. I hope to get to Hardy soon, sadly it
needs some more backports.
There is a public server available on gobby.0x539.de if you want to try
it out (connections are encrypted, but all contents are public and can be
deleted by everybody, wiki-like; so please be considerate and non-destructive).
On local networks Gobby will look for servers through Avahi. That said Gobby
is not yet able to host servers by itself, like the old version did. You have
to install the infinoted server. You can do this on any computer, even on
the local network and take advantage of Avahi. If you want encrypted
connections you need to create certificates but if an unencrypted session is
acceptable you can get an ad-hoc server by invoking infinoted
--security-policy=no-tls without further configuration. It will persist
content across invocations, too.
Thanks to Greg Heynes, there will also be a KDE client for Gobby, called...
Kobby! The first time someone actually uses the fact that the software
uses a library for interoperability. Interesting that it's done now
when the library's C instead of C++ (which obby was written in), with Qt
being C++ anyway. I'm not yet aware of packages but I really hope that
we can release with both.
If you encounter problems, please report them to our Trac (preferred)
or Launchpad.
The other day, while "travelling Deutsche Bahn", I read an
article in the venerable Frankfurter Allgemeine
Zeitung. To my slight surprise, it included a link to
some government document in form of a tinyurl.com URL - (http://tinyurl.com/cr8qso).
As I had no internet acccess at the time I read the
article, I had to defer checking out the link to some later
time. Then I began to wonder how stable those URLs are
compared to the stability of the link they service and of
the newspaper article itself. The description of the link
would probably allow for some targetted google searching,
but is it not the responsibility of the newspaper to allow
their readers to research the link in a couple of days,
months or maybe years? Does tinyurl.com maybe have an
enterprise feature where they guarantee long-living links to
newspapers and similar customers?
gnutls_handshake returns once the handshake is finished. So how does one adapt this to asynchronous transput? Fortunately, there are (badly documented) hooks for this purpose.
An application can tell gnutls to call application-supplied functions instead of the read(2) and write(2) system calls. Thus, when setting up a TLS session but before the handshake, I do the following:
gnutls_transport_set_ptr(gs, this);
gnutls_transport_set_push_function(gs, push_static);
gnutls_transport_set_pull_function(gs, pull_static);
gnutls_transport_set_lowat(gs, 0);
Here, gs is my private copy of the gnutls session structure, and the push_static and pull_static are static member functions in my sesssion wrapper class. The first line tells gnutls to give the current this pointer (a pointer to the current session wrapper) as the first argument to them. The last line tells gnutls not to try treating the this pointer as a Berkeley socket.
The pull_static static member function just passes control on to a non-static member, for convenience:
ssize_t session::pull_static(void * th, void *b, size_t n)
return static_cast<session *>(th)->pull(b, n);
The basic idea of the pull function is to try to return immediately with data from a buffer, and if the buffer is empty, to fail with an error code signalling the absence of data with the possibility that data may become available later (the POSIX EAGAIN code):
class session
[...]
std::vector<unsigned char> ins;
size_t ins_low, ins_high;
[...]
;
ssize_t session::pull(void *b, size_t n_wanted)
unsigned char *cs = static_cast<unsigned char *>(b);
if (ins_high - ins_low > 0)
errno = EAGAIN;
return -1;
size_t n = ins_high - ins_low < n_wanted
? ins_high - ins_low
: n_wanted;
for (size_t i = 0; i < n; i++)
cs[i] = ins[ins_low+i];
ins_low += n;
return n;
Here, ins_low is an index to the ins vector specifying the first byte which has not already been passed on to gnutls, while ins_high is an index to the ins vector specifying the first byte that does not contain data read from the network. The assertions 0 <= ins_low, ins_low <= ins_high and ins_high <= ins.size() are obvious invariants in this buffering scheme.
The push case is simpler: all one needs to do is buffer the data that gnutls wants to send, for later transmission:
class session
[...]
std::vector<unsigned char> outs;
size_t outs_low;
[...]
;
ssize_t session::push(const void *b, size_t n)
const unsigned char *cs = static_cast<const unsigned char *>(b);
for (size_t i = 0; i < n; i++)
outs.push_back(cs[i]);
return n;
The low water mark outs_low (indicating the first byte that has not yet been sent to the network) is not needed in the push function. It would be possible for the push callback to signal EAGAIN, but it is not necessary in this scheme (assuming that one does not need to establish hard buffer limits).
Once gnutls receives an EAGAIN condition from the pull callback, it suspends the current operation and returns to its caller with the gnutls condition GNUTLS_E_AGAIN. The caller must arrange for more data to become available to the pull callback (in this case by scheduling an asynchronous write of the data in the outs buffer scheme and scheduling an asynchronous read to the ins buffer scheme) and then call the operation again, allowing the operation to resume.
The code so far does not actually perform any network transput. For this, I have written two auxiliary methods:
class session
[...]
bool read_active, write_active;
[...]
;
void session::post_write()
if (write_active) return;
if (outs_low > 0 && outs_low == outs.size())
outs.clear();
outs_low = 0;
else if (outs_low > 4096)
outs.erase(outs.begin(), outs.begin() + outs_low);
outs_low = 0;
if (outs_low < outs.size())
stream.async_write_some
(boost::asio::buffer(outs.data()+outs_low,
outs.size()-outs_low),
boost::bind(&session::sent_some,
this, _1, _2));
write_active = true;
void session::post_read()
if (read_active) return;
if (ins_low > 0 && ins_low == ins.size())
ins.clear();
ins_low = 0;
ins_high = 0;
else if (ins_low > 4096)
ins.erase(ins.begin(), ins.begin() + ins_low);
ins_high -= ins_low;
ins_low = 0;
if (ins_high + 4096 >= ins.size()) ins.resize(ins_high + 4096);
stream.async_read_some(boost::asio::buffer(ins.data()+ins_high,
ins.size()-ins_high),
boost::bind(&session::received_some,
this, _1, _2));
read_active = true;
Both helpers prune the buffers when necessary. (I should really remove those magic 4096s and make them a symbolic constant.)
The data members read_active and write_active ensure that at most one asynchronous read and at most one asynchronous write is pending at any given time. My first version did not have this safeguard (instead trying to rely on the ASIO stream reset method to cancel any outstanding asynchronous transput at need), and the code sent some TLS records twice which is not good: sending the ServerHello twice is guaranteed to confuse the client.
Once ASIO completes an asynchronous transput request, it calls the corresponding handler:
void session::received_some(boost::system::error_code ec, size_t n)
read_active = false;
if (ec) pending_error = ec; return;
ins_high += n;
post_pending_actions();
void session::sent_some(boost::system::error_code ec, size_t n)
write_active = false;
if (ec) pending_error = ec; return;
outs_low += n;
post_pending_actions();
Their job is to update the bookkeeping and to trigger the resumption of suspended gnutls operations (which is done by post_pending_actions).
Now we have all the main pieces of the puzzle. The remaining pieces are obvious but rather messy, and I d rather not repeat them here (not even in a cleaned-up form). But their essential idea goes as follows:
When called by the application code or when resumed by post_pending_actions, an asynchronous wrapper of a gnutls operation first examines the session state for a saved error code. If one is found, it is propagated to the application using the usual ASIO techniques, and the operation is cancelled. Otherwise, the wrapper calls the actual gnutls operation. When it returns, the wrapper examines the return value. If successful completion is indicated, the handler given by the application is posted in the ASIO io_service for later execution. If GNUTLS_E_AGAIN is indicated, post_read and post_write are called to schedule actual network transput, and the wrapper is suspended (by pushing it into a queue of pending actions). If any other kind of failure is indicated, it is propagated to the application using the usual ASIO techniques.
The post_pending_actions merely empties the queue of pending actions and schedules the actions that it found in the queue for resumption.
The code snippets above are not my actual working code. I have mainly removed from them some irrelevant details (mostly certain template parameters, debug logging and mutex handling). I don t expect the snippets to compile. I expect I will be able to post my actual git repository to the web in a couple of days.
Please note that my (actual) code has received only rudimentary testing. I believe it is correct, but I won t be surprised to find it contains bugs in the edge cases. I hope this is, still, of some use to somebody 
@do_something or die("blah"); being written in libraries which is a problem as it clearly violates rule (3) I wrote above, because there is no way to catch this die() error.
Furthemore, PHP has no clear concept of namespaces in the current stable versions (< 5.3). PHP 5.3 finally gets namespaces, but using a backslash character, which looks really bad and makes everyone think of eithers Windows systems or escape characters.
There are even more problems with PHP. From time to time, PHP just crashes with a segmentation fault.
MySQL vs. PostgreSQL
MySQL seems to have many problems with checking data, which can confuse me. It got better in MySQL 5 with the introduction of the new sql_modes like STRICT_ALL_TABLES and TRADITIONAL. And using storage engines like InnoDB also allows you to take advantage of stuff like foreign keys and transactions. But MySQL is still problematic.
For example, in an application I wrote I had multiple entries identified by a numeric id (an INTEGER). One day, I wanted to check whether my error handling was OK and requested an entry with the invalid ID 1X , which resulted in SELECT * from A where id='1X';. Now I expected to get an error or at least no result, but MySQL happily proceeded with the query and returned the entry with the ID 1. I tried the same thing on PostgreSQL and got an error message that 1X is not integer, as I expected it.
As another example, when I run the same PHP application using PDO on a MySQL database and a PostgreSQL database, I always get strings from the MySQL database whereas I get integers from PostgreSQL if the columns are integers.
Another thing which PostgreSQL does better than MySQL is password encryption. MySQL only provides access to the standard crypt() function and by default only generates DES passwords. It allows other methods to be used (although not all are available everywhere and therefore not documented), but it provides no way to create a salt for them, which makes them not very useful. PostgreSQL in contrast provides less encryption methods but it provides the gen_salt() function which can be used to generate salts for blowfish encryption (gen_salt('bf')).
Another problem is that MySQL simply ignores references when they are added to the column definition instead of a separate FOREIGN KEY . Because MySQL gives no errors when a feature like REFERENCES is not supported, developers may think that the feature is supported and write code using this feature just to realize later that what they wrote does not what they want.
And you can t write AGPL applications using MySQL client libraries, because MySQL is GPL-2 which is incompatible with the AGPL. You might think that the FOSS License Exception helps here, but this only lists (among others) the GPL-3. Or am I wrong?
But PostgreSQL has other problems, at least when it comes to the performance of the information_schema.columns view. This may be related to the fact that PostgreSQL has > 1000 rows whereas MySQL only has 250. And PostgreSQL may be a bit harder to configure.
Final words
All in all, PHP and MySQL are not the best choices for writing applications. They are to tolerant, and do not treat errors in an acceptable way by aborting the current action. I can get around some problems in PHP by using error handlers which raise exceptions and setting error_reporting to E_ALL, but in fact this should be the default. MySQL problems can be solved to a large extend by setting sql_mode to TRADTIONAL and using InnoDB databases, which should in my opinion be the default and not an option.
My recommendation is to use PostgreSQL servers and for the applications: use frameworks (if you are allowed to, which may not be always the case [e.g. when you are learning SQL, using a ORM is clearly the wrong way]), document your code, separate your code into classes and functions, and make sure you are not using to many global variables. And use a SCM to track your changes.
Posted in General 
With all the discussion about SHA-1 weaknesses and generation of new OpenPGP
keys going on there's some concern about how the web of trust will be affected.
I'm particularly interested in the impact on Debian; while it's possible to add
new keys and keep the old ones around that hasn't worked so well for us with
the migration away from PGPv3 keys. We still have 125 v3 keys left, many of
them for users who also have a v4 key but haven't asked for the v3 key to be
removed or responded to my email prodding them about it. I don't want to repeat
that.
| Total | SCS | Reachable | MSD | Centrality | |||
|---|---|---|---|---|---|---|---|
| 1999-02-06 (slink) | 228 | 36 | (15.78%) | 50 | (21.92%) | 2.9022 | |
| 2000-01-03 (potato) | 375 | 104 | (27.73%) | 180 | (48.00%) | 4.3382 | |
| 2001-09-22 (woody) | 948 | 538 | (56.75%) | 704 | (74.26%) | 4.7320 | 2008.6249 |
| 2005-05-28 (sarge/etch) | 1106 | 883 | (79.83%) | 969 | (87.61%) | 3.3485 | 2074.6604 |
| 2007-12-04 | 1191 | 1001 | (84.04%) | 1062 | (89.16%) | 3.1103 | 2113.3747 |
| 2009-01-18 (lenny) | 1126 | 947 | (84.10%) | 1010 | (89.69%) | 3.0489 | 1941.2594 |
| 2009-04-04 (squeeze/sid) | 1121 | 946 | (84.38%) | 1008 | (89.91%) | 3.0466 | 1936.9761 |
| 2009-05-06 (current) | 1067 | 894 | (83.78%) | 958 | (89.78%) | 2.9670 | 1759.4363 |
| Total | SCS | Reachable | MSD | Centrality | |||
| base | 1067 | 904 | (84.72%) | 959 | (89.87%) | 2.9640 | 1776.4389 |
| update-93sam | 1067 | 902 | (84.53%) | 958 | (89.78%) | 2.9734 | 1780.9874 |
| update-joerg | 1067 | 900 | (84.34%) | 958 | (89.78%) | 2.9776 | 1780.7578 |
| update-aurel32 | 1067 | 898 | (84.16%) | 957 | (89.69%) | 2.9803 | 1779.2497 |
| update-noodles | 1067 | 896 | (83.97%) | 956 | (89.59%) | 2.9831 | 1777.8326 |
| update-jaldhar | 1067 | 896 | (83.97%) | 955 | (89.50%) | 2.9855 | 1779.9193 |
| update-srivasta | 1067 | 896 | (83.97%) | 955 | (89.50%) | 2.9904 | 1784.3382 |
| update-ana | 1067 | 895 | (83.88%) | 954 | (89.40%) | 2.9926 | 1784.3102 |
| update-nobse | 1067 | 893 | (83.69%) | 953 | (89.31%) | 2.9947 | 1782.2392 |
| update-neilm | 1067 | 892 | (83.59%) | 951 | (89.12%) | 2.9974 | 1782.6098 |
| update-reg | 1067 | 891 | (83.50%) | 950 | (89.03%) | 2.9977 | 1780.8515 |
| update-rmayorga | 1067 | 890 | (83.41%) | 949 | (88.94%) | 2.9984 | 1779.4910 |
| update-evgeni | 1067 | 889 | (83.31%) | 948 | (88.84%) | 2.9974 | 1776.6445 |
Debian Times asked me a few questions
I received my first invitation to AARP membership. Aren't you supposed to be atleast 50?
Speaking of old age, the other day I overheard Shailaja tell her little brother while a Seinfeld rerun was on TV, "That's a classic. That's what they [indicating me and my wife] used to watch in the olden times."
Look what came in the mail today:
udd=> SELECT source, sum(installed_size)/1024 AS mib FROM packages WHERE distribution = 'debian' AND release = 'sid' AND component = 'main' AND architecture IN ('all', 'i386') AND section 'debug' GROUP BY source, version ORDER BY mib DESC LIMIT 30;
source mib
------------------+------
openoffice.org 1797
kde-l10n 648
gcj-4.4 544
vtk 465
linux-2.6 404
openclipart 353
vegastrike-data 311
ghc6 308
gclcvs 303
wesnoth 300
fpc 269
axiom 256
webkit 255
gcc-snapshot 255
lazarus 241
kdebase-workspace 226
plt-scheme 221
torcs-data-tracks 219
scilab 213
openscenegraph 211
eclipse 210
sagemath 201
insighttoolkit 198
acl2 195
kdebindings 181
atlas 165
gcl 163
trilinos 153
paraview 153
asterisk 144
(30 rows)
This produces a few well-known packages, but also a number of obscure ones. If you look closer, many of them appear to be themed around scientific, numerical, visualization, Scheme, Lisp, that sort of thing. Hmm.
Here is another idea. Take a package's installation footprint and multiply it by its popularity contest installation count. So you get some kind of maintenance effort score, either because the package is large or because you have a lot of users or both.
SELECT rank() OVER (ORDER BY score DESC), source, sum(installed_size::numeric * insts) AS score FROM packages JOIN popcon USING (package) WHERE distribution = 'debian' AND release = 'sid' AND component = 'main' AND architecture IN ('all', 'i386') GROUP BY source, version ORDER BY score DESC LIMIT 30;
rank source score
-----+-----------------------------+-------------
1 openoffice.org 12638492332
2 mysql-dfsg-5.0 3411344560
3 eglibc 3371485240
4 perl 3019183024
5 evolution 2669948000
6 samba 2308923872
7 mesa 1853902860
8 texlive-base 1684245516
9 gcj-4.3 1610495484
10 foomatic-db-engine 1608178104
11 foomatic-db 1423947704
12 inkscape 1413910080
13 qt4-x11 1258220636
14 gcc-4.3 1248741312
15 kdelibs 1021058256
16 gnome-applets 998434136
17 xulrunner 958232688
18 coreutils 954766896
19 openssl 877067672
20 ncurses 827679424
21 python2.5 815826384
22 aptitude 808161380
23 gimp 786015124
24 gnome-utils 781756328
25 nautilus 774319690
26 openoffice.org-dictionaries 761075576
27 eclipse 756072380
28 dpkg 736626200
29 openclipart 731244240
30 wine 707967500
(30 rows)
(Yeah, they run this thing on PostgreSQL 8.4 beta 1.)
I noticed linux-2.6 is suspiciously absent because of a low popcon score (?!?).
I don't want to dump the entire database into this blog post, but if you try this yourself you can look at about the first 200 to 300 places to find reasonably large and complex projects before it gets a bit more obscure. A few highlights:
51 gnupg 455660464
59 php5 386417572
60 mutt 381148176
83 icu 258602756
84 xorg-server 255186332
101 exim4 224857700
107 openssh 215792828
113 tar 201520400
114 postgresql-8.3 196844584
115 libx11 195856564
116 ruby1.8 194681656
272 emacs22 62047476
This is obviously still biased in a lot of ways, but it does show the major projects.
The UDD is also an interesting use case that shows how you can deploy a PostgreSQL database as a semi-public service with direct access. A great tool, and a great tool to build other great tools on top of.
I mentioned a few weeks ago that initial kernel support for the QNAP
TS-119 and TS-219 Turbo NAS got accepted for 2.6.30 but that there were
some
issues left. Here's an update:
Val ncia is again taken over by football fans, who have come from all over
the Basque Country and Catalunya to watch the Copa de Espa a final
in Mestalla stadium. The city is literally tinted in red, white, blue and
maroon and thousands of supporters (more than 60.000) have flooded the
streets.
If you can't beat them, join them, so for a change, I'm going to
join the crazyness and will go to the
Athletic Hiria to watch
the game in the middle of the leonera.
I can't wait to see how the Basque and Catalan supporters who have tickets
for the game will react when the King of Spain enters the VIP area in Mestalla.
Apparently, more powerful loudspeakers have been installed in the stadium in
an attempt to mitigate what I expect to be the biggest catcall in a Spanish
football stadium since we have a King...
Gora Bar a, Visca l'Athletic!
M. Fran ois Vannson. Les logiciels pare-feu ne risquent-ils pas, madame la ministre, de nuire aux logiciels libres ?From: http://www.assemblee-nationale.fr/13/rapports/r1626.asp Which could be approximately translated as follows:
Mme la ministre. Il n y a aucune menace sur les logiciels libres, qui peuvent tout fait accepter des logiciels de s curisation.
M. Fran ois Vannson. Mrs Minister, aren t firewall a threat on Free Software?I must hence express my real disappointment.
Mrs Minister. There is no threat on Free Software who can accept well accept securisation software.
pub 4096R/6AA15948 2009-05-10
Key fingerprint = 7A33 ECAA 188B 96F2 7C91 7288 B346 4F89 6AA1 5948
uid Ana Beatriz Guerrero L pez <ana@ekaia.org>
uid Ana Beatriz Guerrero L pez <ana@debian.org>
sub 4096R/2497B8B2 2009-05-10
personal-digest-preferences SHA256 cert-digest-algo SHA256Create key
ana@pryan:~$ gpg --gen-key
...
Please select what kind of key you want:
(1) DSA and Elgamal (default)
(2) DSA (sign only)
(5) RSA (sign only)
Your selection? 5
RSA keys may be between 1024 and 4096 bits long.
What keysize do you want? (2048) 4096
Requested keysize is 4096 bits
Please specify how long the key should be valid.
0 = key does not expire
= key expires in n days
w = key expires in n weeks
m = key expires in n months
y = key expires in n years
Key is valid for? (0)
Key does not expire at all
Is this correct? (y/N) y
Real name: Ana Beatriz Guerrero L pez
Email address: ana@ekaia.org
Comment:
You are using the utf-8 character set.
You selected this USER-ID:
Ana Beatriz Guerrero L pez <ana@ekaia.org>
Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? o
You need a Passphrase to protect your secret key.
gpg: key 6AA15948 marked as ultimately trusted
public and secret key created and signed.
gpg: checking the trustdb
gpg: 3 marginal(s) needed, 1 complete(s) needed, classic trust model
gpg: depth: 0 valid: 3 signed: 0 trust: 0-, 0q, 0n, 0m, 0f, 3u
pub 4096R/6AA15948 2009-05-10
Key fingerprint = 7A33 ECAA 188B 96F2 7C91 7288 B346 4F89 6AA1 5948
uid Ana Beatriz Guerrero L pez <ana@ekaia.org>
Note that this key cannot be used for encryption. You may want to use
the command edit-key to generate a subkey for this purpose.
Add other UID
ana@pryan:~$ gpg --edit-key 0x6AA15948
...
command> adduid
Real name: Ana Beatriz Guerrero L pez
Email address: ana@debian.org
Comment:
You are using the utf-8' character set.
You selected this USER-ID:
"Ana Beatriz Guerrero L pez <ana@debian.org>"
Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? o
You need a passphrase to unlock the secret key for
user: "Ana Beatriz Guerrero L pez <ana@ekaia.org>"
4096-bit RSA key, ID 6AA15948, created 2009-05-10
pub 4096R/6AA15948 created: 2009-05-10 expires: never usage: SC
trust: ultimate validity: ultimate
[ultimate] (1) Ana Beatriz Guerrero L pez <ana@ekaia.org>
[ unknown] (2). Ana Beatriz Guerrero L pez <ana@debian.org>
Command> save
Change preferences
ana@pryan:~$ gpg --edit-key 0x6AA15948
...
Command> showpref
[ultimate] (1). Ana Beatriz Guerrero L pez <ana@debian.org>
Cipher: AES256, AES192, AES, CAST5, 3DES
Digest: SHA1, SHA256, RIPEMD160
Compression: ZLIB, BZIP2, ZIP, Uncompressed
Features: MDC, Keyserver no-modify
[ultimate] (2) Ana Beatriz Guerrero L pez <ana@ekaia.org>
Cipher: AES256, AES192, AES, CAST5, 3DES
Digest: SHA1, SHA256, RIPEMD160
Compression: ZLIB, BZIP2, ZIP, Uncompressed
Features: MDC, Keyserver no-modify
Command> setpref SHA512 SHA384 SHA256 SHA224 AES256 AES192 AES CAST5 ZLIB BZIP2 ZIP Uncompressed
Set preference list to:
Cipher: AES256, AES192, AES, CAST5, 3DES
Digest: SHA512, SHA384, SHA256, SHA224, SHA1
Compression: ZLIB, BZIP2, ZIP, Uncompressed
Features: MDC, Keyserver no-modify
Really update the preferences? (y/N) y
gpg: WARNING: no user ID has been marked as primary. This command may
cause a different user ID to become the assumed primary.
You need a passphrase to unlock the secret key for
user: Ana Beatriz Guerrero L pez <ana@debian.org>
4096-bit RSA key, ID 6AA15948, created 2009-05-10
pub 4096R/6AA15948 created: 2009-05-10 expires: never usage: SC
trust: ultimate validity: ultimate
[ultimate] (1) Ana Beatriz Guerrero L pez <ana@debian.org>
[ultimate] (2). Ana Beatriz Guerrero L pez <ana@ekaia.org>
Command> showpref
[ultimate] (1) Ana Beatriz Guerrero L pez <ana@debian.org>
Cipher: AES256, AES192, AES, CAST5, 3DES
Digest: SHA512, SHA384, SHA256, SHA224, SHA1
Compression: ZLIB, BZIP2, ZIP, Uncompressed
Features: MDC, Keyserver no-modify
[ultimate] (2). Ana Beatriz Guerrero L pez <ana@ekaia.org>
Cipher: AES256, AES192, AES, CAST5, 3DES
Digest: SHA512, SHA384, SHA256, SHA224, SHA1
Compression: ZLIB, BZIP2, ZIP, Uncompressed
Features: MDC, Keyserver no-modify
Command> save
Set primary UID
ana@pryan:~$ gpg --edit-key 0x6AA15948
...
Command> uid 1
pub 4096R/6AA15948 created: 2009-05-10 expires: never usage: SC
trust: ultimate validity: ultimate
[ultimate] (1)* Ana Beatriz Guerrero L pez <ana@ekaia.org>
[ultimate] (2) Ana Beatriz Guerrero L pez <ana@debian.org>
Command> primary
You need a passphrase to unlock the secret key for
user: Ana Beatriz Guerrero L pez <ana@ekaia.org>
4096-bit RSA key, ID 6AA15948, created 2009-05-10
pub 4096R/6AA15948 created: 2009-05-10 expires: never usage: SC
trust: ultimate validity: ultimate
[ultimate] (1)* Ana Beatriz Guerrero L pez <ana@ekaia.org>
[ultimate] (2) Ana Beatriz Guerrero L pez <ana@debian.org>
Command> save
Add subkey for encryption
ana@pryan:~$ gpg --edit-key 0x6AA15948
...
Command> addkey
Key is protected.
You need a passphrase to unlock the secret key for
user: "Ana Beatriz Guerrero L pez <ana@ekaia.org>"
4096-bit RSA key, ID 6AA15948, created 2009-05-10
Please select what kind of key you want:
(2) DSA (sign only)
(4) Elgamal (encrypt only)
(5) RSA (sign only)
(6) RSA (encrypt only)
Your selection? 6
RSA keys may be between 1024 and 4096 bits long.
What keysize do you want? (2048) 4096
Requested keysize is 4096 bits
Please specify how long the key should be valid.
0 = key does not expire
= key expires in n days
w = key expires in n weeks
m = key expires in n months
y = key expires in n years
Key is valid for? (0) 0
Key does not expire at all
Is this correct? (y/N) y
Really create? (y/N) y
pub 4096R/6AA15948 created: 2009-05-10 expires: never usage: SC
trust: ultimate validity: ultimate
sub 4096R/2497B8B2 created: 2009-05-10 expires: never usage: E
[ultimate] (1). Ana Beatriz Guerrero L pez <ana@ekaia.org>
[ultimate] (2) Ana Beatriz Guerrero L pez <ana@debian.org>
Command> save
Sign my new key with my old key
ana@pryan:~$ gpg --default-key E8C43461 --sign-key 6AA15948
pub 4096R/6AA15948 created: 2009-05-10 expires: never usage: SC
trust: ultimate validity: ultimate
sub 4096R/2497B8B2 created: 2009-05-10 expires: never usage: E
[ultimate] (1). Ana Beatriz Guerrero L pez <ana@ekaia.org>
[ultimate] (2) Ana Beatriz Guerrero L pez <ana@debian.org>
Really sign all user IDs? (y/N) y
pub 4096R/6AA15948 created: 2009-05-10 expires: never usage: SC
trust: ultimate validity: ultimate
Primary key fingerprint: 7A33 ECAA 188B 96F2 7C91 7288 B346 4F89 6AA1 5948
Ana Beatriz Guerrero L pez <ana@ekaia.org>
Ana Beatriz Guerrero L pez <ana@debian.org>
Are you sure that you want to sign this key with your
key Ana Beatriz Guerrero L pez <ana@ekaia.org> (E8C43461)
Really sign? (y/N) y
You need a passphrase to unlock the secret key for
user: Ana Beatriz Guerrero L pez <ana@ekaia.org>
1024-bit DSA key, ID E8C43461, created 2004-12-06
Send new key to key server
gpg --keyserver pgp.mit.edu --send-key 6AA15948
Hi Ondrej,
How are you?
I am about to publish a new free software project, a new simple PHP framework, and I am interested in your advice.
You started SymPy and were able to make other people join you and develop it with you.
How did you do it?
How did it happen?
Did you actively call for other people or they spontaneously showed interest and joined you?
Are the other major contributor people who were your friends before you started the project?
Did you need to create or manage the project in a particular way to make it attractive to other people?
Are there things you are aware of that promote collaboration or demote it?
I was never successful in doing the same with Winpdb, which while it became reasonably popular, no one has ever joined me to develop it, except for a notable tutorial contribution by Chris Lasher which was developed independently.
Now with the new project, I am wondering what are my chances of making other people try it and take it on. On the one hand it is a new and fresh code base in an interesting field, on the other hand, why would anyone bother to spend their energy on this new project when they have Symfony or Drupal?
What do you think?
BTW, Ohloh believes you have a median of 19,000 lines of changed code per month since the start of their log. Can this be true? Is this humanly possible? According to it SymPy has over 1,000,000 lines of code? I can't understand these numbers. Winpdb has about 25,000 lines after 3 years of development. And from my experience 1,000,000 lines of code projects need about 20-50 full time developers to work on for 2-5 years which is about 40-250 man years. And as if this is not enough you are listed as owner in a dozen other projects in Google code and have enough time to become an awarded scientist. How is this possible?
http://www.ohloh.net/p/sympy/contributors/
BTW2, do you still use Winpdb? If you find yourself using it less, can you say what are the reasons, or what it would take to make it more useful?
BTW3, How is SymPy doing?
Cheers,
Nir
Are the other major contributor people who were your friends before you started the project?No, not a single major contributor was my friend before I started the project. Every single one of them become a developer using the procedure I described above, e.g. first showed on the list or in the issues, and maybe even the very first patch was not a high quality one (and if I was stupid and arrogant, or didn't see the big potential, I would just ignore them). But when given a chance, they became extremely good developers and sympy would simply just not be here without them.
Did you actively call for other people or they spontaneously showed interest and joined you?I very much encourage everyone to contribute, but the initial interest must be in them, e.g. they at least have to show around the mailinglist/issues, so that I know about them. But once I know they are interested in some issue, yes, I try to invite them to fix it, with my help.
Next.