Since my last report, a lot of visible changes have been made. First, if you edit the value of any of the text inputs on the dashboard table, your changes will be saved when you hit the submit button. This feature still needs a bit of work though. For example, I noticed during testing that if you submit the form multiple times, you will sometimes end up with approximately duplicate entries in the changes file (the file that stores user-submitted changes to the table). I also need to implement some checks to ensure that the values submitted are actually valid (i.e. we probably only want to allow a few possible values for the status column). Another big improvement is the addition of a graph at the bottom of each project s page. This graph tracks the number of open tasks versus time. The graph is re-generated once each day based on a progress file. This file simply lists a date and the number of open tasks on each line. While the graph does correctly show a line tracking the progress, it still has some issues to sort out. For example, the x-axis does not properly display a time increment. Work on the graph was blocked for a while due to not having a fully-functioning dashboard. Now that the dashboard is functioning, I should be able to start working on fixing these graph issues. Several people approached me to request the ability to link directly to a particular project. That way, users do not need to mess around with the select boxes. Above each table is a title. If you click on the title, the URL should update to link directly to the project. This is done with the distro and project query parameters. For example, a link to dex.html?distro=ubuntu&project=ancient-patches will take you directly to Ubuntu s ancient-patches project. Attempting to specify an invalid distro or project will cause the query parameters to be ignored. While looking at the table title, you will also notice a new (Graph) link. This link will take you directly to the bottom of the page so you can view the graph. Currently, it is not possible to link directly to the graph anchor tag of a specific project. Attempting to do so will cause the #graph portion of the URL to be ignored. However, if you click on the actual graph, it will load a full-size graph.png that you can link to. One rather significant issue that has shown up is that the table is very wide. On most monitors, you will have to scroll horizontally to view all of the columns. This is due to the fact that most of the cells are made up of text inputs (they don t have a background or border, but if you click on some text in the table, you should see the box) rather than plain text. The size parameter for a text input was meant to specify how many characters should be visible. However, while I have the size parameter set correctly for each input, it is showing up much wider than the text. I will continue to play around with this, since excluding the size parameter results in some values getting cut-off. As mentioned in my last report, I have continued to work with Allison Randal to make the dashboard more useful. I have received feedback directly from her and indirectly from other people working on the python2.7 project. During my last phone call with my mentor, Stefano Zacchiroli, we agreed that it would be very helpful to spend some time documenting how to go about adding a new project to the dashboard and how to utilize all of the dashboard s features. The goal of this is to not require people to go through me or any other individual to be able to use the dashboard. So while the Summer of Code is meant to focus on programming, I will probably spend some time adding some documentation, as a program is rather useless if nobody knows how to use it. Another bug that I will be working on has to do with sorting the table. I had this feature working early on in the project. However, the sorting broke when I starting using text inputs instead of plain text. I have done some research, and I hope to get the sorting re-implemented in the near future. One other feature that has been requested by several people is the ability to hide completed tasks. This has been on my todo list for a while. Matt Zimmerman wanted to take this a step farther. He felt that the status of the task should be separate from the bug. This means that you can have an open task with a closed bug or a closed task with an open bug. The user could then have some more control over what data they are seeing. I will probably attempt to add some check boxes to control whether certain information is displayed. The user will also be able to specify this directly in the URL. This means that a team could link directly to all open tasks in a project. It is also worth noting that thanks to some help from the fantastic Alioth admins, I now have a live instance of the dashboard running on Alioth. While the URL might change in the future, I hope to keep it running on Alioth. If you are interested in helping out with DEX or dextools, my code is available in a git repository on alioth. You can also join the debian-derivatives mailing list, join #debian-derivatives on OFTC, or contact me directly via email or on IRC (nhandler).

This is a rant. I've made it in various forms in various places over the years, and I think it's time to put it somewhere so I can point people at it, and stop repeating myself. Here's the pebble in my shoe: people keep claiming that strncpy is a safer alternative to strcpy. It's not. It's a slower function that is not going to make you significantly safer. The problem with strcpy is that it's somewhat easy to overflow the target array:

char src[] = "hello, world";
char dst[5];
strcpy(dst, src); // Oops.

The suggested alternative with strncpy would be this:

strncpy(dst, src, sizeof(dst));

strncpy gets the size of the destination array as a parameter, and can therefore make sure the string will fit there. Problem solved? Not so much. Every time you do any string handling with C's native string types and functions, you need to make sure you have allocated enough memory for each string, you need to keep track of how much memory you've allocated, and you neet to make sure the zero byte ('NUL', '\0') is put there correctly. This is a lot of details to keep track of, and it's not surprising things sometimes go wrong. If your arrays are allocated statically, which is the simplest, but least useful case, you need to something like this with strcpy:

if (strlen(src) + 1 > sizeof(dst))
    panic(); // or however you want to handle your errors
strcpy(dst, src);

The +1 after strlen is to make room for the zero byte (NUL, '\0') that is used in C to indicate the end of a string. With strncpy, your code is slightly simpler. This is its attraction. Problem is, this only works if dst is an array defined within the scope of the code. If, as is common, you have received dst as an argument to a function, sizeof(dst) is the size of a pointer to an element of the array, not the size of the array. So now you need to track the size explicitly in a second variable:

strncpy(dst, src, dst_size);

Further, strncpy has an unfortunate error handling mode: it merely fills the destination array and stops. It does not indicate failure in any way. It does not terminate the target array with a zero byte. Since all C string processing assume the zero byte is there, as soon as you try to use dst as a string, you're in trouble. But luckily you can check for that before you call strncpy:

if (strlen(src) + 1 > dst_size)
    panic(); // or whatever
strncpy(dst, src, dst_size);

By this time, the code is equally hard whether you use strcpy or strncpy. With strncpy you often waste some extra time, though, filling the unused parts of the target array with zero bytes: they're almost never necessary. Alternatively, you can force the zero byte there yourself, silently truncating a string.

strncpy(dst, src, dst_size);
dst[dst_size - 1] = '\0'; // let's hope dst_size > 0

If your string handling is OK with strings being randomly shorter than they should be, possibly you could replace all of them with empty strings, and save yourself a world of pain. Instead of strncpy, use snprintf. It has sensible error behavior, and is much more versatile:

n = snprintf(dst, dst_size, "%s", src);
if (n >= dst_size)
    panic();

snprintf still isn't very nice to use. It still requires you to allocate memory yourself, and keep track of the size of each array manually. It's also slower for small destination buffers (but faster for large ones). The real solution is to abandon C's antiquated string handling, and use a more sensible helper library instead. There's a bunch of them around, though none have gained much popularity. I'm not going to name any of them, but just imagine how much nicer it would be to write code like this instead:

Dynstr *src = dynstr_new_from_cstring("hello, world\n");
Dynstr *cat = dynstr_cat_many(src, src, src, src, src, NULL);
Dynstr *dog = dynstr_substr(cat, 15, 5);

Let the library take care of memory allocations and related tracking, and concentrate on real problem solving instead. We did this in Kannel, back in 1999, and it wiped out all the buffer overflow problems (until a co-worker decided it was "more efficient" to use native C strings instead, promptly introducing a buffer overflow). All of this string handling is, of course, way nicer in higher level languages. In fact, whenever you need to deal with string handling in C, it is probably a sign that you should consider switching to a higher level language instead. But whatever you do, stop claiming that strncpy is safe or solves any real problems strcpy has. Update: Fixed "It does terminate" to say "It does not terminate". Thanks to Matthew Garrett for pointing that out. Update2: Fixed example call to snprintf. Update3: Fixed the test for snprintf example. Thanks, Redditor axylone. Also pointed out that snprintf is slower than the rest.

You want to start contributing to Debian and/or Ubuntu, you decided to help a package maintainer and you re now looking for how to change a source package and how to submit your changes. 1. Retrieve the source package and install build-dependencies The first step is to retrieve the latest version of the source package and to install the required build-dependencies. I already covered how to do this with apt-get in the article explaining how to rebuild a source package. If you prefer, you can use dget (from the devscripts package) to directly grab the source package. You can find the URL of the .dsc files in the Package Tracking System for example. Sometimes apt-get will warn you that the source package is maintained in a VCS repository, like this:

$ apt-get source wordpress
[...]
NOTICE: 'wordpress' packaging is maintained in the 'Git' version control system at:
git://git.debian.org/git/collab-maint/wordpress.git
[...]

In that case, you can use debcheckout to retrieve the VCS repository instead (provided that you have the corresponding VCS installed):

$ debcheckout wordpress
declared git repository at git://git.debian.org/git/collab-maint/wordpress.git
git clone git://git.debian.org/git/collab-maint/wordpress.git wordpress ...
Cloning into wordpress...

Note however that some maintainers use their VCS in a way that s not really compatible with the explanations that I will give below. It s also a good idea to install the package packaging-dev . It s a meta-package depending on the most common tools that are used for Debian packaging work. 2. Do the changes Execute dch --nmu to record the fact that you re working on an update prepared by someone who is not the maintainer (NMU means Non Maintainer Upload). This also ensures that if we build the package, we won t overwrite the original source package that we downloaded, thus making it possible to generate a diff between both versions. 2.1. Modify Debian packaging files Now fire your text editor and do the required changes in the debian sub-directory. You will probably run dch -a multiple times to document each subsequent change. 2.2. Modify upstream files If you have to modify upstream files, the proper way to do it depends on the source package format ( 1.0 vs 3.0 (quilt) vs 3.0 (native) , see the debian/source/format file) and on the presence or not of a patch system (the what-patch can help you identify it). In this explanation, I ll assume that the package is using the recommended format: 3.0 (quilt) . (It also works for 1.0 if quilt is used and if you configured ~/.quiltrc as recommended by /usr/share/doc/quilt/README.source). First you should ensure that all patches have been applied with quilt push -a. If there s no patch yet, you want to create the debian/patches directory (with mkdir debian/patches). 2.2.1. Import a patch If the upstream changes are already in a patch file (say /tmp/patch that you downloaded from the upstream VCS repository) you can import that patch like this:

$ quilt import -P fix-foobar.patch /tmp/patch
Importing patch /tmp/patch (stored as fix-foobar.patch)
$ quilt push
Applying patch fix-foobar.patch
[...]
Now at patch fix-foobar.patch

The -P option allows to select the name of the patch file created in debian/patches/. As you see, the new patch file is recorded in debian/patches/series but not applied by default, we re thus doing it with quilt push. 2.2.1. Create a new patch If the upstream changes that you want to make are not in a patch yet, you should tell quilt that you re going to create one:

$ quilt new fix-foobar.patch
Patch fix-foobar.patch is now on top

Then you must record every file that you re going to modify with a quilt add invocation. quilt then makes a backup of those files so that it can generate the patch later on. If you re going to modify the files with your text editor you can just do quilt edit file-to-modify, it s the same than quilt add file-to-modify followed by sensible-editor file-to-modify.

$ quilt edit foobar.c
File foobar.c added to patch fix-foobar.patch

The last step is tell quilt to generate the patch:

$ quilt refresh
Refreshed patch fix-foobar.patch

3. Test your changes You should build your modified package with debuild -us -uc . You can easily install the resulting package with debi . Verify that everything works as expected. If not, continue your modifications until you re satisfied with the result. 4. Generate a patch and mail it If you followed the instructions, you should have two .dsc files in the parent directory, like this:

$ cd ..
$ ls wordpress_*.dsc
../wordpress_3.0.5+dfsg-1.1.dsc
../wordpress_3.0.5+dfsg-1.dsc

Generating the patch to send to the maintainer is then just a matter of running debdiff:

$ debdiff wordpress_3.0.5+dfsg-1.dsc wordpress_3.0.5+dfsg-1.1.dsc >/tmp/wp-debdiff

You can send the /tmp/wp-debdiff file to the wordpress maintainer. Usually you send it via the bugreport that your update is fixing and you add the patch tag to the report. This can be automated with the nmudiff utility. By default it assumes that you re using mutt but it can also directly feed the resulting mail to sendmail. The default text that nmudiff proposes assumes that you re actually performing an NMU and that the result has been uploaded. If that s not the case, you should edit the text and make it clear that you re just sending a patch. If you have been working in a VCS repository, instead of using debdiff you can simply use the diff feature integrated in your VCS (git diff, svn diff, etc.). But note that with a distributed VCS (like git/bzr/mercurial, unlike svn) you should probably have committed all individual changes in separate changesets. And instead of sending a single patch, you re probably going to send a series of patches (though it might be easier to just upload your branch in a public repository and give the corresponding URL to the maintainer).

Found it useful? Be sure to not miss other packaging tips (or lessons), click here to subscribe to my free newsletter and get new articles by email.

No comment Liked this article? Click here. My blog is Flattr-enabled.

There are two different uses (within Debian/Ubuntu) of the initialism UDD: Ubuntu Distributed Development and the Ultimate Debian Database. This article talks about the latter. At UDS-O in Budapest a few weeks ago we had a session (video) on DEX: an initiative to improve collaboration between Debian and its derivatives. There's an Ubuntu DEX subproject, and this is what the session was about, thinking up ideas. I thought of something that could possibly yield some results --- my action from the session was:

[laney] measure relative uploads between ubuntu and debian and see if anything interesting pops up: TODO

It seems to me that if we could tell which packages were uploaded with the most frequency to Ubuntu with respect to Debian, then we may be able to figure out why this is and if there's any way we can do the work in Debian instead. Clearly for this to happen we are going to need data! And where better to house a load of data than the Ultimate Debian Database itself? So I wrote some scripts to download the Ubuntu changes, handily made available as mbox archives, chop them around a bit and eventually split them out in a format that UDD can understand. Lucas Nussbaum then kindly integrated the output into UDD itself, and the end result is that an as-near-as-possible complete history of Ubuntu uploads is now available for your creative querying. For example, Wednesday sees the most uploads and Sunday the fewest:

udd=> SELECT EXTRACT (DOW FROM date) AS dow, COUNT(EXTRACT (DOW
FROM date)) AS count FROM ubuntu_upload_history GROUP BY dow ORDER
BY count DESC;
 dow   count 
-----+-------
   3   26739
   2   26699
   1   25361
   5   23497
   4   22154
   6    9687
   0    8752
(7 rows)

I'll lovingly craft a script to generate the data we need for the DEX project in the next few days. But I hope that the data proves useful for other things too. The three tables you want to look at (with \d+ at a psql prompt) are ubuntu_upload_history, ubuntu_upload_history_launchpad_closes (LP bugs closed by uploads) and ubuntu_upload_history_closes (Debian bugs closed, probably not so interesting). Information on how to use UDD yourself is on this wiki page.

Since some time I ve been thinking about a possible way to delete my Gmail s Trash & Spam folders content automatically without having to bother doing it manually every single time I wanted to check my mail and clean it up. (I love keeping everything in place and having my Trash&Spam folders empty as they should be makes me pretty happy) A few years ago when Mutt was my main mail client I had the need to filter my mail through IMAP and while googling around for that I found out a great piece of software: imapfilter. Today while analyzing the above quoted issue I suddenly told myself: Hey, but why don t you use your dear and old friend imapfilter to fulfil your needs? After a few minutes I came up with a small lua script that was doing exactly what I wanted: my Trash&Spam folders are no longer crowded and I finally don t have to delete mails twice! But here they come a few details about my script:

options.timeout = 120
options.subscribe = true
account = IMAP  
 server = 'imap.gmail.com',
 username = 'example@gmail.com',
 password = 'password',
 ssl = 'ssl3'
  
trash = account['[Gmail]/Trash']:is_undeleted()
account['[Gmail]/Trash']:delete_messages(trash)
spam = account['[Gmail]/Spam']:is_unanswered()
account['[Gmail]/Spam']:delete_messages(spam)

The script does two things:

1. It checks whether a mail is not marked as deleted (moving an e-mail into the Trash does not mark it as to be deleted automatically) already and removes it.
2. It checks whether a mail on the Spam folder has been answered (I never had to answer a single e-mail contained into my Spam folder) and if not removes it.

Using the above script is really easy (you should run imapfilter on interactive mode first to generate Gmail s certificates, do that before having cron to run the script for you or otherwise it ll just hang), just make sure to have imapfilter installed on your system and then run it through cron every half an hour or less depending on your needs:

crontab -e

*/30 * * * * imapfilter -c /home/user/imapfilter.lua >> /home/user/imapfilter.log

Please also remember to setup appropriate permissions on the config file since it contains your Gmail s password and most of all make sure that your Spam folder is visible through IMAP (this option can be found on the label menu available under your Gmail s settings) otherwise imapfilter will just report an error. Enjoy!

<meta content="text/html; charset=utf-8" equiv="content-type">

Hello,

This is the third report of my GSoC project for Debian. During last two weeks

I've done most of things about the main tool. Before that weeks I have been

working on project's API. API is almost finished. Still I find missing/wrong

parts during the coding and changing the API design.

Like I said, main tool's coding process is nearly finished. For now It is doing

its job for building Python3.X packages. You can find the source code at [1].

I also coded an example plugin (actually only return values) and plugins/__init__.py

file which contains like Base class, exception classes and plugin listing method.

When I started coding I realized that I need extra parameters. I've added --install-dir

parameter. You can specify the install directory if there is no install directory the tools

pretends that the install directory is "./debian/tmp". After that it checks whether if the

directory exists. If not it creates the directory.

I've changed py3build.py a bit. First, I've thought for every process I invoke py3build's

methods but they are redundant. I've removed them. Now they are called in main

method of py3build.

I use logging module for printing messages, errors etc.

get_build_system() method retieves list from plugins/__init__.py file. In that file there is

scan_for_plugins() method. It finds all plugins in plugins/ directory which files match for

"[a-zA-Z].py". It keeps the list in plugins_list variable and returns that. When I'm writing

scan_for_plugins() method I've read codes (f.e. [2]) about that and decided to name all

plugins' class name as *BuildSystem*. Now, detecting method works with that way.

(all plugins' class' are called BuildSystem) And I've changed base class name to

BuildSystemBase.

get_build_system() method is used by determine_build_system() method. If there is given

build system (by parameter) it tries to use that. If that build system is not usable then give

error and exits. If there is no given build system. It invokes get_build_system() and iterates

that methods return value which is a list. When iterating It create an instance of that build

system class (if can not create then it gives warning and skips that build system plugin)

and invokes detect method and append plugin name and detect method's return value to

the bs_list which contains plugin list which are available. And then sorts it. If there is error

while sorting list or there is no usable plugin then give error and exits. If there is multiple

highest detect values then lists that plugins and exists. (maintainer should select one of

these plugins to build extension.)

In main method, after getting arguments, it checks for verbose mode (or 'DH_VERBOSE'

environment variable) and sets log level (logging.DEBUG or logging.WARN)

If there is no given build_dir then it pretends it is "./".

After that It try to determine build system. If can not then prints error and exits.

And finally it starts to build extension. Firstly it checks if there is given version list. If not,

it uses default values and iterates build loop with that values.

First cleans, configures, builds (if there is no --test-only arg) and tests (if there is --test-only

arg or there is not --no-tests argument)

If there is error (raises plugin.Error) while invoking that method then prints error and exits.

In verbose mode tool prints every process. (like "configuring for PythonX.Y" etc.)

I've started writing a plugin for detecting *distutils*. For now, I only wrote its detect method. It

returns between 0-1.0 floating point value. If everything is OK it returns 1.0. Other methods does

nothing, only returns True but this week I'll finish this plugin and work for others.

For detecting distutils I've used the code at [3]. First I try to import that module if not, I've copied

required methods to plugins/_packaging.py file, I use them.

I don't use is_distutils() method directly. For give probability I invoke what it invokes and give them

a number. (f.e. _has_setup_py has 0.5, _has_pkg_info has 0.25 and _has_distutil_text has 0.25.

But if _has_setup_py returns False then detect method doesn't look other methods since they

require setup.py.)

For now that's all. See you at next time.

[1] git://git.debian.org/~<wbr>mesutcan-guest/py3multibuild.<wbr>git
[2] http://svn.berlios.de/<wbr>viewvc/griffith/trunk/lib/<wbr>plugins/extensions/

[3] http://hg.python.org/<wbr>cpython/file/a7e00934baf1/Lib/<wbr>packaging/util.py

--
Mesutcan Kurt

Hello Creating a configuration model for Config::Model can be a boring task: When a configuration system has a lot of parameters, a lot of details need to be specified to have a complete configuration validation tool. So, all tricks are permitted to make this task more bearable. Let s see how this can be applied to lcdproc. This project requires a configuration file that supports more than 40 drivers. Each driver has it own set of parameters for a total of about 250 parameters. That s a lot to specify.. Fortunately, LCDd.conf template file is written in a way which makes it relatively easy to parse and get all required information to build a model. All drivers are listed in this template file, most parameters have default values and legal values are written in comments in a uniform way. Hence this file (and comments) can be parsed to retrieve information required for the model. This blog will describe how this task can be implemented in Perl using Config::Model modules. The code will be slightly trimmed down to keep this blog (relatively) short. All initialisation code and error checks are removed. You can get the whole script on sourceforge. The main steps are:

1. prepare a parser for LCDd.conf template
2. parse LCDd.conf template
3. prepare a LCDd model skeleton
4. translate LCDd.conf template information in a format suitable to create a model.
5. Write the resulting LCDd model

First, here s the necessary incantation to start properly a Perl program:

use strict;
use warnings;

Then load all required libraries. Their use will be described later:

use Config::Model;
use Config::Model::Itself 1.225;
use Config::Model::Backend::IniFile;
use 5.10.0;
use IO::File;
use IO::String;

Perform some initialisation to avoid upsetting the logging system:

use Log::Log4perl qw(:easy);
Log::Log4perl->easy_init($WARN);

Let s also initialize a model object that will be very useful later:

my $model = Config::Model->new();

Now we can really start the fun part. Step 1: Create a parser for LCDd.conf (INI file)
Problem: LCDd.conf comments must also be analysed to get more semantic information. The comments must be retrieved and associated with the INI classes and parameters found in this file. Fortunately, Config::Model::Backend::IniFile is able to associate comments with the relevant parameters. But Config::Model::Backend::IniFile must store its values in a configuration tree. So let s create a model suitable for LCDd.conf template that accepts any INI class and any INI parameter Dummy::Class is used to store any parameter found in an INI class. The accept keyword with .* regular expression means that any INI parameter will be accepted in each INI class.

$model->create_config_class(
  name   => 'Dummy::Class',
  accept => [
    '.*' =>  
       type => 'leaf',
       value_type => 'uniline'
        ,
    ],
);

Now the Dummy model must provide a configuration class to hold all INI classes:

$model->create_config_class(
    name   => 'Dummy',
    accept => [
       '.*' =>  
           type => 'node',
           config_class_name => 'Dummy::Class'
        ,
    ],
);

Note that an INI backend could be specified directly here. But, some useful parameters are commented out in LCD.conf. Some some processing is required to be able to create a model with these commented parameters. See below for this processing. Step 2: Parse LCDd.conf Now the dummy configuration class is created. Let s create a configuration tree from this dummy class to store the data from LCDd.conf

my $dummy = $model->instance(
    instance_name   => 'dummy',
    root_class_name => 'Dummy',
)-> config_root;

Now, read LCDd.conf:

my $lcd_file
  = IO::File->new('examples/lcdproc/LCDd.conf');
my @lines    = $lcd_file->getlines;

And perform the pre-processing on LCDd.conf mentioned above. Just un-commenting commented parameters is required:

foreach (@lines)   s/^#(\w+=)/$1/  

Store the pre-processed LCDd.conf in a IO::Handle usable by INI backend

my $ioh = IO::String->new( join( '', @lines ) );

Create an INI backend to read the data from pre-processed LCDd.conf and store it in the configuration tree:

my $ini_backend = Config::Model::Backend::IniFile
  ->new( node => $dummy );
$ini_backend->read( io_handle => $ioh );

Step 3: Prepare LCDd configuration model Your attention please. You may remember that we invoked instance on object to create the dummy configuration tree to hold LCDd.conf data. The instance method used the Dummy configuration class. Now, we must invoke instance again on object. But this time, instance will use Config::Model::Itself. This model is Config::Model s way of eating its own dog food. The configuration tree created there will be able to hold LCDd model. Hence the root of this special configuration tree will be called meta_root to avoid confusion with the other configuration root defined above (for the Dummy model). Still with me? Good, let s create this meta tree that will contain LCDd model:

my $meta_root = $model->instance(
    root_class_name => 'Itself::Model',
    instance_name   => 'meta_model',
) -> config_root;

Create the main LCDd configuration class and store the first comment from LCDd.conf as class description:

my $first_comment = $dummy->annotation ;
$meta_root
  ->grab("class:LCDd class_description")
  ->store( $first_comment );

Append my own text in this description:

my $extra_description =
    "Model information extracted from template /etc/LCDd.conf"
  . "\n\n=head1 BUGS\n\nThis model does not support to load "
  . "several drivers. Loading several drivers is probably a "
  . "marginal case. Please complain to the author if this "
  . "assumption is false";
$meta_root->load(
     qq!class:LCDd
        class_description.="\n\n$extra_description"!
);

Add legal stuff in the model:

$meta_root->load( qq!
  class:LCDd
    copyright:0="2011, Dominique Dumont"
    copyright:1="1999-2011, William Ferrell and others"
    license="GPL-2"
!
);

Add INI backend so configuration object created from LCDd model will be able to read user s INI files:

$meta_root->load( qq!
  class:LCDd
    read_config:0
      backend=ini_file
      config_dir="/etc"
      file="LCDd.conf"
!
);

Note that all the load calls above could be done as one call. They were split for better clarity. Step 4:Extracting LCDd.conf information Now that all information from LCDd.conf template file is stored in $dummy configuration tree, it s time to use this information to enhance the skeleton of LCDd model created above. Let s first find all the INI classes defined in LCDd.conf:

my @ini_classes = $dummy->get_element_name;

Now before actually mining LCDd.conf information, we must prepare several subs to handle them. This is done using a dispatch table:

my %dispatch;

First create the default entry which will be used for most parameters. This subs is passed: the INI class name, the INI parameter name, the comment attached to the parameter, the INI value, and an optional value type. It will return a string describing a model for simple elements. For instance: class:"LCDd::MD8800" element:Device type=leaf value_type=uniline default="/dev/ttyS1" description="device to use [default: /dev/ttyS1]" Here s the subroutine invoked by default. Let s create a an anonymous subroutine and store it in the dispatch table:

$dispatch _default_  = sub  
    my ( $ini_class, $ini_param, $ini_note,
         $ini_v, $value_type ) = @_;
    $value_type  = 'uniline';

Now, prepare a string that will be used later to create the ini_class model. This string uses Config::Model s serialisation format described in Config::Model::Loader:

    my $load = qq!class:"$ini_class"
                  element:$ini_param type=leaf !;

Get semantic information from LCDd.conf template comments (written between square brackets):

    my $info = $ini_note =~ /\[(.*)\]/ ? $1 : '';
    $info =~ s/\s+//g;

Use this semantic information to better specify the parameter:

    $load .=
        $info =~ /legal:(\d+)-(\d+)/
           ? " value_type=integer min=$1 max=$2"
      : $info =~ /legal:([\w\,]+)/
           ? " value_type=enum choice=$1"
      :    " value_type=$value_type";
    if ( $info =~ /default:(\w+)/ )  
        # specify upstream default value
        # if it was found in the comment
        $load .= qq! upstream_default="$1"!
           if length($1);
     
    else  
        # or use the value found in INI
        # file as default
        $ini_v =~ s/^"//g;
        $ini_v =~ s/"$//g;
        $load .= qq! default="$ini_v"!
           if length($ini_v);
     

The model (in serialised format) is ready, let s return it and exit the subroutine:

    return $load;
 ;

Now let s take care of the special cases. This one deals with Driver parameter found in INI [server] class. This sub will create a string for an enum element. The possible values of the enum are extracted from the comment found in the LCDd.conf file:

$dispatch "LCDd::server" Driver  = sub  
    my ( $class, $elt, $info, $ini_v ) = @_;
    my $load = qq!class:"$class"
                    element:$elt
                      type=leaf
                      value_type=enum
    !;
    my @drivers = split /\W+/, $info;
    while ( @drivers
            and ( shift @drivers ) !~ /supported/ )  
      # nothing, the work is done by shift operator
      # in loop test to skip all words in $info until
      # 'The following drivers are supported:'.
      # ok, that's a hack
     
    $load .= 'choice='
          . join( ',', @drivers ) . ' ';
    return $load;
 ;

This sub invokes the default sub and add a small feature to the model to ensure that DriverPath ends with / . This is ensured by the match="/$" snippet:

$dispatch "LCDd::server" DriverPath  = sub  
    my ( $class, $elt, $info, $ini_v ) = @_;
    return   $dispatch _default_ ->(@_)
           . q! match="/$" !;
 ;

Likewise, this sub invokes the default sub with the optional 4th parameter that forces the value_type of the element. In this case, we enforce an integer element:

 $dispatch "LCDd::server" WaitTime 
 = $dispatch "LCDd::server" ReportLevel 
 = $dispatch "LCDd::server" Port 
 = sub  
    my ( $class, $elt, $info, $ini_v ) = @_;
    return $dispatch _default_ ->( @_, 'integer' );
   ;

This sub returns a string for a list element with default values:

  $dispatch "LCDd::server" GoodBye 
= $dispatch "LCDd::server" Hello 
= sub  
    my ( $class, $elt, $info, $ini_v ) = @_;
    my $ret = qq( class:"$class"
                    element:$elt
                      type=list
                ) ;
    $ret .= 'cargo type=leaf
             value_type=uniline - ' ;
    $ret .= 'default_with_init:0="\" '.$elt.'\"" ' ;
    $ret .= 'default_with_init:1="\" LCDproc!\""';
    return $ret ;
 ;

Now that the dispatch table is ready, let s really mine LCDd.conf information. First handle all INI classes in the foreach loop:

foreach my $ini_class (@ini_classes)  

Let s retrieve parameters for this INI class from the LCDd.conf template:

  my $ini_obj = $dummy->grab($ini_class);

This will be the name of the lcdproc driver configuration class (e.g. LCDd::imonlcd):

  my $config_class   = "LCDd::$ini_class";

The configuration class will also be created in a serialised format. Let s create config class in case there s no parameter in INI file:

  $meta_root->load(qq!class:"LCDd::$ini_class"!);

Now, loop over all INI parameters and create LCDd::$ini_class elements:

  my @all_ini_params = $ini_obj->get_element_name ;
  foreach my $ini_param ( @all_ini_params )  

Retrieve the value provided in template LCDd.conf:

    my $ini_v = $ini_obj->grab_value($ini_param);

Retrieve the comment attached to the value provided in template LCDd.conf:

    my $ini_comment
      = $ini_obj->grab($ini_param)->annotation;
    # escape embedded quotes
    $ini_comment =~ s/"/\\"/g;

Retrieve the correct sub from dispatch table:

    my $sub = $dispatch $config_class $ini_param 
              $dispatch _default_ ;

Run the sub to get the model string:

    my $model_spec
      = $sub->( $config_class, $ini_param,
                   $ini_comment, $ini_v );

Attach the comment from LCDd.conf as class description:

    $model_spec .= qq! description="$ini_comment"!
      if length($ini_comment);

Then, last but not least, load class specification in the model, and end the loop aroung INI parameters:

    $meta_root->load($model_spec);
   

Now create a an $ini_class element in LCDd class (to link LCDd class and LCDd::$ini_class):

  my $driver_class_spec = qq!
    class:LCDd
      element:$ini_class
  ! ;

Since server and menu are not drivers, they have a special treatment:

  if (   $ini_class eq 'server'
      or $ini_class eq 'menu' )  
    $driver_class_spec .= qq!
      type=node
      config_class_name="LCDd::$ini_class"
    ! ;
   

Here s the treatment for LCDd drivers. Setup the model with a warped node so that a driver configuration class is shown only if the driver was selected in the [server] class:

  else  
    $driver_class_spec .= qq!
      type=warped_node
      config_class_name="LCDd::$ini_class"
      level=hidden
      follow:selected="- server Driver"
      rules:"\$selected eq '$ini_class'"
        level=normal
    !;
   

Now we can load the LCDd driver model in meta root:

  $meta_root->load($driver_class_spec);
 

Step 5: save LCDd model Still with me ? Good. Here s the last part, now that the model was created in memory, let s save it in files. Config::Model::Itself constructor returns an object to read or write the data:

my $rw_obj = Config::Model::Itself
     ->new( model_object => $meta_root );
say "Writing all models in file (please wait)";
$rw_obj->write_all(
    model_dir => 'lib/Config/Model/models/'
);

That s it. Once this script is run, the directory lib/Config/Model/models/ contains a bunch of files:

• The main LCDd model
• All models for the drivers

You can see the resulting editor in my first blog on lcdproc. Conclusion and future steps (with you?) As mentioned at the end of this blog, the result of this model generation process is far from perfect, because the information retrieved from LCDd.conf template are not complete for this task (which is normal since these comments were targeted for users). Now, I d really like to discuss with project developers to improve configuration template to generate better models. For project owner, such an improvement would give them:

• command lines to validate configuration
• graphical configuration editor with integrated help
• Configuration doc generated from the information stored in the template file

Please leave a comment on this blog (or send a message to config-model-users at lists.sourceforge.net) if you are interested in configuration tool generation for a project you contribute to. All the best

Anne Rettenberg wrote an article for Psychology Today that is critical of the idea of online dating [1], she cites one example of a man who visited a prostitute due to being depressed at his lack of success in online dating to support her claim. The first big problem with her article is that she doesn t mention the different experiences that male and female customers presumably have on online dating sites. I don t know what it s like for women on the dating sites so I can t comment on that. But I m sure that someone who works as a counselor could provide some useful insight into this matter. Also she didn t even give a mention to the issue of gay/lesbian dating sites. The next issue is that she didn t offer any good advice for who should use online dating sites and what their aims should be. Rejection In dating in real life (IRL) it s expected that the man make the first move, and therefore women end up rejecting lots of guys for various reasons. Anne seems to think that rejection online is somehow worse than rejection IRL, it probably is for some people but that certainly isn t the case for everyone. The way that lots of dating sites seem to work is that women place adverts, men respond to them, and then the women reply to a small subset of the email that they receive. Rejection in this case isn t a matter of telling someone that you aren t interested, but of merely not replying to their mail. From my discussions with a few men who ve used online dating sites the strategy seems to be to send out initial messages to a few dozen women every day and then maybe get a few responses a day. For the messages that get no response you will never know whether the other person found someone else first, wasn t interested in you for some reason, or just didn t bother checking their email. The only comparison to IRL rejection is that which happens after phone numbers have been exchanged, which isn t going to be that common (and has the same issues regardless of how the people met). Guys, relax about the women who don t respond to your first message. They probably get 100+ responses to their advert and don t have time to even read half of them. If you get rejected later in the process then you can look through your email archive at a later date to try and discover what went wrong. The Aim of Online Dating The fact that Anne s client visited a prostitute suggests that maybe he wasn t really after a relationship. In which case using one of the many online services for finding sex partners might have been a better option. Generally it seems that a good strategy is to try and have fun. I don t know any men who have married someone from a dating site, but the general opinion seems to be that they are still worth using. If you meet someone in a bar then you might end up having a drunken conversation that is drowned out by loud music. If you meet someone over the Internet then you can have a quiet conversation over the phone which seems to be a better way to get to know someone (and generally more pleasant for anyone who s not an alcoholic). I think that men who have no immediate aim other than finding someone nice to talk to will do better than those who aim to score quickly. Of the men I know who married women they met over the Internet (but not through online dating sites) I wonder how many of them would have ended up married if they hadn t used the online dating sites first. It seems that men who regularly communicate with women outside formal situations (work etc) will have a better chance of impressing someone that they like than those who lack such experience. Introverts There are a lot of people who really can t function in a bar. With the way our society works it seems that anyone who can t handle the bar scene really should consider online dating. How Counselors can Help It seems to me that someone who is seeing a counselor and who is considering a new way of finding a SO should ask their counselor for advice first. It also seems to me that a good option might be to ask their counselor for advice in online dating. Instead of being unsuccessful and depressed a man who was seeing a female counselor could do well to ask her advice for how to impress women on the Internet. This is probably a business opportunity for female counselors who can advise men on such things, among other things it seems that seeing an online relationship coach would be perceived in a more positive manner than seeing a counselor or psychologist for the more traditional reasons.

• [1] http://www.psychologytoday.com/blog/beyond-don-juan/201107/can-online-dating-make-you-depressed

• Privacy: as far as I can tell, Plus design had started with privacy in mind, whereas for Facebook it is still an unloved child, a spare tire. Facebook keeps on getting bad reviews here; people don't get their UI and mis-share things. I read somewhere that FB is actually losing users in the US: kids who leave Facebook because their parents are getting on, and they don't want them to see what they shared with their friends.
• UI: the Circles UI is very easy to use. The same functionality on Facebook is awful to manage. And the notifications UI in Plus also are a lot better than the tiny indicator in Facebook.
• New stuff: Hangout and group chats are pretty interesting (well, actually I don't like Video at all ...). This puts pressure on Facebook to move, and it looks like they might present some Skype integration soon. But they need to make this really good to keep up with hangout. Just adding "Call me" buttons for all users that have specified a Skype account in their profile won't do the trick.
• Future integration: Wave already had many of these things, but the wrong way. You would put a map into a chat; the Plus way will probably be to share a map session with a circle and add chat and collaborative editing there. Google "Office" is another place where it is trivial but very useful to add Plus.

---

So, I haven t read the planets in a couple of days but I m going to guess that, like twitter, a lot of people are going to be raving about Google+. For those of you who haven t heard about Google+, it s a new Social Networking site from Google and it brings a new take to the world of social networking. In real life, we communicate differently to the different people in our social contexts. For example, most people will converse differently to their work colleagues than they will to their family. They ll converse differently with their drinking buddies than they will with their Family. There are things that you want only certain people to know, and there are aspects of your personality that you only want to show to certain people. I think giving an example here might be better. This is an example of a friend of mine, who we ll call Fred. Fred is a teacher. Fred also likes to go out of a weekend and party till dawn, while consuming lots of tequila. I, as Fred s friend, am on a night out of drunken debauchery, and I ve my camera with me, and decide to post lots of pictures online of the night s revelry. The next day, one of Fred s colleagues logs into a Social Networking site, and sees said pictures in Fred s profile. Now things aren t looking so bright for Fred. He no longer gives across that professional image that he should do at work. Some may say that this is Fred s fault for going out and doing these things, but everyone s entitled to a personal life, right? At the moment, on Fred s social networking site, it s an all or nothing option. If he allows you access to his profile, he allows you access to see anything and everything that might show up on there. (It s a bit more complicated than this, but let s keep it simple for now). Every time Fred has someone add him as a contact he has to make that decision Do I want this person to be able to see everything I may or may not get up to . Put simply, how many of you have refused someone to add you on the Social Networking site du jour because what you show on their may be harmful to you, or give over an impression that isn t what you want to give across. Enter Circles. Circles brings to the social networking scene the way we interact with people in real life. We show a different aspect of our personality to different people we interact with. I m a bit of an organisational freak, and I know that there are different groups of people that either for the fact of portraying a persona, or not annoying everyone else, I might want to push in different ways. For example, I might want to send out a BBQ at my house, bring lots of alcohol message to all my friends, but don t really want my co-workers coming along and making it so I have to be on my best behaviour. Below is a screenshot of my (current) circles. As you can see I have a fair few, and only a relative few people. Now, let s say that I wanted to send out that BBQ message As you can see I ve the option to send this to the 5 people that are in my Friends group (it s currently mostly geeks on there there are only a few of my real life friends on there so far!) Now with only a few clicks, I ve managed to send out a message to just those few people that I want to Over the next few days, I ll write a bit more about Google+, but I thought I d start with an introduction into the Circles feature. I ll be giving a bit of feedback about what I like, what I dislike, and also making a few suggestions for what I d like to see happen in the future (and probably use the word siloing a fair bit!) For now, toodles.

I've just released git-annex version 3, which stops cluttering the filesystem with .git-annex directories. Instead it stores its data in a git-annex branch, which it manages entirely transparently to the user. It is essentially now using git as a distributed NOSQL database. Let's call it a databranch. This is not an unheard of thing to do with git. The git notes built into recent git does something similar, using a dynamically balanced tree in a hidden branch to store notes. My own pristine-tar injects data into a git branch. (Thanks to Alexander Wirt for showing me how to do that when I was a git newbie.) Some distributed bug trackers store their data in git in various ways. What I think takes git-annex beyond these is that it not only injects data into git, but it does it in a way that's efficient for large quantities of changing data, and it automates merging remote changes into its databranch. This is novel enough to write up how I did it, especially the latter which tends to be a weak spot in things that use git this way. Indeed, it's important to approach your design for using git as a database from the perspective of automated merging. Get the merging right and the rest will follow. I've chosen to use the simplest possible merge, the union merge: When merging parent trees A and B, the result will have all files that are in either A or B, and files present in both will have their lines merged (and possibly reordered or uniqed). The main thing git-annex stores in its databranch is a bunch of presence logs. Each log file corresponds to one item, and has lines with this form:

timestamp [0 1] id

This records whether the item was present at the specified id at a given time. It can be easily union merged, since only the newest timestamp for an id is relevant. Older lines can be compacted away whenever the log is updated. Generalizing this technique for other kinds of data is probably an interesting problem. :) While git can union merge changes into the currently checked out branch, when using git as a database, you want to merge into your internal-use databranch instead, and maintaining a checkout of that branch is inefficient. So git-annex includes a general purpose git-union-merge command that can union merge changes into a git branch, efficiently, without needing the branch to be checked out. Another problem is how to trigger the merge when git pulls changes from remotes. There is no suitible git hook (post-merge won't do because the checked out branch may not change at all). git-annex works around this problem by automatically merging */git-annex into git-annex each time it is run. I hope that git might eventually get such capabilities built into it to better support this type of thing. So that's the data. Now, how to efficiently inject it into your databranch? And how to efficiently retrieve it? The second question is easier to answer, although it took me a while to find the right way ... Which is two orders of magnitude faster than the wrong way, and fairly close in speed to reading data files directly from the filesystem. The right choice is to use git-cat-file --batch; starting it up the first time data is requested, and leaving it running for further queries. This would be straightforward, except git-cat-file --batch is a little difficult when a file is requested that does not exist. To detect that, you'll have to examine its stderr for error messages too. Perhaps git-cat-file --batch could be improved to print something machine parseable to stdout when it cannot find a file. Takes some careful parsing, but straightforward. Efficiently injecting changes into the databranch was another place where my first attempt was an order of magnitude slower than my final code. The key trick is to maintain a separate index file for the branch. (Set GIT_INDEX_FILE to make git use it.) Then changes can be fed into git by using git hash-object, and those hashes recorded into the branch's index file with git update-index --index-info. Finally, just commit the separate index file and update the branch's ref. That works ok, but the sad truth is that git's index files don't scale well as the number of files in the tree grows. Once you have a hundred thousand or so files, updating an index file becomes slow, since for every update, git has to rewrite the entire file. I hope that git will be improved to scale better, perhaps by some git wizard who understands index files (does anyone except Junio and Linus?) arranging for them to be modified in-place. In the meantime, I use a workaround: Each change that will be committed to the databranch is first recorded into a journal file, and when git-annex shuts down, it runs git hash-object just once, passing it all the journal files, and feeds the resulting hashes into a single call to git update-index. Of course, my database code has to make sure to check the journal when retrieving data. And of course, it has to deal with possibly being interrupted in the middle of updating the journal, or before it can commit it, and so forth. If gory details interest you, the complete code for using a git branch as a database, with journaling, is here. After all that, git-annex turned out to be nearly as fast as before when it was simply reading files from the filesystem, and actually faster in some cases. And without the clutter of the .git-annex/ directory, git use is overall faster, commits are uncluttered, and there's no difficulty with branching. Using a git branch as a database is not always the right choice, and git's plumbing could be improved to better support it, but it is an interesting technique.

I think the old-fashioned coredump is a little under-appreciated these days. I'm not sure when it changed, but I even had to add myself to /etc/security/limits.conf to raise my ulimit to even create one. Anyway, debugging __thread variables from coredumps is a bit of a pain. For the uninitiated, the __thread identifier specifies a thread-local variable, i.e. every thread gets its own copy of the variable automatically. The implementation of this is highly architecture specific. The reason is that TLS entries need to be accessed via a register kept as part of the thread state, and thus every architecture chooses their own register and builds their own ABI. On x86-32, which is very register-limited, you certainly don't want to dedicate a register to a pointer to TLS variables and take it out of operation. Luckily there is the hang-over from the 70's (60's? 50's?) segmentation. Without going into real detail, segment registers can be used to offset into a region of memory based on a look-up of a region descriptor stored in a table. Above, you see a simplified example of the %gs register loaded with the index value 2, and thus when you access %gs:20 what you are saying is "find entry 2 in the global descriptor table (GDT), follow it and offset 20 into that region. The kernel gives each thread its own GDT (i.e. the GDT register is part of the thread-state). Thus __thread variables are stored based on segment offsets and voila thread-local storage. Now, there's a few tricks here. For various reasons, a process can not setup entries in the GDT; this is a privileged operation that must be done by the kernel. There is actually a special system call for threads to setup their TLS areas in the GDT set_thread_area. When a new thread starts, the thread-library and dynamic linker conspire to allocate and load any static TLS data (i.e. if you have a global __thread variable initialised to some value, then every thread must see that value when it starts) and then calls this to make sure the variables are ready to go. After that, the gs register is filled with the index of that GDT entry, and all TLS access goes via it. That, in a nut-shell, is TLS for x86-32. Now, to the problem. Take, for example, the following short program: We start a few threads and then abort to make it dump core. But, if you try and examine foo: It seems that gdb doesn't know how to find the value of foo because its not a variable in the usual sense ("this target", in this case, means a coredump). It relies on accessing via the gs register, which relies on the current processes' GDT state, which has since been destroyed. If you care to consult the canonical source of TLS info, you can find out exactly why this is so hard to figure out generically. However, with some work, we can start to figure out the value by hand. A coredump is really a ELF file full of just two things: a bunch of LOAD segments that are just dumps of the process memory regions, and a NOTE section that includes a bunch of notes that the kernel dumps out for us such as the current register state, the process id, the signal that killed us, etc. Here's an example of a core file under readelf You can examine the notes; with readelf we see: This shows 5 notes of type NT_PRSTATUS; it should be little surprise that each of these notes describes the process status of one running thread. When gdb pops up [New Thread 4973] that's because it just hit another note that describes the new thread. To actually make sense of the note, however, we need some other tools that look deeper. The elfutils based tools give us a better description of the various notes in the coredump; more akin to what GDB is interpreting them as. Below I've extracted one thread's info: What's particularly interesting here is that the note type that was previously unknown (Unknown note type: (0x00000200)) has been resolved for us it is in fact of type NT_386_TLS and is a dump of the GDT entries for the thread. So, if we examine how our function is accessing our TLS variable by disassembling the thread function: Examining the disassembly we can see that foo is accessed via an offset of -4 from %gs (this is OK, as our limit value is maxed out. See the TLS ABI doc for more info). Now, we can examine gs and see which selector it is telling us to use: Above we shift out the last 3 bits, as these refer to the privilege level (bits 0 and 1) and if this is a GDT or LDT reference (bit 2). Thus, looking at the GDT descriptor for index 6: we can finally do some maths from the base-address to figure out the value: Thus we have found our TLS value; in this case for thread 3 the value is indeed 3. I caution this is the simplest possible case; other "models" (see the TLS doc, again) may not be so simple to work out by hand, but this would certainly be how you would start.

TED has published a list of resources for suicide prevention and to help survivors and their families [1]. Psychology Today has an interesting article by Paula J. Caplan, Ph.D about the recent US Supreme Court decision denying female employees of Walmart the ability to file a class action lawsuit about their poor pay and working conditions [2]. She describes the problem as a focus on rights of the ruling class vs fairness to the workers, it could also be described as prioritising perceived rights of the rich over the rights of workers to fair treatment. It seems to me that her article has relevance to some of the discussions related to the treatment of women in the Free Software community. New Scientist has an interesting article by Ferris Jabr about the use of MRI to discover brain-wave patterns correlated with Autism in sleeping toddlers [3]. This doesn t seem likely to be useful for scanning the entire population as it currently has a false-positive rate of 7/43 (which would make false-positives outnumber true-positives by about 15:1). But it does seem likely to do some good in identifying young children who might be on the Autism Spectrum. Shea Hembry gave an amusing TED talk about how he created art works for 100 fictional artists for his own exhibition [4]. He created a biography for each artist and every one had a unique style of art. Steve Keil gave a passionate TED talk about the benefits of play for children and for adults [5]. He focussed on the benefits for Bulgaria (where his talk was given) but it all applies to all humans. Frederic Bastiat s 1850 essays on economics are interesting, informatice, and well written [6]. Some of the themes such as the supposed economic benefits of maintaining a large army are the subject of political debate today. Paula J. Caplan, Ph.D wrote an insightful article for Psychology Today about the recent US supreme court decision in regard to the Wal-mart case [7]. Her article seems to have some obvious parallels to the situation in the FOSS community. The idea of rights vs fairness, beliefs that are unconscious or unexamined, and the comparison of attitudes towards racial vs sexual discrimination (in terms of not treating sexual discrimination and harassment seriously) seem to all apply clearly. Al Jazeera reports that the Fukushima disaster is worse than is reported in the mainstream western press [8]. Generally I wouldn t be inclined to trust al Jazeera if other news sources were reliable. But unfortunately reliable news related to contentious issues such as nuclear power is quite rare. It will probably be quite a long time before we can be confident that we know much about Fukushima, everyone who knows seems to be lying.

• [1] http://blog.ted.com/2011/06/11/resources-for-suicide-prevention-post-attempt-survivors-and-their-families/
• [2] http://www.psychologytoday.com/blog/science-isnt-golden/201106/americans-fail-take-sexism-seriously-the-supreme-courts-wal-mart-rul
• [3] http://www.newscientist.com/blogs/shortsharpscience/2011/06/brain-out-of-sync-in-toddlers.html
• [4] http://www.ted.com/talks/shea_hembrey_how_i_became_100_artists.html
• [5] http://www.ted.com/talks/steve_keil_a_manifesto_for_play_for_bulgaria_and_beyond.html
• [6] http://bastiat.org/en/twisatwins.html
• [7] http://www.psychologytoday.com/blog/science-isnt-golden/201106/americans-fail-take-sexism-seriously-the-supreme-courts-wal-mart-rul
• [8] http://english.aljazeera.net/indepth/features/2011/06/201161664828302638.html

I often get requests of persons who would like to contribute to Debian but who don t know where to start. Let s try to answer this question properly so that I can give out this URL the next time that I am asked. The Debian website has a page explaining how to help Debian. While it provides no less than 10 suggestions in a daunting text-only list, it s difficult to know what to do next once you picked up something that you could do. I will try to fix this by providing concrete information for each cases in upcoming articles but in the mean time I propose you another approach to start with. Before answering your question ( what can I do for Debian ), we need to know some information about you. What motivates you? You re a volunteer, you re not doing stuff for Debian because someone told you so. You must have some intrinsic motivation and the ultimate motivation is usually that you re enjoying what you re doing. So what are you enjoying and/or what are your motivations ?

• Is there something that you would like to learn? A new programming language? Packaging? Coding? System administration? A specific software?
• Do you want to interact with smart people?
• Do you like to help users?
• Do you like to fix software just so that it works for you?
• Do you like to build something remarkable and useful for millions of people?

On the opposite, make sure to know what you hate and what you want to stay far away from. Maybe you dislike a programming language so much that you don t want to be involved in a project where you would have to use it, etc. Write down the answers to the questions, you might need them later when you ll ask other Debian contributors how you can help. What are your skills? If you re not interested in learning new skills, then you must obviously select a task where your current skills are sufficient. Again make a list of your skills and in particular of skills that you d like to practice! Here s a non-exhaustive list of skills to consider:

• What languages are you fluent with? Are you confident to write documentation or translate documentation in those languages?
• Are you a programmer? If yes, which languages do you know?
• Can you diagnose problems? Can you debug problems with strace and/or gdb?
• Can you triage bugs?
• Do you know some Debian packaging?
• Are you an artist and/or a web designer?
• Do you know how to work with VCS (subversion, git, bzr, )?

How much time can you spend on Debian? This is the last important information that you need to communicate whenever you re asking someone else what you could do for Debian. There s no point giving you a big task if you can only spend 30 minutes every week. On the opposite if you can work on Debian full time during a week (because you re between 2 contracts or because you re in vacation), it s equally important to know. In general contributing to Debian requires time, you should be ready to spend at least several hours per week and possibly more at the start while you re learning everything. Find something to do At this point you have a generic idea of what you d like to do but you re still missing a concrete objective. Let s try to find one, we ll explore several ways to do this. Scratch your itch The best objectives are those that satisfy your own needs. Here are some examples:

• Did you notice a missing feature? Try to implement it.
• Have you been annoyed by a bug? Try to fix it.
• Did you lose too much time on something because there was no documentation? Write the missing documentation and submit it where appropriate.
• File bug reports for the things that you can t fix yourself. Even wishlist bug reports for new features.
• Do you use software that are not packaged for Debian? Create the package(s) and maintain it/them.
• Do you need a newer version of a package compared to what s in Debian unstable? Contact the maintainer and propose your help to update the package.
• Do you need a newer version of a package compared to what s in Debian stable? Contact the maintainer and propose your help to create a backport.

If you re a good Debian citizen, you have already filed bugs for issues that bugged you. Then you can browse http://bugs.debian.org/from:hertzog@debian.org to find out some ideas of stuff to do (obviously replace hertzog@debian.org by your own email address). Join a team If you don t have a specific itch to scratch, you might want to focus your work on a specific team. Head over to wiki.debian.org/Teams and browse the list of teams. You ll surely find one that works in an area that you like. If you select a packaging team, pick one that works on packages that you re actually using. Some of the teams have instructions for newcomers, follow them when that is the case. Otherwise join the mailing list and the IRC channel, and get a feel of how the team works. See if it suits you, you can follow several teams at the same time and pick the one that you prefer after a few days/weeks. Once you have lurked a bit, if you still don t know how you could help, then just ask on the mailing list. Include all the answers you have collected to the 3 questions on your motivations, your skills and your time available. Focus on a specific package You can concentrate your work on a specific package even if you re in a team, it s often a good idea. But I list it separately because not all packages are team maintained and you might want to help maintain a package where there s a single maintainer currently. I leave it up to you to find a way to select a package that interests you Then head over to the package tracking system: http://packages.qa.debian.org/dpkg (replace dpkg by the name of the package that interests you) Fill in the form in the bottom-left corner with your email and select opts in the drop-down list, then click go . You get a new form where you can select the information that you ll receive, I recommend you to keep everything except upload-binary and to validate the form. From now on, you get the same mails than the maintainer (and a bit more actually), and it s a good idea to inform the maintainer that you subscribed and that you re going to help a bit. Maybe he s willing to grant you commit rights immediately, maybe he will ask you to send patches for a start. The important thing is to create a good relationship. In any case (even if you did not get any answer from the maintainer), you should be free to help triage existing bugs and to help deal with the flow of incoming bugs (including forwarding bugs when appropriate). Help a Debian developer Paul Tagliamonte once blogged Hey, DDs, need help?. He offered his help to overworked Debian developers . His post was missing all the information required (cf motivations/skills/time) but the approach is a good one. The best way to start contributing is to work with existing Debian developers. Even if you just want to be sponsored for your own pet package, you should consider that mentoring is a burden for many Debian developers and that you re more likely to get a sponsor if you have an existing relationship with a Debian developer that you helped. Pick a developer that works in an area that is of interest, and offer him your help. To simplify things even further, I have created a wiki page where you can find out how you can help me and thus build a relationship with me. That s it for now, I hope this article will help you to start contributing. Don t forget to subscribe to my newsletter to not miss future articles for new contributors. Note that you can refer to this article with the following URL: http://raphaelhertzog.com/go/contributing/ (easier to type and remember).

13 comments Liked this article? Click here. My blog is Flattr-enabled.

Today I was made aware of bug #607839, and it makes me sad. I think its a great example about how trademarks and free software do not coexist well together. This bug started out as a question of the copyright license of a file, but quickly turned into a discussion about the GNOME [0] trademark, and the GNOME Foundation Trademark Usage Guidelines for Third Parties I believe that the GNOME project would be standing on solid legal ground if they wanted to actually try to aggressively protect their trademark. It is easy for me to find hundreds of places we use their mark in ways that are expressly forbidden by their trademark license. Here are some examples:

grep -i "gnome's" /var/lib/dpkg/available  # many package descriptions invalidly using a possessive
grep -i "gnome[^ ]" /var/lib/dpkg/availabe # returns nothing, trademark symbol is required

Probably to be in strict compliance, we would need to allow uppercase letters in package names. (I doubt this would happen, especially not for a reason like this). I am sure that a lawyer would tell us that the safest legal place for Debian would be to abandon all use of their marks. This would unfortunately be a herculean effort on Debian's part; we'd have to rename tons of packages, create and maintain tons of patches, create new logos, etc. And, I believe it would have the unfortunate side-effect of having a negative impact on both projects from a publicity standpoint. As with the firefox/iceweasel situation, the company trying to protect its trademark looks like a bully, and Debian looks like out-of-touch zealots. From the tone of the conversation in the bug report, I do not believe that this is something that the GNOME Foundation would force on us, and I certainly don't believe it would be an outcome they would hope for. They could give us written permission to use their marks in the ways we are already using them, however, lots of people would point at DFSG#8, and this would fail the 'tentacles of evil' test. (We are, of course, already under the grasp of these tentacles, as their current trademark license is subject to change) Better for us would be if they modified their trademark license agreement in order to allow our current usages of their mark. I feel that this quite unlikely to happen. It would certainly weaken their hold on the mark. Trying to apply the DFSG in these situations also doesn't work as well as it should. The DFSG is written with copyrights in mind and not patents and trademarks. This feels to me similar to situations we have with patent encumbered software which is under a perfectly free license. We can distribute the software without fear of a claim that we are violating copyrights, but the software is still not perfectly free since as I user, my rights to use and distribute the software might be severely limited from a legal standpoint. Also by Debian choosing to distribute software which has these kinds of potential problems, we might be putting a legal liability on members of the project and people that run mirrors for us. Of course, to add to the conflict I feel, Debian has its own trademarks, which it is poor at enforcing. On one hand, its bad for all the reasons that this situation is bad. On the other hand I sometimes wish that Debian did a better job of protecting its own trademark. It's clear from how many people show up in #debian expecting support for linuxmint or aptosid claiming "linux mint debian edition IS Debian, it says so on linuxmint.org" that we have allowed our mark to be diluted. NOTE: I became aware of this bug because of Zack CCing the FTP team. I serve on this team as an assistant, however, My comments here come from me personally without any discussion with other members of FTP team and should not be taken as any kind of offical word from the FTP team. [0] I suddently feel compelled to mention that The GNOME name is a registered trademark of GNOME Foundation in the United States or other countries.

I have bought a Samsung Galaxy S Android phone some time ago. One of the first thing that you try to do with such a phone is to switch to some alternate firmware (the coined term is ROM ), like CyanogenMod. Flashing a firmware is a whole new universe. You are told about a lot of proprietary tools, you need to download a lot of things from MegaUpload and friends and nobody really explains what s going on under the hood. I was a bit lost at first. I will not provide a step-by-step guide since I cannot afford to stay up-to-date. Such a guide is usually posted on some forum. It assumes you are running Windows and provides appropriate download links. You must follow it (and check that it is recent enough). I will just provide you some background to understand those guides.

Android firmwares There are several Android firmwares. Most of them are specific to a given phone (or its close variations). They can be sorted in three categories:

1. Official firmwares from the phone manufacturer include a kernel with appropriate drivers and, usually, a customized interface (HTC Sense for HTC, TouchWiz for Samsung). Some manufacturers will lag a lot behind Android schedule and you may not get all the new shiny features.
2. Modified firmwares are built from the official firmwares. They usually add a lot of hacks on top of the original firmware, remove some cruft but are still closely related to the official firmware.
3. Firmwares built from Android Open Source Project (the open source version of Android) do not share nothing with the official firmware except some bits related to the kernel (drivers, hacks).

Darky ROM Darky ROM is a modified firmware. It takes the original firmware from Samsung, adds a custom kernel (based on the source of the official kernel), adds a lot of hacks, some additional applications and removes some unwanted cruft. This is a very nice firmware which is quite snappy. This is a drop-in replacement for the default firmware: you will only get improvements. The latest version is 10.1 and there is a neat guide on how to install it.

CyanogenMod CyanogenMod is a firmware built from AOSP. It supports several devices and provide an unified firmware on all of them. Support for the Samsung Galaxy S is recent and is not present in the stable build. However, some RC build is now available. Be sure to read the FAQ (most notably the part saying that you need to install from Froyo 2.2 because the bootloader provided by Samsung with Gingerbread 2.3 is not compatible). You can also grab some information from the wiki. The major drawback of using this firmware instead of going for a modified firmware like Darky ROM is that the battery life may suck. I now need to charge my phone every day instead of once every two days. On the plus side, you get a very clean firmware with a lot of features and almost no cruft. For example, the notification bar can be customized and OpenVPN is available out of the box. Moreover, almost everything is open source (if you don t install Google apps). You can see this firmware as an open source distribution for your phone. There is no need to download random files linked from a random forum to make it work. There is no package manager, though. Despite the bad battery life, I prefer this firmware.

Prerequisites Before trying to flash a new firmware, there are some tools that you need to have. Note that you don t need root access to flash a new firmware.

Download & recovery mode Before trying to install any firmware, you must check that your phone can access download and recovery modes. The download mode allows to flash any new firmware, even if the current firmware does not boot at all. As long as this mode is available, there is very little chance that you brick your phone. To access those modes, you need to use the three-button combo . Here are the steps (your phone should not be plugged):

1. Power off the phone (wait for the phone to vibrate).
2. Hold the Volume down button for download mode. Hold the Volume up button for recovery mode.
3. Hold the Home button.
4. Hold the Power button. The phone will start.
5. Once you have the Samsung logo displayed on the screen, release the Power button.
6. Only release the two other buttons when you enter the recovery mode or download mode. The recovery mode displays a menu. The download mode displays a big yellow Android logo.

If after ten tries, you are unable to access those modes, maybe they have been disabled. Keep reading, there is a solution.

Heimdall Heimdall is a tool to flash a new firmware to your phone. This is not a mandatory tool but you will have to use it if something goes wrong. Most guides will tell you to use Odin. It is a leaked Samsung tool running on Windows only. You can use Heimdall instead.

$ git clone https://github.com/Benjamin-Dobell/Heimdall.git
Cloning into Heimdall...
remote: Counting objects: 423, done.
remote: Compressing objects: 100% (271/271), done.
remote: Total 423 (delta 220), reused 332 (delta 138)
Receiving objects: 100% (423/423), 5.91 MiB   962 KiB/s, done.
Resolving deltas: 100% (220/220), done.
$ cd Heimdall/heimdall
$ sudo apt-get install libusb-1.0-0-dev
$ ./configure && make
[...]
$ sudo cp 60-heimdall-galaxy-s.rules /etc/udev/rules.d
$ sudo /etc/init.d/udev reload

You need to use this tool while in download mode. You can test it, while in download mode, with:

$ ./heimdall print-pit

This will display the partition table of your phone. This is enlightening to know what it contains:

• IBL+PBL is the boot loader, be extra careful with it;
• PIT is the partition table;
• EFS contains some important data like IEMEI, be extra careful with it;
• SBL is the second boot loader;
• SBL2 is the backup of the second boot loader;
• PARAM contains some images used as various stage of the boot process;
• KERNEL is the kernel;
• RECOVERY is the backup of the kernel;
• FACTORYFS contains /system file system which is some kind of root file system;
• DBDATA contains /data file system which is the database to store application parameters;
• CACHE contains /cache file system which contains the Dalvik cache;
• MODEM contains the firmware for the modem.

As we will see, Heimdall is able to flash any of those partitions with flash subcommand.

adb adb stands for Android Debug Bridge. Among other things, this tool allow you to drive your phone through the USB connection. You can get a shell, reboot into download or recovery mode, push files to your Android device, etc. To get this tool, you need to download the full Android SDK.

$ tar zxvf ~/download/android-sdk_r11-linux_x86.tgz
$ cd android-sdk-linux_x86
$ tools/android update sdk
[select Android SDK Platform-tools]
$ platform-tools/adb shell
# uname -a
Linux localhost 2.6.35.7-I9000XWJVH-CL184813 #Dark_Core_2.4 PREEMPT Mon May 16 15:02:38 EEST 2011 armv7l GNU/Linux

The important commands are:

• shell to get a shell on the device;
• reboot recovery to reboot in recovery mode;
• reboot download to reboot in download mode;
• push to put a file on the phone;
• pull to retrieve a file;
• install to install an application from its APK;
• logcat to see the logs and what happens, very useful!

Restore download mode If your phone is missing download mode, you can recover it with the help of adb and heimdall. You need to grab some official firmware and extract boot.bin and Sbl.bin from it.

$ adb reboot download
$ heimdall flash --primary-boot ~/download/boot.bin --secondary-boot ~/download/Sbl.bin

Be extra careful here. You may brick your phone. Double check that you grab the correct files and that they are really for your phone (there exists some variants). It worked for me.

Installing a new firmware While you can switch from one firmware to another without losing your data, it is better to wipe your system and your data before switching firmware. This means that you will lose all your applications and the data inside them (list of contacts, SMS, preferences). However, nothing will ever modify your data in the internal SD card or in the external SD card (which are both inside the phone). They usually contains your music and your photos. There are two ways to install a firmware:

1. via Heimdall, or
2. using the recovery.

Heimdall When a guide tells you to use Odin, you can use Heimdall instead. Odin takes a TAR file. You should untar it and uses its content with Heimdall. For example:

$ ./heimdall flash --repartition --pit s1_odin_20100512.pit \
>                  --factoryfs factoryfs.rfs \
>                  --cache cache.rfs --dbdata dbdata.rfs \
>                  --param param.lfs --modem modem.bin \
>                  --kernel zImage --secondary-boot Sbl.bin 
Claiming interface again...   Success
Setting up interface...   Success
Beginning session...
Handshaking with Loke...   Success
Uploading PIT
PIT upload successful
Uploading SBL
100%
SBL upload successful
Uploading KERNEL
100%
KERNEL upload successful
Uploading MODEM
100%
MODEM upload successful
Uploading PARAM
100%
PARAM upload successful
Uploading FACTORYFS
100%
FACTORYFS upload successful
Uploading DBDATAFS
100%
DBDATAFS upload successful
Uploading CACHE
100%
CACHE upload successful
Ending session...
Rebooting device...

You may be interested to update your modem using Heimdall. This is usually independent of the firmware you want to use. Also, no need to provide a PIT file if you don t want to repartition. Heimdall can read the current one and flash only one of the partitions.

Recovery mode The base recovery shipped with the Samsung Galaxy S does not have a lot of options. Moreover, it may be locked and will accept to install only signed firmwares. Most firmwares come with ClockWorkMod, a more advanced recovery. Once you get it, you will be able to use any ZIP file to install a new firmware. My advice is to try to get ClockWorkMod as soon as possible. If you are root, you can install ROM Manager that will install it for you. If you are not root, most custom firmwares will provide ClockWorkMod. Therefore, you can just install a custom firmware with Heimdall. There exists special firmwares just for this, like EZbase. If you want to install CyanogenMod, be sure to download a Froyo firmware, not a Gingerbread one. Keep in mind that the recovery is located inside the kernel. It is not self-contained in the bootloader (unlike download mode). If you don t want to flash a whole firmware (and lose everything on your phone) just to get the recovery, you can just flash the kernel. The ClockWorkMod recovery comes with an utility to backup your whole phone. This way, you can easily restore your previous firmware if you don t like the current one. This is what is called a Nandroid backup.

Lagfix A lot of guides will tell you to enable or disable lagfix. Samsung shipped the Galaxy S with some crappy filesystem called RFS (which is some kind of enhanced FAT). It is slow and seems responsible for some lags that may happen on the interface. A lagfix is a way to replace this filesystem by something better, like EXT4 or YAFFS2. The process is pretty straighforward and consist to backup all the partition to the SD card, reformat it with another filesystem and restore the backup. The lagfix may be applied automatically with some firmware (like CyanogenMod) or you may request it in the recovery mode (most modified firmwares work this way). In this case, it will be applied automatically on the next boot. You should disable it before installing another firmware with the recovery mode (no need if you install it with Heimdall).

Applications You can find many free (as in free speech) applications for Android. Unfortunately, Android Market does not allow to use this criteria as a filter. F-Droid is an alternate market featuring only FOSS applications. Here is a short selection of free applications:

• ADW.Launcher is a home replacement app. It is very configurable and themable.
• K-9 Mail is a very nice email client.
• Xabber is a XMPP/Jabber client.
• ConnectBot is an SSH client (as well as a terminal).
• DroidWall is an application firewall (using Netfilter). It lets you select which application can access to Internet depending on your connection (wifi or 3G).
• Barcode Scanner decodes 1D/2D barcode including QR codes.
• OsmAnd is a replacement for Google Maps using OpenStreetMap. The free version on Android Market is out-dated. Buy the plus version or download it throught F-Droid.

Another must-have application is Titanium Backup which allows to backup any application with data. Unfortunately, this is not a free (as in free speech) application.

I have bought a Samsung Galaxy S Android phone some time ago. One of the first thing that you try to do with such a phone is to switch to some alternate firmware (the coined term is ROM ), like CyanogenMod. Flashing a firmware is a whole new universe. You are told about a lot of proprietary tools, you need to download a lot of things from MegaUpload and friends and nobody really explains what s going on under the hood. I was a bit lost at first. I will not provide a step-by-step guide since I cannot afford to stay up-to-date. Such a guide is usually posted on some forum. It assumes you are running Windows and provides appropriate download links. You must follow it (and check that it is recent enough). I will just provide you some background to understand those guides.

Android firmwares There are several Android firmwares. Most of them are specific to a given phone (or its close variations). They can be sorted in three categories:

1. Official firmwares from the phone manufacturer include a kernel with appropriate drivers and, usually, a customized interface (HTC Sense for HTC, TouchWiz for Samsung). Some manufacturers will lag a lot behind Android schedule and you may not get all the new shiny features.
2. Modified firmwares are built from the official firmwares. They usually add a lot of hacks on top of the original firmware, remove some cruft but are still closely related to the official firmware.
3. Firmwares built from Android Open Source Project (the open source version of Android) do not share nothing with the official firmware except some bits related to the kernel (drivers, hacks).

Darky ROM Darky ROM is a modified firmware. It takes the original firmware from Samsung, adds a custom kernel (based on the source of the official kernel), adds a lot of hacks, some additional applications and removes some unwanted cruft. This is a very nice firmware which is quite snappy. This is a drop-in replacement for the default firmware: you will only get improvements. The latest version is 10.1 and there is a neat guide on how to install it.

CyanogenMod CyanogenMod is a firmware built from AOSP. It supports several devices and provide an unified firmware on all of them. Support for the Samsung Galaxy S is recent and is not present in the stable build. However, some RC build is now available. Be sure to read the FAQ (most notably the part saying that you need to install from Froyo 2.2 because the bootloader provided by Samsung with Gingerbread 2.3 is not compatible). You can also grab some information from the wiki. The major drawback of using this firmware instead of going for a modified firmware like Darky ROM is that the battery life may suck. I now need to charge my phone every day instead of once every two days. On the plus side, you get a very clean firmware with a lot of features and almost no cruft. For example, the notification bar can be customized and OpenVPN is available out of the box. Moreover, almost everything is open source (if you don t install Google apps). You can see this firmware as an open source distribution for your phone. There is no need to download random files linked from a random forum to make it work. There is no package manager, though. Despite the bad battery life, I prefer this firmware.

Prerequisites Before trying to flash a new firmware, there are some tools that you need to have. Note that you don t need root access to flash a new firmware.

Download & recovery mode Before trying to install any firmware, you must check that your phone can access download and recovery modes. The download mode allows to flash any new firmware, even if the current firmware does not boot at all. As long as this mode is available, there is very little chance that you brick your phone. To access those modes, you need to use the three-button combo . Here are the steps (your phone should not be plugged):

1. Power off the phone (wait for the phone to vibrate).
2. Hold the Volume down button for download mode. Hold the Volume up button for recovery mode.
3. Hold the Home button.
4. Hold the Power button. The phone will start.
5. Once you have the Samsung logo displayed on the screen, release the Power button.
6. Only release the two other buttons when you enter the recovery mode or download mode. The recovery mode displays a menu. The download mode displays a big yellow Android logo.

If after ten tries, you are unable to access those modes, maybe they have been disabled. Keep reading, there is a solution.

Heimdall Heimdall is a tool to flash a new firmware to your phone. This is not a mandatory tool but you will have to use it if something goes wrong. Most guides will tell you to use Odin. It is a leaked Samsung tool running on Windows only. You can use Heimdall instead.

$ git clone https://github.com/Benjamin-Dobell/Heimdall.git
Cloning into Heimdall...
remote: Counting objects: 423, done.
remote: Compressing objects: 100% (271/271), done.
remote: Total 423 (delta 220), reused 332 (delta 138)
Receiving objects: 100% (423/423), 5.91 MiB   962 KiB/s, done.
Resolving deltas: 100% (220/220), done.
$ cd Heimdall/heimdall
$ sudo apt-get install libusb-1.0-0-dev
$ ./configure && make
[...]
$ sudo cp 60-heimdall-galaxy-s.rules /etc/udev/rules.d
$ sudo /etc/init.d/udev reload

You need to use this tool while in download mode. You can test it, while in download mode, with:

$ ./heimdall print-pit

This will display the partition table of your phone. This is enlightening to know what it contains:

• IBL+PBL is the boot loader, be extra careful with it;
• PIT is the partition table;
• EFS contains some important data like IEMEI, be extra careful with it;
• SBL is the second boot loader;
• SBL2 is the backup of the second boot loader;
• PARAM contains some images used as various stage of the boot process;
• KERNEL is the kernel;
• RECOVERY is the backup of the kernel;
• FACTORYFS contains /system file system which is some kind of root file system;
• DBDATA contains /data file system which is the database to store application parameters;
• CACHE contains /cache file system which contains the Dalvik cache;
• MODEM contains the firmware for the modem.

As we will see, Heimdall is able to flash any of those partitions with flash subcommand.

adb adb stands for Android Debug Bridge. Among other things, this tool allow you to drive your phone through the USB connection. You can get a shell, reboot into download or recovery mode, push files to your Android device, etc. To get this tool, you need to download the full Android SDK.

$ tar zxvf ~/download/android-sdk_r11-linux_x86.tgz
$ cd android-sdk-linux_x86
$ tools/android update sdk
[select Android SDK Platform-tools]
$ platform-tools/adb shell
# uname -a
Linux localhost 2.6.35.7-I9000XWJVH-CL184813 #Dark_Core_2.4 PREEMPT Mon May 16 15:02:38 EEST 2011 armv7l GNU/Linux

The important commands are:

• shell to get a shell on the device;
• reboot recovery to reboot in recovery mode;
• reboot download to reboot in download mode;
• push to put a file on the phone;
• pull to retrieve a file;
• install to install an application from its APK;
• logcat to see the logs and what happens, very useful!

Restore download mode If your phone is missing download mode, you can recover it with the help of adb and heimdall. You need to grab some official firmware and extract boot.bin and Sbl.bin from it.

$ adb reboot download
$ heimdall flash --primary-boot ~/download/boot.bin --secondary-boot ~/download/Sbl.bin

Be extra careful here. You may brick your phone. Double check that you grab the correct files and that they are really for your phone (there exists some variants). It worked for me.

Installing a new firmware While you can switch from one firmware to another without losing your data, it is better to wipe your system and your data before switching firmware. This means that you will lose all your applications and the data inside them (list of contacts, SMS, preferences). However, nothing will ever modify your data in the internal SD card or in the external SD card (which are both inside the phone). They usually contains your music and your photos. There are two ways to install a firmware:

1. via Heimdall, or
2. using the recovery.

Heimdall When a guide tells you to use Odin, you can use Heimdall instead. Odin takes a TAR file. You should untar it and uses its content with Heimdall. For example:

$ ./heimdall flash --repartition --pit s1_odin_20100512.pit \
>                  --factoryfs factoryfs.rfs \
>                  --cache cache.rfs --dbdata dbdata.rfs \
>                  --param param.lfs --modem modem.bin \
>                  --kernel zImage --secondary-boot Sbl.bin 
Claiming interface again...   Success
Setting up interface...   Success
Beginning session...
Handshaking with Loke...   Success
Uploading PIT
PIT upload successful
Uploading SBL
100%
SBL upload successful
Uploading KERNEL
100%
KERNEL upload successful
Uploading MODEM
100%
MODEM upload successful
Uploading PARAM
100%
PARAM upload successful
Uploading FACTORYFS
100%
FACTORYFS upload successful
Uploading DBDATAFS
100%
DBDATAFS upload successful
Uploading CACHE
100%
CACHE upload successful
Ending session...
Rebooting device...

You may be interested to update your modem using Heimdall. This is usually independent of the firmware you want to use. Also, no need to provide a PIT file if you don t want to repartition. Heimdall can read the current one and flash only one of the partitions.

Recovery mode The base recovery shipped with the Samsung Galaxy S does not have a lot of options. Moreover, it may be locked and will accept to install only signed firmwares. Most firmwares come with ClockWorkMod, a more advanced recovery. Once you get it, you will be able to use any ZIP file to install a new firmware. My advice is to try to get ClockWorkMod as soon as possible. If you are root, you can install ROM Manager that will install it for you. If you are not root, most custom firmwares will provide ClockWorkMod. Therefore, you can just install a custom firmware with Heimdall. There exists special firmwares just for this, like EZbase. If you want to install CyanogenMod, be sure to download a Froyo firmware, not a Gingerbread one. Keep in mind that the recovery is located inside the kernel. It is not self-contained in the bootloader (unlike download mode). If you don t want to flash a whole firmware (and lose everything on your phone) just to get the recovery, you can just flash the kernel. The ClockWorkMod recovery comes with an utility to backup your whole phone. This way, you can easily restore your previous firmware if you don t like the current one. This is what is called a Nandroid backup.

Lagfix A lot of guides will tell you to enable or disable lagfix. Samsung shipped the Galaxy S with some crappy filesystem called RFS (which is some kind of enhanced FAT). It is slow and seems responsible for some lags that may happen on the interface. A lagfix is a way to replace this filesystem by something better, like EXT4 or YAFFS2. The process is pretty straighforward and consist to backup all the partition to the SD card, reformat it with another filesystem and restore the backup. The lagfix may be applied automatically with some firmware (like CyanogenMod) or you may request it in the recovery mode (most modified firmwares work this way). In this case, it will be applied automatically on the next boot. You should disable it before installing another firmware with the recovery mode (no need if you install it with Heimdall).

Applications You can find many free (as in free speech) applications for Android. Unfortunately, Android Market does not allow to use this criteria as a filter. F-Droid is an alternate market featuring only FOSS applications. Here is a short selection of free applications:

• ADW.Launcher is a home replacement app. It is very configurable and themable.
• K-9 Mail is a very nice email client.
• Xabber is a XMPP/Jabber client.
• ConnectBot is an SSH client (as well as a terminal).
• DroidWall is an application firewall (using Netfilter). It lets you select which application can access to Internet depending on your connection (wifi or 3G).
• Barcode Scanner decodes 1D/2D barcode including QR codes.
• OsmAnd is a replacement for Google Maps using OpenStreetMap. The free version on Android Market is out-dated. Buy the plus version or download it throught F-Droid.

Another must-have application is Titanium Backup which allows to backup any application with data. Unfortunately, this is not a free (as in free speech) application.

It s Co-operatives Fortnight until 9 July and our co-op is supporting many events:

• This afternoon (Wednesday 29 June), we ll be at Co-operative lending and ethical investment in Bristol;
• Tomorrow, we ll be at a Co-operative Business Breakfast in Glastonbury;
• Next Friday, we re hosting a Co-operative Business Breakfast in Weston-super-Mare;
• and we ll post more events to our co-op s news page as we find out about them.

As you may know, we ve already done two events: First, last Friday, Michael attended the Co-operatives UK Annual Meeting on our behalf. Co-operatives UK is our national co-ordinating co-op and the figurehead for the movement in this country. I didn t see the meeting (the live video link didn t work for me, as before), but Michael reports:

The AGM seemed well organized; it was good to hear about the work they re doing and the concerns regarding the co-operative situation in Poland. I think it s important to exercise one s ability to vote, but would have liked more opportunity to speak with people.

We couldn t send anyone to the Mary Portas session on Friday evening, but Paul Dale Smith summarised the session and RuthRosselson described it as Mary Portas tells Co-op Congress how it is . Then on Saturday, I was in Manchester for The Power of Co-operation, thanks to support from software.coop and The Co-operative Bank. The first session I attended, on mutualisting the Post Office was interesting and pretty full, but I found some later sessions rather frustrating. As well as my usual concerns about surrendering to uncritical use of private-sector software, I got quite annoyed by two sessions on member engagement where the presenters seemed to talk for almost the full time without letting the audience engage! Not good for my blood pressure, that, especially when I feel they re missing something! But the real value of the day was in the fascinating discussions I had with many people in the short breaks in the plenary, or while wandering around the exhibition and the marketplace. For example, it was great to see Revolver still selling the T-shirts I mentioned here in 2009 as well as giving out free samples of their brilliant fairtrade coffee. What comes from the various discussions will probably decide whether we attend similar events in future. Were you in Birmingham? How was it for you? Are you supporting Co-operatives Fortnight? How, where, when and why?

Hello The last version (1.247, available in Debian unstable as libconfig-model-perl) of Config::Model provides a brand new model for /etc/LCDd.conf. With this model, you can edit or validate your lcdproc configuration. To edit this configuration the first time, you will have to run:

config-edit -application lcdproc -force

The -force option is required to load the file the first time. (because the default LCDd.conf file contains configuration options for all drivers, even though only one them if usefull). You will get this screen:
In this screen, you can setup the driver for your adress with a right click on Driver parameter. Once youv e selected the driver and clicked on store , the driver will be listed in the left part of the window so you can tune the parameters of this specific driver: In this screen, I ve turned the backlight off and added a small note to remember why. This small note will be writted in a comment in /etc/LCDd.conf. Once all required modifications are done, it s time to save the configuration file using the menu entry File -> save Now, let s see the resulting file (which is a bit long due to the amount of comments in the original LCDd.conf file):

## This file was written by Config::Model
## You may modify the content of this file. Configuration
## modifications will be preserved. Modifications in
## comments may be mangled.
# LCDd.conf -- configuration file for the LCDproc server daemon LCDd
#
# This file contains the configuration for the LCDd server.
#
# The format is ini-file-like. It is divided into sections that start at
# markers that look like [section]. Comments are all line-based comments,
# and are lines that start with '
#
# The server has a 'central' section named [server]. For the menu there is
# a section called [menu]. Further each driver has a section which
# defines how the driver acts.
#
# The drivers are activated by specifiying them in a driver= line in the
# server section, like:
#
#   Driver=curses
#
# This tells LCDd to use the curses driver.
# The first driver that is loaded and is capable of output defines the
# size of the display. The default driver to use is curses.
# If the driver is specified using the -d  command line option,
# the Driver= options in the config file are ignored.
#
# The drivers read their own options from the respective sections.
[server]
# Where can we find the driver modules ?
# IMPORTANT: Make sure to change this setting to reflect your
#            specific setup! Otherwise LCDd won't be able to find
#            the driver modules and will thus not be able to
#            function properly.
# NOTE: Always place a slash as last character !
DriverPath=/usr/lib/lcdproc/
# Tells the server to load the given drivers. Multiple lines can be given.
# The name of the driver is case sensitive and determines the section
# where to look for further configuration options of the specific driver
# as well as the name of the dynamic driver module to load at runtime.
# The latter one can be changed by giving af File= directive in the
# driver specific section.
#
# The following drivers are supported:
#   bayrad, CFontz, CFontz633, CFontzPacket, curses, CwLnx, ea65,
#   EyeboxOne, g15, glcdlib, glk, hd44780, icp_a106, imon, imonlcd,
#   IOWarrior, irman, joy, lb216, lcdm001, lcterm, lirc, lis, MD8800,
#   mdm166a, ms6931, mtc_s16209x, MtxOrb, mx5000, NoritakeVFD, picolcd,
#   pyramid, sed1330, sed1520, serialPOS, serialVFD, shuttleVFD, sli,
#   stv5730, svga, t6963, text, tyan, ula200, xosd
Driver=imonlcd
# Tells the driver to bind to the given interface
Bind=127.0.0.1
# Listen on this specified port; defaults to 13666.
Port=13666
ReportLevel=3
# Sets the reporting level; defaults to 2 (warnings and errors only).
# ReportLevel=3
# Should we report to syslog instead of stderr ? [default: no; legal: yes, no]
# ReportToSyslog=yes
# User to run as.  LCDd will drop its root priviledges, if any,
# and run as this user instead.
User=nobody
Foreground=no
Hello=    Hello
Hello=    LCDproc!
GoodBye=    GoodBye
GoodBye=    LCDproc!
# The server will stay in the foreground if set to true.
# Foreground=no
# Hello message: each entry represents a display line; default: builtin
# Hello="  Welcome to"
# Hello="   LCDproc!"
# GoodBye message: each entry represents a display line; default: builtin
# GoodBye="Thanks for using"
# GoodBye="   LCDproc!"
# Sets the default time in seconds to displays a screen.
WaitTime=5
# If yes, the the serverscreen will be rotated as a usual info screen. If no,
# it will be a background screen, only visible when no other screens are
# active. The special value 'blank' is similar to no, but only a blank screen
# is displayed. [default: on; legal: on, off, blank]
# ServerScreen=no
# Set master backlight setting. If set to 'open' a client may control the
# backlight for its own screens (only). [default: open; legal: off, open, on]
# Backlight=open
# Set master heartbeat setting. If set to 'open' a client may control the
# heartbeat for its own screens (only). [default: open; legal: off, open, on]
# Heartbeat=open
# set title scrolling speed [default: 10; legal: 0-10]
# TitleSpeed=10
# The "...Key=" lines define what the server does with keypresses that
# don't go to any client.
# These are the defaults:
ToggleRotateKey=Enter
PrevScreenKey=Left
NextScreenKey=Right
ScrollUpKey=Up
ScrollDownKey=Down
# ScrollUpKey=Up
# ScrollDownKey=Down
# If you have only 4 keys, you can choose to use this:
# ToggleRotateKey=Enter
# PrevScreenKey=Up
# NextScreenKey=Down
# If you have only 3 keys, you can choose to use this:
# ToggleRotateKey=Enter
# PrevScreenKey=Up
[menu]
# You can configure what keys the menu should use. Note that the MenuKey
# will be reserved exclusively, the others work in shared mode.
# The following works excellent with 4 keys or more.
MenuKey=Escape
EnterKey=Enter
UpKey=Up
DownKey=Down
LeftKey=Left
RightKey=Right
[imonlcd]
Protocol=0
OnExit=2
Device=/dev/lcd0
# My fishes dont like the glow
Backlight=off
DiscMode=0

You will notice a few things:

• Comments from the original file in section [server] and [menu] were kept
• Unused drivers sections are gone
• The fishy note about backlight was saved as a comment above the backlight parameter

If you need to update the file, you will have to run:

sudo config-edit -application lcdproc

The -force option is no longer needed. If you don t like to click, you can run:

config-edit -application lcdproc -ui none -save

For instance:

config-edit -application lcdproc -ui none -save 'server driver=imonlcd - imonlcd Device="/dev/lcd1" '

Further reading If you want to know more about lcdproc model, you can view:

• LCDd model and all models for the drivers
• LCDd documentation and the drivers documentationgenerated from these models

Under the hood I was too lazy to write myself the models of all drivers (about 3500 lines for them all). All models were generated from the LCDd.conf template file provided by lcdproc project. This blog explains how this was done. But the result is far from perfect. Event though LCDd.conf comments give a lot of semantic information, parameter types could be better adjusted and some of the examples provided in LCDd.conf confuse the model generator.
So the current comments are good, but not enough. In the future, we could decide on better structural conventions for the comments that would give users the information they need and give a model generator enough information to generate better models. This will hopefully be the topic of yet another blog All the best

Two weeks ago, I updated the packages containing the BioPerl modules bioperl and bioperl-run, which allowed to resume the work done on GBrowse, one of the genome browsers available in Debian. As many Perl modules, BioPerl has extensive regression tests. Some of them need access to external services, and are disabled by default as the Internet is not available in our build farm. Nevertheless, they can be triggered through the environment variable DEB_MAINTAINER_MODE when building the package locally. Bioperl successfully passes all off-line tests, and a part of the on-line tests is already corrected its development branch. In contrary, BioPerl-Run fail the tests for Bowtie, DBA, EMBOSS, Genewise, Hmmer, PAML, SABlastPlus, T-Coffee and gmap-run. In some cases, like EMBOSS, it is because a command has been renamed in Debian. In other cases, in particular DBA et Genewise, it is much more difficult to figure out on which side is the problem. Regression tests are essential to determine if a package works or not on ports others than the one used by the packager (amd64 in my case). Even simple ones can be useful. In the case of T-Coffee, that has a rudimentary test that I have activated recently, it showed that the packages distributed on armel are not working at all. Running regression tests when building packages have advantages, in particular to have the results published for each port automatically as part of the build logs. But is also cause problems. First, a package would need to build-depend on every software it can interact with, in order to test it comprehensively. In the example of bioperl-run, it makes it impossible to build on armel as long as t-coffee is broken there. Second, this approach does not help the user to test similarly a program installed on his computer. Maybe the Debian Enhancement Proposal 8, to test binary packages with material provided by their source packages, will solve these two problems.