Search Results: "xam"

25 December 2009

Steve Kemp: And if someone gets upset you say, "chill out"!

It was interesting to see Clint Adams describe love and dissatification with configuration management. At work I've got control of 150(ish) machines which are managed via CFEngine. These machines are exclusively running Debian Lenny. In addition to these hosts we also have several machines running Solaris, OpenBSD, and various Ubuntu releases for different purposes. Unfortunately I made a mistake when I setup the CFEngine infrastructure and when writing all the policies, files, etc, I essentially said "OK CFEngine controlled? Then it is Debian". (This has been slowly changing over time, but not very quickly.) But in short this means that the machines running *BSD, Solaris, and non-Debian distributions haven't been managed as well via CFEngine as the rest, even though technically they could have been. A while back I decided that it was time to deal with this situation. Looking around the various options it seemed Puppet was the way of the future and using that we could rewrite/port our policies and make sure they were both cleanly organised and made no assumptions. So I setup a puppetmaster machine, then I installed the client on a range of client machines (openbsd, debian lenny, ubuntu, solaris) so that I could convince myself my approach was valid, and that the tool itself could do everything I wanted it to do. Unfortunately using puppet soon became painful. It has primitives for doing various things such as maintaining local users, working with cronjobs, and similar. Unfortunately not all primitives work upon all platforms, which kinda makes me think "what's the point?". For example the puppet client running upon FreeBSD will let you add a local user, setup a ~/.ssh/authorized_keys file but will not let you setup a password. (Which means you can add users who can login, but then cannot use sudo. Subpar) At this point I've taken a step back. As I think I've mentioned before I don't actually do too much with CFEngine. Just a few jobs:

Fetch a file from the master machine and copy into the local filesystem. (Making no changes.)
Fetch a file from the master machine, move it to the local system after applying a simple edit. (e.g "s/##HOSTNAME##/ hostname /g")
Install a package.
Purge a package.
Setup local user accounts, with ~/.ssh handled properly.
Apply one-line sed-style edits to files. (e.g. "s/ENABLED=no/ENABLED=yes/" /etc/default/foo)

(i.e. I don't use cron facilities, I add files to cron directories. Similarly I don't use process monitoring, instead I install the monit package and drop /etc/monit/monitrc into place.) There is a pretty big decision to make in the future with the alternatives being:

Look at Chef.
Stick with CFEngine but start again with a better layout, with more care and attention to portability things.
Replace the whole mess with in-house-fu.

If we ignore the handling of local users, and sudo setup, then the tasks that remain are almost trivial. Creating a simple parser for a "toy-language" which can let you define copies, edits, and package operations would be an afternoons work. Then add some openssl key authentication and you've got a cfengine-lite. For the moment I'm punting the decision but I'm 90% certain that the choice is CFEngine vs. Chef vs. In-House-Fu - and that puppet is no longer under consideration. Anyway despite having taken months to arrive at this point I'm going to continue to punt. Instead my plan is to move toward using LDAP for all user management, login stuff, and sudo management. That will be useful in its own right, and it will coincidentally mean that whatever management system we do end up using will have on less task to deal with. (Which can only be a good thing.) ObFilm: Terminator II

Steve Kemp: 907

Fetch a file from the master machine and copy into the local filesystem. (Making no changes.)
Fetch a file from the master machine, move it to the local system after applying a simple edit. (e.g "s/##HOSTNAME##/ hostname /g")
Install a package.
Purge a package.
Setup local user accounts, with ~/.ssh handled properly.
Apply one-line sed-style edits to files. (e.g. "s/ENABLED=no/ENABLED=yes/" /etc/default/foo)

Look at Chef.
Stick with CFEngine but start again with a better layout, with more care and attention to portability things.
Replace the whole mess with in-house-fu.

24 December 2009

Mark Brown: EIFF 2009

The Edinburgh Film Festival finished a couple of weeks ago. As ever, I went along and saw a bunch of films and stage interviews. The programme had been a little disappointing, mostly due to feeling a little constricted some things that are usually present were dropped (Mirrorball being the most obvious example) and the late night movies weren t very late night, starting about 10:30 for the most part. That said, it was a lot of fun I was more successful than normal in avoiding duds and there were several things that really stood out. Two films I saw early on that really stood out were Moon and Exam, two low budget indie science fiction films. Moon has had an awful lot of publicity already so I won t repeat what other people have said about it. Exam is a very tight, taut thriller eight people in a room in the final test of a long interview process, one of whom will get the job. Both films looked great a testament to how affordable good CGI has become. I ll be interested to see what follows them now there s some examples of low budget SF out there, and I m not sure what to make of the fact that both of the films were British. Pontypool was also excellent, a zombie movie about the dangers, or possibily salvation, of talk radio. If you see it (which you should) make sure you stay for the end of the credits. Also good was Modern Love is Automatic. It s a low budget indie flick which reminded me an awful lot of The Unbelievable Truth partly in terms of visual design but more in the way it decided to just jump off and handle things in a totally non-naturalistic fashion. It s a really tricky thing to pull off without looking like you just don t care about the audience (witness a lot of experimental films) but it s very impressive when it works and it worked here. On the down side Dario Argento s Giallo had the audience laughing, and I m fairly sure it was an at laugh rather than a with laugh. There came a point in the film where it felt like they d just run out of enthusiasm for the whole thing and were just throwing anything on the screen to tie up the loose ends. Very disappointing at a film festival. That was the only real blip, though overall it was good though there was cost cutting in evidence.

Muammar El Khatib: Installing Adobe Air in Debian AMD 64 (TweetDeck can be used)

I love twitting using TweetDeck. So that, I started to investigate how to install Adobe Air in Debian AMD64 and, it was a real pain in the ass. I have found a way of installing it in Debian AMD64.

First of all, you need to install the following packages:

$ sudo apt-get install ia32-libs lib32nss-mdns lib32asound2 lib32gcc1 lib32ncurses5 lib32stdc++6 lib32z1 libc6 libc6-i386 lib32nss-mdns libcanberra-gtk-module

After installing them, we have to create some symlinks from the location of some of those libraries in /emul/ia32-linux/usr/lib/ to /usr/lib32/. For accomplishing it, do the next:

$ cd /usr/lib32/
$sudo ln -s /emul/ia32-linux/usr/lib/libnss3.so .
$sudo ln -s /emul/ia32-linux/usr/lib/libssl3.so .
$sudo ln -s /emul/ia32-linux/usr/lib/libsmime3.so .
$sudo ln -s /emul/ia32-linux/usr/lib/libnspr4.so .
$sudo ln -s /emul/ia32-linux/usr/lib/libnssutil3.so.1d .
$sudo ln -s /emul/ia32-linux/usr/lib/libplc4.so.0d .
$sudo ln -s /emul/ia32-linux/usr/lib/libplds4.so .

In the case that you have problems with libplds4.so.0d, create the symlink like this:

$sudo ln -s /emul/ia32-linux/usr/lib/libplds4.so.0d libplds4.so.0d

Now, it is time to download Adobe AIR (in this case version 1.5).

$wget http://airdownload.adobe.com/air/lin/download/latest/AdobeAIRInstaller.bin

Once it is downloaded, we have to make the .bin file an executable:

$ sudo chmod +x AdobeAIRInstaller.bin

Then, we can proceed to install it:

$ sudo ./AdobeAIRInstaller.bin

If you get an error like this:

Error loading the runtime (libnss3.so: wrong ELF class: ELFCLASS64)

It means that you haven't done the symlink from /emul/ia32-linux/usr/lib/libnss3.so to /usr/lib32/ correctly or that you haven't installed the library which is missed ('apt-file search' is your friend).

If everything goes fine, then the last steps are:

sudo cp /usr/lib/libadobecertstore.so /usr/lib32

Note: make a symlink if you prefer.

And for TweetDeck I had to do this:

Download libgnome-keyring0 for 32 bits (in this example I used the one from SID, if you are using testing, download the one from that release that you are using), extract it with dpkg and copy the libraries to /usr/lib32:

$ wget http://ftp.us.debian.org/debian/pool/main/g/gnome-keyring/libgnome-keyring0_2.28.2-1_i386.deb
$ sudo dpkg-deb -x libgnome-keyring0_2.28.2-1_i386.deb libgnome-keyring0_2.28.2-1
$ sudo cp libgnome-keyring0_2.28.2-1/usr/lib/* /usr/lib32/

Then install TweetDeck (http://tweetdeck.com/download/) and Enjoy

Note: If you find any problem, please do not hesitate to write a comment so that I can help you or we can improve this post.

23 December 2009

Torsten Landschoff: Hitting the dynamic linker wall

I was working on replacing some mockup code for testing an internal library with python. Basically, the C code is incredibly big and I would rather mock the 3rd party API we are using in python. I wrote a generator to create wrapping code that forwards the C API calls to my python module. Now that most of the work is finished, I get the following error message from my python mock:

1
2
3
4
5
6

Traceback (most recent call last):
File "simple_mockup.py", line 2, in <module>
import threading
File "/usr/lib/python2.5/threading.py", line 11, in <module>
from time import time as _time, sleep as _sleep
ImportError: /usr/lib/python2.5/lib-dynload/time.so: undefined symbol: PyExc_ValueError

What s going on here? This issue reminds me of a bug of my ancient Debian times which affected loading of GTK theme engines from python-gtk. The bug (#38138) is so old, it s not even in the BTS archive anymore The problem is illustrated by the following example program (consisting of three files):

demo.c

1
2
3
4
5
6
7

#include <Python.h>

void test_python()

Py_Initialize();
PyRun_SimpleString("import threading\n");

main.c

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

#include <stdio.h>
#include <dlfcn.h>

int main(void)

const char *error;

void *handle = dlopen("./demo.so", RTLD_LAZY EXTRA_RTLD_FLAGS);
void (*test_python)();
*(void**) &test_python = dlsym(handle, "test_python");
if ((error = dlerror()))
fprintf(stderr, "%s\n", error);
return 1;

printf("Calling test_python @ %p in shared object @ %p.\n", test_python, handle);
(*test_python)();
dlclose(handle);
printf("Feels fine, finishing.\n");
return 0;

run_it.sh

1
2
3
4
5
6
7
8
9
10
11
12
13

#! /bin/sh

gcc -shared python-config --includes -o demo.so demo.c python-config --ldflags

echo "Running example with default RTLD flags:"
gcc -DEXTRA_RTLD_FLAGS=0 -o main main.c -ldl
./main
echo

echo "Passing RTLD_GLOBAL in addition:"
gcc -DEXTRA_RTLD_FLAGS=RTLD_GLOBAL -o main main.c -ldl
./main
echo

On my system, this results in the following output:

torsten@pulsar:~/sh_bug$ ./run_it.sh
Running example with default RTLD flags:
Calling test_python @ 0xb77084bc in shared object @ 0x96bc018.
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/python2.5/threading.py", line 11, in <module>
    from time import time as _time, sleep as _sleep
ImportError: /usr/lib/python2.5/lib-dynload/time.so: undefined symbol: PyExc_ValueError
Feels fine, finishing.
Passing RTLD_GLOBAL in addition:
Calling test_python @ 0xb783f4bc in shared object @ 0x95e1018.
Feels fine, finishing.

Sucky. So embedding Python into an application is easy but if you want to use the interpreter and its modules from a plugin, you are out of luck. Somebody knows a way around this problem? The only solution I can think of is to link libpython.so into each of the python plugins. Update: Work around Small update: For my current problem, the work around is to run the application with the python library preloaded: LD_PRELOAD=/usr/lib/libpython2.5.so.1.0 app. Back to adding functionality

Benjamin Mako Hill: What's in a name?

Over the summer, there was a bit of a tussle at the highest level of Ubuntu governance over whether or not Canonical Ltd., the company that funds the majority of work done directly in Ubuntu, should name its file syncing and backup service Ubuntu One. Canonical's service involved a freely licensed client included in the Ubuntu distribution but, as a network service running on Canonical servers, it was not clearly a part of Ubuntu (the GNU/Linux distribution) or Ubuntu (the community) in the way the term was usually used within the community. Although the network service itself was not Franklin Street Statement free, this was not the most important issue for everyone who objected to the name. The major issue for many seemed to boil down to the fact that, free or not, Ubuntu One is a service run entirely by Canonical outside the reaches of the Ubuntu governance structures. Decisions were made and not everybody -- and maybe not anybody -- was absolutely happy with the outcome. My goal is not try to revive old arguments here. As far as I'm concerned, the issues are settled and the service is called Ubuntu One. That said, the questions raised during the episode are fundamental to Ubuntu and to other firm-sponsored FLOSS projects. Now that the dust has settled, they are worth reflecting on. From a legal perspective, there never was any ambiguity. Canonical "owns" the Ubuntu trademark. In this important sense, "Ubuntu" means whatever Canonical says it means. This is hardly new. As just one example, the Official Ubuntu Book (of which I am an author) was written by community members but became official because Canonical blessed it. But despite the fact that they don't need to, Canonical has often consulted with the community and its governance structures about trademark licensing policy. This was also not a case of Canonical not listening to the community. Canonical employees approached the Ubuntu Community Council (Ubuntu's highest governance board of which I am a member), listened carefully to concerns, and responded thoughtfully. The question was not even about a clash between what Canonical and the CC thought about the issue. An unambiguous majority of the board, including all the Canonical employees and several of the community members, supported the idea of Canonical using the trademark. The question was one about who gets to make the decisions about the Ubuntu name and about what role the community and Canonical would each play. Despite the fact that a majority of the Ubuntu community council was likely to support the proposed name, the CC was told by Canonical CEO Mark Shuttleworth that a vote was irrelevant. Canonical made it clear that decisions about the Ubuntu trademark were simply not in the CC's purview. The decision on how the name was to be used was something that Canonical was not willing to delegate to an "outside" (the firm) governance body. Few businesses would. And although I don't agree with the decision as a community representative, I might have even made the same call from Canonical's shoes. In traditional firms, it's usually pretty clear where the organization's boundary lies. In FLOSS projects -- and especially in FLOSS project like Ubuntu who are sponsored in very large part by a single for-profit company -- boundaries are fuzzier. The conversation about "Ubuntu One" can be seen as a fight over what "Ubuntu" refers to, and, more importantly, who gets to answer that question. In deciding whether to call a service "Ubuntu", a decision is made on what Ubuntu is. Names are powerful. Is Ubuntu just a Canonical project? Are Ubuntu's contributors really just Canonical contributors by proxy? I think the answer to both questions is "no" but the boundary issues involved are complicated and under constant negotiation. Every time Canonical uses the Ubuntu name itself or grants others the ability to do so, these boundary issues are negotiated, one way or another. This boundary setting work reveals an important tension that firms releasing FLOSS must all struggle with. To what extent and in what ways do communities get to decide what a FLOSS project is and to what extent do sponsoring firms get to do so? How should projects and firms do this most effectively? What should we even be optimizing for? I think that any resulting balance has a huge effect on whether a FLOSS project is, on one hand, released under a free license but run like any old corporate project or, on the other, a true "bazaar" style project where no single firm dominates -- or where they fall on that spectrum. Names and trademarks are one way that projects define their own identities and act as an important frontier in this balancing act. As every firm/project negotiates their own answers to questions of names and boundaries, there are important implications for the project's ability to attract volunteers, solicit contributions from other firms, and more. The confusion around conversations about Ubuntu One shows that we still have a lot to learn.

Ian Wienand: Stripping shared libraries

So, how to strip a shared library? --strip-unneeded states that it removes all symbols that are not needed for relocation processing. This is a little cryptic, because one might reasonably assume that a shared library can be "relocated", in that it can be loaded anywhere. However, what this really refers to is object files that are usually built and bundled into a .a archive for static linking. For an object in an static library archive to still be useful, global symbols must be kept, although static symbols can be removed. Take the following small example:

$ cat libtest.c
static int static_var = 100;
int global_var = 100;
static int static_function(void)  
       return static_var;
 
int global_function(int i)  
    return static_function() + global_var + i;

Before stripping:

$ gcc -c -fPIC -o libtest.o libtest.c
$ readelf --symbols ./libtest.o
Symbol table '.symtab' contains 18 entries:
   Num:    Value  Size Type    Bind   Vis      Ndx Name
...
     5: 00000000     4 OBJECT  LOCAL  DEFAULT    5 static_var
     6: 00000000    22 FUNC    LOCAL  DEFAULT    3 static_function
    13: 00000004     4 OBJECT  GLOBAL DEFAULT    5 global_var
    16: 00000016    36 FUNC    GLOBAL DEFAULT    3 global_function

After stripping:

$ strip --strip-unneeded libtest.o
$ readelf --symbols ./libtest.o 
Symbol table '.symtab' contains 15 entries:
   Num:    Value  Size Type    Bind   Vis      Ndx Name
...
    10: 00000004     4 OBJECT  GLOBAL DEFAULT    5 global_var
    13: 00000016    36 FUNC    GLOBAL DEFAULT    3 global_function

If you --strip-all from this object file, it will remove the entire .symtab section and will be useless for further linking, because you'll never be able to find global_function to call it!. Shared libraries are different, however. Shared libraries keep global symbols in a separate ELF section called .dynsym. --strip-all will not touch the dynamic symbol entires, and thus it is therefore safe to remove all the "standard" symbols from the output file, without affecting the usability of the shared library. For example, readelf will still show the .dynsym symbols even after stripping:

$ gcc -shared -fPIC -o libtest.so libtest.c
$ strip --strip-all ./libtest.so 
$ readelf  --syms ./libtest.so 
Symbol table '.dynsym' contains 11 entries:
   Num:    Value  Size Type    Bind   Vis      Ndx Name
...
     6: 00000452    36 FUNC    GLOBAL DEFAULT   12 global_function
    10: 000015e0     4 OBJECT  GLOBAL DEFAULT   21 global_var

However, --strip-unneeded is smart enough to realise that a shared-object library doesn't need the .symtab section as well and remove it. So, conclusions? --strip-all is safe on shared libraries, because global symbols remain in a separate section, but not on objects for inclusion in static libraries (relocatable objects). --strip-unneeded is safe for both, and automatically understands shared objects do not need any .symtab entries to function and removes them; effectively doing the same work as --strip-all. So, --strip-unneeded is essentially the only tool you need for standard stripping needs! See also

Debian Policy Manual

Dirk Eddelbuettel: RInside release 0.2.0

RInside, a set of convenience classes to facilitate embedding of R inside of C++ applications, was updated to release 0.2.0 over the weekend. Sources as well as binary packages are at CRAN; sources are also at my RInside page The biggest news is that we now support builds on Windows -- if and only if you use the R toolchain -- more on that below. Since I had first released RInside people had emailed about support on Windows. However, Richard Holbrey deserves special mention as he actually sat down and tried to build it. Over a few email exchanges we made decent progress, but stalled. More recently, I sat down and had another go and lo and behold, it works. This required a few #ifdef statements here and there, as well as a function setenv() which Windows does not have -- Richard kindly helped here and borrowed most of the code from R itself. Now, to make this plain: you need gcc (as this is how R itself is built). This gcc compiler is the native compiler on Linux and OS X. But on Windows this requires installing the Rtools compiled by Duncan Murdoch as detailed in the The Windows Toolset appendix to the R Installation manual. Do not, I repeat, do not report an error if you fail to build this with another compiler as this is not supported. That said, if you have the proper tools, things are peachy. Install the package, change into the examples directory installed with it and say make (or on Windows: make -f Makefile.win). That's it -- now run the examples (which on Windows will most likely require setting the environment variable R_HOME). Now you can run simple examples like the one from the previous blog post or any of the other ones. Happy embedding!

20 December 2009

Dirk Eddelbuettel: Rcpp and inline example

Following up on yesterday's Rcpp 0.7.0 release, here is a quick example of why the new feastures can be so useful. The following R code defines a character variable gslrng. This variable contains a short C++ code segment, which is then transformed by the function cfunction into a function of two arguments assigned to funx: <body style="background-color:#ffffff">

    ## now use Rcpp to pass down a parameter for the seed, and a vector size
    gslrng <- '
       int seed = RcppSexp(s).asInt();
       int len = RcppSexp(n).asInt();
       gsl_rng *r;
       gsl_rng_env_setup();
       std::vector<double> v(len);
       r = gsl_rng_alloc (gsl_rng_default);
       gsl_rng_set (r, (unsigned long) seed);
       for (int i=0; i<len; i++)  
           v[i] = gsl_rng_get (r);
        
       gsl_rng_free(r);
       return RcppSexp(v).asSexp();
    '
    ## turn into a function that R can call
    ## compileargs redundant on Debian/Ubuntu as gsl headers are found anyway
    funx <- cfunction(signature(s="numeric", n="numeric"),
                      gslrng,
                      includes="#include <gsl/gsl_rng.h>",
                      Rcpp=TRUE,
                      cppargs="-I/usr/include",
                      libargs="-lgsl -lgslcblas")
    print(funx(0, 5))

</body> The signature argument to cfunction defines two variables s and n -- which the C++ function then reads in from R and converts to two integers seed and len. seed is used to initialize the random-number generator, and len draws are then taken and stored in the STL vector v which returned at the end. As the R level, we now have a function of two arguments returning a vector of RNG draws of the given lenth and using the given seed. Also note how we tell cfunction to add the GSL include line, specify that we want to compile and link against Rcpp and provide -I and -L arguments to compile and link with the GSL. (The include statement is not needed as the compiler would have found them in /usr/include anyway, but it shows how to set this if needed.) Finally, we simply call our freshly compiled, linked and loaded C++ function with arguments zero for the seed and five for the length, and print the results vector returned to R from C++.

Wouter Verhelst: Re: Grrr

I received a number of comments on my "Grrr" post, all of which missed the point: Yes, I am aware that there are many more ways to fix this issue beyond a memcpy. However, the example code is legal and would not crash the application, if not for the fact that libc thinks I am doing something wrong. On top of that, this kind of overflow "protection" only kicks in when the code is compiled with -O2 rather than with -g -O0. While I am not sure whether the difference is due to the absense of debugger symbols, or rather due to the different optimization levels, fact is that software which runs fine in debugging should also run fine in production. There are good arguments for compiling all C code with -Wall -Werror, and I do that as a matter of course for all C software that I write. However, sometimes automated tools are just wrong in their compile- or run-time bug detection, in which case such it should be possible to disable that detection. This is one such case, and my blog post was more about ranting about the inability to do so, rather than about the fact that I had to memcpy when in fact there were other options available. But yeah, perhaps I should have been clearer about that. Forgive me for not being clear after having fought with compilers for far too long.

Wouter Verhelst: Good, not evil

There is a bit of a fluff online currently about the following clause in the "jsmin" code (whatever that is):

The software shall be used for good, not evil

This seems to have started when Google rejected a project based on that code due to its license being not free or open source according to their standards, and therefore not welcome anymore. The arguments then quickly degenerated into things like 'when did google stop being against evil'. But those are all besides the point. One of the most important properties of free and open source software is that anyone can use it for any purpose; there are no restrictions to using them. The DFSG (and hence, the OSD which was derived from the DFSG) encode this as follows:

No Discrimination Against Fields of Endeavor
The license must not restrict anyone from making use of the program in a specific field of endeavor. For example, it may not restrict the program from being used in a business, or from being used for genetic research.

You may think "how is 'evil' a field of endeavor", but that is the wrong question. To "use software for evil" can mean any of a number of things, including "nuclear research", "weapons development", "abortion", or, heck "the cash register in a butchery shop", depending on the ethical and moral framework through which the person writing the license sees the world. The ability to give someone a CD or DVD with a bunch of software on there, being able to tell them that they can just use this in any which way they see fit, is a very strong and important feature of the free and open source community. Every time someone comes up with a clause like the above, however, this ability is diluted somewhat; and if it is readily accepted within the greater free and open source movement, then eventually everyone interested in using a piece of software must first check whether they're not trying to use software that forbids someone's pet evil, and we lose one of the greatest strenghts that does exist for free software, but not for proprietary software. The sad thing is, the jsmin author seems to agree. From a video/transcript on which he talks about his absurd license clause is the following quote:

Also about once a year, I get a letter from a lawyer, every year a different lawyer, at a company I don't want to embarrass the company by saying their name, so I'll just say their initials IBM
[laughter]
saying that they want to use something I wrote. Because I put this on everything I write, now. They want to use something that I wrote in something that they wrote, and they were pretty sure they weren't going to use it for evil, but they couldn't say for sure about their customers. So could I give them a special license for that? Of course. So I wrote back this happened literally two weeks ago "I give permission for IBM, its customers, partners, and minions, to use JSLint for evil."

Or, in other words, all you have to do if you want to use this software for evil is set up a second company, tell Douglas that this second company wants to sell software that uses his software to people who might use it for "evil", even though the first company won't, and you're in business. Because Douglas doesn't really oppose evildoers. So the question is, why is that clause in there in the first place? There are only two possibilities; either Douglas didn't really think about those issues, in which case I hope he will one day see the light and remove the clause; or he did, and decided to go ahead and put that clause there anyway. And that would be evil.

Christian Perrier: Font packages moved to source v3

Today, Hideki Yamane completed his work on the last four packages maintained by the Font packaging Team to make them switch to source v3. Along with these changes, we also moved all our packages to debhelper v7 and minimal rules files. We also dropped the dependency on the obsoleted defoma. The team maintains 77 source packages for fonts and has now built a quite solid experience in this. In the upcoming weeks, I'll try to get in touch with other people who maintain fonts, inviting them to join the team and maintain their packages in our SVN (we don't do very fancy things so SVN is very well suited for such work). The maingoals will be to continue dropping support for defoma (dropping defoma hints files: you can see good examples on how to do this properly in packages maintained by the team, such as ttf-dejavu). We also intend to drop useless Recommends or Suggests that appear in some font packages (often on "x-ttcidfontconf fontconfig" which is now completely useless) as well as harmonizing package naming schemes (with a "(ttf otf)-<foundry>-<fontname>" scheme). Feel free to join and share your advice and thoughts about The Right Way To Maintain Font Packages on our mailing list.

Debian News: Updates on LXDE

Several LXDE components have been updated recently in sid. They bring some new features and improve flexibility of customization. Below you have a summary of the most important changes:

lxde-common 0.5.0-2:
The nuoveXT2 icon theme has been removed from lxde-common and it is has been moved to lxde-icon-theme package instead.
The startlxde script now executes 'lxsession -s LXDE -e LXDE' rather than 'lxsession -s LXDE' to reflect the changes in lxsession.
lxsession 0.4.1-1:
There is a new command line argument: -e. This let you set the internal name of the DE. This value is exported via the environment variable $XDG_CURRENT_DESKTOP. The DE name can be used to match desktop files which contain compatible OnlyShowIn and NotShowIn key values.
The -s argument is reserved for session name only and was used to load related config files for the desktop session. For example, you can execute: lxsession -s Lubuntu -e LXDE This should load config files from Lubuntu directory, but the DE is still recognized as LXDE by applications.
lxpanel 0.5.4.1-1:
Now you can determine the visibility of applications in the menu by comparing OnlyShowIn and NotShowIn key values and $XDG_CURRENT_DESKTOP. This means, you can have lxpanel show Gnome menu by the following export XDG_CURRENT_DESKTOP=3DGNOME . Then lxpanel will show gnome applications instead of LXDE ones.
Finally, lxnm and lxpanel-netstat-plugin are now deprecated.
lxnm has been already requested to be removed from the archive and it is recommended to use wicd instead. lxpanel-netstat-plugin has also dropped from lxpanel in the upload 0.5.4.1-1.

Contributed by: Andrew Lee

19 December 2009

Dirk Eddelbuettel: Rcpp 0.7.0

A new release of Rcpp awaits inclusion on CRAN and has also been sent as a .deb to Debian. It will hit mirrors over the next few days, in the meantime you can get it here as well. This release has a couple new features :

inline support: the cfunction from Oleg Sklyar's excellent inline package has been imported and adapted. This means simple C++ programs can be defined in an R character vector and passed to cfunction which will create a complete file that it then compiles, links and loads --- giving you access to compiled C++ code right from the R prompt without having to worry about compiler flags, linker options, ... Better still, we extended this to not only support Rcpp but any external library via addtional header / linker arguments that will be passed to R via the PKG_CPPFLAGS, PKG_CXXFLAGS and PKG_LIBS environment variables.
this even works on Windoze (if you have the Rtools installed as detailed in the Windows Toolset appendix to the R Installation manual) in exactly the same way. And no folks, that still does NOT mean you can use Visual Whatever -- R really requires MinGW as the links in this parapgraph document very plainly. But if and when you have the tools, R's remarkable consistency across operating systems allows you to use Rcpp and inline in pretty much the same way.
A handful of new examples for the inline support have been added.
A new type RcppSexp for simple int, double or std::string scalars as well as vectors; this is particularly useful for the inline support.
This also completes the source code reorginsation: every class now has its own header and implementation file
Last but not least, the package has been relicensed from LGPL-2.1 (or later) to GPL 2 (or later).

Fuller details are in the ChangeLog on Rcpp page.

Russell Coker: Shared Objects and Big Applications

Some time ago I wrote a little utility named memlockd [1]. Memlockd will lock files into memory which allows significantly faster access when the system pages heavily, in my simulated tests I have found that having the programs and shared objects needed for logging in locked in memory can make it possible to login without a timeout when there is heavy paging, this can make the difference between recovering a system with some processes that are out of control and having to reboot it (often without discovering the root cause). As always happens some people use my software in ways that I never planned. One guy is using it to try and make OpenOffice.org load faster. I m not sure that this is a good idea. In a typical installation when configured for the purpose that I intended it (system recovery from a rabbit process) memlockd will take a bit less than 10M of RAM on an i386 platform (that is for bash, login, sshd, getty, busybox, and all necessary shared objects and a few data files. Since RHEL 4 Red Hat distributions have whinged at boot time if there was less than 256M of RAM available, installation of a Red Hat based system on anything less than 128M of RAM has been impossible for some years, and Debian systems perform very poorly with less than about 128M of RAM when you run apt-get. I initially designed memlockd to run on my SE Linux Play Machine which has 128M of RAM in it s current incarnation. Locking 7.5% of RAM on the system may impact performance, but as a large part of that RAM is used for things like libc and bash which tend to be partially paged in at all times this shouldn t be a noticeable impact. But locking 100M or more of OpenOffice seems more likely to have the potential to hurt performance, I often run OpenOffice on a machine with 512M of RAM and the biggest desktop machine I use has 1.5G of RAM for me it wouldn t make sense to lock OpenOffice into memory. But it could be that there is some unusual aspect of his system that makes running memlockd with OpenOffice likely to give worthy benefits in performance without significantly hurting performance for other programs, for example it could have 4G of RAM and a really slow disk. It is also a possibility that the usage of the system makes OpenOffice so much more important than other programs that any decrease in performance in other areas is not relevant. In any case I m happy to help people use my software to do unusual things so I ll support this use. I ve been asked why memlockd doesn t seem to give much benefit when starting up OpenOffice when run with +/opt/openoffice.org3/program/soffice.bin in the config file, where the + means to lock all shared objects that ldd reports that the binary needs. for n in ldd /usr/lib/openoffice/program/soffice.bin sed -e s/^.*=..// -e s/\ .*// sort -u ; do readlink -f $n ; done sort -u > ldd.txt I used the above command to get a list of all shared objects that ldd reports for the soffice.bin program. On my system (Debian/Lenny i386) it reports 77 shared objects loaded. When memlockd locks all those ps reports that the RSS is 51944K. cat /proc/1234/maps sed -e s/^.*\ // sort -u > /tmp/map.txt Then I used the above command to get a list of the files that are memory mapped by OpenOffice when running OpenOffice calc where 1234 is the PID of the soffice.bin process (I expect that the numbers will be similar for writer, impress, etc I just happened to have a spreadsheet open). It reports 172 memory mapped files which include 9 files related to fonts and 64 shared objects under /usr/lib/openoffice/program which are not found by ldd among other things. It s quite common for a large application to use dlopen(3) at run-time to map shared objects instead of linking against them. Running memlockd with this list gave an RSS of 118644K, which is more likely to give a useful performance boost to OpenOffice load times.

[1] http://www.coker.com.au/memlockd/

Adrian von Bidder: True Support

I know, preaching to the choir and all that. But this can't be said often enough (and perhaps the odd non-Debian person stumbles upon this...): Here's another example of how amazingly fast free tech support works: The newest Linux kernel (Debian package version: 2.6.32-1) wouldn't boot on my QNAP TS-419P NAS (see also my earlier posting about the device). Now, since I don't have a serial console cable, I can't really help a lot to debug this (and I am extremely happy that the people at QNAP have thought about recovery: it's trivial to just flash a working kernel or firmware image via DHCP and TFTP), bu still Martin Michlmayr immediately took the time to help me and soon could reproduce it on one of his own machines. The problem has been identified, I got a fixed kernel, and a patch is on his way to the kernel maintainers. All in the after literally just a few days.

18 December 2009

Asheesh Laroia: Diversity in Free Software: South Asians as an example

As someone born in India, I sometimes look around and wonder, Where are the Indians (and other South Asians) in Free Software? (I don't mean to exclude South Asians from other countries, so I will lump us together. I believe that we are more similar than we are different, although I know more about India than about the rest of South Asia.) There is no shortage of Indians performing information technology jobs in the United States. The same is true in academia; the Computing Research Association uses National Science Foundation data to show about 15% of computer science bacholor's degrees are awarded to "Asians or Pacific Islanders." These are not precise numbers targeted at South Asians in particular, but they confirm a general feeling that plenty of technologists in the United States are from that part of the world. South Asia is quite a populous region, coming in at over one billion people. It, too, has plenty of technology workers. So much FLOSS conversation happens in English, and India is well-suited to handle this; English is an "official language". Indian academia reports that there are 350 million English users and about 90 million English speakers. So let's visually compare the Debian developers map for South Asia (over one billion people) and that of New Zealand, a country of four million. India:

New Zealand:

These two countries have about the same number of Debian developers (at least, who have marked their location in the Debian LDAP database). About four. South Asians comprise about one sixth of the world's population. There are about one thousand Debian developers; we represent at best 1% of that. These numbers are comparable to the under-representation of women in Free Software, especially when you compare the figure to South Asians' over-representation in the rest of information technology. That makes me sad. Take a look at the Debian developer map again. You'll see that Debian is certainly not an Americans-only project, or even an English-speakers-only project. South America has a respectable dotting of developers, and Western- to Central-Europe are packed. I have strong feelings about Free Software. It emerges from an ethos of personal empowerment, and with open source it has become a dominant force in computing. Yet there are plenty of sharp people -- at least women and South Asians -- who, somehow, become culturally excluded from participating.

Why care about diversity? Consider the diversity of contributors we already have. Some contribute to Free Software because of particular business needs, such as what caused Avi Kivity to write KVM, the new leader in Linux-based virtualization. Everaldo's art background gave us the "Crystal" icon set that set the standard for sharp-looking icons on the Free Desktop for years. Josh Coalson knew about compressing sound, and his Free Lossless Audio Codec is now the standard in high quality audio. We already have a great deal of diversity. We should be celebrating! Back in 2001, FLAC's users were celebrating. In that year, I decided to ditch proprietary operating systems because I felt I could achieve all my computing needs in the Free world. A happy user of FLAC myself, I lurked on the mailing list as I watched grateful people thank Josh for the great software he wrote. Different contributions will excite different sorts of users. The more different people we have improving FLOSS, the more happy users we can make. Happy users of FLOSS are Free users. Happy users can become contributors, putting forth code, documentation, translations, and word-of-mouth marketing. The first reason to improve diversity in FLOSS is to better suit our users' needs. The more diversity we have in our contributors, the more chance we have of tickling our users in the ways that please them the most. I wish to see an end to software that restricts users' freedom, so I want to see us build the tools that users want. One thing that pleases me is when I see other people contributing who seem similar to me. When I went to Debconf, I was thrilled to be surrounded by people who cared about software freedom and technical excellence. I had even more fun being social, chatting about rainforests, mutual friends, websites, and music. I might have had the most fun playing the card game Mao. A second reason, then, to improve diversity in FLOSS is to increase contributor retention by increasing joy. Mao was an example of a cultural bond I happened to share with a handful of Debianites. The more diversity we have, the more frequent these sorts of coincidences will be. The final, most obvious, reason to reach out to groups of people who do not typically contribute is that we can increase our numbers. That by itself is so valuable. Ubuntu sees 100 new bugs per week, even after the bug squad's efforts. If we can do a better job of recruiting new contributors, the raw numbers give us more strength in creating and maintaining world-class software as well as letting the world know about it.

Changing the balance I believe that there are plenty of South Asians quite capable of contributing to FLOSS. I believe the same of women. I believe the same of men. Back to the topic at hand. Why do the South Asians vanish when we look at Free Software, not tech in general? There are plenty of reasons I can dream up, based on my experience with Indians.
Plenty of South Asian parents urge their children to make low-risk career choices.
My mother reports that schools in India focus on memorization instead of creativity. This can leave little room for extracurricular pursuits.
It's tough for FLOSS advocates to work directly on these distant issues. But I think we can focus some problems we can help solve. Crucially, awareness of Free Software spreads best by social circles. I learned about Linux from a friend at a summer camp. I'll repeat that:
Awareness of Free Software spreads best by social networks.
So if you want to spread that awareness, try to be a bridge. If you meet someone from an unusual background for open source who needs support or mentorship, try to help. That is an investment in the diversity and growth of Free Software. Those people can now unlock more "open source minorities."

What success looks like Google Summer of Code helps some new contributors get started and provides that mentorship. Rachel McCreary was invited to the SciPy conference after a successful summer. Her father left a comment explaining how her sisters participated in FLOSS via Google's Highly Open Participation (GHOP) Contest:
Rachel was inspired and motivated by BOTH of her little sisters, each completing six GHOP tasks (if memory serves).

GHOP and GSOC has been a game-changer for these girls. Rachel's younger sister is applying to schools such as MIT with an interest in a science major. The youngest daughter now has a Caltech poster on her wall with the intent to eventually attend.

Their proud Dad
Soon, these stories will be commonplace. Until then, we have work to do. (I'm still researching these topics. If you can help me find any sort of data to help me learn more about diversity in FLOSS, even if it seems like I wouldn't like it, leave a comment.)

14 December 2009

Phil Hands: arduino for the java challenged

Having excitedly bought an Arduino from the nice folks at tuxbrain.com while they were at DebConf9, I was then a little deflated to discover (while attending the UKUUG's Arduino Workshop in August), that the IDE is written in Java. While I can see that the authors might perceive this as the easiest way to provide cross-platform pointy-clicky-ness, JREs have a half-life of about an hour on my systems.Anyway, I fought with Java to show willing, until I'd determined that the IDE didn't seem to be happy running under xmonad (rendering menus so that they were unclickable) and gave up -- spending the rest of the tutorial proving to myself that I could compile and upload C code for the little beastie, but failing to compile Arduino sketches on the command line.Ever since, I've been meaning to work out the required CLI incantations, and finally made time over the weekend. The incantations turn out to be rather simple, if not very obvious, so I thought the world could benefit from a package that encapsulates the required knowledge.I've pared the source tree down to the c++ code and some examples and tweaked the Makefile so that it'll do it's work out of tree, referring to the files that the package installs for you, which allows you to apt-get install aurduino-core and then cut&paste a few lines from the contained README.Debian in order to get an LED blinking for the first time.The resulting git repo is now on Alioth. The binary-indep package it produces I've called arduino-core -- hopefully I've left enough room for any fan of the Java IDE to shove that back into the upstream branch and add packages for the full IDE, and maybe throwing in another package for the GTK IDE that's out there.I'm very mildly concerned by the fact that Arduino is a trademark of the Arduino Team, but that trademark is for the circuits, not the software, so this is probably irrelevant. If you think I should care enough to rename the package to freeduino, say, please get in touch.Anyway, if you have an Arduino (or similar) please grab a copy and test it.

13 December 2009

Dirk Eddelbuettel: RQuantLib 0.3.1 released

A new version of RQuantLib is now out at CRAN and in Debian. It updates the package to the 0.9.9 release of QuantLib and is mostly a maintenance release with a few minor fixes along the way. Full changelog details, examples and more details about this package are at my RQuantLib page.

David Bremner: Reading MPS files with glpk

Recently I was asked how to read mps (old school linear programming input) files. I couldn't think of a completely off the shelf way to do, so I write a simple c program to use the glpk library. Of course in general you would want to do something other than print it out again.

Next.

Previous.