Search Results: "spa"

6 March 2012

Ian Wienand: Investigating the Python bound method

You work with Python for a while and you'll become familiar with printing a method and getting
<bound method Foo.function of <__main__.Foo instance at 0xb736960c>>
I think there is room for one more explanation on the internet, since I've never seen it diagrammed out (maybe for good reason!). An illustration of Python bound methods In the above diagram on the left, we have the fairly simple conceptual model of a class with a function. One naturally tends to think of the function as a part of the class and your instance calls into that function. This is conceptually correct, but a little abstracted from what's actually happening. The right attempts to illustrate the underlying process in some more depth. The first step, on the top right, is building something like the following:
class Foo():
      def function(self):
      	  print "hi!"
As this illustrates, the above code results in two things happening; firstly a function object for function is created and secondly the __dict__ attribute of the class is given a key function that points to this function object. Now the thing about this function object is that it implements the descriptor protocol. In short, if an object implements a __get__ function; then when that object is accessed as an attribute of an object the __get__ function is called. You can read up on the descriptor protocol, but the important part to remember is that it passes in the context from which it is called; that is the object that is calling the function. So, for example, when we then do the following:
f = Foo()
f.function()
what happens is that we get the attribute function of f and then call it. f above doesn't actually know anything about function as such what it does know is its class inheritance and so Python goes searching the parent's class __dict__ to try and find the function attribute. It finds this, and as per the descriptor protocol when the attribute is accessed it calls the __get__ function of the underlying function object. What happens now is that the function's __get__ method returns essentially a wrapper object that stores the information to bind the function to the object. This wrapper object is of type types.MethodType and you can see it stores some important attributes in the object im_func which is the function to call, and im_self which is the object who called it. Passing the object through to im_self is how function gets it's first self argument (the calling object). So when you print the value of f.function() you see it report itself as a bound method. So hopefully this illustrates that a bound method is a just a special object that knows how to call an underlying function with context about the object that's calling it. To try and make this a little more concrete, consider the following program:
import types
class Foo():
    def function(self):
        print "hi!"
f = Foo()
# this is a function object
print Foo.__dict__['function']
# this is a method as returned by
#   Foo.__dict__['function'].__get__()
print f.function
# we can check that this is an instance of MethodType
print type(f.function) == types.MethodType
# the im_func field of the MethodType is the underlying function
print f.function.im_func
print Foo.__dict__['function']
# these are the same object
print f.function.im_self
print f
Running this gives output something like
$ python ./foo.py
<function function at 0xb73540d4>
<bound method Foo.function of <__main__.Foo instance at 0xb736960c>>
True
<function function at 0xb73540d4>
<function function at 0xb73540d4>
<__main__.Foo instance at 0xb72c060c>
<__main__.Foo instance at 0xb72c060c>
To pull it apart; we can see that Foo.__dict__['function'] is a function object, but then f.function is a bound method. The bound method's im_func is the underlying function object, and the im_self is the object f: thus im_func(im_self) is calling function with the correct object as the first argument self. So the main point is to kind of shift thinking about a function as some particular intrinsic part of a class, but rather as a separate object abstracted from the class that gets bound into an instance as required. The class is in some ways a template and namespacing tool to allow you to find the right function objects; it doesn't actually implement the functions as such. There is plenty more information if you search for "descriptor protocol" and Python binding rules and lots of advanced tricks you can play. But hopefully this is a useful introduction to get an initial handle on what's going on!

Russell Coker: SE Linux Status in Debian 2012-03

I have just finished updating the user-space SE Linux code in Debian/Unstable to the version released on 2012-02-16. There were some changes to the build system from upstream which combined with the new Debian multi-arch support involved a fair bit of work for me. While I was at it I converted more of them to the new Quilt format to make it easier to send patches upstream. In the past I have been a bit slack about sending patches upstream, my aim for the next upstream release of user-space is to have at least half of my patches included upstream this will make things easier for everyone. Recently Mika Pfl ger and Laurent Bigonville have started work on Debian SE Linux, they have done some good work converting the refpolicy source (which is used to build selinux-policy-default) to Quilt. Now it will be a lot easier to send policy patches upstream and porting them to newer versions of the upstream refpolicy. Now the next significant thing that I want to do is to get systemd working correctly with SE Linux. But first I have to get it working correctly wit cryptsetup. Related posts:
  1. SE Linux Status in Debian 2011-10 Debian/Unstable Development deb http://www.coker.com.au wheezy selinux The above APT sources.list...
  2. SE Linux Status in Debian 2012-01 Since my last SE Linux in Debian status report [1]...
  3. Debian SE Linux Status At the moment I ve got more time to work on...

5 March 2012

Vincent Bernat: Asynchronicity & Net-SNMP AgentX protocol

One of the best way to add SNMP support to an application is the use of AgentX protocol: an application can act as a subagent and handle requests from a master agent using AgentX protocol. The most widespread implementation for such a task is the one from Net-SNMP. TL;DR: An application asynchronously leveraging Net-SNMP AgentX implementation has many opportunities to get stuck when the master agent becomes temporarily unavailable. A mostly efficient workaround is to dedicate a master agent for AgentX and delegate MIB handling to a subagent.

Pinging the master agent Net-SNMP uses an event-based model. It also provides some synchronous functions but they are built on top of the asynchronous ones. It is shipped with its own event loop. Therefore, Net-SNMP AgentX protocol implementation seems a fine choice to add SNMP support to an event-based program, for example in Keepalived, a VRRP daemon and a monitoring daemon for LVS clusters. Unfortunately, an AgentX subagent is expected to check, at a regular interval, if the master agent is alive . This is handled automatically with Net-SNMP implementation as long as you call run_alarms() function which will call agentx_check_session():
/*
 * check a session validity for connectivity to the master agent.  If
 * not functioning, close and start attempts to reopen the session 
 */
void
agentx_check_session(unsigned int clientreg, void *clientarg)
 
    /* [...] */
    DEBUGMSGTL(("agentx/subagent", "checking status of session %p\n", ss));
    if (!agentx_send_ping(ss))  
        snmp_log(LOG_WARNING,
                 "AgentX master agent failed to respond to ping.  Attempting to re-register.\n");
        /*
         * master agent disappeared?  Try and re-register.
         * close first, just to be sure .
         */
        agentx_unregister_callbacks(ss);
        agentx_close_session(ss, AGENTX_CLOSE_TIMEOUT);
        /* [...] */
        if (main_session != NULL)  
            /* [...] */
            main_session = NULL;
            agentx_reopen_session(0, NULL);
         
        else  
            snmp_close(main_session);
            main_session = NULL;
         
      else  
        DEBUGMSGTL(("agentx/subagent", "session %p responded to ping\n",
                    ss));
     
 
This piece of code will end up calling (directly or indirectly) the following functions1 which are using agentx_synch_response():
  • agentx_send_ping()
  • agentx_open_session()
  • agentx_reopen_session()
  • agentx_close_session()
  • agentx_register()
  • agentx_unregister()
  • agentx_register_index()
  • agentx_unregister_index()
  • agentx_add_agentcaps()
  • agentx_remove_agentcaps()
agentx_synch_response() will synchronously wait for an answer from the master agent. Your event-based program will just sit here doing nothing while this happens. For example, keepalived may be unable to send VRRP probes resulting in some serious havoc in your cluster. Functions prefixed by agentx_ are not part of NetSNMP API and could therefore be rewritten to do their work asynchronously. Unfortunately, talk is cheap. Moreover, there exists other deep paths that will block a subagent. For example, the low-level connection to the master agent (using TCP or a Unix socket, but not with UDP) will use a blocking connect() call. It seems quite difficult to fix this. Here are some (partial) workarounds:
  • Call agentx_check_session() less often. The default interval is 15 seconds. Disabling it completely or using a very high value is not advised because this function is the only way to reestablish a connection to the master agent when it is, for example, restarted. Here is how to change the delay to 120 seconds:
    netsnmp_ds_set_int(NETSNMP_DS_APPLICATION_ID,
                       NETSNMP_DS_AGENT_AGENTX_PING_INTERVAL, 120);
    
  • Don t call agentx_check_session() if we know the master agent is alive. For example, if it just sent us a new request, no need to ping it. I have written a patch implementing such a workaround. The ABI is left untouched, therefore, you only need to recompile Net-SNMP libraries.
  • Use a lower timeout and no retry. By default, the timeout for synchronous requests is 1 second and a failed request will be retried 5 times. Because several synchronous functions are used in a row, you may be trapped for as long as 30 seconds with those settings. Here is how2 to disable the retry mechanism and to set the timeout to a very low value:
    static int
    snmp_setup_session_cb(int majorID, int minorID,
                  void *serverarg, void *clientarg)
     
        netsnmp_session *sess = serverarg;
        sess->timeout = ONE_SEC / 3;
        sess->retries = 0;
        return 0;
     
    void some_init_function()
     
        /* [...] */
        snmp_register_callback(SNMP_CALLBACK_LIBRARY,
           SNMP_CALLBACK_SESSION_INIT,
           snmp_setup_session_cb, NULL);
        /* [...] */
     
    

Unresponsive master agent Why is the master agent unresponsive? snmpd is also an event-based program and, trust me, there are a lot of places where it will become unresponsive. Here are two examples:
  • When using the pass or the pass_persist directive, Net-SNMP will block until the delegated command outputs a complete line.
    /* var_extensible_pass() in agent/mibgroup/ucd-snmp/pass.c */
    /*
     * valid call.  Exec and get output 
     */
    if ((fd = get_exec_output(passthru)) != -1)  
        file = fdopen(fd, "r");
        if (fgets(buf, sizeof(buf), file) == NULL)  
            fclose(file);
            wait_on_exec(passthru);
            /* [...] */
            continue;
         
        /* [...] */
        fclose(file);
        wait_on_exec(passthru);
        /* [...] */
     
    
  • DISMAN-PING-MIB implementation will just hang while pinging the remote host. If the host is down and you have requested 5 probes with a timeout of 1 second, snmpd will be unresponsive for 5 seconds. A similar problem exists with DISMAN-TRACEROUTE-MIB.

Minimal master agent More and more code is running in the main agent. You cannot expect it to not block. But we could add a minimal master agent to the mix: it will handle only a few MIB modules by itself while most MIB modules will be delagated to a complete agent running as a subagent. Your own subagent will connect to the minimal master agent. Schema showing the use of a minimal master agent It is quite easy to turn snmpd into a subagent: just use -X flag. We should also disable unwanted MIB modules for each agent with -I flag. For the minimal master agent, we only enable some modules. All other modules3 will be served by snmpd acting as a subagent.
$ MODS="snmp_mib,sysORTable,usmConf,usmStats,usmUser,vacm_conf,vacm_context,vacm_vars"
$ snmpd -Lsd -Lf /dev/null -u snmp -g snmp \
>   -C -c /etc/snmp/snmpd.master.conf \
>   -p /var/run/snmpd.master.pid \
>   -I $MODS
$ snmpd -Lsd -Lf /dev/null -u snmp -g snmp \
>   -p /var/run/snmpd.pid -X \
>   -I -$MODS
Access control configuration should be placed in /etc/snmp/snmpd.master.conf, as well as master agentx directive. Other directives (like pass, pass_persist, load, sysname, ) in /etc/snmp/snmpd.conf. The minimal master agent is very unlikely to block and while the ground cause is still here, it is now very much mitigated. You may extend this solution by splicing snmpd into several subagents: one handling safe modules, one handling pass and pass_persist stuff and a last one for unsafe modules (like DISMAN-PING-MIB).

  1. agentx_reopen_session() will call agentx_close_session() and agentx_open_session(). agentx_register() and the following functions will be called by those two functions.
  2. NETSNMP_DS_AGENT_AGENTX_TIMEOUT and NETSNMP_DS_AGENT_AGENTX_RETRIES variables are only used by the master agent, not by a subagent.
  3. Use snmpd -Dmib_init -H to get the list of modules. You may want to add smux module to the master agent.

Russell Coker: LUV Hardware Library

What is it? Last month I started what I am calling the LUV Hardware Library. It s a briefcase full of computer parts that are free to LUV members which I plan to bring to all LUV meetings [1]. The issue is that there is a lot of hardware which has no great value to people and is often excess to requirements but which is still somewhat expensive or difficult to obtain and thus transferring it from the people who don t need it to people who do provides a significant benefit to the recipient but no real cost to the donor. Currently my briefcase has a range of different types of DIMM, some PCI cards, some assorted cables, a laptop SATA disk, and a bunch of other random things. Most of the stuff has been in my spare parts pile for a year or two, but some of it has been donated by other people. What we Need The next thing we need is more packaging, anti-static bags for RAM and PCI cards, sealable bags for screws, and some way of storing motherboard batteries. In terms of hardware that can be donated one thing that would be really useful is SATA disks. I m sure that there are lots of people who have upgraded the storage of their computer and kept the old drive with no real chance of using it. But there are lots of people who need such disks, for example I ve been giving away a bunch of P4 systems which lack disks and are not cabled for IDE disks a supply of SATA disks would make them a lot more usable. Also any other random electronic stuff is of interest, including non-Linux things such as mobile phones (but only if not network locked). If you have something that s big or heavy to donate then contact me via email first. Other Options Computerbank does great work in rebuilding old PCs and selling them cheaply to worthy people [2], but most of the spare hardware I get is below the minimum specs that they will accept. I m not planning to compete with Computerbank in any way, I just want to provide a useful service to LUV members who want to upgrade their PC for free. I encourage other people to do the same at other LUG meetings! Related posts:
  1. Donating old Hardware On a recent visit to my local e-waste disposal place...
  2. Giving Away Hardware For the last few years I have been actively seeking...
  3. Shelf-life of Hardware Recently I ve been having some problems with hardware dying. Having...

John Sullivan: Plus One

In mailing list discussions, I've noticed that the practice of replying "+1" to messages with which one agrees has become very widespread. I think this is interesting for a lot of reasons, such as what it shows about habits picked up in communicating via social network sites that grew out of e-mail percolating back to e-mail. We seem to have really grown to like this idea of "voting" for people's messages, and if there's no "Like" or "+1" button on their message, we'll make our own. (Likewise, apparently a person's name is no longer sufficient to indicate that one is addressing them in an e-mail -- am I right, @dear_reader?) The problem for me is that I have my venerable awesome e-mail client Gnus set to hide quoted text in messages by default with (add-hook 'gnus-article-prepare-hook (lambda () (gnus-article-hide-citation 1))), and it thinks that "+" is the start of a quoted line -- so it hides the "+1". While there's a certain poetic justice to that, it confuses me because it makes the message look empty, and someone could do something like "+1 I'm pregnant too", causing me to miss some very important news. The fix for this is to change the regexp pattern Gnus uses to decide if a line is a message quote or not. gnus-message-cite-prefix-regexp is built using message-cite-prefix-regexp, so I changed the value of the latter, removing the literal "+". I also found while investigating this that I already configure that variable in order to add "#>" which someone sometime during my 9 years of using Gnus must have used. The net result is (setq message-cite-prefix-regexp "\\([ ]*[-_.#[:word:]]+>+\\ [ ]*[]> ]\\)+"). (Note that some of those whitespace characters are tabs, which probably won't display properly here, but you can compare to the default value to see what I actually changed). +1?

4 March 2012

Eddy Petri&#537;or: HOWTO: Fix: Baobab opens directories in Totem/VLC (and some Xfce4 related things)

If you ever used filelight or baobab you probably know how useful they are. If you didn't, then you missed a lot on how you can spot (and fix) where your disk space is wasted.

With my recent attempt to upgrade to GNOME 3 which, because of its innate property to be useless and counter-productive, actually made me to use Xfce4 with a mix of GNOME aplications (since Xfce lacks a few functionalities here and there), I ran into all sorts of problems.

As a side note, Xfce4 is quite decent, but if you like some icons on your panels to be left aligned and some right aligned, you should know that you can add a Separator item to the panel, right click on it -> Properties and tick the Expand* check box. And if you also set the Transparent style, it will look nice, too.

Back to the topic. With my mix of Xfce and Gnome apps, I configured my top panel to contain a Free Space Checker for my /home file system and today it alerted me that I was low on disk space, so I started baobab to check what I can clean up.

When I found a possible suspect, I wanted to open the directory with a file browser, but, instead, Totem was started and started trying to queue all the files in the offending directory. The problem is that one way or another, Totem (or VLC) was configured to be the default handler for directories instead of the file manager.

The solution is simple, open with an editor the file ~/.local/share/applications/mimeapps.list and search for the line starting with inode/directory= and you'll see something like:

inode/directory=nautilus-folder-handler.desktop;baobab-usercreated.desktop;vlc-usercustom.desktop;


Remove the offending part, vlc-usercustom.desktop;, save the file and try again to open that directory from baobab. If you are double-lucky :P and now it opens with Totem, you will have to remove a reference to a "totem-usercustom.desktop;" or something of that sort. Now, on my system, that line looks like this:

inode/directory=nautilus-folder-handler.desktop;baobab-usercreated.desktop;


And now it works as expected**

* I suppose it's called like that in English, I have my desktop in Romanian
** Except that I would like it to start my desktop preferred file manager, not Nautilus, but that's another issue.

Craig Small: JFFNMS 0.9.2 Released

JFFNMS version 0.9.2 was released today both as an upstream tar.gz file and a new debian package. This version fixes some bugs including making sure it works with PHP5.4. The biggest change in PHP 5.4 is that you can no longer call by reference. Previously you could call a function like myfunc(&blah); which would send a pointer to blah and not the item itself. Now the function definition needs to define what it wants rather than change it each time.

3 March 2012

Sylvain Beucler: Cross-compiling with CMake

I'm working on porting FreeGLUT for Android, and FreeGLUT is switching to CMake. I'm more familiar with the GNU Autotools myself, and I'm used to cross-compile with a simple:
./configure --host=arm-linux-androideabi
Can we do that with CMake? Well, it's not well-documented, but yes :) Let's see how to do that for Android (smartphones) and MinGW (.exe for windows).
The principle is: Here's a simple toolchain script:
SET(CMAKE_SYSTEM_NAME Linux)  # Tell CMake we're cross-compiling
include(CMakeForceCompiler)
# Prefix detection only works with compiler id "GNU"
# CMake will look for prefixed g++, cpp, ld, etc. automatically
CMAKE_FORCE_C_COMPILER(arm-linux-androideabi-gcc GNU)
SET(ANDROID TRUE)
Note: SET(ANDROID TRUE) is not mandatory for CMake, it's just used to detect an Android build in the particular case of FreeGLUT development. Here's how to invoke it:
PATH=/usr/src/ndk-standalone-9/bin:$PATH
cd /usr/src/freeglut-3.0.0/
mkdir cross-android/ && cd cross-android/
cmake \
  -D CMAKE_TOOLCHAIN_FILE=../android_toolchain.cmake \
  -D CMAKE_INSTALL_PREFIX=/freeglut \
  ..
make -j4
make install DESTDIR=$(pwd)

Now here's a slightly more complex toolchain for MinGW. The main issue with MinGW is that there are at least 4 possible prefixes depending on which version and which distro you're using. So I ask the user to specify it with GNU_HOST:
SET(CMAKE_SYSTEM_NAME Windows)
include(CMakeForceCompiler)
IF("$ GNU_HOST " STREQUAL "")
    SET(GNU_HOST i586-mingw32msvc)
ENDIF()
# Prefix detection only works with compiler id "GNU"
CMAKE_FORCE_C_COMPILER($ GNU_HOST -gcc GNU)
# CMake doesn't automatically look for prefixed 'windres', do it manually:
SET(CMAKE_RC_COMPILER $ GNU_HOST -windres)
I wanted to print an error when GNU_HOST is not specified, but the toolchain script is loaded multiple times, and the command-line variables are not always set (CMake is not super-clean in that regard). So I defined a default, that works. Sample invocation:
  apt-get install mingw-w64
  mkdir cross-woe/ && cd cross-woe/
  cmake \
    -D GNU_HOST=x86_64-w64-mingw32 \
    -D CMAKE_TOOLCHAIN_FILE=../mingw_cross_toolchain.cmake \
    -D CMAKE_INSTALL_PREFIX=/freeglut \
    ..
  make -j4
  make install DESTDIR=$(pwd)

I used this technique to cross-compile my projects, or other projects that use CMake. mingw-cross-env uses this technique (abeilt with explicit definition of all compiler tools) to provide pre-built windows libraries without altering the existing CMake scripts. Do you people have tips or a better technique to cross-compile with CMake?

Christian Perrier: 2012 update 13 for Debian Installer localization

Status for D-I level 1 (core D-I files): Status for D-I level 2 (packages that have localized material that may appear during default installs, such as iso-codes, tasksel, etc.): Status for D-I level 3 (packages that have localized material that may appear during non-default installs, such as win32-loader) Full 100% completeness (hall of fame) for 13 languages: Bulgarian, Czech, German, French, Indonesian, Italian, Kazakh, Dutch, Polish, Portuguese, Russian, Slovak, Turkish

1 March 2012

Mike Hommey: Introducing faulty.lib

TL;DR link: faulty.lib is on github. Android applications are essentially zip archives, with the APK extension instead of ZIP. While most of Android is java, and java classes are usually loaded from a ZIP archive (usually with the JAR extension), Android applications using native code need to have native libraries on the file system. These native libraries are found under /data/data/$appid/lib, where $appid is the package name, as defined in the AndroidManifest.xml file. So, when Android installs an application, it puts that APK file under /data/app. Then, if the APK contains native libraries under a lib/$ABI subdirectory (where $ABI is armeabi, armeabi-v7a or x86), it also decompresses the files and places them under /data/data/$appid/lib. This means native libraries are actually stored twice on internal flash: once compressed and once decompressed. This is why Chrome for Android takes almost 50MB of internal flash space after installation. Firefox for Android used to have that problem, and we decided we should stop doing that. Michael Wu thus implemented a custom dynamic linker, which would load most of Firefox libraries directly off the APK. This involves decompressing the zipped data somewhere in memory, and doing ld.so s job to make the library usable (please note that on Android, ld.so is actually named linker). There were initially circumstances under which we would decompress into a file and reuse it the next time Firefox starts, but we subsequently removed that possibility (except for debugging purpose) because it ended up being slower than decompressing each time (thanks to internal flash being so slow). Anyways, in order to do ld.so s job, our custom linker was directly derived from Android s system linker, with many tweaks. This custom linker has done its job quite well for some time, now, but has been recently replaced, see further below. Considering Firefox can t do anything useful involving Gecko until its libraries are loaded, in practice, this means Firefox can t display a web page faster than completely decompressing the libraries. Or can it? Just don t sit down cause I ve moved your chair We know that a lot of code and data is not used during Firefox startup. Based on that knowledge, we started working on only loading the necessary bits. The core of the idea is, when a library is requested to be loaded, to reserve anonymous memory for its decompressed size, and that s all. That memory is protected such that any access to it triggers a segmentation fault. When a segmentation fault happens, the required bits are decompressed, and execution is resumed where it was before the segmentation fault. The original prototype was decompressing from a normal zip deflated stream, which means it was impossible to seek in it. So, if an access was made at the end of the library, it was necessary to decompress the whole library. With some nasty binary reordering, and some difficulty, it was possible to avoid accessing the end of the library, but the approach is very much fragile. It only takes an unexpected code path to make things much slower than they should be. Consequently, for the past months, I ve been working on improving the original idea and, with some assistance from Julian Seward, implemented the scheme with seekable compressed streams. Instead of letting the zip archive creation tool deflate libraries, we store specially crafted files. Essentially, files are cut in small chunks, and each chunk is compressed individually. This means a less efficient compression, but it also means random access to chunks is possible. However, instead of stacking on top of our existing custom linker, I started over, from the ground up. First, because it needed a serious clean up (a good part of linker.c is leftovers from the Android linker that we don t use, and APKOpen.cpp is a messy mix of JNI stubs, library decompression handling (which in itself was also a mess) and Gecko initialization code) and most importantly, because it relied on some Android system linker internals and thus required binary compatibility with the system linker. Which, according to Google engineers that contacted us a few months ago, was going to break in what we now know will be called Android Jelly Bean. The benefit of the clean slate approach is that the new code is not tied to Gecko at all and was designed to work on Android as well as on desktop Linux systems (which made debugging much much easier). We re thus releasing the code as a separate project: faulty.lib. It is licensed under the Mozilla Public License version 2.0. Please feel free to try, contribute, and/or fork it. This dynamic linker is not meant to completely follow standard ELF rules (most notably for symbol resolution), and as a result does some assumptions. It s also still work in progress, with some obvious optimizations pending (like, avoiding to resolve the same symbols again and again during relocations), and some features missing (for example, symbol versioning). The next blog post will give some information about how to build Firefox for Android to benefit from on-demand decompression. I will also detail a few of the tricks involved in this dynamic linker in subsequent blog posts.

Rapha&#235;l Hertzog: My Debian Activities in February 2012

This is my monthly summary of my Debian related activities. If you re among the people who made a donation to support my work (384.14 , thanks everybody!), then you can learn how I spent your money. Otherwise it s just an interesting status update on my various projects. Dpkg and multiarch The month started with a decision of the technical committee which allowed me to proceed with an upload of a multiarch dpkg even if Guillem had not yet finished his review (and related changes). Given this decision, Guillem made the experimental upload himself. I announced the availability of this test version and invited people to test it. This lead to new discussions on debian-devel. We learned in those discussions that Guillem changed his mind about the possibility of sharing (identical) files between multiple Multi-Arch: same packages, and that he dropped that feature. But if this point of the multiarch design had been reverted, it would mean that we had to update again all library packages which had already been updated for multi-arch. The discussions mostly stalled at this point with a final note of Guillem explaining that there was a tension between convenience and doing the right things every time that we discuss far-reaching changes. After a few weeks (and a helpful summary from Russ Allbery), Guillem said that he remained unconvinced but that he put back the feature. He also announced that he s close to having completed the work and that he would push the remaining parts of the multiarch branch to master this week (with the 1.16.2 upload planned next week). That s it for the summary. Obviously I participated in the discussions but I didn t do much besides this I have a mandate to upload a multiarch dpkg to sid but I did not want to make use of it while those discussions remained pretty unconclusive. Also Guillem made it pretty clear that the multiarch implementation was buggy , not right and not finished and that he had reworked code fixing at least some of the issues since he never shared that work in progress, I also had no way to help even just by reviewing what he s doing. We also got a few multiarch bug reports, but I couldn t care to get them fixed since Guillem clearly held a lock on the codebase having done many private changes it s not quite like this that I expect to collaborate on a free software project but life is full of surprises! I ll be relieved once this story is over. In the mean time, I have added one new thing on my TODO list since I made a proposal to handle bin-nmu changelogs and it s something that could also fix #440094. Misc dpkg stuff After a discussion with Guillem, we agreed that copyright notices should only appear in the sources and not in manual pages or --version output, both of which are translated and cause useless work to translators when updated. Guillem already had some code to do it for --version strings, and I took care of the changes for the manual pages. I merged some minor documentation updates, fixed a bug with a missing manpage. Later I discovered that some recent changes lead to the loss of all the translated manual pages. I suggested an improvement to dh_installman to fix this (and even prepared a patch). In the end, Guillem opted for another way of installing translated manual pages. Triggered by a discussion on debian-devel, I added a new entry to my TODO list: implementing dpkg-maintscript-helper rm_conffile_if_owner to deal with the case where a conffile is taken over by another package which might (or might not) be installed. Misc packaging At the start of the month, I packaged quilt 0.51. The number of Debian specific patches is slowly getting down. With version 0.51, we dropped 5 patches and introduced a new one. Later in the month I submitted 4 supplementary patches upstream which have been accepted for version 0.60. This new version (just released, I will package it soon) is an important milestone since it s the first version without any C code (Debian had this for a long time but we were carrying an intrusive patch for this). Upstream developer Jean Delvare worked on this and based his work on our patch, but he went further to make it much more efficient. Besides quilt, I also uploaded dh-linktree 0.2 (minor doc update), sql-ledger 2.8.36 (new upstream version), logidee-tools 1.2.12 (minor fixes) and publican 2.8-2 (to fix release critical bug #660795). Debian Consultants The Debian Project Leader is working on federating Debian Companies. As the owner of Freexian SARL, I was highly interested in it since Freexian contributes to Debian, offers support for Debian and has a strategic interest in Debian . There s only one problem, you need to have at least 2 Debian developers on staff but I have no employees (it s me only). I tried to argue that I have already worked with multiple Debian developers (as contractors) when projects were too big for me alone (or when I did not have enough time). Alas this argument was not accepted. Instead, and since our fearless leader is never afraid to propose compromises, he suggested me (and MJ Ray who argued something similar than me) to try to bring life to the Debian Consultants list which (in his mind) would be more appropriate for one-man companies like mine. I accepted to help animate the list, and on his side, he s going to promote both the Debian Companies and the Debian Consultants lists. In any case, the list has seen some traffic lately and you re encouraged to join if you re a freelancer offering services around Debian. The most promising thing is that James Bromberger offered to implement a real database of consultants instead of the current static page. Book update We made quite some progress this month. There s only one chapter left to translate. I thus decided to start with proofreading. I made a call for volunteers and I submitted one (different) chapter to 5 proofreaders. The liberation campaign made a nice leap forwards thanks to good coverage on barrapunto.com. We have reached 80% while we were only at 72% at the start of the month (thanks to the 113 new supporters!). There s thus less than 5000 EUR to raise before the book gets published under a free license. Looking at the progression in the past months, this is unlikely to be completed on time for the release of the book in April. It would be nice though so please share the news around you. Speaking of the book s release, I m slowly preparing it. Translating docbook files is not enough, I must be able to generate HTML, ePub and PDF versions of the book. I m using Publican for most formats, but for the PDF version Publican is moving away of fop and the replacement (webkit-based) is far from being satisfactory to generate a book ready for print. So I plan to use dblatex and get Publican to support dblatex as a backend. I have hired Beno t Guillon, the upstream author of dblatex, to fix some annoying bugs and to improve it to suit my needs for the book (some results are already in the upstream CVS repository). I m also working with a professional book designer to get a nice design. I have also started to look for a Python Django developer to build the website that I will use to commercialize the book. The website will have a larger goal than just this though ( helping to fund free software developers ) but in free software it s always good to start with your own case. :-) Hopefully everything will be ready in April. I m working hard to meet that deadline (you might have noticed that my blog has been relatively quiet in the last month ). Thanks See you next month for a new summary of my activities.

No comment Liked this article? Click here. My blog is Flattr-enabled.

29 February 2012

Amaya Rodrigo: Why "Colussion" is brilliantly named

Collusion is "an experimental add-on for Firefox and allows you to see all the third parties that are tracking your movements across the Web."
If you are not paying for something... you, the customer, are the product being sold.
-- Andrew Lewis , Mozilla.
I am very eager to see how this evolves given that things like Adsense, no less, belong to Google, and Google (i believe, and please correct me if I am wrong), is one of Mozilla's biggest, erm, how to put it, "sponsors". Will Chrome get a similar feature? I am very veruy curious. This might be the biggest White Rabbit I've seen in a long time.

Joey Hess: leap day

This leap day saw me driving along the river on a rainy, with 4 chickens in the car's trunk, and 3 terabytes of disk (and a half a bale of straw) in the back seat. I may have not been blogging much lately about life, because these situations can be hard to explain. (Or because "joined the Debian haskell team and spent two days working on rebuilds for the ghc 7.4 transition" is not thrilling reading.)
hens in a car
The Light Sussex chickens are my sister's spare flock, which are "too tame". They're now cozily installed into a coop we built last weekend. In return I gave her a 6 foot long APC power strip, which had been mounted on the wall of my office. I'm preparing my house in town to be rented, and have little need for two dozen power outlets here in solar power land. Google <3s Your Work Indeed, today is a gift economy day all around -- when I arrived at the cabin, there on the porch was an unexpected package from Google. Particularly surprising since I never get deliveries here, since the driveway is a mile long and often seems like it could dead-end into the woods at any moment. The combination of technological wackiness (I also debugged a laptop whose USB hub hangs when a particular trackball is plugged in) and in your face country texture (including coal trains, being stuck behind a tractor, and miles of amazing tree-height mist) made this a memorable day.

Antonio Terceiro: Thesis submitted

Last Friday, after 5 long years, I have finally submitted my PhD thesis. It was quite a relief, more or less as if an elephant was taken off my back. An English title for my thesis would be Structural Complexity Characterization in Software Systems. Here is an abstract:
This thesis proposes a theory to characterize structural complexity in software systems. This theory aims to identify (i) the contribution of several factors to the structural complexity variation and (ii) the effects of structural complexity in software projects. Possible factors in the structural complexity variation include: human factors, such as general experience of the developers and their familiarity with the different parts of the system; factors related to the changes performed on the system, such as size variation and change diffusion; and organizational factors, such as the maturity of the software development process and the communication structure of the project. Effects of structural complexity include higher effort, and consequently higher cost, in software comprehension and maintenance activities. To test the validity of the proposed theory, four empirical studies were performed, mining data from free software project repositories. We analyzed historical data from changes performed in 13 systems from different application domains and written in different programming languages. The results of these studies indicated that all the factors studied influenced the structural complexity variation significantly in at least one of the projects, but different projects were influenced by different sets of factors. The models obtained were capable of describing up to 93% of the structural complexity variation in the projects analyzed. Keywords: Structural Complexity, Software Maintainance, Human factors in Software Engineering, Mining Software Repositories, Theories in Software Engineering, Empirical Software Engineering, Free/Open Source Software Projects.
Those who read Portuguese can check out the actual thesis text as a PDF file. Most of the studies discussed in the thesis are presented in English in papers I have published during the last years. My defense is going to be on March 23rd. If you happen to be at Salvador at that day, please feel cordially invited.

28 February 2012

Paul Wise: Debian/Ubuntu games screenshot party results

The Debian/Ubuntu games team recently organised a half-day screenshots party for creating screenshots for all the games that are available in Debian/Ubuntu. Unfortunately, only 5 people participated in the screenshots party, I had hoped that we would get more folks turn up. Only one of those was not already involved in the games team and that one person found out via our spam on the #debian-mentors IRC channel. This means that most of our promotion of the event was ineffective. Hopefully for future games parties we can improve this, please let us know if you have any ideas about that. We also had 7 other people on the channel during the party lurking, discussing and or working on other things, in particular Ben Armstrong was working on a games live CD. Over the 7 hours of the party, we uploaded 100 screenshots for 40 different packages. Of the packages, 2 are available only in Debian (auralquiz, cutechess), one is available only in Ubuntu (plasma-widget-tictactoe) and the rest are available in both. I think we got quite a bit done for such a small group. Some of us could not resist filing some bugs on packages that had issues preventing or delaying us from taking screenshots. I also suggested to one maintainer that his package (acm4) be removed from Debian, since it is obsoleted by acm and unusable. If you want to play some games, check out goplay, which is a tool for browsing available games using debtags that displays screenshots from the games-thumbnails package, which has just been updated with the screenshots created during the party. goplay needs some more development, if you would like to help out with that, you would be most welcome. If you are interested to see our screenshots, they are all on the screenshots.debian.net website and the packages are auralquiz, cardstories, londonlaw, cube2font, ardentryst, cutechess, pianobooster, emacs-chess, xye, flare, tmw, garden-of-coloured-lights, flight-of-the-amazon-queen, geekcode, glpeces, fofix, unknown-horizons, klickety, polygen, tictactoe, xscreensaver-screensaver-dizzy, goban-original-games, xjokes, mudlet, xabacus, xtux-client, connectagram, crystalspace, pescetti, purity-ng, qstat, slashem, sudoku, minetest, xscavenger, plasma-widget-tictactoe, xbomb, xfrisk, xpilot-ng and xsok.

27 February 2012

Jon Dowland: American Psycho

American Psycho is a novel by Bret Easton Ellis. It's a first person stream-of-consciousness novel about a mid-to-late twenties Wall Street banker during the late-eighties-early-nineties. Patrick Bateman is a vain, self-obsessed pleasure-seeker, surrounded by a gaggle of similarly self-obsessed bankers and socialites. Bateman and his friends idle away their days thinking about what restaurants to secure dinner bookings; which nightclubs to be seen at; where to score cocaine and pick up women. They compete with one another on points of dress etiquette, and the relative degree of luxury demonstrated by their attire. They are all relatively interchangeable: the reader could be excused for mixing the identi-kit bankers up, and indeed they are frequently confused for each other, and other people, throughout the novel. Bateman describes in vivid detail nearly every item of clothing worn by both his male and female associates, describing both their cut and which fashion house they have come from, occasionally passing judgement. The murders are typically described with the same vivid detail as his more public activities. As well as a yuppie, Bateman is a psychotic, and probably a serial killer: there is a small amount of ambiguity, partially due to the frequent confusions of identity between the various male banker characters in the book. It's possible that Bateman has fantasised his killer side, or embellished it. He's clearly an unreliable narrator. It takes a relatively long time before this aspect of his personality emerges. It's also possible that he is a killer, but the people he kills are so interchangeable that nobody notices, or cares. There is a passage towards the end of the book that strays into Hollywood action-thriller levels of absurdity, possibly the only moment in the book where you have difficulty believing that the events could have occurred as described. Bateman tries repeatedly to talk to people about his crimes, but never succeeds in holding people's attention, or being taken seriously. This particular theme reaches a particularly amusing climax. Reportedly Ellis wrote the novel without the violent scenes and later added them in. I can believe that. The novel is carefully constructed to manipulate the reader in a way I've never been aware of before. The everyday and the horrific mesh to make the whole seem very consistent and lucid. There is a small surreal element woven into the story which is so subtle and downplayed that you can't withdraw into the safety of "this is a fantasy". Similar scenes recur over and over: heading to a restaurant. Heading to a club. Sex. Murder. Ellis uses repetition to lull you into a false sense of security before hitting you hard with an about-face, mid-sentence, like a movie jump-cut. This book has the dubious distinction of being the second novel ever to have made me feel physically queezy after reading a passage (and I was reading the other book in a car, which may have been a contributing factor). The book has been criticised for the level of violence meted out against women. Numerically, Bateman describes a roughly even number of male and female murders. However his female victims are dispatched with a shockingly higher level of planned sadism: most of his male victims are opportunistic rather than planned. Some critics theorise that the even male/female victim count is a deliberate pre-emption on the part of the author from criticism for the violence against women. I don't quite see the argument, here: Bateman is not a sympathetic character. He's a monster, and the fact he's a sexist monster is only one unpleasant facet of many. To me, the disdain and hatred shown towards homeless people was a far more prevalent problem, not least because it wasn't limited to murders, or even Bateman: many of the other characters are cruel towards them. What would this book be, without the murder? Quite similar to a number of Ellis' other stories, if their blurbs are anything to go by: explorations of the emptiness of the lives of privileged, upper class contemporary Americans. Is the psychopathy central to the story? I think so. The contrast between the corporate and the murderous is at the core. Would I recommend this novel? I'd hesitate to do so to anyone with a nervous disposition: otherwise, it has left a lasting impression and a desire to discuss it further, which you can take as praise. There's a good critical review at the M/C Journal.

Mark Brown: Yahoo! Groups spam

Yahoo! Groups have for a long time had big problems with allowing people to create obvious spam groups (randomly generated names, no meaningful description and so on) and then add people without any confirmation. This was annoying but at least they have an obvious abuse reporting link in the subscription mails. Sadly rather than improve the situation they appear to have just changed the link to not be a direct link to the reporting form but instead to be a generic help page. The link didn t change, the content just got even less useful. I m strongly tempted to just block all of Yahoo! from the mail systems I control.

Michal &#268;iha&#345;: phpMyAdmin translations status

With first RC for phpMyAdmin 3.5, we're really close to final release, so let's again look at translations status. Here comes fourth round of translation status update. Since last update we have three new translations at 100%: There are few which are really close to 100% and I hope they will get there soon: As you can see, there is still lot of languages missing, this might be your opportunity to contribute to phpMyAdmin.

Filed under: English Phpmyadmin 2 comments Flattr this!

Andrew McMillan: 2012 DAViCal User Survey

Back in 2005 I saw that a new open standard was being developed for calendaring, and I thought it would be a great idea to implement it. Nothing too complicated - just a really simple implementation... And thus was born the "Really Simple CalDAV Store". A few years later, when I got about 90% of the way through implementing the base CalDAV specification, I realised that "Really Simple" and "Calendar" don't actually happen in the universe we inhabit, so after much deliberation the project got renamed to "DAViCal". Now, in 2012, DAViCal is one of the leading CalDAV servers available, and I spend quite a lot of my time helping people who want to use it. Earlier in the year I was looking at the web server logs and noticed that in a four week period (i.e. as far as my logs go back) there had been several thousand unique sources of hits on the URL that DAViCal uses internally to find out what the latest version is when you browse to the '/setup.php' page. This got me wondering how many DAViCal installations there are out there, and how big they might be, and so forth but since DAViCal is free open source software, there isn't a simple way to answer those questions. I thought that it must be time to run a survey of DAViCal users everywhere to try and find out what the scale of the penetration is. How big (and how small) are the installations running DAViCal? What... Well: lets save the questions for the actual place where you can put answers :-) So click here and take the survey right now... you know you want to :-)

26 February 2012

Joachim Breitner: GHC 7.4.1 speeds up arbtt by a factor of 22

More than two years ago I wrote arbtt, a tool that silently records what programs you are using and allows you to do statistics on that data later, based on rules that you define afterwards, hence the name automatic rule based time tracker. I wasn t doing much with it recently (the last release has been half a year ago), but it nevertheless was running on my machine and by now has tracked a total time span of 248 days in 350000 records. Yesterday, I had a use for it again: measuring the time spent creating a certain document with LaTeX. So I added a rule to my categorize.cfg and ran arbtt-stats. I knew that it was not very fast, and that my data set has grown considerably since I last used it. And indeed, it took more than 6 minutes to process the data and spit out the result. Since I m currently working on the GHC 7.4.1 transition in Debian anyways, I decided to check what happens if I compile the code with that version of the Haskell compiler, instead of the previous version 7.0.4. And behold: The whole process took merely 17.3 seconds to complete! At first I did not believe it, but the result was identical, both binaries were built with the same option, i.e. no profiling enabled or anything like that. Wouldn t you also like to have such speed ups for free, just by waiting for someone else to improve their work?

I tried to find out the reason for the speed up and created profiling output from both the old and the new binary. The old binary spends 83% of the time in Categorize.checkRegex, which basically just call Text.Regex.PCRE.Light.match. Since the version of pcre-light is the same in both binaries, I conclude that the Foreign Function Interface that GHC provides to interact with C libraries (libpcre in this case) is much faster now, although I do not find any mention in the release notes. And even if I do not count the 83% time spent in checkRegex, the code from the new compiler is still 2.7 times faster. Thanks, GHC devs, great work!

Flattr this post

Next.

Previous.