One part of my yearly activity at DebConf is exploring the conference neighbourhood for good running spots. So, the first one this year has been an up and down run on "Camino Las Viudas", heading south from Managua. The path is easy to find: when exiting from Hotel Seminole, turn right in the street, cross the large boulevard on continue staright ahead for how many kilometers as you want. It climbs up in the hills all along, so that's quite a though one with about 5% slope all along. I stopped in the middle of nothing, just because I didn't want to run for more than one hour. I'll have to come back and see where this road goes. It seems to be climbing continuously..:-) So, one of my forthcoming plans is going to *the end* of this road, just to see what's there. From OpenStreetMap, it seems to end up in the hills. The run trace can be seen there.

Every now and then I have to kill mutt, which if done at the wrong time, can leave the header cache in an open state. Then attempting to restart mutt will cause it to hang at the "Selecting INBOX" stage. If mutt is compiled with support for the Tokyo Cabinet header cache backend (as it appears to be in Debian), a quick fix for this issue is to repair the header cache database using the Tokyo Cabinet utility tcbmgr from the tokyocabinet-bin package. Simply running tcbmgr optimize -nl ~/.mutt/cache/headers does the trick, and is much nicer of a fix than deleting the header cache.

Jan Dejemyr reported Debian bug #680000 on Monday July 2nd, against update-manager-core. Bug #670000 was reported as of April 22nd: 2 months and 10 days for 10,000 bugs. About a constant bug report rate (2 months and 7 days last time). How will the wheezy freeze affect this? We'll see in two months!

I'm a heavy user of Debian's Bug Tracking System (especially during this period of the year ;)) and I quite like it. And as a Debian developer, I have to check multiple sources when looking at some source package. Namely, I mainly check the following sources:

• Packages Tracking System gives almost all information concerning a source package (maintainer, available versions, number of bugs, etc )
• Buildd Status Pages build status for the package on each architecture
• and sometimes Popularity Contest numbers

All this is great since you can move from the first one to the second one (and vice-versa) just by clicking with your mouse. It becomes less handy when you are looking at any bug report. In fact, the BTS doesn't link back to the previously mentioned services [1]. So, you have to copy/paste the name of the package or start a new tab and write down the address of the desired service. All this is not very nice for a lazy person like me :)

A few days ago, I remembered that I was already using GreaseMonkey and thought : "Hey, it would be nice to hack quickly something using GM to add the missing links". After a few tests, I've come up with a (very simple, yet extremely useful to me) user script that I called Debian BTS extra links [2]. It adds what I've been missing for years. If you are lazy and don't want to install right now, you can check the following pages to see what it adds:

• A regular bug report page
• A modified bug report page

The userscript will add links to PTS, Buildd status pages and Popcon near every pkgreport.cgi?src= link. Simple and useful. I hope you'll enjoy it as much as I do.

Of course, I've submitted a bug (#679282) against the pseudo package bugs.debian.org kindly asking its maintainer to add a link to (for now) the PTS. But I know that they are busy so I'm not expecting a result soon. Fow now, my simple and dumb userscript WFM.

[1] This is partly true since it links back to the PTS on main pages (pages that list all bug reports of a given set of packages).

[2] Only tested with a recent Iceweasel (from experimental) and Chromium (from unstable).

P.S.: If you do wonder, #4300 is the least bug number known to the BTS. Well, or at least, the result of running ls -1 /srv/bugs.debian.org/spool/archive/00/ sort -h head -n1 on bugs.debian.org.

It's that time of the year again. No, not only the Debian biyearly freeze that has just happened, again (YAY!). It's DebConf time again. Yesterday night I arrived in Managua, Nicaragua, for DebConf12. I'm still jetlagged a bit, but the first impression is great. And that impression is also very green, shockingly green. The city is gorgeous: full of trees and plants (well, at least compared to my metropolitan European standards), and the campus of Universidad Centroamericana where the conference is hosted is even more so. Also, it's a great pleasure to have DebConf in an university campus where students are still swarming. My only reason for sadness thus far. While I did bring my traditional Debian kilt at DebConf, I discovered this morning that I forgot my sporran at home
So if you don't see me wearing my kilt at this DebConf often, that's why

This is my monthly summary of my Debian related activities. If you re among the people who made a donation to support my work (168.12 , thanks everybody!), then you can learn how I spent your money. Otherwise it s just an interesting status update on my various projects. Dpkg This month, I resumed my work on dpkg. I concentrated my efforts on some polishing of the 3.0 (quilt) format. With the latest version (1.16.6 which was uploaded to unstable shortly before the freeze), dpkg-source restores the source tree in a clean state after a failed patch application (#652970), doesn t overwrite the patch header from the pre-existing automatic patch, updates automatically debian/source/include-binaries during dpkg-source commit, and supports a new no-unapply-patches option for those who dislike the auto-unapplication at the end of the process when the patches were not applied at the start. I wanted to go further and offer a new feature that could insert the automatic patch at the bottom of the quilt series but I have been short on time to complete this feature. I just managed to factorize all the quilt handling in a dedicated Perl module (Dpkg::Source::Quilt) to have cleaner code in the module handling the source format (Dpkg::Source::Package::V3::quilt). For those who wonder, this feature is meant primarily for the X Strike Force team which maintains packages in Git and are doings lots of upstream cherry-picks (to fix regressions, etc.). But they also use quilt on top of that tree to keep some lasting Debian specific changes. With the 1.0 format, the automatic diff is a bit messy but at least it gets smaller automatically when a new upstream release gets out, there s nothing to clean out. I d like them to be able to use 3.0 (quilt) while keeping their workflow. I m leaning towards allowing --auto-commit=first:cherry-picks that would name the automatic patch cherry-picks and put it in the first position in the quilt series. (Opinions welcome on that feature, BTW) Packaging There s been quite some packaging in this last month before the freeze:

• I packaged CppUTest (a test framework for C/C++), and I wrote an article about it.
• I prepared a stable update of Publican to fix a missing dependency. I also updated the unstable version to include a backport of a fix that some user requested me to include.
• I updated dh-linktree to improve its documentation (following a discussion that happened on debian-devel) and to deal properly with trailing slashes in its input (#673408).
• I sponsored dblatex 0.3.4-1 and ledgersmb 1.3.18-1.
• I updated gnome-shell-timer to a new upstream snapshot that was tagged as compatible with GNOME 3.4 (#6776516).
• I packaged wordpress 3.4 and spent a whole day triaging the old bugs that accumulated. A few days later I developed a new infrastructure to properly manage plugins/themes/language files. The canonical directory where the user is expected to drop his custom plugins/themes is now in /var/lib/wordpress/wp-content/ and the official plugins/themes are installed there with symlinks pointing back to /usr/share/wordpress/wp-content/ where they actually reside.
• I wanted to commit 2 patches for the developers-reference but then I noticed that some translations were complete and were waiting for an upload. So I cleaned the packaging (switch to dh) and I uploaded version 3.4.8 before committing the patches for #678710 and #678712.

While doing all this packaging work, I found 2 possible improvements that I filed as bug reports:

• #676606: debcommit should be able to identify alone that a new release is prepared (when the distribution field of the changelog changes from UNRELEASED to something else).
• #679132: lintian outputs false positives for the tag package-uses-local-diversion when neither local nor package is given on the dpkg-divert command line.

Debian France Booth at Solutions Linux From June 19th to June 21th, I manned the Debian France booth at Solutions Linux together with Carl Chenet, Tanguy Ortolo and other members of the association. We answered lots of questions, sold all t-shirts and umbrellas that Carl imported from Germany and Switzerland (we really need to get our own merchandising stuff produced in France!), got people to join the association. We also presented a printed copy of the Debian Administrator s Handbook and of the corresponding French book. You can see Carl, me and Tanguy on this picture (click on it to see a bigger picture, thanks to S bastien Dubois of Evolix for this one!):
I know lots of people are preparing for Debconf but I decided to not attend this year, the price of the air plane ticket was a bit too hefty for me and it was also in partial conflict with our family vacations. I thought about attending the Libre Software Meeting instead but alas I won t go there either (but Roland Mas will be there!), I have too much work to complete before my own vacation in 2 weeks. Thanks See you next month for a new summary of my activities.

2 comments Liked this article? Click here. My blog is Flattr-enabled.

Hi! As you may know, I'm working on multiarch cross-toolchains as part of the Google summer of Code (see my previous post for details), and although there is still a lot to do, the project is doing fairly well, and a cross-gcc is already installable and working. I'll proceed with detailed instructions on how to install the cross-compiler, along with some explanations on multiarch itself, and how to build cross-toolchains from source. Prerequisites First, you need specific versions apt (>= 0.9.6) and dpkg (>= 1.16.5).
This is because the cross-toolchains packages I'm working on depend on specific cross-arch dependencies, such as libc6:armhf. Support for such dependencies has been added in the aforementioned versions of apt and dpkg. This means you should be running unstable, or at least use dpkg from unstable. Adding a foreign architecture Adding a foreign architecture is straightforward, but I've seen users do overcomplicated, or just plain wrong things. So, let's do this step by step (well, there are only two of them), with some explanations.

1. Tell dpkg to handle an additional architecture by typing dpkg --add-architecture armhf. There isn't much to say. It will only add an architecture to the list of architectures dpkg would accept to handle.
2. Update APT's database. You need to run apt-get update after adding or removing an architecture to get package lists for all wanted architectures and rebuild APT's database, including the implicit cross-arch dependencies/conflicts brought by multiarch.

And that's it for the common case. Now, a word on architecture restrictions: a common misconception about multiarch seems to be that you have to tell APT which architectures you want from the mirrors by modifying your sources.list file(s). This is not true, as, by default, APT gets the list of accepted architectures from dpkg, and gets the correct package lists from the mirrors configured in the sources.list. You only need to use arch restrictions ( deb [arch=arch1,arch2,...] ... ) when you want to get packages from a mirror only for a subset of your accepted architectures. This is very unlikely, and in the general you do not need to modify your sources.list, at all. Getting the cross-compilers from my repository I provide compiled versions of gcc-4.7-arm-linux-gnueabihf for i386 and amd64 in my repository. If you have followed the previous paragraphs, nothing should be easier than adding a deb http://emdebian.org/~thibg/repo/ sid main line to your sources.list and doing the following:

# apt-get update
[...]
# apt-get --no-install-recommends install gcc-4.7-arm-linux-gnueabihf
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following extra packages will be installed:
  binutils-arm-linux-gnueabihf cpp-4.7-arm-linux-gnueabihf gcc-4.7-arm-linux-gnueabihf-base gcc-4.7-base:armhf libc6:armhf libgcc1:armhf
  libgmp10 libgomp1:armhf libmpc2 libmpfr4
Suggested packages:
  binutils-doc gcc-4.7-locales libmudflap0-4.7-dev:armhf gcc-4.7-doc libgcc1-dbg:armhf libgomp1-dbg:armhf libitm1-dbg:armhf
  libquadmath-dbg:armhf libmudflap0-dbg:armhf binutils-gold glibc-doc:armhf locales:armhf
Recommended packages:
  libc6-dev:armhf
The following NEW packages will be installed:
  binutils-arm-linux-gnueabihf cpp-4.7-arm-linux-gnueabihf gcc-4.7-arm-linux-gnueabihf gcc-4.7-arm-linux-gnueabihf-base gcc-4.7-base:armhf
  libc6:armhf libgcc1:armhf libgmp10 libgomp1:armhf libmpc2 libmpfr4
0 upgraded, 11 newly installed, 0 to remove and 0 not upgraded.
Need to get 23.5 MB of archives.
After this operation, 57.2 MB of additional disk space will be used.
Do you want to continue [Y/n]?

I've disabled the recommends since, libc6-dev:armhf would recommend gcc:armhf itself, which is not co-installable with your native gcc if you have it installed. Building the cross-toolchain yourselves If you don't trust the repository above, or simply want to rebuild the toolchain yourself, it is rather simple: get my gcc-4.7 patches and:

$ apt-get source binutils
$ cd binutils-*/
$ TARGET=arm-linux-gnueabihf dpkg-buildpackage -uc -us -b
$ cd ../
# dpkg -i binutils*arm-linux-gnueabihf*.deb
$ apt-get source gcc-4.7
$ cd gcc-4.7/
$ for patch in path/to/the/patches/*.patch; do patch -p1 < $patch ; done
$ GCC_TARGET=armhf debian/rules control
$ GCC_TARGET=armhf dpkg-buildpackage -uc -us -b

Using the cross-compiler It's easy, but there is a trick, and dpkg-buildpackage -aarmhf won't work out of the box yet, as gcc-4.7-arm-linux-gnueabihf provides the arm-linux-gnueabihf-gcc-4.7 executable, but no arm-linux-gnueabihf-gcc symlink. So, for the moment:

$ cat > test.c << __EOF__
#include <stdio.h>
int main(void)
 
  printf("Hello world\\n");
  return 0;
 
__EOF__
$ arm-linux-gnueabihf-gcc-4.7 test.c
$ qemu-arm ./a.out 
Hello world

Conclusion The toolchain is not complete yet (there is some more work to do on binutils and libstdc++, and equivalents to the gcc , cpp , and build-essential packages to provide), but it should still be useful to cross-compile C programs. Multi-arch cross-toolchains for wheezy can definitely be hosted at emdebian, and I hope to clean up the build process so that it could get included in the archive for wheezy+1.

A copy of this post is sent to soc-coordination@lists.alioth.debian.org as well as to debian-bootstrap@lists.mister-muffin.de.

Diary

June 18 Pietro suggests a faster way to generate installation sets for a list of packages. In my case, I need an installation set for every source package in the archive to find out how often a binary package is needed to build a source package. As a result, the speed is doubled in contrast to the original approach.

June 19

• adapt code to work with new dose release 3.0
• remove unneeded parts of code
• add different possibilities to find amount of source packages that need a binary package
• add code to get multiple installation sets using Depsolver_int.solve

June 20

• add ~global_constraints:false to Depsolver.listcheck, Depsolver.trim and Depsolver.edos_install calls
• adapt output graph to limited xdot capabilities

June 21 I formulate an email to the list, reporting of dependency graphs of debhelper, cdbs, pkg-config and libgtk2.0-dev. My current technique gets an installation set for a source package, removes all those that are already installable and adds the others as a dependency of that source package. This dependency will include an installation set of that binary as well minus all packages that are already available. The problem with that approach are dependency cycles created by long dependency chains. Example: src:A needs B needs C needs A. B and C would both be added as a dependency of src:A. B as well as C would also include their installation set which in both cases includes A. So now there are two cycles: src:A->B->A and src:A->C->A. For a real life example, look at the following situation of cdbs and src:sqlite3. It is created because src:sqlite3 needs cdbs needs python-scour needs python needs python2.7 needs libsqlite3-0. Therfor libsqlite3-0 is in the installation set of cdbs, python-scour, python and python2.7. This creates five cycles in the graph even though there is only one. It would be better to reduce the dependencies added to src:sqlite3 to its direct dependency which is cdbs. Package dependencies are disjunctions from which the solver chooses one or the other to build an installation set. To solve the problem above I would need to know which disjunction the solver chose and then only add the direct dependency of a package to the dependency graph.

• improve build_compile_rounds performance
• big overhaul of menu structure
• fix subgraph extraction code

June 22

• do not create a universe if not needed - use hashtables instead
• for sorting packages, generating difference of package sets and otherwise comparing packages, always use CudfAdd.compare
• as a custom list membership function, use List.exists instead of trying List.find
• more speedup for build_compile_rounds
• the number of source packages that can be built does NOT include the cross built packages
• print closure members in graph
• refactor code and move common functions to bootstrapCommon.ml
• add breakcycles.ml for future code to break cycles using staged build dependencies
• use more extlib functionality
• extended package list input format

June 23 After several emails with Pietro I learn about syntactic dependency graphs. To document my progress and prove my efforts I committed the code as commit 6684c13. But this code was soon found out to be unecessary so it will be removed later and just serves as documentation.

June 24 I came up with another (better?) solution to get the chosen disjunctions. It simply uses the calculated installation set to decide for each disjunction which one was taken by the solver. I reported that important step and the open questions involved with it in an email to the list. The problem always was, that an installation set can easily contain more than one package of a disjunction. In this case it is not clear which branch was chosen by the solver. I found, that in Ubuntu Natty there are only 6 such packages and for each of them the situation can be solved. It can be solved because in all of those cases it is that either one package of a disjunction provides the other or that both packages depend upon each other, which means that both have to be included.

June 27

• use installation set to flatten build dependencies of source packages
• refactor code and move common functions to bootstrapCommon.ml

June 25 I have to have an algorithm that finds all circuits in a given graph. This is necessary so that:

1. cycles can be enumerated for huge dependency graphs where cycles are hard to see
2. cycles can be enumerated to find a cycle that can be broken using staged build dependencies

It seems that Johnson's algorithm is the best way to do this complexity wise and Pietro already blogged about the problem together with an implementation of the algorithm in ocaml. Unfortunately it turns out that his code doesnt implement the algorithm correctly and hence misses out on some cycles. The fix seems not to be too trivial so I'm still investigating it.

June 28

• add crosseverything.ml to obtain a list of source packages that, if cross compiled, would make the whole archive available

Results While the first week was productive as usual, I had to work some time on a University project during the second week as well as attend a family meeting. I will catch up with the lost time over the course of the next week.

dose3 Using dose 3.0 (which fixes a bug about essential packages) the output of the algorithms is now likely less wrong then before.

performance Performance was improved in the generation of installation sets as well as in the code that tries out how many packages can be built in multiple rounds. This was achieved by more caching, less unnecessary operations in looping constructs, replacing lists with hashtables, not creating universes where not necessary.

user interface The main program, basenocycles.ml now has a much better menu structure.

input format The programs take two package file inputs. The list of source packages that has to be cross built for a minimal build system and the list of source packages that was chosen to be cross compiled in addition to that. Both files list one source package per line and now allow comments.

refactoring As more and more scripts are added, more and more functionality is moved to bootstrapCommon.ml which makes each script much cleaner.

what to test for cross building As discussed in the "Future" section of the last report, I now automated the process of finding out which packages, if they were cross compiled, would make the whole archive available because they break all cycles and allow native compilation of the rest. The outcome: to build 3333 out of 3339 packages in natty natively, at most 186 source packages must be cross compiled. The other six cannot be compiled because of version mismatches in the Natty Sources.bz2 and Packages.bz2. The code can be run from crosseverything.ml.

limit source dependencies to direct dependencies Reducing the dependencies of source packages from their full installation set to their direct dependencies by finding out which disjunction of their dependency list were taken, greatly simplifies the dependency graphs. The dependency graph created for libgtk2.0-dev could be reduced from 491 to 247 vertices for a depth of three. For cdbs it is now clearly visible that cdbs depends on libsqlite3-0 which builds from src:sqlite3 which depends on cdbs. Before: After: For pkg-config the graph also has been reduced to the one single cycle that matters: src:pkg-config needs libglib2.0-dev which depends on pkg-config which builds from src:pkg-config. Before: After:

Future I will prepare content for wookey's debconf talk on crossbuilding and bootstrapping. As this will include directions how to use the current code, I will kill two birds with one stone and write some proper documentation for my current source. The following two lists will be displayed after a dependency graph is calculated and reduced to its scc:

• those source packages that have the least build dependencies not fulfilled. Those might be candidates for easy staged build dependencies. Since the source package is part of the scc, it will definitely be involved in some cycle somewhere.
• those binary packages that most source packages depend upon. Those could be candidates for cross compilation as it might be easier to cross compile the source package than using staged build dependencies.

Patrick managed to cross build packages with sbuild now. So the list of packages that crosseverything.ml produces can now be checked efficiently for cross buildability. With this list, potentially more cycles can be broken out of the box. A feature will be added that allows the user to remove all packages from a dependency graph that can be cross compiled without any additional effort. Version mismatches between source and binary packages in Sources.bz2 and Packages.bz2 respectively in Ubuntu make the scripts fail and/or produce wrong results. Debian (even Sid) doesnt have this problem so I should find out where to report this problem to Ubuntu. I need to write a working version of Johnson's algorithm because much functionality depends upon it. I have the option to improve Pietro's version or write one from scratch. Writing one from scratch might be easier as I have Pietro's code as template as well as a Java implementation of Johnson's algorithm which seems to work. The following functionalities need working cycle enumeration:

• given source packages with staged build dependencies, an enumeration of cycles is needed to find out which cycles can be broken by building packages staged. It makes less sense to blindly build a package stage and then check if this makes more packages available.
• display cycles of a dependency graph to the user. After obtaining all cycles in the graph it makes sense to sort them by their length. The user would then investigate the situation of the smallest cycles first. This makes sense because breaking small cycles can potentially break bigger cycles. Since in the end, all cycles have to be eliminated anyway, it makes sense for the user to first tackle the small ones.
• display the feedback arc set to the user. The packages in the feedback arc set might be very good candidates for reduced build dependencies or cross compilation.

For the last 10 years, I've been working on many Free Software projects. From Debian to OpenStack, through awesome, Emacs, XCB and many more. This obviously allowed me to enhance my technical skills, but it also taught me about Free Software and Open Source development processes, and how to work with and close to the community. Working for almost 6 years at Easter-eggs taught me how to work in an autonomous manner, how to lead and manage a project. And how to run a company, thanks to the cooperative status of this great one. These are the reasons why I decided to leave my latest job and run my own company to work as a freelance consultant & developer specialized in Free Software, starting today. Therefore, I am now able and available to provide expertise and development on Free Software, including upstream contribution. Especially on projects I already worked on recently, like OpenStack.

Yesterday night, Regina and me arrived to Nicaragua. Ready to greet us, we found quite a good number of good friends. We had a nice pizza+beer dinner at Diana's house, and some of the foreigners among us were distributed among the houses of several locals. This morning, we woke up together with V ctor, Moray and Gaudenz in Norman's brother's beautiful house. We had breakfast with the family, were picked up to go to the hotel that will have the ho(n rr)or to host us all for the following two weeks, and walked to the Universidad Centroamericana (UCA) campus. Contrary to our usual practices... It seems everything is working fine! I mean, I'm sure we will stumble with some unforseen details and what not... But coming on the very first day to the university, to find that food is all sorted out, that we have food tickets (and they are all printed!), that network works (and it's by a fiber connection that was laid out expressly for us), that we have all the hardware I was worried about, that people are arriving and getting accepted at the hotel. I mean, things work! So, I'm quite optimistic this DebCamp will have everything ready to be a success And the DebConf following it as well, of course! If you have not yet arrived - See you soon!

The other day, I found myself confronted with a large number of large files. Which were presented in (gzip-)compressed ascii format---which R reads directly via gzfile() connections---as well as (compressed) numpy files. The numpy can be read very efficiently into Python. We can do the same in R via save() and load(), of course. But the trouble is that you need to read them first. And reading hundreds of megabytes from ascii is slow, no matter which language you use. Concerning R, I poked aound scan(), played with the colClasses argument and looked at the recent LaF package written just for this purpose. And all these solutions were still orders of magnitude slower than reading numpy. Which is no surprise as it is really hard to beat binary formats when you have to parse countless ascii tokens. So the obvious next idea was to read the numpy file in Python, and to write a simple binary format. One helpful feature with this data set was that it contained only regular (rectangular) matrices of floats. So we could just store two integers for the dimensions, followed by the total data in either one large binary blob, or a sequence of column vectors. But one minor trouble was that the Intertubes lead to no easy solution to unpack the numpy format. StackOverflow had plenty of question around this topic converned with, say, how to serialize in language-independent way. But no converters. And nobody local knew how to undo the "pickle" format underlying numpy. But a remote friend did: Laurent, well-known for his Rpy2 package, pointed me towards using the struct module and steered me towards the solution shown below. So a shameless plug: if you need a very experienced Python or R consultant for sciece work, consider his consulting firm. Finally, to round out this post, let's show the simple solution we crafted so that the next guy searching the Intertubes will have an easier. Let us start with a minimal Python program writing numpy data to disk:

#!/usr/bin/env python
#
# simple example for creating numpy data to demonstrate converter
import numpy as np
# simple float array
a = np.arange(15).reshape(3,5) * 1.1
outfile = "/tmp/data.npy"
np.save(outfile, a)

Next, the simple Python converter to create a binary file containing two integers for row and column dimension, followed by row times columns of floats:

#!/usr/bin/python
#
# read a numpy file, and write a simple binary file containing
#   two integers 'n' and 'k' for rows and columns
#   n times k floats with the actual matrix
# which can be read by any application or language that can read binary
 
import struct
import numpy as np
inputfile = "/tmp/data.npy"
outputfile = "/tmp/data.bin"
# load from the file
mat = np.load(inputfile)
# create a binary file
binfile = file(outputfile, 'wb')
# and write out two integers with the row and column dimension
header = struct.pack('2I', mat.shape[0], mat.shape[1])
binfile.write(header)
# then loop over columns and write each
for i in range(mat.shape[1]):
    data = struct.pack('%id' % mat.shape[0], *mat[:,i])
    binfile.write(data)
binfile.close()

Lastly, a quick littler script showing how R can read the data in a handful of lines:

#!/usr/bin/r
infile <- "/tmp/data.bin"
con <- file(infile, "rb")
dim <- readBin(con, "integer", 2)
Mat <- matrix( readBin(con, "numeric", prod(dim)), dim[1], dim[2])
close(con)
print(Mat)

That did the job---and I already used to converter to read a few weeks worth of data for further analysis in R. This obviously isn't the last word on possible solutions as the additional temporary file can be wasteful (unless it forms a cache for data read multiple times). If someone has nice solutions, please don't hold back and contact me. Thanks again to Laurent for the winning suggestion concerning struct, and help in getting the examples shown here to work.

DebCamp starts tomorrow and the local team is relaxing and enjoying some free time with a cup of Debian Coffee. See you soon in Managua! (In case you wonder yes!!! both the Debian Coffee and the DebConf coffee mug will be for sale during DebConf12 in Managua)

I'm writing this from a hotel room near Heathrow airport, where I'll spend the night I don't like having to get up at early-o-clock in order to get to the airport, so I took a flight on the 30th in the comfortable afternoon, rather than one on the 1st in the uncomfortable morning. Of course that means I get to spend the night in London tonight. As it happens, it'll be my first time that I'll do that; I've been in London before, but it's always been when I was in transit changing planes or trains in Heathrow, or when I was on my way (by car) to Steve's last summer. But I've never been to London for London. Tonight, that isn't the case either, but at least this time I'll get to see more of this city than the airport, King's Cross and Waterloo station, the underground, and the highway network. And then tomorrow, again in the comfortable afternoon, I'll be on my way to Managua. I can hardly wait.

Whilst nobody was looking, Mono became the most important framework in the world of games. And I mean that with only a smidge more than my usual levels of hyperbole. It really is fantastically important, as far as game development goes. I blogged a long time ago about its use inside EA s multi-million-selling The Sims 3. And the massive performance gains in Second Life when they switched from their in-house scripting language to Mono are well documented. But I m not talking about those, I m talking about the now, the recent stuff. First up is Unity 3D. Not to be confused with the Ubuntu user interface, Unity 3D is a proprietary game development tool which began life for Mac, and has now managed to become the single most popular game engine amongst smartphone developers. It targets a bunch of new platforms not just Android and Windows and iOS, but in the new 4.0 release, Linux too for the first time. And every Unity 3D game is a Mono game the core of Unity is a fork of Mono, which enables it to have the performance and ease of development which it does. A reasonably large percentage of the cross-platform games on Kickstarter, such as Wasteland 2 and Super Retro Squad, are cross-platform because Unity 3D makes it easy and each of those games is powered by Mono. However, I have better things to do today than gush about a proprietary tool. No, far more important are the Free ways to develop games. So today I m going to talk about MonoGame. MonoGame is an implementation of Microsoft XNA a high level framework for developing in .NET languages, originally designed to allow indie developers access to the Xbox 360. XNA covers many of the bases covered by SDL sound, graphics, input, and so on, but can go onto platforms where SDL isn t an option, such as the aforementioned Xbox 360. And with MonoGame, any game written for XNA is a game which can not only run on Microsoft s approved platforms Windows, Xbox 360, Windows Phone but also on other platforms such as Mac OS X, iPhone, iPad, Android, and Linux. Cross-platform is good. More games are good. And MonoGame enables every Xbox 360 indie developer to throw a Linux port out of the door with a relative minimum of fuss. There are 2,521 games on the Xbox Live Indie Games store (games by individuals or small groups, with no major publisher) and each of those can become a game for Debian and Ubuntu with very little work. Not to mention the games with actual publishers (some portion of the 492 games on XBLA), not to mention the mobile games written for Windows Phone, Android and iOS. All of these developers have an easy route to Linux releases, as long as the effort remains low. And with today s publishing in the Debian archive of MonoGame 2.5.1, the effort is as low as can be. Anyone who purchased the recent Humble Indie Bundle 5, and played Bastion, has seen MonoGame in action Bastion began life as an XNA game for Xbox 360, and the Linux port was made possible only by MonoGame. Bastion s 100+ industry awards should indicate the possibilities open to a creative mind with decent developer tools. MonoGame is Free Software, built on top of other Free Software, with an active upstream development team genuinely responsive to the needs of Linux distributions. It s well worth a look not just for porting, but for new projects too. And I m delighted that it will be available to all users of Debian 7.0, as well as Ubuntu 12.10.

The bug webinterface of the Ultimate Debian Database currently knows about the following release critical bugs:

In Total:	1088
Affecting Wheezy:	697
Wheezy only:	128
Remaining to be fixed in Wheezy:	569

Of these 569 bugs, the following tags are set:

Pending in Wheezy:	21
Patched in Wheezy:	77
Duplicates in Wheezy:	57
Can be fixed in a security Update:	22
Contrib or non-free in Wheezy:	8
Claimed in Wheezy:	1
Delayed in Wheezy:	2
Otherwise fixed in Wheezy:	44

Ignoring all the above (multiple tags possible) 375 bugs need to be fixed by Debian Contributors to get Debian 7.0 Wheezy released. However, with the view of the Release Managers, 603 need to be dealt with for the release to happen. Please see Interpreting the release critical bug statistics for an explanation of the different numbers.

OASIS is a tool to integrate a configure, build and install system in your OCaml project. It helps to create standard entry points in your build system and allows external tools to analyse your project easily. It is the building brick of OASIS-DB, a CPAN for OCaml. This release takes almost 18 months to complete. This is too long and I will talk in another blog post on the way I am trying to improve this right now (esp. using continuous integration). This new release fixes a small bug (1 line) that prevents setup.ml to run with OCaml 4.00. If you have projects that was generated with a former release consider upgrading to OASIS 0.3.0. There are several big new stuff that comes with this release. It now supports Pack: field for libraries which allows to pack your library using -for-pack and so on. We are also compiling .cmxs (dynlink object for native) by default and we publish them in the META file. This feature was implemented in order to get more libraries to provide .cmxs and to help project like Ocsigen to take advantage of this. If you want to get rid of this at configure time, you can use ./configure --override native_dynlink false. We introduce two new default flags: --enable-tests and --disable-docs for configure. These are implicit flags that define if we will run Test sections or compile Document sections. They are especially useful to reduce the number of dependencies, because dependencies of Test will be excluded by default. We recommend to set Build$: flag(tests) to any Library or Executable sections that are only useful for tests. This allows you to really cut down the number of dependencies. The last change I want to introduce is about the old setup-dev subcommand which is now deprecated. It has been replaced by 2 different update schemes. I am pretty excited by this feature which in fact comes from OASIS user (esp. Lwt project). The former scheme was to have a big 'setup.ml' that always call the command oasis to update. This was complex and not very useful. We now have 2 mode: dynamic and weak. dynamic allow you to have a small setup.ml and to keep your VCS history clean but you need to install OASIS. weak need a big setup.ml but only need to call the command oasis if someone change something in _oasis. This mode is targeted to project that wants to be able to checkout from VCS an OCaml project without installing OASIS. The difference generated by weak mode doesn't pollute too much the VCS history because most the time they make sense. For example, you upgrade your package version number in _oasis and it produces a change with 6 lines where the version number changes in every META, setup.ml files and so on. This version has been tested on:

• Linux (Debian Sid and Debian Squeeze with GODI, OCaml 3.12)
• Windows Vista with MS Visual Studio 2008 toolchain, OCaml 3.12
• Mac OS X, OCaml 3.12 from GODI

You can download it here or use your favorite package manager: Debian (UPDATE: pending upload)

$ apt-get install -t experimental oasis

GODI

$ godi_console perform -build apps-oasis

odb.ml

$ ocaml odb.ml oasis

Here is the complete changelog: EXTREMLY IMPORTANT changes (read this)

• Fix bug with scanf %S@\n for ocaml 4.00. We were unfortunetaly using an undocumented tolerance of Scanf in the previous version. You should consider making new release using this version that fixed this.

PACKAGES uploaded to oasis-db will be automatically "derived" before OCaml 4.00 release (i.e. oUnit v1.1.1 will be regenerated with this new version as oUnit v1.1.1~oasis1). PACKAGES not uploaded to oasis-db need to be regenerated. In order not to break 3rd party tools that consider a tarball constant, I recommend to create a new version. Thanks to INRIA OCaml team for synchronizing with us on this point. Major changes:

• Handle the field Pack: true to be able to create packed libraries. It also installs .mli files for documentation into the target directory. The pack option is only supported for OASISFormat: 0.3, you will need to update the version of your _oasis file to match it.

• Introduce --enable-tests to disable tests and docs at oasis level. It seems a very common pattern to have a Flag tests to turn off by default the tests. This is now define as a standard var and you should remove you previous Flag tests but you can continue to use flag(tests) where needed.

You can now have the following example:

     ...
     Executable test_exec
       Install: false
       Build$: flag(tests)
       MainIs: testExec.ml
       BuildDepends: oUnit
     
     Test main
       Command: $test_exec
       TestTools: test_exec
     ...

The Run$: flag(tests) is implicit for the section Test main. The default value is false for tests. If all the executable for test are flagged correctly (Build$: flag(tests)), you'll get rid of the dependency on oUnit. It works the same for documentation, however the default is true. (Closes: #866)

• Allow to define interdependent flags

In order to allow interdependent flags, we transform back lazy values into unit -> string functions. This allows to change a flag value on the command line and to update all the dependent values. (Closes: #827, #938)

• Deprecate the subcommand oasis setup-dev in favor of oasis setup -setup-update none weak dynamic

It defines different ways to manage the auto-update of setup.ml:

• none: this is the default mode, and the one you should use when distributing tarballs. No update are performed at al.
• weak: the update is only triggered when something change in _oasis , we keep all files generated
• dynamic: the content of 'setup.ml' is ultra small (<2kB) and we only keep a small setup.ml, Makefile and configure.

The choice between weak and dynamic depends on your need with regard to VCS and to the presence of oasis. The weak allow to checkout the project from VCS and be able to work on it, without the need of installing 'oasis' as long as you don't change the file _oasis. But it clutter your VCS history with changes to the build system each time you change something in _oasis. The 'dynamic' mode gives you no VCS history pollution but makes mandatory to have installed oasis libraries.

• Don't copy executable in ocamlbuild

Avoid copying executable to their real name. It helps to call ocamlbuild a single time for the whole build rather than calling it n time (n = number of executable sections) and copying resulting exec. This speeds up the build process because ocamlbuild doesn't have to compute/scan dependencies each time. The drawback is that you have to use $foo when you want to call Executable foo, because $foo will be '_build/.../main.byte'.

• Change the way we parse command line like option (CCOpt, CCLib and the like). We have implemented a real POSIX command line parser, except that variables are processed by Buffer.add_substitute (except if correctly escaped, using Buffer.add_substitute escaping).

For example:

CCOpt: -DEXTERNAL_EXP10 -L/sw/lib "-framework vecLib"

Will be parsed correctly and outputed according to target OS.

• Minimize the dependencies of the project.

In order to ease building oasis, we have minimize the number of dependencies. You only need to install ocamlmod, ocamlify and ocaml-data-notation for a standard build without tests. Dependencies on pcre, extlib and ocamlgraph has been dropped. The remaining dependencies are hidden behind a flag tests.

• Implement proposal for handling plugin in META, using "plugin" extra directives

OASIS now produces .cmxs file by default and add them to META. Now a META looks like:

     ...
     archive(byte) = "oasis.cma"
     archive(byte, plugin) = "oasis.cma"
     archive(native) = "oasis.cmxa"
     archive(native, plugin) = "oasis.cmxs"
     ...

This will ultimately help to generate automatically .cmxs for all oasis enabled projects. We hope that this new feature will improve dynamic linking use in OCaml (esp. for project like Ocsigen). Other changes

• Produce only subcommands specific helps when asked (Closes: #783)
• Use pager for long textual output (Closes: #782)
• Give the help summary directly when running just "oasis" (Closes: #781)
• Remove pwd filename prefix to avoid too long command line on Windows (Closes: #1007)
• Install required .cmx when installing libraries (Closes: #686, #885)
• Retain blanks at the beginning of line in freeform (Closes: #811)
• Allow PreConfCommand to create setup.data with any variables inside (Closes: #823)
• Allow to use setup.data in shell script and Makefile (Closes: #982)
• Fix execute permissions (Closes: #980)
• Drop extra version in ocaml version (Closes: #964, #888)
• Always output info messages on stderr (Closes: #785)
• Fix interversion in version comparator lexer (Closes: #808)
• Document the fact that setup.data is mandatory (Closes: #946)
• configure translates getopt command line to Arg (Closes: #778, #804)
• Generate valid META when there are line breaks in synopsis (Closes: #933)
• Keep owner of generated files (Closes: #793)
• Set -I +threads for ocamlfind ocamldoc in ocamlbuild (Closes: #949)
• Exclude VCS directories by default (Closes: #847)
• Print the configure summary in the "normal" order.
• Merge findlib virtual containers with the same name
• Synchronize plugin data for section
• Improve documentation
• Verify validity of input when parsing in quickstart (Closes: #797)
• Fix stack overflow when there are twice the same library (Closes: #1130)
• Parse license examples for DEP5 specification (Closes: #1102)
• Be ultra-liberal regarding version syntax (allow '_' and ' ' in the version).
• Allow to add extra content to META files (XMETAExtraLines)
• Allow to use executable name that contains "-" as a variable (replacing "-" by "_").

Thanks to Anil Madhavapeddy, Pierre Chambart, Christophe Troestler, Jeremie Dimino, Ronan Le Hy, Yaron Minsky and Till Varoquaux for their help with this release. Also thanks to all the testers of the numerous release candidates. This was a long work and each time a tester has downloaded oasis has helped me to know that I was working for someone.

The DebConf12 schedule is finally available! As usual, the DebConf conference can only be finalized a couple of days before DebConf itself starts. But hey Take a look at it, and start planning your activities! Not final, never final Remember that, even now that the conference schedule is announced, you can still submit talks to DebConf, following what we call ad-hoc talks. What are the steps to do it?

1. Check the schedule and find the timeslot (day, hour and room) you want to talk in
2. Register your event in Pentabarf, and note the generated event ID (the last number in the URL after the submission is accepted)
3. Send a mail to schedule@debconf.org detailing the event ID, title (just to confirm it), and the requested timeslot
4. All requests _must_ be received at least 24hr before the requested event for the scheduling team to act accordingly and for the video team to be able to plan on resources to cover the requested session
5. Video team coverage (event recording plus streaming on a best-effort basis) will be given to activities scheduled on the Aula Magna and Roberto Ter n auditoriums.
6. We will try to grant requests on a first-come-first-serve basis. If two events are requested for the same timeslot, the first will have precedence, and we will schedule the second as close as possible to it. We will notify the submitters if this happens, of course.

Conference tracks In DebConf12, many talks are organized around the following tracks:

DebianDay

This track is meant for the local (Nicaraguan/Central American) community to be attracted to Debian. It does not usually have technically minded talks; the DebConf general audience is, of course, invited to take part in it, as there will surely be interesting angles on how Debian is used in different use-cases, and there will be interesting Q&A session with the audience.

Getting involved in Debian

Central America is a very special region. It has vibrant Free Software user groups and communities, among which there is an important interest to get involved in Debian.
This track will have some talks in Spanish, some talks in English, explaining different Debian processes and tools, aiming at attracting locals to contribute to Debian.

Debian for the cloud

All about virtualization. Management infrastructure, details on virtual machine engines, insight on how to tweak your kernel for optimum virtualized performance, and of course how to get it all humming perfectly with your Debian packages.

Building and porting

With an every day stronger transit to mobile computing, we are seeing a renaissance of porting. At the same time, we finally have multiarch as part of Debian. And were that not to be enough, we finally have more than one compiler that can be used for most of the software in the archive. Put all this together, and you will have a very interesting track.

Social activities

DebConf is as much about technical stuff as it is about building closer ties within the members of its community. Eating cheese+wine, going on a trip to the guts of a volcano, having a nice banquet Do you want to add an activity to this pseudo-track?

Skills exchange

Knowledge can also be built by all of us putting our grain of sand. We first had explicit skills exchange sessions in Bosnia last year, and they were very well received. Feel free to propose similar sessions!

And Well, three days to DebCamp and counting. See you Real Soon Now in Managua!

Hello!

First of all: I'm going to DebConf :D and I really want to say a big "Thank you!" to every one who made this possible.


So, here's what I've done recently:

I spent some time putting everything in order for the departure to Nicaragua, things like finding a plane ticket through a route that does not require a Visa, making a new Passport, now all is in order.

I've integrated the changes for listing available ESSIDs, in order to close #637128 [0] into a separate branch , which can be found here [1].

The plan is to have only one new template which lists the available networks and which has a option for entering the ESSID manualy. In order to do so, a simple scan is performed on the wireless interface, and the configuration for a particular network resulted after the scan is kept if that one is chosen. The enter manually situation is treated exactly as it was before.

Furthermore, last week (or the week before) I've done some coding style fixes again in a separate branch [2], nothing very fancy, just trimming trailing white spaces and changing some tabs to spaces, where spaces were mostly used. I've also noticed some inconsistencies, regarding white spaces between functions names and the parenthesis. However I think I'm overreacting a little, those matters are not really that relevant, but when I'll have some free time I'll get to this as well.


Next in line: Issues

Testing is still giving me some troubles, the show essids part is not yet fully tested, so I couldn't say for sure if everything works as planned.

Apart from the classical: write the image to an USB and reboot method, I'm using Virtual Box for testing. I've managed to create a passthrough for the wireless interface, but now I have troubles with some firmware. Hopefully I'll solve this as soon as possible since it would really make my life easier.


Future plans:

I've realized that I've underestimated a little what the project implies and the amount of information that is new to me. I feel that I have learned a lot and that there is still a lot to learn and I'm very happy that it is so. I've now started documenting regarding IPv6, beginning with this [3]. We haven't decided yet what the next task will be exactly.


[0] http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=637128
[1] http://anonscm.debian.org/gitweb/?p=d-i/netcfg.git;a=shortlog;h=refs/heads/people/sorina/show_essids
[2] http://anonscm.debian.org/gitweb/?p=d-i/netcfg.git;a=shortlog;h=refs/heads/people/sorina/coding_style
[3] http://tldp.org/HOWTO/Linux+IPv6-HOWTO/

As Paul already posted, desktop-base 7.0 was uploaded to Debian unstable.
I worked on the KDE related integration of Adrien Aubourg s Joy Wallpapers (Congratulations again). I am pretty happy with how well they work with KDE, hence I am sharing the result.

This is what KDE s login manager kdm will look like in debian wheezy:

As you can see, the chooser got a whole rework. It s now based on the kdm theme I prepared for my Roj proposal. The new ksplash theme then includes the Debian logo:
For your workspace wallpaper there also exisits the ink splatted alternative:
I am pretty happy with the artwork. I like it s minimalism and it s not getting boring after using it for a while.

Today, OFCOM issued a press release entitled "New measures to protect online copyright and inform consumers". According to the announcement, internet users will be encouraged to download music and films through legal channels under the measures. Ofcom has published a draft code for consultation that would require large internet service providers (ISPs) to inform customers of allegations that their internet connection has been used to infringe copyright. There does seem to have been a number of concessions made, which is welcome, but I do still have a number of concerns. The code 'only' covers ISPs with more than 400k fixed lines - BT, Everything Everywhere, O2, Sky, TalkTalk Group and Virgin Media. This covers about 93% of the UK retail market. If an allegation of copyright infringement is made, the ISP would need to write to the customer with notice that an infringement has occurred, and in a new change, the number of copyright infringement reports connected to their account. If more than three letters are received in a year, anonymous information may be provided on request to copyright owners showing them which infringement reports are linked to that customer s account. The copyright owner may then seek a court order requiring the ISP to reveal the identity of the customer. Two more changes have been implemented:

• Copyright owners procedures for gathering evidence of infringement must now be approved by Ofcom, rather than by the copyright owners themselves.
• Ofcom has decided that subscribers should have 20 working days to appeal an allegation of infringement, but only on grounds specified in the Digital Economy Act.

Obviously the former of these will require further scrutiny, but I'm happier that this is not with the vested interest themselves. As for the latter, this gets a bit interesting. The Act states that

an appeal on any grounds must be determined in favour of the subscriber unless the copyright owner or internet service provider shows that a) the apparent infringement was an infringement of copyright, and b) the report relates to the subscriber s IP address at the time of that infringement.

This is good as it puts the burden of proof on the copyright owner, although it still uses IP addresses which is problematic. For example, New York Judge Gary Brown has ruled IP addresses are insufficient evidence to identify pirates, and has provided a lengthy and thoughtful explanation in his ruling. There has also been a consultation published on the how to share costs costs. The suggestion is fairly simple:

"Copyright Owners should bear all of the costs incurred by Ofcom, the majority of costs incurred by the appeals body, and 75% of the costs efficiently and reasonably incurred by Qualifying ISPs in carrying out their obligations."

This has the risk of driving up internet access prices as ISPs have to recover the costs of these notifications. Additionally:

"Under the Order, subscribers will have to pay a 20 fee to make an appeal against a report of infringement, which will be refunded in the event the appeal succeeds; the remaining costs of determining an appeal will be met by the Copyright Owner who submitted the CIR which has been appealed."

I disagree with the concept of having to pay for an appeal in this way. If accused, someone should be able to refute the argument without cost. This will introduce a barrier to entry, and even if it is refundable is something I believe should be avoided. Finally, the exact costs of implementing this have been set out. It looks like the Copyright Owners would have to pay 60p per letter (with the ISPs shouldering the remaining 20p), a similar split around capital costs - which are the ones that are actually significant. Taking the above, it seems that in total that content owners will have to shell out 15M to use the DEA scheme. The cost seems to be more or less fixed which could encourage content owners to send out frivolous accusations. There's an iterative process which will run to determine the estimated number of notifications, and if eventually the Content Owners don't agree with the pricing, they will have elected not to take advantage of the provisions of the Initial Obligations Code, as they have not been able to make binding commitments to fund CIR processing. So, in summary, this is something to keep an eye on. It certainly hasn't gone away, it's due to start in March 2014. The revised draft code and consultation, which closes on 26 July 2012, can be found here, and the sharing of costs consultation runs until 18th September 2012, found here.