Search Results: "tar"

4 July 2012

Stefano Zacchiroli: bits from the DPL for June 2012

Monthly DPL bits, fresh from the oven^W^W^W hot from DebConf12, and just posted to d-d-a.
Howdy from DebConf12. It's hot, but it's also time to bother you again with a (not so) brief DPL activity report, this time for June 2012. Time-based freeze: DONE, short freeze: TODO Two highlights for this month. First, you've probably noticed Wheezy is now frozen, YAY. This is huge achievement for the release, but also for the project. It's the first time we do a time-based freeze, and it took some quite heated discussion at the beginning of the release cycle to decide to do this. And we did it properly: respecting the planned month and narrowing down the period later. This exercise has hopefully helped both DDs in their package planning and our upstreams in targeting Wheezy with stable releases of their software. Kudos to the release team for their coordination work! Now we've the second part still TODO: releasing Wheezy, without RC bugs, with a freeze period as short as possible. See the beginning of my last "bits from the DPL" mail for my usual song and dance :-P on how to deliver that, together. DebConf12 A lot of us will attend DebConf12. Enjoy it! ... and take the chance to both have fun and make great plans for Debian's future. But remember that "if it didn't happen on a mailing list, it didn't happen". Not all of us will be lucky enough to attend DebConf (in person or remotely). Make sure that those who don't can take part in your team decisions and get informed of what is going to happen here. Politics Zack's spring tour I spent a significant part of June doing Debian talks ins some sort of "spring tour" between Italy and France. In particular: Many thanks to the organizers of these events for inviting and sponsoring me (as well as other Debian people, in the ESRF case) and for their interest in Debian. Sprints Assets Discussions Some relevant discussions for project evolution has been going on in June and I took part into them. You might want to have a look at them: Misc Cheers.
PS the boring day-to-day activity log for June is available at master:/srv/leader/news/bits-from-the-DPL.txt.201206

Dirk Eddelbuettel: RcppBDT 0.2.0

A new release of the RcppBDT package appeared on CRAN earlier today. RcppBDT uses Rcpp, and in particular the nifty Rcpp modules feature of wrapping C++ code for R just by declaring the (class or function) interfaces. It uses this to bring in some useful functions from Boost Date.Time to R so that one can do things like 
R> library(RcppBDT)
R> sapply(2012:2016, function(year)
+         format(getNthDayOfWeek(first, Mon, Sep, year)))
[1] "2012-09-03" "2013-09-02" "2014-09-01" "2015-09-07" "2016-09-05"
R>
to compute the next five Labor Day dates in the US, given the year and the first Monday of September requirement. More examples are e.g. on the earlier blog post announcing version 0.1.0. Changes are mostly internal. R 2.15.1 brough a better / easier way to load such 'modules' into R, and Rcpp 0.9.13 allows us to use this. So RcppBDT continues to be useful as an example package for Rcpp modules. I also streamlines the interface a little: identifiers are now directly accessible in the package's NAMESPACE rather than just via the an instantiated object. I also added a NEWS file, using the .Rd format so that we can import the marked-up text:
News for Package RcppBDT Changes in version 0.2.0 (2012-07-02)
  • The core module, which wraps what in C++ is boost::gregorian::date, is now exposed as an Rcpp module bdtDate. As all example and demos operated off a (package-global) variable 'bdt', no user visible change was needed outside of the code instantiating at package load.
  • Updated package instantiation to the new facilities offered by the current versions R 2.15.1 and Rcpp 0.9.13 which make Rcpp module initialization easier and more streamlined.
Changes in version 0.1.0 (2011-01-17)
  • First CRAN upload (and see ChangeLog for more granular details) bug fix in svm cross validation
Courtesy of CRANberries, there is also a diffstat report for 0.2.0 relative to 0.1.0. As always, feedback is welcome and the rcpp-devel mailing list off the R-Forge page for Rcpp is the best place to start a discussion.

3 July 2012

C.J. Adams-Collier: Currency platform superior to US Federal Reserve Bank

Should I recommend that Software in the Public Interest implement a monetary stack with the stated purpose of replacing the federal reserve system? We would have to publish the source code, of course, and the transaction db would need to be de-centralized so as to avoid private interests using it for their own benefit at the expense of their fellow humans. Not that that would ever happen, mittens. We d probably experience a period of instability due to scaling issues during the learning state, say the first two to five years. The IRS would need to be convinced that it would be a good idea, though. And I think they only listen to voters. And I don t want to be the one responsible for convincing them. I m not so good at exchanging paperwork with them, and they would likely get confused. Anyway, the idea would be that after the learning state, sessions would be migrated from the Fed db at a sustainable rate. After moving the sessions off of the Fed s current, compromised stack, the increase in tax revenues caused by removal of the current implementation s filters should be more than enough to fund the maintenance of the implementation undergoing strict peer review. http://www.wapublicbankproject.org/

Christian Perrier: DebConf running: take 1

One part of my yearly activity at DebConf is exploring the conference neighbourhood for good running spots. So, the first one this year has been an up and down run on "Camino Las Viudas", heading south from Managua. The path is easy to find: when exiting from Hotel Seminole, turn right in the street, cross the large boulevard on continue staright ahead for how many kilometers as you want. It climbs up in the hills all along, so that's quite a though one with about 5% slope all along. I stopped in the middle of nothing, just because I didn't want to run for more than one hour. I'll have to come back and see where this road goes. It seems to be climbing continuously..:-) So, one of my forthcoming plans is going to *the end* of this road, just to see what's there. From OpenStreetMap, it seems to end up in the hills. The run trace can be seen there.

Ryan Kavanagh: Fixing mutt header cache

Every now and then I have to kill mutt, which if done at the wrong time, can leave the header cache in an open state. Then attempting to restart mutt will cause it to hang at the "Selecting INBOX" stage. If mutt is compiled with support for the Tokyo Cabinet header cache backend (as it appears to be in Debian), a quick fix for this issue is to repair the header cache database using the Tokyo Cabinet utility tcbmgr from the tokyocabinet-bin package. Simply running tcbmgr optimize -nl ~/.mutt/cache/headers does the trick, and is much nicer of a fix than deleting the header cache.

2 July 2012

Matthew Garrett: Fun with firmware ABI confusion

I spent a bunch of the past week trying to figure out why my UEFI bootloader code was crashing in bizarre ways. It turns out to be a known issue that various people have hit in the past. After a lot of staring (and swearing) I finally got to the point where I could run a debugger against a running UEFI binary under qemu and found that I was entering a function with valid arguments but executing it with invalid ones. After spending a while crying on IRC people suggested that maybe the red zone was getting corrupted, and it turned out that this was the problem.

The red zone is part of the AMD64 System V ABI and consists of 128 bytes at the bottom of the stack frame. This region is guaranteed to be left untouched by any interrupt or signal handlers, so it's available for functions to use as scratch space. For the code in question, gcc was copying the registers onto the stack before executing the function. Something then trod on those registers. This also explained why I wasn't seeing the problem when I added debug print statements - signal and interrupt handlers won't touch it, but other functions might, so it tends to only be used by functions that don't call any other functions. Adding a print statement meant that the compiler left the red zone alone.

But why was it getting corrupted? UEFI uses the Microsoft AMD64 ABI rather than the System V one, and the Microsoft ABI doesn't define a red zone. UEFI executables built on Linux tend to use System V because that means we can link in static libraries built for Linux, rather than needing an entirely separate toolchain and libraries to build UEFI executables. The problem arises when we run one of these executables and there's a UEFI interrupt handler still running. Take an interrupt, Microsoft ABI interrupt handler gets called and the red zone gets overwritten.

There's a simple fix - we can just tell gcc not to assume that the red zone is there by passing -mno-red-zone. I did that and suddenly everything was stable. Frustrating, but at least it works now.

comment count unavailable comments

Dominique Dumont: Kerberos configuration editor looking for a maintainer

Hello A while ago, Peter Knowles wrote a configuration model for Kerberos. Unfortunately, his priorities shifted before he completed this work. He was kind enough to send me the prototype in hope someone could take over. Although I do not know Kerberos, I believe the model is rather complete. Just click on the image below to judge for yourself: Class diagram of Kerberos configuration model This work was released under a LGPL-2+ license. There are some tasks left to get a Kerberos configuration editor. These are mentioned in config-model-kerberos readme file. To look at the code, just clone config-model-kerberos with 
git://github.com/dod38fr/config-model-kerberos.git
If you need more information on Config::Model, see: If you are interested in adopting Kerberos model, please keep us informed on config-model-users at lists.sourceforge.net. This will avoid duplicated work. On my side, I ll gladly answer questions related to Config::Model to help you get started. All the best
Tagged: Config::Model, Kerberos, Perl

Mehdi Dogguy: Adding useful links on bugreports pages

I'm a heavy user of Debian's Bug Tracking System (especially during this period of the year ;)) and I quite like it. And as a Debian developer, I have to check multiple sources when looking at some source package. Namely, I mainly check the following sources:
All this is great since you can move from the first one to the second one (and vice-versa) just by clicking with your mouse. It becomes less handy when you are looking at any bug report. In fact, the BTS doesn't link back to the previously mentioned services [1]. So, you have to copy/paste the name of the package or start a new tab and write down the address of the desired service. All this is not very nice for a lazy person like me :)

A few days ago, I remembered that I was already using GreaseMonkey and thought : "Hey, it would be nice to hack quickly something using GM to add the missing links". After a few tests, I've come up with a (very simple, yet extremely useful to me) user script that I called Debian BTS extra links [2]. It adds what I've been missing for years. If you are lazy and don't want to install right now, you can check the following pages to see what it adds:
The userscript will add links to PTS, Buildd status pages and Popcon near every pkgreport.cgi?src= link. Simple and useful. I hope you'll enjoy it as much as I do.

Of course, I've submitted a bug (#679282) against the pseudo package bugs.debian.org kindly asking its maintainer to add a link to (for now) the PTS. But I know that they are busy so I'm not expecting a result soon. Fow now, my simple and dumb userscript WFM.

[1] This is partly true since it links back to the PTS on main pages (pages that list all bug reports of a given set of packages).

[2] Only tested with a recent Iceweasel (from experimental) and Chromium (from unstable).

P.S.: If you do wonder, #4300 is the least bug number known to the BTS. Well, or at least, the result of running ls -1 /srv/bugs.debian.org/spool/archive/00/ sort -h head -n1 on bugs.debian.org.

Stefano Zacchiroli: DebConf no sporran 12

It's that time of the year again. No, not only the Debian biyearly freeze that has just happened, again (YAY!). It's DebConf time again. Yesterday night I arrived in Managua, Nicaragua, for DebConf12. I'm still jetlagged a bit, but the first impression is great. And that impression is also very green, shockingly green. The city is gorgeous: full of trees and plants (well, at least compared to my metropolitan European standards), and the campus of Universidad Centroamericana where the conference is hosted is even more so. Also, it's a great pleasure to have DebConf in an university campus where students are still swarming. My only reason for sadness thus far. While I did bring my traditional Debian kilt at DebConf, I discovered this morning that I forgot my sporran at home :-(
So if you don't see me wearing my kilt at this DebConf often, that's why

Raphaël Hertzog: My Debian Activities in June 2012

This is my monthly summary of my Debian related activities. If you re among the people who made a donation to support my work (168.12 , thanks everybody!), then you can learn how I spent your money. Otherwise it s just an interesting status update on my various projects. Dpkg This month, I resumed my work on dpkg. I concentrated my efforts on some polishing of the 3.0 (quilt) format. With the latest version (1.16.6 which was uploaded to unstable shortly before the freeze), dpkg-source restores the source tree in a clean state after a failed patch application (#652970), doesn t overwrite the patch header from the pre-existing automatic patch, updates automatically debian/source/include-binaries during dpkg-source commit, and supports a new no-unapply-patches option for those who dislike the auto-unapplication at the end of the process when the patches were not applied at the start. I wanted to go further and offer a new feature that could insert the automatic patch at the bottom of the quilt series but I have been short on time to complete this feature. I just managed to factorize all the quilt handling in a dedicated Perl module (Dpkg::Source::Quilt) to have cleaner code in the module handling the source format (Dpkg::Source::Package::V3::quilt). For those who wonder, this feature is meant primarily for the X Strike Force team which maintains packages in Git and are doings lots of upstream cherry-picks (to fix regressions, etc.). But they also use quilt on top of that tree to keep some lasting Debian specific changes. With the 1.0 format, the automatic diff is a bit messy but at least it gets smaller automatically when a new upstream release gets out, there s nothing to clean out. I d like them to be able to use 3.0 (quilt) while keeping their workflow. I m leaning towards allowing --auto-commit=first:cherry-picks that would name the automatic patch cherry-picks and put it in the first position in the quilt series. (Opinions welcome on that feature, BTW) Packaging There s been quite some packaging in this last month before the freeze: While doing all this packaging work, I found 2 possible improvements that I filed as bug reports: Debian France Booth at Solutions Linux From June 19th to June 21th, I manned the Debian France booth at Solutions Linux together with Carl Chenet, Tanguy Ortolo and other members of the association. We answered lots of questions, sold all t-shirts and umbrellas that Carl imported from Germany and Switzerland (we really need to get our own merchandising stuff produced in France!), got people to join the association. We also presented a printed copy of the Debian Administrator s Handbook and of the corresponding French book. You can see Carl, me and Tanguy on this picture (click on it to see a bigger picture, thanks to S bastien Dubois of Evolix for this one!):
I know lots of people are preparing for Debconf but I decided to not attend this year, the price of the air plane ticket was a bit too hefty for me and it was also in partial conflict with our family vacations. I thought about attending the Libre Software Meeting instead but alas I won t go there either (but Roland Mas will be there!), I have too much work to complete before my own vacation in 2 weeks. Thanks See you next month for a new summary of my activities.

2 comments Liked this article? Click here. My blog is Flattr-enabled.

Thibaut Girka: apt-get install gcc-4.7-arm-linux-gnueabihf

Hi! As you may know, I'm working on multiarch cross-toolchains as part of the Google summer of Code (see my previous post for details), and although there is still a lot to do, the project is doing fairly well, and a cross-gcc is already installable and working. I'll proceed with detailed instructions on how to install the cross-compiler, along with some explanations on multiarch itself, and how to build cross-toolchains from source. Prerequisites First, you need specific versions apt (>= 0.9.6) and dpkg (>= 1.16.5).
This is because the cross-toolchains packages I'm working on depend on specific cross-arch dependencies, such as libc6:armhf. Support for such dependencies has been added in the aforementioned versions of apt and dpkg. This means you should be running unstable, or at least use dpkg from unstable. Adding a foreign architecture Adding a foreign architecture is straightforward, but I've seen users do overcomplicated, or just plain wrong things. So, let's do this step by step (well, there are only two of them), with some explanations.
  1. Tell dpkg to handle an additional architecture by typing dpkg --add-architecture armhf. There isn't much to say. It will only add an architecture to the list of architectures dpkg would accept to handle.
  2. Update APT's database. You need to run apt-get update after adding or removing an architecture to get package lists for all wanted architectures and rebuild APT's database, including the implicit cross-arch dependencies/conflicts brought by multiarch.
And that's it for the common case. Now, a word on architecture restrictions: a common misconception about multiarch seems to be that you have to tell APT which architectures you want from the mirrors by modifying your sources.list file(s). This is not true, as, by default, APT gets the list of accepted architectures from dpkg, and gets the correct package lists from the mirrors configured in the sources.list. You only need to use arch restrictions ( deb [arch=arch1,arch2,...] ... ) when you want to get packages from a mirror only for a subset of your accepted architectures. This is very unlikely, and in the general you do not need to modify your sources.list, at all. Getting the cross-compilers from my repository I provide compiled versions of gcc-4.7-arm-linux-gnueabihf for i386 and amd64 in my repository. If you have followed the previous paragraphs, nothing should be easier than adding a deb http://emdebian.org/~thibg/repo/ sid main line to your sources.list and doing the following: 
# apt-get update
[...]
# apt-get --no-install-recommends install gcc-4.7-arm-linux-gnueabihf
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following extra packages will be installed:
  binutils-arm-linux-gnueabihf cpp-4.7-arm-linux-gnueabihf gcc-4.7-arm-linux-gnueabihf-base gcc-4.7-base:armhf libc6:armhf libgcc1:armhf
  libgmp10 libgomp1:armhf libmpc2 libmpfr4
Suggested packages:
  binutils-doc gcc-4.7-locales libmudflap0-4.7-dev:armhf gcc-4.7-doc libgcc1-dbg:armhf libgomp1-dbg:armhf libitm1-dbg:armhf
  libquadmath-dbg:armhf libmudflap0-dbg:armhf binutils-gold glibc-doc:armhf locales:armhf
Recommended packages:
  libc6-dev:armhf
The following NEW packages will be installed:
  binutils-arm-linux-gnueabihf cpp-4.7-arm-linux-gnueabihf gcc-4.7-arm-linux-gnueabihf gcc-4.7-arm-linux-gnueabihf-base gcc-4.7-base:armhf
  libc6:armhf libgcc1:armhf libgmp10 libgomp1:armhf libmpc2 libmpfr4
0 upgraded, 11 newly installed, 0 to remove and 0 not upgraded.
Need to get 23.5 MB of archives.
After this operation, 57.2 MB of additional disk space will be used.
Do you want to continue [Y/n]?
I've disabled the recommends since, libc6-dev:armhf would recommend gcc:armhf itself, which is not co-installable with your native gcc if you have it installed. Building the cross-toolchain yourselves If you don't trust the repository above, or simply want to rebuild the toolchain yourself, it is rather simple: get my gcc-4.7 patches and: 
$ apt-get source binutils
$ cd binutils-*/
$ TARGET=arm-linux-gnueabihf dpkg-buildpackage -uc -us -b
$ cd ../
# dpkg -i binutils*arm-linux-gnueabihf*.deb
$ apt-get source gcc-4.7
$ cd gcc-4.7/
$ for patch in path/to/the/patches/*.patch; do patch -p1 < $patch ; done
$ GCC_TARGET=armhf debian/rules control
$ GCC_TARGET=armhf dpkg-buildpackage -uc -us -b
Using the cross-compiler It's easy, but there is a trick, and dpkg-buildpackage -aarmhf won't work out of the box yet, as gcc-4.7-arm-linux-gnueabihf provides the arm-linux-gnueabihf-gcc-4.7 executable, but no arm-linux-gnueabihf-gcc symlink. So, for the moment: 
$ cat > test.c << __EOF__
#include <stdio.h>
int main(void)
 
  printf("Hello world\\n");
  return 0;
 
__EOF__
$ arm-linux-gnueabihf-gcc-4.7 test.c
$ qemu-arm ./a.out 
Hello world
Conclusion The toolchain is not complete yet (there is some more work to do on binutils and libstdc++, and equivalents to the gcc , cpp , and build-essential packages to provide), but it should still be useful to cross-compile C programs. Multi-arch cross-toolchains for wheezy can definitely be hosted at emdebian, and I hope to clean up the build process so that it could get included in the archive for wheezy+1.

Craig Small: Dejagnu tips

Both the procps and psmisc projects use Dejagnu for their testing. It s interesting to try to understand how it all works and learning a new language TCL. One of the big problems with Dejagnu is its documentation which is very sparse. So here are two things I ve picked up along the way for a reference for myself and others. Locale We recently had a bug report on the procps list where tests for w failed for someone when the headers were shown. After a some further debugging it was a locale problem, where the testsuite was looking for a number like 3.14 but the tester was seeing 3,14. There are many ways around this but for us it was to set the locale, with the following line in config/unix.exp after suggestions by Mike Frysinger should fix it. 
set env(LC_ALL) "C"
Cleaning Up
The documentation on this function is awesome! A title Cleanup Procedure and one line cleanup() .
That s all you get. The cleanup proc or function is called at the end of the testing. I use it
to remove temporary files and things like that. 
proc cleanup      
  global test_file
  exec rm $test_file

Johannes Schauer: port bootstrap build-ordering tool report 3

A copy of this post is sent to soc-coordination@lists.alioth.debian.org as well as to debian-bootstrap@lists.mister-muffin.de.

Diary

June 18 Pietro suggests a faster way to generate installation sets for a list of packages. In my case, I need an installation set for every source package in the archive to find out how often a binary package is needed to build a source package. As a result, the speed is doubled in contrast to the original approach.

June 19
  • adapt code to work with new dose release 3.0
  • remove unneeded parts of code
  • add different possibilities to find amount of source packages that need a binary package
  • add code to get multiple installation sets using Depsolver_int.solve

June 20
  • add ~global_constraints:false to Depsolver.listcheck, Depsolver.trim and Depsolver.edos_install calls
  • adapt output graph to limited xdot capabilities

June 21 I formulate an email to the list, reporting of dependency graphs of debhelper, cdbs, pkg-config and libgtk2.0-dev. My current technique gets an installation set for a source package, removes all those that are already installable and adds the others as a dependency of that source package. This dependency will include an installation set of that binary as well minus all packages that are already available. The problem with that approach are dependency cycles created by long dependency chains. Example: src:A needs B needs C needs A. B and C would both be added as a dependency of src:A. B as well as C would also include their installation set which in both cases includes A. So now there are two cycles: src:A->B->A and src:A->C->A. For a real life example, look at the following situation of cdbs and src:sqlite3. cdbs old situation It is created because src:sqlite3 needs cdbs needs python-scour needs python needs python2.7 needs libsqlite3-0. Therfor libsqlite3-0 is in the installation set of cdbs, python-scour, python and python2.7. This creates five cycles in the graph even though there is only one. It would be better to reduce the dependencies added to src:sqlite3 to its direct dependency which is cdbs. Package dependencies are disjunctions from which the solver chooses one or the other to build an installation set. To solve the problem above I would need to know which disjunction the solver chose and then only add the direct dependency of a package to the dependency graph.
  • improve build_compile_rounds performance
  • big overhaul of menu structure
  • fix subgraph extraction code

June 22
  • do not create a universe if not needed - use hashtables instead
  • for sorting packages, generating difference of package sets and otherwise comparing packages, always use CudfAdd.compare
  • as a custom list membership function, use List.exists instead of trying List.find
  • more speedup for build_compile_rounds
  • the number of source packages that can be built does NOT include the cross built packages
  • print closure members in graph
  • refactor code and move common functions to bootstrapCommon.ml
  • add breakcycles.ml for future code to break cycles using staged build dependencies
  • use more extlib functionality
  • extended package list input format

June 23 After several emails with Pietro I learn about syntactic dependency graphs. To document my progress and prove my efforts I committed the code as commit 6684c13. But this code was soon found out to be unecessary so it will be removed later and just serves as documentation.

June 24 I came up with another (better?) solution to get the chosen disjunctions. It simply uses the calculated installation set to decide for each disjunction which one was taken by the solver. I reported that important step and the open questions involved with it in an email to the list. The problem always was, that an installation set can easily contain more than one package of a disjunction. In this case it is not clear which branch was chosen by the solver. I found, that in Ubuntu Natty there are only 6 such packages and for each of them the situation can be solved. It can be solved because in all of those cases it is that either one package of a disjunction provides the other or that both packages depend upon each other, which means that both have to be included.

June 27
  • use installation set to flatten build dependencies of source packages
  • refactor code and move common functions to bootstrapCommon.ml

June 25 I have to have an algorithm that finds all circuits in a given graph. This is necessary so that:
  1. cycles can be enumerated for huge dependency graphs where cycles are hard to see
  2. cycles can be enumerated to find a cycle that can be broken using staged build dependencies
It seems that Johnson's algorithm is the best way to do this complexity wise and Pietro already blogged about the problem together with an implementation of the algorithm in ocaml. Unfortunately it turns out that his code doesnt implement the algorithm correctly and hence misses out on some cycles. The fix seems not to be too trivial so I'm still investigating it.

June 28
  • add crosseverything.ml to obtain a list of source packages that, if cross compiled, would make the whole archive available

Results While the first week was productive as usual, I had to work some time on a University project during the second week as well as attend a family meeting. I will catch up with the lost time over the course of the next week.

dose3 Using dose 3.0 (which fixes a bug about essential packages) the output of the algorithms is now likely less wrong then before.

performance Performance was improved in the generation of installation sets as well as in the code that tries out how many packages can be built in multiple rounds. This was achieved by more caching, less unnecessary operations in looping constructs, replacing lists with hashtables, not creating universes where not necessary.

user interface The main program, basenocycles.ml now has a much better menu structure.

input format The programs take two package file inputs. The list of source packages that has to be cross built for a minimal build system and the list of source packages that was chosen to be cross compiled in addition to that. Both files list one source package per line and now allow comments.

refactoring As more and more scripts are added, more and more functionality is moved to bootstrapCommon.ml which makes each script much cleaner.

what to test for cross building As discussed in the "Future" section of the last report, I now automated the process of finding out which packages, if they were cross compiled, would make the whole archive available because they break all cycles and allow native compilation of the rest. The outcome: to build 3333 out of 3339 packages in natty natively, at most 186 source packages must be cross compiled. The other six cannot be compiled because of version mismatches in the Natty Sources.bz2 and Packages.bz2. The code can be run from crosseverything.ml.

limit source dependencies to direct dependencies Reducing the dependencies of source packages from their full installation set to their direct dependencies by finding out which disjunction of their dependency list were taken, greatly simplifies the dependency graphs. The dependency graph created for libgtk2.0-dev could be reduced from 491 to 247 vertices for a depth of three. For cdbs it is now clearly visible that cdbs depends on libsqlite3-0 which builds from src:sqlite3 which depends on cdbs. Before: cdbs old situation After: cdbs new situation For pkg-config the graph also has been reduced to the one single cycle that matters: src:pkg-config needs libglib2.0-dev which depends on pkg-config which builds from src:pkg-config. Before: pkg-config old situation After: pkg-config old situation

Future I will prepare content for wookey's debconf talk on crossbuilding and bootstrapping. As this will include directions how to use the current code, I will kill two birds with one stone and write some proper documentation for my current source. The following two lists will be displayed after a dependency graph is calculated and reduced to its scc:
  • those source packages that have the least build dependencies not fulfilled. Those might be candidates for easy staged build dependencies. Since the source package is part of the scc, it will definitely be involved in some cycle somewhere.
  • those binary packages that most source packages depend upon. Those could be candidates for cross compilation as it might be easier to cross compile the source package than using staged build dependencies.
Patrick managed to cross build packages with sbuild now. So the list of packages that crosseverything.ml produces can now be checked efficiently for cross buildability. With this list, potentially more cycles can be broken out of the box. A feature will be added that allows the user to remove all packages from a dependency graph that can be cross compiled without any additional effort. Version mismatches between source and binary packages in Sources.bz2 and Packages.bz2 respectively in Ubuntu make the scripts fail and/or produce wrong results. Debian (even Sid) doesnt have this problem so I should find out where to report this problem to Ubuntu. I need to write a working version of Johnson's algorithm because much functionality depends upon it. I have the option to improve Pietro's version or write one from scratch. Writing one from scratch might be easier as I have Pietro's code as template as well as a Java implementation of Johnson's algorithm which seems to work. The following functionalities need working cycle enumeration:
  • given source packages with staged build dependencies, an enumeration of cycles is needed to find out which cycles can be broken by building packages staged. It makes less sense to blindly build a package stage and then check if this makes more packages available.
  • display cycles of a dependency graph to the user. After obtaining all cycles in the graph it makes sense to sort them by their length. The user would then investigate the situation of the smallest cycles first. This makes sense because breaking small cycles can potentially break bigger cycles. Since in the end, all cycles have to be eliminated anyway, it makes sense for the user to first tackle the small ones.
  • display the feedback arc set to the user. The packages in the feedback arc set might be very good candidates for reduced build dependencies or cross compilation.

Julien Danjou: Making the jump: working freelance

For the last 10 years, I've been working on many Free Software projects. From Debian to OpenStack, through awesome, Emacs, XCB and many more. This obviously allowed me to enhance my technical skills, but it also taught me about Free Software and Open Source development processes, and how to work with and close to the community. Working for almost 6 years at Easter-eggs taught me how to work in an autonomous manner, how to lead and manage a project. And how to run a company, thanks to the cooperative status of this great one. These are the reasons why I decided to leave my latest job and run my own company to work as a freelance consultant & developer specialized in Free Software, starting today. Therefore, I am now able and available to provide expertise and development on Free Software, including upstream contribution. Especially on projects I already worked on recently, like OpenStack.

Joey Hess: notes for a caretaker

I recently had the sort of weird experience of a recent blog post being on the top of Hacker News and there being a fair amount of interest in the details of my wacky living situation. I could write a detailed post explaining everything, but that'd be boring .. instead I'll tease the stalkers with more oblique references to it. So here's my notes for a caretaker who will be here while I'm away at DebConf.
Please make yourself at home!

1 July 2012

Gunnar Wolf: Arrived to Nicaragua. DebCamp has officially started!

Yesterday night, Regina and me arrived to Nicaragua. Ready to greet us, we found quite a good number of good friends. We had a nice pizza+beer dinner at Diana's house, and some of the foreigners among us were distributed among the houses of several locals. This morning, we woke up together with V ctor, Moray and Gaudenz in Norman's brother's beautiful house. We had breakfast with the family, were picked up to go to the hotel that will have the ho(n rr)or to host us all for the following two weeks, and walked to the Universidad Centroamericana (UCA) campus. Contrary to our usual practices... It seems everything is working fine! I mean, I'm sure we will stumble with some unforseen details and what not... But coming on the very first day to the university, to find that food is all sorted out, that we have food tickets (and they are all printed!), that network works (and it's by a fiber connection that was laid out expressly for us), that we have all the hardware I was worried about, that people are arriving and getting accepted at the hotel. I mean, things work! So, I'm quite optimistic this DebCamp will have everything ready to be a success And the DebConf following it as well, of course! If you have not yet arrived - See you soon!

Niels Thykier: Performance bottlenecks in Lintian

Thanks to a heads up from Bastian Blank, I learned that Lintian 2.5.7 and 2.5.8 were horribly slow on the Linux binaries. Bastian had already identified the issue and 2.5.9 fixed the performance regression. But in light of that, I decided to have a look at a couple of other bottlenecks. First, I added a simple benchmark support to Lintian 2.5.10 (enabled with -dd) that prints the approximate run time of a given collection. As an example, when running lintian -dd on lintian 2.5.10, you can see something like: 
N: Collecting info: unpacked for source:lintian/2.5.10 ...
[...]
N: Collection script unpacked for source:lintian/2.5.10 done (0.699s)
When done on linux-image, the slowest 3 things with 2.5.10 are (in order of appearance): 
[...]
N: Collection script strings for binary:linux-image-3.2.0-2-amd64/3.2.20-1/amd64 done (12.333s)
N: Collection script objdump-info for binary:linux-image-3.2.0-2-amd64/3.2.20-1/amd64 done (15.915s)
[...]
N: Finished check: binaries (5.911s)
[...]
(The mileage (and order) probably will vary a bit.) These 3 things makes up about 22 seconds of a total running time on approximately 28-30s on my machine. Now if you wondering how 12, 16 and 6 becomes 22 the answer is parallelization . strings and objdump-info are run in parallel so only the most
expensive of the two counts in practise (with multiple processing units). The version of linux-image I have been testing (3.2.20-1, amd64) has over 2800 ELF binaries (kernel modules). That makes the runtime of strings and objdump-info much more dominating than in your average package . For the fun of it I have done a small informal benchmark of various Lintian versions on the binary. I have used the command line: 
# time is the bash shell built-in and not /usr/bin/time
$ time lintian -EvIL +pedantic linux-image-3.2.0-2-amd64_3.2.20-1_amd64.deb >/dev/null
# This was used with only versions that did not accept -L +pedantic
$ time lintian -EvI --pedantic linux-image-3.2.0-2-amd64_3.2.20-1_amd64.deb >/dev/null
With older versions of Lintian (<= 2.5.3) Perl starts to emit warnings; these have been manually filtered out. I used lintian from the git repository (i.e. I didn t install the packages, but checked out the relevant git tags). I had libperlio-gzip-perl installed (affects the 2.5.10 run). Most results are only from a single run, though I ran it twice on the first version (hoping my kernel would cache the deb for the next run). The results are: 
2.5.10
real  0m28.836s
user  0m36.982s
sys  0m3.280s
2.5.9
real  1m9.378s
user  0m33.702s
sys  0m11.177s
2.5.8
real  4m54.492s
user  4m0.631s
sys  0m30.466s
2.5.7 (not tested, but probably about same as 2.5.8)
2.5. 0..6 
real  1m20s  - 1m22s
user  0m19.0s - 0m20.7s
sys  0m5.1s  - 0m5.6s
I think Bastian s complaint was warranted for 2.5. 7,8 . :) While it would have been easy to attribute the performance gain in 2.5.10 on the new parallelization improvements, it is simply not the case. These improvements only apply to running collections when checking multiple packages. On my machine, the parallelization limit for a package is effectively determined by the dependencies between the collections on my machine. Instead the improvements comes from reducing the number of system(3) (or fork+exec) calls Lintian does. Mostly through using xargs more, even if it meant slightly more complex code. But also, libperlio-gzip-perl shaved off a couple of seconds on binaries check. But as I said, linux-image is not your average package . Most of the improvements mentioned here are hardly visible on other packages. So let s have a look at some more other bottlenecks. In my experience the following are the worst offenders : But enough Lintian for now time to fix some RC bugs!

Ingo Juergensmann: 100% CPU load due to Leap Second

This morning Gregor Samsa woke up... oh, pardon! This morning I woke up and found myself puzzled, because my home server was eating up all of my 4 cores CPU cycles. Especially mysqld was high on CPU load. 100% CPU load for the mysql-server instance and 100% CPU load for akonadiservers own mysqld instance. Restarting KDE and mysql-server didn't help on my Debian unstable machine. Next step was upgrading the system. Sometimes this helps indeed, but not today. Looking at bugs.debian.org for mysql-server didn't reveal any help as well. So my next logical step was to ask on #debian-devel in IRC. And my question was very quick answered:
11:28 < ij> since tonight I've got two mysqld processes running at 100% CPU, one spawned by akonadi and
the other is the mysqld from mysql-server (unstable that is). is this an already known issue?
haven't found anything on b.d.o for mysql-server, though
11:29 < mrvn> ij: topic
11:29 < mrvn> you need to set the time
11:30 < ij> waaaaah!
11:30 < mrvn> ij: indeed.
The topic was at that time:
100% CPU? Reset leap second http://xrl.us/bnde4w
So, it was caused by the leap second. Although you might suspect mysql doing some nasty things (which, IMHO, is always a good guess ;)), the issue is this time within the Linux kernel itself, as a commit on git.kernel.org clearifies. To fix this issue you need to set the time manually using the following command or just reboot:
date -s " date "
So far I found these applications being hit by this kernel bug:
  • mysql-server
  • akonadi (as it uses own mysql instances)
  • Firefox
  • Openfire Jabber server (because it's using Java, which seems to trigger the problem as mysql does)
  • Virtualbox' VBoxSVC process
  • puppetmaster from package puppet, reported by Michael
  • mythfrontend, reported by pos on #debian-devel
  • Jetty, Hudson, Puppet agent and master, reported by Christian
  • milter-greylist, reported by E. Recio
  • dovecot, reported by Diogo Resende
  • Google Chrome, reported by Erik B. Andersen
  • if you find more apps, please comment and I'll include them here...
So, hope this helps and many thanks to mrvn and infinity on #debian-devel for the help!
Kategorie:
Debian
Tags:
Debian
Linux
Bug
Software

Dirk Eddelbuettel: Getting numpy data into R

The other day, I found myself confronted with a large number of large files. Which were presented in (gzip-)compressed ascii format---which R reads directly via gzfile() connections---as well as (compressed) numpy files. The numpy can be read very efficiently into Python. We can do the same in R via save() and load(), of course. But the trouble is that you need to read them first. And reading hundreds of megabytes from ascii is slow, no matter which language you use. Concerning R, I poked aound scan(), played with the colClasses argument and looked at the recent LaF package written just for this purpose. And all these solutions were still orders of magnitude slower than reading numpy. Which is no surprise as it is really hard to beat binary formats when you have to parse countless ascii tokens. So the obvious next idea was to read the numpy file in Python, and to write a simple binary format. One helpful feature with this data set was that it contained only regular (rectangular) matrices of floats. So we could just store two integers for the dimensions, followed by the total data in either one large binary blob, or a sequence of column vectors. But one minor trouble was that the Intertubes lead to no easy solution to unpack the numpy format. StackOverflow had plenty of question around this topic converned with, say, how to serialize in language-independent way. But no converters. And nobody local knew how to undo the "pickle" format underlying numpy. But a remote friend did: Laurent, well-known for his Rpy2 package, pointed me towards using the struct module and steered me towards the solution shown below. So a shameless plug: if you need a very experienced Python or R consultant for sciece work, consider his consulting firm. Finally, to round out this post, let's show the simple solution we crafted so that the next guy searching the Intertubes will have an easier. Let us start with a minimal Python program writing numpy data to disk: 
#!/usr/bin/env python
#
# simple example for creating numpy data to demonstrate converter
import numpy as np
# simple float array
a = np.arange(15).reshape(3,5) * 1.1
outfile = "/tmp/data.npy"
np.save(outfile, a)
Next, the simple Python converter to create a binary file containing two integers for row and column dimension, followed by row times columns of floats: 
#!/usr/bin/python
#
# read a numpy file, and write a simple binary file containing
#   two integers 'n' and 'k' for rows and columns
#   n times k floats with the actual matrix
# which can be read by any application or language that can read binary
 
import struct
import numpy as np
inputfile = "/tmp/data.npy"
outputfile = "/tmp/data.bin"
# load from the file
mat = np.load(inputfile)
# create a binary file
binfile = file(outputfile, 'wb')
# and write out two integers with the row and column dimension
header = struct.pack('2I', mat.shape[0], mat.shape[1])
binfile.write(header)
# then loop over columns and write each
for i in range(mat.shape[1]):
    data = struct.pack('%id' % mat.shape[0], *mat[:,i])
    binfile.write(data)
binfile.close()
Lastly, a quick littler script showing how R can read the data in a handful of lines: 
#!/usr/bin/r
infile <- "/tmp/data.bin"
con <- file(infile, "rb")
dim <- readBin(con, "integer", 2)
Mat <- matrix( readBin(con, "numeric", prod(dim)), dim[1], dim[2])
close(con)
print(Mat)
That did the job---and I already used to converter to read a few weeks worth of data for further analysis in R. This obviously isn't the last word on possible solutions as the additional temporary file can be wasteful (unless it forms a cache for data read multiple times). If someone has nice solutions, please don't hold back and contact me. Thanks again to Laurent for the winning suggestion concerning struct, and help in getting the examples shown here to work.

30 June 2012

Gregor Herrmann: RC bugs 2012/26

wheezy is frozen, the outside temperature in INN at midnight is 30 C, & in 3 hours my trip to MGA starts. & here's the list of RC bugs; this week all but one were sponsored uploads where the BTS already contained a complete debdiff; thanks guys!

Next.

Previous.