Search Results: "will"

2 July 2012

Michal Čihař: Surprising uses of Weblate

When I wrote Weblate I expected that major use cases will be free software projects, which will use it for public translating service. At least this was my intention for it. But I seem to be wrong, at least for now. Most of people who contact me regarding Weblate are doing private installations and I already know few places where it was massively used. Unfortunately most of them rejected to be publicly mentioned, so I can't tell you the names. But they do exist :-). Anyway I hope for some free software projects to adopt Weblate as well, otherwise I should have written it as commercial software :-)

Filed under: English Weblate 6 comments Flattr this!

Raphaël Hertzog: My Debian Activities in June 2012

This is my monthly summary of my Debian related activities. If you re among the people who made a donation to support my work (168.12 , thanks everybody!), then you can learn how I spent your money. Otherwise it s just an interesting status update on my various projects. Dpkg This month, I resumed my work on dpkg. I concentrated my efforts on some polishing of the 3.0 (quilt) format. With the latest version (1.16.6 which was uploaded to unstable shortly before the freeze), dpkg-source restores the source tree in a clean state after a failed patch application (#652970), doesn t overwrite the patch header from the pre-existing automatic patch, updates automatically debian/source/include-binaries during dpkg-source commit, and supports a new no-unapply-patches option for those who dislike the auto-unapplication at the end of the process when the patches were not applied at the start. I wanted to go further and offer a new feature that could insert the automatic patch at the bottom of the quilt series but I have been short on time to complete this feature. I just managed to factorize all the quilt handling in a dedicated Perl module (Dpkg::Source::Quilt) to have cleaner code in the module handling the source format (Dpkg::Source::Package::V3::quilt). For those who wonder, this feature is meant primarily for the X Strike Force team which maintains packages in Git and are doings lots of upstream cherry-picks (to fix regressions, etc.). But they also use quilt on top of that tree to keep some lasting Debian specific changes. With the 1.0 format, the automatic diff is a bit messy but at least it gets smaller automatically when a new upstream release gets out, there s nothing to clean out. I d like them to be able to use 3.0 (quilt) while keeping their workflow. I m leaning towards allowing --auto-commit=first:cherry-picks that would name the automatic patch cherry-picks and put it in the first position in the quilt series. (Opinions welcome on that feature, BTW) Packaging There s been quite some packaging in this last month before the freeze: While doing all this packaging work, I found 2 possible improvements that I filed as bug reports: Debian France Booth at Solutions Linux From June 19th to June 21th, I manned the Debian France booth at Solutions Linux together with Carl Chenet, Tanguy Ortolo and other members of the association. We answered lots of questions, sold all t-shirts and umbrellas that Carl imported from Germany and Switzerland (we really need to get our own merchandising stuff produced in France!), got people to join the association. We also presented a printed copy of the Debian Administrator s Handbook and of the corresponding French book. You can see Carl, me and Tanguy on this picture (click on it to see a bigger picture, thanks to S bastien Dubois of Evolix for this one!):
I know lots of people are preparing for Debconf but I decided to not attend this year, the price of the air plane ticket was a bit too hefty for me and it was also in partial conflict with our family vacations. I thought about attending the Libre Software Meeting instead but alas I won t go there either (but Roland Mas will be there!), I have too much work to complete before my own vacation in 2 weeks. Thanks See you next month for a new summary of my activities.

2 comments Liked this article? Click here. My blog is Flattr-enabled.

Thibaut Girka: apt-get install gcc-4.7-arm-linux-gnueabihf

Hi! As you may know, I'm working on multiarch cross-toolchains as part of the Google summer of Code (see my previous post for details), and although there is still a lot to do, the project is doing fairly well, and a cross-gcc is already installable and working. I'll proceed with detailed instructions on how to install the cross-compiler, along with some explanations on multiarch itself, and how to build cross-toolchains from source. Prerequisites First, you need specific versions apt (>= 0.9.6) and dpkg (>= 1.16.5).
This is because the cross-toolchains packages I'm working on depend on specific cross-arch dependencies, such as libc6:armhf. Support for such dependencies has been added in the aforementioned versions of apt and dpkg. This means you should be running unstable, or at least use dpkg from unstable. Adding a foreign architecture Adding a foreign architecture is straightforward, but I've seen users do overcomplicated, or just plain wrong things. So, let's do this step by step (well, there are only two of them), with some explanations.
  1. Tell dpkg to handle an additional architecture by typing dpkg --add-architecture armhf. There isn't much to say. It will only add an architecture to the list of architectures dpkg would accept to handle.
  2. Update APT's database. You need to run apt-get update after adding or removing an architecture to get package lists for all wanted architectures and rebuild APT's database, including the implicit cross-arch dependencies/conflicts brought by multiarch.
And that's it for the common case. Now, a word on architecture restrictions: a common misconception about multiarch seems to be that you have to tell APT which architectures you want from the mirrors by modifying your sources.list file(s). This is not true, as, by default, APT gets the list of accepted architectures from dpkg, and gets the correct package lists from the mirrors configured in the sources.list. You only need to use arch restrictions ( deb [arch=arch1,arch2,...] ... ) when you want to get packages from a mirror only for a subset of your accepted architectures. This is very unlikely, and in the general you do not need to modify your sources.list, at all. Getting the cross-compilers from my repository I provide compiled versions of gcc-4.7-arm-linux-gnueabihf for i386 and amd64 in my repository. If you have followed the previous paragraphs, nothing should be easier than adding a deb http://emdebian.org/~thibg/repo/ sid main line to your sources.list and doing the following:
# apt-get update
[...]
# apt-get --no-install-recommends install gcc-4.7-arm-linux-gnueabihf
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following extra packages will be installed:
  binutils-arm-linux-gnueabihf cpp-4.7-arm-linux-gnueabihf gcc-4.7-arm-linux-gnueabihf-base gcc-4.7-base:armhf libc6:armhf libgcc1:armhf
  libgmp10 libgomp1:armhf libmpc2 libmpfr4
Suggested packages:
  binutils-doc gcc-4.7-locales libmudflap0-4.7-dev:armhf gcc-4.7-doc libgcc1-dbg:armhf libgomp1-dbg:armhf libitm1-dbg:armhf
  libquadmath-dbg:armhf libmudflap0-dbg:armhf binutils-gold glibc-doc:armhf locales:armhf
Recommended packages:
  libc6-dev:armhf
The following NEW packages will be installed:
  binutils-arm-linux-gnueabihf cpp-4.7-arm-linux-gnueabihf gcc-4.7-arm-linux-gnueabihf gcc-4.7-arm-linux-gnueabihf-base gcc-4.7-base:armhf
  libc6:armhf libgcc1:armhf libgmp10 libgomp1:armhf libmpc2 libmpfr4
0 upgraded, 11 newly installed, 0 to remove and 0 not upgraded.
Need to get 23.5 MB of archives.
After this operation, 57.2 MB of additional disk space will be used.
Do you want to continue [Y/n]?
I've disabled the recommends since, libc6-dev:armhf would recommend gcc:armhf itself, which is not co-installable with your native gcc if you have it installed. Building the cross-toolchain yourselves If you don't trust the repository above, or simply want to rebuild the toolchain yourself, it is rather simple: get my gcc-4.7 patches and:
$ apt-get source binutils
$ cd binutils-*/
$ TARGET=arm-linux-gnueabihf dpkg-buildpackage -uc -us -b
$ cd ../
# dpkg -i binutils*arm-linux-gnueabihf*.deb
$ apt-get source gcc-4.7
$ cd gcc-4.7/
$ for patch in path/to/the/patches/*.patch; do patch -p1 < $patch ; done
$ GCC_TARGET=armhf debian/rules control
$ GCC_TARGET=armhf dpkg-buildpackage -uc -us -b
Using the cross-compiler It's easy, but there is a trick, and dpkg-buildpackage -aarmhf won't work out of the box yet, as gcc-4.7-arm-linux-gnueabihf provides the arm-linux-gnueabihf-gcc-4.7 executable, but no arm-linux-gnueabihf-gcc symlink. So, for the moment:
$ cat > test.c << __EOF__
#include <stdio.h>
int main(void)
 
  printf("Hello world\\n");
  return 0;
 
__EOF__
$ arm-linux-gnueabihf-gcc-4.7 test.c
$ qemu-arm ./a.out 
Hello world
Conclusion The toolchain is not complete yet (there is some more work to do on binutils and libstdc++, and equivalents to the gcc , cpp , and build-essential packages to provide), but it should still be useful to cross-compile C programs. Multi-arch cross-toolchains for wheezy can definitely be hosted at emdebian, and I hope to clean up the build process so that it could get included in the archive for wheezy+1.

Johannes Schauer: port bootstrap build-ordering tool report 3

A copy of this post is sent to soc-coordination@lists.alioth.debian.org as well as to debian-bootstrap@lists.mister-muffin.de.

Diary

June 18 Pietro suggests a faster way to generate installation sets for a list of packages. In my case, I need an installation set for every source package in the archive to find out how often a binary package is needed to build a source package. As a result, the speed is doubled in contrast to the original approach.

June 19
  • adapt code to work with new dose release 3.0
  • remove unneeded parts of code
  • add different possibilities to find amount of source packages that need a binary package
  • add code to get multiple installation sets using Depsolver_int.solve

June 20
  • add ~global_constraints:false to Depsolver.listcheck, Depsolver.trim and Depsolver.edos_install calls
  • adapt output graph to limited xdot capabilities

June 21 I formulate an email to the list, reporting of dependency graphs of debhelper, cdbs, pkg-config and libgtk2.0-dev. My current technique gets an installation set for a source package, removes all those that are already installable and adds the others as a dependency of that source package. This dependency will include an installation set of that binary as well minus all packages that are already available. The problem with that approach are dependency cycles created by long dependency chains. Example: src:A needs B needs C needs A. B and C would both be added as a dependency of src:A. B as well as C would also include their installation set which in both cases includes A. So now there are two cycles: src:A->B->A and src:A->C->A. For a real life example, look at the following situation of cdbs and src:sqlite3. cdbs old situation It is created because src:sqlite3 needs cdbs needs python-scour needs python needs python2.7 needs libsqlite3-0. Therfor libsqlite3-0 is in the installation set of cdbs, python-scour, python and python2.7. This creates five cycles in the graph even though there is only one. It would be better to reduce the dependencies added to src:sqlite3 to its direct dependency which is cdbs. Package dependencies are disjunctions from which the solver chooses one or the other to build an installation set. To solve the problem above I would need to know which disjunction the solver chose and then only add the direct dependency of a package to the dependency graph.
  • improve build_compile_rounds performance
  • big overhaul of menu structure
  • fix subgraph extraction code

June 22
  • do not create a universe if not needed - use hashtables instead
  • for sorting packages, generating difference of package sets and otherwise comparing packages, always use CudfAdd.compare
  • as a custom list membership function, use List.exists instead of trying List.find
  • more speedup for build_compile_rounds
  • the number of source packages that can be built does NOT include the cross built packages
  • print closure members in graph
  • refactor code and move common functions to bootstrapCommon.ml
  • add breakcycles.ml for future code to break cycles using staged build dependencies
  • use more extlib functionality
  • extended package list input format

June 23 After several emails with Pietro I learn about syntactic dependency graphs. To document my progress and prove my efforts I committed the code as commit 6684c13. But this code was soon found out to be unecessary so it will be removed later and just serves as documentation.

June 24 I came up with another (better?) solution to get the chosen disjunctions. It simply uses the calculated installation set to decide for each disjunction which one was taken by the solver. I reported that important step and the open questions involved with it in an email to the list. The problem always was, that an installation set can easily contain more than one package of a disjunction. In this case it is not clear which branch was chosen by the solver. I found, that in Ubuntu Natty there are only 6 such packages and for each of them the situation can be solved. It can be solved because in all of those cases it is that either one package of a disjunction provides the other or that both packages depend upon each other, which means that both have to be included.

June 27
  • use installation set to flatten build dependencies of source packages
  • refactor code and move common functions to bootstrapCommon.ml

June 25 I have to have an algorithm that finds all circuits in a given graph. This is necessary so that:
  1. cycles can be enumerated for huge dependency graphs where cycles are hard to see
  2. cycles can be enumerated to find a cycle that can be broken using staged build dependencies
It seems that Johnson's algorithm is the best way to do this complexity wise and Pietro already blogged about the problem together with an implementation of the algorithm in ocaml. Unfortunately it turns out that his code doesnt implement the algorithm correctly and hence misses out on some cycles. The fix seems not to be too trivial so I'm still investigating it.

June 28
  • add crosseverything.ml to obtain a list of source packages that, if cross compiled, would make the whole archive available

Results While the first week was productive as usual, I had to work some time on a University project during the second week as well as attend a family meeting. I will catch up with the lost time over the course of the next week.

dose3 Using dose 3.0 (which fixes a bug about essential packages) the output of the algorithms is now likely less wrong then before.

performance Performance was improved in the generation of installation sets as well as in the code that tries out how many packages can be built in multiple rounds. This was achieved by more caching, less unnecessary operations in looping constructs, replacing lists with hashtables, not creating universes where not necessary.

user interface The main program, basenocycles.ml now has a much better menu structure.

input format The programs take two package file inputs. The list of source packages that has to be cross built for a minimal build system and the list of source packages that was chosen to be cross compiled in addition to that. Both files list one source package per line and now allow comments.

refactoring As more and more scripts are added, more and more functionality is moved to bootstrapCommon.ml which makes each script much cleaner.

what to test for cross building As discussed in the "Future" section of the last report, I now automated the process of finding out which packages, if they were cross compiled, would make the whole archive available because they break all cycles and allow native compilation of the rest. The outcome: to build 3333 out of 3339 packages in natty natively, at most 186 source packages must be cross compiled. The other six cannot be compiled because of version mismatches in the Natty Sources.bz2 and Packages.bz2. The code can be run from crosseverything.ml.

limit source dependencies to direct dependencies Reducing the dependencies of source packages from their full installation set to their direct dependencies by finding out which disjunction of their dependency list were taken, greatly simplifies the dependency graphs. The dependency graph created for libgtk2.0-dev could be reduced from 491 to 247 vertices for a depth of three. For cdbs it is now clearly visible that cdbs depends on libsqlite3-0 which builds from src:sqlite3 which depends on cdbs. Before: cdbs old situation After: cdbs new situation For pkg-config the graph also has been reduced to the one single cycle that matters: src:pkg-config needs libglib2.0-dev which depends on pkg-config which builds from src:pkg-config. Before: pkg-config old situation After: pkg-config old situation

Future I will prepare content for wookey's debconf talk on crossbuilding and bootstrapping. As this will include directions how to use the current code, I will kill two birds with one stone and write some proper documentation for my current source. The following two lists will be displayed after a dependency graph is calculated and reduced to its scc:
  • those source packages that have the least build dependencies not fulfilled. Those might be candidates for easy staged build dependencies. Since the source package is part of the scc, it will definitely be involved in some cycle somewhere.
  • those binary packages that most source packages depend upon. Those could be candidates for cross compilation as it might be easier to cross compile the source package than using staged build dependencies.
Patrick managed to cross build packages with sbuild now. So the list of packages that crosseverything.ml produces can now be checked efficiently for cross buildability. With this list, potentially more cycles can be broken out of the box. A feature will be added that allows the user to remove all packages from a dependency graph that can be cross compiled without any additional effort. Version mismatches between source and binary packages in Sources.bz2 and Packages.bz2 respectively in Ubuntu make the scripts fail and/or produce wrong results. Debian (even Sid) doesnt have this problem so I should find out where to report this problem to Ubuntu. I need to write a working version of Johnson's algorithm because much functionality depends upon it. I have the option to improve Pietro's version or write one from scratch. Writing one from scratch might be easier as I have Pietro's code as template as well as a Java implementation of Johnson's algorithm which seems to work. The following functionalities need working cycle enumeration:
  • given source packages with staged build dependencies, an enumeration of cycles is needed to find out which cycles can be broken by building packages staged. It makes less sense to blindly build a package stage and then check if this makes more packages available.
  • display cycles of a dependency graph to the user. After obtaining all cycles in the graph it makes sense to sort them by their length. The user would then investigate the situation of the smallest cycles first. This makes sense because breaking small cycles can potentially break bigger cycles. Since in the end, all cycles have to be eliminated anyway, it makes sense for the user to first tackle the small ones.
  • display the feedback arc set to the user. The packages in the feedback arc set might be very good candidates for reduced build dependencies or cross compilation.

Benjamin Mako Hill: Wiki Conferencing

I am in Berlin for the Wikipedia Academy, a very cool hybrid free culture community plus refereed academic conference organized, in part, by Wikimedia Deutschland. On Friday, I was very excited to have been invited to give the conference's opening keynote based on my own hybrid take on learning from failures in peer production and incorporating a bunch of my own research. Today, I was on a panel at the conference about free culture and sharing practices. I'll post talks materials and videos when the conference puts them online. I will be in Berlin for the next week or so before I head to directly to Washington, DC for Wikimania between the 11th and 15th. I'll be giving three talks there: Between then and now, I'm taking the next week in Berlin to catch up on work, and with friends. If you're in either place and want to meet up, please get in touch and lets try to arrange something.

Joey Hess: notes for a caretaker

I recently had the sort of weird experience of a recent blog post being on the top of Hacker News and there being a fair amount of interest in the details of my wacky living situation. I could write a detailed post explaining everything, but that'd be boring .. instead I'll tease the stalkers with more oblique references to it. So here's my notes for a caretaker who will be here while I'm away at DebConf.
Please make yourself at home!

1 July 2012

Gunnar Wolf: Arrived to Nicaragua. DebCamp has officially started!

Yesterday night, Regina and me arrived to Nicaragua. Ready to greet us, we found quite a good number of good friends. We had a nice pizza+beer dinner at Diana's house, and some of the foreigners among us were distributed among the houses of several locals. This morning, we woke up together with V ctor, Moray and Gaudenz in Norman's brother's beautiful house. We had breakfast with the family, were picked up to go to the hotel that will have the ho(n rr)or to host us all for the following two weeks, and walked to the Universidad Centroamericana (UCA) campus. Contrary to our usual practices... It seems everything is working fine! I mean, I'm sure we will stumble with some unforseen details and what not... But coming on the very first day to the university, to find that food is all sorted out, that we have food tickets (and they are all printed!), that network works (and it's by a fiber connection that was laid out expressly for us), that we have all the hardware I was worried about, that people are arriving and getting accepted at the hotel. I mean, things work! So, I'm quite optimistic this DebCamp will have everything ready to be a success And the DebConf following it as well, of course! If you have not yet arrived - See you soon!

Wouter Verhelst: Heathrow tourist trap buses

So I slept in the Sheraton Heathrow hotel last night. It's a half-an-hour walk away from the airport, or a five minute bus drive. If you go to any service desk and ask for the best way to get to the hotel, they'll offer you a ticket for the 'HotelHoppa' service, which will stop right in front of your hotel entrance, and costs 4 for a one-way trip (even though the bus sides declare a 4.5 charge) However, if you take the regular "London United" bus service (which drives all over London, not just near the airport like the HotelHoppa service) then the same trip is completely free of charge. You may have to walk about a hundred yards more to reach your hotel, since the bus will stop at a regular bus stop rather than at the hotel entrance, but that's hardly a problem. What a ripoff. So, to clarify, don't take this bus: Rip-off bus Instead, take this bus: Free bus This message brought by your friendly neighbourhood anti-ripoff service.

Niels Thykier: Performance bottlenecks in Lintian

Thanks to a heads up from Bastian Blank, I learned that Lintian 2.5.7 and 2.5.8 were horribly slow on the Linux binaries. Bastian had already identified the issue and 2.5.9 fixed the performance regression. But in light of that, I decided to have a look at a couple of other bottlenecks. First, I added a simple benchmark support to Lintian 2.5.10 (enabled with -dd) that prints the approximate run time of a given collection. As an example, when running lintian -dd on lintian 2.5.10, you can see something like:
N: Collecting info: unpacked for source:lintian/2.5.10 ...
[...]
N: Collection script unpacked for source:lintian/2.5.10 done (0.699s)
When done on linux-image, the slowest 3 things with 2.5.10 are (in order of appearance):
[...]
N: Collection script strings for binary:linux-image-3.2.0-2-amd64/3.2.20-1/amd64 done (12.333s)
N: Collection script objdump-info for binary:linux-image-3.2.0-2-amd64/3.2.20-1/amd64 done (15.915s)
[...]
N: Finished check: binaries (5.911s)
[...]
(The mileage (and order) probably will vary a bit.) These 3 things makes up about 22 seconds of a total running time on approximately 28-30s on my machine. Now if you wondering how 12, 16 and 6 becomes 22 the answer is parallelization . strings and objdump-info are run in parallel so only the most
expensive of the two counts in practise (with multiple processing units). The version of linux-image I have been testing (3.2.20-1, amd64) has over 2800 ELF binaries (kernel modules). That makes the runtime of strings and objdump-info much more dominating than in your average package . For the fun of it I have done a small informal benchmark of various Lintian versions on the binary. I have used the command line:
# time is the bash shell built-in and not /usr/bin/time
$ time lintian -EvIL +pedantic linux-image-3.2.0-2-amd64_3.2.20-1_amd64.deb >/dev/null
# This was used with only versions that did not accept -L +pedantic
$ time lintian -EvI --pedantic linux-image-3.2.0-2-amd64_3.2.20-1_amd64.deb >/dev/null
With older versions of Lintian (<= 2.5.3) Perl starts to emit warnings; these have been manually filtered out. I used lintian from the git repository (i.e. I didn t install the packages, but checked out the relevant git tags). I had libperlio-gzip-perl installed (affects the 2.5.10 run). Most results are only from a single run, though I ran it twice on the first version (hoping my kernel would cache the deb for the next run). The results are:
2.5.10
real  0m28.836s
user  0m36.982s
sys  0m3.280s
2.5.9
real  1m9.378s
user  0m33.702s
sys  0m11.177s
2.5.8
real  4m54.492s
user  4m0.631s
sys  0m30.466s
2.5.7 (not tested, but probably about same as 2.5.8)
2.5. 0..6 
real  1m20s  - 1m22s
user  0m19.0s - 0m20.7s
sys  0m5.1s  - 0m5.6s
I think Bastian s complaint was warranted for 2.5. 7,8 . :) While it would have been easy to attribute the performance gain in 2.5.10 on the new parallelization improvements, it is simply not the case. These improvements only apply to running collections when checking multiple packages. On my machine, the parallelization limit for a package is effectively determined by the dependencies between the collections on my machine. Instead the improvements comes from reducing the number of system(3) (or fork+exec) calls Lintian does. Mostly through using xargs more, even if it meant slightly more complex code. But also, libperlio-gzip-perl shaved off a couple of seconds on binaries check. But as I said, linux-image is not your average package . Most of the improvements mentioned here are hardly visible on other packages. So let s have a look at some more other bottlenecks. In my experience the following are the worst offenders : But enough Lintian for now time to fix some RC bugs!

Dirk Eddelbuettel: Getting numpy data into R

The other day, I found myself confronted with a large number of large files. Which were presented in (gzip-)compressed ascii format---which R reads directly via gzfile() connections---as well as (compressed) numpy files. The numpy can be read very efficiently into Python. We can do the same in R via save() and load(), of course. But the trouble is that you need to read them first. And reading hundreds of megabytes from ascii is slow, no matter which language you use. Concerning R, I poked aound scan(), played with the colClasses argument and looked at the recent LaF package written just for this purpose. And all these solutions were still orders of magnitude slower than reading numpy. Which is no surprise as it is really hard to beat binary formats when you have to parse countless ascii tokens. So the obvious next idea was to read the numpy file in Python, and to write a simple binary format. One helpful feature with this data set was that it contained only regular (rectangular) matrices of floats. So we could just store two integers for the dimensions, followed by the total data in either one large binary blob, or a sequence of column vectors. But one minor trouble was that the Intertubes lead to no easy solution to unpack the numpy format. StackOverflow had plenty of question around this topic converned with, say, how to serialize in language-independent way. But no converters. And nobody local knew how to undo the "pickle" format underlying numpy. But a remote friend did: Laurent, well-known for his Rpy2 package, pointed me towards using the struct module and steered me towards the solution shown below. So a shameless plug: if you need a very experienced Python or R consultant for sciece work, consider his consulting firm. Finally, to round out this post, let's show the simple solution we crafted so that the next guy searching the Intertubes will have an easier. Let us start with a minimal Python program writing numpy data to disk:
#!/usr/bin/env python
#
# simple example for creating numpy data to demonstrate converter
import numpy as np
# simple float array
a = np.arange(15).reshape(3,5) * 1.1
outfile = "/tmp/data.npy"
np.save(outfile, a)
Next, the simple Python converter to create a binary file containing two integers for row and column dimension, followed by row times columns of floats:
#!/usr/bin/python
#
# read a numpy file, and write a simple binary file containing
#   two integers 'n' and 'k' for rows and columns
#   n times k floats with the actual matrix
# which can be read by any application or language that can read binary
 
import struct
import numpy as np
inputfile = "/tmp/data.npy"
outputfile = "/tmp/data.bin"
# load from the file
mat = np.load(inputfile)
# create a binary file
binfile = file(outputfile, 'wb')
# and write out two integers with the row and column dimension
header = struct.pack('2I', mat.shape[0], mat.shape[1])
binfile.write(header)
# then loop over columns and write each
for i in range(mat.shape[1]):
    data = struct.pack('%id' % mat.shape[0], *mat[:,i])
    binfile.write(data)
binfile.close()
Lastly, a quick littler script showing how R can read the data in a handful of lines:
#!/usr/bin/r
infile <- "/tmp/data.bin"
con <- file(infile, "rb")
dim <- readBin(con, "integer", 2)
Mat <- matrix( readBin(con, "numeric", prod(dim)), dim[1], dim[2])
close(con)
print(Mat)
That did the job---and I already used to converter to read a few weeks worth of data for further analysis in R. This obviously isn't the last word on possible solutions as the additional temporary file can be wasteful (unless it forms a cache for data read multiple times). If someone has nice solutions, please don't hold back and contact me. Thanks again to Laurent for the winning suggestion concerning struct, and help in getting the examples shown here to work.

30 June 2012

Leandro G&oacute;mez: DebCamp -1

DebCamp starts tomorrow and the local team is relaxing and enjoying some free time with a cup of Debian Coffee. See you soon in Managua! (In case you wonder yes!!! both the Debian Coffee and the DebConf coffee mug will be for sale during DebConf12 in Managua)

Andrea Veri: Nagios IRC Notifications

Lately (as I earlier pointed out on my blog) I ve been working on improving GNOME s infrastructure monitoring services. After configuring XMPP it was time to find out a good way for sending out relevant notifications to our IRC channel hosted on GIMPNET. I achieved that with a nice combo: supybot + supybot-notify, all that mixed up with a few grains of Nagios command definitions. But here we go with a little step-by-step guide: Requirements 1. Install supybot and configure a new installation:
apt-get install supybot or yum install supybot
mkdir /home/$user/nagbot && cd /home/$user/nagbot
supybot-wizard (follow the directions to get the bot initially configured)
2. Install and load the supybot-notify plugin by doing:
git clone git://git.fedorahosted.org/supybot-notify.git && cd supybot-notify
mkdir -p /home/$user/nagbot/plugins/notify && cp -r * /home/$user/nagbot/plugins/notify
Finally, load the plugin. (this will require you to authenticate to the bot) Nagios configuration 1. Add the relevant command definitions to the commands.cfg file:
# 'notify-by-ircbot' command definition
define command 
    command_name    notify-by-ircbot
    command_line    /usr/bin/printf "%b" "#channel $NOTIFICATIONTYPE$ - $HOSTALIAS$/$SERVICEDESC$ is $SERVICESTATE$: $SERVICEOUTPUT$ ($$(hostname -s))"   nc -w 1 localhost 5050
     
# 'host-notify-by-ircbot' command definition
define command 
	command_name	host-notify-by-ircbot
	command_line	/usr/bin/printf "%b" "#channel $NOTIFICATIONTYPE$ - $HOSTALIAS$ is $HOSTSTATE$: $HOSTOUTPUT$ ($$(hostname -s))"   nc -w 1 localhost 5050
	 
* adjust the Netcat s host and port to your needs, in my case Supybot and Nagios were running on the same host. In the case of a Supybot running on a different host than Nagios, tweak Iptables to allow the desired port:
-A INPUT -m state --state NEW,ESTABLISHED -m tcp -p tcp --dport 5050 -j ACCEPT
2. Add a new entry on the contacts.cfg file:
define contact 
 	contact_name    nagbot
   	use             generic-contact
        alias		Nagios IRC Bot
        email           example@example.com
        service_notification_commands   notify-by-ircbot
 	host_notification_commands      host-notify-by-ircbot
 
3. Reload Nagios:
sudo /etc/init.d/nagios3 reload
And finally, enjoy the result:
 PROBLEM - $hostalias/load average is CRITICAL: CRITICAL - load average: 30.45, 16.24, 7.16 (nagioshost)
 RECOVERY - $hostalias/load average is OK: OK - load average: 0.06, 0.60, 3.65 (nagioshost)

Bernhard R. Link: ACPI power button for the rest of us

The acpi-support maintainer unfortunately decided 2012-06-21 that having some script installed by a package to cleanly shut down the computer should not be possible without having consolekit and thus dbus installed. So (assuming this package will migrate to wheezy which it most likely will tomorrow) with wheezy you will either have to write your own event script or install consolekit and dbus everywhere. You need two files. You need one in /etc/acpi/events/, for example a /etc/acpi/events/powerbtn:
event=button[ /]power
action=/etc/acpi/powerbtn.sh
Which causes a power-button even to call a script /etc/acpi/powerbtn.sh, which you of course also need:
#!/bin/sh
/sbin/shutdown -h -P now "Power button pressed"
You can also name it differently, but /etc/acpi/powerbtn.sh has the advantage that the script from acpi-support-base (in case it was only removed and not purged) does not call shutdown itself if it is there. (And do not forget to restart acpid, otherwise it does not know about your event script yet). For those too lazy I've also prepared a package acpi-support-minimal, which only contains those scripts (and a postinst to restart acpid to bring it into effect with installation), which can be get via apt-get using
deb http://people.debian.org/~brlink/acpi-minimal wheezy-acpi-minimal main
deb-src http://people.debian.org/~brlink/acpi-minimal wheezy-acpi-minimal main
or directly from http://people.debian.org/~brlink/acpi-minimal/pool/main/a/acpi-support-minimal/. Sadly the acpi-support maintainer sees no issue at all and ftp-master doesn't like so tiny packages (which is understandable but means the solution is more than a apt-get away).

29 June 2012

Jo Shields: How to build a bungalow begin with the chimney

Whilst nobody was looking, Mono became the most important framework in the world of games. And I mean that with only a smidge more than my usual levels of hyperbole. It really is fantastically important, as far as game development goes. I blogged a long time ago about its use inside EA s multi-million-selling The Sims 3. And the massive performance gains in Second Life when they switched from their in-house scripting language to Mono are well documented. But I m not talking about those, I m talking about the now, the recent stuff. First up is Unity 3D. Not to be confused with the Ubuntu user interface, Unity 3D is a proprietary game development tool which began life for Mac, and has now managed to become the single most popular game engine amongst smartphone developers. It targets a bunch of new platforms not just Android and Windows and iOS, but in the new 4.0 release, Linux too for the first time. And every Unity 3D game is a Mono game the core of Unity is a fork of Mono, which enables it to have the performance and ease of development which it does. A reasonably large percentage of the cross-platform games on Kickstarter, such as Wasteland 2 and Super Retro Squad, are cross-platform because Unity 3D makes it easy and each of those games is powered by Mono. However, I have better things to do today than gush about a proprietary tool. No, far more important are the Free ways to develop games. So today I m going to talk about MonoGame. MonoGame is an implementation of Microsoft XNA a high level framework for developing in .NET languages, originally designed to allow indie developers access to the Xbox 360. XNA covers many of the bases covered by SDL sound, graphics, input, and so on, but can go onto platforms where SDL isn t an option, such as the aforementioned Xbox 360. And with MonoGame, any game written for XNA is a game which can not only run on Microsoft s approved platforms Windows, Xbox 360, Windows Phone but also on other platforms such as Mac OS X, iPhone, iPad, Android, and Linux. Cross-platform is good. More games are good. And MonoGame enables every Xbox 360 indie developer to throw a Linux port out of the door with a relative minimum of fuss. There are 2,521 games on the Xbox Live Indie Games store (games by individuals or small groups, with no major publisher) and each of those can become a game for Debian and Ubuntu with very little work. Not to mention the games with actual publishers (some portion of the 492 games on XBLA), not to mention the mobile games written for Windows Phone, Android and iOS. All of these developers have an easy route to Linux releases, as long as the effort remains low. And with today s publishing in the Debian archive of MonoGame 2.5.1, the effort is as low as can be. Anyone who purchased the recent Humble Indie Bundle 5, and played Bastion, has seen MonoGame in action Bastion began life as an XNA game for Xbox 360, and the Linux port was made possible only by MonoGame. Bastion s 100+ industry awards should indicate the possibilities open to a creative mind with decent developer tools. MonoGame is Free Software, built on top of other Free Software, with an active upstream development team genuinely responsive to the needs of Linux distributions. It s well worth a look not just for porting, but for new projects too. And I m delighted that it will be available to all users of Debian 7.0, as well as Ubuntu 12.10.

Dirk Eddelbuettel: Rcpp 0.9.13

The bug-fix in version 0.9.12 of Rcpp turned out to be incomplete, so a new version 0.9.13 is now on CRAN and will get to Debian shortly. The Rcpp::Enviroment constructor is now properly fixed (using the global environment as a default value). As well, a new #define was created to detect the clang++ compiler before/after version 3.0.0 as one exceptions header changed location. Unit tests files were also once more updated. The complete NEWS entry for 0.9.13 is below; more details are in the ChangeLog file in the package and on the Rcpp Changelog page.
0.9.13  2012-06-28
    o   Truly corrected Rcpp::Environment class by having default constructor
        use the global environment, and removing the default argument of
        global environment from the SEXP constructor
    o   Added tests for clang++ version to include bits/exception_defines.h
        for versions 3.0 or higher (similar to g++ 4.6.0 or later), needed to
        include one particular exceptions header
    o   Made more regression tests conditional on the RunAllRcppTests to come
        closer to the CRAN mandate of running tests in sixty seconds
    o   Updated unit test wrapper tests/doRUnit.R as well as unitTests/runTests.R
Thanks to CRANberries, you can also look at a diff to the previous release 0.9.12. As always, even fuller details are on the Rcpp Changelog page and the Rcpp page which also leads to the downloads, the browseable doxygen docs and zip files of doxygen output for the standard formats. A local directory has source and documentation too. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page

Michal &#268;iha&#345;: New job, new challenges

Starting next week, I'll start at new job. I will still stay at SUSE, so it's not that big change, but still it is new challenge for me. I'll be moving out of L3 department, where I've spend almost four years. L3 is really a great team where man can learn basically anything, simply because you can meet there anything from broken swap on S390 up to wrong icons in LibreOffice :-). The new position is inside Security team, still I'll be mostly focused on writing some internal tools, rather than doing security work (at least for first year or so). The current plan is base on work I've done for L3 tools and use Django for the web interface, but that's just a plan for now and might change in the future. Anyway looking forward to new challenges at new job.

Filed under: Django English Suse 0 comments Flattr this!

28 June 2012

Sylvain Le Gall: OASIS 0.3.0 release

Logo OASIS small OASIS is a tool to integrate a configure, build and install system in your OCaml project. It helps to create standard entry points in your build system and allows external tools to analyse your project easily. It is the building brick of OASIS-DB, a CPAN for OCaml. This release takes almost 18 months to complete. This is too long and I will talk in another blog post on the way I am trying to improve this right now (esp. using continuous integration). This new release fixes a small bug (1 line) that prevents setup.ml to run with OCaml 4.00. If you have projects that was generated with a former release consider upgrading to OASIS 0.3.0. There are several big new stuff that comes with this release. It now supports Pack: field for libraries which allows to pack your library using -for-pack and so on. We are also compiling .cmxs (dynlink object for native) by default and we publish them in the META file. This feature was implemented in order to get more libraries to provide .cmxs and to help project like Ocsigen to take advantage of this. If you want to get rid of this at configure time, you can use ./configure --override native_dynlink false. We introduce two new default flags: --enable-tests and --disable-docs for configure. These are implicit flags that define if we will run Test sections or compile Document sections. They are especially useful to reduce the number of dependencies, because dependencies of Test will be excluded by default. We recommend to set Build$: flag(tests) to any Library or Executable sections that are only useful for tests. This allows you to really cut down the number of dependencies. The last change I want to introduce is about the old setup-dev subcommand which is now deprecated. It has been replaced by 2 different update schemes. I am pretty excited by this feature which in fact comes from OASIS user (esp. Lwt project). The former scheme was to have a big 'setup.ml' that always call the command oasis to update. This was complex and not very useful. We now have 2 mode: dynamic and weak. dynamic allow you to have a small setup.ml and to keep your VCS history clean but you need to install OASIS. weak need a big setup.ml but only need to call the command oasis if someone change something in _oasis. This mode is targeted to project that wants to be able to checkout from VCS an OCaml project without installing OASIS. The difference generated by weak mode doesn't pollute too much the VCS history because most the time they make sense. For example, you upgrade your package version number in _oasis and it produces a change with 6 lines where the version number changes in every META, setup.ml files and so on. This version has been tested on: You can download it here or use your favorite package manager: Debian (UPDATE: pending upload)
$ apt-get install -t experimental oasis
GODI
$ godi_console perform -build apps-oasis
odb.ml
$ ocaml odb.ml oasis
Here is the complete changelog: EXTREMLY IMPORTANT changes (read this) PACKAGES uploaded to oasis-db will be automatically "derived" before OCaml 4.00 release (i.e. oUnit v1.1.1 will be regenerated with this new version as oUnit v1.1.1~oasis1). PACKAGES not uploaded to oasis-db need to be regenerated. In order not to break 3rd party tools that consider a tarball constant, I recommend to create a new version. Thanks to INRIA OCaml team for synchronizing with us on this point. Major changes: You can now have the following example:
     ...
     Executable test_exec
       Install: false
       Build$: flag(tests)
       MainIs: testExec.ml
       BuildDepends: oUnit
     
     Test main
       Command: $test_exec
       TestTools: test_exec
     ...
The Run$: flag(tests) is implicit for the section Test main. The default value is false for tests. If all the executable for test are flagged correctly (Build$: flag(tests)), you'll get rid of the dependency on oUnit. It works the same for documentation, however the default is true. (Closes: #866) In order to allow interdependent flags, we transform back lazy values into unit -> string functions. This allows to change a flag value on the command line and to update all the dependent values. (Closes: #827, #938) It defines different ways to manage the auto-update of setup.ml: The choice between weak and dynamic depends on your need with regard to VCS and to the presence of oasis. The weak allow to checkout the project from VCS and be able to work on it, without the need of installing 'oasis' as long as you don't change the file _oasis. But it clutter your VCS history with changes to the build system each time you change something in _oasis. The 'dynamic' mode gives you no VCS history pollution but makes mandatory to have installed oasis libraries. Avoid copying executable to their real name. It helps to call ocamlbuild a single time for the whole build rather than calling it n time (n = number of executable sections) and copying resulting exec. This speeds up the build process because ocamlbuild doesn't have to compute/scan dependencies each time. The drawback is that you have to use $foo when you want to call Executable foo, because $foo will be '_build/.../main.byte'. For example:
CCOpt: -DEXTERNAL_EXP10 -L/sw/lib "-framework vecLib"
Will be parsed correctly and outputed according to target OS. In order to ease building oasis, we have minimize the number of dependencies. You only need to install ocamlmod, ocamlify and ocaml-data-notation for a standard build without tests. Dependencies on pcre, extlib and ocamlgraph has been dropped. The remaining dependencies are hidden behind a flag tests. OASIS now produces .cmxs file by default and add them to META. Now a META looks like:
     ...
     archive(byte) = "oasis.cma"
     archive(byte, plugin) = "oasis.cma"
     archive(native) = "oasis.cmxa"
     archive(native, plugin) = "oasis.cmxs"
     ...
This will ultimately help to generate automatically .cmxs for all oasis enabled projects. We hope that this new feature will improve dynamic linking use in OCaml (esp. for project like Ocsigen). Other changes Thanks to Anil Madhavapeddy, Pierre Chambart, Christophe Troestler, Jeremie Dimino, Ronan Le Hy, Yaron Minsky and Till Varoquaux for their help with this release. Also thanks to all the testers of the numerous release candidates. This was a long work and each time a tester has downloaded oasis has helped me to know that I was working for someone.

DebConf team: DebConf12 schedule ready! (Posted by Gunnar Wolf)

The DebConf12 schedule is finally available! As usual, the DebConf conference can only be finalized a couple of days before DebConf itself starts. But hey Take a look at it, and start planning your activities! Not final, never final Remember that, even now that the conference schedule is announced, you can still submit talks to DebConf, following what we call ad-hoc talks. What are the steps to do it?
  1. Check the schedule and find the timeslot (day, hour and room) you want to talk in
  2. Register your event in Pentabarf, and note the generated event ID (the last number in the URL after the submission is accepted)
  3. Send a mail to schedule@debconf.org detailing the event ID, title (just to confirm it), and the requested timeslot
  4. All requests _must_ be received at least 24hr before the requested event for the scheduling team to act accordingly and for the video team to be able to plan on resources to cover the requested session
  5. Video team coverage (event recording plus streaming on a best-effort basis) will be given to activities scheduled on the Aula Magna and Roberto Ter n auditoriums.
  6. We will try to grant requests on a first-come-first-serve basis. If two events are requested for the same timeslot, the first will have precedence, and we will schedule the second as close as possible to it. We will notify the submitters if this happens, of course.
Conference tracks In DebConf12, many talks are organized around the following tracks:
DebianDay
This track is meant for the local (Nicaraguan/Central American) community to be attracted to Debian. It does not usually have technically minded talks; the DebConf general audience is, of course, invited to take part in it, as there will surely be interesting angles on how Debian is used in different use-cases, and there will be interesting Q&A session with the audience.
Getting involved in Debian
Central America is a very special region. It has vibrant Free Software user groups and communities, among which there is an important interest to get involved in Debian.
This track will have some talks in Spanish, some talks in English, explaining different Debian processes and tools, aiming at attracting locals to contribute to Debian.
Debian for the cloud
All about virtualization. Management infrastructure, details on virtual machine engines, insight on how to tweak your kernel for optimum virtualized performance, and of course how to get it all humming perfectly with your Debian packages.
Building and porting
With an every day stronger transit to mobile computing, we are seeing a renaissance of porting. At the same time, we finally have multiarch as part of Debian. And were that not to be enough, we finally have more than one compiler that can be used for most of the software in the archive. Put all this together, and you will have a very interesting track.
Social activities
DebConf is as much about technical stuff as it is about building closer ties within the members of its community. Eating cheese+wine, going on a trip to the guts of a volcano, having a nice banquet Do you want to add an activity to this pseudo-track?
Skills exchange
Knowledge can also be built by all of us putting our grain of sand. We first had explicit skills exchange sessions in Bosnia last year, and they were very well received. Feel free to propose similar sessions!
And Well, three days to DebCamp and counting. See you Real Soon Now in Managua!

Sorina Sandu: Status update

Hello!

First of all: I'm going to DebConf :D and I really want to say a big "Thank you!" to every one who made this possible.


So, here's what I've done recently:

I spent some time putting everything in order for the departure to Nicaragua, things like finding a plane ticket through a route that does not require a Visa, making a new Passport, now all is in order.

I've integrated the changes for listing available ESSIDs, in order to close #637128 [0] into a separate branch , which can be found here [1].

The plan is to have only one new template which lists the available networks and which has a option for entering the ESSID manualy. In order to do so, a simple scan is performed on the wireless interface, and the configuration for a particular network resulted after the scan is kept if that one is chosen. The enter manually situation is treated exactly as it was before.

Furthermore, last week (or the week before) I've done some coding style fixes again in a separate branch [2], nothing very fancy, just trimming trailing white spaces and changing some tabs to spaces, where spaces were mostly used. I've also noticed some inconsistencies, regarding white spaces between functions names and the parenthesis. However I think I'm overreacting a little, those matters are not really that relevant, but when I'll have some free time I'll get to this as well.


Next in line: Issues

Testing is still giving me some troubles, the show essids part is not yet fully tested, so I couldn't say for sure if everything works as planned.

Apart from the classical: write the image to an USB and reboot method, I'm using Virtual Box for testing. I've managed to create a passthrough for the wireless interface, but now I have troubles with some firmware. Hopefully I'll solve this as soon as possible since it would really make my life easier.


Future plans:

I've realized that I've underestimated a little what the project implies and the amount of information that is new to me. I feel that I have learned a lot and that there is still a lot to learn and I'm very happy that it is so. I've now started documenting regarding IPv6, beginning with this [3]. We haven't decided yet what the next task will be exactly.


[0] http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=637128
[1] http://anonscm.debian.org/gitweb/?p=d-i/netcfg.git;a=shortlog;h=refs/heads/people/sorina/show_essids
[2] http://anonscm.debian.org/gitweb/?p=d-i/netcfg.git;a=shortlog;h=refs/heads/people/sorina/coding_style
[3] http://tldp.org/HOWTO/Linux+IPv6-HOWTO/

27 June 2012

Michal &#268;iha&#345;: Weblate - over 20000 translations in 3 months

Today, I wondered how much translations were done using Weblate since it was born. Of course I can give only numbers for services I maintain, the biggest one being translation server for phpMyAdmin, Gammu and Weblate. Looking at the logs, the server l10n.cihar.com helped to contribute over 20000 (actually little bit more as tracked are only changes after releasing Weblate 0.8), what is quite impressive number. Of course biggest amount of contributions came in time of preparing phpMyAdmin 3.5.0 release, where I gave it huge amount of publicity (AKA spamming my blog), but number of people accessing the website still grows since then (still number of submitted translations is slightly lower). Now as 1.0 release proved to be quite stable, it's time to focus on 1.1 development, which will bring offloading of fulltext indexing (it was proven to be unreliable to update index online for bigger sites). Almost all issues targeted for 1.1 are already fixed and all what needs to be done is testing in real life situations.

Filed under: English Phpmyadmin Weblate 0 comments Flattr this!

Next.

Previous.