Search Results: "Tom Marble"

2 November 2017

Bits from Debian: New Debian Developers and Maintainers (September and October 2017)

The following contributors got their Debian Developer accounts in the last two months: The following contributors were added as Debian Maintainers in the last two months: Congratulations!

10 August 2016

Tom Marble: webica

webica I've just pushed the first version of my new Clojure wrapper for Selenium called webica. The reason I need webica is that I want to do automated browser testing for ClojureScript based web applications. Certainly NodeJS, PhantomJS, Nashorn and the like are useful... but these can't quite emulate the full browser experience. We want to test our ClojureScript web apps in browsers -- ideally via our favorite automated continuous integration tools.

My new approach with the webica library is to do full Java introspection in the spirit that amazonica does for the AWS API. In fact I wanted to take it a step further by actually generating Clojure source code via introspection that can be used by Codox to generate nice API docs (which you don't get with amazonica). That, alas, was a little trickier than expected due to pesky Quine-like problems :) . If you load the library on the REPL you can get a feeling for each namespace by calling the show-functions function. I realize this approach of aggressive introspection, playing fast and loose with types and application level dynamic dispatch are crazy antipatterns. In my defense I started out playing around to see "if I could do it". After seeing the result in the form of a shell script in Clojure -- imitating lmgtfy -- perhaps webica will actually be useful! I plan to talk about webica tonight at -- hope to see you there!

24 July 2016

Tom Marble: jai-gagne-le-tour-de-crosstown-2016

J'ai gagn le Tour de Crosstown 2016! Everyone knows that today the finish line for Le Tour de France was crossed on Les Champs- lys es in Paris... And if you haven't seen some of the videos I highly recommend checking out the onboard camera views and the landscapes! Quel beau pays I'm happy to let you know that today I won the Tour de Crosstown 2016 which is the cycling competition at Lifetime Crosstown inspired by and concurrent to Le Tour de France. There were about twenty cyclists competing to see who could earn the most points -- by attending cycling class bien s r. I earned the maillot jaune with 23 points and my next closest competitor had 16 points (with the peloton far behind). But that's just part of the story.

Tour de Crosstown 2016
For some time I've been coming to Life Time Fitness at Crosstown for yoga (in Josefina's class) and playing racquetball with my friend David. The cycling studio is right next to the racquetball courts and there's been a class on Saturday's at the same time we usually play. I told David that it looked like fun and he said, having tried it, that it is fun (and a big workout). In early June David got busy and then had an injury that has kept him off the court ever since. So one Saturday morning I decided to try cycling. I borrowed a heart rate monitor (but had no idea what it was for) and tried to bike along in my regular gym shorts, shoes and a t-shirt. Despite being a cycling newbie I was immediately captured by Alison's music and enthusiasm. She's dancing on her bike and you can't help but lock in the beat. Of course that's just after she tells you to dial up the resistance... and the sweat just pours out! I admit that workout hit me pretty hard, but I had to come back and try the 5:45 am Wednesday EDGE cycle class (gulp). Despite what sounds like a crazy impossible time to get out and on a bike it actually works out super well. This plan requires one to up-level one's organization and after the workout I can assure you that you're fully awake and charged for the day! Soon I invested in my own heart rate monitor. Then I realized it would work so much better if I had a metabolic assessment to tune my aerobic and anaerobic training zones. While I signed up for the assessment I decided to work with May as my personal trainer. In addition to helping me with my upper body (complementing the cycling) May is a nutritionist and has helped me dial in this critical facet of training. Even though I'm still working to tune my diet around my workouts, I've already learned a lot by using My Fitness Pal and, most importantly, I have a whole new attitude about food. Pour les curieux, la nutritioniste maison s'est absent e en France pendant le mois de juillet. Soon I would invest in bike shoes, jerseys and shorts and begin to push myself into the proper zones during workouts and fuel my body properly afterwords. All these changes have led to dramatic weight loss \o/ A few of you know that the past two years have involved a lot of personal hardship. Upon reflection I have come to appreciate that things in my life that I can actually control are a massive opportunity. I decided that fixing my exercise and nutrition were the opportunities I want to focus on. A note for for my Debian friends... I'm sorry to have missed you in Cape Town, but I hope to join you in Montr al next year. So when the Tour de Crosstown started in July I decided this was the time for me to get serious. I want to thank all the instructors for the great workouts (and for all the calories I've left on the bike): Alison, Kristine, Olivia, Tasha, and Caroline! The result of my lifestyle changes are hard to describe.. I feel an amazing amount of energy every day. The impact of prior back injury is now almost non-existent. And what range of motion I hadn't recovered from the previous summer's being "washing machined" by a 3 meter wave while body surfing at the beach in Hossegor is now fully working. Now I'm thinking it's time to treat myself to a new bike :) I'm looking at large touring frames and am currently thinking of the Surly Disc Trucker. In terms of bike shops I've had a good experience with One on One and Grand Performance has come highly recommended. If anyone has suggestions for bikes, bike features, or good shops please let me know! I would encourage everyone here in Minneapolis to join me as guest for a Wed morning 5:45am EDGE cycle class. I'm betting you'll have as much fun as a I do.. and I guarantee you will sweat! The challenge in waking up will pay off handsomely in making you energized for the whole day. Let's bike allons-y!

29 February 2016

Tom Marble: is-slfc-shooting-open-source-in-the-foot

Is SFLC Shooting Open Source in the Foot? The academic article by SFLC about ZFS is troubling and may unintentionally shoot free software licensing in the foot. When I was at Sun (as part of the team that released the Java Programming Language by starting the OpenJDK project) I often heard community concerns about the CDDL license. At the time the big complaint was about the "Choice of Venue" clause. I got involved because Sun had developed many essential Java libraries and distributed them under CDDL. The community requested a more permissive license and I was able to convince internal project leaders (and Sun's lawyers) to make a licensing change for a handful of these projects. And there was much rejoicing. Based on my experience in helping Java to become open source I came to appreciate the legal hacks on copyright which make open source possible. It's the free software license which uses copyright to enable sharing (vs. the default of disabling sharing).

Open Source Licenses
And so I have appreciated many of the writings and speeches from SFLC on the mechanisms of software freedom. I was particularly moved by the talks about the "Freedom Box" concept. That's why this SFLC post on ZFS sounds so off key: if open source works because of free software licenses it seems weird to weaken that foundation by prioritizing the "equity" (or intended spirit) of the license. Allow me to mention that as I do most of my computing these days on GNU/Linux I miss the super cool features of ZFS from Solaris. I did try an early version of btrfs and was quite disappointed (but that's another story). In this happy case the source code for ZFS is available, but what about the future, when we aren't so lucky and someone asserts in court that the "you know, the software license was really about the spirit of sharing and that means we are allowed to use it -- and not be held to the pesky details as written in the license". A lawyer I respect called this out: "Equity" has no place in US law. The point is that for lawyers software licenses work because they have clear, written rules to guarantee the spirit is upheld; but spirit doesn't work in front of a judge -- clear rules do. Free and open source software has made so much progress in all facets of life why on earth would we second guess the licensing tools that made it possible? And why would SFLC try to shift the spotlight (and in this case the legal burden) to "a good-faith belief that the conduct falls within the equity of the license". Especially given the earlier comment which clearly states "[the combination] is inconsistent with the literal meaning of GPLv2 section 2(b)."

The entire raison d' tre for open source software licenses was so that developers (and users) would have clarity and wouldn't have to ask permission to use the software!!! As stated elsewhere (and like I did with those Java libraries) the easy solution is to have the ZFS copyright holder (now Oracle) reclicense (or dual license) the code under a compatible license (permissive or copyleft). If OpenSolaris was still a thing I might understand some hesitancy, but why not liberate ZFS now? So we have to wonder what could possibly be motivating this odd "spirit of the license" position on the part of SFLC? Fortunately charities that enjoy non-profit status are required to make public filings of their income in something called a "Form 990". The latest SFLC 990 I could find shows SFLC getting 78% (or just over $5 million) from "non public support" (see page 14). A number with "two commas" would even be interesting to for-profit companies. Just whom is making these "donations" and what exactly do they get in return? Apparently I'm not the only one wondering about this question. On one hand it's important to know if SFLC as a non-profit is, indeed, acting in the public interest (as the IRS requires). Yet the even bigger issue here is would "asking for a consensus about the spirit" trump the written copyright license and set a scary precedent for open source software in general?

29 January 2013

Hideki Yamane: upstream and distro has a "same" goal

Tom Marble has a nice blog entry "Crowdsourcing Upstream Refactoring". It's interesting and I agree with most, but I want to say against one thing in "conclusion" in his presentation.

He said "upstreams and distros have different goals" but I don't think so. We distro and upstream has a same goal, "Deliver the value to users", but we distro have much criteria than upstream, like license, non-duplicate library, etc.

If we (distro and upstream) cannot share this view, then we would be failed, IMHO.

14 July 2012

Christian Perrier: My own personal "Best Talk Award" for Debconf12

After attending several talks and BOFs at Debconf12, I'll grant my personal "Best Talk Award" to Hideki Yamane's Let's shrink Debian package archive!. Despite it being his first ever Debconf talk/BOF, Yamane-san did an incredibly complete research work to bring arguments about ways to reduce the size of the archive by using xz compression. He triggerred a very live discussion (and for this we can also thank all participants) and the quality of his slides was really high. So, for this, as Tom Marble said after the talk. You definitely deserve it and I'm proud to have you as teammate in the pkg-fonts team.

30 July 2011

Torsten Werner: DebConf11: Jigsaw Progress in Debian

Tom Marble and Guillaume Mazoyer gave a talk about Jigsaw Progress in Debian during DebConf11 in Banja Luka. Guillaume is a Google Summer of Code student mentored by Tom and Sylvestre Ledru. The project is about modularizing OpenJDK which replaces the old fashioned classpath by a modulepath that knows about versioned dependencies. It can reduce the memory footprint and the startup time for applications running in the JVM. It would be interesting to match Debian package versions to Jigsaw module versions. Debian could influence the JDK version 8 in this area. Upstream is already building Debian packages but they do not follow the Debian policy and do not use the Debian packaging tools. Guillaume has uploaded 2 packages (jtharness, jtreg) to Debian. These packages are needed to run the upstream testsuite which ships 3484 individual tests. More than 99% of them are working in Debian. There is a GIT repository of his Jigsaw work at;a=summary. The slides of the talk are available at

28 July 2011

Philipp Kern: caff harmful unless you know what you're doing

So there are two things I stumbled upon with caff:
Thanks to Tom Marble for the hint. I'm still sad that I'd basically need to re-do yesterday's keysigning (which was about 100 e-mails), just to switch from the default SHA1 to SHA256

21 July 2011

Stefano Zacchiroli: Test Driven Development in Debian

... or TDDD and DEP8 context As a nice byproduct of the huge "rolling" discussion we had back in April/May, various people have brainstormed about applying Test-Driven Development (TDD) techniques to Debian development. Here is a brief summary of some opinions on the matter: ... and hey, they've also coined the cool TDDD acronym, which I hereby took the freedom to re-target to Test-Driven Development in Debian. Having a cool acronym, we are already half-way to actually having the process up and running *g*. more testing I believe Debian needs more testing and I've been advocating for that since quite a while e.g. at DebConf10, as one of the main goals we should pursue in the near future. Of course advocating alone is not enough in Debian to make things happen and that is probably why this goal has been (thus far) less successful that others we have put forward, such as welcoming non-packaging contributors as Debian Project members. There are important reasons for increasing testing in Debian. quality assurance Quality Assurance has always been, and still is, one of the distinctive traits of Debian. I often say that Debian has a widespread culture of technical excellence and that is visible in several places: the Debian Policy process, lintian, piuparts, periodic full archive rebuilds, the EDOS/Mancoosi QA tools, the cultural fact that maintainers tend to know a lot about the software they're packaging rather than only about packaging, the we release when it's ready mantra, etc. But caring about quality is not a boolean, it's rather something that should be continuously cherished, refining quality requirements over time. By simply maintaining the status quo in its QA tools and processes, Debian won't remain for long a distribution who could claim to care about package quality. Others will catch up and are in fact already doing that. In particular, we surely have room for improvements in our quality tools and processes for: reducing inertia Inertia is a recurring topic among Debian lovers (and haters). It is often argued how difficult it is to make changes in Debian, both small and large, due to several (alleged) hindrances such as the size of the archive, the number of ports, the number of maintainers that should agree before proceeding with a cross-package change, etc. It's undeniable that the more code/ports/diversity you have, the more difficult is to apply "global" changes. But at least for what concerns the archive size, I believe that for the most part it's just FUD: simply debunking the self-inflicted culture about how "dangerous" doing NMUs is might go and has already gone, imho a long way to fight inertia. Adding per-package and integration tests will make us go another long way in reducing inertia when it comes to performing archive-wide changes. Indeed if a package you are not entirely familiar with has extensive test suites, and if they still pass after your changes, you can be more confident in your changes. The barrier to contribution, possibly via NMU, gets reduced as a result. And if your change will turn out to be bad but still not spot by the test suites, then you can NMU (or otherwise contribute) again to extend the test suite and make the life easier for future contributors to that package. It smells a lot like an useful virtuous cycle to me. autopkgtest / DEP8 how you can help Of all the above, the topic that intrigues me the most is as-installed package testing. Work on that front has been started a few years ago by Ian Jackson when he was working for Canonical. The status quo is embodied by the autopkgtest package. At present, the package contains of various tools and the following two specs:
  1. README.package-tests provides a standard format to declare per-package tests using the new debian/tests/control file. Tests come as executable files which will be run by the adt-run tool in a testbed where the package(s) to be tested is already installed. This part of the specs has been reified as DEP8 which I'm (supposedly) co-driving with Iustin Pop and Ian (for well-deserved credits).
  2. README.virtualisation-sever describes the interface among the test runner and the testbed. A nice separation is provided among the runner and the testbed, enabling different testbed environments with a varying degree of isolation: you can have a "null" testbed which runs tests on your real machines (needless to say, this is highly discouraged, but is provided by the adt-virt-null tool), a chroot testbed (adt-chroot), or a XEN/LVM based testbed (adt-virt-xenlvm).
The specs allow for several runtime testing scenarios and look quite flexible. The tools on the other hand suffer a bit of bitrot, which is unsurprisingly given they haven't been used much for several years. At the very minimum the following Python development tasks are in need of some love: If you are both interested in TDDD and grok Python, the above and many others tasks might whet your appetite. If this is the case don't hesitate to contact me, I'll be happy to provide some guidance.
Note: this post is the basis for the TDDD BoF that I will co-host with Tom Marble at DebConf11. If you plan to come, we will appreciate your thoughts on this matter as well as your help in getting the autopkgtest toolchain up and running again.

15 July 2011

Guillaume Mazoyer: Travel to DebConf 11

internet.pngMy flights tickets are booked so I guess:

I m really happy to go to Banja Luka. This will be my first DebConf and not my last I hope. I ll be there to meet incredible people that make Debian what it is but I will also give a talk with Tom Marble and Sylvestre Ledru about my Google Summer of Code project at Debian. This talk should be the 2011-07-30 at 15:00:00.

2 July 2011

Guillaume Mazoyer: Status report Jigsaw num. 3 for GSoC 2011

code.pngAt beginning of the last two weeks, I was working on the final package of JTReg. I was working on generating man pages and I finally found a solution. I decided to use help2man to generate the man pages but I had to patch the two scripts and . Basically, these two scripts launch main classes of the jtreg.jar JAR file. The problem with them is that they don t know where they can find the JAR file on Debian. So I patched the two scripts shell so they can locate JAR files in /usr/share/java .
To be able to generate man pages with help2man, I needed to be able to use the scripts. But with and using jtreg.jar which is not in /usr/share/java during the build, I had to find a way to make the scripts work. I decided to patch the scripts (again) and make them depend of some environment variables. So in each script, there are 4 used variables: So during the packaging only JTREG_HOME needed to be changed to be able to use the script. Once the scripts patch done, I finally wrote the rules file of the package. Once again, I used javahelper so the rules file contains the following code:

#!/usr/bin/make -f
JAVA_HOME = /usr/lib/jvm/default-java

ant -f make/build.xml
JTREG_HOME=./dist/jtreg/lib/ help2man \
--name="Regression Test Harness" \
--help-option="-help" \
./dist/jtreg/linux/bin/jtdiff > jtdiff.1
JTREG_HOME=./dist/jtreg/lib/ help2man \
--name="Regression Test Harness" \
--help-option="-help" \
./dist/jtreg/linux/bin/jtreg > jtreg.1

rm -r dist :
rm -r build :
rm jtdiff.1 :
rm jtreg.1 :

dh $@ --with javahelper

As you can see it is not really complicated. The most interesting part is the one when the man pages are generated. The JTREG_HOME variable is set to ./dist/jtreg/lib/ because it is the path where jtreg.jar is once it is built. I also used the help-option option of help2man because the help option of jtreg and jtdiff is -help and not help .
The jtreg package is now on the Debian Java team SVN so anyone can get it using:

svn checkout svn+ssh://$ SVN_USER

Thanks to Sylestre Ledru, jtreg is now available in Sid and the ITP is now closed.

The second part of my work consisted in testing Jigsaw and fixed tests if necessary. All the packaging work is now useful since the tests need JTReg to be run. The problem with the current build system of Jigsaw is that it is enough generic to be used on any system. So I had to patch several makefiles. To find them:

cd jigsaw-tests
find . -name 'Makefile' grep '/test/Makefile'

Then, the path to find jtreg has to be modified to match /usr/bin/jtreg . After that it is possible to run the tests with the following command:

make test

Passing all the tests is something which takes a lot of time. After that, I identified 42 (that s a cool number right?) failing tests. Some of them came from the lack of X11 server and other from unreachable network hosts. Some hackers of IcedTea gave me some ideas to fix several tests. They told me to look here and here because I just had the same problems that they had before.
With this, I was able to fix the tests related to unreachable hosts and to the lack of X server. I used Xvfb to fix the X server depending tests.
After running the tests again, only 13 tests were still failing. That is a good result. Tom Marble and me decided to apply the patch of Alan Bateman and re-run the tests to see if it will break something:

cd jigsaw-tests
mkdir patches && cd patches
cd jdk
patch -p1 < ../patches/jdk.patch
make clean
make sanity
make all
make modules
xvfb-run -e xvfb-errors -a -s -ac make test -k

15 tests failed against 13 before. The 2 new failing tests are: I didn t go further but I ll try to understand why we have 15 failing tests in few days. All the modifications that I have made on the source code of Jigsaw can be found in a patch format here:

This 2 weeks were also dedicated to the understanding of Jigsaw and its modules. Reading the jigsaw-dev mailing list, I have found some interesting conversation about jigsaw and how to write modules. So I decided to write a module to see how it works. I followed the instruction of the quick start guide but when compiling the module didn t work. So I tried to understand why and then to fix it. I eventually compiled and installed my module and here is how I did it:

mkdir -p src/com.greetings/com/greetings/
mkdir -p src/org.astro/org/astro/
mkdir modules
vim src/com.greetings/
vim src/com.greetings/com/greetings/
vim src/org.astro/
vim src/org.astro/org/astro/
./jigsaw/build/linux-amd64/bin/javac -d modules \
-modulepath ./jigsaw/build/linux-amd64/modules/modules \
-sourcepath src find src -name '*.java'
./jigsaw/build/linux-amd64/bin/jmod create -L mlib
./jigsaw/build/linux-amd64/bin/jmod install \
modules org.astro com.greetings -L mlib
./jigsaw/build/linux-amd64/bin/java -L mlib -m com.greetings

And here it is, we have a nice hello world module. The documentation says to put the of org.astro in src/org.astro/org/astro/ but it didn t work for me so I tried to put it in src/org.astro/ and it worked. Also the quick start guide says that we have to use the -modulepath option to specify where the modules are. I first tried to use ./jigsaw/build/linux-amd64/modules/ but javac told me he could not find java.lang so I noticed that there is a subdirectory called modules in ./jigsaw/build/linux-amd64/modules/ so I used it as module path and it worked.

Writing a module and seeing how Jigsaw modules are made started to make me think about the packaging of Jigsaw. Dependencies being available, knowing how to build and test Jigsaw and seeing how modules are made, I think that I will soon try to start the packaging of Jigsaw and getting started with the packaging will be my goal for the next two weeks.

18 June 2011

Guillaume Mazoyer: Status report Jigsaw num. 2 for GSoC 2011

code.pngMy internship is now finished, this means that I will be able to work on my GSoC project a lot more than the previous weeks. I hope to make some good progress in the next days.

This last 2 weeks, I was still studying how Jigsaw works. Even if there is no progress on the Jigsaw packaging I have done some packages that will help later.

I ve spend a lot of time trying to package JT Harness and JTReg.
I ve successfully built JT Harness using CDBS but I ve finally rewritten the package using only debhelper and javahelper. Javahelper was really useful to build JAR files, move them to the right directory, put the Javadoc in the right place as well as the examples. The Debian directory of JT Harness can be found on the Java team SVN.

svn co svn+ssh://$ SVN_USER

The source package of jtharness build 2 binary packages: libjtharness-java and libjtharness-java-doc. You can see that by looking at the control file in the debian directory. There are some interesting files in this package, let s take a look at them.

First of all, some of the source files needed to be patched for the package to build. So to accomplish this, there is a couple of patches in debian/patches : To generate these patches, I ve used the edit-patch command which is a really awesome tool. This command uses a temporary shell where we can edit files we want to patch. When the job is done, CTRL + D terminates the temporary shell and the patch is generated.

There are 4 interesting files which are used to build the binary packages and put the files in the right directories. These files are: Of course rules is the most important one. This file tells us how packages are built. A good thing is that rules is quite simple in the jtharness case.

#!/usr/bin/make -f
JAVA_HOME = /usr/lib/jvm/default-java

ant -f build/build.xml

dh $@ --with javahelper

We just tell where Java is and to use javahelper ( with javahelper). Overriding dh_auto_build is interesting because since the JT Harness uses Ant as build system we need to give to Ant where the build.xml file is. By default Ant is searching it in the current directory but in jtharness the file is in the build directory. Since the Ant build file is correctly written, the build generates JAR files, Javadoc and examples code. And libjtharness-java* files are here to put the generated things in the right places.
The job of libjtharness-java.jlibs is to tell where to find the JAR libraries. In this case, the intesresting JAR files were javatest.jar and jt-junit.jar . So in the .jlibs file we just give where those files are after being built.
The job of libjtharness-java-doc.javadoc is to tell where the Javadoc is and where to put it. So there is only one line that says the generated Javadoc is in jar-build/javadoc and put it in /usr/share/doc/libjtharness-java/api .
And finally, libjtharness-java-doc.install is used to move the examples. The syntax is same as libjtharness-java-doc.javadoc .

The JT Harness package can be found in Sid now thanks to Sylvestre Ledru who uploaded it for me. So the ITP bug that I created is now closed.

Building the package for jtreg is quite similar. In fact, it probably easier but I m still facing a problem. jtharness is just a library, jtreg does have main classes and generate some binaries. The difficulty here is to generate the man pages for the 2 generated binaries: jtdiff and jtreg. These 2 files are shell scripts which just use jtreg.jar with the correct parameters.

I tried several tools to generate the man pages of each script. Sadly, I can t find a way that does not causes any problems.
My first choice was to use help2man. This tool looked like to be perfect for the job. But I couldn t find how to launch the scripts correctly. The scripts needs jtreg.jar but they also need javatest.jar from the libjtharness-java package. The classpath cannot be set correctly due how the scripts are written. So to use help2man I ll probably have to patch and completely rewrite and in the sources.
My second choice was to use txt2man. During the build of jtreg there is a usage.txt file which is generated and which can be used as a man page. txt2man can take this text file and output a man page based on the text. I thought it was working but when building the package, lintian threw some warnings about syntax errors in the man pages.

W: jtreg: manpage-has-bad-whatis-entry usr/share/man/man1/jtdiff.1.gz
W: jtreg: manpage-has-errors-from-man usr/share/man/man1/jtdiff.1.gz 292: warning [p 4, 3.3i]: cannot adjust line

Finally, few days ago I decided to try rst2man. This tool looks cool and easy to use. But the text that I have to convert into a man page is not a reStructured text so it cannot without patching the usage.txt file for example. I was going to do it but there is no such file in the sources of jtreg. Actually this file is generated during the build but the text had to come from somewhere. And it was from two properties files in the source code.
I m still looking for a good solution to generate the man pages. If someone has any idea just let me know ;-)

The debian packaging of jtreg is not in the Java team SVN yet because of the man pages problem. I opened an ITP bug so people know that someone is working on jtreg.

My goal for the next two weeks is to finish the jtreg package, understand how jpkg (which is supposed to create Debian packages) works in Jigsaw and write more examples using the modular JVM.

8 June 2011

Lars Wirzenius: Test driven distribution development

Disclaimer: There is a huge discussion going on right now in the debian-devel mailing list, about different ways to rearrange how Debian development happens, and maybe to provide a Debian users with an additional release. This blog entry is not a comment on that discussion, which I am ignoring. The discussion did, however, prompt a dent about "test driven development". That prompted a short exchange of thoughts with Tom Marble, and this blog post is a cleaned up version of my part in that discussion.
Currently Debian development happens roughly like this: all packages are uploaded to a part of the Debian archive called unstable. Once they've been in use for a while without serious problems, they get automatically copied into another part called testing. Every year or two, testing is frozen and any remaining problems are fixed, after which all of testing gets copied into yet another part of the archive called stable, and that's the actual release. There is a little bit of automatic testing, but only a little. Almost all testing is done by people, by them using unstable or testing on their usual computers. If they have problems, they report them. Contrast this with development using Test Driven Development, or TDD, and other modern development methodologies. Here's a rough summary of what I do when I write (much) of my software. In addition, I measure coverage to make sure all parts of the code gets tested. I usually aim for 100% coverage, except for those parts that very hard or quite pointless to test. (That's easier to achieve than you'd think, but that's the topic of another blog post.) This all sounds like a lot of bureaucratic nonsense, but what I get out of this is this: once all tests pass, I have a strong confidence that the software works. As soon as I've added all the features I want to have for a release, I can push a button to push it out. (Well, actually there's a little bit more to it.) This does mean I never have bugs in my software. Of course I do. However, there's a lot fewer of them. In fact, there's so few of them, after tests pass, that I would almost be happy to make an automatic release after every successful run of "make check". Another aspect of the way I do development is distributed version control. The relevant feature is powerful branching and merging. Most things I develop happen in short-lived single-purpose branches: whenever I start work on a feature or bugfix, I create a branch for it. (Unless I'm feeling particularly lazy, which happens more often than you'd think.) When I've finished the feature, or fixed the bug, I merge the branch back into the trunk branch. This way, the trunk is always in releaseable state. It might not have all the features I want for the next release, but the features that are there, do work. The branch probably has bugs, but if I've written tests that are good enough, I know the software works well enough. Or that's the idea. Sometimes things go wrong. Then I write more tests and the next time goes better. (It doesn't have to be perfect, as long as it gets better every time.) See the contrast between automatic tests with good coverage, and the Debian style of relying on user feedback? There's no need to wait for user feedback with automatic tests. This speeds up development, makes releasing easier, but most importantly takes away any reason to fear making changes, even big changes. Automatic tests and good test coverage are easy achieve in small projects. For a system as huge as Debian, good test coverage is quite hard to achieve. The thing about automatic tests, though, is that even a little bit of it is helpful, and after you have a few tests, it gets easier to add more. The first test is the hardest to get done, since you need to set up all the infrastructure for it. Debian does do a bit of automatic testing already. (See lintian, piuparts, autopkgtest, edos, etc.) I don't want to belittle that, but I think we could do better. Here's what I would love to see: Since full test coverage is going to be impossible for Debian, some subset should be targeted. Perhaps something like this would suffice to start with: These tests would not guarantee that a set of changes would not break Debian, but they would give a high confidence that at least basic stuff would still work after the changes. Now, obviously implementing all of this I'm dreaming of (it is just past midnight, after all) is going to be impossible. There's way too much work to do, there's not enough tools for writing tests, and it would require too much computing power to run, and so on and so forth. But it's late, and I've had a bad day, and I might as well dream of something nice. Anyway, anyone interested in these things should perhaps help drive DEP8.

3 June 2011

Guillaume Mazoyer: Status report Jigsaw num. 1 for GSoC 2011

code.pngThe beginning of this Google Summer of Code has been busy and pretty cool. Not a lot of code was produced but I did learn a lot of things.

During this GSoC I will work on the next generation JDK. This means that I will (and I already have) compile the JDK. That s pretty awesome but it requires a lot of power. That s why Tom Marble created me an account on his 8 CPUs machine to help me.
Since I m probably going to sign some packages, I created a new GPG key with a 4096 RSA strength.
My old key was:

pub 1024D/F144A319 2008-10-18

And the new key is:

pub 4096R/EE2BBBC7 2011-05-03

I cross signed my keys using this great tutorial.

I read some documentation about Jigsaw and watched Tom s talk at DebConf 10.
I have spent a little more than a week to understand the current OpenJDK packaging to be able to rebuild it. I have learnt from where do the sources come from. The current packaging is a little hard to understand. There are sources that come from IcedTea, OpenJDK, and more.

After being able to rebuild the OpenJDK 6 package, I have downloaded the source of the OpenJDK 8 and Jigsaw. I used the Mercurial forest to get the sources.

For OpenJDK 8:
hg clone jdk8
cd jdk8 && sh

For Jigsaw:
hg clone jigsaw
cd jigsaw && sh

I have managed to build the sources of OpenJDK 8 and Jigsaw using the same steps:
cd jigsaw
. jigbuild
. jdk/make/
make sanity
make all

Jigsaw specific:
make modules-sanity
make modules

Each build took several minutes. But they were successful. Tom helped me a lot to build OpenJDK 8 and Jigsaw by giving me a solution. For example, after getting the Mercurial forest, there were still missing source files and the build failed. Tom told me that I needed to export a environement variable: ALLOW_DOWNLOADS=true.
After building Jigsaw, I was able to see that the version string is not legal in Debian and that is a real problem.
I have written some simple Hello World examples to compile and run with Jigsaw. To modularize a program, it needs to contain a files. This file includes the modules needed to run the program and the main class to run. For a classic hello world, the files looks like:

module test.hello @ 1
requires jdk.base @ 7-ea;
class test.hello.Main;

Tom gave me some packages to build: jtharness and jtreg. I succesfully built jtharness using CDBS and its Ant helper but I ll have to repackage it to use javahelper instead of CDBS. I still didn t try to build jtreg because it requires jtharness.

My next plan is to get jtharness and jtreg packaged with javahelper soon and understand what modules does what in Jigsaw. Understaning modules will help me to write some more complex examples. I hope to start working on Jigsaw a little more soon (when I my internship will end [next week]).

25 May 2011

Guillaume Mazoyer: Here is a new GSoC student

java.pngHello Planet Debian reader.

And now you re thinking, who is this crazy guy who does not even know how to write English (is my sentence wrong?). My name is Guillaume Mazoyer and I am the Google Summer of Code student working on Jigsaw with Tom Marble and Sylvestre Ledru. I blog from France and I plan to write my GSoC reports right here.

Maybe you don t know what is Jigsaw yet. If so, go read this wiki page, you should be able to understand the goal of my project. Here is the description:

Since the beginning of Java, the JVM is a big virtual machine will all the features incorporated in it. This JVM provides many features which are useful in only limited areas (Corba) or which are not used at all in some contexts (AWT/Swing are useless for a web server).

The goal of the jigsaw project is to provide a modular JVM.

This will provide many improvements in the performance area (startup, solving of the dependencies, etc) but also the size of the dependencies.

There are already some works done by upstream (Oracle) but it does not complain to the Debian standards. The target is Java 8.

The goal is this Google Summer of Code project is to package the current development version of Jigsaw into Debian.

Please note that this project is not only about packaging. This will probably implies some developments and contributions to upstream (upstream is aware of this effort). Updating javahelper to manage specifities of Jigsaw could be considered.

I m pretty excited to work on this project and the work has already started. Can t wait to go deeper and enjoy working on two things that I love: Debian and Java.

8 May 2011

Stefano Zacchiroli: debconf11 and you

DebConf11 program, the FOSS way
I'm going to DebConf11 and, once more, I'm thrilled at idea of living for about 2 weeks immersed in a very very very, very Debian-ic life. DebConf10, DebConf edition of last year, has been for me one of the most intense DebConfs ever, it would be hard to match it. But one of the nice things about DebConf, no matter the edition, is that it's very "open source" and do-ocratic in they way it is organized: everyone could contribute into making it even better than the previous edition. For one thing, there is no "event organizer" company behind DebConf. Rather, most of the hard work that goes into making a DebConf possible is done by the DebConf team, a volunteer team formed by a mixture of Debian contributors, local team members, and DebConf enthusiasts. But while they take care of most of the work and hence deserve the thanks of all DebConf attendees for that they do not account for all that makes a DebConf a great DebConf. For instance, having a great program is up to DebConf participants and the event proposals they submit: talks, BoF, panel discussions, film projections, contests, you name it. Today, in deadline frenzy mode (read on), I've submitted for review the following events for DebConf11: That's it from me, how about you?
Are you going to DebConf and willing to attend a conference with a great program? Great, then start submitting a great event yourself! Anything you are working on in Debian or you are excited about qualifies. And don't worry if you've never attended DebConf either, video archives from past DebConfs can give you more than a feeling of typical DebConf events. While the first deadline for events submission expires today, submissions will be open all the way to the end of the conference, but you should better hurry up as the early birds get the best scheduling slots (and missing the deadline, you'll miss the round of the review for the main program ... but you still have a few hours left :)). Good luck and see you in Banja Luka.

25 April 2011

Obey Arthur Liu: Welcome to our 2011 Debian Google Summer of Code students!

I d like to extend a warm welcome to our new batch of students selected for the 2011 Debian Google Summer of Code! They should soon be posting on Debian Planet and you re welcome to come talk to them on #debian-soc on Further details will be posted in the coming days to our wiki: Automated Multi-Arch Cross-Building and Bootstrapping aka autocrossbuild , by Gustavo Prado Alkmim, mentored by Wookey
Enable easy and automated setup of cross-platform automated build systems and bootstrapping for QA in the Multi-Arch era. This involves the creation of multi-stage bootstrap build sequencing tools and a reliable automated multi-arch cross-builder. APT/Dpkg Transaction Ordering for Safety and Performance aka aptordering , by Chris Baines, mentored by Michael Vogt
The ordering code in libapt is responsible for ordering the unpacking/configuration of debs so as to ensure dependencies are satisfied etc. Currently it organizes the ordering into big batches. This project further implements an ordering satisfying more constrains such as minimal amounts of dpkg invocations , minimal amount of broken packages at any point . DebDelta APT Native Integration aka debdelta , by Ishan Jayawardena, mentored by Michael Vogt
Improve user experience of APT and its front-ends by speeding up the upgrade process. This provides a better framework for unified handling of debdelta and future APT improvements such as parallelism. Support for stable and security ugprades as well as multiple APT related libraries is expected. Dpkg Declarative Diversions aka declarativediversions , by Sam Dunne, mentored by Steve Langasek
The dpkg-divert command should be replaced with a new control file with a declarative syntax which Dpkg will parse and process directly as part of the package unpack and removal phases, eliminating the problems resulting from non-atomic handling of diversions. Backend Tools and Infrastructure for DEX aka dextools , by Nathan Handler, mentored by Matt Zimmerman
EX is a new program designed to help improve Debian and its derivatives by merging in changes made downstream and encouraging discussions between the various projects. As this is a new project, most of the infrastructure does not exist (or is rather hackish and incomplete). This project will create the necessary backend tools and infrastructure so that all Debian derivatives can easily make use of the DEX project. Jigsaw Modularized Java in Debian aka jigsaw , by Guillaume Mazoyer, mentored by Tom Marble
The Java Development Kit (JDK) is a big monolithic software tool: many of its features are only useful in limited areas (GUI toolkits are useless for a web server). This project will bring the Jigsaw modular JDK to Debian, helping performance (start-up, size, etc) but also the dependency resolution (to match Debian packaging). Some work exists upstream does not fit with Debian. This project will package the current development version of Jigsaw, update Debian Java Policy, and create the necessary packaging tools for software depending on it. Python Multi-Build for Python Extensions Packaging aka pythonmultibuild , by Mesutcan Kurt, mentored by Piotr O arowski
This project creates a tool to build Python extensions for all Python versions supported by Debian at the time. The project should detect the upstream build system and testing frameworks and use them. It will be interfaced with CDBS and the dh sequencer, replacing their Python snippets. Debian Teams Activity Metrics aka teammetrics , by Sukhbir Singh, mentored by Andreas Tille
This project will gauge the performance of teams in Debian by measuring metrics such as: postings on relevant mailing lists, package upload records from the Ultimate Debian Database and commit statistics from project repositories The information gathered will help in evaluating team performance by measuring how people in a team are working together. An interface to access this information easily will also be developed. Compute Clusters Integration for Debian Development and Building aka computeclusters , by Rudy Godoy, mentored by Steffen M ller
The project s main goal is to enable developers to easily use compute clusters (Eucalyptus, OpenStack ) as environments for arch-specific development by providing a set of tools they can use to setup and run an extended platform for their development, testing and building tasks. Good luck to everyone!

20 July 2006

Arnaud Vandyck: Everybody wants to open source

It’s strange to follow state of minds of big companies about open source (they rarely speak about Free Software but that’s another story;-)). Dalibor Topic, Mark Wielaard and all friends from GNU Classpath spent a lot of time talking with Sun to try to open source Java (or even just be able to get the tools to make open source java virtual machines and API). They are some moves in that direction and people like Tom Marble is working hard to make Sun’s JDK license DFSG compatible and integrate Sun’s JDK in the Debian distribution. Robert Brewin, co-CTO of Sun Microsystems’ software group, said: “I believe that we will have components of Java released into open source within the year,” meaning by next June, Brewin said. “I think [release of] the whole thing will take a little bit longer.” One more step. Another amazing interview is the one of David Kaefer, director of Business Development, Intellectual Property and Licensing at Microsoft who called the open source software movement a “very powerful force in the industry.” He also wants us to believe that Microsoft will be more and more open… Wait and See ;-)

16 May 2006

Barry Hawkins: A bit chilly in a hot place; a distributable Java JRE and JDK arrives

So it’s finally OK to mention it now; Java has made it explicitly possible (read legal) to distribute the Sun Java JRE/JDK on a GNU/Linux distrubtion. The new license is for Java SE 5 on Linux only, called the Operating System Distribution License for Java, or DLJ for short. You can read the license in text or pdf form. The FAQ for the DLJ is also available in text and pdf. Heck, go through the README for the JRE and JDK while you’re at it. So what does that mean? Well, GNU/Linux distrubtions like Debian can now package a Java runtime environment or Java development kit in their repositories. That was previously not possible due to restrictions present in Java licensing. Users still have to accept the Java SE 5 binary code license that is totally not free and has the same restrictions Java has always had, but this at least makes packaging and supporting Java less painful for distributions. Sun is coordinating the efforts via a project, jdk-distros. This is an unprecedented level of cooperation from Sun with external parties in anything related to Java. I consider myself fortunate to have been a founding member of the project. It has been a pleasant and refreshing experience to meet a few optimistic and forward-thinking people from Sun who have a keen interest in Free Software; a big thanks to Simon Phipps and Tom Marble. I was encouraged that they allowed our contributions to be covered under the MIT license. If you would have told me that a month ago I would have laughed at you. The Debian announcment should be posted on the debian-devel-announce list today. I am sure this will draw both praise and ire from the Debian community. That’s cool, though; the rich diversity is part of what makes it such a vibrant organism.