Search Results: "tar"

12 December 2011

Christian Perrier: Defoma removal

Thanks to the longstading work by Paul Wise in nearly 3 years...and with some help from my side (and from other members of the font packaging team), we're getting closer of a longstanding goal : removing Defoma from the archive. Defoma is indeed obsoleted by fontconfig, which doesn't need these "hints" file, turning the packaging of fonts into a very simple process. On last Saturday, Paul sent the removal request for defoma. The request mentions blocking bugs that are, for most of them, requests for font packages to drop the dependency on defoma. As we already sent patches for these bugs, I started NMU'ing the remaining packages this week-end. Most of them were apparently nearly abandoned packages, so I had no shame doing that so quickly. Indeed, I no longer fear NMU'ing packages even for non critical issues and I even sometimes no longer fear adding a few other fixes when the packages are very obviously poorly maintained (things like source v3, debhelper compatibility bump, build- indep arch targets support, etc.). After this NMU wave, we should be left, in one week, with only five packages blocking the removal of defoma. I might even NMU some of them, though some are TeX-related packages providing T1 fonts additionnally to TrueType ones...something I'l less familiar with. Whatever happens, I now nearly sure that wheezy will not have defoma. Yay! Hats off to Paul.

Cyril Brulebois: Fixing synaptics on powerpc

(It s a new day, it s a new bug ) A regression appeared in the synaptics input driver, between the 1.4.1 and the 1.5.0 release, as reported in Debian #649002 by many users. Basically, no more touchpad on PowerPC (hi, iBook G4 users!). :-( What follows is how I tracked it down. The upstream bug report is fdo#43728. Anyone could have done so, there s no black magic involved. A little hint maybe: git bisect is really easy on X drivers, since one can build and test without the Debian packaging bits. 1st step: Getting started You need sources and build dependencies, nothing fancy. Since packages might be named a bit differently, there s a reference to the relevant upstream repositories in debian/watch for X packages.
# You need git here:
debcheckout xserver-xorg-input-synaptics synaptics.git
cd synaptics.git
# Let s get the upstream tags and branches:
cat debian/watch
git remote add upstream git://anongit.freedesktop.org/xorg/driver/xf86-input-synaptics
git fetch --all
# Let s install build dependencies:
sudo apt-get build-dep xserver-xorg-input-synaptics
2nd step: Reproducing It was said 1.5.0 was the first known buggy version, so let s make sure:
# Get the appropriate tag:
git checkout xf86-input-synaptics-1.5.0
# Prepare the build system:
autoreconf -vfi
# X finds its modules under /usr, not /usr/local:
./configure --prefix=/usr
# Build and install:
make && sudo make install
Now time to restart the display manager, and confirm that the pointer is not moving bug reproduced. Now let s check the 1.4.1 release isn t affected:
# Clean everything:
git clean -xdf
git checkout xf86-input-synaptics-1.4.1
autoreconf -vfi
./configure --prefix=/usr
make && sudo make install
And after a display manager restart, the pointer is moving again, good! For reference, the log has very different lines about appletouch, so it can be used to programmatically learn about the status (good or bad). 3rd step: Narrowing it down So, we have a known good version and a known bad version. Time for a binary search to find the guilty commit which introduced that regression!
git bisect start
git bisect good xf86-input-synaptics-1.4.1
git bisect bad  xf86-input-synaptics-1.5.0
You don t even need to know that 1.4.x and 1.5.x live in different branches, git figures that out for you:
Bisecting: a merge base must be tested
[de0dfb76444ad4160468d00515876c91a9fa20bf] synaptics 1.4.0
It s just a matter of running autoreconf && ./configure && make && sudo make install && sudo /etc/init.d/xdm restart every step of the way, so it s pretty easy. No wonder, 1.4.0 was working OK, so it can be recorded as such through git bisect good, and the binary search goes between that commit and 1.5.0. After a few iterations of git bisect good and git bisect bad, we get to the commit reported on the upstream bug. As an extra bonus, mentioning the upstream bug number/URL in the Debian bug means it can be marked as forwarded there and we ll receive automatic notifications when its status changes, thanks to bts-link. Now, if you want to provide a patch for this bug, you may want to try and revert this commit. 4th step: Providing a patch Let s go back to the affected release and try to revert the guilty commit:
git checkout xf86-input-synaptics-1.5.0
git revert 13543b156d78bc4d01a19844a5ee8f283269621b
Unfortunately, some later commits modified the affected areas so we re running into conflicts:
error: could not revert 13543b1... eventcomm: replace synaptics-custom TEST_BIT with server's BitIsOn.
hint: after resolving the conflicts, mark the corrected paths
hint: with 'git add <paths>' or 'git rm <paths>'
hint: and commit the result with 'git commit'
Let s try to understand what the offending commit did: it replaced a set of macros with a single one, taken from the server. Both the old TEST_BIT() and the new BitIsOn() take 2 arguments, but the order is reversed. Reverting it should only be a matter of introducing the TEST_BIT() macro again (along with the two other ones it needs), and using it. The problem is replacing all BitIsOn(foo,bar) with TEST_BIT(bar,foo): foo and bar might not be trivial, they could be complex C expressions, and a replacement in your favourite editor might not get all occurrences right. Thankfully, there s a nice tool to handle such things, called coccinelle. I won t go into details, just show how it can help here. Basically you just need to craft a tiny patch, called a semantic patch, which describes what I wrote in the previous paragraph. You only need to declare two expressions, say a and b, and declare that all BitIsOn(a,b) must be replaced with TEST_BIT(b,a). Here s the syntax:
@@
expression a,b;
@@
-BitIsOn(a,b)
+TEST_BIT(b,a)
Save it under testbit.cocci and apply it through spatch (from the coccinelle package), asking it to perform in-place replacement (we re in a git checkout copy, remember):
# Apply:
spatch -sp_file testbit.cocci -in_place .
# Profit:
git diff
Then you can run git commit -s, write a nice commit message, send it upstream, and enjoy the rest of the night. Coming soon to a mirror near you: fixing commit. As mentioned on the upstream bug report, I got the Debian reference wrong in the commit message, I really meant #649002.

11 December 2011

Stefano Zacchiroli: bits from the DPL for November 2011

Mako's IronBlogger is a great idea. I often find myself postponing blog posts for a very long time, simply out of laziness. IronBlogger provides a nice community incentive to counter (my) laziness and blogging more often. As a related challenge, we have to face the fact that different subsets of our communities use different media to stay informed: mailing lists, blog (aggregators), social media, IRC, etc. Disparities in how they stay informed are a pity and can be countered using multiple medias at a time. Although I haven't blogged very often as of lately, I managed to keep the Debian (Developer) community informed of what happens in "DPL land" on a monthly basis, by the means of bits from the DPL mails sent to d-d-a. While the target of bits mails perfectly fits d-d-a, there is no reason to exclude a broader public from them. After all, who knows, maybe we'll find the next DPL victim^W candidate among Planet readers! Bonus point: blogging this also helped me realize that my mails are not as markdown-clean as I thought they were. I still have no IronBlogger squad, though. (And sharing beers with folks in the Boston area is not terribly handy for me ). Anyone interested in setting up a BloggeurDeFer in the Paris area? (SCNR)
Dear Project Members,
another month has passed, it's time to bother you again about what has happened in DPL land in November (this time, with even less delay than the last one, ah!). Call for Help: press/publicity team I'd like to highlight the call for help by the press / publicity teams. They are "hiring" and have sent out a call for new members a couple of weeks ago. The work they do is amazing and is very important for Debian, as important as maintaining packages or fixing RC bugs during a freeze. It is only by letting the world know what Debian is and what we do, that we can keep the Project thriving. And letting the world know is exactly what the publicity and press teams do. If you're into writing, blogging, or simply have a crush for social media, please read the call and "apply"! Interviews November has apparently been the "let's interview the DPL" month. I've spent quite some time giving interviews to interested journalists about various topics. For both my embarrassment and transparency on what I've said on behalf of Debian, here are the relevant links: Assets Legal advice (work in progress) Relationships with others Miscellanea Thanks for reading thus far,
and happy hacking.
PS as usual, the boring day-to-day activity log is available at master:/srv/leader/news/bits-from-the-DPL.*

10 December 2011

Patrick Schoenfeld: Migrating from blogspot to Movable Type

A while ago I decided to migrate my existing blogspot blog to an own domain and webspace again. My reasoning was mostly, that blogspot lacked some features, which I'd like to have in my blog.
Additionally, my requirements have changed a bit since I originally moved to blogspot and, last but not least, Blogspot was a compromise anyway.

So I started re-evaluating a possible software platform for my blog. In my numerous previous attempts to start blogging (there were several blogs of me in the internet since at least 2006), before I moved to Blogspot, I used Wordpress. But there were quiet some reasons against it, one of the biggest concerns being its security history. Also, while I worked a lot with PHP in the past years, I have developed a serious antipathy against software written in PHP, which I couldn't just ignore.

In the end, the decision fell on Movable Type, because its written in Perl, which is the language I prefer for most of my projects, because its features were matching my wishes (mostly) and because I heard some good opinions about it. Also it is used by my employer for our company blog.

So the next question was: How to migrate?

I decided to use Movable Type 5, although, at present, it seems not to be the communities choice. At least the list of plugins supporting MT5 is really short. Foremost there was no plugin to import blogger posts, which, after all, was the most important thing about the migration.
Luckily there is such a plugin for Movable Type and so I basically did the following:

  1. Install Movable Type 4
  2. Install the Blogger Import Plugin
  3. Import posts (it supports either the export file of blogger or directly importing posts via the Google API)
  4. Upgrade to Movable Type 5
  5. Check the result
Check the results, or: The missing parts

Obviously such an import is not perfect. Some posts contain images or in-site links. The importer is not able to detect that and honestly it would have a hard time to track that anyway.
So as soon as content is migrated, its time to look for the missing parts.

The process to find missing parts is basically very easy and common among the various missing parts:
Just search for your blogspot URL via the Search & Replace option in the Movable Type administration.

Now how to fix that? For links its quiet easy (although I forgot about them in the first run), as long as the permalinks have kept the same scheme.
In my case that is the case, since I decided to use the Preferred Archive option "Entry" in the blog settings for the new blog and the default (if there is an option, because I don't know)
in Blogspot. The importer does import the Basename of the document, so fixing links is just a matter of replacing the domain part of the URL.

For images its some more work. One has to get the images somehow and upload them in Movable Type. Eventually it then boils down to search and replace, but I decided to do that manually, since I only have a very low number of images in my posts so far.

After that I did everything else, which is not specific to the migration, like picking a template, modifying it to my wishes, considering the additions of plugin etc.
And here we are. There were some issues during the migration, which I haven't handled here. I will blog about them another time.

9 December 2011

Andrew Pollock: [life/americania] Six years in the US

Late last month our six year anniversary of moving to the US quietly came and went. This also makes it six years with Google. I think if you had have told me at the start of 2005, that I'd be ending the year off by starting a six year (and counting) stint in Silicon Valley, I wouldn't have believed you. Six years is comfortably the longest I've spent in any one place since I moved out of home (the four years living at the Central Park Apartments was the longest I've spent in any one home, also). Six years is also far and away the longest time I've stayed with one employer. I wonder what life will be like in another six years' time?

C.J. Adams-Collier: xend xml parsing failure solution

I got this error when starting xend after a filesystem crash: [2011-12-09 11:05:04 5590] ERROR (SrvDaemon:349) Exception starting xend (not well-formed (invalid token): line 3, column 0) It sounded like an XML parsing error, but the stack trace in the log didn t say which file it had loaded. It turns out that it was trying to load something from /var/lib/xend/state/. Purging the /var/lib/xend/state directory fixed the problem, and hopefully had no side-effects. Thanks to this blog for the solution. I should probably file a bug, shouldn t I? [edit] done: http://bugzilla.xen.org/bugzilla/show_bug.cgi?id=1797

Christian Perrier: 10 years being Debian Developer - part 5: being a newbie DD...and working on l10n

I left you 2.5 months ago with the last question asked by my applicaiton manager, Martin Michlmayr : "Please tell me about about yourself and what you intend to do for Debian". Interesting question to revisit now, indeed. Here is what I answered: About myself first. I'm a 40 year old project manager and system administrator working in French National Aerospace Research Center. My best definition of my skills in computing is "Know more or less about a Lot of Things and be a Specialist of Nothing"...:-). I'm definitely not a programmer, nor a real system administrator, nor a RDBMS administrator, nor a personal workstation designer, though I do all of these daily. I think I'm perfect for finding the good person for having a defined job done. Besides this, I'm a genealogist for several years now. This is what finally decided me to apply for becoming a package maintainer : there are some quite good free genealogy software for Unix, though for various reasons they are not used very widely, even Unix geeks (my main software for genealogy still runs on Another Operatin System and is evertythig but free).I think that I can bring something here to the Free Software World, by helping some of these good programs in getting into the best Linux distribution I know.... For me, this is a mean for giving back to the free software movement what I gives to me since I discovered Linux 6-7 years ago. My very first intention as soon as I get my way into the Debian Developers Heaven is adopting the Geneweb package currently maintained by Brent Flugham. I'm in close contact with the author (who happens to be french, which helps) as well as a daily user of it. The current package which is in the distribution is already my work for a great part. I gave it to Brent, the current maintainer and we both agreed that it would be better for me to apply to becoming anofficial maintainer. I also contributed to the package for lifelines, another genealogy software. The last version of the package is also 80% my work, acknowledged by Javier, the official maintainer. Concerning that package, I do not have "plans" for adopting it (we didn't discussed of this with Javier, and I'm not sure I could bring him that much things). I came to Linux thanks to a great friend of mine, Ren Cougnenc. Ren opened my eyes to the free software world when I still thought that it was only a variant of free beers. I got really involved into Linux when I forced me to remove any other Operating System from my computer at work and tried to do my daily job with Linux. I have now succeeded at ONERA in getting free software to be accepted as a credible alternative for important projects. At this time, especially for server and network-related projects. I absolutely cannot tell why and how I came to be a Debian user. I simply don't remember. But I know why I am still a Debian user : this is a distribution which is controlled by only one organisation--->its users. And I want to be part of it. Finally, I did not mention above the somewhat "political" nature of my personal involvment into free software. Except for the physical appearence, I think I mimic RMS on several points (though he probably speaks better french than I try to speak english....which does not help for expressing complex ideas like the ones above!). As anyone can see, I was already very verbose when writing, sorry for this. Funnily Martin summed this up in one paragraph when he posted his AM report about my application. From what I see, also, my English didn't improve that much since then. It seems this is a desperatecause, I'm afraid. Anyway, all this was apparently OK for Martin and, on July 21st 2001, he wrote and posted his AM report and, on July 30th 2001, I got a mail by James Troup: An account has been created for you on developer-accessible machines with username 'bubulle'. bubulle@debian.org was born. Now I can more easily destro^W contribute to my favourite Linux distro. Indeed, I don't remember that much about the 2001-2003 years. I was probably not that active in Debian. Mostly, I was maintaining geneweb, for which I polished the package to have it reach a quite decent state, with elaborated debconf configuration. Indeed, at that time, I was still also deeply involved in genealogy research and still contributing to several mutual help groups for this. This is about the time where I did setup my web site (including pages to keep the link with our US family, which we visited in 2002). I think that the major turn in my Debian activities happened around september 2002 when Denis Barbier contacted me to add support in geneweb for a new feature he introduced in Debian : po-debconf. At that time, I knew nearly nothing about localization and internationalization. Denis was definitely one of the "leaders" in this effort in Debian. During these years, he did a tremendous job setting up tools and infrastructure to make the translation work easier. One of his achievements was "po-debconf", this set of tools and scripts that allows translation debconf "templates", the questions asked to users when configuring packages. All this lead me to discover an entire new world : the world of translating software. As often when I discover something I like, I jumped into it very deeply. Indeed, in early January 2003, I did my very first contributions to debian-l10n-french and began working on systematic translation of debconf templates. Guess what was the goal : 100%, of course! Have ALL packages that have debconf templates...translated to French. We reached that goal.....on June 2nd 2008 in unstable (indeed "virtually" : all packages were either 100% translated...or had a bug report with a complete translation) and on December 21st 2010 for testing. Squeeze was indeed the first Debian release with full 100% for French. Something to learn with localization work: it's never finished and you have to be patient. So, back in 2003, we were starting this effort. Indeed, debian-l10n-french was, at that time, an incredibly busy list and the translation rate was very high: I still remember spending my summer holidays translating 2-3 packages debconf templates every day for two weeks. Meanwhile, my packaging activities were low: only geneweb and lifelines, that was all. Something suddenly changed this and it has been the other "big turn" in my Debian life. After summer 2003, I suddenly started coming on some strange packages that were needing translation: they were popping up daily in lists with funny names like "languagechooser", "countrychooser", "choose-mirror", etc. I knew nothing about them and started "translating" their strings too, and sending bug reports after a decent review on debian-l10n-french. Then, Denis Barbier mailed me and explained me that these things were belonging to a new shiny project named Debian Installer and meant to replace the good old boot-floppies. Denis explained me that it would maybe be more efficient to work directly in the "D-I" team and "commit" my work instead of sending bug reports. Commit? What's that? You mean this wizard tool that only Real Power Developers use, named "CVS"? But this is an incredibly complicated tool, Denis. Do you really want me, the nerd DD, to play with it? Oh, and in this D-I development, I see people who are close to be semi-gods. Names I read in mailing lists and always impress me with their Knowledge and Cleverness: Martin Michlmayr (my AM, doh), Tollef Fog Heen, Petter Reinholdtsen and so many others and, doh, this impressive person named "Joey Hess" who seems to be so clever and knowledgeable, and able to write things I have no clue about. Joey Hess, really? But this guy has been in Debian forever. Me, really? Work with the Elite of Debian? Doh, doh, doh. Anyway, in about two months time, I switched from the clueless guy status to the status of "the guy who nags people about l10n in D-I", along with another fellow named Denny "seppy" Stampfer". And then we started helping Joey to release well localized D-I alphas and betas at the end of 2003 (the release rate at the time was incredible: Sarge installer beta1 in November 2003, beta2 in January 2004). I really remember spending my 2003 Christmas holidays hunting for....100% completion of languages we were supporting, and helping new translators to work on D-I translation. Yes, 8 years ago, I was already doing all this..:-)...painting the world in red. All this leads up to the year 2004. Certainly the most important year in my Debian life because it has been....the year of my first DebConf. But you'll learn about this....in another post (hopefully not in 2.5 months).

Russell Coker: Some OWS Stuff

There is an interesting interview with Richard Wilkinson on ted.com about the damage to society that is caused by excessive inequality [1]. The conclusion is interesting, it seems that one part of the solution is to send the message that excessive greed is anti-social. The demonstrations against the greedy 1% seem likely to do some good in this regard. He also has a TED talk about this in which he shows that even the rich don t benefit from great inequality [2]. If nothing else the links between great inequality and high rates of mental illness and murder should convince people that they don t want to live in such a society. Buying Cameras Some of the radio news reports have claimed that some of the people in OWS camps in Australia are somehow unworthy because they have smart-phones and digital cameras. They seem to believe that people have to demonstrate great poverty to have standing to complain. The problem here is that owning a mobile phone is essential to getting employment in many industries, so the vast majority of people who are old enough to work but not near retirement age just have to own a mobile phone. Once you have decided to use a phone if you are going to use it a lot (which you probably will if living in a tent in a public place) then being on a contract usually makes economic sense. As the cheapest contracts include a free Android phone that means almost everyone who is old enough to be allowed out alone at night and young enough to be willing to sleep rough for a cause will have such a phone. A related complaint is about people owning good cameras. One reason for this is the fact that the huge number of cameras deters the police from using excessive force that s a good justification for spending money on a good camera. But then a good camera doesn t cost much nowadays, the specs on $100 cameras are quite impressive and a camera that costs $400 new will have features that are useful for professional photographers and cameras are even cheaper on ebay. In Australia social security payments for a single person who is out of work are as much as $486.80 per fortnight [3]. Therefore someone who is living on social security could buy a great camera if they saved up two weeks of social security payments or buy a similar camera on ebay for one week s worth of payments. When at a protest the widespread use of cameras is essential for protection against police brutality. It s worth saving up for a camera. But really they are missing the point, OWS is about representing ~99% of the population. Even people on the average income are suffering because of the greed of the rich. Is Australia such a Great Place? There have been claims that Australia is such a great place that OWS protests are not needed, supposedly we have a very egalitarian society. The ABS data shows that the level of inequality as expressed by numbers isn t that great when restricted to just income if you restrict the analysis to the top 20% [4]. The mean gross household income per week is $340 for the lowest quintile and $2,380 for the highest quintile giving a ratio of 1:7.0. But when you look at total household net worth it s $27,400 vs $1,720,700 a ratio of 1:62.8! The ratio for the richest 1% would surely be a lot greater but a quick search of abs.gov.au didn t turn up any statistics on them. Also Richard Wilkinson s analysis puts Australia quite close to the US. If the people who are in the top 1% earned their money, paid taxes, and didn t pervert the political process then there wouldn t be any OWS protests. The corruption in the political process and rent seeking by corporations isn t as bad here as it is in the US, but it s bad enough that we need some serious changes. Emily Manuel wrote an interesting article for Tiger Beatdown about the need for an Occupy movement in Australia [5]. Her description of the political problems in Australia is quite clear. But I wasn t convinced by her analysis which suggested that Australia is different from the US in any significant way. Really Comparing Income To properly compare income it doesn t make sense to compare a ratio of money earned, or disposable income which is often misleading due to disagreements about what is considered disposable. It s really about which of the things that you want to do can be achieved on your income. One possible way of comparing it is based on Maslow s hierarchy of human needs. The poorest 20% of households in Australia have a gross income of $340 per week. The ABS shows that the average of private rental prices in the 8 capital cities (where most of the population lives) is $218 per week [6]. Obviously the poorest households can t afford an average rent unless you have lots of people packed in one house. No matter how you look at the numbers it s going to be difficult for low income people to pay for rent and food. That puts them on the bottom level of Maslow s higherarchy. It seems to me that the benefits of increased income diminish and it becomes more of a contest to be richer than other people. Owning an old car is a significant benefit over having no car. Owning a new car has some real benefits over an old car. Luxury cars usually have safety features before they are introduced to cheaper cars, but apart from that the benefit of owning a luxury car is small. Owning multiple Rolls-Royces doesn t seem to provide any real benefit, someone who would want to do that would probably be better served by hiring a good psychologist. Update: In a comment Brendan Scott cited The Theory of the Leisure Class by Thorstein Veblen. That s a great point, and I ll have to read that book! I still maintain that owning two Rolls Royces is a symptom of psychological problems, people who can sort out such problems won t feel the need to try and prove that they have more money than others. Charles Stross has an interesting article about wealth disparity etc [7]. Being a sci-fi author he also explores what life might be like after Drexler boxes satisfy most of our current materiel needs. The Age has an informative article by Suzy Freeman-Greene contrasting the OWS protests in Melbourne which were broken up by police to the Catholic anti-abortion protests that have been going on for more than 20 years [8]. It seems that corporations aren t the only large organisations that are pwning democracy!

Dirk Eddelbuettel: RInside 0.2.5

RInside 0.2.5 arrived on CRAN earlier today. RInside is a set of convenience classes which facilitate embedding of R inside of C++ applications and programs, using the classes and functions provided by the Rcpp R and C++ integration package. The big news for this release is that Windows applications once again run! James Bates gets full credit for noticing why Windows binaries died on startup, and provided a short patch which we (after some further simplicifcation) added. So for the first time since the 0.2.0 release, Windows users can just change into the examples/standard directory and issue a simple make -f Makefile.win, and the twelve example binaries will actually work. Yay, and kudos to James. The other main addition is a new examples directory examples/wt containing the web-application of the density estimation demo I had blogged about last week. By combining the Wt toolkit with RInside, we get C++ "web" applications containing R which is pretty nifty. The example otherwise follows the earlier Qt-based example of a standard desktop application with RInside. One minor snafu I just noticed is that I was overzealous with the .Rbuildignore file. By excluding the wt and qtbinaries, I also excluded their source files with the same name (up to the .cpp or .h ending). If you desire to run the examples, just grab these examples files from the SVN repo at R-Forge. All changes since the last release are summarized below:
0.2.5   2011-12-07
   o    Applied (somewhat simplified) patch by James Bates which restores
        RInside to working on Windows -- with a big Thank You! to James for
        fixing a long-standing bug we inadvertendly introduced right after
        0.2.0 almost two years ago
   o    New example embedding R inside a Wt (aka Webtoolkit, pronounced 
        'witty') application, mirroring the previous Qt application
   o    Qt example qtdensity now uses the new svg() device in base R; removed
        test for cairoDevice package as well as fallback png code
   o    Very minor fix to qmake.pro file for Qt app correcting link order
CRANberries also provides a short report with changes from the previous release. More information is on the RInside page. Questions, comments etc should go to the rcpp-devel mailing list off the Rcpp R-Forge page.

8 December 2011

Thomas Koch: On distributing binaries

Many software developers don't care about security. A typical Java developer downloads the Eclipse binary from eclipse.org, a website that doesn't even support https. He then continues to install binary eclipse plugins from several different web sites. Once development starts, maven continues to pull in hundreds of plugins and dependencies without checking their origin.
How do you still want to trust this machine with your private passwords or gpg key? I don't. Therefor I do java development at least in a chroot'ed environment. To make this point a bit less theoretic I've collected examples of compromised downloads or servers in recent months.
And another funny example, why you should not run sudo make install on any downloaded source code: The bumblebee rm -rf /usr disaster. Not so funny: 154 people killed in malware caused airplane crash. or Malware compromises USAF Predator drones
What's the morale? If you're a developer, then please get a gpg key, get it signed by many other developers and sign your release artifacts! And no, just publishing a SHA-1 or (god beware) MD5 hash of the artifact does not provide any security. It may just fool people to believe they'd be secure. (Google Code team, please fix this! Apache people, signing artifacts with gpg keys that don't have any signatures is at best useless!)
This blog post has been triggered by a post relating to the issue of binary incompatibilities between Scala versions. If I understand Josh Suereth correctly, he recommends to not worry about binary incompatibilities but to distribute libraries as source code and recompile them when necessary. AMEN! Please, make this a habbit in Scala land! If you then distribute source code in Git repositories you can make use of gpg signed tags.
And while you're at it, why not help your favorite linux distribution to package Scala stuff?

Dirk Eddelbuettel: Rcpp talk at Seattle RUG

Thanks to Zach and the Seattle R User Group for the invitation. Earlier this evening I gave a talk about R, C++ and our Rcpp package, focussing on the why, how, what, who, and "one more thing" higher-level aspects of extending R. This was supposed to be an hour long, and with the examples at the end went a little over, butat least nobody started to snore. The slides are now up at the top of my presentations page.

7 December 2011

Gustavo Noronha Silva: WebKitGTK+ hackfest \o/

It s been a couple days since I returned from this year s WebKitGTK+ hackfest in A Coru a, Spain. The weather was very nice, not too cold and not too rainy, we had great food, great drinks and I got to meet new people, and hang out with old friends, which is always great!

Hackfest black board, photo by Mario

I think this was a very productive hackfest, and as usual a very well organized one! Thanks to the GNOME Foundation for the travel sponsorship, to our friends at Igalia for doing an awesome job at making it happen, and to Collabora for sponsoring it and granting me the time to go there! We got a lot done, and although, as usual, our goals list had many items not crossed, we did cross a few very important ones. I took part in discussions about the new WebKit2 APIs, got to know the new design for GNOME s Web application, which looks great, discussed about Accelerated Compositing along with Joone, Alex, Nayan and Martin Robinson, hacked libsoup a bit to port the multipart/x-mixed-replace patch I wrote to the awesome gio-based infrastructure Dan Winship is building, and some random misc. The biggest chunk of time, though, ended up being devoted to a very uninteresting (to outsiders, at least), but very important task: making it possible to more easily reproduce our test results. TL;DR? We made our bots and development builds use jhbuild to automatically install dependencies; if you re using tarballs, don t worry, your usual autogen/configure/make/make install have not been touched. Now to the more verbose version! The need

Our three build slaves reporting a few failures

For a couple years now we have supported an increasingly complex and very demanding automated testing infrastructure. We have three buildbot slaves, one provided by Collabora (which I maintain), and two provided by Igalia (maintained by their WebKitGTK+ folks). Those bots build as many check ins as possible with 3 different configurations: 32 bits release, 64 bits release, and 64 bits debug. In addition to those, we have another bot called the EWS, or Early Warning System. There are two of those at this moment: one VM provided by Collabora and my desktop, provided by myself. These bots build every patch uploaded to the bugzilla, and report build failures or passes (you can see the green bubbles). They are very important to our development process because if the patch causes a build failure for our port people can often know that before landing, and try fixes by uploading them to bugzilla instead of doing additional commits. And people are usually very receptive to waiting for EWS output and acting on it, except when they take way too long. You can have an idea of what the life of an EWS bot looks like by looking at the recent status for the WebKitGTK+ bots. Maintaining all of those bots is at times a rather daunting task. The tests require a very specific set of packages, fonts, themes and icons to always report the same size for objects in a render. Upgrades, for instance, had to be synchronized, and usually involve generating new baselines for a large number of tests. You can see in these instructions, for instance, how strict the environment requirements are yes, we need specific versions of fonts, because they often cause layouts to change in size! At one point we had tests fail after a compiler upgrade, which made rounding act a bit different! So stability was a very important aspect of maintaining these bots. All of them have the same version of Debian, and most of the packages are pinned to the same version. On the other hand, and in direct contradition to the stability requirement, we often require bleeding edge versions of some libraries we rely on, such as libsoup. Since we started pushing WebKitGTK+ to be libsoup-only, its own progress has been pretty much driven by WebKitGTK+ s requirements, and Dan Winship has made it possible to make our soup backend much, much simpler and way more featureful. That meant, though, requiring very recent versions of soup. To top it off, for anyone not running Debian testing and tracking the exact same versions of packages as the bots it was virtually impossible to get the tests to pass, which made it very difficult for even ourselves to make sure all patches were still passing before committing something. Wow, what a mess. The explosion^Wsolution So a few weeks back Martin Robinson came up with a proposed solution, which, as he says, is the nuclear bomb solution. We would have a jhbuild environment which would build and install all of the dependencies necessary for reproducing the test expectations the bots have. So over the first three days of the hackfest Martin and myself hacked away in building scripts, buildmaster integration, a jhbuild configuration, a jhbuild modules file, setting up tarballs, and wiring it all in a way that makes it convenient for the contributors to get along with. You ll notice that our buildslaves now have a step just before compiling called updated gtk dependencies (gtk is the name we use for our port in the context of WebKit), which runs jhbuild to install any new dependencies or version bumps we added. You can also see that those instructions I mentioned above became a tad simpler. It took us way more time than we thought for the dust to settle, but it eventually began to. The great thing of doing it during the hackfest was that we could find and fix issues with weird configurations on the spot! Oh, you build with AR_FLAGS=cruT and something doesn t like it? OK, we fix it so that the jhbuild modules are not affected by that variable. Oh, turns out we missed a dependency, no problem, we add it to the modules file or install them on the bots, and then document the dependency. I set up a very clean chroot which we could use for trying out changes so as to not disrupt the tree too much for the other hackfest participants, and I think overall we did good. The aftermath By the time we were done our colleagues who ran other distributions such as Fedora were already being able to get a substantial improvements to the number of tests passing, and so did we! Also, the ability to seamlessly upgrade all the bots with a simple commit made it possible for us to very easily land a change that required a very recent (as in unreleased) version of soup which made our networking backend way simpler. All that red looks great, doesn t it? And we aren t done yet, we ll certainly be making more tweaks to this infrastructure to make it more transparent and more helpful to the users (contributors and other people interested in running the tests). If you ve been hit by the instability we caused, sorry about that, poke mrobinson or myself in the #webkitgtk+ IRC channel on FreeNode, and we ll help you out or fix any issues. If you haven t, we hope you enjoy all the goodness that a reproducible testing suite has to offer! That s it for now, folks, I ll have more to report on follow-up work started at the hackfest soon enough, hopefully =).

Kees Cook: how to throw an EC2 party

Prepare a location to run juju and install it:
mkdir ~/party
cd ~/party
sudo apt-get install juju
Initialize your juju environment. Be sure to add juju-origin: ppa to your environment, along with filling in your access-key and secret-key from your Amazon AWS account. Note that control-bucket and admin-secret should not be used by any other environment or juju won t be able to distinguish them. Other variables are good to set now too. I wanted my instances close to me, use I set region: us-west-1 . I also wanted a 64bit system, so using the AMI list, I chose default-series: oneiric , default-instance-type: m1.large and default-image-id: ami-7b772b3e
juju
$EDITOR ~/.juju/environments.yaml
Get my sbuild charm, and configure some types of builders. The salt should be something used only for this party; it is used to generate the random passwords for the builder accounts. The distro and releases can be set to whatever the mk-sbuild tool understands.
bzr co lp:~kees/charm/oneiric/sbuild/trunk sbuild-charm
cat >local.yaml <<EOM
builder-debian:
    salt: some-secret-phrase-for-this-party
    distro: debian
    releases: unstable
builder-ubuntu:
    salt: some-secret-phrase-for-this-party
    distro: ubuntu
    releases: precise,oneiric
EOM
Bootstrap juju and wait for ec2 instance to come up.
juju bootstrap
Before running the status, you can either accept the SSH key blindly, or use ec2-describe-instances to find the instance and public host name, and use my wait-for-ssh tool to inject the SSH host key into your ~/.ssh/known_hosts file. This requires having set up the environment variables needed by ec2-describe-instances , though.
ec2-describe-instances --region REGION
./sbuild-charm/wait-for-ssh INSTANCE HOST REGION
Get status:
juju status
Deploy a builder:
juju deploy --config local.yaml --repository $PWD local:sbuild-charm builder-debian
Deploy more of the same type:
juju add-unit builder-debian
juju add-unit builder-debian
juju add-unit builder-debian
Now you have to wait for them to finish installing, which will take a while. Once they re at least partially up (the builder user has been created), you can print out the slips of paper to hand out to your party attendees:
./sbuild-charm/slips   mpage -1 > /tmp/slips.ps
ps2pdf /tmp/slips.ps /tmp/slips.pdf
They look like this:
Unit: builder-debian/3
Host: ec2-256-1-1-1.us-west-1.compute.amazonaws.com
SSH key fingerprints:
  1024 3e:f7:66:53:a9:e8:96:c7:27:36:71:ce:2a:cf:65:31 (DSA)
  256 53:a9:e8:96:c7:20:6f:8f:4a:de:b2:a3:b7:6b:34:f7 (ECDSA)
  2048 3b:29:99:20:6f:8f:4a:de:b2:a3:b7:6b:34:bc:7a:e3 (RSA)
Username: builder
Password: 68b329da9893
To admin the machines, you can use juju itself, where N is the machine number from the juju status output:
juju ssh N
To add additional chroots to the entire builder service, add them to the config:
juju set builder-debian release=unstable,testing,stable
juju set builder-ubuntu release=precise,oneiric,lucid,natty
Notes about some of the terrible security hacks this charm does: Enjoy!

2011, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.
Creative Commons License

Kees Cook: juju bug fixing

My earlier post on juju described a number of weird glitches I ran into. I got invited by hazmat on IRC (freenode #juju) to try to reproduce the problems so we could isolate the trouble. Fix #1: use the version from the PPA. The juju setup documentation doesn t mention this, but it seems that adding juju-origin: ppa to your ~/.juju/environment.yaml is a good idea. I suggest it be made the default, and to link to the full list of legal syntax for the environment.yaml file. I was not able to reproduce the missing-machines-at-startup problem after doing this, but perhaps it s a hard race to lose. Fix #2: don t use terminate-machine . :P There seems to be a problem around doing the following series of commands: juju remove-unit FOO/N; juju terminate-machine X; juju add-unit FOO . This makes the provisioner go crazy, and leaves all further attempts to add units stick in pending forever. Big thank you to hazmat and SpamapS for helping debug this.

2011, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.
Creative Commons License

Julien Danjou: New job, new blog

It has been a while since I blogged but I've been very busy, with my new job and this new blog! New job! I quitted my job last September, and found another one that I started in October. I'm now the lead developer of eNovance Labs, where I work on the OpenStack project. So far, this allowed me to contribute heavily to the Debian packaging of OpenStack. New blog! In the meantime, I took some time to redesign my personal homepage and this blog, which is now using Hyde, the Python equivalent of Jekyll, which is in Ruby. Since I dislike Ruby (sorry), I preferred to use a Python based generator, and I admit Hyde is really cool. Since I really suck at Web design, this one is obviously based on Twitter's bootstrap

6 December 2011

Don Armstrong: X200 Bios Update on Debian

Updating the bios on my X200 Tablet was surprisingly simple in Debian. Often, updating bios requires some sort of Windows or Dos boot disk and all kinds of machinations to make that happen. Lenovo provides an iso image to update the bios which alleviates some of these problems, but as the X series don't have built-in CDROM drives (and I don't have an ultrabay), All you have to do is download the iso, install syslinux aptitude install syslinux, and configure grub to start memdisk and boot off of the iso. You can do this by copying the iso to your boot partition (cp 7wuj43uc.iso /boot/;), copying memdisk to the boot partition (cp /usr/lib/syslinux/memdisk /boot/;) and adding an entry to your grub.cfg like the following (putting it in /etc/grub.d/40_custom is probably the most reasonable):
menuentry "Thinkpad x200 BIOS Upgrade to 3.19"  
       insmod part_msdos
       insmod ext2
       set root='(hd0,msdos1)'
       linux16 /memdisk iso
       initrd16 /7wuj43uc.iso
 
Note that your configuration will be slightly different depending on whether you have a separate /boot partition (I do) or not. Then you just need to boot the laptop, select the bios upgrade option in grub, and tell the program that you really do want to upgrade the bios and that you really do have a full battery and the laptop plugged into an AC adapter. The major idea for this method came from ThinkWiki's entry.

Don Armstrong: X200 Bios Update on Debian

Updating the bios on my X200 Tablet was surprisingly simple in Debian. Often, updating bios requires some sort of Windows or Dos boot disk and all kinds of machinations to make that happen. Lenovo provides an iso image to update the bios which alleviates some of these problems, but as the X series don't have built-in CDROM drives (and I don't have an ultrabay), All you have to do is download the iso, install syslinux aptitude install syslinux, and configure grub to start memdisk and boot off of the iso. You can do this by copying the iso to your boot partition (cp 7wuj43uc.iso /boot/;), copying memdisk to the boot partition (cp /usr/lib/syslinux/memdisk /boot/;) and adding an entry to your grub.cfg like the following (putting it in /etc/grub.d/40_custom is probably the most reasonable):
menuentry "Thinkpad x200 BIOS Upgrade to 3.19"  
       insmod part_msdos
       insmod ext2
       set root='(hd0,msdos1)'
       linux16 /memdisk iso
       initrd16 /7wuj43uc.iso
 
Note that your configuration will be slightly different depending on whether you have a separate /boot partition (I do) or not. Then you just need to boot the laptop, select the bios upgrade option in grub, and tell the program that you really do want to upgrade the bios and that you really do have a full battery and the laptop plugged into an AC adapter. The major idea for this method came from ThinkWiki's entry.

Sam Hartman: Moonshot Introduction

I recently put together a reading list on Project Moonshot for a friend. If you have seen discussions of Moonshot but not known where to get started understanding the technology, here is a fairly good initial list. It s long, but take a look starting at the beginning and let us know what you think.Take a look at http://www.project-moonshot.org/. Specifically, http://www.project-moonshot.org/sites/default/files/moonshot-feasibility-analysis.pdf and http://www.project-moonshot.org/sites/default/files/moonshot-briefing-ietf-78.pdf That briefing paper contains outdated versions of the technical
specifications.
Please see http://tools.ietf.org/html/draft-ietf-abfab-arch-00 http://tools.ietf.org/html/draft-ietf-abfab-gss-eap and http://tools.ietf.org/html/draft-ietf-abfab-gss-eap-naming
and http://tools.ietf.org/html/draft-ietf-abfab-aaa-saml O, yeah, and for the totally cool stuff that is still being designed
please see http://tools.ietf.org/html/draft-mrw-abfab-multihop-fed and http://tools.ietf.org/html/draft-mrw-abfab-trust-router

Steve Langasek: Making jam from bugs

This weekend, we held a combined Debian Bug Squashing Party and Ubuntu Local Jam in Portland, OR. A big thank you to PuppetLabs for hosting! Thanks to a brilliant insight from Kees Cook, we were able to give everyone access to their own pre-configured build environment as soon as they walked in the door by deploying schroot/sbuild instances in "the cloud" (in this case, Amazon EC2). Small blips with the mirrors notwithstanding, this worked out pretty well, and let people start to get their hands dirty as soon as they walked in the door instead of spending a lot of time up front doing the boring work of setting up a build environment. This was a big win for people who had never done a package build before, and I highly recommend it for future BSPs. You can read about the build environment setup in the Debian wiki, and details on setting up your own BSP cloud in Kees's blog. (And the cloud instances were running Ubuntu 11.10 guests, with Debian unstable chroots - a perfect pairing for our joint Debian/Ubuntu event!) So how did this curious foray into a combined Ubuntu/Debian event go? Not too shabby: When all was said and done, we didn't get a chance to tackle any wheezy release critical bugs like we'd hoped. That's ok, that leaves us something to do for our next event, which will be bigger and even better than this one. Maybe even big enough to rival one of those crazy, all-weekend BSPs that they have in Germany...

Kees Cook: EC2 instances in support of a BSP

On Sunday, I brought up EC2 instances to support the combined Debian Bug Squashing Party/Ubuntu Local Jam that took place at PuppetLabs in Portland, OR, USA. The intent was to provide each participant with their own sbuild environment on a 64bit machine, since we were going to be working on Multi-Arch support, and having both 64bit and 32bit chroots would be helpful. The host was an Ubuntu 11.10 (Oneiric) instance so it would be possible to do SRU verifications in the cloud too. I was curious about the juju provisioning system, since it has an interesting plugin system, called charms , that can be used to build out services. I decided to write an sbuild charm, which was pretty straight forward and quite powerful (using this charm it would be possible to trigger the creation of new schroots across all instances at any time, etc). The juju service itself works really well when it works correctly. When something goes wrong, unfortunately, it becomes nearly impossible to debug or fix. Repeatedly while working on charm development, the provisioning system would lose its mind, and I d have to destroy the entire environment and re-bootstrap to get things running again. I had hoped this wouldn t be the case while I was using it during production on Sunday, but the provisioner broke spectacularly on Sunday too. Due to the fragility of the juju agents, it wasn t possible to restart the provisioner it lost its mind, the other agent s couldn t talk to it any more, etc. I would expect the master services on a cloud instance manager to be extremely robust since having it die would mean totally losing control of all your instances. On Sunday morning, I started 8 instances. 6 came up perfectly and were excellent work-horses all day at the BSP. 2 never came up. The EC2 instances started, but the service provisioner never noticed them. Adding new units didn t work (instances would start, but no services would notice them), and when I tried to remove the seemingly broken machines, the instance provisioner completely went crazy and started dumping Python traces into the logs (which seems to be related to this bug, though some kind of race condition seems to have confused it much earlier than this total failure), and that was it. We used the instances we had, and I spent 3 hours trying to fix the provisioner, eventually giving up on it. I was very pleased with EC2 and Ubuntu Server itself on the instances. The schroots worked, sbuild worked (though I identified some additional things that the charm should likely do for setup). I think juju has a lot of potential, but I m surprised at how fragile it is. It didn t help that Amazon had rebooted the entire West Coast the day before and there were dead Ubuntu Archive Mirrors in the DNS rotation. For anyone else wanting to spin up builders in the cloud using juju, I have a run-down of what this looks like from the admin s perspective, and even include a little script to produce little slips of paper to hand out to attendees with an instance s hostname, ssh keys, and builder SSH password. Seemed to work pretty well overall; I just wish I could have spun up a few more. :) So, even with the fighting with juju and a few extra instances that came up and I had to shut down again without actually using them, the total cost to run the instances for the whole BSP was about US$40, and including the charm development time, about US$45. UPDATE: some more details on how to avoid the glitches I hit.

2011, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.
Creative Commons License

Next.

Previous.