Search Results: "will"

6 March 2013

Raphael Geissert: A bashism a week: returning

Inspired by Thorsten Glaser's comment about where you can break from, this "bashism a week" is about a behaviour not implemented by bash.

return is a special built-in utility, and it should only be used on functions and scripts executed by the dot utility. That's what the POSIX:2001 specification requires.

If you return from any other scope, for example by accidentally calling it from a script that was not sourced but executed directly, the bash shell won't forgive you: it does not abort the execution of commands. This can lead to undesired behaviour.

A wide variety of shell interpreters silently handle such calls to return as if exit had been called.

An easy way to avoid such undesired behaviours is to follow the best practice of setting the e option, i.e.
set -e
. With that option set at the moment of calling return outside of the allowed scopes, bash will abort the execution, as desired.

The POSIX specification does not guarantee the above behaviour either as the result in such cases is "unspecified", however.

5 March 2013

Tanguy Ortolo: Suspend your computer from a non-mainstream desktop environment

Battery icon UPower for regular users Major desktop environments usually provide a user-friendly menu to suspend your computer. Internally, if seems to use something called UPower, which uses something called PolicyKit, which in turn may or may not use another piece called ConsoleKit (do not ask me what all these pieces are and how they relate to each other, I do not know and I do not want to know). For the regular user, the result is the following: when they click on that button, it suspends their computer.UPower for command line users However, if you are using a more unusual desktop environment, for instance a simple, light window manager, you will not have such a button. And no, UPower does not provide a simple command line tool to suspend your computer (they probably consider that command line is out of fashion ). Well, in case you need it, here is a way to suspend your computer from a command line or whatever: just alias it or bind it to a key shortcut. It uses some dark DBus magic:
$ dbus-send --system --print-reply --dest='org.freedesktop.UPower' \
    /org/freedesktop/UPower org.freedesktop.UPower.Suspend
If you prefer to hibernate your computer, that is, to suspend it, not to RAM but to disk:
$ dbus-send --system --print-reply --dest='org.freedesktop.UPower' \
    /org/freedesktop/UPower org.freedesktop.UPower.Hibernate
As you can see, for some reason it requires some duplication of information: the name org.freedesktop.UPower appears three times in that command line: in the destination , object path and interface member , whatever that could mean. Again, do not ask me why, I know nothing about that and I do not want to. Notes
  1. Actually, UPower does provide a command line tool called upower, but the only thing it does (monitoring power sources) is not really useful for that matter.

Lisandro Damián Nicanor Pérez Meyer: My Debian freeze experience (so far)

This is the first freeze in which I'm involved with upload rights. And it turned out to be a quite interesting ride so far, so I thought it would be nice to write about it.

As some of you may know, I'm a part of the Qt/KDE team. Before the freeze I was mostly involved in leaf packages, with some patch here or there, nothing fancy. And then the freeze came...

Bugs in Qt

...and bugs appeared in Qt. But they didn't get solved, even if the patches were there. Due to personal reasons, the manpower in Qt/KDE land decreased below normal levels (which were already low).

I took the time to review them, apply them in a local branch, build and test the fixes. I did a Qt upload before, but it was a team-consented one. This time there was not much reaction in our IRC channel as it used to be, so I was doubting if going ahead or not. I asked Ana, my great friend and former sponsor, for an opinion on the subject, and she gave me a really important advice: the patches were looking good and there is one really big true: if something get's broken, it can be fixed with a later upload.

You might be asking yourself why I was that afraid of doing the upload. Well, when one maintains such a medular package for many users one has to be careful And I also got used that those "big ones" like Qt where normally handled by hand skilled people. Do not take me wrong here, it's not that those people where keeping them for themselves, it's knowing that one does not has the same skills nor experience as them.

But again, no one was able to upload and I had the chance and will to do another upload if needed, so off it went. That was Qt 4:4.8.2-2.

Then new experiences followed: asking a buildd maintainer for a giveback, asking the Release Team for an unblock (more on this later), etc. While sponsoring me, Ana gave me another excellent advice which I always keep in my mind:

You can't know **everything** about Debian.

And that also includes a not so technical skill: communicating with other teams. But finally we got this new version of Qt in testing. Cool :-)

Of course, new bugs appeared, and my lack of skills (and sometimes, time) where replaced by team work: Pino looking at patches and Sune contacting upstream. The eleven uploads that followed are a nice example of team work, even if I was the one who signed and did the uploads. Whoever uses Qt must know that these wonderful people (including those who are not so active nowadays like Modestas or Fathi) have done lots to bring the better to their users.Thank you guys!

Be careful, they might bite you back!

Coming back to the non-technical skills, sometimes you have to communicate with other teams in Debian. And each team is (naturally) a separate world: possibly different people, different goals, etc. Of course, we share the goal to make Debian the best experience we can, but we do not necessarily agree on the paths to achieve so.

During the freeze, there is a team that gets lots of pressure, and not by chance: the Release Team. They handle a very important task, which is to ride the freeze to get to a release. OK, that's what everyone knows. Now, one thing is knowing that and another is really understanding what does that means.

Of course I was in the first group. From the outside, communicating with the RT was a kind of "special art", and not an easy one. I have even been advised to not ask for more than one or two unblocks per weekend, as they might "bite me back". So I put on my flamesuit on and... launched reportbug release.debian.org.

Now I'm really happy to say that my experience was far from what I described above. And yes, I had the chance to even disagree on some stuff. But remember: non-technical skills, a.k.a. social skills. Once I started to know what was going on inside the RT (joining #debian-release was a big help for that) I learnt some nice tips to approach them. Please allow me to list some of them:

  • Remember: you are the maintainer of the package, they are like gatekeepers that are there to help us coordinate to do a release. But they don't maintain the code, you do that. So try to be verbose when needed, explain the changes and don't forget a nice diff. They need to understand what is going on: they can't read your mind.
  • They are human beings too: not everyday might be their best day (the same goes for you too!). And they are under the pressure of a release. Be patient, that finally pays off.
  • Does your changes seem not so clear? try to improve them.
  • The package has a lot of changes but you really feel they are needed? Try to explain that as good as you can.
  • Try to put yourself in their position: do we really want this? If in doubt, there is a nice way to know what they think: a pre-approval bug.
I want to make a stop in this last point. A pre-approval bug it's an unblock bug in which you edit the subject to add "pre-approval" in it. Easy, isn't it? It gives you the opportunity to know what the RT thinks before doing the upload. In other words: it gives you the chance to communicate and do things in the best possible way for all the parts involved.

I've have also seen pre-approval bugs that were really not needed. But to learn where the threshold of what can be directly uploaded and what deserves a pre-approval bug is you need to know the guidelines the RT gives you. Do you still have doubts? fire a pre-approval bug and try to be clear.

Of course, this are all fruits of my experience with the RT during this time. If the RT thinks different from what I'm writing here, please stand up: we are hear to listen to you and learn :-)

As a side note, I think I should file a wishlist bug to include the pre-approval bug option in reportbug. Yes, I'm lazy :-)

Summing up

Overall this was a very nice and positive experience. We are not done yet. Are we really done at some point? Let's hope not, because this is where the fun comes from :-)

Russell Coker: Kogan Mobile

Kogan Mobile is the newest virtual telco in Australia [1]. They resell Telstra 3G (not NextG or LTE) and while their coverage isn t as good as the full Telstra service it s more than adequate for my needs as they provide 3G coverage to 97% of the population and 2G+3G coverage to 98.5%. Their coverage is probably a lot better than Three who had the worst record of network coverage in Australia yet managed to always provide coverage where I wanted it I was a happy Three customer for more than 6 years. Kogan s main selling point is that they offer unlimited calls to Australian mobile phones and land-lines and unlimited SMS for a pre-paid fee of only $300 per annum ($25 per month on average). My parents have been getting unpleasantly large phone bills which have considerably more than $25 of calls to mobile phones every month so an obvious solution for them is to sign up for a Kogan mobile phone and use it for all such calls. There are other ways my parents could save money on calls (such as VOIP) but a mobile phone is easiest and offers other benefits such as running Android apps (when compared to using a non-smart phone). 6G of Data! Kogan also offers 6G of data per month, the down-side to this is that they bill in 1MB increments per session . I was worried that this might be per TCP connection or something else silly but I decided to sign my parents up for it as they aren t going to use a lot of data (they claim that they don t want to use the Internet on their phone but I know better). I ve done some tests on the SIM I got for my parents. For testing purposes I installed the Kogan SIM in my wife s new Nexus 4 and had it provide Wifi net access to my phone while we were playing Ingress. So far after 2 days which involved a reasonable amount of Ingress (I reached level 7) as well as all the usual stuff that happens in the background for two phones (checking email, news, weather, etc) Kogan considers that 177MB have been used out of the 6144MB for the month, which means that even with what is an unusual amount of traffic for us the account in question still isn t going to use half the quota for the month. Now that 10 days have elapsed with less intense usage Kogan considers that a total of 1373MB have been used.
days received sent Kogan
2 127.24 41.54 177
10 959.39 299.6 1373
Kogan also don t seem to mention whether they bill for transmitted data. I used the 3G Watchdog app to measure the amount data transferred, the above table has the amounts of data that 3G Watchdog considers were sent and received along with the amount that is listed by the Kogan Android app. My past experience with 3G Watchdog and Virgin Mobile is that it s usually quite accurate but has been over-reporting the data transfers recently (I think that Virgin is only billing me for downloads while 3G Watchdog counts uploads). So the relatively small difference between the 3G Watchdog report and what Kogan thinks I ve done means that either rounding the session up to the nearest meg doesn t make any significant difference (which would imply that a session can be a long time) or that Kogan isn t counting uploaded data and the session rounding up only adds about 40% to the total recorded transfer. My current plan with Virgin Mobile gives me 1.5G per month of quota, so as long as Kogan s rounding doesn t increase the recorded data transfer by a factor of 4 I will still be able to transfer more data with Kogan while paying less. One disadvantage of using Kogan is that I might have to tweak programs like my email program to poll less frequently to avoid excessive session charges (a program polling every 5 minutes would use up the 6G quota in 21 days if each poll counted as a session) although current tests indicate that this won t be necessary. But the up-side is that there are no extra fees with Kogan, they merely restrict data access for my use and that of most people I know it s better to have data access cut off than to receive a large bill. The Kogan Android App Kogan has an Android app that will give the status of your account and allow you to change the plan etc. This is quite nice but one major disadvantage is that it s also a sales app for the Kogan online store. This is bad for the user as some aspects of what I consider the core functionality are limited (for example there s no way to force a poll of the data usage count or determine how current the data is). But there s an obvious advantage to Kogan in providing a way to sell their goods that is going to be used by every customer of Kogan Mobile. The sales part of the app isn t very functional IMHO, it doesn t seem to have basic functionality such as sorting a list of items by price. Benefits of Kogan 6G of data is a lot! $300 per annum is quite cheap, anyone who makes any serious use of phones will be paying more than that in Australia. Lack of extra fees means that there is little need to restrict net access. I can risk getting cut off near the end of the month but I can t risk the potential for hundreds of dollars in excess fees. The Kogan app shows me the data used so I will probably uninstall 3G Watchdog, having one less program running is a good thing. You get a free SIM (value $5) when you buy a phone from Kogan. Disadvantages of Kogan They are new to the Telco business and admit that their customer service is lacking due to unexpected demand. If you order a SIM now they state that it will arrive in April. Apparently they are deliberately delaying orders because they can t cope with demand. The included call quota doesn t include international calls. While unlimited free calls in Australia is great if you make many international calls then this could end up costing you more. Other mobile telcos such as Lebara offer good deals for International calls, it could be an option to use a Lebara SIM with an old non-smart phone while using Kogan for your smart phone. I am concerned about the lack of detail about how data is accounted. If the definition of a session changes then 6G could turn out not to be enough. As Kogan is reselling a Telstra service it is possible that Telstra could change the deal without Kogan being able to stop them. Conclusion I will move my phone and my wife s phone to Kogan ASAP. My general idea is to sign us up for Kogan about 2 weeks apart, so if one phone runs out of the 6G data quota then the other phone can be used as a Wifi access point for 2 weeks. If the phones don t both have their quota end at the same time then there is less chance of both phones running out during a high traffic month.

4 March 2013

Bastian Blank: Using SECCOMP to filter sync operations

Linux includes a syscall filter since a long time. It was restricted to a pre-defined set of syscalls. Since some versions Linux got a more generic filter. Linux can use a BPF-filter to define actions for syscalls. This allows a fine granular specification on which syscalls to act. Also it supports different outcomes assigned to the filter. This filter can be used to filter out sync operations. Debian already got a tool to do this called eatmydata. It is pretty limited as it uses a shared library to drop the library functions. It needs to be available at all times, or it will not do anything. I wrote a small tool that asks the kernel to filter out sync operations for all children. It sets a filter with all currently supported sync-like operations and makes them return success. However it can't filter the O_SYNC-flag from the open-syscall, so it just makes it return an error. It executes the command given on the command-line after that. This is just a proof of concept, but lets see.
/*
 * Copyright (C) 2013 Bastian Blank <waldi@debian.org>
 *
 * Redistribution and use in source and binary forms, with or without
 * modification, are permitted provided that the following conditions are met:
 *
 * 1. Redistributions of source code must retain the above copyright notice, this
 *    list of conditions and the following disclaimer.
 * 2. Redistributions in binary form must reproduce the above copyright notice,
 *    this list of conditions and the following disclaimer in the documentation
 *    and/or other materials provided with the distribution.
 *
 * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
 * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
 * WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
 * DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR
 * ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
 * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
 * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
 * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
 * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
 * SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 */
#define _GNU_SOURCE 1
#include <errno.h>
#include <fcntl.h>
#include <seccomp.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#define filter_rule_add(action, syscall, count, ...) \
  if (seccomp_rule_add(filter, action, syscall, count, ##__VA_ARGS__)) abort();
static int filter_init(void)
 
  scmp_filter_ctx filter;
  if (!(filter = seccomp_init(SCMP_ACT_ALLOW))) abort();
  if (seccomp_attr_set(filter, SCMP_FLTATR_CTL_NNP, 1)) abort();
  filter_rule_add(SCMP_ACT_ERRNO(EINVAL), SCMP_SYS(open), 1, SCMP_A1(SCMP_CMP_MASKED_EQ, O_SYNC, O_SYNC));
  filter_rule_add(SCMP_ACT_ERRNO(0), SCMP_SYS(fsync), 0);
  filter_rule_add(SCMP_ACT_ERRNO(0), SCMP_SYS(fdatasync), 0);
  filter_rule_add(SCMP_ACT_ERRNO(0), SCMP_SYS(msync), 0);
  filter_rule_add(SCMP_ACT_ERRNO(0), SCMP_SYS(sync), 0);
  filter_rule_add(SCMP_ACT_ERRNO(0), SCMP_SYS(syncfs), 0);
  filter_rule_add(SCMP_ACT_ERRNO(0), SCMP_SYS(sync_file_range), 0);
  return seccomp_load(filter);
 
int main(__attribute__((unused)) int argc, char *argv[])
 
  if (argc <= 1)
   
    fprintf(stderr, "usage: %s COMMAND [ARG]...\n", argv[0]);
    return 2;
   
  if (filter_init())
   
    fprintf(stderr, "%s: can't initialize seccomp filter\n", argv[0]);
    return 1;
   
  execvp(argv[1], &argv[1]);
  if (errno == ENOENT)
   
    fprintf(stderr, "%s: command not found: %s\n", argv[0], argv[1]);
    return 127;
   
  fprintf(stderr, "%s: failed to execute: %s: %s\n", argv[0], argv[1], strerror(errno));
  return 1;
 

Julien Danjou: Ceilometer bug squash day #2

The Ceilometer team is pleased to announce that tomorrow Tuesday 5th March 2013 will be the second bug squash day for Ceilometer. We wrote an extensive page about how you can contribute to Ceilometer, from updating the documentation, to fixing bugs. There's a lot you can do. We've good support for Ceilometer built into Devstack, so installing a development platform is really easy. The main goal for this bug day will be to put Ceilometer in the best possible shape before the grizzly-rc1 release arrives (14th March 2013). This version of Ceilometer should be the last one before the final Grizzly release, so it's a pretty important one. We'll be hanging out on the #openstack-metering IRC channel on Freenode, as usual, so feel free to come by and join us!

3 March 2013

Philipp Kern: git-annex: encrypted remotes

Due to the data loss I blogged about, I had to reverse engineer the encryption used by git-annex for its encrypted special remotes. The file system on which the content lived has a bullet hole of 8 GB in it, which was helpfully discarded by pvmove. It's pretty unhappy about that fact, parts of the git repository are unusable and directories cannot be accessed anymore. git-annex cannot possibly run anymore.

However, I was still able to access the git-annex branch within said git repository (using porcelain). This branch contains a file called remote.log which contains the keys of the special remotes. There's one per remote, encrypted to a GPG key of your choice and all files within that remote are encrypted with the same symmetric key.

One small detail stopped me from getting the decryption right the first time, though. It seems that git-annex uses randomness generated by GPG and armored into base64. In my na vet I spotted the base64 and decoded it. Instead it's used verbatim: the first 256 bytes as HMAC key (which reduces randomness to 192 bytes) and the remaining bytes for the symmetric key used by GPG (which will do another key derivation for CAST5 with it). A bug about that just hit the git-annex wiki.

With that knowledge in mind I wrote a little tool that's able to generate encrypted content keys from the plain ones used in the symlinks. That helps you to locate the file in the encrypted remote. Fetch it and then use the tool to decrypt the file in question with the right key.

The lesson: Really backup the git repository used with git-annex and especially remote.log. I'm now missing most of the metadata but for some more important files it's luckily still present. Recovery of the file content does not depend on it if you can deduce the filename from the content. If you have many little files it might be a bit futile without it, though.

Petter Reinholdtsen: Frikanalen - Complete TV station organised using the web

Do you want to set up your own TV station, schedule videos and broadcast them on the air? Using free software? With video on demand support using free and open standards? Included a web based video stream as well? And administrate it all in your web browser from anywhere in the world? A few years now the Norwegian public access TV-channel Frikanalen have been building a system to do just this. The source code for the solution is licensed using the GNU LGPL, and available from github. The idea is simple. You upload a video file over the web, and attach meta information to the file. You select a time slot in the program schedule, and when the time come it is played on the air and in the web stream. It is also made available in a video on demand solution for anyone to see it also outside its scheduled time. All you need to run a TV station - using your web browser. There are several parts to this web based solution. I'll mention the three most important ones. The first part is the database of videos and the schedule. This is written in Django and include a REST API. The current database is SQLite, but the plan is to migrate it to PostgreSQL. At the moment this system can be tested on beta.frikanalen.tv. The second part is the video playout, taking the schedule information from the database and providing a video stream to broadcast. This is done using CasparCG from SVT and Media Lovin' Toolkit. Video signal distribution is handled using Open Broadcast Encoder. The third part is the converter, handling the transformation of uploaded video files to a format useful for broadcasting, streaming and video on demand. It is still very much work in progress, so it is not yet decided what it will end up using. Note that the source of the latter two parts are not yet pushed to github. The lead author want to clean them up a bit more first. The development is coordinated on the #frikanalen IRC channel (irc.freenode.net), and discussed on the frikanalen mailing list. The lead developer is Benjamin Bruheim (phed on IRC). Anyone is welcome to participate in the development.

2 March 2013

Rapha&#235;l Hertzog: My Free Software Activities in February 2013

This is my monthly summary of my free software related activities. If you re among the people who made a donation to support my work (78.31 , thanks everybody!), then you can learn how I spent your money. Otherwise it s just an interesting status update on my various projects. Debian packaging I wanted to update publican to the latest upstream release but I stopped after a few hours of work during which I filed two bugs that a modicum of testing should have caught before release. So I decided to wait for the next minor release. I uploaded python-django 1.4.4 and 1.4.5, new upstream maintenance and security releases which thus went into wheezy. I also prepared a stable update of Django (1.2.3-3+squeeze5) which required me to backport the last 2 sets of security patches. I uploaded a new revision of wordpress to fix a problem with TinyMCE (#700289) and to update/add many translation files (#697208). Bug reporting and misc fixes Live-build issue. I experienced some intermittent failures when building HDD live images with live-build on armel. Daniel Baumann directed me to the problematic piece of code (the oversizing of the image size was not enough) so I committed a small fix by increasing the oversizing factor to 6%. Live-config issue. I also reported another issue that I diagnosed in live-config (#701788), namely that the script which setups sudo was failing when the default user is root. git-buildpackage issue. I filed #700411 after noticing that git-import-orig imported the debian directory provided by upstream. Those directories are not used with 3.0 (quilt) source package and their presence in the upstream branch is thus harmful: any change to the upstream debian directory will result in conflicts when you merge a new upstream release in your packaging branch. rubygems integration. Later I had to package a bunch of ruby applications that were using Bundler and I wanted to reuse as many packaged ruby modules that I could. But for this, those modules had to provide the required rubygems meta-information. I filed #700419 to request those on rake-compiler and with the help of C dric Boutillier (and others on #debian-ruby), we identified a bunch of ruby modules which could get those with a simple recompilation. I filed bin-nmu requests in #700605. Misc bugs. simple-cdd offers to select profiles to install but I noticed that the associated debconf template was not translated (#700915). The startup scripts (provided by initscripts) in charge of activating the swap are supposed to handle a noswap kernel command line option to disable swap. In #701301, I reported that the option was not working correctly if quiet was present first in the command line due to spurious break statements. Debian France Administrative work. We were late for some legal procedures so I wrote the report of the last general assembly and sent it to the Tribunal d instance of Sarreguemines to record the changes in the administrative board. I also completed the special register of the association, it s a notebook that is legally required and that must document any important change in the governance structure of the association (new members of the board, headquarters change, new bylaws, etc.). Galette developments. Debian France is funding a few enhancements to the Galette free software that we re using to manage the association. I am in touch with the Galette developer to answer his questions and ensure that his work will meet our needs. Librement I have been looking for talented developers who have a genuine interest in my Librement project. I want to fund the initial development of the project but I don t have the means to fund it entirely. So I really wanted to find developers who would find an interest beside the money that I would pay. I got in touch with the team of developers from Scopyleft and they look like very good candidates. But they re heavy users of the Scrum development method and asked me to play the role of product owner . So I started to describe the project with user stories (i.e. create the backlog in the Scrum jargon), you can have a look at them here on trello.com. If you re interested by the topic of free software funding, feel free to review and to send me your comments. My goal is clearly to have a minimal viable product with the first iteration(s) that I fund and then use the platform itself to fund further developments of the project. Thanks See you next month for a new summary of my activities.

5 comments Liked this article? Click here. My blog is Flattr-enabled.

Philipp Kern: PSA: LVM, pvmove and SSDs

If you use LVM with Wheezy on a solid-state drive, you really want to install the latest lvm2 update (i.e. 2.02.95-6, which contains the changes of -5). Otherwise, if you set issue_discards=1 in /etc/lvm/lvm.conf, you will experience severe data loss when using pvmove. Happened to me twice, once I didn't care (chroot data being lost), the second time (today) I did. Not fun, especially when the backup of the data was scheduled for the same day.

One has to wonder why it takes three months for a bug that trashes data to reach testing. (Obviously I know the answer, but they're not particularly good reasons.) Other distributions, like Ubuntu, were much quicker to notice and incorporate that fix. And in the case of the named distribution not because they auto-synced it from unstable. If somebody notices such a grave bug, please yell at people to get the fix out there to our users. Thanks.

Joerg Jaspert: Goodbye Lenny

So this weekend was removal time, namely Lenny. I started out by getting all the different archives which had a Lenny release placed onto archive.debian.org, namely volatile, backports, security and of course the main archive. Actually, they got put onto one of the two hosts that hold this archive and are now syncing to the other. Today started by removing Lenny from those archives. As volatile is dead as an extra archive that was easy - DSA turned off the machine. Backports and security got all their files removed in one big removal after which I had them forget they ever dealt with Lenny. The main archive is a different story. Due to its size I can t simply remove all files at once. This would break every mirror, as they don t delete too many files at once. For the official mirror scripts that magic number is 40000. Both, backports and security, had less than that for their lenny suites, but the main archive has somewhere around 195k files that need to go away. Slightly more than the limit. For that reason I set out to only delete parts of the files for each mirror push. Lucky our toolset supports this easily, I simply had to declare the lenny suites empty and let our clean-suite tool do its work. Worked flawlessly - if one considers 8 hour runtime to mark the files as delete-able no flaw. Took another two hours to actually delete the first 20000 files out of our pool. The next mirror push will thus delete every lenny related entry in our dists/ tree and some 20000 files out of pool. The following mirror pushes will continue deletion, in batches of 10000, until all of lenny is gone.

Bastian Blank: LDAP, Insignificant Space and Postfix

For some, LDAP, just like X.500, is black magic. I won't argue against. Sometimes it really shows surprising behavior. It always makes sense, if you think about what LDAP is built for. One surprising behavior is the handling of the "Insignificant Space"
LDAP supports syntax and comparator methods. The syntax specifies how the entry should look. Usually this is some form of text, but numbers or stuff like telephone numbers are supported as well. The comparators specifies how the values are compared. Most of the text comparators are defined to apply the handling of insignificant spaces. Insignificant space handling normalizes the use of spaces. First all leading and trailing spaces are removed. All the internal spaces are normalized to at most two spaces. At the end all strings starts and ends with one space to allow proper sub-string matches. The resulting strings are used for comparisons. This behavior makes sense most of the time. If the user want to find something in the directory, he usually don't cares about spaces, but about content. But I found one occurrence where this produces some grieve. Postfix supports LDAP since some time. And lets say, it does not care about spaces in its queries. This is no problem as e-mail addresses do not contain spaces. Or do they?
Yes, e-mail addresses can contain spaces. This is not widely known, but still allowed. This addresses are quoted and the command looks like RCPT TO:<" test"@example.com>. The local part is quoted and contains a space at the beginning. And this is where the problem starts. Postfix sanitizes the address. It uses a simplified internal representation of the address for all lookups. So the address gets test@example.com. This form is used in all table lookups. The LDAP table uses the internal form of the address. This address is copied verbatim into the query. This query may look this way (mail= test@example.com). It is sent to the server this way. The LDAP server applies the insignificant space modifications. The query is now interpreted and the comparator specified modifications are applied. The query gets effectively changed to (mail=test@example.com). And this is where the fun starts. Postfix accepts undeliverable mails. Depending on the setup, such LDAP queries may be used to check for valid addresses. Because of the space handling, the sender can add spaces to the beginning of the address and it will still be considered valid. In later steps this addresses are not longer valid. Addresses starting with spaces are considered invalid in some locations of Postfix. What surprised me a bit is that virtual alias handling did not map them. The unmodified addressed showed up on the LMTP backend server. That's why they showed up my radar. I would say Postfix is wrong in this case. The LDAP server applies the spec correctly and defines spaces in e-mail addresses as insignificant. Postfix on the other side considers them significant. The easiest fix would be to not allow any spaces in the LDAP table.

Russ Allbery: Review: Asimov's, June 2011

Review: Asimov's Science Fiction, June 2011
Editor: Sheila Williams
Issue: Volume 35, No. 6
ISSN: 1065-2698
Pages: 112
Sheila Williams's editorial is about a new ebook-only anthology of Asimov's stories, with some discussion of electronic sales. This sort of business editorial is what she's best at writing, although in this case I'm far enough in my reading that I'm reading it through a bit of a time machine. Silverberg's essay is about popular bubbles and is mildly interesting, although I've read quite a lot about this at greater length. I think he misses the plot at the end, though, where he brings in the 2008 collapse and works on focusing the blame on speculators rather than those selling the fraudulent investments. James Patrick Kelly has a surprisingly graceful introduction to fanfic that does a better job at conveying the underlying enthusiasm than most mainstream coverage. The review column this month is Peter Heck's workman-like survey of recently released books of interest. "The Cold Step Beyond" by Ian R. MacLeod: Most of MacLeod's writing I've read has focused on lush world-building, usually with a mix of magic and industrial revolution. This felt like a departure: a character study of a heavily-modified warrior and monster hunter against a clearly science-fictional background. The protagonist has no memory of her past beyond her training in a religious institution, and is as much machine as woman. She is on one of her first missions to hunt dangerous monsters. But she's given almost no direction, and rather than encountering a monster, she encounters a feral woman with an ancient ray gun. MacLeod has a very SFnal twist waiting in the wings, but what I liked best were the character dynamics and the interplay of duty and self-determination. (8) "All the News That's Fit" by Carol Emshwiller: This story follows Emshwiller's normal pattern of telling the story of an outsider and representative of outsider culture trying to understand a society, but her protagonist here is both more alone and more proactive than her average protagonist. She lives in a distant mountain village only rarely visited by a traveling reporter who brings news from outside. When he doesn't come, she goes out looking for him, at first to rescue him if he ran into trouble and later to find the news herself. Much of the story has the reader worrying nervously for her as she observes the large city while clearly having no idea how to live there. The ending provides an interesting and (at least for me) unexpected twist on the rural person coming to the big city, but it was marred by a level of anger from the protagonist that I never thought was well-motivated. (6) "Walking Stick Fires" by Alan DeNiro: There is a science fiction subgenre that, for lack of a better term, I'll call post-apocalyptic surrealism. This is an entry in that strange subgenre, featuring two alien bikers having mostly unmotivated adventures with strange objects and even stranger creatures on an Earth that's apparently been invaded and mostly devastated by multiple parties. Like a lot of post-apocalyptic surrealism, it never makes much sense. It reminded me of Rucker but with a bit more characterization and less hand-waving math. (4) "Apocalypse Daily" by Felicity Shoulders: I really liked the premise for this story. The protagonist is a concept designer for a social-oriented MMORPG, but one where only scores and resources carry over rather than the whole world. The goal of the game is survival: the world ends regularly, globally or locally, in a huge variety of ways, and one scores points and earns resources for further iterations by finding ways to survive that catastrophe. The game is structured to encourage cooperation, at least at first. But Katrina gets into a mildly paranoid workplace conflict with someone else on her team, leading to some significant changes in the underlying expectations of the game. Unfortunately, while the background and idea are quite solid, the direction the story goes is predictable pessimism, the conclusion doesn't say much that I found interesting, and I ended up detesting the protagonist. I would have preferred a more unique psychological twist. (6) "The Fighter" by Colin P. Davies: This is a rather short story, essentially an idea study in the context of a police stop of a futuristic entertainment fighter. This is another one of those depressing takes on a common SF trope. I didn't entirely follow the world-building, but it was all dark and pointless enough that I didn't really want to dig further. (4) "Kiss Me Twice" by Mary Robinette Kowal: I've read a few police procedurals featuring AIs before, but this is by far the best one I've read. Huang is a detective, and like all the detectives he makes extensive use of the police department AI. Unlike most of the detectives, though, he treats her as a full partner and close to human, and she reciprocates with a Mae West persona and a closer relationship with him than most of the department. They're in the middle of investigating a murder when there is a raid on the police department and the AI herself is stolen. The rest of the story entangles their investigation with the theft of the AI. The police AI is, of course, restored from backup, but that's the first time that's ever happened to her, and she's shaken by it, and more disturbed by the knowledge that there are two copies of her running. The mystery angle of the story is reasonably satisfying, but the AI characterization and Huang's careful feeling-out of their relationship and appropriate behavior on his part is the best part of the story. This is a great story: long, satisfying, and thoroughly excellent. I'd happily read more fiction in this universe. (8) Rating: 7 out of 10

1 March 2013

Joerg Jaspert: tmux - like screen, just nicer. Replacing clusterssh too.

By now I am a long time happy user of tmux instead of screen. I started using it somewhere in 2011 and by now I only have one usage of screen left over (direct ttyS0 access for one serial). As usual I have a custom config for it, and if you are interested there is a link for you: tmux.conf. But the more interesting part and why I blog is that I also wrote a little helper around tmux. Ingeniously called tm. And a colleague said I should let other people use it too and place it somewhere accessible. So, err, here it is. Or better, the usage output of it, the actual script has a link somewhere below.
tmux helper by Joerg Jaspert <joerg@ganneff.de>
Call as: tm CMD [host]...
CMD is one of
 ls          List running sessions
 s           Open ssh session to host
 ms          Open multi ssh sessions to hosts, synchronizing input
 $anything   Either plain tmux session with name of $anything or
             session according to TMDIR file
TMDIR file:
Each file in $TMDIR defines a tmux session.
Filename is the commandline option $anything
Content is defined as:
  First line: Session name
  Second line: extra tmux options
  Any following line: A hostname to open a shell with in the normal
  ssh syntax. (ie [user@]hostname)
Environment variables recognized by this script:
TMPDIR - Where tmux stores its session information
         DEFAULT: If unset: /tmp
TMSORT - Should ms sort the hostnames, so it always opens the same
         session, no matter in which order hostnames are presented
         DEFAULT: true
TMOPTS - Extra options to give to the tmux call
         Note that this ONLY affects the final tmux call to attach
         to the session, not to the earlier ones creating it
         DEFAULT: -2
TMDIR  - Where are session information files stored
         DEFAULT: $ HOME /.tmux.d
TMWIN  - Where does your tmux starts numbering its windows?
         This script tries to find the information in your config,
         but as it only checks $HOME/.tmux.conf it might fail.
         So if your window numbers start at anything different to 0,
         like mine do at 1, then you can set TMWIN to 1

It is mainly useful for using tmux as a ssh-multiplexer and replacement for clusterssh though its happy to do any other tmux session too. As the help text may already have told you, you can use it to list existing tmux sessions (oh, wow!). Honestly, that one is there because tm ls is shorter than tmux ls More interestingly, tm s HOSTNAME opens a tmux session with the first window being a ssh shell to HOSTNAME. Should you later type tm s HOSTNAME again, it will attach to the existing session. But the real kicker for me (and colleagues) actually is the ms subcommand of it. ms being short for multi-session, it is my way of no-longer using clusterssh. Basically you type tm ms HOST1 user@HOST2 and it opens a session with one tmux window consisting of two panes. Showing you host1 and host2 at the same time. And the input to both is synchronized, that is - all your input is send to both of them. You aren t limited to just two hosts, do as many as you wish. Actually, I have no idea if tmux has an upper limit, but I did already have more than 100 hosts open at once. The major limitation there is not tmux - but simply the available screenspace, as all hosts are shown in ONE window! (The one advantage of clusterssh, it uses one xterm per host) This is quite helpful in all kinds of administration work involving clusters - imaging installing a package on 5 hosts of a cluster, you usually want it on all of them - or any kind of situations where one wants to run one command on many systems. As I m already said, I m lazy. And so there is a way to easily define multisessions that one uses regulary, by creating files inside the directory behind $TMDIR, usually $ HOME /.tmux.d. They are simple text files and let you easily define sessions you can then (re)open with a simple tm whateveritsname. And if you want the whole script by now: tm

Daniel Pocock: Monitoring JBoss, Tomcat and other application servers with JMXetric

JMXetric 1.0 was finally released in February, let's have a quick look at how conveniently it can be integrated with an application server. A more thorough analysis of gmetric4j and JMXetric can be found in the sections I contributed to the new O'Reilly book Monitoring with Ganglia General comments about JVM instrumentation and the JVM boot classpath Monitoring agents (there are a number of them out there) are loaded at an early stage by the JVM and it needs to have access to any of their dependencies through the boot classpath. JMXetric explicitly declares it's dependencies in MANIFEST.MF:
Boot-Class-Path: oncrpc-1.0.7.jar gmetric4j-1.0.2.jar
As no absolute path is specified, the JVM will look for those JARs in the same directory where the jmxetric.jar is located. Furthermore, the JVM will expect the exact filenames to match. If other filenames are used (e.g. for a newer version of gmetric), then the JVM command line needs to specify the filenames and locations, for example:
-Xbootclasspath/a:/opt/ganglia/jmxetric/gmetric4j-1.0.3.jar
Starting JBoss 7 with JMXetric To start with, unpack a vanilla JBoss distribution:
$ mkdir ~/opt
$ cd ~/opt
$ tar xzf ~/Downloads/jboss-as-7.1.0.Final.tar.gz
Now get the JMXetric code and put it, together with dependencies, in the same directory:
$ mkdir -p ~/opt/ganglia/jmxetric
$ cd ~/opt/ganglia/jmxetric
$ wget http://search.maven.org/remotecontent?filepath=info/ganglia/jmxetric/jmxetric/1.0.2/jmxetric-1.0.2.jar
$ wget http://search.maven.org/remotecontent?filepath=info/ganglia/gmetric4j/gmetric4j/1.0.2/gmetric4j-1.0.2.jar
$ wget http://search.maven.org/remotecontent?filepath=org/acplt/oncrpc/1.0.7/oncrpc-1.0.7.jar
Set up a config file for JMXetric:
$ mkdir -p ~/opt/ganglia/jmxetric/etc
$ cd ~/opt/ganglia/jmxetric/etc
$ wget https://raw.github.com/ganglia/jmxetric/master/etc/jmxetric.xml
$ cp jmxetric.xml jmxetric-jboss.xml
$ vi jmxetric-jboss.xml
and insert the following code:
                <!-- report web server request count -->
		<mbean name="jboss.as:subsystem=web,connector=http" pname="http">
			<attribute name="requestCount" type="int32"/>
		</mbean>
Finally, tell JBoss about the agent:
$ cd ~/opt/jboss-as-7.1.0.Final
$ vi bin/standalone.conf
and apply the following changes to standalone.conf:
--- bin/standalone.conf
+++ ../jboss-as-7.1.0.Final.for-jmxetric/bin/standalone.conf
@@ -4,6 +4,7 @@
 ##                                                                          ##
 ##############################################################################
 
+JBOSS_MODULES_SYSTEM_PKGS="org.jboss.logmanager"
 #
 # This file is optional; it may be removed if not needed.
 #
@@ -48,6 +49,8 @@
 #
 if [ "x$JAVA_OPTS" = "x" ]; then
    JAVA_OPTS="-Xms64m -Xmx512m -XX:MaxPermSize=256m -Djava.net.preferIPv4Stack=true -Dorg.jboss.resolver.warning=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000"
+   JAVA_OPTS="$JAVA_OPTS -Djava.util.logging.manager=org.jboss.logmanager.LogManager -Xbootclasspath/p:$JBOSS_HOME/modules/org/jboss/logmanager/main/jboss-logmanager-1.2.2.GA.jar -Xbootclasspath/p:$JBOSS_HOME/modules/org/jboss/logmanager/log4j/main/jboss-logmanager-log4j-1.0.0.GA.jar -Xbootclasspath/p:$JBOSS_HOME/modules/org/apache/log4j/main/log4j-1.2.16.jar"
+   JAVA_OPTS="$JAVA_OPTS -javaagent:$HOME/opt/ganglia/jmxetric/jmxetric-1.0.2.jar=host=jboss-host.example.org,port=8649,wireformat31x=true,config=$HOME/opt/ganglia/jmxetric/etc/jmxetric-jboss.xml"
    JAVA_OPTS="$JAVA_OPTS -Djboss.modules.system.pkgs=$JBOSS_MODULES_SYSTEM_PKGS -Djava.awt.headless=true"
    JAVA_OPTS="$JAVA_OPTS -Djboss.server.default.config=standalone.xml"
 else
Note that we are adding some of the core JBoss logging code to the boot classpath - that is because JMXetric depends on the logger, and like many monitoring agents, it is impacted by JBoss bug AS-3681. Adding these extra classpath settings is the workaround. Now go ahead and start JBoss, you should see a log message informing you that JMXetric is operational:
$ cd ~/opt/jboss-as-7.1.0.Final
$ chmod a+x bin/*.sh
$ bin/standalone.sh
=========================================================================
  JBoss Bootstrap Environment
  JBOSS_HOME: ...
.
.
.
=========================================================================
JMXetricAgent instrumented JVM, see https://github.com/ganglia/jmxetric
12:59:34,395 INFO  [info.ganglia.gmetric4j.GMonitor] Setting up 1 samplers
12:59:34,536 INFO  [org.jboss.modules] JBoss Modules version 1.1.1.GA
12:59:34,617 INFO  [org.jboss.msc] JBoss MSC version 1.0.2.GA
12:59:34,653 INFO  [org.jboss.as] JBAS015899: JBoss AS 7.1.0.Final "Thunder" starting
.
.
.
I hope this example is useful for you. Please feel free to share recipes you develop for collecting useful JVM metrics, and also any custom charts/reports through collaboration with the Ganglia community mailing list, the Ganglia wiki or your own blog.

Thomas Goirand: Openstack Grizzly (2013.1~g3) available

This post is just a status update for the Openstack packaging, after the next version froze last week. The Openstack bi-annual summit of next April will take place this year in Portland, Oregon, and if everything goes as plan, Grizzly will be released just before the summit. Grizzly will be out a bit before the next Ubuntu in April, as releases are following the one of Ubuntu. Openstack uses town names for it s release names: Austin, Bextar, Cactus (2011.1), Diablo (2011.2), Essex (2012.1), Folsom (2012.2), Grizzly (2013.1). I started to seriously work on the Openstack packaging in October, and never stopped working on its packaging until now. Slowly, but surely, preparing all the packages and their dependency python modules. One package at a time, working all this every day. Folsom works now pretty well and can be used in production, and I maintain it for security (along with Essex, which is in Wheezy). Then Grizzly was frozen last week, on the 22nd of February, with the G3 release. As I already worked on packaging the G2 release in January, managing the packaging of G3 was fast. On late Sunday, I had a repository with Grizzly built, and its corresponding python (build-)dependencies. But while just building your own repository is easy, having all the dependencies in Debian is a lot more work. As of today, if I include all the python modules, I have (at least) touch around 50 packages in total when working on Openstack. Many of them were simply built from scratch. The only python dependency that needs an upgrade in Experimental, so that all dependencies are satisfied, is a new version of pep8. The rest of is new python modules that were not in Debian, and which are currently waiting in the NEW queue for ftp masters approval: python-pecan, python-tablib, python-wsme and websockify. Some of these python modules have been waiting there for a long time, like python-pecan (it s been waiting in the NEW queue for more than 35 days now), some like websockify and python-wsme have been uploaded only this week. I really hope it can be possible to have all of Grizzly in Debian before the next Openstack summit (this depends mainly on the ftp-masters). Note that I do not intend to apply security patches in Grizzly until it is released as the new Openstack stable. So use my private Grizzly repository as your own risk. I intend to fix this by working on some constant integration to have nightly builds, like many people are doing with Openstack. If you want to try it out: deb http://archive.gplhost.com/debian grizzly main

28 February 2013

Niels Thykier: Wheezy release progress (February)

About 5-6 weeks ago, I wrote about the Wheezy release progress, so it is about time for another update. According to UDD, we are down to 204 RC bugs (down from 249, since my last post). It is not quite the 2.4 RC bugs per day actually it is about 1.1 RC bug a day. Unfortunately, we do not appear to be fixing bugs faster than we are reporting them at the moment. If you look at Richard Hartmann s post from last week, then we had 206 RC bugs left. Even worse, we appear to have regressed between week 7 and 8 of this year (194 to 206). If you want the Wheezy release to happen soon, please consider helping us by providing bug fixes that comply with the Wheezy freeze policy. If the pace of RC bug fixes do not pick up, the alternative is that the release team deals with the bugs. Note that deals generally falls into one of 2 categories. Either we defer/ignore the problem for Wheezy[1] or we remove affected packages from Wheezy/testing. Particularly, if we have to remove packages, they may take reverse dependencies with them as collateral damage. I do not like these tools anymore than you do. But if the RC bugs fixes are not coming in, it is the only two tools we have left. [1] Meaning that at best the bug fix will occur at Wheezy point release at worst, not at all.

Erich Schubert: ELKI data mining in 2013

ELKI, the data mining framework I use for all my research, is coming along nicely, and will see continued progress in 2013. The next release is scheduled for SIGMOD 2013, where we will be presenting the novel 3D parallel coordinates visualization we recently developed. This release will bear the version number 0.6.0.
Version 0.5.5 of ELKI is in Debian unstable since december (Version 0.5.0 will be in the next stable release) and Ubuntu raring. The packaged installation can share the dependencies with other Debian packages, so they are smaller than the download from the ELKI web site.
If you are developing cluster analysis or outlier detection algorithm, I would love to see them contributed to ELKI. If I get a clean and well-integrated code by mid june, your algorithm could be included in the next release, too. Publishing your algorithms in source code in a larger framework such as ELKI will often give you more citations. Because it is easier to compare with your algorithm then and to try it on new problems. And, well, citations counts are a measure that administration loves to judge researchers ...
So what else is happening with ELKI:
  • The new book "Outlier Analysis" by C. C. Aggarwal mentions ELKI for visual evaluation of outlier results as well as in the "Resources for the Practioner" section and cites around 10 publications closely related to ELKI.
  • Some classes for color feature extraction of ELKI have been contributed to jFeatureLib, a Java library for feature detection in image data.
  • I'd love to participate in the Google Summer of Code, but I need a contact at Google to "vouch" for the project, otherwise it is hard to get in. I've been sending a couple of emails, but so far have not heard back much yet.
  • As the performance of SVG/Batik is not too good, I'd like to see more OpenGL based visualizations. This could also lead to an Android based version for use on tablets.
  • As I'm not an UI guy, I would love to have someone make a fancier UI that still exposes all the rich functions we have. The current UI is essentially an automatically generated command line builder - which is nice, as new functionality shows up without the need to modify UI code. It's good for experienced users like me, but hard for beginners to get started.
  • I'd love to see integration of ELKI with e.g. OpenRefine / Google Refine to make it easier to do appropriate data cleaning and preprocessing
  • There is work underway for a distributed version running on Hadoop/YARN.

27 February 2013

Daniel Pocock: Understanding the WebRTC architecture from a free software perspective

WebRTC is here, now. Google launched Chrome 25 with built-in WebRTC support on Monday, 25 February. Within 24 hours, DruCall, an easy-to-use WebRTC module for Drupal was available Free as a shiny new GPL project. While this technology offers a level of convenience for end-users that is unprecedented, just what is involved for the keen server administrator who wants to deploy this today? With 20-30% of web users expected to be running WebRTC browsers within 3 months (due to automatic updates), and 60-70% by Christmas, will the free software community be able to seize this opportunity to redefine the way real-time communications is deployed? I've whipped together a brief architecture diagram to answer those questions, demonstrating how all core components can be built with free software, today. All but one of these components is conveniently packaged in stable Linux distributions like Debian and Ubuntu. The full diagram is taken from the DruCall web site where you can find more details:

Petter Reinholdtsen: Dr. Richard Stallman, founder of Free Software Foundation, give a talk in Oslo March 1st 2013

Dr. Richard Stallman, founder of Free Software Foundation, is giving a talk in Oslo March 1st 2013 17:00 to 19:00. The event is public and organised by Norwegian Unix Users Group (NUUG) (where I am the chair of the board) and The Norwegian Open Source Competence Center. The title of the talk is The Free Software Movement and GNU , with this description:
The Free Software Movement campaigns for computer users' freedom to cooperate and control their own computing. The Free Software Movement developed the GNU operating system, typically used together with the kernel Linux, specifically to make these freedoms possible.
The meeting is open for everyone. Due to space limitations, the doors opens for NUUG members at 16:15, and everyone else at 16:45. I am really curious how many will show up. See the event page for the location details.

Next.

Previous.