Search Results: "reg"

12 February 2012

Nikita Youshchenko: I ve been away from Windows for more than a decade but

but I ve already forseen the unwanted meet for some time :( . My elder daughter does need to run educational software from school. And my wife wants to run education software for our younger son. And making all that running under different emulation environments does eat my time, and result is not always good. And also children start to want games Issue 1. Need hardware to run more or less modern windows on. For last 6 years, my home network consists of a more or less powerful headless server, and 3 terminals to access that. Terminals are definitely too weak for windows. And server is needed in 24/7 mode so sorry no windows there. Likely solution for this is one of 3 terminals is actually dying (it is doing so already for months but still works, just hangs sometimes) and needs a replacement, so I can just replace it now and use more powerful hardware. Issue 2. Windows host or windows VM? While windows VM feels preferable for a such a deep-linux person as myself, I tend to think that dual-boot host is better here. If the very reason of allowing windows into this house is make it easy to install and run random windows software, then doing so in VM environment is explicitly against the goal. Issue 3. Use unlicensed windows or buy a license? Although absolute majority of windows users here is Russia use unlicensed windows, it is somewhat unfair to promote free software and use unlicensed windows at the same time. And windows home basic license is not that expensive after all. The setup that makes me less uncomfortable than others is:
- obtain powerful enough hardware, and install that instead of failing terminal,
- obtain windows 7 home basic license
- install dual boot linux + windows, and under linux install windows VM
- when needed, try windows VM first, and in case of any issues, immediately reboot into windows host. What I can t find out for some time already is does microsoft retail license allow to have both host and VM installation on the same computer? These will be physically different installations, but on the same computer, and will never run at the same time. License wording does not give exact answer on this. I wanted to ask on Microsoft s support forum but failed to register there. After asking me quite a few things it claimed that there is some temporary problem and suggested to retry later. I retried later and got the same. I then asked here but looks like nobody knows. At least I want to know, what will happen if I will try to activate windows in VM after I ve activated windows (with the same key) on host. Will this render both installations non-genuine ? Also I d appreciate any comments on the whole situation :)

Iustin Pop: perf stats for doing nothing

Perf stats for "doing nothing" I've recently discovered the perf Linux tool. I heard that oprofile was deprecated and that there is a new tool, and I noted down to try it sometime. Updated: more languages, fixed typos, more details, some graphs. Apologies if this shows twice in your feed. The problem with perf stats is that I hate bloat, or even perceived bloat. Even when it doesn't affect me in any way, the concept of wasted cycles makes me really sad. You probably can guess where this is going I said, well, let's see what perf says about a simple "null" program. Surely doing nothing should be just a small number of instructions, right? Note: I think that perf also records kernel-side code, because the lowest I could get was about ~50K instructions for starting a null program in assembler that doesn't use libc and just executes the syscall asm instruction. However, these ~50K instructions are noise the moment you start to use more high-level languages. Yes, this is expected, but the I was still shocked. And there's lots of delta between languages I'd expected to behave somewhat identical. Again, this is not important in the real world. At all. They are just numbers, and probably the noise (due to short runtime) has lots of influence on the resulting numbers. And I might have screwed up the measurements somehow. Test setup Each program was the equivalent of 'exit 0' in the appropriate form for the language. During the measurements, the machine was as much as possible idle (single-user mode, measurements run at real-time priority, etc.). For compiled languages, -O2 was used. For scripts, a simple #!/path/to/interpreter (without options, except in the case of Python, see below) was used. Each program/script was run 500 times (perf's -r 500) and I've checked that the variations were small ( 0.80% on the metrics I used). You can find all the programs I've used at http://git.k1024.org/perf-null.git/, the current tests are for the tag version perf-null-0.1. The raw data for the below tables/graphs is at log-4. Results Compiled languages
Language Cycles Instructions
asm 63K 51K
c-dietlibc 74K 57K
c-libc-static 177K 107K
c-libc-shared 506K 300K
c++-static 178K 107K
c++-dynamic 1,750K 1,675K
haskell-single 2,229K 1,338K
haskell-threaded 2,629K 1,522K
ocaml-bytecode 3,271K 2,741K
ocaml-native 1,042K 666K
Going from dietlibc to glibc doubles the number of instructions, and for libc going from static to dynamic linking again roughly doubles it. I didn't manage to compile a program dynamically-linked against dietlibc. C++ is interesting. Linked statically, it is in the same ballpark as C, but when linked dynamically, it executes an order of magnitude (!) more instructions. I would guess that the initialisation of the standard C++ library is complex? Haskell, which has a GC and quite a complex runtime, executes slightly less instructions than C++, but uses more cycles. Not bad, given the capabilities of the runtime. The two versions of the Haskell program are with the single-threaded runtime and with the multi-threaded one; not much difference. A fully statically-linked Haskell binary (not recommended usually) goes below 1M instructions, but not by much. OCaml is a very nice surprise. The bytecode runtime is a bit slow to startup, but the (native) compiled version is quite fast to start: only 2 number of instructions and cycles compared to C, for an advanced language. And twice as fast as Haskell . Nice! Shells
Language Cycles Instructions
dash 766K 469K
bash 1,680K 1,044K
mksh 1,258K 942K
mksh-static 504K 322K
So, dash takes ~470K instructions to start, which is way below the C++ count and a bit higher than the C one. Hence, I'd guess that dash is implemented in C . Next, bash is indeed slower on startup than dash, and by slightly more than 2 (both instructions and cycles). So yes, switching /bin/sh from bash to dash makes sense. I wasn't aware of mksh, so thanks for the comments. It is, in the static variant, more efficient that dash, by about 1.5 . However, the dynamically linked version doesn't look too great (dash is also dynamically linked; I would guess a statically-linked dash "beats" mksh-static). Text processing I've added perl here (even though it's a 'full' language) just for comparison; it's also in the next section.
Language Cycles Instructions
mawk 849K 514K
gawk 1,363K 980K
perl 2,946K 2,213K
A normal spread. I knew the reason why mawk is Priority: required is that it's faster than gawk, but I wouldn't have guessed it's almost twice as fast. Interpreted languages Here is where the fun starts
Language Cycles Instructions
lua 5.1 1,947K 1,485K
lua 5.2 1,724K 1,335K
lua jit 1,209K 803K
perl 2,946K 2,213K
tcl 8.4 5,011K 4,552K
tcl 8.5 6,888K 6,022K
tcl 8.6 8,196K 7,236K
ruby 1.8 7,013K 6,128K
ruby 1.9.3 35,870K 35,022K
python 2.6 -S 11,752K 10,247K
python 2.7 -S 11,438K 10,198K
python 3.2 -S 29,003K 27,409K
pypy -S 21,106K 10,036K
python 2.6 25,143K 21,989K
python 2.7 47,325K 50,217K
python 2.7 -O 47,341K 50,185K
python 3.2 113,567K 124,133K
python 3.2 -O 113,424K 124,133K
pypy 90,779K 68,455K
The numbers here are not quite what I expected. There's a huge delta between the fastest (hi Lua!) and the slowest (bye Python!). I wasn't familiar with Lua, so I tested it thanks to the comments. It is, I think, the only language which actually improves from one version to the next (bonus points), and where the JIT version also make is faster. In context, lua jit starts faster than C++. Perl is the one that goes above C++'s instructions count, but not by much. From the point of view of the system, a Perl 'hello world' is only about 1.3 -1.6x slower than a C++ one. Not bad, not bad. Next category is composed of TCL and Ruby, both of which had older versions 2-3 slower than Perl, but whose most recent versions are even more slower. TCL has an almost constant slowdown across versions (5M, 6.9M, 8.2M cycles), but Ruby seems to have taken a significant step backwards: 1.9.3 is 5 slower than 1.8. I wonder why? As for TCL, I didn't expect it to be slower to startup than Perl; good to know. Last category is Python. Oh my. If you run perf stat python -c 'pass' you get some unbelievable numbers, like 50M instructions to do, well, nothing. Yes, it has a GC, yes, it does import modules at runtime, but still On closer investigation, the site module and the imports it does do eat a lot of time. Running a simpler python -S brings it back to a more reasonable 10M instructions, which is in-line with the other interpreted languages. However, even with the -S taken into account, Python also slows down across versions: a tiny improvement from 2.6 to 2.7, but (like Ruby) a 3 slowdown from 2.7 to 3.2. Trying the optimised version (-O) doesn't help at all. Trying pypy, which was based on Python 2.7, makes it around 2 slower to startup (both with and without -S). So in the interpreted languages, it seems only Lua is trying to improve, the rest of the languages are piling up bloat with every version. Note: I should have tried multiple perl versions too. Java Java is in its own category; you guess why , right? GCJ was version 4.6, whereas by java below I mean OpenJDK Runtime Environment (IcedTea6 1.11) (6b24-1.11-4).
Language Cycles Instructions
null-gcj 97,156K 74,576K
java -jamvm 85,535K 80,102K
java -server 147,174K 136,803K
java -zero 132,967K 124,977K
java -cacao 229,799K 205,312K
Using gcj to compile to native code (not sure whether that's native-native or something else) results in a binary that uses less than 100M cycles to start, but the jamvm VM is faster than that (85M cycles). Not bad for java! Python 3.2 is slower to startup yes, I think the world has gone crazy. However, the other VMs are a few times slower: server (the default one) is ~150M cycles, and cacao is ~230M cycles. Wow. The other thing about java is that it was the only one that couldn't be put nicely in a file that you just exec (there is binfmt_misc indeed, but that doesn't allow different Java classes to use different Java VMs, so I don't count this), as opposed to every single other thing I tested here. Someone didn't grow on Unix? Comparative analysis Since there are almost 4 orders of magnitude difference between all the things tested here, a graph of cycles or instructions is not really useful. However, cycles/instruction, branches percentage and branches miss-predicted percentage can be. Hence first the cycles/instructions: Cycles/instruction Pypy is jumping out of the graph here, with the top value of over 2 cycles/instruction. Lua JIT is also bigger than Lua non-JIT, so maybe there's something to this (mostly joking, two data points don't make a series). On the other hand, Python wins as best cycles/instruction (0.91). Lots of ILP, to get below 1? Java gets, irrespective of VM, consistently near 1.0-1.1. C++ gets very different numbers between static linking (1.666) and dynamic linking (1.045), whereas C has basically identical numbers. mksh also has a difference between dynamic and static linking. Hmm Ruby, TCL and Python have consistent values across versions. And that's about what I can see from that graph. Next up, percentage of branches out of total instructions and percentage of branches missed: Branch statistics Note that the two lines shouldn't really be on the same graph; for the branch %, the 100% is the total instructions count, but for the branch miss %, the 100% is the total branch count. Anyway. There are two low-value outliers: So it seems the gcj libraries are well optimised? I'm not familiar enough with this topic, but on the graph it does indeed stand out. On the other end, mksh-static has a high branch miss percentage: 11.60%, which jumps clearly ahead of all the others; this might be why it has a high cycles/instruction count, due to all the stalls in misprediction; one has to wonder why it confuses the branch predictor? I find it interesting that the overall branch count is very similar across languages, both when most of the cost is in the kernel (e.g. asm) and when the user-space cost heavily over-weighs the kernel (e.g. Java). The average is 20.85%, minimum is 17.46%, max 22.93%, standard deviation (if I used gnumeric correctly) is just 0.01. This seems a bit suspicious to me . On the other hand, the mispredicted branches percentage varies much more: from a measly 2.82% to 11.60% (5x difference). Summary So to recap, counting just instructions: Overall, you get roughly three orders of magnitude slower startup between a plain C program using dietlibc and Python. And all, to do basically nothing. On the other hand, I learned some interesting things while doing it, so it wasn't quite for nothing .

Gregor Herrmann: RC bugs 2012/05-06

I was at FOSDEM over the last weekend, so here's my report for two weeks now:

10 February 2012

Raphaël Hertzog: People behind Debian: Ana Beatriz Guerrero L pez, member of the Debian KDE team

If you met Ana, you ll easily remember her. She has a great and pronounced Spanish accent :-) I m glad that the existence of the Debian Women project helped her to join Debian because she has been doing a great job. From KDE packaging to publicity/marketing work, her interests shifted over the years but this allowed her to stay very involved. As she explains it very well, Debian is big enough so that you can stop doing something which is no longer fun for you, and still find something new to do in another part of Debian! Read on to learn more about Ana, the KDE team, Debian s participation to the Google Summer of Code, and more. Raphael: Who are you? Ana: I m Ana Guerrero L pez and I m in my early 30s. I was born and raised in the wonderful city of Sevilla, Spain and I live in Lyon, France. I share my life with another Debian Developer and my paid work is doing Debian support and integration, so you won t be surprised to read that Debian is a big part of my life. Raphael: How did you start contributing to Debian? Ana: Although I knew about the existence of Linux since 1997 or so, I didn t really start using Linux until the summer 2001 when I finally got a computer on my own and an Internet link at home. In the beginning, I was using Mandrake in a dual boot with Windows and later around 2003, I happily moved to only using Debian and ditching the Windows partition. Once settled as a Debian user, I knew anybody could help improve the distribution but I hesitated to join mostly due to two reasons, my perception of Debian was the one of a very elitist and aggressive club and who wants to join this kind of cult^wproject? And even if I wanted to join, I did not know how to get started. By the summer of 2004, the Debian Women project started, it made me seeing Debian as a more welcoming project, and I started maintaining my first packages. The following summer 2005, I attended akademy 2005 (the annual KDE conference) where I had the pleasure to meet there some of the people from the KDE team and this really made a difference for me. Christopher Martin and Adeodato Sim , with the help of other people, have started the maintenance of KDE as a team a few months before and by that time most of the KDE modules where under the maintenance umbrella of the team. This was a very good move since it allowed easily to share the KDE maintenance in a more coordinated way and also eased having non-DDs, like me at that time, to join in and help.
The Debian Women project started, it made me seeing Debian as a more welcoming project.
Raphael: You re part of the Debian KDE team. What s your role in the team and what are your plans for Wheezy? Ana: Nowadays, I am not as active in the KDE team as I used to be in the past. The KDE 3 to KDE 4 transition was quite tiring and changes on the KDE side like the successive marketing renames, the shorter 6 months schedule (it used to be at least 9) or the uncoordinated KDE releases mostly burnt me out. Currently, I am mostly working in helping others to get started within the team, some small fixes here and there, and helping with the uploads: an upload of the full KDE suite to the archive requires some building power and upload bandwidth not everybody have. For Wheezy, with the tentative freeze date in June, the plan is to try to ship the latest possible point release of the KDE 4.8 series. The first release of the series, 4.8.0 was released a couple of weeks ago and while writing these lines, the packaging work for 4.8 hasn t started yet. The next move for the team is getting 4.7.4 in unstable, currently sitting in experimental.
For Wheezy, [ ] the plan is to try to ship the latest possible point release of the KDE 4.8 series.
Besides the KDE packages, there is some software which users perceive as KDE, such as amarok, digikam, etc., which are not part of KDE but fall under its umbrella. These other programs have their own maintainers and their updates depend greatly in the availability of them. For the KDE office suite, we have right now KOffice in the archive. KOffice got a fork some time ago named Calligra and we should replace KOffice by Calligra in the archive before the release of Wheezy. Sadly there isn t yet a final release of Calligra to use. My personal goal for Wheezy was to finish the removal of all the remaining packages depending on KDE 3 and Qt 3 that Squeeze still contained. The removal of the KDE 3 libraries and all the packages using them was quickly achieved after the release of Squeeze. The removal of Qt 3 soon showed that it was task harder than expected since some popular packages (sometimes not in the Debian archive, e.g. third-party scientific software) depend on it, and also Qt 3 is a requirement for LSB compatibility. Right now, Qt 3 has been orphaned for 9 months and nobody has shown any interest in adopting it. Raphael: KDE, much like GNOME, has been forked by people who were unhappy by the direction that the project has taken since version 4 (cf Trinity). What s your personal opinion on KDE 4.x and what s the position of the Debian KDE team concerning this fork? Ana: I use KDE 4 on my laptop and I think it is a solid desktop environment and platform. However I am finding it less and less attractive for me. On one side, my usage of the computer has been slightly changing and on the other side, I do not like how the new developments in KDE are evolving, things like plasmoids or activities are not attractive for me. I have switched my other 2 systems to awesome although I continue to use mainly a bunch of KDE applications: dolphin, konsole, kate, juk, kmix, etc. So you might say my desktop environment is an awesome KDE. Regarding the Trinity project, a lot of users complained very loudly when KDE 3 got replaced by KDE 4 in testing/unstable, so I find quite laudable the decision of some users to act instead and try to continue with a forked development of KDE 3. However the Trinity team seems to be about 3 persons (funny for a project named Trinity :) ) while KDE 3 is big. In perspective, it does not look that big because KDE 4 is even larger, but it is still too much for such small team. In addition those developers need to maintain Qt3 that has been end-of-lifed years ago by Nokia/Trolltech . So my guess is that sooner or later the project will fade away. Nobody from the KDE team is interested in Trinity and in case someone wants to package it for Debian, they would have to make a new team. For the reasons mentioned above: Qt3 maintenance and reduced upstream group, this would be a bad idea. My advice if you do not like KDE 4 and you miss KDE 3, would be taking a look at razor-qt based on Qt4 and quite similar to KDE 3. I read they have plans to port it to Qt4, but frankly that could take some years same it took to the KDE project for KDE 4.0.0 ;-) Raphael: You used to maintain news.debian.net, a WordPress blog dedicated to Debian, but you stopped a while ago. A few months later you started to maintain a Debian page on Google+. Why did you stop the blog and what s your goal with the Google+ page? Ana: I blogged about the reasons I started news.debian.net. In short, I thought Debian needed a better system to publish news, something like a blog. I first tried to suggest the idea to the press/publicity team but they weren t interested, so I started the project alone. IMHO the blog worked quite well and I was feeling like it should be made official. I talked about this with some people but at the time I wasn t pushing it because I had other priorities and I knew pushing it to become official would need some extra time and energy. Stefano decided to start the discussion about making news.debian.net official (that s moving it to a debian.org domain) in its own initiative. After the public discussion and some private exchange of emails with DSA, the situation became frustrating and I decided to close news.debian.net after the release of Squeeze. Later, during DebConf, an officer from the press team announced they were launching a blog and I asked Stefano if he could try to have a discussion about this to see if it could still somehow fit my ideas, and maybe contributing myself, but nobody from the press team answered Stefano s email and the blog hasn t started yet either. Irony that communication didn t work when wanting to improve communication. About the Google+ page, everyday I follow what is going in Debian and quite often I find things I want to share. I do not want to clutter my own profiles with Debian stuff or have people following me because of that, so I decided to create the Debian page when Google+ made them available. I like the fact that people can follow that without having an account in Google+ although they can not comment anonymously. I am not happy about the fact that Google+ is a closed platform but hopefully the data will become easier to export in the near future. Right now, there are some services that provides RSS feeds of Google+ pages if you want to follow the page and you are not in Google+ (or I could setup one if several people ask me). Raphael: Last year you helped to manage Debian s participation to the Google Summer of Code. How did it went? Is there something that you can improve for this year? Ana: I think last year we managed to have people in Debian more aware about what the students were doing. That also helped students to get more feedback and therefore get to know more people in the project and get more integrated. Students were sending periodic public reports available to everybody interested in the status of the projects and some of them also held their own sessions in DebConf. We still failed to start looking for mentors early enough and to give them information about how the GSoC worked and how they could have a successful project. Having good projects in Debian is harder than in other projects because the GSoC mostly promotes having students started in Open Source *coding* for a project, while Debian is more a project about integrating software and we overall do not have so many parts that has to be coded. My personal goal for this year is to try getting the projects earlier to attract good students from the very beginning, even if that means we have less projects than in other years. Raphael: What motivates you to continue to contribute year after year? Ana: Three things. I like improving the OS I use, I like the friends I have made while working in Debian through the years and because I have fun. Also Debian is quite a big project, so if you become tired or burn out working in some area, you always can easily find interesting things to do somewhere else. Raphael: Is there someone in Debian that you admire for their contributions? Ana: Adeodato Sim , he is now in a long leave from the project, but it is one of those persons who made a difference in the project in his job in the release team some years ago. Aur lien Jarno because of his tireless work in (e)glibc and porting of several architectures. I also have special admiration for all those people who have been very active in the project for more than 7-8 years because I know it is not always easy to combine it with real life.
Thank you to Ana for the time spent answering my questions. I hope you enjoyed reading her answers as I did. Note that older interviews are indexed on wiki.debian.org/PeopleBehindDebian.

Subscribe to my newsletter to get my monthly summary of the Debian/Ubuntu news and to not miss further interviews. You can also follow along on Identi.ca, Google+, Twitter and Facebook .

7 comments Liked this article? Click here. My blog is Flattr-enabled.

Russell Coker: Starting with BTRFS

Based on my investigation of RAID reliability [1] I have determined that BTRFS [2] is the Linux storage technology that has the best potential to increase data integrity without costing a lot of money. Basically a BTRFS internal RAID-1 should offer equal or greater data protection than RAID-6. As BTRFS is so important and so very different to any prior technology for Linux it s not something that can be easily deployed in the same way as other filesystems. It is possible to easily switch between filesystems such as Ext4 and XFS because they work in much the same way, you have a single block device which the filesystem uses to create a single mount-point. While BTRFS supports internal RAID so it may have multiple block devices and it may offer multiple mountable filesystems and snapshots. Much of the functionality of Linux Software RAID and LVM is covered by BTRFS. So the sensible way to deploy BTRFS is to give it all your storage and not make use of any other RAID or LVM. So I decided to do a test installation. I started with a Debian install CD that was made shortly before the release of Squeeze (it was first to hand) and installed with BTRFS for the root filesystem, I then upgraded to Debian/Unstable to get the latest kernel as BTRFS is developing rapidly. The system failed on the first boot after upgrading to Unstable because the /etc/fstab entry for the root filesystem had the FSCK pass number set to 1 which wasn t going to work as no FSCK program has been written. I changed that number to 0 and it then worked. The initial install was on a desktop system that had a single IDE drive and a CD-ROM drive. For /boot I used a degraded RAID-1 and then after completing the installation I removed the CD-ROM drive and installed a second hard drive, after that it was easy to add the other device to the RAID-1. Then I tried to add a new device to the BTRFS group with the command btrfs device add /dev/sdb2 /dev/sda2 and was informed that it can t do that to a mounted filesystem! That will decrease the possibilities for using BTRFS on systems with hot-swap drives, I hope that the developers regard it as a bug. Then I booted with an ext3 filesystem for root and tried the btrfs device add /dev/sdb2 /dev/sda2 again but got the error message btrfs: sending ioctl 5000940a to a partition! which is not even found by Google. The next thing that I wanted to do was to put a swap file on BTRFS, the benefits for having redundancy and checksums on swap space seem obvious and other BTRFS features such as compression might give a benefit too. So I created a file by using dd to take take from /dev/zero, ran mkswap on it and then tried to run swapon. But I was told that the file has holes and can t be used. Automatically making zero blocks into holes is a useful feature in many situations, but not in this case. So far my experience with BTRFS is that all the basic things work (IE storing files, directories, etc). But the advanced functions I wanted from BTRFS (mirroring and making a reliable swap space) failed. This is a bit disappointing, but BTRFS isn t described as being ready for production yet. Related posts:
  1. Discovering OS Bugs and Using Snapshots I m running Debian/Unstable on an EeePC 701, I ve got an...
  2. Reliability of RAID ZDNet has an insightful article by Robin Harris predicting the...
  3. How I Partition Disks Having had a number of hard drives fail over the...

9 February 2012

Vincent Bernat: Recipes for extending Net-SNMP

SNMP stands for Simple Network Management Protocol. It allows a manager to query information from an agent. A popular use of SNMP is to retrieve network interface counters to plot a bandwidth graph. While the S in SNMP stands for simple, SNMP can be quite difficult to deal with. However, it is still the de facto standard for retrieving metrics in an heterogeneous environment. Net-SNMP is a suite of SNMP applications including an agent. Out of the box, this agent exports a lot of information but if something is missing, there are several ways to extend it. Which ones? tl;dr: in my opinion, extending Net-SNMP should be done with either of those methods:

SNMP in a nutshell The variables accessible with SNMP through the agent are organized in a tree. Each variable is typed and associated to an object identifier (or OID). For example, .1.3.6.1.2.1.31.1.1.1.1.2 is the OID of the name of the second network card. Usually, SNMP is operated over UDP, port 161. For the purpose of this article, we consider a manager can issue three kinds of requests:
GET
retrieve a variable.
GETNEXT
retrieve the next variable, in lexical order of their OID. This operation enables an SNMP manager to walk through all available variables, for example to discover the list of interfaces of an agent. This operation is what makes an SNMP agent difficult to implement but this is also a critical feature.
SET
request the modification of a variable.
Since such an OID may be difficult to handle by humans, available variables are described by a MIB module which is a file that should be parsed to get a mapping between OID and variable names and types. For example, the above OID can also be referred as IF-MIB::ifName.2. Those MIB modules are described using a subset of ASN.1 defined in RFC 2578. A MIB module allows to group several variables into a conceptual table. For example, IF-MIB::ifDescr, IF-MIB::ifType, IF-MIB::ifSpeed are columns of IF-MIB::ifTable. Each row is associated to an index. In the case of IF-MIB::ifTable, it is the interface index. Net-SNMP includes snmptable allowing one to display such a table in a natural manner1: 
$ snmptable localhost IF-MIB::ifTable
SNMP table: IF-MIB::ifTable
 ifIndex ifDescr           ifType ifMtu   ifSpeed
       1      lo softwareLoopback 16436  10000000
       2    eth0   ethernetCsmacd  1500 100000000
       3    eth1   ethernetCsmacd  1500  10000000
       4     br0   ethernetCsmacd  1500         0
Variables inside a table are called columnar objects while those outside are scalar objects. SNMP, as a protocol, does not care of this distinction since it does not know the concept of MIB modules. Other useful tools from Net-SNMP includes snmpget to get a specific variable, snmpwalk to discover available variables (with GETNEXT operation) from an OID and snmptranslate to translate between object names and OID. 
$ snmpget localhost IF-MIB::ifDescr.2
IF-MIB::ifDescr.2 = STRING: eth0
$ snmpwalk localhost IF-MIB::ifDescr
IF-MIB::ifDescr.1 = STRING: lo
IF-MIB::ifDescr.2 = STRING: eth0
IF-MIB::ifDescr.3 = STRING: eth1
IF-MIB::ifDescr.4 = STRING: br0
$ snmpwalk -On public localhost .1.3.6.1.2.1.2.2.1.2
.1.3.6.1.2.1.2.2.1.2.1 = STRING: lo
.1.3.6.1.2.1.2.2.1.2.2 = STRING: eth0
.1.3.6.1.2.1.2.2.1.2.3 = STRING: eth1
.1.3.6.1.2.1.2.2.1.2.4 = STRING: br0
$ snmptranslate -On IF-MIB::ifDescr.3
.1.3.6.1.2.1.2.2.1.2.3

Extending Net-SNMP ethtool -S allows us to access various statistics from a network card, like the number of octets received and transmitted or the number of collisions. While some of those statistics are defined in IF-MIB, in RMON-MIB and in EtherLike-MIB, some of them are not available and very specific. Therefore, we would like to export those statistics, unmodified, through SNMP. To the best of my knowledge, no MIB module exists for such an usage. The first task is to write one. It can be called ETHTOOL-MIB. It only contains one table indexed by the interface index and the name of the statistic. Here is the expected output: 
$ snmpwalk localhost ETHTOOL-MIB::ethtoolStat.2
ethtoolStat.2.'align_errors' = Counter64: 0
ethtoolStat.2.'broadcast' = Counter64: 25
ethtoolStat.2.'multicast' = Counter64: 345
ethtoolStat.2.'rx_errors' = Counter64: 0
ethtoolStat.2.'rx_missed' = Counter64: 0
ethtoolStat.2.'rx_packets' = Counter64: 262011
ethtoolStat.2.'tx_aborted' = Counter64: 0
ethtoolStat.2.'tx_errors' = Counter64: 0
ethtoolStat.2.'tx_multi_collisions' = Counter64: 0
ethtoolStat.2.'tx_packets' = Counter64: 296170
ethtoolStat.2.'tx_single_collisions' = Counter64: 0
ethtoolStat.2.'tx_underrun' = Counter64: 0
ethtoolStat.2.'unicast' = Counter64: 261641
I have coded various implementations of this MIB module and made them available in a GitHub repository.

With arbitrary commands The extend directive allows the agent to execute an arbitrary command and to provide its output and status through NET-SNMP-EXTEND-MIB module. If we want to export the statistics for eth0, we could add something like this in snmpd.conf, using a custom script to format the output of ethtool -S in a way that is more suitable for our use: 
# Export eth0 statistics
extend eth0-names  /full/path/to/ethtool-stats eth0 names
extend eth0-values /full/path/to/ethtool-stats eth0 values
The result can be retrieved through NET-SNMP-EXTEND-MIB:: nsExtendOutput2Table: 
$ snmpwalk localhost NET-SNMP-EXTEND-MIB::nsExtendOutput2Table
nsExtendOutLine."eth0-names".1 = STRING: tx_packets
nsExtendOutLine."eth0-names".2 = STRING: rx_packets
nsExtendOutLine."eth0-names".3 = STRING: tx_errors
nsExtendOutLine."eth0-names".4 = STRING: rx_errors
nsExtendOutLine."eth0-names".5 = STRING: rx_missed
nsExtendOutLine."eth0-names".6 = STRING: align_errors
nsExtendOutLine."eth0-names".7 = STRING: tx_single_collisions
nsExtendOutLine."eth0-names".8 = STRING: tx_multi_collisions
nsExtendOutLine."eth0-values".1 = STRING: 246309
nsExtendOutLine."eth0-values".2 = STRING: 223941
nsExtendOutLine."eth0-values".3 = STRING: 0
nsExtendOutLine."eth0-values".4 = STRING: 0
nsExtendOutLine."eth0-values".5 = STRING: 0
nsExtendOutLine."eth0-values".6 = STRING: 0
nsExtendOutLine."eth0-values".7 = STRING: 0
nsExtendOutLine."eth0-values".8 = STRING: 0
It is also possible to configure such a command using SET requests. For example, if we want to add statistics for eth2, we could issue the following command: 
$ snmpset -m +NET-SNMP-EXTEND-MIB localhost \
>    'nsExtendStatus."eth2-names"'  = createAndGo \
>    'nsExtendCommand."eth2-names"' = /full/path/to/ethtool-stats \
>    'nsExtendArgs."eth2-names"'    = 'eth2 names' \
>    'nsExtendStatus."eth2-values"'  = createAndGo \
>    'nsExtendCommand."eth2-values"' = /full/path/to/ethtool-stats \
>    'nsExtendArgs."eth2-values"'    = 'eth2 values'
nsExtendStatus."eth2-names" = INTEGER: createAndGo(4)
nsExtendCommand."eth2-names" = STRING: /full/path/to/ethtool-stats
nsExtendArgs."eth2-names" = STRING: eth2 names
nsExtendStatus."eth2-values" = INTEGER: createAndGo(4)
nsExtendCommand."eth2-values" = STRING: /full/path/to/ethtool-stats
nsExtendArgs."eth2-values" = STRING: eth2 values
$ snmpgetnext -m +NET-SNMP-EXTEND-MIB localhost \
>    'nsExtendOutLine."eth2-names"'
nsExtendOutLine."eth2-names".1 = STRING: tx_packets
This can be very convenient but also a huge security risk. You should disable this functionality or enable it only for some SNMPv3 user. Using extend does not allow to implement an arbitrary MIB module. If the cache is not disabled, extend is a good solution for various simple needs, like providing the content of a one-line file or the output of some Nagios plugin. However, exporting anything tabular, like interface statistics, is not a good fit.

With pass-through scripts Net-SNMP allows one to extend the agent by delegating some sub-tree to a script defined by the pass_persist directive. Such a script will receive requests on its standard input with a very simple line-oriented protocol described in the manual page of snmpd.conf. Here is an example of interaction: 
   PING
   PONG
   get
   .1.3.6.1.4.1.39178.100.1.1.1.2.2.109.117.108.116.105.99.97.115.116
   .1.3.6.1.4.1.39178.100.1.1.1.2.2.109.117.108.116.105.99.97.115.116
   counter64
   223941
   getnext
   .1.3.6.1.4.1.39178.100.1.1.1.2.2
   .1.3.6.1.4.1.39178.100.1.1.1.2.2.97.108.105.103.110.95.101.114
   counter64
   0
The protocol is simple enough to implement it from scratch in any scripting language. However, in Perl, you may prefer to use SNMP:: ExtensionSNMP:: PassPersist extension from S bastien Aperghis-Tramoni. You can find the complete example on Github. 
use SNMP::Extension::PassPersist;
my $extsnmp = SNMP::Extension::PassPersist->new(
    backend_collect => \&update_tree
    );
$extsnmp->run;
sub update_tree  
    my @interfaces = </sys/class/net/*>;
    foreach my $interface (@interfaces)  
        $interface =~ s/^.*\///;
        open(ETHTOOL, "ethtool -S $interface 2>/dev/null  ") or next;
        while (<ETHTOOL>)  
            /^\s+(\w+): (\d+)$/ or next;
            my $name  = $1;
            my $value = int($2);
            my $oid   = oid_compute($interface, $name);
            $extsnmp->add_oid_entry($oid, "counter", $value);
         
        close(ETHTOOL);
With Python, snmp-passpersist module, from Nicolas Agius, provides a similar interface. Look at the complete example on GitHub. pass_persist allows you to implement an arbitrary MIB module with honest performances. Here are some important things to know about this directive:
  • Unless you have a recent version of Net-SNMP, 64bit types are converted to 32bit equivalents. The counters may overflow quickly.
  • While handling a request, you are not allowed to pause (for example, to fetch a remote information): the agent would become unresponsive. If you need slow-to-get data or data from the agent, grab them in a separate thread and ensure you have them beforehand.

With AgentX protocol AgentX is a protocol defined in RFC 2741 allowing a master agent to be extended by independent sub-agents. For snmpd to become a master agent, you just need to add master agentx in snmpd.conf. Here are some of the benefits of an AgentX sub-agent over pass_persist:
  1. No configuration is needed for the master agent to accept an additional sub-agent. A sub-agent registers to the master agent the MIB modules (or part of them) it wants to take care of.
  2. A sub-agent is decoupled from the master agent. It can run with a different identity or be integrated into another daemon to export its internal metrics, send traps or allow remote configuration through SNMP.
  3. AgentX protocol can be carried over TCP. Sub-agents can therefore run on a foreign host or in a jailed environment.
  4. 64bit types are fully supported. Traps are also supported.

Perl and Python Net-SNMP provides a Perl module, NetSNMP::agent, to write a sub-agent using AgentX protocol. This module is not as convenient as the module for pass_persist described above since it does not provide a convenient layer to put required information into some datastore. Basically, it mimics the available C API. Here are a few examples of its use: There is no similar binding for Python but there is an agentx extension using ctypes. It provides a more high-level view. I did not write an example for this one but you can look at the example provided by the author.

C with new API To write a sub-agent in C, you have to choose between two API:
  1. the API inherited from UCD-SNMP also known as the traditional API;
  2. the new API developed for Net-SNMP 5.x.
With both API, an handler is registered to take care of one or several sub-trees. With the traditional API, the handler is invoked with very little information. It is then up to you to locate the appropriate variable by mapping the index part of the OID to the objects you want to export. It is quite simple for scalar objects but columnar objects with complex indexes are a pain with such an API. The new API provides various helpers. Some of them allow you to copy objects into an array or a list and let Net-SNMP library do the hard work of serving them. Another helper will provide methods to serve objects without putting them in a special structure but by providing an iterator. Some of those helpers can also help you to cache some variables. On top of this new API, Net-SNMP comes with mib2c, a tool that will help you convert a MIB module into C code: after a few questions, you will get a skeleton C code you will have to complete with the logic to retrieve real objects. A generated file also provides directions on what to do (with some magic command to have a step-by-step cookbook). I was able to implement ETHTOOL-MIB by just running it through mib2c, choosing the table container helper with cache and adding some code in ethtoolStatTable_data_access.c. You can look at the result on Github. Here is a stripped-down version2 of what has been modified: 
/* Socket for ioctl */
skfd = socket(AF_INET, SOCK_DGRAM, 0);
/* Iterate through all interfaces */
getifaddrs(&ifap);
for (ifa = ifap; ifa != NULL; ifa = ifa->ifa_next)  
    /* Grab statistics name and values */
    strcpy(ifr.ifr_name, ifa->ifa_name);
    drvinfo.cmd         = ETHTOOL_GDRVINFO;
    ifr.ifr_data        = (caddr_t) &drvinfo;
    ioctl(skfd, SIOCETHTOOL, &ifr);
    n_stats             = drvinfo.n_stats;
    strings->cmd        = ETHTOOL_GSTRINGS;
    strings->string_set = ETH_SS_STATS;
    strings->len        = n_stats;
    ifr.ifr_data        = (caddr_t) strings;
    ioctl(skfd, SIOCETHTOOL, &ifr);
    stats->cmd          = ETHTOOL_GSTATS;
    stats->n_stats      = n_stats;
    ifr.ifr_data        = (caddr_t) stats;
    ioctl(skfd, SIOCETHTOOL, &ifr);
    ifIndex = if_nametoindex(ifa->ifa_name);
    /* Iterate through statistics */
    for (i = 0; i < n_stats; i++)  
        /* Declare the appropriate index */
        strncpy(ethtoolStatName,
                (char *)&strings->data[i * ETH_GSTRING_LEN],
                ETH_GSTRING_LEN);
        ethtoolStatName[sizeof(ethtoolStatName) - 1] = '\0';
        ethtoolStatName_len = strlen(ethtoolStatName);
        /* Append this new variable to the container. */
        rowreq_ctx = ethtoolStatTable_allocate_rowreq_ctx();
        ethtoolStatTable_indexes_set(rowreq_ctx,
                    ifIndex,
                    ethtoolStatName, ethtoolStatName_len);
        rowreq_ctx->data.ethtoolStat.high = stats->data[i] >> 32;
        rowreq_ctx->data.ethtoolStat.low = (uint32_t)stats->data[i];
        CONTAINER_INSERT(container, rowreq_ctx);
See? It is pretty easy. I have only enumerated all the objects I wanted to expose. I didn t even have to convert statistics names to an OID, mib2c built a ethtoolStatTable_indexes_set() for this purpose. OpenHPI project provides an extensive documentation to write a sub-agent this way. Now, on the downside, to implement a single table, I have 14 generated files whose code style is usually unfit to integrate into an existing project : some people, me included, do not like automatically generated code, all the more when it is so verbose. The solution would be to use the new API without mib2c; unfortunately, while the documentation exists, all provided examples rely exclusively on mib2c tool. Unless you are able to grasp the new API without it, I would restrict its use to two cases:
  • Writing a new module for inclusion into Net-SNMP. Since most modules are now written with the help of mib2c, there is no coding style problem.
  • Writing a standalone sub-agent for some MIB. As a standalone project, you won t run into coding style problems either.
For other uses, let s have a look at the traditional API.

C with traditional API The traditional API is way simpler and even if it is quite old, it is easier to find associated examples. Here is how to initialize the agent, again, in a stripped-down version (the complete version is on GitHub): 
static oid ethtool_oid[] =  1, 3, 6, 1, 4, 1, 39178, 100, 1 ;
static struct variable3 ethtool_vars[] =  
   1, ASN_COUNTER64, RONLY, ethtool_stat, 3,  1, 1, 2 
 ;
int main()
 
  netsnmp_enable_subagent();
  snmp_disable_log();
  snmp_enable_stderrlog();
  init_agent("ethtoolAgent");
  REGISTER_MIB("ethtoolStatTable", ethtool_vars,
               variable3, ethtool_oid);
  init_snmp("ethtoolAgent");
  while (1)
    agent_check_and_process(1);
The REGISTER_MIB() macro is used to register the provided root OID (ethtool_oid) to the master agent. It also registers handlers associated to various sub-OID through the ethtool_vars[] array. Each member of this array features the following fields:
  • magic is an integer allowing one to discriminate OID handled by the same handler. For a table, this allows to handle several columns with the same handler since they share a common index.
  • type is the type of the variable that will be returned.
  • acl tells if the object is read-only (RONLY) or read-write (RWRITE).
  • findVar is the handler associated to the object. Its task will be to find the appropriate object and return its value.
  • The two last fields are the relative OID this entry is responsible for along with its length.
The prototype of ethtool_stat() handler function is the following: 
static u_char*
ethtool_stat(struct variable *vp, oid *name, size_t *length,
             int exact, size_t *var_len, WriteMethod **write_method);
This function should return the value of the requested variable or NULL if no appropriate object was found. The value can be statically stored since it will be copied. Here is the description of the arguments:
  • struct variable *vp is the structure the handle was associated with, except that vp->name is now a full OID, not a relative one. You can access vp->magic to determine which column or which scalar to handle if this handler is common to several objects.
  • oid *name and size_t *length describe the OID requested. If you want to return another OID (for a GETNEXT request for example), you can modify them. name points to an OID array large enough to fit any OID you want to return.
  • int exact tells if the requested OID should be exactly matched (case of GET or SET) or not (GETNEXT).
  • size_t *var_len is used to tell the size of the returned object.
  • WriteMethod **write_method is only used if the variable is modifiable. When a SET request is received, the API will call the handler with exact set to 1 and expects to get a pointer to the function that will handle the write operation3.
To implement this function, you are on your own. When relying on externally fetched data, you need to handle a cache yourself. I have used a red-black tree for this purpose (with code stolen from OpenBSD). This also enables an easy implementation of the GETNEXT operation. Here are three projects using this API to provide SNMP support:
  • Keepalived is a VRRP daemon and a monitoring daemon for LVS clusters. I have added a complete SNMP support. The most interesting files are core/snmp.c, check/check_snmp.c and vrrp/vrrp_snmp.c. It features a tight integration into Keepalived event loop that should be reusable by other projects. It includes read-only support of all data structures (configuration, state and statistics), ability to send traps on events and a write support for some values.
  • lldpd is an implementation of 802.1AB (LLDP) which is a protocol allowing an equipment to advertise itself to its neighbors on Ethernet level. 802.1AB comes with a huge MIB module and many extensions. lldpd provides a fairly complete read-only implementation of this MIB module. Some tables have complex indexes. Have a look at agent.c. There is currently no write support and no traps but lldpd features privilege separation and some code allows it to still use an Unix socket to speak with the master agent.
  • Asterisk, an open source PBX, includes an AgentX sub-agent allowing one to grab some metrics. Look at res/snmp/agent.c.

Other solutions The manual page for snmpd.conf explores other ways to extend Net-SNMP agent under the section Extending agent functionality :
  • exec and sh are deprecated. Use extend instead.
  • pass is a bit simpler than pass_persist. The command is run once for each request. Such a command can easily be turned into a pass_persist script. Moreover, Net-SNMP will not answer any request while a pass command is running.
  • proxy redirects requests to another SNMP agent. Unless you have some code that can only act as a complete SNMP agent, it is better to use AgentX instead. For programs using Net-SNMP API, turning an agent into an AgentX sub-agent is just one line of code. Since a sub-agent involves less code than a complete agent, it is more efficient and more reliable.
  • smux has been superseeded by AgentX.
  • perl and dlmod allow one to load and execute external code inside the running agent. Unless you have important performance requirements, it is better to use a sub-agent instead: if some bug happens, only the sub-agent will be affected, not the master agent. A sub-agent sticks more closely to the Unix philosophy: write programs that do one thing and do it well . However, Net-SNMP FAQ says:
Implementing the module in C within the main agent (directly or via dlmod) is probably the most efficient and reliable, closely followed by embedded perl (or python) extensions. These have the advantage of minimal overheads between the code implementing the MIB module, and the agent framework, and no inter-process communication issues.
  1. Additional arguments for snmp* are usually required to specify a version and a community (for SNMPv1 and SNMPv2c) or a user (for SNMPv3). In all examples, I do not specify anything. I assume that ~/.snmp/snmp.conf contains the appropriate bits. Moreover, some outputs are truncated.
  2. No allocations, no variable declarations and no error checkings. Do not do this at home!
  3. If you want to look at an example of how to handle a SET request, look at the
    SNMP support for Keepalived. A SET operation is more complex because SNMP ensures that the
    SET operation is atomic across all provided variables. It uses a three-staged commit. The traditional API is very inefficient for this kind of operation because the object has to be located again for each step.

8 February 2012

Russell Coker: More DRBD Performance tests

I ve previously written Some Notes on DRBD [1] and a post about DRBD Benchmarking [2]. Previously I had determined that replication protocol C gives the best performance for DRBD, that the batch-time parameters for Ext4 aren t worth touching for a single IDE disk, that barrier=0 gives a massive performance boost, and that DRBD gives a significant performance hit even when the secondary is not connected. Below are the results of some more tests of delivering mail from my Postal benchmark to my LMTP server which uses the Dovecot delivery agent to write it to disk, the rates are in messages per minute where each message is an average of 70K in size. The ext4 filesystem is used for all tests and the filesystem features list is has_journal ext_attr resize_inode dir_index filetype extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize .
p4-2.8
Default Ext4 1663
barrier=0 2875
DRBD no secondary al-extents=7 645
DRBD no secondary default 2409
DRBD no secondary al-extents=1024 2513
DRBD no secondary al-extents=3389 2650
DRBD connected 1575
DRBD connected al-extents=1024 1560
DRBD connected al-extents=1024 Gig-E 1544
The al-extents option determines the size of the dirty areas that need to be resynced when a failed node rejoins the cluster. The default is 127 extents of 4M each for a block size of 508MB to be synchronised. The maximum is 3389 for a synchronisation block size of just over 13G. Even with fast disks and gigabit Ethernet it s going to take a while to synchronise things if dirty zones are 13GB in size. In my tests using the maximum size of al-extents gives a 10% performance benefit in disconnected mode while a size of 1024 gives a 4% performance boost. Changing the al-extents size seems to make no significant difference for a connected DRBD device. All the tests on connected DRBD devices were done with 100baseT apart from the last one which was a separate Gigabit Ethernet cable connecting the two systems. Conclusions For the level of traffic that I m using it seems that Gigabit Ethernet provides no performance benefit, the fact that it gave a slightly lower result is not relevant as the difference is within the margin of error. Increasing the al-extents value helps with disconnected performance, a value of 1024 gives a 4% performance boost. I m not sure that a value of 3389 is a good idea though. The ext4 barriers are disabled by DRBD so a disconnected DRBD device gives performance that is closer to a barrier=0 mount than a regular ext4 mount. With the significant performance difference between connected and disconnected modes it seems possible that for some usage scenarios it could be useful to disable the DRBD secondary at times of peak load it depends on whether DRBD is used as a really current backup or a strict mirror. Future Tests I plan to do some tests of DRBD over Linux software RAID-1 and tests to compare RAID-1 with and without bitmap support. I also plan to do some tests with the BTRFS filesystem, I know it s not ready for production but it would still be nice to know what the performance is like. But I won t use the same systems, they don t have enough CPU power. In my previous tests I established that a 1.5GHz P4 isn t capable of driving the 20G IDE disk to it s maximum capacity and I m not sure that the 2.8GHz P4 is capable of running a RAID to it s capacity. So I will use a dual-core 64bit system with a pair of SATA disks for future tests. The difference in performance between 20G IDE disks and 160G SATA disks should be a lot less than the performance difference between a 2.8GHz P4 and a dual-core 64bit CPU. Related posts:
  1. DRBD Benchmarking I ve got some performance problems with a mail server that s...
  2. Some Notes on DRBD DRBD is a system for replicating a block device across...
  3. Ethernet bonding Bonding is one of the terms used to describe multiple...

Ian Wienand: Python and --prefix

Something interesting I discovered about Python and --prefix that I can't see a lot of documentation on... When you build Python you can use the standard --prefix flag to configure to home the installation as you require. You might expect that this would hard-code the location to look for the support libraries to the value you gave; however in reality it doesn't quite work like that. Python will only look in the directory specified by prefix after it first searches relative to the path of the executing binary. Specifically, it looks at argv[0] and works through a few steps is argv[0] a symlink? then dereference it. Does argv[0] have any slashes in it? if not, then search the $PATH for the binary. After this, it starts searching for dirname(argv[0])/lib/pythonX.Y/os.py, then dirname(argv[0])/../lib and so on, until it reaches the root. Only after these searches fail does the interpreter then fall back to the hard-coded path specified in the --prefix when configured. What is the practical implications? It means you can move around a python installation tree and have it all "just work", which is nice. In my situation, I noticed this because we have a completely self-encapsulated build toolchain, but we wish to ship the same interpreter on the thing that we're building (during the build, we run the interpreter to create .pyc files for distribution, and we need to be sure that when we did this we didn't accidentally pick up any of the build hosts python; only the toolchain python). The PYTHONHOME environment variable does override this behaviour; if it is set then the search stops there. Another interesting thing is that sys.prefix is therefore not the value passed in by --prefix during configure, but the value of the current dynamically determined prefix value. If you run an strace, you can see this in operation. 
readlink("/usr/bin/python", "python2.7", 4096) = 9
readlink("/usr/bin/python2.7", 0xbf8b014c, 4096) = -1 EINVAL (Invalid argument)
stat64("/usr/bin/Modules/Setup", 0xbf8af0a0) = -1 ENOENT (No such file or directory)
stat64("/usr/bin/lib/python2.7/os.py", 0xbf8af090) = -1 ENOENT (No such file or directory)
stat64("/usr/bin/lib/python2.7/os.pyc", 0xbf8af090) = -1 ENOENT (No such file or directory)
stat64("/usr/lib/python2.7/os.py",  st_mode=S_IFREG 0644, st_size=26300, ... ) = 0
stat64("/usr/bin/Modules/Setup", 0xbf8af0a0) = -1 ENOENT (No such file or directory)
stat64("/usr/bin/lib/python2.7/lib-dynload", 0xbf8af0a0) = -1 ENOENT (No such file or directory)
stat64("/usr/lib/python2.7/lib-dynload",  st_mode=S_IFDIR 0755, st_size=4096, ... ) = 0
Firstly it dereferences symlinks. Then it looks for Modules/Setup to see if it is running out of the build tree. Then it starts looking for os.py, walking its way upwards. One interesting thing that may either be a bug or a feature, I haven't decided, is that if you set the prefix to / then the interpreter will not go back to the root and then look in /lib. This is probably pretty obscure usage though! All this is implemented in Modules/getpath.c which has a nice big comment at the top explaining the rules in detail.

7 February 2012

Wouter Verhelst: Video at FOSDEM 2012

A year ago, during FOSDEM 2011, I walked up to the NamurLUG folks who'd been doing video coverage of FOSDEM for years, and suggested that this year, maybe they should consider using dvswitch for video coverage. While they seemed agreeable to this idea, they simply had not had the time to look at it in detail, and were therefore not using it. Since I've used dvswitch in the past, both as part of the DebConf video team and for my own concert recordings, I reasoned this a problem that could be solved by joining the video team. However, when it came to be time to start preparing for FOSDEM 2012, we found that many of the NamurLUG people were not going to have as much time to prepare for video coverage as they had during past years, and therefore responsibility of FOSDEM video coverage fell entirely to me. This was fairly unexpected, though not too daunting. For those of you who don't know dvswitch very well: a typical dvswitch installation for coverage of a talk requires: This is a rather large amount of work, and therefore it is not too unexpected that for DebConf, the DebConf video team takes a week to properly set up two rooms so that they can be used for recording. However, for FOSDEM we only have less than a day on-site to set up things, and we were going to cover five rooms. So, I tried to cut corners as much as possible: So in the end, I would need to take care of ten firewire-capable laptops, five cameras, five mixing consoles, several microphones, a number of transcoding machines, and shitloads of cabling (power, firewire, and audio). And about a week or two before the conference, as the massiveness of what we were about to do started to sink in, I was starting to have bad dreams of what would happen, wondering what I'd gotten myself into this time. Now, one day after FOSDEM, I have to say it all turned out okay, but there are some things where there's still room for improvement. So that I don't forget, I thought I'd make a list of things that went well, and one of things that didn't go well. And since that might be interesting for other people, I thought I'd do it here, rather than in a private file. First, things that did not go so well:
  1. Set-up for some rooms took more time than we had, and as such some rooms did not get properly streamed or recorded for their first few talks. This was due to the fact that there was some confused communication between some members of the team, which meant that we lost a day that we'd planned for preparation, and as such we couldn't test as much as we'd hoped. We need to improve on that next year. Ideally, we should run the full set-up somewhere, with all cameras and all laptops running, and making sure that everything is ready to be set up and that we have every cable we need, before dividing everything over a number of boxes (one per room) and bringing it to the conference.
  2. While some of our cameras were semi-professional Panasonic cameras that had a balanced XLR audio input, others were much lower-level camcorder-style cameras that did not have such a thing. When a camera has a good audio input, it's fairly easy to set up a link from the mixing console to the camera and get the audio into the stream that way. When a camera does not have such a thing, the set-up gets much more complex. Since we had so much to worry about, we did not notice that in one room, on saturday, something went wrong with audio. To combat this, we should improve on our audio set-up for next year, and also verify the streams every once in a while (I did the latter on sunday, so everything should be fine there).
  3. Inherent to an ad-hoc network, there were some network-related issues. For instance, on sunday evening, during the last session in that room, Ben informed me that the link to room H.1301 had gone down. As we were investigating, we easily found the source of the problem:
    IMAG0103
    The cable had been put under a door, and had been 'protected' with loads and loads of duct tape, but apparently that wasn't enough. This wasn't something we wanted to fix anymore, at that point; instead, we just let the cable be and focused on other things.
  4. Storage. None of the systems used for the recordings were my own; instead, some were owned by IRILL and some were rented. If I want to do anything with my recordings, that means I have to copy the data off. The system I chose was to bring a USB3 hard disk and copy everything over; but for saturday, this took an hour and thirty minutes. On sunday, my USB3 disk was fairly full, so I had to revert to a USB2 one, which increased the time by much. In the end, we had to abort copying files, and we'll now have to be a bit creative to get them.
  5. Volunteer management. I originally set up a wiki page to allow people to sign up for talks that they wanted to help out with. Unfortunately, that didn't work so well as it did for DebConf; there were several talks that did not have enough people, resulting in the volunteers from the previous talk remaining on post and finishing their talk. Also, there were some people that signed up without contacting me, or without telling me what name they used on the wiki page, which made it somewhat harder to know who to talk to. I lost count of who had signed up to help out. On Sunday, I decided to set up an IRC channel instead, through which volunteers could communicate more directly and just ask for people to take over, instead of registering for a talk first and then hoping nobody would forget. Also, since I'm terribly bad with faces and video volunteers did not have anything that made them stand out from the rest, I couldn't just walk around until I would see someone involved with video work and ask them to take over in a room. Having a separate 'video team' t-shirt could help there.
After the bad news, here's the list of things that did go well:
  1. Streaming. Even though choosing flumotion rather than icecast (as we'd done for streaming in the past) involved extra work for me before the conference (I had to write some code so that I could stream from dvswitch into flumotion, which wasn't possible out of the box), the guys at Flumotion decided to send over one of their support engineers. As such, I simply did not have to worry about streaming servers crashing or failing to do what they were supposed to do. While I have no reason to think any of the Flumotion servers were having issues, even if they were I wouldn't have known about it, since Francesc took care of it all and never ever required my involvement about something on the flumotion side. This was extremely valuable.
  2. Training right before the welcome talk. Explaining to people in a few words how dvswitch works, and then immediately following that up with a live demonstration, isn't a bad way for them to understand.
  3. Despite all the problems we did have, once everything had been set up and all the gear was more or less manned, actually recording and streaming did go fairly well. The feedback I received so far was mostly positive, which makes me very happy.
In the end, while there certainly is still room for improvement and things did not go as smoothly as I'd hoped, they have gone much better than my worst fears. There's still some work to do, which I'll be doing with the NamurLUG people this week, but all in all, it's gone pretty good. As to the usage of the streams, I asked Flumotion for some statistics. Since I was fairly late in asking, they could only provide me with statistics about sunday:

Rapha&#235;l Hertzog: Dpkg with multiarch support available in Debian experimental

As I announced on debian-devel, Guillem Jover uploaded a snapshot of dpkg s multiarch branch to experimental (version 1.16.2~wipmultiarch). Beware: There will
likely be some small interface changes between this version and the version that will be released later in unstable (possibly in the output of dpkg --get-selections, dpkg --list, maybe other commands). multiarch allows you to install packages from different architectures on the same machine. This can be useful if your computer can run programs from 2 architectures (eg. x86 CPU supporting i386 and amd64), or if you often need to cross-compile software and thus need the libraries of your target architecture. Test dpkg with multiarch support If you want to test multiarch support in dpkg, install the package from experimental (apt-get install dpkg/experimental assuming you have experimental in your sources.list). Then you can add a supplementary architecture to your system by doing sudo dpkg --add-architecture <arch> (e.g. i386 if you are on amd64, and vice-versa). APT will automatically pick up the new architecture and start downloading the Packages file for the new architecture (it uses dpkg --print-foreign-architectures to know about them). From there on you can install packages from the foreign architectures with apt-get install foo:<arch> . Many packages will not be installable because some of their dependencies have not yet been updated to work with in a multiarch world (libraries must be installed in a multiarch-compliant path so as to be co-installable, and then marked Multi-Arch: same ). Other dependencies might need to be marked Multi-Arch: foreign . See wiki.debian.org/Multiarch/Implementation for more HOWTO-like explanations. Now is a good time to see if you can install the foreign packages that you could need in such a setup and to help to convert the required libraries. You can also read Cyril Brulebois article which quickly shows how to hunt for the problematic packages which have not been converted to multiarch (in his sample, ucf is not ready. Since it s an Architecture: all package which can run on any architecture, it means that it s lacking a Multi-Arch: foreign field). Report bugs If you discover any bug in dpkg s multiarch implementation, please report it to the Bug Tracking System (against dpkg with the version 1.16.2~wipmultiarch ). If you notice important libraries or packages which are not yet multiarch ready, please open wishlist bug reports requesting the conversion and point the maintainers towards the wiki page linked above. Even better, prepare patches and submit those with your bug reports. Again, you can follow the lead of Cyril Brulebois who filed 6 bugs! Review the multiarch implementation If you re a C programmer and have some good knowledge of dpkg (or are willing to learn more of it), we would certainly benefit from more eyes reviewing the multiarch branch. If you want to discuss some design issues of the multiarch implementation in dpkg (or have questions related to your review), please get in touch via debian-dpkg@lists.debian.org. The latest version of the branch is pu/multiarch/master in Guillem s personal repository. I have my own version of the branch (pu/multiarch/full) which is usually a snapshot of Guillem s branch with my own submitted fixes. 
$ git clone git://git.debian.org/dpkg/dpkg.git
$ cd dpkg
$ git remote add guillem git://git.hadrons.org/git/debian/dpkg/dpkg.git
$ git remote add buxy git://git.debian.org/~hertzog/dpkg.git
$ git fetch guillem && git fetch buxy
If you followed the instructions above, the relevant branches are thus guillem/pu/multiarch/master and buxy/pu/multiarch/full. Both branches are regularly rebased on top of master where Guillem merges progressively the commits from the multi-arch branch as his review progresses. Thank you in advance for your help bringing multiarch in shape for Debian Wheezy,

6 comments Liked this article? Click here. My blog is Flattr-enabled.

Russell Coker: 5 Principles of Backup Software

Everyone agrees that backups are generally a good thing. But it seems that there is a lot less agreement about how backups should work. Here is a list of 5 principles of backup software that seem to get ignored most of the time: (1/5) Backups should not be Application Specific It s quite reasonable for people to want to extract data from a backup on a different platform. Maybe someone will want to extract data a few decades after the platform becomes obsolete. I believe that vendors of backup software have an ethical obligation to make it possible for customers to get their data out with minimal effort regardless of the circumstances. Often when writing a backup application there will be good reasons for not using the existing formats for data storage (tar, cpio, zip, etc). But ideally any data store which involves something conceptually similar to a collection of files in one larger file will use one of those formats. There have been backward compatible extensions to tar and zip for SE Linux contexts and for OS/2 EAs the possibility of extending archive file formats with no consequence other than warnings on extraction with an unpatched utility has been demonstrated. For a backup which doesn t involve source files (EG the contents of some sort of database) then it should be in a format that can be easily understood and parsed. Well designed XML is generally a reasonable option. Generally the format should involve plain text that is readable and easy to understand which is optionally compressed with a common compression utility (pkzip is a reasonable choice). (2/5) Data Store Formats should be Published For every data store there should be public documentation about it s format to allow future developers to write support for it. It really isn t difficult to release some commented header files so that people can easily determine the data structures. This includes all data stores including databases and filesystems. If I suddenly find myself with a 15yo image of a NTFS filesystem containing a proprietary database I should be able to find official header files for the version of NTFS and the database server in question so I can decode the data if it s important enough. When an application vendor hides the data formats it gives the risk of substantial data loss at some future time. Imposing such risk on customers to try and prevent them from migrating to a rival product is unethical. (3/5) Backups should be forward and backward compatible It is entirely unreasonable for a vendor to demand that all their users install the latest versions of their software. There are lots of good reasons for not upgrading which includes hardware not supporting new versions of the OS, lack of Internet access to perform the upgrade, application compatibility, and just liking the way the old version works. Even for the case of a critical security fix it should be possible to restore data without applying the fix. For any pair of versions of software that are only separated by a few versions it should be possible to backup data from one and restore to the other. Even if the data can t be used directly (EG a backup of AMD64 programs that is restored on an i386 system) it should still be accessible. If a new version of the software doesn t support the ancient file formats then it should be possible for the users to get a slightly older version which talks to both the old and new versions. Backups made on 64bit systems running the latest development version of Linux and on 10yo 32bit proprietary Unix systems are interchangeable. Admittedly Unix is really good at preserving file format compatibility, but there is no technical reason why other systems can t do the same. Source code to cpio, tar, and gnuzip, is freely available! Apple TimeMachine fails badly in this regard, even a slightly older version of Mac OS can t do a restore. It is however nice that most of the TimeMachine data is a tree of files which could be just copied to another system. (4/5) Backup Software should not be Dropped Sony Ericsson has made me hate them even more by putting the following message on their update web site: The Backup and Restore app will be overwritten and cannot be used to restore data. Check out Android Market for alternative apps to back up and restore your data, such as MyBackup. So if you own a Sony Ericsson phone and it is lost, stolen, or completely destroyed and all you have is a backup made by the Sony Ericsson tool then the one thing you absolutely can t do is to buy a new Sony Ericsson phone to restore the data. I believe that anyone who releases backup software has an ethical obligation to support restoring to all equivalent systems. How difficult would it be to put a new free app in the Google Market that has as it s sole purpose recovering old Sony Ericsson backups onto newer phones? It really can t be that difficult, so even if they don t want to waste critical ROM space by putting the feature in all new phones they can make it available to everyone who needs it. When compared to the cost of developing a new Android release for a series of phones the cost of writing such a restore program would be almost nothing. It is simply mind-boggling that Sony Ericsson go against their own commercial interests in this regard. Surely it would make good business sense to be able to sell replacements for all the lost and broken Sony Ericsson phones, but instead customers who get burned by broken backups are given an incentive to buy a product from any other vendor. (5/5) The greater the control over data the greater the obligation for protecting it If you have data stored in a simple and standard manner (EG the /DCIM directory containing MP4 and JPEG files that is on the USB accessible storage in every modern phone) then IMHO it s quite OK to leave customers to their own devices in terms of backups. Typical users can work out that if they don t backup their pictures then they risk losing them, and they can work out how to do it. My Sony Ericsson phones have data stored under /data (settings for Android applications) which is apparently only accessible as root. Sony Ericsson have denied me root access which prevents me running backup programs such as Titanium Backup, therefore I believe that they have a great obligation to provide a way of making a backup of this data and restoring it on a new phone or a phone that has been updated. To just provide phone upgrade instructions which tell me that my phone will be entirely wiped and that I should search the App Market for backup programs is unacceptable. I believe that there are two ethical options available to Sony Ericsson at this time, one is to make it easy to root phones so that Titanium Backup and similar programs can be used, and the other option is to release a suitable backup program for older phones. Based on experience I don t expect Sony Ericsson to choose either option. Now it is also a bad thing for the Android application developers to make it difficult or impossible to backup their data. For example the Wiki for one Android game gives instructions for moving the saved game files to a new phone which starts with root your phone . The developers of that game should have read the Wiki, realised that rooting a phone for the mundane task of transferring saved game files is totally unreasonable, and developed a better alternative. The best thing for developers to do is to allow the users to access their own data in the most convenient manner. Then it becomes the user s responsibility to manage it and they can concentrate on improving their application. Why Freedom is Important Installing CyanogenMod on my Galaxy S was painful, but having root access so I can do anything I want is a great benefit. If phone vendors would do the right thing then I could recommend that other people use the vendor release, but it seems that vendors can be expected to act unethically. So I can t recommend that anyone use an un-modded Android phone at any time. I also can t recommend ever buying a Sony Ericsson product, not even when it s really cheap. Google have done a great thing with their Data Liberation Front [1]. Not only are they providing access to the data they store on our behalf (which is a good thing) but they have a mission statement that demands the same behavior from other companies they make it an issue of competitive advantage! So while Sony Ericsson and other companies might not see a benefit in making people like me stop hating them, failing to be as effective in marketing as Google is a real issue. Data Liberation is something that should be discussed at board elections of IT companies. Keep in mind the fact that ethics are not just about doing nice things, they are about establishing expectations of conduct that will be used by people who deal with you in future. Sony Ericsson has shown that I should expect that they will treat the integrity of my data with contempt and I will keep this in mind every time I decline an opportunity to purchase their products. Google has shown that they consider the protection of my data as an important issue and therefore I can be confident when using and recommending their services that I won t get stuck with data that is locked away. While Google has demonstrated that corporations can do the right thing, the vast majority of evidence suggests that we should never trust a corporation with anything that we might want to retrieve when it s not immediately profitable for the corporation. Therefore avoiding commercial services for storing important data is the sensible thing to do. Related posts:
  1. Galaxy S vs Xperia X10 and Android Network Access Galaxy S Review I ve just been given an indefinite loan...
  2. Standardising Android Don Marti wrote an amusing post about the lack of...
  3. On Burning Platforms Nokia is in the news for it s CEO announcing that...

5 February 2012

John Goerzen: Rain, A Funeral, and Excitement

Rain Friday was something of a rare day for February in Kansas. Starting at about 2AM, the wind picked up, blowing so hard that our windows rattled. That part isn t so rare. Then the cold rain started, dropping almost 2.5 throughout the day. As I worked, I had the blinds on the windows open, but they didn t let in very much light. Still, the wind had calmed down, so the intermittent rain outside was peaceful. Jacob went out to play for a little while, so every so often I saw a warmly-dressed and excited-looking 5-year-old run past my window. A little while after he came in, I told Jacob, I saw you playing outside. His response: Oh good! I got wet! Which, despite the fact that it was about 50 degrees, seemed to excite him. After the blustery start, the calm, slow, and peaceful rain was a pleasant thing to see throughout the day. Funeral My great aunt Alice Goerzen passed away last Sunday. So today, for the third time in a little over a year, I was at the funeral of a Goerzen relative and neighbor. Alice s husband, Milt, passed away in late 2010, and it was while I was at his funeral that Jacob got run over by a tractor. That memory certainly came back to me today. But I think I should set the stage and explain what funerals are like in this small, rural Kansas community. At the church, while people file in, family and close friends generally defined as loosely as desired meet in some other room before the funeral. Memories may be shared, or songs sung, or maybe just a brief meditation or prayer. Then the man from the funeral home there s only one in town will step in. Ivan Miller owned the business for decades, and although he s now retired, his replacement seems pretty similar. Kindly, respectful, and pretty much unchanging. This group then files into the church sanctuary to sit up front, while the rest of the congregation is standing and music is played. We typically sing some hymns, hear memories from the family, a message from a pastor, and then do downstairs for faspa: an light meal with coffee, zwieback, funeral cheese , and some relishes and dessert. You can, by the way, go to the local grocery store and find a product labeled funeral cheese . It s a sharp cheddar, sliced thick and cut into pie piece-shaped wedges. After everyone has picked up their food, microphones are passed around, and anybody that wants to can share memories and stories. These are often hilarious, or touching, and can be more random than anyone could expect. Today we heard a lot about how Aunt Alice loved her flowers and garden. We even saw a video of her giving a tour of her garden, with Milt s mower in the background occasionally accidentally causing a distraction (or maybe it wasn t so accidental; he d never miss an opportunity to cause some mischief ) I tend to think of attending funerals around here as a good time. Sadness is inevitable, but there are so many amazing stories that it is hard to leave feeling sad. Excitement This afternoon, Jacob found me in the office and as he often does, said, Dad, I want to do something with you. Usually I ask him what he d like to do, but his first instinct is usually to ask for watching train videos on Youtube. So sometimes I make other suggestions. Today we played hide and seek with radios, in which the person that is counting is supposed to radio to the other person when they are done. Today was the first time that Jacob came up with the trick of talking into the radio while I was hiding so he could hear where I was. I was sort of proud of him, and he failed to completely hide his smile when I told him I had to turn off my radio or else he d find me too fast. Then later, we played with Jacob s computer, a Linux-based command-line-only machine. I have set up a few shell scripts and aliases for him. Since it doesn t play videos, he doesn t use it as much as he does mine, but it is really fun to watch how his interaction with it changes as he gets older. He can now read amazingly well for a 5-year-old, and is starting to learn how to spell. He loves word games, writing, and typing. I thought I would install an ASCII art program for him. I told Jacob I had some ideas for a new game, and he was irresistibly intrigued. I offered him a choice between figlet and toilet. And, as is probably no surprise to anyone with a 5-year-old, he chose toilet based on its name, Jacob and Oliver both loved typing things and seeing them displayed bigger. I showed Jacob how I could make a freight train by typing ,<@-[]-(*)-@> (that s the comma-shaped snowplow, engine, boxcar [], tank car (*), and caboose @>). Then toilet drew them big, and though abstract, caused great excitement. I hooked up one of the speech synthesizers in Debian to a simple shell script named talk , which is a huge hit with the boys. They enjoy typing in nonsense and hearing the funny result, or in typing in real words and hearing how the computer says them right (or doesn t). All told, we had a good hour s worth of excitement up there.

4 February 2012

Stefano Zacchiroli: bits from the DPL for January 2012

Fresh from the oven, monthly report of what I've been working on as DPL during January 2012.
Dear Developers,
here is another monthly report of what happened in DPL-land, this time for January 2012. There's quite a bit to report about --- including an insane amount of legal-ish stuff --- so please bear with me. Or not. Legal stuff Most of the above wouldn't have been possible without the precious help of folks at SFLC working for SPI and Debian. Be sure to thank SFLC for what they're doing for us and many other Free Software projects. Coordination Nobody stepped up to coordinate the artwork collection for Wheezy I've mentioned last month, so I've tried to do a little bit of that myself. The -publicity team is now preparing the call for artwork and hopefully we'll send it out RSN. In case you want to help, there is still a lot of room for that; just show up on the debian-desktop mailing list. Sprints A Debian Med sprint has happened in January, and Andreas Tille has provided a nice and detailed report about it. Some more sprints are forthcoming this spring, how about yours? Money Important stuff going on Other important stuff has been going on in various area of the project in January. I'd like to point your attention to a couple of things: Miscellanea In the unlikely case you've read thus far, thanks for your attention! Happy Debian hacking.
PS as usual, the boring day-to-day activity log is available at master:/srv/leader/news/bits-from-the-DPL.*

Christian Perrier: Debian Installer fully translated into Kannada

I imagine my readers (except those from India) : "Kannada, WTF"? No, this is not about D-I being translated into the variants of English and French spoken with a funny accent in a very big country located north of the United States of America, where they play ice hockey against mooses, wearing red policemen suits, with fur hats and drinking maple syrup. Kannada is the official language of the state of Karnataka, in southern India. The state that has Bangalore, the IT-leading city in India as capital. Kannada is spoken by about 45 million people, roughly the population of Spain. And, since yesterday, thanks to a local group lead by Vikram Vincent, Debian Installer is fully translated to Kannada. As of now, this is the 5th complete language of India along with Gujarati (46M speakers as first language), Hindi (180M, though some references mention 550M), Marathi (68M) and Tamil (62M). Other supported languages of India are Telugu (99% translated, 70M), Bengali (95%, 71M plus 110M in Bangladesh), Punjabi (91%, 28M) and Malayalam (86%, 36M). As you see, translators in India are really incredibly active and the free software community over there deserves some big light. We don't have many opportunities to meet up as traveling is not easy for contributors in India because, among other problems, of visa regulations for Indian citizens in many so-called western countries. So, really, I wanted to thank them again for this work.

3 February 2012

Russell Coker: A Computer Conference on a Cruise Ship

After LCA [1] there was a discussion about possible locations for future conferences, most of the messages in the discussion were jokes or suggestions that don t seriously apply to LCA. So I ll add my suggestion for conferences other than LCA. I ve previously written generally about the issue of conferences at sea [2]. I don t think that LCA would be suitable for running at sea because delegates have specific expectations for LCA which are quite different to what a cruise ship can offer, so I don t think it makes sense to change LCA which is working well as it is. However there are lots of other possible computer conferences which could suite a cruise ship. Price Price is a major factor in running a conference, so obviously getting a cheap cruise price is very important. Here is a link for Vacations To Go which shows cruises from the Australia/NZ region which are of at least 5 nights and cost no more than $800 [3]. The cheapest entry at this moment is $609 for 5 nights and the cheapest on a per-night basis is an 8 night cruise for $779. The cheapest cruise currently on offer which allows a conference similar to LCA is 7 nights for $699. The prices should be regarded as rough approximations as some cruises have some mandatory extra fees and the prices are quoted in US dollars and subject to currency fluctuations. Note that those prices are for dual-occupancy cabins, this can be a double or a twin configuration. Some cruise ships have cabins for 3 or 4 people that are cheaper, but if you have a cabin for a single person then the rate is almost the same as for having two people. The price for LCA accommodation including breakfast was $78 per night for a single room or $92 for a double room. Then lunch cost a minimum of $10 and for dinner there was $80 for the penguin dinner and probably about $20 for dinner every other night. That gave an overall cost for a 6 night stay (which is probably the minimum for someone who lives further away than Melbourne) in Ballarat of 6*78+6*10+5*20+80==$708. For a double room that would be 6*92+6*10+5*20+2*80==$872. Even if we don t count the fact that the Australian dollar is worth more than the US dollar it is obvious that on the basis of accommodation and food two people sharing a twin cabin on a cruise ship could pay LESS than two people in single rooms at the Ballarat University dorms! Now sharing a cabin isn t so great, but the upside is that cruise ships have excellent food and lots of other entertainment options. I previously reviewed the food on the Dawn Princess and determined that it s better than the food I would expect to get if I spent the cost of the cruise on dinner at land based restaurants [4]. I have been led to believe that the use of ship conference facilities is typically free for any organisation that books a sufficient number of cabins. So there s no reason why the conference admission fees should be any greater than for a land based conference. Advantages A common problem with conferences is finding suitable dining options. Most people want to eat with other delegates but finding restaurants that have sufficient space and which are conveniently located is difficult at best and often impossible. On a cruise ship everything is within a short walk and the restaurants are big, usually be at least one restaurant will hold 500 people. The fact that you have to reserve times for the Main Dining Room makes it more difficult to miss one s colleagues. Everything on a cruise ship is luxurious. There are lots of good locations for BoFs, pools, cafes, restaurants, and bars. Basically the ship is filled with comfortable places for groups of people to sit down. A cruise ship typically has a main theater with more than 700 seats more than large enough for most conferences I ve attended. It s common for the size of a conference to be limited to the size of the main theater that is used, for a cruise ship this will probably be less of a problem than for most other conference venues. Disadvantages The first disadvantage of running a computer conference on a cruise ship is the almost total lack of net access. The costs for net access are more expensive than most delegates will pay. Probably many delegates would check their email but it wouldn t be practical for people to download source code, browse Wikipedia, and use the Internet in other ways related to the conference. It would be practical to have mirrors of Wikipedia, the source of several distributions of Linux, and other big things of common interest. Another possible problem is the fact that you need to book it well in advance to avoid the risk of selling out (there is no option to stay at a different hotel). An established conference with financial backing could just pay to reserve the cabins. But when starting a new conference this could be a problem. Alcohol is rather expensive on cruise ships. But getting really drunk isn t compatible with learning about computer science anyway. Finally the requirement to have at least two people in a cabin for good rates is a serious issue. The upside of this is that people travelling with their SO would find that it works really well (regardless of whether the SO is a delegate or not). But anyone who s not travelling with their SO and doesn t want to share with a friend will have to either pay a lot more or skip the conference. Conclusion I think that there is a good potential for running a computer conference around the Australia/NZ region on a cruise ship. It won t be overly expensive for delegates and the facilities that are provided are good. The trade-off for solitary travelers of having to share a cabin (or pay more) for getting much better food and leisure facilities will be appreciated by many people (and admittedly hated by some). Some people won t appreciate the option of swimming, but even if you consider the cruise ship to be just a floating collection of restaurants and cabins it s still fairly luxurious and beats the heck out of most conferences I ve attended. If you are considering the possibility of running a conference then I think that a cruise ship should be considered. VacationsToGo.com is the best site I ve found for cheap cruise prices, their large group department has experience handling groups of more than 500 people so I think that anyone who wants to run a new conference in/around Australia should give them a call. Also cruise ships travel around the world, so the same thing can be done in other countries but at a different time of year. The economic factors will differ by country though. Cruise ships probably aren t a cheap option for a conference in some other countries. Related posts:
  1. My First Cruise A few weeks ago I went on my first cruise,...
  2. Cruises It seems that in theory cruises can make for quite...
  3. Creating a Micro Conference The TEDxVolcano The TED conference franchise has been extended to...

2 February 2012

Joey Hess: unicode ate my homework

I've just spent several days trying to adapt git-annex to changes in ghc 7.4's handling of unicode in filenames. And by spent, I mean, time withdrawn from the bank, and frittered away. In kindergarten, the top of the classrom wall was encircled by the aA bB cC of the alphabet. I'll bet they still put that up on the walls. And all the kids who grow up to become involved with computers learn that was a lie. The alphabet doesn't stop at zZ. It wouldn't all fit on a wall anymore. So we're in a transition period, where we've all learnt deeply the alphabet, but the reality is much more complicated. And the collision between that intuitive sense of the world and the real world makes things more complicated still. And so, until we get much farther along in this transition period, you have to be very lucky indeed to not have wasted time dealing with that complexity, or at least having encountered Mojibake. Most of the pain centers around programming languages, and libraries, which are all at different stages of the transition from ascii and other legacy encodings to unicode. My most recent pain is because the haskell GHC compiler is moving along in the transition, getting closer to the end. Or at least finishing the second 80% and moving into the third 80%. (This is not a quick transition..) The change involves filename encodings, a situation that, at least on unix systems, is a vast mess of its own. Any filename, anywhere, can be in any encoding, and there's no way to know what's the right one, if you dislike guessing. Haskell folk like strongly typed stuff, so this ambiguity about what type of data is contained in a FilePath type was surely anathama. So GHC is changing to always use UTF-8 for operations on FilePath. (Or whatever the system encoding is set to, but let's just assume it's UTF-8.) Which is great and all, unless you need to write a Haskell program that can deal with arbitrary files. Let's say you want to delete a file. Just a simple rm. Now there are two problems:
  1. The input filename is assumed to be in the system encoding aka unicode. What if it cannot be validly interpreted in that encoding? Probably your rm throws an exception.
  2. Once the FilePath is loaded, it's been decoded to unicode characters. In order to call unlink, these have to be re-encoded to get a filename. Will that be the same bytes as the input filename and the filename on disk? Possibly not, and then the rm will delete the wrong thing, or fail.
But haskell people are smart, so they thought of this problem, and provided a separate type that can deal with it. RawFilePath hearks back to kindergarten; the filename is simply a series of bytes with no encoding. Which means it cannot be converted to a FilePath without encountering the above problems. But does let you write a safe rm in ghc 7.4. So I set out to make something more complicated than a rm, that still needs to deal with arbitrary filename encodings. And I soon saw it would be problimatic. Because the things ghc can do with RawFilePaths are limited. It can't even split the directory from the filename. We often do need to manipulate filenames in such ways, even if we don't know their encoding, when we're doing something more complicated than rm. If you use a library that does anything useful with FilePath, it's not available for RawFilePath. If you used standard haskell stuff like readFile and writeFile, it's not available for RawFilePath either. Enjoy your low-level POSIX interface! So, I went lowlevel, and wrote my own RawFilePath versions of pretty much all of System.FilePath, and System.Directory, and parts of MissingH and other libraries. (And noticed that I can understand all this Haskell code.. yay!) And I got it close enough to working that, I'm sure, if I wanted to chase type errors for a week, I could get git-annex, with ghc 7.4, to fully work on any encoding of filenames. But, now I'm left wondering what to do, because all this work is regressive; it's swimming against the tide of the transition. GHC's change is certainly the right change to make for most programs, that are not like rm. And so most programs and libraries won't use RawFilePath. This risks leaving a program that does a fish out of water. At this point, I'm inclined to make git-annex support only unicode (or the system encoding). That's easy. And maybe have a branch that uses RawFilePath, in a hackish and type-unsafe way, with no guarantees of correctness, for those who really need it.
Previously: unicode eye chart wanted on a bumper sticker abc boxes unpacking boxes

Jamie McClelland: Servers4All... unless someone complains

On Wednesday, February 1, a new virtual server May First/People Link recently rented went offline. We contracted the virtual server through Server4All because we need their un-metered 100Mbit connection to help us handle the bandwidth for Sahara Reporters, one of the most important independent news sources for Saharan Africa. With the server offline, the web site was down as well. We scrambled to setup alternative caching servers to handle the bandwidth. When I logged into our control panel, I saw the message: This virtual server has been suspended by the administrator. Please contact support. I immediately contacted support and then received the message: 
Hello
We have received the following complaint associated with your server/service.
IP: 76.73.121.164
To prevent any further abuse we have suspended this service. In order to
resume, we request you to cooperate with our investigation as promptly as
possible. Please respond to us with the following details:
(1) What has caused the complaint
(2) What is the server used for. Purpose?
(3) How can you resolve the complaint and make sure it will not be repeated.
Depending on the nature of the complaint and your response, we will put back
the server online.  Please note, this has violated our Terms Of Service. We
expect your response within 24 hours, otherwise your account will be
terminated permanently.  Thank you
What?? What complaint?? I followed up but had to wait til the next day to get the response. 
Here is the full log,
An email advertizing the Domain Name: saharareporters.com
has been sent to the blacklist.woody.ch spamtrap.
This Domain does resolve to IP addresses one of which your are responsible:
76.73.121.164
Please investigate why this Domain has been advertized.
Attached you find the headers and reports in ARF for automatic processing.
Feedback is appreciated.
Actual listing periods:
Bounce: 1 Hour in DNS.
Whitelisted IP: Not lised in DNS.
Spam: 24 hours in DNS.
Every Hit: 14 days in evidence DB.
For any questions or Feedback, contact abuse@woody.ch
From: is intentionally set to a bit-bucket.
Kind regards
-Benoit Panizzon-
There is no attachment. I went to woody.ch and it was in German. Then tried blacklist.woody.ch, but no luck. Finally I found the Woody's World Blacklist Page. I plugged in our IP address into their checker and I got: 
Output from the Check, if empty the IP is not listed.
164.121.73.76.[name of the blacklist] being tested.
Host 164.121.73.76.blacklist.woody.ch not found: 3(NXDOMAIN)
Host 164.121.73.76.blacklist.woody.ch not found: 3(NXDOMAIN)
Host 164.121.73.76.rbl.maps.vix.com not found: 3(NXDOMAIN)
Host 164.121.73.76.rbl.maps.vix.com not found: 3(NXDOMAIN)
Host 164.121.73.76.relays.mail-abuse.org not found: 3(NXDOMAIN)
Host 164.121.73.76.relays.mail-abuse.org not found: 3(NXDOMAIN)
;; connection timed out; no servers could be reached
;; connection timed out; no servers could be reached
;; connection timed out; no servers could be reached
;; connection timed out; no servers could be reached
Host 164.121.73.76.relays.ordb.org not found: 3(NXDOMAIN)
Host 164.121.73.76.relays.ordb.org not found: 3(NXDOMAIN)
Host 164.121.73.76.dev.null.dk not found: 3(NXDOMAIN)
Host 164.121.73.76.dev.null.dk not found: 3(NXDOMAIN)
Host 164.121.73.76.blackholes.five-ten-sg.com not found: 3(NXDOMAIN)
Host 164.121.73.76.blackholes.five-ten-sg.com not found: 3(NXDOMAIN)
Host 164.121.73.76.bl.spamcop.net not found: 3(NXDOMAIN)
Host 164.121.73.76.bl.spamcop.net not found: 3(NXDOMAIN)
Host 164.121.73.76.relays.visi.com not found: 3(NXDOMAIN)
Host 164.121.73.76.relays.visi.com not found: 3(NXDOMAIN)
164.121.73.76.blacklist.spambag.org has address 208.91.197.182
164.121.73.76.blacklist.spambag.org descriptive text "v=spf1 -all"
So, Woody's World thinks we are listed in spambag.org. I went to spambag.org and learned that the domain is for sale. I then tried blacklist.spambag.org and got the same page. This page has many links all pointing to advertisements. The "RBL List" link takes me to a page advertising "5 foods you must not eat." Next, out of curiousity, I tried Mxtoolbox. I got one hit from Barricuda. Barricuda says the reputation of the IP address is "poor". Why? According to Barricuda, the reasons could be: Keep in mind, this IP address is not used for sending email. It's just the web site. I then took a step back and re-read the complaint and noticed that it says that the domain name saharareporters.com was listed in a spam email. Hm. More concerted searching for the terms "woody spamtrap blacklist" and I found a pattern in URLs that suggested I plug in the following: http://news.scoutnet.org/rblhostlist.php?id=saharareporters.com.uri And sure enough, there was a result. In short, it was a classic Nigerian Oil scam in which the person claims to be "JAMES IBORI ex-governor of DELTA STATE oil city." The scammer acknowledges that he has been arrested, but promises lots of cash to the person who can help him. In an effort to boost their credibility, the scammer included a link to a Sahara Reporters article about the real James Ibori. And that, my friends, is enough to have one of the most prominent independent African news organization taken offline. But, Sahara Reporters shouldn't necessarily feel singled out. Sahara Reporters should feel singled out. They exist to illuminate news from Africa. They were taken offline because a series of individuals don't know the difference between a prominent independent African news service and a criminal scammer. To most of the Internet, Nigeria and email scams are synonymous. That has to change. Although Sahara Reporters is particularly vulnerable, any site hosted with Server4All can potentially be taken down. All you have to do is write a fake spam/scam email, including a link to the web site you want to be taken offline, and then send that email to: listme@blacklist.woody.ch. I'm currently following up with both Woody's World and Server4All. However, once this particular issue is resolved, we're left with a much bigger and ominous problem. If your hosting provider (or their upstream provider) takes your site offline when it receives a complaint first and then asks questions second, you have a big problem. All of our legal fights over our rights to keep content online are moot if our providers, without any legal pressure to do so, still take down our services based on spurious complaints.

1 February 2012

Richard Hartmann: hello-planet

Hello Planet :) You may have seen my ancient blog posts pop up on the planet when XTaran was nice enough to add me to the planet's aggregation. Not quite sure why ikiwiki/planet decided to do this, but I will try to find out. If you have an idea, which does not involve re-spamming the feed, do let me know. In any case, expect mostly shell-related posts from me. And, as every year, I will be going to FOSDEM.

31 January 2012

Paul Tagliamonte: python-sunlight (or: get at some awesome US Political data programmatically)

I ve spent a few days during work, after work and on the weekend working on python-sunlight, a unified API implementation of a few Sunlight services. This is very unstable, and not released yet (so please don t rely on it yet), but it will be shortly. Be sure to sign up for an API Key, and dump the key to ~/.sunlight.key there s a simple script to help with some of this in bin/, but nothing solid yet. I do, however, encourage you to use it and play around with it, and report your bugs. Contributions (in the form of code) are also very welcome, so please do fork the project and play around with it. Just to give everyone a taste of how cool this is this will pull up a list of twitter IDs of people who mention free market more then anyone else according to the congressional record: 
from sunlight import capitolwords
from sunlight import congress
for person in capitolwords.phrases_by_entity(
    "legislator",
    phrase="free market",
    sort="count",
):
    n = congress.legislators( bioguide_id=person['legislator'],
        all_legislators="true" )
    if len(n) >= 1:
        n = n[0]
        if n['twitter_id'] != "":
            print n['twitter_id']
And the output: 
RepRonPaul
stevekingia
SenSherrodBrown
JacksonLeeTX18
SenatorLeahy
ChuckGrassley
SenJohnMcCain
OrrinHatch
DanaRohrabacher
JudgeTedPoe
McConnellPress
edtowns
SenatorDurbin
SenatorSessions
SenatorHarkin
SenChrisDodd
russfeingold
RosLehtinen
sencarllevin
senjonkyl
Dennis_Kucinich
senatorsanders
SenatorReid
RepDanBurton
RepMikePence
SenSamBrownback
joebiden
FrankPallone
ToddAkin
senatorboxer
RepTrentFranks
JohnCornyn
Have fun! Show off what cha got, and please let me know if you do something cool!

30 January 2012

Christoph Berg: How not to edit files

When packaging new backport versions, I diff the old Debian package with the old backport package to extract the changes I did there, and then apply this patch to the new version. There is always a reject in debian/changelog because the topmost bpo entry won't apply cleanly to the new changelog. To fix this, I invoke "vi debian/changelog*" and manually copy the rejected hunk. Unfortunately, I regularly end up copying it from debian/changelog.rej (buffer 3) into debian/changelog.orig (buffer 2) instead of debian/changelog (buffer 1). Here's the fix in my .vimrc: 
" Prevent accidental editing of patch .orig files
autocmd BufRead *.orig set readonly

Next.

Previous.