Search Results: "spa"

16 November 2012

Axel Beckert: Useful but Unknown Unix Tools: dwdiff better than wdiff + colordiff

A year ago I wrote in Useful but Unknown Unix Tools: How wdiff and colordiff help to choose the right Swiss Army Knife about using wdiff and colordiff together. Colordiff ed wdiff output looks like this: 
$ wdiff foobar.txt barfoo.txt   colordiff
[-foo-]bar fnord
gnarz hurz quux
bla  +foo+  fasel
But if you have colour, why still having these hard to read wdiff markers still in the text? There exists a tool named dwdiff which can do word diffs in colour without textual markers and with even less to type (and without being git diff --color-words ;-). Actually it looks like git diff --color-words, just without the git: 
$ dwdiff -c foobar.txt barfoo.txt
foo bar fnord
gnarz hurz quux
bla foo fasel
Another cool thing about dwdiff (and its name giving feature) is that you can defined what you consider whitespace, i.e. which character(s) delimit the words. So lets do the example above again, but this time declare that f is considered the only whitespace character: 
$ dwdiff -W f -c foobar.txt barfoo.txt
foo bar bar fnord
gnarz hurz quux
bla foo fasel
dwdiff can also show line numbers: 
$ dwdiff -c -L foobar.txt barfoo.txt
   1:1    foo bar fnord
   2:2    gnarz hurz quux
   3:3    bla foo fasel
$ dwdiff -c -L foobar.txt quux.txt
   1:1    foo bar fnord
   1:2    foobar floedeldoe
   2:3    gnarz hurz quux
   3:4    bla foo fasel
(coloured shell screenshots by aha)

15 November 2012

Martin Pitt: python-dbusmock templates

With python-dbusmock you can provide mocks for arbitrary D-BUS services for your test suites or if you want to reproduce a bug. However, when writing actual tests for gnome-settings-daemon etc. I noticed that it is rather cumbersome to always have to set up the skeleton of common services such as UPower. python-dbusmock 0.2 now introduces the concept of templates which provide those skeletons for common standard services so that your code only needs to set up the particular properties and specific D-BUS objects that you need. These templates can be parameterized for common customizations, and they can provide additional convenience methods on the org.freedesktop.DBus.Mock interface to provide more abstract functionality like add a battery . So if you want to pretend you have one AC and a half-charged battery, you can now simply do 
  def setUp(self):
     (self.p_mock, self.obj_upower) = self.spawn_server_template('upower',  )
  def test_ac_bat(self):
     self.obj_upower.AddAC('mock_AC', 'Mock AC')
     self.obj_upower.AddChargingBattery('mock_BAT', 'Mock Battery', 50.0, 1200)
Or, if your code is not in Python, use the CLI/D-BUS interface, like in shell: 
  # start a fake system bus
  eval  dbus-launch 
  export DBUS_SYSTEM_BUS_ADDRESS=$DBUS_SESSION_BUS_ADDRESS
  # start mock upower on the fake bus
  python3 -m dbusmock --template upower &
  # add devices
  gdbus call --system -d org.freedesktop.UPower -o /org/freedesktop/UPower \
      -m org.freedesktop.DBus.Mock.AddAC mock_ac 'Mock AC'
  gdbus call --system -d org.freedesktop.UPower -o /org/freedesktop/UPower \
      -m org.freedesktop.DBus.Mock.AddChargingBattery mock_bat 'Mock Bat' 50.0 1200
In both cases upower --dump or gnome-power-statistics will show you the expected devices (of course you need to run that within the environment of the fake $DBUS_SYSTEM_BUS_ADDRESS, or run the mock on the real system bus as root). Iftikhar Ahmad contributed a template for NetworkManager, which allows you to easily set up ethernet and wifi devices and wifi access points. See pydoc3 dbusmock.templates.networkmanager for details and the test cases for how this looks like in practice. I just released python-dbusmock 0.2.1 and uploaded the new version to Debian experimental. I will sync it into Ubuntu Raring in a few hours.

Dirk Eddelbuettel: Rcpp and the new R:: namespace for Rmath.h

We released Rcpp 0.10.0 earlier today. This post will just provide a simple example for one of the smaller new features -- the new namespace for functions from Rmath.h -- and illustrate one of the key features (Rcpp attributes) in passing. R, as a statistical language and environment, has very well written and tested statistical distribution functions providing probability density, cumulative distribution, quantiles and random number draws for dozens of common and not so common distribution functions. This code is used inside R, and available for use from standalone C or C++ programs via the standalone R math library which Debian / Ubuntu have as a package r-mathlib (and which can be built from R sources). User sometimes write code against this interface, and then want to combine the code with other code, possibly even with Rcpp. We allowed for this, but it required a bit of an ugly interface. R provides a C interface; these have no namespaces. Identifiers can clash, and to be safe one can enable a generic prefix Rf_. So functions which could clash such as length or error become Rf_length and Rf_error and are less likely to conflict with symbols from other libraries. Unfortunately, the side-effect is that calling, say, the probability distribution function for the Normal distribution becomes Rf_pnorm5() (with the 5 denoting the five parameters: quantile, mean, std.deviation, lowerTail, logValue). Not pretty, and not obvious. So one of the things we added was another layer of indirection by adding a namespace R with a bunch of inline'd wrapper functions (as well as several handful of unit tests to make sure we avoided typos and argument transposition and what not). The short example below shows this for a simple function taking a vector, and returning its pnorm computed three different ways: 
#include <Rcpp.h>
// [[Rcpp::export]]
Rcpp::DataFrame mypnorm(Rcpp::NumericVector x)  
    int n = x.size();
    Rcpp::NumericVector y1(n), y2(n), y3(n);
    for (int i=0; i<n; i++)  
        // the way we used to do this
        y1[i] = ::Rf_pnorm5(x[i], 0.0, 1.0, 1, 0);
        // the way we can do it now
        y2[i] = R::pnorm(x[i], 0.0, 1.0, 1, 0);
     
    // or using Rcpp sugar in one go
    y3 = Rcpp::pnorm(x);
    return Rcpp::DataFrame::create(Rcpp::Named("Rold")  = y1,
                                   Rcpp::Named("Rnew")  = y2,
                                   Rcpp::Named("sugar") = y3);
This example also uses the new Rcpp attributes described briefly in the announcement blog post and of course in more detail in the corresponding vignette. Let us just state here that we simply provide a complete C++ function, using standard Rcpp types -- along with one 'attribute' declaration of an export via Rcpp. That's it -- even easier than using inline. Now in R we simply do 
R> sourceCpp("mypnorm.cpp")
to obtain a callable R function with the C++ code just shown behind it. No Makefile, no command-line tool invocation -- nothing but a single call to sourceCpp() which takes care of things --- and brings us a compiled C++ function to R just given the source file with its attribute declaration. We can now use the new function to compute the probaility distribution both the old way, the new way with the 'cleaner' R::pnorm(), and of course the Rcpp sugar way in a single call. We build a data frame in C++, and assert that all three variants are the same: 
R> x <- seq(0, 1, length=1e3)
R> res <- mypnorm(x)
R> head(res)
      Rold     Rnew    sugar
1 0.500000 0.500000 0.500000
2 0.500399 0.500399 0.500399
3 0.500799 0.500799 0.500799
4 0.501198 0.501198 0.501198
5 0.501597 0.501597 0.501597
6 0.501997 0.501997 0.501997
R> all.equal(res[,1], res[,2], res[,3])
[1] TRUE
R>
This example hopefully helped to illustrate how Rcpp 0.10.0 brings both something really powerful (Rcpp attributes -- more on this another time, hopefully) and convenient in the new namespace for statistical functions.

14 November 2012

Dirk Eddelbuettel: Rcpp 0.10.0

Rcpp release 0.10.0 is now on CRAN and being uploaded to Debian. This is a new feature release, and we are very exciting about the changes, notably Rcpp attributes which make using C++ from R even easier than inline (see below as well as the new vignette for details and first examples), the extensions to Rcpp modules (see below) and more as for example new Rcpp sugar functions, a new error output device syncing to R, and a new namespace R> for the statistical functions from Rmath.h. The complete NEWS entry for 0.10.0 is below; more details are in the ChangeLog file in the package and on the Rcpp Changelog page.
Changes in Rcpp version 0.10.0 (2012-11-13)
  • Support for C++11 style attributes (embedded in comments) to enable use of C++ within interactive sessions and to automatically generate module declarations for packages:
    • Rcpp::export attribute to export a C++ function to R
    • sourceCpp() function to source exported functions from a file
    • cppFunction() and evalCpp() functions for inline declarations and execution
    • compileAttribtes() function to generate Rcpp modules from exported functions within a package
    • Rcpp::depends attribute for specifying additional build dependencies for sourceCpp()
    • Rcpp::interfaces attribute to specify the external bindings compileAttributes() should generate (defaults to R-only but a C++ include file using R_GetCCallable can also be generated)
    • New vignette "Rcpp-attribute"
  • Rcpp modules feature set has been expanded:
    • Functions and methods can now return objects from classes that are exposed through modules. This uses the make_new_object template internally. This feature requires that some class traits are declared to indicate Rcpp's wrap/as system that these classes are covered by modules. The macro RCPP_EXPOSED_CLASS and RCPP_EXPOSED_CLASS_NODECL can be used to declared these type traits.
    • Classes exposed through modules can also be used as parameters of exposed functions or methods.
    • Exposed classes can declare factories with ".factory". A factory is a c++ function that returns a pointer to the target class. It is assumed that these objects are allocated with new on the factory. On the R side, factories are called just like other constructors, with the "new" function. This feature allows an alternative way to construct objects.
    • "converter" can be used to declare a way to convert an object of a type to another type. This gets translated to the appropriate "as" method on the R side.
    • Inheritance. A class can now declare that it inherits from another class with the .derives<Parent>( "Parent" ) notation. As a result the exposed class gains methods and properties (fields) from its parent class.
  • New sugar functions:
    • which_min implements which.min. Traversing the sugar expression and returning the index of the first time the minimum value is found.
    • which_max idem
    • unique uses unordered_set to find unique values. In particular, the version for CharacterVector is found to be more efficient than R's version
    • sort_unique calculates unique values and then sorts them.
  • Improvements to output facilities:
    • Implemented sync() so that flushing output streams works
    • Added Rcerr output stream (forwarding to REprintf)
  • Provide a namespace 'R' for the standalone Rmath library so that Rcpp users can access those functions too; also added unit tests
  • Development releases sets variable RunAllRcppTests to yes to run all tests (unless it was alredy set to 'no'); CRAN releases do not and still require setting which helps with the desired CRAN default of less testing at the CRAN server farm.
Thanks to CRANberries, you can also look at a diff to the previous release 0.9.15. As always, even fuller details are on the Rcpp Changelog page and the Rcpp page which also leads to the downloads, the browseable doxygen docs and zip files of doxygen output for the standard formats. A local directory has source and documentation too. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page

Update: One link corrected.

Rudy Godoy: Function testing using function pointers in C++

I do like the way programmers think in terms of DRY, for instance, among other forms of optimization. A couple of days ago I wanted to test different implementations of the same algorithm in C++. The usual approach would be to implement those algorithms, call them in main() and see the results. A more interesting approach would be to write a validator for each function and print whether the result is correct or not. Since both are boring and break the rule of DRY, I ve decided to go complicated but productive, so I ve implemented a function testing method using function pointers that will allow me to write a single function and use it to test the algorithms using different input and expected results. Let s say I have 3 algorithms that implement Modular Exponentiation.
uint64 modexp1new(const uint64 &a, const uint64 &_p, const uint64 &n)
...
uint64 modexp2(const uint64 &_a, const uint64 &_p, const uint64 &_n)
...
uint64 modexp3(const uint64 &a, const uint64 &p, const uint64 &n)
...

 If you are not familiar with the syntax, const means that the parameter is constant, thus it will not be modified inside the function. The & means that the parameter is passed as a reference, read the memory address of the calling variable, so we don t overload RAM with new variables. Now, the testing part. For my very own purposes I just want to test those particular functions and know whether if the result they output conforms what I do expect. Some people call this unit testing.
I do also want to test it in different scenarios, meaning input and output. So, I m creating a function that will get the three parameters required to call the functions, and a fourth parameter that is the value I do expect to be the correct answer. Now, since I don t want to repeat myself writing a call for each case, I m going to create a function pointer that is an array and has the function s address as their value. That way I can call them in a loop, and voila! we are done. Finally, after calling the function I check the result with the expected value and print an OK or FAIL. Tricky part for this could be understanding the function pointer. Two things to consider: the returning value from the referenced functions has to be the same for all. Second: the input definition for each function has to be similar too. This is important because function pointers are just pointers and they need to know the size of the data type in order to navigate the memory. For this sample code the uint64 is a typedef for long long, of course. Full code is below.
void testAlgos(const uint64 &a, const uint64 &e, const uint64 &n, const uint64 &r)
uint64 (*fP[3])(const uint64&, const uint64&, const uint64&) = &modexp1new, &modexp2, &modexp3 ;
uint64 t; for(int i=0; i<3; i++) t = fP[i](a, e, n); std::cout < < "F(" << i << ") " << t << "\t";
if (t == r)
std::cout << "OK";
else
std::cout << "FAIL";

std::cout << std::endl;

 Now that I have this, I can use it in main this way.
int main() uint64 a = 3740332;
uint64 e = 44383;
uint64 n = 3130164467;
uint64 r = 1976425102; testAlgos(a, e, n, r); a = 404137263;
r = 2520752541; testAlgos(a, e, n, r); a = 21;
e = 3;
n = 1003;
r = 234;
testAlgos(a, e, n, r); return 0;
 Resulting in:
F(0) 657519405 FAIL
F(1) 1976425102 OK
F(2) 657519405 FAIL
F(0) -2752082808 FAIL
F(1) 2520752541 OK
F(2) -2752082808 FAIL
F(0) 234 OK
F(1) 234 OK
F(2) 234 OK
 I m happy about not having to write tests like this for all use cases:
std::cout < < "f1: " << modexp1new(a, e, n) << std::endl;
std::cout << "f2: " << modexp2(a, e, n) << std::endl;
std::cout << "f3: " << modexp3(a, e, n) << std::endl;
 Now, happiness can be more exciting if I do use templates so I can test any function independent of the data type for the returning and input values. You know, more abstraction. Homework if time allows! Have fun!
Related posts:
  1. Compute Clusters Integration for Debian Development and Building Report 4 Hello, this is the fourth report for the project. First,...
  2. Compute Clusters Integration for Debian Development and Testing Report 3 Hello this is the third report regarding my project. As...
  3. Compute clusters for Debian development and building final report Compute clusters for Debian development and building final report...

13 November 2012

Blars Blarson: thinkpad-x120e

New Laptop: Thinkpad x120e My travels the last few days have not been to anywhere new. I spent a few days at Tumco, then spent today in Algodones and am staying the night at Quechen Casino. Since I never described my new computer due to network problems, I figured now would be a good time to do so. I purchased a discontinued Lenovo Thinkpad x120e laptop. It has 2gb ram, a 320gb hard drive, dual-core AMD processor, and no optical drive. The first thing I did was wipe windows off of it and install Debian Squeeze (stable), then attempt to upgrade to Debian Wheezy (testing). (Squeeze lacked the drivers for some hardware.) Wheezy lacked the drivers for the display, so during the boot sequence it would switch to an unreadable screen. I eventually found that Sid (unstable) had the driver for the display in non-free. (To install it I needed to boot with the Squeeze kernel.) The most significant thing I have not yet debugged is hibernate mode. One thing I really like about this system is the 4-6 hour battery life. This is enough that I can charge during the day when my solar is in absorption mode (meaning the panels are producing more power than the house battery can safely take) and use it at night and in the morning when there is no spare solar power without depleting the house battery. The 12v charger that I had purchased for my old Thinkpad (which died, prompting the replacement) works fine on my new one. I purchased a USB powered DVD writer to give me optical capability when I need it. This uses two of the three USB ports. The network problem I had this summer was the Millenicom 3g USB stick glitched and needed to be reset with their proprietary software that is only available for Windows and Mac. Lacking access to a Windows system that I could install software on, this meant that I was without my network connection. I used the Sandy Library (fast, free wifi) 30 miles from my worksite and occasionally wifi at a closer coffee shop or cafe which were both unreliable.

12 November 2012

Gunnar Wolf: Update on my e-voting-related work @e_voto

For the last couple of years, one of the topics I have tackled with several columns and a couple of articles published, several presentations on conferences and many comments wherever I can is raising awareness on why security experts around the world oppose electronic voting. Slightly over a year ago, I started gathering all the published news (in Spanish, with some notes in English if I feel them relevant enough) that came to my greedy hands and archiving them, categorized as well as I could, into what I came to call an Electronic voting observatory. I don't feed it with the frequency I'd like, but overall I do feel there is an interesting amount of information in there. About a month and a half ago, via Twitterfeed, I set up a Twitter account where my bot echoes everything I get on that regard. So, while you can follow the observatory via RSS (as probably most readers of my blog will do), I know there are many Twitter users among you @e_voto might be interesting to you if you speak Spanish and are interested in the topic. I try not to be partisan on the information I copy there; of course I am subjective (I try not to refer to news which don't provide new insight or data points, or to obvious repeats), but I am doing this effort to follow the development as it unfolds, not to push my viewpoints.

Ingo Juergensmann: Resurrecting m68k

In August I picked up Elgar, a m68k machine, from NMMN in Hamburg, where it was ought to run as a buildd (NMMN donated space and network). Unfortunately it was in some kind of bad state: operating system was out of date, expansion cards were getting loose and NMMN wasn't happy about the CRT monitor in its datacenter as well. Elgar is an Amiga 4000 Desktop built into a custom tower case. It took some weeks and months until I found a little time to care about Elgar, but now it's up and running again:
[ 0.000000] Initializing cgroup subsys cpuset
[ 0.000000] Initializing cgroup subsys cpu
[ 0.000000] Linux version 3.2.0-3-amiga (Debian 3.2.23-1) (debian-kernel@lists.debian.org) (gcc version 4.6.3 (Debian 4.6.3-8+m68k.1) ) #1 Wed Jul 25 13:02:31 UTC 2012
[ 0.000000] Enabling workaround for errata I14
[ 0.000000] console [debug0] enabled
[ 0.000000] Amiga hardware found: [A4000] VIDEO BLITTER AUDIO FLOPPY A4000_IDE KEYBOARD MOUSE SERIAL PARALLEL A3000_CLK CHIP_RAM PAULA LISA ALICE_PAL ZORRO3
That's the stock Debian m68k kernel and it already runs on Arrakis and Vivaldi, two other buildds, as well without any problem. The only problem at the moment is the missing SCSI driver for the CyberStorm Mk1 accelerator card. There were some changes in the kernel that need to be dealt with by someone knowing. The other problem was to upgrade from etch-m68k to unstable. I already blogged about this last year. It's not as easy anymore and you'll need to deal with lots of dependency problems and such nowadays. But anyway:
elgar:~# cat /etc/debian_version
wheezy/sid
elgar:~# dpkg -l libc6
Desired=Unknown/Install/Remove/Purge/Hold
Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
/ Name Version Architecture Description
+++-====================-===============-===============-==============================================
ii libc6:m68k 2.13-35 m68k Embedded GNU C Library: Shared libraries
It's amazing that the m68k port is in such good condition after more than 4 years. That's because of the real great work of Torsten Glaser who is doing much of the porter work for the tool chain. But m68k has currently 4833 packages in state Needs-Build:
wanna-build statistics - Mon Nov 12 06:51:13 CET 2012
----------------------------------------------------- Distribution unstable:
---------------------
Installed : 1321
Needs-Build : 4833
Building : 0
Uploaded : 0
Failed : 0
Dep-Wait : 0
Reupload-Wait : 0
Install-Wait : 0
Failed-Removed : 0
Dep-Wait-Removed: 0
Not-For-Us : 0
total : 9906 13.34% (1321) up-to-date, 13.34% (1321) including uploaded
48.79% (4833) need building
0.00% ( 0) currently building
0.00% ( 0) failed/dep-wait
0.00% ( 0) old failed/dep-wait
0.00% ( 0) need porting or cause the buildd serious grief
So, there's a lot of work to do, but it's apparent that m68k won't keep up with 10.000 packages in unstable. When I started running an autobuilder back in year 2000 there were 2400 packages for m68k, on Aug. 15th 2005 we had a total of 5949 packages in the archive. For m68k this means that we will have to start adding lots of packages to Not-For-Us. I think m68k will/should end up with approx. 4000 packages at most. In the end m68k is a fairly good shape now. It only needs to get some packages built... Let's see when the buildds are operational again... ;-)
Kategorie:
Debian
Tags:
Debian
m68k
buildd
Buildd.net

11 November 2012

Jakub Wilk: All those translations, how did you ever manage it?

I've just released initial version of gettext-inspector, a tool for checking gettext PO/POT/MO files. While it's in an early stage of development, it's already able to detect wide rage of problems. For example, this is what it emits on my system: 
$ gettext-inspector /usr/share/locale/*/LC_MESSAGES/*.mo   cut -d ' ' -f 1,3   sort   uniq -c   sort -rn
   1902 P: no-language-header-field
   1601 P: no-version-in-project-id-version
   1372 W: no-report-msgid-bugs-to-header-field
    273 P: invalid-content-transfer-encoding
    201 W: invalid-date
     78 W: syntax-error-in-plural-forms
     77 I: no-package-name-in-project-id-version
     63 W: boilerplate-in-report-msgid-bugs-to
     50 W: language-disparity
     47 I: unknown-header-field
     38 W: invalid-language
     25 W: boilerplate-in-project-id-version
     10 I: unknown-poedit-language
      8 W: no-date-header-field
      5 W: no-project-id-version-header-field
      5 W: c1-control-characters
      3 P: no-mime-version-header-field
      3 P: no-content-transfer-encoding-header-field
      2 W: non-portable-encoding
      1 W: invalid-report-msgid-bugs-to
      1 W: ancient-date
      1 I: unable-to-determine-language

Andrew Pollock: [life] If you love someone, set them free

I'm going to spare everyone all the gory details at this time, but I guess it's time to acknowledge the facts. Sarah has left me. This has all come to a head extremely quickly (for me), but Sarah has been sitting on a horrible pile of emotions for months, so it's probably been an eternity for her. I still love Sarah and I'm desperately trying to reconcile, but Sarah just wants to move on. Throw a child in the mix and the fact that we're married in Australia but most of our on-paper shared property is in California, and I just don't want to think about it. I ask that our mutual friends not take sides (which I realise is difficult) and to please give Sarah all the support you can. This can't have been an easy decision for her. Of course, I'll also take any support you want to send my way. I start my sabbatical from work on Thursday (I don't return to work until January), and my parents have a prearranged visit for pretty much all of December. I had all this planned before this came out of left field. I'd debated telling my parents not to come, so I could use the time off work to try and repair things, but Zoe needs time with her grandparents, and I can probably do with having them around.

9 November 2012

Gunnar Wolf: Road trip to ECSL 2012 in Guatemala

Map of our overall driving for six days
Road trip to ECSL 2012 in Guatemala
Encuentro Centroamericano de Software Libre! Guatemala! During a national (for us) holiday, so it's easy to go without missing too much work time! How would I miss the opportunity? Several years ago, I started playing with the idea of having a road trip Probably this was first prompted with the UK crew and the Three Intrepid Motorcycle Riders arriving by land to DebConf 9 I don't know. Fact is, I wanted to go to DebConf10 in New York by land, as well as to DebConf12 in Nicaragua. Mostly due to a lack of time, I didn't Although we did start making some longish trips. Of course, my desire to show Regina what Mexico is like also helped! So, up until a week ago, our (according to my standards) long distance driving experience included:
  • M xico Guanajuato Puerto Vallarta Guanajuato M xico, in early November 2011, for Festival de Software Libre and with Regina and our Blender friends Octavio and Claudia. Totalling almost 1900Km, mostly consisting of wide, toll highway.
  • M xico Xilitla San Luis Potos M xico, in April 2012, just for fun and for a nice short vacation, alone with Regina. Totalling almost 1200Km, but through Sierra Gorda de Quer taro, a very tough stretch of about 250Km which we did at about 50Km/h on average. Beautiful route for sure! We didn't originally intend to go through San Luis Potos , and it does not appear to make much sense, as it adds ~350Km to the total, but it was even quicker than going back by the same route and according to those who now, even faster than our planned route, via Tamazunchale and Ixmiquilpan!
  • M xico San Luis Potos Zacatecas Aguascalientes Guanajuato M xico, in May 2012, for Congreso Internacional de Software Libre, again with Octavio and Claudia. Totalling 1250Km, and following very good roads, although most of them were toll-free.
But there is always a certain halo over crossing a border, maybe more so in countries as large as Mexico. We convinced Pooka and Moni, and granted, with some aprehension, as we knew of some important security risks in the more rural areas we wanted to go through we decided to go to Guatemala. And, although we wanted to go with a bit more time, Real Life took its toll: We could not take more time than the intersection of what our respective jobs offered. So, here goes a short(?) recap of our six day long, 3200Km trip. Of course, we have a map detailing this. Mexico Veracruz I came to my office early on Wednesday (31-oct), and left with Regina around 10AM towards Veracruz. We agreed to meet there with Moni and Pooka, who would take the night bus, and continue together. Crossing Mexico City proved to be the longest obstacle We arrived to Veracruz already past 3PM, and spent a nice evening walking down the center and port of the city. Veracruz port can still be seen as part of central Mexico; I knew the road quite well. Veracruz San Andr s Tuxtla Catemaco San Cristobal de las Casas We met with our friends at the iconic Gran Caf de la Parroquia at 6:30AM. Had a nice breakfast with coffee, and by 7:30 we were heading south-west. The reason to have a road trip was to get to know the route, to enjoy the countryside So, given we "only" had to make 650Km this day, we took the non-toll road A narrow path stretching along the coastal plains of Veracruz, until Acayucan. Doing so, we also saved some money, as the equivalent toll road is around MX$300 (~US$25)! Veracruz is a hot state. We ended up all sweaty and tired by 19:00, when we reached San Cristobal. We had agreed not to drive at night, due to security issues, but fortunately there was quite a bit of traffic both ways between Tuxtla Guti rrez (Chiapas State capital, around 1hr from San Cristobal, where darkness got us) and our destination, so we carried on. Now, San Cristobal is a high city, almost as high as Mexico City (2100m), and being more humid, it was quite chilly. We went for a walk, and were convinced that at a later time, we had to stay for several days there. The city is beautiful, the region is breath-taking, there is a lot of great handicrafts as well, and it's overall very cheap. Really lovely place. San Cristobal de las Casas Cd. Cuauht moc La Mesilla Guatemala Once again, this day started early. We woke up ready to leave at 7AM, and not earlier because the hotel's parking didn't open earlier. After a very quick visit to San Cristobal downtown, to take some photos that were not right the night before, we took the road to Comit n, stopping just for some tamales de bola y chipil n for breakfast. Central Chiapas is almost continuously populated, differing from most of my experience in Mexico. It is all humid, and has some very beautiful landscapes. We passed Comit n, which is a much larger city than what we expected, went downhill after La Trinitaria, crossed a plain, and continued until hills started taking over again. We stopped in a very chaotic, dirty place: Just accross the border, where Ciudad Cuauht moc becomes La Mesilla. This border was basically what we expected: There is no half-official place to exchange money, so we had to buy quetzales from somebody who offered them on the street, at MX$2 per Q1 (where the real exchange should be around 1.50 to 1). While on the road, I was half-looking for exchange posts in Comit n and onwards, and found none (and being a festive day, they would probably be closed anyway). But we were expecting this, after all, and exchanged just the basic minimum: MX$600 (US$50, which by magic became Q300, US$40). The tramit consists of:
  • Spraying the car against diseases (which has a cost of Q18)
  • Each of us has to go through migration. Note, in case you cross this border: We didn't expressly cross Mexican migration, so officially there was no record of us going out. Be sure to go through migration to avoid problems at re-entry!
    Migration has no cost.
  • Customs. As we were entering by car, I had to purchase a permit for circulation. I don't remember the exact quote, but it was around Q150, and the permit is valid for 90 days.
  • That's it! Welcome to Guatemala!
La Mesilla is in Guatemala's Huehuetenango Department, and from all of the Departments we crossed until Guatemala city (Huehuetenango, Quetzaltenango, Totonicap n, Solol , Chimaltenango, Sacatep quez and Guatemala), this is the largest one. Huehuetenango is home to the Cuchumatanes mountain ridge. We found beautiful, really steep, really fertile mountains. It is plainly amazing: Mountains over 60 , and quite often full with agricultural use Even at their steepest points! The CA-1 highway is, in general, in very good shape. There are however many (many, many) speed bumps (or topes, in Mexican terminology. Or t mulos in Guatemalan), at least a couple at every village we crossed, not always painted. The road is narrow and quite winding; it follows river streams for most of the way. We feared it would be in much worse shape, from what we have heard, but during the whole way we found only three points where the road was unusable due to landslides and an alternative road was always in place when we needed it. After Totonicap n, the narrow road becomes a wide (four lane) highway. Don't let that fool you! It still goes through the center of every village along the road, so it's really not meant for speeding. Also, even though the pavement is in very good condition, it is really steep quite often. It is not the easiest road to drive, but it's (again) by far not as bad as we expected. We arrived to Guatemala City as dawn was falling, and got promptly lost. Guatemala has a very strange organization scheme: The city is divided in several zones, laid out in a swirl-like fashion. East-west roads are called Calle and North-south roads are called Avenida (except for zona 4, I think, where they are diagonal, and some are Rutas while the others are V as). I won't go into too much detail). Thing is, many people told us it's a foolproof design, and people from different countries understand the system perfectly. We didn't... At least not when we arrived. We got quite lost, and it took us around one hour to arrive to our hotel, at almost 19:00 Almost 12 hours since we left San Cristobal. Went for a quick dinner, and then waited for our friends to arrive after the first day of work of ECSL, which we missed completely. And, of course, we were quite tired, so we didn't stay up much longer. Antigua Guatemala On Saturday, ECSL's activities started after 14:00 so we almost-kidnapped Wences, the local organization lead, and took him to show us around Antigua Guatemala. Antigua was the capital of Guatemala until an earthquake destroyed it in the 1770s; the capital was moved to present-day Guatemala city, but Antigua was never completely abandoned. Today, it is a world heritage site, a beautiful city, where we could/should have stayed for several days. But we were there for the conference, so we were in Antigua just a couple of hours, and headed back to Guatemala. Word of caution: Going from Guatemala to Antigua, we went down via the steepest road I have ever driven. Again, a real four-lane highway... but quite scary! The main focus for this post is to give some roadtrip advice to potential readers... So, this time around, I won't give much detail regarding ECSL. It was quite interesting, we had some very good discussions... but it would take me too much space to talk about it! The road back: Guatemala Tec n Um n; Cd. Hidalgo Arriaga So, about the road back: Yes, we just spent three days getting to Guatemala City. We were there only for ~36 hours. And... We needed to be here by Tuesday morning no matter what. So, Sunday at noon we said goodbye to our good friends in ECSL and started the long way back. To get to know more of Guatemala, we went back by the CA-2 highway, which goes via the coastal plains Not close to the Pacific ocean, which we didn't get to see at all, but not through the mountains. To get to CA-2, we took CA-9 from Guatemala. If I am not mistaken, this is the only toll road in Guatemala (at least, the only we used, and we used some pretty good highways!) It is not expensive; I don't remember right now, but must have been around Q20 (US$3). Went South past Palin and until CA-2, just outside Escuintla city, and headed West. All of Escuintla and Suchitep quez it is again a four lane highway; somewhere in Retalhueu it becomes a two lane highway. We were strongly advised not to take this road at night because, as the population density is significantly lower than in CA-1, it can get lonely at times And there are several reports of robberies. We did feel the place much less populated, but saw nothing suspicious in any way. Something important: There are almost no speedbumps in CA-2! The terrain stayed quite flat and easy as we crossed Quetzaltenango, and only in San Marcos we found some interesting hills and a very strong rain that would intermitetntly accompany us for the rest of the ride. So, we finally arrived to the border city of Tec n Um n at around 16:30 Approximately four hours after leaving the capital. The Tec n Um n Cd. Hidalgo cities and border pass are completely different from the disorderly and dirty Cd. Cuauht moc La Mesilla ones. The city of Tec n Um n could be just a nice town anywhere in the country, it does not feel aggressive as most border cities I have seen in our continent. We stopped to eat at El pollo campero and headed to the border. In the Mexican side, we also saw a very well consolidated, big and ordered migration area. Migration officers were very kind and helpful As we left Cd. Cuauht moc, Regina didn't get a stamp of leaving Mexico, so technically she was ilegally out of the country (as she is not a national... They didn't care about the rest of us). The tramit to fix this was easy, simple, straightforward. We only paid for the fumigation again (MX$60, US$5), and were allowed to leave. Anyway, we crossed the border. There is a ~30Km narrow road between Cd. Hidalgo and Tapachula, but starting in Tapachula we went on Northwards via a very good, four lane and very straight highway. Even though we had agreed not to drive at night... Well, we were quite hurried and still too far from Mexico City, so we decided to push it for three more hours, following the coastline until the city of Arriaga, almost at the border between Chiapas and Oaxaca. Found a little hotel to sleep some hours and collapsed. Word of warning: This road (from Tapachula to Arriaga) is also known for its robberies. We saw only one suspicious thing: Two guys were pushing up their motorcycle, from which they had apparently fallen. We didn't stop, as they looked healthy and not much in need of help, but later on talked about this Even though this was at night, they were not moving as if they had just crashed; nothing was scratched, not the motorcycle and not their clothes. That might have been an attempt to mug us (or whoever stopped by). This highway is very lonely, and the two directions are separated by a wall of vegetation, so nobody would have probably seen us were we to stop for some minutes. Be aware if you use this road! The trip comes to an end: Arriaga Niltepec Istmo C rdoba M xico The next (last, finally!) day, we left at 6:30AM. After driving somewhat over one hour, we arrived to Niltepec, where a group of taxi drivers had the highway closed as a protest against their local government's tolerance of mototaxis. We evaluated going back to Arriaga and continue via the Tuxtla Guti rrez highway, but that would have been too long. We had a nice breakfast of tlayudas (which resulted in Pooka getting an alergic reaction shortly afterwards) and, talking with people here and there, were told about an alternative route by an agricultural road that surrounds the blockade. So, we took this road the best way we could, and after probably 1hr of driving at 20Km/h, finally came back to the main road. We planned on crossing the isthmus using the Acayucan-Juchit n road We were amazed at the La Ventosa ("the windy") area, where we crossed a huge eolic plant for electricity generation, so of course we got our good share of photos. From then onwards, not much more worth mention. Crossed the isthmus via a quite secondary road in not too good shape (although there is a lot of machinery, and the road will much likely improve in the next few months/years), then took the toll freeway along Veracruz until C rdoba. We stopped for a (delilcious and revigorizing!) cup of coffee in Hotel Zeballos, where Agust n de Iturbide signed with Viceroy Juan O'Donoj the treaties that granted Mexico the independence. Traveller, beware: When crossing between Puebla and Veracruz, there is a steep slope of almost 1000m where , you will almost always (except if it's close to noon) find very thick fog; taking the highway from C rdoba, this is in the region known as Cumbres de Maltrata. We had the usual fog, and just as we left it, a thin but constant rain that went on with us up until we got home. Crossed Puebla state with no further eventualities, and arrived to Pooka and Moni's house by 22:00. Less than one hour later, Regina and me arrived home as well. This was four days ago... and I have finally finished writing it all down ;-) Hope you find this useful, or if not, at least entertaining! If you read this post in my blog, you will find many pictures taken along the trip below (Well, if you are reading the right page, not in the general blog index...). If you are reading from a planet or other syndication service... Well, come to the blog! Dreamhost woes Oh, and... Yes, it sometimes happens: My blog is hosted at Dreamhost. This means that usually it works correctly... But sometimes, specially when many people request many nontrivial pages, it just gives an error. If you get an error, reload once or twice... Or until your patience manages ;-)

Olivier Berger: Using a DreamPlug with Debian as an IPv4/IPv6 router for a home LAN behind a FreeBox

The goal of this article is to describe the setup I've made for my home LAN behind a FreeBox (French ISP Free.fr).

1 Principles
The FreeBox is configured :
  • in router mode (with a local IPv4 network on 192.168.0.x/24)
  • with IPv6 support activated (by convention, we'll use the 2a01:e35:1234:5678::/64 prefix below, to represent the IPv6 prefix provided by the FreeBox)
Behind the FreeBox, I have connected a DreamPlug computer (running Debian wheezy with a dual-boot setup as sold by NewIT) on its eth0 etherlink port. Behind the DreamPlug, I have connected a switch (to its eth1 ethernet port), on which the LAN's machines will be connected, on a dual IPv4/IPv6 network. The IPv4 will be on 192.168.4.0/24 (addresses for machines connected to the switch will be static) while the IPv6 will use the addresses auto-generated on the FreeBox's prefix. The DreamPlug will also provide a wifi network (in master mode, through its uap0 interface), only IPv4 on 192.168.3.0/24. On the DreamPlug, the outgoing IPv4 traffic will be masqueraded, while the IPv6 traffic will be bridged so that machines on the LAN can be reached from the Internet directly, passing through the eth0 eth1 bridge. The DreamPlug will also act as a filter/firewall to only allow selected traffic to reach the LAN machines (in particular for the IPv6, since the machines may be reached directly from the Internet otherwise).

1.1 Alternatives
There are a lot of howtos describing the setup of FreeBox + IPv6, but I couldn't find one that was exactly reflecting the kind of setup I have. Some dealt with a FreeBox setup in bridge mode (i.e. not the router mode), but in case the DreamPlug isn't so much reliable, or I've messed with the config, I'm tempted to keep the possibility to connect directly to the FreeBox. Others didn't use the same setup with a router distributing both wired and wifi networks. And note described exactly the setup for a DreamPlug running Debian wheezy and using ferm as a firewall. No, there's no NIH syndrom here ;)

2 DreamPlug router configuration

2.1 Interfaces configuration
The /etc/network/interfaces on the DreamPlug looks like the following (kudos to Dag Bertelsen's Setting up Dreamplug as a router with Shorewall firewall for the wifi configuration bits): 
# Note: Order of the interfaces declarations matters

auto lo
iface lo inet loopback

auto uap0
iface uap0 inet static
       post-up /usr/bin/uaputl sys_cfg_ssid dream-uAP-XXXX
       # Modus WPA2
       post-up /usr/bin/uaputl sys_cfg_protocol 32
       # WLAN password
       post-up /usr/bin/uaputl sys_cfg_wpa_passphrase "WHATEVER PASSWORD"
       # Set the Cipher
       post-up /usr/bin/uaputl sys_cfg_cipher 8 8
       # automatic channel
       post-up /usr/bin/uaputl sys_cfg_channel 0 1
       post-up /usr/bin/uaputl bss_start
       # we deliver addresses for the Wifi with DHCP
       post-up /usr/sbin/service udhcpd start
       address 192.168.3.1
       netmask 255.255.255.0

# Only IP6 traffic passes through the router between the Freebox on eth0 and the LAN on eth1
auto br0
iface br0 inet manual
      bridge_ports eth0 eth1
      bridge_maxwait 0
      pre-up ebtables -t broute -A BROUTING -p ! ipv6 -j DROP
      down ebtables -t broute -F

# Connected to the FreeBox, with DHCP in IPv4, masquerading outgoing IPv4 traffic
auto eth0
iface eth0 inet dhcp
        up iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
        down iptables -t nat -F
        post-up sysctl -q -w net.ipv4.ip_forward=1
iface eth0 inet6 auto

# Connected to the LAN with a static IPv4 address
auto eth1
iface eth1 inet static
      address 192.168.4.1
      netmask 255.255.255.0

2.2 DHCP server configuration
The DHCP config for udhcpd is as such (in /etc/udhcpd.conf): 
start 192.168.3.100
end 192.168.3.200
interface uap0
opt      lease  86400
opt     router  192.168.3.1
opt     subnet  255.255.255.0
opt     dns     192.168.3.1
opt     domain     localdomain
max_leases     101
lease_file     /var/lib/udhcpd.leases
auto_time       5

2.3 Firewall configuration
The firewall I use is ferm. Here's the /etc/ferm/ferm.conf configuration (YMMV). Thanks to Ren Garcia's R gles de firewall pour un pont IPv6 for inspiration : 
## Interfaces of the router

# Our LAN on a switch
@def $DEV_PRIVATE = eth1;

# The ISP router
@def $DEV_WORLD = eth0;

# Our Wifi LAN
@def $DEV_WIFI = uap0;


## IP networks of our LAN

# IPv4 (includes ethernet switch and wifi)
@def $NET_PRIVATE = 192.168.0.0/16;

# IPv6 (the prefix granted by the ISP)
@def $NET6_PRIVATE = 2a01:e35:1234:5678::/64;

# IPv6 Local link
@def $LINK6 = fe80::/10;

# Specific machines on the LAN
@def $MACHINE_MYDESKTOP = 2a01:0e35:1234:5678:1234:5678:9abc:def0;

# Special function to log dropped packets
@def &DROP() =  
  mod limit limit 1/second limit-burst 10 LOG log-prefix "$DOMAIN-$CHAIN: ";
  DROP;
 

# IPv4 iptables rules
table filter  
  chain INPUT  
  policy DROP;

  # connection tracking
  mod state state INVALID DROP;
  mod state state (ESTABLISHED RELATED) ACCEPT;

  # allow local packet
  interface lo ACCEPT;

  # respond to ping
  #proto icmp ACCEPT; 
  proto icmp icmp-type echo-request ACCEPT;

  # allow IPsec
  proto udp dport 500 ACCEPT;
  proto (esp ah) ACCEPT;

  # allow SSH connections from the private network and from some
  # well-known internet hosts
  #saddr ($NET_PRIVATE 123.45.67.89) proto tcp dport ssh ACCEPT;
  saddr $NET_PRIVATE proto tcp dport ssh ACCEPT;

  # we provide DNS and SMTP services for the internal net
  interface $DEV_PRIVATE saddr $NET_PRIVATE  
  proto (udp tcp) dport domain ACCEPT;
  proto tcp dport smtp ACCEPT;
   

  interface $DEV_WIFI  
  # DHCP
  proto udp dport 67:68 ACCEPT;
  proto (udp tcp) dport domain ACCEPT;

  proto udp dport netbios-dgm DROP;
   

  # LOG all dropped packets
  proto (udp tcp) &DROP();

   

  # outgoing connections are not limited
  chain OUTPUT policy ACCEPT;

  chain FORWARD  
  policy DROP;

  # connection tracking
  mod state state INVALID DROP;
  mod state state (ESTABLISHED RELATED) ACCEPT;

  # connections from the internal net to the internet or to other
  # internal nets are allowed
  interface $DEV_PRIVATE ACCEPT;
  interface $DEV_WIFI ACCEPT;

  # LOG all dropped packets
  proto (udp tcp) &DROP();
   
 

# IPv4 masquerade / NAT support

table nat  
  chain POSTROUTING  
  # masquerade private IP addresses
  saddr $NET_PRIVATE outerface $DEV_WORLD MASQUERADE;
   
 

## IPv6:

domain ip6  
  table filter  

  chain INPUT  
  policy DROP;

  # Internal traffic authorized
  saddr $NET6_PRIVATE ACCEPT;
  saddr $LINK6 ACCEPT;

  # accept multicast
  daddr ff00::/8 ACCEPT;

  interface lo ACCEPT;

  # For public traffic
  # Only DNS resolution + NTP acceped for UDP
  interface br0  
  proto udp  

  sport domain ACCEPT;

  dport ntp ACCEPT;

  &DROP();
   

  # TCP on ports < 1024 forbidden except for SSH
  proto tcp  

  dport ssh ACCEPT;

  dport 0:1023 DROP;
   

  # Accepted established traffic
  mod state state INVALID DROP;
  mod state state (ESTABLISHED RELATED) ACCEPT;

  # incoming connections forbidden
  proto tcp syn DROP;

  # Accept ping
  proto icmpv6 ACCEPT;
   

  proto (udp tcp) &DROP();
   
  chain FORWARD  
  policy DROP;

  # All local traffic accepted
  saddr $LINK6 ACCEPT;

  # All ping accepted unless coming from internet
  saddr ! $NET6_PRIVATE proto icmpv6 icmp-type echo-request DROP;
  proto icmpv6 ACCEPT;

  # Allow SSH to specific machines
  proto tcp daddr $MACHINE_MYDESKTOP dport ssh ACCEPT;

  # Accept established traffic
  mod state state INVALID DROP;
  mod state state (ESTABLISHED RELATED) ACCEPT;

  # Accept outgoing traffic
  saddr $NET6_PRIVATE ACCEPT;

  # Accept local outgoing traffic
  saddr ::1 ACCEPT;

  proto (udp tcp) &DROP();
   
   
 

# ebtables
domain eb chain FORWARD  
  policy DROP;

  # Only accept bridged traffic for IPv6
  logical-in br0 logical-out br0   
  #ebtables -t broute -A BROUTING -p ! ipv6 -j DROP
  proto IPv6 ACCEPT;
2.4 DNS config

The DreamPlug will run bind9 to cache the DNS for the wifi network (default config is OK out of the box).

3 Configuration for the LAN machines

3.1 Connection with Wifi
Machines connected with the WPA wifi network should get an IPv4 only address via DHCP on the 192.168.3.0/24 network. There's nothing to be configured specifically.

3.2 Ethernet connection on the switch
Machines connected via ethernet cables to the switch can run either with IPv4 config, in which case they should be configured statically for the 192.168.4.0/24 network, using the router on 192.168.4.1 as a gateway, or with IPv6 auto config. On wheezy, the /etc/network/interfaces will look like : 
auto eth1
iface eth1 inet static
  address 192.168.4.42
  netmask 255.255.255.0
  gateway 192.168.4.1
  dns-nameservers 212.27.40.241 212.27.40.240

iface eth1 inet6 auto

4 To do list

The setup works fine, but there are a few things I'd like to improve :
  • running a regular Debian distro on the DreamPlug (the kernel used at the moment is patched for the wifi driver in master mode support), possibly using only Free Software.
  • using subnets for the IPv6 machines (but that seems not exactly obvious, requiring NDP proxying and stuff), so that the privacy is improved (the IPv6 auto generated global addresses depend on the MAC addresses hmmm probably not optimal).
  • adding more stuff to the DreamPlug router so that it becomes a full fledged FreedomBox.

5 References

Date: 2012-11-09 16:30:34 CET

7 November 2012

MJ Ray: Adrift @coopsutd in @coopParty workshop

image

Welney washes

Sometimes I felt at home at Cooperatives United, like at the wonderful fringe dinner at Eighth Day last Thursday night. Sometimes I felt overwhelmed by the flood, like at the cooperative party workshop on Friday morning. The workshop addressed the key questions: why do we want cooperative politicians, what could they do and what are the needs and challenges facing them? Personally, I felt the first is obvious and the answer is all the second. It s both frustrating (few ask why other movements want to influence politicians) and reassuring (no taboo topics) that the first question is asked. There were many fascinating answers to the other two questions from around the world. As so often with workshops, few hard conclusions were reached, but I think some interesting conversations were started there, which we might hear again in the future. What would I call the top challenge? Persuading open and transparent co-op members to join the party. I ve written about my difficulty with joining political parties in the past, as some forms of collective responsibility are against my beliefs and I m not sure that joining a small party will help anyone much, but I think I ll take another look at the cooperative party, all the same. The picture above is of the Welney washes, some miles south-west of Lynn. It s a stunning wetland environment, and it surprises me every time the train crosses the floods, as it did on the way to Manchester. And yes, I ve shoehorned a picture I liked into this article!

6 November 2012

Stefano Zacchiroli: bits from the DPL for October 2012

Freshly baked, bits from the DPL for October 2012.
Dear Project Members,
another month, another periodic report of DPL-ish activities, this time for October 2012. Highlights Debian on public clouds I've spent quite some time to improve Debian presence on the so called "public clouds". Following up to an inquiry of a fellow developer, I've reached out to Microsoft to investigate the possibility of having Debian as an option on Windows Azure. Around the same time, I've been approached by Amazon to have Debian as an option on the AWS marketplace. In both cases, we will need to overcome challenges of various kinds, at the technical (e.g. image preparation), bureaucratic (e.g. terms of the agreements we'll need to accept to be present), and political (e.g. chain of trust, platform freedom) levels. Up to now, discussions have been going on mostly in private, simply because they started as 1-to-1 inquiries and continued from there, but there is no good reason they should remain so. Hence, thanks to the listmasters and in particular Alexander Wirt, we have setup the new debian-cloud mailing list. If you are interested in these topics please join the list. For both Azure and AWS there is good progress on the technical part already; summaries will soon be posted on the list so that we are all on the same page. Similarly, I'll post there status reports about the bureaucratic requirements. And of course there is no reason to focus on specific clouds, if you'd like to support others and are willing to put some work to that end, please join the list and let us know. DPL helpers meeting I've already bothered you at least in my last platform and DebConf13 talk with observations about how non-scalable the DPL job is. After having collected applications of DPL helpers for a while, I've finally sat down and tried to put those applications into good use. The idea is simple: to the extend of possible, we should shift from a one-man-band job to a more "board-like" job, with people sharing an agenda, a list of outstanding tasks, and public communication. We have started slowly, setting up the #debian-dpl IRC channel and running periodic bi-weekly meetings there. You can find the meeting minutes and full logs at the usual place. We are still ramping up, so we don't have yet "fancy" stuff like a mailing list or an issue tracker entry, but they're in the working. Some of the outcome are starting to show, too (e.g. as part of recent discussions on 3rd party orphaning, or on our inbound trademark policy, or even in the forthcoming DMCA policy to make mentors.d.n an official project service). It's an experiment and a big challenge. I'm, for one thing, not yet convinced there are enough people interested in sharing the load of DPL duties (that look boring, for many tech geeks) in the long run. But I'm also convinced that the sustainability of the Debian organization model depends on this, so it's worth trying. If you're interested in the challenge and willing to volunteer some of your time, please join us on #debian-dpl . I'll take care of keeping the project informed of further evolution, in particular about the communication channels we will pick for day to day activities and accountability. Events / public communication I've spent most of my remaining Debian time in October attending events on behalf of the Project, in particular: On the topic of public communication, I've also coordinated with the press team an answer to a press inquiry about Secure Boot (which has become part of this article), and happily vouched for the Ubuntu charity marathon, adding some Debian challenges to it. Delegations As largely overdue matters, I've finalized the delegations mentioned last month, namely: FTP masters, New Members Front Desk, and Policy editors. Assets and legal stuff Miscellanea Now let's all go back to RC Bug squashing to make Wheezy a reality. SPAM-my link of the month is http://udd.debian.org/bugs.cgi and its various "views" at the bottom of the page. Cheers.
PS the day-to-day activity log for October 2012 is available at the usual place master:/srv/leader/news/bits-from-the-DPL.txt.201210

Andrew Pollock: [life] On leasing a Leaf

Circumstances have required us to become a two-car family, and I've been very keen on the Nissan Leaf for some time. It's an excellent little car. A friend had leased one, and had positive things to say about the whole leasing process, so on Saturday we leased one ourselves. Now that we've been through the process, I'm not sure why you'd ever buy a car instead of lease. We paid $4,628 as a "down payment" (of which we'll get a $2,500 rebate) and it's going to cost I think $257 a month. Maintenance for the 3 years is included in those monthly payments as is any damage to the vehicle at the end of the lease. After the 3 years you hand the car back or pay out the residual. We plan on handing the car back. We get 12,000 miles a year, after which it's 15 cents a mile. There's no way in the world we'll do 12,000 miles a year in it. The last three cars I've owned I've just paid for in cash, because I generally don't like debt and like to minimise my monthly outlays, but in the case of cars, I also don't like forking out a large wad of cash for something that instantly loses value the moment it drives off the lot. I also think that given the pace of innovation in the electric vehicle space, the new hotness in three years time will be even more awesome than the Leaf is today. It gets delivered tomorrow.

5 November 2012

Vincent Sanders: Another NetSurf Developer Workshop

The NetSurf Developers - Bringing you alternative internet meme delivery since 2002
The weekend saw some of the NetSurf developers gather once again to do battle against the Red Queen...no, wait. I mean plan and write code.

The successful event was once again kindly hosted by Collabora at their Cambridge office, the facilities remain as good as at the previous event and provided a comfortable environment in which to work.

Four developers managed to attend from around the UK John-Mark Bell, Vincent Sanders, Daniel Silverstone and Rob Kendrick. The turnout was a little lower than previous events, however Michael Drake managed to assist with some bug fixes remotely.

From Friday to Sunday afternoon we each managed to put in 24 hours of work consisting of almost 90 commits changing over 100 files and removing 10,000 lines of code.

Whiteboard with the 3.0 release tasks
The main theme of the event was consolidation ready for the 3.0 release. While not a glamorous target this is a major release for the project which we want to achieve soon, this requires the completion of numerous ongoing tasks (like finally removing the dependency of libxml) and fixing as many reported bugs as possible.

The Major things we achieved were:

Once the final usage of libXML is removed (from the hotlist code) NetSurf will become substantially smaller on several platforms as we will not have two separate parser implementations linked.

One big feature of the 3.0 release which was not originally on the roadmap is that of JavaScript. We initially intend to provide two builds of with and without the JavaScript interpreter compiled into NetSurf. This allows those on very small systems to continue to run NetSurf and those users who want a richer web experience to enjoy it.

Although JavaScript is a major new feature which is frequently requested the initial implementation will be incomplete. To complete the feature several thousand JavaScript to DOM interface binding functions must still be written. In addition it should be noted that we have yet to add dynamic layout or extensions such as Canvas or WebGL

We had a Raspberry Pi which was used to run RISC OS so we could examine the operation on this system.

The 2.9 edition as shipped works OK but is lacking the substantial progress made in the last six months.

The CI system build was run on the system and works well so we are reassured the 3.0 edition will provide an even better browsing experience.

The event was a great success overall and we are now very close to our 3.0 release which we hope to produce before Christmas.

4 November 2012

Wouter Verhelst: Windows usability: followup.

Yesterday, I posted a blog post containing a list of issues with Windows, in response to a similar list for Linux, which has received quite some response; I've seen people +1 it on google+, and I've received a number of replies on the blog post itself. One of these replies was by the author of the original list; and as expected, he totally and utterly failed to see the point. He also has the audacity to claim I haven't installed Linux yet what?!? The last time I used anything other than Linux on my personal machine for any real work was on Windows 98. And no, that was not because I was too cheap to buy the latest and greatest. I have used Windows in some cases since then, but never on my own hardware. First of all, I should note that the list which I posted was only semi-serious (with tongue in cheek); the fact that it refers to Windows as the operating system which "attempts to replace Linux as the major operating system" should have been a pretty big clue to that effect. Like the original list that it was a reply to, my list contains things that are only true in a particular context (e.g., the one about requiring recent hardware), it contains things that don't matter for the large majority of Windows users (e.g., the point about focus-follows-mouse) but that are major arguments for why I don't use Windows, it contains things that are so alien to what most of us are used to that even most Linux users don't use it, while on the other hand some swear by it (like the SELinux argument), and it contains some things that a seasoned Windows user will call bullshit on but that are real actual problems for the non-seasoned Windows user that I am (such as the mouse argument). It also contains some actual facts, like the statement about most crashes being a direct result of faulty drivers that's pulled directly from a scientific study on the subject conducted by Microsoft, together with some independent researchers. Like the original list, I don't expect this list to change anything for the better, precisely because it is flawed in exactly the same way. But whether the items on that list were right or wrong isn't the point. This is: Nobody (except nutcases) ever said Linux is perfect, or that it's useful for everyone; and while the issues Artem lists may be bugs or may be design flaws or any of a number of other things that could be real problems, the fact is that creating a long "list of issues" isn't going to solve anything. If you find that there are problems with Linux, you basically have three options: fix the problem, get someone to fix it for you, or stop using Linux. Really. Rather than better or worse, Linux is different. It will have its own set of problems, and it will have its own set of advantages. Some of the design decisions cause problems, but also have serious advantages, and focussing on only these problems (the existence of which I'm not denying) isn't very helpful. A good example is the library model: under Linux, libraries are typically installed in a system-wide location, and use a SONAME which is (part of) the filename of that library and which defines the binary interface exported by that library. In contrary to the situation under Windows, Linux libraries also must use position-independent code. The result is a more robust library model. The downside to that model, however, is that when you install older applications that require older SONAME versions of a particular library, getting those older versions may be a challenge. While in theory, installing an older version of a library is not going to cause any problem if the SONAME is different, in practice you'll often find that you'll need to go to (unsupported) older packages from older versions of the distribution, which may not install due to missing or conflicting dependencies, or which may expect data files in a different format in the same location that newer versions of the same library do, or which may have any number of other issues. When that happens, the solution is to either use an older version of the distribution in a chroot or some such, or to patch and/or compile the needed libraries yourself, and install them alongside the newer version of the same library. That's perfectly possible, after all, precisely because of the SONAME. Another good example is the development model: where Windows has a central authority (Microsoft) which defines what it is to be "Windows" and what is and isn't part of that, the same simply isn't true about Linux. Yes, Linus defines what the official Linux kernel has, but he doesn't have any say about the user interface, the boot process, or any number of other things; other people make decisions there. As a result, in effect, who decides what it means to be a "Linux system" is, really, you, the user. If you believe that the Linux user space is completely wrong, you can just change it to something else entirely; and that's precisely what Google did when they created Android. Since there's no central authority making decisions about what will and won't go into the system, this means that inevitably, at any given time, there will be many competing options for many subsystems. Usually this will be an advantage; e.g., the competition between KDE and Gnome benefits both. Sometimes, however, the competing options are so confusingly interwoven that even those who should know the system inside out can't make heads or tails of it, and debugging issues then becomes quite a challenge. Unfortunately, the Linux sound subsystem is currently an example of the latter. I'm not saying that all the items in Artem's list are invalid; a minority are, in fact, actual problems for a majority of Linux users. Most however, are personal opinions, issues affecting only a small minority of Linux users, or, as above, things where he only talks about the downsides of a design decision that also has major advantages. As an example of something in the second category: it's easy, for example, to list hundreds if not thousands of issues relating to particular pieces of hardware that don't work under Linux, simply because the manufacturer doesn't provide any information, and the people developing drivers for Linux have to reverse engineer how everything works. If you're a bit careful in what you buy, however much like MacOS users need to be careful, too then using Linux isn't an issue. My Lenovo Thinkpad X220, for instance, has no parts that don't work; and this includes things like a docking station, and the fact that my (9-cell) battery life is over fourteen hours, which (I believe) is more than on the same laptop with the same battery under Windows. So how is making a list of issues useful? The answer to that is simply, not at all. If you want to help, don't look for other people's problems; we can all do that. Instead, try to look for solutions to these problems. That's not always easy, and in many cases requires that you first understand why something is done in a particular way. If you don't want to spend time doing that, that's perfectly fine; but then don't expect that open source developers (who are already strapped for time, usually) will suddenly jump up and fix the problems you think are most important, as opposed to what they think should be worked on most urgently. After all, most Open Source developers have their own sets of priorities what they think requires most work, and that's what they'll work on not what's shown in a particular list by someone they don't know and have no reason to respect, somewhere on the Internet. Finally, I'd like to note that any statement which claims that Linux isn't "ready" to be used on the desktop, when hundreds of thousands (if not millions) of people are using it in exactly that way for their day-to-day work, is insulting these people's intelligence. Since that includes me, it's also insulting my intelligence. And I don't like to have my intelligence insulted...

Jaldhar Vyas: A Tale To Strike Terror Into The Hearts Of All Men, Ninjas, and Disney Princesses

Normally around this time of year I would post a picture of one of my children in their Halloween costumes but alas this All-Hallows Eve, Hurricane Sandy has put an end to such frivolity. For, from out of this hell-born wind comes the tale of a creature so horrifying that even puissant warlocks of dark power cower in their Batman pajamas at the sound of his doom-laden moans rending the night as he wanders pitiously through the blackened streets searching for his forbidden love. I speak of the one known only as... THE MAN WHO WENT FOUR DAYS WITHOUT INTERNET Halloween Pic

2 November 2012

Vincent Bernat: Network virtualization with VXLAN

Virtual eXtensible Local Area Network (VXLAN) is a protocol to overlay a virtualized L2 network over an existing IP network with little setup. It is currently described in an Internet-Draft. It adds the following perks to VLANs while still providing isolation:
  1. It uses a 24-bit VXLAN Network Identifier (VNI) which should be enough to address any scale-based concerns of multitenancy.
  2. It wraps L2 frames into UDP datagrams. This allows one to rely on some interesting properties of IP networks like availability and scalability. A VXLAN segment can be extended far beyond the typical reach of today VLANs.
The VXLAN Tunnel End Point (VTEP) originates and terminates VXLAN tunnels. Thanks to a serie of patches from Stephen Hemminger, Linux can now act as a VTEP. Let s see how this works.

About IPv6 When possible, I try to use IPv6 for my labs. This is not the case here for several reasons:
  1. IP multicast is required and PIM-SM implementations for IPv6 are not widespread yet. However, they exist. This explains why I use XORP for this lab: it supports PIM-SM for both IPv4 and IPv6.
  2. VXLAN Internet-Draft specifically addresses only IPv4. This seems a bit odd for a protocol running on top of UDP and I hope this will be fixed soon. This is not a major stopper since some VXLAN implementations support IPv6.
  3. However, the current implementation for Linux does not support IPv6. IPv6 support will be added later.
Once IPv6 support is available, the lab should be easy to adapt.

Lab So, here is the lab used. R1, R2 and R3 will act as VTEPs. They do not make use of PIM-SM. Instead, they have a generic multicast route on eth0. E1, E2 and E3 are edge routers while C1, C2 and C3 are core routers. The proposed lab is not resilient but convenient to explain how things work. It is built on top of KVM hosts. Have a look at my previous article for more details on this. VXLAN lab The lab is hosted on GitHub. I have made the lab easier to try by including the kernel I have used for my tests. XORP comes preconfigured, you just have to configure the VXLAN part. For this, you need a recent version of ip. 
$ sudo apt-get install screen vde2 kvm iproute xorp git
$ git clone git://git.kernel.org/pub/scm/linux/kernel/git/shemminger/iproute2.git
$ cd iproute2
$ ./configure && make
You get  ip' as  ip/ip' and  bridge' as  bridge/bridge'.
$ cd ..
$ git clone git://github.com/vincentbernat/network-lab.git
$ cd network-lab/lab-vxlan
$ ./setup

Unicast routing The first step is to setup unicast routing. OSPF is used for this purpose. The chosen routing daemon is XORP. With xorpsh, we can check if OSPF is working as expected: 
root@c1# xorpsh
root@c1$ show ospf4 neighbor   
  Address         Interface             State      ID              Pri  Dead
192.168.11.11    eth0/eth0              Full      3.0.0.1          128    36
192.168.12.22    eth1/eth1              Full      3.0.0.2          128    33
192.168.101.133  eth2/eth2              Full      2.0.0.3          128    36
192.168.102.122  eth3/eth3              Full      2.0.0.2          128    38
root@c1$ show route table ipv4 unicast ospf   
192.168.1.0/24  [ospf(110)/2]
                > to 192.168.11.11 via eth0/eth0
192.168.2.0/24  [ospf(110)/2]
                > to 192.168.12.22 via eth1/eth1
192.168.3.0/24  [ospf(110)/3]
                > to 192.168.102.122 via eth3/eth3
192.168.13.0/24 [ospf(110)/2]
                > to 192.168.102.122 via eth3/eth3
192.168.21.0/24 [ospf(110)/2]
                > to 192.168.101.133 via eth2/eth2
192.168.22.0/24 [ospf(110)/2]
                > to 192.168.12.22 via eth1/eth1
192.168.23.0/24 [ospf(110)/2]
                > to 192.168.101.133 via eth2/eth2
192.168.103.0/24        [ospf(110)/2]
                > to 192.168.102.122 via eth3/eth3

Multicast routing Once unicast routing is up and running, we need to setup multicast routing. There are two protocols for this: IGMP and PIM-SM. The former one allows routers to forward multicast datagrams while the first one allows hosts to subscribe to a multicast group.

IGMP IGMP is used by hosts and adjacent routers to establish multicast group membership. In our case, it will be used by R2 to let E2 know it subscribed to 239.0.0.11 (a multicast group). Configuring XORP to support IGMP is simple. Let s test with iperf to have a multicast listener on R2: 
root@r2# iperf -u -s -l 1000 -i 1 -B 239.0.0.11
------------------------------------------------------------
Server listening on UDP port 5001
Binding to local address 239.0.0.11
Joining multicast group  239.0.0.11
Receiving 1000 byte datagrams
UDP buffer size:  208 KByte (default)
------------------------------------------------------------
On E2, we can now check that R2 is properly registered for 239.0.0.11: 
root@e2$ show igmp group
Interface    Group           Source          LastReported Timeout V State
eth0         239.0.0.11      0.0.0.0         192.168.2.2      248 2     E
XORP documentation contains a good overview of IGMP.

PIM-SM PIM-SM is far more complex. It does not have its own topology discovery protocol and relies on routing information from other protocols, OSPF in our case. I will describe here a simplified view on how PIM-SM works. XORP documentation contains more details about PIM-SM. The first step for all PIM-SM routers is to elect a rendez-vous point (RP). In our lab, only C1, C2 and C3 have been configured to be elected as a RP. Moreover, we give better priority to C3 to ensure it wins. RP election 
root@e1$ show pim rps   
RP              Type      Pri Holdtime Timeout ActiveGroups GroupPrefix       
192.168.101.133 bootstrap 100      150     135            0 239.0.0.0/8
Let s suppose we start iperf on both R2 and R3. Using IGMP, they subscribe to multicast group 239.0.0.11 with E2 and E3 respectively. Then, E2 and E3 send a join message (also known as a (*,G) join) to the RP (C3) for that multicast group. Using the unicast path from E2 and E3 to the RP, the routers along the paths build the RP tree (RPT), rooted at C3. Each router in the tree knows how to send multicast packets to 239.0.0.11: it will send them to the leaves. RP tree 
root@e3$ show pim join   
Group           Source          RP              Flags
239.0.0.11      0.0.0.0         192.168.101.133 WC   
    Upstream interface (RP):   eth2
    Upstream MRIB next hop (RP): 192.168.23.133
    Upstream RPF'(*,G):        192.168.23.133
    Upstream state:            Joined 
    Join timer:                5
    Local receiver include WC: O...
    Joins RP:                  ....
    Joins WC:                  ....
    Join state:                ....
    Prune state:               ....
    Prune pending state:       ....
    I am assert winner state:  ....
    I am assert loser state:   ....
    Assert winner WC:          ....
    Assert lost WC:            ....
    Assert tracking WC:        O.O.
    Could assert WC:           O...
    I am DR:                   O..O
    Immediate olist RP:        ....
    Immediate olist WC:        O...
    Inherited olist SG:        O...
    Inherited olist SG_RPT:    O...
    PIM include WC:            O...
Let s suppose that R1 wants to send multicast packets to 239.0.0.11. It sends them to R1 which does not have any information on how to contact all the members of the multicast group because it is not the RP. Therefore, it encapsulates the multicast packets into PIM Register packets and sends them to the RP. The RP decapsulates them and sends them natively. The multicast packets are routed from the RP to R2 and R3 using the reverse path formed by the join messages. Multicast routing via the RP 
root@r1# iperf -c 239.0.0.11 -u -b 10k -t 30 -T 10
------------------------------------------------------------
Client connecting to 239.0.0.11, UDP port 5001
Sending 1470 byte datagrams
Setting multicast TTL to 10
UDP buffer size:  208 KByte (default)
------------------------------------------------------------
root@e1# tcpdump -pni eth0
10:58:23.424860 IP 192.168.1.1.35277 > 239.0.0.11.5001: UDP, length 1470
root@c3# tcpdump -pni eth0
10:58:23.552903 IP 192.168.11.11 > 192.168.101.133: PIMv2, Register, length 1480
root@e2# tcpdump -pni eth0
10:58:23.896171 IP 192.168.1.1.35277 > 239.0.0.11.5001: UDP, length 1470
root@e3# tcpdump -pni eth0
10:58:23.824647 IP 192.168.1.1.35277 > 239.0.0.11.5001: UDP, length 1470
As presented here, the routing is not optimal: packets from R1 to R2 could avoid the RP. Moreover, encapsulating multicast packets into unicast packets is not efficient either. At some point, the RP will decide to switch to native multicast1. Rooted at R1, the shortest-path tree (SPT) for the multicast group will be built using source-specific join messages (also known as a (S,G) join). Multicast routing without RP From here, each router in the tree knows how to handle multicast packets from R1 to the group without involving the RP. For example, E1 knows it must duplicate the packet and sends one through the interface to C3 and the other one through the interface to C1: 
root@e1$ show pim join   
Group           Source          RP              Flags
239.0.0.11      192.168.1.1     192.168.101.133 SG SPT DirectlyConnectedS 
    Upstream interface (S):    eth0
    Upstream interface (RP):   eth1
    Upstream MRIB next hop (RP): 192.168.11.111
    Upstream MRIB next hop (S):  UNKNOWN
    Upstream RPF'(S,G):        UNKNOWN
    Upstream state:            Joined 
    Register state:            RegisterPrune RegisterCouldRegister 
    Join timer:                7
    KAT(S,G) running:          true
    Local receiver include WC: ....
    Local receiver include SG: ....
    Local receiver exclude SG: ....
    Joins RP:                  ....
    Joins WC:                  ....
    Joins SG:                  .OO.
    Join state:                .OO.
    Prune state:               ....
    Prune pending state:       ....
    I am assert winner state:  ....
    I am assert loser state:   ....
    Assert winner WC:          ....
    Assert winner SG:          ....
    Assert lost WC:            ....
    Assert lost SG:            ....
    Assert lost SG_RPT:        ....
    Assert tracking SG:        OOO.
    Could assert WC:           ....
    Could assert SG:           .OO.
    I am DR:                   O..O
    Immediate olist RP:        ....
    Immediate olist WC:        ....
    Immediate olist SG:        .OO.
    Inherited olist SG:        .OO.
    Inherited olist SG_RPT:    ....
    PIM include WC:            ....
    PIM include SG:            ....
    PIM exclude SG:            ....
root@e1$ show pim mfc  
Group           Source          RP             
239.0.0.11      192.168.1.1     192.168.101.133
    Incoming interface :      eth0
    Outgoing interfaces:      .OO.
root@e1$ exit
[Connection to XORP closed]
root@e1# ip mroute
(192.168.1.1, 239.0.0.11)        Iif: eth0       Oifs: eth1 eth2

Setting up VXLAN Once IP multicast is running, setting up VXLAN is quite easy. Here are the software requirements:
  • A recent kernel. Pick at least 3.7-rc3. You need to enable CONFIG_VXLAN option. You also currently need a patch on top of it to be able to specify a TTL greater than 1 for multicast packets.
  • A recent version of ip. Currently, you need the version from git.
On R1, R2 and R3, we create a vxlan42 interface with the following commands: 
root@rX# ./ip link add vxlan42 type vxlan id 42 \
>                               group 239.0.0.42 \
>                               ttl 10 dev eth0
root@rX# ip link set up dev vxlan42
root@rX# ./ip -d link show vxlan42
10: vxlan42: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1460 qdisc noqueue state UNKNOWN mode DEFAULT 
link/ether 3e:09:1c:e1:09:2e brd ff:ff:ff:ff:ff:ff
vxlan id 42 group 239.0.0.42 dev eth0 port 32768 61000 ttl 10 ageing 300
Let s assign an IP in 192.168.99.0/24 for each router and check they can ping each other: 
root@r1# ip addr add 192.168.99.1/24 dev vxlan42
root@r2# ip addr add 192.168.99.2/24 dev vxlan42
root@r3# ip addr add 192.168.99.3/24 dev vxlan42
root@r1# ping 192.168.99.2                    
PING 192.168.99.2 (192.168.99.2) 56(84) bytes of data.
64 bytes from 192.168.99.2: icmp_req=1 ttl=64 time=3.90 ms
64 bytes from 192.168.99.2: icmp_req=2 ttl=64 time=1.38 ms
64 bytes from 192.168.99.2: icmp_req=3 ttl=64 time=1.82 ms
--- 192.168.99.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 1.389/2.375/3.907/1.098 ms
We can check the packets are encapsulated: 
root@r1# tcpdump -pni eth0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
11:30:36.561185 IP 192.168.1.1.43349 > 192.168.2.2.8472: UDP, length 106
11:30:36.563179 IP 192.168.2.2.33894 > 192.168.1.1.8472: UDP, length 106
11:30:37.562677 IP 192.168.1.1.43349 > 192.168.2.2.8472: UDP, length 106
11:30:37.564316 IP 192.168.2.2.33894 > 192.168.1.1.8472: UDP, length 106
Moreover, if we send broadcast packets (with ping -b or ARP requests), they are encapsulated into multicast packets: 
root@r1# tcpdump -pni eth0
11:31:27.464198 IP 192.168.1.1.41958 > 239.0.0.42.8472: UDP, length 106
11:31:28.463584 IP 192.168.1.1.41958 > 239.0.0.42.8472: UDP, length 106
Recent versions of iproute also comes with bridge, an utility allowing one to inspect the FDB of bridge-like interfaces: 
root@r1# ../bridge/bridge fdb show vxlan42
3e:09:1c:e1:09:2e dev vxlan42 dst 192.168.2.2 self 
0e:98:40:c6:58:10 dev vxlan42 dst 192.168.3.3 self

Demo For a demo, have a look at the following video (it is also available as an Ogg Theora video).
<iframe frameborder="0" height="270" src="http://www.dailymotion.com/embed/video/xusell" width="480"></iframe>
  1. The decision is usually done when the bandwidth used by the follow reachs some threshold. With XORP, this can be controlled with switch-to-spt-threshold. However, I was unable to make this works as expected. XORP never sends the appropriate PIM packets to make the switch. Therefore, for this lab, it has been configured to switch to native multicast at the first received packet.

Erich Schubert: DBSCAN and OPTICS clustering

DBSCAN [wikipedia] and OPTICS [wikipedia] are two of the most well-known density based clustering algorithms. You can read more on them in the Wikipedia articles linked above.
An interesting property of density based clustering is that these algorithms do not assume clusters to have a particular shape. Furthermore, the algorithms allow "noise" objects that do not belong to any of the clusters. K-means for examples partitions the data space in Voronoi cells (some people claim it produces spherical clusters - that is incorrect). See Wikipedia for the true shape of K-means clusters and an example that canot be clustered by K-means. Internal measures for cluster evaluation also usually assume the clusters to be well-separated spheres (and do not allow noise/outlier objects) - not surprisingly, as we tend to experiment with artificial data generated by a number of Gaussian distributions.
The key parameter to DBSCAN and OPTICS is the "minPts" parameter. It roughly controls the minimum size of a cluster. If you set it too low, everything will become clusters (OPTICS with minPts=2 degenerates to a type of single link clustering). If you set it too high, at some point there won't be any clusters anymore, only noise. However, the parameter usually is not hard to choose. If you for example expect clusters to typically have 100 objects, I'd start with a value of 10 or 20. If your clusters are expected to have 10000 objects, then maybe start experimenting with 500.
The more difficult parameter for DBSCAN is the radius. In some cases, it will be very obvious. Say you are clustering users on a map. Then you might know that a good radius is 1 km. Or 10 km. Whatever makes sense for your particular application. In other cases, the parameter will not be obvious, or you might need multiple values. That is when OPTICS comes into play.
OPTICS is based on a very clever idea: instead of fixing MinPts and the Radius, we only fix minpts, and plot the radius at which an object would be considered dense by DBSCAN. In order to sort the objects on this plot, we process them in a priority heap, so that nearby objects are nearby in the plot. this image on Wikipedia shows an example for such a plot.
OPTICS comes at a cost compared to DBSCAN. Largely because of the priority heap, but also as the nearest neighbor queries are more complicated than the radius queries of DBSCAN. So it will be slower, but you no longer need to set the parameter epsilon. However, OPTICS won't produce a strict partitioning. Primarily it produces this plot, and in many situations you will actually want to visually inspect the plot. There are some methods to extract a hierarchical partitioning out of this plot, based on detecting "steep" areas.
The open source ELKI data mining framework (package "elki" in Debian and Ubuntu) has a very fast and flexible implementation of both algorithms. I've benchmarked this against GNU R ("fpc" package") and Weka, and the difference is enormous. ELKI without index support runs in roughly 11 minutes, with index down to 2 minutes for DBSCAN and 3 minutes for OPTICS. Weka takes 11 hours and GNU R/fpc takes 100 minutes (DBSCAN, no OPTICS available). And the implementation of OPTICS in Weka is not even complete (it does not support proper cluster extraction from the plot). Many of the other OPTICS implementations you can find with Google (e.g. in Python or MATLAB) seem to be based on this Weka version ...
ELKI is open source. So if you want to peek at the code, here are direct links: DBSCAN.java, OPTICS.java.
Some part of the code may be a bit confusing at first. The "Parameterizer" classes serve the purpose of allowing automatic UI generation, for example. So there is quite a bit of meta code involved.
Plus, ELKI is quite extensively optimized. For example, it does not use Java Collections much anymore. Java Iterators, for example, require returning an object on next();. The C++ style iterators used by ELKI can have multiple values, and primitive values. 
for(DBIDIter id = relation.iterDBIDs(); id.valid(); id.advance())
is a typical for loop in ELKI, iterating over all objects of a relation, but the whole loop requires creating (and GC'ing) a single object. And actually, this is as literal as a for loop can get. 
ModifiableDBIDs processedIDs = DBIDUtil.newHashSet(size);
is another example. Essentially, this is like a HashSet<DBID>. Except that it is a lot faster, because the object IDs do not need to live a Java objects, but can internally be stored more efficiently (the only currently available implementation of the DBID layer uses primitive integers).
Java advocates always accuse you of premature optimization when you avoid creating objects for primitives. Yet, in all my benchmarking, I have seen this continuously to have a major impact how many objects you allocate. At least when it is inside a loop that is heavily used. Java collections with boxed primitives just eat a lot of memory, and the memory management overhead does often make a huge difference. Which is why libraries such as Trove (which ELKI uses a lot) exist. Because memory usage does make a difference.
(Avoiding boxing/unboxing systematically in ELKI yielded approximately a 4x speedup. But obviously, ELKI involves a lot of numerical computations.)

Next.

Previous.