We introduced Rcpp 0.10.0 with a number of very nice new features a few days ago, and the activity on the rcpp-devel mailing list has been pretty responsive which is awesome. But because few things beat a nice example, this post tries to build some more excitement. We will illustrate how Rcpp attributes makes it really easy to add C++ code to R session, and that that code is as easy to grasp as R code. Our motivating example is everybody's favourite introduction to Monte Carlo simulation: estimating . A common method uses the fact the unit circle has a surface area equal to . We draw two uniform random numbers x and y, each between zero and one. We then check for the distance of the corresponding point (x,y) relative to the origin. If less than one (or equal), it is in the circle (or on it); if more than one it is outside. As the first quadrant is a quarter of a square of area one, the area of the whole circle is -- so our first quadrant approximates over four. The following figure, kindly borrowed from Wikipedia with full attribution and credit, illustrates this: Now, a vectorized version (drawing N such pairs at once) of this approach is provided by the following R function.

piR <- function(N)  
    x <- runif(N)
    y <- runif(N)
    d <- sqrt(x^2 + y^2)
    return(4 * sum(d < 1.0) / N)
 

And in C++ we can write almost exactly the same function thanks the Rcpp sugar vectorisation available via Rcpp:

#include <Rcpp.h>
using namespace Rcpp;
// [[Rcpp::export]]
double piSugar(const int N)  
    RNGScope scope;		// ensure RNG gets set/reset
    NumericVector x = runif(N);
    NumericVector y = runif(N);
    NumericVector d = sqrt(x*x + y*y);
    return 4.0 * sum(d < 1.0) / N;
 

Sure, there are small differences: C++ is statically typed, R is not. We need one include file for declaration, and we need one instantiation of the RNGScope object to ensure random number draws remain coordinated between the calling R process and the C++ function calling into its (compiled C-code based) random number generators. That way we even get the exact same draws for the same seed. But the basic approach is identical: draw a vector x and vector y, compute the distance to the origin and then obtain the proportion within the unit circle -- which we scale by four. Same idea, same vectorised implementation in C++. But the real key here is the one short line with the [[Rcpp::export]] attribute. This is all it takes (along with sourceCpp() from Rcpp 0.10.0) to get the C++ code into R. The full example (which assumes the C++ file is saved as piSugar.cpp in the same directory) is now:

#!/usr/bin/r
library(Rcpp)
library(rbenchmark)
piR <- function(N)  
    x <- runif(N)
    y <- runif(N)
    d <- sqrt(x^2 + y^2)
    return(4 * sum(d < 1.0) / N)
 
sourceCpp("piSugar.cpp")
N <- 1e6
set.seed(42)
resR <- piR(N)
set.seed(42)
resCpp <- piSugar(N)
## important: check results are identical with RNG seeded
stopifnot(identical(resR, resCpp))
res <- benchmark(piR(N), piSugar(N), order="relative")
print(res[,1:4])

and it does a few things: set up the R function, source the C++ function (and presto: we have a callable C++ function just like that), compute two simulations given the same seed and ensure they are in fact identical -- and proceed to compare the timing in a benchmarking exercise. That last aspect is not even that important -- we end up being almost-but-not-quite twice as fast on my machine for different values of N. The real takeaway here is the ease with which we can get a C++ function into R --- and the new process completely takes care of passing parameters in, results out, and does the compilation, linking and loading. More details about Rcpp attributes are in the new vignette. Now enjoy the .

Update:One somewhat bad typo fixed.

Update:Corrected one background tag.

I m trenching a ditch. I m hoping to run a few 25-pair telco cables, some fiber, a few coax cables, 240v power, 120v power, natural gas and perhaps a few cat-6 cables through it. Assuming the authorities approve, of course. .

I recently wrote about tools to handle archives conveniently. If you just have to handle compressed text files, there are some widely known shortcut commands to mimic common commands on files compressed with a specific compression format. In Debian and derivatives, those tools are part of the according package for that compression utility, i.e. the zcat command is part of the gzip package and the xzfgrep command is part of the xz-utils package. But despite this matrix is quite easy to remember, the situation has a few drawbacks:

• Those tools can only handle the format they re written for (which btw. means that all xz-tools can also handle lzma-compressed files as lzma is xz s predecessor)
• zcat and the other cat variants can t even recognize non-compressed files and throw an error instead of just showing their contents.
• I always tend to think that lzcat and friends are for lzip-based compression as xzcat can handle lzma-compressed files anyway.

This is where the zutils project comes in: zutils provides the functionality of most of these utilities, too, but with one big difference: You don t have to remember, think about or type which compression method has been used for your data, just use zcat, zcmp, zdiff, zgrep, zegrep, or zfgrep and it works independently of what compression method has been used if any or if there are different compression types mixed in the parameters to the same command:

$ zfgrep foobar bla.txt fnord.gz hurz.xz quux.lz bar.lzma

Especially if you use logrotate and let logrotate compress old logs, it s very comfortable that one command suffices to concatenate all the available logfiles, including the current uncompressed one:

$ zcat /var/log/syslog*    

Additionally, zutils versions of these tools also support lzip-compressed files. The zutils package is available in Debian starting with Wheezy and in Ubuntu since Oneiric. When being installed, it replaces the original z* utilities from the gzip package by diverting them away. The only drawback so far is that there neither a zless nor a zmore utility from the zutils project, so zless bla.txt fnord.gz hurz.xz quux.lz bar.lzma will not work as expected even after installing zutils as it is still the one from the gzip package and hence it will show you just the first two files in plain text, but not the remaining ones.

In May this year, in a desperate bid to bail water out of a sinking ship, HMV group sold off the Waterstones chain of bookstores. I'm fond of reading so naturally I visit Waterstones from time to time. (but not exclusively.) For many years, their promotions were unanimously structured as "3 for 2" offers: find three books in the promotion, in-store, and get the cheapest of the three for free. I hated these promotions. I always found it very challenging to find three books in the promotion that I wanted. If I was visiting to buy a specific book and found it in the promotion, on more than one occasion I left the shop empty handed: I wasn't prepared to pay full price for a book in a promotion and not get the benefit of the promotion. Since being sold, Waterstones seem to have made a lot of changes. Many new hardbacks from popular authors have debuted at a discount for a short period (Rowling, Pratchett). I've found many more discounted books not tied to others in 3-for-2-style offers, sometimes window-display stock at half price, a month or so after release. They started a loyalty-card scheme, with a 10 gift card the prize you reach once you've spent 100. (I wonder if such schemes are examples of a token economy?) Finally, they've tentatively started to explore putting cafes in their branches. The one in Newcastle is best described as 'fledgling', but it's nice to have one more place to escape for a cup of tea at lunchtimes. The cafe, as well as a very well stocked collection of magazines, were the two killer features of the sadly- defunct Borders bookseller chain. For years before we got our own Borders in the North East, I used to enjoy the York branch if I was passing. I actually managed to fill a loyalty card this year, so clearly their change in attitude has worked on me! I'm generally not a fan of gift cards, especially in today's climate, so I quickly spent it. I hold an American Express credit card, which earns me cashbask on purchases which I make with it. I think the rate is about 1%. However, I can opt to receive the cashback in the form of gift vouchers. If I do so, I get them to the value of 110% my cashback balance. One of the retailers in on the scheme are Waterstones. I'm tempted

Github now provides native hook for Weblate. This means that all you need to do to link these two together is to enter URL of your Weblate installation and you are done. This was actually my first code in Ruby and I don't think I will start to like it, Python still looks nicer to me :-). However it was not that hard as I expected and I've even contribute (simple) testcase for this service.

Filed under: English Weblate 3 comments Flattr this!

The last week was quite productive as I was on vacation and at home town but sadly I couldn't complete this post so again merging the work with this week but this week ain't much productive as I was tired from journey back and didn't get enough time to recover. So here it goes. Debian Related

• I did a little more fine tuning of friendica package to enable http://localhost/friendica on installation and shipped virtual host configuration for Apache and Nginx which user can simply copy to setup a proper friendica instance. I'm not sure if this is right to do or not but Drupal package does this but only it doesn't enable it during installation and leaves it up to user to create required symlink to enable http://localhost/drupal7. Well this is my first time in packaging webapp so I'm all open for suggestions
• Helped Tobias Diekershoff prepare and upload the friendica package to mentors and file RFS bug. We got bug number #693504 and now we are looking for sponsors, if you are a DD reading this post and interested in federated social networking and can sponsor us it would be great :-)
• Started learning CDBS and converted the Kiwix package to CDBS. I'm glad that Jonas is helping me out when I'm struck some where. Also kiwix is now updated to latest version 0.9~rc2 and I need to do some more fine tuning before asking for upload from Jonas.
• I started helping Jonas to put up CDBS documentation. He suggested me to take over the documentation from duckcorp and then merge other documentation in CDBS git tree. As of now I only managed to separate out the chapters from docbook to markdown format. The conversion was done using pandoc but code formatting in docbook was not proper I guess so after conversion it needed some manual intervention in fixing and formatting code blocks. The repository for CDBS doc is here and as of now doesn't contain any new things.
• Converted one more of my font package fonts-gubbi to CDBS format but I'm now facing a issue that CDBS is including docs inside the generated udeb which is not right, I already posted the question on how to handle this to build-common-hackers and waiting for Jonas to reply back.
• I claimed the ITP #651600 on Roboto fonts from Android OS. After a short discussion with Fabian on pkg-fonts-devel I agreed to merge it with existing fonts-droid and renamed the source package to fonts-android which now builds fonts-droid and fonts-roboto. Package is ready but I'm still seeing if I can fix any bugs on fonts-droid before asking for a upload.
• Wrote a simple script which was derived from droid fonts package from Fedora to get the android fonts from the upstream repository automatically. Enabled this in get-orig-source target for fonts-android. Since cloning a more than 600M repository is PITA I have a local copy of fonts-android and use it to create new release of fonts-android. Additionally I should say script is not well tested ;-)
• suckless-tools has entered into Debian Experimental thanks to Jakub Wilk for a very nice review of the package which went in a long thread of more than 50 mails in debian-mentors. The package itself closed a shit load of bugs, but after upload I noticed there are still some pending bugs which I missed from adding to changelog. So I closed #606148 and #689199. Still there are 2 more bugs which seems like fixed in newer version but I need to confirm before closing them, thus making suckless-tools a bug free package.

Upstream Related I raised a pull request #8 on Gubbi fixing the Makefile to more organized and introducing xz compression in it. Additionally I removed distribution specific stuffs from Makefile and made it generic. That's all for the two weeks. This week there will be foss.in and we will be having some Debian specific mini conference, including some Debian basics to newbies and some bug squashing if any :-). So more to report next week, till then Cya.

Review: Inversions, by Iain M. Banks

Publisher:	Pocket
Copyright:	1998
Printing:	May 2001
ISBN:	0-7434-1196-X
Format:	Mass market
Pages:	376

Inversions poses a special challenge for book metadata for reviews that try to avoid spoilers. If you've looked at my index of reviews, I may have arguably already spoiled you. I'm going to discuss the book with that spoiler, as most reviews do, because I think it's a better book knowing that information up-front. But if you want to avoid any spoilers, stop reading here, and avoid looking at its placement in my review index. Inversions consists of the interleaved stories of two people: a strange female doctor to the king of Haspide, and the bodyguard of the Protector of Tassassen. The doctor's story is told in the form of letters written by the doctor's apprentice to his "Master," who has clearly told him to spy on the doctor. (The identity of this person is revealed during the story.) The bodyguard's story is told in the third person and presented as a reconstruction of events. The stories never meet, although the two countries have some indirect political exchanges. The doctor becomes entangled in court intrigue and several murders, and the bodyguard protects the Protector (a general in charge of what was effectively a military coup against the former regime) during a war with neighboring barons who had initially supported him. Both stories reach conclusions, but somewhat unsatisfactory ones. The two characters never meet in the book. All of this is set among countries at a medieval level of technology. The doctor faces scorn and derision, not to mention outright opposition, both because she's a woman (she's only permitted to claim the title of doctor due to the king's favor and her claimed citizenship in a far-away land with strange customs) and because she uses more effective and more modern techniques. Other physicians still bleed patients, but the doctor is clearly aware of the germ theory of disease, dismisses contageon via bad odors, and otherwise strikes the reader as someone with advanced medical knowledge trying to practice it in primitive conditions. The spoiler, which is never stated outright but which is relatively apparent from various hints and allusions dropped by the characters, as well as at least one event that would otherwise have to be supernatural, is that Inversions actually takes place in the Culture milieu. How this planet and these stories fit into that milieu remains unexplained throughout the story, but we do get some clues in supposedly made-up stories that the bodyguard tells the son of the Protector, stories that are recognizable to Culture readers as less fictional than they sound in a medieval context. Banks has a tendency to play with unusual narrative structure and to fall on the mainstream side of SF plot construction. Inversions is a very closely-observed pair of entwined stories told in a way that downplays most of the dramatic structure and rising action and that undermines feelings of conclusion or catharsis. Despite having, on later reflection, two clear plots with clear conclusions, it feels more like parallel slices of life and stubbornly refuses to advertise any conclusions or deeper meaning. Whether that works will probably have a lot to do with the reader's tolerance for trying to draw one's own conclusions. I found it frustrating and unsatisfying, but knowing going in that it's a Culture novel helped by adding some deeper structure to give the story a modicum of meaning. Insofar as there's a larger theme, it's the exploration of the conflict between the two Culture citizens that is presented, thinly disguised, in the bodyguard's stories. This connection is not obvious; even determining how to map the two characters into the stories is a touch challenging. But if one is aware of it and looking for it in advance, it provides a bit of broader structure and some basis for judging the actions of the doctor and the bodyguard against a larger plan. I needed that to wring enjoyment from the book; I never cared enough about the characters otherwise to fully engage with either story. With the puzzle, and the constant search for connections to Culture concepts, it's somewhat more interesting, but it's still maddeningly indirect. (And, from various other Internet reactions, I don't think I'm missing something obvious.) It's possible that I went into the book with some incorrect assumptions that undermined my enjoyment. Hopefully this review will help other readers. Knowing that this is a Culture book and enjoying the process of finding references makes it more fun (and most of the references aren't too hard to find). But don't expect lots of closure from the stories, and don't expend to find out "what's really going on." It's not that sort of book. If you enjoy the individual stories for themselves, you'll have the best chance of enjoying the book as a whole, with the Culture references providing a fun additional layer. If the basic stories don't appeal, the Culture material probably isn't enough to make the book engrossing. Rating: 6 out of 10

As some may have noticed, Google launched something that is unusual for the company: A game. A mobile alternate reality game, to be precise. See allthingsd article for a bit more information. In my opinion this game is most interesting to people who are mildly interested in geo-caching. You spend most of the gaming time either looking for the right portals (geo-location that is an interesting place for one reason or another, many located at sculptures, murals or (more scarcely) interesting local businesses), or trying to solve the puzzles that appear in the game itself or on http://www.nianticproject.com/. I was one of the beta testers and really enjoy the game, especially in areas with more players. As the game is most fun when people coordinate a bit, I started a google group for the munich area for one of the two teams (the enlightenment), but I will only let people enter that group who can be verified to be on that team. On a side note: I still have a few invites to the game left, so if you are interested, please leave a comment and enter your gmail.com email address (you need a Google account to play anyway) while doing so. I will send out invites in the order I received the requests (which I admit might not always be the same in which they were submitted. First come first serve and no guarantee to get an invitation though. Your email address you enter when commenting isn t shown to anyone and I don t share them with anyone but the game (and thus Google). EDIT: So short time went past, but I already ran out of invites (note that I didn t approve comments that really only were like please send me an invite , but still did sent one out). Feel free to still ask me for an invite, I will send some out as I get new ones, but this might take anything between a few days and some weeks. PS: For the lucky ones who got invites: You don t strictly need to do the training offered when you start the game for the first time, but I strongly encourage you to do so.
PPS: I m opening my group to any enlightened players in Germany. I will share some useful information about some game details there in the near future (nothing company confidential of course, just information anyone can gather in the game or from the published information). EDIT: As noted before, I don t have any more invites to give away. As I still get like 4 requests for invites per day, and now have more than 30 unfulfilled requests pending (with no new invites to give in sight), I m now disabling comments. Disclaimer: I work for Google since a few months ago, but this post solely represents my own opinions and hasn t been endorsed by Google in any way.

Powell's was holding an anniversary sale and giving out free gift cards with a qualifying order, so it seemed like a good idea to pick up a few books. Ben Aaronovitch Midnight Riot (sff)
Patricia Briggs Alpha & Omega (sff)
Patricia Briggs Moon Called (sff)
Mark Forster Get Everything Done (non-fiction)
Lynn Galli Wasted Heart (romance)
Sam Harris Lying (non-fiction)
K.E. Lane And Playing the Role of Herself (romance)
Jack McDevitt Firebird (sff)
Marshall Rosenberg Nonviolent Communication (non-fiction)
Deborah Tannen The Argument Culture (non-fiction)
Bill Willingham, et al. Fables: Animal Farm (graphic novel) I'm not sure if the Forster will be worth the cost, but I've really liked his other writing about time management, so I thought I'd take a chance on it. I think the Tannen and the Rosenberg are on roughly the same topic, but I arrived at them through different recommendation paths. I've been feeling like reading about how to communicate lately. The McDevitt novel is the latest in the Alex Benedict series. I'm only a couple of books away from being caught up. I've gotten lots of positive recommendations for Aaronovitch's series, so thought I'd get the first one. There's also a variety of random stuff, mostly on the Kindle, that I'd not added to one of these posts, noted here mostly for my own records. Alpha & Omega is a novella rather than a novel, which I grabbed because I picked up and read Cry Wolf during vacation.

it's seems we're starting to run out of RC bugs with the implicit usertag easy-enough-for-gregoa

what I've done in the last days besides working on the bugs listed below is to tag quite a few bugs pending that were uploaded into a DELAYED queue. please try to remember to set this tag on delayed NMUs, nmudiff is quite convenient here.

• #666334 src:gnustep-back: "FTBFS: build-arch problems with patch applying and installation of files generated only in build-indep target"
  fix debian/rules, upload to DELAYED/2
• #682416 smstools: "smstools: smsd -s produces Segmentation fault"
  backport fix from latest upstream release, upload to DELAYED/2
• #689249 mediawiki-extensions-math: "mediawiki-extensions-math: installs files in /usr/share/mediawiki/extensions which is a symlink in the mediawiki package"
  send a patch to the BTS
• #691570 src:doc-linux-fr: "Build-depends on mirror, scheduled to be removed"
  sponsor QA upload by Ivo De Decker
• #692890 fbdesk: "fbdesk: no icons on desktop"
  tag unreproducible, lower severity
• #693002 libimager-perl: "libimager-perl: breaks libimager-qrcode-perl"
  bug triaging (pkg-perl)

I've played a bit with Project Euler but all of their problems are math-centric, which is nice but not exactly what I'm looking for: some real-world programming problems to get back into the coding field.

So asking my friend Google, I found CodingBat: it has a Python section with several tasks to complete. I must say they are some kinda trivial to solve, once you know some idiomatic Python code, but some are a bit more interesting. If you're a junior Python coder, or want to get a grip on the language, give it a look.

Oh, and if you know some website that would give me some real-world programming coding problems (something that would be useful on the job, not just coding for fun), I would love to hear you.

After prompts from Wookey and Steve McIntyre, I decided to look at #285559 and #633884 for perl cross-build support and then port that support forward to the current perl in Wheezy and on to the version of perl currently in experimental. The first patch is for perl 5.8, the second for perl 5.12, neither of which is available currently in Debian. snapshot.debian.org provided the 5.12 source but then that no longer cross-builds with the patch.

The problem, as with any cross build, is that the build must avoid trying to execute binaries compiled within the build to achieve the test results required by ./configure (or in the case of perl, Configure). dpkg-cross has one collection of cache values but the intention was always to migrate the package-specific support data into the packages themselves and keep the architecture-specific data in dpkg-cross or dpkg-dev. Therefore, the approach taken in #633884 would be correct, if only there was a way of ensuring that the cached values remain in sync with the relevant Debian package.

I'll note here that I am aware of other ways of cross-building perl, this is particularly concerned with cross-building the Debian configuration of perl as a Debian package and using Debian or Emdebian cross-compilers. After all, the objective is to support bootstrapping Debian onto new architectures. However, I fully expect this to be just as usable with Ubuntu packages of perl compiled with, e.g. Linaro cross-compilers but I haven't yet looked at the differences between perl in Debian vs Ubuntu in any detail.

I've just got perl 5.14.2 cross-building for armel using the Emdebian gcc-4.4 cross-compiler (4.4.5-8) on a Debian sid amd64 machine without errors (it needs testing, which I'll look at later), so now is the time to document how it is done and what needs to be fixed. I've already discussed part of this with the current perl maintainers in Debian and, subject to just how the update mechanism works, have outline approval for pushing these changes into the Debian package and working with upstream where appropriate. The cache data itself might live in a separate source package which will use a strict dependency on perl to ensure that it remains in sync with the version which it can cross-build. Alternatively, if I can correctly partition the cache data between architecture-specific (and therefore generated from the existing files) and package_$version specific, then it may be possible to push a much smaller patch into the Debian perl package. This would start with some common data, calculate the arch-specific data and then look for some version-specific data, gleaned from Debian porter boxes whilst the version is in Debian experimental.

The key point is that I've offered to provide this support for the long term, ensuring that we don't end up with future stable releases of Debian having a perl package which cannot be cross-built. (To achieve that, we will also end up with versions of perl in Debian testing which also cross-build.)

This cross-build is still using dpkg-cross paths, not MultiArch paths, and this will need to be changed eventually. (e.g. by the source package providing two binaries, one which uses MultiArch and one which expects dpkg-cross paths.) The changes include patches for the upstream Makefile.SH, debian/rules and the cache data itself. Depending on where the cache data finally lives, the new support might or might not use the upstream Cross/ directory as the current contents date from the Zaurus support and don't appear to be that useful for current versions of perl.

The cache data itself has several problems:

1. It is tightly tied to the version of perl which generated it.


2. It is, as expected, architecture-dependent


3. It is, unfortunately, very sensitive to the specific configuration used by the distribution itself

That last point is important because it means that the cache data is not useful upstream as a block. It also means that generating the cache data for a specific Debian package means running the generation code on the native architecture with all of the Debian build-dependencies installed for the full perl build. This is going to complicate the use of this method for new architectures like arm64.

My objective for the long term maintenance of this code is to create sufficient data that a new architecture can be bootstrapped by judicious use of some form of template. Quite how that works out, only time will tell. I expect that this will involve isolating the data which is truly architecture-specific which doesn't change between perl versions from the data which is related to the tests for build-dependencies which does change between perl versions and then work out how to deal with any remainder. A new architecture for a specific perl version should then just be a case of populating the arch-specific data such as the size of a pointer/char and the format specifiers for long long etc. alongside the existing (and correct) data for the current version of perl.

Generating the cache data natively

The perl build repeats twice (three builds in total) and each build provides and requires slightly different cache data - static, debug and shared. Therefore, the maintenance code will need to provide a script which can run the correct configuration step for each mode, copy out the cache data for each one and clean up. The script will need to run inside a buildd chroot on a porter box (I'm looking at using abel.debian.org and harris.debian.org for this work so far) so that the derived data matches what the corresponding Debian native build would use. The data then needs slight modification - typically to replace the absolute paths with PERL_BUILD_DIR. It may also be necessary to change the value of cc, ranlib and other compiler-related values to the relevant cross-compiler executables. That should be possible to arrange within the build of the cache data support package itself, allowing new cache files to be dropped in directly from the porter box.

The configuration step may need to be optimised within debian/rules of perl itself as it currently proceeds on from the bare configuration to do some actual building but I need to compare the data to see if a bare config is modified later. The test step can be omitted already. Each step is performed as:

DEB_BUILD_OPTIONS="nocheck" fakeroot debian/rules perl.static 

That is repeated for perl.debug and libperl.so.$(VERSION) where $VERSION comes from :

/bin/bash debian/config.debian --full-version

The files to be copied out are:

• bitcount.h - architecture-specific (may be identical between similar architectures like armel & armhf)


• config.h - architecture & build specific, config.h.debug, config.h.static, config.h.shared.


• config.sh - architecture & build specific config.sh.debug, config.sh.static, config.sh.shared.


• uudmap.h - architecture-specific (may be identical between similar architectures like armel & armhf)

There is a lot of scope for templating of some form here, e.g. config.h.debug is 4,686 lines long but only 41 of those lines differ between amd64 and armhf for the same version of perl (and some of those can be identified from existing architecture-specific constants) which should make for a much smaller patch.

Architecture-specific cache data for perl

So far, the existing patches only deal with armel and armhf. If I compare the differences between armel & armhf, it comes down to:

1. compiler names (config.h.*)


2. architecture names (config.sh.*)


3. architecture-based paths (config.sh.*)

However, comparing armel and armhf doesn't provide sufficient info for deriving arm64 or mips etc. Comparing the same versions for armhf and amd64 shows the range of differences more clearly. Typical architecture differences exist like the size of a long, flags to denote if the compiler can cast negative floats to 32bit ints and the sprintf format specifier strings for handling floats and doubles. The data also includes some less expected ones like:

armhf: #define Netdb_host_t		const void * /**/

amd64: #define Netdb_host_t		char * /**/

I'm not at all sure why that is arch-specific - if anyone knows, email codehelp @ d.o - same address if anyone fancies helping out ....

Cross-builds and debclean

When playing with the cross-build, remember to use the cross-build clean support, not just debclean:

dpkg-architecture -aarmel -c fakeroot debian/rules clean

That wasted quite a bit of my time initially with having to blow away the entire tree, unpack it from original apt sources and repatch it. (Once Wheezy is out, may actually investigate getting debclean to support the -a switch).

OK, that's an introduction, I'm planning on pushing the cross-build support code onto github soon-ish and doing some testing of the cross-built perl binaries in a chroot on an armel box. I'll detail that in another blog post when it's available.

Next step is to look at perl 5.16 and then current perl upstream git to see how to get Makefile.SH fixed for the long term.

Another decent article on online privacy was published by the NYT this morning. The article documents the collection and centralized aggregation of consumer data online. It ends with a strong message about consumers becoming the "product" - which echoes what Noam Chomsky and Ben Bagdikian have been saying for decades. The problem with this critique is that it doesn't go far enough. Chomsky and Bagdikian didn't say readers are the product of modern journalism and that's bad because nobody wants to be a product. They critiqued the model because it produces terrible journalism and democracy suffers as a result. Similarly, the online trading of audiences and the invasion of our privacy that results has more important implications than "don't be a sucker" which seems to be the gist of the privacy argument these days. Honestly, I don't stay awake at night worrying that Nike knows I'm looking for a pair of sneakers, or even that they know my income. I do stay awake at night worrying that people are tagging my photo on Facebook, which could allow the New York Police Dept to submit a photo of protesters to Facebook and get a list of names and addresses of the people in the photo. Or it could allow the police to track my movements via existing networks of surveillance cameras by matching my image to my name. Would that require a search warrant? How would that impact my trust in my government to know that my movements are being tracked? Or worse, to know they might be tracked but I'll never know if they are or aren't? Many governments (including the US) have sordid histories of infiltrating political organizations and intentionally sowing dissent. Having access to activists' purchasing habits provides a treasure trove of material for this purpose that previously was labor intensive for cops to collect. It also provides leverage during interrogation, a time of heightened emotional vulnerability where the mere mention of a private detail of one's life can have a powerful impact. How can we prevent law enforcement from having access to these details of our lives? And then there's the dragnet. A central database of consumer habits provides a tantalizing collection of information that could be searched for people who fit a profile of a crime. Is solving the crime at hand worth the invasion of privacy that results from people being investigating only based on their consumer habits? How would we measure the impact an investigation like this would have on our confidence in democracy and government? The corporate advertisers promoting this kind of data collection argue that these problems are the price of progress. However, the ideology behind these technology developments is capitalism, not some intrinsic aspect of the Internet or network communications. Historically, the Internet has developed based on a drive toward de-centralization. Email and web sites, the bedrock of Internet activity, are based on these principles: anyone can add their own email server or web server and everyone can seamlessly access it. In contrast, the logic of capitalism as it relates to the Internet, is one of centralization and aggregating data. Facebook and Google want you to use their platforms for all your online activities so they have as complete a picture as possible of what you like and do. As the New York Times article describes, companies like Rubicon want to track all your movements and aggregate that data for profit. We have a choice. We can carefully evaluate our use of corporate services and consider the implications it has not only on our lives but on society as a whole. And, we can choose to use and support services that promote open standards and interoperability that will respect our personal privacy values. Want a practical example? Friendica is software designed to allow you to post status alerts, photos, videos and other information about yourself in a way similar to Facebook. However, it has one big difference: you can host your account on any server running the software anywhere in the world. You can still "friend" people on different servers, but we don't have to all trust the same organization to host it. Furthermore, Friendica allows you to post once and automatically cross-post to your Twitter, Facebook or other similar services. Sound good? You are welcome to try out an account on the May First/People Link install: https://friends.mayfirst.org/.

As Weblate 1.3 is out, it's time to think about features for next release. My current plan is to go more into two areas. First is improving support for machine translations. I have some ideas how to integrate these in interesting ways, I just hope my (poor) Javascript knowledge will not prevent me doing it that way. The second area is improving testsuite. Right now it really does not cover much and I'd like to cover at least basic functionality with different file formats. I'll see how much I will be able to cover, but at least I will try to cover as much as possible :-).

Filed under: English Weblate 0 comments Flattr this!

Diego s post got me to switch from grub-pc to grub-efi to boot Debian on my Macbook Pro. But I wanted to go further: getting rid of rEFIt. rEFIt is a pretty useful piece of software, but it s essentially dead. There is the rEFInd fork, which keeps it up-to-date, but it doesn t really help with FileVault. Moreover, the boot sequence for a Linux distro with rEFIt/rEFInd looks like: Apple EFI firmware rEFIt/rEFInd GRUB Linux kernel. Each intermediate step adding its own timeout, so rEFIt/rEFInd can be seen as not-so-useful intermediate step. Thankfully, Matthew Garrett did all the research to allow to directly boot GRUB from the Apple EFI firmware. Unfortunately, his blog post didn t have much actual detail on how to do it. So here it is, for a Debian system:

• Install a few packages you ll need in this process:
  

  # apt-get install hfsprogs icnsutils

• Create a small HFS+ partition. I have a 9MB one, but it s only filled by about 500K, so even smaller should work too. If, like me, you were previously using grub-pc, you probably have a GRUB partition, you can repurpose it. In gdisk, it looks like this:
  

  Number  Start (sector)    End (sector)  Size       Code  Name
     5       235284480       235302943   9.0 MiB     AF00  Apple HFS/HFS+
  Partition GUID code: 48465300-0000-11AA-AA11-00306543ECAC (Apple HFS/HFS+)
  Partition unique GUID: AD1F5465-B777-4178-AC4D-1DE8B2EB1B4B
  First sector: 235284480 (at 112.2 GiB)
  Last sector: 235302943 (at 112.2 GiB)
  Partition size: 18464 sectors (9.0 MiB)
  Attribute flags: 0000000000000000
  Partition name: 'Apple HFS/HFS+'
  

• Create a HFS+ filesystem on that partition:
  

  # mkfs.hfsplus /dev/sda5 -v Debian

  (replace /dev/sda5 with whatever your partition is)
• Add a fstab entry for that filesystem:
  

  # echo $(blkid -o export -s UUID /dev/sda5) /boot/efi auto defaults 0 0 >> /etc/fstab

• Mount the filesystem:
  

  # mkdir /boot/efi
  # mount /boot/efi
  

• Edit /usr/sbin/grub-install, look for xfat , and remove the block of code that looks like:
  

  if test "x$efi_fs" = xfat; then :; else
      echo "$ efidir  doesn't look like an EFI partition." 1>&2
      efidir=
  fi
  

• Run grub-install. At this point, there should be a /boot/efi/EFI/debian/grubx64.efi file (if using grub-efi-amd64).
• Create a /boot/efi/System/Library/CoreServices directory:
  

  # mkdir -p /boot/efi/System/Library/CoreServices

• Create a hard link:
  

  # ln /boot/efi/EFI/debian/grubx64.efi /boot/efi/System/Library/CoreServices/boot.efi

• Create a dummy mach_kernel file:
  

  # echo "This file is required for booting" > /boot/efi/mach_kernel

• Grab the mactel-boot source code, unpack and build it:
  

  # wget http://www.codon.org.uk/~mjg59/mactel-boot/mactel-boot-0.9.tar.bz2
  # tar -jxf mactel-boot-0.9.tar.bz2
  # cd mactel-boot-0.9
  # make PRODUCTVERSION=Debian
  

• Copy the SystemVersion.plist file:
  

  # cp SystemVersion.plist /boot/efi/System/Library/CoreServices/

• Bless the boot file:
  

  # ./hfs-bless /boot/efi/System/Library/CoreServices/boot.efi

• (optional) Add an icon:
  

  # rsvg-convert -w 128 -h 128 -o /tmp/debian.png /usr/share/reportbug/debian-swirl.svg
  # png2icns /boot/efi/.VolumeIcon.icns /tmp/debian.png
  # rm /tmp/debian.png
  

Now, the Apple Boot Manager, shown when holding down the option key when booting the Macbook Pro, looks like this:
And the Startup disk preferences dialog under OSX, like this:

Review: A Hero's Tale, by Catherine M. Wilson

Series:	Women Were Warriors #3
Publisher:	Shield Maiden
Copyright:	2008
ISBN:	0-9815636-3-5
Format:	Kindle
Pages:	301

This is an ebook, so metadata may be inaccurate or missing. See notes on ebooks for more information. A Hero's Tale is the third book in the When Women Were Warriors trilogy, which is best thought of as a single long novel. Reading it independently will miss most of the significance of the plot, since one wouldn't know many of the important characters. A Hero's Tale follows the pattern of the second book: even less quiet contemplation, and even more in the way of events and plot. Tamras completes her training, in a somewhat unexpected way (and in a rather interesting reworking of gender roles), but then both Maara's past and the problems of A Journey of the Heart catch up with them. There is, accordingly, quite a bit more action, or at least war councils and political maneuvering, than the previous books, and somewhat less time for thoughtful wisdom. That also means that A Hero's Tale shifts a bit more towards the traditional fantasy coming-of-age story, where the heroine is able to shift armies and win conflicts with an understated and self-deprecating skill that comes inevitably but indirectly out of all of her previous experiences. I think Wilson is still fairly original in how she presents those tropes, with an interesting focus on empathy and compassion that is all too rare in this sort of story, but this book does feel a lot more like a story that one has read before. If you're particularly allergic to that story shape, you may find this entry somewhat disappointing. One of the things I loved about it, though, was Wilson's avoidance of traditional villains. There is one, so it's not a complete avoidance, but nearly everyone in this story has their own viewpoint and perspective and is on the side of right in their own way. Wilson uses Tamras's empathy and willingness to listen and understand to let the reader see various different perspectives without moving away from the first-person perspective. It's very nicely done. I thought the handling of the emotional arc of the story was more mixed. Tamras is seriously tested in some moments that aimed for heartbreaking, but I thought they fell short of the emotional punch they were supposed to carry since the story shape had already made it obvious that everything would turn out okay in the end. The story is so gentle that Wilson hadn't established any willingness to end something badly, which robs a lot of the punch from emotional turmoil. I also had quite a bit of trouble with the principle of love above all else, particularly for someone in Tamras's position. Maybe that's because I'm still too entrenched in the attitude towards politics that this series argues against, but I had a hard time not seeing her as personally indulgent. Both of those factors blunted the emotional climax for me, leaving it moving but not as heartbreaking as I think Wilson was aiming for. (This is the point where a friend will tell me that I think too much about books while I'm reading them.) That said, Wilson continues to write one of the best mentoring relationships that I've seen in fiction. The dynamic between Tamras and Maara is just beautiful, and remains so through the end of the book. So many traditional issues with rivalry, growth, replacement of the mentor, and relative social position that are ignored or dealt with ham-handedly in other stories are handled adeptly and quietly. Maara is by far my favorite character of this series, largely because of how she handles that relationship. I could identify with her more easily than with Tamras at various points in the story, and I thought the understated ending was quietly beautiful. If you enjoyed the rest of the series, definitely read A Hero's Tale for the conclusion. If you've been hungry for more typical action, you'll get some of it here. I wish a few deep emotions other than love had been in play, but, despite that, When Women Were Warriors is still quietly and delightfully different than most things I've read, and worth the price. Rating: 8 out of 10

OK, I m finished my experiment. It s over now and I can break character . First, I have to offer special thanks to tshirtman for being the first to unambiguously exemplify one of my main points. well done! In case it s not blindingly obvious (as it should be), the reason for my post was that I was outraged by the spectacle of one fairly high-profile member of the linux community trying to rally support to shun and exclude another fairly high-profile member because a nightmare had upset her. WTF? Is that really all it s going to take to destroy someone s reputation and perhaps their career? even with the shunning target s own words available and archived to disprove the ridiculous straw-man mis-characterisations of what he actually said? Not one of the arguments against him actually addressed anything he said, they ALL attacked him for things he didn t say, for things that other people claimed he said. I was further outraged by seeing everyone who even suggested that questioning of stats (or, indeed, ANY claim of fact or evidence) may be, in some small way, a valid and reasonable thing to do get instantly put in their place and dismissed as Yet Another Rape Apologist. Are we supposed to be anti-science, anti-scientific method now? or are rape stats a special case like religion where we are just supposed to switch off our analytical brains and accept what we are told on faith, without question? Surely we are capable of better than that? i know we are capable of better than that. Or, at least, i used to know that. Now i m not so sure. In 2011, all it took was Ted Tso ( TT ) making some fairly reasonable statements about the need for any claimed evidence or statistics to be viewed skeptically and that dissenting research should also be considered and he was instantly vilified as a rape apologist . Sorry, but questioning extremely dodgy stats (that even in feminist circles are viewed more as ideological propaganda than as serious research) is NOWHERE NEAR SUFFICIENT to earn the label of rape-apologist. That is not how debate works you can t just refuse to engage with someone s point and simply accuse them of being the enemy for not agreeing 100% with whatever you say at least, not if you have any intellectual honesty or self-respect. (sure, some people are complete jerks and deserve to be told to FOAD in no uncertain terms but a) jerks like that are self-evident and obvious, and b) TT s participation in that thread was at all times civil and reasonable) But that thread is ancient history it was over and done with nearly two years ago.

---

In October this year, for reasons which are not at all clear, Valerie Aurora ( VA ) decided to revive the issue (which had been resolved back in 2011 with a resounding fuck no, we don t want misogynist shit or porn in our conferences from pretty much the entire linux community including near-universal support for improved anti-harassment policies both for linux.conf.au and for geek conferences in general) and use it to attack TT. And she did so by twisting his words and claiming in a post on The Ada Initiative blog that he said something which he didn t, that rape was impossible if both people were drunk enough . If he chose, he could quite easily win a libel case against her and TAI on that. It s not what he said, it s not even close to what he said, and VA is clearly too smart to honestly believe that it is what he said. In another post on her personal, blog she talks about how what he said was so terrible that it even now gives her nightmares, and that she can t bear the thought of working with him. Again, WTF? VA can say I had nightmares and was upset and furious and THAT is enough to justify a call to shun TT?? He didn t attack her, or threaten her (explicitly or implicitly), he was polite and civil. What he did was disagree with VA by referring to other research that disputed VA s preferred studies. I agree with and support many (perhaps most) of VA s and The Ada Initiative s aims, I certainly believe that linux and open source etc should be very welcoming and supportive of human diversity (including gender and sexuality, identity, religion, politics and so on), believe that it s a good thing that The Ada Initiative exists as part of that diversity welcome to be particularly supportive of women in geekdom. And i wholeheartedly agree that Linux leaders should not make public statements belittling and condoning rape BUT: a) I haven t seen one instance of that happening, ever and b) I find VA s choice of tactics here to be despicable. as i do when anyone else uses similar tactics, because they ARE despicable tactics. They are exactly the same as accusing someone of being a child pornographer for being against net censorship: You dared to disagree with me so I m going to accuse you of being a monster. (and, i must admit, the enthusiasm level of my support for The Ada Initiative is somewhat .diminished by this tactical blunder by the spokesperson and co-founder) There are far more deserving targets of VA s ire than TT. And there are far better ways for the Ada Initiative to achieve their aims.

---

Why? So why did I decide to comment when I knew that I was inevitably going to be accused of hating women and being a rape apologist? Mostly because I thought it would be gutless of me not to. Hardly anyone else had, and they quickly backed down under the accusations of misogyny and since I consider myself to be psychologically fairly strong, I felt that I am capable of wearing a little shit (or even a lot) for a while. In my egocentric fashion, I thought if I can t do then it s no wonder that no one else dares . And also because anyone who cared to make even the slightest effort to find out what my actual views on sexual harassment, rape, women s rights and numerous inter-related issues are can fairly easily see a very consistent record of the kinds of things I argue for and against, and my scathing responses to actual misogynists when they appear on lists that I participate in. they are not my kind of people. and even then, i hesitated. it s scary and intimidating to be putting yourself forward to be accused of being one of the things you hate. this is, of course, an instance of the chilling effect. So, I found the prospect scary, almost terrifying .and I can t think of a single person who has met me online or in real life who would even remotely describe me as being any kind of delicate or sensitive wall-flower. again, if i can t do it, it s no wonder no one else dares . so i clicked the Publish button. an ego is useful for some things. Also, I took VA s words but don t be silent as inspiration. In the process, i discovered why it is that some people just simply refuse to engage in rational discourse. I ve seen it many times from the other side, but i ve never experienced the seductive pleasure of indulging in it myself before there s a liberating freedom in just ignoring any and every point that someone makes and simply accuse them of being the enemy. You don t have to try to understand what they wrote, hell you don t even have to really read it you just need to quickly scan it for overall tone and if they don t seem like they re 100% supportive, you just accuse them of being the enemy or an apologist for the enemy. it s that fucking simple and easy. well, sort of easy. easy for some, perhaps. i personally found it extremely difficult a struggle to refrain from engaging, to remain in character (i m not much of an actor). especially when i kind of agreed with whoever was arguing against my experimental character or if i thought they made a good point. and even more so when i thought that some comments skirted a bit too close to being the kind of misogynist crap that i didn t want to tolerate having on MY blog. (i resolved that issue by just approving any reply that didn t squick me or that i could squint at and think hmmm borderline, give benefit of doubt ) but even though i don t like it, i can recognise the attraction it holds for some people.

---

Other thoughts: I m particularly disgusted by the men who intervene way too early without an explicit invitation or request for help or a clear need such as an immediate threat of violence in womens issues. Many or maybe even most may not realise it, but they are just taking over and asserting male strength and control by protecting women rather than giving them the support and space to discover and practice their own strength and their own voices. These uninvited interventions do not help women, they weaken and undermine them, they perpetuate dependence, they steal strength from the movement. It is patronising and enfeebling. But mostly, they just re-assert male dominance and are an attempt to make women s spaces more comfortable, more palatable, for men.(*) (it s also quite often very transparent self-serving and ingratiating behaviour from blokes who want to lay the groundwork for perhaps getting laid one day) IMO this goes far beyond a problem with men over-involving themselves in feminist causes i feel the same way for any relatively less-privileged group with a need to find their own voice and their own power they ll never find it if members of the privileged class (i.e. white males like me) just ride roughshod over the movement and speak FOR them rather then just silently lending their strength in support. For the most part, they (we) should just shut up and listen we already have more than enough opportunities to have our say. (*) yes, i m well aware of the difficulty in writing something like that paragraph as a member of a privileged class, without coming across as either self-hating or patronising or both. if i ve failed here, it s not for want of trying. done is a post from: Errata

A short announcement to mark the release of the first stable OpenID provider for Drupal. Started by James Walker around 4 years ago, I have since then got commit access and taken care of the module for the last 2 years. And while it was not an easy task, with the help of the community, this module was able to mature to a fully featured, flexible and standard implementation during that time. One has to wonder however, is it too late for OpenID?

A bit of history Drupal was one of the pioneers in OpenID support. Back in 2006 when the protocol was still in its infancy (less than a year old!), Drupal was adding OpenID support to its 4.7 release through a contrib module and two years later, the Drupal 6.0 release was sporting OpenID in core, although not without significant interoperability issues. Still, we could say that rapid adoption by such a popular CMS as Drupal really boosted OpenID's adoption. At about the same time, Yahoo, Sourceforge, Myspace, Microsoft and Google would all join the OpenID bandwagon. On the OpenID "provider" side of things, things were not so great however. Back when I started experimenting with this, it wasn't actually that easy to host your own OpenID provider, to allow your users to login to other OpenID-enabled sites. While there are a lot of different libraries to provide OpenID support in your application there was not (and still isn't) any software that gives you an OpenID provider and is dedicated to that. No official reference implementation either, other than the code from the one of the founding companies of OpenID, JanRain. There was an OpenID provider module in Drupal, but there was no official release done until a two beta releases in 2010.

Koumbit's community involvement It is about that time I started getting involved in the project. Koumbit needed an easy way for our workers to log into multiple sites without too much trouble, so I started looking at OpenID provider implementations. The lack of dedicated software somehow made the Drupal implementation a natural fit, and I started experimenting with the module, filing bugs and reviewing patches, as there were a lot of interoperability and design issues with the provider. After a while, walkah was kind enough to give me full access to the project and I started doing regular releases (every few months), culminating in today's official 1.0 release for both Drupal 6 and Drupal 7, which I am very proud and happy about. While I am here, I want to send many thanks to the people behind the Drupal 7 port: paranojik, mikecar, xamanu and testing from neizod. A lot of issues were also patched, reviewed and tested by numerous contributors that are unfortunately difficult to track down other than reading the commit log. So we at Koumbit have been running our own OpenID provider for a good while now, and it has the key advantage of being connected to our LDAP directory (with the D6 LDAP integration module so it is, in a sense, a central point for authentication at Koumbit. Very often people don't clearly understand what their LDAP password does, and I can easily remind them "it's the OpenID thing", or "try your password in the OpenID site" if they think they have forgotten it. So much goes on in the web browser these days that it makes everything much easier.

The problem with Drupal's OpenID provider However, it is kind of embarrassing that it took so long. More than 6 years after the OpenID protocol was written, we have a stable OpenID provider. Now, this may be because I am overly conservative about stable releases, but I don't think so: the OpenID provider module had, for a long time, serious interoperability issues with OpenID sites. There was also a clear lack of features on the OpenID provider, a gap that various third party modules, now documented on the OpenID provider homepage are now trying to fill. There is also serious security issues with the protocol itself. The complexity and lack of clarity of the OpenID specification have rendered my work quite difficult at time, as the protocol has significant ambiguities and no clear reference implementation. OpenID is unfortunately renowned to be vulnerable to phishing and while Koumbit's use of the HTTPS protocol alleviates some of those issues (especially the redirect session hijack), tools like sslstrip can bypass those protections easily. The privacy aspect is also problematic, as the user is usually not free to choose which information from his or her profile is divulged to the site they are logging into. Of course those issues may be overlooked if the security trade off is not worth it. In the case of Koumbit, we use OpenID on a lot of client sites, but usually nothing critical, and we train our users to pay attention to the HTTPS icon when login in their OpenID account.

Are there alternatives? But unfortuantely, the reality is that there are not really any alternatives to OpenID out there. Mozilla's new BrowserID initiative (now rebranded Persona, boy those guys sure love rebranding..) is interesting, but actually has even more flaws than OpenID itself. Not only is it not resolving the phishing problem, it actually creates new problems, including my personal favorite, trusting server-side Javascript code, which OpenID is not using that much. I should also mention the Shibboleth module which has a surprising number of installs recorded on Drupal.org, but that depends on the much more complicated Shibboleth federation, for which there is no server implementation for Drupal, from what I understand. Oauth, on the other end, is even more widely used, and has server-side components like Oauth Login Provider and Services 3.x. We could also simply give up and use our souls stored at Facebook, Google and Twitter for authentication everywhere. Maybe OpenID opened the door for those corporations to see that there is a real opportunity in leveraging user's profile to track their online activity. It is not something that OpenID was necessarily designed for but it's an idea that certainly permeates Facebook connect and "like" buttons or Twitter and Google online business strategies, where you are the product sold to advertisers. I personally believe we should keep the struggle for distributed and standard identity mechanisms for the web that respect users' privacy. With Drupal's OpenID Provider, it is possible to create a secure, privacy-aware identity provider, although I still have to perform that damn security audit. Experiments with those third party modules should also be encouraged and I welcome any feedback on those modules. While I do not plan to attend OpenID conferences or do significant development on the OpenID provider right now, I will certainly make sure Koumbit keeps its commitment to maintaining stewardship on the OpenID provider module in Drupal. I encourage people to try it out, submit issues and patches to make it even better. Long life to OpenID! PS: if people are interested in funding that security audit, I would be happy to perform this through my regular work at Koumbit.org, just file a request at services@koumbit.org.

Another month, another update to http.debian.net. This time around most of the work was done outside the redirector's code base, as strange as it may sound.

The redirector heavily relies on the mirrors doing at least a couple of things right, for the rest it can and does compensate. When it needs to compensate, certain requests are redirected to automatically-detected good mirrors, thus avoiding mirrors that might work fine for some parts of the day but cause headaches during the rest.

So, part of the work done since the last update was to prod more mirror administrators to upgrade to the latest version of ftpsync. This reduces the number of mirrors for which compensation is needed in order to avoid errors during installations and upgrades. Hopefully, no additional work is needed for the redirector to notice the upgrades. This results in immediate improvements.

However, not all mirrors comply with the bare minimum requirements. As stated in my previous blog post, running rsync once is not enough. When mirrors break these assumptions they lead to the "bad apple" effect. The effects in this case are temporary errors, as experienced by some people. The interesting part of those issues is that the affected population may quickly change given the redirector's use of geo location and the way it creates mirror subsets.

As interesting as the distribution of the effects may be, they are not really welcome. So I put together some code to attempt to detect the bad apples. This resulted in a list of mirrors that have now been disabled in the redirector and whose administrators are going to be contacted so that they comply with the minimum requirements. Given that detection is time-sensitive, there's no 100% guarantee that all of them have been identified so far. The code to detect them will have to be adapted and integrated into the redirector's code base to be proactive on avoiding this kind of issues.

Last but not least, the redirector is now using a database of AS peers for better (re-)routing. This is the next move towards a decision making based more on network location/topology than in geographic location. This first use of a peers database is limited to IPv4 and is based on a recent routing table dump and on feedback provided by interested people. If you are a mirror or network administrator, or you are familiar with the topology of your network please drop me an email so that the redirector can make a better use of your peering agreements.

N.b. in the case of the database, the term peer may also include transit providers. It is used to refer to and establish a relationship between two AS(N)s.


Feedback is, as always, welcome. I read each and every email but it may take me some time to get to it, or reply.