Search Results: "beh"

9 December 2012

Stefano Zacchiroli: bits from the DPL for November 2012

Just posted, bits from the DPL for November 2012. 
 ---------------------------------------------------------------------------------
/ Pop-up advertisement. Breaking news. Help the Release Team [0] by reviewing     \
  pending unblock requests. Oh, and by fixing RC bugs too.                         
                                                                                   
  [0]: http://nthykier.wordpress.com/2012/12/09/wheezy-release-progress-december/  
\---------------------------------------------------------------------------------/
Dear Project Members, here is another (delayed) monthly report of DPL activities, this time for November 2012. Highlights Events Assets That's all for now,
let's go back releasing Wheezy,
Cheers.
PS the day-to-day activity log for November 2012 is available at the usual place master:/srv/leader/news/bits-from-the-DPL.txt.201211

7 December 2012

Thorsten Glaser: Collision resolution in open addressing hashtables

Before we begin, everyone should read up on hashtables and what open addressing / closed hashing is. The context is lines 111 190 of Python s Objects/dictobject.c as of today (so we get the line numbers straight). (I ve reworded this wlog entry a bit; I originally wrote it too late at night for it to read coherent.) Basically, I ve got an application where I d like to use a hashtable for a number of things not as generic as Python, and with focus on small footprint. I d like to offer associative arrays in a scripting language, where the keys are always arbitrary byte strings excluding NUL. Also, I d like to use the hashtable as backend for indexed arrays, where the keys are uint32_t and the usual use case is sequential. Finally, I m using it for several internal tables, such as a list of keywords, one of builtins, one of special variables, etc. which is a reason for me to not use a self-balancing binary tree as data structure (reading further below might suggest that, but getting a sorted list of hashtable keys is not the focus, though not unimportant).
My questions on this are:

Why is the shift on perturb done after its first use? In my experiments (using 32-bit width everywhere), for the pathological case of an 8-element (i = 3) table with three entries 0, 0x40000000 and 0x800000000, the second round yields 1 for all three, so it cannot have to do with the upper bits. My lookup looks like:

	mask = 2  - 1;
	j = perturb = hash(key);
	goto find_first_slot;
	 find_next_slot:
	j = (j << 2) + j + perturb + 1;
	perturb >>= PERTURB_SHIFT;
	/* FALLTHROUGH */
	 find_first_slot:
	entry = table[j & mask];
	if (!match(entry)) goto find_next_empty_slot;

This means that my first check is always the bare hash (so only do it if needed is no reason) and, since I m using gotos, I could just move the perturb >>= PERTURB_SHIFT; line before the line recalculating the next j to use. This seems to make more sense, even in the face of Python. (I actually looked at the Python file s comments again today because I thought to use a different resolution, but they have a good rationale for using the multiplication by 5.) Why can t we just use i as the PERTURB_SHIFT? Sure, this changes a shift-right by a constant, which can possibly be encoded as immediate value in assembly (unless you re on a pre-80186, which can only do SHR AX,1 and SHR AX,CL but not SHR AX,4, but that s outside of mksh s scope) into a right-shift by a variable, but i is already known, and I think the behaviour is better (it wouldn t eat any bits; assume the same 8-entry hashtable and pathologic keys 0, 8 and 16). Again: who do I think I am to go against the wisdom of the Python people, who seem to have shed more thought on this than everyone else I saw, asked, read about (including Spammipedia). That s why I m asking here. On that reference: I don t support spammers or people nagging for donations or premium accounts, like Xing and Groundspeak/Geocaching.COM, at all. In fact, I urge others to do the same, so it really hurts them; it may be their business model, but not if they spam me. Besides, OpenCaching.DE exists. Another thing is: to avoid CVE-2011-4815, I m randomising the hash used, with one seed value per hashtable, changed before a resize operation. I originally thought to seed it with nonzero, but then I have to rehash on hashtable resize, so I ll be XORing the final hash value instead (thanks ciruZ for the idea). I m thinking of omitting that for indexed arrays, as an attacker almost certainly cannot determine the keys there. (To directly use the indexed array keys, which are already uint32_t, as hashes makes using i from even more important.) The hash I m using is a modified Jenkins one-at-a-time called NZAAT: it s my new generic standard n n-cryptographic hash, and the changes are thus: while adding a byte, another increment of the hash is done (so NUL counts), and the finaliser got prefixed with the shift-left-add+shift-right-xor sequence of the adder (but not adding any value or the +1), to get best avalanche for all bytes. I actually compiled several versions of Hash.cs on a Windows VM at work to analyse the original one-at-a-time and all of my modifications; these turned out to be the simplest ones (I originally had added 0x100 instead of 1, but the effect was the same, and +1 is usually cheaper on most CPUs). Also, to avoid people being able to get to the seed, a user will always get only a sorted list of hashtable keys (numeric for indexed arrays, ASCIIbetically otherwise; see also my thoughts on JSON from the previous wlog entry). What algorithm do I use? For strings, comparisons are much more expensive, so I d like to keep them low. Memory use is also a factor; allocating one large(r) block is better than many small ones due to the pool allocator overhead and due to portability to ancient Unic s (which is another reason for me to use a hashtable which is a small struct plus an array of pointers, and then pass the list of keys as array of string pointers, instead of a tree). For both reasons, I m thinking a relatively simple MergeSort: I need to allocate the result array anyway, so I can just get two and free the one that isn t the end result, and it s AFAICT the cheapest on comparisons other than Tree Sort (which nobody really seems to use, and which would effect to using a balanced binary tree again). Since keys are unique, stability and duplicate handling is never an issue. I d like to use only one algorithm and one data structure, not a combination, as compactness is a design goal. Please drop your thoughts on Freenode, e.g. by /msg MemoServ send mirabilos your text here or per eMail to the domains debian, freewrt or mirbsd, which are organisations, with the localpart tg. Or just contact me as usual, if you re already acquainted. Or lookup 0xE99007E0. Thanks in advance! (Especially, Python Developers thoughts are welcome.)

5 December 2012

Raphael Geissert: Introducing: a bashism a week

No matter how many scripting programming languages exist, it appears that shell programming is here to stay around. In many cases it is fast, it "does the job", and best of all: it is available "everywhere". The shell is used by makefiles, on every call to system(), and whatnot.

However, it is a real pain, implementations differ from the standards, some implementations still in use pre-date them, they leave room for undefined behaviour, and bugs in the implementations are nothing but unknown. You can't just specify a given shell interpreter and think you've dealt with the problem. Writing shell scripts that are portable among many platforms is a nightmare, if even possible.

Surprisingly, in spite of all that, a great amount of shell scripts appear to work flawlessly in many systems.

The switch from bash to dash as the default shell interpreter in Debian wasn't done without quite some work (more if you list archived bug reports), and the work ain't over.

For the following months I will be writing about different "bashisms" every Wednesday, hopefully helping people write slightly-more-portable shell scripts. The posts are going to be focused on widely-seen bashisms, probably ignoring those that Debian's policy defines as required to be implemented.

The term "bashism" must be understood as any feature or behaviour not required by SUSv3 (aka POSIX:2001), no matter what its origins are or even if the behaviour is not exhibited by the bash shell.

One of the key points is documenting the script's requirements, starting by specifying the right shell interpreter in the shebang.

Let's see what comes out of this experiment.

As a matter of fact, I have a few months worth of posts written already. All posts are going to be published at scheduled dates, just like this very post.

4 December 2012

Russ Allbery: Review: The Borrowers

Review: The Borrowers, by Mary Norton
Series: The Borrowers #1
Illustrator: Beth Krush
Illustrator: Joe Krush
Publisher: Sandpiper
Copyright: 1953
Printing: 1986
ISBN: 0-15-204737-9
Format: Trade paperback
Pages: 180
Arrietty is a 14-year-old girl who lives under the clock. Well, actually, she lives under the kitchen floor, but the entrance to her family's passageway is under the clock, from which they get their name. She sleeps in an old cigar box, her living room is wallpapered with strips cut from letters (and decorated with postage stamps), and she keeps a journal in a miniature book that's half the size she is. Arrietty, and her parents Pod and Homily, are Borrowers. The Borrowers is a classic series of children's books by Mary Norton about miniature people who, at least in this first book, live in the walls and take things. Not big things, but all the little things: the blotter paper, the pins, the stray sock, old postage stamps, empty matchboxes, and the occasional thimble. With these things, they build their furniture, their clothes, and their homes, safely hidden in the walls and other secret places, always wary of the "human beans" (as Arrietty's mother tends to call them). That life is, of course, a remarkable simulation of middle-class, class-conscious British life, which is much of the fun. Arrietty's family lives in the home of an elderly woman who's confined to bed, so it's quiet, safe, and driven entirely by routine. They only have to watch out for the maid and the gardener, and they keep regular schedules. That's just the way the Borrowers like it; predictable and quiet. Which is why they get caught entirely by surprise when a boy shows up, recovering from illness in the country, and spots Pod on one of his borrowing runs. This is technically a re-read, but I read the original series so long ago that I barely remembered it. Not that there is all that much to remember, at least in the way of plot. This first book follows one of the standard plots for children's books of this type (although it follows Arrietty as the viewpoint character rather than the more common choice of the boy): boy discovers hidden creatures, hidden creatures are afraid, boy turns out to want to make friends with them, they strike up a friendship, peril nonetheless ensues. If you've read many children's books, you've probably read this before; The Mouse and the Motorcycle is an obvious example (and, in terms of plot, a somewhat better book). Arrietty is a typical protagonist, her parents are silly (and a bit flighty, particularly her mother), and the boy is, disappointingly, almost a non-entity. But you don't read The Borrowers for the plot. The brilliance of this book is in the descriptions of what they use, how they use it, and how they construct their ingenious homes. Norton captures much of the wonder of other miniaturization stories (the movie Honey, I Shrunk the Kids is an obvious example), but takes it one step further into clever and thoughtful repurposing. The book is full of off-hand descriptions of how everyday objects become different everyday objects for miniature people, giving it the same delight that one gets from descriptions of how model railroad dioramas or hand-made doll houses are constructed. The detailed (if busy) illustrations by the Krushes help tremendously with visualization and make the descriptions even more fun. This is very much a children's book based on an exploration of a single great idea, so don't expect too much in the way of plot or deep characterization. It's also not science fiction; there isn't any realistic extrapolation of culture. The Borrowers all behave like perfectly ordinary English people without any further explanation because, of course, that's what they would do. Re-reading it as an adult, I wished there was a bit more meat to it. But the ingenuity is still a delight. It's great fun to spend a couple of hours imagining how one would construct the miniature accouterments of everyday life. Followed by The Borrowers Afield. Rating: 7 out of 10

3 December 2012

Erich Schubert: ResearchGate Spam

Update Dec 2012: ResearchGate still keeps on sending me their spam. Most of the colleagues I had that tried out RG now deleted their account there, apparently, so the invitation mails become fewer. Please do not try to push this link on Wikipedia just because you are also annoyed by their emails. My blog is not a "reliable source" by Wikipedia standards. It solely reflects my personal view of that web site, not journalistic or scientific research. The reason why I call ResearchGate spam is the weasel words they use to trick authors into sending the invitation spam. Here's the text coming with the checkbox you need to uncheck (from the ResearchGate "blog")
Add my co-authors that are already using ResearchGate as contacts and invite those who are not yet members.
See how it is worded so it sounds much more like "link my colleagues that are already on researchgate" instead of "send invitation emails to my colleagues"? It deliberately avoids the mentioning of "email", too. And according to the researchgate news post, this is hidden in "Edit Settings", too (I never bothered to try it -- I do not see any benefit to me in their offers, so why should I?). Original post below:
If you are in science, you probably already received a couple of copies of the ResearchGate spam. They are trying to build a "Facebook for scienctists", and so far, their main strategy seems to be aggressive inivitation spam. So far, I've received around 5 of their "inivitations", which essentially sound like "Claim your papers now!" (without actually getting any benefit). When I asked my colleagues about these invitations none actually meant to invite me! This is why I consider this behaviour of ResearchGate to be spam. Plus, at least one of these messages was a reminder, not triggered by user interaction. Right now, they claim to have 1.9 million users. They also claim "20% interact at least once a month". However, they have around 4000 Twitter followers and Facebook fans, and their top topics on their network are at like 10000-50000 users. That is probably a much more real user count estimation: 4k-40k. And these "20%" that interact, might just be those 20% the site grew in this timeframe and that happened to click on the sign up link. For a "social networking" site, these numbers are pointless anyway. That is probably even less than MySpace. Because I do not see any benefit in their offers! Before going on an extremely aggressive marketing campaign like this, they really should consider to actually have something to offer... And the science community is a lot about not wasting their time. It is a dangerous game that ResearchGate is playing here. It may appeal to their techies and investors to artificially inflate their user numbers in the millions. But if you pay for the user numbers with your reputation, that is a bad deal! Once you have the reputation as being a spammer (and mind it, every scientist I've talked to so far complained about the spam and "I clicked on it only to make it stop sending me emails") it's hard to be taken serious again. The scientific community is a lot about reputation, and ResearchGate is screwing up badly on this. In particular, according to researchgate founder on quora, the invitations are opt-out on "claiming" a paper. Sorry, this is wrong. Don't make users annoy other users by sending them unwanted invitations to a worthless service! And after all, there are alternatives such as Academia and Mendeley that do offer much more benefit. (I do not use these either, though. In my opinion, they also do not offer enough benefit to bother going to their website. I've mentioned the inaccuracy of Mendeleys data - and the lack of an option to get them corrected - before in an earlier blog post. Don't rely on Mendeley as citation manager! Their citation data is unreviewed. I'm considering to send ResearchGate (they're Berlin based, but there maybe also is a US office you could direct this to) a cease and desist letter, denying them to store personal information on me, and to use my name on their websites to promote their "services". They may have visions of a more connected and more collaborative science, but they actually don't have new solutions. You can't solve everything by creating yet another web forum and "web2.0izing" everything. Although many of the web 2.0 bubble boys don't want to hear it: you won't solve world hunger and AIDS by doing another website. And there is a life outside the web.

1 December 2012

Rapha&#235;l Hertzog: My Free Software Activities in November 2012

This is my monthly summary of my free software related activities. If you re among the people who made a donation to support my work (692.20 , thanks everybody!), then you can learn how I spent your money. Otherwise it s just an interesting status update on my various projects. Misc packaging I updated the publican package (a tool for publishing material authored in DocBook XML) with version 3.0, a major new upstream version. As with any important update, it had its share of problems and I created two patches that I sent upstream. I uploaded the package to experimental since we re in freeze. The Debian Administrator s Handbook Since the translation teams have been working for a few months, I wanted to put the result of their work online. I did it and I blogged about it on debian-handbook.info. By the way, we have a Polish translation that just started. This took quite some time because many translators were not well versed with Docbook XML and its structure. So I fixed their mistakes and asked the Weblate developer (Michal Cihar) to implement new checks to avoid those basic XML mistakes. I also added a couple of build scripts to the git repository to make it easier to rebuild translations in multiple formats. I used this opportunity to file a couple of bugs I encountered with Publican (concerning ePub output mainly, and custom brands). I also blogged about our plans to update the book for Wheezy. Roland started to work on it but I did not have the time yet. Debian France The officers (president, treasurer, secretary) have just changed and we had to organize the transition. As the new president, I got administrator access on our Gandi virtual machine (france.debian.net) as well as access to our bank account. I got also got a bunch of administrative papers retracing the history of the association. Carl Chenet (the former president) gave them to me during the mini-debconf that was organized in Paris. Indeed, Sylvestre Ledru and Mehdi Dogguy organized our second mini-debconf Paris and they did it very well. It was a great success with over 100 attendants each of the 2 days it lasted (November 24-25th). Carl managed a merchandising booth that was well stuffed (Luca Capello also brought goodies of Debian.ch) I gave small lightning talk to present the ideas behind my Librement project (it s about funding free software developers). BTW I have not been very good at it, it was only my second lightning talk and I have been a bit too verbose. The talk did not fit in my 5 minutes time slot ;-) Back from the mini-debconf, I have been trying to delegate some projects (like get a real website, improve the work-flow of members management, update our server which was still running Lenny). Julien Cristau was willing to upgrade the server did not exactly knew how to upgrade the kernel (it s a bit special since Gandi manages the kernel on the Xen hypervisor side). So I took care of this part and also did some cleanup (adding a backup with its associated remote disk, tweaking the email configuration). And Julien completed the upgrade on November 30th. Alexandre Delano volunteered to have a try at the website and Emmanuel Bouthenot has been looking a bit to see if there was something better than Galette to handle our members. It looks like we ll stay with Galette but have to take care of upgrading it to a newer version. I also processed the first membership applications and organized a vote to extend the board of administrators (since we have two vacant seats). On Monday, we should be back to 9 administrators. Librement Except for the talk during the mini-debconf, I did not do much on this project. That said I got an answer from the Autorit de Contr le Prudentiel saying that I might be eligible for the exemption case (see discussion of last month) and that I should fill out a form to get a confirmation. I also contacted Tunz.com who might be able to provide the services I need (their E-money manager product in particular). They have the required accreditation as a banking/credit institution and are willing to partner with enterprises who setup platforms where you must manage flows of money between several parties. I m now waiting for details such as the cost of their various services. I expect to have much more to show next month I m working with two developers to implement the first building blocks of all this. Thanks See you next month for a new summary of my activities.

No comment Liked this article? Click here. My blog is Flattr-enabled.

30 November 2012

Josselin Mouette: Shameless self-advertisement

Have you ever wondered one of those? For those who weren t at Mini-DebConf Paris 2012 last week, let me share here again the slides:
Large deployment of GNOME from the administrator s perspective
These questions, and many others, will find the beginning of an answer here. At the very least, I hope to show people leads on where to find relevant information for their needs about administrating GNOME systems.

28 November 2012

Alberto Garc a: QEMU and open hardware: SPEC and FMC TDC

Working with open hardware Some weeks ago at LinuxCon EU in Barcelona I talked about how to use QEMU to improve the reliability of device drivers. At Igalia we have been using this for some projects. One of them is the Linux IndustryPack driver. For this project I virtualized two boards: the TEWS TPCI200 PCI carrier and the GE IP-Octal 232 module. This work helped us find some bugs in the device driver and improve its quality. Now, those two boards are examples of products available in the market. But fortunately we can use the same approach to develop for hardware that doesn t exist yet, or is still in a prototype phase. Such is the case of a project we are working on: adding Linux support for this FMC Time-to-digital converter.
FMC TDC
This piece of hardware is designed by CERN and is published under the CERN Open Hardware Licence, which, in their own words is to hardware what the General Public Licence (GPL) is to software . The Open Hardware repository hosts a number of projects that have been published under this license. Why we use QEMU So we are developing the device driver for this hardware, as my colleague Samuel explains in his blog. I m the responsible of virtualizing it using QEMU. There are two main reasons why we want to do this:
  1. Limited availability of the hardware: although the specification is pretty much ready, this is still a prototype. The board is not (yet) commercially available. With virtual hardware, the whole development team can have as many boards as it needs.
  2. Testing: we can test the software against the virtual driver, force all kinds of conditions and scenarios, including the ones that would probably require us to physically damage the board.
While the first point might be the most obvious one, testing the software is actually the one we re more interested in. My colleague Miguel wrote a detailed blog post on how we have been using QEMU to do testing. Writing the virtual hardware Writing a virtual version of a particular piece of hardware for this purpose is not as hard as it might look. First, the point is not to reproduce accurately how the hardware works, but rather how it behaves from the operating system point of view: the hardware is a black box that the OS talks to. Second, it s not necessary to have a complete emulation of the hardware, there s no need to support every single feature, particularly if your software is not going to use it. The emulation can start with the basic functionality and then grow as needed. The FMC TDC, for example, is an FMC card which is in our case connected to a PCIe bridge called SPEC (also available in the Open Hardware repository). We need to emulate both cards in order to have a working system, but the emulation is, at the moment, treating both as if they were just one, which makes it a bit easier to have a prototype and from the device driver point of view doesn t really make a difference. Later the emulation can be split in two as I did with with TPCI200 and IP-Octal 232. This would allow us to support more FMC hardware without having to rewrite the bridging code. There s also code in the emulation to force different kind of scenarios that we are using to test if the driver behaves as expected and handles errors correctly. Those tests include the simulation of input in the any of the lines, simulation of noise, DMA errors, etc.
Tests
And we have written a set of test cases and a continuous integration system, so the driver is automatically tested every time the code is updated. If you want details on this I recommend you again to read Miguel s post.

26 November 2012

Steve Langasek: Upstart in Debian

Good news, everyone! So as of last Sunday, this works on all Linux archs in Debian unstable and gives you a modern version of upstart: 
echo 'Yes, do as I say!'   apt-get -o DPkg::options=--force-remove-essential -y --force-yes install upstart
Thanks to the ifupdown, sysvinit, and udev maintainers for their cooperation in getting upstart support in place; to the Debian release team for accomodating the late changes needed for upstart to be supported in wheezy; and to Scott for his past maintenance of upstart in Debian. Benchmarking One of the consequences is that it's now possible to do meaningful head-to-head comparisons of boot speed between sysvinit (with startpar), upstart, and systemd. At one time or another people have tested systemd vs. sysvinit when using bash as /bin/sh, and upstart vs. sysvinit, and systemd vs. sysvinit+startpar, and there are plenty of bootcharts floating around showing results of one init system or another on one distro or another, but I'm not aware of anyone having done a real, fair comparison of the three solutions, changing nothing but the init system. I've done some initial comparisons in a barebones sid VM, and the results are definitely interesting. Sysvinit with startpar (the default in Debian) can boot a stock sid install, with no added services, in somewhere between 3.37 and 3.42 seconds (three runs). That's not a whole lot, but on the other hand this is a system with a single filesystem and no interesting services yet. Is this really as fast as we can boot? No, even this minimal system can boot faster. Testing with upstart shows that upstart can do the same job in between 3.03 and 3.19 seconds (n=3, mean=3.09). This confirms what we'd already seen in Ubuntu, that it makes a difference to boot speed to have filesystem mounting handled by an integrated process that understands the whole system instead of as a group of serialized shell scripts. What about systemd? The same test gives a boot time between 2.32 and 2.85 seconds (n=4, mean=2.48). Interesting; what would make systemd faster than upstart in sid? Well, a quick look at the system shows one possible contributing factor: the rsyslog package in Debian has a systemd unit file, but not an upstart job file. Dropping in the /etc/init/rsyslog.conf from Ubuntu has a noticeable impact, and brings the upstart boot time down nearer to that of systemd (2.78-3.03s, n=5, mean=2.92). Besides telling me that it's time to start spamming Debian maintainers with wishlist bugs asking them to include upstart jobs in their packages, this suggests that the remaining difference in boot time may be due to the outstanding init scripts in rcS.d that are made built-in no-ops by systemd but not (yet) by upstart in Debian (e.g., hwclock, hostname, udev-mtab). (In Ubuntu, /etc/rcS.d has long since been emptied out in favor of upstart jobs in the common case, since the time it takes to get to runlevel=2 is definitely a major issue for boot speed and boot parallelization.) It also gives the lie to the claim that's been made in various places that spawning shells is a major bottleneck for upstart vs. systemd. More study is certainly needed to confirm this, but at least this naive first test suggests that in spite of the purported benefits of hard-coding boot-time policies in C code, upstart with its default degree of runtime configurability is at least in the ballpark of systemd. Indeed, when OpenSuSE switched from upstart to systemd, it seems that something else in the stack managed to nullify any benefit from improving the boot-time performance of apparmor. Contrary to what some would have you believe, systemd is not some kind of silver bullet for boot speed. Upstart, with its boot-time flexibility and its long history of real-world testing in Ubuntu, is a formidable competitor to systemd in the boot speed department - and a solid solution to the many longstanding boot-time ordering bugs in Debian, which still affect users of sysvinit. I've published the bootcharts for the above tests here. Between the fact that Debian's bootchart package logs by default to /var/log/bootchart.tgz (thus overwriting on every boot) and the fact that these tests are in a VM, I haven't bothered to include the raw data, just the bootcharts themselves. The interested reader can probably generate more interesting boot charts of their own anyway - in particular, it will be interesting to see how the different init systems perform with more complicated filesystem layouts, or when booting a less trivial set of services. Other musings The boot charts have been created with the bootchart package rather than with bootchart2. For one thing, it turns out that bootchart2 includes systemd units, not init scripts; so when replacing bootchart with bootchart2, the non-matching init script is left behind and systemd in particular gets terribly confused. This is now reported as Bug #694403. In an amusing twist, while I was experimenting with bootchart2, I also noticed that having systemd installed would slow down booting with other init systems, because systemd installs udev rules which take a noticeable amount of time to run a helper command at boot even though the helper should be a no-op. So if you're doing boot speed testing of other init systems, be sure you don't have systemd on the system at the time!

Russell Coker: Links November 2012

Julian Treasure gave an informative TED talk about The 4 Ways Sound Affects US [1]. Among other things he claims that open plan offices reduce productivity by 66%! He suggests that people who work in such offices wear headphones and play bird-songs. Naked Capitalism has an interesting interview between John Cusack and Jonathan Turley about how the US government policy of killing US citizens without trial demonstrates the failure of their political system [2]. Washington s blog has an interesting article on the economy in Iceland [3]. Allowing the insolvent banks to go bankrupt was the best thing that they have ever done for their economy. Clay Shirky wrote an insightful article about the social environment of mailing lists and ways to limit flame-wars [4]. ZRep is an interesting program that mirrors ZFS filesystems via regular snapshots and send/recv operations [5]. It seems that it could offer similar benefits to DRBD but at the file level and with greater reliability. James Lockyer gave a movingTEDx talk about his work in providing a legal defence for the wrongly convicted [6]. This has included overturning convictions after as much as half a century in which the falsely accused had already served a life sentence. Nathan Myers wrote an epic polemic about US government policy since 9-11 [7]. It s good to see that some Americans realise it s wrong. There is an insightful TED blog post about TED Fellow Salvatore Iaconesi who has brain cancer [8]. Apparently he had some problems with medical records in proprietary formats which made it difficult to get experts to properly assess his condition. Open document standards can be a matter of life and death and should be mandated by federal law. Paul Wayper wrote an interesting and amusing post about Emotional Computing which compares the strategies of Apple, MS, and the FOSS community among other things [9]. Kevin Allocca of Youtube gave an insightful TED talk about why videos go viral [10]. Jason Fried gave an interesting TED talk Why Work Doesn t Happen at Work [11]. His main issues are distraction and wasted time in meetings. He gives some good ideas for how to improve productivity. But they can also be used for sabotage. If someone doesn t like their employer then they could call for meetings, incite managers to call meetings, and book meetings so that they don t follow each other and thus waste more of the day (EG meetings at 1PM and 3PM instead of having the second meeting when the first finishes). Shyam Sankar gave an interesting TED talk about human computer cooperation [12]. He describes the success of human-computer partnerships in winning chess tournaments, protein folding, and other computational challenges. It seems that the limit for many types of computation will be the ability to get people and computers to work together efficiently. Cory Doctorow wrote an interesting and amusing article for Locus Magazine about some of the failings of modern sci-fi movies [13]. He is mainly concerned with pointless movies that get the science and technology aspects wrong and the way that the blockbuster budget process drives the development of such movies. Of course there are many other things wrong with sci-fi movies such as the fact that most of them are totally implausible (EG aliens who look like humans). The TED blog has an interesting interview with Catarina Mota about hacker spaces and open hardware [14]. Sociological Images has an interesting article about sporting behaviour [15]. They link to a very funny youtube video of a US high school football team who make the other team believe that they aren t playing until they win [16] Related posts:
  1. Links April 2012 Karen Tse gave an interesting TED talk about how to...
  2. Links March 2012 Washington s Blog has an informative summary of recent articles about...
  3. Links November 2011 Forbes has an interesting article about crowd-sourcing by criminals and...

25 November 2012

Steve Kemp: A busy few weeks - bah humbug

The following companies are amongst those showing Christmas Adverts on television before the start of December: I will boycott these companies until next year. In happier news I've spent the past week or two replacing the monitoring system that we use at work. Our previous monitoring system had been struggling to keep up with the sheer number of tests it was being asked to process. This was partly because we carry out many ping-tests, ssh-tests, http-tests, dns-tests, etc. The other reason was that our monitoring system was a behemoth of threaded-ruby, which all ran upon a single host. This made adding another monitoring host a complex undertaking. The new solution uses a work-queue: The code is open-source, written in Ruby, and available here: I've completed the process of tidying up the code to the extent I'm happy with it, and I believe I've also abstracted away the work-specific pieces of the code. That said I'd not be surprised if it needs a few minor tweaks before it it useful for other people.

23 November 2012

Benjamin Mako Hill: Cultivated Disinterest in Professional Sports

Like many of my friends, I have treated professional sports with cultivated indifference. But a year and a half ago, I decided to become a football fan. Several years ago, I was at a talk by Michael Albert at MIT where he chastised American intellectuals for what he claimed was cultivated disdain of professional sports. Albert suggested that sports reflect the go-to topic for small talk and building rapport across class and context. But he suggested that almost everybody who used the term "working class struggle" was incapable of making small talk with members of the working class because unlike most working class people (and most people in general) educated people systematically cultivate ignorance in sports. Professional sports are deeply popular. In the US, Sunday Night Football is now the most popular television show among women in its time slot and the third most popular television in America among 18-49 year old women. That it is also the most popular television show in general is old news. There are very few things that anywhere near half of Americans have in common. Interest in football is one of them. An enormous proportion of the US population watches the Superbowl each year. I recognized myself in Albert's critique. So I decided to follow a local team. I picked football because it is the most popular sport in America and because their strong revenue sharing system means that either team has a chance to win any given match. My local team is the New England Patriots and I've watched many of the team's games or highlights over the last season and a half. I've also followed a couple football blogs. A year and half in, I can call myself a football fan. And I've learned a few things in the process:
  1. With a little effort, getting into sports is easy. Although learning the rules of a sport can be complicated, sports are popular because people, in general, find them fun to watch. If you watch a few games with someone who can explain the rules, and if you begin to cheer for a team, you will find yourself getting emotionally invested and excited.
  2. Sports really do, as Albert implied, allow one to build rapport and small talk across society. I used to dread the local cab driver who would try to make small talk by mentioning Tom Brady or the Red Sox. No more! Some of these conversations turn into broader conversations about life and politics.
  3. Interest in sports can expand or shrink to fill the time you're willing to give it. It can mean just glancing through the sports sections of the paper and watching some highlights here or there. Or it can turn into a lifestyle.
  4. It's not all great. Football, like most professional sports, is deeply permeated with advertisements, commercialism, and money. Like other sports, it is also violent. I don't think I could ever get behind a fight sport where the goal is to hurt someone else. The machoness and absence of women in the highest levels of most professional sports bothers me deeply.
I've also tried to think a lot about why I, like most of my friends, avoided sports in the past. Disinterest in sports among academics and the highly educated is, in my experience, far from passive. I've heard people almost compete to explain the depth of their ignorance in sports one doesn't even know the rules, one doesn't own a television, one doesn't know the first thing about the game. I did the same thing myself. Bethany Bryson, a sociologist at JMU has shown that increased education is associated with increased inclusiveness in musical taste (i.e., highly educated people like more types of music) but that these people are most likely to reject music that is highly favored by the least educated people. Her paper's title sums up the attitude: "Anything But Heavy Metal". For highly educated folks, it's a sign of cultivation to be eclectic in one's tastes. But to signal to others that you belong in the intellectual elite, it can pay in cultural capital to dislike things, like sports, that are enormously popular among the least educated parts of society. This ignorance among highly educated people limits our ability to communicate, bond, and build relationships across different segments of society. It limits our ability to engage in conversations and build a common culture that crosses our highly stratified and segmented societies. Sports are not politically or culturally unproblematic. But they provide an easy and enjoyable way to build common ground with our neighbors and fellow citizens that transcend social boundaries.

18 November 2012

Jamie McClelland: New Privacy Arguments are Needed

Another decent article on online privacy was published by the NYT this morning. The article documents the collection and centralized aggregation of consumer data online. It ends with a strong message about consumers becoming the "product" - which echoes what Noam Chomsky and Ben Bagdikian have been saying for decades. The problem with this critique is that it doesn't go far enough. Chomsky and Bagdikian didn't say readers are the product of modern journalism and that's bad because nobody wants to be a product. They critiqued the model because it produces terrible journalism and democracy suffers as a result. Similarly, the online trading of audiences and the invasion of our privacy that results has more important implications than "don't be a sucker" which seems to be the gist of the privacy argument these days. Honestly, I don't stay awake at night worrying that Nike knows I'm looking for a pair of sneakers, or even that they know my income. I do stay awake at night worrying that people are tagging my photo on Facebook, which could allow the New York Police Dept to submit a photo of protesters to Facebook and get a list of names and addresses of the people in the photo. Or it could allow the police to track my movements via existing networks of surveillance cameras by matching my image to my name. Would that require a search warrant? How would that impact my trust in my government to know that my movements are being tracked? Or worse, to know they might be tracked but I'll never know if they are or aren't? Many governments (including the US) have sordid histories of infiltrating political organizations and intentionally sowing dissent. Having access to activists' purchasing habits provides a treasure trove of material for this purpose that previously was labor intensive for cops to collect. It also provides leverage during interrogation, a time of heightened emotional vulnerability where the mere mention of a private detail of one's life can have a powerful impact. How can we prevent law enforcement from having access to these details of our lives? And then there's the dragnet. A central database of consumer habits provides a tantalizing collection of information that could be searched for people who fit a profile of a crime. Is solving the crime at hand worth the invasion of privacy that results from people being investigating only based on their consumer habits? How would we measure the impact an investigation like this would have on our confidence in democracy and government? The corporate advertisers promoting this kind of data collection argue that these problems are the price of progress. However, the ideology behind these technology developments is capitalism, not some intrinsic aspect of the Internet or network communications. Historically, the Internet has developed based on a drive toward de-centralization. Email and web sites, the bedrock of Internet activity, are based on these principles: anyone can add their own email server or web server and everyone can seamlessly access it. In contrast, the logic of capitalism as it relates to the Internet, is one of centralization and aggregating data. Facebook and Google want you to use their platforms for all your online activities so they have as complete a picture as possible of what you like and do. As the New York Times article describes, companies like Rubicon want to track all your movements and aggregate that data for profit. We have a choice. We can carefully evaluate our use of corporate services and consider the implications it has not only on our lives but on society as a whole. And, we can choose to use and support services that promote open standards and interoperability that will respect our personal privacy values. Want a practical example? Friendica is software designed to allow you to post status alerts, photos, videos and other information about yourself in a way similar to Facebook. However, it has one big difference: you can host your account on any server running the software anywhere in the world. You can still "friend" people on different servers, but we don't have to all trust the same organization to host it. Furthermore, Friendica allows you to post once and automatically cross-post to your Twitter, Facebook or other similar services. Sound good? You are welcome to try out an account on the May First/People Link install: https://friends.mayfirst.org/.

17 November 2012

Craig Sanders: done

OK, I m finished my experiment. It s over now and I can break character . First, I have to offer special thanks to tshirtman for being the first to unambiguously exemplify one of my main points. well done! In case it s not blindingly obvious (as it should be), the reason for my post was that I was outraged by the spectacle of one fairly high-profile member of the linux community trying to rally support to shun and exclude another fairly high-profile member because a nightmare had upset her. WTF? Is that really all it s going to take to destroy someone s reputation and perhaps their career? even with the shunning target s own words available and archived to disprove the ridiculous straw-man mis-characterisations of what he actually said? Not one of the arguments against him actually addressed anything he said, they ALL attacked him for things he didn t say, for things that other people claimed he said. I was further outraged by seeing everyone who even suggested that questioning of stats (or, indeed, ANY claim of fact or evidence) may be, in some small way, a valid and reasonable thing to do get instantly put in their place and dismissed as Yet Another Rape Apologist. Are we supposed to be anti-science, anti-scientific method now? or are rape stats a special case like religion where we are just supposed to switch off our analytical brains and accept what we are told on faith, without question? Surely we are capable of better than that? i know we are capable of better than that. Or, at least, i used to know that. Now i m not so sure. In 2011, all it took was Ted Tso ( TT ) making some fairly reasonable statements about the need for any claimed evidence or statistics to be viewed skeptically and that dissenting research should also be considered and he was instantly vilified as a rape apologist . Sorry, but questioning extremely dodgy stats (that even in feminist circles are viewed more as ideological propaganda than as serious research) is NOWHERE NEAR SUFFICIENT to earn the label of rape-apologist. That is not how debate works you can t just refuse to engage with someone s point and simply accuse them of being the enemy for not agreeing 100% with whatever you say at least, not if you have any intellectual honesty or self-respect. (sure, some people are complete jerks and deserve to be told to FOAD in no uncertain terms but a) jerks like that are self-evident and obvious, and b) TT s participation in that thread was at all times civil and reasonable) But that thread is ancient history it was over and done with nearly two years ago.
In October this year, for reasons which are not at all clear, Valerie Aurora ( VA ) decided to revive the issue (which had been resolved back in 2011 with a resounding fuck no, we don t want misogynist shit or porn in our conferences from pretty much the entire linux community including near-universal support for improved anti-harassment policies both for linux.conf.au and for geek conferences in general) and use it to attack TT. And she did so by twisting his words and claiming in a post on The Ada Initiative blog that he said something which he didn t, that rape was impossible if both people were drunk enough . If he chose, he could quite easily win a libel case against her and TAI on that. It s not what he said, it s not even close to what he said, and VA is clearly too smart to honestly believe that it is what he said. In another post on her personal, blog she talks about how what he said was so terrible that it even now gives her nightmares, and that she can t bear the thought of working with him. Again, WTF? VA can say I had nightmares and was upset and furious and THAT is enough to justify a call to shun TT?? He didn t attack her, or threaten her (explicitly or implicitly), he was polite and civil. What he did was disagree with VA by referring to other research that disputed VA s preferred studies. I agree with and support many (perhaps most) of VA s and The Ada Initiative s aims, I certainly believe that linux and open source etc should be very welcoming and supportive of human diversity (including gender and sexuality, identity, religion, politics and so on), believe that it s a good thing that The Ada Initiative exists as part of that diversity welcome to be particularly supportive of women in geekdom. And i wholeheartedly agree that Linux leaders should not make public statements belittling and condoning rape BUT: a) I haven t seen one instance of that happening, ever and b) I find VA s choice of tactics here to be despicable. as i do when anyone else uses similar tactics, because they ARE despicable tactics. They are exactly the same as accusing someone of being a child pornographer for being against net censorship: You dared to disagree with me so I m going to accuse you of being a monster. (and, i must admit, the enthusiasm level of my support for The Ada Initiative is somewhat .diminished by this tactical blunder by the spokesperson and co-founder) There are far more deserving targets of VA s ire than TT. And there are far better ways for the Ada Initiative to achieve their aims.
Why? So why did I decide to comment when I knew that I was inevitably going to be accused of hating women and being a rape apologist? Mostly because I thought it would be gutless of me not to. Hardly anyone else had, and they quickly backed down under the accusations of misogyny and since I consider myself to be psychologically fairly strong, I felt that I am capable of wearing a little shit (or even a lot) for a while. In my egocentric fashion, I thought if I can t do then it s no wonder that no one else dares . And also because anyone who cared to make even the slightest effort to find out what my actual views on sexual harassment, rape, women s rights and numerous inter-related issues are can fairly easily see a very consistent record of the kinds of things I argue for and against, and my scathing responses to actual misogynists when they appear on lists that I participate in. they are not my kind of people. and even then, i hesitated. it s scary and intimidating to be putting yourself forward to be accused of being one of the things you hate. this is, of course, an instance of the chilling effect. So, I found the prospect scary, almost terrifying .and I can t think of a single person who has met me online or in real life who would even remotely describe me as being any kind of delicate or sensitive wall-flower. again, if i can t do it, it s no wonder no one else dares . so i clicked the Publish button. an ego is useful for some things. Also, I took VA s words but don t be silent as inspiration. In the process, i discovered why it is that some people just simply refuse to engage in rational discourse. I ve seen it many times from the other side, but i ve never experienced the seductive pleasure of indulging in it myself before there s a liberating freedom in just ignoring any and every point that someone makes and simply accuse them of being the enemy. You don t have to try to understand what they wrote, hell you don t even have to really read it you just need to quickly scan it for overall tone and if they don t seem like they re 100% supportive, you just accuse them of being the enemy or an apologist for the enemy. it s that fucking simple and easy. well, sort of easy. easy for some, perhaps. i personally found it extremely difficult a struggle to refrain from engaging, to remain in character (i m not much of an actor). especially when i kind of agreed with whoever was arguing against my experimental character or if i thought they made a good point. and even more so when i thought that some comments skirted a bit too close to being the kind of misogynist crap that i didn t want to tolerate having on MY blog. (i resolved that issue by just approving any reply that didn t squick me or that i could squint at and think hmmm borderline, give benefit of doubt ) but even though i don t like it, i can recognise the attraction it holds for some people.
Other thoughts: I m particularly disgusted by the men who intervene way too early without an explicit invitation or request for help or a clear need such as an immediate threat of violence in womens issues. Many or maybe even most may not realise it, but they are just taking over and asserting male strength and control by protecting women rather than giving them the support and space to discover and practice their own strength and their own voices. These uninvited interventions do not help women, they weaken and undermine them, they perpetuate dependence, they steal strength from the movement. It is patronising and enfeebling. But mostly, they just re-assert male dominance and are an attempt to make women s spaces more comfortable, more palatable, for men.(*) (it s also quite often very transparent self-serving and ingratiating behaviour from blokes who want to lay the groundwork for perhaps getting laid one day) IMO this goes far beyond a problem with men over-involving themselves in feminist causes i feel the same way for any relatively less-privileged group with a need to find their own voice and their own power they ll never find it if members of the privileged class (i.e. white males like me) just ride roughshod over the movement and speak FOR them rather then just silently lending their strength in support. For the most part, they (we) should just shut up and listen we already have more than enough opportunities to have our say. (*) yes, i m well aware of the difficulty in writing something like that paragraph as a member of a privileged class, without coming across as either self-hating or patronising or both. if i ve failed here, it s not for want of trying. done is a post from: Errata

Antoine Beaupr : First stable release of the Drupal OpenID Provider

A short announcement to mark the release of the first stable OpenID provider for Drupal. Started by James Walker around 4 years ago, I have since then got commit access and taken care of the module for the last 2 years. And while it was not an easy task, with the help of the community, this module was able to mature to a fully featured, flexible and standard implementation during that time. One has to wonder however, is it too late for OpenID?

A bit of history Drupal was one of the pioneers in OpenID support. Back in 2006 when the protocol was still in its infancy (less than a year old!), Drupal was adding OpenID support to its 4.7 release through a contrib module and two years later, the Drupal 6.0 release was sporting OpenID in core, although not without significant interoperability issues. Still, we could say that rapid adoption by such a popular CMS as Drupal really boosted OpenID's adoption. At about the same time, Yahoo, Sourceforge, Myspace, Microsoft and Google would all join the OpenID bandwagon. On the OpenID "provider" side of things, things were not so great however. Back when I started experimenting with this, it wasn't actually that easy to host your own OpenID provider, to allow your users to login to other OpenID-enabled sites. While there are a lot of different libraries to provide OpenID support in your application there was not (and still isn't) any software that gives you an OpenID provider and is dedicated to that. No official reference implementation either, other than the code from the one of the founding companies of OpenID, JanRain. There was an OpenID provider module in Drupal, but there was no official release done until a two beta releases in 2010.

Koumbit's community involvement It is about that time I started getting involved in the project. Koumbit needed an easy way for our workers to log into multiple sites without too much trouble, so I started looking at OpenID provider implementations. The lack of dedicated software somehow made the Drupal implementation a natural fit, and I started experimenting with the module, filing bugs and reviewing patches, as there were a lot of interoperability and design issues with the provider. After a while, walkah was kind enough to give me full access to the project and I started doing regular releases (every few months), culminating in today's official 1.0 release for both Drupal 6 and Drupal 7, which I am very proud and happy about. While I am here, I want to send many thanks to the people behind the Drupal 7 port: paranojik, mikecar, xamanu and testing from neizod. A lot of issues were also patched, reviewed and tested by numerous contributors that are unfortunately difficult to track down other than reading the commit log. So we at Koumbit have been running our own OpenID provider for a good while now, and it has the key advantage of being connected to our LDAP directory (with the D6 LDAP integration module so it is, in a sense, a central point for authentication at Koumbit. Very often people don't clearly understand what their LDAP password does, and I can easily remind them "it's the OpenID thing", or "try your password in the OpenID site" if they think they have forgotten it. So much goes on in the web browser these days that it makes everything much easier.

The problem with Drupal's OpenID provider However, it is kind of embarrassing that it took so long. More than 6 years after the OpenID protocol was written, we have a stable OpenID provider. Now, this may be because I am overly conservative about stable releases, but I don't think so: the OpenID provider module had, for a long time, serious interoperability issues with OpenID sites. There was also a clear lack of features on the OpenID provider, a gap that various third party modules, now documented on the OpenID provider homepage are now trying to fill. There is also serious security issues with the protocol itself. The complexity and lack of clarity of the OpenID specification have rendered my work quite difficult at time, as the protocol has significant ambiguities and no clear reference implementation. OpenID is unfortunately renowned to be vulnerable to phishing and while Koumbit's use of the HTTPS protocol alleviates some of those issues (especially the redirect session hijack), tools like sslstrip can bypass those protections easily. The privacy aspect is also problematic, as the user is usually not free to choose which information from his or her profile is divulged to the site they are logging into. Of course those issues may be overlooked if the security trade off is not worth it. In the case of Koumbit, we use OpenID on a lot of client sites, but usually nothing critical, and we train our users to pay attention to the HTTPS icon when login in their OpenID account.

Are there alternatives? But unfortuantely, the reality is that there are not really any alternatives to OpenID out there. Mozilla's new BrowserID initiative (now rebranded Persona, boy those guys sure love rebranding..) is interesting, but actually has even more flaws than OpenID itself. Not only is it not resolving the phishing problem, it actually creates new problems, including my personal favorite, trusting server-side Javascript code, which OpenID is not using that much. I should also mention the Shibboleth module which has a surprising number of installs recorded on Drupal.org, but that depends on the much more complicated Shibboleth federation, for which there is no server implementation for Drupal, from what I understand. Oauth, on the other end, is even more widely used, and has server-side components like Oauth Login Provider and Services 3.x. We could also simply give up and use our souls stored at Facebook, Google and Twitter for authentication everywhere. Maybe OpenID opened the door for those corporations to see that there is a real opportunity in leveraging user's profile to track their online activity. It is not something that OpenID was necessarily designed for but it's an idea that certainly permeates Facebook connect and "like" buttons or Twitter and Google online business strategies, where you are the product sold to advertisers. I personally believe we should keep the struggle for distributed and standard identity mechanisms for the web that respect users' privacy. With Drupal's OpenID Provider, it is possible to create a secure, privacy-aware identity provider, although I still have to perform that damn security audit. Experiments with those third party modules should also be encouraged and I welcome any feedback on those modules. While I do not plan to attend OpenID conferences or do significant development on the OpenID provider right now, I will certainly make sure Koumbit keeps its commitment to maintaining stewardship on the OpenID provider module in Drupal. I encourage people to try it out, submit issues and patches to make it even better. Long life to OpenID! PS: if people are interested in funding that security audit, I would be happy to perform this through my regular work at Koumbit.org, just file a request at services@koumbit.org.

15 November 2012

Dirk Eddelbuettel: Rcpp and the new R:: namespace for Rmath.h

We released Rcpp 0.10.0 earlier today. This post will just provide a simple example for one of the smaller new features -- the new namespace for functions from Rmath.h -- and illustrate one of the key features (Rcpp attributes) in passing. R, as a statistical language and environment, has very well written and tested statistical distribution functions providing probability density, cumulative distribution, quantiles and random number draws for dozens of common and not so common distribution functions. This code is used inside R, and available for use from standalone C or C++ programs via the standalone R math library which Debian / Ubuntu have as a package r-mathlib (and which can be built from R sources). User sometimes write code against this interface, and then want to combine the code with other code, possibly even with Rcpp. We allowed for this, but it required a bit of an ugly interface. R provides a C interface; these have no namespaces. Identifiers can clash, and to be safe one can enable a generic prefix Rf_. So functions which could clash such as length or error become Rf_length and Rf_error and are less likely to conflict with symbols from other libraries. Unfortunately, the side-effect is that calling, say, the probability distribution function for the Normal distribution becomes Rf_pnorm5() (with the 5 denoting the five parameters: quantile, mean, std.deviation, lowerTail, logValue). Not pretty, and not obvious. So one of the things we added was another layer of indirection by adding a namespace R with a bunch of inline'd wrapper functions (as well as several handful of unit tests to make sure we avoided typos and argument transposition and what not). The short example below shows this for a simple function taking a vector, and returning its pnorm computed three different ways: 
#include <Rcpp.h>
// [[Rcpp::export]]
Rcpp::DataFrame mypnorm(Rcpp::NumericVector x)  
    int n = x.size();
    Rcpp::NumericVector y1(n), y2(n), y3(n);
    for (int i=0; i<n; i++)  
        // the way we used to do this
        y1[i] = ::Rf_pnorm5(x[i], 0.0, 1.0, 1, 0);
        // the way we can do it now
        y2[i] = R::pnorm(x[i], 0.0, 1.0, 1, 0);
     
    // or using Rcpp sugar in one go
    y3 = Rcpp::pnorm(x);
    return Rcpp::DataFrame::create(Rcpp::Named("Rold")  = y1,
                                   Rcpp::Named("Rnew")  = y2,
                                   Rcpp::Named("sugar") = y3);
This example also uses the new Rcpp attributes described briefly in the announcement blog post and of course in more detail in the corresponding vignette. Let us just state here that we simply provide a complete C++ function, using standard Rcpp types -- along with one 'attribute' declaration of an export via Rcpp. That's it -- even easier than using inline. Now in R we simply do 
R> sourceCpp("mypnorm.cpp")
to obtain a callable R function with the C++ code just shown behind it. No Makefile, no command-line tool invocation -- nothing but a single call to sourceCpp() which takes care of things --- and brings us a compiled C++ function to R just given the source file with its attribute declaration. We can now use the new function to compute the probaility distribution both the old way, the new way with the 'cleaner' R::pnorm(), and of course the Rcpp sugar way in a single call. We build a data frame in C++, and assert that all three variants are the same: 
R> x <- seq(0, 1, length=1e3)
R> res <- mypnorm(x)
R> head(res)
      Rold     Rnew    sugar
1 0.500000 0.500000 0.500000
2 0.500399 0.500399 0.500399
3 0.500799 0.500799 0.500799
4 0.501198 0.501198 0.501198
5 0.501597 0.501597 0.501597
6 0.501997 0.501997 0.501997
R> all.equal(res[,1], res[,2], res[,3])
[1] TRUE
R>
This example hopefully helped to illustrate how Rcpp 0.10.0 brings both something really powerful (Rcpp attributes -- more on this another time, hopefully) and convenient in the new namespace for statistical functions.

14 November 2012

Petter Reinholdtsen: Debian Edu interview: Angela Fu

Here is another interview with one of the people in the Debian Edu and Skolelinux community. I am running short on people willing to be interviewed, so if you know about someone I should interview, Please send me an email. After asking for many months, I finally managed to lure another one of the people behind the German "IT-Zukunft Schule" project out from maternity leave to conduct an interview. Give a warm welcome to Angela Fu . :) Who are you, and how do you spend your days? I am a 39-year-old woman living in the very north of Germany near Denmark. I live in a patchwork family with "my man" Mike Gabriel, my two daughters, Mikes daughter and Mikes and my rather newborn son. At the moment - because of our little baby - I am spending most of the day by being a caring and organising mom for all the kids. Besides that I am really involved into and occupied with several inner growth processes: New born souls always bring the whole familiar system into movement and that needs time and focus ;-). We are also in the middle of buying a house and moving to it. In 2013 I will work again in my job in a German foundation for nature conservation. I am doing public relation work there. Besides that - and that is the connection to Skolelinux / Debian Edu - I am working in our own school project "IT-Zukunft Schule" in North Germany. I am responsible for the quality assurance, the customer relationship management and the communication processes in the project. Since 2001 I constantly have been training myself in communication and leadership. Besides that I am a forester, a landscaping gardener and a yoga teacher. How did you get in contact with the Skolelinux / Debian Edu project? I fell in love with Mike ;-). Very soon after getting to know him I was completely enrolled into Free Software. At this time Mike did IT-services for one newly founded school in Kiel. Other schools in Kiel needed concepts for their IT environment. Often when Mike came home from working at the newly founded school I found myself listening to his complaints about several points where the communication with the schools head or the teachers did not work. So we were clear that he would not work for one more school if we did not set up a structure for communication between him, the schools head, the teachers, the students and the parents. Together with our friend and hardware supplier Andreas Buchholz we started to get an overview of free software solutions suitable for schools. One day before Christmas 2010 Mike and I had a date with Kurt Gramlich in G tersloh. As Kurt and I are really interested in building networks of people and in being in communication we dived into Skolelinux and brought it to the first grammar schools in Northern Germany. For information about our school project you can read the interview with Mike Gabriel. What do you see as the advantages of Skolelinux / Debian Edu? First I have to say: I cannot answer this question technically. My answer comes rather from a social point of view. The biggest advantage of Skolelinux / Debian Edu I see is the large and strong international community of Debian Developers in the background which is very alive and connected over mailinglists, blogs and meetings. My constant feeling for the Debian Community is: If something does not work they will somehow fix it. All is well ;-). This is of course a user experience. What I also get as a big advantage of Skolelinux / Debian Edu is that everybody who uses it and works with it can also contribute to it - that includes students, teachers, parents... What do you see as the disadvantages of Skolelinux / Debian Edu? I will answer this question relating to the internal structure of Skolelinux / Debian Edu. What I see as a major disadvantage is that there is a gap between the group of developers for Debian Edu and the people who make the marketing, that means the people that bring Skolelinux to the schools. There is a lack of communication between these two groups and I think that does not really work for Skolelinux / Debian Edu. Further I appreciate that Skolelinux / Debian Edu is known as a do-ocracy. Nevertheless I keep asking myself if at some points a democracy or some kind of hierarchical project structure would be good and helpful. I am also missing some kind of contact between the Skolelinux / Debian Edu communities in Europe or on an international level. I think it would be good if there was more sharing between the different countries using Skolelinux / Debian Edu. Which free software do you use daily? On my laptop I am still using an Ubuntu 10.04 with a Gnome Desktop on. As applications I use Openoffice.org, Gedit, Firefox, Pidgin, LaTeX and GnuCash. For mails I am using Horde. And I am really fond of my N900 running with Maemo. Which strategy do you believe is the right one to use to get schools to use free software? I am really convinced that in our school project "IT-Zukunft Schule" we have developed (and keep developing) a great way to get schools to use Free Software. We have written a detailed concept for that so I cannot explain the whole thing here. But in a nutshell the strategy has three crucial pillars:

12 November 2012

Stefano Zacchiroli: Debian newcomer experience survey

In recent times we have worked quite a bit to improve the NM process, i.e. the process newcomers go through to become members of the Debian Project. As it happens, I've just read Nathan's recent post on his NM experience and I think it is a perfect example of the joining experience we are trying to offer to all newcomers. But examples, be them positive or negative, are only anecdotal. To evaluate a process one needs actual data and someone analyzing them, ideally with a scientific approach. This is why I'm happy to host below a guest blog post by Kevin Carillo, who is doing a pretty thorough scientific study about how newcomers join a wide range of Free Software projects, including KDE, OpenSUSE, GNOME, and Debian, of course! TL;DR: if you started contributing to Debian after January 2010, there's a survey for you; participating will help us improving the NM process even further. Kevin's guest blog post follows.
Newcomer experience in Debian and other FOSS communities - Survey My name is Kevin Carillo. I am a PhD student currently living in Wellington (New Zealand) and I am doing some research on Free/Open Source Software communities. If you have started contributing to the Debian project after January 2010 (within approximately the last 3 years), I would like to kindly request your help. I am interested in hearing from people who are either technical or non-technical contributors, and who have had either positive or negative newcomer experiences. The purpose of the research is to work out how newcomers to a FOSS community become valued sustainable contributors. The survey is online and will be available until Tuesday, 27 November, 2012. Inspiration from Debian New Member Debian is a successful community that keeps attracting new contributors and that relies on a very unique way to handle the integration of new contributors: the New Member process. The idea behind the NM process is that it is some sort of filtering procedure allowing to only retain the individuals who have the potential to become valued sustainable contributors in Debian. Within Debian, there is a lot of enthusiasm and pride around the NM process as it seems to be functioning pretty well but the question is: Is this really enough to ensure that Debian remains a healthy and growing community? How does it compare to the way newcomers are integrated in other large projects such as KDE, GNOME, or in other non-Linux related communities such as Mozilla? I have to admit that the Debian NM process has been among the main sources of inspiration that made me embark in this research project. I have kept being quite impressed when talking to people who had gone through the process as all of them came out of it with a real passion for the project and love for its community. When reflecting on the reasons why the NM process succeeds, I have a feeling it is some instance of ritualized socialization. In other words, barriers and initiation rituals that require some effort from newcomers, generate members with higher commitment and sense of identification towards the Debian community. What do newcomers really experience? The main assumption that motivated this project is that attracting new members has become crucial for a large majority of FOSS communities but this is not a sufficient condition to ensure the success and prosperity of a project. A proportion of a community's newcomers must contribute to the well-being and growth of the community. Keeping all that in mind, FOSS projects have thus to do a good job at "socializing" their newcomers and turning them into 'good' contributors. Doing a good job here means that FOSS projects shall ensure that they help generate citizenship-like behaviors from newcomers by designing appropriate newcomer programs and procedures. FOSS communities rely on a wide array of initiatives to facilitate the integration of newcomers but it seems like the other side of the coin is less understood: What do newcomers really experience? And how does this influence their contributions and actions within a project? How is this study going to help Debian? The data will help gain insights about the experience of newcomers within the Debian community. In addition, it will allow to understand how to design effective newcomer initiatives to ensure that Debian will remain a successful and healthy community. The dataset will be released under a share-alike ODbL license so that Debian contributors can extract as much value as possible from the data. Since this survey also involves other large FOSS projects such as Mozilla, KDE, Gnome, Ubuntu, Gentoo, OpenSUSE, and NetBSD, it will also be possible to compare practices across projects in order to identify what works from what does not work when facilitating the integration of newcomers. About the survey This survey is anonymous. The raw dataset of everything one fills in the survey will be released under the ODbL. Since all the questions but one are optional, one is free to control the amount of information they are giving away about themselves. I expect the survey to take around 20 minutes of your time. If you know members of the Debian community who you think would be interested in completing it, please do not hesitate to let them know about this research. I will post news about my progress with this research, and the results on my blog. Don't hesitate to contact me. --- Kevin Carillo

9 November 2012

Olivier Berger: Using a DreamPlug with Debian as an IPv4/IPv6 router for a home LAN behind a FreeBox

The goal of this article is to describe the setup I've made for my home LAN behind a FreeBox (French ISP Free.fr).

1 Principles
The FreeBox is configured :
  • in router mode (with a local IPv4 network on 192.168.0.x/24)
  • with IPv6 support activated (by convention, we'll use the 2a01:e35:1234:5678::/64 prefix below, to represent the IPv6 prefix provided by the FreeBox)
Behind the FreeBox, I have connected a DreamPlug computer (running Debian wheezy with a dual-boot setup as sold by NewIT) on its eth0 etherlink port. Behind the DreamPlug, I have connected a switch (to its eth1 ethernet port), on which the LAN's machines will be connected, on a dual IPv4/IPv6 network. The IPv4 will be on 192.168.4.0/24 (addresses for machines connected to the switch will be static) while the IPv6 will use the addresses auto-generated on the FreeBox's prefix. The DreamPlug will also provide a wifi network (in master mode, through its uap0 interface), only IPv4 on 192.168.3.0/24. On the DreamPlug, the outgoing IPv4 traffic will be masqueraded, while the IPv6 traffic will be bridged so that machines on the LAN can be reached from the Internet directly, passing through the eth0 eth1 bridge. The DreamPlug will also act as a filter/firewall to only allow selected traffic to reach the LAN machines (in particular for the IPv6, since the machines may be reached directly from the Internet otherwise).

1.1 Alternatives
There are a lot of howtos describing the setup of FreeBox + IPv6, but I couldn't find one that was exactly reflecting the kind of setup I have. Some dealt with a FreeBox setup in bridge mode (i.e. not the router mode), but in case the DreamPlug isn't so much reliable, or I've messed with the config, I'm tempted to keep the possibility to connect directly to the FreeBox. Others didn't use the same setup with a router distributing both wired and wifi networks. And note described exactly the setup for a DreamPlug running Debian wheezy and using ferm as a firewall. No, there's no NIH syndrom here ;)

2 DreamPlug router configuration

2.1 Interfaces configuration
The /etc/network/interfaces on the DreamPlug looks like the following (kudos to Dag Bertelsen's Setting up Dreamplug as a router with Shorewall firewall for the wifi configuration bits): 
# Note: Order of the interfaces declarations matters

auto lo
iface lo inet loopback

auto uap0
iface uap0 inet static
       post-up /usr/bin/uaputl sys_cfg_ssid dream-uAP-XXXX
       # Modus WPA2
       post-up /usr/bin/uaputl sys_cfg_protocol 32
       # WLAN password
       post-up /usr/bin/uaputl sys_cfg_wpa_passphrase "WHATEVER PASSWORD"
       # Set the Cipher
       post-up /usr/bin/uaputl sys_cfg_cipher 8 8
       # automatic channel
       post-up /usr/bin/uaputl sys_cfg_channel 0 1
       post-up /usr/bin/uaputl bss_start
       # we deliver addresses for the Wifi with DHCP
       post-up /usr/sbin/service udhcpd start
       address 192.168.3.1
       netmask 255.255.255.0

# Only IP6 traffic passes through the router between the Freebox on eth0 and the LAN on eth1
auto br0
iface br0 inet manual
      bridge_ports eth0 eth1
      bridge_maxwait 0
      pre-up ebtables -t broute -A BROUTING -p ! ipv6 -j DROP
      down ebtables -t broute -F

# Connected to the FreeBox, with DHCP in IPv4, masquerading outgoing IPv4 traffic
auto eth0
iface eth0 inet dhcp
        up iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
        down iptables -t nat -F
        post-up sysctl -q -w net.ipv4.ip_forward=1
iface eth0 inet6 auto

# Connected to the LAN with a static IPv4 address
auto eth1
iface eth1 inet static
      address 192.168.4.1
      netmask 255.255.255.0

2.2 DHCP server configuration
The DHCP config for udhcpd is as such (in /etc/udhcpd.conf): 
start 192.168.3.100
end 192.168.3.200
interface uap0
opt      lease  86400
opt     router  192.168.3.1
opt     subnet  255.255.255.0
opt     dns     192.168.3.1
opt     domain     localdomain
max_leases     101
lease_file     /var/lib/udhcpd.leases
auto_time       5

2.3 Firewall configuration
The firewall I use is ferm. Here's the /etc/ferm/ferm.conf configuration (YMMV). Thanks to Ren Garcia's R gles de firewall pour un pont IPv6 for inspiration : 
## Interfaces of the router

# Our LAN on a switch
@def $DEV_PRIVATE = eth1;

# The ISP router
@def $DEV_WORLD = eth0;

# Our Wifi LAN
@def $DEV_WIFI = uap0;


## IP networks of our LAN

# IPv4 (includes ethernet switch and wifi)
@def $NET_PRIVATE = 192.168.0.0/16;

# IPv6 (the prefix granted by the ISP)
@def $NET6_PRIVATE = 2a01:e35:1234:5678::/64;

# IPv6 Local link
@def $LINK6 = fe80::/10;

# Specific machines on the LAN
@def $MACHINE_MYDESKTOP = 2a01:0e35:1234:5678:1234:5678:9abc:def0;

# Special function to log dropped packets
@def &DROP() =  
  mod limit limit 1/second limit-burst 10 LOG log-prefix "$DOMAIN-$CHAIN: ";
  DROP;
 

# IPv4 iptables rules
table filter  
  chain INPUT  
  policy DROP;

  # connection tracking
  mod state state INVALID DROP;
  mod state state (ESTABLISHED RELATED) ACCEPT;

  # allow local packet
  interface lo ACCEPT;

  # respond to ping
  #proto icmp ACCEPT; 
  proto icmp icmp-type echo-request ACCEPT;

  # allow IPsec
  proto udp dport 500 ACCEPT;
  proto (esp ah) ACCEPT;

  # allow SSH connections from the private network and from some
  # well-known internet hosts
  #saddr ($NET_PRIVATE 123.45.67.89) proto tcp dport ssh ACCEPT;
  saddr $NET_PRIVATE proto tcp dport ssh ACCEPT;

  # we provide DNS and SMTP services for the internal net
  interface $DEV_PRIVATE saddr $NET_PRIVATE  
  proto (udp tcp) dport domain ACCEPT;
  proto tcp dport smtp ACCEPT;
   

  interface $DEV_WIFI  
  # DHCP
  proto udp dport 67:68 ACCEPT;
  proto (udp tcp) dport domain ACCEPT;

  proto udp dport netbios-dgm DROP;
   

  # LOG all dropped packets
  proto (udp tcp) &DROP();

   

  # outgoing connections are not limited
  chain OUTPUT policy ACCEPT;

  chain FORWARD  
  policy DROP;

  # connection tracking
  mod state state INVALID DROP;
  mod state state (ESTABLISHED RELATED) ACCEPT;

  # connections from the internal net to the internet or to other
  # internal nets are allowed
  interface $DEV_PRIVATE ACCEPT;
  interface $DEV_WIFI ACCEPT;

  # LOG all dropped packets
  proto (udp tcp) &DROP();
   
 

# IPv4 masquerade / NAT support

table nat  
  chain POSTROUTING  
  # masquerade private IP addresses
  saddr $NET_PRIVATE outerface $DEV_WORLD MASQUERADE;
   
 

## IPv6:

domain ip6  
  table filter  

  chain INPUT  
  policy DROP;

  # Internal traffic authorized
  saddr $NET6_PRIVATE ACCEPT;
  saddr $LINK6 ACCEPT;

  # accept multicast
  daddr ff00::/8 ACCEPT;

  interface lo ACCEPT;

  # For public traffic
  # Only DNS resolution + NTP acceped for UDP
  interface br0  
  proto udp  

  sport domain ACCEPT;

  dport ntp ACCEPT;

  &DROP();
   

  # TCP on ports < 1024 forbidden except for SSH
  proto tcp  

  dport ssh ACCEPT;

  dport 0:1023 DROP;
   

  # Accepted established traffic
  mod state state INVALID DROP;
  mod state state (ESTABLISHED RELATED) ACCEPT;

  # incoming connections forbidden
  proto tcp syn DROP;

  # Accept ping
  proto icmpv6 ACCEPT;
   

  proto (udp tcp) &DROP();
   
  chain FORWARD  
  policy DROP;

  # All local traffic accepted
  saddr $LINK6 ACCEPT;

  # All ping accepted unless coming from internet
  saddr ! $NET6_PRIVATE proto icmpv6 icmp-type echo-request DROP;
  proto icmpv6 ACCEPT;

  # Allow SSH to specific machines
  proto tcp daddr $MACHINE_MYDESKTOP dport ssh ACCEPT;

  # Accept established traffic
  mod state state INVALID DROP;
  mod state state (ESTABLISHED RELATED) ACCEPT;

  # Accept outgoing traffic
  saddr $NET6_PRIVATE ACCEPT;

  # Accept local outgoing traffic
  saddr ::1 ACCEPT;

  proto (udp tcp) &DROP();
   
   
 

# ebtables
domain eb chain FORWARD  
  policy DROP;

  # Only accept bridged traffic for IPv6
  logical-in br0 logical-out br0   
  #ebtables -t broute -A BROUTING -p ! ipv6 -j DROP
  proto IPv6 ACCEPT;
2.4 DNS config

The DreamPlug will run bind9 to cache the DNS for the wifi network (default config is OK out of the box).

3 Configuration for the LAN machines

3.1 Connection with Wifi
Machines connected with the WPA wifi network should get an IPv4 only address via DHCP on the 192.168.3.0/24 network. There's nothing to be configured specifically.

3.2 Ethernet connection on the switch
Machines connected via ethernet cables to the switch can run either with IPv4 config, in which case they should be configured statically for the 192.168.4.0/24 network, using the router on 192.168.4.1 as a gateway, or with IPv6 auto config. On wheezy, the /etc/network/interfaces will look like : 
auto eth1
iface eth1 inet static
  address 192.168.4.42
  netmask 255.255.255.0
  gateway 192.168.4.1
  dns-nameservers 212.27.40.241 212.27.40.240

iface eth1 inet6 auto

4 To do list

The setup works fine, but there are a few things I'd like to improve :
  • running a regular Debian distro on the DreamPlug (the kernel used at the moment is patched for the wifi driver in master mode support), possibly using only Free Software.
  • using subnets for the IPv6 machines (but that seems not exactly obvious, requiring NDP proxying and stuff), so that the privacy is improved (the IPv6 auto generated global addresses depend on the MAC addresses hmmm probably not optimal).
  • adding more stuff to the DreamPlug router so that it becomes a full fledged FreedomBox.

5 References

Date: 2012-11-09 16:30:34 CET

8 November 2012

Jo Shields: Evil, or why Douglas Crockford is harmful to Free Software

Yesterday I received a new serious bug report against Mono in Debian. For those not in the know, serious severity is release critical, and can trigger removal of a package from the distribution in order to make a shipping release (e.g. if Debian is in deep freeze, as is the case right now). Bug 692614 relates to a single source file, ./mcs/class/System.Web.Extensions/System.Web.Script.Serialization/JsonDeserializer.cs, which is part of the System.Web.Extensions assembly. This file was written in 2008 by an upstream Mono developer, and is based on some code from json.org specifically JSON_parser.c and JSON_checker.c json.org code carries a non-standard license. Specifically, it s MIT/X11 with an added clause: The Software shall be used for Good, not Evil. I ve had some discussions with Mono upstream, and they believe that they have resolved the matter to their satisfaction their solution will make its way into Debian soon. But in the meantime, let s talk about the clause. The Free Software Foundation s Freedom 0 reads:
A program is free software if the program s users have the four essential freedoms:
  • The freedom to run the program, for any purpose (freedom 0).
The Debian Free Software Guidelines and Open Source Initiative s Open Source Definition clause 6 reads:
No Discrimination Against Fields of Endeavor The license must not restrict anyone from making use of the program in a specific field of endeavor. For example, it may not restrict the program from being used in a business, or from being used for genetic research.
It s fairly clear that the JSON.org license clause goes against both of these, making any piece of software using that license neither Open Source nor Free Software. It is not distributable by any organization which mandates Freedom for its users not in Debian, not in Fedora, not on Google Code. Anybody who cares about their users will reject a clause like this, because it has awful chilling effects. Think it s funny? It really isn t. Who can use the code without being at risk of a lawyer knocking on their doors? Can the Catholic Church? The ACLU? Republicans? Democrats? Military contractors? Genetic engineers? Big pharma? Petrochemicals firms? Communists? Fascists? Mono developers? WikiLeaks? Without a clear definition of good and evil , people need to seriously consider whether they are safe to use the code because if the developer and users interpretations of the terms differ, there could be hell to pay. Would you want to ship a hardware device let s say you re making a smart TV and have some developer of a library send his lawyers around to tell you sorry, TV rots kids brains, it s clearly evil to get your entire distribution channel shut down by injunction? It s not an oversight, kids. Upstream are well aware of the pain this childishness causes anyone who takes software licenses seriously. They giggle about it at conferences, like tweenagers who think they ve one-upped the adults in the room. This non-Free license is intended to mock people who take licensing seriously. In fact, one could easily categorise efforts to pollute the Free Software ecosystem with fake, non-Free software as evil, which would mean all JSON.org code fails to comply with its own license, due to shipping with its license (Inception, anyone?). Before the jokers in the room claim that this kind of problem is deserved by Mono, let s take a look at all the software in Debian which Douglas Crockford endangers with his childishness. PHP? OwnCloud? jQuery? You think Debian serves its users well by pulling jQuery from the next release in order to serve the ego of a man behaving like an eleven year old? It is also interesting to note that the author of this do no evil clause works at PayPal. Read that twice, go and repair your irony meters, then come back. In conclusion: thank you so VERY much Doug Crockford for making the world of Free Software measurably worse.

Next.

Previous.