Postman was on a kind mood yesterday. My OpenPhoenux GTA04 arrived! I had my newer Neo FreeRunner upgraded via the service from Golden Delicious.

First, something rare to behold in phones nowadays - Made in Bavaria:






As a sidenote, also rare now - my Nokia N9 is one of the last phones that had this:

It's sad that's now a thing of the past for both hardware assembly and software! But that's just a hint for new companies to step in.

But back to GTA04 - a quick boot to the pre-installed Debian w/ LXDE:






And then as a shortcut unpacked and booted into QtMoko (well, it's also Debian) instead to test that phone functionality also works - and it does:



Next up: brewing something of my own!

Software

I've been using redis for a while now. It is a fast in-memory storage system which offers persistence (unlike memcached), as well as several primitive data-types such as lists & hashes. Anyway it crossed my mind that I don't have a backup of the data it contains, so I knocked up a simple script to dump the contents in plain-text:

• redis-dump

In other software-news I've had some interesting and useful feedback and made two new releases of my slaughter sysadmin tool - it now contains a wee test suite and more robustness.

Hardware

I received an email last night to say that my Raspberry PI has shipped. Ordered 24/05/2012, and dispatched 12/10/2012 - I'd almost forgotten about it. My plan is to make it a media-serving machine, SNES emulator, or similar. Not 100% decided yet. Finally I've taken the time to repaint my office. When I last wrote about working from home I didn't include pictures - I just described the process of using a "work computer" and a "personal computer". So this is what my office used to look like. As you can see there are two machines and a huge desk. With a few changes I now have an office which looks like this - the two machines are glued-together with a KVM. and I have much more room behind it for another desk, more books, and similar toys. Additionally my dedication is now enforced - I simply cannot play with both computer as the same time. The chair was used to mount the picture - usually I sit on a kneeling chair, which is almost visible. What inspired the painting? Partly the need for more space, but mostly water damage. I had a leaking ceiling. (Local people will know all about my horrible leaking roof situation).

The end?

Thanks to a blog post by Eddy Petri or, I became aware of yet another "alternative medicine" company using legal intimidation tactics to scare off critics. According to the originating blog post about the detox "cure" ColonHelp and its producers Zenyth Pharmaceuticals actions, the producer sues Wordpress to get rid of the critical information. To check if the story was for real, I contacted Automattic, the company behind wordpress.com, and they reply was "We can confirm that Zenyth is seeking a court order against WordPress / Automattic. However, we don't believe the Terms of Service have been violated in this matter". The story seem to be simply that a blogger checked the scientific foundation for a popular health product in Rumania, ColonHelp, and reported that there was no reason at all to believe it improved the health of its users. This caused the company behind the product, Zenyth Pharmaceuticals, to use legal intimidation to try to silence the critic, instead of presenting its views and scientific foundation to argue its side. This is the usual story, and the Zenyth Pharmaceuticals company deserve everyone to know how it failed to act properly. Lets hope the Streisand effect can make it rethink its strategy. What is the harm, you might think. I suggest you take a look at a list of victims of detoxification.

As many of you know, I love Ubuntu. I ve given years of my life for Ubuntu. I ve served as much as anyone s expected to, and long upheld the ideology and technical motivations behind Ubuntu as second to none. Until recently. Now, with advertisements in the default DE (and don t tell me they re not, when I search for term and get The Terminator on DVD, it s a ad if you care so much about whatever binding to search applications, why not disable ads by default, and let super-s do your shopping ), and now, begging for a handout when people Download Ubuntu without an option to donate to Debian, which composes over 70% of the Desktop, I get upset. I feel cheated, and I m not even involved on a day-to-day anymore. I m ashamed of what s going on now, and I hope we find a better way to serve our users.

Last weekend I was in Southwest Harbour, Maine again for the annual LTSP hackfest (called By The Sea). It s a fun and productive event and as always it s been good catching up with LTSP folk, even though we were missing Oliver, Alkis and Vagrant. Here is a summary of what I can recall from the discussions of the weekend Recent Happenings in LTSP

• New LTSP Website After a really long time, LTSP finally has a new website. It s a big improvement, there s a new success stories page and the wiki is now self-hosted.
• LTSP PNP - This is a new package (ltsp-pnp) that can safely be installed on existing machines to allow a user to log into an LTSP server using LDM.

Going Forward Social

• On Friday, Chuck Liebow took us out for a boat ride around the harbour on the Sea Princess

• On Saturday night we had the big lobster dinner as per BTS tradition (I had steak since I m not a big seafood person). We were to busy eating and telling stories to be taking any pictures
• And last but certainly not least, I finally met Eric Harrisson, who did a lot of work on K12-LTSP in the Portland schools district. He has an awesome hobby where he builds guitars out of well, almost anything he can find.
  He brought along one of the experimental guitars he slapped together recently and told me all about how it s put together. Even more amazing, he said I could have it! I was planning on getting an acoustic guitar anyway so I m very thrilled about it. Not only is it a completely unique guitar but it sounds great too. I m going to have to think of something to make back for him!

It was great seeing everyone again. I first started using LTSP around 9 years ago and never imagined back then that I d get to meet the people behind it. Ron Colcernian sourced us some really cool LTSP tops that you can see us wearing in the group photo. I hope to get to BTS again next year!

I am a big fan of Ruby on Rails: it does a lot of things, and it does most of them pretty well. When starting a new web project, it's the first thing I would reach for: most of the time, your problem is going to be figuring out a good product/market fit, or whipping up some internal tool without wasting a lot of programmer time. Once you've got a firm grasp of the problem, then maybe you can consider optimizing. Who cares if you do something that no one wants really really fast? However, Ruby on Rails is not beautiful in terms of being particularly fast or lightweight. No complaints from me: most of the time, I'm happy to have something that does so much for me, leaving me to work on the actual problem at hand. Once in a while, though, you do need fast and relatively lightweight, and that space has been getting more interesting over the past few years, at least in terms of the web. First of all, technologies like "Comet", utilizing web sockets or some other always-on connection are becoming more common, where a socket with the server remains open in order to quickly exchange data from the server to the client - and back. That seems to be a poor fit for something like Rails, where it can tie up a lot of resources if one isn't careful. And while computing costs continue to decline, no one minds getting more for less in terms of what their server can do. Furthermore, with frameworks like Backbone.js, pushing more and more code to the client, the server can afford to be a bit simpler and do less, so it may as well be snappy to boot. Java has long been fairly popular for "heavy lifting" types of applications, partially because it does end up being reasonably fast. But it's not something I've ever had much fun using and is usually kind of wordy, and makes you feel like you need a crew of people in Palo Alto, one in Bangalore, and one in Stockholm just to churn out all the code. And it certainly is no lightweight in terms of memory either. So... while it can certainly do pretty much anything you need, I don't see it as being the strongest player when someone needs "real time web" code, and needs it to be reasonably light weight. Ruby, outside of standard Rails stuff like Passenger, seems to offer some interesting possibilities for this kind of work, like Reel but they don't seem to have the traction other solutions do. Python is in a similar situation: the Twisted framework has been around for a while, and while it has some success stories, never seems to have really 'caught fire'. Neither of these languages was built for "concurrency" from the ground up, and that seems to have, to date, inhibited people from using them extensively for this kind of job. My gut feeling is that the need for speed and "concurrency" (or at least handling a lot of concurrent users) will drive adoption of languages heretofore not so popular on the server. Let's have a look at them. Erlang: This is by no means a new language, having been developed in Ericsson in the late 1980ies; and it implements several interesting concepts. First and foremost, concurrency is handled in the form of many small "processes", which are not actually Unix processes at all, but processes internal to the Erlang VM. The Erlang system contains a scheduler that allocates resources to all of these processes, so even if one of them dies or behaves badly, it's not a problem: the system as a whole can continue to function well. The way Erlang is built, the scheduler is preemptive: the internal "processes" don't need to yield to let other processes run. Beyond the scheduler and simple processes, Erlang gives you the tools to create elaborate trees of supervisors and workers that are quite robust to failures in any one portion of the system, as well as giving you the tools to set up a system to run on multiple, distributed computers. This kind of thinking is necessary when you write applications, such as phone switches, where downtime is really, really not ok. Erlang processes comunicate almost exclusively via message passing, meaning that state is not shared. Altogether, this makes for a fast, rock solid system that can easily handle thousands of concurrent connections without breaking a sweat. The computing world being what it is though, Erlang is likely to be more of a Lisp or Smalltalk: it's a trailblazer that did many things years before their time in other languages, but I don't see it as ever quite catching on amongst 'the masses'. It has a wonky syntax, it's a functional programming language, and because it is used in environments where too much experimentation is not good, it does not have a lot of room to break with its own past and innovate in terms of the language itself: it's slow to change and improve, even where the need to is clearly perceived. Node.js: is the opposite, in some ways: being based on Javascript, it draws on a huge number of potential programmers - orders of magnitude more than Erlang. And thanks to design decisions enforcing the use of asynchronous code and callbacks for anything that could block, it deals quite well with concurrent connections, even if the language and libraries don't really give you much in terms of "true" concurrency. This is a simple model that works pretty well, even if, theoretically, all it would take is one "while (1)" loop in a callback to block the entire system. In practice, this doesn't seem to be a big problem, though. More of a problem is writing maintainable code, when everything is a callback. Keeping the network of callbacks straight can be a bit of a chore, and is probably not an optimal model in terms of programmer productivity. That said, people seem to do make due with it, although Node is young enough that we haven't seem projects that are 5 or 6 years old and maintained by people who didn't write them. One of the advantages of being such a popular language is that lots of people have a strong interest in seeing Javascript being very, very performant. That need begat the V8 Javascript engine, from Google, which they were kind enough to release as free software. So one of the advantages that Node has is that the underlying implementation is extremely fast for a dynamic language. For many people who know Javascript, picking up Node.js is also an easy choice, even though people used to browser side programming with Javascript will have to adjust their way of thinking to succesfully tackle server projects. Go language: this is an interesting one, written by some luminaries who work at Google. It can best be described as something akin to C with some updated features, such as garbage collection, that make it more suitable for working on large projects, where things like memory leaks are going to make life very frustrating. It has the feel of a "real language, for real programmers" in that it doesn't stray far too far from what people are used to - it's not going to cause the "what the hell is this?!" reactions that Erlang might in more close-minded circles. Between the big company backing it, and the approachable syntax and concepts, Go looks like it has a good shot at the mainstream. Where it gets interesting is their concurrency model, which is apparently based on something called "communicating sequential processes" which, superficially at least, looks like it has some things in common with Erlang's "actor model" of concurrency. Under the hood, Go apparently hives off its "goroutines" to different OS level threads, but does not have a preemptive scheduler like Erlang: http://code.google.com/p/go/issues/detail?id=543 - although according to this, they may change that in the future. I'm not enough of a computer science guy to comment much on the details of CSP vs Actors, but both seem like valid models with strengths compared to trying to keep threads straight, which always seems to be a source of problems for programmers. Conclusions So, what's actually going to happen? I see Node.js as the clear front runner. It takes a worse-is-better approach that seems to work well enough as people get started. If they encounter difficulties later, they can always rewrite in something else, if needs be, but by "luring people in", Node.js has gathered a large group of users who continue, in turn, to churn out more code for use with the system, making it more attractive to new users. Programming languages are not winner-take-all markets though, so perhaps there is room for a few more languages to have decent followings in this space. Hopefully the competition will lead to ever better tools for those of us utilizing them! What do you think?

Just posted to d-d-a, bits from the DPL for Septemer 2012.

---

Heya,
here is the periodic report of DPL activities, this time related to September 2012 (posted here with some delay, as I've been traveling oversea on behalf of Debian for the past few days). Help needed: Google Code-In The Google Code-In (GCI) initiative is about to start again. We don't have a great track record of participating into it, and that's a pity. The initiative revolves around small tasks that other Free Software projects have come to call "easy hacks", showing how effective they're in attracting new volunteers. To participate, we need both mentors and admins. If you're interested see Ana's call for help and please volunteer on the soc-coordination list. Related to this, you've probably seen the report of our participation in GSoC 2012. I'd like to thank all students, mentors, and admins for a very well organized edition. Logo relicensing & other assets The DFSG-relicensing of the Debian logo, which I've mentioned in recent updates, has now happened. The so called Debian "Open Use" logo, in both variants (with and without "Debian") is now dually licensed under LGPL3+ / CC-BY-SA 3.0. The change has been made effective with a resolution of the SPI Board of Directors. Note that the so called "official" logo, which we seem to use very little, is still released under the terms of a non-free license. Discussion is ongoing on -project to stop promoting it. If you're using it or if you've arguments in favor of against that, please participate in the discussion. I've got from SFLC an updated draft of the new proposed trademark policy, implementing most of the changes requested during related August discussions on -project. I'll post it for review there shortly. The Japanese Trademark Office (JPO) has sent us the final confirmation that trademark transfer request (from individuals Debian contributors in the area to SPI) has happened. We've paid the needed fee of 457 CHF to conclude the transaction. We finally settled a domain transfer agreement with the current owner of the debian.eu. Technically, the domain will be transferred to FFIS in the coming weeks. Core teams (non-)news Quite some core teams are in the process of changing their formations these weeks, but most are still pending proper delegation (or equivalent). So here are just a few "teasers":

• as you might have read in the ftp-master sprint report, we have a new ftp-master: congrats Ansgar! (delegation update pending, but no suspense here)
• we have a new front-desk member (delegation pending + suspense)
• we have a new debian policy team (delegation pending + suspense)
• the tech-ctte is in the process of selecting a new candidate to fill the vacant seat. I've nominated a couple of candidates myself, and I'm now waiting for tech-ctte recommendations as per Constitution 6.2. On related news, the tech-ctte has now dealt with the conflict on Python interpreter maintenership, which was also the longest standing issue brought before them. Having invested quite some energy in that issue myself, I'm delighted to see it finally addressed.

I hope to finalize all the pending tasks above this month. Future events I've spent quite some time to plan my participation in events I'll be attending during October on behalf of Debian, such as the ACM Reflections conference in Urbana-Champaign (at the time of writing, this has already happened), the Ubuntu Developer Summit in Copenhagen, and the LinuxDay event in Turin. Cheers.

---

PS the day-to-day activity log for September 2012 is available at the usual place master:/srv/leader/news/bits-from-the-DPL.txt.201209

Direct harassment and overt sexism are certainly a significant part of why the gender ratios in the Linux community[1] are so skewed, but they're not the whole story. Part of feeling comfortable with a community is the knowledge that those in positions of power can be relied upon to engage in appropriate action when undesirable situations arise - no matter how compelling your code of conduct, no matter how detailed your diversity policy, if someone contravenes them and nothing happens, those documents are worthless and your community is unwelcoming. You can't rely on enforcement unless the community can trust its leadership.

Setting a good example in terms of behaviour is obviously vital, but it's not sufficient. Leaders who engage in sexist behaviour or who harass community members are clearly untrustworthy and undermine any attempts the rest of the community may be making. Failing to step in when they see examples of unacceptable behaviour is as bad. But less obvious is that it's possible to destroy that community trust without clearly contravening those community standards.

An example of this is Todd Akin, a candidate for the upcoming US Senate elections. In August he justified his support for making abortion illegal even in cases of rape, on the basis that women can't become pregnant through being victims of "legitimate rape". Of course, this had little to do with his fiscal or wider social policies, positions that are more directly relevant to most of his potential constituents, and in an election year that's primarily about the economy and healthcare it might be expected that a single issue would have little effect on the election standings. Instead, there was a huge uproar and a previously safe victory has now turned into a real chance of a loss.

Why? A significant part of it is because many people now have difficulty believing that they can trust him to represent them. If he understands women so little that he's willing to make entirely spurious justifications for his positions, how could any woman trust him to consider any other problems they may face? If he's willing to use fake science to back himself up, how could anyone trust him to consider any issue involving science? Through a single statement to a journalist he demonstrated to many that he was unsuited to be their representative in government. Many people who would otherwise have voted for him simply don't feel that they can rely on him to be there for them. Whether he wins his election or not, that distrust is going to remain and it's going to compromise his ability to work with his constituents.

It's the same in technical communities. If a member of a project's technical leadership frequently and loudly expresses their disdain for Python coders, anyone advocating for increased use of Python in that project is unlikely to feel that they can get unbiased opinions from that leader. If the project as a whole is uninterested in adopting Python then that's probably for the best, but if the rest of the project is open to using Python then there's a problem. People working with Python are less likely to join the project, and perhaps the project will lose out as a result.

But it's worth remembering that technical communities are still communities. If a member of a project's technical leadership vociferously argues that slavery benefited minority groups in the long run, members of those minority groups are going to feel that they're not going to be well represented by that leader. If they blog about how homosexuality is a lifestyle choice then homosexuals are unlikely to flock to the project. Likewise, if they downplay the prevalence or impact of rape, women are going to feel marginalised and find something better to do with their time.

It's impossible to separate these things. No matter how technically competent a community leader is, no matter how much code review they perform or how much mentorship they provide, if they're expressing unacceptable social opinions then they're diminishing the community. People I know and respect have left technical communities simply because people in positions of responsibility have engaged in this kind of behaviour without it causing them any problems.

We shouldn't be willing to give people a pass simply because they aren't actually groping anyone or because they're not members of the KKK. Those who drive people away from the community on the basis of race, gender or sexual orientation deserve vocal condemnation, and if they're unwilling to change their behaviour then the community should instead act to drive them away.

We're pretty bad at that in the Linux world at the moment. Too many people have behaved in this way and are left with positions of power or responsibility. There's no easy solution to the problem, but the Ada Initiative is continuing to work to improve awareness and work with communities to reduce the alienation that results from this kind of behaviour. If you want this to be anything other than a straight white male dominated community, you should give them money.

[1] (and computing in general, though to a lesser extent)

comments

Release 0.2.6 of RProtoBuf arrived on CRAN earlier this morning. RProtoBuf provides GNU R bindings for the Google Protobuf data encoding library used and released by Google. This release was once more driven largely by Murray whom we have now added among the list of authors of the package too. The NEWS file entry follows below:

Changes in version 0.2.6 (2012-10-04)

• Applied several more patches by Murray to
  • correct '_' and '__' mismatches in wrapper calls
  • update a few manual pages for style, and add examples
  • fix a bug where NAs were silently treated as TRUE for logical/bool types
  • fix a bug that caused crashes when adding vectors to optional fields
  • fix bugs in readASCII that returned empty protocol buffers when the file or connection could not be opened
  • distinguish between non-existant and not-set fieldswith has() by returning NULL in the former case.
  • fix a bug that caused non-deterministic behavior when setting a repeated message field in a protobuf to a single Message.
  • add unit tests for all of the above.
• Added Murray to Authors: field in DESCRIPTION
• Removed old and unconvincing example on RProtoBuf for storage and serialization in an imagined HighFrequencyFinance context

CRANberries also provides a diff to the previous release 0.2.5. More information is at the RProtoBuf page which has a draft package vignette, a 'quick' overview vignette and a unit test summary vignette. Questions, comments etc should go to the rprotobuf mailing list off the RProtoBuf page at R-Forge.

I was working on writing tests for gnome-settings-daemon a week or so ago, and finally got blocked on being unable to set up upower/ConsoleKit/etc. the way I need them. Also, doing so needs root privileges, I don t want my test suite to actually suspend my machine, and using the real service is generally not suitable for test suites that are supposed to run during make check , in jhbuild, and the like these do not have the polkit privileges to do all that, and may not even have a system D-Bus running in the first place. So I wrote a little test_upower.py helper, then realized that I need another one for systemd/ConsoleKit (for the system idle property), also looked at the mock polkit in udisks and finally sat down for two days to generalize this and do this properly. The result is python-dbusmock, I just released the first tarball. With this you can easily create mock objects on D-Bus from any programming language with a D-Bus binding, or even from the shell. The mock objects look like the real API (or at least the parts that you actually need), but they do not actually do anything (or only some action that you specify yourself). You can configure their state, behaviour and responses as you like in your test, without making any assumptions about the real system status. When using a local system/session bus, you can do unit or integration testing without needing root privileges or disturbing a running system. The Python API offers some convenience functions like start_session_bus() and start_system_bus() for this, in a DBusTestCase class (subclass of the standard unittest.TestCase ). Surprisingly I found very little precedence here. There is a Perl module, but that s not particuarly helpful for test suites in C/Vala/Python. And there is Phil s excellent Bendy Bus, but this has a different goal: If you want to thoroughly test a particular D-BUS service, such as ensuring that it does the right thing, doesn t crash on bad input, etc., then Bendy Bus is for you (and python-dbusmock isn t). However, it is too much overhead and rather inconvenient if you want to test a client-side program and just need a few system services around it which you want to set up in different states for each test. You can use python-dbusmock with any programming language, as you can run the mocker as a normal program. The actual setup of the mock (adding objects, methods, properties, etc.) all happen via D-Bus methods on the org.freedesktop.DBus.Mock interface. You just don t have the convenience D-Bus launch API. The simplest possible example is to create a mock upower with a single Suspend() method, which you can set up like this from Python:

import dbus
import dbusmock
class TestMyProgram(dbusmock.DBusTestCase):
[...]
    def setUp(self):
        self.p_mock = self.spawn_server('org.freedesktop.UPower',
                                        '/org/freedesktop/UPower',
                                        'org.freedesktop.UPower',
                                        system_bus=True,
                                        stdout=subprocess.PIPE)
        # Get a proxy for the UPower object's Mock interface
        self.dbus_upower_mock = dbus.Interface(self.dbus_con.get_object(
            'org.freedesktop.UPower', '/org/freedesktop/UPower'),
            'org.freedesktop.DBus.Mock')
        self.dbus_upower_mock.AddMethod('', 'Suspend', '', '', '')
[...]
    def test_suspend_on_idle(self):
        # run your program in a way that should trigger one suspend call
        # now check the log that we got one Suspend() call
        self.assertRegex(self.p_mock.stdout.readline(), b'^[0-9.]+ Suspend$')

This doesn t depend on Python, you can just as well run the mocker like this:

python3 -m dbusmock org.freedesktop.UPower /org/freedesktop/UPower org.freedesktop.UPower

and then set up the mocks through D-Bus like

gdbus call --system -d org.freedesktop.UPower -o /org/freedesktop/UPower \
      -m org.freedesktop.DBus.Mock.AddMethod '' Suspend '' '' ''

If you use it with Python, you get access to the dbusmock.DBusTestCase class which provides some convenience functions to set up and tear down local private session and system buses. If you use it from another language, you have to call dbus-launch yourself. Please see the README for some more details, pointers to documentation and examples. Update: You can now install this via pip from PyPI or from the daily builds PPA. Update 2: Adjusted blog entry for version 0.0.3 API, to avoid spreading now false information too far.

We've been thinking about getting a dog for ages, but it's been difficult to work out the logistics when we've been travelling so much. Well, today's the day!

Pepper is a Parson Russell Terrier, and she's 3 years old. We just collected her from the lovely people at Wood Green Animals Shelter. Her previous owner passed away recently, so she was looking for a new home. We've just brought her home and things are going great - she's very well behaved (so far!) and is loving a new place with lots of fuss and cuddles.

I ve just read Diego s response to an ill-informed NYT article about data-center power efficiency [1]. This makes me wonder, how much server use does each person have? Google Almost everyone uses Google, most of us use it a lot. The main Google product is also probably the most demanding, their search engine. In a typical day I probably do about 50 to 100 Google searches, that sounds like a lot, but half of them would probably be for one topic that is difficult to find. I don t think that I do that many Google searches because I generally know what I m doing and when I find what I need I spend a lot of time reading it. I m sure that many people do a lot more. Each Google search takes a few seconds to complete (or maybe more if it s an image search and I m on a slow link), but I think it s safe to assume that more than a few seconds of CPU time are involved. How much work would each Google search take if performed on a single system? Presumably Google uses the RAM of many systems as cache which gives a result more similar to a NUMA system than one regular server working for a longer time so there is no way of asking how long it would take to do a Google search with a single server. But I m sure that Google has some ratio of servers to the rate of requests coming in, it s almost certainly a great secret, but we can make some guesses. If the main Google user base comprises people who collectively do an average of 100 searches per day then we can probably guess at the amount of server use required for each search based on the number of servers Google would run. I think it s safe to assume that Google doesn t plan to buy one server for every person on the planet and that they want to have users significantly outnumbering servers. So even for core users they should be aiming to have each user only take a fraction of the resources that one server adds to the pool. So 100 searches probably each take more than 1 second of server use. But they almost certainly take a lot less than 864 seconds (the server use if Google had one server for every 100 daily requests which would imply one server for each of the heavier users). Maybe it takes 10 seconds of server use (CPU, disk, or network whichever is the bottleneck) to complete one search request. That would mean that if the Google network was at 50% utilisation on average then they would have 86400*.5/10/100 == 43 users per server for the core user base who average 100 daily requests. If there are 80M core users that would be about 2M servers, and then maybe something like another 4M servers for the rest of the world. So I could be using 1000 seconds of server time per day on Google searches. I also have a Gmail account which probably uses a few seconds for storing email and giving it to Fetchmail, and I have a bunch of Android devices which use Google calendars, play store, etc. The total Google server use on my behalf for everything other than search is probably a rounding error. But I could be out by an order of magnitude, if it only took 1 second of server use for a Google search then I would be at 100 server seconds per day and Google would only need one server for every 430 users like me. Google also serves lots of adverts on web sites that I visit, I presume that serving the adverts doesn t take much resources by Google standards. But accounting for it, paying the people who host content, and detecting fraud probably takes some significant resources. Other Big Services There are many people who spend hours per day using services such as Facebook. No matter how I try to estimate the server requirements it s probably going to be fairly wrong. But I ll make a guess at a minute of server time per hour. So someone who averages 3 hours of social networking per day (which probably isn t that uncommon) would be using 180 seconds of server time. Personal Servers The server that hosts my blog is reasonably powerful and has two other people as core users. So that could count as 33% of a fairly powerful server in my name. But if we are counting server use per USER then most of the resources of my blog server would be divided among the readers. My blog has about 10,000 people casually reading it through Planet syndication, that could mean that each person who casually reads my blog has 1/30,000 of a server allocated to them for that. Another way of considering it is that 10% of a server (8640 seconds) is covered by me maintaining my blog and writing posts, 20% is for users who visit my blog directly, and 3% is for the users who just see a Planet feed. That would mean that a Planet reader gets 1/330,000 of a server (250ms per day) and someone who reads directly gets 1/50,000 of a server (1.72s per day) as I have about 10,000 people visiting my blog directly in a month. My mail server which is also shared by a dozen or so people (maybe that counts as 5% of a server for me or 4320 seconds per day). Then there s the server I use for SE Linux development (including my Play Machine) and a server I use as a DNS secondary and a shell server for various testing and proxying. Other People s Servers If every reader of a Planet instance like Planet Debian and Planet Linux Australia counts as 1/330,000 of a server for their usage of my blog, then how would that count for my own use of blogs? I tend to read blogs written by the type of people who like to run things themselves, so there would be a lot of fairly under-utilised servers that run blogs. Through Planet Debian and Planet Linux Australia I could be reading 100 or more blogs which are run in the same manner as mine, and in a typical day I probably directly visit a dozen blogs that are run in such a manner. This could give me 50 seconds of server time for blog reading. Home Servers I have a file server at home which is also a desktop system for my wife. In terms of buying and running systems that doesn t count as an extra server as she needs to have a desktop system anyway and using bigger disks doesn t make much difference to the power use (7W is the difference between a RAID-1 server and a single disk desktop system). I also have a PC running as an Internet gateway and firewall. Running servers at home isn t making that much of an impact on my computer power use as there is only one dedicated 24*7 server and that is reasonably low power. But having two desktop systems on 24*7 is a significant factor. Where Power is Used/Wasted No matter how things are counted or what numbers we make up it seems clear that having a desktop system running 24*7 is the biggest use of power that will be assigned to one person. Making PCs more energy efficient through better hardware design and better OS support for suspending would be the best way of saving energy. Nothing that can be done at the server side can compare. Running a server that is only really used by three people is a significant waste by the standards of the NYT article. Of course the thing is that Hetzner is really cheap (and I m not contributing any money) so there isn t a great incentive to be more efficient in this regard. Even if I allocate some portion of the server use to blog readers then there s still a significant portion that has to be assigned to me for my choice to not use a managed service. Running a mail server for a small number of users and running a DNS server and a SE Linux development server are all ways of wasting more power. But the vast majority of the population don t have the skills to run their own server directly, so this sort of use doesn t affect the average power use for the population. Nothing else really matters. No matter what Google does in terms of power use it just doesn t matter when compared to all the desktop systems running 24*7. Small companies may be less efficient, but that will be due to issues of how to share servers among more people and the fact that below a certain limit you can t save money by using less resources particularly if you pay people to develop software. Conclusion I blame Intel for most of the power waste. Android phones and tablets can do some amazing things, which is hardly surprising as by almost every measure they are more powerful than the desktop systems we were all using 10 years ago and by many measures they beat desktop systems from 5 years ago. The same technology should be available in affordable desktop systems. I d like to have a desktop system running Debian based on a multi-core ARM CPU that can drive a monitor at better than FullHD resolution and which uses so little power that it is passively cooled almost all the time. A 64bit ARM system with 8G of RAM a GPU that can decode video (with full Linux driver support) and a fast SSD should compete well enough with typical desktop systems on performance while being quiet, reliable, and energy efficient. Finally please note that most of this post relies on just making stuff up. I don t think that this is wrong given the NYT article that started this. I also think that my estimates are good enough to draw some sensible conclusions.

• [1] http://blog.diegodoval.com/2012/09/23/a-lot-of-lead-bullets-a-response-to-the-new-york-times-article-on-data-center-efficiency/

Related posts:

1. Name Server IP and a Dead Server About 24 hours ago I rebooted the system that runs...
2. Google Custom Search Engine I ve just been experimenting with Google Custom Search [1]. Below...
3. Google Server Design Cnet has an article on the design of the Google...

It seems I am still having problems with spam on this site now. Instead of getting tons of spam comments posted, I get false positive, that is legitimate content marked as spam. First off, I wish to apologize for the people that haven't seen their comments published immediately. Next, I will work on improving this and explaining the current situation. Here are the statistics that the spam module gives me:

• prevented comment spam: 100 (27% aka positives)
• marked comment as spam: 242 (65% aka positives)
• manually marked comment as spam: 3 (0.8%, aka false negatives)
• marked comment as not spam: 17 (5% negatives)
• manually marked comment as not spam: 8 (2.2% false positives)
• total: 370 comments

Another way to put it:

• spam: 93%, 99% of which is automatically detected
• ham: 7%, 32% of which is marked as spam

In other words, the filter works very well at marking spam, so well in fact that it also marks non-spammy comments as spam... Which is a shame because it breaks the interactivity of the site. Basically, you have one chance out of 3 of not seeing your content posted online, which leads people to try to submit it three times, which actually gets them blocked even more. And this is only the spam module filters: the CAPTCHA module tells me it's blocked 26535 attempts (26 thousand) - amazing. So overall, the solutions I put in place a year ago work, somehow, but are too aggressive. I have therefore bumped the threshold from 65% to 80%, hopefully that will help with this. I have also updated the Honeypot module since they have done lots of releases since 1.5, fixed the issue I reported and it's now in use on Drupal.org after some tweaking by our venerable webmaster killes. If this fails, I will look again at other solutions, like the blogspam plugin that relies on the communnity-run blogspam.net site which powers, amongst other things, Debian Administration and Ikiwiki sites. Back then, I also was considering the Bad Behavior and Block Anonymous links modules. (Note to self: I have also removed around 150 comments sitting in the spam module moderation queue, that it didn't seem to remove as it should have done.)

If you want to add some rules to your Apache based on the clients country, mod_geoip is perfect for it. Installation On Squeeze following is enough: # apt-get install libapache2-mod-geoip geoip-database/squeeze-backports Note that you should use the geoip-database version from squeeze-backports to have got the most up to date database version, I am updating it every month. Configuration You can add the rules to your VirtualHost, Directory, Location directives and also to your apache2.conf ( serverwide ). So you are flexible with where to use it. Blocking countries On some servers I have got more than 90 percent of spam requests only from three countries, so I blocked them with: <DirectoryMatch ^/var/www/.*/html >
SetEnvIf GEOIP_COUNTRY_CODE RU BlockCountry
SetEnvIf GEOIP_COUNTRY_CODE CN BlockCountry
SetEnvIf GEOIP_COUNTRY_CODE UA BlockCountry
Deny from env=BlockCountry
</DirectoryMatch> Allow only specific countries In the other way you also can allow specific countries to have got access to your website, this also may be a good idea for extranets, where you know from where your customers are: <Directory /var/www/my.site.com/html/login >
SetEnvIf GEOIP_COUNTRY_CODE DE AllowCountry
SetEnvIf GEOIP_COUNTRY_CODE CH AllowCountry
Deny from all
Allow from env=AllowCountry
</Directory> Very easy! Rewrite Rules You can also use it for mod_rewrite. Within a project, customers from CN and TW should be redirected to the chinese page: RewriteCond % ENV:GEOIP_COUNTRY_CODE ^(CN TW)$
RewriteRule ^(.*)$ http://some.example.cn/site.php [L] mod_geoip with proxy frontends Normaly mod_geoip works behinds load balancers and proxy servers, since it also take care of the HTTP_X_FORWARDED_FOR header. But with haproxy it looks problematic, since it does not add the HTTP_X_FORWARDED_FOR header to KeepAlive d requests :( Disabling KeepAlive is a bad idea on this cluster, so we decided to also use php5-geoip in our application, so everything is working nice now.. What mod_geoip is NOT is mod_geoip helps you to block/allow specific countries, but it does not protect you from them.
Also keep in mind that the database is only ~ 99,8% accurate, so you may have got false positives/negatives. If you only allow german users, a german IP could be listed as russian.
This is much more problematic with mobile/satellite connections and surely you can also not access your page, if you are on vacation in another country. ;)

Some of my Romanian readers might know that for the last two years I've got involved in the skeptical movement to such a degree that I am a co-producer of a bi-weekly podcast on science and skepticism (in Romanian) called Skeptics in Romania . Some might even be regular listeners of the show.

(There isn't much to see now visually on the site, but me and the other people behind the project have some ongoing plans to change that.)

In spite of our modest site, up until now we had some successes, one of them being the publication of an article on us in a known Romanian printed publication and another being the invitation to a live show face to face with Oreste Teodorescu, a well known Romanian mysticist and woo promoter.

During that live show we managed to show a demonstration (video below, in Romanian) of how astrology gives the impression of working, without actually working, and, taking into account we had no prior TV camera experience and that it was a live show, I think we managed an honourable presence.


We also have a series of interviews in English with some really interesting people: Dr. Eugenie Scott, Prof. Christopher French, Prof. Edzard Ernst, Samantha Stein and others. We did these interviews at Denkfest 2011, in Zurich, and we integrated the translated (voice over) interviews in our podcast. The conlusion is that most of our activities revolve around the podcast, so let me tell you more about that.

The podcast has a somewhat fixed structure, it starts with a conversation between ourselves, then we have a segment on the history of science, technology, skepticism and woo, and then we have a segment called The dangers of not being skeptical . In this segment we present cases of people who lost their lives, their health, their money or any combination of the former because they were duped into some scam, science-y sounding non-science, unfounded claim or some other woo.

Having lost recently my brother-in-law to a form of cancer known as Hodgkin lymphoma, I became especially sensitive about miracle-cure claims for cancer, and this section of the show has lately seen its fair share of such cases. Honestly, if there could be a way to prosecute the irresponsible, ignorant and/or cynical people promoting all sorts of quack "therapies", especially for cancer*, I would really like to see it happen. But there isn't, and we're trying the best accessible approach, informing the public.

During my brother-in-law's last two years of his life, he went through lots of chemotherapy and radiotherapy sessions, repeated periods of hospitalisation, and lots of drugs. This is the best of what we currently have for treating and curing most forms of cancer, and too many times this isn't enough. I can't even imagine how stressful and discouraging it must feel when the best of what we have doesn't help.

Here is where the desperation and hopes of patients and their families meet the purely irresponsible cynical or ignorant promoters of woo and quack therapies. Because it takes either an ignorant or a really cynical (I really feel this word isn't enough) person to prey on the suffering of other people to make easy money under the false pretence of offering a cure.

It almost happened to my brother-in-law and his family, because they almost went for some herbal concoction promoted as a cancer cure on some forum, blog or page of a seller of this fake therapy. It was really hard for me to make them understand why using such a product it not advisable, not even in parallel with the medical treatment due its possible counter effect or interactions with the real medical treatment, without them getting the wrong idea that I wasn't trying to help. While trying to be brief and informative not to lose their attention, I told them how "natural" doesn't necessarily mean "good" (uranium, lead and Irukandji's venom are all natural), and how plants are drugs because they all contain chemical substances (and no, "chemical" does not mean "human made" or "artificial") which could interact with the medical treatment.


But most people don't even have the chance of having close by a person with a more science-leaning thought process and a skeptical mind. Those unfortunate people are the most vulnerable people and constitute the biggest chunk of the victims of baseless pseudo-cures or pseudo-treatments.

On our last show, I presented the case of Yvonne Main, a cancer suffering patient who mistook an invasive carcinoma for a cyst, and irdologist Ruth Nelson for a real healthcare giver.

Yvonne Main, died from an invasive carcinoma after seeking help from a iridologist, and delaying real medical treament for 18 months

Yvonne, after seeking medical advice from a person that essentially promotes the dead idea of guessing diseases by looking at the eyes**, used natural treatments for about 18 months and, after all this time, her carcinoma grew to a size of 10 to 11 cm, eating through her skull and causing damage which was later attempted to be countered through bone transplant from her ribs.

Ruth Nelson wasn't prosecuted in any way and continues her practice of quackery unharmed.

This is not the only case, nor even one case from a select few where woo and quackery lead to grave consequences for patients. There are many, many more; they're so many that even after splitting them in categories they seem too many per category, especially when you realise these are only the findings of, essentially, a single man:




This is part of what I have been doing in the last few years, instead of working on Debian. Is it a good thing? Is it a bad thing? Maybe it's good. I want to know what do you think?


* you will, most likely, never hear such a promoter of non-therapies say that there isn't just one cancer, and that, in fact, cancer is a name for a certain family of diseases which are all called cancer - that's a first sign that you might be dealing with quack
** probably in the line of thought that the eyes are the gates to the soul so they must tell something significant about health

Last Saturday I have held a talk at the Meta Main-Rhein Chaosdays 2012 in Darmstadt, an event of the local Chaos Computer Club groups and not unsimilar to Karlsruhe s yearly GPN. I was asked to give an talk on a advanced, Haskell related topic, so I took my audience on a guided tour through the memory of a running Haskell program, explained the different types of closures and their layout, and showed how info tables are used to allow the garbage collector to uniformly treat the objects. We also used that knowledge to roughly predict the memory footprint of a program processing a large file using the String data type. Besides showing raw bytes (using my ghc-heap-view package and a small utility function) I included two visualizations: One is the video of a copying garbage collector at work that I blogged about last week, and the other is a nice tool that a student of mine creates for his bachelor thesis: It allows you to visualize the heap structure of a (potentially partially evaluated) value from GHCi and interactively evaluate parts of the structure if you want, even thunks that are hidden behind other thunks, something that you cannot do from Haskell code. The following picture is a screen shot of ghc-vis visualizing the evaluation of the famous two-line primes definition: The slides of my talk do not cover everything that was included, but the transcript does. It is in German, so as before, if you want me to translate it, then (convince your professor or employer to) invite me to hold the talk again. The ghc-vis tool can be found on Hackage, more information about it is on the author s website.
Flattr this post

Debian GNU/Linux Debian 7.0 (aka Wheezy) will be a general hardened distribution in my eyes. Not only that it now enabled hardened building of packages (see http://wiki.debian.org/Hardening), the Kernel team also backported with 3.2.20-1 the IMO very interesting hidepid option (already available in Wheezy since some weeks)! What is the job of hidepid ? hidepid is an new mount option for the procfs (/proc), with that you can hide processes and its information to other users, like other shell users and to web scripts. hidepid accepts three different values:

• hidepid=0 (default): This is the default setting and gives you the default behaviour.
• hidepid=1: With this option an normal user would not see other processes but their own about ps, top etc, but he is still able to see process IDs in /proc
• hidepid=2: Users are only able too see their own processes (like with hidepid=1), but also the other process IDs are hidden for them in /proc!

Additionaly you can specifiy an user/group ID which is still able to look up the processes with the gid option. So if you want to hide all processes to other users, except root (uid=0) and in this example gid=1001 (some semi administrative user in this example) your /etc/fstab has to look like this: proc /proc proc defaults,hidepid=2,gid=1001 0 0 It was a good descision to backport this feature IMO, but also be careful, it *may* break programs. I did not found any server related application which will break with hidepid=2, but we had to adjust our Nagios monitoring to execute some process checks with another UID, since the nagios user itself could not see anymore, if process A and B is still running. UPDATE 1:
Since a few people asked (thanks for it) with hidepid=2 the process IDs are not invisible, they are unavailable:
$ ls /proc/1
ls: cannot access /proc/1: No such file or directory
$

Continuing the ZFS evaluation journey (which I will summarize here with the things that I learned), I was able to fit about 2.5TB of data on a single 2TB drive, with deduplication enabled. Unfortunately, even moving files around in ZFS (which you would think would be a cheap operation) takes ages. Removing files also takes ages. In a completely unloaded Phenom II X4 910 system running Linux 3.5.2 (actually, from Debian's experimental), and zfsonlinux version 0.6.0.71-0ubuntu1~precise1 (recompiled from source to work with my Debian system), I tried to remove a subdirectory that had about 4000 files. According to time, it took:

real    42m51.487s
user    0m0.188s
sys     0m16.769s

Which is a bit too slow. I just (read: "yesterday") upgraded this machine from 4GB to 6GB of RAM (well, I would have updated it to 8GB of RAM as that is what I ordered, but it one of the RAM chips arrived dead here and I simply took one of the new 4GB card and one of the older 2GB card). But, honestly, I don't see change in upgrading the RAM from 4GB to 6GB and, unless something magic happens with 2GB more, I wouldn't expect a whole different story with 8GB of RAM. Oh, no swap is being used (well, it is an unloaded machine after all), and the Linux kernel swappiness knob is at its default of 60. What I do see, though, is that even for a single file removal, the disk is thrashing a lot. I mean like crazy and that is, of course, the prime suspect of the bad performance (of course, of course). So, there must be some really crazy metadata churning going on here and I guess that the people from the LLNL would like to know of their modules producing this behavior. Keep tuned for some future impressions.

Lessons learned from trying to do some simple non-linear video editing on Linux: Kdenlive has reached a point where it's usable. Not good by any means there are glaring problems such as the color management support being very rudimentary, almost no multicore support, general lack of optimization, frequent near-crashes, and so on but usable to the point where you can actually create a video from start to end. I'd say it's very roughly comparable to Premiere 5 was, circa 1998, but it's not something I'd recommend to my mother. slowmoVideo, on the other hand, was not very useful to me for framerate up-conversions; the end results were simply too poor given the general complexity in my scenes. It can seemingly do an okay (but not perfect) job for down-conversion if you're willing to hack it a bit and live with general slowness and demand for disk space. So, well, behold the end result, I guess. If you like frisbee, that is.

On the 3rd of September, I applied to Google's AdSense program, as I told in my previous post. Well, that was a short-lived experience. On the 5th of September, I received the following e-mail, which, sincerely, is as uninformative as it could get:

Date: Wed, 05 Sep 2012 16:52:22 -0000
From: adsense-noreply@google.com
To: rbrito@gmail.com
Subject: Google AdSense Support
Hello,
As mentioned in our welcome email, we conduct a second review of your
AdSense application once AdSense code is placed on your site(s). As a
result of this review, we have disapproved your account for the following
violation(s):
Issues:
- Site does not comply with Google policies
---------------------
Further detail:
Site does not comply with Google policies: We're unable to approve your
AdSense application at this time because we feel that your site does not
comply with Google AdSense policies or webmaster quality guidelines. It's
our goal to provide our advertisers sites that offer rich and meaningful
content, receive organic traffic, and allow us to serve well-targeted ads
to users. We believe that currently your site does not fulfill this
criteria. 
For more details, please read the webmaster quality guidelines at
http://www.google.com/support/webmasters/bin/answer.py?answer=35769 and
the AdSense program policies at
http://support.google.com/adsense/bin/answer.py?answer=48182,
https://support.google.com/adsense/bin/topic.py?topic=1271507
(...)

OK, I went on and read all the documentation on program policies etc. I investigate each point here:

• Invalid clicks and impressions: No violation here.
• Encouraging clicks: No violation here.
• Content guidelines: No violation here.
• Copyrighted material: No violation here.
• Webmaster guidelines: No violation here.
• Traffic sources: No violation here.
• Ad behavior: No violation here.
• Ad placement: No violation here.
• Site behavior: No violation here.
• Competitive ads and services: No violation here.
• Google advertising cookies: No violation here.

To cut a long story short, I RTFM and did everything correctly. So, dear lazyweb, what would my options be to fund a tiny bit of my software contributions?