I just had a conversation with someone who thinks that their office should have no servers. The office in question has four servers, an Internet gateway/firewall system, the old file server (and also Xen server), the new file server, and the VOIP server. The Internet gateway system could possibly be replaced by a suitably smart ADSL modem type device, but that would reduce the control over the network and wouldn t provide much of a benefit. The VOIP server has to be a separate system for low latency IMHO. In theory you could use a Xen DomU for running Asterisk or you could run Asterisk on the Dom0 of the file/Xen server. But that just makes things difficult. A VOIP server needs to be reliable and is something that you typically don t want to touch once it s working, in this case the Asterisk server has been a few more years without upgrades than the Xen server. An Asterisk system could be replaced by a dedicated telephony device which some people might consider to be removing a server, but really a dedicated VOIP server device is just as much of a server as a P4 running Asterisk but with greater expense. A major advantage of a P4 running Asterisk is that you can easily replace the system at no cost if there is a hardware problem. Having two file servers is excessive for a relatively small office. But running two servers is the common practice when one server is being replaced. The alternative is to just immediately cut things over which has the potential for a lot of people to arrive at work on Monday and find multiple things not working as desired. Having two file servers is a temporary problem. File Servers The first real problem when trying to remove servers from an office is the file server. ADSL links with Annex M can theoretically upload data at 3Mb/s which means almost 400KB/s. So if you have an office with a theoretically perfect ADSL2+ Annex M installation then you could save a 4MB file to a file server on the Internet in not much more than 10 seconds if no-one else is using the Internet connection. Note that 4MB isn t THAT big by today s standards, the organisation in question has many files which are considerably bigger than that. Large files include TIFF and RAW files used for high quality image processing, MS-Office documents, and data files for most accounting programs. Saving a 65MB quick-books file in 3 minutes (assuming that your Annex M connection is perfect and no-one else is using the Internet) would have to suck. Then there s the issue of reading files, video files (which are often used for training and promotion) are generally larger than 100MB which would be more than 30 seconds of download time at ADSL2+ speed but if someone sends an email to everyone in the office saying please watch this video then the average time to load it would be a lot more. Through quickly examining my collection of Youtube downloads I found a video which averaged 590KB/s, if an office using a theoretically perfect ADSL2+ connection giving 24Mb/s (3MB/s) download speed had such a file on a remote file server then a maximum of five people could view it at one time if no-one else in the office was using the Internet. Now when the NBN is connected (which won t happen in areas like the Melbourne CBD for at least another 3 years) it will be possible to get speeds like 100Mb/s download and 25Mb/s upload. That would allow up to 20 people to view videos at once and a 65MB quick-books file could be saved in a mere 22 seconds if everyone else was idle. Of course that relies on the size of data files remaining the same for another 3 years which seems unlikely, currently no Youtube videos use resolutions higher than 1920*1080 (so they don t take full advantage of a $400 Dell monitor) and there s always potential for storing more financial data. I expect that by the time we all have 100Mb/25Mb speeds on the NBN it will be as useful to us as 24Mb/3Mb ADSL2+ Annex M speeds are today (great for home use but limited for an office full of people). There are of course various ways of caching data, but all of them involve something which would be considered to be a server and I expect that all of them are more difficult to install and manage than just having a local file server. Of course instead of crunching the numbers for ADSL speeds etc you could just think for a moment about the way that 100baseT networking to the desktop has been replaced by Gigabit networking. When people expect each workstation to have 1000Mb/s send and receive speed it seems quite obvious that one ADSL connection shared by an entire office isn t going to work well if all the work that is done depends on it. Management could dictate that there is to be no server in the office, but if that was to happen then the users would create file shares on their workstations so you would end up with ad-hoc servers which aren t correctly managed or backed up. That wouldn t be an improvement and technically wouldn t achieve the goal of not having servers. Home Networking Without Servers It is becoming increasingly common to have various servers in a home network. Due to a lack of space and power and the low requirements a home file server will usually be a workstation with some big disks, but there are cheap NAS devices which some people are installing at home. I don t recommend the cheap NAS devices, I m merely noting that they are being used. Home entertainment is also something that can benefit from a server. A MythTV system for recording TV and playing music has more features than a dedicated PVR box. But even the most basic PVR ($169 for a 1TB device in Aldi now) is still a fairly complex computer which would probably conflict with any aim to have a house free of servers. The home network design of having a workstation run as a file and print server can work reasonably well as long as the desktop tasks aren t particularly demanding (IE no games) and the system doesn t change much (IE don t track Debian/Testing or otherwise have new versions of software). But this is really something that only works if you only have a few workstations. Running an office without servers seems rather silly as it seems that none of my friends are able to have a home without a server. Running Internet Services Hypothetically speaking if one was to run an office without servers then that would require running all the servers in question on the Internet somewhere. For some things this can work better than a local server, for example most of my clients who insist on running a mail server in their office would probably get a better result if they had a mail server running on Linode or Hetzner or one of the Hosted Exchange offerings if they want a Windows mail sever. But for a file server if you were to get around the issue of bandwidth required to access the files in normal use there s the issue of managing a server (which is going to take more effort and expense than for a server on the LAN). Then there s the issue of backups. In my previous post about Hard Drives for Backup [1] I considered some of the issues related to backing data up over the Internet. The big problem however is a complete restore, if you have even a few dozen gigs of data that you want to transfer to a remote server in a hurry it can be a difficult problem. If you have hundreds of gigs then it becomes a very difficult problem. I m sure that I could find a Melbourne based Data Center (DC) that gives the option of bringing a USB attached SATA disk for a restore but even that case would give a significant delay when compared to backing things up on a LAN. If a server on the office LAN breaks in the afternoon my client can make arrangements to let me work in their office in the evening to fix it, but sometimes DCs don t allow 24*7 access and sometimes when they do allow access there are organisational problems that make it impossible when you want it (EG the people at the client company who are authorised become unavailable). The Growth of Servers Generally it s a really bad idea to build a server that has exactly the hardware you need. The smart thing to do is to install more of every resource (disk, RAM, CPU, etc) than is needed and to allow expansion when possible (EG have some RAM slots and drive bays free). No matter how well you know your environment and it s users you can get surprised by the way that requirements change. Buying a slightly bigger server at the start costs hardly any money but upgrading a server will cost a lot. Once you have a server that s somewhat over-specced you will always find other things to run on it. Many things could be run elsewhere at some cost, but if you have unused hardware then you may as well use it. Xen and other virtualisation systems are really good in this regard as they allow you to add more services without making upgrades difficult. This means that it s quite common to have a server that is purchased for one task but which ends up being used for many tasks. Anyone who would aspire to an office without servers would probably regard adding extra features in such a manner to be a problem. But really if you want to allow the workers to do their jobs then it s best to be able to add new services as needed without going through a budget approval process for each one. Conclusion There probably are some offices where no-one does any serious file access and everyone s work is based around a web browser or some client software that is suited to storing data on the Internet. But for an office where the workers use traditional Office software such as MS-Office or Libre-Office a file server is necessary. Some sort of telephony server is necessary no matter how you do things. If you have a traditional telephone system then you might try not to call the PABX a server , but really that s what it is. Then when the traditional phone service becomes too expensive you have to consider whether to use Asterisk or a proprietary system, in either case it s really a server. In almost every case the issue isn t whether to have a server in the office, but how many servers to have and how to manage them.

• [1] http://etbe.coker.com.au/2012/08/08/hard-drives-backup/

Related posts:

1. Planning Servers for Failure Sometimes computers fail. If you run enough computers then you...
2. a good security design for an office One issue that is rarely considered is how to...
3. Efficiency of Cooling Servers One thing I had wondered was why home air-conditioning systems...

Ever wondered how you can update your website (in our case a static website with a bunch of HTML and PHP files) by committing to a Git repository hosted on a different server? if the answer to the previous question is yes, then you are in the right place. The scenario: - Website hosted on server A.
- Git repository hosted on server B. and a few details about why would you opt for maintaining your website through Git:

1. You need multiple people to access the static content of your website and you also want to maintain all the history of changes together with all the Git s magic.
2. You think using an FTP server is not secure enough.
3. You think giving out SSH access or more permissions on the server to multiple users it s not what you want. (also using scp will overwrite the files directly with all its consequences)

the setup, on server A: 1. Clone the Git repository over the directory that your Web server will serve.

cd /srv/www.domain.com/http && sudo git clone http://git.example.com/git/domain.git

2. Grab the needed package:

apt-get install fishpolld or yum install fishpolld

3. Set up a topic (called website_update) that will call a git pull each time the repository hosted on server B receives an update. (the file has to be placed into the /etc/fishpoll.d directory)

#!/bin/bash
PATH="/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin"
WEBSITEDIR="/srv/www.domain.com/http"
if [ -d "$ WEBSITEDIR " ]; then
cd "$ WEBSITEDIR "
else
echo "Unable to access theme directory. Failing."
exit 1
fi
git pull

4. Add a little configuration file that will enable the website_update s topic at daemon s startup. (you should name it website_update.conf)

[fishpoll]
on_start = True

5. Open the relevant port on Iptables so that server A and server B can communicate as expected, the daemon runs over port 27527.

-A INPUT -m state --state NEW -m tcp -p tcp -s server-B-IP --dport 27527 -j ACCEPT

6. Start the daemon. (by default, logs are sent to syslog but you can run the daemon in Debug mode by using the -D flag)

sudo /etc/init.d/fishpolld start or sudo systemctl start fishpolld.service or sudo /usr/sbin/fishpolld -D

and on server B: 1. Grab the needed package:

apt-get install fishpoke or yum install fishpoke

2. Configure the relevant Git hook (post-update, located into the hooks directory) and make it executable.

#!/bin/sh
echo "Triggering update of configuration on server A"
fishpoke server-A-IP-or-DNS website_update

Finally test the whole setup by committing to the repository hosted on server B and verify your changes being sent live on your website!

Debian GNU/Linux Debian 7.0 (aka Wheezy) will be a general hardened distribution in my eyes. Not only that it now enabled hardened building of packages (see http://wiki.debian.org/Hardening), the Kernel team also backported with 3.2.20-1 the IMO very interesting hidepid option (already available in Wheezy since some weeks)! What is the job of hidepid ? hidepid is an new mount option for the procfs (/proc), with that you can hide processes and its information to other users, like other shell users and to web scripts. hidepid accepts three different values:

• hidepid=0 (default): This is the default setting and gives you the default behaviour.
• hidepid=1: With this option an normal user would not see other processes but their own about ps, top etc, but he is still able to see process IDs in /proc
• hidepid=2: Users are only able too see their own processes (like with hidepid=1), but also the other process IDs are hidden for them in /proc!

Additionaly you can specifiy an user/group ID which is still able to look up the processes with the gid option. So if you want to hide all processes to other users, except root (uid=0) and in this example gid=1001 (some semi administrative user in this example) your /etc/fstab has to look like this: proc /proc proc defaults,hidepid=2,gid=1001 0 0 It was a good descision to backport this feature IMO, but also be careful, it *may* break programs. I did not found any server related application which will break with hidepid=2, but we had to adjust our Nagios monitoring to execute some process checks with another UID, since the nagios user itself could not see anymore, if process A and B is still running. UPDATE 1:
Since a few people asked (thanks for it) with hidepid=2 the process IDs are not invisible, they are unavailable:
$ ls /proc/1
ls: cannot access /proc/1: No such file or directory
$

This morning version 0.2.8 of RInside arrived on the CRAN sites. RInside provides a set of convenience classes which facilitate embedding of R inside of C++ applications and programs, using the classes and functions provided by the Rcpp R and C++ integration package. This release adds no new features but improves the build process a little and should make use on Windows a little easier. All changes since the last release are summarized below in the NEWS file entry:

Changes in RInside version 0.2.8 (2012-09-07)

• Added CMake build support for armadillo and eigen examples, once again kindly contributed by Peter Aberline
• Corrected Windows package build to always generate a 64 bit static library too
• Updated package build to no longer require configire and configure.win to update the two header file supplying compile-time information; tightened build dependencies on headers in Makevars / Makevars.win
• Improved examples/standard/Makefile.win by detecting architecture

CRANberries also provides a short report with changes from the previous release. More information is on the RInside page. Questions, comments etc should go to the rcpp-devel mailing list off the Rcpp R-Forge page.

Sarah is back in Australia attending the graduation ceremony for her Bachelor of Business degree, at the University of Southern Queensland, an Australia University that specialises in distance education. To say I'm immensely proud of her achievement is putting it lightly. Consider for a moment all of the challenges she has had to face in the past 6 years since she started it:

• studying remotely
• working full time (for a bit)
• losing a baby
• having major heart surgery (with a staph infection complication)
• getting her sternal wires removed (with another staph infection complication)
• having a baby
• looking after a baby at home while continuing to study

Me, I'd fail at "studying remotely". Heck, I've had two stabs at a Bachelor of Information Technology on campus, and what do I have to show for it? No, I'm so very very proud of her for sticking at it, and doing so very well (she twice received the Dean's Award for having a high GPA for the semester). She decided to embark on a business degree after having a really hard time finding meaningful work when we moved to the US. She had been working in business project management for the Australian Federal Government (and really enjoying the work) before we moved, and couldn't find any project management that wasn't high-tech, which wasn't really her thing, and everything wanted a degree. She briefly contracted at VMWare, in a marketing position, and Google, in a Security Vulnerability Management role, but neither of these were really up her alley, so to speak. So she went "bugger it" and enrolled in the degree. Ironically, mid-way through the course, she decided that nursing was more what she wanted to do, so she has also gone out and done short courses in health care. She has been licensed as a Certified Nurse Assistant and also a Phlebotomist, but never worked as either. She's just started an Emergency Medical Technician course at Mission College, and is keen to try and work in that field for a little while before starting on a post-graduate Nursing degree in Australia. Sarah had indicated to me that she'd really love to go back for the on-campus graduation ceremony, since she's barely set foot on campus in the six years she's been studying at the university. I thought that was an excellent idea, so I sprang for a Business Class flight back for just her, and am staying behind to look after Zoe. It's only a flying visit. She departed Wednesday night, returns on Monday. As I write, she's probably at the ceremony right now. Normally it would take 6 years to complete a degree externally, assuming two subjects a semester for 12 semesters. She's obviously had to take a few semesters off here and there because of some of the aforementioned challenges. To still finish the degree in 6 years, she's had to plow through the "summer" break (I say "summer" because it's during the Northern Hemisphere winter) and use the accelerated "summer semester", also known as "semester 3" to do an additional subject. That's meant that summers over here have been her shorter winter break. This summer has been the first summer where she hasn't had to worry about studying, and it's been really wonderful to be a whole family again for an extended period of time, and we've had a jam packed summer as a result. We bought a tent and did a small amount of camping. We've had weekends away with friends. We've been to Raging Waters. We've been to Disneyland (granted that was in between final exams). It's been go go go. So now that summer is almost over, I'm looking forward to the pace slowing down a bit more and things returning to "normal" (well the new "normal" is Sarah doing her EMT class a couple of nights a week until December). She's only just started, but she's really loving the on-campus aspect of it. I'm also looking forward to getting a bit more "me" time. Since Zoe was born, I have sacrificed my personal time to facilitate Sarah's study. I haven't had much time to tinker and I've barely had time for Debian. I'm looking forward to trying to carve out a few hours a week for some of both, plus a bit of reading.

This is my monthly summary of my Debian related activities. If you re among the people who made a donation to support my work (88.41 , thanks everybody!), then you can learn how I spent your money. Otherwise it s just an interesting status update on my various projects. This month has again been a short one since I have mostly been in vacation during the last 2 weeks. Dpkg Things are relatively quiet during the freeze. I only took care of fixing 3 bugs: a regression of 3.0 (quilt) (#683547), a segfault of dpkg-query -W -f (commit) and a bad auto-completion for French users (#685863). Testing the upgrade to wheezy We got several reports of wheezy upgrade that failed because dpkg ran the trigger while the dependencies of the package with pending triggers are not satisfied. Unfortunately fixing this in dpkg is not without problems (see #671711 for details) so Guillem decided to defer this fix for Jessie. My suggestion of an intermediary solution has fallen in limbo. Instead we now have to find solutions for each case where this can fail (example of failure: 680626). Another way to avoid those errors is to ensure that triggers are run as late as possible. We can improve this in multiple ways. The first way is to modify most triggers so that they use the interest-noawait directive. In that case, the packages activating the trigger will be immediately marked as configured (instead of triggers-awaited ) and the trigger will thus not need to be run as part of further dependency solving logic. But as of today, there s no package using this new feature yet despite a nudge on debian-devel-announce. The second way is to modify APT to use dpkg no-triggers, and to let the trigger processing for the end (with a last dpkg configure -a call). I requested this early in the wheezy timeframe but for various reasons, the APT maintainers did not act on it. I pinged them again in #626599 but it s now too late for wheezy. I find this a bit sad because I have been using those options for the entire wheezy cycle and it worked fine for me (and I used them for a dist-upgrade on my wife s laptop too). It would have been good to have all this in place for wheezy so that we don t have to suffer from the same problems during the jessie upgrade, but unless someone steps up to steer those changes, it seems unlikely to happen. Instead, we re back to finding klumsy work-arounds in individual packages. Packaging I prepared security updates for python-django (1.4.1 for unstable,
1.2.3-3+squeeze3 for stable). I packaged a new upstream version for cpputest (3.2-1). I reviewed ledgersmb 1.3.21-1 prepared by Robert James Clay and asked him to prepare another version with further fixes. I released nautilus-dropbox 1.4.0-2 with supplementary changes of my own to support https_proxy and to display better diagnostic information when the download fails. With the help of Paul van der Vlis and Michael Ziegler, we did what was required to be able to migrate python-django-registration 0.8 to Wheezy even though it s a new upstream version with backwards incompatible changes. Thanks to Adam D. Barratt who unblocked the package, we now have the right version in Wheezy despite the fact that I missed the freeze deadline. Debian France Julien Cristau reminded the board of Debian France that we have to elect officers (President, Secretary, Treasurer) as the current officers have withdrawn. I was somewhat afraid that nobody would take over so I pinged each member to try to get new volunteers. We now have volunteers (me, Julien Danjou and Sylvestre Ledru) and we re waiting until Julien finds some time to run the election. Misc With the help of DSA, I setup antispam rules for the owner@packages.qa.debian.org alias because I was getting tired by the amount of spam. In the process, they asked me to write a wiki page for dsa.debian.org to document everything so that they can refer to it for future queries. I did it but it looks like that they did not apply my patch yet. I also tested an upstream patch for gnome-keyring (see bugzilla #681081) that reintroduces the support of forgetting GPG passphrases after a specified amount of time. Thanks See you next month for a new summary of my activities.

No comment Liked this article? Click here. My blog is Flattr-enabled.

I've been slowly noticing, over a number of years now, a place where I don't react to other people the way that I want to. I've been stumbling over defensive reactions that I didn't realize I had. These are a few rules of thumb that I'm formulating for myself to try to improve that, and I thought it might help to write them down. And maybe they'll be meaningful for someone else.

1. Things people are excited about are probably exciting. If I don't find them exciting, that's probably because I don't know enough about them to understand why they're exciting. Or perhaps I have a different mindset, and therefore the exciting parts don't connect with me. Both of those are okay! There are way more things to learn about than I will ever possibly have time to learn, and I don't have to get excited about everything other people are excited about. But, equally, just because I'm not excited doesn't mean that the thing is not exciting.
2. My misplaced sense of obligation should not be someone else's buzzkill. This requires a bit of unpacking, since I only intermittantly realize that I'm doing this. But (partly for cultural reasons, partly due to natural inclinations) I've always felt a strong sense of obligation to "keep up" with things. To know a lot about a lot of topics. And one of the things I've realized is that my brain occasionally twists an interest in knowing lots about the world into a sense of obligation to "stay current" and know something about everything important, at least within my vague field of expertise. And usually that part of my brain has a very expansive notion of field of expertise; all of software development, for example, which is, when I pull it out into the light, ridiculously broad. This then sparks one of the standard failure modes of human cognition. I have an unrealistic expectation of myself. I fail to meet that expectation. This makes me feel guilty and stressed. At that point, my brain starts trying to find targets for me to displace negative feelings onto, and that generally ends up being the next person who is excited about something new that I don't know anything about but feel like I should. This is all very interesting psychologically, in the abstract. However, the concrete result is that I sometimes get all snarky and negative about something that someone else is excited about because it happens to be the moment I hit overload, or because I didn't want to hear about another thing that I feel like I should know more about. And that, in turn, can be either confrontational or discouraging to someone who is legitimately excited (and may rarely get the chance to be excited). And that's not the person I want to be.
3. It's okay not to have an opinion. Seriously, it's okay. The world is full of stuff, far more stuff than any one person can know anything about. It's okay not to go read about everything people say is exciting. It's okay not to know about something new. It's okay to focus on the things I'm already doing. Really. What's not okay is to make up an opinion about something I know nothing about just so that I can feel better about myself by having an opinion. Particularly since that opinion formed under those circumstances tends to be negative, despite the fact that I have no actual basis for having any opinion at all. I need to practice just quietly not having an opinion. No one else needs to know about my lack of opinion.
4. Before bristling at imposition, be sure someone is actually imposing. I frequently leap to the conclusion that, when someone raises a topic, they want me to do something about it. It's a gut emotional reaction, I think in part because the emotional part of my brain likes to make everything about me. Intellectually, if I can engage that part of my brain, it's often obvious that it's not about me, or that they're not asking me to do anything. Often, when someone brings something up, they just want to tell someone else about it, or make me aware of it. If I've listened to, or read, all the sentences and understood then, then congratulations! I've done everything that I needed to do and was expected to do and need not feel any further obligation. Even if something further needs to be done, it doesn't necessarily need to be done by me. I'm very protective of my time and available resources because I almost always feel overwhelmed by all the things that I want to work on. This does not entitle me to snap at random people who are being excited around me on the grounds that they're trying to put more obligations on me. They're probably not. If it becomes clear that they really want me to do something I don't have time to do, I can politely decline, but if I just shut up and listen for longer, usually that's not the case at all.

(Please note that this is not prompted by any current discussion happening anywhere, and paticularly not by anything in Debian. Insofar as it was prompted by anything, it was by a throwaway comment in an unrelated private discussion. It's just something that gelled enough to write down while I was walking home from work.)

Back in June or thereabouts I learned about a project to re-create Nine Inch Nails' first album "Pretty Hate Machine" as a chiptune cover/tribute project. This project is called Pretty Eight Machine. The artist InversePhase was kind enough to send me a copy of the CD as a random prize draw, so I thought I would write a few words about it. I don't know a great deal about the world of chiptunes, besides having heard a variety of "chiptune" covers of various well-known songs, some good, some bad. the backing track to this video, a cover of Tears for Fears' "Mad World", is, in my opinion, good. The front cover for P8M lovingly reflects the recent reworking of the classic PHM cover by Rob Sheridan, for the source album's recent deluxe reissue. There's a great in-depth interview with Rob about the process of reviving that cover at sleevage. That little homage is a good indication of the amount of effort and attention to detail that has gone into this project. One interesting thing about this project is that each track has been composed for a particular vintage chipset, such as the Commodore 64 in the case of the opening track "Head like I/O". InversePhase is very faithful to the original chips here, not just using their tones as part of a modern composition but constraining himself with the actual chipset limitations: for example, the number of voices that can be playing at once. (By contrast, that Tears for Fears chiptune above is the non-authentic type of chiptune). InversePhase tracks Trent's vocal lines as an instrument, so all of the notes that are heard in the original are present on this remake. It would be interesting to hear an "instrumental" mix of these tracks, and perhaps mix isolated vocal tracks from the originals on top My favourite track is probably "Kinda I Want To". I actually really like the original track, in particular the extended breakdown, where a synth noise gradually breaks down and disintegrates bar after bar. It suffered terribly from dated lyrics compared to other PHM tracks, but InversePhase's cover is saved that embarrassment. This particular track (apparently) makes use of a VCR6, a Konami chipset apparently used to supplement the hardware in the Famicom/NES. Yes, back then games came on cartidges that could supply bits of their own hardware if required. Those were the days! And yes InversePhase, you've successfully made the NES sound "dirty": and the synth breakdown sounds great. Any fan of the original album or anyone with an interest in what kind of noises vintage sound synthesis chipsets could produce are thoroughly recommended to give this album a try. You can preview all of the tracks and read per-track information on the bandcamp page.

I've never really used facebook so it's not too hard for me to delete my account there. The only thing that's a pity is that I found (or others found me) many people that I once met and which I'd love to meet again once. So I'll go through my facebook contacts list and will try to copy all contact data especially the email adresses and plan to write emails to everybody every quarter or half a year about happenings in our lifes. I'd be glad to receive such emails from others too! They'll surely be read more carefully then my facebook timeline!

This gets me to another point: I still don't have a satisfying free (as in freedom) solution to manage my contacts across devices and available on my web server. The same for image galleries. So I can understand how convenient facebook for many people is. Still facebook is evil and it's an important and ongoing project to provide a better alternative.

It'll be harder for me to leave Twitter. This one comes next. If people would just switch to the free and privacy aware alternative identi.ca...

And then I've to finally provide a better alternative for my family to share baby pictures then Google Plus... Leaving Xing and Linkedin also doesn't worry me, I just have to collect the addresses of my important contacts there. Not so pressing is to leave Couchsurfing for bewelcome.org.

So if you want to hear news from me just pass by www.koch.ro from time to time or put my blogs feed into your feed reader. I'll also keep my identi.ca account.

update: One tiny example of facebooks evilness is, that they don't show me your real email address (anymore). They only show me an @facebook address. So if I don't know your real email address from any other source, I would not be able to send you messages without sending it over facebooks servers. I read that you can change the email address shown back to your real email address somewhere in the options.

On year ago, we (the mancoosi team) published a comparison study regarding the state of the art of dependency solving in debian. As few noticed, the data presented had few glitches that I promised to fix. So we've repeated our tests using exactly the same data we used one year ago, but now using the latest available versions of all package managers as available in debian unstable. During the last year, three out of four solver that we evaluated release a major upgrade so I expected many improvements in performances and accuracy.

• apt-get 0.8.10 -> 0.9.7
• aptitude 0.6.3 -> 0.6.7
• smart 1.3-1 -> 1.4
• cupt 1.5.14.1 -> 2.5.6

Mpm, our test-bench for new technologies, changed quite a bit under the wood as a consequence of the evolution of apt-cudf and the recent work done in apt-get to integrate external cudf dependency solvers. Overall the results of our study are not changed. All solvers but mpm, that is based on aspcud, are not scalable as the number of packages (and alternatives) grows. It seems that Smart is the solver that does not give up, incurring in a timeout (fixed at 60 seconds) most of the time. Aptitude is the solver that tried to give you a solution, doesn't matter what and as result providing solutions that do not satisfy the user request for one reason or the other. Apt-get does surprisingly well, but it gives up pretty often showing the incomplete nature of it's internal solver. Cupt sometimes timeout, sometimes gives up, but when it is able to provide an answer it is usually optimal and it is very fast ... Mpm consistently finds an optimal solution, but sometimes it takes really a long time to do it. Since mpm is written in python and not optimized for speed this is not a big problem for us. The technology used by mpm is now integrated in apt-get and I hope this will alleviate this problem. All the details of our study can be found one the Mancoosi Website as usual with a lot of details. For example here you can find the results when mixing four major releases : sarge-etch-lenny-squeeze. Comments are more then welcome.

I just released PyGObject 3.3.91, for GNOME 3.5.91. The big new feature in this release (thanks to the release team for granting an exception) is Simon Feltman s new Signal helper class, which makes defining custom signals a whole lot simpler and more obvious. In the past, you had to do

 class C(GObject.GObject):
    __gsignals__ =  
        'my_signal': (GObject.SIGNAL_RUN_FIRST, GObject.TYPE_NONE,
                      (GObject.TYPE_INT,))
     
    def do_my_signal(self, arg):
        print("my_signal called with %i" % arg)

whereas now this looks like

class C(GObject.GObject):
    @GObject.Signal(arg_types=(int,))
    def my_signal(self, arg):
        print("my_signal called with %i" % arg)

or even more elegantly when using Python 3 and its new type annotations:

class C(GObject.GObject):
    @GObject.Signal
    def my_signal(self, arg:int):
        print("my_signal called with %i" % arg)

Check out the updated example and docstring for other ways how to use it. Overrides can now be in a directory different from the one that pygobject installs itself into. These overrides need to put this into their __init__.py at the top:

from pkgutil import extend_path
__path__ = extend_path(__path__, __name__)

and put themselves somewhere into the default PYTHONPATH. This should make it a lot easier for library packages to ship their own overrides for Python. This new version also comes with a couple of new overrides and bug fixes. See the detailled list of changes below. Thanks to all contributors!

• Fix exception test case for Python 2 (Martin Pitt)
• Bump g-i dependency to >= 1.3.9 (Martin Pitt)
• Show proper exception when trying to allocate a disguised struct (Martin Pitt) (#639972)
• Support marshalling GParamSpec signal arguments (Mark Nauwelaerts) (#683099)
• Add test for a signal that returns a GParamSpec (Martin Pitt) (#683265)
• [API add] Add Signal class for adding and connecting custom signals. (Simon Feltman) (#434924)
• Fix pygtkcompat s Gtk.TreeView.insert_column_with_attributes() (Martin Pitt)
• Add override for Gtk.TreeView.insert_column_with_attributes() (Marta Maria Casetti) (#679415)
• .gitignore: Add missing built files (Martin Pitt)
• Ship tests/gi in tarball (Martin Pitt)
• Split test_overrides.py (Martin Pitt) (#683188)
• _pygi_argument_to_object(): Clean up array unmarshalling (Martin Pitt)
• Fix memory leak in _pygi_argument_to_object() (Alban Browaeys) (#682979)
• Fix setting pointer fields/arguments to NULL using None. (Simon Feltman) (#683150)
• Fix for python 2.6, officially drop support for < 2.6 (Martin Pitt) (#682422)
• Allow overrides in other directories than gi itself (Thibault Saunier) (#680913)
• Clean up sys.path handling in tests (Simon Feltman) (#680913)
• Fix dynamic creation of enum and flag gi types for Python 3.3 (Simon Feltman) (#682323)
• [API add] Override g_menu_item_set_attribute (Paolo Borelli) (#682436)

During lunch the other day, I discussed the shortcomings of the tar file format with friend and co-worker Daniel. The tar file format has a lot of legacy by now, and it's not quite up to date with the latest developments in file systems, such as extended attributes. This makes tar badly suited for things such as backups and other situations where precise reproduction of the input data matters. There are several variants of the tar file format, and various more or less standard extensions to it. GNU tar, for example, added support for pathnames longer than 100 bytes many years ago, and it is now commonly supported. Other problems in the tar file format:

• It has no native support for compression. The Unix Way is to use an external compressor, which is nice, but it makes it necessary to decompress the entire file to get a list of its contents. For large archives, this is very time consuming.
• Even when uncompressed, the file format works badly for some kinds of operations, such as deleting files from the archive, or updating them with new versions.
• The file format is entirely linear. When creating a tar file, it would sometimes be possible to write data from multiple sources at the same time, perhaps compressing them separately, maybe with file type specific compressors. With a linear format, this is not possible without spooling some files into temporary files. An interleaved format, similar to multimedia files, which mix audio and video data into a single stream, would make it possible to be more efficient at writing.
• The supported meta data for files is limited, and it's hard to extend the support without breaking the file format.

This led us to discuss the possibility of a new file format. We had a bit of fun exploring the solution space for a while. However, almost all use of tar these days is for distributing sets of files, where the filename and basic set of file permissions is enough. In other words, for things such as source code, tar is just fine. The archives are small enough, and the other limitations are rarely a problem, but the pain of switching to a new format would be great. Thus, with some reluctance, we concluded that a new format would be a waste of time. But I thought I'd write this up anyway, in case one of my readers wants to start working on this.

Scott Chamberlain blogged about faster creation of binomial matrices the other day, and even referred to our RcppArmadillo package as a possible solution (though claiming he didn't get it to work, tst tst -- that is what the rcpp-devel list is here to help with). The post also fell short of a good aggregated timing comparison for which we love the rbenchmark package. So in order to rectify this, and to see what we can do here with Rcpp, a quick post revisiting the issue. As preliminaries, we need to load three packages: inline to create compiled code on the fly (which, I should mention, is also used together with Rcpp by the Stan / RStan MCMC sampler which is creating some buzz this week), the compiler package included with R to create byte-compiled code and lastly the aforementioned rbenchmark package to do the timings. We also set row and column dimension, and set them a little higher than the original example to actually have something measurable:

library(inline)
library(compiler)
library(rbenchmark)
n <- 500
k <- 100

The first suggestion was the one by Scott himself. We will wrap this one, and all the following ones, in a function so that all approaches are comparable as being in a function of two dimension arguments:

scott <- function(N, K)  
    mm <- matrix(0, N, K)
    apply(mm, c(1, 2), function(x) sample(c(0, 1), 1))
 
scottComp <- cmpfun(scott)

We also immediatly compute a byte-compiled version (just because we now can) to see if this helps at all with the code. As there are no (explicit !) loops, we do not expect a big pickup. Scott's function works, but sweeps the sample() function across all rows and columns which is probably going to be (relatively) expensive. Next is the first improvement suggested to Scott which came from Ted Hart.

ted <- function(N, K)  
    matrix(rbinom(N * K, 1, 0.5), ncol = K, nrow = N)
 

This is quite a bit smarter as it vectorises the approach, generating N times K elements at once which are then reshaped into a matrix. Another suggestion came from David Smith as well as Rafael Maia. We rewrite it slightly to make it a function with two arguments for the desired dimensions:

david <- function(m, n)  
    matrix(sample(0:1, m * n, replace = TRUE), m, n)
 

This is very clever as it uses sample() over zero and one rather than making (expensive) draws from random number generator. Next we have a version from Luis Apiolaza:

luis <- function(m, n)  
     round(matrix(runif(m * n), m, n))
 

It draws from a random uniform and rounds to zero and one, rather than deploying the binomial. Then we have the version using RcppArmadillo hinted at by Scott, but with actual arguments and a correction for row/column dimensions. Thanks to inline we can write the C++ code as an R character string; inline takes care of everything and we end up with C++-based solution directly callable from R:

arma <- cxxfunction(signature(ns="integer", ks="integer"), plugin = "RcppArmadillo", body='
   int n = Rcpp::as<int>(ns);
   int k = Rcpp::as<int>(ks);
   return wrap(arma::randu(n, k));
')

This works, and is pretty fast. The only problem is that it answers the wrong question as it returns U(0,1) draws and not binomials. We need to truncate or round. So a corrected version is

armaFloor <- cxxfunction(signature(ns="integer", ks="integer"), plugin = "RcppArmadillo", body='
   int n = Rcpp::as<int>(ns);
   int k = Rcpp::as<int>(ks);
   return wrap(arma::floor(arma::randu(n, k) + 0.5));
')

which uses the the old rounding approximation of adding 1/2 before truncating. With Armadillo in the picture, we do wonder how Rcpp sugar would do. Rcpp sugar, described in one of the eight vignettes of the Rcpp package, is using template meta-programming to provide R-like expressiveness (aka "syntactic sugar") at the C++ level. In particular, it gives access to R's RNG functions using the exact same RNGs as R making the results directly substitutable (whereas Armadillo uses its own RNG).

sugar <- cxxfunction(signature(ns="integer", ks="integer"), plugin = "Rcpp", body='
   int n = Rcpp::as<int>(ns);
   int k = Rcpp::as<int>(ks);
   Rcpp::RNGScope tmp;
   Rcpp::NumericVector draws = Rcpp::runif(n*k);
   return Rcpp::NumericMatrix(n, k, draws.begin());
')

Here Rcpp::RNGScope deals with setting/resetting the R RNG state. This draws a vector of N time K uniforms similar to Luis' function -- and just like Luis' R function does so without looping -- and then shapes a matrix of dimension N by K from it. And it does of course have the same problem as the RcppArmadillo approach earlier and we can use the same solution:

sugarFloor <- cxxfunction(signature(ns="integer", ks="integer"), plugin = "Rcpp", body='
   int n = Rcpp::as<int>(ns);
   int k = Rcpp::as<int>(ks);
   Rcpp::RNGScope tmp;
   Rcpp::NumericVector draws = Rcpp::floor(Rcpp::runif(n*k)+0.5);
   return Rcpp::NumericMatrix(n, k, draws.begin());
')

Now that we have all the pieces in place, we can compare:

res <- benchmark(scott(n, k), scottComp(n,k),
                 ted(n, k), david(n, k), luis(n, k),
                 arma(n, k), sugar(n,k),
                 armaFloor(n, k), sugarFloor(n, k),
                 order="relative", replications=100)
print(res[,1:4])

With all the above code example in a small R script we call via littler, we get

edd@max:~/svn/rcpp/pkg$ r /tmp/scott.r 
Loading required package: methods
              test replications elapsed   relative
7      sugar(n, k)          100   0.072   1.000000
9 sugarFloor(n, k)          100   0.088   1.222222
6       arma(n, k)          100   0.126   1.750000
4      david(n, k)          100   0.136   1.888889
8  armaFloor(n, k)          100   0.138   1.916667
3        ted(n, k)          100   0.384   5.333333
5       luis(n, k)          100   0.410   5.694444
1      scott(n, k)          100  33.045 458.958333
2  scottComp(n, k)          100  33.767 468.986111

We can see several takeaways:

• Rcpp sugar wins, which is something we have seen in previous posts on this blog. One hundred replication take only 72 milliseconds (or 88 in the corrected version) --- less than one millisecond per matrix creation.
• RcppArmadillo does well too, and I presume that the small difference is due not to code in Armadillo but the fact that we need one more 'mapping' of data types on the way back to R
• The sample() idea by David and Rafael is very, very fast too. This proves once again that well-written R code can be competitive. It also suggest how to make the C++ solution by foregoing (expensive) RNG draws in favour of sampling
• The approaches by Ted and Luis are also pretty good. In practice, the are probably good enough.
• Scott's function is not looking so hot (particularly as we increased the problem dimensions) and byte-compilation does not help at all.

Thanks to Scott and everybody for suggesting this interesting problem. Trying the rbinom() Rcpp sugar function, or implementing sample() at the C++ level is, as the saying goes, left as an exercise to the reader.

I claim that the way we conduct discussions on debian-devel has gone quickly downhill this year. The discussions around the default init implementation are particularly sad examples of how Debian debates technical topics. I've whined about this before, and people have suggested various things. Here's a list of things that I think we can conclude do not work:

• "Do not feed the troll" does not work, because there is always someone who can't resist responding. Also, some of the bad behavior is not trolling. Further, having everyone ignore people who behave badly does not give negative feedback for bad behavior, and does not build shared social norms for what is acceptable behavior.
• Unorganized private feedback (people mailing in response to public list posts) sometimes works, but it's a bit hit-and-miss, and again does not help build shared social norms. In other words, new members of the community see that others behave badly, and then nothing happens as far as they can see, whereas in private there may be quite a lot of activity.
• Up/down-voting a la Reddit or Stack Overflow works on those sites, but it is not the only thing that keeps their sites in shape. It also works badly on mailing lists, I think, and we can't very well switch away from the lists for all sorts of reasons. Voting can also result in a stronger "echo chamber", where the popular people or opinions are upvoted, regardless of technical merit.

Things that can work:

• Real-life meetings between participants. Debconf, sprints, FOSDEM, other such conferences. Unfortunately, this is expensive, and we can't reach everyone.
• Active public, but constructive confrontation when people behave badly. This needs good social skills to be done well.

Things that might be worthwhile to try:

• Changes to how our mailing list software works. Not sure what they would be, but Clay Shirky's Group as User: Flaming and the Design of Social Software may be a good source of inspiration.
• If not technical changes to the list software, possibly ways in which we use the tools we have. For example, a friend of mine suggested a while ago that when there is a topic on which consensus is difficult to achieve, each side of the debate (there may be any number) writes up their position and arguments for it and against the others on a wiki page. This might avoid going in circles in the discussions, repeating the same arguments over and over, during weeks and months of discussion.
• Public, but anonymous, feedback tools, a la Reddit and Stack Overflow. Perhaps as simple as adding "+1" and "-1" buttons to the web archive pages for messages, along the "Spam" button, and then publishing aggregate statistics? (A voting system, if you wish.)
• A consensus on the rough bounds of acceptable behavior and some group of people who are delegated the authority to enforce that.

The current situation, where a few hard-nosed, uncompromising people can prevent any rational discussion about init systems cannot be allowed to continue.

Some months ago I have run into a pretty interesting bug while working on a Ubuntu-based remote desktop system. The whole OS was installed on a server somewhere and users could access their desktop remotely. Some call this stuff Desktop-as-a-Service. The operating system we chose was Ubuntu Oneiric (11.10) and the remote access part was implemented with x2go, which uses nxagent to provide NX transport of X sessions. Users could access their Ubuntu machines remotely, with sound, video, youtube, and all you would expect from a desktop machine. The whole thing was working quite well. Now, as I said that was in May. Ubuntu 12.04 LTS was available, and the choice of upgrading to it sounded pretty much obvious. So we upgraded our test system to Precise and everything seemed to work smoothly. Till we tried to open a PDF document, actually. It took evince about 50 seconds to display a relatively small document. Same story with images opened with the default image viewer, eog. The fix delivered to our users was simple: we have set shotwell-viewer as the default image viewer, and epdfview as the default PDF viewer. Everybody happy. In the meantime, obviously, I was interested in this issue so I ve tried to run run evince from a terminal, getting the following output:

(evince:15833): GRIP-WARNING **: Failed to initialize gesture manager.

Funny. On another test system running Debian Sid (unstable) everything was working smoothly. The diff between Ubuntu s version of evince and Debian s is a 6MB monster. Among other changes, I noticed that Ubuntu s version build-depends on libgrip-dev, which depends on libutouch-geis. Multitouch stuff. Why should multitouch support break my remote session? So on May the 10th I filed a bug on launchpad. How this issue got handled is in my opinion one of the many fine examples of the inherent superiority of free software, coupled with a we won t hide problems mindset. For an example of how bad is the proprietary approach, just check a random bug in Adobe s bug tracking system. But let s go back to the evince bug. Other users reported that their VNC sessions were also affected by the same problem. After a few days it was clear that the culprit was utouch-geis, and a patch appeared. Unfortunately it did not actually address the issue. Somebody else reported that RDP sessions were broken too. At the beginning of June Precise was still affected. Finally, on August the 6th a working patch was submitted by Bradley M. Froehle and included by Chase Douglas (thank you guys). End of August, fixed version of geis accepted into precise-updates, case closed. Now for some considerations. The problem was clearly of a certain importance. A Long Term Support, stable version of Ubuntu shipped with broken PDF and image viewing functionalities. It got fixed properly, even though 3 good months are quite a long time for such a bug to get solved. However, the issue only affected a pretty limited number of users, also certainly not Ubuntu s main target audience. This bug never affected Debian, simply because utouch-geis has not made its way into the archive yet. It takes longer to make changes like this in Debian, but for some categories of users stability is more important than new, cool features. Choice is a good thing.

ELKI is a data mining software project that I have been working on for the last years as part of my PhD research. It is open source (AGPL-3 licensed) and avilable as both a Debian package and Ubuntu package in the official repositories. So a simple aptitude install elki should get you going and give you a menu entry for ELKI. These packages come with the scripts elki to launch the MiniGUI and elki-cli to run from command line. The key feature that sets ELKI apart from existing open source tools used in data mining (e.g. Weka and R) is that it has support for index structures to speed up algorithms, and a very modular architecture that allows various combinations of data types, distance functions, index structures and algorithms. When looking for performance regressions and optimization potential in ELKI, I recently ran some benchmarks on a data set with 110250 images described by 8 dimensional color histograms. This is a decently sized dataset: it takes long enough (usually in the range of 1-10 minutes) to measure true hotspots. When including Weka and R in the comarison I was quite surprised: our k-means implementation runs at the same speed as Rs implementation in C (and around twice that of the more flexible "flexclus" version). For some of the key agorithms (DBSCAN, OPTICS, LOF) we are an order of magnitude faster than Weka and R, and adding index support speeds up the computation by another factor of 5-10x. In the most extreme case - DBSCAN in Weka vs. DBSCAN with R-tree in ELKI - the speedup was a factor of 330x, or 2 minutes (ELKI) as opposed to 11 hours (Weka).
The reason why I was suprised is that I expected ELKI to perform much worse. It is written in Java (as opposed to R's kmeans, which is in C), uses a very flexible architecture which for example does not assume distances to be of type double and just has a lot of glue code inbetween. However, obviously, the Java Hotspot compiler actually lives up to its expectations and manages to inline the whole distance computations into k-means, and then compiles it at a level comparable to C. R executes vectorized operations quite fast, but on non-native code as in the LOF example it can become quite slow, too. (I would not take Weka as reference, in particular with DBSCAN and OPTICS there seems to be something seriously broken. Judging from a quick look at it, the OPTICS implementation actually is not even complete, and both implementations actually copy all data out of Weka into a custom linear database, process it there, then feed back the result into Weka. They should just drop that "extension" altogether. The much newer and Weka-like LOF module is much more comparable.) Note that we also have a different focus than Weka. Weka is really popular for machine learning, in particular for classification. In ELKI, we do not have a single classification algorithm because there is Weka for that. Instead, ELKI focuses on cluster analysis and outlier detection. And ELKI has a lot of algorithms in this domain, I dare to say the largest collection. In particular, they are all in the same framework, so they can be easily compared. R does of course have an impressive collection in CRAN, but in the end they do not really fit together. Anyway, ELKI is a cool research project. It keeps on growing, we have a number of students writing extensions as part of their thesis. It has been extremely helpful for me in my own research, as I could quickly prototype some algorithms, then try different combinations and use my existing evaluation and benchmarking. You need some time to get started (largely because of the modular architecture, Java generics and such hurdles), but then it is a very powerful research tool. But there are just many more algorithms, published sometime, somewhere, but barely with source code available. We'd love to get all these published algorithms into ELKI, so researchers can try them out. And enhance them. And use them for their actual data. So far, ELKI was mostly used for algorithmic research, but it's starting to move out into the "real" world. More and more people that are not computer scientists start using ELKI to analyze their data. Because it has algorithms that no other tools have. I tried to get ELKI into the "Google Summer of Code", but it was not accepted. But I'd really like to see it gain more traction outside the university world. There are a number of cool projects associated with ELKI that I will not be able to do myself the next years, unfortunately.

• A web browser frontend would be cool. Maybe even connected to Google Refine, using Refine for preprocessing the data, then migrating it into ELKI for analysis. The current visualization engine of ELKI is using SVG - this should be fairly easy to port into the web browser. Likely, the web browers will even be faster than the current Apache Batik renderer.
• Visual programming frontend. Weka, RapidMiner, Orange: they all have visual programming style UIs. This seems to work quite well to model the data flow within the analysis. I'd love to see this for ELKI, too.
• Cluster/Cloud backend. ELKI can already handle fairly large data sets on a big enough system. If someone spends extra effort on the index structures, the data won't even need to fit into main memory anymore. Yet, everybody now wants "big data", and parallel computation probably is the future. I'm currently working on some first Hadoop YARN based experiments with ELKI. But this is a huge field, turning this into true "elk yarn". I will likely only lay some foundations (unless I get funding to continue as a PostDoc on this project. I sure hope to get to do at least a few years of postdoc somewhere, as I really enjoy working with students on this kind of project)
• New visualization engine. The current visualization engine, based on Apache Batik and SVG is quite okay. It does what I need, which is to get a quick glance at the results and the ability to export them for publications in print quality. (in particular, I can easily edit the SVG files with Inkscape) But it is not really something fancy (although we have a lot of cool visualizations). And it is slow. I havn't found a portable and fast graphics toolkit for Java yet that can produce SVG files. There is a lot of hype around processing, for example, but it seems to be too much about art for me. In fact, I'd love to use either something like Clutter or Cairo. But getting them to work for Windows and Mac OSX will likely be a pain.
• Human Computer Interaction (HCI). This is in my opinion the biggest challenge we are facing with all the "big data" stuff. If you really go into big data (and not just run Hadoop and Mahout on a single system; yes - a lot of people seem to do this), you will at some point need to go beyond just crunching the numbers. So far, the challenges that we are tackling are largely data summarization and selection. TeraSort is a cool project, and a challenge. Yet, what do you actually get from sorting this large amount of data? What do you get from running k-means on a terabyte? When doing data mining on a small data set, you quickly learn that the main challenge actually is preprocessing the data and choosing parameters the right way so that your result is not bogus. Unless you are doing simple prediction tasks, you often don't have a clearly defined objective. Sure, when predicting churn rates, you can hope to just throw all the data into a big cloud and hope you get some enlightement out. But when you are doing cluster analysis or outlier detection - unsupervised methods - the actual objective by definition cannot be hardcoded into a computer. The key objective then is learn something new on the data set. But if you want to have your user learn something on the data set, you will have to have the user guide the whole process, and you will have to present results to the user. Which gets immensely more difficult with larger data. Big data just does no longer look like this. And neither are the algorithms as simple as k-means or hierarchical clustering. Hierarchical clustering is good for teaching the basic ideas of clustering. But you will not be using a dendrogram for a huge data set. Plus, it has a naive complexity of O(n^3) and for some special cases O(n^2) - too slow for truly big data.
  For the "big data future" once we get over all the excitement of being able to just somehow crunch these numbers we will need to seriously look into what to do with the results (in particular, how to present them to the user), and how to make the algorithms accessible and usable for non-techies. Right now, you cannot expect a social sciences researcher to be able to use a Hadoop cluster. Yet to make sense of the results. But if you are a smart guy to actually solve this, and open up "big data processing" to the average non-IT user, this will be big.
• Oh, and of course there are just hundreds of algorithms not yet available (accessible) as open source. Not in ELKI, and usually not anywhere else either. Just to name a few from my wishlist (I could probably implement many of them in a few hours in ELKI, but I don't have the time to do so myself, plus they are good student or starter project to get used to ELKI): BIRCH, CLARA, CLARANS, CLINK, COBWEB, CURE, DOC, DUSC, EDSC, INSCY, MAFIA, P3C, SCHISM, STATPC, SURFING, ... just to name a few.

If you are a researcher in cluster analysis or outlier detection, consider contributing your algorithms to ELKI. Spend some time optimizing them, adding some documentation. Because, if ELKI keeps on growing and gaining popularity, it will be the future benchmark platform. And this can give you citations, which are somewhat the currency of science these days. Algorithms available in the major toolkits just do get cited more, because people compare to them. See this list for an overview of work cited by ELKI - scientific work that we reimplemented at least to some extend for ELKI. It is one of the services that we provide with ELKI for researchers: not only the algorithm, but also the appropriate citation.

DPU, as I ve mentioned casually, is a testing framework for Debian tools. I ll expand a bit more about what DPU does, what it s going to be used for, and how it can help you (yes, you!) create tools to work with Debian packages. Firstly, DPU is a test framework. It s sole purpose is only to run tests, and give each test a pass / fail grade. The test syntax will be as declarative and simple as one can keep such things, and tries to do the right thing(tm). It s not currently stable, but it s getting there. DPU aims to replace Lintian s test suite, and clean up the tests to make them a bit more declarative, and less hackish. At it s core, DPU uses a templating system (using Jinja2) to render out a policy-perfect Debian source directory, including (if needed) any number of fake Upstream files, such as Hello World! in Shell (for arch:all), in C (for arch:any), and so forth. DPU tests mangle (slightly) the files before a run simple stuff like removing a manpage, or adding a file where it shouldn t be. After the builder builds the package (with any of the global or test-local builders), any number of checkers (again, global or local) are run, and output compared. This is the part that s currently undergoing work, but the basics are alive, and an example test directory is on my github. Everything s in Python, and new contributors are welcome!

---

Today, I've merged my Gnus notifications module inside Gnus git repository. This way, it will be available for everybody in Emacs 24.2. This module allows you to be notified via notifications-notify (the Emacs implementation of the Freedesktop desktop notifications) on new messages received in Gnus. It can also retrieves contacts photo via gravatar.el and google-contacts.el to include them in the notification. To enable it in Emacs > 24.1, you just have to add the following line to your Gnus configuration file:

(add-hook 'gnus-after-getting-new-news-hook 'gnus-notifications)

If you want to download it and use it stand-alone for a previous Emacs version, you can fetch the latest file revision and load it before adding the previously given line.