Search Results: "mario"

13 March 2014

Mario Lang: Living in RAM

I switched to SSD on my home server a few years ago, at a time when SSH was still new and sort of hyped. However, I immediately knew that this type of technology is for me: it is fast and rather silent. SSD storage has this air of being fragile though. You are supposed to not wear it out too much. It's not like in the good old days where you would write to your hard disk without any extra thought.
tmpfs Before I switched to SSD I was already using tmpfs to store /tmp in RAM, for performance reasons mostly. /etc/default/tmpfs:
RAMTMP=yes
Since RAM is relatively cheap these days I have lots of it in my home server.
/var/cache/apt/archives When I switched to SSD, I extended this concept by mounting /var/cache/apt/archives via tmpfs. This is to save disk I/O for package updates. And for a machine running sid which is more or less regularily updated, this saves quite some useless writes to disk. Roughly 100MB per update.
/tmp/src Nowadays, I have almost all of my source code checkouts and build-trees in /tmp/src. myrepos makes this very easy to handle. Simply clone all repositories you are interested in, and remember their details with mr register. After a reboot, all you need to do is:
$ mkdir /tmp/src
$ cd !$
$ mr up
Your repository locations have (likely) been recorded in ~/.mrconfig. Of course there is the danger of accidentally loosing data due to a power outage. However, I feel this leads to a rather clean workflow. Changes need to be pushed anyway. So since you can not trust your local repository to really persist, you are forced to push your changes regularily, which is a good thing! Besides, I have a UPS at home which gives me roughly one hour of backup power. This was enough to sustain every power outage I have witnessed in the past years. Typically, outages are rather short lived in my area, somewhere between a few seconds and 5 minutes. It is rather rare that electricity goes missing for more than half an hour. The only downside of this approach is that from time to time, you will use more bandwidth by re-cloning repositories after a reboot. But really, who needs reboots on Linux? Last time I had half a year uptime, and only rebooted because I wanted to jump from linux 3.8 to 3.13. This approach only works on long-running machines though. It is probably not very useful for a laptop.

4 March 2014

Mario Lang: Personal mail server

I used to use Fetchmail and Gnus with its mail splitting capabilites and nnml backend to handle my private mail setup for many years. This worked pretty fine since I am used to accessing my home server through SSH. I have Gnus running inside of a GNU screen session. So I can check mail from remote by using a SSH terminal. This works fine as long as I have a good SSH terminal on a desktop or laptop computer. However, it does not work very well on tablets or smaller mobile devices. Additionally, mail splitting has become a performance burden over the years. I do not want to wait for Gnus to sort incoming mail into different folders while checking for new mail. That is something which should have been done in the background already, and thats what we are going to cover with the setup described below. So I had to change my simple setup to accomodate for the new trends in mobile computing. The obvious core of such a setup is an IMAP server which receives and stores your mail such that different clients can access it. So the time of my Gnus nnml storage are definitely over. Mail is no longer stored in and by my mail client.
Dovecot While there are several IMAP server solutions out there, I find Dovecot fits my needs quite nicely. I've decided to store my mail in Maildir format in ~/Maildir. I prefer storing data like that in the home directory to avoid having to backup separate files from /var/. Maildir also features some index files which should help performance in the long run. Incoming mail will solely be delivered by fetchmail and should be checked for spam. While I can probably configure Exim to run SpamAssassin on mails before delivering them to Dovecot, there is a much more elegant solution: the dovecot local delivery agent (LDA). /usr/lib/dovecot/deliver takes mail from standard input and performs Sieve filtering and updates the mail indexes. We will call this executable more or less directly from fetchmail. Incoming mail from mailing lists will be sorted into different folders using Sieve. Dovecot needs to be told to enable the sieve plugin and to create new folders on demand. /etc/dovecot/local.conf:
disable_plaintext_auth = yes
mail_location = maildir:~/Maildir
lda_mailbox_autocreate = yes
lda_mailbox_autosubscribe = yes
protocol lda  
  mail_plugins = sieve
 
Sieve scripts are actually quite intuitive once you have a template to start from. ~/.dovecot.sieve:
require "fileinto";
if exists "X-Spam-Flag"  
  # Store spam tagged by SpamAssassin into dedicated Spam folder
  if header :contains "X-Spam-Flag" "YES"  
    fileinto "Spam";
   
  elsif exists "X-Cron-Env"  
  # Store mails from Cron daemon in dedicated folder
  fileinto "cron";
  elsif exists "List-Id"  
  # File list-mail into dedicated folders, matching on List-Id
  if header :contains "List-Id" "boost-users.lists.boost.org"  
    fileinto "boost-users";
    elsif header :contains "List-Id" "brltty.mielke.cc"  
    fileinto "brltty";
    elsif header :contains "List-Id" "debian-accessibility.lists.debian.org"  
    fileinto "debian-accessibility";
    elsif header :contains "List-Id" "debian-devel-announce.lists.debian.org"  
    fileinto "debian-devel-announce";
    elsif header :contains "List-Id" "debian-devel.lists.debian.org"  
    fileinto "debian-devel";
    elsif header :contains "List-Id" "spirit-general.lists.sourceforge.net"  
    fileinto "spirit-general";
   
  # ...
 
SpamAssassin Since I want automatic classification of spam messages, I use SpamAssassin. Just install spamassasin and enable spamd in /etc/default/spamassassin:
ENABLE=1
We will use spamc in the Fetchmail configuration.
Fetchmail My ~/.fetchmailrc is a straightforward list of some mailboxes to fetch mail from. I use the mda directive to skip the MTA and send mail through SpamAssassin and deliver it to Dovecot via its LDA mechanism. ~/.fetchmailrc:
set daemon 1200 # Poll at 10 minute intervals
poll blind.guru protocol IMAP: ssl;
# ... add more sources here ...
mda "/usr/bin/spamc -u %T -e /usr/lib/dovecot/deliver -d %T"
To avoid spreading access information in too many configuration files I am using the ability of Fetchmail to use netrc to retrieve account passwords. ~/.netrc:
machine blind.guru login mlang password <hidden>
Gnus I am using Gnus to read mail, newsgroups and RSS feeds since many years now. It would be quite a mouthful to explain all the customizations I am using by now. But there is one very important bit in the context of this article: How to access the IMAP server? In my setup, Emacs and therefore Gnus is running on the same machine as the IMAP server. So I can avoid authentication at all. This configuration will avoid unnecessary password prompts or caching. In ~/.emacs or ~/.gnus:
(setq gnus-secondary-select-methods '((nnimap "localhost"
                                       (nnimap-stream shell)))
      nnimap-shell-program "/usr/lib/dovecot/imap")
With this you should be able to subscribe to your IMAP folders from within Gnus with ease. Sorting incoming mails into folders is now performed by the IMAP server through Sieve scripts. Instead of changing Gnus' configuration I now edit ~/.dovecot.sieve when I subscribe to a new mailing list. If you add a new Sieve rule for a mailing list and the associated folder does not exist yet, Dovecot will autocreate it, very convenient.
Mobile devices Now all that is left is a way for your mobile devices to read and eventually send mail. This is very much dependant on your network setup, so I am not going to go into any detail here. If you are accessing your mail setup from a tablet in your local network you might get away without tinkering with your router configuration. If you want to read/send mail on the go you need some way to get to your external IP. Either it is stable enough or you need some dynamic DNS service. You will definitely want to forward IMAP and maybe SMTP ports from your router to your home server. If you don't have an existing SMTP server for your mobile device that accepts your outgoing mails you can also set one up yourself and deliver outgoing mails from your mobile device to the world with Exim or qmail. I am personally using Exim since I am going with the default MTA for Debian. Configuring Exim to take mail from iOS devices was as simple as enabling an appropriate authentication method and adding an account to /etc/exim4/passwd. I have to admit though that I don't particularily like Exim's configuration files. That is why I ended up using dovecot's LDA in the first place.

1 March 2014

Sune Vuorela: Diploma thesis about media choice and usage in Free Software communities: I need your help

My hard-working KDE friend Mario asked me to help him to get Debian people to help him with his thesis. Here is what he writes: Dear Free Software contributor* I m currently in the process of writing my diploma thesis. I ve worked hard during the last few weeks and months on a questionnaire which shall collect some data for my thesis. Furthermore the data of this survey will be interesting for the Free Software communities as well. So please take some time or add it to your todo list or, even better, go directly to my questionnaire and help me make a great diploma thesis and improve the Free Software community in some ways. The questionnaire takes some 20 to 30 minutes. At the end of the questionnaire you ll find a way to participate in a draw where you can even win something nice. In a first round I got the feedback that the length of the questionnaire and that some questions (mostly the ones at the beginning of the questionnaire about the 12 different tasks) are quite abstract and difficult. But please try it, try your best and take the time and brain power. The remaining part of the questionnaire [1] (after these two pages with the tasks questions) is quite
easy and quickly done. And you have the possibility to come back to where you
have left filling in the questionnaire after a shorter or longer break. And if there are any questions, feedback or you need help don t hesitate a moment to write me an email or ping me on IRC (freenode.net and oftc.net) as unormal. This survey will be open till Sunday, the 9th of March 2014, 23.59 UTC. Thanks to all for reading and helping and towards the summer of 2014 you can read here what all the data you gave me showed us and where we can learn and improve. Thanks in advance and best regards
Mario Fux * By contributor I mean not just developers but translators, artist, usability
people, documentation writers and many more. Everybody who contributes in one
way or the other to Free Software.

25 February 2014

Mario Lang: I am a programming language

I love learning about programming languages. But this one really took me by surprise. Yes, apparently there is a BrainFuck alike two-dimensional esoteric programming language called MarioLANG. And GitHub even has an implementation written in Ruby. I should really allocate a bit of spare time to write at least something in myself.

19 February 2014

Mario Lang: Enhancing Hub to support GitHub social features

In my article Contributing on GitHub I recently explained that I can not use certain social features of GitHub on their website directly. I also explained that I make great use of Hub to circumvent some accessibility issues I have with GitHub. Well, it turns out that adding star, unstar, follow and unfollow to Hub was actually a lot easier then I thought. It is long ago (probably 10 years) that I touched Ruby code the last time. However, since I do learn a lot from examples, I was actually able to add what I needed just by looking at surrounding code. Well, I had to lookup how you add an element to the end of an array (push) but that was basically it. You can have a look at the final outcome in Hub pull request #490. The code resides in the social branch of my hub fork. If you already have an OAuth token for hub, you need to login to GitHub and edit your token to add the user:follow scope if you want to use the follow and unfollow commands. Interestingly, this type of operation is accessible enough to still work with Lynx! I hope this gets merged, I don't particularily like to maintain my own forks of quite obvious functionality. Anyway, problem solved, mostly. All I am missing now is the ability to delete a repository. A dangerous feature, so probably controversial. Anyway, something for another day.

18 February 2014

Mario Lang: Conntributing on GitHub

I really like to read code written by different people. I've always been the type of learner that can cover a lot of ground by looking at examples. A few of the programming tricks I acquired over time have definitely been learnt by reading programs written by other people. There are of course cases of disagrement with coding style (lets put that very mildly), so not every project is fun to read. But I tend to find projects I like enough to actually look at their source code. Now, if you get into this habit, there is also the occasional spotting of a bug. Many are really cosmetic, like typos in comments or documentation. Others are maybe more serious, who knows what can happen if you read a few lines of code written by someone you have never met in person. This is one of the original arguments of open source and free software. It basically goes like: "People will read your code and report problems." I know that happens, but I also know that there are cases of people finding relatively minor things during a small code review, which they don't report, or which get lost in some developers INBOX. I remember waiting months to see memory corruption fixes for a particular project I submitted by email to actually appear in the development repository. In some cases you really need to track your patches and prod someone to remind them about your contribution if need be. True, some of these issues are probably minor enough to not really hurt if they get lost, but I have a feel that a lot of minor things also sort of add up. This is where GitHub comes into play. We have had project hosting services since at least a decade now, but none of them (to my experience) have ever made it so easy for all involved parties to submit and merge changes into projects. Pull requests are just amazing. Well, at least for me, they are since a few months. I have a sort of mixed hate-love relationship with GitHub when it comes to a topic that is very important to me, Accessibility. The last project hosting site I really liked for their simple web interface (read, still usable with Lynx) was Google Code. That was in the days when Google still seemed to care about such things. These days, their newer services are next to unusable to me (with the tools I prefer). SourceForge was always a catastrophy. Many things were cumbersome to do, because the site is so loaded with links to all sorts of things I never need. However, at the time I was using it, all the things I really needed did work. GitHub is a different beast, in a different era of the internet. While I can read project pages and a few basic things perfectly fine, some parts of the GitHub site are simply unusable for no particular reason. For instance, I can edit (some of my) preferences and even save them. But I can not star a project, nor can I follow other developers. Invoking these links gives me a 404, which seems to indicate that some JavaScript (read, client side code execution) is required which my browser does not perform. However, I really wonder why. Because something like "I want to star this project" will eventually have to end up as a request on the server anyway, so it seems technically unnecessary to rely on Javascript for such an operation to succeed. Luckily, these days, hip projects do have an API. An GitHub is no exception. I want to emphasis that an API alone is not an accessibility fix, since you rarely have the time to implement a clinet for a random API, just to get access to a service you want to use. Sometimes you do, and an API is a great enabler in such situations, but a reference client with a rich feature-set is much better. I have to admit that I haven't evaluated any of the possible alternatives, but one client I found for GitHub's API did really boost my productivity in the last year. I am talking about Hub. It is a simple wrapper around Git that adds a few and enhances some other git commands. Most importantly, hub fork will fork the current checkout on GitHub, and hub pull-request allows you to quickly (and I really mean quickly!) open a pull request for some commits you've just done to your fork of some project. This is very helpful, and very accessible. While it does not solve all my accessibility problems with GitHub, it at least enables me to take part in the GitHub workflow. I can easily fork projects and submit pull reuqests for changes I did. And that is what I did, at least a bit, in roughly the past year. I am not sure exactly, but I think it was spring when someone made me aware of Hub. One approach would be to check my GitHub activity graph, but I probably don't have to mention that it is not accessible. Here is a list of some of my contributions to open source projects on GitHub since I discovered a good client for the GitHub API: To summarize: I love GitHub for what it does to the free software community. I think the simplicity (and transparency) of contributions is to the benefit of free and open source software. However, I also hate GitHub sometimes when I need to ask a friend to do relatively trivial changes for me. What I actually want, is a command-line client like Hub that makes most or even all functionality provided on the website available. True, some things like graphs are not immediately easy to adapt. But I really shouldn't need to use a modern full-fledged web browser to tell GitHub that I am interested in the activities of a certain user. Or delete a fork. Yes, thats right. I can use Hub to fork a project, but there is no way to delete it again. Sort of strange for casual small-scale contributions. I guess the workflow there is: fork, commit, pull-request, wait for merge, delete. I don't want to have all sorts of forks on GitHub that I actually don't actively use/maintain. So once a pull request was merged, and I don't have any other things I am working on right now, I actually want to get rid of the fork again. It is easy enough to do a fork should I ever need it again.

Mario Lang: Contributing on GitHub

I really like to read code written by different people. I've always been the type of learner that can cover a lot of ground by looking at examples. A few of the programming tricks I acquired over time have definitely been learnt by reading programs written by other people. There are of course cases of disagrement with coding style (lets put that very mildly), so not every project is fun to read. But I tend to find projects I like enough to actually look at their source code. Now, if you get into this habit, there is also the occasional spotting of a bug. Many are really cosmetic, like typos in comments or documentation. Others are maybe more serious, who knows what can happen if you read a few lines of code written by someone you have never met in person. This is one of the original arguments of open source and free software. It basically goes like: "People will read your code and report problems." I know that happens, but I also know that there are cases of people finding relatively minor things during a small code review, which they don't report, or which get lost in some developers INBOX. I remember waiting months to see memory corruption fixes for a particular project I submitted by email to actually appear in the development repository. In some cases you really need to track your patches and prod someone to remind them about your contribution if need be. True, some of these issues are probably minor enough to not really hurt if they get lost, but I have a feel that a lot of minor things also sort of add up. This is where GitHub comes into play. We have had project hosting services since at least a decade now, but none of them (to my experience) have ever made it so easy for all involved parties to submit and merge changes into projects. Pull requests are just amazing. Well, at least for me, they are since a few months. I have a sort of mixed hate-love relationship with GitHub when it comes to a topic that is very important to me, Accessibility. The last project hosting site I really liked for their simple web interface (read, still usable with Lynx) was Google Code. That was in the days when Google still seemed to care about such things. These days, their newer services are next to unusable to me (with the tools I prefer). SourceForge was always a catastrophy. Many things were cumbersome to do, because the site is so loaded with links to all sorts of things I never need. However, at the time I was using it, all the things I really needed did work. GitHub is a different beast, in a different era of the internet. While I can read project pages and a few basic things perfectly fine, some parts of the GitHub site are simply unusable for no particular reason. For instance, I can edit (some of my) preferences and even save them. But I can not star a project, nor can I follow other developers. Invoking these links gives me a 404, which seems to indicate that some JavaScript (read, client side code execution) is required which my browser does not perform. However, I really wonder why. Because something like "I want to star this project" will eventually have to end up as a request on the server anyway, so it seems technically unnecessary to rely on Javascript for such an operation to succeed. Luckily, these days, hip projects do have an API. An GitHub is no exception. I want to emphasis that an API alone is not an accessibility fix, since you rarely have the time to implement a clinet for a random API, just to get access to a service you want to use. Sometimes you do, and an API is a great enabler in such situations, but a reference client with a rich feature-set is much better. I have to admit that I haven't evaluated any of the possible alternatives, but one client I found for GitHub's API did really boost my productivity in the last year. I am talking about Hub. It is a simple wrapper around Git that adds a few and enhances some other git commands. Most importantly, hub fork will fork the current checkout on GitHub, and hub pull-request allows you to quickly (and I really mean quickly!) open a pull request for some commits you've just done to your fork of some project. This is very helpful, and very accessible. While it does not solve all my accessibility problems with GitHub, it at least enables me to take part in the GitHub workflow. I can easily fork projects and submit pull reuqests for changes I did. And that is what I did, at least a bit, in roughly the past year. I am not sure exactly, but I think it was spring when someone made me aware of Hub. One approach would be to check my GitHub activity graph, but I probably don't have to mention that it is not accessible. Here is a list of some of my contributions to open source projects on GitHub since I discovered a good client for the GitHub API: To summarize: I love GitHub for what it does to the free software community. I think the simplicity (and transparency) of contributions is to the benefit of free and open source software. However, I also hate GitHub sometimes when I need to ask a friend to do relatively trivial changes for me. What I actually want, is a command-line client like Hub that makes most or even all functionality provided on the website available. True, some things like graphs are not immediately easy to adapt. But I really shouldn't need to use a modern full-fledged web browser to tell GitHub that I am interested in the activities of a certain user. Or delete a fork. Yes, thats right. I can use Hub to fork a project, but there is no way to delete it again. Sort of strange for casual small-scale contributions. I guess the workflow there is: fork, commit, pull-request, wait for merge, delete. I don't want to have all sorts of forks on GitHub that I actually don't actively use/maintain. So once a pull request was merged, and I don't have any other things I am working on right now, I actually want to get rid of the fork again. It is easy enough to do a fork should I ever need it again.

14 February 2014

Mario Lang: Type erasure

Andrzej's C++ blog has a nice series on type erasure. I found it interesting to read and have learnt some things from it. For instance, I vaguely guessed, but did not fully realize, that std::function<> is of course a performance hit, since the way how std::function<> is implemented makes it impossible for the compiler to do anything useful with that function call. Inlining is totally out of the question, since we have just erased the type information. You might say, "of course!", but for me it was sort of a revelation. The series is in four parts: part I, part II, part III, part IV.

12 February 2014

Mario Lang: Roughly 1500 source packages have possibly broken links in debian/control

There are currently roughly 1500 source packages in Debian which possibly (very likely actually) do have broken URLs in debian/control. While it is quite useful that we have VCS information and Homepage URLs in the Packages file these days, we also created a rather big source of bitrot. These URLs are typically paste-and-forget. Sure, people occasionally catch the fact that a homepage or VCS has moved, especially if they are active and in good contact with their upstreams. However, there are also other cases...
DUCK to the rescue! My coworker Simon Kainz has worked on a service that helps at least to track which URLs are currently broken. We've initially discussed some way to make this a part of Lintian, which is what I would have prefered. However, for good reasons, Lintian doesn't want to call out to the net by default, so these checks would likely not get run by many developers anyway. So Simon ended up creating DUCK - the Debian Url ChecKer which actually goes out to the net and verifies that all the Vcs-* and Homepage fields of debian/control are actually reachable. The frontend allows developers to search for packages they maintain, to quickly see if they have any URLs which are possibly broken. There is a slight chance of temporary network problems of course, so what DUCK does is to show the status of checks in the last few days, so that you quickly see if you are dealing with a typical false positive. First of all, thanks to Simon! I think DUCK is an excellent project for a future NM candidate. I actually already wanted to advocate him for DD, but we ended up on a website which suggested that new contributors should start by applying for DM these days, and only later go for DD. I find that actually quite strange, especially in Simons case, but well, we did not feel like argueing. Secondly, and thats also very important: whats needed to improve the overall quality of URLs in the package system is your attention! You can easily search for email addresses of maintainers or uploaders. Team members can create a bookmark entry that checks for problems in all packages maintained by the teams they are a member of. You just need to actually visit these pages from time to time. It would probably not be well received if we filed all these bugs at once :-). So we need you to care, since we don't want to generate too much noise regarding "just these broken URLs". Or are they? If your vcs fields are broken, debcheckout will not work properly. Which defeats the purpose of debcheckout. If your homepage URL is broken, packages.d.o will also have a wrong link. 1500 packages with broken URLs. Don't you think we can do better then that?

10 February 2014

Mario Lang: Neurofunkcasts

I have always loved Drum and Bass. In 2013 I rediscovered my love for Darkstep and Neurofunk, and found that these genres have developed quite a lot in the recent years. Some labels like Black Sun Empire and Evol Intent produce mixes/sets on a regular basis as podcasts these days. This article aggregates some neurofunk podcasts I like a lot, most recent first. Enjoy 33 hours and 57 minutes of fun with dark and energizing beats. Thanks to BSE Contrax and Evol Intent for providing such high quality sets. You can also see the Python source for the program that was used to generate this page.

8 February 2014

Mario Lang: Uploading Into and Downloading From The Universe

Have you ever wondered how new ideas, concepts, artworks etc. get into the Superconscious Collective of the universe, so that they then can be downloaded by exactly the right person who is meant to do something with them? yep, I, too, think that many of these ideas, concepts etc. come from the Great Unknown, where the seeds of limitless ideas and possibilities reside. But a lot of them also come from our own minds, from people who upload these things into the Superconscious Collective of our universe. Such ideas and concepts can be uploaded in several ways. Some of them are simply uploaded by a person who sources them, connects to the Superconscious Collective of the universe and streams them up. Others are maybe even uploaded by somebody virtually typing an article with their fingertips into the spread out hands or soles of the feet of their loved one, playfully teasing him/her and thereby streaming information into him/her, which s/he then again uploads, mainly subconsciously, into the Superconscious Collective of the universe. Once uploaded, all these limitless ideas and concepts become available to everyone and every living being, on Earth and throughout the whole universe. The thing is, that most people on Earth don't know how to consciously access information from there. Some do unconsciously and it again shows up in them as entirely new concepts and ideas, which they then spice with the flavor of their own essence and then stream them out into the world, be it as music, as software, as hardware, as artwork, as science and in limitless other ways. But it is also accessed by living beings throughout the whole universe, who also add their own ideas and concepts to the Great Superconscious Collective . Wanna know how to access this limitless information, ideas, concepts and/or possibilities from the Superconscious Collective? Just breathe, relax, expand ... expand and get into your biggest, greatest, awesomest being of you. Become all that you are, connected to everything. Now, consciously plug into the Great Superconscious Collective, the Great Cosmic Internet. Allow the ideas, information, possibilities stream into you and through you and look which ones you resonate with. Call them, feel them, let them land in you, and ask them what they want you to do with them, as you are the one who can express them in the highest possible way, otherwise they wouldn't have come to you in the first place. Then do one small thing to make them real in the world and watch them unfold. Or do whatever feels right for you to do, and create the most amazing new thing in the world.

Mario Lang: boost::python and boost::variant

I have unsuccessfully tried to find a solution for the following problem on the internet several times. Now that I have come at least closer to a usable approach, I thought I'd document what I have found so that others trying to achieve a similar thing can use this as a starting point. Boost.Python offers a very nice and flexible way to interface C++ data types with Python. With just a few lines of code, and the proper linker flags, you get a Python importable shared object from your C++ compiler. This can be very productive. However, there is one aspect of C++ data types that I couldn't figure out how to interface with Python, which are C++ discriminated unions, or more specifically, heterogeneous containers. While Python has no problems with containers containing objects of different types, C++ does not make this very easy by default. Usually the problem is solved with a container of pointers to a base class, and various subclasses with virtual functions. However, this approach is not always practical, especially if the different types of objects in an heterogeneous container dont have many things in common. This is where discriminated unions come to the rescue. They basically behave like a normal union in C, but have an additional field which indicates the type of object currently stored in the union. Boost.Variant does exactly that, with a nice visitor interface added on top of it.
Heterogeneous containers in C++ If we put the boost::variant<> template inside a STL container like std::vector<>, the result is a heterogeneous container. For the purpose of illustration, lets implement such a container. The example below is deliberately simple. In reality, the various types allowed in your variant will probably have more fields then just one.
#include <boost/variant.hpp>
#include <vector>
struct a   int x;  ;
struct b   std::string y;  ;
typedef boost::variant<a, b> variant;
typedef std::vector<variant> vector;
To ease creation of these two types of objects, we are going to write a few factory functions. We are going to wrap them in Python later on.
variant make_variant()   return variant();  
vector make_vector()   return vector a(), b(), a() ;  
Boost.Python Now lets create a Python module which exports the above functionality to Python.
#include <boost/python/class.hpp>
#include <boost/python/def.hpp>
#include <boost/python/implicit.hpp>
#include <boost/python/init.hpp>
#include <boost/python/module.hpp>
#include <boost/python/object.hpp>
#include <boost/python/suite/indexing/vector_indexing_suite.hpp>
vector_indexing_suite apparently needs operator== defined on the value_type of the container. In our case, this is our boost::variant<a, b> type. Luckily, boost::variant<> already provides operator==. However, that operator== relies on operator== being defined for the underlying types. Since equality comparison is probably useful for other things as well, lets just create operator== for our two classes a and b.
bool operator==(a const &lhs, a const &rhs)   return lhs.x == rhs.x;  
bool operator==(b const &lhs, b const &rhs)   return lhs.y == rhs.y;  
Convert a boost::variant<> to PyObject * Boost.Python needs a way to convert our discriminated union to a Python object. This code relies on Python class definitions being present for all underlying variant types. We will define them later.
struct variant_to_object : boost::static_visitor<PyObject *>  
  static result_type convert(variant const &v)  
    return apply_visitor(variant_to_object(), v);
   
  template<typename T>
  result_type operator()(T const &t) const  
    return boost::python::incref(boost::python::object(t).ptr());
   
 ;
And finally, lets create our Python module.
BOOST_PYTHON_MODULE(bpv)  
  using namespace boost::python;
  class_<a>("a", init<a>()).def(init<>()).def_readwrite("x", &a::x);
  class_<b>("b", init<b>()).def(init<>()).def_readwrite("y", &b::y);
  to_python_converter<variant, variant_to_object>();
  implicitly_convertible<a, variant>();
  implicitly_convertible<b, variant>();
  def("make_variant", make_variant);
  class_<vector>("vector").def(vector_indexing_suite<vector, true>());
  def("make_vector", make_vector);
 
Compiling Lets create a shared object for Python.
$ g++ -std=c++11 -fPIC -shared $(python-config --includes) -o bpv.so file.cpp -lboost_python
Running We can load the module into Python and see what it does.
>>> import bpv
>>> variant=bpv.make_variant()
>>> variant
<bpv.a object at 0x7f06bb2130c0>
>>> variant.x
0
>>> variant.x=2
>>> variant.x
2
Nice. We can access the underlying type, and even modify it. Lets see how our heterogeneous container wrapping code behaves.
>>> vector=bpv.make_vector()
>>> vector
<bpv.vector object at 0x7f20693289d0>
>>> len(vector)
3
>>> list(vector)
[<bpv.a object at 0x7f20693190c0>, <bpv.b object at 0x7f20693193d0>, <bpv.a object at 0x7f2069319440>]
So far, so good. This will at least make it possible to convert heterogeneous containers from C++ to Python, which was my initial goal. Unfortunately, contained objects are not treated as references. Whenever retrieved, we get a copy. So in-place modification does not work.
>>> vector[0].x
0
>>> vector[0].x=2
>>> vector[0].x
0
However, we can override an existing element with a modified copy.
>>> e0=vector[0]
>>> type(e0)
<class 'bpv.a'>
>>> e0.x = 2
>>> vector[0] = e0
>>> vector[0].x
2
And we can also use the append and extend methods of Python containers.
>>> len(vector)
3
>>> vector.extend(vector)
>>> vector.append(bpv.a())
>>> len(vector)
7
>>> len(filter(lambda x: type(x)==bpv.b, vector))
2
>>> len(filter(lambda x: type(x)==bpv.a, vector))
5
>>> map(lambda x: x.x, filter(lambda x: type(x)==bpv.a, vector))
[2, 0, 2, 0, 0]
All that is missing for a perfect world is reference semantics for container elements. If anyone has a hint on how to achieve this, please let me know.

7 February 2014

Mario Lang: Solara: Fun with accessible RPGs on iOS

Solara is a fun turn-based fantasy strategy game for iOS which works perfectly well with VoiceOver. AppleVis has a review and all the details like a link into the AppStore. You build your castle level by level, and train your heroes to be stronger in battle. While your heroes are at battle, you can not control any real-time aspect of a battle. The combination of your heroes and their individual strengths (levels) is essential to determine if you can win a battle. So it sort of resembles rolling dice. At the beginning of a battle, you can choose which of your various heroes you'd like to send. You normally have four slots to fill, in earlier levels of Solara even less (to ease you into game play). Fighting is done in two different parts of the game. You need to use your heroes to solve quests, which will be rewarded by experience points and gold, which in turn will increase your playing level. From level 10 on, you will also be able to play in the arena against other player of solara. These arena fights are rewarded with arena points which in turn determine your place in an arena turnament. These turnaments usually last a few days. At the end, the top places are rewarded with gold. The way solara is structured makes it a perfect fit for VoiceOver. There are no real-time gesture problems to solve. It is basically an interactive text adventure with a fantasy theme and a weighted skills mechanism for determining the outcome of a fight. And luckily, the authors of solara realized this and did everything necessary to make this game really nicely playable with VoiceOver. My personal comment: I initially felt its implemented nicely, but playing it would be a waste of time. Now, a few months later, I am at level 39, and have seen hints that there are 50 levels in total. While I am not really hooked, I guess I will finish it eventually :-)

2 July 2013

Ond&#345;ej &#268;ert&iacute;k: My impressions from the SciPy 2013 conference

I have attended the SciPy 2013 conference in Austin, Texas. Here are my impressions.

Number one is the fact that the IPython notebook was used by pretty much everyone. I use it a lot myself, but I didn't realize how ubiquitous it has become. It is quickly becoming the standard now. The IPython notebook is using Markdown and in fact it is better than Rest. The way to remember the "[]()" syntax for links is that in regular text you put links into () parentheses, so you do the same in Markdown, and append [] for the text of the link. The other way to remember is that [] feel more serious and thus are used for the text of the link. I stressed several times to +Fernando Perez and +Brian Granger how awesome it would be to have interactive widgets in the notebook. Fortunately that was pretty much preaching to the choir, as that's one of the first things they plan to implement good foundations for and I just can't wait to use that.

It is now clear, that the IPython notebook is the way to store computations that I want to share with other people, or to use it as a "lab notebook" for myself, so that I can remember what exactly I did to obtain the results (for example how exactly I obtained some figures from raw data). In other words --- instead of having sets of scripts and manual bash commands that have to be executed in particular order to do what I want, just use IPython notebook and put everything in there.

Number two is that how big the conference has become since the last time I attended (couple years ago), yet it still has the friendly feeling. Unfortunately, I had to miss a lot of talks, due to scheduling conflicts (there were three parallel sessions), so I look forward to seeing them on video.

+Aaron Meurer and I have done the SymPy tutorial (see the link for videos and other tutorial materials). It's been nice to finally meet +Matthew Rocklin (very active SymPy contributor) in person. He also had an interesting presentation
about symbolic matrices + Lapack code generation. +Jason Moore presented PyDy.
It's been a great pleasure for us to invite +David Li (still a high school student) to attend the conference and give a presentation about his work on sympygamma.com and live.sympy.org.

It was nice to meet the Julia guys, +Jeff Bezanson and +Stefan Karpinski. I contributed the Fortran benchmarks on the Julia's website some time ago, but I had the feeling that a lot of them are quite artificial and not very meaningful. I think Jeff and Stefan confirmed my feeling. Julia seems to have quite interesting type system and multiple dispatch, that SymPy should learn from.

I met the VTK guys +Matthew McCormick and +Pat Marion. One of the keynotes was given by +Will Schroeder from Kitware about publishing. I remember him stressing to manage dependencies well as well as to use BSD like license (as opposed to viral licenses like GPL or LGPL). That opensource has pretty much won (i.e. it is now clear that that is the way to go).

I had great discussions with +Francesc Alted, +Andy Terrel, +Brett Murphy, +Jonathan Rocher, +Eric Jones, +Travis Oliphant, +Mark Wiebe, +Ilan Schnell, +St fan van der Walt, +David Cournapeau, +Anthony Scopatz, +Paul Ivanov, +Michael Droettboom, +Wes McKinney, +Jake Vanderplas, +Kurt Smith, +Aron Ahmadia, +Kyle Mandli, +Benjamin Root and others.


It's also been nice to have a chat with +Jason Vertrees and other guys from Schr dinger.

One other thing that I realized last week at the conference is that pretty much everyone agreed on the fact that NumPy should act as the default way to represent memory (no matter if the array was created in Fortran or other code) and allow manipulations on it. Faster libraries like Blaze or ODIN should then hook themselves up into NumPy using multiple dispatch. Also SymPy would then hook itself up so that it can be used with array operations natively. Currently SymPy does work with NumPy (see our tests for some examples what works), but the solution is a bit fragile (it is not possible to override NumPy behavior, but because NumPy supports general objects, we simply give it SymPy objects and things mostly work).

Similar to this, I would like to create multiple dispatch in SymPy core itself, so that other (faster) libraries for symbolic manipulation can hook themselves up, so that their own (faster) multiplication, expansion or series expansion would get called instead of the SymPy default one implemented in pure Python.

Other blog posts from the conference:

7 February 2013

Russell Coker: Links February 2013

Aaron on Software wrote an interesting series of blog posts about psychology and personal development collectively Titled Raw Nerve , here s a link to part 2 [1]. The best sections IMHO are 2, 3, and 7. The Atlantic has an insightful article by Thomas E. Ricks about the failures in leadership in the US military that made the problems in Afghanistan and Iraq a lot worse than they needed to be [2] Kent Larson gave an interesting TED talk about how to fit more people in cities [3]. He covers issues of power use, transport, space use, and sharing. I particularly liked the apartments that transform and the design for autonomous vehicles that make eye contact with pedestrians. Andrew McAfee gave an interesting TED talk titled Are Droids Taking Our Jobs [4]. I don t think he adequately supported his conclusion that computers and robots are making things better for everyone (he also presented evidence that things are getting worse for many people), but it was an interesting talk anyway. I Psychopath is an interesting documentary about Sam Vaknin who is the world s most famous narcissist [5]. The entire documentary is available from Youtube and it s really worth watching. The movie Toy Story has been recreated in live action by a couple of teenagers [6]. That s a huge amount of work. Rory Stewart gave an interesting TED talk about how to rebuild democracy [7]. I think that his arguments against using the consequences to argue for democracy and freedom (he suggests not using the torture doesn t work and women s equality doubles the workforce arguments) are weak, but he made interesting points all through his talk. Ernesto Sirolli gave an interesting TED talk about aid work and development work which had a theme of Want to help someone? Shut up and listen! [8]. That made me think of Mary Gardiner s much quoted line from the comments section of her Wikimania talk which was also shut up and listen . Waterloo Labs has some really good engineering Youtube videos [9]. The real life Mario Kart game has just gone viral but there are lots of other good things like the iPhone controlled car and eye controlled Mario Brothers. Robin Chase of Zipcar gave an interesting TED talk about various car sharing systems (Zipcar among others), congestion taxes, the environmental damage that s caused by cars, mesh networks, and other things [10]. She has a vision of a future where most cars are shared and act as nodes in a giant mesh network. Madeleine Albright gave an interesting TED talk about being a female diplomat [11]. She s an amazing speaker. Ron Englash gave an interesting TED talk about the traditional African use of fractals [12]. Among the many interesting anecdotes concerning his research in Africa he was initiated as a priest after explaining Georg Cantor s set theories. Racialicious has an insightful article about the low expectations that members of marginalised groups have of members of the privileged groups [13]. Rick Falkvinge has a radical proposal for reforming copyrights with a declared value system [14]. I don t think that this will ever get legislative support, but if it did I think it would work well for books and songs. I think that some thought should be given to how this would work for Blogs and other sources of periodical content. Obviously filing for every blog post would be an unreasonable burden. Maybe aggregating a year of posts into one copyright assignment block would work. Scott Fraser gave an interesting TED talk about the problem with eyewitness testimony [15]. He gave a real-world example of what had to be done to get an innocent man acquitted, it s quite amazing. Sarah Kendzior wrote an interesting article for al Jazeera about the common practice in American universities to pay Adjunct Professors wages that are below the poverty line [16]. That s just crazy, when students pay record tuition fees there s more than enough money to pay academics decent wages, where does all the money go to anyway?

22 January 2013

Russ Allbery: Review: Fantasy & Science Fiction, March/April 2011

Review: Fantasy & Science Fiction, March/April 2011
Editor: Gordon van Gelder
Issue: Volume 120, No. 3 & 4
ISSN: 1095-8258
Pages: 258
Charles de Lint's book review column sticks with the sorts of things he normally reviews: urban and contemporary fantasy and young adult. Predictably, I didn't find that much of interest. But I was happy to see that not all the reviews were positive, and he talked some about how a few books didn't work. I do prefer seeing a mix of positive and negative (or at least critical) reviews. James Sallis's review column focuses entirely on Henry Kuttner and C.L. Moore (by way of reviewing a collection). I'm always happy to see this sort of review. But between that and de Lint's normal subject matter, this issue of F&SF was left without any current science fiction reviews, which was disappointing. Lucius Shepard's movie review column features stunning amounts of whining, even by Shepard's standards. The topic du jour is how indie films aren't indie enough, mixed with large amounts of cane-shaking and decrying of all popular art. I find it entertaining that the F&SF film review column regularly contains exactly the sort of analysis that one expects from literary gatekeepers who are reviewing science fiction and fantasy. Perhaps David Langford should consider adding an "As We See Others" feature to Ansible cataloging the things genre fiction fans say about popular movies. "Scatter My Ashes" by Albert E. Cowdrey: The protagonist of this story is an itinerant author who has been contracted to write a family history (for $100K, which I suspect is a bit of tongue-in-cheek wish fulfillment) and has promptly tumbled into bed with his employer. But he is somewhat serious about the writing as well, and is poking around in family archives and asking relatives about past details. There is a murder (maybe) in the family history, not to mention some supernatural connections. Those familiar with Cowdrey's writing will recognize the mix of historical drama, investigation, and the supernatural. Puzzles are, of course, untangled, not without a bit of physical danger. Experienced fantasy readers will probably guess at some of the explanation long before the protagonist does. Like most Cowdrey, it's reliably entertaining, but I found it a bit thin. (6) "A Pocketful of Faces" by Paul Di Filippo: Here's a bit of science fiction, and another mystery, this time following the basic model of a police procedural. The police in this case are enforcing laws around acceptable use of "faces" in a future world where one can clone someone's appearance from their DNA and then mount it on a programmable android. As you might expect from that setup, the possibilities are lurid, occasionally disgusting, and inclined to give the police nightmares. After some scene-setting, the story kicks in with the discovery of the face of a dead woman who, at least on the surface, no one should have any motive to clone. There were a few elements of the story that were a bit too disgusting for me, but the basic mystery plot was satisfying. I thought the ending was a let-down, however. Di Filippo tries to complicate the story and, I thought, went just a little too far, leaving motives and intent more confusing than illuminating. (6) "The Paper Menagerie" by Ken Liu: Back to fantasy, this time using a small bit of magic to illustrate the emotional conflicts and difficulties of allegiance for second-generation immigrants. Jack is the son of an American farther and a Chinese mother who was a mail-order bride. He's young at the start of the story and struggling with the embarassment and humiliation that he feels at his mother's history and the difficulties he has blending in with other kids, leading to the sort of breathtaking cruelty that comes so easily from teenagers who are too self-focused and haven't yet developed adult empathy. I found this very hard to read. The magic is beautiful, personal, and very badly damaged by the cruelty in ways that can never really be fixed. It's a sharp reminder of the importance of being open-hearted, but it's also a devastating reminder that the lesson is normally learned too late. Not the story to read if you're prone to worrying about how you might have hurt other people. (6) "The Evening and the Morning" by Sheila Finch: This long novella is about a third of the issue and is, for once, straight science fiction, a somewhat rare beast in F&SF these days. It's set in the far future, among humans who are members of the Guild of Xenolinguists and among aliens called the Venatixi, and it's about an expedition back to the long-abandoned planet of Earth. I had a lot of suspension of disbelief problems with the setup here. While Earth has mostly dropped out of memory, there's a startling lack of curiosity about its current condition among the humans. Finch plays some with transportation systems and leaves humanity largely dependent on other races to explain the failure to return to Earth, but I never quite bought it. It was necessary to set up the plot, which is an exploration story with touches of first contact set on an Earth that's become alien to the characters, but it seemed remarkably artificial to me. But, putting that aside, I did get pulled into the story. Its emotional focus is one of decline and senescence, a growing sense of futility, that's halted by exploration, mystery, and analysis. The question of what's happened on Earth is inherently interesting and engaging, and the slow movement of the story provides opportunities to build up to some eerie moments. The problem, continuing a theme for this issue, is the ending. Some of the reader's questions are answered, but most of the answers are old, well-worn paths in science fiction. The emotional arc of the story is decidedly unsatisfying, at least to me. I think I see what Finch was trying to do: there's an attempted undermining of the normal conclusion of this sort of investigation to make a broader point about how to stay engaged in the world. But it lacked triumph and catharsis for me, partly because the revelations that we get are too pedestrian for the build-up they received. It's still an interesting story, but I don't think it entirely worked. (6) "Night Gauntlet" by Walter C. DeBill, Jr., et al.: The full list of authors for this story (Walter C. DeBill, Jr., Richard Gavin, Robert M. Price, W.H. Pugmire, Jeffrey Thomas, and Don Webb) provides one with the first clue that it's gone off the rails. Collaborative storytelling, where each author tries to riff off the work of the previous author while spinning the story in a different direction, is something that I think works much better orally, particularly if you can watch facial expressions while the authors try to stump each other. In written form, it's a recipe for a poorly-told story. That's indeed what we get here. The setup is typical Cthulhu mythos stuff: a strange scientist obsessed with conspiracy theories goes insane, leaving behind an office with a map of linkages between apparently unrelated places. The characters in the story also start going insane for similar reasons, leading up to a typical confrontation with things man was not meant to know, or at least pay attention to. If you like that sort of thing, you may like this story better than I did, but I thought it was shallow and predictable. (3) "Happy Ending 2.0" by James Patrick Kelly: More fantasy, this time of the time travel variety. (I call it fantasy since there's no scientific explanation for the time travel and it plays a pure fantasy role in the story.) That's about as much as I can say without giving away the plot entirely (it's rather short). I can see what Kelly was going for, and I think he was largely successful, but I'm not sure how to react to it. The story felt like it reinforced some rather uncomfortable stereotypes about romantic relationships, and the so-called happy ending struck me as the sort of situation that was going to turn very nasty and very uncomfortable about five or ten pages past where Kelly ended the story. (5) "The Second Kalandar's Tale" by Francis Marion Soty: The main question I have about this story is one that I can't answer without doing more research than I feel like doing right now: how much of this is original to Soty and how much if it is straight from Burton's translation of One Thousand and One Nights. Burton is credited for the story, so I suspect quite a lot of this is from the original. Whether one would be better off just reading the original, or if Soty's retelling adds anything substantial, are good questions that I don't have the background to answer. Taken as a stand-alone story, it's not a bad one. It's a twisting magical adventure involving a djinn, a captive woman, some rather predictable fighting over the woman, and then a subsequent adventure involving physical transformation and a magical battle reminiscent of T.H. White. (Although I have quite likely reversed the order of inspiration if as much of this is straight from Burton as I suspect.) Gender roles, however, are kind of appalling, despite the presence of a stunningly powerful princess, due to the amount of self-sacrifice expected from every woman in the story. Personally, I don't think any of the men in the story are worth anywhere near the amount of loyalty and bravery that the women show. Still, it was reasonably entertaining throughout, in exactly the way that I would expect a One Thousand and One Nights tale to be. Whether there's any point in reading it instead of the original is a question I'll leave to others. (7) "Bodyguard" by Karl Bunker: This is probably the best science fiction of the issue. The first person protagonist is an explorer living with an alien race, partly in order to flee the post-singularity world of uploaded minds and apparent stagnation that Earth has become. It's a personal story that uses his analysis of alien mindsets (and his interaction with his assigned alien bodyguard) to flesh out his own desires, emotional background, and reactions to the world. There are some neat linguistic bits here that I quite enjoyed, although I wish they'd been developed at even more length. (The alien language is more realistic than it might sound; there are some human languages that construct sentences in a vaguely similar way.) It's a sad, elegiac story, but it grew on me. (7) "Botanical Exercises for Curious Girls" by Kali Wallace: One has to count this story as science fiction as well, although for me it had a fantasy tone because the scientific world seems to play by fantasy rules from the perspective of the protagonist. Unpacking that perspective is part of the enjoyment of the story. At the start, she seems to be a disabled girl who is being cared for by a strange succession of nurses who vary by the time of day, but as the story progresses, it becomes clear that something much stranger is going on. There are moments that capture a sense of wonder, reinforced by the persistantly curious and happy narrative voice, but both are undercut by a pervasive sense of danger and dread. This is a light story written about rather dark actions. My biggest complaint with the story is that it doesn't so much end as wander off into the sunset. It set up conflict within a claustrophobic frame, so I can understand the thematic intent of breaking free of that frame, but in the process I felt like the story walked away from all of the questions and structure that it created and ended in a place that felt less alive with potential than formless and oddly pointless. I think I wanted it to stay involved and engaged with the environment it had created. (6) "Ping" by Dixon Wragg: I probably should just skip this, since despite the table of contents billing and the full title introduction, it's not a story. It's a two-line joke. But it's such a bad two-line joke that I had to complain about it. I have no idea why F&SF bothered to reprint it. (1) "The Ifs of Time" by James Stoddard: This certainly fits with the Arabian Nights story in this issue. The timekeeper of a vast and rambling castle (think Gormenghast taken to the extreme) wanders into a story-telling session in a distant part of the castle. The reader gets to listen to four (fairly good) short stories about time, knowledge, and memory, told in four very different genres. All of this does relate to why the timekeeper is there, and the frame story is resolved by the end, but the embedded stories are the best part; each of them is interesting in a different way, and none of them outlast their welcome. This was probably the strongest story of this issue. (7) Rating: 6 out of 10

20 October 2012

Vincent Bernat: Network lab with KVM

To experiment with network stuff, I was using UML-based network labs. Many alternatives exist, like GNS3, Netkit, Marionnet or Cloonix. All of them are great viable solutions but I still prefer to stick to my minimal home-made solution with UML virtual machines. Here is why: The use of UML had some drawbacks: However, UML features HostFS, a filesystem providing access to any part of the host filesystem. This is the killer feature which allows me to not use any virtual disk image and to get access to my home directory right from the guest. I discovered recently that KVM provided 9P, a similar filesystem on top of VirtIO, the paravirtualized IO framework.

Setting up the lab The setup of the lab is done with a single self-contained shell file. The layout is similar to what I have done with UML. I will only highlight here the most interesting steps.

Booting KVM with a minimal kernel My initial goal was to experiment with Nicolas Dichtel s IPv6 ECMP patch. Therefore, I needed to configure a custom kernel. I have started from make defconfig, removed everything that was not necessary, added what I needed for my lab (mostly network stuff) and added the appropriate options for VirtIO drivers:
CONFIG_NET_9P_VIRTIO=y
CONFIG_VIRTIO_BLK=y
CONFIG_VIRTIO_NET=y
CONFIG_VIRTIO_CONSOLE=y
CONFIG_HW_RANDOM_VIRTIO=y
CONFIG_VIRTIO=y
CONFIG_VIRTIO_RING=y
CONFIG_VIRTIO_PCI=y
CONFIG_VIRTIO_BALLOON=y
CONFIG_VIRTIO_MMIO=y
No modules. Grab the complete configuration if you want to have a look. From here, you can start your kernel with the following command ($LINUX is the appropriate bzImage):
kvm \
  -m 256m \
  -display none \
  -nodefconfig -no-user-config -nodefaults \
  \
  -chardev stdio,id=charserial0,signal=off \
  -device isa-serial,chardev=charserial0,id=serial0 \
  \
  -chardev socket,id=con0,path=$TMP/vm-$name-console.pipe,server,nowait \
  -mon chardev=con0,mode=readline,default \
  \
  -kernel $LINUX \
  -append "init=/bin/sh console=ttyS0"
Of course, since there is no disk to boot from, the kernel will panic when trying to mount the root filesystem. KVM is configured to not display video output (-display none). A serial port is defined and uses stdio as a backend1. The kernel is configured to use this serial port as a console (console=ttyS0). A VirtIO console could have been used instead but it seems this is not possible to make it work early in the boot process. The KVM monitor is setup to listen on an Unix socket. It is possible to connect to it with socat UNIX:$TMP/vm-$name-console.pipe -.

Initial ramdisk UPDATED: I was initially unable to mount the host filesystem as the root filesystem for the guest directly by the kernel. In a comment, Josh Triplett told me to use /dev/root as the mount tag to solve this problem. I keep using an initrd in this post but the lab on Github has been updated to not use one. Here is how to build a small initial ramdisk:
# Setup initrd
setup_initrd()  
    info "Build initrd"
    DESTDIR=$TMP/initrd
    mkdir -p $DESTDIR
    # Setup busybox
    copy_exec $($WHICH busybox) /bin/busybox
    for applet in $($ DESTDIR /bin/busybox --list); do
        ln -s busybox $ DESTDIR /bin/$ applet 
    done
    # Setup init
    cp $PROGNAME $ DESTDIR /init
    cd "$ DESTDIR " && find .   \
       cpio --quiet -R 0:0 -o -H newc   \
       gzip > $TMP/initrd.gz
 
The copy_exec function is stolen from the initramfs-tools package in Debian. It will ensure that the appropriate libraries are also copied. Another solution would have been to use a static busybox. The setup script is copied as /init in the initial ramdisk. It will detect it has been invoked as such. If it was omitted, a shell would be spawned instead. Remove the cp call if you want to experiment manually. The flag -initrd allows KVM to use this initial ramdisk.

Root filesystem Let s mount our root filesystem using 9P. This is quite easy. First KVM needs to be configured to export the host filesystem to the guest:
kvm \
  $ PREVIOUS_ARGS  \
  -fsdev local,security_model=passthrough,id=fsdev-root,path=$ ROOT ,readonly \
  -device virtio-9p-pci,id=fs-root,fsdev=fsdev-root,mount_tag=rootshare
$ ROOT can either be / or any directory containing a complete filesystem. Mounting it from the guest is quite easy:
mkdir -p /target/ro
mount -t 9p rootshare /target/ro -o trans=virtio,version=9p2000.u
You should find a complete root filesystem inside /target/ro. I have used version=9p2000.u instead of version=9p2000.L because the later does not allow a program to mount() a host mount point2. Now, you have a read-only root filesystem (because you don t want to mess with your existing root filesystem and moreover, you did not run this lab as root, did you?). Let s use an union filesystem. Debian comes with AUFS while Ubuntu and OpenWRT have migrated to overlayfs. I was previously using AUFS but got errors on some specific cases. It is still not clear which one will end up in the kernel. So, let s try overlayfs. I didn t find any patchset ready to be applied on top of my kernel tree. I was working with David Miller s net-next tree. Here is how I have applied the overlayfs patch on top of it:
$ git remote add torvalds git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git
$ git fetch torvalds
$ git remote add overlayfs git://git.kernel.org/pub/scm/linux/kernel/git/mszeredi/vfs.git
$ git fetch overlayfs
$ git merge-base overlayfs.v15 v3.6
4cbe5a555fa58a79b6ecbb6c531b8bab0650778d
$ git checkout -b net-next+overlayfs
$ git cherry-pick 4cbe5a555fa58a79b6ecbb6c531b8bab0650778d..overlayfs.v15
Don t forget to enable CONFIG_OVERLAYFS_FS in .config. Here is how I configured the whole root filesystem:
info "Setup overlayfs"
mkdir /target
mkdir /target/ro
mkdir /target/rw
mkdir /target/overlay
# Version 9p2000.u allows to access /dev, /sys and mount new
# partitions over them. This is not the case for 9p2000.L.
mount -t 9p        rootshare /target/ro      -o trans=virtio,version=9p2000.u
mount -t tmpfs     tmpfs     /target/rw      -o rw
mount -t overlayfs overlayfs /target/overlay -o lowerdir=/target/ro,upperdir=/target/rw
mount -n -t proc  proc /target/overlay/proc
mount -n -t sysfs sys  /target/overlay/sys
info "Mount home directory on /root"
mount -t 9p homeshare /target/overlay/root -o trans=virtio,version=9p2000.L,access=0,rw
info "Mount lab directory on /lab"
mkdir /target/overlay/lab
mount -t 9p labshare /target/overlay/lab -o trans=virtio,version=9p2000.L,access=0,rw
info "Chroot"
export STATE=1
cp "$PROGNAME" /target/overlay
exec chroot /target/overlay "$PROGNAME"
You have to export your $ HOME and the lab directory from host:
kvm \
  $ PREVIOUS_ARGS  \
  -fsdev local,security_model=passthrough,id=fsdev-root,path=$ ROOT ,readonly \
  -device virtio-9p-pci,id=fs-root,fsdev=fsdev-root,mount_tag=rootshare \
  -fsdev local,security_model=none,id=fsdev-home,path=$ HOME  \
  -device virtio-9p-pci,id=fs-home,fsdev=fsdev-home,mount_tag=homeshare \
  -fsdev local,security_model=none,id=fsdev-lab,path=$(dirname "$PROGNAME") \
  -device virtio-9p-pci,id=fs-lab,fsdev=fsdev-lab,mount_tag=labshare

Network You know what is missing from our network lab? Network setup. For each LAN that I will need, I spawn a VDE switch:
# Setup a VDE switch
setup_switch()  
    info "Setup switch $1"
    screen -t "sw-$1" \
        start-stop-daemon --make-pidfile --pidfile "$TMP/switch-$1.pid" \
        --start --startas $($WHICH vde_switch) -- \
        --sock "$TMP/switch-$1.sock"
    screen -X select 0
 
To attach an interface to the newly created LAN, I use:
mac=$(echo $name-$net   sha1sum   \
            awk ' print "52:54:" substr($1,0,2) ":" substr($1, 2, 2) ":" substr($1, 4, 2) ":" substr($1, 6, 2) ')
kvm \
  $ PREVIOUS_ARGS  \
  -net nic,model=virtio,macaddr=$mac,vlan=$net \
  -net vde,sock=$TMP/switch-$net.sock,vlan=$net
The use of a VDE switch allows me to run the lab as a non-root user. It is possible to give Internet access to each VM, either by using -net user flag or using slirpvde on a special switch. I prefer the latest solution since it will allow the VM to speak to each others.

Debugging This lab was mostly done to debug both the kernel and Quagga. Each of them can be debugged remotely.

Kernel debugging While the kernel features KGDB, its own debugger, compatible with GDB, it is easier to use the remote GDB server built inside KVM.
kvm \
  $ PREVIOUS_ARGS  \
  -gdb unix:$TMP/vm-$name-gdb.pipe,server,nowait
To connect to the remote GDB server from the host, first locate the vmlinux file at the root of the source tree and run GDB on it. The kernel has to be compiled with CONFIG_DEBUG_INFO=y to get the appropriate debugging symbols. Then, use socat with the Unix socket to attach to the remote debugger:
$ gdb vmlinux
GNU gdb (GDB) 7.4.1-debian
Reading symbols from /home/bernat/src/linux/vmlinux...done.
(gdb) target remote   socat UNIX:$TMP/vm-$name-gdb.pipe -
Remote debugging using   socat UNIX:/tmp/tmp.W36qWnrCEj/vm-r1-gdb.pipe -
native_safe_halt () at /home/bernat/src/linux/arch/x86/include/asm/irqflags.h:50
50   
(gdb)
You can now set breakpoints and resume the execution of the kernel. It is easier to debug the kernel if optimizations are not enabled. However, it is not possible to disable them globally. You can however disable them for some files. For example, to debug net/ipv6/route.c, just add CFLAGS_route.o = -O0 to net/ipv6/Makefile, remove net/ipv6/route.o and type make.

Userland debugging To debug a program inside KVM, you can just use gdb as usual. Your $HOME directory is available and it should be therefore straightforward. However, if you want to perform some remote debugging, that s quite easy. Add a new serial port to KVM:
kvm \
  $ PREVIOUS_ARGS  \
  -chardev socket,id=charserial1,path=$TMP/vm-$name-serial.pipe,server,nowait \
  -device isa-serial,chardev=charserial1,id=serial1
Starts gdbserver in the guest:
$ libtool execute gdbserver /dev/ttyS1 zebra/zebra
Process /root/code/orange/quagga/build/zebra/.libs/lt-zebra created; pid = 800
Remote debugging using /dev/ttyS1
And from the host, you can attach to the remote process:
$ libtool execute gdb zebra/zebra
GNU gdb (GDB) 7.4.1-debian
Reading symbols from /home/bernat/code/orange/quagga/build/zebra/.libs/lt-zebra...done.
(gdb) target remote   socat UNIX:/tmp/tmp.W36qWnrCEj/vm-r1-serial.pipe
Remote debugging using   socat UNIX:/tmp/tmp.W36qWnrCEj/vm-r1-serial.pipe
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
0x00007ffff7dddaf0 in ?? () from /lib64/ld-linux-x86-64.so.2
(gdb)

Demo For a demo, have a look at the following video (it is also available as an Ogg Theora video).
<iframe frameborder="0" height="270" src="http://www.dailymotion.com/embed/video/xuglsg" width="480"></iframe>

  1. stdio is configured such that signals are not enabled. KVM won t stop when receiving SIGINT. This is important for the usage we want to have.
  2. Therefore, it is not possible to mound a fresh /proc on top of the existing one. I have searched a bit but didn t find why. Any comments on this is welcome.

12 September 2012

Martin F. Krafft: A black day for democracy

Today was a black day for democracy in Germany. The German constitutional court ruled in favour of the European Stability Mechanism. In combination with last week s announcement by the European Central Bank to purchase government bonds without limits (breaking the No-Bail-Out clause at the core of their mandate more obviously and irreversably than ever before), the German people have lost a good deal of democracy today. Why? you may ask because from now on, fiscal and financial policy will be made in Brussels, by people enjoying full immunity, but who are not elected democratically by the European people, let alone the Germans, and they will freely decide over who has to pay and be liable for whom. I am talking about people like Klaus Regling, who was already involved the very first time the Maastricht Criteria were violated. He is now at the front of the largest and most powerful financial weapon ever conceived. With immunity. And people like Mario Draghi, whom I would possibly call the most corrupt person I know. His announcement to save the Euro at whatever cost accidentally came only a day before his motherland Italy had to go to the market for more money and was able to place a bond at such ridiculously low interest rates that anyone who s kept up to speed with Italy s development had to rightfully ask how that was possible. While in the past, for whatever reason, the European people have let the ECB get by saying that they are not bailing out countries when they buy bonds on the secondary market (wtf!), they have finally dropped that restriction (the law). And as of today, the ESM is ready to go, along with the fiscal pact. Germany is now liable for more than quarter of all of the Eurozone s past and future debts. And no citizen will be able to have any more influence in this, or reverse it. Budget, fiscal policy and currency control are forever gone. Not that parliamentarian democracies were ever direct. Yet, in the past, one could at least vote for those people whose promises one was inclined to believe the most. You can still do that in the future, but those people won t be able to influence fiscal or financial policy anymore. There is no way back. The ESM and its employees enjoy full immunity, and the ESM is forever-binding. There is no exit clause. Thanks to the ECB s law breaking and the ESM, which I consider highly unconstitutional, at least in Germany, Eurozone-countries may refinance their debts at interest rates that are in no way related to their ability to pay back loans. All other countries foremost Germany are henceforth liable for others debts. The fundamental rule of the EU that no country would have to stand up for another country, is gone with the wind. Within an hour, the markets reacted. Germany, which previously had to pay negative interest (a sign of stability) saw interests on its bond shoot up. And Spain, Portugal, Greece and others who couldn t previously refinance their old debts, are now getting fresh money cheaper than ever. Spain s president Rajoy today didn t even bother beating around the bush anymore, he s now going to apply for fresh money but won t bother with any saving schemes or other restructurings. Monti in Italy has suggested the same. Wouldn t you take money if you were offered it for free, without the need to pay it back? This is more than inflation, in my opinion. What is currently happening in Europe is active depreciation of individual wealth. Our heads of state are actively working against the people. The Euro has lost all credibility and everyone knows it. It is only a question of time until it will tremble and fall. Meanwhile, the market celebrates and continues their gambles while they still can, on the backs of our currency and our wealth. Most affected are the people who have savings in Euros, whose life insurances are decreasing in worth and who cannot afford to diversify into other asset classes or currencies. On the other hand, those who let their money do the work are being saved. Whoever previously invested into bonds of struggling states, hoping to reap massive interest gains, is now proven right. Brussels has eliminated the risk factor. What kind of message does this send??? Hands up if you thought that our politicians are even interested in closing the rapidly widening gap between rich and poor. Really? That s naive. The Eurozone is corrupt, and our currency has never been as virtual as today. Nobody can say whether saving the Euro at all cost is the right thing and noone knows whether what s currently happening is just bad. I would have wished that our politicians had taken the crisis as an incentive to fix the system in the interest of the people and with a long-term focus: But on the contrary! Europe s policiticans are making it crystal clear that the foundation upon which it was built, the laws and rules, the promises and guarantees, no longer apply. The people were not asked. The promises once made were broken. Our politicians have ruled over our heads. More debts are being made, and more debts to pay off debts, and so on. It s long gotten out of control, now the process is institutionalised. I feel sorry for our kids. I find it irresponsible what is being done to them (in addition to the way we rape the environment). I also feel deeply with the people in the struggling countries who are being screwed by the crisis and are not at fault. What our politicians are doing is unfortunately not going to help long term. The problems are just postponed, and with every day, the inevitable crash will be more painful. I am sorry. Today is a black day for democracy. We have lost souvereignity. We have lost control over our currency. We have lost our budget rights. And I have lost my faith in the last instance of the German government that I trusted. As of today, I know that the German constitutional court is nothing more than a puppet in the hands of the politicians (who are themselves puppets of Brussels and the banks). The limit they imposed (Germany s liability must not increase beyond 190 billion Euros without the federal parliament s consent) is worthless. Soon the politicians will explain to us why it s inevitable that we must raise this limit. Not that the people could prevent it, but still I had hoped for a fundamental ruling. They should not have touched numbers. The EU had a no-bailout-clause from day one. It was conditional from the start. If one of the fundamental principles of a contract is broken, the contract becomes invalid. Not only did I expect the court to rule against socialised debt, I would have wished them to go a step further. The German national bank gave up control over the currency to the ECB only because the ECB incorporated the principles of the German national bank. Once the ECB overturned those principles, Germany should have reclaimed their souvereignity. But noone else in Europe would have wanted that. Merkel became a puppet herself. I am grateful that our daughter has dual citizenship. NP: Porcupine Tree: Live at Atlanta 2010

7 December 2011

Gustavo Noronha Silva: WebKitGTK+ hackfest \o/

It s been a couple days since I returned from this year s WebKitGTK+ hackfest in A Coru a, Spain. The weather was very nice, not too cold and not too rainy, we had great food, great drinks and I got to meet new people, and hang out with old friends, which is always great!

Hackfest black board, photo by Mario

I think this was a very productive hackfest, and as usual a very well organized one! Thanks to the GNOME Foundation for the travel sponsorship, to our friends at Igalia for doing an awesome job at making it happen, and to Collabora for sponsoring it and granting me the time to go there! We got a lot done, and although, as usual, our goals list had many items not crossed, we did cross a few very important ones. I took part in discussions about the new WebKit2 APIs, got to know the new design for GNOME s Web application, which looks great, discussed about Accelerated Compositing along with Joone, Alex, Nayan and Martin Robinson, hacked libsoup a bit to port the multipart/x-mixed-replace patch I wrote to the awesome gio-based infrastructure Dan Winship is building, and some random misc. The biggest chunk of time, though, ended up being devoted to a very uninteresting (to outsiders, at least), but very important task: making it possible to more easily reproduce our test results. TL;DR? We made our bots and development builds use jhbuild to automatically install dependencies; if you re using tarballs, don t worry, your usual autogen/configure/make/make install have not been touched. Now to the more verbose version! The need

Our three build slaves reporting a few failures

For a couple years now we have supported an increasingly complex and very demanding automated testing infrastructure. We have three buildbot slaves, one provided by Collabora (which I maintain), and two provided by Igalia (maintained by their WebKitGTK+ folks). Those bots build as many check ins as possible with 3 different configurations: 32 bits release, 64 bits release, and 64 bits debug. In addition to those, we have another bot called the EWS, or Early Warning System. There are two of those at this moment: one VM provided by Collabora and my desktop, provided by myself. These bots build every patch uploaded to the bugzilla, and report build failures or passes (you can see the green bubbles). They are very important to our development process because if the patch causes a build failure for our port people can often know that before landing, and try fixes by uploading them to bugzilla instead of doing additional commits. And people are usually very receptive to waiting for EWS output and acting on it, except when they take way too long. You can have an idea of what the life of an EWS bot looks like by looking at the recent status for the WebKitGTK+ bots. Maintaining all of those bots is at times a rather daunting task. The tests require a very specific set of packages, fonts, themes and icons to always report the same size for objects in a render. Upgrades, for instance, had to be synchronized, and usually involve generating new baselines for a large number of tests. You can see in these instructions, for instance, how strict the environment requirements are yes, we need specific versions of fonts, because they often cause layouts to change in size! At one point we had tests fail after a compiler upgrade, which made rounding act a bit different! So stability was a very important aspect of maintaining these bots. All of them have the same version of Debian, and most of the packages are pinned to the same version. On the other hand, and in direct contradition to the stability requirement, we often require bleeding edge versions of some libraries we rely on, such as libsoup. Since we started pushing WebKitGTK+ to be libsoup-only, its own progress has been pretty much driven by WebKitGTK+ s requirements, and Dan Winship has made it possible to make our soup backend much, much simpler and way more featureful. That meant, though, requiring very recent versions of soup. To top it off, for anyone not running Debian testing and tracking the exact same versions of packages as the bots it was virtually impossible to get the tests to pass, which made it very difficult for even ourselves to make sure all patches were still passing before committing something. Wow, what a mess. The explosion^Wsolution So a few weeks back Martin Robinson came up with a proposed solution, which, as he says, is the nuclear bomb solution. We would have a jhbuild environment which would build and install all of the dependencies necessary for reproducing the test expectations the bots have. So over the first three days of the hackfest Martin and myself hacked away in building scripts, buildmaster integration, a jhbuild configuration, a jhbuild modules file, setting up tarballs, and wiring it all in a way that makes it convenient for the contributors to get along with. You ll notice that our buildslaves now have a step just before compiling called updated gtk dependencies (gtk is the name we use for our port in the context of WebKit), which runs jhbuild to install any new dependencies or version bumps we added. You can also see that those instructions I mentioned above became a tad simpler. It took us way more time than we thought for the dust to settle, but it eventually began to. The great thing of doing it during the hackfest was that we could find and fix issues with weird configurations on the spot! Oh, you build with AR_FLAGS=cruT and something doesn t like it? OK, we fix it so that the jhbuild modules are not affected by that variable. Oh, turns out we missed a dependency, no problem, we add it to the modules file or install them on the bots, and then document the dependency. I set up a very clean chroot which we could use for trying out changes so as to not disrupt the tree too much for the other hackfest participants, and I think overall we did good. The aftermath By the time we were done our colleagues who ran other distributions such as Fedora were already being able to get a substantial improvements to the number of tests passing, and so did we! Also, the ability to seamlessly upgrade all the bots with a simple commit made it possible for us to very easily land a change that required a very recent (as in unreleased) version of soup which made our networking backend way simpler. All that red looks great, doesn t it? And we aren t done yet, we ll certainly be making more tweaks to this infrastructure to make it more transparent and more helpful to the users (contributors and other people interested in running the tests). If you ve been hit by the instability we caused, sorry about that, poke mrobinson or myself in the #webkitgtk+ IRC channel on FreeNode, and we ll help you out or fix any issues. If you haven t, we hope you enjoy all the goodness that a reproducible testing suite has to offer! That s it for now, folks, I ll have more to report on follow-up work started at the hackfest soon enough, hopefully =).

11 October 2011

Christian Perrier: RWC 2011 : after 1/4 finals

I was expecting to write more about the 7th Rugby World Cup, but real life, Debian work and running activities prevented me to do so.. Still, I watched several games and I can share my feelings now. First of all, this is a tremendous organization from New Zealand. It seems that about the entire country is working on making this even a great success and I really appreciate to see a place I certainly have to visit some day receiving such wordlwide attention (OK, admitedly, mostly in the part of the world that understands rugby). First round already gave several surprises even if the 1/4 finals were after all kinda expected (at least, the list of countries). Which lead us to the following 1/4 finals (in parenthesis are my bets): If you know about the results, you know I screwed it nearly completely..:-) The Welsh team played a great game against an uninspired Irish team. It has certainly been the best 1/4 final and they certainly deserve their win. Even though I usually tend to be supprotive of Ireland, I was very balanced here, and finally turned out to be in favor of the Welsh. We really have to fear them in semi-finals. Australia-South Africa was theoretically the most exciting 1/4 final but finally turned out to be quite boring. Both teams insisted on playing mostly to occupy their opponents part of the field, more than trying to score, then relying on penalties to score. That was apparently the good tactics for Australia ad they also deserve their win against a South-African team where forward players were not as decisive as they sometimes are. Argentina was again there and really there. It has been the only team up to now who lead score against New Zealand. And that was deserved. What a wonderful 1st halftime! Obviously, it was impossible for them to resist during second halftime and it slowly became obvious that the Blacks (saved during 1st half by the kicks of a very inspired Kiri Weepu) would finally manage to score tries. But, still, our argentinian friends, for instance the inoxydable "Super Mario" Ledesma, or the tireless Felipe Contepomi, were not here as a sacrificial victim. What to say about England-France? First half was astonishing for us, of course. This is what we love (and hate) with our beloved French team. Definitely the team that can make surprises and, imho, the only one that can beat New Zealand if that has to happen (but also the only one that can be entirely crushed by them). The defeat against Tonga and the week that followed completely transformed them. A stunning 3rd row, defending each and every single bit of England trying to invade "la patrie en danger". Rear lanes with the magicians of Toulouse (Clerc and M dard) as the ideal finishers of magic play by Parra, Trinh Duc, Mermoz, Palisson (the good surprise of this world cup, Alexis). And, during second half, a trilling resistance to assaults of the British White Knights, concluded by this delivering drop-goal by Trinh Duc. For sure, with games like this, they can beat everybody and by everybody, I mean everybody. Remember Millenium Stadium in 2007..:-) So, well. Australia-New Zealand and France-Wales. I know where my heart is balancing for both games. The Blacks and Les Bleus in final, thi is what we hope (and fear...), but both teams, particularly France, will have to first climb a quite big wall before reaching this.

Next.

Previous.