The first Debian-NYC Novice Night was a great success. We had about 25 people, about fifteen of which we had never seen before. We've been trying hard to reach out to new groups, and it seems to have worked even though we have a lot yet to do. One idea we'll probably use in the future is Novice Nights targeted to specific groups. A group of tech teachers in NYC has already contacted us and we will probably do a night in conjunction with them in the near future.People came for all sorts of reasons. We had a lot of installations, some people came for help configuration or upgrading, and some people came just to talk and learn more. Those who came to help learned a lot, too. Most everyone wants to come back.As with all Debian-NYC's activities, our plans and what we have learned are online so that anyone can adapt and reproduce what we do.We have a social event on December 10th, and will have another workshop, and another Novice Night, early next year.Tags: debian-nyc

A continuation of my intermittent series on Emdebian. This time covering how Emdebian changes component behaviourin Emdebian Grip.
Debian divides packages principally between those which comply with the DFSG (main), those which do not (non-free) and those which could but depend on those which do not (contrib). This leads to an enormous bias towards packages in main, which is good in terms of DFSG compliance but not so good for the size of the Packages files which every machine has to download to be able to install new stuff or upgrade existing stuff.
Emdebian is as concerned about the runtime space requirements of the system as much as the install time space requirements. Little point gaining 20Mb by removing stuff from packages if the next apt-get update adds 30Mb of cache data.
Therefore, Emdebian Grip sub-divides main (we don't have contrib or non-free packages at this time). This gives most users the smallest cache data size whilst allowing other users to add the extra components and get the full list of packages provided in Emdebian.
Alongside this division, the aim with emdebian-grip-server is to help others run secondary Emdebian Grip build machines which have a different range of packages. This allows one server to focus on XFCE and one to use other desktops or no desktops.
Components in Grip
Grip subdivides Debian main into main (~60%), dev (~30%), java, debug and doc components. The main emphasis, therefore, is that most Emdebian Grip machines will not be used to actually build other packages. I covered some of the theory on this step on the debian-embedded mailing list in 2009.
Component allocation
Packages are allocated to particular components by three main mechanisms:

1. Package name suffix - -dev packages into dev, -dbg packages into debug, -doc packages into doc.
2. Section names - devel goes into dev, java goes into java
3. Override files which are visible via the emdebian website.

Override files cover situations where a package is in a problematic Section (e.g. dbus is in Section: devel when it is almost impossible to have a netbook or graphical Debian installation without it). Once Squeeze is released, I might take up these overrides with ftpmaster to see if some rationale can be constructed to make things easier.
Sources
The use of components means that an Emdebian Grip device can choose a range of components in the apt sources line. Any permutation as long as main is preserved.

deb http://www.emdebian.org/grip squeeze main dev debug java doc

More on Filtering
I mentioned last time that Emdebian does not carry all packages available in Debian. As well as filtering the files contained within packages, Emdebian Grip filters the list of packages which are allowed into the repository for each server. This is why Emdebian Grip has under 3,000 packages to be released alongside Squeeze. (Emdebian Grip 1.0 based on Lenny had some 1,000 packages.)
Filtering the list of packages means that the maintainer of the server gets to choose their package set in a similar way to how tasksel can be used to select a functionally complete set of packages at installation time. (Indeed, Emdebian Grip provides some bespoke tasks which optimise the combination of tasksel and repository filtering.) Processing packages for Emdebian Grip takes a finite amount of time per package, per architecture and restricting the total number of packages in Grip has several advantages:

1. Further reductions in cache size - less files in the Packages file, less Packages data to download and cache.
2. Less processing time which makes it easier to maintain an Emdebian Grip server
3. Less data to mirror which makes it attractive for admins already running a full Debian mirror to consider squeezing in Emdebian Grip alongside.

Combining component changes and filters needs a little bit of careful handling in the server side code so that packages remain installable. Override files need to be adjusted manually, library transitions mean that once libfoo0 has been replaced by libfoo1, anything depending on libfoo1 is broken in Emdebian Grip until libfoo1 can be added to the repository (which is also a manual task). I'm glad that edos-debcheck is regaining a machine-parseable output format because this will add the chance to make at least some of these transition additions automatic. Other than these tasks, emdebian-grip-server is almost entirely automated and runs as a cron task creating verbose log files each run.
The server task runs edos-debcheck for each supported architecture and for each suite, carefully combining the various Packages files from the available components. As an alternative assessment, Debian Weather runs edos-debcheck over the Packages file for the main component only, for each suite and each architecture in Emdebian Grip. The combination of the two provides invaluable data to spot missing packages.
Equally, there are times when packages disappear from Debian. Currently, the automated scripts don't handle this aggressively but manual support scripts are available, usually for use before a release when the suites are changing less frequently.
I'll cover migrations between suites next time. In the meantime, if you want to prepare your own package set for Emdebian Grip, install emdebian-grip-server from experimental and file bugs to improve the documentation.
Thanks to those who took on the Herculean task of translating the emdebian-grip-server manpages into German and French - more translations are always welcome. POT file contained inside the source package.

The screenshots.debian.net site is a public repository of screenshots taken from applications contained in the Debian GNU/Linux distribution and its derivates like Ubuntu. It was created by Cristoph Haas to help people get an impression of what a certain software will look like on your desktop before you install it. Everybody can take screenshots and upload them through the upload form. So far there are about 3100 screenshots, which is actually a small number if you consider Debian Squeeze contains about 15000 source packages After I hacked a bit on KSnapshot, the KDE snapshot tool, Sune asked me to add an export to screnshots.debian.net command. I had not thought of it before but that looked like a great idea, it would make the process or taking-screenshot, visit-upload-form, upload-each-screen so much easier!. So I started to develop my first KIPI plugin. Here is the resulting Debian Screenshots plugin, as you would use it in KSnapshot:

Being a KIPI plugin, it is available not only from KSnapshot but also from Digikam, Gwenview and the other applications which support KIPI plugins:

Which makes me think: what about having a debshots (the software powering screenshots.debian.net) installation over at kde.org and have screenshots for all the KDE applications? What do you think? Promo team? Sysadmins? Setting it up shouldn t be difficult: I m no Python or Pylons expert (I m more of a Ruby and Wt C++ guy and got it running in a VM in an afternoon, even solving some SQLAlchemy 0.6 issues.

This is the second post about distcheck. I want to give a quick overview of the differences between edos-distcheck and the new version. First despite using the same sat solver and encoding of the problem, Distcheck has been re-written from scratch. Dose2 has several architectural problems and not very well documented. Adding new features had become too difficult and error-prone, so this was a natural choice (at least for me). Hopefully Dose3 will survive the Mancoosi project and provide a base for dependency reasoning. The framework is well documented and the architecture pretty modular. It's is written in ocaml, so sadly, I don't expect many people to join the development team, but we'll be very open to it. These are the main differences with edos-debcheck .

Performances distcheck is about two times faster than edos-debcheck (from dose2), but it is a "bit" slower then debcheck (say the original debcheck), that is the tool wrote by Jerome Vouillon and that was then superseded in debian by edos-debcheck. The original debcheck was a all-in-one tool that did the parsing, encoding and solving without converting the problem to any intermediate format. distcheck trades a bit of speed for generality. Since it is based on Cudf, it can handle different formats and can be easily adapted in a range of situation just by changing the encoding of the original problem to cudf. Below there are a couple of test I've performed on my machine (debian unstable). The numbers speak alone.

$time cat tmp/squeeze.packages edos-debcheck -failures > /dev/null
Completing conflicts... * 100.0%
Conflicts and dependencies... * 100.0%
Solving * 100.0%

real 0m19.515s
user 0m19.193s
sys 0m0.276s

$time ./distcheck.native -f deb://tmp/squeeze.packages > /dev/null

real 0m10.859s
user 0m10.669s
sys 0m0.172s

Input The second big difference is about different input format. In fact, at the moment, we have two different tools in debian, one edos-debcheck and the other edos-rpmcheck. Despite using the same underlying library these two tools have different code bases. distcheck basically is a multiplexer that convert different inputs to a common format and then uses it (agnostically) to solve the installation problem. It can be called in different ways (via symlinks) to behave similarly to its predecessors. At the moment we are able to handle 5 different formats

1. deb:// Packages 822 format for debian based distributions
2. hdlist:// a binary format used by rpm based distribution
3. synth:// a simplified format to describe rpm based package repositories
4. eclipse:// a 822 based format that encoded OSGi plugings metadata
5. cudf:// the native cudf format

distcheck handles gz and bz2 compressed file transparently . However if you care about performances, you should decompress your input file first and the parse it with distcheck and it often takes more time to decompress the file on the fly that run the installability test itself. There is also an experimental database backend that is not compiled by default at them moment.

Output Regarding the output, I've already explained the main differences in an old post. As a quick reminder, the old edos-debcheck had two output options. The first is a human readable - unstructured output - that was a handy source of information when running the tool interactively. The second was a xml based format (without a dtd or a schema, I believe) that was used for batch processing. distcheck has only one output type in the YAML format that aims to be human and machine readable. Hopefully this will cater for both needs. Moreover, just recently I've added the output of distcheck a summary of who is breaking what. The output of edos-debcheck was basically a map of packages to the reasons of the breakage. In addition to this information distcheck gives also a maps between reason (a missing dependency or a conflict) to the list of packages that are broken by this problem.This additional info is off by default, but I think it can be nice to know what is the missing dependency that is responsible for the majority of problems in a distribution... For example, calling distcheck with --summary :

$./distcheck.native --summary deb://tests/sid.packages
backgroud-packages: 29589
foreground-packages: 29589
broken-packages: 143
missing-packages: 138
conflict-packages: 5
unique-missing-packages: 52
unique-conflict-packages: 5
summary:
-
missing:
missingdep: libevas-svn-05-engines-x (>= 0.9.9.063)
packages:
-
package: enna-dbg
version: 0.4.0-4
architecture: amd64
source: enna (= 0.4.0-4)
-
package: enna
version: 0.4.0-4
architecture: amd64
source: enna (= 0.4.0-4)
-
missing:
missingdep: libopenscenegraph56 (>= 2.8.1)
packages:
-
package: libosgal1
version: 0.6.1-2+b3
architecture: amd64
source: osgal (= 0.6.1-2)
-
package: libosgal-dev
version: 0.6.1-2+b3
architecture: amd64
source: osgal (= 0.6.1-2)

Below I give a small example of the edos-debcheck output compared to the new yaml based output.

$cat tests/sid.packages edos-debcheck -failures -explain
Completing conflicts... * 100.0%
Conflicts and dependencies... * 100.0%
Solving * 100.0%
zope-zms (= 1:2.11.1-03-1): FAILED
zope-zms (= 1:2.11.1-03-1) depends on missing:
- zope2.10
- zope2.9
zope-tinytableplus (= 0.9-19): FAILED
zope-tinytableplus (= 0.9-19) depends on missing:
- zope2.11
- zope2.10
- zope2.9
...

And an extract from the distcheck output (the order is different. I cut and pasted parts of the output here...)

$./distcheck.native -f -e deb://tests/sid.packages
report:
-
package: zope-zms
version: 1:2.11.1-03-1
architecture: all
source: zope-zms (= 1:2.11.1-03-1)
status: broken
reasons:
-
missing:
pkg:
package: zope-zms
version: 1:2.11.1-03-1
architecture: all
missingdep: zope2.9 zope2.10
-
package: zope-tinytableplus
version: 0.9-19
architecture: all
source: zope-tinytableplus (= 0.9-19)
status: broken
reasons:
-
missing:
pkg:
package: zope-tinytableplus
version: 0.9-19
architecture: all
missingdep: zope2.9 zope2.10 zope2.11
...

Future The roadmap to release version 1.0 of distcheck is as follows:

1. add background and foreground package selection. This feature will allow the use to specify a larger universe (background packages), but check only a subset of this universe (foreground packages). This should allow users to select packages using grep-dctrl and then pipe them to discheck . At the moment we can select individual packages on the command line or we can use expression like bash (<= 2.7) to check all version of bash in the universe with version greater than 2.7.
2. code cleanup and a bit of refactoring between distcheck and buildcheck (that is a frontend for distcheck that allow us to report broken build dependencies)
3. consider essential packages while performing the installation test. Here there are few things we have to understand, but the idea would be to detect possible problems related the implicit presence of essential packages in the distribution. At the moment, distcheck performs the installation test in the empty universe, while ideally, the universe should contain all essential packages.
4. finish the documentation. The effort in underway and we hope to finalize shortly to release the debian package in experimental.

LWN has an article by Valerie Aurora titled The dark side of open source conferences [1] which is about sexual harassment and sexual assault at Free Software conferences. Apparently some conferences create such a bad environment that some people won t attend, it s a well researched article that everyone in the community should read.The Autism DerailmentThe comments have the usual mix of insight, foolishness, and derailment that you expect from such discussions. One derailment thread that annoyed me is the discussion about men on the Autism Spectrum started by Joe Buck [2]. Joe seems to believe that the 1% of males on the Autism Spectrum (and something greater than 1% but a lot less than 50% in the Free Software community) are a serious part of the problem because they supposedly hit on women who aren t interested in them in spite of the fact that the article in question is about women who are being insulted, harassed, and groped at at open source conferences . The article had no mention of men who try to chat up women presumably this was a deliberate decision to focus on sexual assault and harassment rather than what Joe wanted to talk about.In response Mackenzie made the following insightful point:I don t think any autistic person who is high-functioning enough to A) contribute to open source B) want to be at an event with so many people and C) carry on any sort of conversation is low-functioning enough not to understand stop or no. If you can understand your patch has been rejected, you can likely understand don t do that again. Understanding how Other People FeelBruce Perens claimed What they [Aspies] don t understand is how the other person in the situation feels . Like many (possibly most) people Bruce doesn t seem to get the fact that no-one can really understand how other people feel. The best logical analysis of this seems to be the Changing Emotions article on Less Wrong [3]. While Less Wrong deals with Male to Female conversion as the example (which may be relevant to the discussion about the treatment of women) the same logic also applies to smaller changes. Anyone who even thinks that if they would always be able understand how their identical twin felt (if they had one) probably hasn t considered these issues much. As an aside, having a psychologist diagnose you as being on the Autism Spectrum and therefore by implication thinking differently to 99% of the population really makes you consider the ways in which other people might have different thought processes and experiences.Every time we have a discussion about issues related to sexism in the Free Software community we get a lot of documented evidence that there are many people who are apparently neuro-typical (IE not Autistic) who don t understand how other people think in many cases they go so far as to tell other people what their emotional state should be.What Really HappensNix said However, in that situation our natural reflex is to *get out of there*, not to jump on women like some sort of slobbering caveman which is a really good summary.In more detail, I think that the vast majority of guys who are on the Autism Spectrum and who are able to do things like attend computer conferences (*) realise that chatting up a random girl that they meet is something that just isn t going to work out. Generally people don t attempt things that they expect to fail so I don t think that Autistic guys are going to be hitting on girls at conferences.(*) Having never met any Autistic people who aren t capable of attending such conferences I can t speak for them. I really doubt that the Low Functioning Autistic guys are as much of a problem as some people claim, but lack evidence. In any case the actions of people who don t attend conferences aren t relevant to a discussion about things that happen at conferences.Update: It Keeps GoingDion claims that the misogyny at conferences is due to socially inept people, he also casually switches between discussing people who misunderstand when someone is flirting and people who hire almost-naked booth-babes (two very different classes of action) [4]. Several people asked for supporting evidence, naturally none was provided.In response njs posted a link to Marissa Lingen s blog post Don t blame autism, dammit [5]. Marissa points out that people who offend other people due to lacking social skills will tend to do so in times and places that are likely to get a bad reaction if you don t know that you are doing something wrong then there s no reason to hide it. If someone offends a senior manager at a corporate event then it could be because they are on the Autism Spectrum (I ve apparently done that). If someone offends junior people at a times and places where there are no witnesses but is always nice to managers and other powerful people then it s not related to Autism.One final note, I have little tolerance for anyone who claims to be an Aspie when they do something wrong. You are either on the Autism Spectrum all the time or none of it. Anyone who wants any sympathy for me for an occasion where they stuffed up due to being an Aspie can start by making a clear statement about where they are on the Autism Spectrum.Update2: Yet More from Bruce PerensBruce wrote IMO, the kind of men who go in to software engineering suffer a lack of healthy interaction with women who are their peers, and it may be that the high incidence of empathy disorders in our field is involved (which seems to be part of the inspiration for Joe Buck later in that thread) and now claims Nobody here was trying to connect Asperger s or autism with the touching incidents or violent crime .Matthew Garrett responded to that with If you weren t trying to say that the high incidence of empathy disorders in our field was related to a lack of healthy interaction with women who are their peers, and that that has something to do with incidents of sexual harassment or assault at conferences, what were you trying to say? Because that sounds awfully like We wouldn t have so many problems if it weren t for all the autists .Bruce s latest comment is If you choose to read something that nasty into my writing, that s your problem. Get therapy .Through this discussion I ve been unsure of whether to interpret the statements by Bruce et al the way Matthew does or whether I should consider them as merely a desperate attempt to derail the discussion. I can t imagine any possible way of interpreting such comments in connection with the discussion of sexual assault as anything other than either trivialising violent crimes against women (suggesting that they are no worse than asking out someone who s not interested) or claiming that anyone who lacks social skills should be treated as a violent sexual predator. It s just not reasonable to believe that every single person who wrote such comments referring to Autism was misunderstood and really meant something nice.As a general rule I don t think that it s the responsibility of other people to try and find a non-offensive interpretation of something that one might say. I don t think that all the people who strongly disagree with the most obvious and reasonable interpretations of Bruce s comments should get therapy. I think that Bruce should explain what he means clearly.

• [1] http://lwn.net/Articles/417952/
• [2] http://lwn.net/Articles/418209/
• [3] http://lesswrong.com/lw/xe/changing_emotions/
• [4] http://lwn.net/Articles/418281/
• [5] http://mrissa.livejournal.com/746160.html

I have this project in my head, I want to work on it but I always lack the time. In order to go forward, I thought I could write about it, at least it would let me clarify my ideas and the core goals. So here I am, I will present you Librement (I have registered the alioth project but it s empty). The core goal is to make it easy for every user to contribute to free software in some way. I will now present the main features that I envision. Defining skills and interests In order to propose tasks that the user can do, we must have an idea of his skills. So on the first run (and later through a preferences menu) the user will be invited to define his skills:

• his native languages (multiple allowed)
• other languages he can understand
• programming languages he knows
• version control systems he can use
• markup language he knows (HTML, DocBook, Wiki-like formats, etc.)
• etc.

Maybe we can also ask which skills he would like to learn. Because contributing to free software is a nice opportunity to learn new skills! We should also find out what the user is interested in. What are his favorite free software projects? What kind of contributions would he like to do (documentation, translation, coding, bug fixing, bug triaging, creating artwork, donations, etc.)? Choose activities and pick concrete tasks Based on the user s skills and his interests, the software shows a list of possible activities. The user can then sort that list, from the most interesting one to those that he doesn t want to do. Each activity can generate concrete tasks. For example, the activity Do translation for Debian could generate a task Translate strings in debconf/fr.po or Review translations in partman/fr.po . Work on tasks When the user decides to work on a task, a step-by-step assistant helps him/her. It can automate some steps and provide explanations for the remaining ones, for example in the case of a translation for Debian:

• grab the PO file (from a VCS, from an HTTP URL, from a translation server, etc.);
• select and install a software to work with PO file (if not already done);
• edit the PO file with the preferred program;
• check the PO file (is it complete? is there no mistakes like missing substitutions?);
• send back the completed PO file in a mail to the Debian bugtracking system.

If the tasks is not completed in one go, the user can resume it the next time. Each free software project must provide some meta-information describing the various workflows involved for contributing to the different parts of the project. If necessary the project can also provide new plugins to support new operations that are not available in the default library. Setting goals In order to keep the user motivated, the software could track how much time he spent contributing to free software and it could verify if the user reached the goals he picked up for himself. Maybe it can also hook into the OMG Trophy Awarding System. The sky is the limit I hope that you now have a clearer idea of what this desktop application is supposed to be. There are literally hundreds of ways to contribute to free software and I like the idea that we can streamline the process for most users. All the plugins implementing activities can use local information (list of packages installed with their versions, configuration settings, etc.) to propose tasks targetted to the user and highly beneficial for the corresponding free software projects. For example, a bug tagged unreproducible might benefit from a few more users trying to reproduce it. The software could direct the user to this bug report if it detects that he/she runs the same version on the same architecture and that this software is regularly run on the system. Many projects have created operations or events to encourage people to contribute, they could all be implemented as dedicated activities in Librement. I m thinking of stuff like Gnome Love, Ubuntu s 5-a-day, Ubuntu s 100 papercuts, etc. Even for people who have no time to contribute, the application can still be useful by referencing the various ways to donate money (or material) to projects that they are using. Feedback welcome I m excited by the potential of such an application, but it s normal since it s my idea. Do you believe it can be useful and popular? Do you have ideas of exciting activities that such a framework can offer? PS: If you wonder how I came up with the name Librement , here s the explanation. It s a French word which means freely . And users who want to give back are trying to live up to the principles of free software, which I sum up by they are trying to live freely .

Follow me on Identi.ca, Twitter and Facebook. Or subscribe to this blog by RSS or by email.

11 comments Liked this article? Click here. My blog is Flattr-enabled.

I ve written previously about my reading habits, online and offline, and the patterns I extrapolate to content consumption in general. I ve been talking with other people about this as well, and am beginning to develop a model to apply to my daily life. There are plenty of unsatisfactory metaphors circulating in this area: we eat the information diet , drink from the fire hose , endure the information explosion , and so on. None of them describe the richness of my experience: the profound variations in style, texture, speed, depth and movement are lost in this kind of dry imagery. Instead, I think of it like respiration. We inhale information, and we also exhale it transformed. We do this, consciously or unconsciously, every moment of our lives. Sometimes we do it quickly, other times slowly, and it can be relaxing or stimulating. We only retain a small amount of what we take in, but it becomes a part of us. We can immerse ourselves deeply, meditatively, in a series of breaths, or fail to notice as we breathe shallowly or pause altogether. One breath may be virtually silent, the next filled with a song or a question or a piercing whistle. We maintain a balance in our breathing, and I aspire to do the same with information: reading and writing, listening and speaking, seeing and being seen. I don t mean this simplistically, that I should do both in equal proportion (imagine trying to write as many words as you read!), but doing both consistently. It is often only when I share an idea that I come to understand it deeply, no matter how much I have read about it. By writing it down, or telling someone about it, I naturally fill in the gaps in my understanding and create mental structures which help me recall and apply what I ve learned. My input and output should be balanced appropriately for my circumstances. Rapid-fire email is like hyperventilation: I can do it for a while, reading and replying in quick succession, and even feel energized by it, but if I go on for too long, I get dizzy. Running might call for a certain breathing ratio, and Yoga quite a different one. Both can be healthy practices, but they are different, and each requires focus and consistency. Another key lesson from breathing is to let go. A friend recently told me about his daily online news routine, which included closing any unread tabs at the end of the session. I will sometimes hang onto an article or a video for days or weeks before I find the opportunity to take it in. I eventually get around to most of them, but meanwhile they are taking up space and causing a continuous low level of anxiety or guilt. This information isn t going anywhere, and if it s truly significant for me, it will most likely turn up again. If it doesn t, that s OK too. Similarly, it s not wise to breathe stale, indoor air for too long. I will try to step outside regularly, and engage with people and media from outside my usual sphere. When the weather is tolerable, I ll open up the house and let plenty of fresh air circulate. This will help me avoid getting stuck in the echo chamber of my own ideas, or in groupthink. Perhaps most significantly, I will try to remember that information exchange, like breathing, is not an end in itself. It is a means to action. This will remind me to get out of my head regularly, and do something significant with what I ve learned. Ready? Breathe.

As I previously blogged about, I wrote an Annual Review of Anthropology on digital media last year. About a month ago, I found out that anyone can download it thanks to a link provided by the ARA, which we are allowed to put on one institutional web page. So go here (and go to the citation for the link) for those who are interested in a way too short review of some of the ethnographic literature on digital media. Writing the piece left me many psychological wounds and scars, one of which had to do with the fact that I probably overlooked some folks. I have been left out of review type essays and honestly, it sucks. I tried to be as comprehensive as possible: I chose not to massively whittle down the scope (which was an option) and was able to smuggle in more citations than originally allowed and yet I still cut out 200 citations. But in the end I overlooked some folks as I found out about them too late. If I could go back in time, this is who I would include (well there are others but I have chosen these for now). So Shaka McGlotten: not only does he have a cool name, he studies some cool stuff like DIY online porn, race, and zombies. He has published a bunch of articles and a book is forthcoming. Check out his work here. I missed this book Online a Lot of the Time: Ritual, Fetish, and Sign by Ken Hills which looks quite pertinent and a great read. Jonathan Marshall is an anthropologist in Sydney who has been working on digital stuff for a long while now and recently published Living Cybermind, which covers in detail modes of interacting and communicating in a detailed examination of a mailing list by the name of Cybermind. Although this book is not out yet, it will be soon and looks fascinating: Digital Jesus (great cover). From what I understand, and one of the reasons I want to dive into the book , is because it is so longitudinal, at least when measured in Internet years. Rob Howard has studied Christians online since the days of Usenet to the present and thus this ethnography promises to have some real meat to it. I am eagerly anticipating the forthcoming book by Beth Coleman . I had a chance to read a chapter recently and it looks fantastic. While in some respects grounded in virtual worlds, it is far more expansive than that topic, addressing a range of issues from desire, experience, emotion and race. Can t wait. I cite four dissertations in the ARA mostly because I read them, thought they were great, and material based on the dissertation is en route to being published in some form in the next future. The one dissertation I wish I had read is Jenny Cool s thesis on cyberorganic.

I've just released new version of Gammu, which comes with some things, which are worth mentioning. First thing is m-obex support (which I've already announced). Since last announcement, there were some improvements, but I'm affraid it still does not work, because it was not completely tested. So the very same request applies - testers welcome and we need feedback. The second big change is universal SQL backend for Gammu SMSD, which consolidates code of former MySQL, PostgreSQL and DBI backends. It also adds option to configure SQL queries it will use. This was tested using usual testsuite, so it is working, but there still might be some glitches, which will need polishing before we can declare it stable.

Filed under: Gammu 0 comments Flattr this!

i suspect a lot of people are used to forwarding TCP sockets with SSH -- for example, to connect locally to a mysql daemon that runs only on the loopback interface of a remote machine (this is debian's default mysql-server configuration):

ssh -N -T -oExitOnForwardFailure=yes -L 3306:localhost:3306 remoteuser@mysqlserver.example

But sometimes, the remote service runs on a UNIX-domain socket, not on a TCP socket -- for example, debian's default configuration for postgresql is to have it listen only on a UNIX domain socket in /var/run/postgresql, and use SO_PEERCRED with a simple system account == psql account mapping scheme to authenticate users without needing any extra credentials. This is not quite as simple to forward over ssh, but it's doable as long as socat is installed on both your local host and on the remote postgres server. Here's one way to do it if $SOCKET_DIR points to the full path of a directory under the user's control (this is all one command, split across lines for easier reading):

socat "UNIX-LISTEN:$SOCKET_DIR/.s.PGSQL.5432,reuseaddr,fork" \
   EXEC:'ssh remoteuser@psqlserver.example socat STDIO UNIX-CONNECT\:/var/run/postgresql/.s.PGSQL.5432'

Then, you'd connect with something like:

psql "user=remoteuser host=$SOCKET_DIR"

Each such psql connection will trigger an ssh connection to be made. Of course, this won't work well if ssh has to prompt for passwords, but you should be using ssh-agent anyway, right? There are at least a couple nice features of being able to use postgresql from a local client like this:

• your psql client can load files from your local machine, and can dump/export files to the local machine.
• your ~/.psql_history stays local, so you can review what you did even when you're offline
• you can run local RDBMS administrative GUIs like pgadmin3 with minimal network traffic and no extra packages installed on the server.
• unlike forwarding TCP ports (where any other user account on the machine can hop onto your connection), you can control access to your local UNIX-domain socket with standard filesystem permissions on $SOCKET_DIR.

Of course, postgresql itself already comes with a nice range of high-quality network-capable authentication mechanisms you could use. But many of them (like GSSAPI or X.509 mutual key-based authentication over TLS) require additional infrastructure setup; and you probably already have sshd up and running on that machine -- so why not make use of it?Tags: postgresql, socat, ssh, tip, unix-domain socket

Ok, so micro rant time: this is the effect of not taking things upstream: hardware doesn t work Out Of The Box. Very briefly, I purchased a Vodafone prepaid mobile broadband package today, which comes with a modem and SIM. The modem is a K3571-Z, and Ubuntu *thinks* it knows how they work (it doesn t). So it fails to connect in NetworkManager with a rather opaque NO CARRIER message. Thanks to excellent assistance from Matt Trudel, we tracked this down to a theory that perhaps modemmanager is using the wrong serial port and voila, it is. From there, the config file (/lib/udev/rules.d/77-mm-zte-port-types.rules) was an obvious next step and indeed there is no entry in there for the 19d2:1010 the K3571-Z. Google found one immediately though, on a Vodafone research site. The awful shame is this: that was committed to the bcm project in March this year. If Vodafone had shipped off a patch to modemmanager, we could have had that in 10.10, and possibly even in 10.04. There are plenty of users having trouble on Whirlpool etc with this model who would have had a better experience helping Vodafone s users be happier. All it would have taken is an email I m sure Vodafone want a great experience for their users, but I think they re failing to separate out platform improvements share and share alike, and branding / custom facilities. The net impact is harmful, not helpful. Anyhow, Natty will support this modem.

This is a mental note really, since my certificates last two years and I ve always forgotten what to do about it.

1. Generate a new request:
   openssl req -new -key <keyfile> -out <csrfile>
   StartSSL throw away all properties of the request except the key, so any answers will do.
2. Re-use the request you sent last time (thanks Noel).
3. Get the certificate signed.
4. Dovecot expects a key in /etc/ssl/private/dovecot.pem and a certificate chain in /etc/ssl/certs/dovecot.pem. Build the chain, CERTIFICATE FIRST:
   cat <crt> sub_class2.pem ca.pem > /etc/ssl/certs/dovecot.pem
5. Reload Dovecot and test from somewhere remote:
   openssl s_client -connect <server>:imaps
6. Coffee time.

Lighttpd is basically the same, but additionally expects the key to be in the top of the certificate chain.

(This is a part of a series of posts on my recent experience with Windows. See Windows (I) for the first post.)

For the past month, I got disconnected from basically all of my usual activities. Not only from Debian work (well, yes, I followed up on a couple of important bugs and releases, but I was in a very low power consumption mode of sorts), DebConf organization/following (some people have contacted me regarding the DebConf 12 venue decision, for which we should start working and have a presentation soon) Even the always-so-important Real Life got pushed aside. I barely participated in this year's EDUSOL (On-line Encounter of Education and Free Software). Even work suffered I asked my boss to hold any incoming requests for me (save, of course, for urgent stuff). I am now shyly coming up from the hole I dug myself in for this time. Why did I do such a thing? I do not have a formal education beyond high school. Due to several personal issues, I was accepted to the best university in my country, and to the first generation of the study program I would definitively love to have followed, but didn't attend. Of course, I had a fair bit of computing background, enough to be competitive and start working. After amazingly few years, I didn't only get a decent job, but I got a job precisely at the very university I wanted to study at (and where my father has worked for the past 40 years). And, even with some changes along the years, here I am with 10 years of laboral history in this university, with an academic position (although the smallest academic position, but still). It is one of my life's achievements, one of the things that makes me proudest of. Still, I don't want to stay at this level for the rest of my life. And an institution such as this one has rules Yes, flexible enough to allow me in, but rules whose importance I cannot deny. I aspire to become a professor/researcher at some point, and that is just not possible without filling in the prerequisites. Besides, I want to try my hand at teaching at a universitary level. So, what choices did I have besides devoting at least four years to full-time study? Well, there is a government entity that evaluates and regulates formal education given by smaller universities CENEVAL. For probably around ten years now, they have offered the ACREL (Acuerdo 286 Licenciatura) program to validate professional knowledge acquired through work experience And I'm writing this detail as several people have asked me for details on it. So, once you are accepted in the program (basic requisites: Having a high school certificate, over 30 year old, and with over 10 years of provable work experience), they can grant a Licenciatura degree (i.e. the equivalent to a four-year study program). So, in order to get it granted, I have to:

• Present a written exam, covering all of the areas of study. In my case (it varies depending on the program you are presenting), it was a 12 hour exam (over three sessions IIRC I finished each session in around three hours of the four allowed). Questions are all multiple choice. For the Software Engineering degree, you are allowed to enter each session with up to five reference books. Rules vary depending on the area. The exam was long, although not too hard. And I'm proud to say that I reached outstanding level on each of the areas of the exam
  If this information intersts you, do take a look at the links under the Study guides bullet at CENEVAL's information page. The areas to study, the style of the questions, and the general information on the exams are very close to what you will actually receive.
• The second phase (oral exam) is split in two sub-phases. The first one, the one I have just finished, is the longest one: I was assigned a practical case to develop. In my case, I was requested to write a document detailing the methodology for writing a request-tracking system for a restaurant. I was given a month for this task And in part, the reason that kept me so busy is that I subscribe to the idea that the best methodology to write an application is to write it, not just to document what you are about to do (and then fail to recognize you missed so many details). Then, there is yet another period (between one and three months, assigned randomly AFAICT) during which I would have to develop the application and a presentation, and defend it against a jury.
• Some study areas (mainly those related to health) have a third step: A practical exam. Of course, I cannot go into details on this one.

Anyway... Enough of a blog posting. I hope this is useful for somebody. And I hope it explains why I have disappeared so much lately. If everything goes as planned, I expect to be receiving my "licenciatura" title around March, and afterwards... Well, we shall see! Now... Back to work. And back to life. And back to my huge backlog of other pending things :)

We ve come a long way since the times when you needed to configure 2 X servers in XDM just to be able to use 2 X sessions at once. However there was still some way to go until recently. A number of bugs that could be wrongly attributed to bugs in the X server or in the desktop environment were actually caused by the display manager doing crap. GDM up to 2.20 Since the introduction of the flexible X servers feature, GDM hadn t evolved much on the matter of user switching. What it used to do was pretty straightforward:

• a specific protocol can be invoked by the gdmflexiserver command;
• the gdm daemon spawns a new X server on an empty console;
• it initiates another login process in it;
• when the session exits, or if the user clicks on Quit instead of logging in, the X server exits.

It is interesting to note that VT (console) switching is purely handled by the X server. When starting, the new server switches the current VT to where it is. When exiting, it automatically switches back to the VT from which it was launched. While very simple, this idea fails to work correctly every time you try to do something more complicated than starting a temporary session for a guest and exiting it. For example, if you start two of them, there is a chance that, when the X server switches back to the console it was run from, there is nothing left running in this console, leaving you with the funny Control-Alt-Fn shortcuts to find your way back to a X server. You will also meet interesting race conditions when trying to switch back to an existing session from the login window. GDM 2.28 and above In the process of rewriting the code entirely, the GDM developers tried to address a number of those shortcomings, making use of D-Bus and ConsoleKit. The new design is slightly more complicated, however.

• The gdmflexiserver tool will first try to look for an existing login window in another X server, and just switch to the VT it is in if it finds one.
• Otherwise, the daemon starts a new slave process with a new X server and a new login window, in a very similar way to what older versions did.
• When logging in as a user with an existing session, it switches to the VT it is in, but leaves the login window and its X server running.
• When going into a new session, the X server is simply left to die at the end of the session, and to switch back to the VT from which it was launched.

Not killing the X server in some cases partly addresses the problems caused by letting it switch back to the original VT when exiting. However in several ways the cure is worse than the disease.

• First of all, it will leave unused X servers, with all processes used by the login window - and that makes quite a number of them, with GDM now using a minimal GNOME session.
• When there is such a login window remaining, ConsoleKit will refuse to let you shut down your computer, being lured into thinking there is someone else using it.
• It doesn t solve the inconsistency issue. When you leave a session, you can find either of: a login window, a screensaver unlock dialog, or a black screen.

Getting it to work The modular architecture of GDM makes it possible to improve the situation. (Possible but not easy because of the millefeuille of classes.) However, it is merely a band-aid unless you fix the root issue: the X server knowing better than you which VT it should switch to when exiting. Fortunately Xorg now features an option to avoid that behavior: -novtswitch. So the first step is to enable it, and let the GDM daemon (or slave) handle VT switching through ConsoleKit. With that, the following changes are possible.

• When switching to an existing session, don t leave a X server behind. You can now kill it safely without risking a VT switch.
• On the opposite, when exiting a session, always respawn a login window on the same VT.
• The last step is to stop making a difference between the first launched X server (called a static server) and the flexible servers. The only remaining difference between a static display and a flexible one is that the static one honors automatic login/timed login settings.

The result With all these changes the behavior of the display manager is finally completely consistent.

• When exiting a session, regardless of it, you always find the same login window.
• There is never an unused process left.
• You will never find yourself facing a black screen with only keyboard shortcuts to leave you out of it.

Interestingly enough, this is very similar to what user switching looks like on Vista or MacOS X. So what now? These changes are stabilized for Debian squeeze, but of course it has been long overdue to get them accepted upstream, along with the very large number of Debian-specific changes that still lie in our packages.

Earlier in the year, Romain and I did a bunch of initial work on a wrapper from R to the GNU GSL by way of our Rcpp package for seamless R and C++ integration. But other work kept us busy and this fell a little to the side. We have now found some time to finish this work for a first release, together with a nicely detailed eleven page package vignette. As of today, the package is now a CRAN package, and Romain already posted a nice announcement on his blog and on the rcpp-devel list. So what does RcppGSL do? I gave the package its own webpage here as well and listed these points as key features of RcppGSL:

• templated vector and matrix classes: these are similar to Rcpp's own vector and matrix classes, but really are just smart pointers around the C structure expected by the library
• this means you can transfer data from R to your GSL-using programs in pretty much the same way you would in other C++ programs using Rcpp---by relying on the Rcpp::as() and Rcpp::wrap() converterrs
• at the C++ level you can use these GSL vectors in a more C++-alike way (using eg foo[i] to access an element at index i)
• yet at the same time you can just pass these vector and matrix objects to the GSL functions expecting its C objects: thanks to some cleverness in these classes they pass the right object on (see the example below)
• we also provide the lightweight views for vectors and matrices as the GSL API uses these in many places.

Also provided is a simple example which is a simple implementation of a column norm (which we could easily compute directly in R, but we are simply re-using an example from Section 8.4.14 of the GSL manual):

#include <RcppGSL.h>
#include <gsl/gsl_matrix.h>
#include <gsl/gsl_blas.h>
extern "C" SEXP colNorm(SEXP sM)  
  try  
        RcppGSL::matrix<double> M = sM;     // create gsl data structures from SEXP
        int k = M.ncol();
        Rcpp::NumericVector n(k);           // to store results
        for (int j = 0; j < k; j++)  
            RcppGSL::vector_view<double> colview = gsl_matrix_column (M, j);
            n[j] = gsl_blas_dnrm2(colview);
         
        M.free() ;
        return n;                           // return vector
    catch( std::exception &ex )  
        forward_exception_to_r( ex );
    catch(...)  
        ::Rf_error( "c++ exception (unknown reason)" );
   
  return R_NilValue; // -Wall
 

This example function is implemented in an example package contained in the RcppGSL package itself -- so that users have a complete stanza to use in their packages. This will then build a user package on Linux, OS X and Windows provided the GSL is installed (and on Windows you have to do all the extra steps of defining an environment variable pointing to and of course install Rtools to build in the first place---Linux and OS X are so much easier for development). Another complete example is in the package itself and provides a faster (compiled) alternative to the standard lm() function in R; this example is the continuation of the same example I had in several versions of my Intro to HPC with R tutorials and in the Rcpp package itself as an early example. We will try to touch base with CRAN package authors using both GSL and Rcpp to see how this can help them. The API in our package may well be incomplete, but we are always happy to try to respond to requests for additional features brought to our attention, preferably via the rcpp-devel list. More information is on the RcppGSL page. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page.

I have been helping with an overdue web app project lately, and the project uses ASP.NET MVC, so I have had to use Windows for most of my day. Since I had been using GNU/Linux almost exclusively since 2003, this is quite an adventure (imagine trying out GNU/Linux after a seven year break). It is not nearly as bad as I thought it would be, even though I am using the not-so-well-liked Windows Vista which came pre-installed with my computer.

The first thing that struck me is how much less important the OS is when compared to the situation a few years ago: a lot of the applications I depend on are web-based (GMail, Google Calendar, Google Docs, Springpad), and the ones that are not web-based are multiplatform and installable with a few clicks (GVim comes to mind). In fact, I do not think there is a single piece of software that I really missed in Windows. (An interesting corollary of this observation is that it should be equally easy to migrate to GNU/Linux nowadays.)

The one thing where the operating system matters is hardware support, and I do not think I have to say much here. After years of whack-a-mole GNU/Linux upgrades which would fix support for one component but break another (for things as essential as video and sound), it was absolutely weird having all hardware work properly at the same time (sound after suspend! no video glitches! high quality webcam picture!). There is no point recounting the reasons why hardware support in Linux is as it is, but now I am starting to think that Linux users do tend to suffer from a sort of Stockholm syndrome regarding hardware support; I know I did.

The other important thing that operating systems do is resource management, and I did not have problems here either. In fact, I liked the virtual memory behaviour much better: when Windows ran out of memory (I had filled up the smallish Windows partition and there was no space left for the swap file to grow), it started rejecting malloc calls from applications, but the system was kept alive and responsive, and I could close apps and clean up the mess without much trouble. In Ubuntu, with the default settings, whenever a runaway process ate too much RAM, the system would start thrashing so hard that it would be very difficult to fix. My standard procedure (switch to text console, log in, find and kill app using command line tools) would take minutes and was so annoying that I would sometimes just reset the computer instead. And in the end, even if I did not reset, an important app would possibly be killed anyway. I know there are kernel parameters for tweaking this behaviour (swappiness?), but I never got around to figuring them out as this situation does not arise that frequently, and testing is painful.

I did not like the Windows FS security model. ACLs are much more complicated and thus less transparent that the Unix approach, even though they are more powerful. My main gripe with them is that I do not know a good way to display them easily (as with ls -l in Unix). I have not had to deal with them much though.

Here at mancoosi we have been working for quite a while to promote and advance solver technology for FOSS distributions. We are almost at the end of the project and it is important to make the mancoosi technology relevant for the community. On goal of the project is to provide a prototype that uses part of the results of mancoosi that can based to install/remove/upgrade packages on a user machine. We certainly don't want to create yet another meta installer. This would be very time consuming and certainly going beyond the scope of the project. The idea is to create a prototype, that can work as an apt-get drop in replacement that will allow everybody to play with different solvers and installation criteria. A very first integration step is a small shell script apt-mancoosi that tries to put together different tools that we have implemented during the project. Roberto wrote extensively about his experience with apt-mancoosi a while ago showing that somehow the mancoosi tools are already usable, as proof of concept, to experiment with all solvers participating to the Misc competition. On notable obstacle we encountered with apt-mancoosi is how to pipe the result of an external solver to apt-get to effectively install the packages proposed as solution. Apt-mancoosi fails to be a drop-in replacement for apt-get exactly for this reason. The "problem" is quite simple : The idea at the beginning was to pass to apt-get, on the command line, a request that effectively represents a complete solution. We expected that, since this was already a locked-down solution, apt-get would have just installed all packages without any further modification to the proposed installation set. Of course, since apt-get is designed to satisfy a user request, and not just to install packages, we quickly realized that our evil plan was doomed to failure. The only option left, was to use libapt directly, but the idea of programming in c++ quickly made me to desist. After a bit of research (not that much after all), I finally found a viable solution to our problems in ''python-apt'' that is a low level and wrapper around libapt. This definitely made my day. Now the juicy details. the problem was to convince apt to completely bypass the solver part and just call the installer. First a small intro. python-apt has an extensive documentation with a couple of tutorials. Using python-apt is actually pretty easy (some snippet from the python-apt doco) : here we open the cache (apt.Cache is a wrapper around the low level binding in the apt_pkg module), then we update the package list. This is equivalent to apt-get update . Installing a package is equally easy : Now, the method mark_install of the module package will effectively run the solver to resolve and mark all the dependencies of the package python-apt. This is the default behavior when apt-get is used on the command line. This method has however three optional arguments that are just what I was looking for, that is autoFix, autoInst and fromUser . The explanation from the python-apt doco is quite clear. What we want is to set autoFix and autoInst to false to completely bypass the solver. So imagine that an external solver can give use a string of the form : bash+ dash=1.4 baobab- that basically asks to install bash to the newest version, dash at version 1.4 and remove baobab. Suppose also that this is a complete solution, that is, all dependencies are satisfied and there are no conflicts. The work flow of mpm (mancoosi package manager) is as follows :

• init apt-get,
• convert all packages lists + status in a cudf description,
• pass this cudf to an external solver,
• get the result and set all packages to add/remove in the apt.cache of python-apt
• download the packages
• commit the changes (effectively, run dpkg)

We already have a first prototype on the mancoosi svn. It's not released yet as we are waiting to do more testing, add more options and make it stable enough for testing. Maybe one day, this will be uploaded to debian. This is the trace of a successful installation of a package in a lenny chroot. The solver used here is the p2 solver I think we'll keep working on this python prototype for a while, but this is not certainly what we want to propose to the community. The mancoosi package manager is probably going to be written in Ocaml and integrated with dose3 and libcudf. This will allow us to gain speed and have a solid language to develop with (nothing against python, but we don't feel that a scripting language is suitable for an essential component as a package manager). Time will tell. For the moment this is just vapor-ware ...

When I accidentally looked at my personal Debian repository, I realized that it is time to remove obsolete things from it. There were packages for polld, for which which I don't see need anymore (it is now properly handled by kernel), hacked acpid packages, which were doing special things for Toshiba M30 notebook (which I don't use with Linux anymore and nobody else probably knew about these packages) or wmmp, a Window Maker applet for MPD, which I haven't used for quite a long time. All those are gone now and I don't think anybody will miss them (if you do, please tell me). What stayed there are daily snapshots of dictionaries for StarDict and gnaughty package, which was rejected as possibly causing troubles.

Filed under: Debian 0 comments Flattr this!

I've been participating in National Novel Writing Month for several years now but this is the first time I've actually managed to complete writing 50,000 words in one month. My Science Fiction novel "The Anaclepses of Darwin Fenderman" is set in a future where the last individualist must fight for free will against, robots, telepathic dogs, and Phillip K. Dick. It came out fairly well I think considering the haste in which it was written. Of course here's a hell of a lot of clean up that needs to be done before I can dare to show anyone else but in the coming months I want to see if I can get it into a fit state for publishing. Meanwhile, here's a teaser:

"Where is it then?"
"Out there somewhere.", he said gesturing vaguely at the window.
"Beyond the Hudson? But there's nothing but wasteland out there."