Search Results: "reg"

5 December 2010

Christoph Egger: The erlang experience

This week I had to write a little tool that would collect input on different channels, via socket / netcat, via a http server, .... Calls for a parralel Design -- Maybe a good place to write something real in Erlang. While erlang itself was really nice to write -- I do like Prolog as well as the bunch of functional languages -- doing networking in erlang seems a bit special, the interfaces just aren't thin wrappers around the libc stuff. Getting a Socket Text interface What sounds like a easy challenge to accomplish was actually harder than expected. All I found was some way to pass binaries representing erlang code over a socket and evaluating it remotle. While it's nice that such things are as easy to do as they are it doesn't help me with my task of moving simple strings.
start() ->
     ok, Listen  = gen_tcp:listen(51622, [binary,  packet, 0 ,
                                          reuseaddr, true ,
                                          active, true ]),
    spawn(fun () -> accept_loop(Listen) end).
accept_loop(Listen) ->
     ok, Socket  = gen_tcp:accept(Listen),
    accept_loop(Listen),
    handle(Socket)
handle(Socket) ->
    receive
         tcp, Socket, Bin  ->
            io:format("~p~n", binary_to_list(Bin));
         tcp_closed, Socket  ->
            Buffer
    end.
So the socket is now up and receives text as wanted. However, as we are already runnign a parralel program it would be nice to be hable to handle multiple socket connections in parralel right? For that we just need to add a spawn() at the right place. The right place is not the handle(Socket) but the accept_loop(Listen) because the process that called accept will receive all the tcp messages. This last part was quite obvious after finding the documentation of the active, _ properties for the socket. always here means that you'll receive all data from the socket as erlang Messages, once delivers one package and waits until it is activated again and false requires calling a method -- this would have been possible as well when forking handle(Socket). The web server Ok we also want a webserver. We do not want to run some webapplication inside appache or so, just do some post/get and simple pages. Erlang here provides a built-in httpd with some mod_esi that calls some function depending on the URL used. It doesn't do anything fancy like templating or DB backends or stuff, just taking strings and building the http answers. Unfortunately there are no examples around and basically noone seems to be using this combination (apart from some hacks like mine probably). So as I needed to get some additional information into the handler function (a Pid to connect to some service), I, as a novice, just couldn't find a way. Asking on IRC the solution was rather simple: Just using erlang's process registry. For more complex stuf gproc might prove usefull here. Result I guess I've got a huge step farther in programming erlang now. The manpages are easily found by your search engine -- for python I have to either walk through the (well structured) online documentation or search for the right link in the search results, for erlang they're typically first. Also Joe Armstrong's Books as proven usefull. The most dificult part probably is getting around all the nice extras you can do (transfering functions over sockets et al) and find out how to do the thing you need.

3 December 2010

Neil Williams: Documenting Emdebian - components and filters

A continuation of my intermittent series on Emdebian. This time covering how Emdebian changes component behaviourin Emdebian Grip.
Debian divides packages principally between those which comply with the DFSG (main), those which do not (non-free) and those which could but depend on those which do not (contrib). This leads to an enormous bias towards packages in main, which is good in terms of DFSG compliance but not so good for the size of the Packages files which every machine has to download to be able to install new stuff or upgrade existing stuff.
Emdebian is as concerned about the runtime space requirements of the system as much as the install time space requirements. Little point gaining 20Mb by removing stuff from packages if the next apt-get update adds 30Mb of cache data.
Therefore, Emdebian Grip sub-divides main (we don't have contrib or non-free packages at this time). This gives most users the smallest cache data size whilst allowing other users to add the extra components and get the full list of packages provided in Emdebian.
Alongside this division, the aim with emdebian-grip-server is to help others run secondary Emdebian Grip build machines which have a different range of packages. This allows one server to focus on XFCE and one to use other desktops or no desktops.
Components in Grip
Grip subdivides Debian main into main (~60%), dev (~30%), java, debug and doc components. The main emphasis, therefore, is that most Emdebian Grip machines will not be used to actually build other packages. I covered some of the theory on this step on the debian-embedded mailing list in 2009.
Component allocation
Packages are allocated to particular components by three main mechanisms:
  1. Package name suffix - -dev packages into dev, -dbg packages into debug, -doc packages into doc.
  2. Section names - devel goes into dev, java goes into java
  3. Override files which are visible via the emdebian website.

Override files cover situations where a package is in a problematic Section (e.g. dbus is in Section: devel when it is almost impossible to have a netbook or graphical Debian installation without it). Once Squeeze is released, I might take up these overrides with ftpmaster to see if some rationale can be constructed to make things easier.
Sources
The use of components means that an Emdebian Grip device can choose a range of components in the apt sources line. Any permutation as long as main is preserved.

deb http://www.emdebian.org/grip squeeze main dev debug java doc

More on Filtering
I mentioned last time that Emdebian does not carry all packages available in Debian. As well as filtering the files contained within packages, Emdebian Grip filters the list of packages which are allowed into the repository for each server. This is why Emdebian Grip has under 3,000 packages to be released alongside Squeeze. (Emdebian Grip 1.0 based on Lenny had some 1,000 packages.)
Filtering the list of packages means that the maintainer of the server gets to choose their package set in a similar way to how tasksel can be used to select a functionally complete set of packages at installation time. (Indeed, Emdebian Grip provides some bespoke tasks which optimise the combination of tasksel and repository filtering.) Processing packages for Emdebian Grip takes a finite amount of time per package, per architecture and restricting the total number of packages in Grip has several advantages:
  1. Further reductions in cache size - less files in the Packages file, less Packages data to download and cache.
  2. Less processing time which makes it easier to maintain an Emdebian Grip server
  3. Less data to mirror which makes it attractive for admins already running a full Debian mirror to consider squeezing in Emdebian Grip alongside.

Combining component changes and filters needs a little bit of careful handling in the server side code so that packages remain installable. Override files need to be adjusted manually, library transitions mean that once libfoo0 has been replaced by libfoo1, anything depending on libfoo1 is broken in Emdebian Grip until libfoo1 can be added to the repository (which is also a manual task). I'm glad that edos-debcheck is regaining a machine-parseable output format because this will add the chance to make at least some of these transition additions automatic. Other than these tasks, emdebian-grip-server is almost entirely automated and runs as a cron task creating verbose log files each run.
The server task runs edos-debcheck for each supported architecture and for each suite, carefully combining the various Packages files from the available components. As an alternative assessment, Debian Weather runs edos-debcheck over the Packages file for the main component only, for each suite and each architecture in Emdebian Grip. The combination of the two provides invaluable data to spot missing packages.
Equally, there are times when packages disappear from Debian. Currently, the automated scripts don't handle this aggressively but manual support scripts are available, usually for use before a release when the suites are changing less frequently.
I'll cover migrations between suites next time. In the meantime, if you want to prepare your own package set for Emdebian Grip, install emdebian-grip-server from experimental and file bugs to improve the documentation. :-)
Thanks to those who took on the Herculean task of translating the emdebian-grip-server manpages into German and French - more translations are always welcome. POT file contained inside the source package.

Jos Parrella: Escenarios de uso de Likewise Open

Hace ya alg n tiempo escrib sobre el uso de libpam-ccreds y libnss-db para conseguir la implementaci n de un servicio de cach de credenciales. Sin embargo, hace casi un a o revisit el producto Likewise Open, en un proyecto grande de autenticaci n de Linux contra un Active Directory de Microsoft, o contra un dominio Windows desarrollado con Samba. En realidad Likewise Open elimina la incertidumbre de un setup artesanal de cach de credenciales, siempre y cuando el equipo se est loggeando contra un dominio Windows. Si usted tiene la necesidad de integrar una distribuci n de Linux moderna (una versi n reciente de Debian, Ubuntu, Fedora, por ejemplo) con un dominio Windows servidor por un Active Directory o por Samba, mi recomendaci n es que use Likewise Open. Likewise es libre, abierto y gratuito, y viene en los repositorios de muchas distribuciones, o se baja de la p gina Web de Likewise. Tiene dos versiones (paquetes), likewise-open-gui y likewise-open; el primero instala el segundo y ofrece una interfaz gr fica sencilla para unir el equipo al dominio. Solo se requiere el nombre del dominio, el nombre del equipo (se toma de hostname) y el usuario y la clave de un usuario que pueda agregar equipos al dominio. Como siempre, la resoluci n del dominio se hace consultando al DNS provisto en la configuraci n de red (u otorgado por el DHCP) por ciertos registros SRV, para obtener la IP de uno de los controladores de dominio, y luego ciertas conexiones a algunos puertos del controlador de dominio. Por lo tanto es importante que, si va a probar este escenario en una m quina virtual, la conexi n est en modo puente con la red en la que se encuentra el controlador de dominio. Una vez que el equipo se une al dominio, se crea la cuenta de m quina en el controlador de dominio, y al cerrar la sesi n, ya pueden ingresar a Linux los usuarios del dominio, usando la sintaxis DOMINIO.AD\usuario o DOMINIO\usuario (formas largas y cortas) y la contrase a del dominio. Las pol ticas de inicio de sesi n y contrase as ser n respetadas. Tambi n se puede usar domainjoin-cli para unirse por CLI. Obviamente, Likewise ya ofrece el servicio de cach de credenciales y de cach de usuarios y grupos (importa los grupos del dominio) y lo hace de forma transparente. Por ello, en muchos casos yo recomendar a que si tienes un OpenLDAP y quieres hacer cach con pam-ccreds y nss-db, haz un upgrade de OpenLDAP a un dominio usando Samba y usa Likewise. Ahorra muchos dolores de cabeza. En Likewise Open no hay soporte para GPOs. En ning n caso funcionar n las GPO que se hacen para Windows, pero en versiones pagadas de Likewise hay una funci n que a ade nuevas GPO para GNOME y otras aplicaciones. No he probado estas funcionalidades.

Pietro Abate: distcheck vs edos-debcheck

This is the second post about distcheck. I want to give a quick overview of the differences between edos-distcheck and the new version. First despite using the same sat solver and encoding of the problem, Distcheck has been re-written from scratch. Dose2 has several architectural problems and not very well documented. Adding new features had become too difficult and error-prone, so this was a natural choice (at least for me). Hopefully Dose3 will survive the Mancoosi project and provide a base for dependency reasoning. The framework is well documented and the architecture pretty modular. It's is written in ocaml, so sadly, I don't expect many people to join the development team, but we'll be very open to it. These are the main differences with edos-debcheck .

Performances distcheck is about two times faster than edos-debcheck (from dose2), but it is a "bit" slower then debcheck (say the original debcheck), that is the tool wrote by Jerome Vouillon and that was then superseded in debian by edos-debcheck. The original debcheck was a all-in-one tool that did the parsing, encoding and solving without converting the problem to any intermediate format. distcheck trades a bit of speed for generality. Since it is based on Cudf, it can handle different formats and can be easily adapted in a range of situation just by changing the encoding of the original problem to cudf. Below there are a couple of test I've performed on my machine (debian unstable). The numbers speak alone.
$time cat tmp/squeeze.packages edos-debcheck -failures > /dev/null
Completing conflicts... * 100.0%
Conflicts and dependencies... * 100.0%
Solving * 100.0%

real 0m19.515s
user 0m19.193s
sys 0m0.276s
$time ./distcheck.native -f deb://tmp/squeeze.packages > /dev/null

real 0m10.859s
user 0m10.669s
sys 0m0.172s

Input The second big difference is about different input format. In fact, at the moment, we have two different tools in debian, one edos-debcheck and the other edos-rpmcheck. Despite using the same underlying library these two tools have different code bases. distcheck basically is a multiplexer that convert different inputs to a common format and then uses it (agnostically) to solve the installation problem. It can be called in different ways (via symlinks) to behave similarly to its predecessors. At the moment we are able to handle 5 different formats
  1. deb:// Packages 822 format for debian based distributions
  2. hdlist:// a binary format used by rpm based distribution
  3. synth:// a simplified format to describe rpm based package repositories
  4. eclipse:// a 822 based format that encoded OSGi plugings metadata
  5. cudf:// the native cudf format
distcheck handles gz and bz2 compressed file transparently . However if you care about performances, you should decompress your input file first and the parse it with distcheck and it often takes more time to decompress the file on the fly that run the installability test itself. There is also an experimental database backend that is not compiled by default at them moment.

Output Regarding the output, I've already explained the main differences in an old post. As a quick reminder, the old edos-debcheck had two output options. The first is a human readable - unstructured output - that was a handy source of information when running the tool interactively. The second was a xml based format (without a dtd or a schema, I believe) that was used for batch processing. distcheck has only one output type in the YAML format that aims to be human and machine readable. Hopefully this will cater for both needs. Moreover, just recently I've added the output of distcheck a summary of who is breaking what. The output of edos-debcheck was basically a map of packages to the reasons of the breakage. In addition to this information distcheck gives also a maps between reason (a missing dependency or a conflict) to the list of packages that are broken by this problem.This additional info is off by default, but I think it can be nice to know what is the missing dependency that is responsible for the majority of problems in a distribution... For example, calling distcheck with --summary :
$./distcheck.native --summary deb://tests/sid.packages
backgroud-packages: 29589
foreground-packages: 29589
broken-packages: 143
missing-packages: 138
conflict-packages: 5
unique-missing-packages: 52
unique-conflict-packages: 5
summary:
-
missing:
missingdep: libevas-svn-05-engines-x (>= 0.9.9.063)
packages:
-
package: enna-dbg
version: 0.4.0-4
architecture: amd64
source: enna (= 0.4.0-4)
-
package: enna
version: 0.4.0-4
architecture: amd64
source: enna (= 0.4.0-4)
-
missing:
missingdep: libopenscenegraph56 (>= 2.8.1)
packages:
-
package: libosgal1
version: 0.6.1-2+b3
architecture: amd64
source: osgal (= 0.6.1-2)
-
package: libosgal-dev
version: 0.6.1-2+b3
architecture: amd64
source: osgal (= 0.6.1-2)
Below I give a small example of the edos-debcheck output compared to the new yaml based output.
$cat tests/sid.packages edos-debcheck -failures -explain
Completing conflicts... * 100.0%
Conflicts and dependencies... * 100.0%
Solving * 100.0%
zope-zms (= 1:2.11.1-03-1): FAILED
zope-zms (= 1:2.11.1-03-1) depends on missing:
- zope2.10
- zope2.9
zope-tinytableplus (= 0.9-19): FAILED
zope-tinytableplus (= 0.9-19) depends on missing:
- zope2.11
- zope2.10
- zope2.9
...
And an extract from the distcheck output (the order is different. I cut and pasted parts of the output here...)
$./distcheck.native -f -e deb://tests/sid.packages
report:
-
package: zope-zms
version: 1:2.11.1-03-1
architecture: all
source: zope-zms (= 1:2.11.1-03-1)
status: broken
reasons:
-
missing:
pkg:
package: zope-zms
version: 1:2.11.1-03-1
architecture: all
missingdep: zope2.9 zope2.10
-
package: zope-tinytableplus
version: 0.9-19
architecture: all
source: zope-tinytableplus (= 0.9-19)
status: broken
reasons:
-
missing:
pkg:
package: zope-tinytableplus
version: 0.9-19
architecture: all
missingdep: zope2.9 zope2.10 zope2.11
...

Future The roadmap to release version 1.0 of distcheck is as follows:
  1. add background and foreground package selection. This feature will allow the use to specify a larger universe (background packages), but check only a subset of this universe (foreground packages). This should allow users to select packages using grep-dctrl and then pipe them to discheck . At the moment we can select individual packages on the command line or we can use expression like bash (<= 2.7) to check all version of bash in the universe with version greater than 2.7.
  2. code cleanup and a bit of refactoring between distcheck and buildcheck (that is a frontend for distcheck that allow us to report broken build dependencies)
  3. consider essential packages while performing the installation test. Here there are few things we have to understand, but the idea would be to detect possible problems related the implicit presence of essential packages in the distribution. At the moment, distcheck performs the installation test in the empty universe, while ideally, the universe should contain all essential packages.
  4. finish the documentation. The effort in underway and we hope to finalize shortly to release the debian package in experimental.

Sam Hartman: Bad Hair Day for Kerberos

Tuesday, MIT Kerberos had a bad hair day one of those days where you re looking through your hair and realize that it s turned to Medusa s snakes while you weren t looking. Apparently, since the introduction of RC4, MIT Kerberos has had significant problems handling checksums. Recall that when Kerberos talks about checksums it s conflating two things: unkeyed checksums like SHA-1 and message authentication codes like HMAC-SHA1 used with an AES key derivation. The protocol doesn t have a well defined concept of an unkeyed checksum, although it does have the concept of checksums like CRC32 that ignore their keys and can be modified by an attacker. One way of looking at it is that checksums were over-abstracted and generalized. Around the time that 3DES was introduced, there was a belief that we d have a generalized mechanism for introducing new crypto systems. By the time RFC 3961 actually got written, we d realized that we could not abstract things quite as far as we d done for 3DES. The code however was written as part of adding 3DES support. There are two major classes of problem. The first is that the 3DES (and I believe AES) checksums don t actually depend on the crypto system: they re just HMACs. They do end up needing to perform encryption operations as part of key derivation. However the code permitted these checksums to be used with any key, not just the kind of key that was intended. In a nice abstract way, the operations of the crypto system associated with the key were used rather than those of the crypto system loosely associated with the checksum. I guess that s good: feeding a 128-bit key into 3DES might kind of confuse 3DES which expects a 168-bit key. On the other hand, RC4 has a block size of 1 because it is a stream cipher. For various reasons, that means that regardless of what RC4 key you start with, if you use the 3DES checksum with that key, there are only 256 possible outpus for the HMAC. Sadly, that s not a lot of work for the attacker. To make matters worse, one of the common interfaces for choosing the right checksum to use was to enumerate through the set of available checksums and pick the first one that would accept the kind of key in question. Unfortunately, 3DES came before RC4 and there are some cases where the wrong checksum would be used. Another serious set of problems stems from the handling of unkeyed checksums. It s important to check and make sure that a received checksum is keyed if you are in a context where an attacker could have modified it. Using an md5 outside of encrypted text to integrity protect a message doesn t make sense. Some of the code was not good about checking this. What worries me most about this set of issues is how many new vulnerabilities were introduced recently. The set of things you can do with 1.6 based on these errors was significant, but not nearly as impressive as 1.7. A whole new set of attacks were added for the 1.8 release. In my mind, the most serious attack was added for the 1.7 release. A remote attacker can send an integrity-protected GSS-API token using an unkeyed checksum. Since there s no key the attacker doesn t need to worry about not knowing it. However the checksum verifies, and the code is happy to go forward. I think we need to take a close look at how we got here and what went wrong. The fact that multiple future releases made the problem worse made it clear that we produced a set of APIs where doing the worng thing is easier than doing the right thing. It seems like there is something important to fix here about our existing APIs and documentation. It might be possible to add tests or things to look for when adding new crypto systems. However I also think there is an important lesson to take away at a design level. Right now I don t know what the answers our, but I encourage the community to think closely about this issue. I m speaking about MIT Kerberos because I m familiar with the details there. However it s my understanding that the entire Kerberos community has been thinking about checksums lately, and MIT is not the only implementation with improvements to make here.

Rapha&#235;l Hertzog: Librement: a new way to help people who want to contribute to free software

Find your way in the free software worldI have this project in my head, I want to work on it but I always lack the time. In order to go forward, I thought I could write about it, at least it would let me clarify my ideas and the core goals. So here I am, I will present you Librement (I have registered the alioth project but it s empty). The core goal is to make it easy for every user to contribute to free software in some way. I will now present the main features that I envision. Defining skills and interests In order to propose tasks that the user can do, we must have an idea of his skills. So on the first run (and later through a preferences menu) the user will be invited to define his skills: Maybe we can also ask which skills he would like to learn. Because contributing to free software is a nice opportunity to learn new skills! We should also find out what the user is interested in. What are his favorite free software projects? What kind of contributions would he like to do (documentation, translation, coding, bug fixing, bug triaging, creating artwork, donations, etc.)? Choose activities and pick concrete tasks Based on the user s skills and his interests, the software shows a list of possible activities. The user can then sort that list, from the most interesting one to those that he doesn t want to do. Each activity can generate concrete tasks. For example, the activity Do translation for Debian could generate a task Translate strings in debconf/fr.po or Review translations in partman/fr.po . Work on tasks When the user decides to work on a task, a step-by-step assistant helps him/her. It can automate some steps and provide explanations for the remaining ones, for example in the case of a translation for Debian: If the tasks is not completed in one go, the user can resume it the next time. Each free software project must provide some meta-information describing the various workflows involved for contributing to the different parts of the project. If necessary the project can also provide new plugins to support new operations that are not available in the default library. Setting goals In order to keep the user motivated, the software could track how much time he spent contributing to free software and it could verify if the user reached the goals he picked up for himself. Maybe it can also hook into the OMG Trophy Awarding System. The sky is the limit I hope that you now have a clearer idea of what this desktop application is supposed to be. There are literally hundreds of ways to contribute to free software and I like the idea that we can streamline the process for most users. All the plugins implementing activities can use local information (list of packages installed with their versions, configuration settings, etc.) to propose tasks targetted to the user and highly beneficial for the corresponding free software projects. For example, a bug tagged unreproducible might benefit from a few more users trying to reproduce it. The software could direct the user to this bug report if it detects that he/she runs the same version on the same architecture and that this software is regularly run on the system. Many projects have created operations or events to encourage people to contribute, they could all be implemented as dedicated activities in Librement. I m thinking of stuff like Gnome Love, Ubuntu s 5-a-day, Ubuntu s 100 papercuts, etc. Even for people who have no time to contribute, the application can still be useful by referencing the various ways to donate money (or material) to projects that they are using. Feedback welcome I m excited by the potential of such an application, but it s normal since it s my idea. Do you believe it can be useful and popular? Do you have ideas of exciting activities that such a framework can offer? PS: If you wonder how I came up with the name Librement , here s the explanation. It s a French word which means freely . And users who want to give back are trying to live up to the principles of free software, which I sum up by they are trying to live freely .
Follow me on Identi.ca, Twitter and Facebook. Or subscribe to this blog by RSS or by email.

11 comments Liked this article? Click here. My blog is Flattr-enabled.

2 December 2010

Matt Zimmerman: Breathing information

I ve written previously about my reading habits, online and offline, and the patterns I extrapolate to content consumption in general. I ve been talking with other people about this as well, and am beginning to develop a model to apply to my daily life. There are plenty of unsatisfactory metaphors circulating in this area: we eat the information diet , drink from the fire hose , endure the information explosion , and so on. None of them describe the richness of my experience: the profound variations in style, texture, speed, depth and movement are lost in this kind of dry imagery. Instead, I think of it like respiration. We inhale information, and we also exhale it transformed. We do this, consciously or unconsciously, every moment of our lives. Sometimes we do it quickly, other times slowly, and it can be relaxing or stimulating. We only retain a small amount of what we take in, but it becomes a part of us. We can immerse ourselves deeply, meditatively, in a series of breaths, or fail to notice as we breathe shallowly or pause altogether. One breath may be virtually silent, the next filled with a song or a question or a piercing whistle. We maintain a balance in our breathing, and I aspire to do the same with information: reading and writing, listening and speaking, seeing and being seen. I don t mean this simplistically, that I should do both in equal proportion (imagine trying to write as many words as you read!), but doing both consistently. It is often only when I share an idea that I come to understand it deeply, no matter how much I have read about it. By writing it down, or telling someone about it, I naturally fill in the gaps in my understanding and create mental structures which help me recall and apply what I ve learned. My input and output should be balanced appropriately for my circumstances. Rapid-fire email is like hyperventilation: I can do it for a while, reading and replying in quick succession, and even feel energized by it, but if I go on for too long, I get dizzy. Running might call for a certain breathing ratio, and Yoga quite a different one. Both can be healthy practices, but they are different, and each requires focus and consistency. Another key lesson from breathing is to let go. A friend recently told me about his daily online news routine, which included closing any unread tabs at the end of the session. I will sometimes hang onto an article or a video for days or weeks before I find the opportunity to take it in. I eventually get around to most of them, but meanwhile they are taking up space and causing a continuous low level of anxiety or guilt. This information isn t going anywhere, and if it s truly significant for me, it will most likely turn up again. If it doesn t, that s OK too. Similarly, it s not wise to breathe stale, indoor air for too long. I will try to step outside regularly, and engage with people and media from outside my usual sphere. When the weather is tolerable, I ll open up the house and let plenty of fresh air circulate. This will help me avoid getting stuck in the echo chamber of my own ideas, or in groupthink. Perhaps most significantly, I will try to remember that information exchange, like breathing, is not an end in itself. It is a means to action. This will remind me to get out of my head regularly, and do something significant with what I ve learned. Ready? Breathe.

1 December 2010

Gunnar Wolf: Back to... Normality?

For the past month, I got disconnected from basically all of my usual activities. Not only from Debian work (well, yes, I followed up on a couple of important bugs and releases, but I was in a very low power consumption mode of sorts), DebConf organization/following (some people have contacted me regarding the DebConf 12 venue decision, for which we should start working and have a presentation soon) Even the always-so-important Real Life got pushed aside. I barely participated in this year's EDUSOL (On-line Encounter of Education and Free Software). Even work suffered I asked my boss to hold any incoming requests for me (save, of course, for urgent stuff). I am now shyly coming up from the hole I dug myself in for this time. Why did I do such a thing? I do not have a formal education beyond high school. Due to several personal issues, I was accepted to the best university in my country, and to the first generation of the study program I would definitively love to have followed, but didn't attend. Of course, I had a fair bit of computing background, enough to be competitive and start working. After amazingly few years, I didn't only get a decent job, but I got a job precisely at the very university I wanted to study at (and where my father has worked for the past 40 years). And, even with some changes along the years, here I am with 10 years of laboral history in this university, with an academic position (although the smallest academic position, but still). It is one of my life's achievements, one of the things that makes me proudest of. Still, I don't want to stay at this level for the rest of my life. And an institution such as this one has rules Yes, flexible enough to allow me in, but rules whose importance I cannot deny. I aspire to become a professor/researcher at some point, and that is just not possible without filling in the prerequisites. Besides, I want to try my hand at teaching at a universitary level. So, what choices did I have besides devoting at least four years to full-time study? Well, there is a government entity that evaluates and regulates formal education given by smaller universities CENEVAL. For probably around ten years now, they have offered the ACREL (Acuerdo 286 Licenciatura) program to validate professional knowledge acquired through work experience And I'm writing this detail as several people have asked me for details on it. So, once you are accepted in the program (basic requisites: Having a high school certificate, over 30 year old, and with over 10 years of provable work experience), they can grant a Licenciatura degree (i.e. the equivalent to a four-year study program). So, in order to get it granted, I have to: Anyway... Enough of a blog posting. I hope this is useful for somebody. And I hope it explains why I have disappeared so much lately. If everything goes as planned, I expect to be receiving my "licenciatura" title around March, and afterwards... Well, we shall see! Now... Back to work. And back to life. And back to my huge backlog of other pending things :)

Josselin Mouette: Getting user switching to not suck

We ve come a long way since the times when you needed to configure 2 X servers in XDM just to be able to use 2 X sessions at once. However there was still some way to go until recently. A number of bugs that could be wrongly attributed to bugs in the X server or in the desktop environment were actually caused by the display manager doing crap. GDM up to 2.20 Since the introduction of the flexible X servers feature, GDM hadn t evolved much on the matter of user switching. What it used to do was pretty straightforward: It is interesting to note that VT (console) switching is purely handled by the X server. When starting, the new server switches the current VT to where it is. When exiting, it automatically switches back to the VT from which it was launched. While very simple, this idea fails to work correctly every time you try to do something more complicated than starting a temporary session for a guest and exiting it. For example, if you start two of them, there is a chance that, when the X server switches back to the console it was run from, there is nothing left running in this console, leaving you with the funny Control-Alt-Fn shortcuts to find your way back to a X server. You will also meet interesting race conditions when trying to switch back to an existing session from the login window. GDM 2.28 and above In the process of rewriting the code entirely, the GDM developers tried to address a number of those shortcomings, making use of D-Bus and ConsoleKit. The new design is slightly more complicated, however. Not killing the X server in some cases partly addresses the problems caused by letting it switch back to the original VT when exiting. However in several ways the cure is worse than the disease. Getting it to work The modular architecture of GDM makes it possible to improve the situation. (Possible but not easy because of the millefeuille of classes.) However, it is merely a band-aid unless you fix the root issue: the X server knowing better than you which VT it should switch to when exiting. Fortunately Xorg now features an option to avoid that behavior: -novtswitch. So the first step is to enable it, and let the GDM daemon (or slave) handle VT switching through ConsoleKit. With that, the following changes are possible. The result With all these changes the behavior of the display manager is finally completely consistent. Interestingly enough, this is very similar to what user switching looks like on Vista or MacOS X. So what now? These changes are stabilized for Debian squeeze, but of course it has been long overdue to get them accepted upstream, along with the very large number of Debian-specific changes that still lie in our packages.

Gintautas Miliauskas: Windows (I)

I have been helping with an overdue web app project lately, and the project uses ASP.NET MVC, so I have had to use Windows for most of my day. Since I had been using GNU/Linux almost exclusively since 2003, this is quite an adventure (imagine trying out GNU/Linux after a seven year break). It is not nearly as bad as I thought it would be, even though I am using the not-so-well-liked Windows Vista which came pre-installed with my computer.

The first thing that struck me is how much less important the OS is when compared to the situation a few years ago: a lot of the applications I depend on are web-based (GMail, Google Calendar, Google Docs, Springpad), and the ones that are not web-based are multiplatform and installable with a few clicks (GVim comes to mind). In fact, I do not think there is a single piece of software that I really missed in Windows. (An interesting corollary of this observation is that it should be equally easy to migrate to GNU/Linux nowadays.)

The one thing where the operating system matters is hardware support, and I do not think I have to say much here. After years of whack-a-mole GNU/Linux upgrades which would fix support for one component but break another (for things as essential as video and sound), it was absolutely weird having all hardware work properly at the same time (sound after suspend! no video glitches! high quality webcam picture!). There is no point recounting the reasons why hardware support in Linux is as it is, but now I am starting to think that Linux users do tend to suffer from a sort of Stockholm syndrome regarding hardware support; I know I did.

The other important thing that operating systems do is resource management, and I did not have problems here either. In fact, I liked the virtual memory behaviour much better: when Windows ran out of memory (I had filled up the smallish Windows partition and there was no space left for the swap file to grow), it started rejecting malloc calls from applications, but the system was kept alive and responsive, and I could close apps and clean up the mess without much trouble. In Ubuntu, with the default settings, whenever a runaway process ate too much RAM, the system would start thrashing so hard that it would be very difficult to fix. My standard procedure (switch to text console, log in, find and kill app using command line tools) would take minutes and was so annoying that I would sometimes just reset the computer instead. And in the end, even if I did not reset, an important app would possibly be killed anyway. I know there are kernel parameters for tweaking this behaviour (swappiness?), but I never got around to figuring them out as this situation does not arise that frequently, and testing is painful.

I did not like the Windows FS security model. ACLs are much more complicated and thus less transparent that the Unix approach, even though they are more powerful. My main gripe with them is that I do not know a good way to display them easily (as with ls -l in Unix). I have not had to deal with them much though.

I have had no stability issues. It seems that this argument should be put to rest.

Long blog posts are annoying, so I will leave the user interface, the developer experience and my complaints with the system for separate posts (edit: you can now read my next post about the UI).

30 November 2010

Jordi Mallach: The unforeseen consequences of our GR 11 Summer walk

I knew walking all over the Pyrenees during a whole month would come with some side effects. I could imagine having a few muscular issues in my legs and some back pain for a while after getting back; after all we did over 7 hours of exercise every single day during a month. What I got after our hike was totally unexpected... I should have run a half marathon in Gandia on the 21st, and instead I stayed at home doing some assignments. When we came back, I was in a really good condition, and wanted to keep the good shape we had built the month before. Given I haven't been able to swim for nearly a year now, due to the Misterious Shoulder Injury and I don't have enough time to go out cycling regularly, I centered my efforts in running, with the idea of starting to get prepared for cross/mountain races this season. Training had to wait a little because some days after getting back, I got a sudden pain on both knees, which even made climbing stairs difficult. It stuck for a while, and when it finally went away, more or less, I went ahead and tried going out for a run. I haven't been training for over two years, so I indended to increase the distances proggressively. The first day I went out, I ran really nicely for the first 25 minutes, and suddenly I started getting an intense pain in the outer side of my right knee, to the point I could barely walk back home. I've tried letting them rest for weeks, and every time I try them, the pain comes back. My physiotherapist says my body has developed quite a few asymmetries, probably caused by the many days I had to walk crippled by a big blister in my right foot. As time passes, I feel I'm losing all the physical improvements I had developed during the summer, and I can't do anything about it. The new approach to tackle this is yoga. I know my body isn't too elastic, and the lack of stretching during the walk made it even worse, so I'll try to forget about real traning during some time and focus on healing my muscles. Hopefully, this will help me resolve the nastiest physical problem I've ever come across.

Debian News: DebConf 11 to take place July 24-30, 2011

DebConf 11 will take place from July 24 to 30, 2011, in the city of Banja Luka, Republika Srpska, Bosnia and Herzegovina.

The final calendar of the event is:

The call for papers and registration dates are still to be decided.

More information at the DebConf 11 website.

Robert Collins: testrepository iteration for python projects

Tesetrepository has a really nice workflow for fixing a set of failing tests:
  1. Tell it about the failing tests (e.g. by doing a full test run, or running a single known failing test)
  2. Run just the known failing tests (testr run failing)
  3. Make a change
  4. Goto step 2
As you fix up the tests testr will just give your test runner a smaller and smaller list of tests to run. However I haven t been able to use that feature when developing (most) Python programs. Today though, I added the necessary support to testtools, and as a result subunit (which inherits its thin test runner shim from testtools) now supports load-list. With this a simple .testr.conf can support this lovely workflow. This is the one used in testrepository itself: it runs the testrepository tests, which are regular unittest tests, using subunit.run this gives it subunit output, and tells testrepository how to run a subset of tests.

[DEFAULT]
test_command=python -m subunit.run $IDOPTION testrepository.tests.test_suite
test_id_option=--load-list $IDFILE

29 November 2010

Axel Beckert: Useful but Unknown Unix Tools: colored cal(endar)

Another thing I regard as useful on the commandline are colors. (Others commonly refer to my color preferences as eye cancer, but that s not relevant here. ;-) Colors help to easily distinguish between relevant and non-relevant things or to separate different things. One colored command line program I use nearly everyday is the colored fork of bsdmainutils s well know (or at least better known) cal program whose output looks like this:
$ cal
    November 2010    
Su Mo Tu We Th Fr Sa
    1  2  3  4  5  6
 7  8  9 10 11 12 13
14 15 16 17 18 19 20
21 22 23 24 25 26 27
28 29 30
$ 
Since the colored fork calls itself still cal (as the Debian source package is called), to avoid conflicts with bsdmainutils s cal, the binary package and the binary itself are called ccal (short for colored cal) and its output looks like this:
$ ccal
    November 2010     
 Mo Tu We Th Fr Sa Su 
  1  2  3  4  5  6  7 
  8  9 10 11 12 13 14 
 15 16 17 18 19 20 21 
 22 23 24 25 26 27 28 
 29<30>               
                      
$ 
ccal also use the locale definitions (LC_TIME), if available, that s why the one screenshot is Sun-Sat and the other one is Mon-Sun. According to the man page, ccal has some more improvements over the classic cal: If displaying the single-month format, ccal will look for a date file (the default file or whatever you specify with the -d option). If found, ccal will read the file, looking for special date descriptions for that month which will be displayed to the right of the calendar. By default, up to 24 appointments (number may be changed with -m) may be displayed per month. If the current date happens to fall on one of these special dates, it will be flagged by an asterisk. If there is room, appointments for the next month may also be displayed (next month s dates having definitions like 2nd Thursday will be skipped). But I must admit, I don t use that feature. I just use it as on-demand calendar sheet. The colored screenshot above is btw. generated with Adam Borowski s ansi2html which popped up after I published my Intent to package aha (for Ansi HTML Adapter), the first ANSI colors to HTML converter I found on the net. (And yes I searched for it because of this and some yet to come blog postings. No spoilers here though. ;-) While capturing colored output of ls and some other tools was easy, it needed a little hack to capture ccal s output, because if you pipe ccal output to anything, it drops its colors. Always. The solution was to run it in GNU Screen, log the output and then pipe the log file through aha or ansi2html :
$ screen -c /dev/null -L ccal; cat screenlog.0   ansi2html > ccal.html
Probably the output of every program which drops colors when writing to a pipe instead of a terminal can be fetched that way. Update, 2:04 CET: As Adam Borowski points out in message 62 of #605380, script is the better tool for making tools believe they talk to a terminal:
$ script -q /dev/null -c "ccal"   ansi2html
Catching the output of a whole year s calender ( ccal 2011 ) failed with GNU screen though, as it was truncated on the 24th of September by GNU screen (as it was already truncated in the screenlog.0), so no screenshot of that for the moment. And with script, I d get the whole output, but would have to manually fix the styles so they don t get posted literally to Planet Debian. So look for yourself how the output of ccal 2011 looks like by installing ccal. ;-)

Theodore Ts'o: Google has a problem retaining great engineers? Bullcrap.

Once again, there s been another story about how Google is having trouble retaining talent. Despite all Eric Schmidt s attempts to tell folks that Google s regretted attrition rate has not changed in seven years, this story just doesn t want to seem to die. (And those stories about Google paying $3.5 million and $7 million to keep an engineer from defecting to Facebook? As far as I know, total bull. I bet it s something made up by some Facebook recruiter who needed to explain how she let a live prospect get away. :-) At least for me, the complete opposite is true. There are very few companies where I can do the work that I want to do, and Google is one of them. A startup is totally the wrong place for me. Why? Because if you talk to any venture capitalist, a startup has one and only one reason to exist: to prove that it has a scalable, viable business model. Take diapers.com for example. As Business Week described, while they were proving that they had a business model that worked, they purchased their diapers at the local BJ s and shipped them via Fedex. Another startup, Chegg, proved its business model by using Amazon.com to drop ship text books to their first customers. (The venture capitalist Mark Maples talked about this in a brilliant talk at the Founders Showcase; the Chegg example starts around 20:50 minutes in, but I d recommend listening to the whole thing, since it s such a great talk.) You don t negotiate volume discounts with textbook publishers, or build huge warehouses to hold all of the diapers that you re going to buy until you prove that you have a business model that works. Similarly, you don t work on great technology at a startup. Startups, by and large, aren t about technology at least, not the Web 2.0 startups like Facebook, Foursquare, Twitter, Groupon, etc. They are about business model discovery. So if you are fundamentally a technologist at heart, whose heart sings when you re making a better file system, or fixing a kernel bug, you re not going to be happy at a startup. At least, not if the startup is run competently. If you have the heart of an entrepreneur, and you are willing to roll the dice (since 9 out of 10 startups go belly up; those are the ones that failed to find a viable business model), then sure, go for a startup. And understand that your job will be to make something that works well at a small scale, quick, dirty, and cheap. If that means using some proprietary software, then that s what you should do. Hopefully you ll get lucky and win the IPO lottery. But if your primary interest is to doing great engineering work, then you want go to company that has a proven business model. Google has enough scale that I can sleep well knowing that improvements I am making to Linux and the ext4 file system are saving enough money for the company that Google is making a profit on me. That is, the value Google is getting out of my work is worth many multiples of my total compensation. And that s a position that every engineer should strive for, since that s how you can be confident that you will remain gainfully employed. The fact that I can do what I love, and go to conferences, and all of my work is open source that s just total icing on the cake. :-) No related posts.

Petter Reinholdtsen: Debian Edu development gathering and General Assembly for FRiSK

On friday, the first Debian Edu / Skolelinux development gathering in a long time take place here in Oslo, Norway. I really look forward to seeing all the good people working on the Squeeze release. The gathering is open for everyone interested in learning more about Debian Edu / Skolelinux. On Saturday, the Norwegian member organization taking care of organizing these development gatherings, Fri Programvare i Skolen, will hold its General Assembly for 2010. Membership is open for all, and currently there are 388 people registered as members. Last year 32 members cast their vote in the memberdb based election system. I hope more people find time to vote this year.

Rapha&#235;l Hertzog: How to find the right Debian packages: high-level search interface

The Debian archive is known to be one of the largest software collections available in the free software world. With more than 16,000 source packages and 30,000 binary packages, users sometimes have trouble finding packages that are relevant to them. Debian developer Enrico Zini has been working on infrastructure to solve this problem. During the recent mini-debconf Paris, Enrico gave a talk presenting what he has been working on in the last few years, which hasn t gotten yet the attention it deserves . Enrico is known in the Debian community for the introduction of debtags, a system used to classify all packages using facets. Each facet describes a specific kind of property: type of user-interface, programming language it s written in, type of document manipulated, purpose of the software, etc. His most recent work builds on that. It is available in Debian and Ubuntu in the apt-xapian-index package. Its purpose is to allow advanced queries over the database of available packages. Users of apt-xapian-index He started by presenting some early users of the infrastructure. The most widely know is Ubuntu s software center. Its search feature provides results almost instantly thanks to apt-xapian-index. But it is a very simple interface that doesn t exploit many of the advanced features provided by the apt-xapian-index. Another early adopter, making use of some more advanced features, is GoPlay!. It s a graphical user interface to find games. It makes use of debtags to classify games so that you can browse, for example, all 3D action/arcade games related to cars. GoPlay has even been extended to be a more generic debtags based package browser and the package now also provides GoLearn!, GoAdmin!, GoNet!, GoOffice!, GoSafe!, and GoWeb!. Fuss-launcher is an application launcher and not a package browser, but by using apt-xapian-index, it s able to reuse information provided at the package level to make it easier to find installed applications. Package descriptions tend to be more verbose than those embedded in .desktop files. Enrico also showed another nice feature to the audience: if you drag a document onto its window, it will show you a list of applications that can open it. Last but not least, apt-xapian-index provides a command line search tool that is vastly superior to the traditional apt-cache search: it s axi-cache search (axi stands for apt-xapian-index). Enrico compared the output of a search on the letter r . While apt-cache spits out an infinite list of packages containing this letter somewhere in the description, axi-cache only listed packages related to GNU R. He also demonstrated the contextual tab completion. It makes it easy to use debtags and to refine your search. Once you have typed a first keyword, the tab-completion for the second one only contains keywords or debtags that are actually able to provide more restrictive results. Advanced queries with logical operations (AND, OR, NOT, XOR) are also supported. Features of the backend Enrico then dived into the internals. Xapian s search engine is at the root of this infrastructure. He likes it because it s a simple library (i.e. no daemon) and it has nice Python bindings. While apt-xapian-index s core work is to index the descriptions of all the packages, it actually stores much more and can be easily extended with plugins (written in Python). For instance, the information stored encompasses: He already has plans to store more: adding popularity contest data (see wishlist bugs #602180 and #602182) will make it possible to sort query results in a useful way. The most widely used applications are good choices when it comes to community support, and they are likely of better quality due to the larger user base. Adding timestamps of the last installation/upgrade/removal, will make it easier to pin-point a regression to a specific package update. The generated index is world-readable and can be used from any application provided it can use the Xapian library which is written in C++ but has bindings for Perl, Python, PHP, Java, Tcl, C#, and Ruby. Call for experimentation Enrico believes that many useful applications have yet to be invented on top of apt-xapian-index s features. He s calling for experimentation and asking for new ideas. The only practical limit that he has encountered is the size of the index: currently it varies between 50 Mb (Debian unstable without translation) and 70 Mb (Debian stable/testing/unstable with one translation). He would like it to not grow over 100 Mb since it s installed by default (due to aptitude recommending it) and he s not comfortable with the idea of using more than 20% of the disk footprint of a basic install just for this service. That s why the index was configured to not store the position of the terms: it s thus not possible to find out packages whose description contains the word statistical immediately followed by the word computing . You can however find those which have both terms somewhere in their description. Enrico wondered if apt-xapian-index offers too much freedom. That could explain why few people experimented with it despite his numerous blog posts with code samples and information on how to get started using it. But it s not difficult to imagine use cases for this data. It could be used to extend tools like rc-alert or wnpp-alert, for example. They provide a long list of Debian packages that are looking for some help and are installed on the machine. With apt-xapian-index, it would be possible to restrict the results to the set of packages written in a specific programming language or for a particular desktop environment. The more likely explanation is that too few people know about the tool. There are many more itches to scratch where apt-xapian-index s features could be very useful, and my guess is that Enrico s wishes will eventually come true.
This article was first published in Linux Weekly News. Click here to subscribe to my newsletter and keep up with the Debian/Ubuntu news thanks to my monthly digest.

No comment Liked this article? Click here. My blog is Flattr-enabled.

28 November 2010

Gregor Herrmann: RC bugs 2010/46 - 2010/47

this time my report on my work on RC bugs contains the list of what I did during the bug squashing party in bern. thanks to the nice guys from switzerland, france, & italy for the great time, & especially to gaudenz for organising the meeting!

27 November 2010

Paul van Tilburg: 10 Years of Spacelabs

A few months ago I realised it has been 10 years already since Spacelabs 1 was founded. Today we will celebrate that with a reunion and probably reminisce about past projects, events and experiences. Therefore, I thought it would be nice to accompany this with a post. The past Spacelabs was founded to create something that sits in between research and consumer appliances. Since we are associated to the Electro-Optical Communication group these things often had to do with bandwidth. For example, showing the ease of use of plastic optical fiber or participate in GPRS/UMTS trials. However, due to the loose structure of Spacelabs, we ventured into many other projects of our own such a information system-enabled fridge with a barcode reader, a dynamic DNS system, etc. At some point we got our own Internet connection with an accompanying subnet. It has been fun and rewarding to learn how to setup and run a mini ISP. Spacelabs doesn t have any formal structure: when you are on the mailing list, you are a member. There are no regular meetings, no hierarchy, no obligations. So in some ways, we are a group that is much like a free/open source software community. I think that this approach has provided us the drive to work both on projects during our free time as well as our own ideas. I feel that Spacelabs has contributed so much to my life. It might not be the typical study association, but it was a big supplement to my studies, it was fun and it was completely ours. Not only did it teach me many, many technical things from networks to program languages to software design but it also gave me experiences on other levels, such as very attuned group work and responsibilities for key systems. Spacelabs in 2006 The future Soon, everyone connected to Spacelabs will have finished their Master s or PhD (in my case) and nobody will be left to continue the project. Although this is a bit sad, I think we can be proud of what we have accomplished, learnt and experienced. I will miss our location high in the Potentiaal/E-hoog building overlooking Eindhoven, and our being together as a group. We are however not gone yet, there is still a lot to plan and to work on if we want to rescue the huge amount of servers and services that we have accumulated over the years if we were to leave

1 Yes, the website is very much from the past.

Russell Coker: Open Respect

On the 5th of November Jono Bacon wrote a blog post titled MAKING OUR WORLD MORE RESPECTFUL [1]. He then registered the domain OpenRespect.org for his manifesto [2].Aaron Siego supports the general concept of being more respectful while listing some of the practical problems that occur in a multi-cultural world [3]. If you read nothing else about this issue I recommend reading Aaron s post (which I summarised very poorly, I can t think of a good one-sentence explanation).Sam Varghese wrote an article suggesting that it was an attempt to deflect criticism from Ubuntu [4]. I think that Ubuntu receives a lot of unfair criticism, but don t think it s really relevant to the issue at hand. Sam criticises some of the Open Respect supporters for failing to show respect in the past.Jono wrote a follow-up blog post titled REFLECTIONS ON RESPECT [5], in that post he said Sometimes we were disrespectful, and frankly, sometimes we were also inadvertently assholes. We never set out to be assholes, but we did set out to be edgy in a satirical way, but we sometimes went too far and I apologize for that. But you know what, we all grow and mature in different parts of our lives about his past LugRadio work. Maybe instead of focusing on being respectful of others it would give more benefit to focus on how to apologise, back down, and move on after being disrespectful. It seems that the situations of epic disrespect tend not to be one-off incidents but are instead arguments of greatly expanded scope that arose from someone refusing to admit any error after doing something that is considered to be offensive. [Someone has registered OpenApology.org, just in case you were wondering.]Maybe advocating Rogerian Argument would be another way of reducing some of the problems in the community.Finally Kirrily wrote an insightful post about the ineffectiveness of discussing offensive things [6]. In terms of issues such as Feminism (which Kirrily uses as an example and which is also a matter of significant interest for her) discussions about offense apparently tend to end up being about the person who was offended rather than the person who might be described as acting in an offensive manner. She suggests using terms such as marginalised, belittled, stereotyped, frustrated, humiliated, threatened, patronised, silenced, intimidated, misrepresented, etc instead. It seems to me that there may be a similar issue with respect . Objective claims about people being dismissed, ignored, patronised, etc can be evaluated more easily than a claim about being disrespected.

Next.

Previous.