Search Results: "beh"

23 September 2014

Enrico Zini: pressure

Pressure I've just stumbled on this bit that seems relevant to me:

Insist on using objective criteria The final step is to use mutually agreed and objective criteria for evaluating the candidate solutions. During this stage they encourage openness and surrender to principle not pressure. http://www.wikisummaries.org/Getting_to_Yes

I find the concept of "pressure" very relevant, and I like the idea of discussions being guided by content rather than pressure. I'm exploring the idea of filing under this concept of "pressure" most of the things described in code of conducts, and I'm toying with looking at gender or race issues from the point of view of making people surrender to pressure. In that context, most code of conducts seem to be giving a partial definition of "pressure". I've been uncomfortable at DebConf this year, because the conference PG12 code of conduct would cause me trouble for talking about what lessons can Debian learn from consent culture in BDSM communities, but it would still allow situations in which people would have to yield to pressure, as long as the pressure was done avoiding the behaviours blacklisted by the CoC. Pressure could be the phrase "you are wrong" without further explanation, spoken by someone with more reputation than I have in a project. It could be someone with the time for writing ten emails a day discussing with someone with barely the time to write one. It could be someone using elaborate English discussing with someone who needs to look up every other word in a dictionary. It could be just ignoring emails from people who have issues different than mine. I like the idea of having "please do not use pressure to bring your issues forward" written somewhere, rather than spend time blacklisting all possible ways of pressuring people. I love how the Diversity Statement is elegantly getting all this where it says: We welcome contributions from everyone as long as they interact constructively with our community. However, I also find it hard not to fall back to using pressure, even just for self-preservation: I have often found myself in the situation of having the responsibility to get a job done, and not having the time or emotional resources to even read the emails I get about the subject. All my life I've seen people in such a situation yell "shut up and let me work!", and I feel a burning thirst for other kinds of role models. A CoC saying "do not use pressure" would not help me much here, but being around people who do that, learning to notice when and how they do it, and knowing that I could learn from them, that certainly would. If you can link to examples, I'd like to add them here.

19 September 2014

Paul Tagliamonte: Docker PostgreSQL Foreign Data Wrapper

For the tl;dr: Docker FDW is a thing. Star it, hack it, try it out. File bugs, be happy. If you want to see what it's like to read, there's some example SQL down below.

This post was edited on Sep 21st to add information about the DELETE and INSERT operators

The question is first, what the heck is a PostgreSQL Foreign Data Wrapper? PostgreSQL Foreign Data Wrappers are plugins that allow C libraries to provide an adaptor for PostgreSQL to talk to an external database. Some folks have used this to wrap stuff like MongoDB, which I always found to be hilarous (and an epic hack). Enter Multicorn During my time at PyGotham, I saw a talk from Wes Chow about something called Multicorn. He was showing off some really neat plugins, such as the git revision history of CPython, and parsed logfiles from some stuff over at Chartbeat. This basically blew my mind. All throughout the talk I was coming up with all sorts of things that I wanted to do -- this whole library is basically exactly what I've been dreaming about for years. I've always wanted to provide a SQL-like interface into querying API data, joining data cross-API using common crosswalks, such as using Capitol Words to query for Legislators, and use the bioguide ids to JOIN against the congress api to get their Twitter account names. My first shot was to Multicorn the new Open Civic Data API I was working on, chuckled and put it aside as a really awesome hack. Enter Docker It wasn't until tianon connected the dots for me and suggested a Docker FDW did I get really excited. Cue a few hours of hacking, and I'm proud to say -- here's Docker FDW. This lets us ask all sorts of really interesting questions out of the API, and might even help folks writing webapps avoid adding too much Docker-aware logic. Abstractions can be fun! Setting it up I'm going to assume you have a working Multicorn, PostgreSQL and Docker setup (including adding the postgres user to the docker group) So, now let's pop open a psql session. Create a database (I called mine dockerfdw, but it can be anything), and let's create some tables. Before we create the tables, we need to let PostgreSQL know where our objects are. This takes a name for the server, and the Python importable path to our FDW.

CREATE SERVER docker_containers FOREIGN DATA WRAPPER multicorn options (
    wrapper 'dockerfdw.wrappers.containers.ContainerFdw');
CREATE SERVER docker_image FOREIGN DATA WRAPPER multicorn options (
    wrapper 'dockerfdw.wrappers.images.ImageFdw');

Now that we have the server in place, we can tell PostgreSQL to create a table backed by the FDW by creating a foreign table. I won't go too much into the syntax here, but you might also note that we pass in some options - these are passed to the constructor of the FDW, letting us set stuff like the Docker host.

CREATE foreign table docker_containers (
    "id"          TEXT,
    "image"       TEXT,
    "name"        TEXT,
    "names"       TEXT[],
    "privileged"  BOOLEAN,
    "ip"          TEXT,
    "bridge"      TEXT,
    "running"     BOOLEAN,
    "pid"         INT,
    "exit_code"   INT,
    "command"     TEXT[]
) server docker_containers options (
    host 'unix:///run/docker.sock'
);
CREATE foreign table docker_images (
    "id"              TEXT,
    "architecture"    TEXT,
    "author"          TEXT,
    "comment"         TEXT,
    "parent"          TEXT,
    "tags"            TEXT[]
) server docker_image options (
    host 'unix:///run/docker.sock'
);

And, now that we have tables in place, we can try to learn something about the Docker containers. Let's start with something fun - a join from containers to images, showing all image tag names, the container names and the ip of the container (if it has one!).

SELECT docker_containers.ip, docker_containers.names, docker_images.tags
  FROM docker_containers
  RIGHT JOIN docker_images
  ON docker_containers.image=docker_images.id;

     ip                   names                               tags                   
-------------+-----------------------------+-----------------------------------------
                                              ruby:latest 
                                              paultag/vcs-mirror:latest 
                /de-openstates-to-ocd         sunlightlabs/scrapers-us-state:latest 
                /ny-openstates-to-ocd         sunlightlabs/scrapers-us-state:latest 
                /ar-openstates-to-ocd         sunlightlabs/scrapers-us-state:latest 
 172.17.0.47    /ms-openstates-to-ocd         sunlightlabs/scrapers-us-state:latest 
 172.17.0.46    /nc-openstates-to-ocd         sunlightlabs/scrapers-us-state:latest 
                /ia-openstates-to-ocd         sunlightlabs/scrapers-us-state:latest 
                /az-openstates-to-ocd         sunlightlabs/scrapers-us-state:latest 
                /oh-openstates-to-ocd         sunlightlabs/scrapers-us-state:latest 
                /va-openstates-to-ocd         sunlightlabs/scrapers-us-state:latest 
 172.17.0.41    /wa-openstates-to-ocd         sunlightlabs/scrapers-us-state:latest 
                /jovial_poincare              <none>:<none> 
                /jolly_goldstine              <none>:<none> 
                /cranky_torvalds              <none>:<none> 
                /backstabbing_wilson          <none>:<none> 
                /desperate_hoover             <none>:<none> 
                /backstabbing_ardinghelli     <none>:<none> 
                /cocky_feynman                <none>:<none> 
                                              paultag/postgres:latest 
                                              debian:testing 
                                              paultag/crank:latest 
                                              <none>:<none> 
                                              <none>:<none> 
                /stupefied_fermat             hackerschool/doorbot:latest 
                /focused_euclid               debian:unstable 
                /focused_babbage              debian:unstable 
                /clever_torvalds              debian:unstable 
                /stoic_tesla                  debian:unstable 
                /evil_torvalds                debian:unstable 
                /foo                          debian:unstable 
(31 rows)

OK, let's see if we can bring this to the next level now. I finally got around to implementing INSERT and DELETE operations, which turned out to be pretty simple to do. Check this out:

DELETE FROM docker_containers;

DELETE 1

This will do a stop + kill after a 10 second hang behind the scenes. It's actually a lot of fun to spawn up a container and terminate it from PostgreSQL.

INSERT INTO docker_containers (name, image) VALUES ('hello', 'debian:unstable') RETURNING id;

                                id                                
------------------------------------------------------------------
 0a903dcf5ae10ee1923064e25ab0f46e0debd513f54860beb44b2a187643ff05
INSERT 0 1
(1 row)

Spawning containers works too - this is still very immature and not super practical, but I figure while I'm showing off, I might as well go all the way.

SELECT ip FROM docker_containers WHERE id='0a903dcf5ae10ee1923064e25ab0f46e0debd513f54860beb44b2a187643ff05';

     ip      
-------------
 172.17.0.12
(1 row)

Success! This is just a taste of what's to come, so please feel free to hack on Docker FDW, tweet me @paultag, file bugs / feature requests. It's currently a bit of a hack, and it's something that I think has long-term potential after some work goes into making sure that this is a rock solid interface to the Docker API.

16 September 2014

Matthew Garrett: ACPI, kernels and contracts with firmware

ACPI is a complicated specification - the latest version is 980 pages long. But that's because it's trying to define something complicated: an entire interface for abstracting away hardware details and making it easier for an unmodified OS to boot diverse platforms.

Inevitably, though, it can't define the full behaviour of an ACPI system. It doesn't explicitly state what should happen if you violate the spec, for instance. Obviously, in a just and fair world, no systems would violate the spec. But in the grim meathook future that we actually inhabit, systems do. We lack the technology to go back in time and retroactively prevent this, and so we're forced to deal with making these systems work.

This ends up being a pain in the neck in the x86 world, but it could be much worse. Way back in 2008 I wrote something about why the Linux kernel reports itself to firmware as "Windows" but refuses to identify itself as Linux. The short version is that "Linux" doesn't actually identify the behaviour of the kernel in a meaningful way. "Linux" doesn't tell you whether the kernel can deal with buffers being passed when the spec says it should be a package. "Linux" doesn't tell you whether the OS knows how to deal with an HPET. "Linux" doesn't tell you whether the OS can reinitialise graphics hardware.

Back then I was writing from the perspective of the firmware changing its behaviour in response to the OS, but it turns out that it's also relevant from the perspective of the OS changing its behaviour in response to the firmware. Windows 8 handles backlights differently to older versions. Firmware that's intended to support Windows 8 may expect this behaviour. If the OS tells the firmware that it's compatible with Windows 8, the OS has to behave compatibly with Windows 8.

In essence, if the firmware asks for Windows 8 support and the OS says yes, the OS is forming a contract with the firmware that it will behave in a specific way. If Windows 8 allows certain spec violations, the OS must permit those violations. If Windows 8 makes certain ACPI calls in a certain order, the OS must make those calls in the same order. Any firmware bug that is triggered by the OS not behaving identically to Windows 8 must be dealt with by modifying the OS to behave like Windows 8.

This sounds horrifying, but it's actually important. The existence of well-defined[1] OS behaviours means that the industry has something to target. Vendors test their hardware against Windows, and because Windows has consistent behaviour within a version[2] the vendors know that their machines won't suddenly stop working after an update. Linux benefits from this because we know that we can make hardware work as long as we're compatible with the Windows behaviour.

That's fine for x86. But remember when I said it could be worse? What if there were a platform that Microsoft weren't targeting? A platform where Linux was the dominant OS? A platform where vendors all test their hardware against Linux and expect it to have a consistent ACPI implementation?

Our even grimmer meathook future welcomes ARM to the ACPI world.

Software development is hard, and firmware development is software development with worse compilers. Firmware is inevitably going to rely on undefined behaviour. It's going to make assumptions about ordering. It's going to mishandle some cases. And it's the operating system's job to handle that. On x86 we know that systems are tested against Windows, and so we simply implement that behaviour. On ARM, we don't have that convenient reference. We are the reference. And that means that systems will end up accidentally depending on Linux-specific behaviour. Which means that if we ever change that behaviour, those systems will break.

So far we've resisted calls for Linux to provide a contract to the firmware in the way that Windows does, simply because there's been no need to - we can just implement the same contract as Windows. How are we going to manage this on ARM? The worst case scenario is that a system is tested against, say, Linux 3.19 and works fine. We make a change in 3.21 that breaks this system, but nobody notices at the time. Another system is tested against 3.21 and works fine. A few months later somebody finally notices that 3.21 broke their system and the change gets reverted, but oh no! Reverting it breaks the other system. What do we do now? The systems aren't telling us which behaviour they expect, so we're left with the prospect of adding machine-specific quirks. This isn't scalable.

Supporting ACPI on ARM means developing a sense of discipline around ACPI development that we simply haven't had so far. If we want to avoid breaking systems we have two options:

1) Commit to never modifying the ACPI behaviour of Linux.
2) Exposing an interface that indicates which well-defined ACPI behaviour a specific kernel implements, and bumping that whenever an incompatible change is made. Backward compatibility paths will be required if firmware only supports an older interface.

(1) is unlikely to be practical, but (2) isn't a great deal easier. Somebody is going to need to take responsibility for tracking ACPI behaviour and incrementing the exported interface whenever it changes, and we need to know who that's going to be before any of these systems start shipping. The alternative is a sea of ARM devices that only run specific kernel versions, which is exactly the scenario that ACPI was supposed to be fixing.

[1] Defined by implementation, not defined by specification
[2] Windows may change behaviour between versions, but always adds a new _OSI string when it does so. It can then modify its behaviour depending on whether the firmware knows about later versions of Windows.

comments

13 September 2014

Laura Arjona: Disabling comments in the blog

I m getting more spam than the amount that I can stand in this blog. Comments are moderated, so the public is not suffering that, only me. From time to time I go to my dashboard and clean the spam. I m afraid that I delete some legit comment in these spam-cleaning-fevers, or, more probably, that a legit comment waits in the queue for several days (weeks?), just because I m lazy to deal with spam and let days pass by (until the fever comes). I think I m going to follow the wisdom of Bradley M. Kuhn and link to a pump.io note for comments on my blog posts (disabling them here in WordPress.com). I usually post a notice when I write something in my blog, so the only task is to update the blog post with the pump.io URL of the thread for comments. While WordPress.com allows to write comments quickly, without need of an account (you write just a name and an email, and the comment), in pump you need to have an account and sign in to comment. That looks as a bad thing, a barrier for people to participate. But of course, it stops spam :) After thinking about it a bit, pump.io it s a federated network, you can choose the pump server that they want, you can create a fake account, you don t need to provide personal information and it s another way to promote one of the social networks where I live. Other systems link to facebook, twitter, or other places, and nobody complains! Even when those services don t have any of the advantages of being in a federated free-software powered social network :) If anybody don t want to use pump.io but wants to comment, other ways to reach me or the related blog post are:

Comment in the GNUSocial fediverse: the post announcing the thread for each blog post will be propagated to my quitter.se account too.
While I m still using Twitter, they can comment on the corresponding tweet, but beware that I m seriously thinking about closing my account there, since I rarely use it and don t like the platform.
Drop me an email, I can post the comment on behalf of that person (if you want your comment to be anonymous , please state it in the email).

So now it s decided, and this is the first post of this new experiment. This text is posted in pump.io too, and you can comment there :)
Filed under: My experiences and opinion, Tools Tagged: Blog, English, federation, pump.io, social networks, Wordpress

11 September 2014

Sylvestre Ledru: Rebuild of Debian using Clang 3.5.0

Clang 3.5.0 has just been released. A new rebuild has been done highlight the progress to get Debian built with clang. tl;dr: Great progress. We decreased from 9.5% to 5.7% of failures. Full results are available on http://clang.debian.net At time of the rebuild with 3.4.2, we had 2040 packages failing to build with clang. With 3.5.0, this dropped to 1261 packages. Fixes With Arthur Marble and Alexander Ovchinnikov, both GSoC students, we worked on various ways to decrease the number of errors. Upstream fixes First, the most obvious way, we fixed programming bugs/mistakes in upstream sources. Basically, we took categories of failure and fixed issues one after the other. We started with simple bugs like 'Wrong main declaration', 'non-void function should return a value' or 'Void function should not return a value'.

They are trivial to fix. We continued with harder fixes like ' Undefined reference' or 'Variable length array for a non POD (plain old data) element'.

So, besides these one, we worked on:

In total, we reported 295 bugs with patches. 85 of them have been fixed (meaning that the Debian maintainer uploaded a new version with the fix).

In parallel, I think that the switch by FreeBSD and Mac OS X to Clang also helped to fix various issues by upstreams. Hacking in clang As a parallel approach, we started to implement a suggestion from Linus Torvalds and a few others. Instead of trying to fix all upstream, where we can, we tried to update clang to improve the gcc compatibility.

gcc has many flags to disable or enable optimizations. Some of them are legacy, others have no sense in clang, etc. Instead of failing in clang with an error, we create a new category of warnings (showing optimization flag '%0' is not supported) and moved all relevant flags into it. Some examples, r212805, r213365, r214906 or r214907

We also updated clang to silent some useless arguments like -finput_charset=UTF-8 (r212110), clang being UTF-8 compliant.

Finally, we worked on the forwarding of linker flags. Clang and gcc have a very different behavior: when gcc does not know an argument, it is going to forward the argument to the linker. Clang, in this case, is going to reject the argument and fail with an error. In clang, we have to explicitly declare which arguments are going to be transfer to the linker. Of course, the correct way to pass arguments to the linker is to use -Xlinker or -Wl but the Debian rebuild proved that these shortcuts are used. Two of these arguments are now forwarded:

-z keyword - r213198
-u Force symbol to be entered in the output file as an undefined symbol - r211756. This one fixed most of the haskell build failures. It fixed the most common issue that we had (701 occurrences but this does not mean that all these packages build fine now, some haskell-based package are failing later in the process)

New errors Just like in other releases, new warnings are added in clang. With (bad) usage of -Werror by upstream software, this causes new build failures:

Absolute value error - 6 occurences
Tautological pointer comparison - 10 occurences

I also took the opportunity to add some further categorizations in the list of errors. Some examples:

Next steps The Debile project being close to ready with Cl ment Schreiner's GSoC, we will now have an automatic and transparent way to rebuild packages using clang. Conclusion As stated, we can see a huge drop in term of number of failures over time:

Hopefully, Clang getting better and better, more and more projects adopting it as the default compiler or as a base for plugin/extension developments, this percentage will continue to decrease.
Having some kind of release goal with clang for Jessie+1 can now be considered as potentially reachable. Want to help? There are several things which can be done to help:

Point me common error patterns in the Not categorized list of errors to create new categories
Report and fix packages
As an upstream, integrate clang as part of your continuous integration system
Hack on cqa-scanlogs, the error detection tool to detect error patterns (example: Undetected error). This tool is used also for the regular rebuilds of the archive.
Improve clang.debian.net website

Acknowledgments Thanks to David Suarez for the rebuilds of the archive, Arthur Marble and Alexander Ovchinnikov for their GSoC works and Nicolas S velin-Radiguet for the few fixes.

Original post blogged on b2evolution.

10 September 2014

Petter Reinholdtsen: Good bye subkeys.pgp.net, welcome pool.sks-keyservers.net

Yesterday, I had the pleasure of attending a talk with the Norwegian Unix User Group about the OpenPGP keyserver pool sks-keyservers.net, and was very happy to learn that there is a large set of publicly available key servers to use when looking for peoples public key. So far I have used subkeys.pgp.net, and some times wwwkeys.nl.pgp.net when the former were misbehaving, but those days are ended. The servers I have used up until yesterday have been slow and some times unavailable. I hope those problems are gone now. Behind the round robin DNS entry of the sks-keyservers.net service there is a pool of more than 100 keyservers which are checked every day to ensure they are well connected and up to date. It must be better than what I have used so far. :) Yesterdays speaker told me that the service is the default keyserver provided by the default configuration in GnuPG, but this do not seem to be used in Debian. Perhaps it should? Anyway, I've updated my ~/.gnupg/options file to now include this line:

keyserver pool.sks-keyservers.net

With GnuPG version 2 one can also locate the keyserver using SRV entries in DNS. Just for fun, I did just that at work, so now every user of GnuPG at the University of Oslo should find a OpenGPG keyserver automatically should their need it:

% host -t srv _pgpkey-http._tcp.uio.no
_pgpkey-http._tcp.uio.no has SRV record 0 100 11371 pool.sks-keyservers.net.
%

Now if only the HKP lookup protocol supported finding signature paths, I would be very happy. It can look up a given key or search for a user ID, but I normally do not want that, but to find a trust path from my key to another key. Given a user ID or key ID, I would like to find (and download) the keys representing a signature path from my key to the key in question, to be able to get a trust path between the two keys. This is as far as I can tell not possible today. Perhaps something for a future version of the protocol?

7 September 2014

Craig Small: How not to get Galaxy Tab into Safe Mode

For weeks my Galaxy Tab 10.1 has reasonably consistently gone into safe mode. Not booting into it but I d use it fine then put it away and next time I looked at it, Safe Mode was there. It wasn t every time, but averaged to be about every second time. So the first thing was a bit of googling to see what this Safe Mode was. Most of the suggestions were around how to put it into safe mode during the boot process but my problem was opposite; it wasn t during booting and I wanted something to stop safe mode, not put the device into it. The closest I got to it was there was some misbehaving program that kicked the thing into safe mode. The problem was, I checked several times and there were no running programs. I really did start to worry I had a hardware fault or something wrong deep within the OS. When you have problems in IT, you re usually asked What s new? What s changed? . The answer is generally Nothing which gets a switch No really, what did change . The only answer I could come up with was a hardware keyboard. This slim aluminum uses bluetooth to communicate to the tablet and clips onto the front screen to protect it when not in use. Could this be the change I was looking for? The clue was that sometimes when you boot Android, if you hold down some keys it boots into safemode. It seems that holding down some combination of keys (volume up/down, power) puts into safe mode. The keyboard can clip onto the tablet in two ways, one long edge has some raised edges while one doesn t. If the raised edge was connected to the same side as the buttons, I d get safe mode sometimes as the edge pushed some of those buttons. More importantly, putting the raised edge on the side with no buttons meant no more safe mode. Not really a software or electrical fault, more one of just mechanics.

2 September 2014

Antonio Terceiro: DebConf 14: Community, Debian CI, Ruby, Redmine, and Noosfero

This time, for personal reasons I wasn t able to attend the full DebConf, which started on the Saturday August 22nd. I arrived at Portland on the Tuesday the 26th by noon, at the 4th of the conference. Even though I would like to arrive earlier, the loss was alleviated by the work of the amazing DebConf video team. I was able to follow remotely most of the sessions I would like to attend if I were there already. As I will say to everyone, DebConf is for sure the best conference I have ever attended. The technical and philosophical discussions that take place in talks, BoF sessions or even unplanned ad-hoc gathering are deep. The hacking moments where you have a chance to pair with fellow developers, with whom you usually only have contact remotely via IRC or email, are precious. That is all great. But definitively, catching up with old friends, and making new ones, is what makes DebConf so special. Your old friends are your old friends, and meeting them again after so much time is always a pleasure. New friendships will already start with a powerful bond, which is being part of the Debian community. Being only 4 hours behind my home time zone, jetlag wasn t a big problem during the day. However, I was waking up too early in the morning and consequently getting tired very early at night, so I mostly didn t go out after hacklabs were closed at 10PM. Despite all of the discussion, being in the audience for several talks, other social interactions and whatnot, during this DebConf I have managed to do quite some useful work. debci and the Debian Continuous Integration project I gave a talk where I discussed past, present, and future of debci and the Debian Continuous Integration project. The slides are available, as well as the video recording. One thing I want you to take away is that there is a difference between debci and the Debian Continuous Integration project:

debci is a platform for Continuous Integration specifically tailored for the Debian repository and similar ones. If you work on a Debian derivative, or otherwise provides Debian packages in a repository, you can use debci to run tests for your stuff.
- a (very) few thinks in debci, though, are currently hardcoded for Debian. Other projects using it would be a nice and needed peer pressure to get rid of those.
Debian Continuous Integration is Debian s instance of debci, which currently runs tests for all packages in the unstable distribution that provide autopkgtest support. It will be expanded in the future to run tests on other suites and architectures.

A few days before DebConf, C dric Boutillier managed to extract gem2deb-test-runner from gem2deb, so that autopkgtest tests can be run against any Ruby package that has tests by running gem2deb-test-runner --autopkgtest. gem2deb-test-runner will do the right thing, make sure that the tests don t use code from the source package, but instead run them against the installed package. Then, right after my talk I was glad to discover that the Perl team is also working on a similar tool that will automate running tests for their packages against the installed package. We agreed that they will send me a whitelist of packages in which we could just call that tool and have it do The Right Thing. We might be talking here about getting autopkgtest support (and consequentially continuous integration) for free for almost ~~2000~~ 4000 packages. The missing bits for this to happen are:

making debci use a whitelist of packages that, while not having the appropriate Testsuite: autopkgtest field in the Sources file, could be assumed to have autopkgtest support by calling the right tool (gem2deb-test-runner for Ruby, or the Perl team s new tool for Perl packages).
make the autopkgtest test runner assume a corresponding, implicit, debian/tests/control when it not exists in those packages

During a few days I have mentored Lucas Kanashiro, who also attended DebConf, on writing a patch to add support for email notifications in debci so maintainers can be pro-actively notified of status changes (pass/fail, fail/pass) in their packages. I have also started hacking on the support for distributed workers, based on the initial work by Martin Pitt:

updated the amqp branch against the code in the master branch.
added a debci enqueue command that can be used to force test runs for packages given on the command line.
I sent a patch for librabbitmq that adds support for limiting the number of messages the server will send to a connected client. With this patch applied, the debci workers were modified to request being sent only 1 message at a time, so late workers will start to receive packages to process as soon as they are up. Without this, a single connected worker would receive all messages right away, while a second worker that comes up 1 second later would sit idle until new packages are queued for testing.

Ruby I had some discusion with Christian about making Rubygems install to $HOME by default when the user is not root. We discussed a few implementation options, and while I don t have a solution yet, we have a better understanding of the potential pitfalls. The Ruby BoF session on Friday produced a few interesting discussions. Some take away point include, but are not limited to:

Since the last DebConf, we were able to remove all obsolete Ruby interpreters, and now only have Ruby 2.1 in unstable. Ruby 2.1 will be the default version in Debian 8 (jessie).
There is user interest is being able to run the interpreter from Debian, but install everything else from Rubygems.
We are lacking in all the documentation-related goals for jessie that were proposed at the previous DebConf.

Redmine I was able to make Redmine work with the Rails 4 stack we currently have in unstable/testing. This required using a snapshot of the still unreleased version 3.0 based on the rails-4.1 branch in the upstream Subversion repository as source. I am a little nervous about using a upstream snapshot, though. According to the "roadmap of the project ":http://www.redmine.org/projects/redmine/roadmap the only purpose of the 3.0 release will be to upgrade to Rails 4, but before that happens there should be a 2.6.0 release that is also not released yet. 3.0 should be equivalent to that 2.6.0 version both feature-wise and, specially, bug-wise. The only problem is that we don t know what that 2.6.0 looks like yet. According to the roadmap it seems there is not much left in term of features for 2.6.0, though. The updated package is not in unstable yet, but will be soon. It needs more testing, and a good update to the documentation. Those interested in helping to test Redmine on jessie before the freeze please get in touch with me. Noosfero I gave a lighting talk on Noosfero, a platform for social networking websites I am upstream for. It is a Rails appplication licensed under the AGPLv3, and there are packages for wheezy. You can checkout the slides I used. Video recording is not available yet, but should be soon. That s it. I am looking forward to DebConf 15 at Heidelberg. :-)

30 August 2014

John Goerzen: 2AM to Seattle

Monday morning, 1:45AM. Laura and I walk into the boys room. We turn on the light. Nothing happens. (They re sound sleepers.) Boys, it s time to get up to go get on the train! Four eyes pop open. Yay! Oh I m so excited! And then, Meow! (They enjoy playing with their stuffed cats that Laura got them for Christmas.) Before long, it was out the door to the train station. We even had time to stop at a donut shop along the way. We climbed into our family bedroom (a sleeping car room on Amtrak specifically designed for families of four), and as the train started to move, the excitement of what was going on crept in. Yes, it s 2:42AM, but these are two happy boys:

Jacob and Oliver love trains, and this was the beginning of a 3-day train trip from Newton to Seattle that would take us through Kansas, Colorado, the Rocky Mountains of New Mexico, Arizona, Los Angeles, up the California coast, through the Cascades, and on to Seattle. Whew! Here we are later that morning before breakfast:

Here s our train at a station stop in La Junta, CO:

And at the beautiful small mountain town of Raton, NM:

Some of the passing scenery in New Mexico:

Through it all, we found many things to pass the time. I don t think anybody was bored. I took the boys exploring the train several times we d walk from one end to the other and see what all was there. There was always the dining car for our meals, the lounge car for watching the passing scenery, and on the Coast Starlight, the Pacific Parlor Car. Here we are getting ready for breakfast one morning.

Getting to select meals and order in the train restaurant was a big deal for the boys.

Laura brought one of her origami books, which even managed to pull the boys away from the passing scenery in the lounge car for quite some time.

Origami is serious business:

They had some fun wrapping themselves around my feet and challenging me to move. And were delighted when I could move even though they were trying to weight me down!

Several games of Uno were played, but even those sometimes couldn t compete with the passing scenery:

The Coast Starlight features the Pacific Parlor Car, which was built over 50 years ago for the Santa Fe Hi-Level trains. They ve been updated; the upper level is a lounge and small restaurant, and the lower level has been turned into a small theater. They show movies in there twice a day, but most of the time, the place is empty. A great place to go with little boys to run around and play games.

The boys and I sort of invented a new game: roadrunner and coyote, loosely based on the old Looney Tunes cartoons. Jacob and Oliver would be roadrunners, running around and yelling MEEP MEEP! Meanwhile, I was the coyote, who would try to catch them even briefly succeeding sometimes but ultimately fail in some hilarious way. It burned a lot of energy. And, of course, the parlor car was good for scenery-watching too:

We were right along the Pacific Ocean for several hours sometimes there would be a highway or a town between us and the beach, but usually there was nothing at all between us and the coast. It was beautiful to watch the jagged coastline go by, to gaze out onto the ocean, watching the birds apparently so beautiful that I didn t even think to take some photos. Laura s parents live in California, and took a connecting train. I had arranged for them to have a sleeping car room near ours, so for the last day of the trip, we had a group of 6. Here are the boys with their grandparents at lunch Wednesday:

We stepped off the train in Seattle into beautiful King Street Station.

Our first day in Seattle was a quiet day of not too much. Laura s relatives live near Lake Washington, so we went out there to play. The boys enjoyed gathering black rocks along the shore.

We went blackberry picking after that filled up buckets for a cobbler. The next day, we rode the Seattle Monorail. The boys have been talking about this for months a kind of train they ve never been on. That was the biggest thing in their minds that they were waiting for. They got to ride in the very front, by the operator.

Nice view from up there.

We walked through the Pike Market I hadn t been in such a large and crowded place like that since I was in Guadalajara:

At the Seattle Aquarium, we all had a great time checking out all the exhibits. The please touch one was a particular hit.

Walking underneath the salmon tank was fun too. We spent a couple of days doing things closer to downtown. Laura s cousin works at MOHAI, the Museum of History and Industry, so we spent a morning there. The boys particularly enjoyed the old periscope mounted to the top of the building, and the exhibit on chocolate (of course!)

They love any kind of transportation, so of course we had to get a ride on the Seattle Streetcar that comes by MOHAI.

All weekend long, we had been noticing the seaplanes taking off from Lake Washington and Lake Union (near MOHAI). So finally I decided to investigate, and one morning while Laura was doing things with her cousin, the boys and I took a short seaplane ride from one lake to another, and then rode every method of transportation we could except for ferries (we did that the next day). Here is our Kenmore Air plane:

The view of Lake Washington from 1000 feet was beautiful:

I think we got a better view than the Space Needle, and it probably cost about the same anyhow.

After splashdown, we took the streetcar to a place where we could eat lunch right by the monorail tracks. Then we rode the monorail again. Then we caught a train (it went underground a bit so it was a subway to them!) and rode it a few blocks. There is even scenery underground, it seems.

We rode a bus back, and saved one last adventure for the next day: a ferry to Bainbridge Island.

Laura and I even got some time to ourselves to go have lunch at an amazing Greek restaurant to celebrate a year since we got engaged. It s amazing to think that, by now, it s only a few months until our wedding anniversary too! There are many special memories of the weekend I could mention visiting with Laura s family, watching the boys play with her uncle s pipe organ (it s in his house!), watching the boys play with their grandparents, having all six of us on the train for a day, flying paper airplanes off the balcony, enjoying the cool breeze on the ferry and the beautiful mountains behind the lake. One of my favorites is waking up to high-pitched Meow? Meow meow meow! Wake up, brother! sorts of sounds. There was so much cat-play on the trip, and it was cute to hear. I have the feeling we won t hear things like that much more. So many times on the trip I heard, Oh dad, I am so excited! I never get tired of hearing that. And, of course, I was excited, too.

24 August 2014

Noah Meyerhans: Debconf by train

Today is the first time I've taken an interstate train trip in something like 15 years. A few things about the trip were pleasantly surprising. Most of these will come as no surprise:

Less time wasted in security theater at the station prior to departure.
On-time departure
More comfortable seats than a plane or bus.
Quiet.
Permissive free wifi

Wifi was the biggest surprise. Not that it existed, since we're living in the future and wifi is expected everywhere. It's IPv4 only and stuck behind a NAT, which isn't a big surprise, but it is reasonably open. There isn't any port filtering of non-web TCP ports, and even non-TCP protocols are allowed out. Even my aiccu IPv6 tunnel worked fine from the train, although I did experience some weird behavior with it. I haven't used aiccu much in quite a while, since I have a native IPv6 connection at home, but it can be convenient while travelling. I'm still trying to figure out happened today, though. The first symptoms were that, although I could ping IPv6 hosts, I could not actually log in via IMAP or ssh. Tcpdump showed all the standard symptoms of a PMTU blackhole. Small packets flow fine, large ones are dropped. The interface MTU is set to 1280, which is the minimum MTU for IPv6 and any path on the internet is expected to handle packets of at least that size. Experimentation via ping6 reveals that the largest payload size I can successfully exchange with a peer is 820 bytes. Add 8 bytes for the ICMPv6 header for 828 bytes of payload, plus 40 bytes for the IPv6 header gives an 868 byte packet, which is well under what should be the MTU for this path. I've worked around this problem with an ip6tables rule to rewrite the MSS on outgoing SYN packets to 760 bytes, which should leave 40 for the IPv6 header and 20 for any extension headers:

sudo ip6tables -t mangle -A OUTPUT -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --set-mss 760

It is working well and will allow me to publish this from the train, which I'd otherwise have been unable to do. But... weird.

Vincent Sanders: Without craftsmanship, inspiration is a mere reed shaken in the wind.

While I imagine Johannes Brahms was referring to music I think the sentiment applies to other endeavours just as well. The trap of believing an idea is worth something without an implementation occurs all too often, however this is not such an unhappy tale.

Lars Wirzenius, Steve McIntyre and myself were chatting a few weeks ago about several of the ongoing Debian discussions. As is often the case these discussions had devolved into somewhat unproductive noise and yet amongst all this was a voice of reason in Russ Allbery.

Lars decided that would take the opportunity of the upcoming opportunity of Debconf 14 to say thank you to Russ for his work. It was decided that a plaque would be a nice gift and I volunteered to do the physical manufacture. Lars came up with the idea of a DEBCON scale similar to the DEFCON scale and got some text together with an initial design idea.

CAD drawing of cut paths in clear acrylic

I took the initial design and as is often the case what is practically possible forced several changes. The prototype was a steep learning curve on using the Cambridge makespace laser cutter to create all the separate pieces.

The construction is pretty simple and consisted of three layers of transparent acrylic plastic. The base layer is a single piece of plastic with the correct outline. The next layer has the DEBCON title, the Debian swirl and level numbers. The top layer has the text engraved in its back surface giving the impression the text floats above the layer behind it.

For the prototype I attempted to glue the pieces together. This was a complete disaster and required discarding the entire piece and starting again with new materials.

The final version with stand ready to be presented

For the second version I used four small nylon bolts to hold the sandwich of layers together which worked very well.

Presentation of plaque photo by Aigars Mahinovs

Yesterday at the Debconf 14 opening Steve McIntyre presented it to Russ and I think he was pleased, certainly he was surprised (photo from Aigars Mahinovs).

The design files are available from my design git repo, though why anyone would want to reproduce it I have no idea ;-)

23 August 2014

Steve Kemp: Updating Debian Administration, the code

So I previously talked about the setup behind Debian Administration, and my complaints about the slownes. The previous post talked about the logical setup, and the hardware. This post talks about the more interesting thing. The code. The code behind the site was originally written by Denny De La Haye. I found it and reworked it a lot, most obviously adding structure and test cases. Once I did that the early version of the site was born. Later my version became the official version, as when Denny setup Police State UK he used my codebase rather than his. So the code huh? Well as you might expect it is written in Perl. There used to be this layout:

]
yawns/cgi-bin/index.cgi
yawns/cgi-bin/Pages.pl
yawns/lib/...
yawns/htdocs/

Almost every request would hit the index.cgi script, which would parse the request and return the appropriate output via the standard CGI interface. How did it know what you wanted? Well sometimes there would be a paramater set which would be looked up in a dispatch-table:

/cgi-bin/index.cgi?article=40         - Show article 40
/cgi-bin/index.cgi?view_user=Steve    - Show the user Steve
/cgi-bin/index.cgi?recent_comments=10 - Show the most recent comments.

Over time the code became hard to update because there was no consistency, and over time the site became slow because this is not a quick setup. Spiders, bots, and just average users would cause a lot of perl processes to run. So? What did I do? I moved the thing to using FastCGI, which avoids the cost of forking Perl and loading (100k+) the code. Unfortunately this required a bit of work because all the parameter handling was messy and caused issues if I just renamed index.cgi -> index.fcgi. The most obvious solution was to use one parameter, globally, to specify the requested mode of operation. Hang on? One parameter to control the page requested? A persistant environment? What does that remind me of? Yes. CGI::Application. I started small, and pulled some of the code out of index.cgi + Pages.pl, and over into a dedicated CGI::Application class:

Application::Feeds - Called via /cgi-bin/f.fcgi.
Application::Ajax - Called via /cgi-bin/a.fcgi.

So now every part of the site that is called by Ajax has one persistent handler, and every part of the site which returns RSS feeds has another. I had some fun setting up the sessions to match those created by the old stuff, but I quickly made it work, as this example shows:

Viewing recent comments:
- https://www.debian-administration.org/recent/comments
- http://debian-administration.org/cgi-bin/f.fcgi?mode=recent_comments

The final job was the biggest, moving all the other (non-feed, non-ajax) modes over to a similar CGI::Application structure. There were 53 modes that had to be ported, and I did them methodically, first porting all the Poll-related requests, then all the article-releated ones, & etc. I think I did about 15 a day for three days. Then the rest in a sudden rush. In conclusion the code is now fast because we don't use CGI, and instead use FastCGI. This allowed minor changes to be carried out, such as compiling the HTML::Template templates which determine the look and feel, etc. Those things don't make sense in the CGI environment, but with persistence they are essentially free. The site got a little more of a speed boost when I updated DNS, and a lot more when I blacklisted a bunch of IP-space. As I was wrapping this up I realized that the code had accidentally become closed - because the old repository no longer exists. That is not deliberate, or intentional, and will be rectified soon. The site would never have been started if I'd not seen Dennys original project, and although I don't think others would use the code it should be possible. I remember at the time I was searching for things like "Perl CMS" and finding Slashcode, and Scoop, which I knew were too heavyweight for my little toy blog. In conclusion Debian Administration website is 10 years old now. It might not have changed the world, it might have become less relevant, but I'm glad I tried, and I'm glad there were years when it really was the best place to be. These days there are HowtoForges, blogs, spam posts titled "How to install SSH on Trusty", "How to install SSH on Wheezy", "How to install SSH on Precise", and all that. No shortage of content, just finding the good from the bad is the challenge. Me? The single best resource I read these days is probably LWN.net. Starting to ramble now. Go look at my quick hack for remote command execution https://github.com/skx/nanoexec ?

22 August 2014

Russell Coker: Men Commenting on Women s Issues

A lecture at LCA 2011 which included some inappropriate slides was followed by long discussions on mailing lists. In February 2011 I wrote a blog post debunking some of the bogus arguments in two lists [1]. One of the noteworthy incidents in the mailing list discussion concerned Ted Ts o (an influential member of the Linux community) debating the definition of rape. My main point on that issue in Feb 2011 was that it s insensitive to needlessly debate the statistics. Recently Valerie Aurora wrote about another aspect of this on The Ada Initiative blog [2] and on her personal blog. Some of her significant points are that conference harassment doesn t end when the conference ends (it can continue on mailing lists etc), that good people shouldn t do nothing when bad things happen, and that free speech doesn t mean freedom from consequences or the freedom to use private resources (such as conference mailing lists) without restriction. Craig Sanders wrote a very misguided post about the Ted Ts o situation [3]. One of the many things wrong with his post is his statement I m particularly disgusted by the men who intervene way too early without an explicit invitation or request for help or a clear need such as an immediate threat of violence in womens issues . I believe that as a general rule when any group of people are involved in causing a problem they should be involved in fixing it. So when we have problems that are broadly based around men treating women badly the prime responsibility should be upon men to fix them. It seems very clear that no matter what scope is chosen for fixing the problems (whether it be lobbying for new legislation, sociological research, blogging, or directly discussing issues with people to change their attitudes) women are doing considerably more than half the work. I believe that this is an indication that overall men are failing. Asking for Help I don t believe that members of minority groups should have to ask for help. Asking isn t easy, having someone spontaneously offer help because it s the right thing to do can be a lot easier to accept psychologically than having to beg for help. There is a book named Women Don t Ask which has a page on the geek feminism Wiki [4]. I think the fact that so many women relate to a book named Women Don t Ask is an indication that we shouldn t expect women to ask directly, particularly in times of stress. The Wiki page notes a criticism of the book that some specific requests are framed as complaining , so I think we should consider a complaint from a woman as a direct request to do something. The geek feminism blog has an article titled How To Exclude Women Without Really Trying which covers many aspects of one incident [5]. Near the end of the article is a direct call for men to be involved in dealing with such problems. The geek feminism Wiki has a page on Allies which includes Even a blog post helps [6]. It seems clear from public web sites run by women that women really want men to be involved. Finally when I get blog comments and private email from women who thank me for my posts I take it as an implied request to do more of the same. One thing that we really don t want is to have men wait and do nothing until there is an immediate threat of violence. There are two massive problems with that plan, one is that being saved from a violent situation isn t a fun experience, the other is that an immediate threat of violence is most likely to happen when there is no-one around to intervene. Men Don t Listen to Women Rebecca Solnit wrote an article about being ignored by men titled Men Explain Things to Me [7]. When discussing women s issues the term Mansplaining is often used for that sort of thing, the geek feminism Wiki has some background [8]. It seems obvious that the men who have the greatest need to be taught some things related to women s issues are the ones who are least likely to listen to women. This implies that other men have to teach them. Craig says that women need space to discover and practice their own strength and their own voices . I think that the best way to achieve that goal is to listen when women speak. Of course that doesn t preclude speaking as well, just listen first, listen carefully, and listen more than you speak. Craig claims that when men like me and Matthew Garrett comment on such issues we are making women s spaces more comfortable, more palatable, for men . From all the discussion on this it seems quite obvious that what would make things more comfortable for men would be for the issue to never be discussed at all. It seems to me that two of the ways of making such discussions uncomfortable for most men are to discuss sexual assault and to discuss what should be done when you have a friend who treats women in a way that you don t like. Matthew has covered both of those so it seems that he s doing a good job of making men uncomfortable I think that this is a good thing, a discussion that is comfortable and palatable for the people in power is not going to be any good for the people who aren t in power. The Voting Aspect It seems to me that when certain issues are discussed we have a social process that is some form of vote. If one person complains then they are portrayed as crazy. When other people agree with the complaint then their comments are marginalised to try and preserve the narrative of one crazy person. It seems that in the case of the discussion about Rape Apology and LCA2011 most men who comment regard it as one person (either Valeria Aurora or Matthew Garrett) causing a dispute. There is even some commentary which references my blog post about Rape Apology [9] but somehow manages to ignore me when it comes to counting more than one person agreeing with Valerie. For reference David Zanetti was the first person to use the term apologist for rapists in connection with the LCA 2011 discussion [10]. So we have a count of at least three men already. These same patterns always happen so making a comment in support makes a difference. It doesn t have to be insightful, long, or well written, merely I agree and a link to a web page will help. Note that a blog post is much better than a comment in this regard, comments are much like conversation while a blog post is a stronger commitment to a position. I don t believe that the majority is necessarily correct. But an opinion which is supported by too small a minority isn t going to be considered much by most people. The Cost of Commenting The Internet is a hostile environment, when you comment on a contentious issue there will be people who demonstrate their disagreement in uncivilised and even criminal ways. S. E. Smith wrote an informative post for Tiger Beatdown about the terrorism that feminist bloggers face [11]. I believe that men face fewer threats than women when they write about such things and the threats are less credible. I don t believe that any of the men who have threatened me have the ability to carry out their threats but I expect that many women who receive such threats will consider them to be credible. The difference in the frequency and nature of the terrorism (and there is no other word for what S. E. Smith describes) experienced by men and women gives a vastly different cost to commenting. So when men fail to address issues related to the behavior of other men that isn t helping women in any way. It s imposing a significant cost on women for covering issues which could be addressed by men for minimal cost. It s interesting to note that there are men who consider themselves to be brave because they write things which will cause women to criticise them or even accuse them of misogyny. I think that the women who write about such issues even though they will receive threats of significant violence are the brave ones. Not Being Patronising Craig raises the issue of not being patronising, which is of course very important. I think that the first thing to do to avoid being perceived as patronising in a blog post is to cite adequate references. I ve spent a lot of time reading what women have written about such issues and cited the articles that seem most useful in describing the issues. I m sure that some women will disagree with my choice of references and some will disagree with some of my conclusions, but I think that most women will appreciate that I read what women write (it seems that most men don t). It seems to me that a significant part of feminism is about women not having men tell them what to do. So when men offer advice on how to go about feminist advocacy it s likely to be taken badly. It s not just that women don t want advice from men, but that advice from men is usually wrong. There are patterns in communication which mean that the effective strategies for women communicating with men are different from the effective strategies for men communicating with men (see my previous section on men not listening to women). Also there s a common trend of men offering simplistic advice on how to solve problems, one thing to keep in mind is that any problem which affects many people and is easy to solve has probably been solved a long time ago. Often when social issues are discussed there is some background in the life experience of the people involved. For example Rookie Mag has an article about the street harassment women face which includes many disturbing anecdotes (some of which concern primary school students) [12]. Obviously anyone who has lived through that sort of thing (which means most women) will instinctively understand some issues related to threatening sexual behavior that I can t easily understand even when I spend some time considering the matter. So there will be things which don t immediately appear to be serious problems to me but which are interpreted very differently by women. The non-patronising approach to such things is to accept the concerns women express as legitimate, to try to understand them, and not to argue about it. For example the issue that Valerie recently raised wasn t something that seemed significant when I first read the email in question, but I carefully considered it when I saw her posts explaining the issue and what she wrote makes sense to me. I don t think it s possible for a man to make a useful comment on any issue related to the treatment of women without consulting multiple women first. I suggest a pre-requisite for any man who wants to write any sort of long article about the treatment of women is to have conversations with multiple women who have relevant knowledge. I ve had some long discussions with more than a few women who are involved with the FOSS community. This has given me a reasonable understanding of some of the issues (I won t claim to be any sort of expert). I think that if you just go and imagine things about a group of people who have a significantly different life-experience then you will be wrong in many ways and often offensively wrong. Just reading isn t enough, you need to have conversations with multiple people so that they can point out the things you don t understand. This isn t any sort of comprehensive list of ways to avoid being patronising, but it s a few things which seem like common mistakes. Anne Onne wrote a detailed post advising men who want to comment on feminist blogs etc [13], most of it applies to any situation where men comment on women s issues.

21 August 2014

Steve Kemp: Updating Debian Administration

Recently I've been getting annoyed with the Debian Administration website; too often it would be slower than it should be considering the resources behind it. As a brief recap I have six nodes:

1 x MySQL Database - The only MySQL database I personally manage these days.
4 x Web Nodes.
1 x Misc server.

The misc server is designed to display events. There is a node.js listener which receives UDP messages and stores them in a rotating buffer. The messages might contain things like "User bob logged in", "Slaughter ran", etc. It's a neat hack which gives a good feeling of what is going on cluster-wide. I need to rationalize that code - but there's a very simple predecessor posted on github for the curious. Anyway enough diversions, the database is tuned, and "small". The misc server is almost entirely irrelevent, non-public, and not explicitly advertised. So what do the web nodes run? Well they run a lot. Potentially. Each web node has four services configured:

Apache 2.x - All nodes.
uCarp - All nodes.
Pound - Master node.
Varnish - Master node.

Apache runs the main site, listening on *:8080. One of the nodes will be special and will claim a virtual IP provided via ucarp. The virtual IP is actually the end-point visitors hit, meaning we have:

Master Host	Other hosts
Running: Apache. Pound. Varnish	Running: Apache.

Pound is configured to listen on the virtual IP and perform SSL termination. That means that incoming requests get proxied from "vip:443 -> vip:80". Varnish listens on "vip:80" and proxies to the back-end apache instances. The end result should be high availability. In the typical case all four servers are alive, and all is well. If one server dies, and it is not the master, then it will simply be dropped as a valid back-end. If a single server dies and it is the master then a new one will appear, thanks to the magic of ucarp, and the remaining three will be used as expected. I'm sure there is a pathological case when all four hosts die, and at that point the site will be down, but that's something that should be atypical. Yes, I am prone to over-engineering. The site doesn't have any availability requirements that justify this setup, but it is good to experiment and learn things. So, with this setup in mind, with incoming requests (on average) being divided at random onto one of four hosts, why is the damn thing so slow? We'll come back to that in the next post. (Good news though; I fixed it ;)

18 August 2014

Junichi Uekawa: sigaction bit me.

sigaction bit me. There's a system call and a libc function of the similar (sigaction vs rt_sigaction) name but they behave differently.

17 August 2014

Jamie McClelland: Getting to know systemd

Update 2014-08-20: apcid needs tweaking. See update section below. Somehow both my regular work laptop and home entertainment computers (both running Debian Jessie) were switched to systemd without me noticing. Judging from by dpkg.log it may have happened quite a while ago. I'm pretty sure that's a compliment to the backwards compatibility efforts made by the systemd developers and a criticism of me (I should be paying closer attention to what's happening on my own machines!). In any event, I've started trying to pay more attention now - particularly learning how to take advantage of this new software. I'll try to keep this blog updated as I learn. For now, I have made a few changes and discoveries. First - I have a convenient bash wrapper I use that both starts my OpenVPN client to a samba server and then mounts the drive. I only connect when I need to and rarely do I properly disconnect (the OpenVPN connection does not start automatically). So, I've written the script to carefully check if my openvpn connection is present and either restart or start depending on the results. I had something like this:

if ps -eFH   egrep [o]penvpn; then
  sudo /etc/init.d/openvpn restart
else
  sudo /etc/init.d/openvpn start
fi

One of the advertised advantages of systemd is the ability to more accurately detect if a service is running. So, first I changed this to:

if systemctl -q is-active openvpn.service; then
  sudo systemctl restart openvpn.service
else
  sudo systemctl start openvpn.service
fi

However, after reviewing the man page I realized I can shorten if further to simply:

  sudo systemctl restart openvpn.service

According to the man page, restart means:

Restart one or more units specified on the command line. If the units are not
running yet, they will be started.

After discovering this meaning for "restart" in systemd, I tested and realized that "restart" works the same way for openvpn using the old sysv style init system. Oh well. At least there's a man page and a stronger guarantee that it will work with all services, not just the ones that happen to respect that convention in their init.d scripts. The next step was to disable openvpn on start up. I confess, I never bothered to really learn update-rc.d. Everytime I read the manpage I ended up throwing up my hands and renaming symlinks by hand. In the case of openvpn I had previously edited /etc/default/openvpn to indicate that "none" of the virtual private networks should be started. Now, I've returned that file to the default configuration and instead I ran:

systemctl disable openvpn.service

UPDATES 2014-08-20: I've recently noticed strange behavior when I wake my laptop. Seems to sometimes go right back to sleep. After doing some digging I traced the problem to some customizations I have made to my laptop's acpid behavior combined with systemd taking over some apci events. Up to now, I have created my own /etc/acpi files so I have full control over the acpi events. In particular, I don't want my laptop to suspend when I close the lid. I only want it to suspend when I press the suspend key. And, when it does suspend, I want it to run my own personal suspend script so I can do things like lock the screen and restart tint2. I've found that systemd launches it's own acpi event monitoring that ignores my /etc/acpid rules (the systemd "unit" that monitors events is called acpid.socket which exists in addition to acpid.service). The systemd reaction to events is controlled by the systemd-logind.service which has a configuration file: /etc/systemd/logind.conf. By default, systemd-logind.service will put my laptop to sleep when the lid is closed and when the suspend button is pushed. systemd seems to get the signal first, putting the laptop to sleep. After I wake it up, acpid gets the signal - so it goes right back to sleep. Reading man logind.conf helps. I was able to restore my desired behavior by adding these lines to /etc/systemd/logind.conf:

HandleSuspendKey=ignore
HandleLidSwitch=ignore

Then: sudo systemctl restart systemd-logind.service.

15 August 2014

Aurelien Jarno: Intel about to disable TSX instructions?

Last time I changed my desktop computer I bought a CPU from the Intel Haswell family, the one available on the market at that time. I carefully selected the CPU to make sure it supports as many instructions extensions as possible in this family (Intel likes segmentation, even high-end CPUs like the Core i7-4770k do not support all possible instructions). I ended-up choosing the Core i7-4771 as it supports the Transactional Synchronization Extensions (Intel TSX) instructions, which provide transactional memory support. Support for it has been recently added in the GNU libc, and has been activated in Debian. By choosing this CPU, I wanted to be sure that I can debug this support in case of bug report, like for example in bug#751147. Recently some computing websites started to mention that the TSX instructions have bugs on Xeon E3 v3 family (and likely on Core i7-4771 as they share the same silicon and stepping), quoting this Intel document. Indeed one can read on page 49:

HSW136. Software Using Intel TSX May Result in Unpredictable System Behavior

Problem: Under a complex set of internal timing conditions and system events, software using the Intel TSX (Transactional Synchronization Extensions) instructions may result in unpredictable system behavior.
Implication: This erratum may result in unpredictable system behavior.
Workaround: It is possible for the BIOS to contain a workaround for this erratum.

And later on page 51:

Due to Erratum HSw136, TSX instructions are disabled and are only supported for software development. See your Intel representative for details.

The same websites tell that Intel is going to disable the TSX instructions via a microcode update. I hope it won t be the case and that they are going to be able to find a microcode fix. Otherwise it would mean I will have to upgrade my desktop computer earlier than expected. It s a bit expensive to upgrade it every year and that s a the reason why I skipped the Ivy Bridge generation which didn t bring a lot from the instructions point of view. Alternatively I can also skip microcode and BIOS updates, in the hope I won t need another fix from them at some point.

11 August 2014

Juliana Louback: JSCommunicator - Setup and Architecture

Preface During Google Summer of Code 2014, I got to work on the Debian WebRTC portal under the mentorship of Daniel Pocock. Now I had every intention of meticulously documenting my progress at each step of development in this blog, but I was a bit late in getting the blog up and running. I ll now be publishing a series of posts to recap all I ve done during GSoC. Better late than never. Intro JSCommunicator is a SIP communication tool developed in HTML and JavaScript. The code was designed to make integrating JSCommunicator with a website or web app as simple as possible. It s quite easy, really. However, I do think a more detailed explanation on how to set things up and a guide to the JSCommunicator architecture could be of use, particularly for those wanting to modify the code in any way. Setup To begin, please fork the JSCommunicator repo. If you are new to git, feel free to follow the steps in section Setup and Clone in this post. If you read the README file (which you always should), you ll see that JSCommunicator needs a SIP proxy that supports SIP over Websockets transport. Some options are:

I didn t find a tutorial for Kamailio setup, I did find one for repro setup. And bonus, here you have a great tutorial on how to setup and configure your SIP proxy AND your TURN server. In your project s root directory, you ll see a file named config-sample.js. Make a copy of that file named config.js. The config-sample.js file has comments that are very instructive. In sum, the only thing you have to modify is the turn_servers property and the websocket property. In my case, debrtc.org was the domain registered for testing my project, so my config file has:

turn_servers: [
	  server:"turn:debrtc.org"       
],

Note that unlike the sample, I didn t set a username and password so SIP credentials will be used. Now fill in the websocket property here we use sip5060.net.

websocket:  
    servers: 'wss://ws.sip5060.net',
    connection_recovery_min_interval: 2,
    connection_recovery_max_interval: 30,
   ,

I m pretty sure you can set the connection_recovery properties to whatever you like. Everything else is optional. If you set the user property, specifically display_name and uri, that will be used to fill in the Login form and takes preference over any Remember me data. If you also set sip_auth_password, JSCommunicator will automatically log in. All the other properties are for other optional functionalities and are well explained. You ll need some third-party javascript that is not included in the JSCommunicator git repo. Namely jQuery version 1.4 or higher and ArbiterJS version 1.0. Download jQuery here and ArbiterJS here and place the .js files in your project s root directory. Do make sure that you are including the correct filename in your html file. For example, in phone-dev.shtml, a file named jquery.js is included. The file you downloaded will likely have version numbers in it s file name. So rename the downloaded file or change the content of src in your includes. This is uber trivial, but I ve made the mistake several times. You ll also need JsSIP.js which can be downloaded here. Same naming care must be taken as is the case for jQuery and ArbiterJS. The recently added Instant Messaging component and some of the new features need jQuery-UI - so far version 1.11.* is known to work. From the downloaded .zip all you need is the jquery-ui-...js file and the jquery-ui-...css file, also to be placed in the project s root directory. If you ll be using the internationalization support you ll also need jquery.i18n.properties.js. To try out JSCommunicator, deploy the website locally by copying your project directory to the apache document root directory (provided you are using apache, which is a good idea.). You ll likely have to restart your server before this works. Now the demo .shtml pages only have a header with all the necessary includes, then a Server Side Include for the body of the page, with all the JSCommunicator html. The html content is in the file jscommunicator.inc. You can enable SSI on apache, OR you can simply copy and paste the content of jscommunicator.inc into phone-dev.shmtl. Start up apache, open a browser and navigate to localhost/jscommunicator/phone-dev.shmtl and you should see: jscommunicatorRaw

Actually, pending updates to JSCommunicator, you should see a brand new UI! But all the core stuff will be the same. Architecture Disclaimer: I m explaining my view of the JSCommunicator architecture, which currently may not be 100% correct. But so far it s been enough for me to make my modifications and additions to the code, so it could be of use. One I get a JSCommunicator Founding Father s stamp of approval, I ll be sure to confirm the accuracy. Now to explain how the JSCommunicator code interacts, the use of each code item is described, ignoring all the html and css which will vary according to how you choose to use JSCommunicator. I m also not going to explain jQuery which is a dependency but not specific to WebRTC. The core JSCommunicator code is the following:

config.js
jssip-helper.js
parseuri.js
webrtc-check.js
JSCommUI.js
JSCommManager.js
make-release
init.js
Arbiter.js
JsSIP.js

Each of these files will be presented in what I hope is an intuitive order.

config.js - As expected, this file contains your custom configuration specifications, i.e. the servers being used to direct traffic, authentication credentials, and a series of properties to enable/disable optional functionalities in JSCommunicator.

The next three files could be considered utils :

jssip-helper.js - This will load the data from config.js necessary to run JSCommunicator, such as configurations relating to servers, websockets, connection specifications (timeout, recovery intervals), and user credentials. Properties for optional features are ignored of course.
parseuri.js - Contains a function to segregate data from a URI.
webrtc-check.js - Verifies if browser is WebRTC compatible by checking if it s possible to enable camera and microphone access.

These two are where the magic happens:

JSCommUI.js - Responsible for the UI component, controling what becomes visible to the user, audio effects, client-side error handling, and gathering the data that will be fed to JSCommManager.js.
JSCommManager.js - Initializes the SIP User Agent to manage the session including (but not limited to) beginning/terminating the session, sending/receiving SIP messages and calls, and signaling the state of the session and other important notifications such as incoming calls, messages received, etc.

Now for some extras:

make-release - Combines the main .js files into one big file. Gets jssip-helper.js, parseuri.js, webrtc-check.js, JSCommUI.js and JSCommManager.js and spits out a new file JSComm.js with all that content. Now you understand why phone-dev.shtml included each of the 5 files mentioned above whereas phone.shmtl includes only JSComm.js which didn t exist until you ran make-release. That confused me a bit.
init.js - On page load, calls JSCommManager.js init function, which eventually calls JSCommUI.js init function. In other words, starts up the JSCommunicator app. I guess it s only used to show how to start up the app. This could be done directly on the page you re embedding JSCommunicator from. So maybe not entirely needed.

Third party code:

Arbiter.js - Javascript implmentation of the publish/subscribe patten, written by Matt Kruse. In JSCommunicator it s used in JSCommManager.js to publish signals to describe direct the app s behavior. For example, JSCommManager will publish a message indicating that the user received a call from a certain URI. In event-demo.js we subscribe to this kind of message and when said message is received, an action can be performed such as adding to the app s call history. Very cool.
JsSIP.js - Implements the SIP WebSocket transport in Javascript. This ensures the transportantion of data is done in adherence to the WebSocket protocol. In JSCommManager.js we initialize a SIP User Agent based in the implementation in JsSIP.js. The User Agent will translate all of the JSCommunicator actions into SIP WebSocket format. For example, when sending an IM, the JSCommunicator app will collect the essential data such as say the origin URI, destination URI and an IM message, while the User Agent is in charge of formating the data so that it can be transported in a SIP message unit. A SIP message contains a lot more information than just the sender, receiver and message. Of course, a lot of the info in a SIP message is irrelevant to the user and in JSCommUI.js we filter through all that data and only display what the user needs to see.

Here s a diagram of sorts to help you visualize how the code interacts: jscommArch

In sum, 1 - JSCommUI.js handles what is displayed in the UI and feeds data to JSCommManager.js; 2 - JSCommManager.js actually does stuff, feeding data to be displayed to JSCommUI.js; 3 - JSCommManager.js calls functions from the three utils , parseuri.js, webrtc-check.js and jssip-helpher.js which organizes the data from config.js; 4 - JSCommManager.js initializes a SIP User Agent based on the implementation in Arbiter.js. When making any changes to JSCommunicator, you will likely only be working with JSCommUI.js and JSCommManager.js.

Juliana Louback: JSCommunicator - Setup and Architecture

turn_servers: [
	  server:"turn:debrtc.org"       
],

Note that unlike the sample, I didn t set a username and password so SIP credentials will be used. Now fill in the websocket property here we use sip5060.net.

websocket:  
    servers: 'wss://ws.sip5060.net',
    connection_recovery_min_interval: 2,
    connection_recovery_max_interval: 30,
   ,

config.js
jssip-helper.js
parseuri.js
webrtc-check.js
JSCommUI.js
JSCommManager.js
make-release
init.js
Arbiter.js
JsSIP.js

Each of these files will be presented in what I hope is an intuitive order.

config.js - As expected, this file contains your custom configuration specifications, i.e. the servers being used to direct traffic, authentication credentials, and a series of properties to enable/disable optional functionalities in JSCommunicator.

The next three files could be considered utils :

jssip-helper.js - This will load the data from config.js necessary to run JSCommunicator, such as configurations relating to servers, websockets, connection specifications (timeout, recovery intervals), and user credentials. Properties for optional features are ignored of course.
parseuri.js - Contains a function to segregate data from a URI.
webrtc-check.js - Verifies if browser is WebRTC compatible by checking if it s possible to enable camera and microphone access.

These two are where the magic happens:

JSCommUI.js - Responsible for the UI component, controling what becomes visible to the user, audio effects, client-side error handling, and gathering the data that will be fed to JSCommManager.js.
JSCommManager.js - Initializes the SIP User Agent to manage the session including (but not limited to) beginning/terminating the session, sending/receiving SIP messages and calls, and signaling the state of the session and other important notifications such as incoming calls, messages received, etc.

Now for some extras:

make-release - Combines the main .js files into one big file. Gets jssip-helper.js, parseuri.js, webrtc-check.js, JSCommUI.js and JSCommManager.js and spits out a new file JSComm.js with all that content. Now you understand why phone-dev.shtml included each of the 5 files mentioned above whereas phone.shmtl includes only JSComm.js which didn t exist until you ran make-release. That confused me a bit.
init.js - On page load, calls JSCommManager.js init function, which eventually calls JSCommUI.js init function. In other words, starts up the JSCommunicator app. I guess it s only used to show how to start up the app. This could be done directly on the page you re embedding JSCommunicator from. So maybe not entirely needed.

Third party code:

Arbiter.js - Javascript implmentation of the publish/subscribe patten, written by Matt Kruse. In JSCommunicator it s used in JSCommManager.js to publish signals to describe direct the app s behavior. For example, JSCommManager will publish a message indicating that the user received a call from a certain URI. In event-demo.js we subscribe to this kind of message and when said message is received, an action can be performed such as adding to the app s call history. Very cool.
JsSIP.js - Implements the SIP WebSocket transport in Javascript. This ensures the transportantion of data is done in adherence to the WebSocket protocol. In JSCommManager.js we initialize a SIP User Agent based in the implementation in JsSIP.js. The User Agent will translate all of the JSCommunicator actions into SIP WebSocket format. For example, when sending an IM, the JSCommunicator app will collect the essential data such as say the origin URI, destination URI and an IM message, while the User Agent is in charge of formating the data so that it can be transported in a SIP message unit. A SIP message contains a lot more information than just the sender, receiver and message. Of course, a lot of the info in a SIP message is irrelevant to the user and in JSCommUI.js we filter through all that data and only display what the user needs to see.

Here s a diagram of sorts to help you visualize how the code interacts: jscommArch

10 August 2014

Cyril Brulebois: Why is my package blocked?

A bit of history: A while ago udeb-producing packages were getting frozen on a regular fashion, when a d-i release was about to be cut. While I wasn t looking at the time, I can easily understand the reasons behind that: d-i is built upon many components, it takes some time to make sure it s basically in shape for a release, and it s very annoying when a regression sneaks in right before the installation images get built. I took over d-i release maintenance in May 2012 and only a few uploads happened before the wheezy freeze. I was only discovering the job at the time, and I basically released whatever was in testing then. The freeze began right after that (end of June), so I started double checking things affecting d-i (in addition to or instead of the review performed by other release team members), and unblocking packages when changes seemed safe, or once they were tested. A few uploads happened after the wheezy release and there s already a Jessie Alpha 1 release. I was about to release Jessie Beta 1 after some fair bits of testing, a debian-installer upload, and the only remaining bits were: building installation images (hello Steve), and of course communication (mail announce and website update). Unfortunately a new upstream release reached testing in the meanwhile, breaking the installer in several ways. I ll give details below, of course not because I want to point finger at the maintainer, but to illustrate the ramifications that a single package s migrating to testing can induce.

parted 3.2-1 was uploaded on 2014-07-30 and migrated on 2014-08-05.
parted 3.2-2 fixed a regression reported in Ubuntu only (LP#1352252) which I also hit with images built locally after that migration.
I then built some images locally using fixed parted packages but then discovered that auto-lvm was still broken, which I reported in #757417.
After some investigation Colin confirmed some behavioral changes in this new parted release, which imply the need for an update of several other partman-* components: #757661, #757662, #757663, #757664, #757665, #757666.
Thankfully fixes have been added for all of those, but more testing is needed before possibly urgenting those packages so that they get into testing as soon as possible.

Since I d like to avoid such experience in the future, I ll probably reintroduce the old method and freeze all udeb-producing packages during next d-i releases. So you know why it might happen. Your next question might be: What to do when your package is getting caught in that net? . In that case, please get in touch with both debian-release@ and debian-boot@ asking for an unblock. I ll then review your particular package, and either let it migrate to testing, or delay it until after the release. Update: official announcement.

Next.

Previous.