Search Results: "spa"

16 July 2012

Thorsten Glaser: Apache 2, https clients linked against GnuTLS, connection errors

I ve been debugging a weird problem at work after upgrading a complex system from lenny to wheezy, some https clients failed to connect: GNU wget and Debian s version of lynx(1) which is linked against libgnutls26 fail. NSS applications continue to work, as does cURL; wget and lynx on MirBSD (linked with OpenSSL of course) work. Even Debian s gnutls-cli tools from both gnutls26 and gnutls28 work. Huh. The error_log shows renegotiation problems, yet setting the new Apache 2 configuration option to use insecure renegotiation doesn t help either. (The option is a total #FAIL: its only other value is use secure TLSv1.x renegotiation , but I don t want/need SSL renegortiation at all, anyway.) Natureshadow told me this was a hot issue on Debianforum at the moment, yet, nobody had a clue or enough information to file a formal bugreport against (initially) apache2, as that s what changed. I tracked it down on a new VM with no configuration otherwise, and here are my findings so others don t run into it.

Tracking down the problem, this can be reduced to the following configuration (minimised, to show the problem) in /etc/apache2/sites-enabled/1one:

	<VirtualHost *:443>
		ServerName wiki-70.lan.tarent.de
		RedirectMatch permanent . https://evolvis-70.lan.tarent.de/
		SSLEngine on
		SSLCertificateFile /etc/ssl/W_lan_tarent_de.cer
		SSLCertificateKeyFile /etc/ssl/private/W_lan_tarent_de.key
		SSLCertificateChainFile /etc/ssl/godaddy.ca
	</VirtualHost>
 

Do not mind the actual content, this is a very stripped-down demo on a not-actually-set-up-yet box.

Same is valid for the companion configuration file /etc/apache2/sites-enabled/2two:

	NameVirtualHost *:443
	<VirtualHost *:443>
		ServerName evolvis-70.lan.tarent.de
		SSLEngine on
		# workaround for BEAST (CVE-2011-3389), short-term
		SSLCipherSuite RC4-SHA
		SSLCertificateFile /etc/ssl/W_lan_tarent_de.cer
		SSLCertificateKeyFile /etc/ssl/private/W_lan_tarent_de.key
		SSLCertificateChainFile /etc/ssl/godaddy.ca
		SSLProtocol TLSv1
	</VirtualHost>
 

Turns out the BEAST workaround was at fault here: the differing SSLCipherSuites between the vhosts (on the same Legacy IP / TCP Port tuple, as we use Wildcard SSL Certificates) made Apache 2 want to renegotiate, so either commenting it on 2two or, better, adding it to 1one helped. Interestingly enough, the SSLProtocol directive did not matter (in my tests). So, keep SSL settings synchronised between vhosts. In fact, those were already from include files, but 2two was from the Evolvis 5 generation, whereas we added to 1one an Include of the httpd.ssl1.inc file generated by the previous releases of EvolvisForge and had not switched those legacy vhosts to the new configuration, as everything worked on lenny. This wlog entry brought to you by the system administrators of tarent solutions GmbH and the Evolvis Project, based on FusionForge.

15 July 2012

Jordi Mallach: Season of change

It feels like I'm sitting in a roller-coaster wagon. There's probably too much change going on for me to assimilate naturally. In particular: I just wrapped up (well, mostly) one of the toughest Uni semestres. I had to deal with lots of very time consuming assignments, and then the usual round of final exams. Even if this semester I got the best marks in my journey (or shall we say Via Crucis) through University, I still managed to fail one exam, for the Advanced Networks subject, which is quite annoying, given I got high marks (even the highest in one case) in other subjects I really don't master at all. In any case, this is the end of the pain. The only thing that's left is just one exam and a project based on GNU/Linux technologies which will basically mean formatting for prettyness the sysadmin docs we've been collecting at the office during the last few years. This effort will be nothing to what I've been doing during the past 18 months, so I'm really relieved to have it past me already. Getting rid of studies comes just two weeks before a big change in my professional career. Friday was my last day at the Institut Tecnol gic d'Inform tica, after five and a half fantastic years working with awesome people in a very friendly atmosphere. I've learned a great deal, and taking this decision wasn't easy at all. I leave lots of good friends behind, people I really love, and tomorrow will be difficult to not have them around me. I wish my ITI ex-workmates the best of luck in these difficult times for everyone in Spain and specially in the Valencian Country with the massive cuts going on. I feel the timing for this jump couldn't be better. Tomorrow, when I get ready to go for work, I won't be leaving home at all, instead I will just sit where I am right now, at home, and log into some corporate IRC server. Tomorrow I'll be joining Collabora, and I'm a mix of excited, curious and happy about this incredible opportunity. Thanks to Sjoerd for nags, I might not be writing this if it wasn't for you!

Collabora When I was first approached about this, I thought Collabora was a small company. But as I looked more into it, I discovered that's not longer the case, there's many more people than I imagined working there (here!), and was delighted to see I knew many of them, and many other are well known members of the major Free Sofware communities. I'll be joining the sysadmin team to work closely with Jo Shields. See you tomorrow, folks. :) This opportunity to work from home is godsend, given the third bit of change that'll be happening soon: sometime in late September, Maria and I should join the ranks of first-time parents, following the baby boom wave surrounding us. While you can imagine we're really happy about this, we're also freaking out because this is going to happen in just two months and a half, and weeks go by really fast lately. So yeah, being able to be at home with this really small baby will be a big bonus for the incredible experience we're about to enjoy. We've been both busy with other stuff, but during the summer we should be focusing on preparing the baby's arrival. There's a whole lot to do! Expect my Debian and other Free Software activities to get a hit, of course. :) If I am normally sleep-deprived, this is going to be the next level.

Paul Tagliamonte: usage of vcs-git in the Debian archive

I ve recently gone through some of the unpacked source sitting in the lintian lab, and I ve got a few cool stats. First of all, the types of VCS tags I ve found:
      1 vcs-git
    123 Vcs-git
   6480 Vcs-Git
      1 Vcs-GIT
      5 VCS-git
     86 VCS-Git
Interesting! It seems the (IMHO correct) Vcs-Git style has a lot of adoption
     17 XS-Vcs-Git
and a few old style git tags :) Well, let s take a look at the most common prefixes:
   6498 git
    196 http
     15 https
All pretty stock, when I see (to my shock) a:
      4 git+ssh
Oh noes! Let s get rid of those, folks! :) So, let s check out the most popular domains to host git packages:
    5375 git.debian.org
     555 anonscm.debian.org
     171 github.com
OK, I d like to stop right here. As much as I hate nonfree software, and I think relying on as part of our infrastructure sucks, there is a degree of sense to keeping stuff on GitHub. If upstream is on GitHub, it may make it easier to coordinate and work with them. However! These are repos sitting in people s personal accounts which means taking over the package in the event someone needed to is not practical. The person with the highest number of Debian repos on GitHub is 19, followed by 12, 5 and a few 4s. Enter github.com/debian IMHO, we should consider putting the repos that are already on GitHub under Debian namespace, so that the team of maintainers may be able to add new collaborators. I mean, we have the account and some friendly DDs who offered to take care of this organization outside the project. That, or get us off GitHub entirely. I m not sure that s wise, though. However, even though there are a ton of repos on GitHub, it s about 1/40th of the total number of VCS tagged packages in Debian. On with the show:
      60 git.debian-maintainers.org
      39 gitorious.org
      26 git.b9.com
      22 git.lxde.org
      19 git.kitenet.net
      19 git.brad-smith.co.uk
      18 git.gag.com
And some of the more interesting ones:
      1 code.launchpad.net
for Vcs-Git??? :)
      1 git.debian.org:
Looks like it was a login@git.d.o: URL that was just changed slightly. Let s double-check our VCS tags, people :) The breakdown of Debian VCS d URLs:
      1 anonscm.alioth.debian.org
      1 anonscp.debian.org
      1 git.debian.org:
      2 scm.alioth.debian.org
     12 alioth.debian.org
    555 anonscm.debian.org
   5375 git.debian.org
And out of git.d.o, 2544 of them use the git://git.debian.org/git/pkg-foo pattern rather then the git://git.debian.org/pkg-foo pattern. Interesting! roughly 78 use /users and roughly 113 use /git/users I had no idea we had so many complexities in our VCS pointers :) When I get some more time, I ll go through and figure out which team has the most Git VCS tags :)

Christian Perrier: 2012 update 29 for Debian Installer localization

Not much progress ATM (most translations are complete and D-I has been frozen to prepare its release anyway). Status for D-I level 1 (core D-I files): Status for D-I level 2 (packages that have localized material that may appear during default installs, such as iso-codes, tasksel, etc.): Status for D-I level 3 (packages that have localized material that may appear during non-default installs, such as win32-loader) Full 100% completeness (hall of fame) for 32 languages: Asturian, Belarusian, Bulgarian, Catalan, Czech, Welsh, Danish, German, Esperanto, Spanish, Persian, French, Galician, Hebrew, Indonesian, Italian, Japanese, Kazakh, Khmer, Latvian, Norwegian Bokm l, Dutch, Polish, Portuguese, Russian, Slovak, Slovenian, Swedish, Thai, Turkish, Simplified Chinese, Traditional Chinese

Mike Hommey: Comment spam

Three weeks ago, I slightly modified the comment system on this blog for an experiment. This blog is a standard wordpress installation. Comments are normally directed to the wp-comments-post.php script by the HTML form. What I did is: During the past three weeks, on this blog, there were 7170 comments, 8 of which were actual comments. 7162 were spam (~99.9%). This means a large portion of spammers didn t care about actually checking the comment forms and used the standard wordpress url, and another large portion don t run javascript on their bots, although a very few do.

14 July 2012

Thorsten Glaser: bubulle s Cool prompt for git users with mksh

Originally posted by bubulle on Planet Debian, a shell prompt that displays the current git branch, in colour on some terminals, after the current working directory. The following snippet does similar things for mksh users, except it doesn t redefine your prompt but amend it just throw it at the bottom of your ~/.mkshrc before that last line beginning with a colon (copy from /etc/skel/.mkshrc if you haven t done that yet):

	function parse_git_branch  
		git branch 2>/dev/null   sed -n '/^\* \(.*\)/s//(\1)/p'
	 
	function amend_prompt_with_git  
		local p q='$(parse_git_branch)' r
		if [[ $TERM = @(xterm-color xterm screen*) ]]; then
			if [[ $ PS1:1:1  = $'\r' ]]; then
				p=$ PS1:0:1 
			else
				p=$'\001'
				PS1=$p$'\r'$PS1
			fi
			q=$p$'\e[1;33m'$p$q$p$'\e[0m'$p
		fi
		p=$ PS1%%*( )[#$]*( ) 
		if [[ $p != "$PS1" ]]; then
			# prompt ends with space + #-or-$ + space, we can amend
			r=$ PS1: $ #p 
			PS1=$p$q$r
		fi
	 
	amend_prompt_with_git
	unset -f amend_prompt_with_git
 

The indirection by use of a function is not strictly necessary but allows the use of locals. I took the liberty of adding an asterisk after screen to match the GNU/Linux nonsense of having TERM=screen.xterm or somesuch.

Christian Perrier: Debconf 12 work

As already written, even though it seems, I didn't spend my time running at Debconf12. Or eating cheese... Or (tentatively) hiking volcanoes... Or helping people to kill each other with socks... Or drinking beers... Not *only* all of these (some of them at the same time, though hiking a volcano while eating cheese and drinking beers is not particularly easy)....but also some Debian work. So, I uploaded a backport of samba to squeeze backports and our squeeze users should now have the same samba version than wheezy ones. I also stopped several cronjobs on i18n.debian.net and moved some material there as links have been (or should be) moved to the brand new i18n.debian.org (and its alias l10n.debian.org). I did a major cleanup in tasksel, committed several fixes, proposed others for review (mostly to Joey Hess). All this in preparation for a soon-to-come upload, probably after D-I beta1 which has been prepared by Cyril Brulebois while he was.....not attending Debconf12.. I also followed the integration of Sorina Sandu's work on netcfg for link detection and network ESSID choice in Debian Installer. Sorina is doing well in her GSOC work, because she's clearly someone with great capabilities who we will, I hope, be able to keep contributing to Debian. We can also thank the mentoring work of Gaudenz Steinlin to guide Sorina through D-I's arcanes. I also went through the current status of debconf translation completeness in testing for the 7 languages that target 100% (Czech, German, Spanish, French, Portuguese, Russian, Swedish). Here, the point is mostly taking care that fixes are either: I can we have good chances to achieve this goal for at least 6 of these 7 languages (Czech might be more tricky as several translations are not yet made). Finally, I also did most of my regular Debian work (which usually takes 1-2 hours every day)...and bits of my paid work (sorry for those people who I shared the table at breakfast, but that was my only common time window with my team at work). So, well, quite productive weeks, once again.

Jonathan Carter: Debconf 12 Managua, Nicaragua


It s the last day of Debconf 12 in Managua, it s the first ever Decbonf I ve attended and it has been just awesome.

There s so much I could talk about and it would take forever to put down, but here s a few highlights for me:
  • My first time in Nicaragua and also Central America! First time I ve ever seen a vulcano!
  • The Bits from the DPL talk was great, Zack pretty much hit several nails on the head of why I love the Debian project so much
  • The Debian Cheese and Wine party was off the wall, the food and the drinks were awesome and it was a lot of fun
  • I ve been attending some of the Debconf organising sessions, I was considering getting a few people together so that we could put in a bit for Sherbrooke for 2016, but then I learned that a bunch of people are already getting together to put a bid together for Montr al in 2014, so I ll get involved with that instead. Stefano and I have also been talking about a hypothetical Debconf in 2018 or 2020 in Cape Town or Stellenbosch. As much as we want a Debconf in Africa, neither of us will have time to organise a team for that in the short term. I m hoping that being involved in the Montr al bid (and hopefully an event) will give me good experience for the Cape Town one.
  • The day trip in the middle of the week was great. I just spent the whole day lying in a hammock, drinking beer and staring at the ocean. I think I needed that.
  • The Debian key signing party was good, I got my key signed by more than 20 Debian contributors this week.
  • The sessions and BoFs were great. I learned a lot about the Debian project this week and filled up a few gaps in my knowledge.
  • I got involved with the video team and played camera man for a few sessions for the live video feeds. It was surprisingly fun. I took some photos too during the week but my camera is really awful, I need to get a new one by the next Debconf.
  • The people of Nicaragua are awesome. Everyone here is so friendly and helpful, it turned
  • There s probably too much other things to mention and I ll never get a blog entry out if I even try, but thanks to everyone who made this Debconf possible, even Pollito.

Gunnar Wolf: La mochila viajera The travelling backpack

TitleLa mochila viajera The travelling backpack
Publication TypeBroadcast
Year of Publication2012
AuthorsWolf, G, Bello L, Ram n E, Taleno A, Ortega J
Refereed DesignationNon-Refereed
Full TextDuring DebConf12, and thanks to Fito's kind invitation, a bunch of us were part of a radio program that's transmitted in Managua and Chinandega, La mochila viajera . This program talks about different viewpoints and problematics about migration, seen from the eyes of Nicaraguans, which share quite a bit with us in Mexico But at the same time, are very different.
I was invited as part of a research group in UNAM on the topic Communication, trasmigration and emergent systems . To be honest, I'm way behind schedule with the part of the research I should be doing, but here I very roughly describe (part of) the topic I'm interested in.
The session was quite interesting. Of course, the reason I'm sending this to the blog is mainly that I want to keep it at hand, but I think some of you might find it interesting. Of course, it is in Spanish, so YMMV ;-)
AttachmentSize
La mochila viajera, 13 de julio del 2012: Migrantes y software libre84.19 MB

Romain Francoise: Getting clickable URLs in xterm (sort of)

My terminal emulator of choice is xterm: it's fast, light, and (since it's pretty much the reference implementation) it has excellent support for everything a console user could desire... except for one thing that would be very, very convenient: making URLs clickable. Other terminal emulators have this feature, but they also have problems that make them inferior to xterm in different ways. Over the years people have come up with various workarounds for this situation, like screen scraping utilities (urlview, urlscan) that can be hooked up to other console programs to extract and browse URLs, but they're still not as convenient as just using the mouse, and often require the program to run on the same machine as the browser. Fortunately, xterm #277 (released in January 2012) added a new feature that provides almost exactly what I had been looking for: it can now spawn programs using the exec-formatted action and give them as argument the contents of the current selection or clipboard. So you can add the following to your ~/.Xresources: *VT100*translations: #override Meta <Btn1Up>: exec-formatted("x-www-browser '%t'", PRIMARY) which makes xterm run x-www-browser on the selection when it receives Alt + left click. (Adjust for whatever your Meta key is.) This is advantageously combined with a charClass setting to make xterm treat URLs as a single word, so that you can just double-click on them to select them: XTerm*charClass: 33:48,36-47:48,58-59:48,61:48,63-64:48,95:48,126:48 With both of these enabled, opening URLs is now just a matter of:
  1. Double-clicking the URL to select it
  2. Doing Alt + click anywhere on the xterm window to run the browser
which, while more involved than a single click, it still much faster than having to copy the URL manually to the browser.

12 July 2012

Richard Hartmann: Do what I mean

And I still get into arguments about why I prefer apt-get over aptitude... Sometimes, stupid is good.
# apt-get update
[...get Debian sid package list...]
# aptitude install libdbd-csv-perl
The following NEW packages will be installed:
  libdbd-csv-perl libparams-util-perl a  libsql-statement-perl a  libtext-csv-xs-perl a 
The following packages will be upgraded:
  libdbi-perl perl perl-base perl-modules
4 packages upgraded, 4 newly installed, 0 to remove and 1459 not upgraded.
Need to get 9,845 kB of archives. After unpacking 1,069 kB will be used.
The following packages have unmet dependencies:
  libperl5.14: Depends: perl-base (= 5.14.2-7) but 5.14.2-12 is to be installed.
The following actions will resolve these dependencies:
      Remove the following packages:
1)      elinks
2)      irssi
3)      irssi-scripts
4)      libperl5.14
5)      perlmagick
6)      vim-gtk
7)      xchat
      Leave the following dependencies unresolved:
8)      elinks-data recommends elinks (= 0.12~pre5-7)
9)      vim-gui-common recommends vim-gnome   vim-gtk   vim-athena
10)     xchat-common recommends xchat
11)     inkscape recommends perlmagick
Accept this solution? [Y/n/q/?] ^C
# apt-get install libdbd-csv-perl
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following extra packages will be installed:
  libdbi-perl libparams-util-perl libperl5.14 libsql-statement-perl libtext-csv-xs-perl perl perl-base perl-modules
The following NEW packages will be installed:
  libdbd-csv-perl libparams-util-perl libsql-statement-perl libtext-csv-xs-perl
The following packages will be upgraded:
  libdbi-perl libperl5.14 perl perl-base perl-modules
5 upgraded, 4 newly installed, 0 to remove and 1458 not upgraded.
Need to get 10.6 MB of archives.
After this operation, 1,081 kB of additional disk space will be used.
Do you want to continue [Y/n]?
Get:1 http://ftp5.gwdg.de/pub/linux/debian/debian/ unstable/main perl i386 5.14.2-12 [3,701 kB]
Get:2 http://ftp5.gwdg.de/pub/linux/debian/debian/ unstable/main libperl5.14 i386 5.14.2-12 [731 kB]
Get:3 http://ftp5.gwdg.de/pub/linux/debian/debian/ unstable/main perl-base i386 5.14.2-12 [1,494 kB]
Get:4 http://ftp5.gwdg.de/pub/linux/debian/debian/ unstable/main perl-modules all 5.14.2-12 [3,438 kB]
Get:5 http://ftp5.gwdg.de/pub/linux/debian/debian/ unstable/main libtext-csv-xs-perl i386 0.90-1 [73.1 kB]
Get:6 http://ftp5.gwdg.de/pub/linux/debian/debian/ unstable/main libdbi-perl i386 1.622-1 [897 kB]
Get:7 http://ftp5.gwdg.de/pub/linux/debian/debian/ unstable/main libparams-util-perl i386 1.07-1 [25.4 kB]
Get:8 http://ftp5.gwdg.de/pub/linux/debian/debian/ unstable/main libsql-statement-perl all 1.33-1 [179 kB]
Get:9 http://ftp5.gwdg.de/pub/linux/debian/debian/ unstable/main libdbd-csv-perl all 0.3500-1 [36.4 kB]
Fetched 10.6 MB in 10s (980 kB/s)
Retrieving bug reports... Done
Parsing Found/Fixed information... Done
Reading changelogs... Done
(Reading database ... 278465 files and directories currently installed.)
Preparing to replace perl 5.14.2-7 (using .../perl_5.14.2-12_i386.deb) ...
Unpacking replacement perl ...
Preparing to replace libperl5.14 5.14.2-7 (using .../libperl5.14_5.14.2-12_i386.deb) ...
Unpacking replacement libperl5.14 ...
Preparing to replace perl-base 5.14.2-7 (using .../perl-base_5.14.2-12_i386.deb) ...
Unpacking replacement perl-base ...
Processing triggers for man-db ...
Setting up perl-base (5.14.2-12) ...
(Reading database ... 278465 files and directories currently installed.)
Preparing to replace perl-modules 5.14.2-7 (using .../perl-modules_5.14.2-12_all.deb) ...
Unpacking replacement perl-modules ...
Selecting previously unselected package libtext-csv-xs-perl.
Unpacking libtext-csv-xs-perl (from .../libtext-csv-xs-perl_0.90-1_i386.deb) ...
Preparing to replace libdbi-perl 1.617-1 (using .../libdbi-perl_1.622-1_i386.deb) ...
Unpacking replacement libdbi-perl ...
Selecting previously unselected package libparams-util-perl.
Unpacking libparams-util-perl (from .../libparams-util-perl_1.07-1_i386.deb) ...
Selecting previously unselected package libsql-statement-perl.
Unpacking libsql-statement-perl (from .../libsql-statement-perl_1.33-1_all.deb) ...
Selecting previously unselected package libdbd-csv-perl.
Unpacking libdbd-csv-perl (from .../libdbd-csv-perl_0.3500-1_all.deb) ...
Processing triggers for man-db ...
Setting up libperl5.14 (5.14.2-12) ...
Setting up perl-modules (5.14.2-12) ...
Setting up perl (5.14.2-12) ...
Setting up libtext-csv-xs-perl (0.90-1) ...
Setting up libdbi-perl (1.622-1) ...
Setting up libparams-util-perl (1.07-1) ...
Setting up libsql-statement-perl (1.33-1) ...
Setting up libdbd-csv-perl (0.3500-1) ...
#

And don't get me started about waiting for aptitude *upgrade to resolve its dependency graph...

Enrico Zini: More diversity in Debian skills

More diversity in Debian skills This blog post has been co-authored with Francesca Ciceri. In his Debconf talk, zack said:
We need to understand how to invite people with different backgrounds than packaging to join the Debian project [...] I don't know what exactly, but we need to do more to attract those kinds of people.
Francesca and I know what we could do: make other kinds of contributions visible. Basically, we should track and acknowledge the contributions of webmasters, translators, programmers, sysadmins, event organisers, and so on, at the same level as what we do for packagers: DDPO, minechangelogs, Portfolio... For any non-packaging activity that we can make visible and credited, we get: Here's an example: who's the lead translator for German? And if you are German, who's the lead translator for Spanish? Czech? Thai? I (Enrico) don't know the answers, not even for Italian, but we all should! Or at least it should be trivial to find out. To start to change this, is just a matter of programming. Francesca already worked on a list of trackable data sources, at least for translators. Here are some more details, related to translation: And here are some notes about other fields: And finally, if you are still wondering who those translation coordinators are, they are listed here, although not all teams keep that page up to date. Of course, when a data source is too hard to mine, it can make sense to see if the workflow could be improved, rather than spending months writing compicated mining code. This is a fun project for people at Debconf to get together and try. If by the end of the conference we had a way to credit some group of non-packaging contributors, even if just one like translators or website contributors, at least we would finally have started having official trackers for the activities of non-packagers.

Ritesh Raj Sarraf: Reporting bugs with Apport

So yet another bug reporting tool. :-)When I prepared Apport for Debian, I wasn't sure how it will look going forward. If you look at the current one lying in experimental, it just detects your crashes and pops up in your systray. It doesn't have the mechanism to interact with BTS.So, like the title of this post says, this is the first worthy feature for Apport in the context of Debian.Apport Detail ReportNothing special here. You would have seen a similar window before if you have installed apport. What changes with this release is, that now, when you check the Send an error report to help fix this problem, it will really file a bug report on the BTS.Here's what the emailed bug report will look like:
Package: leafnode
Version: 2.0.0.alpha20090406a-1
=============================
ProblemType: Crash
Architecture: amd64
Date: Tue Jul  3 00:08:02 2012
Dependencies:
 adduser 3.113+nmu3
 base-passwd 3.5.24
 cron 3.0pl1-123
 debconf 1.5.44
 debianutils 4.3.1
 dpkg 1.16.4.3
 gcc-4.7-base 4.7.1-2
 libbz2-1.0 1.0.6-3
 libc-bin 2.13-33
 libc6 2.13-33
 libclass-isa-perl 0.36-3
 libdb5.1 5.1.29-4
 libfile-copy-recursive-perl 0.38-1
 libgcc1 1:4.7.1-2
 libgdbm3 1.8.3-11
 liblzma5 5.1.1alpha+20120614-1
 libpam-modules 1.1.3-7.1
 libpam-modules-bin 1.1.3-7.1
 libpam-runtime 1.1.3-7.1
 libpam0g 1.1.3-7.1
 libpcre3 1:8.30-5
 libpopt0 1.16-7
 libselinux1 2.1.9-5
 libsemanage-common 2.1.6-6
 libsemanage1 2.1.6-6
 libsepol1 2.1.4-3
 libswitch-perl 2.16-2
 libustr-1.0-1 1.0.4-3
 libwrap0 7.6.q-23
 logrotate 3.8.1-4
 lsb-base 4.1+Debian7 [modified: lib/lsb/init-functions]
 multiarch-support 2.13-33
 netbase 5.0
 openbsd-inetd 0.20091229-2
 passwd 1:4.1.5.1-1
 perl 5.14.2-12
 perl-base 5.14.2-12
 perl-modules 5.14.2-12
 sensible-utils 0.0.7
 tar 1.26-4
 tcpd 7.6.q-23
 update-inetd 4.43
 zlib1g 1:1.2.7.dfsg-13
Disassembly:
 => 0x7f8e69738475 <*__GI_raise+53>:	cmp    $0xfffffffffffff000,%rax
    0x7f8e6973847b <*__GI_raise+59>:	ja     0x7f8e69738492 <*__GI_raise+82>
    0x7f8e6973847d <*__GI_raise+61>:	repz retq 
    0x7f8e6973847f <*__GI_raise+63>:	nop
    0x7f8e69738480 <*__GI_raise+64>:	test   %eax,%eax
    0x7f8e69738482 <*__GI_raise+66>:	jg     0x7f8e69738465 <*__GI_raise+37>
    0x7f8e69738484 <*__GI_raise+68>:	test   $0x7fffffff,%eax
    0x7f8e69738489 <*__GI_raise+73>:	jne    0x7f8e697384a2 <*__GI_raise+98>
    0x7f8e6973848b <*__GI_raise+75>:	mov    %esi,%eax
    0x7f8e6973848d <*__GI_raise+77>:	nopl   (%rax)
    0x7f8e69738490 <*__GI_raise+80>:	jmp    0x7f8e69738465 <*__GI_raise+37>
    0x7f8e69738492 <*__GI_raise+82>:	mov    0x34e97f(%rip),%rdx        # 0x7f8e69a86e18
    0x7f8e69738499 <*__GI_raise+89>:	neg    %eax
    0x7f8e6973849b <*__GI_raise+91>:	mov    %eax,%fs:(%rdx)
    0x7f8e6973849e <*__GI_raise+94>:	or     $0xffffffff,%eax
    0x7f8e697384a1 <*__GI_raise+97>:	retq
DistroRelease: Debian 7.0
ExecutablePath: /usr/sbin/fetchnews
ExecutableTimestamp: 1265584779
Package: leafnode 2.0.0.alpha20090406a-1
PackageArchitecture: amd64
ProcCmdline: /usr/sbin/fetchnews
ProcCwd: /
ProcEnviron:
 LANGUAGE=en_US:en
 LC_TIME=en_IN.UTF-8
 LC_MONETARY=en_IN.UTF-8
 PATH=(custom, no user)
 LC_ADDRESS=en_IN.UTF-8
 LANG=en_US.UTF-8
 LC_TELEPHONE=en_IN.UTF-8
 LC_NAME=en_IN.UTF-8
 SHELL=/bin/sh
 LC_MEASUREMENT=en_IN.UTF-8
 LC_NUMERIC=en_IN.UTF-8
 LC_PAPER=en_IN.UTF-8
ProcMaps:
 00400000-00421000 r-xp 00000000 08:06 4464934                            /usr/sbin/fetchnews
 00621000-00622000 rw-p 00021000 08:06 4464934                            /usr/sbin/fetchnews
 00622000-00623000 rw-p 00000000 00:00 0 
 00be4000-00c05000 rw-p 00000000 00:00 0                                  [heap]
 7f8e68edc000-7f8e68eef000 r-xp 00000000 08:06 1179776                    /lib/x86_64-linux-gnu/libresolv-2.13.so
 7f8e68eef000-7f8e690ee000 ---p 00013000 08:06 1179776                    /lib/x86_64-linux-gnu/libresolv-2.13.so
 7f8e690ee000-7f8e690ef000 r--p 00012000 08:06 1179776                    /lib/x86_64-linux-gnu/libresolv-2.13.so
 7f8e690ef000-7f8e690f0000 rw-p 00013000 08:06 1179776                    /lib/x86_64-linux-gnu/libresolv-2.13.so
 7f8e690f0000-7f8e690f2000 rw-p 00000000 00:00 0 
 7f8e690f2000-7f8e690f7000 r-xp 00000000 08:06 1180036                    /lib/x86_64-linux-gnu/libnss_dns-2.13.so
 7f8e690f7000-7f8e692f6000 ---p 00005000 08:06 1180036                    /lib/x86_64-linux-gnu/libnss_dns-2.13.so
 7f8e692f6000-7f8e692f7000 r--p 00004000 08:06 1180036                    /lib/x86_64-linux-gnu/libnss_dns-2.13.so
 7f8e692f7000-7f8e692f8000 rw-p 00005000 08:06 1180036                    /lib/x86_64-linux-gnu/libnss_dns-2.13.so
 7f8e692f8000-7f8e692fa000 r-xp 00000000 08:06 1183392                    /lib/libnss_mdns4_minimal.so.2
 7f8e692fa000-7f8e694f9000 ---p 00002000 08:06 1183392                    /lib/libnss_mdns4_minimal.so.2
 7f8e694f9000-7f8e694fa000 rw-p 00001000 08:06 1183392                    /lib/libnss_mdns4_minimal.so.2
 7f8e694fa000-7f8e69505000 r-xp 00000000 08:06 1180055                    /lib/x86_64-linux-gnu/libnss_files-2.13.so
 7f8e69505000-7f8e69704000 ---p 0000b000 08:06 1180055                    /lib/x86_64-linux-gnu/libnss_files-2.13.so
 7f8e69704000-7f8e69705000 r--p 0000a000 08:06 1180055                    /lib/x86_64-linux-gnu/libnss_files-2.13.so
 7f8e69705000-7f8e69706000 rw-p 0000b000 08:06 1180055                    /lib/x86_64-linux-gnu/libnss_files-2.13.so
 7f8e69706000-7f8e69883000 r-xp 00000000 08:06 1179700                    /lib/x86_64-linux-gnu/libc-2.13.so
 7f8e69883000-7f8e69a83000 ---p 0017d000 08:06 1179700                    /lib/x86_64-linux-gnu/libc-2.13.so
 7f8e69a83000-7f8e69a87000 r--p 0017d000 08:06 1179700                    /lib/x86_64-linux-gnu/libc-2.13.so
 7f8e69a87000-7f8e69a88000 rw-p 00181000 08:06 1179700                    /lib/x86_64-linux-gnu/libc-2.13.so
 7f8e69a88000-7f8e69a8d000 rw-p 00000000 00:00 0 
 7f8e69a8d000-7f8e69a95000 r-xp 00000000 08:06 1180031                    /lib/x86_64-linux-gnu/libcrypt-2.13.so
 7f8e69a95000-7f8e69c94000 ---p 00008000 08:06 1180031                    /lib/x86_64-linux-gnu/libcrypt-2.13.so
 7f8e69c94000-7f8e69c95000 r--p 00007000 08:06 1180031                    /lib/x86_64-linux-gnu/libcrypt-2.13.so
 7f8e69c95000-7f8e69c96000 rw-p 00008000 08:06 1180031                    /lib/x86_64-linux-gnu/libcrypt-2.13.so
 7f8e69c96000-7f8e69cc4000 rw-p 00000000 00:00 0 
 7f8e69cc4000-7f8e69cc6000 r-xp 00000000 08:06 1180047                    /lib/x86_64-linux-gnu/libdl-2.13.so
 7f8e69cc6000-7f8e69ec6000 ---p 00002000 08:06 1180047                    /lib/x86_64-linux-gnu/libdl-2.13.so
 7f8e69ec6000-7f8e69ec7000 r--p 00002000 08:06 1180047                    /lib/x86_64-linux-gnu/libdl-2.13.so
 7f8e69ec7000-7f8e69ec8000 rw-p 00003000 08:06 1180047                    /lib/x86_64-linux-gnu/libdl-2.13.so
 7f8e69ec8000-7f8e69ed5000 r-xp 00000000 08:06 1179893                    /lib/x86_64-linux-gnu/libpam.so.0.83.0
 7f8e69ed5000-7f8e6a0d4000 ---p 0000d000 08:06 1179893                    /lib/x86_64-linux-gnu/libpam.so.0.83.0
 7f8e6a0d4000-7f8e6a0d5000 r--p 0000c000 08:06 1179893                    /lib/x86_64-linux-gnu/libpam.so.0.83.0
 7f8e6a0d5000-7f8e6a0d6000 rw-p 0000d000 08:06 1179893                    /lib/x86_64-linux-gnu/libpam.so.0.83.0
 7f8e6a0d6000-7f8e6a112000 r-xp 00000000 08:06 1182916                    /lib/x86_64-linux-gnu/libpcre.so.3.13.1
 7f8e6a112000-7f8e6a312000 ---p 0003c000 08:06 1182916                    /lib/x86_64-linux-gnu/libpcre.so.3.13.1
 7f8e6a312000-7f8e6a313000 rw-p 0003c000 08:06 1182916                    /lib/x86_64-linux-gnu/libpcre.so.3.13.1
 7f8e6a313000-7f8e6a333000 r-xp 00000000 08:06 1180134                    /lib/x86_64-linux-gnu/ld-2.13.so
 7f8e6a503000-7f8e6a507000 rw-p 00000000 00:00 0 
 7f8e6a530000-7f8e6a532000 rw-p 00000000 00:00 0 
 7f8e6a532000-7f8e6a533000 r--p 0001f000 08:06 1180134                    /lib/x86_64-linux-gnu/ld-2.13.so
 7f8e6a533000-7f8e6a534000 rw-p 00020000 08:06 1180134                    /lib/x86_64-linux-gnu/ld-2.13.so
 7f8e6a534000-7f8e6a535000 rw-p 00000000 00:00 0 
 7fffbc06e000-7fffbc08f000 rw-p 00000000 00:00 0                          [stack]
 7fffbc1ff000-7fffbc200000 r-xp 00000000 00:00 0                          [vdso]
 ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
ProcStatus:
 Name:	fetchnews
 State:	S (sleeping)
 Tgid:	6872
 Pid:	6872
 PPid:	6871
 TracerPid:	0
 Uid:	9	9	9	9
 Gid:	9	9	9	9
 FDSize:	64
 Groups:	9 
 VmPeak:	   21440 kB
 VmSize:	   21276 kB
 VmLck:	       0 kB
 VmPin:	       0 kB
 VmHWM:	     984 kB
 VmRSS:	     984 kB
 VmData:	     380 kB
 VmStk:	     136 kB
 VmExe:	     132 kB
 VmLib:	    2132 kB
 VmPTE:	      64 kB
 VmSwap:	       0 kB
 Threads:	1
 SigQ:	0/23227
 SigPnd:	0000000000000000
 ShdPnd:	0000000000000000
 SigBlk:	0000000000000000
 SigIgn:	0000000000000000
 SigCgt:	0000000000000000
 CapInh:	0000000000000000
 CapPrm:	0000000000000000
 CapEff:	0000000000000000
 CapBnd:	ffffffffffffffff
 Cpus_allowed:	f
 Cpus_allowed_list:	0-3
 Mems_allowed:	00000000,00000001
 Mems_allowed_list:	0
 voluntary_ctxt_switches:	6
 nonvoluntary_ctxt_switches:	1
Registers:
 rax            0x0	0
 rbx            0x0	0
 rcx            0xffffffffffffffff	-1
 rdx            0x6	6
 rsi            0x1ad8	6872
 rdi            0x1ad8	6872
 rbp            0x0	0x0
 rsp            0x7fffbc08d4f8	0x7fffbc08d4f8
 r8             0x7f8e6a504700	140249645729536
 r9             0x6d6f642064656966	7885631562835126630
 r10            0x8	8
 r11            0x246	582
 r12            0x0	0
 r13            0x7fffbc08d920	140736348084512
 r14            0x0	0
 r15            0x0	0
 rip            0x7f8e69738475	0x7f8e69738475 <*__GI_raise+53>
 eflags         0x246	[ PF ZF IF ]
 cs             0x33	51
 ss             0x2b	43
 ds             0x0	0
 es             0x0	0
 fs             0x0	0
 gs             0x0	0
Signal: 6
SourcePackage: leafnode
Stacktrace:
 #0  0x00007f8e69738475 in *__GI_raise (sig=<optimized out>) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
         pid = <optimized out>
         selftid = <optimized out>
 #1  0x00007f8e6973b6f0 in *__GI_abort () at abort.c:92
         act =  __sigaction_handler =  sa_handler = 0x7f8e69850d3d, sa_sigaction = 0x7f8e69850d3d , sa_mask =  __val =  140736348083740, 140249634747968, 0, 0, 140736348084512, 140249631083424, 140249645738440, 0, 4294967295, 206158430232, 1, 6427272, 0, 0, 0, 0 , sa_flags = 1781664242, sa_restorer = 0x1 
         sigs =  __val =  32, 0 <repeats 15 times> 
 #2  0x0000000000416292 in ?? ()
 No symbol table info available.
 #3  0x0000000000411c80 in ?? ()
 No symbol table info available.
 #4  0x0000000000406952 in ?? ()
 No symbol table info available.
 #5  0x00007f8e69724ead in __libc_start_main (main=<optimized out>, argc=<optimized out>, ubp_av=<optimized out>, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffbc08d918) at libc-start.c:228
         result = <optimized out>
         unwind_buf =  cancel_jmp_buf =  jmp_buf =  0, 498162418289285118, 4206480, 140736348084512, 0, 0, -498021567843461122, -435434157313288194 , mask_was_saved = 0 , priv =  pad =  0x0, 0x0, 0x418360, 0x7fffbc08d928 , data =  prev = 0x0, cleanup = 0x0, canceltype = 4293472 
         not_first_call = <optimized out>
 #6  0x0000000000402fb9 in ?? ()
 No symbol table info available.
 #7  0x00007fffbc08d918 in ?? ()
 No symbol table info available.
 #8  0x000000000000001c in ?? ()
 No symbol table info available.
 #9  0x0000000000000001 in ?? ()
 No symbol table info available.
 #10 0x00007fffbc08ee96 in ?? ()
 No symbol table info available.
 #11 0x0000000000000000 in ?? ()
 No symbol table info available.
StacktraceTop:
 ?? ()
 ?? ()
 ?? ()
 __libc_start_main (main=<optimized out>, argc=<optimized out>, ubp_av=<optimized out>, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffbc08d918) at libc-start.c:228
 ?? ()
ThreadStacktrace:
 .
 Thread 1 (LWP 6872):
 #0  0x00007f8e69738475 in *__GI_raise (sig=<optimized out>) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
         pid = <optimized out>
         selftid = <optimized out>
 #1  0x00007f8e6973b6f0 in *__GI_abort () at abort.c:92
         act =  __sigaction_handler =  sa_handler = 0x7f8e69850d3d, sa_sigaction = 0x7f8e69850d3d , sa_mask =  __val =  140736348083740, 140249634747968, 0, 0, 140736348084512, 140249631083424, 140249645738440, 0, 4294967295, 206158430232, 1, 6427272, 0, 0, 0, 0 , sa_flags = 1781664242, sa_restorer = 0x1 
         sigs =  __val =  32, 0 <repeats 15 times> 
 #2  0x0000000000416292 in ?? ()
 No symbol table info available.
 #3  0x0000000000411c80 in ?? ()
 No symbol table info available.
 #4  0x0000000000406952 in ?? ()
 No symbol table info available.
 #5  0x00007f8e69724ead in __libc_start_main (main=<optimized out>, argc=<optimized out>, ubp_av=<optimized out>, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffbc08d918) at libc-start.c:228
         result = <optimized out>
         unwind_buf =  cancel_jmp_buf =  jmp_buf =  0, 498162418289285118, 4206480, 140736348084512, 0, 0, -498021567843461122, -435434157313288194 , mask_was_saved = 0 , priv =  pad =  0x0, 0x0, 0x418360, 0x7fffbc08d928 , data =  prev = 0x0, cleanup = 0x0, canceltype = 4293472 
         not_first_call = <optimized out>
 #6  0x0000000000402fb9 in ?? ()
 No symbol table info available.
 #7  0x00007fffbc08d918 in ?? ()
 No symbol table info available.
 #8  0x000000000000001c in ?? ()
 No symbol table info available.
 #9  0x0000000000000001 in ?? ()
 No symbol table info available.
 #10 0x00007fffbc08ee96 in ?? ()
 No symbol table info available.
 #11 0x0000000000000000 in ?? ()
 No symbol table info available.
Title: fetchnews crashed with SIGABRT in __libc_start_main()
Uname: Linux 3.4-trunk-amd64 x86_64
UserGroups: 
The first 2 lines should be enough for the BTS server to file the bug report to the correct package and against the correct version.If you paid attention in the screen shot and the email report, you will notice that the email report does not include the CoreDump section. This was knocked off intentionally as we currently might not need it. If there is a need, the data is always available on the user's apport database, and the dev/user and seek it on a need basis. But in case, if a day comes when we really want it, that too is possible. The same email report will have the CoreDump seciton with a section like the following:
CoreDump: base64 
H4sICAAAAAAC/0NvcmVEdW1wAOx9C0AU1f7/7LLAggirouF7VDQ0xcVXWGqLoq4Kur6KMmV5CoqwwqJYVquggYqhlVm3B3mt7OWlbnXtZeubHtfobc9Lb6xUemiU5f7PmfkcmBl3VMzbvff3Px89fPb7nfOeM2fOOfOdMzeNT5pgNBgEBpMwRmiRBMEmnA6bEC8MbfYvI6eN4BePBfvX0zQC6Q9RJyULY6NKpuGC6A8rO56uE07wH44lI7YynINlc6cmnFE3n1L5nNB7j+Ssv8lPelbDaeEkuBCu8ag6HENjz9PCmZTh6sNz/aanVz4PS6+V4aoQTohQhxM1rK2XGoQTI/yn5xD814sX4VyacGerFxbOObh15atj6Z17OKl89Qjn0Qkn6pSvEeGqB59z+YKU4aqmtS6fQgDS0wlXo5NPC8I5HK07DyyczdW68yCy9FytK58V4Zw64eou8l8+G8JZy/2Xj51Abfmaw21Th7Np+LTrD+Fc21p5/SGcRxPOoX8dye0T4ep00vPotU92Hh5u3Xln4Wwvt6587A7jaGU4C8I5X/afz3qd/pqFs73WuutWZOm91rr20hzuu9aVz4pwru9adz3YEM6jE65Op3wOdh6OtO68s3C2tgtaVT4nS+/cw0nlc7H61AnnDfBfPg/Cie0WnOv5C1KFi2tdPqsQzqoTTtAZ91SzcI4Frepf6tlIbeaCc71PyyECcXxm686fBeGsrQwnIpytleGsCFcX9IJPVT6jmrXnwRHM2q863NnOn5MNbJ0+irOFo2EcBjn8uGkzxgvN/ZuMID8t7lAXQXiPuHfhLjTE983qMmswAVVmhpydm5UpZhYsEiaq254PCFLkW8nfQF9cViT1HUM18Te9LefjEhahJv6zgV3DtI7LMaJhdSzMwMFo+cIIgdPWbXaWOyMnP2tpEbwPLi4qHFyUnps/uPmI+EfqOhBD/SBF3nq19J1S+VmT8Z389iVVm8CBdhAzwQHNl 7xNNReLV6SpPH/NUzH0OWZNHkPAVYPl+Npo9GEaua1GDtfIF2ni7wz+5QP5fOOOIXxxTJYjWPgP/bfLQHSLRkUd3hTgv749PWm7FP7r0FcDP15U/cmO9fL4jLFNgzkaiBr4zwWL//QaYtey3vH/NtBrxEbbBHFJk6bOTuFt4v9cmwj4N/vn4ODg4ODg4ODg4OD4/wE3aZ7/G/H8n60B2aDfFmVs9kOf/5vJ325CV2n+HSgol59tKvYiasZmxRzNJCeIhG0q7gY1Y4OCA1UlsKl4b5hRxWxhu2Vdmq0Dp6vYi8UxS5SgCmdEuBiEi4F

Now, Is this going to be useful? Will we get flooded with bug reports?
 
Usefulness: I think this will uncover many bugs that might be getting unnoticed today. Take this example itself. This bug was detect against the leafnode package's fetchnews binary. That binary does a quiet job in the background, fetching news. I never was aware that it had been crashing. As a system-wide monitoring tool, apport was able to detect, trap and inform the user of the crash. So I think this will be useful and improve the overall quality.
Abuse/Flood: This is something I can't predict. Perhaps it will bring in challenges if people just blindly click on the Send Report button. One option can be to have the "Send Report" checkbox unchecked by default. That'll hopefully lower down the possibilities.
 
What do you think? Let me know your comments.
This change will soon be pushed to apport in experimental. Hopefully this weekend.
 
I want to wrap up this post with a Thank You note to Martin Pitt and his team who created Apport, and with such simplicity. It took me 30 lines of code to adapt it to Debian. My intent with Apport is to keep the changes to the minimum so that we can always leverage new features and fixes that Apport brings in. So, as far as I'll try, there will be no drift from its original shape.
Categories:
Keywords:

Craig Small: debhelper versions Guilty!

After reading Jakub s pet peeve about debhelper build-dependencies I decided to check my own and sponsored packages to see how they fare.
find debian/ -type f -name control   xargs grep -h -o 'debhelper (&gt;= 9[^)]*)'   sort   uniq -c
      2 debhelper (&gt;= 9)
      2 debhelper (&gt;= 9.0)
      3 debhelper (&gt;= 9.0.0)
      1 debhelper (&gt;= 9.20120115)
Oops! Something to fix in all subsequent releases.

11 July 2012

Dirk Eddelbuettel: Getting numpy data into R -- Take Two

A couple of days ago, I had posted a short Python script to convert numpy files into a simple binary format which R can read quickly. Nice, but still needing an extra file. Shortly thereafter, I found Carl Rogers cnpy library which makes reading and writing numpy files from C++ a breeze, and I quickly wrapped this up into a new package RcppCNPy which was released a few days ago. This post will show a quick example, also summarized in the short pdf vignette describing the package, and provided as a demo within the package.
R> library(RcppCNPy)
Loading required package: Rcpp
R> library(rbenchmark)
R> 
R> n <- 1e5
R> k <- 50
R> 
R> M <- matrix(seq(1.0, n*k, by=1.0), n, k)
R> 
R> txtfile <- tempfile(fileext=".txt")
R> write.table(M, file=txtfile)
R> 
R> pyfile <- tempfile(fileext=".py")
R> npySave(pyfile, M)
R> 
R> pygzfile <- tempfile(fileext=".py")
R> npySave(pygzfile, M)
R> system(paste("gzip -9", pygzfile))
R> pygzfile <- paste(pygzfile, ".gz", sep="")
R> 
R> 
We first load the new package (as well as the rbenchmark package used for the benchmarking example) into R. We then create a large matrix of 100,000 rows and 50 columns. Not quite big data by any stretch, but large enough for ascii reading to be painfully slow. We also write two npy files and compress the second one. Next, we use the benchmark function to time the three approaches:
R> res <- benchmark(read.table(txtfile),
+                  npyLoad(pyfile),
+                  npyLoad(pygzfile),
+                  order="relative",
+                  columns=c("test", "replications", "elapsed", "relative"),
+                  replications=10)
R> print(res)
                 test replications elapsed relative
2     npyLoad(pyfile)           10   1.241  1.00000
3   npyLoad(pygzfile)           10   3.098  2.49637
1 read.table(txtfile)           10  96.744 77.95649
R> 
As shown by this example, loading a numpy file directly beats the pants off reading the data from ascii: it is about 78 times faster. Reading a compressed file is somewhat slower as the data stream has to be passed through the uncompressor provide by the zlib library. So instead of reading a binary blob in one go (once the file header has been parsed) we have to operate piecemeal---which is bound to be slower. It does however save in storage space (and users can make this tradeoff between speed and size) and is still orders of magnitude faster than parsing the ascii file. Finally, and not shown here, we unlink the temporary files. Summing up, this post demonstrated how the RcppCNPy package can be a useful to access data in numpy files (which may even be compressed). Data can also be written from R to be accessed later by numpy.

10 July 2012

Keith Packard: hotplug-displaylink

Getting Hotplugging working with a DisplayLink USB to DVI adapter I merged Dave Airlie s randr provider patches to the server and pushed them out in preparation for freezing the X server for the version 1.13 release. Before freezing, I figured I should at least test hotplugging my DisplayLink adapter. Well, that took all day Sources For the impatient, here s where all of the bits are. These work on my machine. Upstream bits:
  1. randrproto. git://anongit.freedesktop.org/git/xorg/proto/randrproto master
Bits from Dave Airlie s trees:
  1. libdrm. git://people.freedesktop.org/~airlied/drm.git prime
  2. libXrandr. git://people.freedesktop.org/~airlied/libXrandr.git prime
  3. xrandr. git://people.freedesktop.org/~airlied/xrandr.git prime
  4. xf86-video-intel. git://people.freedesktop.org/~airlied/xf86-video-intel prime
Bits from my trees:
  1. xf86-video-modesetting. git://people.freedesktop.org/~keithp/xf86-video-modesetting prime
  2. kernel. git://people.freedesktop.org/~keithp/linux prime
  3. X server. git://people.freedesktop.org/~keithp/xserver master
Kernel adventures The 3.5-rc6 bits have all of the necessary driver changes to support DisplayLink hot-plug. At least, they do if you re not running 32-bit user space atop a 64-bit kernel. If you are, then most of the DRM drivers don t work at all they re missing the .compat_ioctl entry in the file_operations structure, which makes all ioctl calls return -ENOTTY. In particular, the udl driver is missing this entry, so when you plug in the device, the server tries to talk to it and nothing works. X server adventures Once I got the kernel working, I discovered some nice crashing behaviour in the X server. If I started the server with the DisplayLink device plugged in, as soon as I tried to enable the output, the server would segfault. Turns out this cases requires special code in the server to ensure that the DisplayLink screen privates are adjust correctly while the intel driver loads. Modesetting driver fixup The modesetting driver was missing a rename that happened to a structure member in the last couple of days. I d bet Dave has the same patch sitting on a disk somewhere. xrandr changes The command line additions to xrandr are sufficient to get this stuff working, but I think it could use some improvements to make the interface a bit friendlier. In particular, requiring you to use XIDs to identify providers is a bit harsh. Schedules I ll be pushing the X server changes out this evening; review would be appreciated, but I don t think any of the patches I made there are scary. I sent the kernel patches necessary to lkml, but with Dave in transit, I m not sure who is minding the store. As for the rest of the bits, they re sitting in Dave s repositories, presumably they ll get pushed upstream soon. But, does it actually work? Almost. Everything says it s working, but I m not getting any signal to my DVI monitor. My DisplayLink device is ancient, and the kernel complains about it, saying
[drm:udl_parse_vendor_descriptor] *ERROR* Unrecognized vendor firmware descriptor".
So close, and yet

9 July 2012

Julien Danjou: Logitech K750 keyboard and Unifying Receiver Linux support

A year ago, I bought a Logitech Wireless Solar Keyboard K750. I'm particularly picky on keyboards, but this one is good. It has an incredible useful feature: while being wireless, it has no need for disposable or rechargeable batteries, it uses solar power! Logitech Wireless Solar Keyboard K750 My problem is that there's obviously no way to know the battery status from Linux, the provided application only working on Windows. And one dark night, while fragging on QuakeLive, my keyboard stopped working: it had no battery left. This activity being quite energy consuming, it emptied the whole battery. Someone should write code to get the battery status and light meter from Linux: challenge accepted! How the keyboard works Logitech Unifying Receiver This keyboard, like many of the new wireless devices from Logitech, uses the Unifying interface. It's an USB receiver that can be attached up to 6 differents devices (mouse, keyboards ). On old Linux kernel, the Unifying receiver is recognized as only one keyboard and/or one mouse device. Recently, a driver called hid-logitech-dj has been added to the Linux kernel. With this driver, each device attached to the receiver is recognized as one different device. What the Logitech application does Logitech Solar App The Logitech application under Windows works that way: you launch it, and it displays the battery charge level. On the keyboard, there's a special "light" button (up right). When pressed, a LED will light up on the keyboard: green if the keyboard is receiving enough light and is charging, red if the keyboard does not receive enough light and is therefore discharging. Pushing this same button while the application is running will makes the light meter activated: the application will tell you how much lux your keyboard is receiving. Let's reverse engineer this As far as I know, there's nothing in the USB HID protocol that handles this kind of functionality (battery status, light meter ) in a standard way. So the first task to accomplish is, unfortunately, to reverse engineer the program. I discovered a bit too late that Drew Fisher did a good presentation on USB reverse engineering at 28c3. You might want to take a look at it if you want to reverse engineer on USB. I did not need it, but I learned a few things. Anyway, my plan was the following: run the Logitech application inside a virtual machine running Windows, give it direct access to the USB keyboard, and sniff what happens on the USB wire. To achieve that, you need a virtual machine emulator that can do USB pass-through. Both KVM and VirtualBox can do that, but VirtualBox works much better with USB and allow hot(un)plugging of devices, so I used it. To sniff what happens on the USB, you need to load the usbmon Linux kernel module. Simply doing modprobe usbmon will work. You can then use Wireshark which know how to use usbmon devices and understand the USB protocol. USB stuff you need to know You don't need to know much about USB to understand what I'll write about below, but for the sake of comprehensibility I'll write a couple of things here before jumping in. To communicate with an USB device, we communicate with one of its endpoints. Endpoints are regrouped into an interface. Interfaces are regrouped into a configuration. A device might contains one or several configurations. There's also several types of packets in the USB wire protocol, and at least two of them interest us there, they are: All of this and more is well (and better) explained in the chapter 13 of Linux Device Drivers, Third Edition. Sniffed data Once everything was set-up, I ran my beloved Wireshark. There's a an URB of type interrupt sent each time you press any key with some data in it. Therefore I advise you to plug another keyboard (or use the laptop keyboard if you're doing this on a laptop), otherwise you'll get crazy trying to sniff the keyboard you're typing on. At this point, just launching the application does a bunch of USB traffic. Pressing the "light" button on the keyboard makes even more USB packets coming in and out. Here's the interesting packets that I noticed once I excluded the noise: With all this, the next step was clear: understand the packets and reproduce that exchange under Linux. What the packets mean The "go for the light meter" packet The packet sent from the computer to the keyboard is the following. <figure>
Frame 17: 71 bytes on wire (568 bits), 71 bytes captured (568 bits)
    Frame Length: 71 bytes (568 bits)
    Capture Length: 71 bytes (568 bits)
USB URB
    URB id: 0xffff880161997240
    URB type: URB_SUBMIT ('S')
    URB transfer type: URB_CONTROL (0x02)
    Endpoint: 0x00, Direction: OUT
        0... .... = Direction: OUT (0)
        .000 0000 = Endpoint value: 0
    Device: 6
    URB bus id: 1
    Device setup request: relevant (0)
    Data: present (0)
    URB sec: 1340124450
    URB usec: 495643
    URB status: Operation now in progress (-EINPROGRESS) (-115)
    URB length [bytes]: 7
    Data length [bytes]: 7
    [Response in: 18]
    [bInterfaceClass: HID (0x03)]
    URB setup
        bmRequestType: 0x21
            0... .... = Direction: Host-to-device
            .01. .... = Type: Class (0x01)
            ...0 0001 = Recipient: Interface (0x01)
    bRequest: SET_REPORT (0x09)
    wValue: 0x0210
        ReportID: 16
        ReportType: Output (2)
    wIndex: 2
    wLength: 7
0000  40 72 99 61 01 88 ff ff 53 02 00 06 01 00 00 00   @r.a....S.......
0010  22 ad e0 4f 00 00 00 00 1b 90 07 00 8d ff ff ff   "..O............
0020  07 00 00 00 07 00 00 00 21 09 10 02 02 00 07 00   ........!.......
0030  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
0040  10 01 09 03 78 01 00                              ....x..
</figure> What's here interesting is the last part representing the data. wLength says that the length of the data is 7 bytes, so let's take a look at those 7 bytes: 10 01 09 03 78 01 00. Well, actually, you can't decode them like that, unless you're a freak or a Logitech engineer. And I have actually no idea what they mean. But sending this to the keyboard will trigger an interesting thing: the keyboard will start sending URB interrupt with some data without you pressing any more key. The "light meter and battery values" packet This is most interesting packet. This is the one sent by the keyboard to the host and that contains the data we want to retrieve. <figure>
Frame 1467: 84 bytes on wire (672 bits), 84 bytes captured (672 bits)
    Frame Length: 84 bytes (672 bits)
    Capture Length: 84 bytes (672 bits)
USB URB
    URB id: 0xffff88010c43c380
    URB type: URB_COMPLETE ('C')
    URB transfer type: URB_INTERRUPT (0x01)
    Endpoint: 0x83, Direction: IN
        1... .... = Direction: IN (1)
        .000 0011 = Endpoint value: 3
    Device: 2
    URB bus id: 6
    Device setup request: not relevant ('-')
    Data: present (0)
    URB sec: 1334953309
    URB usec: 728740
    URB status: Success (0)
    URB length [bytes]: 20
    Data length [bytes]: 20
    [Request in: 1466]
    [Time from request: 0.992374000 seconds]
    [bInterfaceClass: Unknown (0xffff)]
Leftover Capture Data: 1102091039000c061d474f4f4400000000000000
0000  80 c3 43 0c 01 88 ff ff 43 01 83 02 06 00 2d 00   ..C.....C.....-.
0010  5d c5 91 4f 00 00 00 00 a4 1e 0b 00 00 00 00 00   ]..O............
0020  14 00 00 00 14 00 00 00 00 00 00 00 00 00 00 00   ................
0030  02 00 00 00 00 00 00 00 00 02 00 00 00 00 00 00   ................
0040  11 02 09 10 39 00 0c 06 1d 47 4f 4f 44 00 00 00   ....9....GOOD...
0050  00 00 00 00                                       ....
</figure> This packets come in regularly (1 per second) on the wire for some time once you sent the "go for the light meter" packet. At one point they are emitted less often and do not contain the value for the light meter anymore, suggesting that the control packet sent earlier triggers the activation of the light meter for a defined period. Now you probably wonder where the data are in this. They're in the 20 bytes leftover in the capture data part, indicated by Wireshark, at the end of the packet: 11 02 09 10 39 00 0c 06 1d 47 4f 4f 44 00 00 00 00 00 00 00. Fortunately, it was easy to decode. Knowing we're looking for 2 values (battery charge and light meter), we just need to observe and compare the packet emitted on the wire with the values displayed by the Logitech Solar App. To achieve this, I looked both at the Logitech Solar App and Wireshark while bringing more and more light near the keyboard, increasing the lux value received by the meter on the Solar App, and saw that the fields represented in blue (see below) where changing in Wireshark. Since 2 bytes were changing, I guessed that it was coded on 16 bits, and therefore it was easy to correlate the value with the Solar App. <figure>
[ ....9....GOOD....... ]
11 02 09 10 39 00 0c 06 1d 47 4f 4f 44 00 00 00 00 00 00 00
4 bytes - 1 byte for battery charge - 2 bytes for light meter - 2 bytes - 4 bytes for GOOD - 7 bytes
</figure> In this example, the battery has a charge of 0x39 = 57 % and the light meter receives 0x0c = 12 lux of light. It's basically dark, and that makes sense: it was night and the light was off in my office, the only light being the one coming from my screen. I've no idea what the GOOD part of the packet is about, but it's present in every packet and it's actually very handy to recognize such a packet. Therefore I'm considering this as some sort of useful mark for now. For the other bytes, they were always the same (0x11 0x2 0x9 0x10 at the beginning, 7 times 0x00 at the end). The 2 bytes between the light meter and GOOD probably mean something, but I've no idea what for now. Building our solar app Now we've enough information to build our own very basic solar application. We know how to triggers the light meter, and we know how to decode the packets. We're going to write a small application using libusb. Here's a quick example. It's not perfect and does not check for error codes, be careful. <figure>
/* Written by Julien Danjou <julien> in 2012 */
#include <linux/hid.h>
#include <libusb.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
int main(void)
 
    libusb_context *ctx;
    libusb_init(&ctx);
    libusb_set_debug(ctx, 3);
    /* Look at the keyboard based on vendor and device id */
    libusb_device_handle *device_handle = libusb_open_device_with_vid_pid(ctx, 0x046d, 0xc52b);
    fprintf(stderr, "Found keyboard 0x%p\n", device_handle);
    libusb_device *device = libusb_get_device(device_handle);
    struct libusb_device_descriptor desc;
    libusb_get_device_descriptor(device, &desc);
    for(uint8_t config_index = 0; config_index < desc.bNumConfigurations; config_index++)
     
        struct libusb_config_descriptor *config;
        libusb_get_config_descriptor(device, config_index, &config);
        /* We know we want interface 2 */
        int iface_index = 2;
        const struct libusb_interface *iface = &config->interface[iface_index];
        for (int altsetting_index = 0; altsetting_index < iface->num_altsetting; altsetting_index++)
         
            const struct libusb_interface_descriptor *iface_desc = &iface->altsetting[al>tsetting_index];
            if (iface_desc->bInterfaceClass == LIBUSB_CLASS_HID)
             
                libusb_detach_kernel_driver(device_handle, iface_index);
                libusb_claim_interface(device_handle, iface_index);
                unsigned char ret[65535];
                unsigned char payload[] = "\x10\x02\x09\x03\x78\x01\x00";
                if(libusb_control_transfer(device_handle,
                                           LIBUSB_REQUEST_TYPE_CLASS   LIBUSB_RECIPIENT_INTERFACE,
                                           HID_REQ_SET_REPORT,
                                           0x0210, iface_index, payload, sizeof(payload) - 1, 10000))
                 
                    int actual_length = 0;
                    while(actual_length != 20   strncmp((const char *) &ret[9], "GOOD", 4))
                        libusb_interrupt_transfer(device_handle,
                                                  iface_desc->endpoint[0].bEndpointAddress,
                                                  ret, sizeof(ret), &actual_length, 100000);
                    uint16_t lux = ret[5] << 8   ret[6];
                    fprintf(stderr, "Charge: %d %%\nLight: %d lux\n", ret[4], lux);
                 
                libusb_release_interface(device_handle, iface_index);
                libusb_attach_kernel_driver(device_handle, iface_index);
             
         
     
    libusb_close(device_handle);
    libusb_exit(ctx);
 
</figure> What the program is doing is the following: This gives the following:
Found keyboard 0x0x24ec8e0
Charge: 64 %
Light: 21 lux
Challenge accomplished! To be continued Unfortunately, this approach has at least one major drawback. We have to disconnect the Logitech Unifying Receiver from the kernel. That means that while we're waiting for the packet, we're dropping packets corresponding to other events from every connected device (key presses, pointer motions ). In order to solve that, I sent a request for help on the linux-input mailing list. That way, I learned that Logitech is actually using the HID++ protocol to communicate with the devices using the Unifying Receiver. Lars-Dominik Braun managed to get the HID++ specifications from Logitech and published them with their authorization. This opens a whole new world. With that document, I may be able to understand the part I reverse engineered and convert this to a more useful and generic library using the hidraw interface (so we don't have to disconnect the devices from the kernel driver).

Russell Coker: Postfwd and Local Only Email

Over a year ago when I was considering my first Android phone purchase I setup a test account on my mail server so that I could test email clients on phones and tablets. I used a short password because I didn t want to type a lot on small screens and because typing a password into a random system owned by someone else isn t particularly secure anyway. Then I forgot about the account until I noticed that my mail server was sending out spam. Next time I setup such a test account I ll put rules similar to the following in my Postfwd [1] configuration to stop Postfix from sending such messages. That will prevent the test account from receiving mail from outside or sending mail out of the server. The former is optional (getting a few thousand spam messages in an unused test account is no big deal) but the latter is needed to prevent getting my server blacklisted. id=R_test_recipient ; recipient==test@coker.com.au ; sender!~.*@coker.com.au ; action=REJECT
id=R_test_sender ; sender==test@coker.com.au ; recipient!~.*@coker.com.au ; action=REJECT Related posts:
  1. email disclaimers Andre Pang blogs about the annoyance of email disclaimers. For...
  2. Email Passwords I was doing some routine sysadmin work for a client...
  3. Some Postfix Scripts for dealing with Outbound Spamming I ve just written some small scripts to help me manage...

Joey Hess: debian-cd work at DebCamp

I've been working the past two days on debian-cd, which was the main thing (besides git-annex assistant) I planned to work on at DebCamp. Yesterday, Steve McIntyre and I cleaned up some cruft in debian-cd's package lists. This freed somewhere in the area of 30 mb. I also took a grep through d-i and made sure the CDs include packages that d-i can optionally install in some situations. Today, I investigated how Recommends are ordered on the CD, and concluded it's as close to optimal as can be easily achieved. So was not able to save space there, but I did find a way to reorganize the desktop task that avoids needing to include a lot of printing stuff and some other stuff on the first CD. While that helped some, it still didn't get either Gnome or Kde to entirely fit, so getting there will probably involve rebuilding 100 or so packages with xz, if someone decides to do that. So it's stil TBD whether Gnome or Kde will fit on a single CD in Debian wheezy. At this point I think most of us are getting tired of fighting this increasingly losing battle every release, and so other options like only having a desktop DVD are looking more appealing, as they solve the problem long-term. This would also free up the first CD for other interesting use cases, perhaps xfce, or perhaps a CD targeted at server users, and/or containing high-popularity non-desktop packages.

8 July 2012

Petter Reinholdtsen: Debian Edu interview: George Bredberg

The Debian Edu / Skolelinux project have users all over the globe, but until recently we have not known about any users in Norway's neighbour country Sweden. This changed when George Bredberg showed up in March this year on the mailing list, asking interesting questions about how to adjust and scale the just released Debian Edu Wheezy setup to his liking. He granted me an interview, and I am happy to share his answers with you here. Who are you, and how do you spend your days? I'm a 44 year old country guy that have been working 12 years at the same school as 50% IT-manager and 50% Teacher. My educational background is fil.kand in history and religious beliefs, an exam as a "folkhighschool" teacher, that is, for teaching grownups. In Norwegian I believe it's called "Vuxenupplaring". I also have a master in "Technology and social change". So I'm not really a tech guy, I just like to study how humans and technology interact and that is my perspective when working with IT. How did you get in contact with the Skolelinux/Debian Edu project? I have followed the Skolelinux project for quite some time by now. Earlier I tested out the K12-LTSP project, which we used for some time, but I really like the idea of having a distribution aimed to be a complete solution for schools with necessary tools integrated. When K12-LTSP abandoned that idea some years ago, I started to look more seriously into Skolelinux instead. What do you see as the advantages of Skolelinux/Debian Edu? The big point of Skolelinux to me is that it is a complete distribution, ready to install. It has LDAP-support, MS Windows integration tools and so forth already configured, saving an administrator a lot of time and headache. We were using another Linux based thin-client system called Thinlinc, that has served us very well. But that Skolelinux is based on VNC and LTSP, to me, is better when it comes to the kind of multimedia used in schools. That is showing videos from Youtube or educational TV. It is also easier to mix thin clients with workstations, since the user settings will be the same. In our VNC-based solution you had to "beat around the bush" by setting up a second, hidden, home-directory for user settings for the workstations, because they will be different from the ones used on the thin clients. Skolelinux support for diskless workstations are very convenient since a school today often need to use a class room projector showing videos in full screen. That is easily done with a small integrated media computer running as a diskless workstation. You have only two installs to update and configure. One for the thin clients and one for the workstations. Also saving a lot of time. Our old system was also based on Redhat and CentOS. They are both very nice distributions, but they are sometimes painfully slow when it comes to updating multimedia support and multimedia programs (even such as Gimp), leaving us with a bit "oldish" applications. Debian is quicker to update. What do you see as the disadvantages of Skolelinux/Debian Edu? Debian is a bit too quick when it comes to updating. As an example we use old HP terminals as thinclients, and two times already this year (2012) the updates you get from the repositories has stopped sound from working with them. It's a kernel/ALSA issue. So you have to be more careful properly testing the updates before you run them in a production environment. This has never happened with CentOS. I also would like to be able to set my own domain-settings at install time. In Skolelinux they are kind of hard coded into the distribution, when it comes to LDAP and at least samba integration. That is more a cosmetic/translation issue, and not a real problem. Running MS Windows applications within the Skolelinux environment needs to be better supported. That is, running them seamlessly via RDP, and support for single-sign on. That will make the transition to free software easier, because you can keep the applications you really need. No support will make it impossible if you work in a school where some applications can't be open source. As for us we really need to run Adobe InDesign in our journalist classes. We run a journalist education, and is one of the very few non university ones that is ok:d by Svenska journalistf rbundet (Swedish journalist association). Our education gives the pupils the right of membership there, once they are done. This is important if you want to get a job. Adobe InDesign is the program most commonly used in newspapers and magazines. We used Quark Express before, but they seem to loose there market to Adobe. The only "equivalent" to InDesign in the opensource world is Scribus, and its not advanced enough. At least not according to the teacher. I think it would be possible to use it, because they are not supposed to learn a program, they are supposed to learn how to edit and compile a newspaper. But politically at our school we are not there yet. And Scribus lacks a lot of things you find i InDesign. We used even a windows program for sound editing when it comes to the radio-journalist part. The year to come we are going to try Audacity. That software has the same kind of limitations compared to Adobe Audition, but that teacher is a bit more open minded. We have tried Ardour also, but that instead is more like a music studio program, not intended for the kind of editing taking place in a radio studio. Its way to complex and the GUI is to scattered when you only want to cut, make pass-overs, add extra channels and normalise. Those things you can do in Audacity, but its not as easy as in Audition. You have to do more things manually with envelopes, and that is a bit old fashion and timewasting. Its also harder to cut and move sound from one channel to another, which is a thing that you do frequently because you often find yourself needing to rearrange parts of the sound file. So, I am not sure we will succeed in replacing even Audition, but we will try. The problem is the students have certain expectations when they start an education towards a profession. So the programs has to look and feel professional. Good thing with radio, there are many programs out there, that radio studios use, so its not as standardised as Newspaper editing. That means, it does not really matter what program they learn, because once they start working they still have to learn the program the studio uses, so instead focus has to be to learn the editing part without to much focus on a specific software. Which free software do you use daily? Myself I'm running Linux Mint, or Ubuntu these days. I use almost only open source software, and preferably Linux based. When it comes to most used applications its OpenOffice, and Firefox (of course ;) ) Which strategy do you believe is the right one to use to get schools to use free software? To get schools to use free software there has to be good open source software that are windows based, to ease the transition. But it's also very important that the multimedia support is working flawlessly. The problems with Youtube, Twitter, Facebook and whatever will create problems when it comes to both teachers and students. Economy are also important for schools, so using thin clients, as long as they have good multimedia support, is a very good idea. It's also important that the open source software works even for the administration. It's hard to convince the teachers to stick with open source, if the principal has to run Windows. It also creates a problem if some classes has to use Windows for there tasks, since that will create a difference in "status" between classes, so a good support for running windows applications via the thin client (Linux) desktop is essential. At least at our school, where we have mixed level of educations, from high-school to journalist-school. Update 2012-07-09 08:30: Paul Wise tipped me on IRC about three useful sources related to Free Software for radio stations: the LWN article Radio station management with Airtime, Airtime which claim to be a Free open source radio automation software and Rivendell which claim to be complete radio broadcast automation solution. All of them seem useful to the aspiring radio producer.

Next.

Previous.