Search Results: "spa"

30 November 2013

Gunnar Wolf: On errors in exams - Short rant

Blogging from a phone... first time ever. I don't want to forget some specifics for this :) I have just completed an exam to try to enter a postgraduate program (I'll talk more about it once it becomes real ). The exam is administered by CENEVAL, the same evaluation agency Where I presented my graduation equivalency exam some years ago - Only this exam is for all of the postgraduate studies on many national universities and is thus basically just a psychometric test. The exam had 162 questions, all to be filled in a optical reader sheet, on five subjects: mathematical reasoning, Spanish grammar and comprehension, Project management, Computers and technology, And English reading and understanding. It was all in all a fun exam to take, mostly due to the math reasoning part. But... on the Subject I Know I am an expert, I have to complain (and intend to find a easy to do so formally). First, I spotted two absolute mistakes (and answered based on What I knew others would, but knowing the answer is wrong technically). One was a subtlety, on how and why have hard drives should be defragmented (and part of my quip is that it's an obsolete habit, but besides, the answers were all erroneous), but a second one was... just wrong. It asked on what should not be part of an "Internet link" (can only guess they meant An URL). The 4 options were valid parts of a URL - including one very seldom used by most people, but very often by many of us: the @ sign. Anyway, answered it, but my other main gripe is that most of the section was in specific use of Office software. Not only In Office-like, which would be bad enough to begin with, but on specific ways of using Mainly Excel And PowerPoint. Syntax issues, or the name of the menu under which to look for specific functions.
Anyway, I will wait the stipulated 10 days for the exam to be rated, but will anyway look die a way to contact the very opaque and secretive CENEVAL. Not to demand to be better treated, but to try to correct those known mistakes and errors.

Russell Coker: Links November 2013

Shanley wrote an insightful article about microagressions and management [1]. It s interesting to read that and think of past work experiences, even the best managers do it. Bill Stone gave an inspiring TED talk about exploring huge caves, autonamous probes to explore underground lakes (which can be used on Europa) and building a refuelling station on the Moon [2]. Simon Lewis gave an interesting TED talk about consciousness and the technology needed to help him recover from injuries sustained in a serious car crash [3]. Paul Wayper wrote an interesting article about reforming the patent system [4]. He also notes that the patent system is claimed to be protecting the mythical home inventor when it s really about patent trolls (and ex-inventors who work for them). This is similar to the way that ex-musicians work for organisations that promote extreme copyright legislation. Amanda Palmer gave an interesting TED talk about asking for donations/assistance, and the interactions between musicians and the audience [5]. Some part of this are NSFW. Hans Rakers wrote a useful post about how to solve a Dovecot problem with too many files open [6]. His solution was for a Red Hat based system, for Debian you can do the same but by editing /etc/init.d/dovecot. The use of the /proc/N/limits file was interesting, I ve never had a cause to deliberately use that file before. Krebs on Security has an interesting article about Android malware being used to defeat SMS systems to prevent bank fraud [7]. Apparently an infected PC will instruct the user to install an Android app to complete the process. Rick Falkvinge wrote an interesting article about how to apply basic economics terminology to so-called Intellectual Property [8]. Matthew Garrett wrote an interesting post about the way that Ubuntu gets a better result than Debian and Fedora because it has clear fixed goals [9]. He states that many people regard Fedora as a playground to produce a range of niche derivatives , probably a large portion of the Fedora and Debian developers consider this a feature not a bug. Ming Thein wrote an interesting article about the demise of the DSLR [10]. Bruce Schneier wrote an interesting post on the detention of David Miranda by the British authorities [11]. It s mostly speculation as to why they would do such a thing (which seems to go against their own best interests) and whether the NSA even knows which documents Edward Snowden copied. Jaclyn Friedman wrote an interesting article on Mens Rights Movements (MRAs) and how they are bad for MEN as well as for women [12]. Rodney S. Tucker wrote an insightful article for the IEEE about the NBN [13]. Basically the Liberal party are going to spend most of the tax money needed for a full NBN but get a significantly less than the full benefit. Lauren Drell wrote an interesting article for Mashable about TellSpec, a portable spectrometer that communicates with an Android phone to analyse food for allergens [14]. I guess this will stop schools from banning phones. Katie McDonough wrote an interesting article for Salon about the Pope s statements about the problems with unchecked capitalism [15]. His ideas are really nothing new to anyone who has read the Bible and read the news. It seems to me that the most newsworthy part of this is that most Christian leaders don t make similar statements. Daniel Leidert wrote an interesting post about power saving when running Debian on a HP Microserver [16]. Most of it is relevant to other AMD64 hardware too, I ll have to investigate the PCIE ASPM and spin down options on some of my systems that are mostly idle.

29 November 2013

Steve Kemp: A difficult day

Today was my last day working at Bytemark, and I found it a lot harder than expected. For better or worse I finished earlier than expected; having been gradually removing my accounts and privileges over the past few weeks I'd revoked my OpenVPN key this morning. Mid-afternoon my openvpn connection tried to renegotiate session keys, or similar, and failed. So I stopped work a few hours early. That meant I managed to avoid sending my "goodbye world" email, which is probably for the best - after all a lovely company, lovely people, and a good environment, what can you say besides things that are lovely? I think I largely wrapped things up neatly, and I'm pleased that one of my photos is hanging on the office wall. (I look forward to seeing that actually, I've only rarely made canvas prints.) The only other thing of note this week has been the sharp rise in blogspam I've detected. Black Friday alive and well, on the internets ..

Tanguy Ortolo: Mutt: encrypt all messages sent to known PGP users

GnuPG logo This is one thing I have wanted to do for a long time: configure Mutt to encrypt all messages sent to addresses for which I have a valid public key. Well, here is an awk-based script to generate that configuration. (Yes, I know, a similar script was already written. But I did not see it at first, and I find mine more readable. :-) )Code
#! /bin/sh
set -e
# Specific hooks set global configuration values for Mutt, which are not
# reset for new messages, so a default hook is needed to set the default
# value (do not try encrypting all messages).
printf "send-hook . unset crypt_autoencrypt\n\n"
cat <<EOF
# This group contains all known PGP users, and can be used to define
# hooks, for instance to encrypt all messages sent to PGP users, or to
# add a specific signature to promote PGP usage to non-PGP users.
EOF
# Generate a machine-readable list of known keys
gpg2 --batch --with-colons --list-keys \
      awk -- '
        BEGIN  
            FS = ":"
            # A flag that indicates whether or not we are in a public
            # key with usable encryption capability
            usable = 0
         
        # Detect beginning of public keys with usable encryption
        # capability
        $1 == "pub" && $12 ~ /E/   usable = 1  
        # Detect beginning of public keys without usable encryption
        # capability
        $1 == "pub" && $12 !~ /E/   usable = 0  
        
        # Only consider user IDs from public keys with usable encryption
        # capability and with regular, marginal, full or ultimate
        # validity (n, m, f or u in the second field)
        usable && $1 == "uid" && $2 ~ /[nmfu]/ && $10 ~ /@/  
            # Remove everything before and after the email address in
            # brackets
            sub(/^[^<]*</, "", $10)
            sub(/>[^>]*$/, "", $10)
            # Finally print the email address
            print $10
         ' \
      sort -u \
       
        # Group addresses by ten to reduce the number of Mutt commands
        finished=0
        while [ $finished -ne 1 ]
        do
            addresses=""
            for i in 0 1 2 3 4 5 6 7 8 9
            do
                if read address
                then
                    addresses="$addresses $address"
                else
                    finished=1
                    break
                fi
            done
            # Add PGP users' addresses to a pgp-users group
            printf "group -group pgp-users -addr %s\n" \
                "$addresses"
        done
     
# Build a hook to try encrypting messages sent to PGP users
printf "\n# Encrypt all messages sent to known PGP users\n"
printf "send-hook \"%%C pgp-users\" set crypt_autoencrypt\n" \
printf "\n# Add a specific signature to promote PGP usage to non-PGP users\n"
printf "#send-hook \"!%%C pgp-users\" set signature=~/.signature.use-pgp\n"
Usage Script usage This script generates uses GnuPG to print a list of all known keys, takes only valid ones, and uses them to build a series of configuration instructions for Mutt, that define hooks that are triggered when you write a message to them. This configuration has to be written to your Mutt configuration file:
$ bin/gen-pgp-hooks.sh >> ~/.muttrc
Or better (far better, in fact), to a dedicated configuration file you will refer to in your main one:
$ bin/gen-pgp-hooks.sh > ~/.muttrc.crypt-recipients
$ echo "source ~/.muttrc.crypt-recipients" >> ~/.muttrc
You may want to adapt this script to it directly writes to that file rather than having to redirect it. Also, remember to run it when you refresh your public keyring or add new keys to it. Mutt usage These hooks will make Mutt try to encrypt all messages you send to at least one known PGP user. That means it will try it even for messages you send to a PGP user and to non-PGP ones. In these cases, it will not find keys for all the recipient, and ask you which key to use; as you have none, you will have to quit that prompt by typing ^G, and manually disable encryption by accessing the PGP options menu with the key p.

27 November 2013

Timo Jyrinki: Workaround for setting Full RGB when Intel driver's Automatic setting does not work

Background I upgraded from Linux 3.8 to 3.11 among with newer Mesa, X.Org and Intel driver recently and I found a small workaround was needed because of upstream changes.

The upstream change was the Add "Automatic" mode for "Broadcast RGB" property, and defaulting to the Automatic. This is a sensible default, since many (most?) TVs default to the more limited 16-235, and continuing to default to Full from the driver side would mean wrong colors on the TV. I've set my screen to support the full 0-255 range available to not cut the amount of available shades of colors down.

Unfortunately it seems the Automatic setting does not work for my HDMI input, ie blacks become grey since the driver still outputs the more limited range. Maybe there could be something to improve on the driver side, but I'd guess it's more about my 2008 Sony TV actually having a mode that the standard suggests limited range for. I remember the TV did default to limited range, so maybe the EDID data from TV does not change when setting the RGB range to Full.

I hope the Automatic setting works to offer full range on newer screens and the modes they have, but that's probably up to the manufacturers and standards.

Below is an illustration of the correct setting on my Haswell CPU. When the Broadcast RGB is left to its default Automatic setting, the above image is displayed. When set to Full, the image below with deeper blacks is seen instead. I used manual settings on my camera so it's the same exposure.


Workaround For me the workaround has evolved to the following so far. Create a /etc/X11/Xsession.d/95fullrgb file:

if [ "$(/usr/bin/xrandr -q --prop   grep 'Broadcast RGB: Full'   wc -l)" = "0" ] ; then
/usr/bin/xrandr --output HDMI3 --set "Broadcast RGB" "Full"
fi
And since I'm using lightdm, adding the following to /etc/lightdm/lightdm.conf means the flicker only happens once during bootup:

display-setup-script=/etc/X11/Xsession.d/95fullrgb

Important: when using the LightDM setting, enable executable bits (chmod +x) to /etc/X11/Xsession.d/95fullrgb for it to work. Obviously also check your output, for me it was HDMI3.

If there is no situation where it'd set back to "Limited 16:235" setting on its own, the display manager script should be enough and having it in /etc/X11/Xsession.d is redundant and slows login time down. I think for me it maybe went from 2 seconds to 3 seconds since executing xrandr query is not cheap.
Misc Note that unrelated to Full range usage, the Limited range at the moment behaves incorrectly on Haswell until the patch in bug #71769 is accepted. That means, the blacks are grey in Limited mode even if the screen is also set to Limited.

I'd prefer there would be a kernel parameter for the Broadcast RGB setting, although my Haswell machine does boot so fast I don't get to see too many seconds of wrong colors...

23 November 2013

Steve Kemp: Things have settled down nicely

I've now completed all my KVM migrations. Moving my personal virtual machines from one host to another. There were a few niggles, for example I didn't have a working IPv6 allocation at the time I moved things so I had to set that up post-migration. I've also joined each of the hosts into a VPN which makes cross-guest communication secure and simple. Finally I've overhauled my firewalls and service lists. I installed a couple of extra guests, using libvirt and booting from the Debian ISO. The Debian installer continues to impress, though it did make me think I should overhaul my PXE setup at home. It wouldn't be hard to have a Raspberry PI running as a TFTP + DHCP server. You could plug it into a network, reboot your desktop, and then have it boot into the imager. At the moment I run DHCP + TFTPD + etc on my main desktop, and that allows me to reimage any of the hosts in the flat easily, except itself obviously. The last time I reinstalled this system I had to reconfigure DHCP + PXE + TFTP on another host. I think the next time I need to reinstall any system I'll "waste" an SD-card on an image-server host. Finally I've recently read the Rick Cook Wizardy Series: Fun idea. Horrible puns. Some of the books were too long, or left plot elements dangling, but on average they were more good than bad. Albeit a little predictable and "simple".

22 November 2013

Enrico Zini: loncity

Airports Photo of a commercial in London City airport saying 'In the lap of luxury - Want to reach a captive audience with dwell time? Why advertise anywhere else? - London City Airport Media Sales' In the airport, we are not travellers. We are a captive audience with dwell time. In other words, suckers stuck in a room where the only pastime provided is spending money and staring at advertisements selling advertisement space in rooms full of suckers like them.

Russ Allbery: Review: The Pathway to Awesomeness

Review: The Pathway to Awesomeness, by Mark Forster
Publisher: Hyperink
Copyright: October 2013
ASIN: B00FXGXMPQ
Format: Kindle
Pages: 97
Mark Forster is probably my favorite time management (or, if you prefer, attention management) writer. The place to start with Forster if you're not familiar with him is Do It Tomorrow, not here. Unlike the other books by him I've reviewed, this is just an edited collection of blog posts presented in a convenient ebook form, without even an ISBN number. But if you've read his books (at least the ones that aren't out of print and cost a fortune), this is a convenient way to not have to trawl back through his blog and read the articles in a more convenient fashion. For me, it was also an easy way to give him a bit of money for his blog. Unless you have a similar motive, I don't think the content to price ratio is worth it. But I think everything is still up at Get Everything Done in case anything here strikes your fancy. As with most blog compilations, The Pathway to Awesomeness (I really hate that title) is composed of a whole bunch of fairly short articles, rarely more than three pages each, vaguely collected by topic. The first few are rather general and focused on the normal self-help platitudes that try to build up your sense that you're in control of your own life. But Forster doesn't take too long to get to what he's best at: short ideas, mental tools, interesting tricks, and simple techniques. Even more so than his books, due to the nature of this collection, there's a huge variety here, which means you can probably find something that strikes a chord no matter what attention management problem you've been running into. That's one of the main reasons why I prefer his writing: he tries to provide a giant toolbox instead of a single, streamlined System For Everyone. The other thing I like about Forster is that he is much more blunt than many authors about the biggest problem I have with attention management: trying to do more than anyone could possibly do given the available time. There are some more essays here about that, and several of them are quite good. I think my favorite is the short essay in which he points out how often people decide to do some task for a particular reason, and then get so engrossed in the performance of that task that they forget the original reason, risking sinking far more energy into that task than is actually warranted. Like a lot of Forster's observations, it's obvious in retrospect, but he has a way of phrasing those observation so that they shake the right thing loose in my head and let me re-evaluate something I was doing. Regular readers of Forster's blog know that he loves playing around with different simple to-do list management systems. Unfortunately, none of those are here, so don't pick this up expecting a collection of articles about Superfocus, Autofocus, etc. It makes sense given the topic of this collection, which is more general advice rather than systems, but this collection is so short that I wish he'd added a section on system development. I would have enjoyed reading a collection of all of his miniature systems together in one place and showing how he tweaked and changed them over time, particularly if he'd added some discussion of why he made each change and what goals he was trying to meet with each system. Ah well. If you aren't already familiar with Forster, there's no point in this, but if you are, I got a few new tidbits out of it and will probably skim through it a few more times. Buying it is mostly an exercise in putting money in the tip jar, and his writing is much less polished here (and more prone to cliches), but I still enjoyed reading through it. Rating: 7 out of 10

21 November 2013

Petter Reinholdtsen: All drones should be radio marked with what they do and who they belong to

Drones, flying robots, are getting more and more popular. The most know ones are the killer drones used by some government to murder people they do not like without giving them the chance of a fair trial, but the technology have many good uses too, from mapping and forest maintenance to photography and search and rescue. I am sure it is just a question of time before "bad drones" are in the hands of private enterprises and not only state criminals but petty criminals too. The drone technology is very useful and very dangerous. To have some control over the use of drones, I agree with Daniel Suarez in his TED talk "The kill decision shouldn't belong to a robot", where he suggested this little gem to keep the good while limiting the bad use of drones:
Each robot and drone should have a cryptographically signed I.D. burned in at the factory that can be used to track its movement through public spaces. We have license plates on cars, tail numbers on aircraft. This is no different. And every citizen should be able to download an app that shows the population of drones and autonomous vehicles moving through public spaces around them, both right now and historically. And civic leaders should deploy sensors and civic drones to detect rogue drones, and instead of sending killer drones of their own up to shoot them down, they should notify humans to their presence. And in certain very high-security areas, perhaps civic drones would snare them and drag them off to a bomb disposal facility. But notice, this is more an immune system than a weapons system. It would allow us to avail ourselves of the use of autonomous vehicles and drones while still preserving our open, civil society.
The key is that every citizen should be able to read the radio beacons sent from the drones in the area, to be able to check both the government and others use of drones. For such control to be effective, everyone must be able to do it. What should such beacon contain? At least formal owner, purpose, contact information and GPS location. Probably also the origin and target position of the current flight. And perhaps some registration number to be able to look up the drone in a central database tracking their movement. Robots should not have privacy. It is people who need privacy.

Craig Small: Quieting dbus

I run a program called logcheck which regularly scans the logfiles looking for interesting things . One of these interesting things (to logcheck, not to me at first) was dbus complaining about mythtv. The logs look something like this:
Nov 17 05:08:18 elmo dbus-daemon[1621]: dbus[1621]: [system] Rejected send message, 2 matched rules; type="method_call", sender=":1.236" (uid=999 pid=3669 comm="/usr/bin/mythfilldatabase --verbose general --logl") interface="org.freedesktop.NetworkManager" member="GetDevices" error name="(unset)" requested_reply="0" destination="org.freedesktop.NetworkManager" (uid=0 pid=1930 comm="/usr/sbin/NetworkManager ")
What this is is the dbus daemon rejecting a message being sent from mythfilldatabase that wants to go to NetworkManager. Looking at NetworkManager s dbus configuration, its doing all the right things. So it s a simple matter of just fixing up the dbus configuration and you re done. Then I hit the first snag; the dbus documentation is a little sad. I remember seeing a show where they MRI scanned some artifacts to get to the writing and then used a maker-bot to recreate the hidden writing. They then painstakingly translated it using a team of experts to work out the English translation. Working out the dbus configurations is not quite that difficult, but it s close. The easiest way is to copy something and see if it works, which is what I did. I created a file /etc/dbus-1/system.d/mythtv.conf which permitted the myth user access to the GetDevices method. It looks like:
<!DOCTYPE busconfig PUBLIC
 "-//freedesktop//DTD D-BUS Bus Configuration 1.0//EN"
 "http://www.freedesktop.org/standards/dbus/1.0/busconfig.dtd">
<busconfig>
        <policy user="mythtv">
                <allow send_destination="org.freedesktop.NetworkManager"
                       send_interface="org.freedesktop.NetworkManager"
                       send_member="GetDevices"/>
        </policy>
</busconfig>
Reloaded dbus and all went quiet.

19 November 2013

Carl Chenet: Yarn: a scenario testing tool

Follow me also on Twitter Those who follow me on Twitter may have read my recent tweet about Yarn, a really nice software by Lars Wirzenius I discovered the last week-end while attending the Mini-Debconf UK in Cambridge. To make a long story short, Yarn allows the user to write scenarios of the executions of softwares. This scenario will be automatically played by Yarn which also returns the overview of the execution, with the goal to test the behaviour of these softwares. 1. Small example of how to use Yarn Let s have a look of how to use Yarn with a real example. The scenario will test if the software Brebis, a fully automatized backup checker, executes in an expected way. We can proceed as follow :
  1. set up the test environment
  2. launch a command with the software under test in the test environment
  3. verify the execution of the test
This is an example of a Yarn scenario file, the file is formatted using the Markdown syntax:
    SCENARIO basic brebis execution
    GIVEN setting up brebis
    AND generating backup configuration with brebis
    WHEN brebis verifies a backup
    THEN verify brebis output
    IMPLEMENTS GIVEN setting up brebis
    hg clone http://hg.brebisproject.org $DATADIR/brebis
    mkdir -p $DATADIR/brebis/yarn-test
    cp $DATADIR/brebis/functional-tests/expected-generated-list-for-tar-archive/expected-generated-list-for-tar-archive.tar.gz $DATADIR/brebis/yarn-test
    IMPLEMENTS GIVEN generating backup configuration with brebis
    $DATADIR/brebis/brebis.py -G $DATADIR/brebis/yarn-test/expected-generated-list-for-tar-archive.tar.gz
    IMPLEMENTS WHEN brebis verifies a backup
    $DATADIR/brebis/brebis.py -c $DATADIR/brebis/yarn-test -l $DATADIR/brebis/yarn-test/brebis.log
    IMPLEMENTS THEN verify brebis output
    if [ -s $DATADIR/brebis/yarn-test/brebis.log ]; then return 1; else return 0; fi
2. Result of the execution of the Yarn scenario Yarn provides at the end of the scenario an overview of the execution, as show below: $ yarn brebis-scenario
Scenario test suite PASS, with 1 scenarios (4 total steps), in 16.4 seconds 3. Details of the Yarn scenarios The five first lines are basically your scenario. Each step takes a keyword and a sentence to identify a step. You define a name with the SCENARIO keyword. The next step, GIVEN, allows generally the setup of the test environment. It is possible of course to define different steps with the keywork AND. Once the environment setup has been completed, we launch the test with the keyword WHEN. At last we check the result of the execution with THEN. It is worth noting that if you need further steps after checking the result of the execution, you can use FINALLY. Very helpful while setting up the test environment is the variable $DATADIR, automatically initialized by Yarn. It provides the path to a temporary directory where you can store files during the execution of your scenario. This directory is by default removed at the end of the scenario. Now the real trick: at each step I described above is associated shell commands to execute thanks to the IMPLEMENTS keyword.It s in my opinion a great way to increase the understanding of what the scenario really does. In a short glance of the first lines, if the names of the steps are carefully chosen, you will understand what the scenario does without needing to read the actual shell code. Simple and efficient. 4. More about Yarn Yarn is written in Python and now available in Debian in the package cmdtest. The sources are available here. In my opinion, if you were looking for this kind of tool, Yarn is in a really good direction. This project is still young but the core features do the job and the upstream is eager to receive feedbacks (and patches) :) Some links: Interested in Yarn? What do you think about it? Let us know in the comments of this post.

17 November 2013

Gregor Herrmann: RC bugs 2013/46

not my most active RC bug squashing week but still, a few things done:
  • #713575 src:libgcal: "libgcal: FTBFS: ld: /usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/libcheck_pic.a(check_pack.o): undefined reference to symbol '__pthread_unregister_cancel@@GLIBC_2.3.3'"
    add patch from Ubuntu / Andreas Moog, upload to DELAYED/2
  • #713662 src:heartbeat: "heartbeat: FTBFS: ucast.c:468:6: error: conflicting types for 'i'"
    add patch from Ubuntu / Andres Rodriguez, upload to DELAYED/2
  • #718068 src:csh: "csh: FTBFS: Makefile:24: *** missing separator. Stop."
    propose a different solution
  • #718098 src:menhir: "menhir: FTBFS: Makefile:34: *** Please define PREFIX. Stop."
    "backport" fix from experimental, upload to DELAYED/2, then rescheduled to 0-day with maintainer's permission
  • #726410 src:spandsp: "spandsp: FTBFS in parallel mode"
    tag unreproducible
  • #727806 tunnelx: "tunnelx: Depends on openjdk-6, which is going away"
    update Debian-Java-Home in .manifest file; upload to DELAYED/2

Dmitrijs Ledkovs: libnih (upstart dependency) ported to kFreeBSD/eglibc


Back in September at the Linux Plumbers Conference in New Orleans, James Hunt and I presented Upstart roadmap, which included porting Upstart to kFreeBSD/glibc. The first milestone is now complete.

Libnih is ported to kFreeBSD/eglibc. During the course to get there the following has happened:

  • Later libnih test-suite gets a test case failure, which is then reported to FreeBSD.org. Turns out some syscalls on FreeBSD returned bogus siginfo_t.si_status information in some cases. Luckily there is now a full test-case and kernel patch posted by Jilles Tjoelker. Once that's committed upstream, I'll push for uploads to the affected kernels in Debian.


  • A few small patches applied to libnih, mostly adding POSIX compliant header includes and the like.
So with a patched libc and patched kernel one can compile libnih & run its test-suite. There are however some caveats feature miss-partiy:

  • nih_watch / inotify support is disabled. This is TODO on my list to port nih_watch to use kevent/kqueue EVFILT_VNODE.
  • FreeBSD kernel doesn't have abstract namespace domain sockets, so at the moment I have disabled some abstract namespace test-cases & changed dbus-connection tests to use pathname sockets instead. This is still a TODO because unlink before bind is not done, and the socket is not cleaned up after servers close it at the moment.




With all caveats above, here are the test-suite results:

================================================
libnih 1.0.4: nih-dbus-tool/test-suite.log
================================================

# TOTAL: 2492
# PASS: 2492
# SKIP: 0
# XFAIL: 0
# FAIL: 0
# XPASS: 0
# ERROR: 0

.. contents:: :depth: 2

===========================================
libnih 1.0.4: nih-dbus/test-suite.log
===========================================

# TOTAL: 110
# PASS: 110
# SKIP: 0
# XFAIL: 0
# FAIL: 0
# XPASS: 0
# ERROR: 0

.. contents:: :depth: 2

======================================
libnih 1.0.4: nih/test-suite.log
======================================

# TOTAL: 788
# PASS: 787
# SKIP: 0
# XFAIL: 1
# FAIL: 0
# XPASS: 0
# ERROR: 0

.. contents:: :depth: 2

PASS: test_watch
================

not ok 1 - nih_watch not supprted yet # TODO
XFAIL: test_watch 1 - nih_watch not supprted yet # TODO

at tests/test_watch.c:1594 (main).
1..1

16 November 2013

Russ Allbery: Review: Wanderlust

Review: Wanderlust, by Ann Aguirre
Series: Sirantha Jax #2
Publisher: Ace
Copyright: September 2008
ISBN: 0-441-01627-8
Format: Mass market
Pages: 312
Wanderlust is the second book in the Sirantha Jax series and a direct sequel to Grimspace. There's no point in reading them out of order. The new shape of Sirantha's universe is still coming together after the climactic events of Grimspace, but they seem to involve a new job. In politics, even, but one for which she's uniquely qualified. Before she can even accept it, she finds out that multiple parties are interested in her taking this position, for widely varying reasons. But she starts her new position by taking a detour to visit some old friends (and the planet I liked least in Grimspace's settings), which turns into a war and a comprehensive distraction that lasts the rest of the book. The biggest problem with Wanderlust (and unfortunately there are several) is that it doesn't feel like a complete book. The first few chapters set up what appears to be the next step of Sirantha's adventures, as well as establishing a new villain, and then the rest of the book is an extended detour. She does learn more about the new villain, but almost everything that's set up at the start of this book is still unresolved at the end of it. The detours are moderately entertaining and (at least after an initial decision based on unfinished business from the first book) are at least related to the plot, but this feels like the setup and middle-book distractions of a book twice as long. I came away wondering why all of that was really necessary and why I had to wait to the next book to read the actual plot. The second problem with this book is the whining. And the angst. And the self-esteem issues. And did I mention the whining? Okay, I get that one of the distinguishing elements of this series is that it's space opera with an urban fantasy first-person perspective, and the latter usually comes with extensive angst and insecurity about one's love life. But, really, I am not interested in Sirantha complaining about her appearance yet again, and I am particularly not interested in the obnoxiously tedious "I love you and therefore I must push you away" dance that both she and the male romance lead do through half this book, and only stop doing because they can't any more. I realize that self-destructiveness is a thing, and that people do react this way, but I don't want to read about it, and I particularly don't want to listen to the narrator whine about how hard it all is for much of the book. Some of the appeal of Sirantha is that she's fragile, paranoid, and all sorts of messed up, but she muddles through and manages to do the right thing anyway. However, for me, the key word there is "do." When she's twitchy and angry and sarcastic and scared but doing something (which does also happen in this book), I really enjoy this series. When she's living in her own head and building elaborate plans for how she's going to destroy her life this time, it starts getting tiresome. It doesn't help that I don't see what she sees in her romantic partner (at least most of the time; he has his moments), and therefore I have even less sympathy for her grand self-sacrificing gestures. Thankfully, there are good parts too. Aguirre introduces a few new characters, all of whom I liked. Sirantha is assembling quite the ragtag band of misfits, and they're misfits in all the right sorts of ways. They play off each other well and they're full of plot hooks. We also get an amusing twist on Sirantha's relationship with her personal assistant AI, which, while bizarrely tangential to the plot, looks like it will hold a lot of promise for the future. And I love Vel, who made the end of Grimspace and who continues to be a highlight of these books, partly because he doesn't throw emotional temper tantrums or indulge in bizarre silent treatments. I always love the quietly competent characters in stories like this. This is not up to Grimspace, but hopefully it's an aberration. It's not bad, exactly, but it went off the rails. Thankfully, the next book promises quite a lot of Vel, which will hopefully be an improvement. Followed by Doubleblind. Rating: 6 out of 10

Russ Allbery: Review: Blue Remembered Earth

Review: Blue Remembered Earth, by Alastair Reynolds
Series: Poseidon's Children #1
Publisher: Ace
Copyright: January 2012
Printing: June 2012
ISBN: 0-441-02071-2
Format: Hardcover
Pages: 505
Geoffrey Akinya is one heir to the vast Akinya business empire, which straddles the solar system and has created vast riches from exploration and mining. He wants nothing to do with it. His passion is elephants: long and methodical study of wild herds in Africa, including a slow and careful investigation into their thought processes. His obnoxious and superior cousins have a passion for business and finance and can have the running of the company for all he cares. But Geoffrey's grandmother Eunice, the family matriarch and driving force behind much of their business expansion, has died. Her ashes are brought back to Africa from the Winter Palace in orbit around the Moon where she lived out her final days and scattered in a family ceremony. And, shortly thereafter, Geoffrey's cousins convince him to investigate a safe-deposit box left on the Moon by Eunice to ensure that it doesn't contain anything damaging to the family. What it contains is the first step in a puzzle. Despite himself, Geoffrey and his sister Sunday an artist and family black sheep who lives in a region of the Moon that is one of the sole holdouts against the ubiquitous monitoring common in the rest of the inhabited solar system are slowly pulled into unraveling that puzzle. This leads them into an uneasy alliance with one of the major political forces in the solar system and some startling revelations about their grandmother's actual plans. Blue Remembered Earth is another entry in the currently-popular sub-genre of solar system SF. Interstellar travel is still a dream, but a combination of improved technology, suspended animation, and heavy use of robots and robotic factories has let humans slowly expand into more of the solar system. On Earth, powers have risen and fallen, and the leap into the solar system coincided with a surging and powerful Africa. Extensive undersea colonization has established a new international consortium of sea-floor countries, and Earth politics have realigned into a diplomatic and business struggle between surface and oceanic nations. Unusually for this sort of fiction, most characters of significance in this story are African. The United States and Europe have quietly disappeared in the way that all non-US countries tend to in most other SF. Reynolds doesn't do very much with this setup (and the characters don't seem distinctly African to me). It's just there as the unremarked normal. I liked that. I kind of wish more of African culture had come through in the story, but there are enough SF novels with an assumed and unremarked white (and usually US) future that an assumed and unremarked black African culture is a nice change of pace. I picked up this book in part because it was described as a better 2312. That's a fairly accurate summary (although only helpful to those who have read Kim Stanley Robinson's novel). The scope and setting are similar, although Reynolds's future cuts back on the scope of off-world human colonization, relies more heavily on robotics, and keeps the power base on Earth. Both books are structured as a mystery, although in Blue Remembered Earth that mystery is deliberately planted for the characters. Reynolds is nowhere near as good at set pieces as Robinson, sadly. There's nothing here at the level of some of the scenes on Mercury in 2312. But he is much better at plotting and characterization, and Blue Remembered Earth tells a coherent story with increasing tension and a satisfying ending. Reynolds here uses an interesting strategy for spinning a mystery out of pieces of knowledge and discovery: set the discovery process prior to the novel, use protagonists who were unaware of it, and turn the details into a mystery that they have to investigate. This is an interesting way to address the problem that most discovery processes are slow and not particularly dramatic. It is artificial Eunice creates the plot for the novel but that worked for me. By the end of the book, I think I understand her reasons for doing so, and both Geoffrey and Sunday have good reasons to not be aware of any of the details before they start following the clues that Eunice leaves behind. The mystery of course sends Geoffrey and Sunday on a bit of a tour of the solar system as well as a tour of the past of their family business and the politics of the present. Here, Reynolds draws on an old SF idea: a conflict between groups that want to mechanize space exploration and groups that want to adapt humans for space. Long-time SF readers will be familiar with this approach, and I don't think Reynolds does anything startlingly new with it, but I thought it was a well-executed side plot. And, for once, an SF author realizes that Earth's oceans are as compelling of a place for that conflict to play out as outer space. There are, of course, significant mysteries to be uncovered, but it takes much of the book before those revelations start (even if one of them is almost spoiled in the dust jacket summary). The story up until that point is driven by character, particularly Geoffrey's stubborn annoyance at the rest of his family and his complete lack of interest in being the protagonist of a novel. I should probably warn here that many of the Amazon reviews disliked this book because of the characters, particularly Geoffrey. Apparently he comes across to some as whiny, resentful, and annoying. I, however, didn't have this reaction at all, possibly in part because I wholeheartedly share his opinion of both big business and his cousins. I thoroughly enjoyed reading an SF novel whose main characters are not driven either typical capitalist motives or a desire for personal exploration, but instead are pulled into the plot through a sense of loyalty and an inability to get out of the way of the problem. I also liked the way Reynolds manages to broaden and deepen the reader's appreciation for the characters as the story unfolds, and even salvages some characters I thought were unsalvageable. The last third of the book features some excellent plot twists and some very nicely-handled revelations. This is apparently the first book of a trilogy, and I had a suspicion at the end of the book that sequels were coming, but I think it stands on its own quite well. Most of the questions raised by the plot are resolved by the end of the book, and those that remain are of a size that other books have left unanswered intentionally. Despite not reaching the same moments of awe-inspiring beauty, and despite more workman-like descriptions, the stronger plot and more interesting characters do make Blue Remembered Earth a better 2312 for me. Given that 2312 won a Nebula, that's saying something (although I don't agree with that award). It's a bit deliberate in its pacing, and the plot built around a constructed puzzle can feel a bit artificial, but I think it's a solid and satisfying near-space SF novel. Followed by On the Steel Breeze. Rating: 7 out of 10

15 November 2013

Jo Shields: The directhex! In an Adventure with MIPS

We ve been trying to build Mono on MIPS in Debian for a long time. Just under a decade, in fact. Mono 0.29.99.20040114-3 was the first attempt, back when the Mono source was 10 meg, not today s ~80. It never worked though. Not once. At the end of 2004, 10 upstream versions later, we gave up, and turned off mipsel as a target architecture in the package. When I became a Debian Developer in 2009, one of the first things I did was try again with Mono on various previously unsupported architectures, by running a build on the Debian porterboxes. Supposedly MIPS support was there, as evidenced by all the MIPS-related files in the source tree. But I d try it, and it d fail, and upstream would say works for me . I repeat the process every major release or so, to see what s changed. The breakthrough Spurred by the removal of two bitrotted architectures (IA64 and SPARC), I decided to try again, this time investigating more deeply into the reason for the failures, with the help of upstream developer Alex R nne Petersen. A couple of hours of IRC-driven gdb and test program disassembly later, a seemingly innocent comment flagged something in my brain: 09-07-2013 15:10:11 < alexrp!~zor@baldr.rfw.name: TIL that mul and mult are not the same thing on mips Why is this notable? Well, MIPS processors lack a whole bunch of instructions which are commonly used in assembler. MUL is one of them it s valid MIPS assembler, which is expanded to MULT/MFLO when compiling. Call it a macro, or a mnemonic, or shorthand the preferred term is pseudoinstructions . So what s the issue? 09-07-2013 15:18:27 > directhex: mono isn't trying to use a mul instruction, right? i mean, that instruction doesn't exist as far as the cpu is concerned, it's a macro the compiler does things with See where this is headed? 09-07-2013 15:23:59 > directhex: mini/mini-mips.c:#define USE_MUL 1 /* use mul instead of mult/mflo for multiply */ Argh. See, the thing about MIPS pseudoinstructions is they may be real instructions on a given CPU implementation. Strictly speaking MUL isn t a standard instruction, but a given CPU might have it anyway, to make multiplication a little faster (by using only one instruction, not two, for multiplication). In this case, the Debian MIPS infrastructure is based around ICT Loongson-2E processors which don t have that extension but the upstream Mono developers were building and testing on an extended CPU, never seeing the issue themselves. Flipping that define to 0 (and amending the instruction length setting in another file) fixed the build. Mono was running on MIPS for me for the first time ever. Digging through the history in git showed just how annoying this implementation quirk was. USE_MUL was added in late 2008 replacing a previously used #if 1 . The mult/mflo version of the code existed in the Mono source since the first time the full MIPS port was committed in 2006, but we never saw it. The breakage So, with that patched to work, I added mipsel to the Experimental build of Mono which still failed. The runtime would build fine, but the class library build would fail at random times, with random meaningless stack traces. Unrepeatable. Some kind of race condition. The build would eventually succeed if I hammered make a few times, but that s no good for the Debian build daemons. Back to square one except I had an epiphany yesterday. I have heard more than once that Loongson processors are missing a few instructions. What if one of those was being hit, intermittently? I started doing a search in places that might need to work around that kind of issue, and found this. A patch to binutils in 2009, replacing one no-op instruction with another, when /usr/bin/as is fed the -mfix-loongson2f-nop flag. Turns out NOP is another pseudoinstruction on MIPS. Well, more of an alias. The opcode 0 000000 is Shift Left Logical with 0 registers and 0 data, which is a no-op. But on all but the latest generation of Loongson-2F chips, that opcode can, under heavy load, fail causing inconsistent state in the CPU registers. The flag to as replaces sll 0,0,0 with or $at,$at,0 , which is also a no-op instruction, but doesn t trigger the failure on Loongson-2F chips (and 2E chips, although that s not stated in the documentation). As long as ALL your programs get fed through as , you don t have a problem, since it uses the replacement opcode but what if you use a JITter to generate your own opcodes? Oh fuck, it couldn t be diff --git a/mono/arch/mips/mips-codegen.h b/mono/arch/mips/mips-codegen.h
index dc4df7d..1dbd1c6 100644
--- a/mono/arch/mips/mips-codegen.h
+++ b/mono/arch/mips/mips-codegen.h
@@ -334,7 +334,7 @@ enum
/* misc and coprocessor ops */
#define mips_move(c,dest,src) mips_addu(c,dest,src,mips_zero)
#define mips_dmove(c,dest,src) mips_daddu(c,dest,src,mips_zero)
-#define mips_nop(c) mips_sll(c,0,0,0)
+#define mips_nop(c) mips_or(c,mips_at,mips_at,0)
#define mips_break(c,code) mips_emit32(c, ((code)<<6) 13)
#define mips_mfhi(c,dest) mips_format_r(c,0,0,0,dest,0,16)
#define mips_mflo(c,dest) mips_format_r(c,0,0,0,dest,0,18)
Oh yes it could! Mono was using sll 0,0,0 (the recommended no-op instruction from the MIPS instruction reference manual), causing failures in my tests, because Debian s build and test infrastructure just happens to use defective silicon. And, again, upstream were unable to reproduce a problem because they use better silicon than we do. So what now? Well, last night I uploaded mono_3.2.3+dfsg-3, which includes the above patch to force the replacement no-op instruction. It test built fine on the porterbox, and it should (when the damn experimental buildd gets around to it), just work. Finally. After just under a decade, Mono packages will be available on MIPS in Debian. And after all this time, all we had to change was 4 lines to work around 7 year old Chinese knock-off processors. The edit So, things are finally built. It turns out that despite everything, the replacement NOP opcode is not enough. If you re-read the post to the binutils list, pay close attention to: In theory this is still not enough to fully eliminate possible hangs, but the possiblity is extremely low now and hard to be hit in real code. It s a filthy lie. It s easy to hit the issue in real code: just do a from-source build of the whole Mono class library. With the replacement instruction it builds .NET 2.0, 3.0, 3.5, 4.0, and most of 4.5, before dying in the same way as before an improvement on failing early in the 2.0 build, but not enough. Thankfully, 2 out of the 5 Debian mipsel build servers are not Loongson 2 they re 11 year old Broadcom SWARM developer boards. Not fast but also not broken. Luck smiled on me, and caused my build to go to one of these Broadcom machines. As a result (experimental_mipsel-dchroot)directhex@eder:~$ mono --version head -1
Mono JIT compiler version 3.2.3 (Debian 3.2.3+dfsg-3)
It s been a long time coming.

14 November 2013

Tanguy Ortolo: Using Wine with sound under Debian testing (Jessie)

If you are using Wine under Debian testing with PulseAudio, you probably noticed that you cannot get sound playback any more. This is because: Loudspeaker icon
  1. Wine uses ALSA, which uses a plugin to play through PulseAudio;
  2. Wine being in 32 bits, all that has to be installed in i386 versions;
  3. recent versions of libasound2-plugins depend on libavcodec54 which depends on libopus0 which is not multiarch-capable and thus cannot be installed in both i386 and amd64 versions;
  4. libopus0:amd64 cannot be reasonably removed to install only libopus0:i386 because many multimedia software depend on it (in other words: try that and you will end removing VLC and everything similar too).
No need to despair though, as there are several ways to work around that problem until the maintainer of libasound2-plugins has converted it to multiarch.Use ALSA directly Since the problem is that Wine tries to use the ALSA PulseAudio plugin which is not available in 32 bits, the most direct solution is to configure it not to use it. Run winecfg, go to the audio tab and manually select your audio devices, which will then be used directly with ALSA without trying to use the PulseAudio plugin.
Screenshot of winecfg

Wine configuration

Downgrade ALSA plugins Since the problem is caused by a dependency chain that only exists with the testing (Jessie) version of the libasound2-plugins package, another solution is to downgrade it to a previous version. To do that, make sure you have enabled the Debian repositories for both the stable (Wheezy) and testing (Jessie) in /etc/apt/sources.list: deb http://ftp.fr.debian.org/debian/ jessie main deb http://security.debian.org/ jessie/updates main contrib non-free deb http://ftp.fr.debian.org/debian/ wheezy main deb http://security.debian.org/ wheezy/updates main contrib non-free Then, forcefully install the stable version of libasound2-plugins for both i386 and amd64: # apt-get install libasound2-plugins:i386/stable \ libasound2-plugins:amd64/stable

13 November 2013

Andrea Veri: Configuring DNSSEC on your personal domain

Today I ll be working out how to properly configure DNSSEC on a BIND9 installation, I ll also make sure to give you all the needed instructions to properly verify if a specific domain is being correctly covered by DNSSEC itself. In addition to that a few more details will be provided about adding the relevant SSHFP s entries on your DNS zone files to be able to automatically verify the authenticity of your domain when connecting to it with SSH avoiding any possible MITM attack. First of all, let s create the Zone Signing Key (ZSK) which is the key that will be responsible to sign any other record on the zone file which is not a DNSKEY record itself:
dnssec-keygen -a RSASHA1 -b 1024 -n ZONE gnome.org


Note: the dnssec-keygen binary file should be part of bind97 (RHEL 5) or bind (RHEL6) package according to yum whatprovides: RHEL 5:
32:bind97-9.7.0-17.P2.el5_9.2.x86_64 : The Berkeley Internet Name Domain (BIND) DNS (Domain Name System) server
Repo : rhel-x86_64-server-5
Matched from:
Filename : /usr/sbin/dnssec-keygen


RHEL 6:
32:bind-9.8.2-0.17.rc1.el6.3.x86_64 : The Berkeley Internet Name Domain (BIND) DNS (Domain Name System) server
Repo : rhel-x86_64-server-6
Matched from:
Filename : /usr/sbin/dnssec-keygen


Then, create the Key Signing Key (KSK), which will be used to sign all the DNSKEY records:
dnssec-keygen -a RSASHA1 -b 2048 -n ZONE -f KSK gnome.org


Creating the above keys can take several minutes, when done copy the public keys to the zone file this way:
cat Kgnome.org*.key >> gnome.org


When done you can clean out the useless bits from the zone file and just leave the DNSKEY records (which are not commented out as you will notice) An additional and cleaner way of accomplishing the above is to use the INCLUDE rule on the zone file itself as follows:
$INCLUDE /srv/dnssec-keys/Kgnome.org+005+12345.key
$INCLUDE /srv/dnssec-keys/Kgnome.org+005+67890.key


Choosing which method to use is really up to you. Once that is done you can go ahead and sign the zone file. As of myself I m making use of the do-domain script taken from the Fedora Infrastructure Team s repositories. If you are going to use it yourself, make sure to adjust all the relevant variables to match your setup, especially keyspath, region_zones, template_zone, signed_zones and AREA. The do-domain script also checks your zone file through named-checkzone before signing it.
/me while editing the do-domains script with the preview of gnome-code-assistance!

/me while editing the do-domains script with the preview of gnome-code-assistance!

If instead you don t want to use the script above, you can sign the zone file manually in the following way:
dnssec-signzone -K /path/to/your/dnssec/keys -e +3024000 -N INCREMENT gnome.org


By default, the above command will append .signed to the file name, you can modify that behaviour by appending the -f flag to the dnssec-signzone call. The -N INCREMENT will increment the serial number automatically making use of the RFC 1982 arithmetics while the -e flag will extend the zone signature end date from the default 30 days to 35. (this way we can safely run a monthly cron job that will sign the zone file automatically) You can make use of the following script to achieve the above:
#!/bin/sh
SIGNZONE="/usr/sbin/dnssec-signzone"
DNSSEC_KEYS="/srv/dnssec-keys"
NAMEDCHROOT="/var/named/chroot"
ZONEFILES="gnome.org"
cd $NAMEDCHROOT
for ZONE in $ZONEFILES; do
$SIGNZONE -K $DNSSEC_KEYS -e +3024000 -f $ZONE.signed -N INCREMENT $ZONE
done
/sbin/service named reload


Once the zone file has been signed just make sure to include it on named.conf and restart named:
zone "gnome.org"  
file "gnome.org.signed";
 ;


When you re done with that we should be moving ahead adding a DS record for our domain at our domain registrar. My example is taken from the known gandi.net registrar.
gandi

Gandi s DNSSEC interface

Select KSK (257) and (RSA/SHA-1) on the dropdown list and paste your public key on the box. You will find the public key you need on one of the Kgnome.org*.key files, you should look for the DNSKEY 257 entry as dig DNSKEY gnome.org shows:
;; ANSWER SECTION:
gnome.org. 888 IN DNSKEY 257 3 5 AwEAAbRD7AymDFuKc2iXta7HXZMleMkUMwjOZTsn4f75ZUp0of8TJdlU DtFtqifEBnFcGJU5r+ZVvkBKQ0qDTTjayL54Nz56XGGoIBj6XxbG8Es+ VbZCg0RsetDk5EsxLst0egrvOXga27jbsJ+7Me3D5Xp1bkBnQMrXEXQ9 C43QfO2KUWJVljo1Bii3fTfnHSLRUsbRn8Puz+orK71qxs3G9mgGR6rm n91brkpfmHKr3S9Rbxq8iDRWDPiCaWkI7qfASdFk4TLV0gSVlA3OxyW9 TCkPZStZ5r/WRW2jhUY/kjHERQd4qX5dHAuYrjJSV99P6FfCFXoJ3ty5 s3fl1RZaTo8=


Once that is done you should have a fully covered DNSSEC domain, you can verify that this way:
dig . DNSKEY   grep -Ev '^($ ;)' > root.keys
dig +sigchase +trusted-key=./root.keys gnome.org. A   cat -n


The result:
105 ;; WE HAVE MATERIAL, WE NOW DO VALIDATION
106 ;; VERIFYING DS RRset for org. with DNSKEY:59085: success
107 ;; OK We found DNSKEY (or more) to validate the RRset
108 ;; Ok, find a Trusted Key in the DNSKEY RRset: 19036
109 ;; VERIFYING DNSKEY RRset for . with DNSKEY:19036: success
110
111 ;; Ok this DNSKEY is a Trusted Key, DNSSEC validation is ok: SUCCESS

Bonus content: Adding SSHFP entries for your domain and verifying them You can retrieve the SSHFP entries for a specific host with the following command:
ssh-keygen -r $(hostname --fqdn) -f /etc/ssh/ssh_host_rsa_key.pub


When retrieved just add the SSHFP entry on the zone file for your domain and verify it:
ssh -oVerifyHostKeyDNS=yes -v subdomain.gnome.org


Or directly add the above parameter into your /etc/ssh/ssh_config file this way:
VerifyHostKeyDNS=yes


And run ssh -v subdomain.gnome.org , the result you should receive:
debug1: Server host key: RSA 00:39:fd:1a:a4:2c:6b:28:b8:2e:95:31:c2:90:72:03
debug1: found 1 secure fingerprints in DNS
debug1: matching host key fingerprint found in DNS
debug1: ssh_rsa_verify: signature correct

That s it! Enjoy!

Joachim Breitner: Constructing a list in a Monad

In Haskell, lists are ubiquitous. In real-world Haskell, monads are ubiquitous. So it should be an easy task to fill a list with values obtained in a monad, i.e. a list of n random numbers (in the Rand monad), or getting some input from the user, and storing it in a list, until the user chooses to finish the input. The task For now, assume a function getInput :: IO Int; we want to run it 10000 and then have the list of the values returned. Sounds simple and straight forward, but it is actually really hard to get the run-time behaviour that we naively and optimistically expect. The expectation In an imperative setting, we would have a simple counting loop. In the loop, we run getInput, allocate a new cell for the linked list, and append it to the list. Clearly, this
  • consumes no stack space and
  • at the end of the loop, the list is already fully evaluated in memory, ready to be used.
Can we achieve that with Haskell? Let s look at a few alternatives (complete code available)
Attempt 1: The accumulator I ll start with the probably worst attempt. Unfortunately, from what I have seen from correcting Haskell exams, this is what beginners would write, especially if they have the above imperative algorithm in mind, and know about the convert loops to recursion with accumulators approach to solving Haskell problems:
accumAppend :: Int -> IO [Int]
accumAppend n = go n []
  where
    go 0 l = return l
    go n l = do  x <- getInput; go (n-1) (l++[x]) 
To check the space usage of this, I pass -Kn to the GHC runtime and see what limit is just enough to let this program calculate accumAppend 10000 and fully evaluate the resulting list. It turns out that we need 328336 bytes of stack space. So what about the second requirement; that the list is fully evaluated? Experienced Haskellers see it immediatelly: The accumulator is not used in the loop, so each iteration adds a thunk around the previous value, that would add the next entry to the end. Only when the list is actually used, these are run, each traversing the whole list, causing quadratic cost. We can verify this using ghc-vis:
Clearly (and not surprisingly) this is not the best solution. Attempt 2: The accumulator (forced) Lets try to fix the second issue, by making sure that the list is always fully evaluated. We import Control.DeepSeq and write
accumAppendSeq :: Int -> IO [Int]
accumAppendSeq n = go n []
  where
    go 0 l = return l
    go n l = do  x <- getInput ; go (n-1) $!! (l++[x]) 
As the accumulator gets fully evaluated before being passed to the next invocation of go, the result will not contain thunks. But note that this has also fixed the stack consumption: The code now runs the smallest setting for -K, 8 bytes. This works as both go and deepseq are tail-recursive. But this is still not the solution we are looking for, as the quadradic cost caused by using ++ is still there. Attempt 3: The accumulator (reversed) Since we know that l++[x] is expensive, but x:l is cheap, we can simply use this to update the accumulator. This has the slightly annoying consequence that the entries of the list are in the reverse order, but we can fix that in the base case:
accumReverse :: Int -> IO [Int]
accumReverse n = go n []
  where
    go 0 l = return (reverse l)
    go n l = do  x <- getInput ; go (n-1) (x:l) 
And indeed, we get noticably faster code that still runs in 8 bytes of stack. Unfortunately, we still don t construct the final list directly. Attempt 4: Naive recursion Maybe we can do without an accumulator. The most naive such code would be
listRecurse :: Int -> IO [Int]
listRecurse n = go n
  where
    go 0 = return []
    go n = do  x <- getInput; r <- go (n-1); return (x:r) 
and I d expect that most experienced Haskellers would write that code (if they are asked not to use a library function). And indeed, this code does create the final list directly. But it requires a large stack or 164616 bytes. The reason is that this is no longer tail recursive: After calling go again we need to take the result and prepend x Attempt 5: library functions Maybe we should simply not worry about the implementation, and use library functions? Clearly, the authors of such libraries have found the best solution. So how does
listReplicateM :: Int -> IO [Int]
listReplicateM n = Control.Monad.replicateM n getInput
fare? Unfortunately, not better than the naive recursion. In fact, the stack space requirement is the same 164616 bytes. The same holds when using special libraries for generating and consuming list, e.g. the Streams module from the vector package:
listStreams :: Int -> IO [Int]
listStreams n = MStream.toList $ MStream.replicateM n getInput
Attempt 6: Difference lists It is time to dig deeper into our box of tricks. What if we have lists with constant time Snoc (a.k.a. appending a new element)? Then Attempt 3 would not require reversing the list. Such lists are provided by difference lists; there are libraries for that, but we can simply implement them using lambdas:
accumDList :: Int -> IO [Int]
accumDList n = go n id
  where
    go 0 l = return (l [])
    go n l = do  x <- getInput; go (n-1) (l . (x:)) 
Indeed, no stack is required (8 bytes are enough). But we are cheating here: We are still not constructing the final list, but rather we combine functions that append elements to lists. So when accumDList returns, what we have in memory, is a thunk and sequence of partial function applications in the wrong order, and evaluating the thunk does basically the same thing as reverse in Attempt 3. It might be a bit faster, is still not satisfying enough:
Attempt 7: Unsafe functions Even deepter in the box of tricks, somewhere in the grit and dust where one usually should not reach, there is a function that can help us out here: unsafeInterleaveIO. This allow us to modify the naive recursion in a way that first creates the Cons-cell, and then continues the loop:
listUnsafeInterleave :: Int -> IO [Int]
listUnsafeInterleave n = do  l <- go n; return $!! l 
  where
    go 0 = return []
    go n = do x <- getInput
              r <- unsafeInterleaveIO (go (n-1))
              return (x:r)
The stack consumption is 8 bytes. It is worth noting that the all to deepseq (in $!!) will actually drive the evaluation of go. Also, it guarantees that the unsafe effects of unsafeInterleaveIO are confined to the execution of listUnsafeInterleave, i.e. are actually safe. Run time statistics I ran criterion over the functions (generating lists of length 1000), and found these results:
  • accumAppend: 6.5 ms
  • accumAppendSeq: 8.2 ms
  • accumReverse: 14.0 us
  • listRecurse: 8.0 us (same as listReplicateM and listStreams)
  • accumDList: 12.0 us
  • listUnsafeInterleave: 174.0 us
Conclusion There are many ways to write a list-constructing list in Haskell, none are perfect they either fill the heap, or do not construct the list directly (usually requiring a second pass through it), or rely on IO-specific unsafe functions. The naive recursion performs the best, but may cause a stack overflow (note, though that from GHC 7.8, the stack will be unlimited). The next best option seems to be the difference lists
I m still wondering: What would be required from Haskell, GHC or the monads in question to have a fully satisfactory solution here, i.e. one that is as fast as the naive recursion, as stack efficient as the difference list solution, and allocates no thunks, only list cells?

Raphael Geissert: A bashism a week: heredocs

One great feature of POSIX shells is the so-called heredoc. They are even available in languages such as Perl, PHP, and Ruby.

So where is the bashism?

It's in the implementation. What odd thing do you see below?


$ strace -fqe open bash -c 'cat <<EOF
foo
EOF' 2>&1 grep -v /lib
open("/etc/ld.so.cache", O_RDONLY O_CLOEXEC) = 3
open("/dev/tty", O_RDWR O_NONBLOCK O_LARGEFILE) = 3
open("/proc/meminfo", O_RDONLY O_CLOEXEC) = 3
[pid 6696] open("/tmp/sh-thd-1384296303", O_WRONLY O_CREAT O_EXCL O_TRUNC O_LARGEFILE, 0600) = 3
[pid 6696] open("/tmp/sh-thd-1384296303", O_RDONLY O_LARGEFILE) = 4
[pid 6696] open("/etc/ld.so.cache", O_RDONLY O_CLOEXEC) = 3
foo
--- SIGCHLD (Child exited) @ 0 (0) ---


Yes, it uses temporary files!

So do ksh, pdksh, mksh, posh and possible other shells. Busybox's sh and dash do not use temporary files, though:


$ strace -fqe open dash -c 'cat <<EOF
foo
EOF' 2>&1 grep -v /lib
open("/etc/ld.so.cache", O_RDONLY O_CLOEXEC) = 3
[pid 6767] open("/etc/ld.so.cache", O_RDONLY O_CLOEXEC) = 3
foo
--- SIGCHLD (Child exited) @ 0 (0) ---


Next time you want data to never hit a hard disk, beware that heredocuments and herestrings are best avoided.

Next.

Previous.