Search Results: "ema"

12 July 2013

Justus Winter: Hi, I'm the one who wrote umount...

Everyone enjoys ASCII screenshots as much as I do, right? So here is the Hurd booting with my mtab-prototype bound to /run/mtab: 
INIT: version 2.88 booting
Using makefile-style concurrent boot in runlevel S.
Activating swap...done.
Checking root file system...fsck from util-linux 2.20.1
hd2 : tray open or drive not ready
hd2 : tray open or drive not ready
hd2 : tray open or drive not ready
hd2 : tray open or drive not ready
end_request: I/O error, dev 02:00, sector 0
/dev/hd0s1: clean, 44292/181056 files, 287759/723200 blocks
done.
Activating lvm and md swap...(default pager): Already paging to partition hd0s5!
done.
Checking file systems...fsck from util-linux 2.20.1
hd2 : tray open or drive not ready
hd2 : tray open or drive not ready
end_request: I/O error, dev 02:00, sector 0
done.
Cleaning up temporary files... /tmp.
Mounting local filesystems...done.
Activating swapfile swap...(default pager): Already paging to partition hd0s5!
done.
Cleaning up temporary files....
Configuring network interfaces...inetutils-ifconfig: invalid arguments
ifup: failed to open pid file /run/network/ifup-/dev/eth0.pid: No such file or directory
Internet Systems Consortium DHCP Client 4.2.2
Copyright 2004-2011 Internet Systems Consortium.
All rights reserved.
For info, please visit https://www.isc.org/software/dhcp/
can't create /var/lib/dhcp/dhclient./dev/eth0.leases: No such file or directory
Listening on Socket//dev/eth0
Sending on   Socket//dev/eth0
DHCPDISCOVER on /dev/eth0 to 255.255.255.255 port 67 interval 4
DHCPREQUEST on /dev/eth0 to 255.255.255.255 port 67
DHCPOFFER from 10.0.2.2
DHCPACK from 10.0.2.2
can't create /var/lib/dhcp/dhclient./dev/eth0.leases: No such file or directory
bound to 10.0.2.15 -- renewal in 42550 seconds.
done.
Cleaning up temporary files....
Setting up X socket directories... /tmp/.X11-unix /tmp/.ICE-unix.
INIT: Entering runlevel: 2
Using makefile-style concurrent boot in runlevel 2.
Starting enhanced syslogd: rsyslogd.
Starting deferred execution scheduler: atd.
Starting periodic command scheduler: cron.
Starting system message bus: dbusFailed to set socket option"/var/run/dbus/system_bus_socket": Protocol not available.
Starting OpenBSD Secure Shell server: sshd.
GNU 0.3 (debian) (console)
login:
As you can see, the boot process looks quite clean. In particular, there are no issues left with Debians initscripts. The remaining noise comes from: This is accomplished by my mtab translator prototype. It's work in progress, but I have reworked and broken up my initial prototype. My initial prototype only patched libdiskfs, one of three libraries translator authors can use to write filesystem-like translators. I have made the necessary changes to the other two libraries and patched up any loose ends. This is the translator in action: 
% fsysopts /run/mtab
/hurd/mtab /
% cat /run/mtab
/dev/hd0s1 / ext2fs writable,no-inherit-dir-group,store-type=typed 0 0
none /dev/ttyp0 /hurd/term name,/dev/ptyp0,type,pty-master 0 0
none /run /hurd/tmpfs writable,no-suid,no-exec,no-inherit-dir-group,no-sync,size=80484K 0 0
none /run/lock /hurd/tmpfs writable,no-suid,no-exec,no-inherit-dir-group,no-sync,size=5M 0 0
none /run/shm /hurd/tmpfs writable,no-suid,no-exec,no-inherit-dir-group,no-sync,size=259480K 0 0
I also patched the sysvinit package. My version includes and switches to runsystem.sysv that uses /sbin/init to start the system. It also carries all the necessary workarounds for the remaining sysvinit related issues. The resulting Debian/Hurd system is very nice, it is getting more and more similar to what a Debian user would expect: I have rebuild the hurd and sysvinit package and uploaded them into an apt repository: 
deb http://teythoon.cryptobitch.de/gsoc/heap/debian unstable main
Please use unstable for now. Also make sure that you have a recovery plan for your Debian/Hurd installation if anything goes wrong. For your convenience there's a seed tarball containing packages with the appropriate sources.list.d snippets and the repository key: https://teythoon.cryptobitch.de/gsoc/heap/debian/seed.tar Currently /hurd/console is not started (I'll add a init script for that later), but for now you have to start a getty on the console. Make sure that your /etc/inittab contains a line like this: 
7:2345:respawn:/sbin/getty 38400 console
The patched initscripts package contains the runsystem.sysv file and uses the Debian alternatives system to replace the runsystem.gnu variant. It also switches to the appropriate halt and reboot utilities. If you install the package, you must use halt-hurd or reboot-hurd to halt or reboot the system. Running halt or reboot is not harmful though, it just doesn't work. So here I am. I've been coding for the Hurd for four weeks now, and suddenly I'm the guy who wrote umount. My point is, if you ever wanted to work on an operating system, you might want to consider working on the Hurd. It the Debian/Hurd port is very decent and (according to #debian-hurds "Debian GNU/Hurd Doomsday-o-meter") 78.44% of all Debian packages are available, so you've got a familiar environment. Also, if your solution is right and your code is reasonably clean, it is not that hard to get your changes accepted. The list of open issues is long and might contain something that interests you. You could be the one implementing a read-ahead solution to massively improve Hurds overall performance. Also since Hurd is just a Hird of Unix replacing daemons running on-top the GNU Mach microkernel, most of the functionality lies within userspace processes. This makes the whole system very extensible and quite easy to work with. For an overview over the Hurd, see this page. Next week I'll focus on improving my mtab prototype, improving the sysvinit patches and upstreaming the ones that are ready. As always, in case I run out of stuff to do, I'll pick something from my list of issues, most likely the network related issues. In completely unrelated news, my Debian/Linux workstation died with a kernel panic while I was writing this blog entry. It's a good thing Emacs honors "save early, save often". M-x insert-monolithic-kernel-rant-here. This is it for this week, thanks for your attention :)

Daniel Pocock: Practical VPNs with strongSwan, Shorewall, Linux firewalls and OpenWRT routers

There is intense interest in communications privacy at the moment thanks to the Snowden scandal. Open source software has offered credible solutions for privacy and encryption for many years. Sadly, making these solutions work together is not always plug-and-play. In fact, secure networking, like VoIP, has been plagued by problems with interoperability and firewall/filtering issues although now the solutions are starting to become apparent. Here I will look at some of them, such as the use of firewall zones to simplify management or the use of ECDSA certificates to avoid UDP fragmentation problems. I've drawn together a lot of essential tips from different documents and mailing list discussions to demonstrate how to solve a real-world networking problem. A typical network scenario and requirements Here is a diagram of the network that will be used to help us examine the capabilities of these open source solutions. Some comments about the diagram:
  • The names in square brackets are the zones for Shorewall, they are explained later.
  • The dotted lines are IPsec tunnels over the untrusted Internet.
  • The road-warrior users (mobiles and laptops) get virtual IP addresses from the VPN gateway. The branch offices/home routers do not use virtual IPs.
  • The mobile phones are largely untrusted: easily lost or stolen, many apps have malware, they can only tunnel to the central VPN and only access a very limited range of services.
  • The laptops are managed devices so they are trusted with access to more services. For efficiency they can connect directly to branch office/home VPNs as well as the central server.
  • Smart-phone user browsing habits are systematically monitored by mobile companies with insidious links into mass-media and advertising. Road-warriors sometimes plug-in at client sites or hotels where IT staff may monitor their browsing. Therefore, all these users want to tunnel all their browsing through the VPN.
  • The central VPN gateway/firewall is running strongSwan VPN and Shorewall firewall on Linux. It could be Debian, Fedora or Ubuntu. Other open source platforms such as OpenBSD are also very well respected for building firewall and VPN solutions, but Shorewall, which is one of the key ingredients in this recipe, only works on Linux at present.
  • The branch office/home network could be another strongSwan/Shorewall server or it could also be an OpenWRT router
  • The default configuration for most wifi routers creates a bridge joining wifi users with wired LAN users. Not here, it has been deliberately configured as an independent network. Road-warriors who attach to the wifi must use VPN tunnelling to access the local wired network. In OpenWRT, it is relatively easy to make the Wifi network an independent subnet. This is an essential security precaution because wifi passwords should not be considered secure: they are often transmitted to third parties, for example, by the cloud backup service in many newer smart phones.
Package mayhem The major components are packaged on all the major Linux distributions. Nonetheless, in every case, I found that it is necessary to re-compile fresh strongSwan packages from sources. It is not so hard to do but it is necessary and worth the effort. Here are related blog entries where I provide the details about how to re-build fresh versions of the official packages with all necessary features enabled: Using X.509 certificates as a standard feature of the VPN For convenience, many people building a point-to-point VPN start with passwords (sometimes referred to as pre-shared keys (PSK)) as a security mechanism. As the VPN grows, passwords become unmanageable. In this solution, we only look at how to build a VPN secured by X.509 certificates. The certificate concept is not hard. In this scenario, there are a few things that make it particularly easy to work with certificates:
  • strongSwan comes with a convenient command line tool, ipsec pki. Many of it's functions are equivalent to things you can do with OpenSSL or GnuTLS. However, the ipsec pki syntax is much more lightweight and simply provides a convenient way to do the things you need to do when maintaining a VPN.
  • Many of the routine activities involved in certificate maintainence can be scripted. My recent blog about using Android clients with strongSwan gives a sample script demonstrating how ipsec pki commands can be used to prepare a PKCS#12 (.p12) file that can be easily loaded into an Android device using the SD-card.
  • For building a VPN, there is no need to use a public certificate authority such as Verisign. Consequently, there is no need to fill in all their forms or make any payments for each device/certificate. Some larger organisations do choose to outsource their certificate management to such firms. For smaller organisations, an effective and sometimes better solution can be achieved by maintaining the process in-house with a private root CA.
UDP fragmentation during IPsec IKEv2 key exchange and ECDSA A common problem for IPsec VPNs using X.509 certificates is the fragmentation of key exchange datagrams during session setup. Sometimes it works, sometimes it doesn't. Various workarounds exist, such as keeping copies of all certificates from potential peers on every host. As the network grows, this becomes inconvenient to maintain and to some extent it eliminates the benefits of using PKI. Fortunately, there is a solution: Elliptic Curve Cryptography (ECC). Many people currently use RSA key-pairs. Best practice suggests using RSA keys of at least 2048 bits and often 4096 bits. Using ECC with a smaller 384-bit key is considered to be equivalent to a 7680 bit RSA key pair. Consequently, ECDSA certificates are much smaller than RSA certificates. Furthermore, at these key sizes, the key exchange packets are almost always smaller than the typical 1500 byte MTU. A further demand for ECDSA is arising due to the use of ECC within smart cards. Many smartcards don't support any RSA key larger than 2048 bits. The highly secure 384-bit ECC key is implemented in quite a few common smart cards. Smart card vendors have shown a preference for the ECC keys due to the US Government's preference for ECC and the lower computational overheads make them more suitable for constrained execution environments. Anyone who wants to use smart cards as part of their VPN or general IT security now, or in the future, needs to consider ECC/ECDSA. Making the network simple with Shorewall zones For this example, we are not just taking some easy point-to-point example. We have a real-world, multi-site, multi-device network with road warriors. Simplifying this architecture is important to help us understand and secure it. The solution? Each of these is abstracted to a "zone" in Shorewall. In the diagram above, the zone names are in square brackets. The purpose of each zone is described below:
Zone name Description
loc This is the private LAN and contains servers like databases, private source code respositories and NFS file servers
dmz This is the DMZ and contains web servers that are accessible from the public internet. Some of these servers talk to databases or message queues in the LAN network loc
vpn_a These are road warriors that are not very trustworthy, such as mobile devices. They are occasionally stolen and usually full of spyware (referred to by users as "apps"). They have limited access to ports on some DMZ servers, e.g. for sending and receiving mail using SMTP and IMAP (those ports are not exposed to the public Internet at large). They use the VPN tunnel for general internet access/browsing, to avoid surveillance by their mobile carrier.
vpn_b These are managed laptops that have a low probability of malware infection. They may well be using smart cards for access control. Consequently, they are more trusted than the vpn_a users and have access to some extra intranet pages and file servers. Like the smart-phone users, they use the VPN tunnel for general internet access/browsing, to avoid surveillance by third-party wifi hotspot operators.
vpn_c This firewall zone represents remote sites with managed hardware, such as branch offices or home networks with IPsec routers running OpenWRT.
cust These are servers hosted for third-parties or collaborative testing/development purposes. They have their own firewall arrangements if necessary.
net This zone represents traffic from the public Internet.
A practical Shorewall configuration Shorewall is chosen to manage the iptables and ip6tables firewall rules. Shorewall provides a level of abstraction that makes netfilter much more manageable than manual iptables scripting. The Shorewall concept of zones is very similar to the zones implemented in OpenWRT and this is an extremely useful paradigm for firewall management. Practical configuration of Shorewall is very well explained in the Shorewall quick start. The one thing that is not immediately obvious is a strategy for planning the contents of the /etc/shorewall/policy and /etc/shorewall/rules files. The exact details for making it work effectively with a modern IPsec VPN are not explained in a single document, so I've gathered those details below as well. An effective way to plan the Shorewall zone configuration is with a table like this:
Destination zone
loc dmz vpn_a vpn_b vpn_c cust net
Source zone loc \
dmz ? \ X X X
vpn_a ? ? \ X X
vpn_b ? ? X \ X
vpn_c X \
cust X ? X X X \
net X ? X X X \
The symbols in the table are defined:
Symbol Meaning
ACCEPT in policy file
X REJECT or DROP in policy file
? REJECT or DROP in policy file, but ACCEPT some specific ports in the rules file
Naturally, this modelling technique is valid for both IPv4 and IPv6 firewalling (with Shorewall6) Looking at the diagram in two dimensions, it is easy to spot patterns. Each pattern can be condensed into a single entry in the rules file. For example, it is clear from the first row that the loc zone can access all other zones. That can be expressed very concisely with a single line in the policy file: 
loc all ACCEPT
Specific Shorewall tips for use with IPsec VPNs and strongSwan Shorewall has several web pages dedicated to VPNs, including the IPsec specific documentation.. Personally, I found that I had to gather a few details from several of these pages to make an optimal solution. Here are those tips:
  • Ignore everything about the /etc/shorewall/tunnels file. It is not needed and not used any more
  • Name the VPN zones (we call them vpn_a, vpn_b and vpn_c) in the zones file but there is no need to put them in the /etc/shorewall/interfaces file.
  • The /etc/shorewall/hosts file is not just for hosts and can be used to specify network ranges, such as those associated with the VPN virtual IP addresses. The ranges you put in this file should match the rightsourceip pool assignments in strongSwan's /etc/ipsec.conf
  • One of the examples suggests using mss=1400 in the /etc/shorewall/zones file. I found that is too big and leads to packets being dropped in some situations. To start with, try a small value such as 1024 and then try larger values later after you prove everything works. Setting mss for IPsec appears to be essential.
  • Do not use the routefilter feature in the /etc/shorewall/interfaces file as it is not compatible with IPsec
Otherwise, just follow the typical examples from the Shorewall quick start guide and configure it to work the way you want. Here is an example /etc/shorewall/zones file: 
fw firewall
net ipv4
dmz ipv4
loc ipv4
cust ipv4
vpn_a ipsec mode=tunnel mss=1024
vpn_b ipsec mode=tunnel mss=1024
vpn_c ipsec mode=tunnel mss=1024
Here is an example /etc/shorewall/hosts file describing the VPN ranges from the diagram: 
vpn_a eth0:10.1.100.0/24 ipsec
vpn_b eth0:10.1.200.0/24 ipsec
vpn_c eth0:192.168.1.0/24 ipsec
Here is an example /etc/shorewall/policy file based on the table above: 
loc all ACCEPT
vpn_c all ACCEPT
cust net ACCEPT
net cust ACCEPT
all all REJECT
Here is an example /etc/shorewall/rules file based on the network: 
SECTION ALL
# allow connections to the firewall itself to start VPNs:
# Rule  source    dest    protocol/port details
ACCEPT   all       fw                ah
ACCEPT   all       fw                esp
ACCEPT   all       fw                udp 500
ACCEPT   all       fw                udp 4500
# allow access to HTTP servers in DMZ:
ACCEPT   all       dmz               tcp 80
# allow connections from HTTP servers to MySQL database in private LAN:
ACCEPT   dmz       loc:10.2.0.43     tcp 3306
# allow connections from all VPN users to IMAPS server in private LAN:
ACCEPT vpn_a,vpn_b,vpn_c loc:10.2.0.58 tcp 993
# allow VPN users (but not the smartphones in vpn_a) to the
# PostgresQL database for PostBooks accounting system:
ACCEPT vpn_b,vpn_c loc:10.2.0.48      tcp 5432
SECTION ESTABLISHED
ACCEPT   all       all
SECTION RELATED
ACCEPT   all       all
Once the files are created, Shorewall can be easily activated with: 
# shorewall compile && shorewall restart
strongSwan IPsec VPN setup Like Shorewall, strongSwan is also very well documented and I'm just going to focus on those specific areas that are relevant to this type of VPN project.
  • Allow the road-warriors to send all browsing traffic over the VPN means including leftsubnet=0.0.0.0/0 in the VPN server's /etc/ipsec.conf file. Be wary though: sometimes the road-warriors start sending the DHCP renewal over the tunnel instead of to their local DHCP server.
  • As we are using Shorewall zones for firewalling, you must set the options leftfirewall=no and lefthostaccess=no in ipsec.conf. Shorewall already knows about the remote networks as they are defined in the /etc/shorewall/hosts file and so firewall rules don't need to be manipulated each time a tunnel goes up or down.
  • As discussed above, X.509 certificates are used for peer authentication. In the certificate Distinguished Name (DN), store the zone name in the Organizational Unit (OU) component, for example, OU=vpn_c, CN=gw.branch1.example.org
  • In the ipsec.conf file, match the users to connections using wildcard specifications such as rightid="OU=vpn_a, CN=*"
  • Put a subjectAltName with hostname in every certificate. The --san option to the ipsec pki commands adds the subjectAltName.
  • Keep the certificate distinguished names (DN) short, this makes the certificate smaller and reduces the risk of fragmented authentication packets. Many examples show a long and verbose DN such as C=GB, O=Acme World Wide Widget Corporation, OU=Engineering, CN=laptop1.eng.acme.example.org. On a private VPN, it is rarely necessary to have all that in the DN, just include OU for the zone name and CN for the host name.
  • As discussed above, use the ECDSA scheme for keypairs (not RSA) to ensure that the key exchange datagrams don't suffer from fragmentation problems. For example, generate a keypair with the command ipsec pki --gen --type ecdsa --size 384 > user1Key.der
  • Road warriors should have leftid=%fromcert in their ipsec.conf file. This forces them to use the Distinguished Name and not the subjectAltName (SAN) to identify themselves.
  • For road warriors who require virtual tunnel IPs, configure them to request both IPv4 and IPv6 addresses (dual stack) with leftsourceip=%config4,%config6 and on the VPN gateway, configure the ranges as arguments to rightsourceip
  • To ensure that roadwarriors query the LAN DNS, add the DNS settings to strongswan.conf and make sure road warriors are using a more recent strongSwan version that can dynamically update /etc/resolv.conf. Protecting DNS traffic is important for privacy reasons. It also makes sure that the road-warriors can discover servers that are not advertised in public DNS records.
Central firewall/VPN gateway configuration Although they are not present in the diagram, IPv6 networks are also configured in these strongSwan examples. It is very easy to combine IPv4 and IPv6 into a single /etc/ipsec.conf file. As long as the road-warriors have leftsourceip=%config4,%config6 in their own configurations, they will operate dual-stack IPv4/IPv6 whenver they connect to the VPN. Here is an example /etc/ipsec.conf for the central VPN gateway: 
config setup
        charonstart=yes
        charondebug=all
        plutostart=no
conn %default
        ikelifetime=60m
        keylife=20m
        rekeymargin=3m
        keyingtries=1
        keyexchange=ikev2
conn branch1
        left=198.51.100.1
        leftsubnet=203.0.113.0/24,10.0.0.0/8,2001:DB8:12:80:/64
        leftcert=fw1Cert.der
        leftid=@fw1.example.org
        leftfirewall=no
        lefthostaccess=no
        right=%any
        rightid=@branch1-fw.example.org
        rightsubnet=192.168.1.0/24
        auto=add
conn rw_vpn_a
        left=198.51.100.1
        leftsubnet=0.0.0.0/0,::0/0
        leftcert=fw1Cert.der
        leftid=@fw1.example.org
        leftfirewall=no
        lefthostaccess=no
        right=%any
        rightid="OU=vpn_a, CN=*"
        rightsourceip=10.1.100.0/24,2001:DB8:1000:100::/64
        auto=add
conn rw_vpn_b
        left=198.51.100.1
        leftsubnet=0.0.0.0/0,::0/0
        leftcert=fw1Cert.der
        leftid=@fw1.example.org
        leftfirewall=no
        lefthostaccess=no
        right=%any
        rightid="OU=vpn_b, CN=*"
        rightsourceip=10.1.200.0/24,2001:DB8:1000:200::/64
        auto=add
Sample branch office or home router VPN configuration Here is an example /etc/ipsec.conf for the Linux server or OpenWRT VPN at the branch office or home: 
conn head_office
        left=%defaultroute
        leftid=@branch1-fw.example.org
        leftcert=branch1-fwCert.der
        leftsubnet=192.168.1.0/24,2001:DB8:12:80:/64
        leftfirewall=no
        lefthostaccess=no
        right=fw1.example.org
        rightid=@fw1.example.org
        rightsubnet=203.0.113.0/24,10.0.0.0/8,2001:DB8:1000::/52
        auto=start
# notice we only allow vpn_b users, not vpn_a
# these users are given virtual IPs from our own
# 192.168.1.0 subnet
conn rw_vpn_b
        left=branch1-fw.example.org
        leftsubnet=192.168.1.0/24,2001:DB8:12:80:/64
        leftcert=branch1-fwCert.der
        leftid=@branch1-fw.example.org
        leftfirewall=no
        lefthostaccess=no
        right=%any
        rightid="OU=vpn_b, CN=*"
        rightsourceip=192.168.1.160/27,2001:DB8:12:80::8000/116
        auto=add
Further topics Shorewall and Shorewall6 don't currently support a unified configuration. This can make it slightly tedious to duplicate rules between the two IP variations. However, the syntax for IPv4 and IPv6 configuration is virtually identical. Shorewall only currently supports Linux netfilter rules. In theory it could be extended to support other types of firewall API, such as pf used by OpenBSD and the related BSD family of systems. A more advanced architecture would split the single firewall into multiple firewall hosts, like the inner and outer walls of a large castle. The VPN gateway would also become a standalone host in the DMZ. This would require more complex routing table entries. Smart cards with PIN numbers provide an effective form of two-factor authentication that can protect certificates for remote users. Smart cards are documented well by strongSwan already so I haven't repeated any of that material in this article. Managing a private X.509 certificate authority in practice may require slightly more effort than I've described, especially when an organisation grows. Small networks and home users don't need to worry about these details too much, but for most deployments it is necessary to consider things like certificate revocation lists and special schemes to protect the root certificate's private key. EJBCA is one open source project that might help. Some users may want to consider ways to prevent the road-warriors from accidentally browsing the Internet when the IPsec tunnel is not active. Such policies could be implemented with some basic iptables firewalls rules in the road-warrior devices. Summary Using these strategies and configuration tips, planning and building a VPN will hopefully be much simpler. Please feel free to ask questions on the mailing lists for any of the projects discussed in this blog.

Keith Packard: Present

The Present Extension I ve finally gotten an initial implementation of the Present extension written and running and thought I should write up the current design and status. Present Design The current Present extension consists of two requests:
  1. PresentRegion. Puts new bits from a pixmap in a window.
  2. PresentSelectInput. Asks for Present events to be delivered.
PresentRegion This request takes a pile of arguments: 
 
    PresentRegion
    window: WINDOW
    pixmap: PIXMAP
    valid-area: REGION or None
    update-area: REGION or None
    x-off, y-off: INT16
    target-msc: CARD64
    divisor: CARD64
    remainder: CARD64
    idle-fence: FENCE
 
Errors: Drawable, Pixmap, Match
Provides new content for the specified window, to be made visible at the specified time (defined by target-msc , divisor and remainder ). update-area defines the subset of the window to be updated, or None if the whole window is to be updated. valid-area defines the portion of pixmap which contains valid window contents, or None if the pixmap contains valid contents for the whole window. PresentRegion may use any region of pixmap which contains update-area and which is contained by valid-area . In other words, areas inside update-area will be presented from pixmap , areas outside valid-area will not be presented from pixmap and areas inside valid-area but outside update-area may or may not be presented at the discretion of the X server. x-off and y-off define the location in the window where the 0,0 location of the pixmap will be presented. valid-area and update-area are relative to the pixmap. If target-msc is greater than the current msc for window , the presentation will occur at (or after) the target-msc field. Otherwise, the presentation will occur after the next field where msc % divisor == remainder . If divisor is zero, then the presentation will occur after the current field. idle-fence is triggered when pixmap is no longer in use. This may be at any time following the PresentRegion request, the contents may be immediately copied to another buffer, copied just in time for the vblank interrupt or the pixmap may be used directly for display, in which case it will be busy until some future PresentRegion operation. If window is destroyed before the presentation occurs, then the presentation action will not be completed. PresentRegion holds a reference to pixmap until the presentation occurs, so pixmap may be immediately freed after the request executes, even if that is before the presentation occurs. If idle-fence is destroyed before the presentation occurs, then idle-fence will not be signaled but the presentation will occur normally. PresentSelectInput 
 
    PresentSelectInput
    event-id: PRESENTEVENTID
    window: WINDOW
    eventMask: SETofPRESENTEVENT
 
Errors: Window, Value, Match, IDchoice, Access
Selects the set of Present events to be delivered for the specified window and event context. PresentSelectInput can create, modify or delete event contexts. An event context is associated with a specific window; using an existing event context with a different window generates a Match error. If eventContext specifies an existing event context, then if eventMask is empty, PresentSelectInput deletes the specified context, otherwise the specified event context is changed to select a different set of events. If eventContext is an unused XID, then if eventMask is empty no operation is performed. Otherwise, a new event context is created selecting the specified events. Present Extension Events There are three different events for the Present extension:
  1. PresentConfigureNotify
  2. PresentCompleteNotify
  3. PresentRedirectNotify
PresentConfigureNotify This event is moving from the DRI3 extension where it doesn t belong. 
 
    PresentConfigureNotify
    type: CARD8         XGE event type (35)
    extension: CARD8        Present extension request number
    length: CARD16          2
    evtype: CARD16          Present_ConfigureNotify
    eventID: PRESENTEVENTID
    window: WINDOW
    x: INT16
    y: INT16
    width: CARD16
    height: CARD16
    off_x: INT16
    off_y: INT16
    pixmap_width: CARD16
    pixmap_height: CARD16
    pixmap_flags: CARD32
PresentConfigureNotify events are sent when the window configuration changes if PresentSelectInput has requested it. PresentConfigureNotify events are XGE events and so do not have a unique event type. x and y are the parent-relative location of window . PresentCompleteNotify 
 
    PresentCompleteNotify
    type: CARD8         XGE event type (35)
    extension: CARD8        Present extension request number
    length: CARD16          2
    evtype: CARD16          Present_CompleteNotify
    eventID: PRESENTEVENTID
    window: WINDOW
    ust: CARD64
    msc: CARD64
    sbc: CARD64
Notify events are delivered when a PresentRegion operation has completed and the specified contents are being displayed. sbc, msc and ust indicate the swap count, frame count and system time of the related PresentRegion request. PresentRedirectNotify This one is not specified yet, but the intent is for it to contain sufficient information for the compositing manager to be able to construct a suitable screen update that includes an application window update. Finishing GLXOMLsync_control At this point, Mesa only exposes the old swapinterval configuration value, it doesn t provide any of the GLXOMLsynccontrol APIs. However, the Present protocol does have the bits necessary to support glXSwapBuffersMscOML in the PresentRegion request. Let s see how the remaining APIs in this GL extension will be supported. glXGetSyncValuesOML This one is easy; it only needs to return the UST/MSC/SBC values from the most recent SwapBuffers request. Those are returned in the PresentCompleteNotify event, so Mesa just needs to capture that event and save the values away for return to the application. We don t need any new Present protocol for this. glXGetMscRateOML This returns the refresh rate of the monitor associated with the specified drawable. RandR exposes all of the necessary data for each monitor, but the monitor each window is going to be synchronized against isn t exposed anywhere, and is effectively implementation-dependent. So, I think the easy thing to do here is to have a Present request which reports which RandR output a window will be synchronized with. glXSwapBuffersMscOML This is the one API which is already directly supported by the Present extension. glXWaitForMscOML This is effectively the same as glXSwapBuffersMscOML, except that it doesn t actually perform a swap. For this, I think we want a new request that generates an X event when the target values are reached, and then have the client block until that event is received. This will avoid blocking other threads using the same X connection. glXWaitForSbcOML Like glXWaitForMscOML, this just needs to trigger an event to be delivered at the right time. Presentation Redirection I m focusing on finishing the above stuff before I start writing the redirection spec and code, but I ve been thinking a bit about it. What we want is for applications that are presenting a new frame within a redirected window to have that presentation delivered directly to the compositing manager instead of having the bits copied to the redirected buffer and have the compositing manager only learn about this when the damage event is received from copying the bits. Once the compositing manager receives notification that there are new bits available for a portion of a window, it can then get those bits onto the screen in one of two ways:
  1. Copy them to the back buffer for the whole screen and then present that back buffer to the screen.
  2. Present them directly to the screen, bypassing the compositing manager s back buffer entirely.
The first option is what you d do if there were more updates than just a single window; it would update the whole screen at the same time. The second option is what you d do if there was only the one window update to act on, and this would automatically take advantage of page flipping to avoid copying data at all. No need to un-redirect windows for the page flip to work. In both cases, we need to inform the original application both when its pixmap is idle and when the swap actually hits the screen. And we need to make sure the final swap happens when the application requested. For the pixmap idle notification, the current PresentRegion idlefence argument should suffice; we just need to pass the idlefence XID along in the redirection event. Simple. For completion notification, we need to make sure the SBC value for the original window gets incremented and that a PresentCompleteNotify event is delivered. I think that means we want to append a chunk of data to the PresentRegion request so that suitable events can be delivered when the compositing manager PresentRegion occur. I think that just needs to be the window of that original window. To make sure the swap happens at the right time, we just need to have the targetmsc value provided to the compositing manager. I m hoping that having this all event driven is that when a later PresentRegion is redirected that has an *earlier* targetmsc, we can simply queue that as well and things should just work . Current Status I ve got the protocol definitions (both X server and XCB) done, and libxcb supporting the extension. I ve got the X server bits working, but the actual updates are not synchronized to the monitor; they re using OS timers and CopyArea for now, mostly so I can test things without also hacking drivers. Mesa is using the extension, but it only provides the swapinterval value and is not yet supporting the full GLXOMLsynccontrol extension. I ve also got a simple 2D core X application using the extension which is in the shmfd repository in my home directory on freedesktop.org. Availability As always, these bits are already published in my home directory on freedesktop.org in various git repositories.

11 July 2013

Johannes Schauer: Using botch to generate transition build orders

In October 2012, Joachim Breitner asked me whether botch (well, it didnt have a name back then) can also be used to calculate a build order for recompiling ~450 haskell-packages with a new ghc version (it was probably the 7.6.1 release) to upload them to experimental. What is still blocking this ability is the inability of botch to directly read *.dsc files instead of having to rely on Packages and Sources files. On the other hand (in case there exists a set of Packages and Sources files) it is now much easier to use botch for such a use case for which it was not originally designed. To demonstrate how botch can be used to calculate the build order for library transitions, I wrote the script create-transition-order.sh which executes the individual tools in the correct order with the correct arguments. To validate the correctness of botch, I compared the output to the order which ben produces for transitions. The create-transition-order.sh script is called with a ben query string and optionally a snapshot.debian.org timestamp as its arguments. The script relies on ben being installed because ben query strings cannot be translated into grep-dctrl query strings as ben query splits fields which contain comma separated values at the comma before searching for strings. Unfortunately the script also currently relies on a yet unreleased ben feature which allows ben download to create a ben.cache file. You can track the progress of this feature as bug#714703. Creating a ben.cache file is necessary because some queries rely on an association between binary and source packages which is not present in all packages in a Packages file. For example the haskell transition query makes use of this feature by including the .source field in its query. The result of these trials was, that botch produced nearly the same build order for nearly all transitions. The only differences are due to shortcomings of ben and botch. For example, ben is not able to create an order for transitions where involved packages form one or more cycles. A prominent example is the haskell transition where ghc itself is only built during step 15 after many other packages which would have needed ghc to be compiled. Botch solves this problem by reducing all strongly connected components in a cyclic graph to a single vertex before creating the order. This operation makes the graph acyclic and creating a build order trivial. The only remaining problems can then be found within the strongly connected components (for Haskell they are of maximum size two) but the overall build order is correct. On the other hand botch has no notion of packages which are affected by a transition and thus creates a build order which in some cases is longer than the one created by ben. When generating the interdependencies between packages, ben only considers those which are part of the transition. Botch on the other hand considers all dependency relationships. It would be simple to solve this issue in botch by removing unaffected packages from the dependency graph through edge contraction (an operation already used by botch for other tasks). This exercise also let me find another bug in dose3 where libdose would not associate a binary package with a binNMU version with its associated source package but instead report a version mismatch error. This problem was also reported in the process.

10 July 2013

Russell Coker: Samsung Galaxy Note 2

A few weeks ago I bought a new Samsung Galaxy Note 2 Android phone. As I predicted in my post about Phone and Tablet Sizes [1] the Note 2 with a 5.55 display is a bit too big for me to have an ideal single handed side grasp (I estimate that about 5.2 would be ideal). But I can stretch a bit and move it around in my hand to touch all parts of the screen with my thumb. Although when doing that I don t have a tight grip, with my previous phone the Samsung Galaxy S3 [2] I could properly wrap my hand around it to grip it tightly while using it with one hand. The Note 2 will be easier for me to drop or for a thief to snatch it from me. While the big screen makes the phone difficult to use with one hand it does allow viewing more data. The ConnectBot SSH client (Play Store link) [3] (F-Droid repository link) [4] works a lot better on a larger screen I ve also discovered that the volume control buttons can be used to change the font size in ConnectBot which is handy as the default is really tiny. Also Klaus Weidner s Hacker s Keyboard (Play Store link) [5] (F-Droid repository link) [6] works a lot better with a larger screen. When I tested the Hacker s keyboard on a smaller phone I found the 5 row layout too difficult to use, but on the Note 2 it works well. As an aside I wish I could quickly and easily toggle between 4 row mode (good for SMS) and 5 row mode (good for sysadmin work) in the Hackers Keyboard. For less serious use the large screen on the Note 2 is good for watching TV. I ve got a collection of mp4s of TV shows that I ve been meaning to watch, now I watch them in bed on my phone. Another advantage of the Note 2 is the battery life. When playing Ingress and doing all the usual email checks etc my Note 2 will last about as long as my wife s Galaxy S3 with it s power case. So without any extra batteries a Note 2 will probably last about twice as long as a Galaxy S3. The Note 2 has more RAM than the S3 I used to use (I had the S3 with 1G of RAM) and it also apparently has a faster CPU. The CPU speed hasn t been an issue for me but the extra RAM is a real benefit, it means that I can usually switch between Ingress and other programs without having to restart Ingress each time. As an aside I think that Google and LG should release a Nexus Ingress 4 phone or some other device that s optimised for Ingress, I m sure it would sell well. Some people make a big deal about the stylus that comes with the Note 2. It is a neat feature to have the device know when the stylus tip is hovering over the screen but it s not very useful for me. If I was going to create art work on a phone (as some people do) then it would interest me, but I m more interested in email and ssh for fine input and my fingers are generally good enough for those tasks. I have got thin fingers, so I think that people with thicker fingers could really benefit from the stylus. I recently bought a batch of stylus pens from Kogan which have a pen at one end and a rubber stylus tip at the other for fat fingered people I know who need to use an Android phone but can t afford a Note or Note 2. I also like the software build on it which is almost the same as that on the Galaxy S3. It seems that many people prefer the apparently stock features on the Nexus 4 but I like the way Samsung does things. Conclusion I m very happy with my new phone. The bigger screen allows me to see things more clearly, this is good for web browsing, reading email, and now that I can use the Hackers Keyboard I can type more effectively. The longer battery life is really good too, although I think that Samsung could have done better if the phone was 3mm thicker then it could have a much bigger battery and have a larger CCD for the camera. I don t think that the phone is really different, at least not for my use. Samsung has promoted it for artistic use and I ve seen evidence to support their claims. But for most people it ll just be a phone with a larger screen and a bigger battery. Some people criticise it for being too big, but it s still smaller than the handsets on most desk phones so it s not big by the standards of old fashioned phones. The increased size has not only allowed me to do the same things more effectively but also allowed me to do things I hadn t tried doing on a phone before such as watching TV. This isn t because of the phone being particularly special in any way, it s just that the change in size gives more possibilities for ways of using it. The Samsung Galaxy Mega is the largest smart phone. The 5.8 version of the Mega has a resolution of only 960*540 (less than the Note and Note 2 not good enough IMHO) and the 6.3 version has the same resolution as the Note 2 of 1280*720. I think that both Mega variants are too big for me, I need to be able to use a phone with one hand. So it seems that the Note 2 is probably the best phone for me right now.
Related posts:
  1. Samsung Galaxy S3 First Review with Power Case My new Samsung Galaxy S3 arrived a couple of days...
  2. Samsung Galaxy Camera a Quick Review I recently had a chance to briefly play with the...
  3. CyanogenMod and the Galaxy S Thanks to some advice from Philipp Kern I have now...

9 July 2013

Steve Kemp: lumail is complete

So, writing an email client, how did that turn out? Pretty damn well as it happens. My views continue to range from "email is easy" to "email is hard". But I'm now using my home-grown mail-client for 100% of my personal mail-handling. Writing a mail client did seem a little crazy when I started, but the problem breaks down into a few distinct steps and individually they're not so hard: Once you've got that you're almost 80% complete. You're just missing things like "delete", "reply", "compose", and those things are pretty simple to implement. I definitely made the right call in making it scriptable with Lua, because I've been able to write so many functions for working with mail. For example marking messages in all folders matching a regular expression. The code is pretty well structured, and now I've got TAB-completion support on all primitives and user-additions I'm finding it a lot nicer to use. The future? I'm not sure. But right now I'm very happy.

Wouter Verhelst: Mark groet 's morgens de dingen

Or, Mark greeting the things in the morning
Dag ventje met de fiets op de vaas met de bloem Hi little guy with the bike on the vase with the flower
ploem ploem ploum ploum
Dag stoel naast de tafel Hi chair besides the table
Dag brood op de tafel Hi bread on the table
Dag visserke-vis met de pijp Hi fishery-man with the pipe
en and
Dag visserke-vis met de pet Hi fishery-man with the hat
goeiendag good day
Daa-ag vis Bye-ee fish
Dag lieve vis Bye dear fish
Dag klein visselijn mijn Bye my little fishery-fish
The above is one of Paul van Ostaijen's poems, and is, in my opinion, one of his finest. I tried translating it to English so that my non-Dutch-speaking readers could appreciate it (Dutch-speaking readers will most likely already know it anyway), but it's almost impossible to properly translate it. In this poem, van Ostaijen invents a few words that aren't part of the Dutch language proper. A good example is 'visserke-vis'. The first part, 'visserke', is (now) a dialect diminutive of 'visser', which translates to "fisherman" (though it is likely that in Van Ostaijen's days, 'visserke' wasn't considered to be a dialect word). The second part, 'vis', is just Dutch for 'fish'. A literal translation would thus be 'fishereman-fish' On the other hand, given the relative similarity of both words, it is conceivable that Van Ostaijen is just playing with the sound of those words here (he does that later in the poem, anyway), and as such, it's not entirely clear whether he's talking about a fish with a pipe and a had, or about a fisherman with a pipe and a hat. It could be both. His use of the word 'Dag' adds to confusion. In Dutch, the word 'Dag' can be used as a greeting when seeing people for the first time in the morning, but it can also be used as a greeting when leaving people in the evening. Since this is used near the end of the poem, I translated it by using its second meaning; but it's also possible that Van Ostaijen is just playing with us. Of that final line, I should add that the size of the letters is not my doing; Van Ostaijen's original version contained this. In fact, he is known for his expressionistic way of using letters, and some of his other poems go into extremes with that; his wikipedia entry has a nice example. The final verse of the poem sounds pretty well, but defies all grammar. In proper grammar, the verse would read "Dag mijn klein visje". Van Ostaijen turns 'visje' into 'visselijn' so that it rhymes on 'mijn', and puts 'mijn' at the end. Whereas it's still possible to figure out a meaning of the first part of 'visserke-vis', this certainly isn't possible with 'visselijn'. It just isn't a word; all you can make of it is that he's talking about a fish. This certainly contradicts the idea that he was talking about a fisherman earlier. All in all, "Mark groet 's morgens de dingen" is a very funny poem; it describes a man or a boy who does nothing else than say hello to objects when he gets up in the morning; and though it's necessary to explain the language if you don't speak Dutch to fully appreciate it, the idea that there's much more to be found behind the words is most likely wrong. This is just silly amusement.

Raphaël Hertzog: My Free Software Activities in June 2013

This is my monthly summary of my free software related activities. If you re among the people who made a donation to support my work (106.39 , thanks everybody!), then you can learn how I spent your money. Otherwise it s just an interesting status update on my various projects. Debian France I handled some subscriptions that we collected during Solutions Linux at the end of last month and I also worked on updating the accounts in preparation of our annual general assembly. As the current president, I wrote a moral assessment of the organization and I helped Sylvestre Ledru (the treasurer) to prepare an overview of our financial situation (which is rather good). The plan for this general assembly included the renewal of 6 of our 9 board members so I tried to motivate some persons to candidate and I m pretty happy to see that we managed to have 9 candidates for 6 seats, a real election for once. :-) And as usual, I updated Galette to version 0.7.4.2 and I filed a couple of tickets (#621, #622). Debian Package Tracking System We re getting close to one month of work on the Google Summer of Code Project that rewrites the Package Tracking System. On average I spent a good 1h30 per day on reviewing the code, giving my feedback, and preparing the user stories for the next iteration, plus a 2 hour meeting each Wednesday afternoon. This is more than I expected but at I m rather happy with the result so far. We have a live test instance updated each week (after the end of the iteration) at pts.debian.net. The web interface is empty because the bulk of the work has been spent on the email interface for now, but it s now feature complete and already has seen some improvements compared to the current implementation (fixing #340863 for example). You can interact with it by sending mails to control@pts.debian.net. The Debian Administrator s Handbook The fundraising for the liberation of the original book in French met its initial target very quickly and we added new targets with supplementary rewards (more ebook from Eyrolles, donation to Debian). All the supplementary targets were met, although the last one took until the last day! Thanks to all the supporters, we now have a French translation of the Debian Administrator s Handbook which is free and we have collected about 2700 EUR for Debian. Debian Packaging I got a couple of release critical bug reports that I had to fix: And trying to anticipate future issues, I filed an upstream ticket on feed2omb (#33) to see whether upstream had plans to support the new pump.io API that identi.ca is supposed to use soonish. Kali Linux I filed a couple of Debian bugs: #711866 to request a new upstream version of w3af in Debian, #711044 to report a missing dependency in LSB meta-informations of the vsftpd init script. I worked on packaging passing-the-hash. Basically this is a fork of several software just to override some NTLM authentication functions. In some cases, I managed to avoid the fork by transforming the patches in LD_PRELOAD libraries that override the specific symbol of the patched functions. Among the forked software, there was winexe which wasn t packaged yet. So I packaged it but we quickly ran into some segfaults (which was already reported to upstream since a few months). I investigated the problem and prepared a patch. It has been submitted to upstream and merged. Thanks See you next month for a new summary of my activities.

No comment Liked this article? Click here. My blog is Flattr-enabled.

5 July 2013

Justus Winter: mount, umount and mtab

This week was all about getting mount into shape, addind umount and hacking up a prototype for the mtab issue. With all my patches to mount the boot log looks like this: 
INIT: version 2.88 booting
Using makefile-style concurrent boot in runlevel S.
Activating swap...done.
Checking root file system...fsck from util-linux 2.20.1
hd2 : tray open or drive not ready
hd2 : tray open or drive not ready
hd2 : tray open or drive not ready
hd2 : tray open or drive not ready
end_request: I/O error, dev 02:00, sector 0
ext2fs_check_if_mount: Can't check if filesystem is mounted due to missing mtab file while determining whether /dev/hd0s1 is mounted.
/dev/hd0s1: clean, 44101/181056 files, 283413/723200 blocks
done.
Cleaning up temporary files... /tmp.
/etc/rcS.d/S06mtab.sh: 48: /etc/rcS.d/S06mtab.sh: cannot open /proc/mounts: No such file
Activating lvm and md swap...(default pager): Already paging to partition hd0s5!
done.
Checking file systems...fsck from util-linux 2.20.1
hd2 : tray open or drive not ready
hd2 : tray open or drive not ready
end_request: I/O error, dev 02:00, sector 0
done.
Mounting local filesystems...done.
Activating swapfile swap...(default pager): Already paging to partition hd0s5!
done.
df: Warning: cannot read table of mounted file systems: No such file or directory
Cleaning up temporary files....
Configuring network interfaces...inetutils-ifconfig: invalid arguments
ifup: failed to open pid file /run/network/ifup-/dev/eth0.pid: No such file or directory
Internet Systems Consortium DHCP Client 4.2.2
Copyright 2004-2011 Internet Systems Consortium.
All rights reserved.
For info, please visit https://www.isc.org/software/dhcp/
can't create /var/lib/dhcp/dhclient./dev/eth0.leases: No such file or directory
Listening on Socket//dev/eth0
Sending on   Socket//dev/eth0
DHCPDISCOVER on /dev/eth0 to 255.255.255.255 port 67 interval 7
DHCPREQUEST on /dev/eth0 to 255.255.255.255 port 67
DHCPOFFER from 10.0.2.2
DHCPACK from 10.0.2.2
can't create /var/lib/dhcp/dhclient./dev/eth0.leases: No such file or directory
bound to 10.0.2.15 -- renewal in 42107 seconds.
done.
Cleaning up temporary files....
Setting up X socket directories... /tmp/.X11-unix /tmp/.ICE-unix.
INIT: Entering runlevel: 2
Using makefile-style concurrent boot in runlevel 2.
Starting enhanced syslogd: rsyslogd.
Starting deferred execution scheduler: atd.
Starting periodic command scheduler: cron.
Starting system message bus: dbusFailed to set socket option"/var/run/dbus/system_bus_socket": Protocol not available.
Starting OpenBSD Secure Shell server: sshd.
GNU 0.3 (debian) (console)
login:
As you can see, mountall.sh works fine now (it's the line about mounting local filesystems). This is a list of issues I tackled this week: Note that even though I've implemented umount, it doesn't actually work yet. That is because umount needs a mtab file or /proc/mounts to do anything, because you can either invoke umount with the mount point or the device and umount needs an mtab style file to figure out whether the argument is referring to the former or the latter. So the next big issue is the mtab issue. There is some information about that in the Hurd wiki, it even is a gsoc project idea of its own: http://www.gnu.org/software/hurd/community/gsoc/project_ideas/mtab.html But I need this now, this cannot wait for another gsoc. The implementation outlined by that page consists roughly of: If this hypothetical mtab translator is bound to say /proc/mounts this would solve all our problems: 
% settrans -ca /proc/mounts /hurd/mtab /
As this involves adding a RPC procedure and keeping a list of passive and active translators, this involves some design decisions. As discussed in #hurd I hacked up a prototype and post it for review: http://lists.gnu.org/archive/html/bug-hurd/2013-07/msg00022.html As soon as I get the necessary feedback on this issue, I will get the implementation up to speed. Until then, I will pick another item from my status list, most probably the ifupdown issue. Speaking of feedback, all of my patches I sent last week were merged at the appropriate locations. That was really quick, many thanks for that :) While working on the mount issues, I frequently looked at the implementation of Linux' and FreeBSD's mount to see how --types is handled there. I must admit that I was not to keen about the idea of writing c again, but I was positively surprised by the code I've seen from Hurd so far. As Hurd is the operating system of the GNU system, its code is free to use any GNU extension to libc and gcc including nice stuff like nested functions. If you look at the different mount implementations, you'll find the Hurd version to be the most modern and clean one imho:
Linux
http://sources.debian.net/src/util-linux/2.20.1-5.4/mount/mount.c
FreeBSD
http://sources.debian.net/src/freebsd-utils/9.0%2Bds1-11/sbin/mount/mount.c
Hurd
http://sources.debian.net/src/hurd/20130620-1/utils/mount.c
See you next week :)

4 July 2013

Emanuele Rocca: Useful tools for Python developers

Python is a great language with an impressive number of tools designed to make developers' life easier. Sometimes, however, the problem is getting to know that these tools exist in the first place. By contributing to projects like OpenStack's Nova client and Falcon, I have recently come across some useful tools that can seriously improve the quality of your code. The first one is called pyflakes, a passive checker of Python programs developed by Phil Frost. What it does is parsing your source files and checking for possible errors such as undefined names and unused imports. Let's consider the following example: <figure class="code">
1
2
3
4

import urllib

print "pyflakes example"
urlib.urlopen('http://www.linux.it')
</figure>The code above contains a typo, we have misspelled urllib. Here is what pyflakes thinks about our program: 
$ pyflakes example.py
example.py:1: 'urllib' imported but unused
example.py:4: undefined name 'urlib'
On line 4 we try to use urlib which is not defined. Also, we import urllib on line 1 and we do nothing with it. Our typo has been spotted! Notice that, even though our program contains a print statement, 'pyflakes example' has not been printed. That is because pyflakes parses the source files it checks, without importing them, making it safe to use on modules with side effects. pyflakes can be installed with pip or apt-get. The second tool I want to talk about is Ned Batchelder's coverage.py. No doubt you write unit tests for your programs. Right? Good. coverage.py is out there to help you checking how much of your program is actually covered. Let's use as an example codicefiscale, a Python project of mine. First we install coverage: 
pip install coverage
Then we run our unit tests: 
$ coverage run --source=codicefiscale tests.py
.......
----------------------------------------------------------------------
Ran 7 tests in 0.003s
We pass the module we want to test with --source=codicefiscale so that coverage will only report information about that specific module. Now that our tests have been performed successfully it is time to check how much of our code is covered by unit tests: 
$ coverage report -m
Name            Stmts   Miss  Cover   Missing
---------------------------------------------
codicefiscale      73      4    95%   61, 67, 95, 100
Not bad, 95% of our module is covered! Still, coverage let us know that 4 lines have not been touched by the unit tests. With this information, we can go write some meaningful test cases that will also cover the missing lines.

Daniel Pocock: My Linux server IPv6 deployment approach

I previously discussed the ease of deploying IPv6 for Linux servers. Whether it is Debian, Fedora or another distribution the IPv6 stack should "just work" these days. However, for maintaining a production network with minimum risk of interruption, there are a few extra things to be aware of during IPv6 deployment. Here I present a rough set of steps that can be followed, usually with minimal or no downtime, to deploy IPv6. The steps are in the order that they should be used in practice Plenty of other web sites already explain the theory of IPv6 networks or practical configuration detail (like how to set a static IPv6 address in Debian). This page ignores all of that detail and looks at the overall project strategy. Borrow the last decimal octet (or host bits) of IPv4 addresses As mentioned in my earlier blog:
  • Of the 128 bits in IPv6 addresses, most sites now use a fixed 64 bit netmask for all their subnets. This is the practice encouraged by RFC 4291.
  • Therefore, there are always more bits in the 64-bit "host" portion of the address than in the host portion of any 32-bit IPv4 address.
  • Consequently, it is possible to just borrow the host portion (typically the last octet for a class-C network) and use it as the host portion of an IPv6 address
IPv4 address Host portion of address IPv6 address
192.168.1.5/24 5 2001:1234:567::5
Gotcha: remember that the IPv4 addresses are usually written in decimal while IPv6 addresses, including the example above, are hex. This doesn't really matter in practice unless the addresses are being manipulated by code that is sensitive to the host bits. For all other intents and purposes, it just makes it easy to read the addresses and associate them with IPv4. A completely valid alternative is to use the full IPv4 address as the host portion of the IPv6 address. For example, if the IPv4 address = 192.168.1.5, the IPv6 address may be 2001:1234:567::192.168.1.5. This approach is slighly more verbose and is completely valid. Which approach you use is at your discretion. Create reverse mappings (PTR records) in DNS Just as IPv4 has the in-addr.arpa zone and telephone numbers have the e164.arpa zone (for ENUM), IPv6 addresses have a reverse mapping zone too: ip6.arpa The recommended way to proceed is to create reverse mappings for all hosts that have IPv4 reverse mappings. Some services may be using reverse lookups for authentication (e.g. MySQL will check this for any user who has a "Host" restriction defined). This is why it is recommended to create all the reverse mappings very early in the IPv6 program, before any hosts start making connections over IPv6. Create extra A records in DNS It's possible that some applications will be more stubborn about IPv6 adoption than others. For example, if you have a host called wolf.example.org and it hosts a mail server, web server, DNS server and XMPP server all using the name wolf then all those server processes will have to support IPv6 connectivity from the moment you add an AAAA record. A good practice is to create independent DNS A records (such as ns1.example.org and mail.example.org) for each application (not just CNAME records). Existing CNAME records can be converted to A records and converted back to CNAME records after the IPv6 deployment is 100% complete. Duplicate firewall entries Duplicate all IPv4 firewall entries to create an IPv6 firewall For many types of network, this is a trivial (although possibly tedious) task. There are some small gotchas:
  • There is no NAT. Any firewall entries for NAT need to be reviewed: for source NAT, it is necessary to use a mechanism like connection tracking to protect workstations that previously had asymmetrical access to the public Internet. For destination NAT, see the TPROXY feature in ip6tables
  • IPsec works slightly differently. Rules for AH and ESP packets can't simply be duplicated for IPv6, they need further tweaks or warnings will be generated.
  • IPv6 makes some ICMP packages mandatory. If IPv4 ICMP is completely blocked, you still need to enable the IPv6 ICMP (or at least the subset of ICMP packets that are mandatory).
For more comments about firewalling IPv6, please see my earlier blog on the subject Put the IPv6 addresses on the hosts Add the necessary records to the appropriate place, for example, the /etc/network/interfaces files on Debian/Ubuntu style systems. For hosts that have multiple addresses on a single interface, it is often desirable to ensure that outbound connections always appear to come from just one of the addresses. This can be achieved using the preferred_lft option when configuring the non-default addresses. For example, if configuring three addresses on eth0: 
ip addr add dev eth0 preferred_lft 0 2001:1234:567::10/64
ip addr add dev eth0  2001:1234:567::11/64
ip addr add dev eth0 preferred_lft 0 2001:1234:567::12/64
In the example above, all outgoing IPv6 connections will appear to come from 2001:1234:567::11/64 Important: this is only a very shallow discussion of the issues around source addresses. If it doesn't provide a valid solution for you, there are many dedicated articles about this specific topic, madduck's blog looks at it for firewalls and tunnels but the concepts are valid for any type of multi-homed host Review listening processes with netstat Now that hosts have IPv6 addresses, it is worthwhile looking through the output of 
netstat -nlp
to see which processes have bound to both IPv4 and IPv6 addresses and which processes remain stuck on IPv4 only. For processes that are only listening on IPv4 or only bound to a specific address (rather than 0.0.0.0), it is a good idea to investigate the way they are configured and find out if they will support IPv6. Review the monitoring infrastructure (Nagios, Ganglia, etc) Now is a good time to review the monitoring infrastructure. Make sure that tools like Nagios are testing the services using their dedicated DNS names and not the host names. For example, if the host wolf.example.org runs an LDAP server accessible as ldap1.example.org, make sure Nagios is configured to poll ldap1.example.org. Look at the check_v46 wrapper for Nagios and similar solutions to make sure that Nagios is testing both the IPv4 and IPv6 version of each service. Just as developers use unit testing as part of test-driven development workflows, administrators deploying IPv6 can use the monitoring framework to validate the progress of their IPv6 program. Check for ACLs Do any server processes use IP-based ACLs to control access? For example, an Apache web server may be using an IP ACL to restrict access to some pages. MySQL or Postgres databases may have IP ACLs defined for some users or databases. When hosts start using IPv6 to connect to services, their IPv6 source addresses won't pass the ACL and services may be inaccessible. Therefore, it is worthwhile checking for such ACLs early on. Prepare DNS servers first Preparing the DNS servers first is a good step. The DNS servers should be configured to accept queries over IPv6. Once the DNS servers are listening on IPv6 addresses, go ahead and do some of the following:
  • Insert IPv6 AAAA records for the name servers in their own zone files. As mentioned in the previous tip, the name server records should have their own independent A records already with names such as ns1.example.org. Consequently, no other application should notice when the AAAA record is created.
  • Create IPv6 glue records for any name servers that have IPv4 glue records
Configure secondary name servers to poll over IPv6 Configure the secondary nameservers to poll the primary nameserver using it's new IPv6 address. Update resolv.conf on all hosts Now that DNS servers are listening on IPv6, it is possible to put IPv6 nameserver addresses in /etc/resolv.conf on all the other hosts. Configure and test other low-level services Although services such as LDAP and MySQL may not be exposed to public Internet users, they are an essential foundation for many other applications to run. Make sure all these low level services are accessible over IPv6. Test them. After testing each of these services, it is OK to go ahead and create the AAAA records for the service. Configure and test other services Continue testing each service is accessible over it's IPv6 address and then add the necessary AAAA record. This includes all the high level services now, such as mail and web servers. Configure and test hostnames Finally, start adding AAAA records associated with hostnames to the DNS zone. Now it should be possible to ssh or ping6 the hostnames using IPv6. You may also want to add the IPv6 address of each host into it's own /etc/hosts file at this stage.

3 July 2013

Matthew Garrett: Explaining Intel Rapid Start Technology

Recent Intel-based systems often implement something called Intel Rapid Start Technology. Like many things with the word "Technology" in the name, there's a large part of this that's marketing. The relatively small amount of technical documentation available implies that it's tied to your motherboard chipset and CPU, but as far as I can tell it's entirely implemented in firmware and could work just as well on, say, a Cyrix on a circa 1996 SIS-based motherboard if someone wrote the BIOS code[1]. But since nobody has, we're stuck with the vendors who've met Intel's requirements and licensed the code.

The concept of IRST is pretty simple. There's a firmware mechanism for setting a sleep timeout. If you suspend your computer and this timeout expires, it'll resume. However, instead of handing control back to the OS, the firmware just copies the entire contents of RAM to a special partition and turns the computer off. Next time you hit the power button, the firmware dumps the partition contents back into RAM and resumes as if nothing had changed. This takes a few seconds longer than resume from S3 but is far faster than resume from hibernation since it starts the moment the system gets power.

At a more technical level, it's a little more complicated. The first thing to know about this feature is that it's entirely invisible unless your hard drive is set up correctly. There needs to be a partition that's at least the size of your system's physical RAM. For GPT systems, this needs to have a type GUID of D3BFE2DE-3DAF-11DF-BA-40-E3A556D89593. For MBR systems, you need a partition type of 0x84[2]. If the firmware doesn't find an appropriate partition then the OS will get no indication that the firmware supports it. Boo.

(The second thing is that it seems like it really does have to be on an SSD, and if you try to do this on spinning media your firmware will ignore it anyway)

If all the prerequisites are in place, an ACPI device with an HID of INT3392 will exist. It has four methods associated with it: GFFS, SFFS, GFTV and SFTV. GFFS returns an integer representing the events that will cause the system to wake up from S3 and suspend to SSD. The system will wake after the timeout expires if bit 0 is set, and will wake when the battery becomes critically low if bit 1 is set. The other bits appear to be unused at the moment[3]. SFFS sets the wakeup events, using the same bit values as GFFS. GFTV returns an integer containing the current wakeup timeout in minutes. SFTV sets it. Values above 1440 (ie, 24 hours) seem to be considered invalid - if I set them the value instead ends up as 10 and the timeout flag gets cleared from the wakeup events field.

I've submitted a patch that adds a sysfs interface for setting these values, and unless anyone objects it'll probably end up in 3.11. There's still the remaining question of how userspace should make use of these, and also how installers should behave when it comes to systems that support IRST. As previously mentioned, there's no obvious indication to the OS that the feature is supported unless the appropriate partition already exists. The easiest way to deal with this is for installers to default to retaining any partitions with the magic IDs, but I'm still looking into whether it's possible to get the firmware to cough up some more information so it can be created automatically even if the drive's entirely blank.

And now, having got this working on a test machine, I just need to split my Thinkpad's swap partition in half and make sure it works here as well. Woo.

[1] Note: I am not going to do this.
[2] Conveniently, the same as the partition type that APM systems used for suspend to disk back when dubstep hadn't been invented yet
[3] At least, if you attempt to set them they get ignored.

comment count unavailable comments

Ian Campbell: Emesinae -- Yet Another Bug Tracker

Quite a while ago at a Xen hackathon (I think the Cambridge one in 2011, has it really been 2 years?) the topic of bug tracking came up. The developers who were present (myself included) expressed a general dislike for web based bug trackers, and in particular the existing bugzilla wasn't working for us for a variety of reasons (primarily lack of resource for care and general tending of it). They also liked the existing work flow based around mails to the Xen mailing lists, there was a feeling that this fits well with the demographics of the bugs which get reported to Xen: a sizable number of the issues are configuration errors or misunderstandings which can usually quickly be dealt with through a small email thread (followed by a request to please update the wiki). The project also gets quite a few bug reports which are initially "unactionable" due to of lack logs or a coherent description, in these cases a little bit of back and forth can quickly determine whether the reporter is going to provide the necessary additional information (this is something we are also trying to fix with clearer guidance and an ongoing effort to provide a reportbug-like tool for Xen). However there were concerns that bigger issues which couldn't be dealt with quickly were falling through the cracks. We kicked around a few ideas for how a bug tracker that met our needs might work and that was mostly that. Since then I've kicked the ideas around in my head a bit and searched for an existing system which would fit what was required. In the end I didn't find anything which quite fitted the bill and about six months ago I was looking for an interesting hacking project (perhaps something a bit different to my day job, hypervisor and kernel hacking) so I decided to bite the bullet, dust of my rusty Perl skills and hack something together. Yes, that's right, I've written Yet Another Bug Tracker(tm). I confess I feel a bit dirty for doing so. I've taken a fair bit of inspiration from debbugs (the Debian bug tracker) but I've turned the concept a little on its head. Debbugs is effectively instantiating a new mailing list for each bug report (each NNNNNN@bugs.debian.org can be thought of as a mailing list plus some additional bug specific metadata). Instead I implemented a model which takes an existing mailing list and acts as a list archive which provides a mechanism to track individual threads (or subthreads) as bugs (each bug is effectively a root Message-Id plus a little bit of metadata). In common with debbugs bugs are created and subsequently manipulated by sending mail to a special control@ address. Commands are available to create bugs as well as to graft and prune subthreads onto existing bugs (to cope with people who break threading or offtopic subthreads etc). There is also a read-only web interface for browsing the bugs. Internally the system is subscribed to the list and records details of each message and their relationships (e.g. References headers etc) in a sqlite database. A set of CGI scripts (using the Perl CGI module) provide an interface to browse bugs. Since the system tracks bugs based on mailing list threads I've named it Emesinae which is a "thread-legged bug" (yes, I spent longer than is strictly necessary browsing entomological websites). I've made the code available on gitorious under a GPLv2 license, see https://gitorious.org/emesinae. I announced an alpha instance of Emesinae back in May which tracks the xen-devel mailing list. You can find the bug tracker at http://bugs.xenproject.org/xen/. Since then I've been slowly fleshing out some more useful features (such as tagging of affected branches). It's not seen a great deal of uptake from the Xen developers yet but there are already some bugs being tracked and I'm hopeful that it will prove useful in the long run. In any case it was a fun hacking project for a few rainy evenings (and a couple more train and plane journeys).

1 July 2013

Russ Allbery: Review: Dark Lies the Island

Review: Dark Lies the Island, by Kevin Barry
Publisher: Greywolf
Copyright: 2012
Printing: 2013
ISBN: 1-55597-651-4
Format: ARC
Pages: 185
This is a literary short story collection by a mainstream Irish writer, rather far from my normal fare. It was the second book in a shipment of Powell's Indiespensable, which is generally (when there is one) a proof or Advanced Reading Copy of something relatively obscure. I'm not really the target audience, not being much of a short story aficionado, but the whole point of doing this Indiespensable experiment is to stretch my reading range. First off, as I would generally expect by mainstream fiction, the quality of the writing is excellent. One thing I particularly like about Barry is that he's refreshingly and unhesitatingly profane. His characters curse like I expect people to curse, some of them at the drop of a hat, and for me it gives a feel of solid realism to his dialogue. He's also very good with eye dialect, giving the impression of accents without making the text hard to read. Some of these stories I liked a great deal, rather more than I had expected. But if there was a general undermining flaw for me, it was that I prefer stories that go somewhere, that have a plot or lead up to a definite point. Some of Barry's do, but a lot of them seem more vignettes or just scenes, sketches of characters, that drift off without resolution. I can admire those as writing exercises, but for me they don't fill the desire I have when I sit down with a story. I'm not sure how useful these reviews will be, given that I don't have a lot of mainstream fiction background to compare them with, but here are some reactions, for whatever they're worth. "Across the Rooftops": This is one of those sketches, but it's one of the best of them. It's a single moment on the rooftops, an attempt to capture the moment of possibility around the first romantic overture. It's sadly quite conventional in gender roles (nervous man initiating, woman as relationship gatekeeper), but apart from that it's very well-written. This kind of thing is normally not to my taste, but Barry does such a great job capturing the sense of uncertainty and the sharp focus on detail that I couldn't help but like it. (7) "Wifey Redux": This is by far my favorite story of this book. I think it's both hilarious and brilliant, right from the starting line.
This is the story of a happy marriage but before you throw up and turn the page let me say that it will end with my face pressed hard into the cold metal of the Volvo's bonnet, my hands cuffed behind my back, and my rights droned into my ear this will occur in the car park of a big-box retail unit on the Naas Road in Dublin.
It's the story of a marriage and how it changes over time, but also the story of being a father and his baffled, flailing reactions to his daughter's sexual explorations. But that's not what makes the story. What makes the story is what leads up to the arrest at the end, which had me laughing out-loud in delight. Barry is great at writing a sort of devilish humor; I wish more of the collection was like this. (9) "Fjord of Killary": This is another one of the ones I enjoyed. It's about an author who buys a hotel (with a bar) on the fjord of Killary thinking that the local culture and contact along with the wild beauty would inspire his poetry. At the start of the story, he's thoroughly sick of both the climate and the people, and the collision between his idyllic dreams and the reality, as well as his acerbic first-person complaints, is quite entertaining. A huge storm and resulting flood provides the conflict and impetus for change, and a few moments of humor in the understated local reactions. I thought the closing epiphany was a bit too obvious and expected, and the ending didn't quite work for me, but it has some good moments. (7) "A Cruelty": Here's where the collection started losing me. This is still well-written and closely observed, but in the case of this story, I wish it weren't, since I found it extremely uncomfortable. The viewpoint character is vulnerable to an encounter of unalloyed nastiness. I'm sure that's the point, but, similar to how I avoid horror, it's the sort of story I'd rather not read. (3) "Beer Trip to Llandudno": This was my second-favorite story of the collection. It's about a group of men who take periodic beer-tasting trips and rate beers. They're just normal men, from a variety of backgrounds and professions, and I thought Barry did a great job capturing the sort of camaraderie that comes from long association with a hobby. That includes, here, the awkward limits on what one talks about and the tendency to use the hobby as a shield from anything that drifts into less certain territory. I thought the story suffered somewhat from not having much of a plot or conclusion, but that's also necessary from the setup. This group of people isn't actually going to solve problems, but will be there for each other in their own way. (8) "Ernestine and Kit": Another very dark story, although that's not quite as obvious at the start. As with "A Cruelty," it's well-written but disturbing and lacks any sort of positive resolution. Here, I think Barry went a bit far into portraying people who are a large part of the popular imagination but who are exceptionally rare in reality. It's the kind of story that, at least for me, plays badly with my brain's threat analysis: too memorable for the level of actual risk, and feeling like it was playing a bit too much with popular boogeymen. (4) "The Mainland Campaign": This one was just too subtle for me. Some of the summaries for this book hint at a meaning for the story that I didn't pick up at all on my first reading. On re-reading, I suspect that analysis is right, but even re-reading the story with that interpretation in mind (this is the sort of story where I'm not sure the average American would pick it up on first reading, but others might), I still don't understand exactly what happened. I think some mainstream writing, and some short stories, tends to err on the side of subtle. (4) "Wistful England": Another one that's way over on the sketch of a moment side of stories. Unlike "Across the Rooftops," it failed to make any impression on me whatsoever; I had to skim it just now to remind myself of any details at all, and by the time I edited this review for posting, I'd forgotten it again. I think it was aiming at capturing a mood, but the mopey viewpoint character didn't mean anything to me. (3) "Doctor Sot": This is another odd one, and I'm not quite sure what I think of it. Either it's another character sketch without much of a plot, or the plot was too subtle. The protagonist is a rather unlikable drunk who doubles as a small town's incompetent doctor. The story involves his encounter with a local sort of hippie encampment, which is moderately interesting even though I couldn't stand the man. Then it leads to an encounter that clearly meant something, but which was entirely lost on me. (4) "The Girls and the Dogs": And from an absence of plot back to the deeply disturbing. This story is about a moderately unlikable drug dealer who gets refuge from an extremely unlikable crazy person in a weird relationship with a woman and her sister. (As is sadly typical, polyamory only makes an appearance when the author wants to paint a scenario that's utterly fucked up and psychotic.) There's a pseudo-fantasy element here in that the characters talk about spells, but I think the only spell involved is authorial fiat to create a deeply broken and sick game that plays into a host of negative stereotypes. Despite myself, I did get drawn into the suspense of the story (Barry's undoubtedly a good writer), but it left me feeling dirty on several levels. (4) "White Hitachi": Finally, another story that I liked, although not as good as the ones earlier in the collection. This one follows a con man and two-bit thief as he gets his brother out of lockup and tries to resolve, or at least stay ahead of, various problems in his life. Again, no real ending, but I thought Barry did a good job at taking a snapshot of a life from a perspective that I don't read much in literature (along with all the sexism and racism that comes along with it). It sort of drifts off rather than ends (although the last sentence is a great bit of characterization), but I found it oddly enjoyable. (6) "Dark Lies the Island": I'm not sure what to make of this. It's another character sketch, a moment of deep psychological significance for a character, floating somewhat alone but described in detail. But the character is a girl struggling with cutting, from a male author whose other stories did not fill me with confidence in his handling of female characters. Barry makes this psychologically believable, but I'm a little nervous about trusting his portrayal of that mindset. It's also, again, just a bit too subtle; I had a hard time getting an emotional feel for how the events of the story connect. (5) "Berlin Arkonaplatz My Lesbian Summer": The final story of the collection is fairly typical of the problem I had with most of it. It seems reasonably well-written, I had a hard time connecting with the protagonist or understanding his motivations. It looks at a slice of life with which I have no familiarity at all (and is interesting for that reason), but I don't quite trust the story enough to submerge in it. Barry seems to write a lot of protagonists who seem aimless, befuddled, or just largely out of control, and I like a bit more agency in the primary character. Silvija is an interesting character, even shown from another's perspective, but the story is almost purely observational: showing her without comment, without obvious motive, and mostly without plot. It's strangely compelling, but not quite compelling enough that I'd actually seek it out. Which, really, is the whole book in a nutshell. (6) Rating: 6 out of 10

Russ Allbery: Pod::Thread 1.01 and faq2html 1.32

Julien LIE pointed out a rendering problem with one of my pages, and some further investigation revealed that the changes that I made to faq2html when moving away from Subversion caused all sorts of problems. I started testing a new version of it, and then found a bunch of other problems with pod2thread. One I already knew about, but others I'd not seen before. Since I was working on this anyway, I decided to fix all the problems I knew about. Pod::Thread 1.01 now correctly encodes its output for UTF-8, which will fix some long-standing problems with ISO 8859-1 characters used to render non-breaking spaces. It also fixes some minor formatting issues and adds a workaround for the inability to format formatting codes in headings when generating an artificial table of contents. faq2html 1.32 now supports multiple subheadings at the top of a page provided that the start of the second subheading is still a centered line. I use this format in a bunch of my package README files, all of which had been misformatted due to not having this support. I did a large test generation of my web pages, and I don't think I added any new problems. (faq2html is inherently tricky, since it's converting plain text to HTML and has to do a bunch of markup guessing. I suppose one of these days I should just give up and switch over to writing README files in Markdown or something.) While I was at it, I also modernized Pod::Thread's module packaging and switched it to Module::Build, which is the direction I'm (slowly) going for all of my Perl module work. The generic parts of the test suite also got a refresh to the latest versions of my generic test suite code (which I'm currently maintaining in rra-c-util for lack of a better location). That had one interesting wrinkle. Pod::Thread was the one Perl distribution where I'd played with using a module filter program (something ExtUtils::MakeMaker supports) to set the module $VERSION automatically during the build. This had the advantage of letting me maintain the distribution version in one place (Makefile.PL) no matter how many Perl modules there are. However, Module::Build doesn't support that. So, this afternoon I wrote Module::Build::SetVersion, which overrides the processing of *.pm files to set the $VERSION string to whatever one sets dist_version to in Build.PL. This lets you just put placeholder strings in all of the modules, and the process of building them into blib will substitute in the current distribution version. Currently, this module is only in this package and subclasses Module::Build (it really should be a mixin of some kind, but I didn't work out a good way of doing that). I'm going to experiment with copying it into my various module packages, but I suppose at some point I should wrap it up in its own distribution, add a test suite, and release it properly. (If you would find this useful, drop me email and I'll prioritize releasing it properly on CPAN somewhat higher.) Note that you can do the same thing with the existing Module::Build::PM_Filter module, as well as a bunch of other things, but that requires you write and maintain a Perl script at the top level of the distribution that does the desired transformations and seemed unnecessarily complex for what I was trying to do. I really wanted something I could just drop in and forget about. You can get the latest releases of Pod::Thread and faq2html from my web tools distribution page.

30 June 2013

Russ Allbery: Random haul

A random book haul, for no particular reason other than that it had been a while and several books came out that I have been eagerly awaiting (in one case for many years): Ann Aguirre Wanderlust (sff)
Ann Aguirre Killbox (sff)
Ann Aguirre Doubleblind (sff)
Ann Aguirre Aftermath (sff)
Ann Aguirre Endgame (sff)
Yoon Ha Lee A Conservation of Shadows (sff collection)
M.J. Locke Up Against It (sff)
China Mi ville Railsea (sff)
Chris Moriarty Ghost Spin (sff)
Paul Nahin An Imaginary Tale (non-fiction) Yes, Ghost Spin finally exists, and I have held it in my hands. Now if I can just remember what was happening in Spin State and Spin Control. I may just have to re-read them, which would be such a hardship.... The explanation of all the Aguirre purchases is that if you say sensible things amid the latest round of obnoxious sexism in the SFF community, and particularly when anonymous cowards claim your writing is obviously bad because of what you say, there are people like me who may well buy your entire series, new, as a show of support. Particularly if it was already on their radar as something to read. I already had the first book, but I bought all the rest sight unseen, based just on the strength of her commentary and some of the things other people have said about the books. This afternoon, I finished reading Grimspace, the first in the series, and now I'm delighted that I already have the others in hand. Excellent stuff; a bit trying on suspension of disbelief in a few spots, but the pacing was so good I didn't care. Review coming eventually when I catch up further. Yoon Ha Lee is my absolute favorite SF short fiction author writing currently, and that's saying something since I generally don't like short fiction. I've been eagerly awaiting her first collection ever since I heard it had gone to print. Also, in the context of picking up eight more SF novels or collections by women, I want to note just how irritating it is when people claim that women don't write science fiction. I read quite a lot of science fiction, and I haven't had any trouble finding female authors. I haven't even needed to try; the books I read run about half female authors with no special attention or effort on my part. If someone honestly hasn't heard of those writers, I have a bunch of suggestions in my collected book reviews, but too often it seems less a plea for help and more a rhetorical point about how science fiction is somehow more naturally male. Which is complete bullshit, and I have the library and reviews to prove it. Last year I read 13 unambiguous science fiction novels by women (not counting any fantasy). Some of them I liked, some of them I didn't, some of them were more believable than others, and some of them were harder (in the sense of being more scientifically plausible) than others just like the 11 science fiction novels, not counting fantasy, by men that I read in the same time frame. So far this year, I've reviewed one science fiction novel by a woman and two by men, and I have two more by women (and one by a man) waiting to be reviewed. I'm not doing anything special, just reading what's published, what's been recommended to me, and what wins awards. Anyone else can do exactly the same thing.

28 June 2013

Vincent Bernat: lan o: a task launcher powered by cgroups

A few months ago, I was looking for a piece of software to spawn long-running tasks on behalf of some daemon with the benefit of the tasks not being interrupted when this daemon is restarted. Here are the specifications: The last requirement explains why tasks are not spawned directly by the daemon requesting them: its complexity or the way it needs to be operated may need frequent restarts. Even if it is possible to re-execute a daemon while keeping its children, like stateful re-exec support in Upstart, this is quite difficult: internal state should be serialized and restored. Part of this state can be contained into a third-party library. While Upstart or systemd seemed to be good candidates for this purpose, I didn t find a straightforward way to run arbitrary tasks without any privilege1. Here comes lan o2. It is a very simple task launcher. It can run any tasks, stop them and check if they are still running. It leverages cgroups in recent Linux kernels and avoids the use of any daemon.

Quick tour Before looking at how lan o works, let s have a look at how it can be used. To avoid usage conflicts, each task is run in the context of a namespace that needs to be initialized: 
$ sudo lanco testns init -u $(id -un) -g $(id -gn)
This is the only command that needs to be run as root. Subsequent ones can be run as a normal user. Let s run some task: 
$ lanco testns run first-task openssl speed aes
$ lanco testns check first-task && echo "Still running"
Still running
$ lanco testns ls
testns
   first-task
      28456 openssl speed aes
The output of the task is logged into a file: 
$ head -3 /var/log/lanco-testns/task-first-task.log
Doing aes-128 cbc for 3s on 16 size blocks: 8678442 aes-128 cbc's in 2.85s
Doing aes-128 cbc for 3s on 64 size blocks: 2478283 aes-128 cbc's in 2.99s
Doing aes-128 cbc for 3s on 256 size blocks: 628105 aes-128 cbc's in 3.00s
If the task is too long, we can kill it: 
$ lanco testns run first-task openssl speed aes
$ lanco testns stop first-task
You cannot run a task that already exists or kill a task that does not exist: 
$ lanco testns run first-task openssl speed aes
2013-06-09T22:50:34 [WARN/run] task first-task is already running
$ lanco testns stop second-task
2013-06-09T22:50:45 [WARN/stop] task second-task is not running
Thanks to the use of cgroups, lan o is able to track multiple processes even when they are forking away3: 
$ lanco testns run second-task sh -c \
>  "while true; do (sleep 30 &)& sleep 1; done"
$ lanco testns ls
testns
   first-task
      28456 openssl speed aes 
   second-task
      29572 sh -c while true; do (sleep 30 &)& sleep 1; done 
      29575 sleep 30 
      29593 sleep 30 
      29596 sleep 30 
      29599 sleep 30 
      29622 sleep 30 
      29644 sleep 1 
      29645 sleep 30 
  
$ lanco testns stop second-task
$ lanco testns check second-task   echo "Killed!"
Killed!
Also, there is a top-like command (lanco testns top): lanco top

Using cgroups Control groups (cgroups) is a mechanism to partition a set of tasks and their future children into hierarchical groups with a set of parameters.

Hierarchy Let s start with the hierarchical stuff first. To create a new hierarchy, you have to mount the cgroup filesystem on some empty directory. Usually, cgroups hierarchies are setup in /sys/fs/cgroups which is a tmpfs filesystem: 
# mount -t tmpfs tmpfs /sys/fs/cgroup -o nosuid,nodev,noexec,relatime,mode=755
Now, we can create our first hierarchy: 
# cd /sys/fs/cgroup
# mkdir my-first-hierarchy
# mount -t cgroup cgroup my-first-hierarchy -o name=my-first-hierarchy,none
# ls -1 my-first-hierarchy
cgroup.clone_children
cgroup.event_control
cgroup.procs
notify_on_release
release_agent
tasks
We ll see later the purpose of none. The most interesting file is tasks. It contains the PIDs of all processes in our group. Since there are currently no sub-group, all processes are part of it. Let s create a sub-group and attach a process to it: 
# mkdir first-child
# cd first-child
# ls -1
cgroup.clone_children
cgroup.event_control
cgroup.procs
notify_on_release
tasks
# echo $$ > tasks
# cat tasks
23184
23311
# cat /proc/$$/cgroup 
9:name=my-first-hierarchy:/first-child
8:perf_event:/
7:blkio:/
6:net_cls:/
5:freezer:/
4:devices:/
3:cpuacct,cpu:/
2:cpuset:/
1:name=systemd:/user/bernat/1
We have added our shell to the new cgroup. Moreover, all its children will also be part of this group. This explains why we have two tasks: the shell and cat. The last command is quite interesting: for each hierarchy, it shows which cgroup the task belongs to. For a given hierarchy, each task is exactly in one group. The most useful features of lan o are done using just this: a namespace is a named hierarchy and each task is enclosed into a cgroup so it can be tracked properly.

Subsystems A subsystem makes use of the task grouping provided by cgroups to assign resources to a set of tasks. Each available subsystem can only be in one hierarchy, so the usual way to setup things is to assign a hierarchy for each subsystem4. Let s have a look at the cpuset subsystem: it allows to assign tasks to specific CPUs and memory nodes (for NUMA systems). Let s suppose we have 4 cores and we want to assign the first core to common system tasks and the three remaining ones to nginx: 
# cd /sys/fs/cgroup
# mkdir cpuset
# mount -t cgroup cgroup cpuset -o cpuset
# echo 0-3 > cpuset/cpuset.cpus
# echo 0 > cpuset/cpuset.mems
# mkdir cpuset/system
# echo 0 > cpuset/system/cpuset.cpus
# echo 0 > cpuset/system/cpuset.mems
# for task in $(cat cpuset/tasks); do
>    echo $task > cpuset/system/tasks
> done
# mkdir cpuset/nginx
# echo 1-3 > cpuset/nginx/cpuset.cpus
# echo 0 > cpuset/nginx/cpuset.mems
# for task in $(pidof nginx); do
>    echo $task > cpuset/nginx/tasks
> done
The first mount is not needed if something else (like systemd) has setup the appropriate subsystem. Now, even if some system process is going crazy, it won t affect the performance of your webserver. The kernel comes with a more complete documentation if you need additional details on cgroups. There are high level tools to manipulate them, like the tools provided by libcg, but they are currently quite buggy and invasive.

Use in lan o Here are the features used by lan o:
  • For each namespace, a specific named hierarchy is created.
  • By setting the appropriate permissions on the hierarchy, an unprivileged user can create subgroups.
  • Each submitted task has its own sub-group for tracking purpose.
  • Actions can be executed when a task terminates by using the release agent mechanism5.
  • The cpuacct subsystem is used to track CPU usage: a group is created in this subsystem for this purpose.
  1. Both systemd and Upstart are difficult to run without being PID 1 and without being root. They both support user sessions but this requires to use them as PID 1 as well. I discovered later than runit would have been a good fit: it does not require to be PID 1, it does not need to be run as root and the service directory can be specified by an environment variable. It could have been tweaked to meet the above requirements.
  2. Lan o means launcher in esperanto. The unusual diacritic on the letter c is not present in many web fonts, so the rendering may be a little odd.
  3. Linux cgroups do not provide any facility to kill a whole group. We need to iterate through the processes and kill them one by one. lan o does not try to freeze processes before killing them and therefore may be inefficient against violent fork bombs.
  4. The cpu and cpuacct subsystems are the notable exception. There seems to be little use to keep them in separate hierarchies, so they are part of the same hierarchy.
  5. Unfortunately, only one global release agent per hierarchy can be used. Since lan o provides with the ability to execute an arbitrary command when a task terminates, the command needs to be stored on the filesystem for the release agent of the namespace to execute it.

Niels Thykier: Optimizing autodie

After converting Lintian to using autodie, I was a bit trouble by the effects it had on Lintian s (start-up) performance. This overhead accumulated to at least a 1s when checking packages. Indeed, I am not the first to complain about this problem. For those who have not heard of it, autodie is a Perl pragma that basically alters subs and built-ins from a check return value for errors -sub (e.g. like C) to throws exception on error -sub (e.g. like Python). So, you can replace open( ) or die with open( ) when autodie is in effect. autodie achieves this by exporting a little sub that wraps the target sub (or built-in). This little wrapper is compiled[1] into the importing package. The problem with this solution is that it ends up compiling these subs once per importing package. Secondly, all of these wrappers are wrapped in something called a leak guard . As autodie is a lexical pragma it is not permitted to leak across file boundaries. The leak guards take care of that by doing a runtime check of the caller and reverting to the original sub/built-in if a leak occurred. These leak guards were also compiled into the importing package. It is probably not much of a surprise to anyone that autodie spent quite a bit of its runtime compiling these wrappers and their leak guards. Together with upstream, I have addressed some of these problems. The first optimization was to allow some wrappers for built-ins to be reused that does not take glob or bareword arguments[2]. Secondly, we started using closures to generate the leak guards. Combined these two effectively cut the load time in half for subsequent loads of autodie, though it did have a negative effect on the call overhead for these wrappers (~1s for 1M calls). That landed in autodie v2.18. The extra runtime overhead of v2.18 was basically caused by the leak guard being split into two (a small closure, which called the real leak guard with an extra argument). We managed to merge these two parts into one without (too much) loss of code maintainability. With that, the call overhead of these wrappers dropped by ~3s (for 1M calls) and thus the wrappers in autodie v2.20 were even faster than the ones in v2.17. Beyond these improvements, I have sent upstream several patches (still pending review) that will improve the performance of autodie even further. If merged, we are looking at another 1.5s (per 1M calls) removed and almost a factor 4 reduction on the load time of autodie. This factor 4 comes from lazily compiling the wrapper on demand, so some of the price is moved to the call overhead. That said, it is a net gain since it is highly unlikely that every module using autodie will use every single wrapped built-in (how often do you use both read and sysread in the same module)? [1] It is compiled via a call to eval using some template code. [2] For the rest of the built-ins, it could probably just use Symbol::qualify to qualify bareword filehandles. Perhaps that will come in a future version of autodie. :)

Russ Allbery: Review: This Alien Shore

Review: This Alien Shore, by C.S. Friedman
Publisher: DAW
Copyright: 1998
Printing: July 1999
ISBN: 0-88677-799-2
Format: Mass market
Pages: 564
Jamisia is an orphan taken in by the Shido corporation, but she has a fairly comfortable and happy life in Shido Station. She lives in the care of her tutor, has friends, has something that feels like a relatively normal life. Then the station is attacked and destroyed by a rival corporation and she barely escapes in a one-person lifeboat, sent inexplicably by her tutor with a fake identity to rendezvous with a spaceship heading for the up-and-out. That begins a long and quite confusing flight away from everything she's known, and a discovery that her life is far more complicated than she thought. Meanwhile, far from the backwater of corporate tyranny that Earth has become, a new computer virus is wreaking havoc on Guild outpilots. The Guild navigate the ainniq: a sort of subspace that allows ships to slip between nodes without regard to distance. This is the only acceptable method of interstellar travel now that humans know that the more conventional Hausman Drive causes massive genetic mutations, and only Guerans have the secret of navigating it safely. But a virus of a sophistication no one has seen before appears to be targeting outpilot brains through their implants. This Alien Shore is told in two threads but multiple characters and viewpoints. Jamisia (yes, I stumbled over that name the entire book) spends nearly the whole book running while discovering that her head is a very complicated place for very bad reasons. Meanwhile, various other people (most notably the careful and meticulous Kio Masada) are analyzing and tracing the virus, which is of deep concern to the powers that be in the Guaran-controlled network of human colonies. The threads do come together eventually, helped by a young and much more impulsive hacker named Phoenix who is also on the trail of the virus, but it takes much longer than I expected. The book is a bit of cyberpunk, a bit of space opera, and is stuffed to the brim with world-building ideas. There's always something that feels slightly emotionally off about C.S. Friedman novels. Her characters are a bit too intense, a bit too full of primary colors and inherent natures. They don't do a lot of changing; they're just a little too much purely themselves to feel quite accurate. Sometimes it matters and sometimes it doesn't. This is one of the books where it mostly doesn't, partly because it's full of such an amazing variety of characters, and partly because Friedman does a great job adding situational complexity by playing them off of each other and showing them through each other's eyes. Pyschological realism, though, this isn't, nor would I look to Friedman for that. That's somewhat unfortunate given how central psychology is to this book. The focus that slowly emerges from this novel is the whole-hearted embrace of psychological difference as a force for creativity, power, and perspective. Acceptance, respect for, and integration of difference borders on a religious belief for Guerans, which makes this book one of the most wholeheartedly inclusive stories (at least for psychology) I've read. I loved that message enough to overlook the fact that those psychological differences are not treated with much subtlety. The most central of the differences (a very mild spoiler, but one that's essentially given away on the back cover and that is easy to guess early in the book) is multiple personalities. Sadly, I thought that was also the least successful. Friedman treats the topic respectfully and as a difficult asset, which I liked, but it's bog-standard fictional multiple personalities. The presentation owes much more to popular fiction than to the reality of disassociative identity, and the notes that Friedman hits (and the characteristics that the multiples embody) are predictable. However, her second major psychological idea is simply brilliant. The Guerans have developed a coding system for people to communicate their psychological profile to anyone who sees them, and all of the Gueran characters in the novel actively engage with that system and make extensive use of it. Friedman hints at a great deal of subtlety: a wealth of primary patterns (many of which are described in quite enjoyable animal metaphors as chapter epigraphs), secondary patterns that can be layered underneath, and a complex interaction of base patterns and patterns for daily mood. Here, I think Friedman's mild tendency towards archetypal characters helps, since it makes the psychological coding system clearer for the reader. I wish it had been more central to the story and explained further, although I can see the risk of turning the novel into an RPG sourcebook. (Although, now that I mention it, I would love to read an RPG sourcebook for this universe.) The other point for which I have to give Friedman huge credit is that at least half of the book is about a computer virus, the analysis of that virus, and the hacking culture of her future universe, and at no point during the story did I want to start throwing things. There are no completely unbelievable magic computer programs, the sense of movement of data and virtual presence through computer networks seemed vaguely plausible and less steeped in unrealistic VR metaphors, the sense of aesthetics of code is captured remarkably well for a story of this sort, and Friedman even brings some variety to her computer-savvy characters. There are a few mildly stereotyped young hackers (although even those have a tendency to have more depth than the stereotype), but Friedman also populates her electronic world with the other types you rarely see in these novels: thoughtful, brilliant, but meticulous researchers, competent technicians who don't engage as much on the aesthetic aspects, and even moderately-intelligent users who pay attention to the computers their lives are built around. I could actually believe in both these people and the software they use with only the suspension of disbelief required for any future SF. That's not to say there are no flaws, quibbles, or places where computers probably won't ever work that way. But that's part of what makes me so impressed. Friedman is still using computers and hacking in the direct service of a fairly typical SF plot to achieve specific dramatic ends and to create a mystery. She's not building the story around actual technology; she's staying fairly close to the sort of technology and plots that a lot of cyberpunk and hacking novels have. And yet she manages not to make the computing entirely stupid or unbelievable. It's existence proof that an SF author can pay attention to experts and research without compromising the story they want to tell. I want to take this book and smack Robert Sawyer upside the head with it. This is a great idea book with a heavily computer-driven plot strand that I quite enjoyed. It is, however, not without its flaws. The virus-hunting part of the plot is significantly more interesting than Jamisia's flight, particularly given that she keeps running, and running, and running, encountering various people who seem important and end up not being important at all. It takes more than half the book before anything particularly interesting happens to her; up until then, I was dragging myself through her segments so that I could get back to Masada and the other Guerans. She's also annoyingly dim about some things. I figured out what the deep secret in her brain was within the first four or five chapters (it's entirely obvious from the political structure of Friedman's universe), and I have no idea why it took everyone else so long to figure it out. There is also, as mentioned, the slightly off feeling of the characters that I find typical of Friedman, and a cut-throat power struggle at the heart of the book that just didn't have enough psychological believability to fully engage me. But the ideas are wonderful: technology-driven political structures, corporate-controlled states, ideological clashes, lots of believable cultural interactions, and a neat two-option interstellar travel system (even if the nature of the Hausman Drive damage bears no resemblence to any genetic science known to our reality). And that's on top of a wonderful visual language of psychology and discussions of computer security that I actually enjoyed reading. Friedman's the sort of author who won't be to everyone's tastes. You have to have some tolerance for her tendency to write grand clashes of power between highly flawed, aggressively emotional people. Sometimes you have to put up with rather a lot of setup. But I really liked this novel. I suspect I'm editing out a lot of the boring bits in my memory, but I fondly remember that edited version as a ton of fun. Rating: 8 out of 10

26 June 2013

Daniel Pocock: The irrational fear of the Internet, foreign hackers and blue ties

In recent times, many people around the world have come to appreciate the fears of Australians. Fear of the Internet drove our communications minister to pursue a policy of Internet censorship that has been compared to countries like North Korea. Senator Stephen Conroy's policies achieved the global ridicule that they deserve when he was awarded the title of 2009 Internet Villain of the year. Today, a change of leadership has censored him out of a job and his scare campaign will hopefully disappear with him. Fear of people who work surfaced more recently. Our Prime Minister spent much of this year lashing out at highly skilled workers, especially IT workers she disrespectfully accused of "stealing jobs". Promoting a fear of men in blue ties took her into uncharted territory. In London, the omniprescent security theatre constantly reminds people to fear suspicious packages. But is Australia really to be concerned about blue ties? Blue ties get their revenge As always in politics, it is a story of hyopcrisy. Gillard herself was born in the UK and has no right to be criticising other hard working, highly skilled professionals because they have a foreign background. As for that pesky bunch of bloggers running wild on the Internet her communications minister aspired to suppress: when the Prime Minister did a widely derided photo shoot for a women's magazine, the bloggers lit up the web like a Christmas tree. Within 24 hours, a petition was circulating within her party to sack her Remarkably, while accusing foreigners of "stealing" jobs, it has been a subject of some debate that she only obtained her own job by taking it from another member of her own party in the middle of the night. Today, that same man came to work in a blue tie and took the job back. Hopefully Gillard's fears will leave with her.

Next.

Previous.