Search Results: "nate"

4 August 2013

Francois Marier: Debugging Gearman configuration

Gearman is a queuing system that has been in Debian for a long time and is quite reliable. I ran into problems however when upgrading a server from Debian squeeze to wheezy however. Here's how I debugged my Gearman setup.

Log verbosity First of all, I started by increasing the verbosity level of the daemon by adding --verbose=INFO to /etc/default/gearman-job-server (the possible values of the verbose option are in the libgearman documentation) and restarting the daemon:
/etc/init.d/gearman-job-server restart
I opened a second terminal to keep an eye on the logs:
tail -f /var/log/gearman-job-server/gearman.log

Listing available workers Next, I registered a very simple worker:
gearman -w -f mytest cat
and made sure it was connected properly by telneting into the Gearman process:
telnet localhost 4730
and listing all currently connected workers using the workers command (one of the commands available in the Gearman TEXT protocol). There should be an entry similar to this one:
30 127.0.0.1 - : mytest
Because there are no exit or quit commands in the TEXT protocol, you need to terminate the telnet connection like this:
  1. Press Return
  2. Press Ctrl + ]
  3. Press Return
  4. Type quit at the telnet prompt and press Return.
Finally, I sent some input to the simple worker I setup earlier:
echo "hi there"   gearman -f mytest
and got my input repeated on the terminal:
hi there

Gearman bug I traced my problems down to this error message when I sent input to the worker:
gearman: gearman_client_run_tasks : connect_poll(Connection refused)
getsockopt() failed -> libgearman/connection.cc:104
It turns out that it is a known bug that was fixed upstream but still affects Debian wheezy and some versions of Ubuntu. The bug report is pretty unhelpful since the work-around is hidden away in the comments of this "invalid" answer: be explicit about the hostname and port number in both gearman calls. So I was able to make it work like this:
gearman -w -h 127.0.0.1 -p 4730 -f mytest cat    
echo "hi there"   gearman -h 127.0.0.1 -p 4730 -f mytest
where the hostname matches exactly what's in /etc/default/gearman-job-server.

3 August 2013

Steve Langasek: Network services and Upstart

So Debian bug #710747 led to an interesting discussion the other night on IRC which made me realize there are a lot of people who have yet to understand why sysvinit needs replacing. I can't speak to whether upstart solves this bug in particular. The tftpd-hpa package in Ubuntu (and in Debian experimental) does have an upstart job, and I have used tftpd-hpa on systems whose network interfaces are managed entirely with NetworkManager, and I have not seen this bug; but I can't say that this is more than coincidence. However, I can speak to how upstart provides facilities that address the general problem of starting services that require a working network connection to be present before they start up. Traditionally on Linux systems, init scripts could assume that the network connection is up before starting any services for the target runlevel. With LSB dependencies, you could also express this as
Required-Start:    $network
The trouble with both of these approaches is that they assume that there is a single point in time, early in the boot of the system, when the "network" is up. This was an ok assumption 15 years ago, when the Linux kernel initialized all devices serially and all our network configuration was static. But nowadays, you simply cannot rely on systems booting in such a way that the network is fully up shortly after boot - or in a way that you can block the system boot waiting for the network to be up. If all of our services would cope gracefully with being started without the network, this would be a non-issue. Unfortunately, not all network services are written in such a way as to work without a network; nor do they all cope with dynamic changes to interfaces or addresses. While it would be better if these services were written more robustly, since there's no shortage of daemons written this way, it's convenient that upstart provides tools for coping with this behavior in the meantime. Here's a real (simplified) example of an upstart job for a service that needs to wait for a non-loopback interface before it starts:
start on (local-filesystems and net-device-up IFACE!=lo)
stop on runlevel [!2345]
expect fork
respawn
exec nmbd -D
Now, you might also have a service that you only want to start up when a particular network device comes up. This is easily expressed in upstart, and might look like this hypothetical job:
start on net-device-up IFACE=wlan0
stop on net-device-down IFACE=wlan0
expect daemon
respawn
exec mydaemon
If you need to restart a daemon whenever the set of active network connections changes, that's less straightforward. Upstart doesn't have the notion of a 'restart' rule, so you would need two jobs to handle this - one for the daemon itself that starts on the first network connection, and a second job to trigger a restart of the daemon when the network status changes. For the tftpd-hpa case in the abovementioned bug, this might look like:
$ cat /etc/init/tftpd-hpa.conf
start on runlevel [2345]
stop on runlevel [!2345]
expect fork
respawn
script
        if [ -f $ DEFAULTS  ]; then
                . $ DEFAULTS 
        fi
        exec /usr/sbin/in.tftpd --listen  --user $ TFTP_USERNAME  --address $ TFTP_ADDRESS  $ TFTP_OPTIONS  $ TFTP_DIRECTORY 
end script
$ cat /etc/init/tftpd-hpa-restart.conf
start on net-device-up
task
instance $IFACE
script
        status tftpd-hpa   grep -q start/   stop
        restart tftpd-hpa
end script
For this case, upstart doesn't provide quite so great an advantage. It's nice to be able to use upstart natively for both pieces, but you can do the same thing with an init script plus an if-up.d script for ifupdown, which is what maintainers do today. I think adding a restart on stanza to upstart in the future to address this would be a good idea. Though in any event, this is far simpler to do with upstart than with any if-up.d script I've ever seen for managing an initscript-based service. Between the more friendly declarative syntax, and the enhanced semantics for expressing when to run (or restart) a job, upstart offers clear advantages over other init systems.

2 August 2013

Raphaël Hertzog: My Free Software Activities in July 2013

This is my monthly summary of my free software related activities. If you re among the people who made a donation to support my work (167.67 , thanks everybody!), then you can learn how I spent your money. Otherwise it s just an interesting status update on my various projects. The Debian Administrator s Handbook After the successful crowdfunding campaign, I had a bunch of rewards to ship: I subcontracted most of the job but I had to take care of the books with dedication. I also dealt regularly with books/stickers coming back to the sender (due to invalid address or people not picking up their parcels in the post-office). After the rewards, we had to take care to actually finalize the liberation of the French translation. I merged the translations we had in Git and Roland updated/translated a few strings that weren t in the original book in French. Then I have put the book online. Last but not least, I started to work on updating the English book for Debian 7 (Roland started way before me) and we have put some updated chapters up for review. Debian France Elections. After Debian France s general assembly, the new board of administrators voted the officers: I have been re-elected as President, Sylvestre continues as Treasurer but we have a new Secretary in the person of Alexandre Delano . Welcome Alexandre! I did the administrative work to register the new board/officers in the Tribunal d instance and to give access to the internal git repositories to the new members. Galette. I also did a bunch of tests on Galette s new features that Debian France ordered to the upstream author. They should all land in the next upstream release due in the next weeks. \o/ Accounting. I worked on the accounting to bring it up-to-date so that Sylvestre can pick up the work from now on. We re learning how to best use ledger for our needs. PTS rewrite I continued to spend about 12 hours a week to mentor Marko Lalic who is rewriting the Package Tracking System. I m pretty happy with the results so far so I marked him as pass for the mid-term evaluation required by Google. You can have a look at the documentation and the web interface is starting to show some content. The email interface is fully working and I have configured the real PTS to forward all mails to our test instance (pts.debian.net) so that you can use the rewritten PTS for real-life work. Mail your subscription commands to control@pts.debian.net and start using it! Thanks to the test driven development methodology we re using, we re pretty confident that it works reasonably well! :-) I also packaged python-django-jsonfield (still in NEW) since Marko has been using this python module in his code, and filed bug #717900 on sqlite3 to raise a limit that we have hit with queries made by the PTS. Kali Linux I used the Calxeda Highbank node donated to Debian by Offensive Security to test the new -armmp kernel flavor on it. It seemed to work except for a missing network driver (filed in #717269). Misc Debian work Issues with social networks. With the move of identi.ca to pump.io, we don t have any possibility to auto-post status updates based on RSS feeds. Identi.ca s @debian account was also configured to push updates to the @debian account on twitter.com (and from there it was grabbed in the Debian page on Facebook). This is also gone so to limit the damage, I setup twitterfeed.com so that the twitter/facebook accounts continue to have updates). If you re looking for a development project, here s an area that is not well covered by free software! We need code to do what twitterfeed does and we need that code to also support pump.io. Dpkg work. It s been a long time since I last pushed some code to dpkg s git repository. I took care of reworking and merging a patch submitted by Steve Langasek to fix #716948 (an issue with dpkg-maintscript-helper rm_conffile messing with conffiles that the package no longer owns). Git mail notification. When I was still administrator of Alioth, I wrote git-commit-notice (a fork of Git s post-receive-email) and many packaging projects are using this hook script to send commit notices to mailing lists. This script has not been updated for multiple years and it started spewing warnings recently due to deprecated features in Wheezy s git. So I looked at updating it and while doing so I discovered a much better replacement with git-multimail. Thus I adapted git-commit-notice to work on top of this new script. The result has now been installed on git.debian.org (this is to be properly announced in the next DeveloperNews). Misc work. I packaged sql-ledger 3.0.5-1, forwarded #714739 on publican, and I participated in discussions to move the French Debian planets to planet.debian.org. Thanks See you next month for a new summary of my activities.

10 comments Liked this article? Click here. My blog is Flattr-enabled.

Justus Winter: There is no spoon...

This is the Hurd shutting down:
root@debian:~# init 0
INIT: Switching to runlevel: 0
INIT: Sending processes the TERM signal
INIT: Sending processes the KILL signal
Using makefile-style concurrent boot in runlevel 0.
Stopping deferred execution scheduler: atd.
Asking all remaining processes to terminate...done.
All processes ended within 1 seconds...done.
Stopping enhanced syslogd: rsyslogd.
Deconfiguring network interfaces...ifdown: failed to open pid file /run/network/ifdown-/dev/eth0.pid: No such file or directory
Internet Systems Consortium DHCP Client 4.2.2
Copyright 2004-2011 Internet Systems Consortium.
All rights reserved.
For info, please visit https://www.isc.org/software/dhcp/
can't create /var/lib/dhcp/dhclient./dev/eth0.leases: No such file or directory
Listening on Socket//dev/eth0
Sending on   Socket//dev/eth0
/bin/sh: 1: ifconfig: not found
done.
Deactivating swap...swapoff: /dev/hd0s5: 177152k swap space
done.
mount: cannot remount /: Device or resource busy
Will now halt.
INIT: no more processes left in this runlevel
The important line is the one saying Asking all remaining processes to terminate...done.. What happens there is that /sbin/killall5 is run, actually does its job and the system survives that. What was needed to fix killall5 is to mark some processes (all translators being started as root as well as any essential processes) as important and to exempt them from being frozen and killed by killall5. Furthermore it was necessary to fill in the correct values for the start_code and end_code fields of the /proc/*/stat records (this used to be an issue for Debian/kFreeBSD as well). I am still in the process of cleaning up the patch series, I will finish that first thing Monday morning. I also spent my time polishing my mtab translator, the patch series is in its fourth revision and I consider it ready for inclusion: http://lists.gnu.org/archive/html/bug-hurd/2013-07/msg00259.html I also had this moment of clarity and enlightenment. For the first time I thought I finally understood what the Hurd really was. It is just a bunch of Mach programs that talk to each other in this really strange language and as a result they behave very much like the equivalent programs would on a different POSIX-like system. This is of course hidden away from the application programmer in our libc. This language is a mere convention, Richard calls this system personality. On some other level I have known this for a long time, but I never grasped the profound implications for users and developers, and for composability and the security of the system as a whole. I will try to show some of the cool stuff that can be done with such a system, stuff that is hard to do on monolithic kernels. And I had to patch the exec server. You know what they say, once you looked into /hurd/exec, there is no going back. I am very happy to report that I addressed all three major issues I identified in my second week. Next week I will clean up my initscripts patch series and submit it for inclusion. I will also rebuild the hurd, sysvinit and libc packages with all my patches included for broader testing. And if I ever run out of stuff to do first, I will have to look at the network related issues.

Joey Hess: DIY crowdfunding and bitcoin

Well, my git-annex crowdfunding campaign is half way to its August 15th conclusion. So far it's raised more than five times what I hoped it would. I wish I could say I'm like some canny NASA engineer who intentionally sets low expectations for their Mars rover, but in both the previous kickstarter and this campaign I've really had no idea how far it'd go. I'm glad that I'll be working on git-annex for another year. I was particularly unsure if it'd be successful to move off Kickstarter. During the git-annex assistant Kickstarter campaign, I saw many small contributions from people who learned of it due to it being a successfully funded project, a staff pick, etc. Losing that easy network effect is a gamble. So far I've had only half the number of contributors that I got on Kickstarter. I've basically missed out entirely on the $5 level casual contributors. On the other hand, my backers have generally been more generous (and some have been exceedingly generous). And I've avoided rewards that will cost much money, so I may end up in the same ballpark funding level in the end! Incidentially, I'm really enjoying getting in touch to let people know when I make their sponsored commits. There's still time to sponsor one of your own ;) Bitcoin I also was curious to experiment with Bitcoin in this campaign. Partly because Paypal isn't available everywhere internationally, and takes really obnoxious percentages of transactions (though probably not as bad as Kickstarter taking its percentage followed by Amazon payments taking its percentage..) and partly because there seem to be interesting possibilities for supporting free software with Bitcoin. (Especially if any of the microtransactions on top of Bitcoin take off.) So far 5% of backers have used Bitcoin. It's been quite strange to actually have significant amounts of bitcoins in my wallet. Wordpress has had 94 bitcoin payments over 9 months since starting accepting them. I've had 47 payments in the two weeks my campaign has run so far. Wow! Most of the bitcoin payments have come in via Coinbase (a few people have found my direct payment address), but of those very few were using bitcoin purchased on Coinbase. Most are probably transfers of bitcoin they already had, or perhaps bitcoin purchased on other sites. The one technical issue I've had with using bitcoin is that Coinbase has not provided details about who sent most of the donations. Probably some of them are intentionally anonymous, but I suspect Coinbase's interface to claim incoming bitcoin transactions failed for some of them. (If you donated bitcoin and want to actually get a reward, please email me.) By the way, I'm converting most of the bitcoins back to USD pretty quickly. I'm not interested in speculating on currency exchange rates with money that has been donated so I can accomplish a particular task.. DIY I put up the campaign website without any means in place to handle updating it. This is because I never automate anything until I've done it at least 10 times by hand. ;) After the first trickle of donations became a flood, I quickly realized I needed at least something to handle keeping the numbers straight. What I whipped up in an hour of coding is a system where I enter incoming payments into a hledger file and a small haskell program parses that and writes out various files that are included into the website. Amusingly the percentage calculation and display code was copied from git-annex, so part of git-annex is helping run its own fundraising campaign. The campaign video is itself hosted in a public git-annex repository, come to think of it. The rest of the site is built using ikiwiki. Given that it's hosted at Branchable, this is a high level of dogfooding and DIY. There are certianly better crowdfunding platforms, but all I miss in this one is automated transaction entry. And I have total flexability, double entry accounting, and a powerful static website generator that handled being on the top of Hacker News without a sweat. Oh, and some money. What's not to like?

1 August 2013

Matthew Garrett: Don't ship 32-bit UEFI firmware on x86

Shipping a UEFI-bootable Linux distribution is a touch awkward[1], with the main sticking point being the necessity to produce boot media with multiple El-Torito images. An El-Torito image is either an image of a floppy disk or a small hard drive, embedded into the ISO. This allows BIOS systems to look for an El Torito image, hook some interrupts and then boot without the BIOS having to care about the fact that the image is embedded in an ISO-9660 filesystem. UEFI systems will look for an El-Torito image with a special tag - if they find it they'll mount it as a FAT filesystem and look for a bootloader, and if not they'll fall back to BIOS compatibility.

So, if you want a CD that'll boot on both BIOS and UEFI systems, you need two El-Torito images - one for BIOS, one for UEFI. Unfortunately, various BIOSes deal exceptionally badly with CD images that contain more than one El-Torito image. The most common failure case is to print a menu asking you to choose an option without labelling the options, but some will just fail outright. Thankfully, this is pretty much exclusively limited to 32-bit systems.

Things get irritatingly more complicated due to a quirk of UEFI. UEFI is based on executing code in native mode. That means that 32-bit UEFI systems can't execute 64-bit code in firmware, even if the CPU is capable of it. A 64-bit OS can only boot on 32-bit UEFI if it has very ugly compatibility hacks, including having to rewrite structures and register state every time it makes a UEFI call. The only OS I'm aware of that implements this is MacOS X. Having looked into what it'd take to implement it in Linux, I decided that hammering rusty nails through my feet would be a preferable use of time. Thankfully, I went drinking instead.

So distributions have a choice. They can either produce UEFI-bootable CD images for 32-bit x86 and risk failures on actual 32-bit systems, or they can ignore UEFI entirely on 32-bit and succeed in booting on all the hardware that people actually own[2]. Unsurprisingly, they tend to choose the second option.

So, if you're building an x86 hardware platform, don't ship with 32-bit UEFI. If you're stuck with a 32-bit CPU then just ship BIOS. If you have a 64-bit CPU then ship a 64-bit UEFI. If you ship with 32-bit UEFI, no significant existing Linux distribution will support you, and you'll face an uphill battle to convince them to do so.

32-bit UEFI. Just say what on earth were you thinking, please, no, can't you find a solution that doesn't involve me getting tetanus jabs.

(If you're worried about the extra memory consumption of 64-bit OSes, just encourage 32-bit userspace on a 64-bit kernel. Or just boot via BIOS)

[1] See the number that still don't manage it despite having had several years to adapt
[2] Until recently, the only vendor to ship 32-bit UEFI firmware on volume hardware was Apple. This was fine on their 32-bit systems, but on their 64-bit systems with 32-bit UEFI booting a 64-bit version of Windows would result in boot failure. Apple rectified this by stating that 64-bit Windows wasn't supported on platforms with 32-bit UEFI, which is a neat trick if you can manage it. 32-bit Windows (and Linux) was fine because it didn't include a UEFI boot image and so didn't trigger the bug.

comment count unavailable comments

31 July 2013

Jan Wagner: Updated roundcube for wheezy

Roundcube is one of the well known OpenSource solutions for accessing Mails via webinterface. Unfortunately the package in wheezy is ~2 years old.
In Debian unstable the package was updated, but #714727 is blocking the migration to jessie. In the meanwhile I backported the package to wheezy. For everybody still using the sqlite (v2) backend, see here. There is no automatic migration from sqlite to sqlite3 backend.
For everybody using the package roundcube-plugins-extra, I created an updated package for roundcube 0.9 and submitted my changes to the Debian Roundcube Maintainers. If you want to give it a try, you could install the packages from my wheezy-backports repository:
# wget "http://ftp.cyconet.org/debian/sources.list.d/wheezy-backports-cyconet.list" \
    -O /etc/apt/sources.list.d/wheezy-backports-cyconet.list
# aptitude update
# aptitude -t wheezy-backports install roundcube-mysql roundcube-plugins roundcube-plugins-extra
For adding my archive key, you can just install the package "debian-cyconet-archive-keyring"

24 July 2013

Jan Wagner: Frag is bigger than frame.

Since 2 weeks I'm faced with some problems at a Xen hosting environment based on Debian wheezy. The issue is, that one specific domU is regularly failing network. The investigations showed, that no network packets from and to this system are seen on the the network interface in the domU and on the bridge interface in the dom0.
The funny part is, that the firewall on the dom0 now is logging packet rejects on the interface of the effected system and one interface of an other specific domU for the destination address. One pack and rejects on both interfaces ... really strange!
Shutting down the domU and recreating them doesn't help. Even if restarting other domUs, those are also effected by now. The only way to fix is to reboot the entire dom0. After several investigations I stumpled upon the following in the dmesg:
vif-xm116: Frag is bigger than frame.
vif-xm116: fatal error; disabling device
xenbr0: port 4(vif-xm116) entering disabled state
A quick search at the famouse search engine revealed, there seems something bigger. Even a Debian bug is open since February. Unfortunately this problem is still unfixed in wheezy and squeeze.
But ... there is hope in sight! Ian Campbell provided packages based on top of the latest kernel packages with a call for testing. Installing them seems to fix the issue in my environment. So we just have to hope this will get fixed soon, as this regression is really annoying. Thanks to all who make this happen in advance!

Keith Packard: present-redirect

Present Extension Redirection Multi-buffered applications have always behaved poorly in the presence of the Composite extension: Owen Taylor suggested that Present should offer a way to redirect operations to the compositing manager as a way to solve these problems. This posting is my attempt to make that idea a bit more concrete given the current Present design. Design Goals Here s a list of features I think we should try to provide:
  1. Provide accurate information to applications about when presentation to the screen actually occurs. In particular, GLX applications using GLX_OML_sync_control should receive the correct information in terms of UST and MSC for each Swap Buffers request.
  2. Ensure that applications still receive correct information as to the contents of their buffers, in particular we want to be able to implement EGL_EXT_buffer_age in a useful manner.
  3. Avoid needing to un-redirect full-screen windows to get page flipping behavior.
  4. Eliminate all extra copies. A windowed application may still perform one copy from back buffer to scanout buffer, but there shouldn t be any reason to copy contents to the composite redirection buffer or the compositing manager back buffer.
Simple Present Redirection With those goals in mind, here s what I see as the sequence of events for a simple windowed application doing a new full-window update without any translucency or window transformation in effect:
  1. Application creates back buffer, draws new frame to it.
  2. Application executes PresentRegion. In this example, the valid and update parameters are None , indicating that the full window should be redrawn.
  3. The server captures the PresentRegion request and constructs a PresentRedirectNotify event containing sufficient information for the compositor to place that image correctly on the screen:
    • target window for the presentation
    • source pixmap containing the new image
    • idle fence to notify when the source pixmap is no longer in use.
    • serial number from the request.
    • target MSC for the presentation. This should probably just be the computed absolute MSC value, and not the original target/divisor/numerator values provided by the application.
  4. The compositing manager receives this event and constructs a new PresentRegion request using the provided source pixmap, but this time targeting the root window, and constructing a valid region which clips the pixmap to the shape of the window on the screen. This request would use the original application s idle fence value so that when complete, the application would get notified. This request would need to also include the original target window and serial number so that a suitable PresentCompleteNotify event can be constructed and delivered when the final presentation is complete.
  5. The server executes this new PresentRegion operation. When complete, it delivers PresentCompleteNotify events to both the compositing manager and the application.
  6. Once the source pixmap is no longer in use (either the copy has completed, or the screen has flipped away from this pixmap), the server triggers the idle fence.
Multiple Application Redirection If multiple applications perform PresentRegion operations within the same frame, then the compositing manager will receive multiple PresentRedirectNotify events, and can simply construct multiple new PresentRegion requests. If these are all queued to the same global MSC, they will execute at the same frame boundary. No inter-operation dependency exists here. Complex Presentations Ok, so the simple case looks like it s pretty darn simple to implement, and satisfies the design goals nicely. Let s look at a couple of more complicated cases in common usage; the first is with translucency, the second with scaling application images down to thumbnails and the third with partial application window updates. Redirection with Translucency If the compositing manager discovers that a portion of the updated region overlays or is overlaid by something translucent (either another window, or drop shadows, or something else), then a composite image for that area must be constructed before the whole can be presented. Starting when the compositing manager receives the event, we have:
  1. The compositing manager receives this event. Using the new pixmap, along with pixmaps for the other involved windows and graphical elements, the compositing manager constructs an updated view for the affected portion of the screen in a back buffer pixmap. Once complete, a PresentRegion operation that uses this back buffer pixmap is sent to the X server. Again, the original target window and serial number are also delivered to the server so that a suitable PresentCompleteNotify event can be delivered to the application.
  2. The server executes this new PresentRegion operation; PresentCompleteNotify events are delivered, and idle fences triggered as appropriate.
Redirection with Transformation Transformation of the window contents means that we cannot always update a portion of the back buffer directly from the provided application pixmap as that will not contain the window border. Contents generated from a region that includes both application pixels and window border pixels must be sourced from a single pixmap containing both sets of pixels. One option that I ve discussed in the past to solve this would be to have the original application allocate the pixmap large enough to hold both the application contents and the surrounding window border. Have it draw the application contents at the correct offset within this pixmap, and then have the window manager contents drawn around that; either automatically by the X server, or even manually by the compositing manager. That would be mighty convenient for the compositing manager, but would require significant additional infrastructure throughout the X server and even harder the drawing system (OpenGL or some other system). There s another reason to want this though, and that s for sub-frame buffer scanout page swapping. The second option would be for the compositing manager to combine these images itself; there s a nice pixmap already containing the window manager image the composite redirect buffer. Taking the provided source pixmap and copying it directly to the target window will construct our composite image, just as if we had no Present redirection in place. This will cost an additional copy though, which we ve promised to avoid. Of course, as it s just for thumb-nailing or other visual effects, perhaps the compositing manager could perform this operation at a reduced frame rate, so that overall system performance didn t suffer. Retaining Access to the Application Buffer Above, I discussed having the idle fence from the redirected PresentRegion operation be sent along with the replacement PresentRegion operation. This ignores the fact that the composting manager may well need the contents of that application frame again in the future, when displaying changes for other applications that involve the same region of the screen. With the goal of making sure the idle fences are triggered as soon as possible so that applications can re-use or free the buffers quickly, let s think about when the triggering can occur.
  1. Full-screen flipped applications. In this case, the application s idle fence can be triggered once the application provides a new frame and the X server has flipped to that new frame, or some other scanout buffer.
  2. Windowed, copied applications. In this case, the application s idle fence can be triggered once the application provides a new frame to the compositing manager, and the X server doesn t have any presentations queued.
In both cases, we require that both the X server and the compositing manager be finished with the buffer before the application s idle fence should be triggered. One easy way to get this behavior is for the composting manager to create a new idle fence for its operations. When that is triggered, it would receive an X event and then trigger the applications idle fence as appropriate. This would add considerable latency to the application s idle fence a round trip through the compositing manager. The alternative would be to construct some additional protocol to make the applications idle fence dependent on the Present operation and some additional state provided by the compositing manager. Some experimentation here is warranted, but my experience with latency in this area is that it causes applications to end up allocating another back buffer as the idle notification arrives just after a buffer allocation request comes down to the rendering library. Definitely sub-optimal. An Aside on Media Stream Counters The GLX_OML_sync_control extension defines the Media Stream Counter (MSC) as a counter unique to the graphics subsystem which is incremented each time a vertical retrace occurs. That would be trivial if we had only one vertical retrace sequence in the world. However, we actually have N+1 such counters, one for each of the N active monitors in the system and a separate fake counter to be used when none of the other counters is available. In the current Present implementation, windows transition among the various Media Stream Counter domains as they move between the various monitors, and those monitors get turned on and off. As they move between these counter domains, Present tracks a global offset from their original domain. This offset ensures that the MSC value remains monotonically increasing as seen by each window. What it does not ensure is that all windows have comparable MSC sequence values; two windows on the same monitor may well have different MSC values for the same physical retrace event. And, even moving a window from one MSC domain to another and back won t make it return to the original MSC sequence values due to differences in refresh rates between the monitors. Internally, Present asserts that each CRTC in the system identifies a unique MSC domain, and it has a driver API which identifies which CRTC a particular window should be associated with. Once a particular CRTC has been identified for a window, client-relative MSC values and CRTC-relative MSC values can be exchanged using an offset between that CRTC MSC domain and the window MSC domain. The Intel driver assigns CRTCs to windows by picking the CRTC showing the greatest number of pixels for a particular window. When two CRTCs show the same number of pixels, the Intel driver picks the first in the list. Vblank Synchronization and Multiple Monitors Ok, so each window lives in a particular MSC domain, clocked by the MSC of the CRTC the driver has associated it with. In an un-composited world, this makes picking when to update the screen pretty simple; Present updates the screen when vblank happens in the CRTC associated with the window. In the composite redirected case, it s a bit harder all of the PresentRegion operations are going to target the root window, and yet we want updates for each window to be synchronized with the monitor containing that window. Of course, the root window belongs to a single MSC domain (likely the largest monitor, using the selection algorithm described above from the Intel driver). So, any PresentRegion requests will be timed relative to that single monitor. I think what is required here is for the PresentRegion extension to take an optional CRTC argument, which would then be used as the MSC domain instead of the window MSC domain. All of the timing arguments would be interpreted relative to that CRTC MSC domain. The PresentRedirectNotify event would then contain the relevant CRTC and the MSC value would be relative to that CRTC. A clever Compositing manager could then decompose a global PresentRegion operation into per-CRTC PresentRegion operations and ensure that multiple monitors were all synchronized correctly. We could take this even further and have the PresentRegion capable of passing a smaller CRTC-sized pixmap down to the kernel, effectively providing per-CRTC pixmaps with no visible explicit protocol Other Composite Users Ok, so the above discussion is clearly focused on getting the correct contents onto the screen with minimal copies along the way. However, what I ve ignored is how to deal with other applications, also using Composite at the same time. They re all going to expect that the composite redirect buffers will contain correct window contents at all times, and yet we ve just spent a bunch of time making that not be the case to avoid copying data into those buffers and instead copying directly to the compositing manager back or front buffers. Obviously the X server is aware of when this happens; the compositing manager will have selected for manual redirection on all top-level windows, while our other application will have only been able to select for automatic redirection. So, we ve got two pretty clear choices here:
  1. Have the X server change how Present redirection works when some other application selects for Automatic redirection on a window. It would copy the source pixmap into the window buffer and then send (a modified?) PresentRedirectNotify event to the compositing manager.
  2. Include a flag in the PresentRedirectNotify event that the composite redirect buffer needs to eventually get the contents of the source pixmap, and then expect the compositing manager to figure out what to do.
Development Plans As usual, I m going to pick the path of least resistance for all of the above options and see how things look; where the easy thing works, we can keep using it. Where the easy thing fails, I ll try something else. The changes required for this are pretty minimal. The PresentRegion request needs to gain a list of window/serial pairs that are also to be notified when the operation completes:
PRESENTNOTIFY  
    window: WINDOW
    serial: CARD32
     
 
    PresentRegion
    window: WINDOW
    pixmap: PIXMAP
    serial: CARD32
    valid-area: REGION or None
    update-area: REGION or None
    x-off, y-off: INT16
    idle-fence: FENCE
    target-crtc: CRTC or None
    target-msc: CARD64
    divisor: CARD64
    remainder: CARD64
    notifies: LISTofPRESENTNOTIFY
 
    Errors: Window, Pixmap, Match
The target-crtc parameter explicitly identifies a CRTC MSC domain. If None, then this request implicitly uses the window MSC domain. notifies provides a list of windows that will also receive PresentCompleteNotify events with the associated serial number when this PresentRegion operation completes.
 
    PresentRedirectNotify
    type: CARD8         XGE event type (35)
    extension: CARD8        Present extension request number
    length: CARD16          2
    evtype: CARD16          Present_RedirectNotify
    eventID: PRESENTEVENTID
    event-window: WINDOW
    window: WINDOW
    pixmap: PIXMAP
    serial: CARD32
    valid-area: REGION
    valid-rect: RECTANGLE
    update-area: REGION
    update-rect: RECTANGLE
    x-off, y-off: INT16
    target-crtc: CRTC
    target-msc: CARD64
    idle-fence: FENCE
    update-window: BOOL
 
The target-crtc identifies which CRTC MSC domain the target-msc value relates to. divisor and remainder have been removed as the target-msc value has been adjusted using the application values. If update-window is True, then the recipient of this event is instructed to provide reasonably up-to-date contents directly to the window by copying the contents of pixmap to the window manually. Beyond these two protocol changes, the compositing manager is expected to receive Sync events when the idle-fence is triggered and then manually perform a Sync operation to trigger the client s idle-fence when appropriate. I m planning to work on these changes, and then go re-work xcompmgr (or perhaps unagi, which certainly looks less messy) to incorporate support for Present redirection. The goal is to have something to demonstrate at Guadec, which doesn t seem impossible, aside from leaving on vacation in four days

22 July 2013

Daniel Pocock: Building existing autotools C/C++ projects on Android

I recently came across a requirement to build a project involving C++ code on Android. I've done this before with small C projects, for example, patching the JNI code in Lumicall. Google supplies a toolchain dubbed the Native Development Kit (NDK) for Android. Using the NDK typically involves using it's custom build system, usually duplicating all Makefile content into a file called Android.mk in each subdirectory. For large projects, that duplication doesn't sound like fun to implement and it definitely doesn't sound like fun keeping it up to date. reSIProcate: 39 Makefiles + dependencies reSIProcate is quite a large project involving multiple shared libraries, some useful demonstration apps, a SIP proxy (repro) and a TURN server (reTurn). There are 39 Makefiles and numerous dependencies. There are numerous test cases for each library and sub-project too. Sounds like a perfect fit for Android Alternative ways to proceed The Android NDK documentation pack describes a process for bypassing Android.mk files, invoking the toolchain (compiler, linker, etc) directly to build standalone apps with their own Makefile system I felt this was worth a go Two approaches are described. The first approach is to simply reference toolchain binaries (e.g. arm-linux-androideabi-gcc) using the well-known environment variables such as CC and LD. The documentation warns that in this case, support for the Standard Template Library (STL) is extremely limited. Nonetheless, I decided to try this first and found that I couldn't easily link binaries with the full STL feature set required by reSIProcate. I moved on to the second approach. The second approach involves calling the script $ NDK_HOME /build/tools/make-standalone-toolchain.sh to export a custom toolchain directory tree with full STL support. Using a little more disk space, but not so hard. Initially I had NDK version r7 on my system and I found that make-standalone-toolchain.sh was not working reliably. It appeared to complete successfully (exit status = 0) but in reality it was not copying the STL components to the custom directory. I upgraded to version r8e and the problem was resolved. Making it easy Finally, I wrote a little wrapper script that can do everything: build the custom toolchain, set up all those environment variables for the tools and then invoke the configure script with some of the special settings that are required for a cross compile. Coding problems To gain some confidence in this progress, I had already run the full reSIProcate build, including test cases, on all hosts in the Debian build farm, including some ARM hosts. This helped confirm that the code is correct for such CPUs as well as helping to identify some previously unknown errors on the more typical platforms. Nonetheless, some minor problems appeared with the build for Android:
  • Android's pthread capabilities are built into the C library so -lpthread must not be included on the linker command line. This is easily adapted using Makefile variables.
  • Although Android is Linux, some of the header locations are not quite the same. For example, references to sys/fcntl.h must be wrapped in a pre-processor conditional that uses just the path fcntl.h when __ANDROID__ is defined
After fixing a few of these minor issues, everything fell into place and the build was successful. High quality portable code The reSIProcate project has a strong tradition of providing cross-platform support. The code quality is quite good and well documented. Despite the project's large size and complicated build system, there was already a good chance of success. This may not be the case for every C/C++ project, especially those making extensive use of other shared libraries. Building the dependencies: OpenSSL on Android For the core features I wanted to use, I only needed one major dependency, the OpenSSL project. libcrypto and libssl already appear in Android. Nonetheless, their headers are not present in the NDK tree and therefore their API (and compile-time flags) can not be considered 100% stable across all handsets past and future. The solution is to build a standalone libcrypto and libssl and ship them with your own app. Fortunately this has been done before: I grabbed the code from Guardian Project's OpenSSL in github and found that it built immediately with my NDK version. To use it, simply make sure that CPPFLAGS and LDFLAGS include the necessary references to OpenSSL, as demonstrated in my script Test cases I found that make -i check would enable me to build all test cases on the build host. Naturally it would fail to run any of them as they are ARM binaries. Using -i in the make command line makes it possible to build them all anyway, ignoring the failures, for running them later. Copying the test* binaries from the .libs directories to the phone, I was able to run them immediately What next There are three further steps in this work:
  • Generalising the script to make binaries for all supported Android architectures (not just ARM)
  • Extending the script to copy the shared libraries into the build tree of a traditional Android project where they will be automatically scooped up when the apk file is assembled.
  • Writing the necessary JNI code to access the C/C++ code from Java

Matthew Garrett: ARM and firmware specifications

Jon Masters, Chief ARM Architect at Red Hat, recently posted a description of his expectations for baseline arm64 servers. The quick summary is that systems should implement UEFI and ACPI, and any more traditional ARM boot mechanisms should be ignored. This is an interesting departure from the status quo in the ARM world, and it's worth thinking about the benefits and drawbacks of this approach.

It's very easy to build a generic kernel for most x86 systems, since the PC platform is fairly well defined even if not terribly well specified. Where system hardware does vary, it's almost always exposed on an enumerable bus (such as PCI or USB) which allows the OS to bind appropriate drivers. Things are different in the ARM world. Even once you're past the point of different SoC vendors requiring different kernel setup code and drivers, you still have to cope with the fact that system vendors can wire these SoCs up very differently. Hardware is often attached via GPIO lines without any means to enumerate them. The end result is that you've traditionally needed a different kernel for every ARM board. This is viable if you're selling the OS and hardware as a single product, but less viable if there's any desire to run a generic OS on the hardware.

The solution that's been adopted for this in the Linux world is called Device Tree. Device Tree actually has significant history, having been used as the device descriptor format in Open Firmware. Since there was already support for it in the Linux kernel, adapting it for use in ARM devices was straightforward. Device Tree aware devices can pass a descriptor blob to the kernel at startup[1], and devices without that knowledge can have a blob build into the kernel.

So, if this problem is already solved, why the push to move to UEFI and ACPI? This push didn't actually originate in the Linux world - Microsoft mandate that Windows RT devices implement UEFI and ACPI, and were they to launch a Windows ARM server product would probably carry that over. That makes sense for Microsoft, since recent versions of Windows have been x86 only and so have grown all kinds of support for ACPI and UEFI. Supporting Device Tree would require Microsoft to rewrite large parts of Windows, whereas mandating UEFI and ACPI allowed them to reuse most of their existing Windows boot and driver code. As a result, largely at Microsoft's behest, ACPI 5 has grown a range of additional features for describing things like GPIO pinouts and I2C connections. Whatever your weird device layout, you can probably express it via ACPI.

This argument works less well for Linux. Linux already supports Device Tree, whereas it currently doesn't support ACPI or UEFI on ARM[2]. Hardware vendors are already used to working with Device Tree. Moving to UEFI and ACPI has the potential to uncover a range of exciting new kernel issues and vendor bugs. It's not obviously an engineering win.

So how about users? There's an argument that since server vendors are now mostly shipping ACPI and UEFI systems, having ARM support these technologies makes it easier for customers to replace x86 systems with ARM systems. This really doesn't fly for ACPI, which is entirely invisible to the user. There are no standard ACPI entry points for system configuration, and the bits of ACPI that are generically useful (such as configuring system wakeup times) are already abstracted away to a standard interface by the kernel. It's somewhat more compelling for UEFI. UEFI supports a platform-independent bytecode language (EFI Byte Code, or EBC), which means that customers can write their own system management utilities, build them for EBC and then deploy them to their servers without caring about whether they're x86 or ARM. Want a bootloader that'll hit an internal HTTP server in order to determine which system image to deploy, and which works on both x86 and ARM? Straightforward.

Arnd Bergmann has a interesting counterargument. In a nutshell, ARM servers aren't currently aiming for the same market as x86 servers, and as a result customers are unlikely to gain any significant benefit from shared functionality between the two.

So if there's no real benefit to users, and if there's no benefit to kernel developers, what's the point? The main one that springs to mind is that there is a benefit to distributions. Moving to UEFI means that there's a standard mechanism for distributions to interact with the firmware and configure the bootloader. The traditional ARM approach has been for vendors to ship their own version of u-boot. If that's in flash then it's not much of a problem[3], but if it's on disk then you have to ship a range of different bootloaders and know which one to install (and let's not even talk about initial bootstrapping).

This seems like the most compelling argument. UEFI provides a genuine benefit for distributions, and long term it probably provides some benefit to customers. The question is whether that benefit is worth the flux. The same distribution benefit could be gained by simply mandating a minimum set of u-boot functionality, which would seem much more straightforward. The customer benefit is currently unclear.

In the end it'll probably be a market decision. If Red Hat produce an ARM product that has these requirements, and if Suse produce an ARM product that will work with u-boot and Device Tree, it'll be up to vendors to decide whether the additional work to support UEFI/ACPI is worth it in order to be able to sell to customers who want Red Hat. I expect that large vendors like HP and Dell will probably do it, but the smaller ones may not. The customer demand issue is also going to be unclear until we learn whether using UEFI is something that customers actually care about, rather than a theoretical benefit.

Overall, I'm on the fence as to whether a UEFI requirement is going to stick, and I suspect that the ACPI requirement is tilting at windmills. There's nothing stopping vendors from providing a Device Tree blob from UEFI, and I can't think of any benefits they gain from using ACPI instead. Vendor interest in the generic parts of the ACPI spec has been tepid even in the x86 world (the vast majority of ACPI spec updates come from Microsoft and Intel, not any of the system vendors), and I don't see that changing with the introduction of a range of ARM vendors who are already happy with Device Tree.

We'll see. Linux is going to need to gain the support for UEFI and ACPI on ARM in any case, since there's already hardware shipping in that configuration. But with ARM vendors still getting to grips with Device Tree, forcing them to go through another change in how they do things is going to be hard work. Red Hat may be successful in enforcing these requirements at the cost of some vendor unhappiness, or Red Hat may find that their product doesn't boot on most of the available hardware. It's an aggressive gamble, and while it'll be interesting to see how it plays out, I'm not that optimistic.

[1] The blob could be pulled from the firmware, but it's not uncommon for it to be built into u-boot instead. This does mean that you have a device-specific u-boot even if you have a generic kernel, but that's typically true anyway.
[2] Patches have been posted for ARM UEFI support. They're not mergeable in their current form, but they should be in the near future. ACPI support is in development.
[3] Although not all u-boots are created equal - some vendors ship versions that will only boot off FAT, some vendors ship versions that will only boot off ext2. Having to special case this stuff in your installer is a pain.

comment count unavailable comments

Daniel Pocock: Winning at any cost

It's not every day that a student messing around with keystroke loggers comes to fame through slashdot. Nonetheless, systematically rigging an election and getting sentenced to 12 months in a dorm with bars has helped raise 22 year old Matthew Weaver's profile well above that of the average script kiddie. Now let's stop and reflect on poor Weaver's future. You may be thinking that with an exchange program like this on his academic record he won't be so popular with employers. Given that he was busted by campus security rather than the FBI he won't even attract the interest of those companies who hire ex-hackers. So where could he go? How is it done in Australia? Not too long ago, when I was a student myself, one of our prominent universities was subjected to a very similar scam. Four members of the Tin Tin for NUS ticket at La Trobe University were implicated in stuffing the ballot the old fashioned way. The incidents even share the characteristics of chronic stupidity: just as Weaver had been caught voting for himself 259 times from the same IP address in a campus computer lab, team Tin Tin had tried to hand their bag of manipulated postal votes directly to the deputy returning officer rather than discretely posting them through the internal mail. According to an official report by the Deputy Returning Officer, Karsten Haley, all four candidates were charged with Dishonest Conduct and Interfering with Ballot Papers. Unfortunately, the report notes that
La Trobe University SRC Electoral Regulations do not empower the Returning Officer or Deputy to enforce charges or disciplinary procedures and the charges were never faced by the accused.
Given the seriousness of the matter, Haley did not give up his attempts to hold them to account. He escalated it to the Dean of the college and then to the University Secretary. He reports that "their disinterest was extraordinary" and that nobody would involve the police. Young Labor suspended Just over a year later, in 1997, the ALP's youth division for the state of Victoria, Young Labor, was suspended after attempts to rig the ballot to elect the Young Labor leadership team. The guilty parties were never publicly named. Nobody was formally suspended or expelled and this simply left them with more time on their hands to invest their energy in other elections. The suspension of Victorian Young Labor remained in effect for a number of years. The specific allegations about the Young Labor ballot suggest that those people particularly keen to win had printed fake student cards and given them to stooges who would impersonate other Young Labor members who had not attended to vote in person. Where are they now? It is no co-incidence that these students were (and still are) members of Labor Unity, a powerful faction within Australia's ruling Labor Party, the ALP. Most political organisations would presumably express concern about these allegations. The ALP does things differently. One of the students who withdrew his nomination in La Trobe, Mr Larocca, subsequently became Mayor in the City of Moreland, one of the ALP's strongholds. Even more remarkably from an outsider's viewpoint, another of these figures, Stephen Donnelly, is currently employed as the Assistant State Secretary of the ALP in Victoria. Communications like this newsletter reveal that he is one of the key figures in the party's pre-selection process. He has recently been appointed to direct the ALP's 2013 federal election campaign for the state of Victoria. Another co-incidence On the same weekend that Weaver was in the news for his antics, Donnelly's latest employer, the ALP's Victorian branch, was conducting pre-selection ballots to choose candidates for the upcoming federal election. So it's no surprise that Monday's newspaper headlines report fresh allegations of voting irregularities. Sadly, I've seen some of Labor Unity's bad behavior first hand. About 10 years ago I was living in South Melbourne, which is in the federal electoral district of Melbourne Ports. A young female friend of mine, a member of the local Elwood branch of the ALP, had spent election day handing out brochures for an ALP candidate in a marginal seat rather than assisting the controversial local ALP candidate, Michael Danby. A few days later I was witness to an incident where Danby aggressively confronted this young woman and demanded to know why he hadn't seen her handing out his own leaflets on polling day. He stood within centimeters of her and was literally looking down on her as he demanded some kind of apology to sooth his bruised ego. She looked terrified and barely responded. Within moments one of his handlers approached and physically moved Danby away from this young woman, I dare to think where things would have gone otherwise. Eye for talent Remarkably, at the same time, the infamous Stephen Donnelly had started shadowing Danby in his movements about the district. Fresh out of university, his talents had been recognised by Danby and he was employed in Danby's office, enabling him to continue honing his skills on a full-time basis with a tax-payer funded salary. What a remarkable contrast to the story of Weaver. Can anybody imagine a US congressman collecting Weaver from the prison gates and deploying him to an office on Capitol Hill? The biggest bankruptcy in student history Around the same time, Donnelly's Student Unity, the student arm of Labor Unity were successful in taking over the student union of my own campus, the University of Melbourne. Not long after I graduated I heard that they had been accused of skimming off $1 million from catering providers and a high-risk $46 million property transaction that put the organisation into liquidation. Unlike Mr Weaver, who's scheme at Cal State barely got off the ground, none of those involved in the Melbourne University incident has faced criminal proceedings. One ALP figure, Andrew Landeryou, spent several months in Costa Rica while wanted for questioning in the Supreme Court. His wife has just been endorsed for a seat in the Senate with support from various Labor Unity figures including Danby. The Gillard questions In 1996, around the same time that Donnelly & Co. were romping around student unions learning the tricks of the political trade, a lawyer quietly departed from the firm Slater and Gorden after an internal investigation into a property transaction linked to a union slush fund. Like Donnelly, this lawyer's next move was to take employment in the office of a Labor Party MP. More recently she was backed by Labor Unity to become Prime Minister. The union slush fund remains under investigation, frustrated by the disappearance of documents. The $60 million heist Recently I blogged about Gillard and Abbott, leaders of the two main political parties in Australia, agreeing to take $60 million of taxpayer money to fund their parties' campaigns in the upcoming federal election, giving themselves an obscenely unfair unadvantage over all other contestants. Where would that money end up? In the case of the ALP, does it appear likely that figures like the Victorian ALP's federal campaign director, Mr Donnelly, would be involved in the expenditure? National shame With this background, it becomes easier to understand the quality (or lack of it) in Australia's national leadership. When you consider that the generation responsible for the La Trobe incident, the Young Labor suspension and the MUSU bankruptcy are now growing into positions of greater responsibility in the ALP it leaves me feeling the quality of leadership is only going to get a lot worse before it starts getting better. For example, the recent incident where coloured people were fed to the sharks has nothing to do with the worldwide refugee crisis and everything to do with maintaining the dumbed-down level of political discourse that Labor Unity thugs and their followers can cope with. Real issues like climate change and energy policy, for example, appear to be beyond the pay grade of Australia's political class Ranjini - coloured, indefinite detention It is startling that up to her own recent demise, Gillard herself had repeatedly begged the public to stop asking questions about her own past and remember that Labor politicians are innocent until proven guilty - yet she had a pregnant coloured woman thrown into a concentration camp on unfounded fears about "national security". No evidence has ever been presented that poor Ranjini committed a crime, but the houses bought with money from trade unions, transactions handled through Gillard's own office, seem to be as solid as bricks and mortar. If only poor Matthew Weaver had been an Australian, how much further would his star have risen? Update: please sign the petition at change.org asking La Trobe university to re-examine the report and refer it formally to the police. If you are concerned about the plight of poor Ranjini and other people subject to Australia's domestic rendition program, please take a moment to see Letters for Ranjini

21 July 2013

Benjamin Mako Hill: The Wikipedia Gender Gap Revisited

In a new paper, recently published in the open access journal PLOSONE, Aaron Shaw and I build on new research in survey methodology to describe a method for estimating bias in opt-in surveys of contributors to online communities. We use the technique to reevaluate the most widely cited estimate of the gender gap in Wikipedia. A series of studies have shown that Wikipedia s editor-base is overwhelmingly male. This extreme gender imbalance threatens to undermine Wikipedia s capacity to produce high quality information from a full range of perspectives. For example, many articles on topics of particular interest to women tend to be under-produced or of poor quality. Given the open and often anonymous nature of online communities, measuring contributor demographics is a challenge. Most demographic data on Wikipedia editors come from opt-in surveys where people respond to open, public invitations. Unfortunately, very few people answer these invitations. Results from opt-in surveys are unreliable because respondents are rarely representative of the community as a whole. The most widely-cited estimate from a large 2008 survey by the Wikimedia Foundation (WMF) and UN University in Maastrict (UNU-MERIT) suggested that only 13% of contributors were female. However, the very same survey suggested that less than 40% of Wikipedia s readers were female. We know, from several reliable sources, that Wikipedia s readership is evenly split by gender a sign of bias in the WMF/UNU-MERIT survey. In our paper, we combine data from a nationally representative survey of the US by the Pew Internet and American Life Project with the opt-in data from the 2008 WMF/UNU-MERIT survey to come up with revised estimates of the Wikipedia gender gap. The details of the estimation technique are in the paper, but the core steps are:
  1. We use the Pew dataset to provide baseline information about Wikipedia readers.
  2. We apply a statistical technique called propensity scoring to estimate the likelihood that a US adult Wikipedia reader would have volunteered to participate in the WMF/UNU-MERIT survey.
  3. We follow a process originally developed by Valliant and Dever to weight the WMF/UNU-MERIT survey to correct for estimated bias.
  4. We extend this weighting technique to Wikipedia editors in the WMF/UNU data to produce adjusted estimates of the demographics of their sample.
Using this method, we estimate that the proportion of female US adult editors was 27.5% higher than the original study reported (22.7%, versus 17.8%), and that the total proportion of female editors was 26.8% higher (16.1%, versus 12.7%). These findings are consistent with other work showing that opt-in surveys tend to undercount women. Overall, these results reinforce the basic substantive finding that women are vastly under-represented among Wikipedia editors. Beyond Wikipedia, our paper describes a method online communities can adopt to estimate contributor demographics using opt-in surveys, but that is more credible than relying entirely on opt-in data. Advertising-intelligence firms like ComScore and Quantcast provide demographic data on the readership of an enormous proportion of websites. With these sources, almost any community can use our method (and source code) to replicate a similar analysis by: (1) surveying a community s readers (or a random subset) with the same instrument used to survey contributors; (2) combining results for readers with reliable demographic data about the readership population from a credible source; (3) reweighting survey results using the method we describe. Although our new estimates will not help us us close the gender gap in Wikipedia or address its troubling implications, they give us a better picture of the problem. Additionally, our method offers an improved tool to build a clearer demographic picture of other online communities in general.

19 July 2013

Justus Winter: A story about virtualization

I'm sure you all will be a bit disappointed (I know I am) that there are no ascii screenshots in this weeks report. But let me make it up to you by telling you a story. I was at the FOSDEM in the year 2012 and I went to a nice workshop on sunday afternoon in the Virtualization Devroom. Renzo Davoli was the host of this workshop and he started with a little introductory talk to get everyone to agree on a common terminology first. He began by asking the question, what virtualization meant in the most general way. He defined the ability to virtualize a resource as the ability to freely and transparently control how someone (say a process) is interacting with said resource. So for example if you LD_PRELOAD a library to impersonate fopen(3) and friends, you have virtualized the filesystem (for some small values of virtualized). He then went on to discuss various methods of virtualization that are commonly available on Linux, not only full system virtualization solutions, but also all kinds of methods allowing a more fine-grained control over what resources are virtualized. Of course every method had its strengths and its weaknesses. Having seen Samuels talk about all the awesome things Hurd can do for virtualization I approached him with a question in the free discussion part of his workshop. I asked whether he would agree that once a resource is moved from the kernel to the userspace, the problem of virtualizing that resource is almost trivially solved, and he agreed. So I said that this was awesome, because then the trivial and elegant solution for all his virtualization (and tracing) needs are micro kernel operating systems, and he also agreed to that but (as far as I can recall) he mentioned that there is none that would meet his needs (I believe he is a computer science professor and heavily relies on virtualization techniques for his lab and for teaching purposes). And even though a microkernel operating systems has a cost (message passing instead of function calls for example) it also has its merits such as scaling better from a development point of view. Also, many cool features can emerge just from the design. For example think about the container support in Linux and how painfully long it took to make all the resources namespace aware (with one of the most critical, the user namespace being the most difficult one). On Hurd you get the same functionality for free. If you are curious, read Samuels awesome slides. So what have I done this week? Next week I'm going to work on the two remaining sysvinit related issues. These are the only ones preventing me to come up with a clean patch series against the sysvinit package and I figured that it would be nice to propose such a series rather sooner than later so that I will have plenty of time to discuss any issues with the sysvinit maintainers. See you next week :)

Michal Čihař: Speed improvements in Weblate

Yesterday, I've spent some time trying to improve speed of translation import in Weblate. It got annoyingly slow while adding new features and I felt there has to be some area for improvements even with all the features. I was testing on development server with debug stuff enabled (what slows down things quite a lot already) and first import took about 30 seconds. Quick look revealed that for linked repositories, the link lookup happens too often, so some caching could be used. This has saved about 3 seconds... With offloaded indexing (what is anyway only useful setup for production use), every reindex request was checking whether there is already one before creating new one. In most cases there is none before, so it's simply faster to try to create one and ignore error in case it is already there. We're now 5 seconds faster. Another big resource consumer are quality checks. They provide useful feedback to translators, but it means that quite a lot of code needs to be executed for every single string. First thing I've realized is that unit check state is recalculated everytime regardless the fact whether there was any change, what was easy to fix. The other speed issue is in consistency check - it does too complex SQL query, which usually does not match anything. Converting this to much simpler one (which will in most cases still have no results) improves performance for usual path while having minimal impact for the real checking. Another 4 seconds down on my test import. Last thing which I changed is way how per translation statistics are updated. In past this required quite a lot of queries, but yesterday I've realized this can be much reduced. During test suite run on PostgreSQL, I saw that it's not that easy - PostgreSQL does not allow to sum boolean fields, but fortunately this can be fixed as well. This has saved about 2 seconds. In the end the tested import took 16 seconds instead of original 30 and issued 2500 SQL queries instead of 5000 at the beginning. I hope you will enjoy regained speed of imports :-).

Filed under: English Weblate 0 comments Flattr this!

18 July 2013

Vincent Bernat: Packaging a daemon for OS X

There are three main ways to distribute a command-line daemon for OS X:
  1. Distributing source code and instructions on how to compile it.
  2. Using a third-party package manager, like Homebrew.
  3. Providing an installer package.

Homebrew Homebrew is a popular package management system. It works like the BSDs ports collections by downloading, compiling and installing the requested software, while also installing any required dependencies automatically. Creating a new package is quite easy and there are a lot of examples available. However, there are some limitations:
  • You don t really build a package but execute a recipe to locally install the software.
  • You need to install development tools, either a whole Xcode installation1 or the command line version.
  • If you need to execute some steps as root, you will have to explain them to the user for her to execute by hand. This includes the creation of a system user or the installation of a daemon through launchd.
If you can t live with those limitations, you may need to build an installer package.

Building an installer package OS X comes with a graphical and a command-line installer. The graphical one is run by opening a package from the Finder. Then, the user experiences some familiar wizard allowing her to install the software in a matter of a few seconds. OS X graphical installer The documentation around how to build such a package is somewhat clumsy. You may find outdated information or pieces of information that only apply to projects using Xcode. I will try to provide here accurate bits in the following context:
  1. You are packaging a pure command-line tool.
  2. You are using Autoconf and Automake as a build system.
  3. You want to support several architectures.
  4. You want to support older versions of OS X.

Creating a package Building such a package was previously done with some graphical tool called PackageMaker. This tool is not available any more and developers are asked to switch to pkgbuild and productbuild. There is a very neat Stackoverflow article on how those tools work. A package is built in two steps:
  1. Build component packages.
  2. Combine them into a product archive.
A component package contains a set of files and a set of scripts to execute at various steps of the installation. You can have several component packages, for example a package for the daemon and a package for the client. They are built with pkgbuild. A product archive contains the previously created component packages as well as a file describing various options of the installer (required and optional components, license, welcome text, ). To create a component package, we need to install the needed files in some directory:
$ ./configure --prefix=/usr --sysconfdir=/private/etc
$ make
$ make install DESTDIR=$PWD/osx-pkg
We will now put the content of osx-pkg into a component package with pkgbuild:
$ mkdir pkg1
$ pkgbuild --root osx-pkg \
>    --identifier org.someid.daemon \
>    --version 0.47 \
>    --ownership recommended
>    pkg1/output.pkg
pkgbuild: Inferring bundle components from contents of osx-pkg
pkgbuild: Wrote package to output.pkg
You need to be careful with the identifier. It needs to be unique and identify your software as well as the specific component. Then, you need to create an XML file describing the installer. Let s call it distribution.xml:
<?xml version="1.0" encoding="utf-8" standalone="no"?>
<installer-gui-script minSpecVersion="1">
    <title>Some daemon</title>
    <organization>org.someid</organization>
    <domains enable_localSystem="true"/>
    <options customize="never" require-scripts="true" rootVolumeOnly="true" />
    <!-- Define documents displayed at various steps -->
    <welcome    file="welcome.html"    mime-type="text/html" />
    <license    file="license.html"    mime-type="text/html" />
    <conclusion file="conclusion.html" mime-type="text/html" />
    <!-- List all component packages -->
    <pkg-ref id="org.someid.daemon"
             version="0"
             auth="root">output.pkg</pkg-ref>
    <!-- List them again here. They can now be organized
         as a hierarchy if you want. -->
    <choices-outline>
        <line choice="org.someid.daemon"/>
    </choices-outline>
    <!-- Define each choice above -->
    <choice
        id="org.someid.daemon"
        visible="false"
        title="some daemon"
        description="The daemon"
        start_selected="true">
      <pkg-ref id="org.someid.daemon"/>
    </choice>
</installer-gui-script>
Fortunately, this file is documented on Apple Developer Library. Since we only describe one package, we pass customize="never" as an option to skip the choice part. You can however remove this attribute when you have several component packages. The attribute rootVolumeOnly="true" explains that this daemon can only be installed system-wide. It is marked as deprecated but still works. Its replacement (domains tag) displays an unusual and buggy pane which Apple doesn t use in any of its packages. You need to put the HTML2 documents into a resources directory. It is also possible to choose the background by specifying a <background/> tag. The following command will generate the product archive:
$ productbuild --distribution distribution.xml \
>              --resources resources \
>              --package-path pkg1 \
>              --version 0.47 \
>              ../final.pkg
productbuild: Wrote product to ../final.pkg

Scripts The installer can execute some scripts during installation. For example, suppose we would like to register our daemon with launchd for it to be run at system start. You first need to write some org.someid.plist file and ensure it will get installed in /Library/LaunchDaemons:
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
  <key>Label</key>
  <string>org.someid</string>
  <key>ProgramArguments</key>
  <array>
    <string>/usr/sbin/mydaemon</string>
    <string>-d</string>
  </array>
  <key>RunAtLoad</key><true/>
  <key>KeepAlive</key><true/>
</dict>
</plist>
Then create a scripts/postinstall file with the following content:
#!/bin/sh
set -e
/bin/launchctl load "/Library/LaunchDaemons/org.someid.plist"
You also need a scripts/preinstall file which will allow your program to be smoothly upgraded:
#!/bin/bash
set -e
if /bin/launchctl list "org.someid" &> /dev/null; then
    /bin/launchctl unload "/Library/LaunchDaemons/org.someid.plist"
fi
Ensure that those scripts are executable and add --scripts scripts to pkgbuild invocation. If you need to add a system user, use dscl.

Dependencies There is no dependency management built into OS X installer. Therefore, you should ensure that everything is present in the package or in the base system. For example, lldpd relies on libevent, an event notification library. This library is not provided with OS X. When it is not found, the build system will use an embedded copy. However, if you have installed libevent with Homebrew, you will get binaries linked to this local libevent installation. Your package won t work on another host. The output of otool -L can detect unwanted dependencies:
$ otool -L build/usr/sbin/lldpd
build/usr/sbin/lldpd:
        /usr/lib/libresolv.9.dylib (compatibility version 1.0.0, current version 1.0.0)
        /System/Library/Frameworks/IOKit.framework/Versions/A/IOKit (compatibility version 1.0.0, current version 275.0.0)
        /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 744.19.0)
        /System/Library/Frameworks/Foundation.framework/Versions/C/Foundation (compatibility version 300.0.0, current version 945.18.0)
        /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 169.3.0)

Compilation for older versions of OS X The package as built above may only work on the same version of OS X that you use to compile. If you want a package that will work on OS X 10.6, you will need to download the appropriate SDK3. Then, you need to tell the compiler the target version of OS X and the SDK location. You can do this by setting CFLAGS and LDFLAGS (and CXXFLAGS if you use C++):
$ SDK=/Developer/SDKs/MacOSX10.6.sdk
$ ./configure --prefix=/usr --sysconfdir=/private/etc \
>         CFLAGS="-mmacosx-version-min=10.6 -isysroot $SDK" \
>         LDFLAGS="-mmacosx-version-min=10.6 -isysroot $SDK"
The -mmacosx-version-min flag will be used by various macros in the headers to mark some functions as available or not, depending on the version of OS X you target. It will also be used by the linker. The -isysroot will tell the location of the SDK. Headers and libraries will be looked up in this location first.

Universal binaries Mac OS X 10.6 was available for both IA-32 and x86-64 architectures. If you want to support both installations with a single package, you need to build universal binaries. The Mach object file format, used by OS X, allows several versions of the executable in the same file. The operating system will select the most appropriate one. An easy way to generate such files is to pass -arch i386 -arch x86_64 to the compiler:
$ ./configure --prefix=/usr --sysconfdir=/private/etc \
>     CC="gcc -arch i386 -arch x86_64" \
>     CPP="gcc -E"
However, this is a dangerous option. Suppose that your ./configure script tries to determine some architecture-dependent parameter, like the size of an integer (with AC_CHECK_SIZEOF); it will compute it for the host architecture. The binary generated for the other architecture will therefore use a wrong value and may crash at some point. The correct way to generate an universal binary is to execute two separate compilations and to build the universal binary with lipo:
$ for arch in i386 x86_64; do
>   mkdir $arch ; cd $arch
>   ../configure --prefix=/usr --sysconfdir=/private/etc \
>       CC="gcc -arch $ARCH" \
>       CPP="gcc -E"
>   make
>   make install DESTDIR=$PWD/../target-$ARCH
>   cd ..
> done
[...]
$ lipo -create -output daemon i386/usr/sbin/daemon x86_64/usr/sbin/daemon
Since lipo only works on file, I have written a Python script applying lipo recursively to several directories.

Putting everything together Now, we need to automate a bit. You could provide some nifty script. I propose an appropriate Makefile.am instead. This way, we can use the output variables, like @VERSION@, from ./configure to generate some of the files, like distribution.xml. You need to add that in your configure.ac:
AC_CONFIG_FILES([osx/Makefile osx/distribution.xml osx/im.bernat.lldpd.plist])
AC_CONFIG_FILES([osx/scripts/preinstall],  [chmod +x osx/scripts/preinstall])
AC_CONFIG_FILES([osx/scripts/postinstall], [chmod +x osx/scripts/postinstall])
AC_SUBST([CONFIGURE_ARGS], [$ac_configure_args])
Let s have a look at osx/Makefile.am. First, we define some variables:
PKG_NAME=@PACKAGE@-@VERSION@.pkg
PKG_TITLE=@PACKAGE@ @VERSION@
PKG_DIR=@PACKAGE@-@VERSION@
ARCHS=@host_cpu@
If we want to build for several architectures, we will type make ARCHS="x86_64 i386". Otherwise it defaults to the current host s architecture. We use install-data-local target to install files specific to OS X:
install-data-local:
    install -m 0755 -d $(DESTDIR)/Library/LaunchDaemons
    install -m 0644 im.bernat.@PACKAGE@.plist $(DESTDIR)/Library/LaunchDaemons
uninstall-local:
    rm -f $(DESTDIR)/Library/LaunchDaemons/im.bernat.@PACKAGE@.plist
The main target is the product archive, built with productbuild:
../$(PKG_NAME): pkg.1/$(PKG_NAME) distribution.xml resources
    $(PRODUCTBUILD) \
        --distribution distribution.xml \
        --resources resources \
        --package-path pkg.1 \
        --version @VERSION@ \
        $@
Its main dependency is the component package:
pkg.1/$(PKG_NAME): $(PKG_DIR) scripts
    [ -d pkg.1 ]   mkdir pkg.1
    $(PKGBUILD) \
        --root $(PKG_DIR) \
        --identifier im.bernat.@PACKAGE@.daemon \
        --version @VERSION@ \
        --ownership recommended \
        --scripts scripts \
        $@
Now, we need to build $(PKG_DIR):
$(PKG_DIR): stamp-$(PKG_DIR)
stamp-$(PKG_DIR): $(ARCHS:%=%/$(PKG_DIR))
    $(srcdir)/lipo $(PKG_DIR) $^
    touch $@
The $(ARCHS:%=%/$(PKG_DIR)) will expand to x86_64/$(PKG_DIR) i386/$(PKG_DIR). This is a substitution reference4. Before applying our lipo script, we need to be able to build the architecture-dependent package directories:
pkg_curarch = $(@:stamp-%=%)
$(ARCHS:%=%/$(PKG_DIR)): %/$(PKG_DIR): stamp-%
$(ARCHS:%=stamp-%): stamp-%: im.bernat.lldpd.plist
    [ -d $(pkg_curarch) ]   mkdir -p $(pkg_curarch)
    (cd $(pkg_curarch) && \
        $(abs_top_srcdir)/configure @CONFIGURE_ARGS@ \
            CC="@CC@ -arch $(pkg_curarch)" \
            CPP="@CPP@")
    (cd $(pkg_curarch) && \
        $(MAKE) install DESTDIR=$(abs_builddir)/$(pkg_curarch)/$(PKG_DIR))
    touch $@
This may seem a bit obscure, but this is really like what we have described previously. Note that we use @CONFIGURE_ARGS@ which is a variable we have defined in configure.ac. Here is how a user would create its own OS X package:
$ SDK=/Developer/SDKs/MacOSX10.6.sdk
$ mkdir build && cd build
$ ../configure --prefix=/usr --sysconfdir=/private/etc --with-embedded-libevent \
   CFLAGS="-mmacosx-version-min=10.6 -isysroot $SDK" \
   LDFLAGS="-mmacosx-version-min=10.6 -isysroot $SDK"
[...]
$ make -C osx pkg ARCHS="i386 x86_64"
[...]
productbuild: Wrote product to ../lldpd-0.7.5-21-g5b90c4f-dirty.pkg
The package has been built in ../lldpd-0.7.5-21-g5b90c4f-dirty.pkg.

  1. Even if Xcode is free, you will need to register to Apple and show a valid credit card number. That s quite an interesting requirement.
  2. It is possible to use other formats (like RTF or just plain text) for documents but HTML seems the best choice if you want some formatting without fiddling with RTF.
  3. Unfortunately, Apple makes it difficult to find them. The easiest way to get the SDK for Mac OS X 10.6 is to download Xcode 4.3.3, mount the image and copy the SDK to /Developer/SDKs.
  4. This is equivalent to the use of patsubst function. However, this function is specific to GNU make. The substitution reference pattern was accepted recently in POSIX and is coined as pattern macro expansion.

16 July 2013

Joachim Breitner: Real World Haskell Applications

Today I held a lightning talk at the Karlsruhe Functional Programmer s Group, where I presented Real World Haskell applications, i.e. programs that are written in Haskell, are somewhat mature and well-known and have non-programmers as their target group. Here is the (surely subjective) list that I have come up (also see the slides): There is also an impressive list of industrial users that use Haskell without talking about it much, but that does not work well to show off that you can do any kind of application in Haskell. If I missed anything important, let me know!

15 July 2013

Daniel Pocock: Linux init-systems

Several value-added init systems have emerged in recent times. Most notably, Upstart has become the default on Ubuntu while systemd is now default in Fedora. Let's not forget the crusade to move beyond traditional init was given a big boost back in 2005 by Sun's introduction of SMF on Solaris 10 some years before the main Linux solutions gained prominence. Debian is currently debating whether to use one of these alternatives as a default. My own position on the matter is not decided, but I do feel that some of the debates are asking the wrong questions. Some of the most interesting issues here are likely to have ramifications for other distributions. My experience with systemd I've come across systemd in my efforts to package reSIProcate on Fedora. One of the challenges I came across was trying to both create a /var/run/repro directory and telling systemd to launch the process as a non-root user. I tried using the ExecStartPre directive in the systemd manifest to invoke mkdir. Unfortunately, it runs as the non-root user too and could not create the PID directory. The conventional init script doesn't suffer this problem. In the end, I have patched the repro binary to start as root and drop privileges itself, but that was only when it became apparent that we need to do that for WebRTC support (listening on port 443). While most daemons have the ability to drop privileges, there will be edge cases like this that need more than just a systemd manifest. Maintainers who encounter such issues may have to give feedback to the systemd and Upstart upstreams and give them the opportunity to provide generalized solutions. Then again, the most generalized solution would be another scripting language and then we'd be back where we started. The reason to change and the reason not to Some people have argued that Debian should change simply because other distributions did so. This is not a compelling technical argument alone. The fragmentation of init-alternatives is not something that makes upstreams excited either. It would be interesting to see if Upstart and systemd could agree on a common manifest format - maybe even a format that could be parsed by a wrapper from SysV init. Then Debian wouldn't really have to give up support for kFreeBSD and other non-Linux kernel efforts. A slight alternative to this approach would be using a system like cmake to read a single service XML manifest and spit out permutations readable by each of the different init schemes. It has also been well argued that the simple and well known solution of scripts is much less complicated to understand and maintain and is extremely stable. Wishlist items for all distributions While I don't know which solution is the best, one thing I do believe is that if there is a change and if it is going to be disruptive to maintainers and users then it should aim to bring in as many benefits as possible to offset the pain and encourage people to support the migration effort. Some of the potential road-map items are not only possible for Debian but could also become part of other platforms. Here are some of the issues that came to my mind:
  • Better support for starting multiple instances of a single process
  • Support for distributed solutions, for example, making sure that process A only starts on host 1 after process B on host 2
  • Support for clustering, for example, making sure that at least 2 instances of process A are running across 3 hosts.
  • Automated integration with monitoring and fault recovery systems: for example, can Nagios (or another tool) automatically detect whether a particular process should be running and whether it is running? Given that init-replacements typically need to start processes in the correct order, can they share that dependency information with Nagios to avoid manual/duplicated dependency mappings? Can Nagios restart the process without any manual scripting? Can this all be enabled for all processes on a given system in a plug-and-play manner?
  • Scheduler interaction: can and should any of the above, particularly the dependency mappings, be integrated with scheduling? For example, should some cron jobs be deferred when MySQL is down?
What other potential improvements do people feel to be worthwhile?

14 July 2013

Matt Zimmerman: Liberty and justice for all, but not in equal measure

200302020-001 As Americans we might like to believe that the US legal system is intended to protect all of our citizens. Unfortunately, it doesn t protect us all equally, and in fact disproportionately fails to protect the most vulnerable. We re surrounded by instances of injustice related to gender, race and other axes of social privilege, and the machinations of law are not exempt. The state of Florida has recently provided an especially stark example in the application of its self-defense laws in two cases: Marissa Alexander and George Zimmerman. This example is notable because although there were many similarities between the cases, the outcomes were very different. Alexander s case was tried in May 2012 , Zimmerman s in July 2013, both prosecuted by Florida state s attorney Angela Corey. Both cases involved the use of firearms which were legally purchased and carried, and their owners were trained in their use. Both prosecutions cast the defendant as the aggressor, who could have avoided the confrontation. Both of the encounters were with unarmed persons. Both defenses were based on Florida self-defense laws, which include stand your ground laws justifying the use of deadly force without the obligation to retreat. Both shooters admitted to firing a single shot with the intent of defending themselves. Beyond those similarities, each case had its own unique circumstances. The events of Alexander s case took place in her home. Her altercation was with her husband, Rico Gray Sr., who was under a restraining order following a conviction for domestic battery which put Alexander in the hospital. After Gray threatened to kill her, Alexander retrieved a handgun from her car, returned to confront him, firing once. She was arrested and charged the same day. She had had no prior criminal record. A jury deliberated for just 12 minutes before convicting her. A judge sentenced her to 20 years in prison, in accord with mandatory minimums specified by law. Gray, previously sentenced to probation for his earlier conviction, remains free. Zimmerman,_George_-_Seminole_County_MugZimmerman s shot was fired in his neighborhood, in an altercation with a teenager, Trayvon Martin, who was a guest in the community and walking by himself. The two were not acquainted. Zimmerman called police from his car, claiming that Martin appeared suspicious, and began to follow him. Some of the facts of their encounter remain in dispute, but that Zimmerman fired his gun is not in question. Afterward, Zimmerman was detained by police, questioned and released the same night without being arrested or charged. Following a public outcry, a new investigation was launched and two months later he was arrested and charged. He had been previously arrested and charged with assaulting a police officer, but the charges were later dropped. After 16 hours of deliberation, the jury found Zimmerman not guilty, and he is free today. The most striking difference between the two cases is where each defendant aimed their gun: George Zimmerman shot Trayvon Martin in the chest and killed him, while Marissa Alexander fired at a wall and injured no one. Alexander, a black woman, is in prison for scaring her abusive husband away, while Zimmerman, who killed a young black man, walks free. Alexander and Martin s families have lost a mother and a son. The outcomes for the people involved in these cases could not be more different. Regardless of the merits of the relevant laws themselves, their radically unequal application is deeply troubling. What does this tell us about the relative value of these human lives, as weighed by the judicial system?
The Florida criminal justice system has sent two clear messages today. One is that if women who are victims of domestic violence try to protect themselves, the Stand Your Ground Law will not apply to them. [...] The second message is that if you are black, the system will treat you differently. - U.S. Rep. Corrine Brown
References

Michael Stapelberg: Systemd support in Debian packages: how to help

Sometimes, people show up in our IRC channel #debian-systemd or on our mailing list pkg-systemd-maintainers@lists.alioth.debian.org and ask how they can help. This blog post answers that question. First of all, whatever you end up doing, please coordinate with us first! Saying hi in our IRC channel or on our mailing list and explaining what you intend to do is all we ask for. In the past, people have filed bug reports with service files that are not idiomatic, and we really need to avoid that. Currently, our biggest task is to add/improve systemd support to individual Debian packages, and this is where we can use a lot of help. We have a dashboard which contains two lists:
  1. Packages that ship a service file, but don t use dh-systemd yet.
    These need to be updated to get proper maintscripts (e.g. deb-systemd-helper enable/disable calls, possibly start/stop/restart calls).
  2. Packages that ship an init script but no service file yet.
    These need a service file plus dh-systemd support.
Adding a service file and dh-systemd support After you picked a service from the second list in our dashboard, first double-check that there is no bug filed against it yet (the dashboard might be a few hours behind). Then, search for a service file, either upstream or in other distributions such as Arch Linux, Fedora, Gentoo, etc. In case there is no service file yet, write one. Please carefully test the service file (e.g. does starting/stopping/restarting and reloading work) and send it to us for review. Be careful to avoid obvious regressions from the sysvinit init script. That is, don t just write a very simplistic service file that breaks a use-case which formerly worked. If in doubt, ask pkg-systemd-maintainers or chose a different package to work on. Afterwards, add systemd support by using the dh-systemd helper, see https://wiki.debian.org/Systemd/Packaging. The last step is to file a proper bug report against the package that uses our usertag and CCs pkg-systemd-maintainers. See http://bugs.debian.org/714190 for an example. Replacing custom maintscripts with dh-systemd Check if the package currently has any direct systemctl calls in its maintscripts and remove them if so (see http://bugs.debian.org/713853 for an example). Follow https://wiki.debian.org/Systemd/Packaging to add systemd support using the dh-systemd helper. Check whether the maintscripts look good and generate a debdiff to highlight the differences between the old and the new version:
debdiff --controlfiles=ALL OLD_amd64.changes NEW_amd64.changes
Then file a bug report which uses our usertag and CCs pkg-systemd-maintainers. See http://bugs.debian.org/713853 for an example. Thanks! Thank you for helping us and Debian in general!

Next.

Previous.