Search Results: "ed"

23 July 2013

MJ Ray: Misusing a Royal Baby and Child Porn to Censor The Internet

There s been some media coverage at the start of this week about blocking child porn. Except it s not about child porn that s a trojan horse. People who want to access pornography that is already illegal (Protection of Children Act 1978) are probably already using security tools to hide their downloading and will be unaffected by this unless they re pretty stupid. And the announcement, about the same time as the predicted birth of a royal baby, third in line to the throne, seems like a cynical attempt to bury bad news taken straight from the Blair Government. That would almost be enough reason to oppose it: they don t want the media to look at this too closely for some reason. So what s this actually about? It looks like a way to force through widespread acceptance of the ability to censor most UK internet users by shouting won t somebody think of the children? If you doubt it, take a look at the list of filtered topics: So if they get away with this censorship, you won t be able to use Twitter or contact the Samaritans until you deactivate it. Except I suspect you will because they re pretty big and the Cameron Government won t want to pick a fight with them: it ll be the next Twitter and the next Samaritans, currently much smaller and unable to defend themselves, who get shut out of UK homes. So what can we do, besides explaining this and writing to our MPs? Are we better off joining parties who oppose this censorship, like the Pirate Party, or joining existing parties and trying to overturn their stupid support for it?

Keith Packard: present sync

Implementing Vblank Synchronization in the Present Extension This is mostly a status update on how the Present extension is doing; the big news this week is that I ve finished implementing vblank synchronized blts and flips, and things seem to be working quite well. Vblank Synchronized Blts The goal here is to have the hardware executing the blt operation in such a way as to avoid any tearing artifacts. In current drivers, there are essentially two different ways to make this happen:
  1. Insert a command into the ring which blocks execution until a suitable time immediately preceding the blt operation.
  2. Queue the blt operation at vblank time so that it executes before the scanout starts.
Option 1. provides the fewest artifacts; if the hardware can blt faster than scanout, there shouldn t ever be anything untoward visible on the screen. However, it also blocks future command execution within the same context. For example, if two vblank synchronized blts are queued at the same time, its possible for the second blt to be delayed by yet another frame, causing both applications to run at half of the frame rate. Option 2. avoids blocking the hardware, allowing for ongoing operations to proceed on the hardware without waiting for the synchronized blt to complete. However, it can cause artifacts if the delay from the vblank event to the eventual execution of the blt command is too long. Queuing the blt right when it needs to execute means that we also have the opportunity to skip some blts; if the application presents two buffers within the same frame time, the blt of the first buffer can be skipped, saving memory bandwidth and time. Present uses Option 2, which may occasionally cause a tearing artifact, but avoids slowing down applications while allowing the X server to discard overlapping blt operations when possible. Queuing the Blt at Vblank There are several options for getting the blt queued and executed when the vblank occurs:
  1. Queue the blt from the interrupt handler
  2. Queue the blt from a kernel thread running in response to the interrupt
  3. Send an event up to user space and have the X server construct the blt command.
These are listed in increasing maximum latency, but also in decreasing complexity. Option 1. is made more complicated as much of the work necessary to get a command queued to the hardware cannot be done from interrupt context. One can imagine having the desired command already present in the ring buffer and have the interrupt handler simply move the ring tail pointer value. Future operations to be queued before the vblank operation could then re-write the ring as necessary. A queued operation could also be adjusted by the X server as necessary to keep it correct across changes to the window system state. Option 2. is similar, but the kernel implementation should be quite a bit simpler as the queuing operation is done in process context and can use the existing driver infrastructure. For the X server, this is the same as Option 1, requiring that it construct a queued blt operation and deliver that to the kernel, and then revoke and re-queue if the X server state changed before the operation was completed. Option 3 is the simplest of all, requiring no changes within the kernel and few within the X server. The X server waits to receive a vblank notification event for the appropriate frame and then simply invokes existing mechanisms to construct and queue the blt operation to the kernel. Oddly, Present currently uses Option 3. If that proves to generate too many display artifacts, we can come back and change the code to try something more complicated. Flipping the Frame Buffer Taking advantage of the hardware s ability to quickly shift scanout from one chunk of memory to another is critical to providing efficient buffer presentation within the X server. It is slightly more complicated to implement than simply copying data to the current scanout buffer for a few reasons:
  1. The presented pixmap is owned by the application, and so it shouldn t be used except when the presented window covers the whole screen. When the window gets reconfigured, we end up copying the window s pixmap to the regular screen pixmap.
  2. The kernel flipping API is asynchronous, and doesn t provide any abort mechanism. This isn t usually much of an issue; we simply delay reporting the actual time of flip until the kernel sends the notification event to the X server. However, if the window is reconfigured or destroyed while the flip is still pending, cleaning that up must wait until the flip has finished.
  3. The applications buffer remains busy until it is no longer being used for scanout; that means that applications will have to be aware of this and ensure that they don t deadlock waiting for the current scanout buffer to become idle before switching to a new scanout buffer.
Present is different from DRI2 in using application-allocated buffers for this operation. For DRI2, when flipping to a window buffer, that buffer becomes the screen pixmap the driver flips the new buffer object into the screen pixmap and releases the previous buffer object for other use. For Present, as the buffer is owned by the application, I figured it would be better to switch back to the real screen buffer when necessary. This also means that applications aren t left holding a handle to the frame buffer, which seems like it might be a nice feature. The hardest part of this work was dealing with client and server shutdown, dealing with objects getting deleted in random orders while other data structures retained references. (The kernel DRM drivers use the term page flipping to mean an atomic flip between one frame buffer and another, generally implemented by simply switching the address used for the scanout buffer. I d like to avoid using the word page in this context as we re not flipping memory pages individually, but rather a huge group of memory that forms an entire frame buffer. We could use plane flipping (as intel docs do), frame buffer flipping (but that s a mouthful), display flipping or almost anything but page flipping ). Overall DRI3000 Status At this point, the DRI3 extension is complete and the Present extension is largely complete, except for redirection for compositors. The few piglit tests for GLX_OML_sync_control all pass now, which is at least better than DRI2 does. I think I ve effectively replicated the essential features of DRI2 while offering room to implement a couple of new GL extensions: The code for this stuff has all been pushed to a number of repositories:

22 July 2013

Keith Packard: async flip

Asynchronous Display Flipping for Linux Using display (or plane or page) flipping is a nice optimization for screen updates as it avoids copying the application render buffer to the separate scanout buffer. However, in the current DRM interface, the flip operation is not performed immediately. Rather, the flip operation must be presented to the kernel before the vertical blank interval preceding the desired display time. If the flip operation is even slightly delayed, the new scanout image will not be visible until the following frame. With many applications pushing the performance of the graphics hardware right to the edge, it s not at all uncommon for an occasional frame to just miss and cause an ugly jerk on the screen. The GLX_EXT_swap_control_tear extension provides a way for applications to request that when the swap request is delivered too late, instead of delaying for the next frame, the contents should be presented as soon as possible, even if that might cause some tearing artifacts on the screen. This gives the application developer the choice between the two visual artifacts (tearing vs judder). A Linux interface for Async Flipping This part was pretty easy to implement. First, a new capability value, DRM_CAP_ASYNC_PAGE_FLIP was added so that applications could discover whether the driver offered this feature or not. Second, a new flag value, DRM_MODE_PAGE_FLIP_ASYNC was added to the DRM_IOCTL_MODE_PAGE_FLIP ioctl. To get the new flag down into the driver required adding the flag parameter to the page_flip crtc function. Async Display Flipping support in Intel Hardware Recent Intel display hardware has support for asynchronous flipping, either through the ring using the MI_DISPLAY_FLIP instruction or directly through the registers. There are some limitations however: I ve also only implemented flipping on Sandybridge and Ivybridge hardware as that s what I ve got for testing. Using Async Display Flipping with Present I m using async display flipping when the kernel supports it in the current Present implementation. This offers the ability to short-circuit multiple presentations targeting the same scanout frame. I haven t exposed the ability to select this to applications over the Present protocol yet, but that s not a big change. Implementing the GLX_EXT_swap_control_tear extension I haven t even looked at implementing this extension; I m hoping that once I ve gotten the functionality exposed over the X protocol that someone will at least help me figure out how to hook this up. Getting the Bits The interesting pieces here are the new kernel bits; the DRM library changes simply add new defines to existing APIs.

Daniel Pocock: Building existing autotools C/C++ projects on Android

I recently came across a requirement to build a project involving C++ code on Android. I've done this before with small C projects, for example, patching the JNI code in Lumicall. Google supplies a toolchain dubbed the Native Development Kit (NDK) for Android. Using the NDK typically involves using it's custom build system, usually duplicating all Makefile content into a file called Android.mk in each subdirectory. For large projects, that duplication doesn't sound like fun to implement and it definitely doesn't sound like fun keeping it up to date. reSIProcate: 39 Makefiles + dependencies reSIProcate is quite a large project involving multiple shared libraries, some useful demonstration apps, a SIP proxy (repro) and a TURN server (reTurn). There are 39 Makefiles and numerous dependencies. There are numerous test cases for each library and sub-project too. Sounds like a perfect fit for Android Alternative ways to proceed The Android NDK documentation pack describes a process for bypassing Android.mk files, invoking the toolchain (compiler, linker, etc) directly to build standalone apps with their own Makefile system I felt this was worth a go Two approaches are described. The first approach is to simply reference toolchain binaries (e.g. arm-linux-androideabi-gcc) using the well-known environment variables such as CC and LD. The documentation warns that in this case, support for the Standard Template Library (STL) is extremely limited. Nonetheless, I decided to try this first and found that I couldn't easily link binaries with the full STL feature set required by reSIProcate. I moved on to the second approach. The second approach involves calling the script $ NDK_HOME /build/tools/make-standalone-toolchain.sh to export a custom toolchain directory tree with full STL support. Using a little more disk space, but not so hard. Initially I had NDK version r7 on my system and I found that make-standalone-toolchain.sh was not working reliably. It appeared to complete successfully (exit status = 0) but in reality it was not copying the STL components to the custom directory. I upgraded to version r8e and the problem was resolved. Making it easy Finally, I wrote a little wrapper script that can do everything: build the custom toolchain, set up all those environment variables for the tools and then invoke the configure script with some of the special settings that are required for a cross compile. Coding problems To gain some confidence in this progress, I had already run the full reSIProcate build, including test cases, on all hosts in the Debian build farm, including some ARM hosts. This helped confirm that the code is correct for such CPUs as well as helping to identify some previously unknown errors on the more typical platforms. Nonetheless, some minor problems appeared with the build for Android:
  • Android's pthread capabilities are built into the C library so -lpthread must not be included on the linker command line. This is easily adapted using Makefile variables.
  • Although Android is Linux, some of the header locations are not quite the same. For example, references to sys/fcntl.h must be wrapped in a pre-processor conditional that uses just the path fcntl.h when __ANDROID__ is defined
After fixing a few of these minor issues, everything fell into place and the build was successful. High quality portable code The reSIProcate project has a strong tradition of providing cross-platform support. The code quality is quite good and well documented. Despite the project's large size and complicated build system, there was already a good chance of success. This may not be the case for every C/C++ project, especially those making extensive use of other shared libraries. Building the dependencies: OpenSSL on Android For the core features I wanted to use, I only needed one major dependency, the OpenSSL project. libcrypto and libssl already appear in Android. Nonetheless, their headers are not present in the NDK tree and therefore their API (and compile-time flags) can not be considered 100% stable across all handsets past and future. The solution is to build a standalone libcrypto and libssl and ship them with your own app. Fortunately this has been done before: I grabbed the code from Guardian Project's OpenSSL in github and found that it built immediately with my NDK version. To use it, simply make sure that CPPFLAGS and LDFLAGS include the necessary references to OpenSSL, as demonstrated in my script Test cases I found that make -i check would enable me to build all test cases on the build host. Naturally it would fail to run any of them as they are ARM binaries. Using -i in the make command line makes it possible to build them all anyway, ignoring the failures, for running them later. Copying the test* binaries from the .libs directories to the phone, I was able to run them immediately What next There are three further steps in this work:
  • Generalising the script to make binaries for all supported Android architectures (not just ARM)
  • Extending the script to copy the shared libraries into the build tree of a traditional Android project where they will be automatically scooped up when the apk file is assembled.
  • Writing the necessary JNI code to access the C/C++ code from Java

Jamie McClelland: Sharing screen with remote Linux user behind a firewall

Intentionally helping someone poke a hole through their firewall to allow any users on a remote machine access to their computer is generally a bad idea... unless they want you to help them with their computer. In that case, I find it really useful. Toward this end, I've setup a dedicated user account (jamie-share@chavez.mayfirst.org in this example). I granted myself shell access to this user and generated an ssh key pair:
ssh-keygen -t rsa
Then, as a convenience I move my public key into the home directory:
cp ~/.ssh/id_rsa.pub ~/
Next, I give the user I want to help ssh access to this shell account on an Internet connected computer (in this example: jamie-share@chavez.mayfirst.org). Either I share the password with them or if they have an ssh key or monkeysphere identity I use that. Then, I ask them to grant teh jamie-share user access to their user account by running (which downloads the key I just created):
mkdir -p ~/.ssh
ssh jamie-share@chavez.mayfirst.org "cat id_rsa.pub" >> ~/.ssh/authorized_keys
Next, ask them to install openssh-server on their local computer:
sudo apt-get install openssh-server screen
Next, they run the following command on their local computer:
ssh -R 2222:localhost:22 jamie-share@chavez.mayfirst.org
This command says: forward port 2222 on chavez.mayfirst.org to port 22 on your local computer. Lastly, I log in to jamie-share@chavez.mayfirst.org and run, replacing with their local username.
ssh <their-username>localhost -p 2222
You can do whatever you want now, but running screen is a good way to share a session so the person you are working with can see what you are doing as you do it. p.s. Thanks Ross for the tips!

Brett Parker: Replacing Planet Planet

I'm just replacing the code running planet alug from a (very old) Planet 2.0 to Venus, appears that the transition has mostly worked, but maybe it's time to think about what to replace that with too, it'd be nice, for instance, to use something shiny and django based so that some more people can add feeds (with out me having to go edit the file!).

Matthew Garrett: ARM and firmware specifications

Jon Masters, Chief ARM Architect at Red Hat, recently posted a description of his expectations for baseline arm64 servers. The quick summary is that systems should implement UEFI and ACPI, and any more traditional ARM boot mechanisms should be ignored. This is an interesting departure from the status quo in the ARM world, and it's worth thinking about the benefits and drawbacks of this approach.

It's very easy to build a generic kernel for most x86 systems, since the PC platform is fairly well defined even if not terribly well specified. Where system hardware does vary, it's almost always exposed on an enumerable bus (such as PCI or USB) which allows the OS to bind appropriate drivers. Things are different in the ARM world. Even once you're past the point of different SoC vendors requiring different kernel setup code and drivers, you still have to cope with the fact that system vendors can wire these SoCs up very differently. Hardware is often attached via GPIO lines without any means to enumerate them. The end result is that you've traditionally needed a different kernel for every ARM board. This is viable if you're selling the OS and hardware as a single product, but less viable if there's any desire to run a generic OS on the hardware.

The solution that's been adopted for this in the Linux world is called Device Tree. Device Tree actually has significant history, having been used as the device descriptor format in Open Firmware. Since there was already support for it in the Linux kernel, adapting it for use in ARM devices was straightforward. Device Tree aware devices can pass a descriptor blob to the kernel at startup[1], and devices without that knowledge can have a blob build into the kernel.

So, if this problem is already solved, why the push to move to UEFI and ACPI? This push didn't actually originate in the Linux world - Microsoft mandate that Windows RT devices implement UEFI and ACPI, and were they to launch a Windows ARM server product would probably carry that over. That makes sense for Microsoft, since recent versions of Windows have been x86 only and so have grown all kinds of support for ACPI and UEFI. Supporting Device Tree would require Microsoft to rewrite large parts of Windows, whereas mandating UEFI and ACPI allowed them to reuse most of their existing Windows boot and driver code. As a result, largely at Microsoft's behest, ACPI 5 has grown a range of additional features for describing things like GPIO pinouts and I2C connections. Whatever your weird device layout, you can probably express it via ACPI.

This argument works less well for Linux. Linux already supports Device Tree, whereas it currently doesn't support ACPI or UEFI on ARM[2]. Hardware vendors are already used to working with Device Tree. Moving to UEFI and ACPI has the potential to uncover a range of exciting new kernel issues and vendor bugs. It's not obviously an engineering win.

So how about users? There's an argument that since server vendors are now mostly shipping ACPI and UEFI systems, having ARM support these technologies makes it easier for customers to replace x86 systems with ARM systems. This really doesn't fly for ACPI, which is entirely invisible to the user. There are no standard ACPI entry points for system configuration, and the bits of ACPI that are generically useful (such as configuring system wakeup times) are already abstracted away to a standard interface by the kernel. It's somewhat more compelling for UEFI. UEFI supports a platform-independent bytecode language (EFI Byte Code, or EBC), which means that customers can write their own system management utilities, build them for EBC and then deploy them to their servers without caring about whether they're x86 or ARM. Want a bootloader that'll hit an internal HTTP server in order to determine which system image to deploy, and which works on both x86 and ARM? Straightforward.

Arnd Bergmann has a interesting counterargument. In a nutshell, ARM servers aren't currently aiming for the same market as x86 servers, and as a result customers are unlikely to gain any significant benefit from shared functionality between the two.

So if there's no real benefit to users, and if there's no benefit to kernel developers, what's the point? The main one that springs to mind is that there is a benefit to distributions. Moving to UEFI means that there's a standard mechanism for distributions to interact with the firmware and configure the bootloader. The traditional ARM approach has been for vendors to ship their own version of u-boot. If that's in flash then it's not much of a problem[3], but if it's on disk then you have to ship a range of different bootloaders and know which one to install (and let's not even talk about initial bootstrapping).

This seems like the most compelling argument. UEFI provides a genuine benefit for distributions, and long term it probably provides some benefit to customers. The question is whether that benefit is worth the flux. The same distribution benefit could be gained by simply mandating a minimum set of u-boot functionality, which would seem much more straightforward. The customer benefit is currently unclear.

In the end it'll probably be a market decision. If Red Hat produce an ARM product that has these requirements, and if Suse produce an ARM product that will work with u-boot and Device Tree, it'll be up to vendors to decide whether the additional work to support UEFI/ACPI is worth it in order to be able to sell to customers who want Red Hat. I expect that large vendors like HP and Dell will probably do it, but the smaller ones may not. The customer demand issue is also going to be unclear until we learn whether using UEFI is something that customers actually care about, rather than a theoretical benefit.

Overall, I'm on the fence as to whether a UEFI requirement is going to stick, and I suspect that the ACPI requirement is tilting at windmills. There's nothing stopping vendors from providing a Device Tree blob from UEFI, and I can't think of any benefits they gain from using ACPI instead. Vendor interest in the generic parts of the ACPI spec has been tepid even in the x86 world (the vast majority of ACPI spec updates come from Microsoft and Intel, not any of the system vendors), and I don't see that changing with the introduction of a range of ARM vendors who are already happy with Device Tree.

We'll see. Linux is going to need to gain the support for UEFI and ACPI on ARM in any case, since there's already hardware shipping in that configuration. But with ARM vendors still getting to grips with Device Tree, forcing them to go through another change in how they do things is going to be hard work. Red Hat may be successful in enforcing these requirements at the cost of some vendor unhappiness, or Red Hat may find that their product doesn't boot on most of the available hardware. It's an aggressive gamble, and while it'll be interesting to see how it plays out, I'm not that optimistic.

[1] The blob could be pulled from the firmware, but it's not uncommon for it to be built into u-boot instead. This does mean that you have a device-specific u-boot even if you have a generic kernel, but that's typically true anyway.
[2] Patches have been posted for ARM UEFI support. They're not mergeable in their current form, but they should be in the near future. ACPI support is in development.
[3] Although not all u-boots are created equal - some vendors ship versions that will only boot off FAT, some vendors ship versions that will only boot off ext2. Having to special case this stuff in your installer is a pain.

comment count unavailable comments

Riku Voipio: ACPI on ARM storm in teacup

A recent google+ post by Jon Masters caused some stormy and some less stormy responses. A lot of BIOS/UEFI/ACPI hate comes X86, where ACPI is used from everything from suspending devices to reading buttons and setting leds. So when X86 kernel suspends, it does magic calls to ACPI and prays that the firmware vendor did not screw it up. Now vendors do screw up, hence lots of cursing and ugly workarounds in the kernel follows. My Lenovo has a firwmare bug where the FN-buttons and Fan stops working if the laptop is attached on AC adapter for too long. The fan is probably a simple i2c device the kernel could control directly without jumping through ACPI hiding layer hoops. But the X86 people hold the view it is better to trust the firmware engineer to control devices instead of having the kernel folk to write device drivers to ... control devices! Now on ARM(64) the idea of using ACPI is to have none of that. Instead the idea is to use ACPI only to provide tables for enumerating what devices are available in the platform. Just like what device tree does. Now if this is the same as device tree, why bother? The main reason is to allow the distribution installer behave same on X86 / ARM / ARM64. This is crucial for distributions like fedora and RHEL where a cabal holds the point of view that X86 distribution development must not be constrained by ARM support. But it also important for everyone that the method of installing your favorite distribution to an ARM64 server is standard and works the same for any server from any vendor. Now while UEFI and ACPI are definitely not my preferred solutions, I can accept them as necessary evil for having a more standard platform.

Daniel Pocock: Winning at any cost

It's not every day that a student messing around with keystroke loggers comes to fame through slashdot. Nonetheless, systematically rigging an election and getting sentenced to 12 months in a dorm with bars has helped raise 22 year old Matthew Weaver's profile well above that of the average script kiddie. Now let's stop and reflect on poor Weaver's future. You may be thinking that with an exchange program like this on his academic record he won't be so popular with employers. Given that he was busted by campus security rather than the FBI he won't even attract the interest of those companies who hire ex-hackers. So where could he go? How is it done in Australia? Not too long ago, when I was a student myself, one of our prominent universities was subjected to a very similar scam. Four members of the Tin Tin for NUS ticket at La Trobe University were implicated in stuffing the ballot the old fashioned way. The incidents even share the characteristics of chronic stupidity: just as Weaver had been caught voting for himself 259 times from the same IP address in a campus computer lab, team Tin Tin had tried to hand their bag of manipulated postal votes directly to the deputy returning officer rather than discretely posting them through the internal mail. According to an official report by the Deputy Returning Officer, Karsten Haley, all four candidates were charged with Dishonest Conduct and Interfering with Ballot Papers. Unfortunately, the report notes that
La Trobe University SRC Electoral Regulations do not empower the Returning Officer or Deputy to enforce charges or disciplinary procedures and the charges were never faced by the accused.
Given the seriousness of the matter, Haley did not give up his attempts to hold them to account. He escalated it to the Dean of the college and then to the University Secretary. He reports that "their disinterest was extraordinary" and that nobody would involve the police. Young Labor suspended Just over a year later, in 1997, the ALP's youth division for the state of Victoria, Young Labor, was suspended after attempts to rig the ballot to elect the Young Labor leadership team. The guilty parties were never publicly named. Nobody was formally suspended or expelled and this simply left them with more time on their hands to invest their energy in other elections. The suspension of Victorian Young Labor remained in effect for a number of years. The specific allegations about the Young Labor ballot suggest that those people particularly keen to win had printed fake student cards and given them to stooges who would impersonate other Young Labor members who had not attended to vote in person. Where are they now? It is no co-incidence that these students were (and still are) members of Labor Unity, a powerful faction within Australia's ruling Labor Party, the ALP. Most political organisations would presumably express concern about these allegations. The ALP does things differently. One of the students who withdrew his nomination in La Trobe, Mr Larocca, subsequently became Mayor in the City of Moreland, one of the ALP's strongholds. Even more remarkably from an outsider's viewpoint, another of these figures, Stephen Donnelly, is currently employed as the Assistant State Secretary of the ALP in Victoria. Communications like this newsletter reveal that he is one of the key figures in the party's pre-selection process. He has recently been appointed to direct the ALP's 2013 federal election campaign for the state of Victoria. Another co-incidence On the same weekend that Weaver was in the news for his antics, Donnelly's latest employer, the ALP's Victorian branch, was conducting pre-selection ballots to choose candidates for the upcoming federal election. So it's no surprise that Monday's newspaper headlines report fresh allegations of voting irregularities. Sadly, I've seen some of Labor Unity's bad behavior first hand. About 10 years ago I was living in South Melbourne, which is in the federal electoral district of Melbourne Ports. A young female friend of mine, a member of the local Elwood branch of the ALP, had spent election day handing out brochures for an ALP candidate in a marginal seat rather than assisting the controversial local ALP candidate, Michael Danby. A few days later I was witness to an incident where Danby aggressively confronted this young woman and demanded to know why he hadn't seen her handing out his own leaflets on polling day. He stood within centimeters of her and was literally looking down on her as he demanded some kind of apology to sooth his bruised ego. She looked terrified and barely responded. Within moments one of his handlers approached and physically moved Danby away from this young woman, I dare to think where things would have gone otherwise. Eye for talent Remarkably, at the same time, the infamous Stephen Donnelly had started shadowing Danby in his movements about the district. Fresh out of university, his talents had been recognised by Danby and he was employed in Danby's office, enabling him to continue honing his skills on a full-time basis with a tax-payer funded salary. What a remarkable contrast to the story of Weaver. Can anybody imagine a US congressman collecting Weaver from the prison gates and deploying him to an office on Capitol Hill? The biggest bankruptcy in student history Around the same time, Donnelly's Student Unity, the student arm of Labor Unity were successful in taking over the student union of my own campus, the University of Melbourne. Not long after I graduated I heard that they had been accused of skimming off $1 million from catering providers and a high-risk $46 million property transaction that put the organisation into liquidation. Unlike Mr Weaver, who's scheme at Cal State barely got off the ground, none of those involved in the Melbourne University incident has faced criminal proceedings. One ALP figure, Andrew Landeryou, spent several months in Costa Rica while wanted for questioning in the Supreme Court. His wife has just been endorsed for a seat in the Senate with support from various Labor Unity figures including Danby. The Gillard questions In 1996, around the same time that Donnelly & Co. were romping around student unions learning the tricks of the political trade, a lawyer quietly departed from the firm Slater and Gorden after an internal investigation into a property transaction linked to a union slush fund. Like Donnelly, this lawyer's next move was to take employment in the office of a Labor Party MP. More recently she was backed by Labor Unity to become Prime Minister. The union slush fund remains under investigation, frustrated by the disappearance of documents. The $60 million heist Recently I blogged about Gillard and Abbott, leaders of the two main political parties in Australia, agreeing to take $60 million of taxpayer money to fund their parties' campaigns in the upcoming federal election, giving themselves an obscenely unfair unadvantage over all other contestants. Where would that money end up? In the case of the ALP, does it appear likely that figures like the Victorian ALP's federal campaign director, Mr Donnelly, would be involved in the expenditure? National shame With this background, it becomes easier to understand the quality (or lack of it) in Australia's national leadership. When you consider that the generation responsible for the La Trobe incident, the Young Labor suspension and the MUSU bankruptcy are now growing into positions of greater responsibility in the ALP it leaves me feeling the quality of leadership is only going to get a lot worse before it starts getting better. For example, the recent incident where coloured people were fed to the sharks has nothing to do with the worldwide refugee crisis and everything to do with maintaining the dumbed-down level of political discourse that Labor Unity thugs and their followers can cope with. Real issues like climate change and energy policy, for example, appear to be beyond the pay grade of Australia's political class Ranjini - coloured, indefinite detention It is startling that up to her own recent demise, Gillard herself had repeatedly begged the public to stop asking questions about her own past and remember that Labor politicians are innocent until proven guilty - yet she had a pregnant coloured woman thrown into a concentration camp on unfounded fears about "national security". No evidence has ever been presented that poor Ranjini committed a crime, but the houses bought with money from trade unions, transactions handled through Gillard's own office, seem to be as solid as bricks and mortar. If only poor Matthew Weaver had been an Australian, how much further would his star have risen? Update: please sign the petition at change.org asking La Trobe university to re-examine the report and refer it formally to the police. If you are concerned about the plight of poor Ranjini and other people subject to Australia's domestic rendition program, please take a moment to see Letters for Ranjini

21 July 2013

Benjamin Mako Hill: The Wikipedia Gender Gap Revisited

In a new paper, recently published in the open access journal PLOSONE, Aaron Shaw and I build on new research in survey methodology to describe a method for estimating bias in opt-in surveys of contributors to online communities. We use the technique to reevaluate the most widely cited estimate of the gender gap in Wikipedia. A series of studies have shown that Wikipedia s editor-base is overwhelmingly male. This extreme gender imbalance threatens to undermine Wikipedia s capacity to produce high quality information from a full range of perspectives. For example, many articles on topics of particular interest to women tend to be under-produced or of poor quality. Given the open and often anonymous nature of online communities, measuring contributor demographics is a challenge. Most demographic data on Wikipedia editors come from opt-in surveys where people respond to open, public invitations. Unfortunately, very few people answer these invitations. Results from opt-in surveys are unreliable because respondents are rarely representative of the community as a whole. The most widely-cited estimate from a large 2008 survey by the Wikimedia Foundation (WMF) and UN University in Maastrict (UNU-MERIT) suggested that only 13% of contributors were female. However, the very same survey suggested that less than 40% of Wikipedia s readers were female. We know, from several reliable sources, that Wikipedia s readership is evenly split by gender a sign of bias in the WMF/UNU-MERIT survey. In our paper, we combine data from a nationally representative survey of the US by the Pew Internet and American Life Project with the opt-in data from the 2008 WMF/UNU-MERIT survey to come up with revised estimates of the Wikipedia gender gap. The details of the estimation technique are in the paper, but the core steps are:
  1. We use the Pew dataset to provide baseline information about Wikipedia readers.
  2. We apply a statistical technique called propensity scoring to estimate the likelihood that a US adult Wikipedia reader would have volunteered to participate in the WMF/UNU-MERIT survey.
  3. We follow a process originally developed by Valliant and Dever to weight the WMF/UNU-MERIT survey to correct for estimated bias.
  4. We extend this weighting technique to Wikipedia editors in the WMF/UNU data to produce adjusted estimates of the demographics of their sample.
Using this method, we estimate that the proportion of female US adult editors was 27.5% higher than the original study reported (22.7%, versus 17.8%), and that the total proportion of female editors was 26.8% higher (16.1%, versus 12.7%). These findings are consistent with other work showing that opt-in surveys tend to undercount women. Overall, these results reinforce the basic substantive finding that women are vastly under-represented among Wikipedia editors. Beyond Wikipedia, our paper describes a method online communities can adopt to estimate contributor demographics using opt-in surveys, but that is more credible than relying entirely on opt-in data. Advertising-intelligence firms like ComScore and Quantcast provide demographic data on the readership of an enormous proportion of websites. With these sources, almost any community can use our method (and source code) to replicate a similar analysis by: (1) surveying a community s readers (or a random subset) with the same instrument used to survey contributors; (2) combining results for readers with reliable demographic data about the readership population from a credible source; (3) reweighting survey results using the method we describe. Although our new estimates will not help us us close the gender gap in Wikipedia or address its troubling implications, they give us a better picture of the problem. Additionally, our method offers an improved tool to build a clearer demographic picture of other online communities in general.

Wouter Verhelst: Two kings and three queens

Now that King Albert has abdicated and Prince (now King) Philippe has taken the oath of office, Belgium has two Kings and three Queens; yes, Albert has abdicated, but in Belgian tradition he retains the title of King, and his spouse, Queen Paola, retains the title of Queen. The third Queen, Fabiola, is the widow of late King Baudouin. As someone pointed out to me, that makes a full house. Does that make Philippe the King of Spades, or the King of Hearts? Or perhaps it's neither, and it's the King of Diamonds

Rog rio Brito: Beginning Ruby

These are some very rough/sketchy notes that I took while doing the exercises on the introductory track of the Ruby language at codeacademy. They were written from the point-of-view of an amateur programmer in C and Python (read: "me"). Since this is my first time with Ruby and I only have about 1 week of experience with it, these notes are likely to contain many mistakes, many simplifications, many omissions or simply gross errors. I would love to be informed of such facts so that I can fix them and not write code that is burdensome for others to maintain. Operators Numbers Basic I/O Strings Control flow If While For Loop Arrays Hashes Functions/methods Symbols Comments Lambda functions Classes Modules Mixins Conventions

20 July 2013

Luke Faraone: Joining the Debian FTPTeam

I'm pleased to say that I have joined the Debian FTPTeam as of the Friday before last. See Joerg Jaspert's announcement on debian-devel-announce.

The FTPTeam is responsible for maintaining the Debian software archive, and ensures that new software in Debian is high-quality and compliant with our policies.

As an "ftpassistant", I (along with Paul, Scott, Gergely, and others) will be helping to process the NEW queue, which is currently at a whopping 297 packages. Here's hoping we'll be able to get that number down over the coming weeks!

Eddy Petri&#537;or: (Not a) GNU Make quirk, or why logs should be provided

About two months ago I was writing about a quirk I found in GNU Make related to the $(patsubst ) function.

I have just tried this on my Debian Wheezy laptop which has make 3.81, but I wasn't able to reproduce the issue with the version from Debian (3.81-8.2).

The makefile looks like this:
PATH := ../some/prefixCPU12suf/include
CPUINC := $(patsubst ../some/prefix%,%,$(PATH))
CPU := $(patsubst %/include,%,$(CPUINC))

default:
@echo "PATH = $(PATH)"
@echo "CPUINC = $(CPUINC)"
@echo "CPU = $(CPU)"
And the result was correct:
0 eddy@heidi /tmp $ make
PATH = ../some/prefixCPU12/include
CPUINC = CPU12/include
CPU = CPU12
0 eddy@heidi /tmp $ make --version
GNU Make 3.81
Copyright (C) 2006 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.

This program built for x86_64-pc-linux-gnu
The worst part is that I know I tested this issue on 3.82 on Cygwin and on Linux with the 3.82 version and it failed, but I wasn't able to remember how I did it. I started searching through the directory where I knew there could be the test makefile, I wasn't able to find it, until I remembered what I was trying to achieve.

From a path like ../some/prefixCPU12suf/include I wanted to use % to remove the parts 'some/prefix' and 'suf/include' because in the directory ../CPU12 there were some files that needed to be processed.

The actual issue is that GNU Make's '%' is not analogous to shell's '*', so that means code like this does not work as I assumed anf the 'pref' part is not an anchor:


PATH := ../some/prefCPU12suf/include
CPUINC := $(patsubst pref%,%,$(PATH))
CPU := $(patsubst %suf/include,%,$(CPUINC))

default:
@echo "PATH = $(PATH)"
@echo "CPUINC = $(CPUINC)"
@echo "CPU = $(CPU)"
Which leads to these results, no matter the version:

0 eddy@heidi ~/usr/src/make/make-profiler/make-3.82 $ ./make -f /tmp/makefile
PATH = ../some/prefCPU12suf/include
CPUINC = ../some/prefCPU12suf/include
CPU = ../some/prefCPU12
0 eddy@heidi ~/usr/src/make/make-profiler/make-3.82 $ ./make --version
GNU Make 3.82
Built for x86_64-unknown-linux-gnu
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http: gnu.org="" gpl.html="" licenses="">
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
0 eddy@heidi ~/usr/src/make/make-profiler/make-3.82 $ make -f /tmp/makefile
PATH = ../some/prefCPU12suf/include
CPUINC = ../some/prefCPU12suf/include
CPU = ../some/prefCPU12
0 eddy@heidi ~/usr/src/make/make-profiler/make-3.82 $ make --version
GNU Make 3.81
Copyright (C) 2006 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.

This program built for x86_64-pc-linux-gnu</http:>
Not sure if this could be qualified as a true bug, or a if the way I expected is a nice to have feature, but, in any case, the behaviour is consistent, unlike my brain which failed to initially identify the inconsistency in my code:

0 eddy@heidi ~/usr/src/make/make-profiler/make-3.82 $ grep patsubst /tmp/makefile
CPUINC := $(patsubst pref%,%,$(PATH))
CPU := $(patsubst %suf,%,$(CPUINC))
0 eddy@heidi ~/usr/src/make/make-profiler/make-3.82 $ make -f /tmp/makefile
PATH = ../some/prefCPU12suf/include
CPUINC = ../some/prefCPU12suf/include
CPU = ../some/prefCPU12suf/include
Note that this behaviour of patsubst is asymtric to how subst works, as explained in the updated old post.

This took some extra effort to remember what was the actual issue, and shows why logs are important when reporting an issue, and why reporting issues as soon as they were encountered: because human brains are faulty. (Yes, yours, too!)

Eddy Petri&#537;or: GNU Make quirk

(Updated 2013-07-20: Correctly describe the issue and provide examples)

Make has several useful built-in functions, among which $(patsubst pattern,replacer,string). The pattern can contain the character % which is a wildcard meaning any string and that can be referred in the replacer parameter. Only the first occurrence of % is treated with this special meaning, just as the manual says.

What the manual doesn't say is that only the form %bla works, while the form bla% does NOT work GNU Make's '%' is not analogous to shell's '*', so you can't expect to remove substrings in the middle of a string without knowing the exact string until the part you want to remove.

This does not work if you want to obtain '../some/CPU12/include' in OTHERPATH:

PATH := ../some/prefCPU12/include
OTHERPATH := $(patsubst pref%,%,$(PATH))

default:
@echo "OTHERPATH = $(OTHERPATH)
The result will be:
OTHERPATH = ../some/prefCPU12/include
This bahaviour is somewhat asymetric to how subst behaves and how one would expect it to, considering '%' is explained as a glob pattern:

0 eddy@heidi /tmp $ cat makefile
PATH := ../some/prefCPU12/include
OTHERPATH := $(patsubst pref%,%,$(PATH))
SPATH := $(subst pref,,$(PATH))

default:
@echo "PATH = $(PATH)"
@echo "OTHERPATH = $(OTHERPATH)"
@echo "SPATH = $(SPATH)"
0 eddy@heidi /tmp $ make
PATH = ../some/prefCPU12/include
OTHERPATH = ../some/prefCPU12/include
SPATH = ../some/CPU12/include
Not sure what make developers would say about this, and I am not sure if having % work as a glob pattern and using it, instead of invoking sed would be better for performance, but sure I would like to have the option :-) .

Francesca Ciceri: #debcraft n.3: Debian swirl reverse appliqu on t-shirt

please, do try this at home the finished result This week's project is a quick t-shirt reconstruction with the reverse appliqu technique which basically means layering two different fabrics, sew them togheter, and then cut away part of the top one to reveal the one underneath.
Ready to start? What you will need the t-shirt before the reconstruction A guitar, Infinite Jest, The Illuminatus Trilogy and The Year of the Flood, while suggested for your personal enjoyment, are not strictly necessary for this project. Step 1 cutting the red fabric Cut a square of red fabric of the desired dimensions, I've used the short sleeve of a ruined red t-shirt.
On the wrong side of your patch, apply the fusible interfacing. Step 2 drawing the swirl on the interfacing Draw a swirl on the interfacing.
IMPORTANT: remember to draw the swirl as mirrored, because the interfacing is the wrong side of our patch.
So, if usually the Debian swirl has the back to the left and the front to the right - like a "(" - when you draw it on the interfacing it will need to have the back to the right and front to the side - like a ")".
I do realize that this explanation isn't really clear: just look at the picture above. Step 3 pin the patch to the t-shirt Turn your t-shirt inside out and pin the patch (with the interfacing up) in the place you want to put it: I decided to put mine on the lower left corner, approximately 10 cm above the bottom hem.
(In the picture is lower, I changed idea on the position just before sewing it)
Now you can sew it, following closely the drawn line.
Because I'm lazy, I decided to do it by hand, but if you're not afraid of sewing curved shapes, doing it with the sewing machine is definitely the right (and quick) way to do it. Step 4 cut the extra fabric Now, you can cut the extra fabric around your swirl. Step 5 revealing the swirl Turn your shirt right side out.
Make a snip through the knit in the fabric on the inside part of your stitches and cut off the extra fabric, to reveal the red swirl underneath.
Be careful to not cut the red fabric! Step 6 ending with a embroidered dashed line If you like it, you can end embroidering a dashed line (with running stitches) in a contrasting thread all around the swirl. And you're finished: now you can enjoy your new Debian t-shirt! :) For any questions feel free to contact me.

Michael Stapelberg: systemd 204 in Debian experimental

As of today, systemd 204 is available in Debian experimental. If you are interested in systemd, please install it and report any issues to the BTS merely reporting them on IRC is not sufficient, we need to have them in the BTS so we don t forget about them. In case you ve been using the unofficial 204 packages from http://people.debian.org/~biebl/systemd, please downgrade first to systemd 44-12 and udev 175-7.2, then upgrade to the versions in Debian experimental. This will ensure that during the upgrade all obsolete conffiles are upgraded properly. Thanks for helping with testing!

Charles Plessy: triggers in Dpkg and RPM: a comparaison.

Dpkg and RPM provide a mechanism called triggers. But are they doing the same thing? There are two different types of triggers for Dpkg: explicit and file. In the case of explicit triggers, the script of a package A will activate a package B. The file type is more frequently used. In that case, the creation, modification or deletion of a file in a path declared by a package B will activate the package B. The triggers are often used when a package A has to register an information to a package B. For instance, when a graphical application is installed, its package will add its description in the /usr/share/applications directory, which will trigger the desktop-file-utils package, which will do what is necessary to add a new menu entry in the window manager. RPM's triggers function differently. The installation, update or removal of a package A will activate a package B if this package B declares its interest for the package A in its SPEC file (with possible restrictions on package A's version). The documentation of RPM and Fedora gives a few examples of use, like the update of symbolic links when one mail server is replaced by another. Altogether, despite both mechanisms have the same name, they do not have the same goal. In the case of Dpkg, the triggers are especially useful to group similar actions. For instance, to update the list of manual pages once after installing a hundred of packages, instead of a hundred of times, after installing each package. They are also useful to transfer commands from a large number of packages to a central one, thus easing the evolution of the system. In the case of RPM, the triggers seem more specialised, as they always are about the relationship between a pair of packages.

Paul Tagliamonte: The Deprecation of Google Latitude

I m a bit disappointed Google is shutting down Google Latitude - I ve been an avid user of latitude since it came out - I always found it really quite neat. It was a bit creepy knowing the last few years of my exact location every day were sitting on a Google server, but at the same time, the API let me use this data for all sorts of personal projects. In fact, all my machines use my current latitude data in some form or another.
As a result, I m looking for a replacement. Anyone know of a good Free Software application I can install on my Android (Nexus 4) phone, and push real-time latitude and longitude data to my server? Any advice at all would be great.

19 July 2013

Justus Winter: A story about virtualization

I'm sure you all will be a bit disappointed (I know I am) that there are no ascii screenshots in this weeks report. But let me make it up to you by telling you a story. I was at the FOSDEM in the year 2012 and I went to a nice workshop on sunday afternoon in the Virtualization Devroom. Renzo Davoli was the host of this workshop and he started with a little introductory talk to get everyone to agree on a common terminology first. He began by asking the question, what virtualization meant in the most general way. He defined the ability to virtualize a resource as the ability to freely and transparently control how someone (say a process) is interacting with said resource. So for example if you LD_PRELOAD a library to impersonate fopen(3) and friends, you have virtualized the filesystem (for some small values of virtualized). He then went on to discuss various methods of virtualization that are commonly available on Linux, not only full system virtualization solutions, but also all kinds of methods allowing a more fine-grained control over what resources are virtualized. Of course every method had its strengths and its weaknesses. Having seen Samuels talk about all the awesome things Hurd can do for virtualization I approached him with a question in the free discussion part of his workshop. I asked whether he would agree that once a resource is moved from the kernel to the userspace, the problem of virtualizing that resource is almost trivially solved, and he agreed. So I said that this was awesome, because then the trivial and elegant solution for all his virtualization (and tracing) needs are micro kernel operating systems, and he also agreed to that but (as far as I can recall) he mentioned that there is none that would meet his needs (I believe he is a computer science professor and heavily relies on virtualization techniques for his lab and for teaching purposes). And even though a microkernel operating systems has a cost (message passing instead of function calls for example) it also has its merits such as scaling better from a development point of view. Also, many cool features can emerge just from the design. For example think about the container support in Linux and how painfully long it took to make all the resources namespace aware (with one of the most critical, the user namespace being the most difficult one). On Hurd you get the same functionality for free. If you are curious, read Samuels awesome slides. So what have I done this week? Next week I'm going to work on the two remaining sysvinit related issues. These are the only ones preventing me to come up with a clean patch series against the sysvinit package and I figured that it would be nice to propose such a series rather sooner than later so that I will have plenty of time to discuss any issues with the sysvinit maintainers. See you next week :)

Next.

Previous.