Weblate 1.6 has been released today. It comes with lot of improvements, especially adds voting for suggestions, improves import performance and brings more helpful admin interface. This is also first release with crowdfunded feature, but there are still some features to fund! Full list of changes for 1.6:

• Nicer error handling on registration.
• Browsing of changes.
• Fixed sorting of machine translation suggestions.
• Improved support for MyMemory machine translation.
• Added support for Amagama machine translation.
• Various optimizations on frequently used pages.
• Highlights searched phrase in search results.
• Support for automatic fixups while saving the message.
• Tracking of translation history and option to revert it.
• Added support for Google Translate API.
• Added support for managing SSH host keys.
• Various form validation improvements.
• Various quality checks improvements.
• Performance improvements for import.
• Added support for voting on suggestions.
• Cleanup of admin interface.

You can find more information about Weblate on it's website, the code is hosted on Github. If you are curious how it looks, you can try it out on demo server. You can login there with demo account using demo password or register your own user. Ready to run appliances will be soon available in SUSE Studio Gallery. Weblate is also being used https://l10n.cihar.com/ as official translating service for phpMyAdmin, Gammu, Weblate itself and others. If you are free software project which would like to use Weblate, I'm happy to help you with set up or even host Weblate for you.

Filed under: English Phpmyadmin Weblate 0 comments Flattr this!

Since 2 weeks I'm faced with some problems at a Xen hosting environment based on Debian wheezy. The issue is, that one specific domU is regularly failing network. The investigations showed, that no network packets from and to this system are seen on the the network interface in the domU and on the bridge interface in the dom0.
The funny part is, that the firewall on the dom0 now is logging packet rejects on the interface of the effected system and one interface of an other specific domU for the destination address. One pack and rejects on both interfaces ... really strange!
Shutting down the domU and recreating them doesn't help. Even if restarting other domUs, those are also effected by now. The only way to fix is to reboot the entire dom0. After several investigations I stumpled upon the following in the dmesg:

vif-xm116: Frag is bigger than frame.
vif-xm116: fatal error; disabling device
xenbr0: port 4(vif-xm116) entering disabled state

A quick search at the famouse search engine revealed, there seems something bigger. Even a Debian bug is open since February. Unfortunately this problem is still unfixed in wheezy and squeeze.
But ... there is hope in sight! Ian Campbell provided packages based on top of the latest kernel packages with a call for testing. Installing them seems to fix the issue in my environment. So we just have to hope this will get fixed soon, as this regression is really annoying. Thanks to all who make this happen in advance!

Present Extension Redirection Multi-buffered applications have always behaved poorly in the presence of the Composite extension:

• Updating involves multiple copies of the window contents, first from back buffer to composite redirect buffer, thence from the composite buffer to the screen back buffer, and then from the screen back buffer to the scanout buffer. The last copy is amenable to page flipping, but that would require the EGL buffer age extension so that the compositor could reduce the cost of presenting following frames by being able to track the contents of the back buffers.
• The application is not informed about when the actual screen presentation occurs.

Owen Taylor suggested that Present should offer a way to redirect operations to the compositing manager as a way to solve these problems. This posting is my attempt to make that idea a bit more concrete given the current Present design. Design Goals Here s a list of features I think we should try to provide:

1. Provide accurate information to applications about when presentation to the screen actually occurs. In particular, GLX applications using GLX_OML_sync_control should receive the correct information in terms of UST and MSC for each Swap Buffers request.
2. Ensure that applications still receive correct information as to the contents of their buffers, in particular we want to be able to implement EGL_EXT_buffer_age in a useful manner.
3. Avoid needing to un-redirect full-screen windows to get page flipping behavior.
4. Eliminate all extra copies. A windowed application may still perform one copy from back buffer to scanout buffer, but there shouldn t be any reason to copy contents to the composite redirection buffer or the compositing manager back buffer.

Simple Present Redirection With those goals in mind, here s what I see as the sequence of events for a simple windowed application doing a new full-window update without any translucency or window transformation in effect:

1. Application creates back buffer, draws new frame to it.
2. Application executes PresentRegion. In this example, the valid and update parameters are None , indicating that the full window should be redrawn.
3. The server captures the PresentRegion request and constructs a PresentRedirectNotify event containing sufficient information for the compositor to place that image correctly on the screen:
   • target window for the presentation
   • source pixmap containing the new image
   • idle fence to notify when the source pixmap is no longer in use.
   • serial number from the request.
   • target MSC for the presentation. This should probably just be the computed absolute MSC value, and not the original target/divisor/numerator values provided by the application.
4. The compositing manager receives this event and constructs a new PresentRegion request using the provided source pixmap, but this time targeting the root window, and constructing a valid region which clips the pixmap to the shape of the window on the screen. This request would use the original application s idle fence value so that when complete, the application would get notified. This request would need to also include the original target window and serial number so that a suitable PresentCompleteNotify event can be constructed and delivered when the final presentation is complete.
5. The server executes this new PresentRegion operation. When complete, it delivers PresentCompleteNotify events to both the compositing manager and the application.
6. Once the source pixmap is no longer in use (either the copy has completed, or the screen has flipped away from this pixmap), the server triggers the idle fence.

Multiple Application Redirection If multiple applications perform PresentRegion operations within the same frame, then the compositing manager will receive multiple PresentRedirectNotify events, and can simply construct multiple new PresentRegion requests. If these are all queued to the same global MSC, they will execute at the same frame boundary. No inter-operation dependency exists here. Complex Presentations Ok, so the simple case looks like it s pretty darn simple to implement, and satisfies the design goals nicely. Let s look at a couple of more complicated cases in common usage; the first is with translucency, the second with scaling application images down to thumbnails and the third with partial application window updates. Redirection with Translucency If the compositing manager discovers that a portion of the updated region overlays or is overlaid by something translucent (either another window, or drop shadows, or something else), then a composite image for that area must be constructed before the whole can be presented. Starting when the compositing manager receives the event, we have:

1. The compositing manager receives this event. Using the new pixmap, along with pixmaps for the other involved windows and graphical elements, the compositing manager constructs an updated view for the affected portion of the screen in a back buffer pixmap. Once complete, a PresentRegion operation that uses this back buffer pixmap is sent to the X server. Again, the original target window and serial number are also delivered to the server so that a suitable PresentCompleteNotify event can be delivered to the application.
2. The server executes this new PresentRegion operation; PresentCompleteNotify events are delivered, and idle fences triggered as appropriate.

Redirection with Transformation Transformation of the window contents means that we cannot always update a portion of the back buffer directly from the provided application pixmap as that will not contain the window border. Contents generated from a region that includes both application pixels and window border pixels must be sourced from a single pixmap containing both sets of pixels. One option that I ve discussed in the past to solve this would be to have the original application allocate the pixmap large enough to hold both the application contents and the surrounding window border. Have it draw the application contents at the correct offset within this pixmap, and then have the window manager contents drawn around that; either automatically by the X server, or even manually by the compositing manager. That would be mighty convenient for the compositing manager, but would require significant additional infrastructure throughout the X server and even harder the drawing system (OpenGL or some other system). There s another reason to want this though, and that s for sub-frame buffer scanout page swapping. The second option would be for the compositing manager to combine these images itself; there s a nice pixmap already containing the window manager image the composite redirect buffer. Taking the provided source pixmap and copying it directly to the target window will construct our composite image, just as if we had no Present redirection in place. This will cost an additional copy though, which we ve promised to avoid. Of course, as it s just for thumb-nailing or other visual effects, perhaps the compositing manager could perform this operation at a reduced frame rate, so that overall system performance didn t suffer. Retaining Access to the Application Buffer Above, I discussed having the idle fence from the redirected PresentRegion operation be sent along with the replacement PresentRegion operation. This ignores the fact that the composting manager may well need the contents of that application frame again in the future, when displaying changes for other applications that involve the same region of the screen. With the goal of making sure the idle fences are triggered as soon as possible so that applications can re-use or free the buffers quickly, let s think about when the triggering can occur.

1. Full-screen flipped applications. In this case, the application s idle fence can be triggered once the application provides a new frame and the X server has flipped to that new frame, or some other scanout buffer.
2. Windowed, copied applications. In this case, the application s idle fence can be triggered once the application provides a new frame to the compositing manager, and the X server doesn t have any presentations queued.

In both cases, we require that both the X server and the compositing manager be finished with the buffer before the application s idle fence should be triggered. One easy way to get this behavior is for the composting manager to create a new idle fence for its operations. When that is triggered, it would receive an X event and then trigger the applications idle fence as appropriate. This would add considerable latency to the application s idle fence a round trip through the compositing manager. The alternative would be to construct some additional protocol to make the applications idle fence dependent on the Present operation and some additional state provided by the compositing manager. Some experimentation here is warranted, but my experience with latency in this area is that it causes applications to end up allocating another back buffer as the idle notification arrives just after a buffer allocation request comes down to the rendering library. Definitely sub-optimal. An Aside on Media Stream Counters The GLX_OML_sync_control extension defines the Media Stream Counter (MSC) as a counter unique to the graphics subsystem which is incremented each time a vertical retrace occurs. That would be trivial if we had only one vertical retrace sequence in the world. However, we actually have N+1 such counters, one for each of the N active monitors in the system and a separate fake counter to be used when none of the other counters is available. In the current Present implementation, windows transition among the various Media Stream Counter domains as they move between the various monitors, and those monitors get turned on and off. As they move between these counter domains, Present tracks a global offset from their original domain. This offset ensures that the MSC value remains monotonically increasing as seen by each window. What it does not ensure is that all windows have comparable MSC sequence values; two windows on the same monitor may well have different MSC values for the same physical retrace event. And, even moving a window from one MSC domain to another and back won t make it return to the original MSC sequence values due to differences in refresh rates between the monitors. Internally, Present asserts that each CRTC in the system identifies a unique MSC domain, and it has a driver API which identifies which CRTC a particular window should be associated with. Once a particular CRTC has been identified for a window, client-relative MSC values and CRTC-relative MSC values can be exchanged using an offset between that CRTC MSC domain and the window MSC domain. The Intel driver assigns CRTCs to windows by picking the CRTC showing the greatest number of pixels for a particular window. When two CRTCs show the same number of pixels, the Intel driver picks the first in the list. Vblank Synchronization and Multiple Monitors Ok, so each window lives in a particular MSC domain, clocked by the MSC of the CRTC the driver has associated it with. In an un-composited world, this makes picking when to update the screen pretty simple; Present updates the screen when vblank happens in the CRTC associated with the window. In the composite redirected case, it s a bit harder all of the PresentRegion operations are going to target the root window, and yet we want updates for each window to be synchronized with the monitor containing that window. Of course, the root window belongs to a single MSC domain (likely the largest monitor, using the selection algorithm described above from the Intel driver). So, any PresentRegion requests will be timed relative to that single monitor. I think what is required here is for the PresentRegion extension to take an optional CRTC argument, which would then be used as the MSC domain instead of the window MSC domain. All of the timing arguments would be interpreted relative to that CRTC MSC domain. The PresentRedirectNotify event would then contain the relevant CRTC and the MSC value would be relative to that CRTC. A clever Compositing manager could then decompose a global PresentRegion operation into per-CRTC PresentRegion operations and ensure that multiple monitors were all synchronized correctly. We could take this even further and have the PresentRegion capable of passing a smaller CRTC-sized pixmap down to the kernel, effectively providing per-CRTC pixmaps with no visible explicit protocol Other Composite Users Ok, so the above discussion is clearly focused on getting the correct contents onto the screen with minimal copies along the way. However, what I ve ignored is how to deal with other applications, also using Composite at the same time. They re all going to expect that the composite redirect buffers will contain correct window contents at all times, and yet we ve just spent a bunch of time making that not be the case to avoid copying data into those buffers and instead copying directly to the compositing manager back or front buffers. Obviously the X server is aware of when this happens; the compositing manager will have selected for manual redirection on all top-level windows, while our other application will have only been able to select for automatic redirection. So, we ve got two pretty clear choices here:

1. Have the X server change how Present redirection works when some other application selects for Automatic redirection on a window. It would copy the source pixmap into the window buffer and then send (a modified?) PresentRedirectNotify event to the compositing manager.
2. Include a flag in the PresentRedirectNotify event that the composite redirect buffer needs to eventually get the contents of the source pixmap, and then expect the compositing manager to figure out what to do.

Development Plans As usual, I m going to pick the path of least resistance for all of the above options and see how things look; where the easy thing works, we can keep using it. Where the easy thing fails, I ll try something else. The changes required for this are pretty minimal. The PresentRegion request needs to gain a list of window/serial pairs that are also to be notified when the operation completes:

PRESENTNOTIFY  
    window: WINDOW
    serial: CARD32
     
 
    PresentRegion
    window: WINDOW
    pixmap: PIXMAP
    serial: CARD32
    valid-area: REGION or None
    update-area: REGION or None
    x-off, y-off: INT16
    idle-fence: FENCE
    target-crtc: CRTC or None
    target-msc: CARD64
    divisor: CARD64
    remainder: CARD64
    notifies: LISTofPRESENTNOTIFY
 
    Errors: Window, Pixmap, Match

The target-crtc parameter explicitly identifies a CRTC MSC domain. If None, then this request implicitly uses the window MSC domain. notifies provides a list of windows that will also receive PresentCompleteNotify events with the associated serial number when this PresentRegion operation completes.

 
    PresentRedirectNotify
    type: CARD8         XGE event type (35)
    extension: CARD8        Present extension request number
    length: CARD16          2
    evtype: CARD16          Present_RedirectNotify
    eventID: PRESENTEVENTID
    event-window: WINDOW
    window: WINDOW
    pixmap: PIXMAP
    serial: CARD32
    valid-area: REGION
    valid-rect: RECTANGLE
    update-area: REGION
    update-rect: RECTANGLE
    x-off, y-off: INT16
    target-crtc: CRTC
    target-msc: CARD64
    idle-fence: FENCE
    update-window: BOOL
 

The target-crtc identifies which CRTC MSC domain the target-msc value relates to. divisor and remainder have been removed as the target-msc value has been adjusted using the application values. If update-window is True, then the recipient of this event is instructed to provide reasonably up-to-date contents directly to the window by copying the contents of pixmap to the window manually. Beyond these two protocol changes, the compositing manager is expected to receive Sync events when the idle-fence is triggered and then manually perform a Sync operation to trigger the client s idle-fence when appropriate. I m planning to work on these changes, and then go re-work xcompmgr (or perhaps unagi, which certainly looks less messy) to incorporate support for Present redirection. The goal is to have something to demonstrate at Guadec, which doesn t seem impossible, aside from leaving on vacation in four days

The Scope of the Problem Security is inherently complex because of the large number of ways of circumventing it. For example Internet facing servers have been successfully attacked based on vulnerabilities in the OS, the server application, public key generation, DNS, SSL key certificates (and many other programs and algorithms in use), as well as the infrastructure and employees of all companies in the chain. When all those layers work reasonably well (not perfectly but well enough to not obviously be the weakest link) there are attacks on the end user systems that access the servers (such as the trojan horse programs used to attack PCs used for online banking). My Area of Interest The area of security that interests me is Linux software development. There are many related areas such as documentation and default configurations to make it easier for people to secure their systems (instead of insecure systems being the default option) which are all important. There are also many related fields such as ensuring that all people with relevant access are trustworthy. There are many interesting problems to solve in such areas most of which aren t a good match for my skills or just require more time than I have available. I sometimes write blog posts commenting on random security problems in other areas. Sometimes I hope to inspire people to new research, sometimes I hope to just inform users who can consider the issues when implementing solutions to security problems. Bugs In the software development side there are ongoing problems of bugs in code that weaken security. The fact that the main area of concern for people who are interested in securing systems is fixing bugs is an indication that the problem of software quality needs a lot of work at the moment. The other area that gets a reasonable amount of obvious work is in access control. Again it s an area that needs a lot of work, but the fact that we re not done with that is an indication of how far there is to go in generally improving computer security. Authenticating Software Releases There have been cases where source code repositories have been compromised to introduce trojan horse code, the ones I ve read about have been discovered reasonably quickly with little harm done but there could be some which weren t discovered. Of course it s likely that such attacks will be discovered because someone will have the original and the copies can be compared. Repositories of binaries are a bigger problem, it s not always possible to recompile a program and get a binary which checks out as being identical (larger programs often include the build time in the binary). Even for build processes which don t include such data it can be very difficult to determine the integrity of a build process. For example programs compiled with different versions of libraries, header files, or compilers will usually differ slightly. As most developers frequently change the versions of such software they will often be unable to verify their own binaries and any automated verification of such binaries will be impossible for anyone else. So if a developer s workstation was compromised without their knowledge it might be impossible for them to later check whether they released trojan binaries without just running the binaries in question and looking for undesired behavior. The problem of verifying past binaries is solvable for large software companies, Linux distributions, and all other organisations that have the resources to keep old versions of all binaries and libraries used to build software. For proprietary software companies the verification process would have to start with faith in the vendor of their OS and compiler doing the right thing. For Linux distributions and other organisations based on free software it would start by having the source to everything which can then be verified in theory although in practice verifying all source for compilers, the OS, and libraries would be a huge undertaking. Espionage There is a well documented history of military espionage, people who are sworn to secrecy have been subverted by money, blackmail, and by having political beliefs which don t agree with their government. The history of corporate espionage is less well documented but as corporations perform less stringent background checks than military organisations I think it s safe to assume that corporate espionage is much more common. Presumably any government organisation that can have any success at subverting employees of a foreign government can be much more successful in subverting programmers (either in companies such as Microsoft or in the FOSS community). One factor that makes it easier to launch such attacks is the global nature of software development. Government jobs that involve access to secret data have requirements about where the applicant was born and has lived, corporate jobs and volunteer positions in free software development don t have such requirements. The effort involved in subverting an existing employee of a software company or contributor to free software or the effort involved in getting an agent accepted in such a project would be quite small when compared to a nuclear weapons program. Therefore I think we should assume that every country which is capable of developing nuclear weapons (even North Korea) can do such things if they wish. Would the government of such a country want to subvert a major software project that is used by hundreds of millions of people? I can imagine ways that such things could benefit a government and while there would be costs for such actions (both in local politics and international relations) it seems most likely that some governments would consider it to be worth the risk and North Korea doesn t seem to have much to lose. Conclusion We would like every computer to be like a castle with a strong wall separating them from the bad things which can t be breached in ways that aren t obvious. But the way things are progressing with increasingly complex systems depending on more people and other systems it s becoming more like biology than engineering. We can think of important government systems as being comparable to the way people with compromised immune systems are isolated from any risk of catching a disease, the consequences of an infection are worse so greater isolation measures are required. For regular desktop PCs getting infected by a trojan is often regarded as being similar to getting a cold in winter. People just accept that their PC will be infected on occasion and don t bother making any serious effort to prevent it. After an infection is discovered the user (or their management for a corporate PC) tend not to be particularly worried about data loss in spite of some high profile data leaks from companies that do security work and the ongoing attacks against online banking and webcam spying on home PCs. I don t know what it will take for users to start taking security risks seriously. I think that a secure boot is a good step in the right direction, but it s a long way from being able to magically solve all security problems. I ve previously described some of the ways that secure boot won t save you [1]. The problems of subverting developers don t seem to be an immediate concern (although we should consider the possibility that it might be happening already without anyone noticing). The ongoing trend is that the value of computers in society is steadily increasing which therefore increases the rewards for criminals and spy agencies who can compromise them. Therefore it seems that we will definitely face the problems of subverted developers if we can adequately address the current technical problems related to flaws in software and inadequate access control. We just need to fix some of the problems which are exploited more easily to force the attackers to use the more difficult and expensive attacks. Note that it is a really good thing to make attacks more difficult, that decreases the number of organisations that are capable of attack even though it won t stop determined attackers. For end user systems the major problem seems to be related to running random programs from the Internet without a security model that adequately protects the system. Both Android and iOS make good efforts at protecting a system in the face of random hostile applications, but they have both been shown to fail in practice (it might be a good idea to have a phone for games that is separate from the phone used for phone calls etc). More research into OS security is needed to address this. But in the mean time users need to refrain from playing games and viewing porn on systems that are used for work, Internet banking, and other important things. While PCs are small and cheap enough that having separate PCs for important and unimportant tasks is practical it seems that most users don t regard the problems as being serious enough to be worth the effort.

• [1] http://etbe.coker.com.au/2011/12/28/secure-boot-protecting-against-root/

Implementing Vblank Synchronization in the Present Extension This is mostly a status update on how the Present extension is doing; the big news this week is that I ve finished implementing vblank synchronized blts and flips, and things seem to be working quite well. Vblank Synchronized Blts The goal here is to have the hardware executing the blt operation in such a way as to avoid any tearing artifacts. In current drivers, there are essentially two different ways to make this happen:

1. Insert a command into the ring which blocks execution until a suitable time immediately preceding the blt operation.
2. Queue the blt operation at vblank time so that it executes before the scanout starts.

Option 1. provides the fewest artifacts; if the hardware can blt faster than scanout, there shouldn t ever be anything untoward visible on the screen. However, it also blocks future command execution within the same context. For example, if two vblank synchronized blts are queued at the same time, its possible for the second blt to be delayed by yet another frame, causing both applications to run at half of the frame rate. Option 2. avoids blocking the hardware, allowing for ongoing operations to proceed on the hardware without waiting for the synchronized blt to complete. However, it can cause artifacts if the delay from the vblank event to the eventual execution of the blt command is too long. Queuing the blt right when it needs to execute means that we also have the opportunity to skip some blts; if the application presents two buffers within the same frame time, the blt of the first buffer can be skipped, saving memory bandwidth and time. Present uses Option 2, which may occasionally cause a tearing artifact, but avoids slowing down applications while allowing the X server to discard overlapping blt operations when possible. Queuing the Blt at Vblank There are several options for getting the blt queued and executed when the vblank occurs:

1. Queue the blt from the interrupt handler
2. Queue the blt from a kernel thread running in response to the interrupt
3. Send an event up to user space and have the X server construct the blt command.

These are listed in increasing maximum latency, but also in decreasing complexity. Option 1. is made more complicated as much of the work necessary to get a command queued to the hardware cannot be done from interrupt context. One can imagine having the desired command already present in the ring buffer and have the interrupt handler simply move the ring tail pointer value. Future operations to be queued before the vblank operation could then re-write the ring as necessary. A queued operation could also be adjusted by the X server as necessary to keep it correct across changes to the window system state. Option 2. is similar, but the kernel implementation should be quite a bit simpler as the queuing operation is done in process context and can use the existing driver infrastructure. For the X server, this is the same as Option 1, requiring that it construct a queued blt operation and deliver that to the kernel, and then revoke and re-queue if the X server state changed before the operation was completed. Option 3 is the simplest of all, requiring no changes within the kernel and few within the X server. The X server waits to receive a vblank notification event for the appropriate frame and then simply invokes existing mechanisms to construct and queue the blt operation to the kernel. Oddly, Present currently uses Option 3. If that proves to generate too many display artifacts, we can come back and change the code to try something more complicated. Flipping the Frame Buffer Taking advantage of the hardware s ability to quickly shift scanout from one chunk of memory to another is critical to providing efficient buffer presentation within the X server. It is slightly more complicated to implement than simply copying data to the current scanout buffer for a few reasons:

1. The presented pixmap is owned by the application, and so it shouldn t be used except when the presented window covers the whole screen. When the window gets reconfigured, we end up copying the window s pixmap to the regular screen pixmap.
2. The kernel flipping API is asynchronous, and doesn t provide any abort mechanism. This isn t usually much of an issue; we simply delay reporting the actual time of flip until the kernel sends the notification event to the X server. However, if the window is reconfigured or destroyed while the flip is still pending, cleaning that up must wait until the flip has finished.
3. The applications buffer remains busy until it is no longer being used for scanout; that means that applications will have to be aware of this and ensure that they don t deadlock waiting for the current scanout buffer to become idle before switching to a new scanout buffer.

Present is different from DRI2 in using application-allocated buffers for this operation. For DRI2, when flipping to a window buffer, that buffer becomes the screen pixmap the driver flips the new buffer object into the screen pixmap and releases the previous buffer object for other use. For Present, as the buffer is owned by the application, I figured it would be better to switch back to the real screen buffer when necessary. This also means that applications aren t left holding a handle to the frame buffer, which seems like it might be a nice feature. The hardest part of this work was dealing with client and server shutdown, dealing with objects getting deleted in random orders while other data structures retained references. (The kernel DRM drivers use the term page flipping to mean an atomic flip between one frame buffer and another, generally implemented by simply switching the address used for the scanout buffer. I d like to avoid using the word page in this context as we re not flipping memory pages individually, but rather a huge group of memory that forms an entire frame buffer. We could use plane flipping (as intel docs do), frame buffer flipping (but that s a mouthful), display flipping or almost anything but page flipping ). Overall DRI3000 Status At this point, the DRI3 extension is complete and the Present extension is largely complete, except for redirection for compositors. The few piglit tests for GLX_OML_sync_control all pass now, which is at least better than DRI2 does. I think I ve effectively replicated the essential features of DRI2 while offering room to implement a couple of new GL extensions:

• GLX_EXT_swap_control_tear. This will provide applications with the ability to avoid dropping frames when pushing the hardware just over the frame rate limit.
• EGL_EXT_buffer_age. (I assume we ll probably want a GLX version as well?) This will allow compositors to more efficiently perform partial updates in a flipping environment, and is enabled by having all of the buffer management within the GL library.

The code for this stuff has all been pushed to a number of repositories:

• git://people.freedesktop.org/~keithp/dri3proto.git master. DRI3 protocol specification and X server headers.
• git://people.freedesktop.org/~keithp/presentproto.git master. Present protocol specification and X server headers.
• git://people.freedesktop.org/~keithp/xcb/proto.git dri3. XCB protocol defines for both DRI3 and Present.
• git://people.freedesktop.org/~keithp/xcb/libxcb.git dri3. XCB library changes for file descriptor passing.
• git://people.freedesktop.org/~keithp/xserver.git dri3. X server with file descriptor passing, DRI3 and Present support.
• git://people.freedesktop.org/~keithp/mesa.git dri3 Mesa with DRI3/Present support for GLX.
• git://people.freedesktop.org/~keithp/drm.git dri3. DRM library with defines for async flipping.
• git://people.freedesktop.org/~keithp/xf86-video-intel.git dri3. Intel driver with DRI3, Present and async flipping support.
• git://git.kernel.org/pub/scm/linux/kernel/git/keithp/linux-2.6 dri3. Kernel with async flipping.

Asynchronous Display Flipping for Linux Using display (or plane or page) flipping is a nice optimization for screen updates as it avoids copying the application render buffer to the separate scanout buffer. However, in the current DRM interface, the flip operation is not performed immediately. Rather, the flip operation must be presented to the kernel before the vertical blank interval preceding the desired display time. If the flip operation is even slightly delayed, the new scanout image will not be visible until the following frame. With many applications pushing the performance of the graphics hardware right to the edge, it s not at all uncommon for an occasional frame to just miss and cause an ugly jerk on the screen. The GLX_EXT_swap_control_tear extension provides a way for applications to request that when the swap request is delivered too late, instead of delaying for the next frame, the contents should be presented as soon as possible, even if that might cause some tearing artifacts on the screen. This gives the application developer the choice between the two visual artifacts (tearing vs judder). A Linux interface for Async Flipping This part was pretty easy to implement. First, a new capability value, DRM_CAP_ASYNC_PAGE_FLIP was added so that applications could discover whether the driver offered this feature or not. Second, a new flag value, DRM_MODE_PAGE_FLIP_ASYNC was added to the DRM_IOCTL_MODE_PAGE_FLIP ioctl. To get the new flag down into the driver required adding the flag parameter to the page_flip crtc function. Async Display Flipping support in Intel Hardware Recent Intel display hardware has support for asynchronous flipping, either through the ring using the MI_DISPLAY_FLIP instruction or directly through the registers. There are some limitations however:

• The pitch of the current and new frame buffers must match
• Both current and new frame buffers must use X tiling

I ve also only implemented flipping on Sandybridge and Ivybridge hardware as that s what I ve got for testing. Using Async Display Flipping with Present I m using async display flipping when the kernel supports it in the current Present implementation. This offers the ability to short-circuit multiple presentations targeting the same scanout frame. I haven t exposed the ability to select this to applications over the Present protocol yet, but that s not a big change. Implementing the GLX_EXT_swap_control_tear extension I haven t even looked at implementing this extension; I m hoping that once I ve gotten the functionality exposed over the X protocol that someone will at least help me figure out how to hook this up. Getting the Bits The interesting pieces here are the new kernel bits; the DRM library changes simply add new defines to existing APIs.

• Kernel bits are in my kernel.org repository git://git.kernel.org/pub/scm/linux/kernel/git/keithp/linux-2.6 dri3
• DRM library bits are in my freedesktop.org drm repository git://people.freedesktop.org/home/keithp/drm.git dri3

These are some very rough/sketchy notes that I took while doing the exercises on the introductory track of the Ruby language at codeacademy. They were written from the point-of-view of an amateur programmer in C and Python (read: "me"). Since this is my first time with Ruby and I only have about 1 week of experience with it, these notes are likely to contain many mistakes, many simplifications, many omissions or simply gross errors. I would love to be informed of such facts so that I can fix them and not write code that is burdensome for others to maintain. Operators

• Math operators: +, -, *, /, %, ** (just like Python)
• Relational operators: ==, <, >, <=, >= (just like other languages)
• Boolean operators: &&, , ! (just like C/C++/Perl)
• The combined comparison operator ("spaceship operator", like Perl): <=> (follows the spirit of what strcmp in C does)
• Conditional assignment operator ( =) only assigns a value to a variable if that variable is bound to nil.

Numbers

• You can write number literals embedding an underscore (_) for readability purposes.

Basic I/O

• Output: print and puts. Both print strings to stdout, but puts also appends a new line to the end.
• Input: gets accepts an input from stdin and returns a string.
• To convert a string foo to an integer, use the Integer(foo) constructor. It throws an exception if it cannot convert the value.

Strings

• Strings have at least some methods: .length, .reverse, .upcase, .downcase, .capitalize, .chomp (like Perl, removes the trailing endline)
• .upcase seems to have problems with non-ASCII characters, like .
• Interpolation is performed by putting the name name of the variable inside a string and delimited as # name at the place we want the interpolation to occur.
• Methods like .capitalize don't change the string (they return a new string capitalized). To change the string, you call the method .capitalize! (with a ! at the end of the name of the method).
• Other useful methods: .gsub for globally substituting a regular expression (and its variant .gsub! for inplace modification), and .include? to verify if the first string includes another.
• Another method: .split(" "), to split on the string specified as argument.

Control flow If

• The syntax of a if is:

    if expression1
      block1
    elsif expression2
      block2
    else
      block3
    end
  

  note that it ends with the word end, there are no need for parentheses, no braces, no colons, and the keyword for "else if" is elsif (different from Python's elif).
• The "if" can also be started with the keyword unless instead, which checks if the expression is false rather than true (as is the case of if).

While

• The syntax of the while is similar to other languages:

    while condition
      block
    end
  

• A variation of the while is the until, with the same syntax, except for the until in place of the while, and that it iterates when the condition is false, instead of true.

For

• The simple case of iterating over a range of numbers is:

    for i in 1...10
      block
    end
  

  Like ranges in Python, the 10 is not included (the variable i iterates from 1 to 9). For an inclusive range, we use two dots (1..10), instead of three dots (1...10).

Loop

• To iterate a block, we can use the loop keyword, as in:

    loop   block   # with braces
  

  or as in

    loop do # note the pair do/end in place of braces
      block
    end
  

• It is useful to use break when using loop to break out of the infinite iterations.
• Similarly, we can use next as in other languages (the same as continue in C or C++).

Arrays

• The equivalent to Python's lists are arrays and they are declared in the same fashion:

    my_array = [1, 2, 3, 4, 5]
  

• Arrays in Ruby are 0-based.
• Every array has an iterator called the .each method. The basic syntax is:

    object.each    item  block   # do something
  

  which can be written as:

    object.each do  item  block end
  

  where the name between is bound to (a copy of?) each element of object in turn. The equivalent in Python would be:

    for item in object:
        block
  

• Arrays can be sorted with the .sort method (or its sibling .sort!). When passing a parameter, it is useful to use the spaceship operator (<=>) (like Perl) to specify a custom order (similar in spirit to what we do with Python's key argument to the sort method of a list). Example:

     my_array.sort!  a, b  a <=> b  # Ascending order
     my_array.sort!  a, b  b <=> a  # Descending order
  

• The .push method appends an element to an Array (similar to Python's .append).
• The empty Array is denoted by [], like an empty list in Python.
• An iterator (like (1..10)) can be converted to an array with the .to_a method.

Hashes

• A literal hash is defined by the syntax:

    my_hash =  
        "key1" => "abc",
        "key2" => 2,
        "key3" => false
     
  

  Quite similar to the version in Python:

    my_hash =  
        "key1": "abc",
        "key2": 2,
        "key3": False
     
  

  In fact, in Ruby 1.9, you can write, instead:

    my_hash =  
        key1: "abc",
        key2: 2,
        key3: false
     
  

  which is very similar to Python or JSON.
• The empty hash, OTOH, is defined by calling the constructor .new of the class Hash:

    my_hash = Hash.new
  

  The syntax used by Python can also be used: my_hash =
• Iterating over a hash is done the same way as done for an array, but the .each method gives both the key and the value, as in:

    my_hash.each do  k, v 
      puts "# k : # v "
    end
  

• To iterate only on keys, we use the method .each_key. To iterate only on values, we use the method .each_value.
• A hash can be sorted with the .sort_by method (to sort by keys, values or any expression involving them) and since it can be ordered, we can reverse (in-place) the order of its elements with the .reverse! method. This makes Ruby's hashes to "taste" like Python's OrderedDict's.
• When one initializes a hash, one can specify a default value to the .new method (e.g., foo = Hash.new(0)), which makes it convenient to do the equivalent of Python's defaultdict or Counter.
• You can select part of a hash by using the .select method and giving it a block. For example:

    good_movies = movie_ratings.select k, v  v > 3 
  

Functions/methods

• Functions in Ruby are defined with the word def (as in Python) and end with the word end, as other composite constructs. Example:

    def square(n)
      return n*n
    end
  

• Parentheses are optional in Ruby, unless they create ambiguity. So, to call the function above, one can use any of the following:

    square(2)
    square 2
  

• A function can take a block as a parameter (and this is what .each does, for instance).
• Functions can take a variable number of elements with the splash operator (denoted by *), which puts the extra parameters in an Array (like Python).
• Functions can have default parameters, which are set in the "prototype", just like in Python or C++.
• If the return keyword is omitted from the method, Ruby uses an implicit return, where the value of the last executed expression is returned to the caller of the method.

Symbols

• Any instances of a given symbol share the same ID. This is due to the fact that they are immutable.
• Symbols start with a colon (:) and they can contain any character that is valid for a variable name.
• It is possible to convert between a symbol and a string with the methods .to_s and .to_sym.

Comments

• Single line comments use the hash symbol (#).
• Ruby also has multiline comments, which are delimited by a line beginning with =begin and a line beginning with =end.

Lambda functions

• A lambda function in Ruby is defined, as expected, as, for example:

    foo = lambda    x  x.to_sym  
  

• The basic differences between a lambda and a proc is that:
  • A lambda checks the number of elements that it receives, while a proc doesn't and assigns nil to parameters that it didn't receive (a proc seems to ignore extra parameter that it wasn't expecting).
  • When a lambda returns, the control goes back to the function that called the lambda, while when a proc returns, the control is returned also from the function that called the proc.

Classes

• "Everything is an object in Ruby." (At least one exception are blocks).
• To test if an object is of a given class, you use the method .is_a? on the object and the type as argument (e.g., foo.is_a? Integer)
• The "null" value in Ruby is nil. It is equivalent to None in Python.
• The only "falsy" values in Ruby are false and nil. The others are "truthy", differently from C/C++/Perl/Python.
• For a given object, the method .object_id returns what is the "address" of that object.
• Classes in Ruby are created with the syntax

    class ClassExample
        # body
    end
  

  (That is for a class that has is not derived from other regular classes).
• Constructors in Ruby are called initialize. An instance of a class is created by calling the method .new on the class name (and passing parameters as needed for the initialize method).
• Instance variables are named with 1 at sign (@) as a prefix. Class variables are named with 2 at signs (@@) as a prefix. Global variables are named with a dollar sign ($) as a prefix.
• Class methods are declared inside a class prefixed with self. and they are called with the class name suffixed with the name of the method. For example, for a class MyClass, with a class method class_method, we call it as MyClass.class_method.
• When overriding a method in a subclass, one can call the method from the parent class with the keyword super, as in super(arg1, arg2).
• Ruby doesn't support multiple inheritance. In place of multiple inheritance, Ruby offers mixins.
• To access one attribute, say @my_attrib, in Ruby classes, one usually creates two methods to manipulate it: a method with the name my_attrib, that returns the value of @my_attrib (a getter) and a method with the name my_attrib= that accepts a parameter to set @my_attrib to (a setter). There is syntatic sugar for creating both getters or setters. For a getter, you declare attr_reader :my_attrib (the member name as a symbol). For a setter, you declare attr_writer :my_attrib. If you need both a getter and a setter, then you can abbreviate them with attr_accessor.
• All methods, by default, are public in Ruby. To explicitly mark something as public, you use the keyword public. To mark something as private, you use the keyword private.

Modules

• You can import a module with the keyword require (as in Perl) and the name of the module in quotes.
• One example of a module is the benchmark module, which provides the Benchmark class, which has the method .realtime, which takes a block and reports how much time it used.
• To create a module, use a syntax similar to that of a class, but substituting the keyword class with module.
• The scope resolution operator in Ruby is the double colon (::), like in Perl.
• To create a method inside a module, you have to prefix it by the name of the module with a dot or with self., if you want to create a method to be called outside the module. The same also holds for creating class methods (as opposed to instance methods): they can be simply named self.foo to get a class method called foo.
• While some modules are already "preloaded" by the interpreter (e.g., Math), others need to be explicitly loaded. We do this by using require 'foo' to load the module (in the file foo?).

Mixins

• A mixin is the use of a module to "enrich" a class, adding "instance" methods of the module to the class. It is, in a sense, a way to have some of multiple inheritance in Ruby (without its headaches?)
• Using the word include Foo imports the items of the module Foo in the current namespace for instances of a class.
• Using the word extend Foo imports the items of the module Foo in the current namespace as class methods/attributes.

Conventions

• Local variables are named in lower case, with parts separated by underscores, as in my_name.
• Methods that return boolean values have a question mark (?) at the end of their name.
• Methods that have destructive side-effects (like changing their input) have an exclamation mark (!) at the end of their name.
• Code is indented by 2 spaces.
• Like other languages, constants are written in all caps, with words separated by underscores. For example, MAX_VALUE.
• Global variables are prefixed with a dollar sign (e.g., $Foo).
• Classes are named in camel case (no initial lowercase).
• The "poetic mode" in Ruby recommends one to omit the parentheses on function calls (whenever unambiguous) and, in essence, to write as little "bureaucracy" (like the explicit return keyword in methods).

Dpkg and RPM provide a mechanism called triggers. But are they doing the same thing? There are two different types of triggers for Dpkg: explicit and file. In the case of explicit triggers, the script of a package A will activate a package B. The file type is more frequently used. In that case, the creation, modification or deletion of a file in a path declared by a package B will activate the package B. The triggers are often used when a package A has to register an information to a package B. For instance, when a graphical application is installed, its package will add its description in the /usr/share/applications directory, which will trigger the desktop-file-utils package, which will do what is necessary to add a new menu entry in the window manager. RPM's triggers function differently. The installation, update or removal of a package A will activate a package B if this package B declares its interest for the package A in its SPEC file (with possible restrictions on package A's version). The documentation of RPM and Fedora gives a few examples of use, like the update of symbolic links when one mail server is replaced by another. Altogether, despite both mechanisms have the same name, they do not have the same goal. In the case of Dpkg, the triggers are especially useful to group similar actions. For instance, to update the list of manual pages once after installing a hundred of packages, instead of a hundred of times, after installing each package. They are also useful to transfer commands from a large number of packages to a central one, thus easing the evolution of the system. In the case of RPM, the triggers seem more specialised, as they always are about the relationship between a pair of packages.

Yesterday, I've spent some time trying to improve speed of translation import in Weblate. It got annoyingly slow while adding new features and I felt there has to be some area for improvements even with all the features. I was testing on development server with debug stuff enabled (what slows down things quite a lot already) and first import took about 30 seconds. Quick look revealed that for linked repositories, the link lookup happens too often, so some caching could be used. This has saved about 3 seconds... With offloaded indexing (what is anyway only useful setup for production use), every reindex request was checking whether there is already one before creating new one. In most cases there is none before, so it's simply faster to try to create one and ignore error in case it is already there. We're now 5 seconds faster. Another big resource consumer are quality checks. They provide useful feedback to translators, but it means that quite a lot of code needs to be executed for every single string. First thing I've realized is that unit check state is recalculated everytime regardless the fact whether there was any change, what was easy to fix. The other speed issue is in consistency check - it does too complex SQL query, which usually does not match anything. Converting this to much simpler one (which will in most cases still have no results) improves performance for usual path while having minimal impact for the real checking. Another 4 seconds down on my test import. Last thing which I changed is way how per translation statistics are updated. In past this required quite a lot of queries, but yesterday I've realized this can be much reduced. During test suite run on PostgreSQL, I saw that it's not that easy - PostgreSQL does not allow to sum boolean fields, but fortunately this can be fixed as well. This has saved about 2 seconds. In the end the tested import took 16 seconds instead of original 30 and issued 2500 SQL queries instead of 5000 at the beginning. I hope you will enjoy regained speed of imports :-).

Filed under: English Weblate 0 comments Flattr this!

Michael Langguth and Scalaris AG asked me to publish the mksh/Win32 Beta 14 source and binary archive, and it is with joy I m doing this. Checksums and Hashes

• RMD160 (ports/mksh-w32-beta14.zip) = 0dc8ef6e95592bd132f701ca77c4e0a3afe46f24
• TIGER (ports/mksh-w32-beta14.zip) = 966e548f9e9c1d5b137ae3ec48e60db4a57c9a0ed15720fb
• 1181543005 517402 /MirOS/dist/mir/mksh/ports/mksh-w32-beta14.zip
• MD5 (ports/mksh-w32-beta14.zip) = b57367b0710bf76a972b493562e2b6b5

Just a few words on it (more in the README.1st file included): this is a port of The MirBSD Korn Shell R39 to the native WinAPI; it s not quite got the full Unix feel (especially as it targets the Weihenstephan unxutils instead of a full Interix or Cygwin environment) but doesn t need a full POSIX emulation layer either. It s intended to replace MKS ksh and the MKS Toolkit. Source for the compatibility library is also included under The MirOS Licence; we aim at publishing it as OSI Certified Open Source Software like mksh itself. (There is a situation with dlmalloc/nedmalloc being resolved, and the icon is derived from the BSD d mon which is a protected unregistered trademark, but we re not Mozilla and allow distro packages to keep using it ) Rebasing it on a newer mksh(1) followed by (partial) integration into the main source code is a goal. Have fun trying it out and hacking on it. It s currently built with -DMKSH_NOPROSPECTOFWORK (so coprocesses and a few other minor things won t work), but a SIGCHLD emulation is being worked on but if you want to help out, I m sure it s welcome, just come on IRC or post on the mailing list, and I ll forward things to Michael as needed. Reports on testing with other toolchain and OS versions are also welcome.

There are three main ways to distribute a command-line daemon for OS X:

1. Distributing source code and instructions on how to compile it.
2. Using a third-party package manager, like Homebrew.
3. Providing an installer package.

Homebrew Homebrew is a popular package management system. It works like the BSDs ports collections by downloading, compiling and installing the requested software, while also installing any required dependencies automatically. Creating a new package is quite easy and there are a lot of examples available. However, there are some limitations:

• You don t really build a package but execute a recipe to locally install the software.
• You need to install development tools, either a whole Xcode installation¹ or the command line version.
• If you need to execute some steps as root, you will have to explain them to the user for her to execute by hand. This includes the creation of a system user or the installation of a daemon through launchd.

If you can t live with those limitations, you may need to build an installer package.

Building an installer package OS X comes with a graphical and a command-line installer. The graphical one is run by opening a package from the Finder. Then, the user experiences some familiar wizard allowing her to install the software in a matter of a few seconds. The documentation around how to build such a package is somewhat clumsy. You may find outdated information or pieces of information that only apply to projects using Xcode. I will try to provide here accurate bits in the following context:

1. You are packaging a pure command-line tool.
2. You are using Autoconf and Automake as a build system.
3. You want to support several architectures.
4. You want to support older versions of OS X.

Creating a package Building such a package was previously done with some graphical tool called PackageMaker. This tool is not available any more and developers are asked to switch to pkgbuild and productbuild. There is a very neat Stackoverflow article on how those tools work. A package is built in two steps:

1. Build component packages.
2. Combine them into a product archive.

A component package contains a set of files and a set of scripts to execute at various steps of the installation. You can have several component packages, for example a package for the daemon and a package for the client. They are built with pkgbuild. A product archive contains the previously created component packages as well as a file describing various options of the installer (required and optional components, license, welcome text, ). To create a component package, we need to install the needed files in some directory:

$ ./configure --prefix=/usr --sysconfdir=/private/etc
$ make
$ make install DESTDIR=$PWD/osx-pkg

We will now put the content of osx-pkg into a component package with pkgbuild:

$ mkdir pkg1
$ pkgbuild --root osx-pkg \
>    --identifier org.someid.daemon \
>    --version 0.47 \
>    --ownership recommended
>    pkg1/output.pkg
pkgbuild: Inferring bundle components from contents of osx-pkg
pkgbuild: Wrote package to output.pkg

You need to be careful with the identifier. It needs to be unique and identify your software as well as the specific component. Then, you need to create an XML file describing the installer. Let s call it distribution.xml:

<?xml version="1.0" encoding="utf-8" standalone="no"?>
<installer-gui-script minSpecVersion="1">
    <title>Some daemon</title>
    <organization>org.someid</organization>
    <domains enable_localSystem="true"/>
    <options customize="never" require-scripts="true" rootVolumeOnly="true" />
    <!-- Define documents displayed at various steps -->
    <welcome    file="welcome.html"    mime-type="text/html" />
    <license    file="license.html"    mime-type="text/html" />
    <conclusion file="conclusion.html" mime-type="text/html" />
    <!-- List all component packages -->
    <pkg-ref id="org.someid.daemon"
             version="0"
             auth="root">output.pkg</pkg-ref>
    <!-- List them again here. They can now be organized
         as a hierarchy if you want. -->
    <choices-outline>
        <line choice="org.someid.daemon"/>
    </choices-outline>
    <!-- Define each choice above -->
    <choice
        id="org.someid.daemon"
        visible="false"
        title="some daemon"
        description="The daemon"
        start_selected="true">
      <pkg-ref id="org.someid.daemon"/>
    </choice>
</installer-gui-script>

Fortunately, this file is documented on Apple Developer Library. Since we only describe one package, we pass customize="never" as an option to skip the choice part. You can however remove this attribute when you have several component packages. The attribute rootVolumeOnly="true" explains that this daemon can only be installed system-wide. It is marked as deprecated but still works. Its replacement (domains tag) displays an unusual and buggy pane which Apple doesn t use in any of its packages. You need to put the HTML² documents into a resources directory. It is also possible to choose the background by specifying a <background/> tag. The following command will generate the product archive:

$ productbuild --distribution distribution.xml \
>              --resources resources \
>              --package-path pkg1 \
>              --version 0.47 \
>              ../final.pkg
productbuild: Wrote product to ../final.pkg

Scripts The installer can execute some scripts during installation. For example, suppose we would like to register our daemon with launchd for it to be run at system start. You first need to write some org.someid.plist file and ensure it will get installed in /Library/LaunchDaemons:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
  <key>Label</key>
  <string>org.someid</string>
  <key>ProgramArguments</key>
  <array>
    <string>/usr/sbin/mydaemon</string>
    <string>-d</string>
  </array>
  <key>RunAtLoad</key><true/>
  <key>KeepAlive</key><true/>
</dict>
</plist>

Then create a scripts/postinstall file with the following content:

#!/bin/sh
set -e
/bin/launchctl load "/Library/LaunchDaemons/org.someid.plist"

You also need a scripts/preinstall file which will allow your program to be smoothly upgraded:

#!/bin/bash
set -e
if /bin/launchctl list "org.someid" &> /dev/null; then
    /bin/launchctl unload "/Library/LaunchDaemons/org.someid.plist"
fi

Ensure that those scripts are executable and add --scripts scripts to pkgbuild invocation. If you need to add a system user, use dscl.

Dependencies There is no dependency management built into OS X installer. Therefore, you should ensure that everything is present in the package or in the base system. For example, lldpd relies on libevent, an event notification library. This library is not provided with OS X. When it is not found, the build system will use an embedded copy. However, if you have installed libevent with Homebrew, you will get binaries linked to this local libevent installation. Your package won t work on another host. The output of otool -L can detect unwanted dependencies:

$ otool -L build/usr/sbin/lldpd
build/usr/sbin/lldpd:
        /usr/lib/libresolv.9.dylib (compatibility version 1.0.0, current version 1.0.0)
        /System/Library/Frameworks/IOKit.framework/Versions/A/IOKit (compatibility version 1.0.0, current version 275.0.0)
        /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 744.19.0)
        /System/Library/Frameworks/Foundation.framework/Versions/C/Foundation (compatibility version 300.0.0, current version 945.18.0)
        /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 169.3.0)

Compilation for older versions of OS X The package as built above may only work on the same version of OS X that you use to compile. If you want a package that will work on OS X 10.6, you will need to download the appropriate SDK³. Then, you need to tell the compiler the target version of OS X and the SDK location. You can do this by setting CFLAGS and LDFLAGS (and CXXFLAGS if you use C++):

$ SDK=/Developer/SDKs/MacOSX10.6.sdk
$ ./configure --prefix=/usr --sysconfdir=/private/etc \
>         CFLAGS="-mmacosx-version-min=10.6 -isysroot $SDK" \
>         LDFLAGS="-mmacosx-version-min=10.6 -isysroot $SDK"

The -mmacosx-version-min flag will be used by various macros in the headers to mark some functions as available or not, depending on the version of OS X you target. It will also be used by the linker. The -isysroot will tell the location of the SDK. Headers and libraries will be looked up in this location first.

Universal binaries Mac OS X 10.6 was available for both IA-32 and x86-64 architectures. If you want to support both installations with a single package, you need to build universal binaries. The Mach object file format, used by OS X, allows several versions of the executable in the same file. The operating system will select the most appropriate one. An easy way to generate such files is to pass -arch i386 -arch x86_64 to the compiler:

$ ./configure --prefix=/usr --sysconfdir=/private/etc \
>     CC="gcc -arch i386 -arch x86_64" \
>     CPP="gcc -E"

However, this is a dangerous option. Suppose that your ./configure script tries to determine some architecture-dependent parameter, like the size of an integer (with AC_CHECK_SIZEOF); it will compute it for the host architecture. The binary generated for the other architecture will therefore use a wrong value and may crash at some point. The correct way to generate an universal binary is to execute two separate compilations and to build the universal binary with lipo:

$ for arch in i386 x86_64; do
>   mkdir $arch ; cd $arch
>   ../configure --prefix=/usr --sysconfdir=/private/etc \
>       CC="gcc -arch $ARCH" \
>       CPP="gcc -E"
>   make
>   make install DESTDIR=$PWD/../target-$ARCH
>   cd ..
> done
[...]
$ lipo -create -output daemon i386/usr/sbin/daemon x86_64/usr/sbin/daemon

Since lipo only works on file, I have written a Python script applying lipo recursively to several directories.

Putting everything together Now, we need to automate a bit. You could provide some nifty script. I propose an appropriate Makefile.am instead. This way, we can use the output variables, like @VERSION@, from ./configure to generate some of the files, like distribution.xml. You need to add that in your configure.ac:

AC_CONFIG_FILES([osx/Makefile osx/distribution.xml osx/im.bernat.lldpd.plist])
AC_CONFIG_FILES([osx/scripts/preinstall],  [chmod +x osx/scripts/preinstall])
AC_CONFIG_FILES([osx/scripts/postinstall], [chmod +x osx/scripts/postinstall])
AC_SUBST([CONFIGURE_ARGS], [$ac_configure_args])

Let s have a look at osx/Makefile.am. First, we define some variables:

PKG_NAME=@PACKAGE@-@VERSION@.pkg
PKG_TITLE=@PACKAGE@ @VERSION@
PKG_DIR=@PACKAGE@-@VERSION@
ARCHS=@host_cpu@

If we want to build for several architectures, we will type make ARCHS="x86_64 i386". Otherwise it defaults to the current host s architecture. We use install-data-local target to install files specific to OS X:

install-data-local:
    install -m 0755 -d $(DESTDIR)/Library/LaunchDaemons
    install -m 0644 im.bernat.@PACKAGE@.plist $(DESTDIR)/Library/LaunchDaemons
uninstall-local:
    rm -f $(DESTDIR)/Library/LaunchDaemons/im.bernat.@PACKAGE@.plist

The main target is the product archive, built with productbuild:

../$(PKG_NAME): pkg.1/$(PKG_NAME) distribution.xml resources
    $(PRODUCTBUILD) \
        --distribution distribution.xml \
        --resources resources \
        --package-path pkg.1 \
        --version @VERSION@ \
        $@

Its main dependency is the component package:

pkg.1/$(PKG_NAME): $(PKG_DIR) scripts
    [ -d pkg.1 ]   mkdir pkg.1
    $(PKGBUILD) \
        --root $(PKG_DIR) \
        --identifier im.bernat.@PACKAGE@.daemon \
        --version @VERSION@ \
        --ownership recommended \
        --scripts scripts \
        $@

Now, we need to build $(PKG_DIR):

$(PKG_DIR): stamp-$(PKG_DIR)
stamp-$(PKG_DIR): $(ARCHS:%=%/$(PKG_DIR))
    $(srcdir)/lipo $(PKG_DIR) $^
    touch $@

The $(ARCHS:%=%/$(PKG_DIR)) will expand to x86_64/$(PKG_DIR) i386/$(PKG_DIR). This is a substitution reference⁴. Before applying our lipo script, we need to be able to build the architecture-dependent package directories:

pkg_curarch = $(@:stamp-%=%)
$(ARCHS:%=%/$(PKG_DIR)): %/$(PKG_DIR): stamp-%
$(ARCHS:%=stamp-%): stamp-%: im.bernat.lldpd.plist
    [ -d $(pkg_curarch) ]   mkdir -p $(pkg_curarch)
    (cd $(pkg_curarch) && \
        $(abs_top_srcdir)/configure @CONFIGURE_ARGS@ \
            CC="@CC@ -arch $(pkg_curarch)" \
            CPP="@CPP@")
    (cd $(pkg_curarch) && \
        $(MAKE) install DESTDIR=$(abs_builddir)/$(pkg_curarch)/$(PKG_DIR))
    touch $@

This may seem a bit obscure, but this is really like what we have described previously. Note that we use @CONFIGURE_ARGS@ which is a variable we have defined in configure.ac. Here is how a user would create its own OS X package:

$ SDK=/Developer/SDKs/MacOSX10.6.sdk
$ mkdir build && cd build
$ ../configure --prefix=/usr --sysconfdir=/private/etc --with-embedded-libevent \
   CFLAGS="-mmacosx-version-min=10.6 -isysroot $SDK" \
   LDFLAGS="-mmacosx-version-min=10.6 -isysroot $SDK"
[...]
$ make -C osx pkg ARCHS="i386 x86_64"
[...]
productbuild: Wrote product to ../lldpd-0.7.5-21-g5b90c4f-dirty.pkg
The package has been built in ../lldpd-0.7.5-21-g5b90c4f-dirty.pkg.

---

1. Even if Xcode is free, you will need to register to Apple and show a valid credit card number. That s quite an interesting requirement.
2. It is possible to use other formats (like RTF or just plain text) for documents but HTML seems the best choice if you want some formatting without fiddling with RTF.
3. Unfortunately, Apple makes it difficult to find them. The easiest way to get the SDK for Mac OS X 10.6 is to download Xcode 4.3.3, mount the image and copy the SDK to /Developer/SDKs.
4. This is equivalent to the use of patsubst function. However, this function is specific to GNU make. The substitution reference pattern was accepted recently in POSIX and is coined as pattern macro expansion.

Even though the backend of Rosenberg NMS uses rrdtool RRD files, the front end graphs are created using jqPlot. The idea is to have a set of templates for the different types of graphs and just apply them to the various data sets. It makes things a lot simpler for new graphs because you just select which one you want; unless you want something a lot different which would involve a new graph template. In any case, anyone that looks enough at the standard rrdtool graphs will know they are a series of steps. While it depends on the RRA, usually they are 5 minute steps, so a graph showing an increasing rate might show 5 minutes of 2 Mbps and then the next 5 minutes of 11 Mbps. jqPlot graphs as I ve currently got them draws a line between two data points, so there would be a sloping line starting at 2.5 minutes (half of the first 5 minute interval) and 2Mbps sloping up to 11 Mbps at 7.5 minutes. At first I thought this was wrong and spent some time attempting to fix the graph by making it look like a rrdtool graph more. Someway through that process I stopped and wondered, what IS the right way? The answer like a lot of other things, is it depends . For a graph showing a rate, such as the output bits per second on an interface, the way this is done is at regular intervals a counter is measured. So if at time 0 the counter is 140 and time 60 the counter is 200 and finally at time 120 the counter is at 800 there has been an average rate of 1 [(200-140)/60] and 100 [(800-200)/60)]. rrdtool would show a horizontal bar at 1 bps and then another horizontal bar at 100 at the next time interval. jqPlot would show a sloping line going from 30,1 up to 90,100. Two graphs looking very different from the same data, what gives? Each graph is right, or rather is showing an estimation of different things, hence their differences. rrdtool is showing the average rate for that time period. It is in some ways accurate because leaving aside missed wrap-arounds and resets that many bits did pass through that interface for the time specified. However often the graphs are interpreted incorrectly as the real rate and not an average. We can be reasonably sure for a rate that it would not be 1 for exactly a minute and then immediately jump to 100 for another minute. This isn t rrdtool s fault, it is just how the graphs can be interpreted. jqplot will show more realistic graphs, with a curve sloping up. However this too makes assumptions that the rate increase is linear which often it is not. It just gives the illusion that the graph knows more about the data than it really does. In the end, both graphs are at the same time accurate and misleading. It s important when looking at any graph in general (not just these two types) that you understand its limits and assumptions. To give one example of the problems that can be missed, traffic interfaces may have microbursts (large amount of traffic in short amount of time) which, due to the averaging that goes on in graphing are invisible to graphs and give an incorrect account of what is going on.

Mir is Canonical's new display server. It fulfils a broadly similar role to Wayland and Android's Surfaceflinger, in that it takes final responsibility for getting pixels onto the screen. XMir is an X server that runs on top of Mir. It permits applications that know how to speak the X protocol but don't know how to speak to Mir (ie, approximately all of them at present) to run in a Mir-based environment.

For Ubuntu 13.10, Canonical are proposing to use Mir by default. This doesn't mean that most applications will be using Mir, though - instead, the default session will run XMir as a full-screen client and a normal X environment will be run on that. This lets Canonical deploy Mir without forcing anyone to update their applications, allowing them to take a gradual approach. By 14.10, Canonical expect the default Unity session to be a Mir client rather than an X client. In theory it will then be possible to run an Ubuntu system without any X applications at all, leaving XMir to do nothing other than run legacy applications.

Of course, this requires a certain amount of replumbing. X would normally be responsible for doing things like setting up the screen and pushing the pixels out to the hardware, but this is now handled by Mir instead. Where a native X server would allocate a framebuffer in video memory and render into it, XMir asks Mir for a window corresponding to the size of the screen and renders into that and then simply asks Mir to display it. This step is actually more interesting than it sounds.

Unless you're willing to throw lots of CPU at them, unaccelerated graphics are slow. Even if you are, you're going to end up consuming more power for the same performance, so XMir would be impractical if it didn't provide access to accelerated hardware graphics functions. It makes use of the existing Xorg accelerated X drivers to do this, which is as simple as telling the drivers to render into the window that XMir requested from Mir rather than into the video framebuffer directly. In other words, when displaying through XMir, you're using exactly the same display driver stack as you would be if you were using Xorg. In theory you'd expect identical performance - in practice there's a 10-20% performance hit right now, but that's being actively worked on. Fullscreen 3D apps will also currently take a hit due to there being no support for skipping compositing, which is being fixed. XMir should certainly be capable of performing around as well as native X, but there's no reason for it to be any faster.

The output drivers aren't the only part of the stack. XMir still needs some way to get input events. You might naively expect that Mir would forward input events to XMir, but the current state of affairs is that XMir loads the existing X input drivers which in turn open the input devices themselves. This explains why XMir currently shows two cursors[1] - the first cursor is the Mir cursor that's being driven by the input events Mir is receiving, while the second is the cursor being drawn by XMir in response to the input events that it's receiving[2].

The final main piece of duplicated functionality is output management. In the case where X is its own display server, it can simply ask the kernel to program the outputs. XMir can't do that, so would have to ask Mir to do it instead. Sadly, that functionality's not implemented yet. Running xrandr on Xmir gives:

Screen 0: minimum 320 x 320, current 1366 x 768, maximum 8192 x 8192
XMIR-1 connected primary 1366x768+0+0 (normal left inverted right x axis y axis) 0mm x 0mm
   XMIR mode of death[3]   60.0*+

which translates as "I have a screen that can be any size from 320x320 to 8192x8192, and have one output of unknown physical size that can only display 1366x768 at 60Hz". Other than the 1366x768, the result will be identical no matter how many outputs you have - XMir currently has no support for configuring multiple displays, and will always report 60Hz. It also has no support for placing monitors into any kind of DPMS state.

Obviously, this is still software under active development. There's three months from now until until Ubuntu is due to release with this functionality enabled by default, and the only significant missing features are xrandr and DPMS support. As long as Mir exposes interfaces for controlling those, they shouldn't be a problem to implement. To most users, the transition from native Xorg to XMir should be completely invisible.

Which is kind of the problem. What benefits does the user gain from using XMir on Mir rather than native Xorg?

Features? XMir is a total of around 1000 lines of code on top of Xorg. It implements no functionality that isn't present in Xorg.

Performance? XMir effectively is Xorg - it's the same code running the same drivers. It's not going to be any faster. At the moment it's actually slower, but that's probably fixable.

Security? Again, XMir is effectively Xorg. It still runs as root. It has the same privileges. There's no security benefit to using XMir instead of Xorg.

Like I said, the transition should be completely invisible - no drawbacks, but no benefits. What it does do is get Mir deployed on a larger number of systems, which means Canonical can both get wider testing of the basic Mir functionality and argue that the number of deployed Mir systems means hardware vendors should support it. Users won't see any of the benefits until Unity transitions to being a native Mir client, which is slated for 14.10 next year.

As a PR move, it's pretty great. The public perception is that Mir has gone from not existing to being sufficient to run a full desktop environment in very little time, and it's natural to compare it to Wayland which has been in development for longer and still isn't running a fully featured desktop session. The problem with that perception is that Mir is doing very little at the moment. It's setting an output mode (by asking the kernel to), allocating a window for XMir to draw into and then pushing that window onto the screen whenever XMir tells it that it's changed. This isn't advanced display server functionality, it's the bare minimum that a display server could possibly do[4]. Which is a shame, because Mir is significantly more functional than that, it's just not being used to do anything we couldn't already do.

In summary: XMir on Mir in Ubuntu provides no user benefits and isn't a compelling technology demo. Mir itself will permit a range of additional features, but isn't slated to be running a user session itself until 14.10. The only obvious benefit to Canonical in shipping XMir on Mir is to gain additional testing, which makes using it in 14.04, a supposedly stable and long term release, a somewhat surprising choice.

[1] Although this is in the process of being fixed
[2] This also explains why, on systems with a trackpoint and a touchpad, only the trackpoint moves the Mir cursor while both move the XMir cursor. The X touchpad driver has put the touchpad in absolute mode, which means it's now reporting events that Mir currently ignores.
[3] "XMIR mode of death" is hardcoded into XMir. I'm really hoping it's a Regular Show reference.
[4] So why hasn't anyone done this with Wayland? Mostly because, as described above, there's no technical benefit in doing so. Wayland does have an X server for compatibility purposes, and if you wanted you could run it as a full screen client and run an entire session underneath it. But you'd gain nothing by doing so. The Wayland X server is intended for running individual clients rather than an entire X session. Run an X client under Wayland and it'll pop up in its own individual window and managed by your Wayland session's window manager, just like it would under X. XMir currently has no support for this "rootless" mode - right now if you want to run X apps under Mir, you'll need to launch an entire X session with its own window manager.

comments

As Americans we might like to believe that the US legal system is intended to protect all of our citizens. Unfortunately, it doesn t protect us all equally, and in fact disproportionately fails to protect the most vulnerable. We re surrounded by instances of injustice related to gender, race and other axes of social privilege, and the machinations of law are not exempt. The state of Florida has recently provided an especially stark example in the application of its self-defense laws in two cases: Marissa Alexander and George Zimmerman. This example is notable because although there were many similarities between the cases, the outcomes were very different. Alexander s case was tried in May 2012 , Zimmerman s in July 2013, both prosecuted by Florida state s attorney Angela Corey. Both cases involved the use of firearms which were legally purchased and carried, and their owners were trained in their use. Both prosecutions cast the defendant as the aggressor, who could have avoided the confrontation. Both of the encounters were with unarmed persons. Both defenses were based on Florida self-defense laws, which include stand your ground laws justifying the use of deadly force without the obligation to retreat. Both shooters admitted to firing a single shot with the intent of defending themselves. Beyond those similarities, each case had its own unique circumstances. The events of Alexander s case took place in her home. Her altercation was with her husband, Rico Gray Sr., who was under a restraining order following a conviction for domestic battery which put Alexander in the hospital. After Gray threatened to kill her, Alexander retrieved a handgun from her car, returned to confront him, firing once. She was arrested and charged the same day. She had had no prior criminal record. A jury deliberated for just 12 minutes before convicting her. A judge sentenced her to 20 years in prison, in accord with mandatory minimums specified by law. Gray, previously sentenced to probation for his earlier conviction, remains free. Zimmerman s shot was fired in his neighborhood, in an altercation with a teenager, Trayvon Martin, who was a guest in the community and walking by himself. The two were not acquainted. Zimmerman called police from his car, claiming that Martin appeared suspicious, and began to follow him. Some of the facts of their encounter remain in dispute, but that Zimmerman fired his gun is not in question. Afterward, Zimmerman was detained by police, questioned and released the same night without being arrested or charged. Following a public outcry, a new investigation was launched and two months later he was arrested and charged. He had been previously arrested and charged with assaulting a police officer, but the charges were later dropped. After 16 hours of deliberation, the jury found Zimmerman not guilty, and he is free today. The most striking difference between the two cases is where each defendant aimed their gun: George Zimmerman shot Trayvon Martin in the chest and killed him, while Marissa Alexander fired at a wall and injured no one. Alexander, a black woman, is in prison for scaring her abusive husband away, while Zimmerman, who killed a young black man, walks free. Alexander and Martin s families have lost a mother and a son. The outcomes for the people involved in these cases could not be more different. Regardless of the merits of the relevant laws themselves, their radically unequal application is deeply troubling. What does this tell us about the relative value of these human lives, as weighed by the judicial system?

The Florida criminal justice system has sent two clear messages today. One is that if women who are victims of domestic violence try to protect themselves, the Stand Your Ground Law will not apply to them. [...] The second message is that if you are black, the system will treat you differently. - U.S. Rep. Corrine Brown

References

• http://en.wikipedia.org/wiki/Shooting_of_Trayvon_Martin
• http://www.thefloridastar.com/new-trial-denied-battered-wife/
• http://www.huffingtonpost.com/2012/05/19/marissa-alexander-gets-20_n_1530035.html
• http://www.huffingtonpost.com/josh-horwitz/protecting-marissa-at-the_b_1551998.html
• http://justiceformarissa.blogspot.com/
• http://www.documentcloud.org/documents/326700-full-transcript-zimmerman.html

With the demise of Google Reader, many of us have scrambled to find alternatives which cover many/most of the features we were used to. For me, synchronised access from web and mobile counts quite high, and I want the (Android) mobile experience to be pretty pleasant too. So with that, and like many other folks, I had shifted over to Feedly. And before I carry on, let me say that Feedly actually does pretty well. They handled the onslaught of new users; they also listened and changed their UI a little to be more compact and Reader-alike And most importantly, it mostly just works. Until it doesn't. Around the time Feedly shifted folks to their own cloud-based backend, posts from my very own CRANberries aggregator of CRAN package changes for R started to show only truncated posts. See this screenshot from reading Feedly: and compare it with this one from TheOldReader: which contains the diffstat output, nicely marked up and all. Now, it so happens that I am the creator of the very RSS feed I am consuming here, and I wrote it so that I could read the very diffstat output that is now missing. And I know full well that neither the code, nor the hosting (on my own box), nor any other aspect changed. Before confirming this via TheOldReader, I looked at the RSS output, and I tried other frontend and apps --- and it became clear that the bug seems to be at the Feedly cloud storage level. And trying to be a good sport, I submitted a bug report / suggestion, but to no avail (apart from two other chiming in that they see this elsewhere too). Given that I am the coder behind the feed that is displayed in a truncated manner, I am aware that I am using a code stack that is stale (the original Blosxom static web generator) yet which has not posed another problem anywhere in the decade I used it. Nor does the CRANberries feed pose a problem when aggregated and viewed via the TheOldReader code path, or for that matter via Planet R which also carries (parts of) CRANberries. So for now, I have to either pivot out of the CRANberries RSS reading in Feedly and go directly to the webpage (fine, but cumbersome and in need of a working connection) or use a second subscription mechanism. I appreciate what the TheOldReader folks are doing---presumably with a minor fraction of the resources available to Feedly---and may hang with them for now. But it would be awfully nice if Feedly could sleep off its hangover and fix this. In which case I'd be much happier recommending its service.

this week(end) I was more in the "RC bug fix flow" than in the last weeks, & here are the results:

• #665518 src:clipit: "clipit: Including individual glib headers no longer supported"
  add patch from Michael Biebl, upload to DELAYED/2 (NMU originally prepared by Sebastian Ramacher)
• #701406 src:nagios-plugins: "nagios-plugins: ftbfs with eglibc-2.17"
  add patch from Ubuntu / Robie Basak, upload to DELAYED/2, then cancelled on maintainer's request, who uploaded the fixed package himself
• #707393 src:gorm.app: "gorm.app: FTBFS: GormCore/./obj/libGormCore.so: undefined reference to class_pose_as'"
  add patch from Ubuntu / Julian Taylor, upload to DELAYED/2
• #710768 ganglia-webfrontend: "ganglia-webfrontend: please update libgd dependencies"
  drop dependency on libgd2*, upload to DELAYED/2
• #711800 src:python-poppler: "python-poppler: FTBFS: /usr/bin/make: unrecognized option '--prefix --exec-prefix --includes --libs --cflags --ldflags --extension-suffix --help --configdir'"
  apply changes from Ubuntu / Matthias Klose, upload to DELAYED/2
• #713204 src:voms-api-java: "voms-api-java: FTBFS: cp: cannot stat 'target/site/javadoc/apidocs': No such file or directory"
  upload NMU with BouncyCastle patch prepared by Emmanuel Bourg to DELAYED/2
• #713360 src:regina-normal: "regina-normal: FTBFS: CMake Error: The following variables are used in this project, but they are set to NOTFOUND. PYTHON_LIBRARY"
  apply improved patch from Peter Green, upload to DELAYED/2
• #713450 src:oregano: "oregano: FTBFS: ld: sheet/libsheet.a(grid.o): undefined reference to symbol 'floor@@GLIBC_2.2.5'"
  add patch from Petter Reinholdtsen, upload to DELAYED/2
• #713627 src:rox: "rox: FTBFS: ld: support.o: undefined reference to symbol 'floor@@GLIBC_2.2.5'"
  add patch to link against math lib, upload to DELAYED/2
• #713649 src:gdis: "gdis: FTBFS: gdis.h:269:18: fatal error: glib.h: No such file or directory"
  propose a patch

Sometimes, people show up in our IRC channel #debian-systemd or on our mailing list pkg-systemd-maintainers@lists.alioth.debian.org and ask how they can help. This blog post answers that question. First of all, whatever you end up doing, please coordinate with us first! Saying hi in our IRC channel or on our mailing list and explaining what you intend to do is all we ask for. In the past, people have filed bug reports with service files that are not idiomatic, and we really need to avoid that. Currently, our biggest task is to add/improve systemd support to individual Debian packages, and this is where we can use a lot of help. We have a dashboard which contains two lists:

1. Packages that ship a service file, but don t use dh-systemd yet.
   These need to be updated to get proper maintscripts (e.g. deb-systemd-helper enable/disable calls, possibly start/stop/restart calls).
2. Packages that ship an init script but no service file yet.
   These need a service file plus dh-systemd support.

Adding a service file and dh-systemd support After you picked a service from the second list in our dashboard, first double-check that there is no bug filed against it yet (the dashboard might be a few hours behind). Then, search for a service file, either upstream or in other distributions such as Arch Linux, Fedora, Gentoo, etc. In case there is no service file yet, write one. Please carefully test the service file (e.g. does starting/stopping/restarting and reloading work) and send it to us for review. Be careful to avoid obvious regressions from the sysvinit init script. That is, don t just write a very simplistic service file that breaks a use-case which formerly worked. If in doubt, ask pkg-systemd-maintainers or chose a different package to work on. Afterwards, add systemd support by using the dh-systemd helper, see https://wiki.debian.org/Systemd/Packaging. The last step is to file a proper bug report against the package that uses our usertag and CCs pkg-systemd-maintainers. See http://bugs.debian.org/714190 for an example. Replacing custom maintscripts with dh-systemd Check if the package currently has any direct systemctl calls in its maintscripts and remove them if so (see http://bugs.debian.org/713853 for an example). Follow https://wiki.debian.org/Systemd/Packaging to add systemd support using the dh-systemd helper. Check whether the maintscripts look good and generate a debdiff to highlight the differences between the old and the new version:

debdiff --controlfiles=ALL OLD_amd64.changes NEW_amd64.changes

Then file a bug report which uses our usertag and CCs pkg-systemd-maintainers. See http://bugs.debian.org/713853 for an example. Thanks! Thank you for helping us and Debian in general!

The German computer magazine c't has covered Debsources in its most recent edition (c't 16/2013). In that article, they also state:

Debsources integriert auch eine Code-Suche, allerdings werden lediglich die
Quellen des Unstable-Zweigs durchsucht, der zirka ein Drittel des Quellcodes
von Debsources ausmacht.

This loosely translates to:

Debsources also integrates a code search engine, but it only searches the
sources of the unstable tree, which makes up for roughly one third of the
sources that Debsources covers.

I suspect the author of the article arrived at this conclusion because Debsources talks about 400 GiB of source code, whereas Debian Code Search talks about 130 GiB of source code. However, it still struck me as odd and giving the wrong impression. I thought that packages don t differ that much between stable, testing and unstable. So I fired up psql and pounded UDD until it revealed how many packages are different between stable, testing and unstable:

• 3581 of wheezy s packages have a newer upstream version in testing. The total amount of packages in wheezy is 17576, so this makes about 20.3%.
• 1131 (6%) of these packages are even newer in unstable.
• 13092 packages (74%) have identical upstream versions in wheezy, testing and unstable.

Conclusion So, in conclusion, Debian Code Search covers 74% of wheezy, testing and unstable. Debsources also covers contrib and non-free, which explains the higher disk usage. In particular, there might be big blobs in non-free that account for a lot of disk space. Also, Debsources keeps sources around for a few weeks whereas Debian Code Search only keeps the most recent snapshot. Query

SELECT
  COUNT(*)
FROM
  (SELECT
     migrations.source,
     sources.version AS stable_version,
     testing_version,
     unstable_version
   FROM sources
   LEFT JOIN migrations
   USING (source)
   WHERE sources.release = 'wheezy') AS x
WHERE
  regexp_replace(stable_version::text, E'-([0-9.]+)$', '') = regexp_replace(testing_version::text, E'-([0-9.]+)$', '');