There will come a time when you believe everything is finished. That will be the beginning. Louis L Amour

Last year, I wrote about the difficult times in my life and hope for future, but never really explained why. I have written little since, because there is little I can write without a bit of explanation. It is partly because of the complexity of the task of telling my story without telling too much of others stories. But it is important I tell this story. So often on the Internet, we hear only the brave face, the positive things that happen. This story involves tears and difficulty. And also we often see only anger and bitterness. This story involves joy and celebration. One afternoon last year, I was working as usual (I work from home) when two sheriff deputies arrived on my doorstep. They gave me paperwork showing that my now ex-wife had filed for divorce, had asked them to serve me the paperwork, and that she had been given temporary possession of the house. I had 2 hours to gather up clothes and a little computer equipment (there was a list) and leave the house. Thus began the most difficult time in my life. I went from reading a bedtime story and singing a bedtime song to my young boys every night to seeing them only a little, from living in the house my grandparents and dad had lived in to having no particular plan for where I d sleep that night, from thinking I had a good idea of what the future held to not knowing when, if ever, I d ever be back home. I worried about how the boys would fare (they have done well so far). And it was incomprehensible; I couldn t find answers to why? . In the time since, the divorce became final, I did return home, the boys spend more time with me, and a new normal emerged. At the time, it seemed like a sudden, deep winter blizzard. I couldn t see very far down the road, spring seemed far off, and I couldn t see very well either forward or backward. But I was determined to find positives in the situation. It started almost immediately; I had never been a person to talk about pain, but just a few hours after the divorce was filed, I knew I needed to talk to someone about it, and did. A week later, I shared about it in church. Amazing friends, locally and all over the world, provided support and encouragement. I had less total time with the boys than before the divorce, but more time with just the three of us, and we used it to play together at home, spend days in town, and even take a train trip to Santa Fe, where none of us had ever been before. I realized how much I could forgive, and that my ex-wife probably did the best she could with the persistent legacy of difficult life events that happened to her long before she met me. I understood this, and was never angry, just sad, for everyone. I have always known nobody is perfect, myself included, but can be hard on myself when I m less than perfect. I forgave myself, too, realizing that I did my best to help in the most unfamiliar of waters, and although I sometimes didn t get it right, my conscience is clear because my heart was in the right place and I tried, very hard. Most incredibly, I became a person with a deep sense of inner peace. I always tried to work hard to set life on a good path; I got good grades in school, am a good employee, and have a strong set of values. But where courts are involved, there s a strong sense of powerlessness. At times, there was nothing I could do to make life better for my boys or for me. I finally had to let go of taking on responsibility for all that on my own shoulders. I simply knew that things would be OK, and in fact were OK, and that there is nothing in life that really deserves worries. That doesn t mean worries are never present, but that mostly they are subdued, like a radio quietly playing in another room. When they aren t, I can sit down at the piano, play my penny whistle, sing, walk to my creek, talk to friends, or any of so many things that let them melt away. I stopped searching for happiness and peace, and let those things find me. In religious terms, my faith became not just an intellectual one, but also a spiritual one. An atheist friend asked me, Just what does religion mean to you anyway? My answer: The certainty that spring always comes, for everyone that understands this.

People speak of misfortunes and sufferings, remarked Pierre, but if at this moment I were asked: Would you rather be what you were before you were taken prisoner, or go through all this again? then for heaven s sake let me again have captivity and horseflesh! We imagine that when we are thrown out of our usual ruts all is lost, but it is only then that what is new and good begins. While there is life there is happiness. There is much, much before us. Tolstoy, War and Peace

By Thanksgiving, I had much to be thankful for. Some of it under my nose waiting to be rediscovered after years of distraction, such as the glorious Kansas sunrises. Some things were new, such as roasting a turkey all by myself (or, rather, with Jacob and Oliver) for the first time, and having it come out absolutely perfectly. And some were just the things of everyday life: that I lived in my own house again, that I could walk out to the creek at the edge of my property whenever I wanted, that I could play piano, that the sounds of laughter and little running feet again could often be heard on my wood floors. That s not to say everything was easy; the courts sometimes made decisions sometimes I didn t think were in the boys best interests, legal things dragged on and on, but in the end, peace endured. Happiness endured. I found myself thinking at Thanksgiving that it was the best Thanksgiving ever. Not much later, I considered myself happier than I ve ever been. I was focusing on the daily gifts of life, marveling at the sunrises, looking forward to life s next adventure, confident that it would be far better than the last. And then, to my complete surprise, I found myself in a relationship again. On top of all the wonderful things happening in my life, I met Laura. I never could have imagined a friend so wonderful, a relationship so loving and joyful, something to treasure so deeply. I can look back at events and shake my head in amazement and wonder, that I found myself happier than I d ever been, and then this wonderful relationship on top of that. I have been blessed to have the life I do, and feel almost embarrassingly lucky. I share this story because friends that had been through divorce years ago shared their stories. They gave me hope. And if I didn t share this story in this public way, I would be squandering an opportunity to find more positives from what happened. I hope that this helps, somehow, someone that is in pain know that there is beauty in the valley, and spring always comes, every single year. And I share it because happiness like this can t be repressed for long. Tolstoy was right. While there is life, there is happiness. I ll end with a story from last Thanksgiving. It was 6:10AM that weekend. I was still asleep, and heard this: Jacob, yelling from his room: Dad? Dad! Me, groggy: Yes, Jacob? Jacob: Can I go down and look at the Christmas tree? Me: Sure, and you can turn it on too. Jacob: Great! At that point, I could have gone back to sleep. I was really short on sleep that morning, and Jacob would have been fine. But I gave him a couple of minutes, then I went downstairs too. He was curled up on the piano bench, looking at the tree. I quietly turned up the downstairs thermostat, got a chair, put it next to the piano bench, and sat down by the tree too. Jacob crawled over onto my lap and snuggled up for awhile. Neither of us said anything. Then: Dad? Yes? This is the best Thanksgiving ever. And he gave me a big hug. And he was right. Yes, he was RIGHT! That will be the beginning. Spring comes!

Present Extension Redirection Multi-buffered applications have always behaved poorly in the presence of the Composite extension:

• Updating involves multiple copies of the window contents, first from back buffer to composite redirect buffer, thence from the composite buffer to the screen back buffer, and then from the screen back buffer to the scanout buffer. The last copy is amenable to page flipping, but that would require the EGL buffer age extension so that the compositor could reduce the cost of presenting following frames by being able to track the contents of the back buffers.
• The application is not informed about when the actual screen presentation occurs.

Owen Taylor suggested that Present should offer a way to redirect operations to the compositing manager as a way to solve these problems. This posting is my attempt to make that idea a bit more concrete given the current Present design. Design Goals Here s a list of features I think we should try to provide:

1. Provide accurate information to applications about when presentation to the screen actually occurs. In particular, GLX applications using GLX_OML_sync_control should receive the correct information in terms of UST and MSC for each Swap Buffers request.
2. Ensure that applications still receive correct information as to the contents of their buffers, in particular we want to be able to implement EGL_EXT_buffer_age in a useful manner.
3. Avoid needing to un-redirect full-screen windows to get page flipping behavior.
4. Eliminate all extra copies. A windowed application may still perform one copy from back buffer to scanout buffer, but there shouldn t be any reason to copy contents to the composite redirection buffer or the compositing manager back buffer.

Simple Present Redirection With those goals in mind, here s what I see as the sequence of events for a simple windowed application doing a new full-window update without any translucency or window transformation in effect:

1. Application creates back buffer, draws new frame to it.
2. Application executes PresentRegion. In this example, the valid and update parameters are None , indicating that the full window should be redrawn.
3. The server captures the PresentRegion request and constructs a PresentRedirectNotify event containing sufficient information for the compositor to place that image correctly on the screen:
   • target window for the presentation
   • source pixmap containing the new image
   • idle fence to notify when the source pixmap is no longer in use.
   • serial number from the request.
   • target MSC for the presentation. This should probably just be the computed absolute MSC value, and not the original target/divisor/numerator values provided by the application.
4. The compositing manager receives this event and constructs a new PresentRegion request using the provided source pixmap, but this time targeting the root window, and constructing a valid region which clips the pixmap to the shape of the window on the screen. This request would use the original application s idle fence value so that when complete, the application would get notified. This request would need to also include the original target window and serial number so that a suitable PresentCompleteNotify event can be constructed and delivered when the final presentation is complete.
5. The server executes this new PresentRegion operation. When complete, it delivers PresentCompleteNotify events to both the compositing manager and the application.
6. Once the source pixmap is no longer in use (either the copy has completed, or the screen has flipped away from this pixmap), the server triggers the idle fence.

Multiple Application Redirection If multiple applications perform PresentRegion operations within the same frame, then the compositing manager will receive multiple PresentRedirectNotify events, and can simply construct multiple new PresentRegion requests. If these are all queued to the same global MSC, they will execute at the same frame boundary. No inter-operation dependency exists here. Complex Presentations Ok, so the simple case looks like it s pretty darn simple to implement, and satisfies the design goals nicely. Let s look at a couple of more complicated cases in common usage; the first is with translucency, the second with scaling application images down to thumbnails and the third with partial application window updates. Redirection with Translucency If the compositing manager discovers that a portion of the updated region overlays or is overlaid by something translucent (either another window, or drop shadows, or something else), then a composite image for that area must be constructed before the whole can be presented. Starting when the compositing manager receives the event, we have:

1. The compositing manager receives this event. Using the new pixmap, along with pixmaps for the other involved windows and graphical elements, the compositing manager constructs an updated view for the affected portion of the screen in a back buffer pixmap. Once complete, a PresentRegion operation that uses this back buffer pixmap is sent to the X server. Again, the original target window and serial number are also delivered to the server so that a suitable PresentCompleteNotify event can be delivered to the application.
2. The server executes this new PresentRegion operation; PresentCompleteNotify events are delivered, and idle fences triggered as appropriate.

Redirection with Transformation Transformation of the window contents means that we cannot always update a portion of the back buffer directly from the provided application pixmap as that will not contain the window border. Contents generated from a region that includes both application pixels and window border pixels must be sourced from a single pixmap containing both sets of pixels. One option that I ve discussed in the past to solve this would be to have the original application allocate the pixmap large enough to hold both the application contents and the surrounding window border. Have it draw the application contents at the correct offset within this pixmap, and then have the window manager contents drawn around that; either automatically by the X server, or even manually by the compositing manager. That would be mighty convenient for the compositing manager, but would require significant additional infrastructure throughout the X server and even harder the drawing system (OpenGL or some other system). There s another reason to want this though, and that s for sub-frame buffer scanout page swapping. The second option would be for the compositing manager to combine these images itself; there s a nice pixmap already containing the window manager image the composite redirect buffer. Taking the provided source pixmap and copying it directly to the target window will construct our composite image, just as if we had no Present redirection in place. This will cost an additional copy though, which we ve promised to avoid. Of course, as it s just for thumb-nailing or other visual effects, perhaps the compositing manager could perform this operation at a reduced frame rate, so that overall system performance didn t suffer. Retaining Access to the Application Buffer Above, I discussed having the idle fence from the redirected PresentRegion operation be sent along with the replacement PresentRegion operation. This ignores the fact that the composting manager may well need the contents of that application frame again in the future, when displaying changes for other applications that involve the same region of the screen. With the goal of making sure the idle fences are triggered as soon as possible so that applications can re-use or free the buffers quickly, let s think about when the triggering can occur.

1. Full-screen flipped applications. In this case, the application s idle fence can be triggered once the application provides a new frame and the X server has flipped to that new frame, or some other scanout buffer.
2. Windowed, copied applications. In this case, the application s idle fence can be triggered once the application provides a new frame to the compositing manager, and the X server doesn t have any presentations queued.

In both cases, we require that both the X server and the compositing manager be finished with the buffer before the application s idle fence should be triggered. One easy way to get this behavior is for the composting manager to create a new idle fence for its operations. When that is triggered, it would receive an X event and then trigger the applications idle fence as appropriate. This would add considerable latency to the application s idle fence a round trip through the compositing manager. The alternative would be to construct some additional protocol to make the applications idle fence dependent on the Present operation and some additional state provided by the compositing manager. Some experimentation here is warranted, but my experience with latency in this area is that it causes applications to end up allocating another back buffer as the idle notification arrives just after a buffer allocation request comes down to the rendering library. Definitely sub-optimal. An Aside on Media Stream Counters The GLX_OML_sync_control extension defines the Media Stream Counter (MSC) as a counter unique to the graphics subsystem which is incremented each time a vertical retrace occurs. That would be trivial if we had only one vertical retrace sequence in the world. However, we actually have N+1 such counters, one for each of the N active monitors in the system and a separate fake counter to be used when none of the other counters is available. In the current Present implementation, windows transition among the various Media Stream Counter domains as they move between the various monitors, and those monitors get turned on and off. As they move between these counter domains, Present tracks a global offset from their original domain. This offset ensures that the MSC value remains monotonically increasing as seen by each window. What it does not ensure is that all windows have comparable MSC sequence values; two windows on the same monitor may well have different MSC values for the same physical retrace event. And, even moving a window from one MSC domain to another and back won t make it return to the original MSC sequence values due to differences in refresh rates between the monitors. Internally, Present asserts that each CRTC in the system identifies a unique MSC domain, and it has a driver API which identifies which CRTC a particular window should be associated with. Once a particular CRTC has been identified for a window, client-relative MSC values and CRTC-relative MSC values can be exchanged using an offset between that CRTC MSC domain and the window MSC domain. The Intel driver assigns CRTCs to windows by picking the CRTC showing the greatest number of pixels for a particular window. When two CRTCs show the same number of pixels, the Intel driver picks the first in the list. Vblank Synchronization and Multiple Monitors Ok, so each window lives in a particular MSC domain, clocked by the MSC of the CRTC the driver has associated it with. In an un-composited world, this makes picking when to update the screen pretty simple; Present updates the screen when vblank happens in the CRTC associated with the window. In the composite redirected case, it s a bit harder all of the PresentRegion operations are going to target the root window, and yet we want updates for each window to be synchronized with the monitor containing that window. Of course, the root window belongs to a single MSC domain (likely the largest monitor, using the selection algorithm described above from the Intel driver). So, any PresentRegion requests will be timed relative to that single monitor. I think what is required here is for the PresentRegion extension to take an optional CRTC argument, which would then be used as the MSC domain instead of the window MSC domain. All of the timing arguments would be interpreted relative to that CRTC MSC domain. The PresentRedirectNotify event would then contain the relevant CRTC and the MSC value would be relative to that CRTC. A clever Compositing manager could then decompose a global PresentRegion operation into per-CRTC PresentRegion operations and ensure that multiple monitors were all synchronized correctly. We could take this even further and have the PresentRegion capable of passing a smaller CRTC-sized pixmap down to the kernel, effectively providing per-CRTC pixmaps with no visible explicit protocol Other Composite Users Ok, so the above discussion is clearly focused on getting the correct contents onto the screen with minimal copies along the way. However, what I ve ignored is how to deal with other applications, also using Composite at the same time. They re all going to expect that the composite redirect buffers will contain correct window contents at all times, and yet we ve just spent a bunch of time making that not be the case to avoid copying data into those buffers and instead copying directly to the compositing manager back or front buffers. Obviously the X server is aware of when this happens; the compositing manager will have selected for manual redirection on all top-level windows, while our other application will have only been able to select for automatic redirection. So, we ve got two pretty clear choices here:

1. Have the X server change how Present redirection works when some other application selects for Automatic redirection on a window. It would copy the source pixmap into the window buffer and then send (a modified?) PresentRedirectNotify event to the compositing manager.
2. Include a flag in the PresentRedirectNotify event that the composite redirect buffer needs to eventually get the contents of the source pixmap, and then expect the compositing manager to figure out what to do.

Development Plans As usual, I m going to pick the path of least resistance for all of the above options and see how things look; where the easy thing works, we can keep using it. Where the easy thing fails, I ll try something else. The changes required for this are pretty minimal. The PresentRegion request needs to gain a list of window/serial pairs that are also to be notified when the operation completes:

PRESENTNOTIFY  
    window: WINDOW
    serial: CARD32
     
 
    PresentRegion
    window: WINDOW
    pixmap: PIXMAP
    serial: CARD32
    valid-area: REGION or None
    update-area: REGION or None
    x-off, y-off: INT16
    idle-fence: FENCE
    target-crtc: CRTC or None
    target-msc: CARD64
    divisor: CARD64
    remainder: CARD64
    notifies: LISTofPRESENTNOTIFY
 
    Errors: Window, Pixmap, Match

The target-crtc parameter explicitly identifies a CRTC MSC domain. If None, then this request implicitly uses the window MSC domain. notifies provides a list of windows that will also receive PresentCompleteNotify events with the associated serial number when this PresentRegion operation completes.

 
    PresentRedirectNotify
    type: CARD8         XGE event type (35)
    extension: CARD8        Present extension request number
    length: CARD16          2
    evtype: CARD16          Present_RedirectNotify
    eventID: PRESENTEVENTID
    event-window: WINDOW
    window: WINDOW
    pixmap: PIXMAP
    serial: CARD32
    valid-area: REGION
    valid-rect: RECTANGLE
    update-area: REGION
    update-rect: RECTANGLE
    x-off, y-off: INT16
    target-crtc: CRTC
    target-msc: CARD64
    idle-fence: FENCE
    update-window: BOOL
 

The target-crtc identifies which CRTC MSC domain the target-msc value relates to. divisor and remainder have been removed as the target-msc value has been adjusted using the application values. If update-window is True, then the recipient of this event is instructed to provide reasonably up-to-date contents directly to the window by copying the contents of pixmap to the window manually. Beyond these two protocol changes, the compositing manager is expected to receive Sync events when the idle-fence is triggered and then manually perform a Sync operation to trigger the client s idle-fence when appropriate. I m planning to work on these changes, and then go re-work xcompmgr (or perhaps unagi, which certainly looks less messy) to incorporate support for Present redirection. The goal is to have something to demonstrate at Guadec, which doesn t seem impossible, aside from leaving on vacation in four days

<section class="field field-name-taxonomy-vocabulary-1 field-type-taxonomy-term-reference field-label-above view-mode-rss">

Categories:

• Debian-Blog
• Computing

</section><section class="field field-name-taxonomy-vocabulary-2 field-type-taxonomy-term-reference field-label-above view-mode-rss">

Keywords:

• laptop-mode-tools
• linux

</section>

• RHUT

The Scope of the Problem Security is inherently complex because of the large number of ways of circumventing it. For example Internet facing servers have been successfully attacked based on vulnerabilities in the OS, the server application, public key generation, DNS, SSL key certificates (and many other programs and algorithms in use), as well as the infrastructure and employees of all companies in the chain. When all those layers work reasonably well (not perfectly but well enough to not obviously be the weakest link) there are attacks on the end user systems that access the servers (such as the trojan horse programs used to attack PCs used for online banking). My Area of Interest The area of security that interests me is Linux software development. There are many related areas such as documentation and default configurations to make it easier for people to secure their systems (instead of insecure systems being the default option) which are all important. There are also many related fields such as ensuring that all people with relevant access are trustworthy. There are many interesting problems to solve in such areas most of which aren t a good match for my skills or just require more time than I have available. I sometimes write blog posts commenting on random security problems in other areas. Sometimes I hope to inspire people to new research, sometimes I hope to just inform users who can consider the issues when implementing solutions to security problems. Bugs In the software development side there are ongoing problems of bugs in code that weaken security. The fact that the main area of concern for people who are interested in securing systems is fixing bugs is an indication that the problem of software quality needs a lot of work at the moment. The other area that gets a reasonable amount of obvious work is in access control. Again it s an area that needs a lot of work, but the fact that we re not done with that is an indication of how far there is to go in generally improving computer security. Authenticating Software Releases There have been cases where source code repositories have been compromised to introduce trojan horse code, the ones I ve read about have been discovered reasonably quickly with little harm done but there could be some which weren t discovered. Of course it s likely that such attacks will be discovered because someone will have the original and the copies can be compared. Repositories of binaries are a bigger problem, it s not always possible to recompile a program and get a binary which checks out as being identical (larger programs often include the build time in the binary). Even for build processes which don t include such data it can be very difficult to determine the integrity of a build process. For example programs compiled with different versions of libraries, header files, or compilers will usually differ slightly. As most developers frequently change the versions of such software they will often be unable to verify their own binaries and any automated verification of such binaries will be impossible for anyone else. So if a developer s workstation was compromised without their knowledge it might be impossible for them to later check whether they released trojan binaries without just running the binaries in question and looking for undesired behavior. The problem of verifying past binaries is solvable for large software companies, Linux distributions, and all other organisations that have the resources to keep old versions of all binaries and libraries used to build software. For proprietary software companies the verification process would have to start with faith in the vendor of their OS and compiler doing the right thing. For Linux distributions and other organisations based on free software it would start by having the source to everything which can then be verified in theory although in practice verifying all source for compilers, the OS, and libraries would be a huge undertaking. Espionage There is a well documented history of military espionage, people who are sworn to secrecy have been subverted by money, blackmail, and by having political beliefs which don t agree with their government. The history of corporate espionage is less well documented but as corporations perform less stringent background checks than military organisations I think it s safe to assume that corporate espionage is much more common. Presumably any government organisation that can have any success at subverting employees of a foreign government can be much more successful in subverting programmers (either in companies such as Microsoft or in the FOSS community). One factor that makes it easier to launch such attacks is the global nature of software development. Government jobs that involve access to secret data have requirements about where the applicant was born and has lived, corporate jobs and volunteer positions in free software development don t have such requirements. The effort involved in subverting an existing employee of a software company or contributor to free software or the effort involved in getting an agent accepted in such a project would be quite small when compared to a nuclear weapons program. Therefore I think we should assume that every country which is capable of developing nuclear weapons (even North Korea) can do such things if they wish. Would the government of such a country want to subvert a major software project that is used by hundreds of millions of people? I can imagine ways that such things could benefit a government and while there would be costs for such actions (both in local politics and international relations) it seems most likely that some governments would consider it to be worth the risk and North Korea doesn t seem to have much to lose. Conclusion We would like every computer to be like a castle with a strong wall separating them from the bad things which can t be breached in ways that aren t obvious. But the way things are progressing with increasingly complex systems depending on more people and other systems it s becoming more like biology than engineering. We can think of important government systems as being comparable to the way people with compromised immune systems are isolated from any risk of catching a disease, the consequences of an infection are worse so greater isolation measures are required. For regular desktop PCs getting infected by a trojan is often regarded as being similar to getting a cold in winter. People just accept that their PC will be infected on occasion and don t bother making any serious effort to prevent it. After an infection is discovered the user (or their management for a corporate PC) tend not to be particularly worried about data loss in spite of some high profile data leaks from companies that do security work and the ongoing attacks against online banking and webcam spying on home PCs. I don t know what it will take for users to start taking security risks seriously. I think that a secure boot is a good step in the right direction, but it s a long way from being able to magically solve all security problems. I ve previously described some of the ways that secure boot won t save you [1]. The problems of subverting developers don t seem to be an immediate concern (although we should consider the possibility that it might be happening already without anyone noticing). The ongoing trend is that the value of computers in society is steadily increasing which therefore increases the rewards for criminals and spy agencies who can compromise them. Therefore it seems that we will definitely face the problems of subverted developers if we can adequately address the current technical problems related to flaws in software and inadequate access control. We just need to fix some of the problems which are exploited more easily to force the attackers to use the more difficult and expensive attacks. Note that it is a really good thing to make attacks more difficult, that decreases the number of organisations that are capable of attack even though it won t stop determined attackers. For end user systems the major problem seems to be related to running random programs from the Internet without a security model that adequately protects the system. Both Android and iOS make good efforts at protecting a system in the face of random hostile applications, but they have both been shown to fail in practice (it might be a good idea to have a phone for games that is separate from the phone used for phone calls etc). More research into OS security is needed to address this. But in the mean time users need to refrain from playing games and viewing porn on systems that are used for work, Internet banking, and other important things. While PCs are small and cheap enough that having separate PCs for important and unimportant tasks is practical it seems that most users don t regard the problems as being serious enough to be worth the effort.

• [1] http://etbe.coker.com.au/2011/12/28/secure-boot-protecting-against-root/

There s been some media coverage at the start of this week about blocking child porn. Except it s not about child porn that s a trojan horse. People who want to access pornography that is already illegal (Protection of Children Act 1978) are probably already using security tools to hide their downloading and will be unaffected by this unless they re pretty stupid. And the announcement, about the same time as the predicted birth of a royal baby, third in line to the throne, seems like a cynical attempt to bury bad news taken straight from the Blair Government. That would almost be enough reason to oppose it: they don t want the media to look at this too closely for some reason. So what s this actually about? It looks like a way to force through widespread acceptance of the ability to censor most UK internet users by shouting won t somebody think of the children? If you doubt it, take a look at the list of filtered topics:

• Dating
• Drugs, Alcohol and Tobacco
• File Sharing Sites
• Gambling
• Games
• Pornography
• Social Networking
• Suicide and Self-Harm
• Weapons and Violence

So if they get away with this censorship, you won t be able to use Twitter or contact the Samaritans until you deactivate it. Except I suspect you will because they re pretty big and the Cameron Government won t want to pick a fight with them: it ll be the next Twitter and the next Samaritans, currently much smaller and unable to defend themselves, who get shut out of UK homes. So what can we do, besides explaining this and writing to our MPs? Are we better off joining parties who oppose this censorship, like the Pirate Party, or joining existing parties and trying to overturn their stupid support for it?

Jon Masters, Chief ARM Architect at Red Hat, recently posted a description of his expectations for baseline arm64 servers. The quick summary is that systems should implement UEFI and ACPI, and any more traditional ARM boot mechanisms should be ignored. This is an interesting departure from the status quo in the ARM world, and it's worth thinking about the benefits and drawbacks of this approach.

It's very easy to build a generic kernel for most x86 systems, since the PC platform is fairly well defined even if not terribly well specified. Where system hardware does vary, it's almost always exposed on an enumerable bus (such as PCI or USB) which allows the OS to bind appropriate drivers. Things are different in the ARM world. Even once you're past the point of different SoC vendors requiring different kernel setup code and drivers, you still have to cope with the fact that system vendors can wire these SoCs up very differently. Hardware is often attached via GPIO lines without any means to enumerate them. The end result is that you've traditionally needed a different kernel for every ARM board. This is viable if you're selling the OS and hardware as a single product, but less viable if there's any desire to run a generic OS on the hardware.

The solution that's been adopted for this in the Linux world is called Device Tree. Device Tree actually has significant history, having been used as the device descriptor format in Open Firmware. Since there was already support for it in the Linux kernel, adapting it for use in ARM devices was straightforward. Device Tree aware devices can pass a descriptor blob to the kernel at startup[1], and devices without that knowledge can have a blob build into the kernel.

So, if this problem is already solved, why the push to move to UEFI and ACPI? This push didn't actually originate in the Linux world - Microsoft mandate that Windows RT devices implement UEFI and ACPI, and were they to launch a Windows ARM server product would probably carry that over. That makes sense for Microsoft, since recent versions of Windows have been x86 only and so have grown all kinds of support for ACPI and UEFI. Supporting Device Tree would require Microsoft to rewrite large parts of Windows, whereas mandating UEFI and ACPI allowed them to reuse most of their existing Windows boot and driver code. As a result, largely at Microsoft's behest, ACPI 5 has grown a range of additional features for describing things like GPIO pinouts and I2C connections. Whatever your weird device layout, you can probably express it via ACPI.

This argument works less well for Linux. Linux already supports Device Tree, whereas it currently doesn't support ACPI or UEFI on ARM[2]. Hardware vendors are already used to working with Device Tree. Moving to UEFI and ACPI has the potential to uncover a range of exciting new kernel issues and vendor bugs. It's not obviously an engineering win.

So how about users? There's an argument that since server vendors are now mostly shipping ACPI and UEFI systems, having ARM support these technologies makes it easier for customers to replace x86 systems with ARM systems. This really doesn't fly for ACPI, which is entirely invisible to the user. There are no standard ACPI entry points for system configuration, and the bits of ACPI that are generically useful (such as configuring system wakeup times) are already abstracted away to a standard interface by the kernel. It's somewhat more compelling for UEFI. UEFI supports a platform-independent bytecode language (EFI Byte Code, or EBC), which means that customers can write their own system management utilities, build them for EBC and then deploy them to their servers without caring about whether they're x86 or ARM. Want a bootloader that'll hit an internal HTTP server in order to determine which system image to deploy, and which works on both x86 and ARM? Straightforward.

Arnd Bergmann has a interesting counterargument. In a nutshell, ARM servers aren't currently aiming for the same market as x86 servers, and as a result customers are unlikely to gain any significant benefit from shared functionality between the two.

So if there's no real benefit to users, and if there's no benefit to kernel developers, what's the point? The main one that springs to mind is that there is a benefit to distributions. Moving to UEFI means that there's a standard mechanism for distributions to interact with the firmware and configure the bootloader. The traditional ARM approach has been for vendors to ship their own version of u-boot. If that's in flash then it's not much of a problem[3], but if it's on disk then you have to ship a range of different bootloaders and know which one to install (and let's not even talk about initial bootstrapping).

This seems like the most compelling argument. UEFI provides a genuine benefit for distributions, and long term it probably provides some benefit to customers. The question is whether that benefit is worth the flux. The same distribution benefit could be gained by simply mandating a minimum set of u-boot functionality, which would seem much more straightforward. The customer benefit is currently unclear.

In the end it'll probably be a market decision. If Red Hat produce an ARM product that has these requirements, and if Suse produce an ARM product that will work with u-boot and Device Tree, it'll be up to vendors to decide whether the additional work to support UEFI/ACPI is worth it in order to be able to sell to customers who want Red Hat. I expect that large vendors like HP and Dell will probably do it, but the smaller ones may not. The customer demand issue is also going to be unclear until we learn whether using UEFI is something that customers actually care about, rather than a theoretical benefit.

Overall, I'm on the fence as to whether a UEFI requirement is going to stick, and I suspect that the ACPI requirement is tilting at windmills. There's nothing stopping vendors from providing a Device Tree blob from UEFI, and I can't think of any benefits they gain from using ACPI instead. Vendor interest in the generic parts of the ACPI spec has been tepid even in the x86 world (the vast majority of ACPI spec updates come from Microsoft and Intel, not any of the system vendors), and I don't see that changing with the introduction of a range of ARM vendors who are already happy with Device Tree.

We'll see. Linux is going to need to gain the support for UEFI and ACPI on ARM in any case, since there's already hardware shipping in that configuration. But with ARM vendors still getting to grips with Device Tree, forcing them to go through another change in how they do things is going to be hard work. Red Hat may be successful in enforcing these requirements at the cost of some vendor unhappiness, or Red Hat may find that their product doesn't boot on most of the available hardware. It's an aggressive gamble, and while it'll be interesting to see how it plays out, I'm not that optimistic.

[1] The blob could be pulled from the firmware, but it's not uncommon for it to be built into u-boot instead. This does mean that you have a device-specific u-boot even if you have a generic kernel, but that's typically true anyway.
[2] Patches have been posted for ARM UEFI support. They're not mergeable in their current form, but they should be in the near future. ACPI support is in development.
[3] Although not all u-boots are created equal - some vendors ship versions that will only boot off FAT, some vendors ship versions that will only boot off ext2. Having to special case this stuff in your installer is a pain.

comments

A recent google+ post by Jon Masters caused some stormy and some less stormy responses. A lot of BIOS/UEFI/ACPI hate comes X86, where ACPI is used from everything from suspending devices to reading buttons and setting leds. So when X86 kernel suspends, it does magic calls to ACPI and prays that the firmware vendor did not screw it up. Now vendors do screw up, hence lots of cursing and ugly workarounds in the kernel follows. My Lenovo has a firwmare bug where the FN-buttons and Fan stops working if the laptop is attached on AC adapter for too long. The fan is probably a simple i2c device the kernel could control directly without jumping through ACPI hiding layer hoops. But the X86 people hold the view it is better to trust the firmware engineer to control devices instead of having the kernel folk to write device drivers to ... control devices! Now on ARM(64) the idea of using ACPI is to have none of that. Instead the idea is to use ACPI only to provide tables for enumerating what devices are available in the platform. Just like what device tree does. Now if this is the same as device tree, why bother? The main reason is to allow the distribution installer behave same on X86 / ARM / ARM64. This is crucial for distributions like fedora and RHEL where a cabal holds the point of view that X86 distribution development must not be constrained by ARM support. But it also important for everyone that the method of installing your favorite distribution to an ARM64 server is standard and works the same for any server from any vendor. Now while UEFI and ACPI are definitely not my preferred solutions, I can accept them as necessary evil for having a more standard platform.

Dpkg and RPM provide a mechanism called triggers. But are they doing the same thing? There are two different types of triggers for Dpkg: explicit and file. In the case of explicit triggers, the script of a package A will activate a package B. The file type is more frequently used. In that case, the creation, modification or deletion of a file in a path declared by a package B will activate the package B. The triggers are often used when a package A has to register an information to a package B. For instance, when a graphical application is installed, its package will add its description in the /usr/share/applications directory, which will trigger the desktop-file-utils package, which will do what is necessary to add a new menu entry in the window manager. RPM's triggers function differently. The installation, update or removal of a package A will activate a package B if this package B declares its interest for the package A in its SPEC file (with possible restrictions on package A's version). The documentation of RPM and Fedora gives a few examples of use, like the update of symbolic links when one mail server is replaced by another. Altogether, despite both mechanisms have the same name, they do not have the same goal. In the case of Dpkg, the triggers are especially useful to group similar actions. For instance, to update the list of manual pages once after installing a hundred of packages, instead of a hundred of times, after installing each package. They are also useful to transfer commands from a large number of packages to a central one, thus easing the evolution of the system. In the case of RPM, the triggers seem more specialised, as they always are about the relationship between a pair of packages.

I'm sure you all will be a bit disappointed (I know I am) that there are no ascii screenshots in this weeks report. But let me make it up to you by telling you a story. I was at the FOSDEM in the year 2012 and I went to a nice workshop on sunday afternoon in the Virtualization Devroom. Renzo Davoli was the host of this workshop and he started with a little introductory talk to get everyone to agree on a common terminology first. He began by asking the question, what virtualization meant in the most general way. He defined the ability to virtualize a resource as the ability to freely and transparently control how someone (say a process) is interacting with said resource. So for example if you LD_PRELOAD a library to impersonate fopen(3) and friends, you have virtualized the filesystem (for some small values of virtualized). He then went on to discuss various methods of virtualization that are commonly available on Linux, not only full system virtualization solutions, but also all kinds of methods allowing a more fine-grained control over what resources are virtualized. Of course every method had its strengths and its weaknesses. Having seen Samuels talk about all the awesome things Hurd can do for virtualization I approached him with a question in the free discussion part of his workshop. I asked whether he would agree that once a resource is moved from the kernel to the userspace, the problem of virtualizing that resource is almost trivially solved, and he agreed. So I said that this was awesome, because then the trivial and elegant solution for all his virtualization (and tracing) needs are micro kernel operating systems, and he also agreed to that but (as far as I can recall) he mentioned that there is none that would meet his needs (I believe he is a computer science professor and heavily relies on virtualization techniques for his lab and for teaching purposes). And even though a microkernel operating systems has a cost (message passing instead of function calls for example) it also has its merits such as scaling better from a development point of view. Also, many cool features can emerge just from the design. For example think about the container support in Linux and how painfully long it took to make all the resources namespace aware (with one of the most critical, the user namespace being the most difficult one). On Hurd you get the same functionality for free. If you are curious, read Samuels awesome slides. So what have I done this week?

• procfs should ignore --no dev,exec,suid : http://lists.gnu.org/archive/html/bug-hurd/2013-07/msg00076.html
• libnetfs does not respond properly to file_get_translator requests on nodes with a passive translator record: http://lists.gnu.org/archive/html/bug-hurd/2013-07/msg00148.html
• As the mtab translator is a translator of its own, there needs to be a way to bind it to /proc/mounts. I figured the easiest way would be to just serve a node with an appropriate passive translator record: http://lists.gnu.org/archive/html/bug-hurd/2013-07/msg00149.html
• There was no clean way to start the console-client. I added daemonizing support to the console-client and created an appropriate initscript: http://lists.gnu.org/archive/html/bug-hurd/2013-07/msg00165.html
• I also worked a lot on my mtab translator prototype: http://lists.gnu.org/archive/html/bug-hurd/2013-07/msg00183.html

Next week I'm going to work on the two remaining sysvinit related issues. These are the only ones preventing me to come up with a clean patch series against the sysvinit package and I figured that it would be nice to propose such a series rather sooner than later so that I will have plenty of time to discuss any issues with the sysvinit maintainers. See you next week :)

Yesterday, I've spent some time trying to improve speed of translation import in Weblate. It got annoyingly slow while adding new features and I felt there has to be some area for improvements even with all the features. I was testing on development server with debug stuff enabled (what slows down things quite a lot already) and first import took about 30 seconds. Quick look revealed that for linked repositories, the link lookup happens too often, so some caching could be used. This has saved about 3 seconds... With offloaded indexing (what is anyway only useful setup for production use), every reindex request was checking whether there is already one before creating new one. In most cases there is none before, so it's simply faster to try to create one and ignore error in case it is already there. We're now 5 seconds faster. Another big resource consumer are quality checks. They provide useful feedback to translators, but it means that quite a lot of code needs to be executed for every single string. First thing I've realized is that unit check state is recalculated everytime regardless the fact whether there was any change, what was easy to fix. The other speed issue is in consistency check - it does too complex SQL query, which usually does not match anything. Converting this to much simpler one (which will in most cases still have no results) improves performance for usual path while having minimal impact for the real checking. Another 4 seconds down on my test import. Last thing which I changed is way how per translation statistics are updated. In past this required quite a lot of queries, but yesterday I've realized this can be much reduced. During test suite run on PostgreSQL, I saw that it's not that easy - PostgreSQL does not allow to sum boolean fields, but fortunately this can be fixed as well. This has saved about 2 seconds. In the end the tested import took 16 seconds instead of original 30 and issued 2500 SQL queries instead of 5000 at the beginning. I hope you will enjoy regained speed of imports :-).

Filed under: English Weblate 0 comments Flattr this!

There are three main ways to distribute a command-line daemon for OS X:

1. Distributing source code and instructions on how to compile it.
2. Using a third-party package manager, like Homebrew.
3. Providing an installer package.

Homebrew Homebrew is a popular package management system. It works like the BSDs ports collections by downloading, compiling and installing the requested software, while also installing any required dependencies automatically. Creating a new package is quite easy and there are a lot of examples available. However, there are some limitations:

• You don t really build a package but execute a recipe to locally install the software.
• You need to install development tools, either a whole Xcode installation¹ or the command line version.
• If you need to execute some steps as root, you will have to explain them to the user for her to execute by hand. This includes the creation of a system user or the installation of a daemon through launchd.

If you can t live with those limitations, you may need to build an installer package.

Building an installer package OS X comes with a graphical and a command-line installer. The graphical one is run by opening a package from the Finder. Then, the user experiences some familiar wizard allowing her to install the software in a matter of a few seconds. The documentation around how to build such a package is somewhat clumsy. You may find outdated information or pieces of information that only apply to projects using Xcode. I will try to provide here accurate bits in the following context:

1. You are packaging a pure command-line tool.
2. You are using Autoconf and Automake as a build system.
3. You want to support several architectures.
4. You want to support older versions of OS X.

Creating a package Building such a package was previously done with some graphical tool called PackageMaker. This tool is not available any more and developers are asked to switch to pkgbuild and productbuild. There is a very neat Stackoverflow article on how those tools work. A package is built in two steps:

1. Build component packages.
2. Combine them into a product archive.

A component package contains a set of files and a set of scripts to execute at various steps of the installation. You can have several component packages, for example a package for the daemon and a package for the client. They are built with pkgbuild. A product archive contains the previously created component packages as well as a file describing various options of the installer (required and optional components, license, welcome text, ). To create a component package, we need to install the needed files in some directory:

$ ./configure --prefix=/usr --sysconfdir=/private/etc
$ make
$ make install DESTDIR=$PWD/osx-pkg

We will now put the content of osx-pkg into a component package with pkgbuild:

$ mkdir pkg1
$ pkgbuild --root osx-pkg \
>    --identifier org.someid.daemon \
>    --version 0.47 \
>    --ownership recommended
>    pkg1/output.pkg
pkgbuild: Inferring bundle components from contents of osx-pkg
pkgbuild: Wrote package to output.pkg

You need to be careful with the identifier. It needs to be unique and identify your software as well as the specific component. Then, you need to create an XML file describing the installer. Let s call it distribution.xml:

<?xml version="1.0" encoding="utf-8" standalone="no"?>
<installer-gui-script minSpecVersion="1">
    <title>Some daemon</title>
    <organization>org.someid</organization>
    <domains enable_localSystem="true"/>
    <options customize="never" require-scripts="true" rootVolumeOnly="true" />
    <!-- Define documents displayed at various steps -->
    <welcome    file="welcome.html"    mime-type="text/html" />
    <license    file="license.html"    mime-type="text/html" />
    <conclusion file="conclusion.html" mime-type="text/html" />
    <!-- List all component packages -->
    <pkg-ref id="org.someid.daemon"
             version="0"
             auth="root">output.pkg</pkg-ref>
    <!-- List them again here. They can now be organized
         as a hierarchy if you want. -->
    <choices-outline>
        <line choice="org.someid.daemon"/>
    </choices-outline>
    <!-- Define each choice above -->
    <choice
        id="org.someid.daemon"
        visible="false"
        title="some daemon"
        description="The daemon"
        start_selected="true">
      <pkg-ref id="org.someid.daemon"/>
    </choice>
</installer-gui-script>

Fortunately, this file is documented on Apple Developer Library. Since we only describe one package, we pass customize="never" as an option to skip the choice part. You can however remove this attribute when you have several component packages. The attribute rootVolumeOnly="true" explains that this daemon can only be installed system-wide. It is marked as deprecated but still works. Its replacement (domains tag) displays an unusual and buggy pane which Apple doesn t use in any of its packages. You need to put the HTML² documents into a resources directory. It is also possible to choose the background by specifying a <background/> tag. The following command will generate the product archive:

$ productbuild --distribution distribution.xml \
>              --resources resources \
>              --package-path pkg1 \
>              --version 0.47 \
>              ../final.pkg
productbuild: Wrote product to ../final.pkg

Scripts The installer can execute some scripts during installation. For example, suppose we would like to register our daemon with launchd for it to be run at system start. You first need to write some org.someid.plist file and ensure it will get installed in /Library/LaunchDaemons:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
  <key>Label</key>
  <string>org.someid</string>
  <key>ProgramArguments</key>
  <array>
    <string>/usr/sbin/mydaemon</string>
    <string>-d</string>
  </array>
  <key>RunAtLoad</key><true/>
  <key>KeepAlive</key><true/>
</dict>
</plist>

Then create a scripts/postinstall file with the following content:

#!/bin/sh
set -e
/bin/launchctl load "/Library/LaunchDaemons/org.someid.plist"

You also need a scripts/preinstall file which will allow your program to be smoothly upgraded:

#!/bin/bash
set -e
if /bin/launchctl list "org.someid" &> /dev/null; then
    /bin/launchctl unload "/Library/LaunchDaemons/org.someid.plist"
fi

Ensure that those scripts are executable and add --scripts scripts to pkgbuild invocation. If you need to add a system user, use dscl.

Dependencies There is no dependency management built into OS X installer. Therefore, you should ensure that everything is present in the package or in the base system. For example, lldpd relies on libevent, an event notification library. This library is not provided with OS X. When it is not found, the build system will use an embedded copy. However, if you have installed libevent with Homebrew, you will get binaries linked to this local libevent installation. Your package won t work on another host. The output of otool -L can detect unwanted dependencies:

$ otool -L build/usr/sbin/lldpd
build/usr/sbin/lldpd:
        /usr/lib/libresolv.9.dylib (compatibility version 1.0.0, current version 1.0.0)
        /System/Library/Frameworks/IOKit.framework/Versions/A/IOKit (compatibility version 1.0.0, current version 275.0.0)
        /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 744.19.0)
        /System/Library/Frameworks/Foundation.framework/Versions/C/Foundation (compatibility version 300.0.0, current version 945.18.0)
        /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 169.3.0)

Compilation for older versions of OS X The package as built above may only work on the same version of OS X that you use to compile. If you want a package that will work on OS X 10.6, you will need to download the appropriate SDK³. Then, you need to tell the compiler the target version of OS X and the SDK location. You can do this by setting CFLAGS and LDFLAGS (and CXXFLAGS if you use C++):

$ SDK=/Developer/SDKs/MacOSX10.6.sdk
$ ./configure --prefix=/usr --sysconfdir=/private/etc \
>         CFLAGS="-mmacosx-version-min=10.6 -isysroot $SDK" \
>         LDFLAGS="-mmacosx-version-min=10.6 -isysroot $SDK"

The -mmacosx-version-min flag will be used by various macros in the headers to mark some functions as available or not, depending on the version of OS X you target. It will also be used by the linker. The -isysroot will tell the location of the SDK. Headers and libraries will be looked up in this location first.

Universal binaries Mac OS X 10.6 was available for both IA-32 and x86-64 architectures. If you want to support both installations with a single package, you need to build universal binaries. The Mach object file format, used by OS X, allows several versions of the executable in the same file. The operating system will select the most appropriate one. An easy way to generate such files is to pass -arch i386 -arch x86_64 to the compiler:

$ ./configure --prefix=/usr --sysconfdir=/private/etc \
>     CC="gcc -arch i386 -arch x86_64" \
>     CPP="gcc -E"

However, this is a dangerous option. Suppose that your ./configure script tries to determine some architecture-dependent parameter, like the size of an integer (with AC_CHECK_SIZEOF); it will compute it for the host architecture. The binary generated for the other architecture will therefore use a wrong value and may crash at some point. The correct way to generate an universal binary is to execute two separate compilations and to build the universal binary with lipo:

$ for arch in i386 x86_64; do
>   mkdir $arch ; cd $arch
>   ../configure --prefix=/usr --sysconfdir=/private/etc \
>       CC="gcc -arch $ARCH" \
>       CPP="gcc -E"
>   make
>   make install DESTDIR=$PWD/../target-$ARCH
>   cd ..
> done
[...]
$ lipo -create -output daemon i386/usr/sbin/daemon x86_64/usr/sbin/daemon

Since lipo only works on file, I have written a Python script applying lipo recursively to several directories.

Putting everything together Now, we need to automate a bit. You could provide some nifty script. I propose an appropriate Makefile.am instead. This way, we can use the output variables, like @VERSION@, from ./configure to generate some of the files, like distribution.xml. You need to add that in your configure.ac:

AC_CONFIG_FILES([osx/Makefile osx/distribution.xml osx/im.bernat.lldpd.plist])
AC_CONFIG_FILES([osx/scripts/preinstall],  [chmod +x osx/scripts/preinstall])
AC_CONFIG_FILES([osx/scripts/postinstall], [chmod +x osx/scripts/postinstall])
AC_SUBST([CONFIGURE_ARGS], [$ac_configure_args])

Let s have a look at osx/Makefile.am. First, we define some variables:

PKG_NAME=@PACKAGE@-@VERSION@.pkg
PKG_TITLE=@PACKAGE@ @VERSION@
PKG_DIR=@PACKAGE@-@VERSION@
ARCHS=@host_cpu@

If we want to build for several architectures, we will type make ARCHS="x86_64 i386". Otherwise it defaults to the current host s architecture. We use install-data-local target to install files specific to OS X:

install-data-local:
    install -m 0755 -d $(DESTDIR)/Library/LaunchDaemons
    install -m 0644 im.bernat.@PACKAGE@.plist $(DESTDIR)/Library/LaunchDaemons
uninstall-local:
    rm -f $(DESTDIR)/Library/LaunchDaemons/im.bernat.@PACKAGE@.plist

The main target is the product archive, built with productbuild:

../$(PKG_NAME): pkg.1/$(PKG_NAME) distribution.xml resources
    $(PRODUCTBUILD) \
        --distribution distribution.xml \
        --resources resources \
        --package-path pkg.1 \
        --version @VERSION@ \
        $@

Its main dependency is the component package:

pkg.1/$(PKG_NAME): $(PKG_DIR) scripts
    [ -d pkg.1 ]   mkdir pkg.1
    $(PKGBUILD) \
        --root $(PKG_DIR) \
        --identifier im.bernat.@PACKAGE@.daemon \
        --version @VERSION@ \
        --ownership recommended \
        --scripts scripts \
        $@

Now, we need to build $(PKG_DIR):

$(PKG_DIR): stamp-$(PKG_DIR)
stamp-$(PKG_DIR): $(ARCHS:%=%/$(PKG_DIR))
    $(srcdir)/lipo $(PKG_DIR) $^
    touch $@

The $(ARCHS:%=%/$(PKG_DIR)) will expand to x86_64/$(PKG_DIR) i386/$(PKG_DIR). This is a substitution reference⁴. Before applying our lipo script, we need to be able to build the architecture-dependent package directories:

pkg_curarch = $(@:stamp-%=%)
$(ARCHS:%=%/$(PKG_DIR)): %/$(PKG_DIR): stamp-%
$(ARCHS:%=stamp-%): stamp-%: im.bernat.lldpd.plist
    [ -d $(pkg_curarch) ]   mkdir -p $(pkg_curarch)
    (cd $(pkg_curarch) && \
        $(abs_top_srcdir)/configure @CONFIGURE_ARGS@ \
            CC="@CC@ -arch $(pkg_curarch)" \
            CPP="@CPP@")
    (cd $(pkg_curarch) && \
        $(MAKE) install DESTDIR=$(abs_builddir)/$(pkg_curarch)/$(PKG_DIR))
    touch $@

This may seem a bit obscure, but this is really like what we have described previously. Note that we use @CONFIGURE_ARGS@ which is a variable we have defined in configure.ac. Here is how a user would create its own OS X package:

$ SDK=/Developer/SDKs/MacOSX10.6.sdk
$ mkdir build && cd build
$ ../configure --prefix=/usr --sysconfdir=/private/etc --with-embedded-libevent \
   CFLAGS="-mmacosx-version-min=10.6 -isysroot $SDK" \
   LDFLAGS="-mmacosx-version-min=10.6 -isysroot $SDK"
[...]
$ make -C osx pkg ARCHS="i386 x86_64"
[...]
productbuild: Wrote product to ../lldpd-0.7.5-21-g5b90c4f-dirty.pkg
The package has been built in ../lldpd-0.7.5-21-g5b90c4f-dirty.pkg.

---

1. Even if Xcode is free, you will need to register to Apple and show a valid credit card number. That s quite an interesting requirement.
2. It is possible to use other formats (like RTF or just plain text) for documents but HTML seems the best choice if you want some formatting without fiddling with RTF.
3. Unfortunately, Apple makes it difficult to find them. The easiest way to get the SDK for Mac OS X 10.6 is to download Xcode 4.3.3, mount the image and copy the SDK to /Developer/SDKs.
4. This is equivalent to the use of patsubst function. However, this function is specific to GNU make. The substitution reference pattern was accepted recently in POSIX and is coined as pattern macro expansion.

Mir is Canonical's new display server. It fulfils a broadly similar role to Wayland and Android's Surfaceflinger, in that it takes final responsibility for getting pixels onto the screen. XMir is an X server that runs on top of Mir. It permits applications that know how to speak the X protocol but don't know how to speak to Mir (ie, approximately all of them at present) to run in a Mir-based environment.

For Ubuntu 13.10, Canonical are proposing to use Mir by default. This doesn't mean that most applications will be using Mir, though - instead, the default session will run XMir as a full-screen client and a normal X environment will be run on that. This lets Canonical deploy Mir without forcing anyone to update their applications, allowing them to take a gradual approach. By 14.10, Canonical expect the default Unity session to be a Mir client rather than an X client. In theory it will then be possible to run an Ubuntu system without any X applications at all, leaving XMir to do nothing other than run legacy applications.

Of course, this requires a certain amount of replumbing. X would normally be responsible for doing things like setting up the screen and pushing the pixels out to the hardware, but this is now handled by Mir instead. Where a native X server would allocate a framebuffer in video memory and render into it, XMir asks Mir for a window corresponding to the size of the screen and renders into that and then simply asks Mir to display it. This step is actually more interesting than it sounds.

Unless you're willing to throw lots of CPU at them, unaccelerated graphics are slow. Even if you are, you're going to end up consuming more power for the same performance, so XMir would be impractical if it didn't provide access to accelerated hardware graphics functions. It makes use of the existing Xorg accelerated X drivers to do this, which is as simple as telling the drivers to render into the window that XMir requested from Mir rather than into the video framebuffer directly. In other words, when displaying through XMir, you're using exactly the same display driver stack as you would be if you were using Xorg. In theory you'd expect identical performance - in practice there's a 10-20% performance hit right now, but that's being actively worked on. Fullscreen 3D apps will also currently take a hit due to there being no support for skipping compositing, which is being fixed. XMir should certainly be capable of performing around as well as native X, but there's no reason for it to be any faster.

The output drivers aren't the only part of the stack. XMir still needs some way to get input events. You might naively expect that Mir would forward input events to XMir, but the current state of affairs is that XMir loads the existing X input drivers which in turn open the input devices themselves. This explains why XMir currently shows two cursors[1] - the first cursor is the Mir cursor that's being driven by the input events Mir is receiving, while the second is the cursor being drawn by XMir in response to the input events that it's receiving[2].

The final main piece of duplicated functionality is output management. In the case where X is its own display server, it can simply ask the kernel to program the outputs. XMir can't do that, so would have to ask Mir to do it instead. Sadly, that functionality's not implemented yet. Running xrandr on Xmir gives:

Screen 0: minimum 320 x 320, current 1366 x 768, maximum 8192 x 8192
XMIR-1 connected primary 1366x768+0+0 (normal left inverted right x axis y axis) 0mm x 0mm
   XMIR mode of death[3]   60.0*+

which translates as "I have a screen that can be any size from 320x320 to 8192x8192, and have one output of unknown physical size that can only display 1366x768 at 60Hz". Other than the 1366x768, the result will be identical no matter how many outputs you have - XMir currently has no support for configuring multiple displays, and will always report 60Hz. It also has no support for placing monitors into any kind of DPMS state.

Obviously, this is still software under active development. There's three months from now until until Ubuntu is due to release with this functionality enabled by default, and the only significant missing features are xrandr and DPMS support. As long as Mir exposes interfaces for controlling those, they shouldn't be a problem to implement. To most users, the transition from native Xorg to XMir should be completely invisible.

Which is kind of the problem. What benefits does the user gain from using XMir on Mir rather than native Xorg?

Features? XMir is a total of around 1000 lines of code on top of Xorg. It implements no functionality that isn't present in Xorg.

Performance? XMir effectively is Xorg - it's the same code running the same drivers. It's not going to be any faster. At the moment it's actually slower, but that's probably fixable.

Security? Again, XMir is effectively Xorg. It still runs as root. It has the same privileges. There's no security benefit to using XMir instead of Xorg.

Like I said, the transition should be completely invisible - no drawbacks, but no benefits. What it does do is get Mir deployed on a larger number of systems, which means Canonical can both get wider testing of the basic Mir functionality and argue that the number of deployed Mir systems means hardware vendors should support it. Users won't see any of the benefits until Unity transitions to being a native Mir client, which is slated for 14.10 next year.

As a PR move, it's pretty great. The public perception is that Mir has gone from not existing to being sufficient to run a full desktop environment in very little time, and it's natural to compare it to Wayland which has been in development for longer and still isn't running a fully featured desktop session. The problem with that perception is that Mir is doing very little at the moment. It's setting an output mode (by asking the kernel to), allocating a window for XMir to draw into and then pushing that window onto the screen whenever XMir tells it that it's changed. This isn't advanced display server functionality, it's the bare minimum that a display server could possibly do[4]. Which is a shame, because Mir is significantly more functional than that, it's just not being used to do anything we couldn't already do.

In summary: XMir on Mir in Ubuntu provides no user benefits and isn't a compelling technology demo. Mir itself will permit a range of additional features, but isn't slated to be running a user session itself until 14.10. The only obvious benefit to Canonical in shipping XMir on Mir is to gain additional testing, which makes using it in 14.04, a supposedly stable and long term release, a somewhat surprising choice.

[1] Although this is in the process of being fixed
[2] This also explains why, on systems with a trackpoint and a touchpad, only the trackpoint moves the Mir cursor while both move the XMir cursor. The X touchpad driver has put the touchpad in absolute mode, which means it's now reporting events that Mir currently ignores.
[3] "XMIR mode of death" is hardcoded into XMir. I'm really hoping it's a Regular Show reference.
[4] So why hasn't anyone done this with Wayland? Mostly because, as described above, there's no technical benefit in doing so. Wayland does have an X server for compatibility purposes, and if you wanted you could run it as a full screen client and run an entire session underneath it. But you'd gain nothing by doing so. The Wayland X server is intended for running individual clients rather than an entire X session. Run an X client under Wayland and it'll pop up in its own individual window and managed by your Wayland session's window manager, just like it would under X. XMir currently has no support for this "rootless" mode - right now if you want to run X apps under Mir, you'll need to launch an entire X session with its own window manager.

comments

Having reached the end of my Kickstarter funded year working on the git-annex assistant, I've decided to try one more crowdfunding campaign, to see if I can get funded to work on it a little while longer. I went back and forth on this for a while. The Kickstarter funded development was extremely successful (one of the most productive years of my life). I certianly want to work on git-annex more, and have lots more stuff to do, particularly around security and encryption. On the other hand, it's hard to frame ongoing development as a normal Kickstarter campaign to start something new. Anyway, I've decided to go ahead and try it, and not do it through Kickstarter this time. So I have my own website set up and accepting donations, and hopefully I'll make enough to spend a few more months working on git-annex. campaign.joeyh.name ... And if not, I'll probably spend a few months working on git-annex part time, while looking for paying work with the rest of my time. By the way, I'm taking payments in both US Dollars (via Paypal) and Bitcoin (via Coinbase). Can't wait to see how this works out!

Thanks to the Qt/KDE team, specially to Maxy who has done most of the packaging and uploading, sid users are now enjoying KDE 4.10.5, including the new KDE PIM stuff that we have held out for the Wheezy release.

I started using KMail2 (inside Kontact) a few days after Wheezy's release, getting it from experimental. And I have to admit that I really like it, just like with KMail1.

But my upgrade did have some bumps on the road, so I'm sharing them here so you can now how I solved them.

Mail import worked as we were waiting: it did work. So it was really useful to hold back Kmail1 until this really worked.

Now, I had a problem with my hard disk: whenever KMail started, it would start accessing it without pause. There where two reasons (for what I could test, I haven't looked at the source to really see if there was some other oddity) for this: I had a nepomuk/virtuoso DB created quite some time ago and initial mail indexing.

The initial mail indexing takes lots of time. For 1GB of DIMAP I had to wait like 5 hours (yes, 5 hours) on a 5600 rpm disk to let it fully finish. My desktop machine, with a faster hard drive, took a little less.

As far as people told me, that should have been enough, but my disk kept crawling. So I remembered someone from the team saying something about people with early-created nepomuk/virtuoso databases will have some speed issues. Mine where more than that, buy trying was worth the shot.

I had nepomuk disabled since I tried it the first version due to this exact problem. So I closed my KDE session and removed the nepomuk/virtuoso data:

rm -r ~/.kde/share/apps/nepomuk/

Then I logged back in KDE and waited (again) the 5 hours to let nepomuk re index my mail, this time totally finishing after 5 hours. Starting from that point, I get some one or two minutes of disk trashing some times I log in (not always), but it's actually not that bad. And I heard that in KDE 4.11 this has been improved a lot, so I should see a better behavior from that point on.

Please understand that this was my trial-and-error fix, it may be possible that someone comes with a better solution :-)

Every package maintainer has probably looked at the Testing migration section of their package s PTS page and wondered: Okay? What the hell does that mean? . Or perhaps Is that something I should do something about? . Lets have a look at an example excuse (from the PTS)[1]:

excuses:
*  Maintainer: Debian Python Modules Team
*  Too young, only 8 of 10 days old
*  Updating python-scipy fixes old bugs: #691254, #707315
*  Not considered

The first thing to look for is the phrase Valid candidate . If that appears, Britney will attempt to migrate your package to testing. Note the migration may fail, but we can come back to that (in a) later (post). On the other hand, if Not considered appears, then Britney believes your package is not ready to migrate yet. At this point, the excuse will contain 1 or more reasons explaining why Britney thinks your package should not migrate. The most common problem is probably age . In the example above, that is indeed the problem denoted by the Too young, only 8 of 10 days old . Age is incremented on the evening run (i.e. the 10 pm UTC run). The excuse of that run will include the up to date age. So when you see:

  Too young, only 9 of 10 days old

It means that the age criteria will be satisfied in the next evening run . If the age requirement is satisfied, the entry will look like:

    11 days old (needed 10 days)

If there is a Not considered , it means there is something else you need to fix to make your package migrate. Note that the age is based on the day the package was uploaded to sid. If the package migrates to testing and is later removed from testing again, the age will usually be ridiculously high. This is because all the days spent the package spent in testing is also included in the age. [1] It looks similar on the real excuses page, except the excuses: header is actually the name of the source. Is also includes information about the versions involved (i.e. [from-version] to [to-version] ).

Half a lifetime ago, somewhere at the end of the last millennia I became a free software enthusiast. On and off, I've been working on Free Software for the past fifteen years - most of them unknown, discontinued by now; a huge part of this work was a hobby: small, infrequent contributions. But it was, and still is, something I enjoy doing, something that satisfies my hunger for knowledge, my desire to give back to the community I received so much from. Over the years, what originally started as a desire became a need. What started as a preference, became a requirement. And all through these years, I had a dream, a clear goal I wanted to achieve, which gave purpose to a lot of things I wrote and worked on.This past Monday, the goal was reached. I left the BalaBit support department this week, and went for greener pastures, to do what I wanted to do since I became a free software fanatic: work on free software for a living. As of July first, I'm officially a syslog-ng OSE developer, working full time on 100% Free Software. And not just any software, but something used by millions all over the world. For the first time in my life, I am able to do what I want, for a living.So how did I end up here?It all started on a boring night in December, 2010, when I wrote the first version of the mongodb destination, which also marks my entrance into the syslog-ng contributor scene (apart from a set of bugfixes I was doing in my role as a software engineer at support). From this grew an agreement between the support and development departments (and myself, of course), that I can spend 20% of my time working on syslog-ng OSE. For a while, I spent my Fridays doing just that, eventually obtaining another hat: maintaining the 3.3 stable branch, and later the 3.4 branch too. Meanwhile, the 20% percent was reconsidered, and became 50%, with the premise that it will become 100% in the not too distant future. That future is now.That future is now, and while there will be a slower ramp-up time, there are great things in store, of which, I'll talk about in a later post. Needless to say, though, even if I achieved a dream, there are other desires to pursue, and I'm not going to spend fifteen years achieving them: it's time to fire up the rocket boots, and see where they take me.Two years. Watch this place.