Search Results: "tar"

30 July 2012

Russell Coker: Links July 2012

The New York Times has an interesting article about hacker hostels [1]. I had an idea for similar things after watching a Japanese movie about Tokiwa-s a shared apartment for Manga artists which among others inspired the creator of Astro Boy [2]. The TED blog has an interesting interview with William Noel about open access to art and historical data [3]. Most of his talk concerns an Archimedes codex which has been recovered and published on the Internet. He advocates publishing all manner of art and historical data under a Creative Commons license. The education system is often criticised for trying too hard to make children feel successful and not teaching them skills needed to be successful, it seems that the US military fails the same way in it s war games [4]. Webroot.com published an interesting article last year about the first BIOS rootkit in the wild [5]. I really wish that they would design motherboards with a switch to enable BIOS writing which would default to off . I recently did a poll at a LUG meeting and found that only half the audience had updated the BIOS on most systems they owned, if the most technical people generally don t need a dangerous feature then it should probably be disabled by default. Matthew Wright wrote an interesting article about the costs of upgrading the electricity grid in Australia vs the costs of upgrading air-conditioners [6]. It seems that it would be a lot cheaper for the government to buy everyone a new air-conditioner than to upgrade the grid. Owl City has a post of 10 Myths About Introverts [7]. That could probably be titled 10 Myths About Aspies and still be correct. Susan Cain gave the most popular talk of TED 2012 about The Power of Introverts , here is an interesting interview about the talk and Introversion [8]. Related posts:
  1. Links April 2012 Karen Tse gave an interesting TED talk about how to...
  2. Links July 2011 The Reid Report has an article about the marriage pledge...
  3. Links March 2012 Washington s Blog has an informative summary of recent articles about...

Johannes Schauer: port bootstrap build-ordering tool report 4

A copy of this post is sent to soc-coordination@lists.alioth.debian.org as well as to debian-bootstrap@lists.mister-muffin.de.

Diary

July 2
  • playing around with syntactic dependency graphs and how to use them to flatten dependencies

July 4
  • make work with dose 3.0.2
  • add linux-amd64 to source architectures
  • remove printing in build_compile_rounds
  • catch Not_found exception and print warning
  • use the whole installation set in crosseverything.ml instead of flattened dependencies
  • detect infinite loop and quit in crosseverything.ml
  • use globbing in _tags file
  • use wildcards and patsubst in makefile

July 5
  • throw a warning if there exist binary packages without source packages
  • add string_of_list and string_of_pkglist and adapt print_pkg_list and print_pkg_list_full to use them
  • fix and extend flatten_deps - now also tested with Debian Sid

July 6
  • do not exclude the crosscompiled packages from being compiled in crosseverything.ml
  • clean up basebuildsystem.ml, remove old code, use BootstrapCommon code
  • clean up basenocycles.ml, remove unused code and commented out code
  • add option to print statistics about the generated dependency graph
  • implement most_needed_fast_wrong as well as most_needed_slow_correct and make both available through the menu

July 7
  • allow to investigate all scc, not only the full graph and the scc containing the investigated package
  • handle Not_found in src_list_from_bin_list with warning message
  • handle the event of the whole archive actually being buildable
  • replace raise Failure with failwith
  • handle incorrectly typed package names
  • add first version of reduced_dist.ml to create a self-contained mini distribution out of a big one

July 8
  • add script to quickly check for binary packages without source package
  • make Debian Sid default in makefile
  • add *.d.byte files to .gitignore
  • README is helpful now
  • more pattern matching and recursiveness everywhere

July 9
  • fix termination condition of reduced_dist.ml
  • have precise as default ubuntu distribution
  • do not allow to investigate an already installable package

July 10
  • milestone: show all cycles in a graph
  • add copyright info (LGPL3+)

July 11
  • advice to use dose tools in README

July 16
  • write apt_pkg based python filter script replacing grep-dctrl

July 17
  • use Depsolver.listcheck more often
  • add dist_graph.ml
  • refactor dependency graph code into its own module

July 18
  • improve package selection for reduced_dist.ml
  • improve performance of cycle enumeration code

July 20
  • implement buildprofile support into dose3

July 22
  • let dist_graph.ml use commandline arguments

July 23
  • allow dose3 to generate source package lists without Build- Depends Conflicts -Indep

July 29
  • implement crosscompile support into dose3

Results

Readme There is not yet a writeup on how everything works and how all the pieces of the code work together but the current README file provides a short introduction on how to use the tools.
  • build and runtime dependencies
  • compile instructions
  • execution examples for each program
  • step by step guide how to analyze the dependency situation
  • explanation of general commandline options
A detailed writeup about the inner workings of everything will be part of a final documentation stage.

License All my code is now released under the terms of the LGPL either version 3, or (at your option) any later version. A special linking exception is made to the license which can be read at the top of the provided COPYING file. The exception is necessary because Ocaml links statically, which means that without that exception, the conditions of distribution would basically equal GPL3+.

reduced_dist.ml Especially the Debian archive is huge and one might want to work on a reduced selection of packages first. Having a smaller selection of the archive would be significantly faster and would also not add thousands of packages that are not important for an extended base system. I call a reduced distribution a set of source packages A and a set of binary packages B which fulfill the following three properties:
  • all source packages A must be buildable with only binary packages B being available
  • all binary packages B except for architecture:all packages must be buildable from source packages A
The set of binary packages B and source packages A can be retrieved using the reduced_dist program. It allows to either build the most minimal reduced distribution or one that includes a certain package selection. To filter out the package control stanzas for a reduced distribution from a full distribution, I originally used a call to grep-dctrl but later replaced that by a custom python script called filter-packages.py. This script uses python-apt to filter Packages and Sources files for a certain package selection.

dist_graph.ml It soon became obvious that there were not many independent dependency cycle situation but just one big scc that would contain 96% of the packages that are involved in build dependency cycles. Therefor it made sense to write a program that does not iteratively build the dependency graph starting from a single package, but which builds a dependency graph for a whole archive.

Cycles I can now enumerate all cycles in the dependency graph. I covered the theoretical part in another blog post and wrote an email about the achievement to the list. Both resources contain more links to the respective sourcecode. The dependency graph generated for Debian Sid has 39486 vertices. It has only one central scc with 1027 vertices and only eight other scc with 2 to 7 vertices. All the other source and binary packages in the dependency graph for the archive are degenerate components of length one. Obtaining the attached result took 4 hours on my machine (Core i5 @ 2.53GHz). 1.5 h of that were needed to build the dependency graph, the other 2.5 hours were needed to run johnson's algorithm on the result. Memory consumption of the program was at about 700 MB. It is to my joy that apparently the runtime of the cycle finding algorithm for a whole Debian Sid repository as well as the memory requirements are within orders of magnitude that are justifiable when being run on off-the-shelf hardware. It must also be noted that nothing is optimized for performance yet. A list of all cycles in Debian Sid up to length 4 can be retrieved from this email. This cycle analysis assumes that only essential packages, build-essential and dependencies and debhelper are available. Debhelper is not an essential or build-essential package but 79% of the archive build-depends on it. The most interesting cycles are probably those of length 2 that need packages that they build themselves. Noticeable examples for these situations are vala, python, mlton, fpc, sbcl and ghc. Languages seem love to need themselves to be built.

Buildprofiles There is a long discussion of how to encode staged build dependency information in source packages. While the initial idea was to use Build-Depends-StageN fields, this solution would duplicate large parts of the Build-Depends field, which leads to bitrot as well as it is inflexible to possible other build "profiles". To remedy the situation it was proposed to use field names like Build-Depends[stage1 embedded] but this would also duplicate information and would break with the rfc822 format of package description files. A document maintained by Guillem Jover gives even more ideas and details. Internally, Patrick and me decided for another idea of Guillem Jover to annotate staged build dependencies. The format reads like:
Build-Depends: huge (>= 1.0) [i386 arm] <!embedded !bootstrap>, tiny
So each build profile would follow a dependency in <> "brackets" an have a similar format as architecture options. Patrick has a patch for dpkg that implements this functionality while I patched dose3.

Dropping Build-Depends-Indep and Build-Conflicts-Indep When representing the dependencies of a source package, dose3 concatenates its Build-Depends and Build-Depends-Indep dependency information. So up to now, a source package could only be compiled, if it manages to compile all of its binary packages including architecture:all packages. But when bootstrapping a new architecture, it should be sufficient to only build the architecture dependent packages and therefor to only build the build-arch target in debian/rules and not the build-indep target. Only considering the Build-Depends field and dismissing the Build-Depends-Indep field, reduced the main scc from 1027 vertices to 979 vertices. The amount of cycles up to length four reduced from 276 to 206. Especially the cycles containing gtk-doc-tools, doxygen, debiandoc-sgml and texlive-latex-base got much less. Patrick managed to add a Build-Depends-Indep field to four packages so far which reduced the scc further by 14 vertices down to 965 vertices. So besides staged build dependencies and cross building there is now a third method that can be applied to break dependency cycles: add Build-Depends-Indep information to them or update existing information. I submitted a list of packages that have a binary-indep and/or a build-indep target in their debian/rules to the list. I also submitted a patch for dose3 to be able to specify to ignore Build-Depends-Indep and Build-Conflicts-Indep information.

Dose3 crossbuilding So far I only looked at dependency situations in the native case. While the native case contains a huge scc of about 1000 packages, the dependency situation will be much nicer when cross building. But dose3 was so far not able to simulate cross building of source packages. I wrote a patch that implements this functionality and will allow me to write programs that help analyze the cross-situation as well.

Debconf Presentation Wookey was giving a talk at debconf12 for which I was supplying him with slides. The slides in their final version can be downloaded here

Future Patrick maintains a list of "weak" build dependencies. Those are dependencies that are very likely to be droppable in either a staged build or using Build-Depends-Indep. I must make use of this list to make it easier to find packages that can easily be removed of their dependencies. I will have to implement support for resolving the main scc using staged build dependencies. Since it is unlikely that Patrick will be fast enough in supplying me with modified packages, I will need to create myself a database of dummy packages. Another open task is to allow to analyze the crossbuilding dependency situation. What I'm currently more or less waiting on is the inclusion of my patches into dose3 as well as a decision on the buildprofile format. More people need to discuss about it until it can be included into tools as well as policy. Every maintainer of a package can help making bootstrapping easier by making sure that as many dependencies as possible are part of the Build-Depends-Indep field.

29 July 2012

Joey Hess: build a dynamic webapp in Yesod in just three days

This screencast shows what I built. Scroll down for my Yesod braindump. <video controls="controls" src="http://joeyh.name/screencasts/git-annex-webapp.ogg"></video> video link I've been astonished how quickly this went together. This is my first time using any sort of web framework, and I used a still unusual one, Yesod. It's my first time using Bootstrap too. It's also the first time I've done any AJAX programming! Bootstrap was something I'd heard of and seen some suspiciously similar looking sites built with, but it was really a pleasant surprise. Being able to make things that work and look good on the web without struggling with CSS is such a nice change. For the first time, it makes the web feel like a UI toolkit to me. As I've learned Yesod, I kept seeing problems the web framework solves that I either solved in an ad-hoc basis in Ikiwiki, or didn't realize were problems. In the former group are things like having a consistent way to store a URL to redirect to after an action (such as logging in) finishes, and generating urls programmatically in ways that avoid broken links. In the latter group are things like exploiting types to guarantee mistakes can't be made. And also the abstraction of widgets, which combine html, css, and javascript, in a way that can be composed with other widgets. Overall, I'm really enjoying Yesod, and it's making me productive in new ways. I also see a lot of potential in Yesod to improve from where it is.
    webpage = bootstrap Dynamic $ do
        setTitle "FooCorp"
        login <- getLogin
        navbar [FixedTop] $ do
            brand "FooCorp"
            link AboutR
            link BlogR
            nav [PullRight] $
                link . maybe LoginR ProfileR login
        div [ContainerFluid] $ content login
        where
            content Nothing = heroUnit $ do
                para $ "We're the FooCorp for you."
                button "Register Today" [Primary, Large] SignUpR
                carousel
                    [ amazingFeatures
                    , aboutFooCorp
                    , pricing
                    ]
            content (Just user)  = do
                para $ "Welcome back " ++ name user ++ "!"
                showProfile user

Gregor Herrmann: RC bugs 2012/27-30

during the last weeks I was quite busy with other things (like DebCamp, DebConf, & vacations), so this is a report covering 4 weeks. at least I managed to catch up during the last days a bit

28 July 2012

Vincent Bernat: Switching to the awesome window manager

I have happily used FVWM as my window manager for more than 10 years. However, I recently got tired of manually arranging windows and using the mouse so much. A window manager is one of the handful pieces of software getting in your way at every moment which explains why there are so many of them and why we might put so much time in it. I decided to try a tiling window manager. While i3 seemed pretty hot and powerful (watch the screencast!), I really wanted something configurable and extensible with some language. So far, the common choices are: I chose awesome, despite the fact that StumpWM vote for Lisp seemed a better fit (but it is more minimalist). I hope there is some parallel universe where I enjoy StumpWM. Visually, here is what I got so far: awesome dual screen setup

Awesome configuration Without a configuration file, awesome does nothing. It does not come with any hard-coded behavior: everything needs to be configured through its Lua configuration file. Of course, a default one is provided but you can also start from scratch. If you like to control your window manager, this is somewhat wonderful. awesome is well documented. The wiki provides a FAQ, a good introduction and the API reference is concise enough to be read from the top to the bottom. Knowing Lua is not mandatory since it is quite easy to dive into such a language. I have posted my configuration on GitHub. It should not be used as is but some snippets may be worth to be stolen and adapted into your own configuration. The following sections put light on some notable points.

Keybindings Ten years ago was the epoch of scavanger hunts to recover IBM Model M keyboards from waste containers. They were great to type on and they did not feature the infamous Windows keys. Nowadays, this is harder to get such a keyboard. All my keyboards now have Windows keys. This is a major change with respect to configure a window manager: the left Windows key is mapped to Mod4 and is usually unused by most applications and can therefore be dedicated to the window manager. The main problem with the ability to define many keybindings is to remember the less frequently used one. I have monkey-patched awful.key module to be able to attach a documentation string to a keybinding. I have documented the whole process on the awesome wiki. awesome online help

Quake console A Quake console is a drop-down terminal which can be toggled with some key. I was heavily relying on it in FVWM. I think this is still a useful addition to any awesome configuration. There are several possible solutions documented in the awesome wiki. I have added my own1 which works great for me. Quake console

XRandR XRandR is an extension which allows to dynamically reconfigure outputs: you plug an external screen to your laptop and you issue some command to enable it:
$ xrandr --output VGA-1 --auto --left-of LVDS-1
awesome detects the change and will restart automatically. Laptops usually come with a special key to enable/disable an external screen. Nowadays, this key does nothing unless configured appropriately. Out of the box, it is mapped to XF86Display symbol. I have associated this key to a function that will cycle through possible configurations depending on the plugged screens. For example, if I plug an external screen to my laptop, I can cycle through the following configurations:
  • only the internal screen,
  • only the external screen,
  • internal screen on the left, external screen on the right,
  • external screen on the left, internal screen on the right,
  • no change.
The proposed configuration is displayed using naughty, the notification system integrated in awesome. Notification of screen reconfiguration

Widgets I was previously using Conky to display various system-related information, like free space, CPU usage and network usage. awesome comes with widgets that can fit the same use. I am relying on vicious, a contributed widget manager, to manage most of them. It allows one to attach a function whose task is to fetch values to be displayed. This is quite powerful. Here is an example with a volume widget:
local volwidget = widget(  type = "textbox"  )
vicious.register(volwidget, vicious.widgets.volume,
         '<span font="Terminus 8">$2 $1%</span>',
        2, "Master")
volwidget:buttons(awful.util.table.join(
             awful.button(   , 1, volume.mixer),
             awful.button(   , 3, volume.toggle),
             awful.button(   , 4, volume.increase),
             awful.button(   , 5, volume.decrease)))
You can also use a function to format the text as you wish. For example, you can display a value in red if it is too low. Have a look at my battery widget for an example. Various widgets

Miscellaneous While I was working on my awesome configuration, I also changed some other desktop-related bits.

Keyboard configuration I happen to setup all my keyboards to use the QWERTY layout. I use a compose key to input special characters like . I have also recently use Caps Lock as a Control key. All this is perfectly supported since ages by X11 I am also mapping the Pause key to XF86ScreenSaver key symbol which will in turn be bound to a function that will trigger xautolock to lock the screen. Thanks to a great article about extending the X keyboard map with xkb, I discovered that X was able to switch from one layout to another using groups2. I finally opted for this simple configuration:
$ setxkbmap us,fr '' compose:rwin ctrl:nocaps grp:rctrl_rshift_toggle
$ xmodmap -e 'keysym Pause = XF86ScreenSaver'
I switch from us to fr by pressing both left Control and left Shift keys.

Getting rid of most GNOME stuff Less than one year ago, to take a step forward to the future, I started to heavily rely on some GNOME components like GNOME Display Manager, GNOME Power Manager, the screen saver, gnome-session, gnome-settings-daemon and others. I had numerous problems when I tried to setup everything without pulling the whole GNOME stack. At each GNOME update, something was broken: the screensaver didn t start automatically anymore until a full session restart or some keybindings were randomly hijacked by gnome-settings-daemon. Therefore, I have decided to get rid of most of those components. I have replaced GNOME Power Manager with system-level tools like sleepd and the PM utilities. I replaced the GNOME screensaver with i3lock and xautolock. GDM has been replaced by SLiM which now features ConsoleKit support3. I use ~/.gtkrc-2.0 and ~/.config/gtk-3.0/settings.ini to configure GTK+. The future will wait.

Terminal color scheme I am using rxvt-unicode as my terminal with a black background (and some light transparency). The default color scheme is suboptimal on the readability front. Sharing terminal color schemes seems a popular activity. I finally opted for the derp color scheme which brings a major improvement over the default configuration. Comparison of terminal color schemes I have also switched to Xft for font rendering using DejaVu Sans Mono as my default font (instead of fixed) with the following configuration in ~/.Xresources:
Xft.antialias: true
Xft.hinting: true
Xft.hintstyle: hintlight
Xft.rgba: rgb
URxvt.font: xft:DejaVu Sans Mono-8
URxvt.letterSpace: -1
The result is less crisp but seems a bit more readable. I may switch back in the future. Comparison of terminal fonts

Next steps My reliance to the mouse has been greatly reduced. However, I still need it for casual browsing. I am looking at luakit a WebKit-based browser extensible with Lua for this purpose.

  1. The console gets its own unique name. This allows awesome to reliably detect when it is spawned, even on restart. It is how the Quake console works in the mod of FVWM I was using.
  2. However, the layout is global, not per-window. If you are interested by a per-window layout, take a look at kbdd.
  3. Nowadays, you cannot really survive without ConsoleKit. Many PolicyKit policies do not rely on groups any more to grant access to your devices.

Jon Dowland: books

I'm forty pages into a battered copy of "Complicity" by Iain Banks. I picked this copy up for the princely sum of 90p at Barter Books last weekend. For the uninitiated, Barter Books is a grand old bookshop situated in a disused train station in Alnwick, Northumberland. It has the dubious pleasure of being the originator of the "keep calm and carry on" fad. At the time that I bought it, I wasn't sure whether it was already in my swollen reading pile, but 90p wasn't a great loss if so. I could leave it in the book pile at my next hotel. (it was, and that's a sorry indictment of the state of my reading pile.) This gnarled paperback has character. There's a faded price sticker on the back revealing a first-hand price in Australian Dollars. The book started its post-sale life on the other side of the world. Where else has it been? I take some small pleasure in planning the next step in its future. To imagine this book with a long tail of readers having a free market value of 90p, or roughly the price of a posh chocolate bar, I'm struck by both wondering how publishers and writers like Mr Banks make a living (and yet they do) but also how the pricing of ebooks, completely lacking in this character I've written about and non-transferable, seems over inflated to the end user. I've been struggling to short list a selection of books to take on holiday. Most of my friends have responded "buy a kindle", with a complete lack of sympathy for my predicament. I'm borrowing one which means I can finally find out whether I'll "take" to it. I'm concerned I'll not be able to get sucked into a story in the same way I can with a paper book. I certainly won't rely on just the kindle for my reading needs. I don't want to prejudge my experiment too much but I suspect I won't be convinced. Where I can see technology supporting my reading is in addition to the paper book: particularly for non fiction, technical books, academic papers, more graphical-oriented things (photo books, graphic novels). I do have a small but growing collection of old, rare-in-print texts that I've only been able to source as PDFs. I think something like the Google Nexus 7 would do the job for these and fit into my life more readily than the kindle.

John Goerzen: How to get started programming?

I have been asked for advice from several people recently on how to get started programming, or how to further develop a nascent interest in coding or software engineering. The people asking the questions range in age from about 10 years old to older than me. These are people that, for various reasons, are not very easily able to take computer science courses right now. One would think that, since I ve been doing this for somewhere around a quarter century (oh I do feel old now), that I d be ready to offer up some great advice. And offer some suggestions I have. But I m not convinced they re good ones. I have two main tensions. The first is that I, like many in the communities I tend to hang out in such as Debian s, have a personality that leads me to take a deep dive into details of anything that holds my interest. Whether it s Linux, Haskell, or amateur radio, I want to do more than skim the surface if I m having fun with it. Many people are not like that. They may have a lot of fun programming in Visual Basic, not really caring that other languages are out there. Or some people are not like this yet. I feel unqualified to provide good advice to people that are different from me in that way. To put it a different way: most people don t want to wait 4 years to be useful, and want to start out right away and get better over time (and I was the same way too.) The second is related. I learned programming at a time when, other than BASIC, interpreted languages were not really available to me. (Yes, they were available, but not to me.) I cut my teeth on BASIC, Pascal, and C. Although I rarely use C anymore, I can still drop into it at a moment s notice and be perfectly comfortable. I feel it was a fundamentally valuable experience, and that it would be very hard to become a great programmer without ever having lived and breathed something like C, where memory and pointers must be managed manually. Having said that, it is probably possible to become a good coder without ever having touched C. Here, then, is an edited version of some rambly advice I sent to someone recently, where learning OOP was particularly mentioned. I would welcome your comments and suggestions. I may point people that ask to this post in the future. For simply learning how to write code, Dive Into Python has long been a decent resource, though it may assume more experience than some have. I haven t read them myself, but I ve also heard good things about the How to Think Like a Computer Scientist series from Green Tea Press. They re all available as free PDF downloads, too! Eric S. Raymond s The Art of Unix Programming is another work I ve heard good things about, despite having never read it myself. A quick glance at the table of contents makes me think that even if people don t wind up working on Unix, the lessons and philosophy should be informative. It seems that many Computer Science programs are using Java for the core of their instruction, or even almost exclusively. Whether that is good or bad, I m not completely sure. It certainly gets people into OOP more deeply, but I m a right tool for the job kind of person. Despite the hype, OO like everything else isn t the right tool for every job. It is fine for people to dive straight into OO and become good programmers/engineers. However, I think it would be difficult to become a great programmer/engineer without ever having a solid understanding of a more low-level language, such as C in particular. I did my CS work when it was mostly based in C, and am glad for it. If someone never has to manage memory or pointers, I suspect they will be at a disadvantage in the long run for not being able to understand or work with the system at a more fundamental level. If a person knows C, plus some concepts of OO and Functional Programming (FP), it should be easy to pick up just about any other language out there. I used to think Python was a great first language, but during the 2.x series they added so much fluff and so many special cases that I m less enthusiastic now, though I don t know how much of that got cleaned up in 3.x. I am not too keen on Java as a first language, because too many things that should be simple aren t. I have a fondness for Haskell, and its close relationship to mathematics could make it a great first language or maybe a poor one, depending on your perspective. One other thing I think it s important for good programmers to have experience with all three major models of programming (procedural, OO, functional.) Even if a person winds up working mostly in one universe, knowledge of and experience with the others is important and informative and, in my experience, leads to better algorithms and architecture all around.

Luke Faraone: Simple two-factor authentication



Do you want to allow "Rebooting of Cluster 19" as Jane Doe, Foo Corp? Yes/NoI've been thinking a lot recently about information security and passwords. It's widely agreed that two-factor authentication, which combines something-you-know (a password) with something you have (eg. a smart card or some one-time-password) is superior to using a password by itself. Most solutions I've seen, short of client-side certificates, both require a trusted third party and provide little auditability.

I'm working on a project which attempts to address those problems. Basically, the idea is that you attempt to perform an action on a website, like logging in, moving some money between accounts, or restarting a cluster. Then, your mobile device ((iPhone used in the graphic to the right because a SVG was handy. Image adapted from work by Virgile Pypaert [CC-BY-SA-3.0], via Wikimedia Commons)) (Android, iPhone, or some custom hardware) starts buzzing, displaying the text of the action. When you approve or deny the action, your device uses a locally stored private key to sign the text of the approval and send it back to the server. The server checks that you approved the same transaction number and text, then allows the action. The server can store the approval notice to allow auditors later to determine that somebody in possession of the private key signed the action request.

So far, I've written a small Django server application, and a Python CLI device and client. They're not yet ready for a public release; I need to write up installation instructions and test it a bit more, but I'll publish them soon on Launchpad. On the usability side of things, I plan to conduct some trials in January comparing it to things like Google Authenticator.

This should be less susceptible to MITM attacks than one-time-passwords, since you can authenticate specific transactions, rather than entire sessions.

On the other hand, I'm no security researcher. Dear lazyweb, are there any problems with the above approach? Also, if I'm going to release this (as I plan to once I get something working), I need a name. Ideas?

27 July 2012

Julien Danjou: Ceilometer, the OpenStack metering project

For the last months, I've been working on a metering project for OpenStack, so it's time to talk a bit about it. OpenStack is a growing cloud platform providing IaaS. A problem easily identified by everyone building a public cloud platform is that nothing is provided to retrieve the platform usage data. Some data are available in some places, but not everything is, and you have to do a lot of processing from the various components to get something useful in the end. But in order to bill customers that are using your public cloud platform, you need to dot his. In this regard, a lot of companies running public OpenStack based infrastructure wrote their own solution to cover this functional areas, and to become able to bill theirs customers. To avoid everybody doing and maintaining such a stack in their corners, the Ceilometer has been created. The project aims to cover the metering aspect of the OpenStack components, pulling usage data from every components and storing them into a single place. It then offer a retrieving point for this data via a REST API. The initial specifications have been written in April this year, and actual implementation started in May. The project is currently worked on by me, Dreamhost and Canonical. We already have designed an architecture that we are implementing, and we hope to release a first usable version with Folsom. Ceilometer architecture I did a presentation of this project yesterday at XLCloud, which has been very well received. <iframe allowfullscreen="true" frameborder="0" height="389" mozallowfullscreen="true" src="https://docs.google.com/presentation/embed?id=11ALGC4xuWcRvXKTSnnsteJUkArsTfQW-7IAfWRRI5kQ&amp;start=false&amp;loop=false&amp;delayms=3000" webkitallowfullscreen="true" width="480"></iframe> If you are interested in helping us and contributing, feel free to join us during one of our weekly IRC meeting or fix some bugs. :-)

Keith Packard: sandybridge-caching

Playing around with LLC controls on Sandybridge today. What I wanted to know was how caching backbuffers affects application performance. The reason for this question is that when the driver uses page flipping to display a new frame, we force the new scanout buffer to be uncached because the display hardware only scans out from main memory. However, when we go back to using this buffer as the next back-buffer, we don t turn the caching bits back on as that requires writing GTT entries to change caching modes. This means that swapping via page-flipping and swapping via copying has a large change in main memory access patterns during rendering page flipping applications use an uncached back buffer while copying applications use a cached back buffer. What effect does the back-buffer caching-mode have on application performance? The changes In the kernel, I have three modes 1) Existing code. Set buffer objects to cache mode none when preparing them as scanout buffers. The first time the application flips to a back buffer, caching is disabled on that buffer forever 2) Flip back and forth. Set buffers to cache mode none when used as scanout. When that stops, flip them back to cache mode llc. This involves a whole lot of clflushing and GTT rewriting, obviously. 3) Flush to memory, but don t disable caching. In this mode, the buffer gets flushed (from both CPU and GPU) before being used as a scanout, but the cache mode isn t changed. Benchmarking with glxgears The first test was pretty simple I hacked up glxgears (yes, glxgears is not a benchmark) to redraw the same frame multiple times (including the clear) per swap to provide a scalable amount of work per swap. No pixel operations, so this is just writing to the back buffer. Thes numbers are the median of 5 runs, except for those obviously limited by the refresh rate.
RepsExistingFlipFlush
10606060
20606060
4042.07034.11337.130
8021.25917.95718.658
Note the linear scaling over reps (once you get below refresh). I think this means that the swap overhead is in the noise, including flushing and rewriting GTT entries. Given that my screen is (yuck) 1366x768 pixels, or just over 4MB, and my total LLC is 4MB, it s seems like the performance changes here are likely caused by the rendering ejecting every other object from the cache. What s wrong with benchmarking with glxgears? Of course, glxgears is not a benchmark, and I realized while collecting this data that, in particular, it never reads from the backbuffer, and most pixels are only written once by the clear operation at the start of the rendering code. So, it forces everything else out of the cache at the start of each drawing cycle, but doesn t stick anything useful into the cache. Benchmarking nexuiz, a (more) real application For the next test, I ran nexuiz demo1.dem in all three modes. Numbers out of that weren t nearly as stable as glxgears, and I m afraid I didn t write all of them down, but I got numbers between 45 and 50 frames per second in all three modes. None significantly faster, none significantly slower. So, for these two tests, caching has no positive effect on overall rendering performance. Obviously, I need to collect data from more applications to see if the effect is general. I sure hope so, because the alternative will be to find some heuristic to direct when to enable caching. The patch This is on top of kernel version 3.5.
diff --git a/drivers/gpu/drm/i915/i915_drv.h b/drivers/gpu/drm/i915/i915_drv.h
index b0b676a..f69fba8 100644
--- a/drivers/gpu/drm/i915/i915_drv.h
+++ b/drivers/gpu/drm/i915/i915_drv.h
@@ -1335,6 +1335,10 @@ int __must_check
 i915_gem_object_pin_to_display_plane(struct drm_i915_gem_object *obj,
                     u32 alignment,
                     struct intel_ring_buffer *pipelined);
+
+void
+i915_gem_object_unpin_from_display_plane(struct drm_i915_gem_object *obj);
+
 int i915_gem_attach_phys_object(struct drm_device *dev,
                struct drm_i915_gem_object *obj,
                int id,
diff --git a/drivers/gpu/drm/i915/i915_gem.c b/drivers/gpu/drm/i915/i915_gem.c
index 288d7b8..b606bd2 100644
--- a/drivers/gpu/drm/i915/i915_gem.c
+++ b/drivers/gpu/drm/i915/i915_gem.c
@@ -2851,6 +2851,9 @@ int i915_gem_object_set_cache_level(struct drm_i915_gem_object *obj,
    return 0;
  
+#define JUST_FLUSH 0
+#define FLIP_CACHING   0
+
 /*
  * Prepare buffer for display plane (scanout, cursors, etc).
  * Can be called from an uninterruptible phase (modesetting) and allows
@@ -2883,7 +2886,11 @@ i915_gem_object_pin_to_display_plane(struct drm_i915_gem_object *obj,
     * of uncaching, which would allow us to flush all the LLC-cached data
     * with that bit in the PTE to main memory with just one PIPE_CONTROL.
     */
+#if JUST_FLUSH
+   ret = i915_gem_object_finish_gpu(obj);
+#else
    ret = i915_gem_object_set_cache_level(obj, I915_CACHE_NONE);
+#endif
    if (ret)
        return ret;
@@ -2913,6 +2920,16 @@ i915_gem_object_pin_to_display_plane(struct drm_i915_gem_object *obj,
    return 0;
  
+void
+i915_gem_object_unpin_from_display_plane(struct drm_i915_gem_object *obj)
+ 
+   i915_gem_object_unpin(obj);
+#if FLIP_CACHING
+   if (HAS_LLC(obj->base.dev))
+       i915_gem_object_set_cache_level(obj, I915_CACHE_LLC);
+#endif
+ 
+
 int
 i915_gem_object_finish_gpu(struct drm_i915_gem_object *obj)
  
diff --git a/drivers/gpu/drm/i915/intel_display.c b/drivers/gpu/drm/i915/intel_display.c
index a8538ac..76a2012 100644
--- a/drivers/gpu/drm/i915/intel_display.c
+++ b/drivers/gpu/drm/i915/intel_display.c
@@ -1821,7 +1821,7 @@ err_interruptible:
 void intel_unpin_fb_obj(struct drm_i915_gem_object *obj)
  
    i915_gem_object_unpin_fence(obj);
-   i915_gem_object_unpin(obj);
+   i915_gem_object_unpin_from_display_plane(obj);
  
 static int i9xx_update_plane(struct drm_crtc *crtc, struct drm_framebuffer *fb,

Paul Tagliamonte: ERMAHGIT!

After seeing this post on reddit, I ported the OP s code to ERMAHGIT after following the following (simple) instructions:
sudo ./setup.py install
then, head over to a friendly git repo, and do the following:
git config core.editor /usr/local/bin/ermahgit
Anyway, here s a few lines from some random git commits I have handy:
Added chamber to vote page heading
ERDDERD CHERMBER TO VERTE PERGE HERERDERNG
Fixing the import I just broke
FERXERNG THE ERMPERT I JERST BRERKER
Bugfix for non-ascii chars using encode
BERGFERX FER NERN-ERSCERI CHERS ERSERNG ERNCERDER
Add this sorry excuse for a manpage.
ERDD THERS SERRER ERXCERSE FER A MAHNPERGER.
Please, never, ever use this :)

25 July 2012

Vincent Sanders: Fowlers Delight?

Fowler published his paper on Continuous Integration (CI) way back in 2000. Personally at the time I was a much more naive developer, I was not really an adherent to the whole XP community and had worked for a series of companies where the idea of a release build was of a month long process and the concept that the whole codebase could be built and tested on every commit was a comical notion at best.

By the time the time the 2006 edition of the paper came out however I was convinced. Ever since then I have tried to make the projects I worked on have as many CI features as possible. It is one reason I ran the ARM Kernel autobuilder for years as an attempt to provide at least build coverage.

I especially like the way these concepts mean that releases are much less "scary" events and release anxiety is reduced to a minimum. When every commit produces something that is of release quality a project really is doing it right.

The reason I mention any of this is I have recently set up the NetSurf projects CI server. The heavy lifting is being done by a Jenkins instance running on a VPS fulfilling the "Every Commit Should Build the Mainline on an Integration Machine" part of the Fowler guidelines.

Setting up Jenkins was much less trouble than many of the previous systems I have used. I have been a little underwhelmed by the support for plain old make based projects which you have to resort to shell scripts for, but leaving that aside the rest was straightforward. The documentation could do with a purge pass for the old hudson name and the error reporting (especially under memory pressure) is not great.

And there lies my one real issue with the tool, memory. The VPS has 512MByte RAM and runs nothing beyond a web server, jenkins and the C compiler to build the project components. One would hope that was plenty of memory, doing the builds by hand it did seem gcc is happy in that space, alas Jenkins is a Java application and guzzles RAM (over 60% of RSS right now) and causes itself out of memory exceptions with distressing frequency.

The project is left with the choices:
It is such a pity too, aside from the RAM issues it is all going well and it has already encouraged us to get the toolchains for the less commonly built targets (ppc amiga and windows for a start) working.

Diego Escalante Urrelo: A Coru a, d a uno

You can follow the latest GUADEC updates from @guadec and @diegoe.

Isa, Alejo, Laura, and Germ n stealing a mug.

GUADEC just started. Or at least, the arrangements for GUADEC.

The local team is hard at work at the GUADEC HQ of the Facultade de Inform tica, they are really working really hard to make sure that there is a proper wired network setup (yes, you read that right!), a great video setup to record all the talks and publish them as soon as possible and setting up the final details of rooms, info desk, accommodation

If you are staying at the Rialta, you can take the hourly bus to the Facultade (just ask the driver). The bus departs from the frontis of Block 3. Ask the driver to let you know when to get off the bus. He is aware of GUADEC so your gringo-spanish should suffice.

We will be holding a pre-registration party tonight at 21:00 (9pm) just outside the Rialta cafeteria. If you are already here drop by for the Queimada by Alejo, our galician wizard.

Vincent Sanders: Travels with Mr. Brown

My return from Debconf12 has been tinged with a little wistfulness, I had a great time but wish I could have spent a little more time there to justify the seventeen hours travel each way. I took a lot of pictures which gave me a good record of my trip.

The talks, BOF and discussions were, as usual, very useful. The release team explaining what needed to be done for Wheezy was both informative and amusing.

The numerous BOF from Steve Mcintyre were a great source of discussion and ideas and appear to have generated progress on some quite contentious issues.

I especially enjoyed the Sylvestre Ledru talk on building the archive with clang and how this might be another useful tool in finding bugs.

Hideki Yamane gave a really useful talk "Let's shrink Debian package archive!" He gave a practical explanation on how Debian could benefit from using xz compression, where it is not appropriate and had a selection of real numbers to help the discussion. Given this was Hideki first talk at a Debconf I must congratulate him on doing an excellent job.

There were many other talks which I have not singled out here but that says nothing about their quality or usefulness, more about why I should blog immediately after an event and not leave it a week. Though the video team have managed to capture many of the talks so you can go and watch them too.

The event was well organised and the accommodation was pleasant, if a little crowded with three to a room. The hotel had a pool which was the centre for evening activities most days, though I did miss Neil McGovern (one of my room mates) unintentionally swimming in his kilt.
The lunch and dinner catering was outdoors which was novel. The food was generally good if a little limited for those of us with less straightforward dietary requirements.

Some of us did venture out to have dinner at the continental hotel on one evening for a change of scene.
There was of course the obligatory conference meal by the lakeside and an awesome day trip where I saw a mangrove swamp and (fortunately) no salt water crocodiles.
All in all I had a fabulous and productive time. I would like to thank Collabora for travel sponsorship to the event and to Neil who was a great travelling companion.

Vincent Sanders: A sad state of affairs

Surprisingly my previous post on my Debconf trip has gathered more queries over its title than anything else.

Mostly my blog titles are, if I can manage it, relevant quotes or book titles. For the last post I used the title of a collection of Mark Twain scripts surrounding his trip through Nicaragua in 1866.

I have not been able to read the publication myself beyond the quoted excerpts in more modern articles because, somehow, almost 150 years after the words were written the only access to these words is to buy a very expensive and rare 1940 publication.

This appears to stem from the fact the original Clements scripts were simply not published before 1940 and hence appear to gain copyright from that date (IANAL I might be wrong here is the source I used) and thanks to the US government effectively making copyrighted works published after 1922 be "protected" forever I may never read it at all.

Rather sad really, but if you do have a copy I could borrow...oh no that is probably illegal too? better just let the words fade to dust eh?

24 July 2012

Gunnar Wolf: Ten theses in favor of free download of cultural goods on the Internet (by Enrique G. Gallegos)

This is one of the days where reading my everyday newspaper was worth more than just getting bitter at the news. I found this text in La Jornada, my usual newspaper. I liked it very much, and decided to translate it for a wider audience. Of course, if you can read Spanish, do yourself a favor and go to the original. It is not that the text is so easy to translate. And, after all, I'm not a native English writer. I'm trying to do a literal translation, even when disagreeing with the author. Ten theses in favor of free download of cultural goods on the Internet Enrique G. Gallegos Poet and philosopher. Currently a researcher in Universidad Aut noma Metropolitana-C
  1. Historical legacy. Criticists who promote the persecution of piracy and free downloads of cultural works from the Internet argue that the patrimonial rights have been wronged; they start off suposing that a cultural work is formed from a historical void, as if there was a "nothing" to begin with, and then a "something" appeared. Nothing more fallacious: All cultural products has its precedents, and thanks to these it generates part of its best forces.
  2. Opening towards the future. Human beings are projected into the future. As the main philosophies of the 20th century have explored, one of the singular characteristics of mankind is the ability to think and imagine the future. Products of culture are the best ways to think and project society, politics, love, hope, needs, failures A song, a poem, an essay, can trigger imaginary worlds with a transformative potential. To deny this to humanity with "patrimonial damage" as an argument is to mutilate man's temporal nature.
  3. Recognition of the present. Cultural works generate moments of reflection, criticism and pleasure; they can trigger actions demanding commitment, solidarity and strong indignation against injustice. If every cultural work is adscribed in some way to a historic tradition, and if cultural products anre necessary to imagine other possible worlds and open our future, it's not less true that they also allow us to recognize our present by making us more sensitive to others' ideas, sensations and emotions. Without culture, there is no present nor others' presence.
  4. Divulgation of the cultural heritage. If culture is "heritage of mankind", it should be also spread by any possible way. But not just spread, but actively sought for the biggest number of people to be able to effectively have access to it. In a potentially interconnected world, Internet is the most adequate world for it (despite its limitations). Therefore, penalizing free downloads means avoiding spreading it and restricting access to those groups who don't have enough economic resources.
  5. Preserving cultural heritage. If we agree that cultural works are heritage of mankind, we need to take every action needed to preserve them. But preserving culture does not only mean keeping the products in museums, galleries or never-polluted drawers; it rather means to keep them in the collective memory and in the flow of constant interpretations and appropiations. Said in other words, the genuine way to preserve culture is to allow universal access to cultural goods. That's why, holding that free copies and downloads of books, music, videos, etc. over the Internet are harmful, is an incompatible argument with the obligation to preserve cultural heritage.
  6. They are not goods. Cultural works are material and spiritual manifestations of mankind, concretions of its historicity; they are, also, expressions of emotions, ideas and projections of other lives and other worlds. That's why they cannot be equated to goods or inserted in patrimonialist logic. The market might want to cheat by giving a certain price to a painting by Orozco or auctioning a Baudelaire manuscript, but it will never be able to aprehend their true meaning as cultural works. And it's not that culture is not faluable, but taht its estimation criteria do not follow the rules of market, but those of the imponderable and unlimited.
  7. Price disproportion. Assuming we accept the possibility that intermediaries charge for the services they perform, the value of a book, record or movie should never exceed the daily wage of a worker or employee. But this ellection would only be one more option within the effective possibility that people opt to download or freely copy the cultural work. The final decision as to which media to go to should be a sovereign resolution of the culture-interested person.
  8. The principle of the most benefit. Even when copies and free downloads over Internet could generate a patrimonial "wrong" to third parties, the cultural benefit obtained by it will always be greater in as much that it carries out intensely the principle to foster, spread and access the cultural heritage of mankind. To think otherwise is to privilege the few over the many.
  9. The real evil is elsewhere. Murder, women- and children-trade, poverty and misery are the real evils that ache mankind. According to the United Nations, in 2010 there were 468 thousand murders in the world; the estimation is that 3500 million people live in poverty; in their 2009 inform, the UN found over 2400 victims of "people trade", kidnapped as sex slaves; in some regions in Africa, close to 30% of children suffer acute malnutrition and six people (between children and adults) die every day from hunger. That's why pretending that free download of cultural works is an evil is clearly a tricky and deceptive speech
  10. Against the intermediary-salesman. Culture does not need intermedieries that reduce cultural works to goods. Culture is too important to leavie it in the hands of salespeople that simplistly equate cultural goods with cakes. An intermediary of this kind will never understand the difference between a work of art and a disposable razor blade. What the world needs is more support from the government from every country to artists, creators and poets, as well as ensuring the conditions for the absolute mobility freedom for cultural works.

Michal &#268;iha&#345;: Weblate hacking #2

Second day of Hackweek VIII is over and there has been again great progress on Weblate. I've started with improving documentation. Most of improvements were possible by using djangodocs extension for Sphinx, which allows better documenting of Django admin commands and settings. Then I've focused on handling languages as there were some things which I did not like. Most importantly, Weblate will now try to use existing languages even if used code does not 100% match (eg. cs_CZ and cs-CZ are treated equal). Also with help of locale aliases, some specific variants won't be created and generic one used instead (eg. cs_CZ will use cs, as these are actually same). If there will be need to differentiate them, you still can create language manually and Weblate will use matched one. Last thing is translation locking - translator can now lock translation for exclusive use and nobody else will be able to work on that. This will help in situations where more people started to work on same translation and basically duplicated the effort. There are 3 open issues targeted for 1.2 release, but I'm not sure if some of these are good ideas, let's keep the decision for next days.

Filed under: English Suse Weblate 0 comments Flattr this!

Martin Pitt: PyGObject hackfest @ GUADEC: Agenda

I started to collect some easy PyGObject bugs which are appropriate for the PyGObject hackfest at GUADEC on July 30th. These are bugs which do not need a lot of previous knowlege and are excellent starters for new contributors, such as adding overrides, fixing build system issues, etc. I also created an initial idea pool/agenda/coordination page, where participants can add or signup for things to work on. Feel free to add your own topics! I m really looking forward to GUADEC and the hackfest, see you there! GUADEC 2012

Wouter Verhelst: Runner scripts

Bert Desmet blogs about having to run a script more than once per minute, and using a runner script (started from cron) to do that. Unfortunately, his implementation contains a race condition:
if ! ps -p $(cat /path/to/runner.pid);
  then
    echo $$ > /path/to/runner.pid
If the script is accidentally started twice on approximately the same moment, chances are pretty high you'll see them both executing the ps call at approximately the same time, concluding that no other script is running, and then both executing the echo call at the same time. Result: your script will be running twice, yet you'll have only one pid file. Of course, if you do indeed run the script only from cron, and just once every 30 minutes, as in Bert's case, chances that you'll hit this race condition are pretty low. However, race conditions are bad, especially if they're easy to avoid.
set -o noclobber
if ! echo $$ > /var/run/runner.pid
then
  echo pidfile exists; exiting
  exit 0
fi
This code uses the noclobber shell option, which causes the shell to issue an error state if the file we try to overwrite with > already exists, and to do so in an atomic operation (so, no race condition there: either the file exists and we fail, or the file does not exist yet and we create it with our PID). Of course, that leaves out the check for is the other process running? Adding a second check for that is fairly complex to do right, as this cannot be done in an atomic operation; so it's usually better use exit traps to clean up the file:
set -o noclobber
if ! echo $$ > /var/run/runner.pid
then
  echo pidfile exists; exiting
  exit 0
fi
trap EXIT rm -f /var/run/runner.pid
This will fail if the system dies when the script is running, but then since we use /var/run (which is cleared on bootup, either because it's a tmpfs or because the initscripts clean it) this means the file should only ever be stale if some genius kills our process with kill -9 or kill -11.

Wouter Verhelst: DebConf 12 is over :(

As I write this, DebConf12 has actually been over for more than a week. Usually during DebConf, I'm a more proficient blogger than during other times of the year, since I'm not doing as much work that I have to remain quiet about at that time, which simply means I have more to blog about. Unfortunately, however, this year the server on which this blog runs managed to become unavailable on the second day of DebCamp, an issue that I could only fix once I was back in Belgium, which meant I haven't been able to blog as much as I would've wanted to. Anyway, all I can say is that both DebConf and DebCamp have been a huge success for me again, this year. I'd started the second rewrite of ipcfg a few weeks before DebCamp, and spent most of DebCamp whipping the basic framework into shape. It's nowhere near ready yet, but I'm confident it'll get further this time than it did before. During DebConf, I was also responsible for doing the group photo. I'd done that (together with Andrew McMillan) during DebConf8 before (if you look closely, you'll see me pressing the remote which actually triggers that picture), and I still like that picture. This time around, we didn't have a beach nearby (at least not as nearby as during DebConf8), so we had to do things slightly differently. Rather than doing one large picture, I took several pictures and used Hugin to stitch them together. Unfortunately however, some things did go wrong with the group photo. For starters, the weather wasn't very helpful, in that I noticed it was about to rain. As such, the main bunch of pictures were taken slightly before the announced time of 14:30, which meant that some people had missed it. To make matters worse, I had rushed to the group and handed the camera to someone else, so that I could gimp myself onto the picture too; but that meant that some people who were arriving as these pictures were being taken thought they were on the picture, when they really weren't. I've fixed most of these by just gimping them into the picture as well, but I might've missed some of them. If you're one of them, talk to me! Preferably now. If you're not one of them (or even if you are), hold off on doing things like printing it in large format or some such. I'll re-upload improved versions onto the same location once they're ready.

Next.

Previous.