Search Results: "will"

2 May 2013

Joey Hess: the #newinwheezy game: STM

Debian wheezy includes a bunch of excellent new Haskell libraries. I'm going to highlight one that should be interesting to non-Haskell developers, who may have struggled with writing non-buggy threaded programs in other languages: libghc-stm-dev I had given up on most threaded programs before learning about Software Transactional Memory. Writing a correct threaded program, when multiple threads needed to modify the same state, needed careful uses of locking. In my experience, locking is almost never gotten right the first time. A real life example I encountered is an app that displays a queue of files to be downloaded, and a list of files currently downloading. Starting a new download would go something like this:
startDownload = do
    file <- getQueuedFile
    startDownloadThread file
    push file currentDownLoads
But there's a point in time in which another thread, that refreshes the display, could then see an inconsistent state, where the file is in neither place. To fix this, you'd need to add lock checking around all accesses to the download queue and current downloads list, and lock them both here. (And be sure to always take the locks in the same order!) But, it's worse than that, because how is getQueuedFile implemented? If the queue is empty, it needs to wait on a file being added. But how can a file be added the queue if we've locked it in order to perform this larger startDownload operation? What should be really simple code has become really complex juggling of locks. STM deals with this in a much nicer way:
startDownload = atomically $ do
    file <- getQueuedFile
    startDownloadThread file
    push file currentDownLoads
Now the two operations are performed as one atomic transaction. It's not possible for any other thread to see an inconsistent state. No explicit locking is needed. And, getQueuedFile can do whatever waiting it needs to, also using STM. This becomes part of the same larger transaction, in a way that cannot deadlock. It might be implemented like this:
getQueuedFile = atomically $
    if empty downloadQueue
        then retry
        else pop downloadQueue
When the queue is empty and this calls "retry", STM automatically waits for the queue to change before restarting the transaction. So this blocks until a file becomes available. It does it without any locking, and without you needing to explicitly tell STM what you're waiting on. I find this beautiful, and am happier with it the more I use it in my code. Functions like getQueuedFile that run entirely in STM are building blocks that can be snapped together without worries to build more and more complex things. For non-Haskell developers, STM is also available in Clojure, and work is underway to add it to gcc. There is also Hardware Transactional Memory coming, to speed it up. Although in my experience it's quite acceptably fast already. However, as far as I know, all these other implementations of STM leave developers with a problem nearly as thorny as the original problem with locking. STM inherently works by detecting when a change is made that conflicts with another transaction, throwing away the change, and retrying. This means that code inside a STM transaction may run more than once. Wait a second.. Doesn't that mean this code has a problem?
startDownload = atomically $ do
    file <- getQueuedFile
    startDownloadThread file
    push file currentDownLoads
Yes, this code is buggy! If the download thread is started, but then STM restarts the transaction, the same file will be downloaded repeatedly. The C, Clojure, etc, STM implementations all let you write this buggy code. Haskell, however, does not. The buggy code I showed won't even compile. The way it prevents this involves, well, monads. But essentially, it is able to use type checking to automatically determine that startDownloadThread is not safe to put in the middle of a STM transaction. You're left with no choice but to change things so the thread is only spawned once the transaction succeeds:
startDownload = do
    file <- atomically $ do
        f <- getQueuedFile
        push f currentDownLoads
        return f
    startDownloadThread file
If you appreciate that, you may want to check out some other #newinwheezy stuff like libghc-yesod-dev, a web framework that uses type checking to avoid broken urls and XSS attacks, and also makes heavy use of threading, so is a great fit for using with STM. And libghc-quickcheck2-dev, which leverages the type system to automatically test properties about your program.

Russ Allbery: Review: The System of the World

Review: The System of the World, by Neal Stephenson
Series: The Baroque Cycle #3
Publisher: William Morrow
Copyright: 2004
ISBN: 0-06-052387-5
Format: Hardcover
Pages: 892
This is the third book of the three-volume Baroque Cycle. I think you could, if you really wanted, read it without reading the previous volumes; Stephenson is certainly long-winded enough that you can pick up most of what's going on while you read. It's been a year since I read the second volume, and I only resorted to Wikipedia a couple of times to remember plot elements (and mostly from the first book). However, I wouldn't recommended starting here. Many of the character relationships, and most of the underpinning of the plot, is established in the previous volumes and given more significance by them. You would also miss The Confusion, which is the best book of the series, although none of this series rises to the level at which I'd recommend it except under specific circumstances. Quicksilver establishes the characters of Daniel Waterhouse, a fictional Puritan whose family was close to Cromwell and who became a friend to Isaac Newton in the days following the Restoration; Jack Shaftoe, a vagabond who wanders Europe in a sequence of improbable adventures; and Eliza, who becomes a friend to Leibniz and a spy for William of Orange. The Waterhouse sections are prominent in Quicksilver: full of the early history of the Royal Society, alchemy, and a small amount of politics. Of those three characters, Eliza is by far the most interesting, which meant that I was delighted when The Confusion dropped Waterhouse almost entirely and mixed Eliza's further story with more improbable but entertaining sea adventures of Jack Shaftoe. You will immediately sense my root problem with The System of the World when you hear that it is almost entirely about Daniel Waterhouse. While Eliza and Jack both appear, they play supporting roles at best, and Eliza's wonderful sharp intelligence and pragmatic survival skills are left out almost entirely. Instead, this is a novel about Waterhouse's return to England after spending quite a bit of time in the American colonies working on calculating machines. He is almost immediately entangled in dangerous politics from multiple directions: the precarious national politics in England near the end of the reign of Queen Anne, Isaac Newton's attempts to maintain the currency of England as Master of the Mint, and a bombing attempt that may have been aimed at him, may have been aimed at Newton, and may have been aimed at someone else entirely. Much of the book consists of an extended investigation of this bombing plot, skullduggery involving counterfeiters, and attempts to use the currency and the Mint as part of the political conflict between Whigs and Tories, mixed in with attempts to construct a very early computer (this is Stephenson, after all). Leibniz and Eliza come into this only as confidants of the Hanoverians. All this may sound exciting, and there are parts of it that hold the attention. But this book sprawls as badly as Quicksilver did. There's just too much detail without either enough plot or enough clarity. Stephenson tries to make you feel, smell, and hear the streets of London and the concerns of an idiosyncratic group of semi-nobles during one of the more interesting junctures of British history, but he does that by nearly drowning you in it, and without providing enough high-level guidance. For most of the book, I felt like I was being given a tour of a house on my hands and knees with a magnifying glass. It's a bad sign when the reader of a historical novel is regularly resorting to Wikipedia, not to follow interesting tangents of supporting material, but to try to get a basic sense of the players and the politics involved because the author never explains them clearly. If you're more familiar with the details of British history than I am, and can more easily follow the casual intermixing of two or three forms of address for the same historical figure, you may not have that problem. But I think other structural issues remain, and one of the largest is Waterhouse himself. Jack Shaftoe, and particularly Eliza, are more interesting characters because they're characters. They're not always particularly believable, but they attack the world with panache and are constantly squirming into the center of things. Stephenson's portrayals of Newton, Leibniz, the Duke of Marlborough, Sophia of Hanover, Peter the Great, and the other historical figures who show up here are interesting for different reasons: Stephenson has history to draw on and elaborate, and it's fascinating to meet those people from a different angle than dry lists of accomplishments. History has a way of providing random details that are too bizarre to make up; Isaac Newton, for example, actually did disguise himself to infiltrate London criminal society in pursuit of counterfeiters while he was Master of the Mint! Waterhouse, for me, has none of these advantages. He is an invented character in whom I have no pre-existing interest. He drifts through events largely through personal connections, all of which seem to be almost accidental. He's welcome in the councils of the Royal Society because he's apparently a scientist, but the amount of actual science we see him doing is quite limited. His nonconformist background allies him squarely with the Whigs, but his actual position on religious matters seems much less set than the others around him. What he seems to want, more than anything else, is to help Leibniz in the development of a computer and to reconcile Newton and Leibniz. And he's not particularly effective at either. In short, he has little in the way of memorable character or dynamism, despite being the primary viewpoint character, and seems to exist mostly to know everyone and be everywhere that's important to the story. He feels like an authorial insertion more than a character. It's quite easy to believe that Stephenson himself would have loved to be in exactly the role and situation that Waterhouse finds himself in throughout the book, in the middle of the councils of the wise and powerful, in just the right position to watch the events of history. I can sympathize, but it doesn't make for engrossing reading. Novels live and die by the strength of their characters, particularly their protagonists; I want more than just a neutral viewpoint. The third major structural problem that I had with this book is that I think Stephenson buries his lede. After finishing it, I think this is a book with a point, a central premise around which all the events of the story turn, and which is the philosophical culmination of The Baroque Cycle as a whole. But Stephenson seems oddly unwilling to state that premise outright until the very end of the book. For the first half, one could be forgiven in thinking this is a story about alchemy and the oddly heavy gold that's been a part of the story since The Confusion, or perhaps about foundational but forgotten work on computation that preceded Babbage by a century. But those all turn out to be side stories, sometimes even without a proper conclusion. I appreciate honoring the intelligence of the reader, and I presume that Stephenson would like to guide the reader through the same process of realization that the characters go through, but I think he takes this much too far and fails to make the realization clear. I'll therefore state what I believe is the premise outright, since I think it's a stronger book with this idea in mind: The System of the World is a continuation of the transformational economics shown in The Confusion into the realm of politics. Specifically, it's about the replacement of people with systems, about the journey towards Parliamentary supremacy, central banking, and the persistent state, and about the application of scientific principles of consistency and reproducibility to politics and economics (however fitfully and arbitrarily). Quicksilver was about the rise of science; The Confusion was, in retrospect, about the rise of economics; and The System of the World tries to be about the rise of technocratic modern politics, barely perceptible among the squabbles between Tories and Whigs. I think that's a fascinating premise, and I would have loved to read a book that tackles it head-on. That's a concept that is much more familiar from the late 19th and early 20th centuries in the context of Marxism, early socialism, technological utopianism, and similar attempted applications of scientific analysis to political and human behavior for the betterment of human civilization. Shifting that 200 years earlier and looking at a similar question from the perspective of the giants of the Enlightenment feels full of of potential. There are moments when I think Stephenson captures the sense of a seismic shift in how economies are run, knowledge is established, and civilizations are knit together. But, most of the time, it just isn't clear. There's so much other stuff in this book, and in the whole series: so many false starts, digressions, abandoned plots, discarded characters, and awkward attempts at romance (as much as I like the characters, Stephenson's portrayal of the relationship between Eliza and Jack is simply ridiculous and not particularly funny) that the whole weight of the edifice crushes what I think is the core concept. Stephenson is never going to be sparse. When you start a Stephenson novel, you know it's going to be full of chunks of partly digested encyclopedia and random research findings that may have nothing to do with the plot. But his best books (Snow Crash, The Diamond Age, even Cryptonomicon) have an underlying structure off of which all of those digressions are hung. You can see the bones beneath the flesh, and the creature they create is one you want to get to know. I'm not sure there are any bones here, and that may be the peril, for Stephenson, of writing historical fiction. I wonder if he felt that the structure of history would provide enough structure by itself that he could wrap a few plots around the outside of it and call it good. If so, it didn't work, at least for me. A lot of things happen. Some of them are even exciting and tense. A lot of people meet, interact, and show off their views of the world. A great deal of history, research, and sense of place is described in painstaking detail. But at the end of the book, I felt like I had to reach for some sort of point and try to retrofit it to the story. Lots happened, but there wasn't a novel. And that makes it quite hard to get enthused by the book. If you adored Quicksilver, I suspect you will also like this. I think they're the most similar. If, like I did, you thought The Confusion was a significant step up in enjoyment in the series and were hoping the trend will continue, I'm sad to report that it didn't. If you were considering whether to read the whole series and were waiting to see what I thought of the end, my advice is to give The Baroque Cycle a pass unless you absolutely love Stephenson's digressions, don't care if they're about history instead of current technology, and cannot live without 3,000 pages of them. It's not that they're bad books, but they're very long books, they take a significant investment of time and attention, and I think that, for most readers, there are other books that would repay a similar investment with more enjoyment. Rating: 5 out of 10

1 May 2013

Daniel Pocock: British Telecom attacks free software with patent demands

The Register is reporting that British Telecom is now attacking the VoIP industry by claiming that it has patents over the IETF standardised Session Initition Protocol (SIP RFC 3261) SIP is widely implemented in free software, in standalone products such as Lumicall and in Linux distributions such as Debian that carry a wide variety of SIP servers, SIP proxies and SIP clients, including the default chat client Empathy It remains to be seen whether they will engage in a similar attack on related protocols like XMPP / Jabber Many of the VoIP companies under attack are heavy users of free software

Daniel Pocock: British Telecom attacks free software with patent demands

The Register is reporting that British Telecom is now attacking the VoIP industry by claiming that it has patents over the IETF standardised Session Initition Protocol (SIP RFC 3261) SIP is widely implemented in free software, in standalone products such as Lumicall and in Linux distributions such as Debian that carry a wide variety of SIP servers, SIP proxies and SIP clients, including the default chat client Empathy It remains to be seen whether they will engage in a similar attack on related protocols like XMPP / Jabber Many of the VoIP companies under attack are heavy users of free software

Antoine Beaupr : New Debian release: my contributions to the #newinwheezy game

So while I haven't work exceptionally hard to help with the Debian Wheezy release (I merely help with a few RC bugs and organised a bug squashing party), I still have a few significant contributions to the release. So here's my claim to fame to the #newinwheezy game:

Russ Allbery: Review: Questions for a Soldier

Review: Questions for a Soldier, by John Scalzi
Publisher: Subterranean
Copyright: 2005
Printing: 2012
ISBN: 1-59606-468-4
Format: Kindle
Pages: 28
Yet another short story from the Subterranean Scalzi Super Bundle collection, which is padding out my numbers this year since I don't seem to be writing as many reviews. Unlike some of the others, this one is fairly hard to come by if you didn't buy the original chapbook or the issue of Subterranean Magazine that included it. But don't worry, you're not really missing anything. Questions for a Soldier is really a short story, and slight even for that. It's a pendant to Old Man's War, a close-up of one stop in a good will tour that John Perry, the protagonist of that novel, does during it. In universe, it's basically a propaganda piece for the Colonial Defense Force that fleshes out some more of the background, mainly from the CDF side, of Scalzi's future universe. What Scalzi is doing here is mildly interesting: he's trying to capture the style of this sort of public affairs, feel-good speaking engagement but balance that against his portrayal of Perry as an honest straight shooter. Perry does a lot of the careful presentation of both sides that one would expect from someone who knows he's standing up to represent one side of an issue but doesn't want to make the propaganda too obvious. As a portrayal of a political speech of the subgenre "trying not to act like a politician," I think Scalzi successfully captures the tone. That's probably also why I found it vaguely annoying, since I think such speeches are a classic form of manipulation. Take someone (who may even honestly be a person of good will), put them up in front of a crowd, don't tell them what they have to say but make sure they're representing something that they're at least somewhat proud of, and let social pressure and expectations take care of the rest. The person will come across as honest and open (since they consider themselves to be), they don't actually know anything damaging, and they'll want to present their side in a good light. Humans are wonderful at rationalizations. The audience comes away feeling like the people they may be politically opposed to are human, and good people, and they can't be doing anything that upsetting since that fellow was a nice, upstanding person who told some great war stories and was clearly putting themselves in danger to protect the rest of us. It blunts political outrage and anger, justified or not, with a bit of "we're all on the same side and we're all good people" emotional camaraderie without any danger of any real revelations, uncomfortable information, or dangerous theories being expressed. Scalzi does a good job holding up a mirror here, and he also gives Perry the opportunity to make the solid (if quite predictable and dangerously incomplete) case for a professional military (and a completely cliched case for world, or galactic, peace). It all sounds quite believable because Perry obviously believes it. (That does not mean that Scalzi believes it; he's writing a character in a time, place, and role when the character believed those things.) But, at the end, it's a bit of political fluff that isn't really a story (no plot) and adds little to the larger series in which it was embedded. The later novels in the Old Man's War series raise the questions asked here in a much more forceful and direct way, in ways that produce at least a few answers. The general style is the sort of speech that you could find almost anywhere in US politics. If you have the collection anyway, it's mildly interesting to take apart what Scalzi is portraying here and think about it a bit, but it's completely missable. Rating: 5 out of 10

30 April 2013

Gunnar Wolf: Activities facing the next round of Trans-Pacific Partnership negotiations ( #yaratpp )

Excuse me for the rush and lack of organization... But this kind of things don't always allow for proper planning. So, please bear with my chaos ;-) What is the Trans-Pacific Partnership? Yet another secretely negotiated international agreement that, among many chapters, aims at pushing a free-market based economy, as defined by a very select few Most important to me, and to many of my readers: It includes important chapters on intellectual property and online rights. Hundreds of thousands of us along the world took part in different ways on the (online and "meat-space") demonstrations against the SOPA/PIPA laws back in February 2012. We knew back then that a similar project would attempt to bite us back: Well, here it is. Only this time, it's not only covering copyright, patents, trademark, reverse engineering, etc. TPP is basically a large-scale free trade agreement on steroids. The issue that we care about now is just one of its aspects. Thus, it's way less probable we can get a full stop for TPP as we got for SOPA. But we have to get it on the minds of as many people as possible! Learn more with this infography distributed by the EFF. Which countries? The countries currently part of TPP are Chile, Peru, New Zealand, Australia, Malaysia, Brunei, Singapore, Vietnam And, of course, the USA. Mexico, Canada and Japan are in the process of joining the partnership. A group of Mexican senators are travelling to Lima to take part of this round. Image by Colin Beardon (It's Our Future, NZ) What are we doing about it? As much as possible! I tried to tune in with Peru's much more organized call The next round of negotiations will be in Lima, Peru, between May 14 and 24. Their activities are wildly more organized than ours: They are planning a weekend-long Camping for Internet freedom, with 28 hours worth of activities. As for us, our activities will be far more limited, but I still hope to have an interesting session: Poster design by Gacela. Thanks! This Friday, we will have Aula Magna, Facultad de Ingenier a, UNAM, M xico DF, from 10AM and until 3PM. We do not have a clear speakers program, as the organization was quite rushed. I have invited several people who I know will be interesting to hear, and I expect a good part of the discussion to be a round table. I expect we will:
  1. Introduce people working on different corners of this topic
  2. Explain in some more detail what TPP is about
  3. Come up with actions we can take to influence Mexico's joining of TPP
  4. And this will be at Facultad de Ingenier a. Another explicit goal of this session will be, of course, to bring the topic closer to the students!
We want you! So... I am posting this message also as a plead for help. Do you think you can participate here? Were you among the local organizers for the anti-SOPA movement? Do you have some insight on TPP you can share? Do you have some gear to film+encode the talks? (as they will surely be interesting!) Or, is the topic just interesting for you? Well, please come and join us! So, again: Friday, 2012-05-03, 10:00-15:00
AttachmentSize
Poster. Desgin by Gacela Thanks!457.49 KB
Infography about TPP distributed by the EFF392.45 KB
"Here, let me sign this for you". Image by Colin Beardon.27.08 KB

Gunnar Wolf: Activities facing the next round of Trans-Pacific Partnership negotiations ( #yaratpp #tpp #internetesnuestra )

Excuse me for the rush and lack of organization... But this kind of things don't always allow for proper planning. So, please bear with my chaos ;-) What is the Trans-Pacific Partnership? Yet another secretely negotiated international agreement that, among many chapters, aims at pushing a free-market based economy, as defined by a very select few Most important to me, and to many of my readers: It includes important chapters on intellectual property and online rights. Hundreds of thousands of us along the world took part in different ways on the (online and "meat-space") demonstrations against the SOPA/PIPA laws back in February 2012. We knew back then that a similar project would attempt to bite us back: Well, here it is. Only this time, it's not only covering copyright, patents, trademark, reverse engineering, etc. TPP is basically a large-scale free trade agreement on steroids. The issue that we care about now is just one of its aspects. Thus, it's way less probable we can get a full stop for TPP as we got for SOPA. But we have to get it on the minds of as many people as possible! Learn more with this infography distributed by the EFF. Which countries? The countries currently part of TPP are Chile, Peru, New Zealand, Australia, Malaysia, Brunei, Singapore, Vietnam And, of course, the USA. Mexico, Canada and Japan are in the process of joining the partnership. A group of Mexican senators are travelling to Lima to take part of this round. Image by Colin Beardon (It's Our Future, NZ) What are we doing about it? As much as possible! I tried to tune in with Peru's much more organized call The next round of negotiations will be in Lima, Peru, between May 14 and 24. Their activities are wildly more organized than ours: They are planning a weekend-long Camping for Internet freedom, with 28 hours worth of activities. As for us, our activities will be far more limited, but I still hope to have an interesting session: Poster design by Gacela. Thanks! This Friday, we will have Aula Magna, Facultad de Ingenier a, UNAM, M xico DF, from 10AM and until 3PM. We do not have a clear speakers program, as the organization was quite rushed. I have invited several people who I know will be interesting to hear, and I expect a good part of the discussion to be a round table. I expect we will:
  1. Introduce people working on different corners of this topic
  2. Explain in some more detail what TPP is about
  3. Come up with actions we can take to influence Mexico's joining of TPP
  4. And this will be at Facultad de Ingenier a. Another explicit goal of this session will be, of course, to bring the topic closer to the students!
We want you! So... I am posting this message also as a plead for help. Do you think you can participate here? Were you among the local organizers for the anti-SOPA movement? Do you have some insight on TPP you can share? Do you have some gear to film+encode the talks? (as they will surely be interesting!) Or, is the topic just interesting for you? Well, please come and join us! Some more informative links BE THERE! So, again: Friday, 2012-05-03, 10:00-15:00 [Update] So, 2012-05-03 came and went. And thankfully, Alfredo was there to record most of the talk! So, you can download the video: Gunnar Wolf, Salvador Alc ntar: Qu es TPP? Por qu me debe preoucpar? Qu podemos hacer?
AttachmentSize
Poster. Design by Gacela Thanks!457.49 KB
Infography about TPP distributed by the EFF392.45 KB
"Here, let me sign this for you". Image by Colin Beardon.27.08 KB

Steve Kemp: After you've started it seems like a bad idea?

To recap: given the absence of other credible alternatives I had two options: I think there is room for a new console client, because mutt is showing its age and does feel like it should have a real extension language - be it guile, lisp, javascript(!), Lua, or something else. So I distilled what I thought I wanted into three sections: So how did I do? I wrote a ncurses-based client which has Lua backed into it. You can fully explore the sidebar-mode - which lets you select multiple folders. From there you can view the messages in a list. What you can't do is anything "real": For a two-day hack it is remarkably robust, and allowing scripting shows awesomeness. Consider this:
--
-- show all folders in the Maildir-list.
--
function all()
   -- ensure that the sidebar displays all folders
   sidebar_mode = "all";
   -- we're going to be in "maildir browsing mode"
   cmail_mode = "sidebar";
   reset_sidebar();
   refresh_screen();
end
--
-- Test code, show that the pattern-searching works.
--
-- To use this press ":" to enter the prompt, then enter "livejournal".
--
-- OR press "l" when in the sidebar-mode.
--
function livejournal()
   sidebar_pattern = "/.livejournal.2";
   sidebar_mode = "pattern";
   reset_sidebar();
   refresh_screen();
end
--
-- There is a different table for each mode.
--
keymap =  
keymap['sidebar'] =  
keymap['index']   =  
keymap['message'] =  
--
-- In the sidebar-mode "b" toggles the sidebar <-> index.
--
-- ":" invokes the evaluator.
-- "q" quits the browser and goes to the index-mode.
-- "Q" quits the program entirely.
--
keymap['sidebar'][':'] = "prompt-eval"
keymap['sidebar']['b'] = "toggle"
keymap['sidebar']['q'] = "toggle"
keymap['sidebar']['Q'] = "exit"
-- show all/unread/livejournal folders
keymap['sidebar']['a'] = "all"
keymap['sidebar']['u'] = "unread"
keymap['sidebar']['l'] = "livejournal"
Neat, huh? See the cmail.lua file on github for more details. My decision hasn't really progressed any further, though I can see that if this client were complete I'd love to use it. Its just that the remaining parts are the fiddly ones. I guess I'll re-hack mutt, and keep this on the back-burner. The code is ropey in places, but should you wish to view: And damn C is kicking my ass.

29 April 2013

Wouter Verhelst: New in wheezy: NBD named exports and installer support

Just after the release of squeeze, I released nbd 2.9.17, which had a new feature that required some backwards-incompatible change: the ability to specify an export by name, rather than by port number. Obviously, that means that wheezy will be the first release to ship with support for such named exports (although a backport was uploaded to squeeze-backports with support for such a named export). After all, names are that more obvious a way to specify an export than is a meaningless number. The init scripts and root-on-NBD support was updated, although a bugfix was denied for r0 (it will hopefully get into r1). In addition, during the wheezy cycle I finally finished the partman-nbd support in the installer. With this, it is possible to install Debian to an NBD device on diskless systems, which is nice.

Thomas Goirand: Jenkins: building debian packages after a git push (my 2cents of a howto)

The below is written in the hope it will be helpful for my fellow DDs. Why using build after push ? Simple answer: to save time, to always use a clean build environment, to automate more tests. Real answer: because you are lazy, and tired of always having to type these build commands, and that watching the IRC channel is more fun than watching the build process. Other less important answers: building takes some CPU time, and makes your computer run slower for other tasks. It is really nice that building doesn t consume CPU cycles on your workstation/laptop, and that a server does the work, not disturbing you while you are working. It is also super nice that it can maintain a Debian repository for you after a successful build, available for everyone to use and test, which would be harder to achieve on your work machine (which may be behind a router doing NAT, or even sometimes turned off, etc.). It s also kind of fun to have an IRC robot telling everyone when a build is successful, so that you don t have to tell them, they can see it and start testing your work. Install a SID box that can build with cowbuilder
Install Jenkins WARNING: Probably, before really installing, you should read what s bellow (eg: Securing Jenkins). Simply apt-get install jenkins from experimental (the SID version has some security issues, and has been removed from Wheezy, on a request of the maintainer). Normally, after installing jenkins, you can access it through:
http://<ip-of-your-server>:8080/
There is no auth by default, so anyone will be able to access your jenkins web GUI and start any script under the jenkins user (sic!). Jenkins auth
Before doing anything else, you have to enable Jenkins auth, otherwise, you have everything accessible from the outside, meaning that more or less, anyone browsing your jenkins server could be allowed to run any command. It might sound simple, but in fact, Jenkins auth is tricky to activate, and it is very easy to get yourself locked out, with no working web access. So here s the steps: 1. Click on Manage Jenkins then on Configure system 2. Check the enable security checkbox 3. Under security realm select Jenkins s own user database and leave allow users to sign up . Important: leave Anyone can do anything for the moment (otherwise, you will lock yourself out). 5. At the bottom of the screen, click on the SAVE button. 6. On the top right, click to login / create an account. Create yourself an account, and stay logged in. 7. Once logged-in, go back again in the Manage Jenkins -> Configure system , under security. 8. Switch to Project based matrix authentication strategy . Under User/group to add , enter the new login you ve just created, and click on Add . 9. Select absolutely all checkboxes for that user, so that you make yourself an administrator. 10. For the Anonymous user, for Job, check Read, Build and Workspace. For Overall select Read. 11. At the bottom of the screen, hit save again. Now, anonymous (eg: not logged-in) users should be able to see all projects, and be able to click on the build now button. Note that if you lock yourself out, the way to fix is to turn off Jenkins, edit config.xml, remove the useSecurity thing, and all what is in authorizationStragegy and securityRealm , then restart Jenkins. I had to do that multiple times until I had it right (as it isn t really obvious you got to leave Jenkins completely insecure when creating a new user). Securing Jenkins: Proxy Jenkins through apache to use it over SSL
When doing a quick $search-engine search, you will see lots of tutorials to use apache as a proxy, which seems to be the standard way to run Jenkins. Add the following to /etc/apache2/sites-available/default-ssl:
ProxyPass / http://localhost:8080/
ProxyPassReverse / http://localhost:8080/
ProxyRequests Off
Then perform the following commands on the shell:
htpasswd -c /etc/apache2/jenkins_htpasswd <your-jenkins-username>
a2enmod proxy
a2enmod proxy_http
a2enmod ssl
a2ensite default-ssl
a2dissite default
apache2ctl restart
Then disable access to the port 8080 of jenkins from outside:
iptables -I INPUT -d <ip-of-your-server> -p tcp --dport 8080 -j REJECT
Of course, this doesn t mean you shouldn t take the steps to activate Jenkins own authentication, which is disabled by default (sic!). Build a script to build packages in a cowbuilder I thought it was hard. In fact it was not. All together, this was kind of fun to hack. Yes hack. What I did yet another kind of 10km ugly shell script. The way to use it is simply: build-openstack-pkg <package-name>. On my build server, I have put that script in /usr/bin, so that it is accessible from the default path.Ugly, but it does the job! Jenkins build script for openstack At the end of the script, scan_repo() generates the necessary files for a Debian repository to work under /home/ftp. I use pure-ftpd to serve it. /home/ftp must be owned by jenkins:jenkins so that the build script can copy packages in it. This build script is by no mean state of the art, and in fact, it s quite hack-ish (so I m not particularly proud of it, but it does its job ). If I am showing it in this blog post, it is just to show an example of what can be done. It is left as an exercise to the reader to create another build script adapted to its own needs, and write something cleaner and more modular. Dependency building Let s say that you are using the Built-Using: field, and that package B needs to be built if package A changes. Well, Jenkins can be configured to do that. Simply edit the configuration of project B (you will find it, it s easy ). My use case: In my case, for building Glance, Heat, Keystone, Nova, Quantum, Cinder and Ceilometer, which are all components of Openstack, I have written a small (about 500 lines) library of shell functions, and an also small (90 lines) Makefile, which are packaged in openstack-pkg-tools (so Nova, Glance, etc. all build-depends on openstack-pkg-tools). The shell functions are included in each maintainer scripts (debian/*.config and debian/*.postinst mainly) to avoid having some pre-depends that would break debconf flow. The Makefile of openstack-pkg-tools is included in debian/rules of each packages. In such a case, trying to manage the build process by hand is boring and time consuming (spending your time watching the build process of package A, so that you can manually start the build of package B, then wait again ). But it is also error prone: it is easy to do a mistakes in the build order, you can forget to dpkg -i the new version of package A, etc. But that s not it. Probably at some point, you will want Jenkins to rebuild everything. Well, that s easy to do. Simply create a dummy project, and have other project to build after that one. The build steps could simply be: echo Dummy project as a shell script (I m not even sure that s needed ). Configuring git to start a build on each push In Jenkins, pass your mouse over the Build now URL. Well, we just need to wget that URL in your Alioth repository. A small drawing is better than long explanation:
for i in  ls /git/openstack  ; do
    echo "wget -q --no-check-certificate \
    https://<ip-of-your-server>/job/$ PROJ_NAME /build?delay=0sec \
    -O /dev/null" >/git/openstack/$ i /hooks/post-receive \
        && chmod 0770 /git/openstack/$ i /hooks/post-receive;
done
The chmod 0770 is necessary if you don t want every Alioth users to have access to your Jenkins server web interface and see an eventual htpassword protection that you can add to your jenkins box (I m not covering that, but it is fairly easy to add such protection). Note that all of the members of your Alioth group will then have access to this post-receive hook, containing the password of your htaccess, so you must trust everyone in your Alioth group to not do nasty things with your Jenkins. Bonus point: IRC robot If you would like to see the result of your build published on IRC, Jenkins can do that. Click on Manage Jenkins , then on Manage Plugins . Then click on available and check the box in front of IRC plugin . Go at the bottom of the screen and click on Add . Then check the box to restart Jenkins automatically. Once it is restarted, go again under Manage jenkins then Configure system . Select the IRC Notification and configure it to join the network and the channel you want. Click on Advanced to select the IRC nick name of your bot, and make sure you change the port (by default jenkins has 194, when IRC normally uses 6667). Be patient when waiting for the IRC robot to connect / disconnect, this can take some time. Now, for each jenkins Job, you can tick the IRC Notification option. Doing piuparts after build One nice thing with automated builds, is that most of the time, you don t need to wait starring at them. So you can add as many tests as you want, the Jenkins IRC robot will anyway let you know sooner or later the result of your build. So adding piuparts tests in the build script seems the correct thing to do. Though that is still on my todo, so maybe that will be for my next blog post.

Richard Hartmann: #newinwheezy

There's a #newinwheezy game which basically presents a few of the 4451 new source packages in Debian/wheezy to a wider audience. My own entry is, obviously, vcsh. Quoth the manpage
vcsh - manage config files in $HOME via fake bare git repositories

You can also consult the (somewhat outdated) readme or just clone it. In related news, I have been asked to hold my "Gitify your life" talk at LinuxTag 2013. This talk can be found here. While the text on that page is in German, all slides will be in English and I will happily use either language based on what the audience prefers.

Mart&iacute;n Ferrari: Setting up my server: netfilter

I was going to start this series with explaining how I did the remote set-up, but instead I will share something that happened today. One of the first things you want to do when putting a server directly connected to the Internet is some filtering. You don't want to have an application listening on the network by mistake, so a simple netfilter firewall is a good way to ensure you are only accepting connections on ports you explicitly allowed. I have been a long-time user of ferm, a simple tool that will read a configuration file written in a special structured syntax, and generates iptables commands from it. I have used it successfully to build very complex firewalls in previous jobs, and it had the huge benefit of keeping your firewall description readable and easy to modify by other people. This time I thought I may go with something simpler, as I only wanted a handful of very simple netfilter rules. I looked at Shorewall, and browsed a bit a few others. But in the end I decided against them: there was the need to learn the tools' concepts about different parts of the network, or there were more slanted towards command-line commands, so your actual configuration will be some files in /var/lib, totally managed by the tool. With ferm, I just need to write a very small configuration file, which reads almost like iptables commands, and that's it. In fact, the default configuration placed by the Debian package, already did 90% of what I wanted: accept incoming SSH connections, ICMP packets, and reject everything else. I took the example IPv6 configuration from /usr/share/doc/ferm/examples/ipv6.ferm and in 10 minutes it was ready:
table filter  
    chain INPUT  
        policy DROP;
        mod state state INVALID DROP;
        mod state state (ESTABLISHED RELATED) ACCEPT;
        interface lo ACCEPT;
        proto icmp ACCEPT; 
        # allow IPsec
        proto udp dport 500 ACCEPT;
        proto (esp ah) ACCEPT;
        proto tcp dport ssh ACCEPT;
        proto tcp dport (http https) ACCEPT;
     
    chain OUTPUT policy ACCEPT;
    chain FORWARD policy DROP;
 
domain ip6 table filter  
    chain INPUT  
        policy DROP;
        mod state state INVALID DROP;
        mod state state (ESTABLISHED RELATED) ACCEPT;
        interface lo ACCEPT;
        proto ipv6-icmp ACCEPT;
        proto tcp dport ssh ACCEPT;
        proto tcp dport (http https) ACCEPT;
     
    chain OUTPUT policy ACCEPT;
    chain FORWARD policy DROP;
 
It is important to note than when doing this kind of thing on a remote machine, you want to make sure you don't get locked out by accident. My method is that before activating any dangerous change, I drop an at job to disable the firewall in a few minutes:
# echo /etc/init.d/ferm stop   at now +10min
warning: commands will be executed using /bin/sh
job 4 at Mon Apr 29 02:47:00 2013
And if everything goes well, I just remove the job:
# atrm 4
Update: As paravoid pointed out in the comments, now (read: since many years ago, but I've never noticed) ferm has a --interactive mode which will revert the changes if you get locked out, much like the screen resolution changing dialog in Gnome.
Another thing that you definitely want to do, is to have some kind of protection against the almost constant influx of brute-force attacks against SSH. Apart from the obvious PermitRootLogin=no setting, there are a couple of popular methods to stop people probing random username/password combinations (I am assuming here that you actually have sensible passwords, or no passwords at all): running SSH in a non-standard port, and the great fail2ban daemon. Since I don't like non-standard stuff, I installed fail2ban, which by default it will inspect /var/log/auth.log for SSH login failures and insert netfilter rules to block the offenders. Problem is, I don't like much how fail2ban inserts rules and chains into my very tidy netfilter configuration which I had just created. So, I added an "action" to do things my way: only create a service-related chain and insert rules there, I will call that chain from my main ferm.conf. Ferm runs early in the boot sequence, so this won't be a problem during normal operation. The only caveat is that after changing a configuration in ferm, I need to restart fail2ban so it will recreate the netfilter chains and rules, which were wiped by ferm. This is my configuration, note that I am ignoring the port and protocol: the whole IP is blocked for a few minutes.
# cat /etc/fail2ban/jail.local 
[DEFAULT]
action = iptables-fixed[name=%(__name__)s]
# cat /etc/fail2ban/action.d/iptables-fixed.conf
[Definition]
actionstart = iptables -N fail2ban-<name>
              iptables -I fail2ban -j fail2ban-<name>
actionstop = iptables -D fail2ban -j fail2ban-<name>
             iptables -F fail2ban-<name>
             iptables -X fail2ban-<name>
actioncheck = iptables -n -L   grep -q fail2ban-<name>
actionban = iptables -I fail2ban-<name> 1 -s <ip> -j DROP
actionunban = iptables -D fail2ban-<name> -s <ip> -j DROP
[Init]
name = default

28 April 2013

Mart&iacute;n Ferrari: Moving my stuff away from home

TL;DR version: I want to get rid of the small server running at home, I tell you here about the service I've chosen, and why I like it. In following posts, I'll explain how did I set it up remotely. Disclaimer: I am in now way affiliated with the companies I mention here (except for Picasa, as I am a Google employee), and don't get any bonuses for this post. I am only sharing this because I think it might be useful information for other people.
Being a frequent migrant means possessions are a burden. In my previous place of residence (in France), I originally intended to only stay for 6 months, and so I arrived with just a couple of suitcases, and in the end that was enough for me to live for almost 2 years. The last time, on the other hand, I was removing my stuff completely from Argentina. I emptied my house, gave away some stuff, send some boxes to my parents' place, and carried the rest with me. That was a lot of stuff, but since the company was paying for the relocation, it was not much of a problem. Later I realised my mistake, and knowing that my time in Ireland is limited, I started to try and get rid of stuff I don't need. I know I will just sell or give away much of my stuff when I finally leave, but there are some things that are not so easy to part with. The main one being my home server, which hosts this website, my VCS repositories, pictures, and many other things I need to have on the net. This all used to be located in a home-made PC tucked in a data centre, co-located by a friendly company. But that computer died almost 2 years ago, and so canterville became abhean, and my stuff started being hosted with my aDSL connection. It worked well for some time, but now I realised I had to revert that change. With this in mind I set off to find a cheap place to host my stuff. I had a few requirements: I don't have that many photos, nor they are too big, but these requirements made it clear that most VPS offerings were not going to work for me. For some reason I fail to understand, local storage in VPS offerings is usually prohibitively expensive. This is OK for most use cases, but not for mine. A friend of mine, with a similar use case, is a happy VPS customer. He told me his trick: he only hosts in the server low-quality versions of the pictures, and keeps the originals (and back-ups) at home. This was a great idea, but with two fatal flaws: I want to only carry around a laptop and one or two external hard drives; and I want to have back-ups that are not physically with me. I was starting to think about hosting my files in Amazon s3 or something like that, since most dedicated servers are way too expensive. But then I heard about two French companies offering dirt-cheap servers: OVH and Online.net. Both of them offered small servers for about 12 a month, cheaper than most VPS offerings! Online seems to mainly cater to the French market, and for some silly reason, they charge a 50 set-up fee to customers outside of France. OVH, on the other hand, has many local branches, including an Irish one, so I went with them. The offering is a low-cost line called Kimsufi, and the smallest one is still very decent for a personal server: Once I had paid the fee for one month, it took a while for it to be activated (their payment system is pretty bad), but it finally was enabled about 24h later. Then the real fun started. On one hand, I was happy to see a wide selections of operating systems to choose from, including Debian stable and testing, and a web console with many functionalities, including some basic monitoring; but on the other hand, I realised that the installed image was not pristine, the online docs are not very good, and the web application is a bit buggy and really awkward to navigate. Having sub-par docs is not something I would usually care much about, but it made it a bit more difficult to me to understand some of the very cool functionalities their system offers (more on that in a bit), and more importantly, it made it clear to me that I won't trust their image: the procedures detailed there were not exactly best-practices, and they allow themselves to log-in as root into my server. I want to describe here what I think are their most interesting features, that made it possible to me to do risky operations, like encrypting the root partition, and setting up a firewall; and being able to fix problems that would usually require physical access. These are found in their web console: a hardware reset, and configurable netboot support with many offered images, including a rescue image based on Ubuntu and one that serves as a virtual KVM. (It is surprising that these servers don't have a serial console, but at least the kernel does not detect any). With these in hand, I didn't have to fear being locked out of my server for ever. Just set up a netboot image and hard-reboot the machine! Also, it made it very simple to install my system from scratch with debootstrap. The virtual KVM is a very interesting trick. It is a netboot image that runs some tests, and fires up a web browser. You get an email with the URL and a password to access it, and then you open a page that offers you what is basically a Qemu connected to a VNC server which will boot from your real hard drive. It is super slow, but that allows you to get console access to your server, which can be very handy to debug booting problems, unless it is some issue with the real hardware. It also offers the possibility of downloading an ISO image off the network and booting that, so it can be used to run a stock installer CD too. In another post I'll describe how I reinstalled my server remotely, and some of the pitfalls that I've encountered in the process.

Christian Perrier: [life] Running update: January-April...and more.

It seems that I didn't send any update to my international friends for quite a while, at least when it comes at my running activities. So, in short, I ran A LOT during the first months of April 2013. I mean it. As of now (April 28th), I cumulated 1424 kilometers, with a peak in March up to 375 kilometers. So, that's an average 12 kilometers per day. How did I achieve this? Among many other things, by doing part of my commute to work by running, which means 14 kilometers in one day, with a backpack containing everything needed to be dressed "normally" while working, plus my laptop, my rain jacket, etc....so up to 4 kilograms on my shoulders. And, yes, I can use a shower at work and I don't stink all day long! This alone already makes a fairly good training. Of course, all alone, it wouldn't be really funny, so I, of course, add some runs during the week-end, mostly trail running, enjoying the nature around our place. Official races have been mostly trail races during these months. The only road race has been an half-marathon in Bullion, back in February (4th year in a row I'm running this one, which is traditionnaly the "resume road races" competition in the area). I completed it in 1h39, quite close to my PB, even if....that was meant to be a training only. In February, still as a preparation race for Paris Ecotrail, I ran a 20km trail in Auffargis (a neat small village in the neighbourhood of our place), again completing it with great success, with a big negative split (for non hard-core runners, a negative split happens when one runs the second half of a race faster than the first one). All this was in preparation for Paris Ecotrail race, my second time on this 80km race that ends up at Eiffel Tower. Last year, being my first attempt on such distance,, I completed it in 11 hours 5 minutes. To make it short, this year, I finished 579th out of more than 2000 runners, in 9h36. That was indeed a really great result, in line with my 8h15 time back in November for the 70km "Le Puy-Firminy" night race. Moreover, I could indeed recover very quickly : the race was run on a Saturday and I resumed my "commute runs" on Wednesday. The next target were two trail races in April : I originally planned a 44km race on April 21st and finally ended up adding to it a 35km trail race on April 7th (the day of Paris Marathon), only 3 weeks after Ecotrail..:-) And I completed both these with a huge success. Indeed my best trail races ever, again with two negative splits and also a very good place. Indeed, I now usually complete races close to the very first women...:-). So, on April 7th, the trail du Josas (35km, 800m positive climb... which means about the equivalent of a marathon) was completed in 3h40....and last Sunday, the trail des Lavoirs (44km, 1100m positive climb) was completed in 4h40, with the last 2 kilometers being run above 13km/h. Describing how one can feel when "flying" in the very last kilometers of such a long run is just....impossible. Great, great, great memories. Then, during the week following the trail des Lavoirs, I ran 101km in 6 days, confirming that recovery is perfect. So, definitely, I am stunned by what I could achieve during these months, without injury, without big pain. Just good training and good results, without suffering and a giant pleasure. Yes, running is definitely a drug and I'm deeply addicted. Well, the result is, in short, that I feel good and well, so I think I won't stop soon...:-) Next challenge : Mont-Blanc marathon, in Chamonix : 42.195km....and 2500meters positive climb, with 1500 meters negative. Start in Chamonix at 1050m high and end at Planpraz (2050m), facing the Mont-Blanc, with a maximum altitude of 2267m during the race. Quite an interesting "marathon", isn't it? That will be my first race in real mountains...and, I guess not the last one. Target time: 6 hours. Secret wish: 5h30. During summer, I will mostly be preparing for the second part of the year....but I'll certainly enjoy the neighbourhood of Vaumarcus, Switzerland, where I'll attend DebConf. Challenge : combine running, hacking, cheese eating and fit all this in 24 hours every day. For the end of the year, challenges should peak between October and December: So, well, see you soon on this blog for another update after Mont-Blanc marathon. Let's hope I'll give you good news.

Vasudev Kamath: Tribute to Beloved Teacher

Today I'm writing this blog with saddened heart. My mentor and a best friend Dr.Ashokkumar is no more. He died yesterday after fighing with Lymph Node cancer. Ashokkumar or Ashok sir that is how we all students used to address him, was a Professor in Information Science Engineering in NMAM Institue of Technology and recently transferred to Computer Science Engineering Departement. My last meeting him with him was last year December during which he was looking every bit okay other than he had knee pain because of which he couldn't walk freely. But I never imagined that it will be my last meeting with him. Ashok sir was also behind the FLOSS event that took place in NMAM Institute of Technology including MiniDebconf 2011 which saw 2 of the foreign DD's Christian Perrier and Jonas Smedegaard. It was because of his first organized FLOSS event which I volunteered called Linux Habba I entered into the FLOSS world. This means its because of him that I started my FLOSS world journey and reached to my current level. Also it was because of his motivation I started writing this blog which I continue till this day.
I whole heartedly thank Ashok sir for teaching me guiding me and motivating me during my difficult times. You will always be remembered through out my life. May your soul Rest In Peace.
Here are the 2 pics of Ashok sir taken during Minidebconf (Credits: Christian Perrier and Kartik Mistry) Ashokkumar with Jonas and Me Ashokkumar during inaugaration of MiniDebconf with Christian and Kartik on stage Good bye Sir :-(

Vagrant Cascadian: intro

For many years, I've been meaning to write about various things in some sort of online journal. I'd like to tie together various parts of my life, such as my technical work in Debian, LTSP and other Free Software, Software Libre, and Open Source Software projects, but also my passion for Aikido, and other ideas that inspire me, such as pleaching. As with many new things, I struggled to name this journal. I even struggled with calling it a journal as opposed to a Blog, though blog always sounded a bit... brief and kind of blunt for my liking. So journal it is. But I still didn't even really have a name for it. Recently introduced to the concept of pleaching, essentially weaving living, growing branches togther, I was drawn to the concept, but admittedly, the sound of "pleach" or "pleaching" also carried some hint of harshness in it. Looking into synonyms, weaving, plait, braid... braid, yes, I typically wear my hair in a pair of braids... It has enough meaning to get by on... But that's too plain... so I borrowed from Spanish to come up with "trenza". So, thankfully to all involved, most of these posts will probably not be about picking names for things. I have a bit of backlog about some technical projects I've worked on that I'd like to write about, so I'll start with a few of those...

Paul Tagliamonte: Recent Hy developments

With some new patches from just oddles of interested hackers, there are some hot new changes, including a major mode for emacs and a hot new bot. I m currently blocking stuff while I m in the middle of the compiler, but new feature development will be wide open soon! Check out Hy, play with the source, install it with pip, and consider doing awesome stuff!

26 April 2013

Keith Packard: Shared Memory Fences

Shared Memory Fences In our last adventure, dri3k first steps, one of the future work items was to deal with synchronization between the direct rendering application and the X server. DRI2 handles this by performing a round trip each time the application starts using a buffer that was being used by the X server. As DRI3 manages buffer allocation within the application, there s really no reason to talk to the server, so this implicit serialization point just isn t available to us. As I mentioned last time, James Jones and Aaron Plattner added an explicit GPU serialization system to the Sync extension. These SyncFences serializing rendering between two X clients, but within the server there are hooks provided for the driver to use hardware-specific serialization primitives. The existing Linux DRM interfaces queue rendering to the GPU in the order requests are made to the kernel, so we don t need the ability to serialize within the GPU, we just need to serialize requests to the kernel. Simple CPU-based serialization gating access to the GPU will suffice here, at least for the current set of drivers. GPU access which is not mediated by the kernel will presumably require serialization that involves the GPU itself. We ll leave that for a future adventure though; the goal today is to build something that works with the current Linux DRM interfaces. SyncFence Semantics The semantics required by SyncFences is for multiple clients to block on a fence which a single client then triggers. All of the blocked clients start executing requests immediately after the trigger fires. There are four basic operations on SyncFences: SyncFences are the same as Events as provided by Python and other systems. Of course all of the names have been changed to keep things interesting. I ll call them Fences here, to be consistent with the current X usage. Using Pthread Primitives One fact about pthreads that I recently learned is that the synchronization primitives (mutexes, barriers and semaphores) are actually supposed to work across process boundaries, if those objects are in shared memory mapped by each process. That seemed like a great simplification for this project; allocate a page of shared memory, map into the X server and direct rendering application and use the existing pthreads APIs. Alas, the pthread objects are architecture specific. I m pretty sure that when that spec was written, no-one ever thought of running multiple architectures within the same memory space. I went and looked at the code to check, and found that each of these objects has a different size and structure on x86 and x86_64 architectures. That makes it pretty hard to use this API within X as we often have both 32- and 64- bit applications talking to the same (presumably 64-bit) X server. As a last resort, I read through a bunch of articles on using futexes directly within applications and decided that it was probably possible to implement what I needed in an architecture-independent fashion. Futexes Linux Futexes live in this strange limbo of being a not-quite-public kernel interface. Glibc uses them internally to implement locking primitives, but it doesn t export any direct interface to the system call. Certainly they re easy to use incorrectly, but it s unusual in the Linux space to have our fundamental tools locked away for our own safety . Fortunately, we can still get at futexes by creating our own syscall wrappers.
static inline long sys_futex(void *addr1, int op, int val1,
                 struct timespec *timeout, void *addr2, int val3)
 
    return syscall(SYS_futex, addr1, op, val1, timeout, addr2, val3);
 
For this little exercise, I created two simple wrappers, one to block on a futex:
static inline int futex_wait(int32_t *addr, int32_t value)  
    return sys_futex(addr, FUTEX_WAIT, value, NULL, NULL, 0);
 
and one to wake up all futex waiters:
static inline int futex_wake(int32_t *addr)  
    return sys_futex(addr, FUTEX_WAKE, MAXINT, NULL, NULL, 0);
 
Atomic Memory Operations I need atomic memory operations to keep separate cores from seeing different values of the fence value, GCC defines a few such primitives and I picked _syncboolcompareandswap and _syncvalcompareandswap. I also need fetch and store operations that the compiler won t shuffle around:
#define barrier() __asm__ __volatile__("": : :"memory")
static inline void atomic_store(int32_t *f, int32_t v)
 
    barrier();
    *f = v;
    barrier();
 
static inline int32_t atomic_fetch(int32_t *a)
 
    int32_t v;
    barrier();
    v = *a;
    barrier();
    return v;
 
If your machine doesn t make these two operations atomic, then you would redefine these as needed. Futex-based Fences These wake-all semantics of Fences greatly simplify reasoning about the operation as there s no need to ensure that only a single thread runs past Await, the only requirement is that no threads pass the Await operation until the fence is triggered. A Fence is defined by a single 32-bit integer which can take one of three values: With those, I built the fence operations as follows. Here s Await:
int fence_await(int32_t *f)
 
    while (__sync_val_compare_and_swap(f, 0, -1) != 1)  
        if (futex_wait(f, -1))  
            if (errno != EWOULDBLOCK)
                return -1;
         
     
    return 0;
 
The basic requirement that the thread not run until the fence is triggered is met by fetching the current value of the fence and comparing it with 1. Until it is signaled, that comparison will return false. The compareandswap operation makes sure the fence is -1 before the thread calls futex_wait, either it was already -1 in the case where there were other waiters, or it was 0 before and is now -1 in the case where there were no waiters before. This needs to be an atomic operation so that the fence value will be seen as -1 by the trigger operation if there are any threads in the syscall. The futex_wait call will return once the value is no longer -1, it also ensures that the thread won t block if the trigger occurs between the swap and the syscall. Here s the Trigger function:
int fence_trigger(int32_t *f)
 
    if (__sync_val_compare_and_swap(f, 0, 1) == -1)  
        atomic_store(f, 1);
        if (futex_wake(f) < 0)
            return -1;
     
    return 0;
 
The atomic compareandswap operation will make sure that no Await thread swaps the 0 for a -1 while the trigger is changing the value from 0 to 1; either the Await switches from 0 to -1 or the Trigger switches from 0 to 1. If the value before the compareandswap was -1, then there may be threads waiting on the Fence. An atomic store, constructed with two memory barriers and a regular store operation, to mark the Fence triggered is followed by the futex_wake call to unblock all Awaiting threads. The Query function is just an atomic fetch:
int fence_query(int32_t *f)
 
    return atomic_fetch(f) == 1;
 
Reset requires a compareandswap so that it doesn t disturb things if the fence has already been reset and there are threads waiting on it:
void fence_reset(int32_t *f)
 
    __sync_bool_compare_and_swap(f, 1, 0);
 
A Request for Review Ok, so we ve all tried to create synchronization primitives only to find that our obvious implementations were full of holes. I d love to hear from you if you ve identified any problems in the above code, or if you can figure out how to use the existing glibc primitives for this operation.

Russ Allbery: WebAuth 4.5.0

Time to finally release three months of work! This is a very large feature release for the multifactor support in WebAuth and for the integration with the user information service, which is the way that WebAuth interacts with local site policy. (Currently, you still have to write your own to a well-defined protocol, although we do hope to provide Perl modules to help with this in the future.) The major driving motivation for this release is to add the infrastructure required to set long-lived persistent cookies in a browser that contribute factors to authentications, which allows WebAuth to support "remember this device" functionality and to only require multifactor from devices from which a user has not previously authenticated. This includes a new token type, new attributes the user information service can send (including a way to invalidate such tokens), and new data that's sent back to the user information service. The user information service also now has the ability to add arbitrary additional factors to the current authentication, something that is intended to provide a hook for a local help desk to bypass multifactor for a user for some time if required. This release also contains substantial contributions by Benjamin Coddington at UVM to improve multifactor interactions, including sending the OTP type back to the user information service if WebLogin knows it, a mechanism for the user information service to communicate a message to the user that's displayed on the multifactor login page, opaque state that can be sent back and forth between WebLogin and the user information service, and the ability for the user information service to add specific authentication factors to the required set for a particular authentication. Other improvements in multifactor handling include the ability to set a lifetime on factors obtained via OTP login, a fix for a long-standing bug where an initial multifactor factor would satisfy a session requirement for random multifactor, and logging of even ignored errors when contacting the user information service. There are other changes too. This release touches almost every part of WebAuth. The change to WebAuthForceLogin in 4.4.0 was reverted since, on further consideration, the original semantics seemed more useful. Password change handling in WebLogin was fixed (it's been broken for some time). Apache 2.4 error logging for all modules is much improved, and mod_webauth and mod_webkdc now produce better error logs for all versions of Apache. And WebLogin now communicates password expiration times to its templates in seconds since epoch in addition to a pre-formatted English time for better localization support. William Orr contributed a new WebAuthLdapOperationalAttribute directive for mod_webauthldap that allows it to query operational attributes and include them in the environemnt. There are two backward-incompatible changes for WebLogin. First, WebAuth now supports a user checkbox indicating either to remember their login on that device or to not remember their login (local site templates can present it either way). However, proper implementation of this matching the normal expected wording of "remember me on this device" required changing the default, so a straight upgrade from an earlier version will result in no single sign-on. To preserve behavior, either a template change to add the checkbox (checked by default) or a configuration change are required. Second, support for getting password expiration times directly with remctl to a kadmin-remctl backend has been removed in favor of using data from the user information service by way of the WebKDC. Finally, I got to do a lot of cleanup of the API, fix diagosis of undef passed to Perl XS functions, and fixed a compilation error with Heimdal. You can get the latest release from the official WebAuth distribution site or from my WebAuth distribution pages.

Next.

Previous.