Search Results: "cas"

4 December 2012

Russ Allbery: Review: The Borrowers

Review: The Borrowers, by Mary Norton
Series: The Borrowers #1
Illustrator: Beth Krush
Illustrator: Joe Krush
Publisher: Sandpiper
Copyright: 1953
Printing: 1986
ISBN: 0-15-204737-9
Format: Trade paperback
Pages: 180
Arrietty is a 14-year-old girl who lives under the clock. Well, actually, she lives under the kitchen floor, but the entrance to her family's passageway is under the clock, from which they get their name. She sleeps in an old cigar box, her living room is wallpapered with strips cut from letters (and decorated with postage stamps), and she keeps a journal in a miniature book that's half the size she is. Arrietty, and her parents Pod and Homily, are Borrowers. The Borrowers is a classic series of children's books by Mary Norton about miniature people who, at least in this first book, live in the walls and take things. Not big things, but all the little things: the blotter paper, the pins, the stray sock, old postage stamps, empty matchboxes, and the occasional thimble. With these things, they build their furniture, their clothes, and their homes, safely hidden in the walls and other secret places, always wary of the "human beans" (as Arrietty's mother tends to call them). That life is, of course, a remarkable simulation of middle-class, class-conscious British life, which is much of the fun. Arrietty's family lives in the home of an elderly woman who's confined to bed, so it's quiet, safe, and driven entirely by routine. They only have to watch out for the maid and the gardener, and they keep regular schedules. That's just the way the Borrowers like it; predictable and quiet. Which is why they get caught entirely by surprise when a boy shows up, recovering from illness in the country, and spots Pod on one of his borrowing runs. This is technically a re-read, but I read the original series so long ago that I barely remembered it. Not that there is all that much to remember, at least in the way of plot. This first book follows one of the standard plots for children's books of this type (although it follows Arrietty as the viewpoint character rather than the more common choice of the boy): boy discovers hidden creatures, hidden creatures are afraid, boy turns out to want to make friends with them, they strike up a friendship, peril nonetheless ensues. If you've read many children's books, you've probably read this before; The Mouse and the Motorcycle is an obvious example (and, in terms of plot, a somewhat better book). Arrietty is a typical protagonist, her parents are silly (and a bit flighty, particularly her mother), and the boy is, disappointingly, almost a non-entity. But you don't read The Borrowers for the plot. The brilliance of this book is in the descriptions of what they use, how they use it, and how they construct their ingenious homes. Norton captures much of the wonder of other miniaturization stories (the movie Honey, I Shrunk the Kids is an obvious example), but takes it one step further into clever and thoughtful repurposing. The book is full of off-hand descriptions of how everyday objects become different everyday objects for miniature people, giving it the same delight that one gets from descriptions of how model railroad dioramas or hand-made doll houses are constructed. The detailed (if busy) illustrations by the Krushes help tremendously with visualization and make the descriptions even more fun. This is very much a children's book based on an exploration of a single great idea, so don't expect too much in the way of plot or deep characterization. It's also not science fiction; there isn't any realistic extrapolation of culture. The Borrowers all behave like perfectly ordinary English people without any further explanation because, of course, that's what they would do. Re-reading it as an adult, I wished there was a bit more meat to it. But the ingenuity is still a delight. It's great fun to spend a couple of hours imagining how one would construct the miniature accouterments of everyday life. Followed by The Borrowers Afield. Rating: 7 out of 10

3 December 2012

Joey Hess: hledger

Apologies in advance for writing a long blog post about the dull and specialised subject of double-entry accounting from the Unix tools perspective, that ends up talking about Monads to boot. I can't believe I'm about to write such a thing, but I'm finding it an oddly interesting subject. double-entry accounting I've known of, though probably not really understood double entry accounting for a while, thanks to GnuCash. I think GnuCash did something rather special in making such a subject approachable to the layman, and I've been happy to recommend GnuCash to friends. I was stoked to find a chapter in my sister Anna's new book that happily and plausibly suggests readers use GnuCash. But for my personal use, GnuCash is clunky. I mean, I'm a programmer, but I can't bring any of my abilities to bear on it, without a deep dive into the code. The file format is opaque (and a real pain to keep checked into git with all the log files); the user interface is often confusing, but there's no benefit to its learning curve, it never seems to get better than manually entering data into its ledger, or perhaps importing data from a bank. I've never found the reports very useful. I've got perhaps a year of data in GnuCash, but it's fragmented and incomplete and not something I've been able to motivate myself to keep up with. So I have less financial data than I'd like. I'm hoping ledger will change that. ledger I've known about ledger for a while, at least since This Linux Weekly News article. It's a quintessential Unix tool, that simply processes text files. The genius of it is the simplicity of the file format, that gets the essence and full power of double entry bookeeping down to something that approaches a meme. Once you get the file format stuck in your head, you're done for.
2004/05/27 Book Store
      Expenses:Books                 $20.00
      Liabilities:Visa
starting to use hledger Now as a Haskell guy, I was immediately drawn to the Haskell clone, hledger. It's nice that there are two (mostly) compatable implementations of ledger too. So from here on, I'll be talking about hledger. I sat down and built a hledger setup for myself the other day. I started by converting the GnuCash books I have been keeping up-to-date, for a small side business (a rental property). It quickly turned into something like programming, in the best way, as I used features like: a modern unix program While I've been diving into hledger, I've been drawing all kinds of parallels between it and other modern Unix-friendly programs I use lately. I think we've gone over a bit of a watershed recently. Unix tools used to be either very simple and crude (though often quite useful), or really baroque and complex with far too many options (often due to 10-30 years of evolution). Or they were a graphical user interface, like GnuCash, and completely divorced from Unix traditions. The new unix programs have some commonalities... importing data After manually converting my GnuCash data, I imported all my PayPal history into hledger. And happily it calculates the same balance Paypal does. It also tells me I've paid PayPal $180 in transaction fees over the years, which is something PayPal certianly doesn't tell you on their website. (However, my current import from PayPal's CSV files is a hackish, and only handles USD currency, so I miss some currency conversion fees.) I also imported my Amazon Payments history, which includes all the Kickstarter transactions. I almost got this to balance, but hledger and Amazon disagree about how many hundreths of a cent remain in my account. Still, pretty good, and I know how much I paid Amazon in fees for my kickstarter, and how much was kicked back to Kickstarter as well. (Look for a cost breakdown in some future blog post.) At this point, hledger stats says I have 3700 transactions on file, which is not bad for what was just a few hours work yesterday. One problem is hledger's gotten a little slow with this many transactions. It takes 5 seconds to get a balance. The ledger program, written in C, is reportedly much faster. hledger recently had a O(n^2) slowdown fixed, which makes me think it's probably only starting to be optimised. With Haskell code, you can get lucky and get near C (language, not speed of light) performace without doing much, or less lucky and get not much better than python performance until you dive into optimising. So there's hope. harnessing haskell If there's one place hledger misses out on being a state of the art modern Unix program, it's in the rules files that are used to drive CSV imports. I found these really hard to use; the manual notes that "rules file parse errors are not the greatest"; and it's just really inflexible. I think the state of the art would be to use a Domain Specific Language here. For both my Amazon and PayPal imports I had CVS data something like:
date, description, amount, fees, gross
I want to take the feeds into account, and make a split transaction, like this:
date description
    assets:accounts:PayPal             $9.90
    expenses:transaction fees:PayPal   $0.10
    income:misc:PayPal                 -$10.00
This does not seem possible with the rules file. I also wanted to combine multiple CVS lines, to do with currency conversions, into a single transaction, and couldn't. The problem is that the rules file is an ad-hoc format, not a fully programmable one. If instead, hledger's rules files were compiled into standalone haskell programs that performed the import, arbitrarily complicated conversions like the above could be done. So, I'm thinking about something like this for a DSL.. I doubt I'll get much further than this, since I have a hacked together multi-pass importer that meets my basic needs. Still, this would be nice, and being able to think about adding thing kind of thing to a program cleanly is one of the reasons I reach for the Haskell version when possible these days. First, here's part of one of my two paypal import rules files (the other one extracts the transaction fees):
amount-field 7
date-field 0
description-field %(3) - %(4) - %(11)
base-account assets:accounts:PayPal
Bank Account
assets:accounts:checking
.*
expenses:misc:PayPal
That fills out the basic fields, and makes things with "Bank Account" in their description be categorised as bank transfers. Here's how it'd look as Haskell, carefully written to avoid the $ operator that's more than a little confusing in this context. :)
main :: IO ()
main = convert paypalConveter
paypalConverter :: [CSVLine] -> [Maybe Transaction]
paypalConverter = map convert
  where
    convert = do
        setAmount =<< field 7
        setDate =<< field 0
        setDescription =<< combine " - " [field 3, field 4, field 11]
        defaultAccounts
            "assets:accounts:PayPal" ==> "expenses:misc:PayPal"
        inDescription "Bank Account" ?
            "assets:accounts:PayPal" ==> "assets:accounts:checking"
That seems like something non-haskell people could get their heads around, especially if they didn't need to write the boilerplate function definitions and types at the top. But I may be biased. :) Internally, this seems to be using a combination Reader and Writer monad that can get at fields from a CSV line and build up a Transaction. But I really just made up a simple DSL as I went along and thew in enough syntax to make it seem practical to implement. :) Of course, a Haskell programmer can skip the monads entirely, or use others they prefer. And could do arbitrarily complicated stuff during imports, including building split transactions, and combining together multiple related CVS lines into a single transaction.

1 December 2012

Thorsten Glaser: Proposed extensions to the JSON specification

The following proposal extends the JSON specification, with the idea of using JSON as an information interchange format, rather than just a way of writing certain ECMAscript values. They do not add anything but only restrict valid JSON content and encoders with some rationale. First of, I d like to remind everyone, including JSON s author, that JSON is case-sensitive, except in the four hexdigits after a backslash-u sequence in a String. Second, I d like to remind everyone that JSON is not binary-safe. No way around that, it implements Unicode (actually, 16-bit UCS-2, and it doesn t guarantee that UTF-16 surrogates are correctly paired) text. I also consider only UTF- 8, 16,32 B,L E valid encodings for JSON. (No PDP endian, either. Sorry, guys.) For my first proposal, I d like to point out CVE-2011-4815 which was about overflowing hashtables. The obvious fix is to randomise the hash per hashtable; to ensure this doesn t leak, we sort ASCIIbetically the keys of an Object in the encoder. (Using Unicode is good here we can just sort the keys as UTF-8 strings by their uint8_t value or as Unicode (UCS-2 or even UCS-4 or UTF-16) strings by the codepoints.) JSON was never preserving the order of elements in an Object anyway so we make it standardised (we still accept any order, and, when parsing, in collision cases, the later value wins). This also helps diffs. For my second proposal, I d like to forbid \u0000, \uFFFE, \uFFFF in strings. The first because many implementations use C strings, and for an information interchange format this is better; it also has security implications to allow NUL in a string. The other two, but not unpaired UTF-16 surrogates (as ECMAscript uses UCS-2 and got UTF-16 only later) because they re not valid Unicode; JSON was not binary-safe already so why bother. Among other benefits, this also helps implementations. For my third proposal, I d like to agree that implementations should impose a nesting depth limit that may be user-defined, and in the face of which, cyclic checking may be ignored by an encoder. I emit nesting depth overflows as literalnull; might also throw an error. Since I was asked, the common standard value is to restrict nesting depth is 32, unless the user specified one. (I also saw 8, but 32 WFM pretty well.) Most seem to use it even if it may seem low at first. Only specialised applications probably need more, and they can always pass a value. For my forth proposal, backslash-escape U+007F U+009F always. It may upset humans, editors, databases, etc. (This paragraph is newly added, after some IRC discussion.) All these do not permit anything that wasn t accepted to be accepted afterwards. I ve got a fifth proposal which changes acceptance rules but only for a subset of parsers: formally JSON is defined in ECMA-262 as industry standard that, in contrast to RFC 4627, always allowed any Value as top-level element of a JSON text. I d like to make it so, and ignore the RFC s requirement for it to be an Object or Array. Even so, the first two characters (after the BOM, if any) of a JSON text always are in the non-NUL 7-bit ASCII range, allowing for encoding detection. (This is done by the NUL octet pattern in the first four octets.) JSON has only taken off because it s a tightly defined simple format that can be used everywhere and isn t too awful for humans (escaping not needed for U+0020 U+D7FF and U+E000 U+FFFD after all, although I d also take the C1 control characters out, see my forth proposal above). I ve started to use a trailing comma in indexed and associative arrays in code I write at work, when the array values are one a line, to help version control systems to do their diffs, but refrain from asking for a JSON extension to permit that in order to not endanger compatibility any (no comment needed, it s just not worth it), but I d like my above proposals to be followed by implementators (and I m one of them). Some more discussion with Jonathan pointed out that JSON5 allows for trailing commata in Object and Array; IMHO the only feature of it that is not bad or outright harmful. I ll probably keep from accepting them because, on their own, they re not that useful, and I usually would run JSON texts, even configs, through a parser/encoder roundtrip to pretty-print them which would lose them anyway. As for binary-safeness: probably best to just use base64 and let the outer layers worry about compression. The data is usually unrelated to the JSON-encoded structure, and even if it s related to other data the base64 representation is usually similar (unless misaligned). Update 02.12.2012 Wrong I was about the first two characters: " " is a valid JSON text. Still possible to peek at four octets and determine the encoding by ordering the tests; updated my notes.

Rapha&#235;l Hertzog: My Free Software Activities in November 2012

This is my monthly summary of my free software related activities. If you re among the people who made a donation to support my work (692.20 , thanks everybody!), then you can learn how I spent your money. Otherwise it s just an interesting status update on my various projects. Misc packaging I updated the publican package (a tool for publishing material authored in DocBook XML) with version 3.0, a major new upstream version. As with any important update, it had its share of problems and I created two patches that I sent upstream. I uploaded the package to experimental since we re in freeze. The Debian Administrator s Handbook Since the translation teams have been working for a few months, I wanted to put the result of their work online. I did it and I blogged about it on debian-handbook.info. By the way, we have a Polish translation that just started. This took quite some time because many translators were not well versed with Docbook XML and its structure. So I fixed their mistakes and asked the Weblate developer (Michal Cihar) to implement new checks to avoid those basic XML mistakes. I also added a couple of build scripts to the git repository to make it easier to rebuild translations in multiple formats. I used this opportunity to file a couple of bugs I encountered with Publican (concerning ePub output mainly, and custom brands). I also blogged about our plans to update the book for Wheezy. Roland started to work on it but I did not have the time yet. Debian France The officers (president, treasurer, secretary) have just changed and we had to organize the transition. As the new president, I got administrator access on our Gandi virtual machine (france.debian.net) as well as access to our bank account. I got also got a bunch of administrative papers retracing the history of the association. Carl Chenet (the former president) gave them to me during the mini-debconf that was organized in Paris. Indeed, Sylvestre Ledru and Mehdi Dogguy organized our second mini-debconf Paris and they did it very well. It was a great success with over 100 attendants each of the 2 days it lasted (November 24-25th). Carl managed a merchandising booth that was well stuffed (Luca Capello also brought goodies of Debian.ch) I gave small lightning talk to present the ideas behind my Librement project (it s about funding free software developers). BTW I have not been very good at it, it was only my second lightning talk and I have been a bit too verbose. The talk did not fit in my 5 minutes time slot ;-) Back from the mini-debconf, I have been trying to delegate some projects (like get a real website, improve the work-flow of members management, update our server which was still running Lenny). Julien Cristau was willing to upgrade the server did not exactly knew how to upgrade the kernel (it s a bit special since Gandi manages the kernel on the Xen hypervisor side). So I took care of this part and also did some cleanup (adding a backup with its associated remote disk, tweaking the email configuration). And Julien completed the upgrade on November 30th. Alexandre Delano volunteered to have a try at the website and Emmanuel Bouthenot has been looking a bit to see if there was something better than Galette to handle our members. It looks like we ll stay with Galette but have to take care of upgrading it to a newer version. I also processed the first membership applications and organized a vote to extend the board of administrators (since we have two vacant seats). On Monday, we should be back to 9 administrators. Librement Except for the talk during the mini-debconf, I did not do much on this project. That said I got an answer from the Autorit de Contr le Prudentiel saying that I might be eligible for the exemption case (see discussion of last month) and that I should fill out a form to get a confirmation. I also contacted Tunz.com who might be able to provide the services I need (their E-money manager product in particular). They have the required accreditation as a banking/credit institution and are willing to partner with enterprises who setup platforms where you must manage flows of money between several parties. I m now waiting for details such as the cost of their various services. I expect to have much more to show next month I m working with two developers to implement the first building blocks of all this. Thanks See you next month for a new summary of my activities.

No comment Liked this article? Click here. My blog is Flattr-enabled.

29 November 2012

Wouter Verhelst: HELMo talk about Debian

About a month and a half ago, I got an email from someone studying at the HELMo institute for higher eduction in Li ge, Belgium, inviting me to talk at their conference which was held today. In the email, they asked me to talk about "Debian and Open Source", which is such a wide subject that I could talk about it all day; and then they gave me 20 minutes to do it. So I picked "Debian", and focused a bit more on that, only to find out when I saw their schedule that they'd put "Open Source" there instead. Hrm. Oh well. The talk itself went pretty good. I believe I managed a good balance between introducing Debian to those who've not heard of it before, and explaining to those who have, how this Debian thing works (or is supposed to work). The fact that the audience consisted of first- and second-year IT students made this easier (in case you were wondering: the third-year students, who're supposed to graduate at the end of this eyar, organized the thing). I got some interesting questions afterwards (which is always a good sign), and only positive comments as well (which is even better). After me, Lionel Dricot held a talk which was titled "Pirate Party", but which could be summarized as his personal history into the open source world, followed by some parallels between the open source world and the pirate party.That, too, was an interesting talk, even if I'm not sure I'd vote for the pirate party (which isn't even possible in Mechelen, but ignoring that). All in all, a nice use of my time.

Russ Allbery: Review: American Gods

Review: American Gods, by Neil Gaiman
Publisher: HarperTorch
Copyright: July 2001
Printing: May 2002
ISBN: 0-380-78903-5
Format: Mass market
Pages: 588
At the start of American Gods, Shadow is in prison. He's done three years of a sentence of six and is about to get released on parole. He has a wife and a job waiting for him and every hope of getting back to a good life. None of that happens. The day he's released, he finds out his wife died in a car accident. The job is gone too; it was an offer from his best friend, who died in the same accident. Numb, an ex-con, with no job prospects, he's recruited by a strange man named Wednesday, with the help of a leprechaun named Mad Sweeney, to drive Wednesday around the country, run errands for him, hurt people if absolutely necessary, and hold his vigil if he dies. And that's how Shadow is drawn into the conflict between old gods and young, between the gods that immigrants brought to America from the old world and the flashy new gods of technology and media that America is creating for itself. With native American gods and cultural legends watching around the edges. I first read American Gods shortly after it came out and before I started writing reviews, so this is a re-read. It's a book I'm glad I re-read before reviewing. The first time I read it, I was thoroughly engrossed and thought it was excellent. This time, I was entertained, but surprised at how slowly it moved, how disconnected it sometimes felt, and how little substance there was to the plot. Reading American Gods involves spending lots of time with Shadow, and I think one's reactions to this book will vary a lot based on how much one likes him as a character. Shadow is very withdrawn, very detached from the world, and quietly matter-of-fact in how he goes through his life. He's been through some major shocks and has been severed fairly thoroughly from society by three years in prison, and he doesn't reconnect easily. To some readers, such as myself, this comes across as taciturn competence and quiet observation, two things that I greatly enjoy reading about. I liked Shadow, liked reading between the lines for hints at his feelings, and sympathized a lot with his emotional state. But I can easily see an alternative reading that makes him a cipher or, worse, a blank spot in the book into which the reader has to insert themselves. One has to work at seeing Shadow's character and finding some sympathy for the degree to which he stays separate from the world. American Gods does have a plot, but it's the sort of plot that allows for endless digressions, background stories, and side meetings. In that, it reminded me less of Gaiman's other novels and more of several arcs of Sandman. I think it's worthwhile to think of this less as a novel and more as an extended ramble through the world as Gaiman sees it, full of mythology, stories, quirky people understanding the world in their own ways, and upwellings of the sacred and significant in the most unexpected of places. For example, the holy places of America in American Gods, analogous to the multiple small sacred places and old ruins in Europe, are roadside attractions, via a line of argument that makes much more sense than it has any right to and demonstrates a canny understanding of the expression of American entrepreneurial culture. As with much Gaiman, reading with Wikipedia or another mythological resource at hand is a good idea. Even if you have a good general knowledge of the most common mythology studied in the US, you probably won't recognize many of the figures Gaiman draws on: Czernobog, for example, or Wisakedjak. Part of the fun of this book is seeing Gaiman humanize them and embed them in modern culture and modern concerns while retaining their mythological edge and strangeness. As any reader of Sandman can tell you, this is one of Gaiman's strengths as a writer, and he does it well here. The bits with the undertakers Mr. Ibis and Mr. Jacquel (Thoth and Anubis) are absolutely brilliant and probably the best sections of the book (although Sam's exchange with the Men in Black is a strong contender).
"Back in my day, we had it all set up. You lined up when you died, and you'd answer for your evil deeds and for your good deeds, and if your evil deeds outweighed a feather, we'd feed your soul and your heart to Ammet, the Eater of Souls." "He must have eaten a lot of people." "Not as many as you'd think. It was a really heavy feather. We had it made special. You had to be pretty damn evil to tip the scales on that baby."
Like so much of Gaiman's work, this is the sort of book that contributed a half-dozen quotes to my quote file during my first reading. Gaiman always impresses me by not only managing to portray gods as human without losing the numinous edge, but sustaining that portrayal through very extended contact with the same god. And despite the interlude with the undertakers, the long interlude in small-town life in Lakeside, and the numerous other digressions and meanderings of a book that's not in much of a hurry to get anywhere, there is a plot and a definite conclusion, and Gaiman has some interesting twists in store at the end of that road. There's a lot of philosophical currents going on under the surface, not all of which cohere, but Gaiman does pull a lot of them together. The resolution can be a bit of an anticlimax, but I think it's a fitting one that's true to the nature of the story. I didn't enjoy this as much on the second reading as I did on the first. It didn't feel quite as coherent as I'd remembered it, and the meandering pace occasionally made the story feel stretched. But I do like Shadow, I love spending time in Gaiman's head and with his portrayals of deities, and American Gods is a treasure of great quotes. If you enjoy Gaiman's other work and aren't looking for a lot of plot and forward-moving energy, this is definitely recommended. Rating: 7 out of 10

28 November 2012

Alberto Garc a: QEMU and open hardware: SPEC and FMC TDC

Working with open hardware Some weeks ago at LinuxCon EU in Barcelona I talked about how to use QEMU to improve the reliability of device drivers. At Igalia we have been using this for some projects. One of them is the Linux IndustryPack driver. For this project I virtualized two boards: the TEWS TPCI200 PCI carrier and the GE IP-Octal 232 module. This work helped us find some bugs in the device driver and improve its quality. Now, those two boards are examples of products available in the market. But fortunately we can use the same approach to develop for hardware that doesn t exist yet, or is still in a prototype phase. Such is the case of a project we are working on: adding Linux support for this FMC Time-to-digital converter.
FMC TDC
This piece of hardware is designed by CERN and is published under the CERN Open Hardware Licence, which, in their own words is to hardware what the General Public Licence (GPL) is to software . The Open Hardware repository hosts a number of projects that have been published under this license. Why we use QEMU So we are developing the device driver for this hardware, as my colleague Samuel explains in his blog. I m the responsible of virtualizing it using QEMU. There are two main reasons why we want to do this:
  1. Limited availability of the hardware: although the specification is pretty much ready, this is still a prototype. The board is not (yet) commercially available. With virtual hardware, the whole development team can have as many boards as it needs.
  2. Testing: we can test the software against the virtual driver, force all kinds of conditions and scenarios, including the ones that would probably require us to physically damage the board.
While the first point might be the most obvious one, testing the software is actually the one we re more interested in. My colleague Miguel wrote a detailed blog post on how we have been using QEMU to do testing. Writing the virtual hardware Writing a virtual version of a particular piece of hardware for this purpose is not as hard as it might look. First, the point is not to reproduce accurately how the hardware works, but rather how it behaves from the operating system point of view: the hardware is a black box that the OS talks to. Second, it s not necessary to have a complete emulation of the hardware, there s no need to support every single feature, particularly if your software is not going to use it. The emulation can start with the basic functionality and then grow as needed. The FMC TDC, for example, is an FMC card which is in our case connected to a PCIe bridge called SPEC (also available in the Open Hardware repository). We need to emulate both cards in order to have a working system, but the emulation is, at the moment, treating both as if they were just one, which makes it a bit easier to have a prototype and from the device driver point of view doesn t really make a difference. Later the emulation can be split in two as I did with with TPCI200 and IP-Octal 232. This would allow us to support more FMC hardware without having to rewrite the bridging code. There s also code in the emulation to force different kind of scenarios that we are using to test if the driver behaves as expected and handles errors correctly. Those tests include the simulation of input in the any of the lines, simulation of noise, DMA errors, etc.
Tests
And we have written a set of test cases and a continuous integration system, so the driver is automatically tested every time the code is updated. If you want details on this I recommend you again to read Miguel s post.

Russ Allbery: Review: Emissaries from the Dead

Review: Emissaries from the Dead, by Adam-Troy Castro
Series: Andrea Cort #1
Publisher: Eos
Copyright: March 2008
ISBN: 0-06-144372-7
Format: Mass market
Pages: 387
As a small child, Andrea Cort was a colonist on Bocai, part of a utopian project to create a mixed human and Bocaian community. Until, that is, all the members of the community woke up one day and started horrifically murdering each other for reasons that none of the survivors could ever explain. Now, she's protected from extradition by a life contract with the Diplomatic Corps and serves them as a special investigator. The scene of the investigation of this book is the strange habitat One One One. The habitat was created by the AIsource, an alliance of artificial intelligences now independent of their original creators. It's in deep space, far from any sun, and would have remained entirely unknown to the other inhabitants of the galaxy if the AIsource hadn't announced its presence. The choice to announce it was baffling, since it contains a sentient species entirely under the control of the AIsource, something that's against the shared rules of all the other species, but which they wouldn't have known about if the AIsource hadn't drawn their attention to it. After extensive negotiations, one unofficial human research group has been allowed access, and now two of the members of that group have been murdered. That's where Andrea comes in. This is Castro's first original solo novel, but he has a background in short fiction in the SF, fantasy, and horror genres and some previous collaborations and shared universe writing. The horror shows through a bit more than I'd personally prefer: the preface briefly describing Andrea's past would have been sufficient, but Castro goes back and shows it in horrific and gruesome detail as a flashback. His fight scenes are also strangely and disturbingly detailed. If you have a sensitive stomach, be aware that you may want to skim some parts of this. The overall construction of this novel, though, is not horror. It's an SF murder mystery, about halfway between a procedural and a noir detective story. Andrea does follow a coherent procedure and uses an investigative technique more sophisticated than stirring the pot and seeing what jumps out, making it not quite noir, but it has the noir tendency towards story progress via attacks on the detective and quite a lot of ongoing tension. It also involves quite a lot of setup; be aware going in that we get over a hundred pages of description of the various research group members, the details of the crimes, possible motives, and the details of the artificial habitat without a lot of forward progress on understanding the crime. Castro puts a lot of work into the design of the habitat despite leaving it mostly in the background. It could easily serve as the backdrop of a more typical Big Dumb Object SF novel. The habitable zone where the research team is and where all of Andrea's investigations take place is the effective "top" of the habitat, covered in dense foilage to which the native sentient species clings and crawls its entire life. Below is an effectively infinite fall into crushing acidic atmosphere. Castro managed to not set off my dislike of heights too badly, in part because he focuses more on Andrea's own dislike of heights rather than vividly describing the drop, but it's a constant presence throughout the novel and adds a persistant background tension. It also leads to some minor but neat clashes of perspective in conversations with the native species. For most of the book, I would have described Emissaries from the Dead as an enjoyable but not horribly original detective novel with a neat science fiction background. But there are several interesting ending twists that I won't spoil and some decisions that quite surprised me. They don't have a large effect on the plot of this book, but they make for a fascinating hook and setup for later books in the series (and I see there is a sequel). It's definitely not the sort of thing that you want spoiled before reading this book, so avoid reading blurbs and details about the sequel before reading this one. I wouldn't go out of my way to hunt down this book, but it's solid, enjoyable detective SF with a surprisingly good ending. I can see why it won the Philip K. Dick award (for best SF paperback original). Mildly recommended. Followed by The Third Claw of God. Rating: 7 out of 10

26 November 2012

Stefano Zacchiroli: mini debconf paris 2012

rc bugs, cloud, and getting involved This past week-end has been rather intense. In addition to another, non free software related event I was volunteering for, I had the pleasure to participate in the 2nd Paris mini DebConf. Once again, the organization has been great and the average quality of the talks have been very high. I'd say talk quality is now totally up to par with the yearly full blown DebConf (and yes, talks have been in English :-P ). If I had to single the talk that intrigued me the most, I'd name Joss' talk on large GNOME deployments: it's full of insights on the GNOME architecture and of tips useful to all power users, no matter the size of your GNOME "deployment". For more info on the talks have a look at the program. To catch up with the talks you missed you can peruse the slides there and/or keep an eye on http://video.debian.net, where we usually post conference videos "when they are ready". At the conference I've also witnessed the usual healthy mix of country origins that I remember from the previous Paris mini DebConf. Once again I've been happy to meet (and host!) Debian friends from many countries including Germany, Switzerland, Italy, Greece, Finland, you name it. Kudos to the organizers (hi Sylvestre and Mehdi!) and to all the volunteers who made this possible. On my part, I didn't have any full blown talk scheduled (ETOOMANYTALKS struck me this month ) but I did book two lightning talks slots that I've used for: On the subject of lightning talks, I also recommend to promote Lucas' talk on how to get involved in Debian. It's dense and straight to the point, able to both convey useful tips and point wannabe contributors to the most useful contributions they can make to Debian.

Steve Langasek: Upstart in Debian

Good news, everyone! So as of last Sunday, this works on all Linux archs in Debian unstable and gives you a modern version of upstart:
echo 'Yes, do as I say!'   apt-get -o DPkg::options=--force-remove-essential -y --force-yes install upstart
Thanks to the ifupdown, sysvinit, and udev maintainers for their cooperation in getting upstart support in place; to the Debian release team for accomodating the late changes needed for upstart to be supported in wheezy; and to Scott for his past maintenance of upstart in Debian. Benchmarking One of the consequences is that it's now possible to do meaningful head-to-head comparisons of boot speed between sysvinit (with startpar), upstart, and systemd. At one time or another people have tested systemd vs. sysvinit when using bash as /bin/sh, and upstart vs. sysvinit, and systemd vs. sysvinit+startpar, and there are plenty of bootcharts floating around showing results of one init system or another on one distro or another, but I'm not aware of anyone having done a real, fair comparison of the three solutions, changing nothing but the init system. I've done some initial comparisons in a barebones sid VM, and the results are definitely interesting. Sysvinit with startpar (the default in Debian) can boot a stock sid install, with no added services, in somewhere between 3.37 and 3.42 seconds (three runs). That's not a whole lot, but on the other hand this is a system with a single filesystem and no interesting services yet. Is this really as fast as we can boot? No, even this minimal system can boot faster. Testing with upstart shows that upstart can do the same job in between 3.03 and 3.19 seconds (n=3, mean=3.09). This confirms what we'd already seen in Ubuntu, that it makes a difference to boot speed to have filesystem mounting handled by an integrated process that understands the whole system instead of as a group of serialized shell scripts. What about systemd? The same test gives a boot time between 2.32 and 2.85 seconds (n=4, mean=2.48). Interesting; what would make systemd faster than upstart in sid? Well, a quick look at the system shows one possible contributing factor: the rsyslog package in Debian has a systemd unit file, but not an upstart job file. Dropping in the /etc/init/rsyslog.conf from Ubuntu has a noticeable impact, and brings the upstart boot time down nearer to that of systemd (2.78-3.03s, n=5, mean=2.92). Besides telling me that it's time to start spamming Debian maintainers with wishlist bugs asking them to include upstart jobs in their packages, this suggests that the remaining difference in boot time may be due to the outstanding init scripts in rcS.d that are made built-in no-ops by systemd but not (yet) by upstart in Debian (e.g., hwclock, hostname, udev-mtab). (In Ubuntu, /etc/rcS.d has long since been emptied out in favor of upstart jobs in the common case, since the time it takes to get to runlevel=2 is definitely a major issue for boot speed and boot parallelization.) It also gives the lie to the claim that's been made in various places that spawning shells is a major bottleneck for upstart vs. systemd. More study is certainly needed to confirm this, but at least this naive first test suggests that in spite of the purported benefits of hard-coding boot-time policies in C code, upstart with its default degree of runtime configurability is at least in the ballpark of systemd. Indeed, when OpenSuSE switched from upstart to systemd, it seems that something else in the stack managed to nullify any benefit from improving the boot-time performance of apparmor. Contrary to what some would have you believe, systemd is not some kind of silver bullet for boot speed. Upstart, with its boot-time flexibility and its long history of real-world testing in Ubuntu, is a formidable competitor to systemd in the boot speed department - and a solid solution to the many longstanding boot-time ordering bugs in Debian, which still affect users of sysvinit. I've published the bootcharts for the above tests here. Between the fact that Debian's bootchart package logs by default to /var/log/bootchart.tgz (thus overwriting on every boot) and the fact that these tests are in a VM, I haven't bothered to include the raw data, just the bootcharts themselves. The interested reader can probably generate more interesting boot charts of their own anyway - in particular, it will be interesting to see how the different init systems perform with more complicated filesystem layouts, or when booting a less trivial set of services. Other musings The boot charts have been created with the bootchart package rather than with bootchart2. For one thing, it turns out that bootchart2 includes systemd units, not init scripts; so when replacing bootchart with bootchart2, the non-matching init script is left behind and systemd in particular gets terribly confused. This is now reported as Bug #694403. In an amusing twist, while I was experimenting with bootchart2, I also noticed that having systemd installed would slow down booting with other init systems, because systemd installs udev rules which take a noticeable amount of time to run a helper command at boot even though the helper should be a no-op. So if you're doing boot speed testing of other init systems, be sure you don't have systemd on the system at the time!

25 November 2012

Lucas Nussbaum: Half of the package maintainers are not DDs or DMs

During the Paris Mini-Debconf, Nicolas Dandrimont talked about The state of mentors.debian.net: GSoC and beyond. He said that Half of Debian s packages are maintained by sponsored maintainers. That statement was actually wrong, as he confirmed later. However, using a few UDD queries, I could come up with: Full UDD notes:
all packages in sid:
select source, version from sources_uniq where release = 'sid'
packages in sid known to upload_history:
select source, version from upload_history where
(source, version) in (select source, version from sources_uniq where release = 'sid')
packages that were uploaded by the changed_by person:
create temporary table sources_not_sponsored as select distinct source, version
 from upload_history, carnivore_keys, carnivore_emails
 where (source, version) in (select source, version from sources_uniq where release = 'sid')
 and fingerprint = key
 and carnivore_keys.id = carnivore_emails.id
 and carnivore_emails.email = changed_by_email;
packages not uploaded by the changed_by person:
create temp table uh_sid as select source, version, fingerprint, changed_by_email
from upload_history
where (source, version) in (select source, version from sources_uniq where release = 'sid');
create temp table uh_sid_sponsored as select source, version, fingerprint, changed_by_email from uh_sid
where (source, version) not in (select source, version from sources_not_sponsored);
list with sponsor login:
select distinct source, version, fingerprint, changed_by_email, login
from uh_sid_sponsored
left join carnivore_keys on fingerprint = key
left join carnivore_login on carnivore_keys.id = carnivore_login.id;
=> 4188 sponsored packages. some of them are in a strange state (changed_by is a DD, but uploaded by another DD). excluding those:
create temp table sponsored_but_dds as select distinct source, version, fingerprint, changed_by_email, login
from uh_sid_sponsored, carnivore_emails, carnivore_login
where changed_by_email = carnivore_emails.email
and carnivore_emails.id = carnivore_login.id;
create temp table really_sponsored as select distinct source, version, fingerprint, changed_by_email, login
from uh_sid_sponsored
left join carnivore_keys on fingerprint = key
left join carnivore_login on carnivore_keys.id = carnivore_login.id
where (source, version) not in (select source, version from sponsored_but_dds);
=> 3147 sponsored packages
select distinct changed_by_email from really_sponsored ;
=> 963 different sponsorees
select distinct changed_by_email from upload_history where
(source, version) in (select source, version from sources_uniq where release = 'sid');
=> 2015 distinct emails.
no DD amongst maintainer or uploader:
create temp table dds_emails as select email from carnivore_emails, carnivore_login
where carnivore_emails.id = carnivore_login.id;
select source, version, maintainer, uploaders from sources_uniq
where release='sid'
and maintainer_email not in (select * from dds_emails)
and not exists (select * from uploaders where release = 'sid' and sources_uniq.source = uploaders.source and sources_uniq.version = uploaders.version and email in (select * from dds_emails))
and maintainer_email != 'packages@qa.debian.org'
and (source, version) in (select source, version from really_sponsored);

Lucas Nussbaum: Mini-debconf lightning talk: Get involved! It s not that hard!

This week-end I attended the Paris Mini-Debconf, which was really a great event, and a nice opportunity to meet everybody again.
I delivered a lightning talk on Get involved! It s not that hard! , which was also a good excuse to mention the Debian packaging tutorial and the Debian Maintainer Dashboard.

24 November 2012

Russ Allbery: Review: Dune

Review: Dune, by Frank Herbert
Series: Dune #1
Publisher: Ace
Copyright: 1965
Printing: September 1990
ISBN: 0-441-17271-7
Format: Mass market
Pages: 537
The Atreides family, distant cousins to the imperial family, have ruled the planet of Caladan for twenty generations. Caladan is a wet farming world, comfortable and pleasant, but not horribly important. But House Atreides is feuding with House Harkonnen, and, at the start of Dune, that feud maneuvers Duke Leto into giving up his holdings and moving his family to take possession of Arrakis. Arrakis is a desert planet, previously controlled by Baron Harkonnen. It is unrelentingly hostile, home to smugglers and dangerous local desert dwellers called Fremen. But it's also one of the most important planets in the galaxy, since it's the sole origin of the chemical called melange, or spice. Spice permits a limited form of prescience, which allows the navigators of the Spacing Guild to successfully steer ships across the interstellar void. Arrakis's production of spice is what makes interstellar travel, and therefore all of interstellar civilization, possible. Dune is the story of Paul Atreides, son and heir to Duke Leto Atreides. His mother, Lady Jessica, is one of the Bene Gesserit, a secretive order of women devoted to mental and physical discipline and to the long-term genetic improvement of mankind. He is not supposed to exist; Lady Jessica was supposed to only bear a daughter of Leto. But he may be something special, the long-sought (but also dangerous) Kwisatz Haderach who can unite male and female Bene Gesserit powers. The Bene Gesserit take great interest in him from the start of Dune. More surprisingly, so do the Fremen of Arrakis; from the moment he arrives there, he seems to be fulfilling prophecies of theirs that are partly, but not entirely, ones planted by the Bene Gesserit long ago. The feud with the Harkonnens, the unstable place of Arrakis in galactic politics, the dreams of the Fremen and the Imperial ecologist on Arrakis of terraforming, Bene Gesserit plans, Paul's abilities, and the legends of the Fremen all combine in a complex mix of politics, battle, and clashes of culture. Dune is an acknowledged SF masterpiece, one of the best-known classics of the genre. It's usually found in short lists of the best SF novels ever written. It spawned five sequels by Frank Herbert (about which more in a moment), as well as numerous additional sequels and prequels by Kevin J. Anderson and Brian Herbert. It's been adopted for the screen twice, not to mention board games, video games, and numerous other projects. This is my second reading, the first in about twenty years, but the story was still immediately familiar from having seen films and having discussed and read about the universe. This is not science fiction in any strict sense. Dune is science fiction in the same way that Star Wars is: a futuristic gloss on top of power structures inspired by feudalism, heavily mixed with mysticism, mental powers, magic, and implausible but convenient science that creates the story effects the author wants. Both Bene Gesserit powers in general and Paul's abilities in particular are effectively magic. There is some hand-waving explanation of their ability to verbally control other people as taking advantage of specific pitches and intonations that people are vulnerable to, but it's effectively spell-casting (and is a direct inspiration for Jedi mind tricks). All of the mysticism (and there's quite a lot of it in Dune, including race memory, precognition, and even molecular transformation) resembles the Force from Star Wars more than anything scientific. Dune is epic fantasy told on a science fiction stage, complete with a young protagonist coming into his powers and dangerous and sometimes hostile mentors. What Dune gets right, and what has put it so high in the pantheon of great science fiction, is the world building. Herbert sets the story tens of thousands of years into the future of humanity and then effectively projects the feeling of deep history over everything in the novel. This is the kind of book that has appendices with more background information; more to the point, it's the kind of book where you may actually read them out of curiosity. Mankind has a vast interstellar empire (Herbert's universe, like Asimov's Foundation universe, admits no aliens) governed by a system akin to the early British monarchy. An emperor rules in balance with the Great Houses, who meet in a sort of parliament. But against both is a third force: the Spacing Guild, who maintains a monopoly over all interstellar travel. (And the Bene Gesserit form an underground, secretive fourth power base.) Herbert plays with vast swaths of time and great forces of history as well as very good epic fantasy and better than nearly all SF I've read. The detailed world-building is equally good. Nearly all of Dune takes place on the desert planet of Arrakis, which has a lovingly-described ecology and local culture built entirely around scarcity of water. (The details of that ecology are much of the plot and mystery of the book, so I won't spoil them further.) While I doubt the precise details hold up to close scientific scrutiny, this is an obvious precursor to the great ecological stories of later SF, such as Kim Stanley Robinson's Mars trilogy. The details all feel right and hang together in satisfying ways, while also generating the great Sand Worms of Arrakis, a key ingredient in several of the best set pieces in the history of SF. This is the sort of book where the fascinating details and discoveries about the world do as much to keep one turning the pages as the plot, although the plot is also satisfyingly twisty and tense. Unfortunately, Dune doesn't get everything right. The amount of mysticism involved is a bit much, and at times the drug-trip mystical experiences of viewpoint characters turn into excessively purple prose and nearly incomprehensible descriptions. Those mystical experiences also involve race and genetic memory, a concept that's just scientific enough to be unbelievable. A few of the other scentific cheats are also rather blantant; for example, Herbert constructs an elaborate, artificial technology of shielding that seems designed primarily as an excuse to add sword combat to a futuristic story, and I have always struggled to suspend disbelief about the way lasers and shields interact in Dune. The Spacer Guild's monopoly on interstellar travel can be explained; their monopoly on local orbital space, or even the high stratosphere, both vital to allow certain things on Arrakis to remain secret, are much more dubious. Herbert mostly doesn't try to explain these things, and as with Star Wars the less explained the cheats are, the better they work as part of the story. But the technological background doesn't hold up against much examination. Worse, for me, is the general quality of the writing. Herbert does some things very well, such as world-building, and avoids awkward infodumps. Characterization and pacing are both fairly solid; he does a good job with Paul and Jessica in particular, and I've always liked the Fremen. But he wants to put the reader in everyone's head, frequently by giving character thoughts as italicized dialogue, and to enable that he uses a perspective that I always find distracting. Most fiction is written in tight third person. This means that the viewpoint character for any given section of the book is referred to in the third person, like all the other characters, but the reader has special access to their thoughts and emotions. We get to know what they're really thinking and feeling, not just the impressions they give to others, while the non-viewpoint characters are shown only from external appearances and the thoughts of the viewpoint character. Some books hold to the same viewpoint character throughout, but more commonly books move between viewpoint characters at scene breaks to provide more angles on the book's events. First person, in which the story is told by a specific character as if they were telling a story or writing it down, is the most common alternative. Third person objective, in which we don't get any special insight into the internal thoughts of any of the characters, is less common but still unsurprising. Dune does not use any of those perspectives. Instead, Dune uses wandering third-person omniscient, in which we get the inner thoughts and emotions of a character in a scene and then a few lines later the inner thoughts and emotions of a different character. This is the sort of thing that may or may not bug you depending on how much you've read, how deep the expectations of perspective are ingrained, and how much you notice perspective. It drives me nuts. I subconsciously align with the viewpoint character of a section, and pay attention to the ways that authors indicate which character will be the viewpoint character at the start of a scene. Herbert's constant flitting from character to character makes me dizzy. We get the verbatim thoughts of everyone almost indiscriminately, making me feel like I'm randomly hopscotching through the scene. For me, this does two things: it hurts my ability to get engrossed in the story, since I'm constantly thrown out of my normal reading mode when the viewpoint unexpectedly shifts, and it makes the writing feel repetitive. One keeps hearing about the same thing from multiple perspectives, and at times the story bogs down in everyone's internal dialogues rather than showing character reactions and letting the reader draw their own conclusions. I think it tries for a cinematic perspective, but ends up making the story feel muddled. The other flaw, which I didn't notice originally but which leaped out at me during this re-read, is that Herbert's world-building uses quite a few stereotypes. The most notorious, and most widely discussed, is of course the Fremen. Herbert draws heavily on Arab and Islamic culture even beyond the obvious similarities of people living in a harsh, arid climate. He borrows some rather loaded terms and cultural markers, such as jihad, to construct a culture of potential religious fanatics. This is not all bad; the Fremen are clearly portrayed as the good guys, which is a refreshing change from more typical current portrayals of Islam. But it becomes clear that they have aligned their entire culture around influences from outside, and the whole plot of Dune can be fairly characterized as an instance of "what these people need is a white man." Paul (and Kynes before him) joins their culture as well, but Paul becomes a better native than the natives, while simultaneously bringing his outside perspective. It's the sort of plot that is more widely noticed today than it would have been in 1965. Another major example of this, and one that I found more blatant, is that Herbert turns the Harkonnen into hissable, one-sided villains and uses some nasty stereotypes to do it. The insane torturer is consistently and repeatedly described as effeminate, fat is used as a marker of moral inferiority and evil, and the primary villain is homosexual and prefers drugged young male slaves. Here too, this sort of characterization short-cut was more common in 1965, but it's not appealing and makes the (already rather camp) scenes set among the Harkonnen even less enjoyable. Less clear-cut is the way women are handled throughout Dune. I do have to give Herbert some credit, particularly for the era in which he was writing. There are powerful female characters in Dune, including both Jessica and Alia, who have their own independent power and successfully pursue their own agendas throughout. The effectively all-female Bene Gesserit is a major political power in the story and is treated by the other players with respect as well as fear. But it's hard not to also notice the general position of women as subservient to men, not only in the general culture of the Great Houses but also in the more positively-portrayed Fremen culture. Indeed, the subservience of women is even worse in Fremen culture, where they're treated like property and where being killed by a woman is a sign of shame. Again, Herbert deserves some credit for doing better than a lot of 1960s fiction, but the sexism fairy has still been at work here. None of these flaws change the fact that Dune is a masterpiece. Herbert brings together history, world building, ecology, politics, and a compelling coming-of-age story about a messiah figure into a fast-paced, sweeping epic with a thoroughly satisfying conclusion. I think they do make it a flawed masterpiece, but it's still one of those SF novels that everyone should read at least once. Sadly, it's also a masterpiece that I think has suffered from its own success in the form of sequels, prequels, and a ton of supporting material. This is one of the problems that truly excellent world building can lead to. Human history is fractal: any specific detail can be examined in more depth and will usually lead (provided that information is available at all) to even more fascinating detail. The best world building conveys that impression of depth. That's what Herbert achieves here with hints, notes, and asides: the sense that galactic history is a vast ediface with the same fractal complexity as real human history. It makes for a compelling background, but it also inspires people to dig into that background and flesh out all of the details the way that we do with human history. But this doesn't actually work; invented history created by one person simply cannot be fractal in the same way. Human history is endlessly complex because it was generated by the complex interactions of many people. Invented history is an illusion that hints at complexity by building the same surface, but one mind, or even a small number of minds, cannot generate the same depth. The result is that if one digs too deep, one removes that convincing surface and ends up with a mundane, simplistic, and unsatisfyingly fake set of events. I think that's what's happened with all of the supporting material that's been written around Dune since its original publication. Dune is of a piece, a single story that's deeply enjoyable on its own terms and leaves the reader with a satisfying impression of complexity. The systemic excavation of that complexity lessens it and reveals too much of the illusion. Yes, I want to know more about the Butlerian Jihad, but that's the point: the wanting is the sign of succesful crafting of imagined history. Reading the definitive account is more likely to leave me unsatisfied than to lead to the recursive curiosity that human history can create. The sequels to Dune written by Herbert himself are, for me, another matter. Some reviewers level the same criticism at them: that Herbert dives too far into background best left unexplored. But they have the advantage of moving forward, telling more of the story set off by Paul, and the end of Dune is a clear setup for a sequel. One of Paul's goals throughout most of the book has been left unaccomplished. I don't think Herbert dove too deep into his creation; rather, my problem with his sequels (all of which I've read, although it's been some years now) is that he took the story in a direction that I actively disliked and found painful to read. Regardless, the general consensus is that the sequels aren't as good as the original, and while Dune doesn't fully resolve its story, it's complete enough that it's possible to stop here. Stopping is the general recommendation, although I still may re-read and review the sequels at some point. Followed by Dune Messiah. Rating: 8 out of 10

22 November 2012

Ian Wienand: Upgrading Samsung Series 9 Laptop BIOS without Windows

Having recently acquired a Samsung Series 9 laptop, I'd like to update the BIOS. This is proving rather difficult without Windows. The first thing you download from the Samsung website isn't actually BIOS update at all; it's a Windows application which then goes and finds a BIOS update for your machine. Some time with a Windows VM, tcpdump and you'll end up at http://sbuservice.samsungmobile.com/. I'll leave interested parties to play around at that website, but with a little fiddling you can find the BIOS you're looking for. In this case, it is ITEM_20121024_754_WIN_P08AAH.exe which is the latest version, P08AAH. Unfortunately, this still won't run in any useful way -- in a virtual-machine it errors out saying the battery isn't connected. On previous endeavors such as this, running various incarnations of zip on the .exe file has found interesting components, but not in this case. Not to be deterred, some work with strings will yield the following
$ strings ./ITEM_20121024_754_WIN_P08AAH.exe   grep Copyright
inflate 1.2.7 Copyright 1995-2012 Mark Adler
deflate 1.2.7 Copyright 1995-2012 Jean-loup Gailly and Mark Adler
Now this is interesting, because it tells us that the components aren't using some fancy compression scheme we have no hope of figuring out. Checking RFC1952 we see the following header mentioned
Each member has the following structure:
+---+---+---+---+---+---+---+---+---+---+
 ID1 ID2 CM  FLG      MTIME      XFL OS   (more-->)
+---+---+---+---+---+---+---+---+---+---+
We can easily look for that in the file. Expanding the values, what we're looking for is 0x1f 0x8b 0x08 0x0 0x0 0x0 0x0 0x0 0x0 0x0 which represents the compressed stream (the OS byte is 0xb for NTFS which we can just ignore). A quick hack will get that done and give us the offsets we need
$ ./look-for-headers
Found potential header at 455212
Found potential header at 677136
Found potential header at 1293930
Found potential header at 1294980
Found potential header at 2771193
Found potential header at 2779874
Found potential header at 2789142
Found potential header at 2935778
Found potential header at 3078639
Found potential header at 3083039
Found potential header at 3338552
Found potential header at 3339443
Found potential header at 3352440
Found potential header at 3366757
Found potential header at 3391734
Found potential header at 3444255
Found potential header at 3494372
So, we can feed this into a shell script and extract the streams
FILE=ITEM_20121024_754_WIN_P08AAH.exe
for offset in 455212 677136 1293930 1294980 \
    2771193 2779874 2789142 2935778 3078639 \
    3083039 3338552 3339443 3352440 3366757 \
    3391734 3444255 3494372
do
    echo "create $offset.gz"
    dd if=$FILE  bs=1 skip=$offset   zcat > $offset.data
done
We don't have to worry about the end-point, because zcat will handily stop outputting and ignore junk at the end of the stream. That will yield the following output files:
$ md5sum *
c014e900b02ae39d6e574c63154809b2  1293930.data
982747cd5215c7d51fa1fba255557fef  1294980.data
942c6a8332d5dd06d8f4b2a9cb386ff4  2771193.data
d98d2f80b94f70780b46d1f079a38d93  2779874.data
33ce3174b0e205f58c7cedc61adc6f32  2789142.data
1d6851288f40fa95895b2fadee8a8b82  2935778.data
3a0a9b2fa9b40bdf5721068acde351ce  3078639.data
fc166141d06934e9c7238326f18a4e68  3083039.data
8cf7f86f84afa81363d3cef44b257d50  3338552.data
85e09e18fd495599041e13b4197e1746  3339443.data
0c72004eae0131d52817f23faf8a46a7  3352440.data
26347f93721e7ac49134a17104f56090  3366757.data
26973e301da4948a0016765e40700e8c  3391734.data
c8e4675cda3aa367d523a471bb205fba  3444255.data
7bf5ea753d4cc056b9462a02ac51b160  3494372.data
a2c6562f7f43aaa0e636cc1995b7ee3d  455212.data
d5c91b007dac3a5c6317f8a8833f9304  677136.data
65c0dcd1e7e9f8f87a8e9fb706eb8768  ITEM_20121024_754_WIN_P08AAH.exe
$ file *
1293930.data:                     MS-DOS executable
1294980.data:                     data
2771193.data:                     PE32 executable (native) Intel 80386, for MS Windows
2779874.data:                     PE32+ executable (native) x86-64, for MS Windows
2789142.data:                     PE32 executable (console) Intel 80386 (stripped to external PDB), for MS Windows
2935778.data:                     PE32+ executable (console) x86-64 (stripped to external PDB), for MS Windows
3078639.data:                     MS-DOS executable
3083039.data:                     PE32 executable (GUI) Intel 80386, for MS Windows
3338552.data:                     ASCII text, with CRLF line terminators
3339443.data:                     PE32 executable (native) Intel 80386, for MS Windows
3352440.data:                     PE32+ executable (native) x86-64, for MS Windows
3366757.data:                     PE32 executable (DLL) (GUI) Intel 80386, for MS Windows
3391734.data:                     PE32 executable (GUI) Intel 80386, for MS Windows
3444255.data:                     PE32 executable (GUI) Intel 80386, for MS Windows
3494372.data:                     PE32 executable (DLL) (GUI) Intel 80386, for MS Windows
455212.data:                      PE32 executable (DLL) (console) Intel 80386, for MS Windows
677136.data:                      data
ITEM_20121024_754_WIN_P08AAH.exe: PE32 executable (GUI) Intel 80386, for MS Windows
This is where I get a bit stuck. There is clearly a bunch of WinFlash stuff that Wine kind-of recognises, but so far I haven't managed to figure out how I might go about flashing this from a DOS boot disk. So far I know it's a "Phoenix SecureCore Tiano" based BIOS, and there is some indication that there is a UEFI app to flash the BIOS, but unfortunately I didn't set the laptop up with UEFI (note to anyone setting this up; probably a good idea to do that). I believe that if you have Windows, you can halt the install part-way through and see these files extracted in a temp directory. Possibly if we can put some file-names to the blobs it might help. There may also be a much easier way to do this that I'm completely missing. If anything comes up, I'll update this post; hopefully a future Googlenaut who enjoys BIOS hacking will drop a comment... Update 2012-11-27 I'm unfortunately less optimistic about being able to update this. Certainly the files identified above as MS-DOS executable are EFI binaries; however they are both 32-bit binaries. You can easily get an EFI shell up and running by putting a 64-bit EFI shell binary on a FAT formatted USB disk in the efi/boot/bootx64.efi but it will not run either of those binaries, presumably because they're 32-bit. A 32-bit EFI binary doesn't seem to be recognised for boot, so no luck there. A comment also mentioned some issues with Samsung's and UEFI boot in this bug though it's not clear what models are affected. Another comment mentioned binwalk which also does a nice job of finding the embedded zip streams.
$ strings ./1293930.data
...
RSDS
D:\flash\Source\ENV\src\Efi863\edk\phoenix\edk\Sample\Platform\DUET\ldr16\IA32\oemslp20.pdb
oemslp20.dll
InitializeDriver
2n3u3
$ strings ./3078639.data
en-us;zh-Hans
slp20.marker
slp20.size
slp20.offset
slp20.offset2
read marker fail!
Marker file '%s''s size(%d)!=%d
Not a valided OemMarker!
Update image data error.
marker file is updated.
marker file size too big!
Not a valided OemMarker in ROM.
Update image data error.
SLP marker is reserved.
-PROT
Prot flag is found, mProgFlag is true.
Prot flag is not found, mProgFlag is false.
Please specify input file name.
-PROT
slpFile
OEM Activation 2.0 processing
Failed to Locate Flash Support protocol.
...
D:\flash\Source\ENV\src\Efi863\edk\phoenix\edk\Sample\Platform\DUET\ldr16\IA32\SLP20.pdb

21 November 2012

Axel Beckert: Suggestions for the GNOME Team

Thanks to Erich Schubert s blog posting on Planet Debian I became aware of the 2012 GNOME User Survey at Phoronix. Like back in 2006 I still use some GNOME applications, so I do consider myself as GNOME user in the widest sense and hence I filled out that survey. Additionally I have to live with GNOME 3 as a system administrator of workstations, and that s some kind of usage, too. ;-) The last question in the survey was Do you have any comments or suggestions for the GNOME team? Sure I have. And since I tried to give constructive feedback instead of only ranting, here s my answer to that question as I submitted it in the survey, too, just spiced up with some hyperlinks and highlighting:
Don t try to change the users. Give the users more possibilities to change GNOME if they don t agree with your own preferences and decisions. (The trend to castrate the user was already starting with GNOME 2 and GNOME 3 made that worse IMHO.) If you really think that you need less configurability because some non-power-users are confused or challenged by too many choices, then please give the other users at least the chance to enable more configuration options. A very good example in that hindsight was Kazehakase (RIP) who offered several user interfaces (novice, intermediate and power user or such). The popular text-mode web browser Lynx does the same, too, btw. GNOME lost me mostly with the change to GNOME 2. The switch from Galeon 1.2 to 1.3/2.0 was horrible and the later switch to Epiphany made things even worse on the browser side. My short trip to GNOME as desktop environment ended with moving back to FVWM (configurable without tons of clicking, especially after moving to some other computer) and for the browser I moved on to Kazehakase back then. Nowadays I m living very well with Awesome and Ratpoison as window managers, Conkeror as web browser (which are all very configurable) and a few selected GNOME applications like Liferea (luckily still quite configurable despite I miss Gecko s about:config since the switch to WebKit), GUCharmap and Gnumeric. For people switching from Windows I nowadays recommend XFCE or maybe LXDE on low-end computers. I likely would recommend GNOME 2, too, if it still would exist. With regards to MATE I m skeptical about its persistance and future, but I m glad it exists as it solves a lot of problems and brings in just a few new ones. Cinnamon as well as SolusOS are based on the current GNOME libraries and are very likely the more persistent projects, but also very likely have the very same multi-head issues we re all barfing about at work with Ubuntu Precise. (Heck, am I glad that I use Awesome at work, too, and all four screens work perfectly as they did with FVWM before.)
Thanks to Dirk Deimeke for his (German written) pointer to Marcus Moeller s interview with Ikey Doherty (in German, too) about his Debian-/GNOME-based distribution SolusOS.

Ian Campbell: Consolidating Qcontrol Configuration Files

A little while ago Michele Cane filed Debian bug #689912 against qcontrol asking that the package install the correct configuration file for a TS-419P II, a newer device which has an LCD. It turns out that to support devices with LCDs properly we need to probe a GPIO line in order to figure out if an LCD is attached or if a serial console is in use (since the two are mutually exclusive on this hardware, when you attach a serial console you must toggle a jumper too). This actually turns out to be rather trivial since qcontrol's configuration files are actually simple Lua programs and the state of the GPIO lines can be accessed via sysfs and so we can read the state directly from the configuration file itself. While looking into this I happened to notice that the various board-specific configuration files are actually very similar and that the differences can all be detected by a combination of the Hardware field in /proc/cpuinfo and the GPIO lines. So I started looking into whether I could write a single consolidated qcontrol.conf file to replace the various variant specific files. It turns out to be quite possible, the resulting consolidated file has been posted to the bug log (see also my mail). The danger with configuration files which are also fully-fledged programming languages is that there is a temptation to write more and more complex configuration files with the danger that the primary purpose of the file (configuration) can be obscured. I think in this case it is a close call but on balance I think the downsides outweigh the advantages, so I've uploaded qcontrol 0.4.2+svn-r40-2 which simply adds LCD support to the relevant board specific configuration file. In process of doing so I seem to have fallen into a trap which seems to catch out many sbuild users at least once and uploaded a package with "UNRELEASED" still in the changelog. To try and save myself (and others) some blushes in the future I've filed Debian bug #693841 (wishlist) against dput to see if it can catch this case, I've also added to my prerelease checklist. As it happens my own TS-419P II arrived today. I intend to use it for second tier storage (mythtv recordings, music, mirrors) as well as light server duties (MTA, internal DNS etc) which should allow me to turn off the existing PC in the cupboard under the stairs which is getting a bit long in the tooth and starting to become a bit unreliable (I also suspect its costing a fortune to run).

20 November 2012

Jon Dowland: Waterstones

In May this year, in a desperate bid to bail water out of a sinking ship, HMV group sold off the Waterstones chain of bookstores. I'm fond of reading so naturally I visit Waterstones from time to time. (but not exclusively.) For many years, their promotions were unanimously structured as "3 for 2" offers: find three books in the promotion, in-store, and get the cheapest of the three for free. I hated these promotions. I always found it very challenging to find three books in the promotion that I wanted. If I was visiting to buy a specific book and found it in the promotion, on more than one occasion I left the shop empty handed: I wasn't prepared to pay full price for a book in a promotion and not get the benefit of the promotion. Since being sold, Waterstones seem to have made a lot of changes. Many new hardbacks from popular authors have debuted at a discount for a short period (Rowling, Pratchett). I've found many more discounted books not tied to others in 3-for-2-style offers, sometimes window-display stock at half price, a month or so after release. They started a loyalty-card scheme, with a 10 gift card the prize you reach once you've spent 100. (I wonder if such schemes are examples of a token economy?) Finally, they've tentatively started to explore putting cafes in their branches. The one in Newcastle is best described as 'fledgling', but it's nice to have one more place to escape for a cup of tea at lunchtimes. The cafe, as well as a very well stocked collection of magazines, were the two killer features of the sadly- defunct Borders bookseller chain. For years before we got our own Borders in the North East, I used to enjoy the York branch if I was passing. I actually managed to fill a loyalty card this year, so clearly their change in attitude has worked on me! I'm generally not a fan of gift cards, especially in today's climate, so I quickly spent it. I hold an American Express credit card, which earns me cashbask on purchases which I make with it. I think the rate is about 1%. However, I can opt to receive the cashback in the form of gift vouchers. If I do so, I get them to the value of 110% my cashback balance. One of the retailers in on the scheme are Waterstones. I'm tempted

Michal &#268;iha&#345;: Weblate hook at Github

Github now provides native hook for Weblate. This means that all you need to do to link these two together is to enter URL of your Weblate installation and you are done. This was actually my first code in Ruby and I don't think I will start to like it, Python still looks nicer to me :-). However it was not that hard as I expected and I've even contribute (simple) testcase for this service.

Filed under: English Weblate 3 comments Flattr this!

18 November 2012

Sven Mueller: Ingress

As some may have noticed, Google launched something that is unusual for the company: A game. A mobile alternate reality game, to be precise. See allthingsd article for a bit more information. In my opinion this game is most interesting to people who are mildly interested in geo-caching. You spend most of the gaming time either looking for the right portals (geo-location that is an interesting place for one reason or another, many located at sculptures, murals or (more scarcely) interesting local businesses), or trying to solve the puzzles that appear in the game itself or on http://www.nianticproject.com/. I was one of the beta testers and really enjoy the game, especially in areas with more players. As the game is most fun when people coordinate a bit, I started a google group for the munich area for one of the two teams (the enlightenment), but I will only let people enter that group who can be verified to be on that team. On a side note: I still have a few invites to the game left, so if you are interested, please leave a comment and enter your gmail.com email address (you need a Google account to play anyway) while doing so. I will send out invites in the order I received the requests (which I admit might not always be the same in which they were submitted. First come first serve and no guarantee to get an invitation though. Your email address you enter when commenting isn t shown to anyone and I don t share them with anyone but the game (and thus Google). EDIT: So short time went past, but I already ran out of invites (note that I didn t approve comments that really only were like please send me an invite , but still did sent one out). Feel free to still ask me for an invite, I will send some out as I get new ones, but this might take anything between a few days and some weeks. PS: For the lucky ones who got invites: You don t strictly need to do the training offered when you start the game for the first time, but I strongly encourage you to do so.
PPS: I m opening my group to any enlightened players in Germany. I will share some useful information about some game details there in the near future (nothing company confidential of course, just information anyone can gather in the game or from the published information). EDIT: As noted before, I don t have any more invites to give away. As I still get like 4 requests for invites per day, and now have more than 30 unfulfilled requests pending (with no new invites to give in sight), I m now disabling comments. Disclaimer: I work for Google since a few months ago, but this post solely represents my own opinions and hasn t been endorsed by Google in any way.

Neil Williams: Long term maintenance of perl cross-build support in Debian

After prompts from Wookey and Steve McIntyre, I decided to look at #285559 and #633884 for perl cross-build support and then port that support forward to the current perl in Wheezy and on to the version of perl currently in experimental. The first patch is for perl 5.8, the second for perl 5.12, neither of which is available currently in Debian. snapshot.debian.org provided the 5.12 source but then that no longer cross-builds with the patch.

The problem, as with any cross build, is that the build must avoid trying to execute binaries compiled within the build to achieve the test results required by ./configure (or in the case of perl, Configure). dpkg-cross has one collection of cache values but the intention was always to migrate the package-specific support data into the packages themselves and keep the architecture-specific data in dpkg-cross or dpkg-dev. Therefore, the approach taken in #633884 would be correct, if only there was a way of ensuring that the cached values remain in sync with the relevant Debian package.

I'll note here that I am aware of other ways of cross-building perl, this is particularly concerned with cross-building the Debian configuration of perl as a Debian package and using Debian or Emdebian cross-compilers. After all, the objective is to support bootstrapping Debian onto new architectures. However, I fully expect this to be just as usable with Ubuntu packages of perl compiled with, e.g. Linaro cross-compilers but I haven't yet looked at the differences between perl in Debian vs Ubuntu in any detail.

I've just got perl 5.14.2 cross-building for armel using the Emdebian gcc-4.4 cross-compiler (4.4.5-8) on a Debian sid amd64 machine without errors (it needs testing, which I'll look at later), so now is the time to document how it is done and what needs to be fixed. I've already discussed part of this with the current perl maintainers in Debian and, subject to just how the update mechanism works, have outline approval for pushing these changes into the Debian package and working with upstream where appropriate. The cache data itself might live in a separate source package which will use a strict dependency on perl to ensure that it remains in sync with the version which it can cross-build. Alternatively, if I can correctly partition the cache data between architecture-specific (and therefore generated from the existing files) and package_$version specific, then it may be possible to push a much smaller patch into the Debian perl package. This would start with some common data, calculate the arch-specific data and then look for some version-specific data, gleaned from Debian porter boxes whilst the version is in Debian experimental.

The key point is that I've offered to provide this support for the long term, ensuring that we don't end up with future stable releases of Debian having a perl package which cannot be cross-built. (To achieve that, we will also end up with versions of perl in Debian testing which also cross-build.)

This cross-build is still using dpkg-cross paths, not MultiArch paths, and this will need to be changed eventually. (e.g. by the source package providing two binaries, one which uses MultiArch and one which expects dpkg-cross paths.) The changes include patches for the upstream Makefile.SH, debian/rules and the cache data itself. Depending on where the cache data finally lives, the new support might or might not use the upstream Cross/ directory as the current contents date from the Zaurus support and don't appear to be that useful for current versions of perl.

The cache data itself has several problems:

  1. It is tightly tied to the version of perl which generated it.

  2. It is, as expected, architecture-dependent

  3. It is, unfortunately, very sensitive to the specific configuration used by the distribution itself


That last point is important because it means that the cache data is not useful upstream as a block. It also means that generating the cache data for a specific Debian package means running the generation code on the native architecture with all of the Debian build-dependencies installed for the full perl build. This is going to complicate the use of this method for new architectures like arm64.

My objective for the long term maintenance of this code is to create sufficient data that a new architecture can be bootstrapped by judicious use of some form of template. Quite how that works out, only time will tell. I expect that this will involve isolating the data which is truly architecture-specific which doesn't change between perl versions from the data which is related to the tests for build-dependencies which does change between perl versions and then work out how to deal with any remainder. A new architecture for a specific perl version should then just be a case of populating the arch-specific data such as the size of a pointer/char and the format specifiers for long long etc. alongside the existing (and correct) data for the current version of perl.

Generating the cache data natively

The perl build repeats twice (three builds in total) and each build provides and requires slightly different cache data - static, debug and shared. Therefore, the maintenance code will need to provide a script which can run the correct configuration step for each mode, copy out the cache data for each one and clean up. The script will need to run inside a buildd chroot on a porter box (I'm looking at using abel.debian.org and harris.debian.org for this work so far) so that the derived data matches what the corresponding Debian native build would use. The data then needs slight modification - typically to replace the absolute paths with PERL_BUILD_DIR. It may also be necessary to change the value of cc, ranlib and other compiler-related values to the relevant cross-compiler executables. That should be possible to arrange within the build of the cache data support package itself, allowing new cache files to be dropped in directly from the porter box.

The configuration step may need to be optimised within debian/rules of perl itself as it currently proceeds on from the bare configuration to do some actual building but I need to compare the data to see if a bare config is modified later. The test step can be omitted already. Each step is performed as:

DEB_BUILD_OPTIONS="nocheck" fakeroot debian/rules perl.static


That is repeated for perl.debug and libperl.so.$(VERSION) where $VERSION comes from :

/bin/bash debian/config.debian --full-version


The files to be copied out are:


There is a lot of scope for templating of some form here, e.g. config.h.debug is 4,686 lines long but only 41 of those lines differ between amd64 and armhf for the same version of perl (and some of those can be identified from existing architecture-specific constants) which should make for a much smaller patch.

Architecture-specific cache data for perl

So far, the existing patches only deal with armel and armhf. If I compare the differences between armel & armhf, it comes down to:

  1. compiler names (config.h.*)

  2. architecture names (config.sh.*)

  3. architecture-based paths (config.sh.*)


However, comparing armel and armhf doesn't provide sufficient info for deriving arm64 or mips etc. Comparing the same versions for armhf and amd64 shows the range of differences more clearly. Typical architecture differences exist like the size of a long, flags to denote if the compiler can cast negative floats to 32bit ints and the sprintf format specifier strings for handling floats and doubles. The data also includes some less expected ones like:

armhf: #define Netdb_host_t const void * /**/
amd64: #define Netdb_host_t char * /**/


I'm not at all sure why that is arch-specific - if anyone knows, email codehelp @ d.o - same address if anyone fancies helping out ....

Cross-builds and debclean

When playing with the cross-build, remember to use the cross-build clean support, not just debclean:


dpkg-architecture -aarmel -c fakeroot debian/rules clean


That wasted quite a bit of my time initially with having to blow away the entire tree, unpack it from original apt sources and repatch it. (Once Wheezy is out, may actually investigate getting debclean to support the -a switch).

OK, that's an introduction, I'm planning on pushing the cross-build support code onto github soon-ish and doing some testing of the cross-built perl binaries in a chroot on an armel box. I'll detail that in another blog post when it's available.

Next step is to look at perl 5.16 and then current perl upstream git to see how to get Makefile.SH fixed for the long term.

Next.

Previous.