Search Results: "rul"

3 December 2012

Joey Hess: hledger

Apologies in advance for writing a long blog post about the dull and specialised subject of double-entry accounting from the Unix tools perspective, that ends up talking about Monads to boot. I can't believe I'm about to write such a thing, but I'm finding it an oddly interesting subject. double-entry accounting I've known of, though probably not really understood double entry accounting for a while, thanks to GnuCash. I think GnuCash did something rather special in making such a subject approachable to the layman, and I've been happy to recommend GnuCash to friends. I was stoked to find a chapter in my sister Anna's new book that happily and plausibly suggests readers use GnuCash. But for my personal use, GnuCash is clunky. I mean, I'm a programmer, but I can't bring any of my abilities to bear on it, without a deep dive into the code. The file format is opaque (and a real pain to keep checked into git with all the log files); the user interface is often confusing, but there's no benefit to its learning curve, it never seems to get better than manually entering data into its ledger, or perhaps importing data from a bank. I've never found the reports very useful. I've got perhaps a year of data in GnuCash, but it's fragmented and incomplete and not something I've been able to motivate myself to keep up with. So I have less financial data than I'd like. I'm hoping ledger will change that. ledger I've known about ledger for a while, at least since This Linux Weekly News article. It's a quintessential Unix tool, that simply processes text files. The genius of it is the simplicity of the file format, that gets the essence and full power of double entry bookeeping down to something that approaches a meme. Once you get the file format stuck in your head, you're done for.
2004/05/27 Book Store
      Expenses:Books                 $20.00
      Liabilities:Visa
starting to use hledger Now as a Haskell guy, I was immediately drawn to the Haskell clone, hledger. It's nice that there are two (mostly) compatable implementations of ledger too. So from here on, I'll be talking about hledger. I sat down and built a hledger setup for myself the other day. I started by converting the GnuCash books I have been keeping up-to-date, for a small side business (a rental property). It quickly turned into something like programming, in the best way, as I used features like: a modern unix program While I've been diving into hledger, I've been drawing all kinds of parallels between it and other modern Unix-friendly programs I use lately. I think we've gone over a bit of a watershed recently. Unix tools used to be either very simple and crude (though often quite useful), or really baroque and complex with far too many options (often due to 10-30 years of evolution). Or they were a graphical user interface, like GnuCash, and completely divorced from Unix traditions. The new unix programs have some commonalities... importing data After manually converting my GnuCash data, I imported all my PayPal history into hledger. And happily it calculates the same balance Paypal does. It also tells me I've paid PayPal $180 in transaction fees over the years, which is something PayPal certianly doesn't tell you on their website. (However, my current import from PayPal's CSV files is a hackish, and only handles USD currency, so I miss some currency conversion fees.) I also imported my Amazon Payments history, which includes all the Kickstarter transactions. I almost got this to balance, but hledger and Amazon disagree about how many hundreths of a cent remain in my account. Still, pretty good, and I know how much I paid Amazon in fees for my kickstarter, and how much was kicked back to Kickstarter as well. (Look for a cost breakdown in some future blog post.) At this point, hledger stats says I have 3700 transactions on file, which is not bad for what was just a few hours work yesterday. One problem is hledger's gotten a little slow with this many transactions. It takes 5 seconds to get a balance. The ledger program, written in C, is reportedly much faster. hledger recently had a O(n^2) slowdown fixed, which makes me think it's probably only starting to be optimised. With Haskell code, you can get lucky and get near C (language, not speed of light) performace without doing much, or less lucky and get not much better than python performance until you dive into optimising. So there's hope. harnessing haskell If there's one place hledger misses out on being a state of the art modern Unix program, it's in the rules files that are used to drive CSV imports. I found these really hard to use; the manual notes that "rules file parse errors are not the greatest"; and it's just really inflexible. I think the state of the art would be to use a Domain Specific Language here. For both my Amazon and PayPal imports I had CVS data something like:
date, description, amount, fees, gross
I want to take the feeds into account, and make a split transaction, like this:
date description
    assets:accounts:PayPal             $9.90
    expenses:transaction fees:PayPal   $0.10
    income:misc:PayPal                 -$10.00
This does not seem possible with the rules file. I also wanted to combine multiple CVS lines, to do with currency conversions, into a single transaction, and couldn't. The problem is that the rules file is an ad-hoc format, not a fully programmable one. If instead, hledger's rules files were compiled into standalone haskell programs that performed the import, arbitrarily complicated conversions like the above could be done. So, I'm thinking about something like this for a DSL.. I doubt I'll get much further than this, since I have a hacked together multi-pass importer that meets my basic needs. Still, this would be nice, and being able to think about adding thing kind of thing to a program cleanly is one of the reasons I reach for the Haskell version when possible these days. First, here's part of one of my two paypal import rules files (the other one extracts the transaction fees):
amount-field 7
date-field 0
description-field %(3) - %(4) - %(11)
base-account assets:accounts:PayPal
Bank Account
assets:accounts:checking
.*
expenses:misc:PayPal
That fills out the basic fields, and makes things with "Bank Account" in their description be categorised as bank transfers. Here's how it'd look as Haskell, carefully written to avoid the $ operator that's more than a little confusing in this context. :)
main :: IO ()
main = convert paypalConveter
paypalConverter :: [CSVLine] -> [Maybe Transaction]
paypalConverter = map convert
  where
    convert = do
        setAmount =<< field 7
        setDate =<< field 0
        setDescription =<< combine " - " [field 3, field 4, field 11]
        defaultAccounts
            "assets:accounts:PayPal" ==> "expenses:misc:PayPal"
        inDescription "Bank Account" ?
            "assets:accounts:PayPal" ==> "assets:accounts:checking"
That seems like something non-haskell people could get their heads around, especially if they didn't need to write the boilerplate function definitions and types at the top. But I may be biased. :) Internally, this seems to be using a combination Reader and Writer monad that can get at fields from a CSV line and build up a Transaction. But I really just made up a simple DSL as I went along and thew in enough syntax to make it seem practical to implement. :) Of course, a Haskell programmer can skip the monads entirely, or use others they prefer. And could do arbitrarily complicated stuff during imports, including building split transactions, and combining together multiple related CVS lines into a single transaction.

1 December 2012

Thorsten Glaser: Proposed extensions to the JSON specification

The following proposal extends the JSON specification, with the idea of using JSON as an information interchange format, rather than just a way of writing certain ECMAscript values. They do not add anything but only restrict valid JSON content and encoders with some rationale. First of, I d like to remind everyone, including JSON s author, that JSON is case-sensitive, except in the four hexdigits after a backslash-u sequence in a String. Second, I d like to remind everyone that JSON is not binary-safe. No way around that, it implements Unicode (actually, 16-bit UCS-2, and it doesn t guarantee that UTF-16 surrogates are correctly paired) text. I also consider only UTF- 8, 16,32 B,L E valid encodings for JSON. (No PDP endian, either. Sorry, guys.) For my first proposal, I d like to point out CVE-2011-4815 which was about overflowing hashtables. The obvious fix is to randomise the hash per hashtable; to ensure this doesn t leak, we sort ASCIIbetically the keys of an Object in the encoder. (Using Unicode is good here we can just sort the keys as UTF-8 strings by their uint8_t value or as Unicode (UCS-2 or even UCS-4 or UTF-16) strings by the codepoints.) JSON was never preserving the order of elements in an Object anyway so we make it standardised (we still accept any order, and, when parsing, in collision cases, the later value wins). This also helps diffs. For my second proposal, I d like to forbid \u0000, \uFFFE, \uFFFF in strings. The first because many implementations use C strings, and for an information interchange format this is better; it also has security implications to allow NUL in a string. The other two, but not unpaired UTF-16 surrogates (as ECMAscript uses UCS-2 and got UTF-16 only later) because they re not valid Unicode; JSON was not binary-safe already so why bother. Among other benefits, this also helps implementations. For my third proposal, I d like to agree that implementations should impose a nesting depth limit that may be user-defined, and in the face of which, cyclic checking may be ignored by an encoder. I emit nesting depth overflows as literalnull; might also throw an error. Since I was asked, the common standard value is to restrict nesting depth is 32, unless the user specified one. (I also saw 8, but 32 WFM pretty well.) Most seem to use it even if it may seem low at first. Only specialised applications probably need more, and they can always pass a value. For my forth proposal, backslash-escape U+007F U+009F always. It may upset humans, editors, databases, etc. (This paragraph is newly added, after some IRC discussion.) All these do not permit anything that wasn t accepted to be accepted afterwards. I ve got a fifth proposal which changes acceptance rules but only for a subset of parsers: formally JSON is defined in ECMA-262 as industry standard that, in contrast to RFC 4627, always allowed any Value as top-level element of a JSON text. I d like to make it so, and ignore the RFC s requirement for it to be an Object or Array. Even so, the first two characters (after the BOM, if any) of a JSON text always are in the non-NUL 7-bit ASCII range, allowing for encoding detection. (This is done by the NUL octet pattern in the first four octets.) JSON has only taken off because it s a tightly defined simple format that can be used everywhere and isn t too awful for humans (escaping not needed for U+0020 U+D7FF and U+E000 U+FFFD after all, although I d also take the C1 control characters out, see my forth proposal above). I ve started to use a trailing comma in indexed and associative arrays in code I write at work, when the array values are one a line, to help version control systems to do their diffs, but refrain from asking for a JSON extension to permit that in order to not endanger compatibility any (no comment needed, it s just not worth it), but I d like my above proposals to be followed by implementators (and I m one of them). Some more discussion with Jonathan pointed out that JSON5 allows for trailing commata in Object and Array; IMHO the only feature of it that is not bad or outright harmful. I ll probably keep from accepting them because, on their own, they re not that useful, and I usually would run JSON texts, even configs, through a parser/encoder roundtrip to pretty-print them which would lose them anyway. As for binary-safeness: probably best to just use base64 and let the outer layers worry about compression. The data is usually unrelated to the JSON-encoded structure, and even if it s related to other data the base64 representation is usually similar (unless misaligned). Update 02.12.2012 Wrong I was about the first two characters: " " is a valid JSON text. Still possible to peek at four octets and determine the encoding by ordering the tests; updated my notes.

Michael Stapelberg: rc bug #694670: kscd does not play any CD

Dear Lazyweb, I recently investigated #694670 - kscd does not play any CD essentially, at least on my machine, phonon-based applications such as kscd won t play Audio CDs when your CD drive is not called /dev/cdrom, but e.g. /dev/cdrom5 because I have had multiple CD drives connected to my machine. Now, I am not very deeply involved into KDE or phonon in particular. Therefore, I am asking: Is there anyone who could have a look into this issue? It should be easy to reproduce, you can simply append "5" to "cdrom" in /etc/udev/rules.d/70-persistent-cd.rules and reboot afterwards. As an outsider, I am somewhat baffled by the complexity of C++, phonon, gstreamer, etc. All these layers of abstraction are not that easy to debug, at least not for me. Good blog posts about digging into this issue are appreciated :-).

28 November 2012

Russ Allbery: Review: Emissaries from the Dead

Review: Emissaries from the Dead, by Adam-Troy Castro
Series: Andrea Cort #1
Publisher: Eos
Copyright: March 2008
ISBN: 0-06-144372-7
Format: Mass market
Pages: 387
As a small child, Andrea Cort was a colonist on Bocai, part of a utopian project to create a mixed human and Bocaian community. Until, that is, all the members of the community woke up one day and started horrifically murdering each other for reasons that none of the survivors could ever explain. Now, she's protected from extradition by a life contract with the Diplomatic Corps and serves them as a special investigator. The scene of the investigation of this book is the strange habitat One One One. The habitat was created by the AIsource, an alliance of artificial intelligences now independent of their original creators. It's in deep space, far from any sun, and would have remained entirely unknown to the other inhabitants of the galaxy if the AIsource hadn't announced its presence. The choice to announce it was baffling, since it contains a sentient species entirely under the control of the AIsource, something that's against the shared rules of all the other species, but which they wouldn't have known about if the AIsource hadn't drawn their attention to it. After extensive negotiations, one unofficial human research group has been allowed access, and now two of the members of that group have been murdered. That's where Andrea comes in. This is Castro's first original solo novel, but he has a background in short fiction in the SF, fantasy, and horror genres and some previous collaborations and shared universe writing. The horror shows through a bit more than I'd personally prefer: the preface briefly describing Andrea's past would have been sufficient, but Castro goes back and shows it in horrific and gruesome detail as a flashback. His fight scenes are also strangely and disturbingly detailed. If you have a sensitive stomach, be aware that you may want to skim some parts of this. The overall construction of this novel, though, is not horror. It's an SF murder mystery, about halfway between a procedural and a noir detective story. Andrea does follow a coherent procedure and uses an investigative technique more sophisticated than stirring the pot and seeing what jumps out, making it not quite noir, but it has the noir tendency towards story progress via attacks on the detective and quite a lot of ongoing tension. It also involves quite a lot of setup; be aware going in that we get over a hundred pages of description of the various research group members, the details of the crimes, possible motives, and the details of the artificial habitat without a lot of forward progress on understanding the crime. Castro puts a lot of work into the design of the habitat despite leaving it mostly in the background. It could easily serve as the backdrop of a more typical Big Dumb Object SF novel. The habitable zone where the research team is and where all of Andrea's investigations take place is the effective "top" of the habitat, covered in dense foilage to which the native sentient species clings and crawls its entire life. Below is an effectively infinite fall into crushing acidic atmosphere. Castro managed to not set off my dislike of heights too badly, in part because he focuses more on Andrea's own dislike of heights rather than vividly describing the drop, but it's a constant presence throughout the novel and adds a persistant background tension. It also leads to some minor but neat clashes of perspective in conversations with the native species. For most of the book, I would have described Emissaries from the Dead as an enjoyable but not horribly original detective novel with a neat science fiction background. But there are several interesting ending twists that I won't spoil and some decisions that quite surprised me. They don't have a large effect on the plot of this book, but they make for a fascinating hook and setup for later books in the series (and I see there is a sequel). It's definitely not the sort of thing that you want spoiled before reading this book, so avoid reading blurbs and details about the sequel before reading this one. I wouldn't go out of my way to hunt down this book, but it's solid, enjoyable detective SF with a surprisingly good ending. I can see why it won the Philip K. Dick award (for best SF paperback original). Mildly recommended. Followed by The Third Claw of God. Rating: 7 out of 10

26 November 2012

Steve Langasek: Upstart in Debian

Good news, everyone! So as of last Sunday, this works on all Linux archs in Debian unstable and gives you a modern version of upstart:
echo 'Yes, do as I say!'   apt-get -o DPkg::options=--force-remove-essential -y --force-yes install upstart
Thanks to the ifupdown, sysvinit, and udev maintainers for their cooperation in getting upstart support in place; to the Debian release team for accomodating the late changes needed for upstart to be supported in wheezy; and to Scott for his past maintenance of upstart in Debian. Benchmarking One of the consequences is that it's now possible to do meaningful head-to-head comparisons of boot speed between sysvinit (with startpar), upstart, and systemd. At one time or another people have tested systemd vs. sysvinit when using bash as /bin/sh, and upstart vs. sysvinit, and systemd vs. sysvinit+startpar, and there are plenty of bootcharts floating around showing results of one init system or another on one distro or another, but I'm not aware of anyone having done a real, fair comparison of the three solutions, changing nothing but the init system. I've done some initial comparisons in a barebones sid VM, and the results are definitely interesting. Sysvinit with startpar (the default in Debian) can boot a stock sid install, with no added services, in somewhere between 3.37 and 3.42 seconds (three runs). That's not a whole lot, but on the other hand this is a system with a single filesystem and no interesting services yet. Is this really as fast as we can boot? No, even this minimal system can boot faster. Testing with upstart shows that upstart can do the same job in between 3.03 and 3.19 seconds (n=3, mean=3.09). This confirms what we'd already seen in Ubuntu, that it makes a difference to boot speed to have filesystem mounting handled by an integrated process that understands the whole system instead of as a group of serialized shell scripts. What about systemd? The same test gives a boot time between 2.32 and 2.85 seconds (n=4, mean=2.48). Interesting; what would make systemd faster than upstart in sid? Well, a quick look at the system shows one possible contributing factor: the rsyslog package in Debian has a systemd unit file, but not an upstart job file. Dropping in the /etc/init/rsyslog.conf from Ubuntu has a noticeable impact, and brings the upstart boot time down nearer to that of systemd (2.78-3.03s, n=5, mean=2.92). Besides telling me that it's time to start spamming Debian maintainers with wishlist bugs asking them to include upstart jobs in their packages, this suggests that the remaining difference in boot time may be due to the outstanding init scripts in rcS.d that are made built-in no-ops by systemd but not (yet) by upstart in Debian (e.g., hwclock, hostname, udev-mtab). (In Ubuntu, /etc/rcS.d has long since been emptied out in favor of upstart jobs in the common case, since the time it takes to get to runlevel=2 is definitely a major issue for boot speed and boot parallelization.) It also gives the lie to the claim that's been made in various places that spawning shells is a major bottleneck for upstart vs. systemd. More study is certainly needed to confirm this, but at least this naive first test suggests that in spite of the purported benefits of hard-coding boot-time policies in C code, upstart with its default degree of runtime configurability is at least in the ballpark of systemd. Indeed, when OpenSuSE switched from upstart to systemd, it seems that something else in the stack managed to nullify any benefit from improving the boot-time performance of apparmor. Contrary to what some would have you believe, systemd is not some kind of silver bullet for boot speed. Upstart, with its boot-time flexibility and its long history of real-world testing in Ubuntu, is a formidable competitor to systemd in the boot speed department - and a solid solution to the many longstanding boot-time ordering bugs in Debian, which still affect users of sysvinit. I've published the bootcharts for the above tests here. Between the fact that Debian's bootchart package logs by default to /var/log/bootchart.tgz (thus overwriting on every boot) and the fact that these tests are in a VM, I haven't bothered to include the raw data, just the bootcharts themselves. The interested reader can probably generate more interesting boot charts of their own anyway - in particular, it will be interesting to see how the different init systems perform with more complicated filesystem layouts, or when booting a less trivial set of services. Other musings The boot charts have been created with the bootchart package rather than with bootchart2. For one thing, it turns out that bootchart2 includes systemd units, not init scripts; so when replacing bootchart with bootchart2, the non-matching init script is left behind and systemd in particular gets terribly confused. This is now reported as Bug #694403. In an amusing twist, while I was experimenting with bootchart2, I also noticed that having systemd installed would slow down booting with other init systems, because systemd installs udev rules which take a noticeable amount of time to run a helper command at boot even though the helper should be a no-op. So if you're doing boot speed testing of other init systems, be sure you don't have systemd on the system at the time!

24 November 2012

Russ Allbery: Review: Dune

Review: Dune, by Frank Herbert
Series: Dune #1
Publisher: Ace
Copyright: 1965
Printing: September 1990
ISBN: 0-441-17271-7
Format: Mass market
Pages: 537
The Atreides family, distant cousins to the imperial family, have ruled the planet of Caladan for twenty generations. Caladan is a wet farming world, comfortable and pleasant, but not horribly important. But House Atreides is feuding with House Harkonnen, and, at the start of Dune, that feud maneuvers Duke Leto into giving up his holdings and moving his family to take possession of Arrakis. Arrakis is a desert planet, previously controlled by Baron Harkonnen. It is unrelentingly hostile, home to smugglers and dangerous local desert dwellers called Fremen. But it's also one of the most important planets in the galaxy, since it's the sole origin of the chemical called melange, or spice. Spice permits a limited form of prescience, which allows the navigators of the Spacing Guild to successfully steer ships across the interstellar void. Arrakis's production of spice is what makes interstellar travel, and therefore all of interstellar civilization, possible. Dune is the story of Paul Atreides, son and heir to Duke Leto Atreides. His mother, Lady Jessica, is one of the Bene Gesserit, a secretive order of women devoted to mental and physical discipline and to the long-term genetic improvement of mankind. He is not supposed to exist; Lady Jessica was supposed to only bear a daughter of Leto. But he may be something special, the long-sought (but also dangerous) Kwisatz Haderach who can unite male and female Bene Gesserit powers. The Bene Gesserit take great interest in him from the start of Dune. More surprisingly, so do the Fremen of Arrakis; from the moment he arrives there, he seems to be fulfilling prophecies of theirs that are partly, but not entirely, ones planted by the Bene Gesserit long ago. The feud with the Harkonnens, the unstable place of Arrakis in galactic politics, the dreams of the Fremen and the Imperial ecologist on Arrakis of terraforming, Bene Gesserit plans, Paul's abilities, and the legends of the Fremen all combine in a complex mix of politics, battle, and clashes of culture. Dune is an acknowledged SF masterpiece, one of the best-known classics of the genre. It's usually found in short lists of the best SF novels ever written. It spawned five sequels by Frank Herbert (about which more in a moment), as well as numerous additional sequels and prequels by Kevin J. Anderson and Brian Herbert. It's been adopted for the screen twice, not to mention board games, video games, and numerous other projects. This is my second reading, the first in about twenty years, but the story was still immediately familiar from having seen films and having discussed and read about the universe. This is not science fiction in any strict sense. Dune is science fiction in the same way that Star Wars is: a futuristic gloss on top of power structures inspired by feudalism, heavily mixed with mysticism, mental powers, magic, and implausible but convenient science that creates the story effects the author wants. Both Bene Gesserit powers in general and Paul's abilities in particular are effectively magic. There is some hand-waving explanation of their ability to verbally control other people as taking advantage of specific pitches and intonations that people are vulnerable to, but it's effectively spell-casting (and is a direct inspiration for Jedi mind tricks). All of the mysticism (and there's quite a lot of it in Dune, including race memory, precognition, and even molecular transformation) resembles the Force from Star Wars more than anything scientific. Dune is epic fantasy told on a science fiction stage, complete with a young protagonist coming into his powers and dangerous and sometimes hostile mentors. What Dune gets right, and what has put it so high in the pantheon of great science fiction, is the world building. Herbert sets the story tens of thousands of years into the future of humanity and then effectively projects the feeling of deep history over everything in the novel. This is the kind of book that has appendices with more background information; more to the point, it's the kind of book where you may actually read them out of curiosity. Mankind has a vast interstellar empire (Herbert's universe, like Asimov's Foundation universe, admits no aliens) governed by a system akin to the early British monarchy. An emperor rules in balance with the Great Houses, who meet in a sort of parliament. But against both is a third force: the Spacing Guild, who maintains a monopoly over all interstellar travel. (And the Bene Gesserit form an underground, secretive fourth power base.) Herbert plays with vast swaths of time and great forces of history as well as very good epic fantasy and better than nearly all SF I've read. The detailed world-building is equally good. Nearly all of Dune takes place on the desert planet of Arrakis, which has a lovingly-described ecology and local culture built entirely around scarcity of water. (The details of that ecology are much of the plot and mystery of the book, so I won't spoil them further.) While I doubt the precise details hold up to close scientific scrutiny, this is an obvious precursor to the great ecological stories of later SF, such as Kim Stanley Robinson's Mars trilogy. The details all feel right and hang together in satisfying ways, while also generating the great Sand Worms of Arrakis, a key ingredient in several of the best set pieces in the history of SF. This is the sort of book where the fascinating details and discoveries about the world do as much to keep one turning the pages as the plot, although the plot is also satisfyingly twisty and tense. Unfortunately, Dune doesn't get everything right. The amount of mysticism involved is a bit much, and at times the drug-trip mystical experiences of viewpoint characters turn into excessively purple prose and nearly incomprehensible descriptions. Those mystical experiences also involve race and genetic memory, a concept that's just scientific enough to be unbelievable. A few of the other scentific cheats are also rather blantant; for example, Herbert constructs an elaborate, artificial technology of shielding that seems designed primarily as an excuse to add sword combat to a futuristic story, and I have always struggled to suspend disbelief about the way lasers and shields interact in Dune. The Spacer Guild's monopoly on interstellar travel can be explained; their monopoly on local orbital space, or even the high stratosphere, both vital to allow certain things on Arrakis to remain secret, are much more dubious. Herbert mostly doesn't try to explain these things, and as with Star Wars the less explained the cheats are, the better they work as part of the story. But the technological background doesn't hold up against much examination. Worse, for me, is the general quality of the writing. Herbert does some things very well, such as world-building, and avoids awkward infodumps. Characterization and pacing are both fairly solid; he does a good job with Paul and Jessica in particular, and I've always liked the Fremen. But he wants to put the reader in everyone's head, frequently by giving character thoughts as italicized dialogue, and to enable that he uses a perspective that I always find distracting. Most fiction is written in tight third person. This means that the viewpoint character for any given section of the book is referred to in the third person, like all the other characters, but the reader has special access to their thoughts and emotions. We get to know what they're really thinking and feeling, not just the impressions they give to others, while the non-viewpoint characters are shown only from external appearances and the thoughts of the viewpoint character. Some books hold to the same viewpoint character throughout, but more commonly books move between viewpoint characters at scene breaks to provide more angles on the book's events. First person, in which the story is told by a specific character as if they were telling a story or writing it down, is the most common alternative. Third person objective, in which we don't get any special insight into the internal thoughts of any of the characters, is less common but still unsurprising. Dune does not use any of those perspectives. Instead, Dune uses wandering third-person omniscient, in which we get the inner thoughts and emotions of a character in a scene and then a few lines later the inner thoughts and emotions of a different character. This is the sort of thing that may or may not bug you depending on how much you've read, how deep the expectations of perspective are ingrained, and how much you notice perspective. It drives me nuts. I subconsciously align with the viewpoint character of a section, and pay attention to the ways that authors indicate which character will be the viewpoint character at the start of a scene. Herbert's constant flitting from character to character makes me dizzy. We get the verbatim thoughts of everyone almost indiscriminately, making me feel like I'm randomly hopscotching through the scene. For me, this does two things: it hurts my ability to get engrossed in the story, since I'm constantly thrown out of my normal reading mode when the viewpoint unexpectedly shifts, and it makes the writing feel repetitive. One keeps hearing about the same thing from multiple perspectives, and at times the story bogs down in everyone's internal dialogues rather than showing character reactions and letting the reader draw their own conclusions. I think it tries for a cinematic perspective, but ends up making the story feel muddled. The other flaw, which I didn't notice originally but which leaped out at me during this re-read, is that Herbert's world-building uses quite a few stereotypes. The most notorious, and most widely discussed, is of course the Fremen. Herbert draws heavily on Arab and Islamic culture even beyond the obvious similarities of people living in a harsh, arid climate. He borrows some rather loaded terms and cultural markers, such as jihad, to construct a culture of potential religious fanatics. This is not all bad; the Fremen are clearly portrayed as the good guys, which is a refreshing change from more typical current portrayals of Islam. But it becomes clear that they have aligned their entire culture around influences from outside, and the whole plot of Dune can be fairly characterized as an instance of "what these people need is a white man." Paul (and Kynes before him) joins their culture as well, but Paul becomes a better native than the natives, while simultaneously bringing his outside perspective. It's the sort of plot that is more widely noticed today than it would have been in 1965. Another major example of this, and one that I found more blatant, is that Herbert turns the Harkonnen into hissable, one-sided villains and uses some nasty stereotypes to do it. The insane torturer is consistently and repeatedly described as effeminate, fat is used as a marker of moral inferiority and evil, and the primary villain is homosexual and prefers drugged young male slaves. Here too, this sort of characterization short-cut was more common in 1965, but it's not appealing and makes the (already rather camp) scenes set among the Harkonnen even less enjoyable. Less clear-cut is the way women are handled throughout Dune. I do have to give Herbert some credit, particularly for the era in which he was writing. There are powerful female characters in Dune, including both Jessica and Alia, who have their own independent power and successfully pursue their own agendas throughout. The effectively all-female Bene Gesserit is a major political power in the story and is treated by the other players with respect as well as fear. But it's hard not to also notice the general position of women as subservient to men, not only in the general culture of the Great Houses but also in the more positively-portrayed Fremen culture. Indeed, the subservience of women is even worse in Fremen culture, where they're treated like property and where being killed by a woman is a sign of shame. Again, Herbert deserves some credit for doing better than a lot of 1960s fiction, but the sexism fairy has still been at work here. None of these flaws change the fact that Dune is a masterpiece. Herbert brings together history, world building, ecology, politics, and a compelling coming-of-age story about a messiah figure into a fast-paced, sweeping epic with a thoroughly satisfying conclusion. I think they do make it a flawed masterpiece, but it's still one of those SF novels that everyone should read at least once. Sadly, it's also a masterpiece that I think has suffered from its own success in the form of sequels, prequels, and a ton of supporting material. This is one of the problems that truly excellent world building can lead to. Human history is fractal: any specific detail can be examined in more depth and will usually lead (provided that information is available at all) to even more fascinating detail. The best world building conveys that impression of depth. That's what Herbert achieves here with hints, notes, and asides: the sense that galactic history is a vast ediface with the same fractal complexity as real human history. It makes for a compelling background, but it also inspires people to dig into that background and flesh out all of the details the way that we do with human history. But this doesn't actually work; invented history created by one person simply cannot be fractal in the same way. Human history is endlessly complex because it was generated by the complex interactions of many people. Invented history is an illusion that hints at complexity by building the same surface, but one mind, or even a small number of minds, cannot generate the same depth. The result is that if one digs too deep, one removes that convincing surface and ends up with a mundane, simplistic, and unsatisfyingly fake set of events. I think that's what's happened with all of the supporting material that's been written around Dune since its original publication. Dune is of a piece, a single story that's deeply enjoyable on its own terms and leaves the reader with a satisfying impression of complexity. The systemic excavation of that complexity lessens it and reveals too much of the illusion. Yes, I want to know more about the Butlerian Jihad, but that's the point: the wanting is the sign of succesful crafting of imagined history. Reading the definitive account is more likely to leave me unsatisfied than to lead to the recursive curiosity that human history can create. The sequels to Dune written by Herbert himself are, for me, another matter. Some reviewers level the same criticism at them: that Herbert dives too far into background best left unexplored. But they have the advantage of moving forward, telling more of the story set off by Paul, and the end of Dune is a clear setup for a sequel. One of Paul's goals throughout most of the book has been left unaccomplished. I don't think Herbert dove too deep into his creation; rather, my problem with his sequels (all of which I've read, although it's been some years now) is that he took the story in a direction that I actively disliked and found painful to read. Regardless, the general consensus is that the sequels aren't as good as the original, and while Dune doesn't fully resolve its story, it's complete enough that it's possible to stop here. Stopping is the general recommendation, although I still may re-read and review the sequels at some point. Followed by Dune Messiah. Rating: 8 out of 10

23 November 2012

Benjamin Mako Hill: Cultivated Disinterest in Professional Sports

Like many of my friends, I have treated professional sports with cultivated indifference. But a year and a half ago, I decided to become a football fan. Several years ago, I was at a talk by Michael Albert at MIT where he chastised American intellectuals for what he claimed was cultivated disdain of professional sports. Albert suggested that sports reflect the go-to topic for small talk and building rapport across class and context. But he suggested that almost everybody who used the term "working class struggle" was incapable of making small talk with members of the working class because unlike most working class people (and most people in general) educated people systematically cultivate ignorance in sports. Professional sports are deeply popular. In the US, Sunday Night Football is now the most popular television show among women in its time slot and the third most popular television in America among 18-49 year old women. That it is also the most popular television show in general is old news. There are very few things that anywhere near half of Americans have in common. Interest in football is one of them. An enormous proportion of the US population watches the Superbowl each year. I recognized myself in Albert's critique. So I decided to follow a local team. I picked football because it is the most popular sport in America and because their strong revenue sharing system means that either team has a chance to win any given match. My local team is the New England Patriots and I've watched many of the team's games or highlights over the last season and a half. I've also followed a couple football blogs. A year and half in, I can call myself a football fan. And I've learned a few things in the process:
  1. With a little effort, getting into sports is easy. Although learning the rules of a sport can be complicated, sports are popular because people, in general, find them fun to watch. If you watch a few games with someone who can explain the rules, and if you begin to cheer for a team, you will find yourself getting emotionally invested and excited.
  2. Sports really do, as Albert implied, allow one to build rapport and small talk across society. I used to dread the local cab driver who would try to make small talk by mentioning Tom Brady or the Red Sox. No more! Some of these conversations turn into broader conversations about life and politics.
  3. Interest in sports can expand or shrink to fill the time you're willing to give it. It can mean just glancing through the sports sections of the paper and watching some highlights here or there. Or it can turn into a lifestyle.
  4. It's not all great. Football, like most professional sports, is deeply permeated with advertisements, commercialism, and money. Like other sports, it is also violent. I don't think I could ever get behind a fight sport where the goal is to hurt someone else. The machoness and absence of women in the highest levels of most professional sports bothers me deeply.
I've also tried to think a lot about why I, like most of my friends, avoided sports in the past. Disinterest in sports among academics and the highly educated is, in my experience, far from passive. I've heard people almost compete to explain the depth of their ignorance in sports one doesn't even know the rules, one doesn't own a television, one doesn't know the first thing about the game. I did the same thing myself. Bethany Bryson, a sociologist at JMU has shown that increased education is associated with increased inclusiveness in musical taste (i.e., highly educated people like more types of music) but that these people are most likely to reject music that is highly favored by the least educated people. Her paper's title sums up the attitude: "Anything But Heavy Metal". For highly educated folks, it's a sign of cultivation to be eclectic in one's tastes. But to signal to others that you belong in the intellectual elite, it can pay in cultural capital to dislike things, like sports, that are enormously popular among the least educated parts of society. This ignorance among highly educated people limits our ability to communicate, bond, and build relationships across different segments of society. It limits our ability to engage in conversations and build a common culture that crosses our highly stratified and segmented societies. Sports are not politically or culturally unproblematic. But they provide an easy and enjoyable way to build common ground with our neighbors and fellow citizens that transcend social boundaries.

22 November 2012

Cyril Brulebois: Another month, another beta

Today s recipe: Result: Debian Installer 7.0 Beta4 release! Speaking of which, shameless plug: I'm giving a Debian Installer talk at mini-DebConf Paris this sunday.

18 November 2012

Gregor Herrmann: RC bugs 2012/46

it's seems we're starting to run out of RC bugs with the implicit usertag easy-enough-for-gregoa

what I've done in the last days besides working on the bugs listed below is to tag quite a few bugs pending that were uploaded into a DELAYED queue. please try to remember to set this tag on delayed NMUs, nmudiff is quite convenient here.

Neil Williams: Long term maintenance of perl cross-build support in Debian

After prompts from Wookey and Steve McIntyre, I decided to look at #285559 and #633884 for perl cross-build support and then port that support forward to the current perl in Wheezy and on to the version of perl currently in experimental. The first patch is for perl 5.8, the second for perl 5.12, neither of which is available currently in Debian. snapshot.debian.org provided the 5.12 source but then that no longer cross-builds with the patch.

The problem, as with any cross build, is that the build must avoid trying to execute binaries compiled within the build to achieve the test results required by ./configure (or in the case of perl, Configure). dpkg-cross has one collection of cache values but the intention was always to migrate the package-specific support data into the packages themselves and keep the architecture-specific data in dpkg-cross or dpkg-dev. Therefore, the approach taken in #633884 would be correct, if only there was a way of ensuring that the cached values remain in sync with the relevant Debian package.

I'll note here that I am aware of other ways of cross-building perl, this is particularly concerned with cross-building the Debian configuration of perl as a Debian package and using Debian or Emdebian cross-compilers. After all, the objective is to support bootstrapping Debian onto new architectures. However, I fully expect this to be just as usable with Ubuntu packages of perl compiled with, e.g. Linaro cross-compilers but I haven't yet looked at the differences between perl in Debian vs Ubuntu in any detail.

I've just got perl 5.14.2 cross-building for armel using the Emdebian gcc-4.4 cross-compiler (4.4.5-8) on a Debian sid amd64 machine without errors (it needs testing, which I'll look at later), so now is the time to document how it is done and what needs to be fixed. I've already discussed part of this with the current perl maintainers in Debian and, subject to just how the update mechanism works, have outline approval for pushing these changes into the Debian package and working with upstream where appropriate. The cache data itself might live in a separate source package which will use a strict dependency on perl to ensure that it remains in sync with the version which it can cross-build. Alternatively, if I can correctly partition the cache data between architecture-specific (and therefore generated from the existing files) and package_$version specific, then it may be possible to push a much smaller patch into the Debian perl package. This would start with some common data, calculate the arch-specific data and then look for some version-specific data, gleaned from Debian porter boxes whilst the version is in Debian experimental.

The key point is that I've offered to provide this support for the long term, ensuring that we don't end up with future stable releases of Debian having a perl package which cannot be cross-built. (To achieve that, we will also end up with versions of perl in Debian testing which also cross-build.)

This cross-build is still using dpkg-cross paths, not MultiArch paths, and this will need to be changed eventually. (e.g. by the source package providing two binaries, one which uses MultiArch and one which expects dpkg-cross paths.) The changes include patches for the upstream Makefile.SH, debian/rules and the cache data itself. Depending on where the cache data finally lives, the new support might or might not use the upstream Cross/ directory as the current contents date from the Zaurus support and don't appear to be that useful for current versions of perl.

The cache data itself has several problems:

  1. It is tightly tied to the version of perl which generated it.

  2. It is, as expected, architecture-dependent

  3. It is, unfortunately, very sensitive to the specific configuration used by the distribution itself


That last point is important because it means that the cache data is not useful upstream as a block. It also means that generating the cache data for a specific Debian package means running the generation code on the native architecture with all of the Debian build-dependencies installed for the full perl build. This is going to complicate the use of this method for new architectures like arm64.

My objective for the long term maintenance of this code is to create sufficient data that a new architecture can be bootstrapped by judicious use of some form of template. Quite how that works out, only time will tell. I expect that this will involve isolating the data which is truly architecture-specific which doesn't change between perl versions from the data which is related to the tests for build-dependencies which does change between perl versions and then work out how to deal with any remainder. A new architecture for a specific perl version should then just be a case of populating the arch-specific data such as the size of a pointer/char and the format specifiers for long long etc. alongside the existing (and correct) data for the current version of perl.

Generating the cache data natively

The perl build repeats twice (three builds in total) and each build provides and requires slightly different cache data - static, debug and shared. Therefore, the maintenance code will need to provide a script which can run the correct configuration step for each mode, copy out the cache data for each one and clean up. The script will need to run inside a buildd chroot on a porter box (I'm looking at using abel.debian.org and harris.debian.org for this work so far) so that the derived data matches what the corresponding Debian native build would use. The data then needs slight modification - typically to replace the absolute paths with PERL_BUILD_DIR. It may also be necessary to change the value of cc, ranlib and other compiler-related values to the relevant cross-compiler executables. That should be possible to arrange within the build of the cache data support package itself, allowing new cache files to be dropped in directly from the porter box.

The configuration step may need to be optimised within debian/rules of perl itself as it currently proceeds on from the bare configuration to do some actual building but I need to compare the data to see if a bare config is modified later. The test step can be omitted already. Each step is performed as:

DEB_BUILD_OPTIONS="nocheck" fakeroot debian/rules perl.static


That is repeated for perl.debug and libperl.so.$(VERSION) where $VERSION comes from :

/bin/bash debian/config.debian --full-version


The files to be copied out are:


There is a lot of scope for templating of some form here, e.g. config.h.debug is 4,686 lines long but only 41 of those lines differ between amd64 and armhf for the same version of perl (and some of those can be identified from existing architecture-specific constants) which should make for a much smaller patch.

Architecture-specific cache data for perl

So far, the existing patches only deal with armel and armhf. If I compare the differences between armel & armhf, it comes down to:

  1. compiler names (config.h.*)

  2. architecture names (config.sh.*)

  3. architecture-based paths (config.sh.*)


However, comparing armel and armhf doesn't provide sufficient info for deriving arm64 or mips etc. Comparing the same versions for armhf and amd64 shows the range of differences more clearly. Typical architecture differences exist like the size of a long, flags to denote if the compiler can cast negative floats to 32bit ints and the sprintf format specifier strings for handling floats and doubles. The data also includes some less expected ones like:

armhf: #define Netdb_host_t const void * /**/
amd64: #define Netdb_host_t char * /**/


I'm not at all sure why that is arch-specific - if anyone knows, email codehelp @ d.o - same address if anyone fancies helping out ....

Cross-builds and debclean

When playing with the cross-build, remember to use the cross-build clean support, not just debclean:


dpkg-architecture -aarmel -c fakeroot debian/rules clean


That wasted quite a bit of my time initially with having to blow away the entire tree, unpack it from original apt sources and repatch it. (Once Wheezy is out, may actually investigate getting debclean to support the -a switch).

OK, that's an introduction, I'm planning on pushing the cross-build support code onto github soon-ish and doing some testing of the cross-built perl binaries in a chroot on an armel box. I'll detail that in another blog post when it's available.

Next step is to look at perl 5.16 and then current perl upstream git to see how to get Makefile.SH fixed for the long term.

17 November 2012

Russell Coker: Geeky Jeans

It s likely that most people make things like comfort, style, and price the main criteria when purchasing clothes. But there are other things that can be more important such as the ability to fit a phone in the pocket. My last pair of jeans was from Rivers (one of the more affordable Australian clothing stores which also has online sales) [1]. They were the long leg version and have front pockets that are 28cm deep (measured from the bottom of the pocket to the lowest part of the lip) and 15cm wide (a 15cm ruler will barely fit sideways in the pocket). I ve just got some new jeans from Rivers which are the regular leg length, they have front pockets which are 21cm deep and slightly more than 16cm wide. The old pair could fit a Nexus 7 tablet in the front pocket. The new pair should more easily fit such a tablet in the pocket but it might be less comfortable to walk with the tablet in the pocket. I don t plan to try using my front pocket for a tablet (I d be more likely to use a backpack or my Scott e vest [2]), but a Galaxy Note 2 (which is about the largest device that most people would want in their pocket) would fit nicely. I find the Rivers jeans to be quite comfortable and I like the way they look. They also only cost $25 online or $30 in the store. When I bought my first pair before I even had a 4 phone they were good value and they will be even better value early next year when I get a bigger phone. Even though Rivers jeans may wear out faster than more expensive brands, for $30 it s easy to just buy a few pairs at a time. During an email discussion of geeky clothing the issue of women s clothes having fewer and smaller pockets was raised. Unfortunately I didn t think to measure the pockets in women s jeans when I was at the Rivers to discover whether they have big pockets too. I ll do that next time I m in the area.

15 November 2012

Ian Campbell: Xen Microcode Loading in Wheezy

No sooner had Henrique de Moraes Holschuh announced the availability of improved microcode updating in Wheezy (sadly non-free) than Stephan Seitz pointed out that this support was broken for systems running Xen, which is unfortunate since as Henrique notes the impact of not applying such an update could be pretty much anything. There's a bit of a saga here: a patch has been available for several years which adds Xen support to the kernel's microcode driver in a nicely self contained way. However the Linux maintainers have decided that loading microcode from the kernel is too late and it should be done earlier (e.g. by the bootloader, or some shim between the bootloader and the kernel) in order to be effective and on that basis rejected the Xen patch implementing the existing kernel based scheme. In the meantime Xen has implemented early loading of microcode via an additional multiboot module passed by the bootloader and this was released in Xen 4.2 in September. (so far I've not seen any progress on making any change to the native side but that's by the by). However Wheezy is going to be shipping with Xen 4.1 which doesn't yet have this support in it. So we are basically faced with two choices, either we backport the support from Xen 4.2 or we apply the kernel patch. The downside of backporting rather than taking the kernel patch is that the backport is non-trivial (although not totally unwieldy) while the kernel patch is pretty much entirely self-contained. The other downside of early the early loading scheme in general (for either native or Xen) is the need to update tools (e.g. update-grub) as well as modifying the microcode packaging to put the files somewhere that the bootloader can see them (so /boot). This certainly can't happen for Wheezy at this stage which basically rules out the backport approach. So, the upshot is I've filed Debian bug #693053 and applied the kernel patch to our tree. It's all a bit unsatisfactory but at least microcode loading will work for users running Xen on Wheezy. With any luck for Jessie the native early microcode loading code will have landed as well and the pain of the infrastructure updates can be shared.

Peter Makholm: Deploying Half of CPAN on Debian using Carton

As systems developer I like just being able to pick the most useful Perl modules from CPAN and preferably use quite recent versions of the modules I need. On the other hand our systems administrators prefer running Debian Stable and deploy software using .deb packages. I do not want to maintain hundred small packages by hand. I just want a single package with all needed the Perl dependencies for a given project. After looking at different possibilities for bundling Perl dependencies I finally found Carton written by Tatsuhiko Miyagawa. Given a list of your direct Perl dependencies, Carton can maintain the full dependency chain and manage which versions of each module you deploy. One of the nice features is that you get a reproducible set of versions each time and not just the latest, greatest, and recently broken versions. The initial step is to set Carton up to install the dependencies for my project. This part is done just like the Carton documentation describes. To ensure independence from whatever is available on CPAN I end by calling carton bundle. The next step is to write a small Perl module making sure you load modules from the right place. I end up with installing modules in /usr/lib/projectname. My module looks like this:
package My::Bundle::ProjectName;
=head1 NAME
My::Bundle::ProjectName - Bundle of CPAN modules used by ProjectName
=cut
our $VERSION = 1.0;
use Config;
use Carp;
use Carp::Heavy;  # Ensure that all parts of Carp are loaded before @INC mangling
use File::Spec;
our $BASE = '/usr/lib/projectname';
sub import  
    carp (__PACKAGE__ . " bundle not installed")
        unless -d $BASE;
    unshift @INC, 
        File::Spec->catdir( $BASE, 'lib', 'perl5' ),
        File::Spec->catdir( $BASE, 'lib', 'perl5', $Config archname );
 
1;
Now I need to install this module and all the modules maintained by Carton. I do this by a enhanced Module::Build script:
use v5.10.0;
use strict;
use warnings;
use Module::Build;
my $class = Module::Build->subclass(
    class => "My::Builder",
    code  => q 
        sub ACTION_build  
            my $self = shift;
            $self->SUPER::ACTION_build;
            !system 'carton', 'install', '--cached';
         
        sub ACTION_install  
            my $self = shift;
            $self->SUPER::ACTION_install;
            
            my $installdir = $self->destdir . "/usr/lib/projectname";
            system 'install', '-d', $installdir;
            !system 'cp', '-a', 'local/lib', $installdir;
         
     ,
);
$class->new(
    module_name       => 'My::Bundle::ProjectName',
    dist_author       => 'Peter Makholm <peter>',
    dist_version_from => 'lib/My/Bundle/ProjectName.pm',
    add_to_cleanup    => [ 'local/lib', 'local/bin' ],
)->create_build_script;
The last step is to create a .deb package. This is mostly just a standard debian/ directory using debhelper. You just need to override the dh_perl to pick up the correct binary dependencies. So this is my debian/rules:
#!/usr/bin/make -f
%:
        dh $@
override_dh_perl:
        dh_perl /usr/lib/projectname/lib
That s it. Then all the scripts and applications making up my project just needes to use My::Bundle::ProjectName and the rest is automatic. I am thinking about enhancing Carton to provide individual .deb packages instead of just installing the modules. This would make it quite easy to maintain the packages in a more The Debian Perl Group-compatible way. That could be cool.

Matthew Garrett: More in the series of bizarre UEFI bugs

A (well, now former) coworker let me know about a problem he was having with a Lenovo Thinkcentre M92p. It booted Fedora UEFI install media fine, but after an apparently successful installation refused to boot. UEFI installs on Windows worked perfectly. Secure Boot was quickly ruled out, but this could still have been a number of things. The most interesting observation was that the Fedora boot option didn't appear in the firmware boot menu at all, but Windows did. We spent a little while comparing the variable contents, gradually ruling out potential issues - Linux was writing an entry that had an extra 6 bytes in a structure, for instance[1], and a sufficiently paranoid firmware implementation may have been tripping up on that. Fixing that didn't help, though. Finally we tried just taking the Windows entry and changing the descriptive string. And it broke.

Every UEFI boot entry has a descriptive string. This is used by the firmware when it's presenting a menu to users - instead of "Hard drive 0" and "USB drive 3", the firmware can list "Windows Boot Manager" and "Fedora Linux". There's no reason at all for the firmware to be parsing these strings. But the evidence seemed pretty strong - given two identical boot entries, one saying "Windows Boot Manager" and one not, only the first would work. At this point I downloaded a copy of the firmware and started poking at it. Turns out that yes, actually, there is a function that compares the descriptive string against "Windows Boot Manager" and appears to return an error if it doesn't match. What's stranger is that it also checks for "Red Hat Enterprise Linux" and lets that one work as well.

This is, obviously, bizarre. A vendor appears to have actually written additional code to check whether an OS claims to be Windows before it'll let it boot. Someone then presumably tested booting RHEL on it and discovered that it didn't work. Rather than take out that check, they then addded another check to let RHEL boot as well. We haven't yet verified whether this is an absolute string match or whether a prefix of "Red Hat Enterprise Linux" is sufficient, and further examination of the code may reveal further workarounds. For now, if you want to run Fedora[2] on these systems you're probably best off changing the firmware to perform a legacy boot.

[1] src/include/efi.h: uint8_t padding[6]; /* Emperically needed */, says the efibootmgr source code. Unhelpful.
[2] Or Ubuntu, or Suse, or

comment count unavailable comments

14 November 2012

Rudy Godoy: Function testing using function pointers in C++

I do like the way programmers think in terms of DRY, for instance, among other forms of optimization. A couple of days ago I wanted to test different implementations of the same algorithm in C++. The usual approach would be to implement those algorithms, call them in main() and see the results. A more interesting approach would be to write a validator for each function and print whether the result is correct or not. Since both are boring and break the rule of DRY, I ve decided to go complicated but productive, so I ve implemented a function testing method using function pointers that will allow me to write a single function and use it to test the algorithms using different input and expected results. Let s say I have 3 algorithms that implement Modular Exponentiation.
uint64 modexp1new(const uint64 &a, const uint64 &_p, const uint64 &n)
...


uint64 modexp2(const uint64 &_a, const uint64 &_p, const uint64 &_n)
...


uint64 modexp3(const uint64 &a, const uint64 &p, const uint64 &n)
...

If you are not familiar with the syntax, const means that the parameter is constant, thus it will not be modified inside the function. The & means that the parameter is passed as a reference, read the memory address of the calling variable, so we don t overload RAM with new variables. Now, the testing part. For my very own purposes I just want to test those particular functions and know whether if the result they output conforms what I do expect. Some people call this unit testing.
I do also want to test it in different scenarios, meaning input and output. So, I m creating a function that will get the three parameters required to call the functions, and a fourth parameter that is the value I do expect to be the correct answer. Now, since I don t want to repeat myself writing a call for each case, I m going to create a function pointer that is an array and has the function s address as their value. That way I can call them in a loop, and voila! we are done. Finally, after calling the function I check the result with the expected value and print an OK or FAIL. Tricky part for this could be understanding the function pointer. Two things to consider: the returning value from the referenced functions has to be the same for all. Second: the input definition for each function has to be similar too. This is important because function pointers are just pointers and they need to know the size of the data type in order to navigate the memory. For this sample code the uint64 is a typedef for long long, of course. Full code is below.
void testAlgos(const uint64 &a, const uint64 &e, const uint64 &n, const uint64 &r)
uint64 (*fP[3])(const uint64&, const uint64&, const uint64&) = &modexp1new, &modexp2, &modexp3 ;
uint64 t;
for(int i=0; i<3; i++) t = fP[i](a, e, n); std::cout < < "F(" << i << ") " << t << "\t";
if (t == r)
std::cout << "OK";
else
std::cout << "FAIL";

std::cout << std::endl;

Now that I have this, I can use it in main this way.
int main()
uint64 a = 3740332;
uint64 e = 44383;
uint64 n = 3130164467;
uint64 r = 1976425102; testAlgos(a, e, n, r); a = 404137263;
r = 2520752541; testAlgos(a, e, n, r); a = 21;
e = 3;
n = 1003;
r = 234;
testAlgos(a, e, n, r); return 0;
Resulting in:
F(0) 657519405 FAIL
F(1) 1976425102 OK
F(2) 657519405 FAIL
F(0) -2752082808 FAIL
F(1) 2520752541 OK
F(2) -2752082808 FAIL
F(0) 234 OK
F(1) 234 OK
F(2) 234 OK
I m happy about not having to write tests like this for all use cases:
std::cout < < "f1: " << modexp1new(a, e, n) << std::endl;
std::cout << "f2: " << modexp2(a, e, n) << std::endl;
std::cout << "f3: " << modexp3(a, e, n) << std::endl;
Now, happiness can be more exciting if I do use templates so I can test any function independent of the data type for the returning and input values. You know, more abstraction. Homework if time allows! Have fun!
Related posts:
  1. Compute Clusters Integration for Debian Development and Building Report 4 Hello, this is the fourth report for the project. First,...
  2. Compute Clusters Integration for Debian Development and Testing Report 3 Hello this is the third report regarding my project. As...
  3. Compute clusters for Debian development and building final report Compute clusters for Debian development and building final report...

9 November 2012

Olivier Berger: Using a DreamPlug with Debian as an IPv4/IPv6 router for a home LAN behind a FreeBox

The goal of this article is to describe the setup I've made for my home LAN behind a FreeBox (French ISP Free.fr).

1 Principles
The FreeBox is configured :
  • in router mode (with a local IPv4 network on 192.168.0.x/24)
  • with IPv6 support activated (by convention, we'll use the 2a01:e35:1234:5678::/64 prefix below, to represent the IPv6 prefix provided by the FreeBox)
Behind the FreeBox, I have connected a DreamPlug computer (running Debian wheezy with a dual-boot setup as sold by NewIT) on its eth0 etherlink port. Behind the DreamPlug, I have connected a switch (to its eth1 ethernet port), on which the LAN's machines will be connected, on a dual IPv4/IPv6 network. The IPv4 will be on 192.168.4.0/24 (addresses for machines connected to the switch will be static) while the IPv6 will use the addresses auto-generated on the FreeBox's prefix. The DreamPlug will also provide a wifi network (in master mode, through its uap0 interface), only IPv4 on 192.168.3.0/24. On the DreamPlug, the outgoing IPv4 traffic will be masqueraded, while the IPv6 traffic will be bridged so that machines on the LAN can be reached from the Internet directly, passing through the eth0 eth1 bridge. The DreamPlug will also act as a filter/firewall to only allow selected traffic to reach the LAN machines (in particular for the IPv6, since the machines may be reached directly from the Internet otherwise).

1.1 Alternatives
There are a lot of howtos describing the setup of FreeBox + IPv6, but I couldn't find one that was exactly reflecting the kind of setup I have. Some dealt with a FreeBox setup in bridge mode (i.e. not the router mode), but in case the DreamPlug isn't so much reliable, or I've messed with the config, I'm tempted to keep the possibility to connect directly to the FreeBox. Others didn't use the same setup with a router distributing both wired and wifi networks. And note described exactly the setup for a DreamPlug running Debian wheezy and using ferm as a firewall. No, there's no NIH syndrom here ;)

2 DreamPlug router configuration

2.1 Interfaces configuration
The /etc/network/interfaces on the DreamPlug looks like the following (kudos to Dag Bertelsen's Setting up Dreamplug as a router with Shorewall firewall for the wifi configuration bits):
# Note: Order of the interfaces declarations matters

auto lo
iface lo inet loopback

auto uap0
iface uap0 inet static
post-up /usr/bin/uaputl sys_cfg_ssid dream-uAP-XXXX
# Modus WPA2
post-up /usr/bin/uaputl sys_cfg_protocol 32
# WLAN password
post-up /usr/bin/uaputl sys_cfg_wpa_passphrase "WHATEVER PASSWORD"
# Set the Cipher
post-up /usr/bin/uaputl sys_cfg_cipher 8 8
# automatic channel
post-up /usr/bin/uaputl sys_cfg_channel 0 1
post-up /usr/bin/uaputl bss_start
# we deliver addresses for the Wifi with DHCP
post-up /usr/sbin/service udhcpd start
address 192.168.3.1
netmask 255.255.255.0

# Only IP6 traffic passes through the router between the Freebox on eth0 and the LAN on eth1
auto br0
iface br0 inet manual
bridge_ports eth0 eth1
bridge_maxwait 0
pre-up ebtables -t broute -A BROUTING -p ! ipv6 -j DROP
down ebtables -t broute -F

# Connected to the FreeBox, with DHCP in IPv4, masquerading outgoing IPv4 traffic
auto eth0
iface eth0 inet dhcp
up iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
down iptables -t nat -F
post-up sysctl -q -w net.ipv4.ip_forward=1
iface eth0 inet6 auto

# Connected to the LAN with a static IPv4 address
auto eth1
iface eth1 inet static
address 192.168.4.1
netmask 255.255.255.0

2.2 DHCP server configuration
The DHCP config for udhcpd is as such (in /etc/udhcpd.conf):
start 192.168.3.100
end 192.168.3.200
interface uap0
opt lease 86400
opt router 192.168.3.1
opt subnet 255.255.255.0
opt dns 192.168.3.1
opt domain localdomain
max_leases 101
lease_file /var/lib/udhcpd.leases
auto_time 5

2.3 Firewall configuration
The firewall I use is ferm. Here's the /etc/ferm/ferm.conf configuration (YMMV). Thanks to Ren Garcia's R gles de firewall pour un pont IPv6 for inspiration :
## Interfaces of the router

# Our LAN on a switch
@def $DEV_PRIVATE = eth1;

# The ISP router
@def $DEV_WORLD = eth0;

# Our Wifi LAN
@def $DEV_WIFI = uap0;


## IP networks of our LAN

# IPv4 (includes ethernet switch and wifi)
@def $NET_PRIVATE = 192.168.0.0/16;

# IPv6 (the prefix granted by the ISP)
@def $NET6_PRIVATE = 2a01:e35:1234:5678::/64;

# IPv6 Local link
@def $LINK6 = fe80::/10;

# Specific machines on the LAN
@def $MACHINE_MYDESKTOP = 2a01:0e35:1234:5678:1234:5678:9abc:def0;

# Special function to log dropped packets
@def &DROP() =
mod limit limit 1/second limit-burst 10 LOG log-prefix "$DOMAIN-$CHAIN: ";
DROP;


# IPv4 iptables rules
table filter
chain INPUT
policy DROP;

# connection tracking
mod state state INVALID DROP;
mod state state (ESTABLISHED RELATED) ACCEPT;

# allow local packet
interface lo ACCEPT;

# respond to ping
#proto icmp ACCEPT;
proto icmp icmp-type echo-request ACCEPT;

# allow IPsec
proto udp dport 500 ACCEPT;
proto (esp ah) ACCEPT;

# allow SSH connections from the private network and from some
# well-known internet hosts
#saddr ($NET_PRIVATE 123.45.67.89) proto tcp dport ssh ACCEPT;
saddr $NET_PRIVATE proto tcp dport ssh ACCEPT;

# we provide DNS and SMTP services for the internal net
interface $DEV_PRIVATE saddr $NET_PRIVATE
proto (udp tcp) dport domain ACCEPT;
proto tcp dport smtp ACCEPT;


interface $DEV_WIFI
# DHCP
proto udp dport 67:68 ACCEPT;
proto (udp tcp) dport domain ACCEPT;

proto udp dport netbios-dgm DROP;


# LOG all dropped packets
proto (udp tcp) &DROP();



# outgoing connections are not limited
chain OUTPUT policy ACCEPT;

chain FORWARD
policy DROP;

# connection tracking
mod state state INVALID DROP;
mod state state (ESTABLISHED RELATED) ACCEPT;

# connections from the internal net to the internet or to other
# internal nets are allowed
interface $DEV_PRIVATE ACCEPT;
interface $DEV_WIFI ACCEPT;

# LOG all dropped packets
proto (udp tcp) &DROP();



# IPv4 masquerade / NAT support

table nat
chain POSTROUTING
# masquerade private IP addresses
saddr $NET_PRIVATE outerface $DEV_WORLD MASQUERADE;



## IPv6:

domain ip6
table filter

chain INPUT
policy DROP;

# Internal traffic authorized
saddr $NET6_PRIVATE ACCEPT;
saddr $LINK6 ACCEPT;

# accept multicast
daddr ff00::/8 ACCEPT;

interface lo ACCEPT;

# For public traffic
# Only DNS resolution + NTP acceped for UDP
interface br0
proto udp

sport domain ACCEPT;

dport ntp ACCEPT;

&DROP();


# TCP on ports < 1024 forbidden except for SSH
proto tcp

dport ssh ACCEPT;

dport 0:1023 DROP;


# Accepted established traffic
mod state state INVALID DROP;
mod state state (ESTABLISHED RELATED) ACCEPT;

# incoming connections forbidden
proto tcp syn DROP;

# Accept ping
proto icmpv6 ACCEPT;


proto (udp tcp) &DROP();

chain FORWARD
policy DROP;

# All local traffic accepted
saddr $LINK6 ACCEPT;

# All ping accepted unless coming from internet
saddr ! $NET6_PRIVATE proto icmpv6 icmp-type echo-request DROP;
proto icmpv6 ACCEPT;

# Allow SSH to specific machines
proto tcp daddr $MACHINE_MYDESKTOP dport ssh ACCEPT;

# Accept established traffic
mod state state INVALID DROP;
mod state state (ESTABLISHED RELATED) ACCEPT;

# Accept outgoing traffic
saddr $NET6_PRIVATE ACCEPT;

# Accept local outgoing traffic
saddr ::1 ACCEPT;

proto (udp tcp) &DROP();




# ebtables
domain eb chain FORWARD
policy DROP;

# Only accept bridged traffic for IPv6
logical-in br0 logical-out br0
#ebtables -t broute -A BROUTING -p ! ipv6 -j DROP
proto IPv6 ACCEPT;


2.4 DNS config

The DreamPlug will run bind9 to cache the DNS for the wifi network (default config is OK out of the box).

3 Configuration for the LAN machines

3.1 Connection with Wifi
Machines connected with the WPA wifi network should get an IPv4 only address via DHCP on the 192.168.3.0/24 network. There's nothing to be configured specifically.

3.2 Ethernet connection on the switch
Machines connected via ethernet cables to the switch can run either with IPv4 config, in which case they should be configured statically for the 192.168.4.0/24 network, using the router on 192.168.4.1 as a gateway, or with IPv6 auto config. On wheezy, the /etc/network/interfaces will look like :
auto eth1
iface eth1 inet static
address 192.168.4.42
netmask 255.255.255.0
gateway 192.168.4.1
dns-nameservers 212.27.40.241 212.27.40.240

iface eth1 inet6 auto

4 To do list

The setup works fine, but there are a few things I'd like to improve :
  • running a regular Debian distro on the DreamPlug (the kernel used at the moment is patched for the wifi driver in master mode support), possibly using only Free Software.
  • using subnets for the IPv6 machines (but that seems not exactly obvious, requiring NDP proxying and stuff), so that the privacy is improved (the IPv6 auto generated global addresses depend on the MAC addresses hmmm probably not optimal).
  • adding more stuff to the DreamPlug router so that it becomes a full fledged FreedomBox.

5 References

Date: 2012-11-09 16:30:34 CET

6 November 2012

MJ Ray: Why co-ops and social enterprises should avoid publishing Word files

I sometimes ask other co-ops and social enterprises to publish things as web pages, PDFs, ODFs or basically any standard format instead of Microsoft Word Doc files. Doc files have the practical problem that they look different even in different versions of Microsoft Word, but also, Microsoft is not a co-operative (a private-sector firm oft accused of bring a monopolist, in fact!), whereas PDF, ODF and so on are more co-operatively developed, so I feel that we should support the better alternatives. The usual first move is to PDFs. I don t like them as much as web pages (it s a file to download and open in an app or plugin, rather than something I can just read without interrupting my flow), but the process is basically the same. Just upload a PDF where you would have uploaded a Doc. Occasionally, I get a reply saying that PDF is no better than Word because Adobe are also a private-sector supplier and therefore not much different from Microsoft (not as big, though?), while doc files are an ISO standard too. I m careful to suggest using ISO PDFs rather than Adobe PDFs. Although PDF started with Adobe, it was given to an independent ISO process (currently co-chaired by someone from Microsoft, which amuses me) and so software to read it has been developed by a wide range of people, including our co-op in a small part. You can find some non-Adobe software for reading PDFs at http://pdfreaders.org there is also alternative PDF creation software, such as that built-in to libreoffice, but I don t know of a good similar listing for them. Microsoft have only given part of Doc (called Docx) to the ISO process and there is a limited range of other readers for it, which suffer the same different on every version problem as Word as far as I ve seen. There s also the added complication that Microsoft Office reportedly won t comply with the Strict standard before Office 2013 More widely, Microsoft are so un-cooperative that they don t seem to want to share a marketplace with anyone else and have been hauled in front of regulators numerous times for monopoly offences. They seem to be on their way there again. So Adobe is not great (does its best to continue marketing and extending PDF as if it had sole control), but Microsoft is much worse. ISO ODF and web pages are other possibilities. The ODF standards are mainly developed by the democratic non-profit Organization for the Advancement of Structured Information Standards, while web standards are mainly developed by an unincorporated trade consortium, the W3C. While they are not themselves co-operatives, co-operatives can be members of both of those. So, whatever the problem, please prefer to publish web pages, PDF or ODF files, rather than Microsoft Word Doc files.

Russ Allbery: Review: Cerebus

Review: Cerebus, by Dave Sim
Series: Cerebus #1
Publisher: Aardvark-Vanaheim
Copyright: August 1987
Printing: July 2003
ISBN: 0-919359-08-6
Format: Graphic novel
Pages: 546
Cerebus is something of a legend in comics. Begun in December of 1977 by Dave Sim, it was one of the first entirely independent, self-published comics in a field dominated by the large work-for-hire companies like Marvel and DC. It ran for 300 issues and nearly 27 years and became one of the most influential independent comic books of all time, in part due to Sim's outspoken views in favor of creator rights and his regular use of the editorial pages in Cerebus issues to air those views. This collection (the first "phonebook") collects issues 1 through 25, with one of the amazing wrap-around covers that makes all of the phonebooks so beautiful (possibly partly by later Cerebus collaborator Gerhard, although if so it's uncredited so far as I can tell). Cerebus reliably has some of the best black-and-white art you will ever see in comics. There is some debate over where to start with Cerebus, and a faction that, for good reasons, argues for starting with the second phonebook (High Society). While these first twenty-five issues do introduce the reader to a bunch of important characters (Elrod, Lord Julius, Jaka, Artemis Roach, and Suenteus Po, for example), all those characters are later reintroduced and nothing that happens here is hugely vital for the overall story. It's also quite rough, starting as Conan parody with almost no depth. The first half or so of this collection features lots of short stories with little or no broader significance, and the early ones are about little other than Cerebus's skills and fighting abilities. That said, when reading the series, I like to start at the beginning. It is nice to follow the characters from their moment of first introduction, and it's delightful to watch Sim's ability grow (surprisingly quickly) through the first few issues. Cerebus #1 is bad: crude, simplistic artwork, almost nothing in the way of a story, and lots of purple narration. But flipping forward even to Cerebus #6 (the first appearance of Jaka), one sees a remarkable difference. By Cerebus #7, Cerebus looks like himself, the plot is getting more complex, and Sim is clearly hitting his stride. And, by the end of this collection, the art has moved from crude past competent and into truly beautiful in places. It's one of the few black-and-white comics where I never miss color. The detailed line work is more enjoyable than I think any coloring could be. The strength of Cerebus as an ongoing character slowly emerges from behind the parody. What I like the most about Cerebus is that he's neither a predestined victor (apart from the early issues that follow the Conan model most closely) nor a pure loner who stands apart from the world. He gets embroiled in political affairs, but almost always for his own reasons (primarily wealth). He has his own moral code, but it's fluid and situational; it's the realistic muddle of impulse and vague principle that most of us fall back on in our everyday life, which is remarkably unlike the typical moral code in comics (or even fiction in general). And while he is in one sense better and more powerful than anyone else in the story, that doesn't mean Cerebus gets what he wants. Most stories here end up going rather poorly for him, forcing daring escapes or frustrating cutting of losses. Sim quickly finds a voice for Cerebus that's irascible, wise, practical, and a bit world-weary, as well as remarkably unflappable. He's one of the best protagonists in comics, and that's already clear by the end of this collection. Parody is the focus of these first issues, which is a mixed bag. The early issues are fairly weak sword-and-sorcery parody (particularly Red Sonja, primarily a vehicle for some tired sexist jokes) and worth reading only for the development in Sim's art style and the growth of Cerebus as a unique voice. Sim gets away from straight parody for the middle of the collection, but then makes an unfortunate return for the final few issues, featuring parodies of Man-Thing and X-Men that I thought were more forced than funny. You have to have some tolerance for this, and (similar to early Pratchett) a lot of it isn't as funny as the author seems to think it is. That said, three of Sim's most brilliant ongoing characters are parodies, just ones that are mixed and inserted into the "wrong" genres in ways that bring them alive. Elrod of Melvinbone, a parody of Moorcock's Elric of Melnibone who speaks exactly like Foghorn Leghorn, should not work and yet does. He's the source of the funniest moments in this collection. His persistant treatment of Cerebus as a kid in a bunny suit shouldn't be as funny as it is, but it reliably makes me laugh each time I re-read this collection. Lord Julius is a straight insertion of Groucho Marx who really comes into his own in the next collection, High Society, but some of the hilarious High Society moments are foreshadowed here. And Artemis Roach, who starts as a parody of Batman and will later parody a huge variety of comic book characters, provides several delightful moments with Cerebus as straight man. I'm not much of a fan of parody, but I still think Cerebus is genuinely funny. High Society is definitely better, but I think one would miss some great bits by skipping over the first collection. Much of what makes it work is the character of Cerebus, who is in turn a wonderful straight man for Sim's wilder characters and an endless source of sharp one-liners. It's easy to really care about and root for Cerebus, even when he's being manipulative and amoral, because he's so straightforward and forthright about it. The world Sim puts him into is full of chaos, ridiculousness, and unfairness, and Cerebus is the sort of character to put his head down, make a few sarcastic comments, and then get on with it. It's fun to watch. One final note: I've always thought the "phonebook" collections were one of Sim's best ideas. Unlike nearly all comic book collections, a Cerebus phonebook provides enough material to be satisfying and has always felt like a good value for the money. I wish more comic book publishers would learn from Sim's example and produce larger collections that aren't hardcover deluxe editions (although Sim has an admitted advantage from not having to reproduce color). Followed by High Society. Rating: 7 out of 10

Cyril Brulebois: Network daemons

While I m not a big fan of clicky-clicky applications, I find it quite nice to be able to select some WEP/WPA settings in a GUI (as opposed to configuring wpa_supplicant manually) once in a while (new home, new couch by a friend s, new phone), and be done with it forever. Having mostly used Xfce over the years, I naturally came to trying out Wicd around 2006 (according to Wikipedia), and it looked like it did the job. It s been on all my laptops since then, and things were quite fine until the last laptop switch. For some (to be determined) reasons, that one comes up with a soft-blocked (rfkill) wireless LAN, which makes Wicd the biggest pain I have ever experienced as far as networking is concerned (see 4th bug report below). So I finally decided to try out that network-manager thing people apparently love to hate. A few bugs were filed in the process: I think nm-applet is the only Gtk3 application I have, meaning slightly bad visuals (no integration with the current Gtk2 theme for other applications), but besides that, everything is now running smoothly. Bottom line: contrary to Wicd, NetworkManager just works! I can t tell for sure since I m not using Gnome, but I m not sure why somebody would want to use Wicd there as it s clearly inferior on a technical level. Maybe something about the GUI/CLI? At least NetworkManager knows about netlink sockets, doesn t waste resources through useless polling, and is able to figure out when connections need kicking. That one just does its job. What triggered this Wicd NetworkManager switch is a look at the linux source, which reminded me of some strange polling noticed while playing around with auditd, as mentioned in #692417; that and the no network at boot-up unless one triggers several actions in a Wicd client bit.

5 November 2012

MJ Ray: Seize the Media! @theBoyler @coopsutd

image

1 Angel Square

So when I arrived on Thursday (ten minutes early despite a cancelled train, thanks to help from the cooperative fellow traveller mentioned last post), I was in time to go to a workshop on media cooperatives led by @theBoyler. It wasn t what I expected. It seemed to concentrate on the opportunity presented by the current awful state of news media companies in many countries and the technology-driven changes to their businesses. For example, WordPress is a viable way to start online first, then move into print later. There are already news co-ops in places where there would be no local news media. The challenge well be how to overcome what I d call spoiling or scorched earth tactics from departing media companies. Some local news media owners are looking for a way out. One obvious one is to turn a local title into a minor variation on a regional-or-worse service or publication, maybe by selling it to a bigger media business. That loses some audience but many will put up with it, so it avoids creating an obvious gap in the market for more truly local reporting. Much else was covered and the audience suggested tons of examples. I m not sure we reached many firm conclusions, but there seem quite a few examples to learn from (Morning Star, taz, ) and Dave Boyle s work is continuing

Next.

Previous.