Search Results: "claus"

29 October 2013

Matthew Palmer: Why Your Puppet Module Sucks

I use Puppet a lot at work, and I use a lot of modules written by other people, as well as writing quite a number of my own. Here s a brief list of reasons why I might say that your module sucks.

1. You use global variables This would have to be both the most common idiom that just makes my teeth grind. Defined types exist for a bloody reason. Global variables make it incredibly difficult to reason about what is going to happen when I use a particular global variable (see also You don t write documentation ), and I get no feedback if I typo a global variable name. Add in a healthy heaping of lack of namespacing , and you ve basically guaranteed that your module will be loudly cursed.

2. You use parameterised classes I ve never managed to work out why parameterised classes even exist. They ve got all the problems of regular classes, as well as all the problems of types. There was a fantastic opportunity with parameterised classes to fix some of the really annoying things about regular resources, such as doing conflict checking on parameters and only barfing if there was a conflict but no, if you define the same class twice, even with identical parameters, Puppet will smack you on the hand and take away your biscuit. FFFFFFUUUUUUUUUUUU-

3. You fail at using fail() Modules are supposed to be reusable. That s their whole reason for existence. One of the benefits of Puppet is that you can provide an OS-independent interface for configuring something. However, unless you re some absolute God of cross-platform compatibility, you will only be writing your module to support those OSes or distros that you, personally, care about. That s cool if you don t know anything about OS X, you probably shouldn t be guessing at how to support something on that platform anyway. However, when you do have some sort of platform-specific code, for the love of pete, have a default or else clause that calls fail() with a useful and descriptive error message, to indicate clearly that you haven t included support for whatever environment the module s being used in. Failing to do this can cause some really spectacular explosions, because the rest of your module assumes that certain things have been done in the platform-specific code, and when it hasn t hoo boy.

4. You don t write documentation Yes, it isn t easy to write good documentation. I know that. But without documentation, your module is going to be practically unuseable. So if you don t have docs, your module is basically useless. Well done, you.

5. You have undeclared dependencies This also includes declaring your dependencies in a non-machine-parseable manner; I m not going to grovel through your README for the list of other modules I might need; I have machines to do that kind of thing for me. If I try to use your module, and it craps out on trying to reference some sort of type or class that doesn t appear at all related to your module, I will call into question your ancestry, and die a little more inside.

6. You use common packages without even trying to avoid the pitfalls OK, to be fair this is, ultimately, Puppet s grand fuckup, not yours, but you at least need to pretend to care There is no sane, standardised way in Puppet for multiple modules to install the same package. Let s say that a module I write needs to have a compiler. So I package "gcc": . Then someone else s module also wants a compiler, so it also package "gcc": . FWAKOOM! says Puppet. What the fuck? says the poor sysadmin, who just wanted both a virtualenv and an RVM environment on the one machine. Basically, using packages is going to make your module suck. Does that mean that this makes wide distribution of a large repository of modules written by different people nigh-on impossible? Yes. Fantastic.

Matthew Palmer: Why Your Puppet Module Sucks

I use Puppet a lot at work, and I use a lot of modules written by other people, as well as writing quite a number of my own. Here s a brief list of reasons why I might say that your module sucks.

1. You use global variables This would have to be both the most common idiom that just makes my teeth grind. Defined types exist for a bloody reason. Global variables make it incredibly difficult to reason about what is going to happen when I use a particular global variable (see also You don t write documentation ), and I get no feedback if I typo a global variable name. Add in a healthy heaping of lack of namespacing , and you ve basically guaranteed that your module will be loudly cursed.

2. You use parameterised classes I ve never managed to work out why parameterised classes even exist. They ve got all the problems of regular classes, as well as all the problems of types. There was a fantastic opportunity with parameterised classes to fix some of the really annoying things about regular resources, such as doing conflict checking on parameters and only barfing if there was a conflict but no, if you define the same class twice, even with identical parameters, Puppet will smack you on the hand and take away your biscuit. FFFFFFUUUUUUUUUUUU-

3. You fail at using fail() Modules are supposed to be reusable. That s their whole reason for existence. One of the benefits of Puppet is that you can provide an OS-independent interface for configuring something. However, unless you re some absolute God of cross-platform compatibility, you will only be writing your module to support those OSes or distros that you, personally, care about. That s cool if you don t know anything about OS X, you probably shouldn t be guessing at how to support something on that platform anyway. However, when you do have some sort of platform-specific code, for the love of pete, have a default or else clause that calls fail() with a useful and descriptive error message, to indicate clearly that you haven t included support for whatever environment the module s being used in. Failing to do this can cause some really spectacular explosions, because the rest of your module assumes that certain things have been done in the platform-specific code, and when it hasn t hoo boy.

4. You don t write documentation Yes, it isn t easy to write good documentation. I know that. But without documentation, your module is going to be practically unuseable. So if you don t have docs, your module is basically useless. Well done, you.

5. You have undeclared dependencies This also includes declaring your dependencies in a non-machine-parseable manner; I m not going to grovel through your README for the list of other modules I might need; I have machines to do that kind of thing for me. If I try to use your module, and it craps out on trying to reference some sort of type or class that doesn t appear at all related to your module, I will call into question your ancestry, and die a little more inside.

6. You use common packages without even trying to avoid the pitfalls OK, to be fair this is, ultimately, Puppet s grand fuckup, not yours, but you at least need to pretend to care There is no sane, standardised way in Puppet for multiple modules to install the same package. Let s say that a module I write needs to have a compiler. So I package "gcc": . Then someone else s module also wants a compiler, so it also package "gcc": . FWAKOOM! says Puppet. What the fuck? says the poor sysadmin, who just wanted both a virtualenv and an RVM environment on the one machine. Basically, using packages is going to make your module suck. Does that mean that this makes wide distribution of a large repository of modules written by different people nigh-on impossible? Yes. Fantastic.

24 September 2013

Simon Josefsson: BLURB: Software repository metadata convention

As a maintainer of several software packages I often find myself copying text snippets from the README file into different places (savannah, github, freecode, emails, etc). Recently I had a need to generate a list of software packages that included every project s name, brief summary, license and URL. I could have generated that list manually by copying the text from every project s README and COPYING files. However then I would have to maintain my list manually to keep it in sync with all the projects. This easily leads to stale information, so I thought a better approach would be to put the information I needed into each projects source code version control system. The advantage is that the manual work to extract the information may be automated by a script, since the data is in a usable format. I ll explain here how my solution works. To be able to find the information using only the URL to the repository, I needed a filename convention. The filename I chose was BLURB; for the etymology, see Wikipedia s page about Blurb. The data format in the file is similar to normal email headers. An example illustrate the principles well. Below is the BLURB file for one of my projects.
Author: Yubico
Basename: libyubikey
Homepage: http://opensource.yubico.com/yubico-c/
License: BSD-2-Clause
Name: Yubico C low-level Library
Project: yubico-c
Summary: C library for manipulating Yubico YubiKey One-Time Passwords (OTPs)
The format is simple: UTF-8 text with each line starting with a header followed by a colon ( : ), some whitespace, and some content. If a line starts with whitespace, it is a continuation of the previous line s content (trim leading whitespace). The following table describes the fields that I use. I may update this blog post in the future with new fields or improved explanations (for reference, current date is 2013-09-24).
Header Meaning
Project Short identifier for the project (e.g., gcc , emacs )
Name Official name of the project in English
Summary Brief one-line summary of the project s purpose in English
Author Origin of the project
Homepage URL to the project s website
License License keyword, preferrably using one of the SPDX license identifiers
Basename The tarball basename, if different from the project name
Finally some reflection of the solution. After quick design, I thought that I couldn t be the first one with this problem, and I tried to find other similar efforts. I haven t been able to find any standardization effort that have the following properties: The related efforts that I found were SPDX which at first look seemed to offer what I wanted. However on closer examination it failed to deliver on all the requirements above, and appeared to have somewhat different goals. However I found the SPDX license list useful and refer to it. Another effort is Eric S. Raymond s freecode-submit and shipper but its primary focus is to encode information about each release. The design of the BLURB file is clearly influenced by these tools. Another influence has been Debian s specification for machine-readable copyright information. The Free Software Foundation s list of software projects seemed like another candidate, but it doesn t suggest any way to store the information in the upstream project itself. flattr this!

1 September 2013

Raphaël Hertzog: My Free Software Activities in August 2013

This is my monthly summary of my free software related activities. If you re among the people who made a donation to support my work (47.50 , thanks everybody!), then you can learn how I spent your money. Otherwise it s just an interesting status update on my various projects. Package Tracking System There are only 2-3 weeks left in the summer of code project dedicated to rewrite the package tracking system. We have come a long way during August check it out yourself in pts.debian.net. The rewrite doesn t have all the features of the old PTS yet, but I opted to keep some of the easy and less interesting features for others to re-implement. Instead I asked Marko to work in the coming weeks on new features that will bring more value, like the possibility to have user accounts with the possibility to easily review and tweak all your subscriptions on the web, and like the possibility to subscribe to groups of packages (i.e. those managed by a team). Our main problem right now is that exim has a pretty poor default behavior of forking hundreds of processes if you get hundreds of mails (in a batch) to an address that delivers via a pipe (postfix is saner, it serializes the deliveries on pipes). The new PTS is much more modular and its memory footprint is bigger (about 3 times more for the process that delivers mails, 30Mb instead of 10Mb), and in such a situation we managed to run out of memory for now we worked around the situation with an exim setting that queues mails once the load gets too high but it s a poor workaround IMO. We could obviously implement our own queue and a daemon but I d like to avoid this. So who knows how to tell exim to behave? :-) On the positive side, Marko has gotten some feedback from people who like the new PTS and are using it daily already. And several persons have expressed their interest to work on the new codebase already. On my side, I created a package so that it s easy to deploy for derivatives. In this process, I revamped the way we manage the Django settings (for development and for production). The package is not finished yet, but it s mostly usable already. But I still want to do some cleanup/refactoring in the models before others start deploying it. We must also enable South to make it possible to upgrade easily afterwards. DebConf 13 in Vaumarcus From August 10th to 17th, I was attending DebConf 13. It matched the only week of vacation that my wife had this summer so we went there with the whole family (that is with a 3 years old son, and 6 months old one). Thus I could not immerse myself in Debconf and missed all the nice things that happen outside of the talk rooms. I picked 3-4 interesting talks per day and I spent the rest with my family. On the positive side, I was pleased that my wife could meet (or at least see) some other Debian people. She knows quite a few (of you) by name because I have been telling her Debian stories for years now Debian France Debian France sold quite some merchandise during Debconf but I didn t take care of that. It was supervised by Sylvestre Ledru but fortunately he got the help of multiple persons, both to bring everything there, to sell it, and to bring back the rest. The good news of the month is that the upstream author of galette published a new version with all the features that we ordered him a few months ago. We send now automatic reminders to members who must renew their subscription, we have automatic update of our accounting books (in a ledger file in a git repository) when we people donate or pay their subscription via the paypal form on our website. I was so pleased to finally have this that I took some hours to finalize the packaging of galette, so that it could be uploaded to Debian. It s now waiting in the NEW queue. I also spent multiple hours to write the python script that is executed by galette and that updates the accounting files. Misc Debian stuff Debian Packaging. I did two uploads of logidee-tools to fix bugs #718671 and #718836. I created a package for Dolibarr a PHP-based CRM and ERP software (it doesn t do accounting however), it s sitting in the NEW queue for almost a month already. I forwarded #719000 to the upstream Publican developers. I filed #720393 to request a new upstream version of libphp-mailer. git-multimail. After its deployment on Alioth last month, Niels Thykier reported me a case where it lead to bounces, I filed this as a new upstream ticket and in fact I fixed it myself a few days after. I got the fixed version installed on Alioth. dpkg. I investigated why the the automatic builds of dpkg were no longer happening and asked Michael Prokop if he could install a newer version of gettext in the build chroot. He told me that he would need a backport for that so I asked Santiago Vila if he was willing to provide it and he kindly accepted. A few days after, the package was in backports and I m now again running the latest dpkg out of git thanks to the nice service provided by Michael. Misc discussions. The thread about user planets drifted into a discussion of how to avoid promotional posts on such planets and in that context someone again brought up the Debian Machine Usage Policy as a way to shut down any kind of (self-)promotional content on planet if there s money involved. This always irritates me and this time I opted to ask James Troup about the origin of that clause in the DMUP. So who is willing to work with DSA to fix the DMUP so that people stop abusing it in contexts where it doesn t make sense? I also participated in some discussions concerning dgit. I like the ideas behind the tool, but I m saddened by the behavior of Ian Jackson. I helped him to fill his gap of knowledge about new sources formats but he keeps on bashing about the 3.0 (quilt) source format both in the manual page and in the output of the program. He believes that dgit is no longer an experiment but the truth is that it s still a poorly commented Perl script doing lots of hackish things. Kali Linux Between Debconf and all, I haven t done much for Kali except a couple of fixes. There s a nice story of how I tracked a bug in live-installer on the Kali blog. That fix has been committed to Debian. I also improved live-build to include xfsprogs/jfsutils on the ISO image when you include the debian-installer (so that you don t end up in problems when you pick JFS or XFS as file systems for your installation). Thanks See you next month for a new summary of my activities.

One comment Liked this article? Click here. My blog is Flattr-enabled.

6 August 2013

Craig Small: Careful with apache upgrades

You might (or not if you don t visit) notice all my websites were down. A rushed apt-get dist-upgrade and I found two problems:
  1. PHP5 got removed, which is bad if you run a wordpress site that uses PHP to run
  2. The apache configuration has changed.
Yes, the NEWS entries did warn me, if I read them fully. Yes, I didn t read them enough. Apache now ignores configuration files that don t end in .conf To give a completely non-theoretical example, if you have your virtual hosts in files such as /etc/apache2/sites-enabled/enc.com.au then this will not be recognised and your sites will show the default It works page. Stuff that doesn t fall in the usual places where website stuff should go, which for my setup is a lot of things, will also be denied as the developers have tightened up the rules around what is permitted. Pretty simple to fix with a few <Directory blah> clauses. This isn t a criticism of the Debian apache developers. They do an awesome job of keeping the package workable, flexible but secure which isn t easy. Now it s all back working, I actually agree with the changes they have made. It is just that the latest changes are, well, tricky so be forewarned.

27 June 2013

Russ Allbery: Review: The Odd Clauses

Review: The Odd Clauses, by Jay Wexler
Publisher: Beacon
Copyright: 2011
ISBN: 0-8070-0089-2
Format: Trade paperback
Pages: 193
The subtitle of this book summarizes it nicely: Understanding the Constitution through Ten of Its Most Curious Provisions. (The constitution in question is the United States Constitution.) It's a sort of popularization of US constitutional law, but rather than look at the most well-known and controversial parts, it takes a look at some of the odd side points, obscure clauses, or clauses that one rarely thinks about because they rarely appear in the news. This is partly a tour of curiosity aimed at those who like legal trivia, but it also provides, indirectly, a look at how constitutional law is made and debated, and at some of the burning concerns at the time of the writing that have since faded away. Sometimes why the concern has faded is as interesting as the original problem. This is the sort of book whose table of contents provides a lot of insight into what you're getting. It has ten chapters, each devoted to a particular clause or amendment: incompatibility, weights and measures, recess appointments (one of the rare ones that has made the news), original jurisdiction, natural-born citizen (odd, but sadly now not at all obscure), letters of marque and reprisal, titles of nobility, bills of attainder, and the 3rd and 21st amendments. (For those who haven't memorized all the amendments to the US Constitution, the 3rd is the prohibition on quartering soldiers in private homes and the 21st is Prohibition, which here is examined not for its core intent but for the implications for federalism.) As you can see, this is quite a selection, and while the clauses are often obscure, the issues raised by them are still generally quite active. There are some that were intensely controversial at the time of the original debate that are little more than amusing trivia at this point (the prohibition on officers of the United States taking titles of nobility), or have become so much a part of the structure of the government that people rarely given them a second thought (such as the incompatibility clause, which prohibits members of Congress from holding positions in the executive branch). But most of these clauses establish ground rules for the US government that are still quite current today, even if the legal terminology used at the time is no longer familiar. A good example is the clause prohibiting bills of attainder, something that the writers of the US Constitution saw no need to define. A bill of attainder is an act of a legislature declaring a person guilty of a crime and assigning some punishment without trial. Framed that way, it seems like an obvious extension of the separation of powers system that the US government is constructed around, but unlikely to provoke much modern controversy. However, as Wexler explains quite lucidly, there's a reason why this clause is right next to the prohibition on ex post facto laws. While no one would stand for the US Congress declaring guilt on a matter of criminal law, it has written some startlingly specific laws that seem aimed solely at specific individuals or organizations, which is exactly the sort of behavior that this prohibition, interpreted broadly, was intended to prevent. Similarly, one might think that Congress's power to issue letters of marque and reprisal is little more than ground for an interesting history lesson. But that clause is deeply entangled with the checkered history of US declarations of war and controversies over Congressional oversight, and Wexler uses it as a jumping-off point to take a look at the implications for the modern habit of undeclared wars fought by military contractors. This is a short book, so these treatments are necessarily brief. Some of them are also fairly minor; for example, the discussion of the weights and measures clause is mostly an extended rant on the US refusal to adopt the metric system. That's a sentiment with which I entirely agree, but I'm not sure it has much to do with constitutional law. But Wexler is an entertaining and concise writer; even when the discussion is relatively shallow, I enjoyed reading his analysis. (I will warn, though, that he takes a few potshots from a liberal perspective, so one's enjoyment of his humor may to some extent depend on your personal political allegiances.) I wouldn't rush out to find this book, but I found it mildly diverting and occasionally quite informative. The chapter on original jurisdiction, for instance, explained how the Supreme Court handles certain classes of cases that must be filed directly with it, despite not being a trial court. This was a question that I didn't even know I had, and a side system of judicial procedure that turned out to be surprisingly interesting. I suspect The Odd Clauses will be the most interesting to people who have only a news media understanding of the US Constitution. I went through a personal libertarian period, during which I read the constitution in detail and sought out some discussions of its specifics, so quite a bit of this book was familiar ground to me. Others without that study background may find more novelty. But even with that background, I was entertained, and don't regret buying it. One note, though: I bought this book on the recommendation of Kevin Underhill, author Lowering the Bar, which is a masterpiece of dry humor. If you're looking for legal humor, parts of The Odd Clauses are mildly funny, but Lowering the Bar is the sort of site that will have you staying up late reading just one more page of archives and reading bits aloud to anyone nearby. Rating: 7 out of 10

11 April 2013

Russ Allbery: Hugo nominee haul

I need to write up new Kindle books, of which there are now quite a few due to various sales plus the Hugo nominee slate, but I got another set of paper books and they're sitting in front of me. So here's a list. Saladin Ahmed Throne of the Crescent Moon (sff)
Elizabeth Bear Shattered Pillars (sff)
Ta-Nehisi Coates The Beautiful Struggle (non-fiction)
Guy Gavriel Kay River of Stars (sff)
Jenny Larson Let's Pretend This Never Happened (non-fiction)
Domenica Ruta With or Without You (non-fiction)
Jay Wexler The Odd Clauses (non-fiction) Ahmed's book is the remaining Hugo nominee that I didn't already pick up. I'm delighted to see the diversity on the Hugo and particularly the Nebula slate this year, and I'm curious to see the spin that Ahmed brings to the epic fantasy genre. Bear's book is the sequel to Range of Ghosts, another book I'm very much looking forward to reading to but haven't yet. I'm rather behind on reading Bear's work right now. Let's Pretend This Never Happened, With or Without You, and The Beautiful Struggle are all memoirs, of varying degrees of seriousness. I've gotten hooked on Coates's writing at The Atlantic, and I highly recommend it if you've not yet seen it. Domenica Ruta's memoir is the latest Indiespensible selection. Jay Wexler's book was recommended by Lowering the Bar, a legal humor blog whose entire archives I'm slowly reading. But the highlight of this order is Kay's River of Stars, which is a sequel of sorts to one of my favorite books ever. This is probably the book I'll read during my next vacation.

10 February 2013

Hideki Yamane: license for MIBs

Unfortunately, most of MIBs doesn't meet DFSG since RFC is non-free. However, RFC5378 was released and it says "code components in RFCs are licensed under Simplified BSD license (BSD-3-clause)", See Trust Legal Provisions (TLP) Documents page.

Code components are

ABNF definitions, XML Schemas, XML DTDs, XML RelaxNG definitions, tables of values, MIBs, PIBs, ASN.1, and classical programming source code
So, recent MIBs can be included to Debian package :)

But it isn't enough because those RFCs are almost 30/260, 90% are not DFSG-free. And 190/260 RFCs are owned by ISOC and IETF trustee, I hope thay will release old MIBs code in RFCs under DFSG-free license as same as after RFC5378.

22 January 2013

Russ Allbery: Review: Fantasy & Science Fiction, March/April 2011

Review: Fantasy & Science Fiction, March/April 2011
Editor: Gordon van Gelder
Issue: Volume 120, No. 3 & 4
ISSN: 1095-8258
Pages: 258
Charles de Lint's book review column sticks with the sorts of things he normally reviews: urban and contemporary fantasy and young adult. Predictably, I didn't find that much of interest. But I was happy to see that not all the reviews were positive, and he talked some about how a few books didn't work. I do prefer seeing a mix of positive and negative (or at least critical) reviews. James Sallis's review column focuses entirely on Henry Kuttner and C.L. Moore (by way of reviewing a collection). I'm always happy to see this sort of review. But between that and de Lint's normal subject matter, this issue of F&SF was left without any current science fiction reviews, which was disappointing. Lucius Shepard's movie review column features stunning amounts of whining, even by Shepard's standards. The topic du jour is how indie films aren't indie enough, mixed with large amounts of cane-shaking and decrying of all popular art. I find it entertaining that the F&SF film review column regularly contains exactly the sort of analysis that one expects from literary gatekeepers who are reviewing science fiction and fantasy. Perhaps David Langford should consider adding an "As We See Others" feature to Ansible cataloging the things genre fiction fans say about popular movies. "Scatter My Ashes" by Albert E. Cowdrey: The protagonist of this story is an itinerant author who has been contracted to write a family history (for $100K, which I suspect is a bit of tongue-in-cheek wish fulfillment) and has promptly tumbled into bed with his employer. But he is somewhat serious about the writing as well, and is poking around in family archives and asking relatives about past details. There is a murder (maybe) in the family history, not to mention some supernatural connections. Those familiar with Cowdrey's writing will recognize the mix of historical drama, investigation, and the supernatural. Puzzles are, of course, untangled, not without a bit of physical danger. Experienced fantasy readers will probably guess at some of the explanation long before the protagonist does. Like most Cowdrey, it's reliably entertaining, but I found it a bit thin. (6) "A Pocketful of Faces" by Paul Di Filippo: Here's a bit of science fiction, and another mystery, this time following the basic model of a police procedural. The police in this case are enforcing laws around acceptable use of "faces" in a future world where one can clone someone's appearance from their DNA and then mount it on a programmable android. As you might expect from that setup, the possibilities are lurid, occasionally disgusting, and inclined to give the police nightmares. After some scene-setting, the story kicks in with the discovery of the face of a dead woman who, at least on the surface, no one should have any motive to clone. There were a few elements of the story that were a bit too disgusting for me, but the basic mystery plot was satisfying. I thought the ending was a let-down, however. Di Filippo tries to complicate the story and, I thought, went just a little too far, leaving motives and intent more confusing than illuminating. (6) "The Paper Menagerie" by Ken Liu: Back to fantasy, this time using a small bit of magic to illustrate the emotional conflicts and difficulties of allegiance for second-generation immigrants. Jack is the son of an American farther and a Chinese mother who was a mail-order bride. He's young at the start of the story and struggling with the embarassment and humiliation that he feels at his mother's history and the difficulties he has blending in with other kids, leading to the sort of breathtaking cruelty that comes so easily from teenagers who are too self-focused and haven't yet developed adult empathy. I found this very hard to read. The magic is beautiful, personal, and very badly damaged by the cruelty in ways that can never really be fixed. It's a sharp reminder of the importance of being open-hearted, but it's also a devastating reminder that the lesson is normally learned too late. Not the story to read if you're prone to worrying about how you might have hurt other people. (6) "The Evening and the Morning" by Sheila Finch: This long novella is about a third of the issue and is, for once, straight science fiction, a somewhat rare beast in F&SF these days. It's set in the far future, among humans who are members of the Guild of Xenolinguists and among aliens called the Venatixi, and it's about an expedition back to the long-abandoned planet of Earth. I had a lot of suspension of disbelief problems with the setup here. While Earth has mostly dropped out of memory, there's a startling lack of curiosity about its current condition among the humans. Finch plays some with transportation systems and leaves humanity largely dependent on other races to explain the failure to return to Earth, but I never quite bought it. It was necessary to set up the plot, which is an exploration story with touches of first contact set on an Earth that's become alien to the characters, but it seemed remarkably artificial to me. But, putting that aside, I did get pulled into the story. Its emotional focus is one of decline and senescence, a growing sense of futility, that's halted by exploration, mystery, and analysis. The question of what's happened on Earth is inherently interesting and engaging, and the slow movement of the story provides opportunities to build up to some eerie moments. The problem, continuing a theme for this issue, is the ending. Some of the reader's questions are answered, but most of the answers are old, well-worn paths in science fiction. The emotional arc of the story is decidedly unsatisfying, at least to me. I think I see what Finch was trying to do: there's an attempted undermining of the normal conclusion of this sort of investigation to make a broader point about how to stay engaged in the world. But it lacked triumph and catharsis for me, partly because the revelations that we get are too pedestrian for the build-up they received. It's still an interesting story, but I don't think it entirely worked. (6) "Night Gauntlet" by Walter C. DeBill, Jr., et al.: The full list of authors for this story (Walter C. DeBill, Jr., Richard Gavin, Robert M. Price, W.H. Pugmire, Jeffrey Thomas, and Don Webb) provides one with the first clue that it's gone off the rails. Collaborative storytelling, where each author tries to riff off the work of the previous author while spinning the story in a different direction, is something that I think works much better orally, particularly if you can watch facial expressions while the authors try to stump each other. In written form, it's a recipe for a poorly-told story. That's indeed what we get here. The setup is typical Cthulhu mythos stuff: a strange scientist obsessed with conspiracy theories goes insane, leaving behind an office with a map of linkages between apparently unrelated places. The characters in the story also start going insane for similar reasons, leading up to a typical confrontation with things man was not meant to know, or at least pay attention to. If you like that sort of thing, you may like this story better than I did, but I thought it was shallow and predictable. (3) "Happy Ending 2.0" by James Patrick Kelly: More fantasy, this time of the time travel variety. (I call it fantasy since there's no scientific explanation for the time travel and it plays a pure fantasy role in the story.) That's about as much as I can say without giving away the plot entirely (it's rather short). I can see what Kelly was going for, and I think he was largely successful, but I'm not sure how to react to it. The story felt like it reinforced some rather uncomfortable stereotypes about romantic relationships, and the so-called happy ending struck me as the sort of situation that was going to turn very nasty and very uncomfortable about five or ten pages past where Kelly ended the story. (5) "The Second Kalandar's Tale" by Francis Marion Soty: The main question I have about this story is one that I can't answer without doing more research than I feel like doing right now: how much of this is original to Soty and how much if it is straight from Burton's translation of One Thousand and One Nights. Burton is credited for the story, so I suspect quite a lot of this is from the original. Whether one would be better off just reading the original, or if Soty's retelling adds anything substantial, are good questions that I don't have the background to answer. Taken as a stand-alone story, it's not a bad one. It's a twisting magical adventure involving a djinn, a captive woman, some rather predictable fighting over the woman, and then a subsequent adventure involving physical transformation and a magical battle reminiscent of T.H. White. (Although I have quite likely reversed the order of inspiration if as much of this is straight from Burton as I suspect.) Gender roles, however, are kind of appalling, despite the presence of a stunningly powerful princess, due to the amount of self-sacrifice expected from every woman in the story. Personally, I don't think any of the men in the story are worth anywhere near the amount of loyalty and bravery that the women show. Still, it was reasonably entertaining throughout, in exactly the way that I would expect a One Thousand and One Nights tale to be. Whether there's any point in reading it instead of the original is a question I'll leave to others. (7) "Bodyguard" by Karl Bunker: This is probably the best science fiction of the issue. The first person protagonist is an explorer living with an alien race, partly in order to flee the post-singularity world of uploaded minds and apparent stagnation that Earth has become. It's a personal story that uses his analysis of alien mindsets (and his interaction with his assigned alien bodyguard) to flesh out his own desires, emotional background, and reactions to the world. There are some neat linguistic bits here that I quite enjoyed, although I wish they'd been developed at even more length. (The alien language is more realistic than it might sound; there are some human languages that construct sentences in a vaguely similar way.) It's a sad, elegiac story, but it grew on me. (7) "Botanical Exercises for Curious Girls" by Kali Wallace: One has to count this story as science fiction as well, although for me it had a fantasy tone because the scientific world seems to play by fantasy rules from the perspective of the protagonist. Unpacking that perspective is part of the enjoyment of the story. At the start, she seems to be a disabled girl who is being cared for by a strange succession of nurses who vary by the time of day, but as the story progresses, it becomes clear that something much stranger is going on. There are moments that capture a sense of wonder, reinforced by the persistantly curious and happy narrative voice, but both are undercut by a pervasive sense of danger and dread. This is a light story written about rather dark actions. My biggest complaint with the story is that it doesn't so much end as wander off into the sunset. It set up conflict within a claustrophobic frame, so I can understand the thematic intent of breaking free of that frame, but in the process I felt like the story walked away from all of the questions and structure that it created and ended in a place that felt less alive with potential than formless and oddly pointless. I think I wanted it to stay involved and engaged with the environment it had created. (6) "Ping" by Dixon Wragg: I probably should just skip this, since despite the table of contents billing and the full title introduction, it's not a story. It's a two-line joke. But it's such a bad two-line joke that I had to complain about it. I have no idea why F&SF bothered to reprint it. (1) "The Ifs of Time" by James Stoddard: This certainly fits with the Arabian Nights story in this issue. The timekeeper of a vast and rambling castle (think Gormenghast taken to the extreme) wanders into a story-telling session in a distant part of the castle. The reader gets to listen to four (fairly good) short stories about time, knowledge, and memory, told in four very different genres. All of this does relate to why the timekeeper is there, and the frame story is resolved by the end, but the embedded stories are the best part; each of them is interesting in a different way, and none of them outlast their welcome. This was probably the strongest story of this issue. (7) Rating: 6 out of 10

6 January 2013

Neil Williams: bootstrapping arm64

I'm still working on perl-cross-debian (just uploaded 0.0.2) and there's more to do on that with upstream but part of the reason to work on perl cross-building is to do what I can to help with the ARM64 port.

So, I went back to Wookey's talk at DebConf12 which leads to the current list of cross-build results for arm64 and started through the list.

coreutils is listed as failing but that was an archive error (MD5sum hash mismatch), so that just needs a retry. I don't have access to that buildd, yet, so nothing I can do there.

Next on the list (excluding those just waiting for build-deps) was klibc.

aarch64-linux-gnu-gcc -Wp,-MD,usr/klibc/.__static_init.o.d -nostdinc -iwithprefix
include -I/ PKGBUILDDIR /usr/include/arch/x86_64 -Iusr/include/arch/x86_64
-I/ PKGBUILDDIR /usr/include/bits64
-Iusr/include/bits64 -I/ PKGBUILDDIR /usr/klibc/../include
-Iusr/klibc/../include -I/ PKGBUILDDIR /usr/include -Iusr/include
-I/ PKGBUILDDIR /linux/include -Ilinux/include
-I/ PKGBUILDDIR /linux/arch/x86/include -Ilinux/arch/x86/include
-D__KLIBC__=2 -D__KLIBC_MINOR__=0 -D_BITSIZE=64 -fno-stack-protector -fwrapv -m64 -Os
-fno-asynchronous-unwind-tables -fomit-frame-pointer -falign-functions=1 -falign-jumps=1 -falign-loops=1
-W -Wall -Wno-sign-compare -Wno-unused-parameter -c -o usr/klibc/__static_init.o usr/klibc/__static_init.c
aarch64-linux-gnu-gcc: error: unrecognized command line option '-m64'
make[4]: *** [usr/klibc/__static_init.o] Error 1


Turns out that this is a build failure I understood, at least initially. A little digging and a trivial patch was begun:

--- a/usr/klibc/arch/arm/MCONFIG
+++ b/usr/klibc/arch/arm/MCONFIG
@@ -30,6 +30,13 @@
ifeq ($(CONFIG_AEABI),y)
KLIBCREQFLAGS += -mabi=aapcs-linux -mno-thumb-interwork
else
+# aarch64
+ifeq ($(CONFIG_AARCH64),y)
+KLIBCREQFLAGS +=
+KLIBCOPTFLAGS += -mgeneral-regs-only
+else
KLIBCREQFLAGS += -mabi=apcs-gnu -mno-thumb-interwork
endif
endif
+endif
+

Alongside a trivial change to debian/rules

ifeq ($(DEB_HOST_ARCH),arm64)
DEB_MAKE_ENVVARS := ARCH=arm CONFIG_AARCH64=y CPU_ARCH=armv8-a CPU_TUNE=generic
endif

OK, then things get a bit more awkward,

....
code/debian/cross/klibc/klibc-2.0.1/linux/arch/arm/include -Ilinux/arch/arm/include -D__KLIBC__=2
-D__KLIBC_MINOR__=0 -D_BITSIZE=32 -fno-stack-protector -fwrapv -fno-exceptions -Os
-march=armv8-a -mtune=generic -mgeneral-regs-only -W -Wall -Wno-sign-compare -Wno-unused-parameter
-c -o ../foo.o usr/klibc/arch/arm/crt0.S
usr/klibc/arch/arm/crt0.S: Assembler messages:
usr/klibc/arch/arm/crt0.S:19: Error: operand 1 should be an integer register -- mov r0,sp'
usr/klibc/arch/arm/crt0.S:20: Error: operand 1 should be an integer register -- mov r1,#0'

Hmm. Assembly, well, yes, I've done assembly before, I know what mov should normally do, sp is likely to be the stack pointer .... where's my AARCH64 assembly PDF again... PRD03-GENC-010197 ...

OK, so maybe the r0 and r1 should be x0 and x1, hmm, that at least doesn't raise assembly errors. So a tentative change:

--- a/usr/klibc/arch/arm/crt0.S
+++ b/usr/klibc/arch/arm/crt0.S
@@ -15,9 +15,13 @@
#ifdef __thumb__
.thumb_func
#endif
-
+#ifdef __aarch64__
+_start: mov x0, sp
+ mov x1, #0
+ bl __libc_init
+#else
_start: mov r0, sp
mov r1, #0
bl __libc_init
-
+#endif
.size _start,.-_start

Next stage, however, leaves me quite a bit more lost:

....
klibc-2.0.1/linux/arch/arm/include -Ilinux/arch/arm/include
-D__KLIBC__=2 -D__KLIBC_MINOR__=0 -D_BITSIZE=32 -fno-stack-protector -fwrapv -fno-exceptions -Os
-march=armv8-a -mtune=generic -mgeneral-regs-only -W -Wall -Wno-sign-compare -Wno-unused-parameter
-c -o usr/klibc/arch/arm/setjmp.o usr/klibc/arch/arm/setjmp.S
usr/klibc/arch/arm/setjmp.S: Assembler messages:
usr/klibc/arch/arm/setjmp.S:32: Error: unknown mnemonic stmia' -- stmia r0, r4,r5,r6,r7,r8,r9,r10,fp,sp,lr '
usr/klibc/arch/arm/setjmp.S:33: Error: operand 1 should be an integer register -- mov r0,#0'
usr/klibc/arch/arm/setjmp.S:34: Error: operand 1 should be an integer register -- mov pc,lr'
usr/klibc/arch/arm/setjmp.S:42: Error: unknown mnemonic ldmia' -- ldmia r0, r4,r5,r6,r7,r8,r9,r10,fp,sp,lr '
usr/klibc/arch/arm/setjmp.S:43: Error: unknown mnemonic movs' -- movs r0,r1'
usr/klibc/arch/arm/setjmp.S:44: Error: unknown mnemonic moveq' -- moveq r0,#1'
usr/klibc/arch/arm/setjmp.S:45: Error: operand 1 should be an integer register -- mov pc,lr'
make[5]: *** [usr/klibc/arch/arm/setjmp.o] Error 1


So now I'm out of my depth of AARCH64 assembly (apart from the recurrence of mov r0 vs mov x0 etc.). If the above is useful then maybe someone can work out what is wrong with setjmp.S or whether AARCH64 just means that klibc needs to gain a arch/arm64/ directory and not try to duplicate each entire assembly block within #ifdef clauses.

I don't really know where else to put an incomplete investigation like this, so it's here for anyone to find.

(Oh, and if you're reading those arm64 cross-build logs, then a few hundred occurrences of ldconfig: /usr/lib/aarch64-linux-gnu/*.so is for unknown machine 183. in every build log (success or fail) is apparently entirely normal until more packages get fixed. Really slows down scanning the build log. :-(

I may try busybox or libusb next. libusb looks like a classic "you might have told me to cross-compile but I'm going to use g++ anyway because I know best" cross-building problem, indicative of yet another BDBS. sigh.

Resources:
Getting started with 64-bit ARM development
ARMv8 images for developers
AArch64 for everyone, Marcin Juszkiewicz
Howto/HelloAarch64 - Linaro wiki
AArch64 gcc options

29 December 2012

John Goerzen: Milk, Cookies, and Delight

Sometimes an attic is all it takes to delight children. This afternoon, the boys and I made cookies. Jacob has been talking about setting out milk and cookies for Santa Claus for several days, and of course the fact that we had made cookies reminded him of this as I figured it would. So after the boys got into their pajamas and all ready for bed, we set out milk and cookies for Santa. The boys have always known that Santa is pretend, but love the stories and traditions anyhow. Never mind that Christmas was 3 days ago, and they ve already opened their presents. It s SANTA! It s magic! It doesn t matter! I asked Jacob, Would you like me to pretend to be Santa tonight? A big grin, then Oh yes, dad! Do it! So after I read them their bedtime story, sang them a song (Jacob chose a Latin hymn that s my boy!), and tucked them into bed, I pretended to be Santa. I went back downstairs. I drank the milk and ate the cookies. Then I went to my small future present stash, selected a few small items, and put them under the tree. I gave it a few minutes. Then I crept up to the attic. I snuck along the wood floors quietly, until I was above the boys room. Then I jumped. And I scraped a wood chair along the floor. And then I yelled out HO HO HO! Merry Christmas! I had a brief conversation with Rudolph, then made some sliding noises. I was silent for a few seconds, then made some more noise and said, Wow, Rudolph, Jacob and Oliver left some great milk and cookies! Let s go deliver the rest of our presents! And made some vague sleigh taking off from the roof of a house noises. I crept back down the stairs. I put my ear to the outside of the closed door to the boys bedroom. I heard Jacob excitedly jabbering, He said milk and cookies! He liked them! He really liked them! Ooo butterfly, he was here! (Butterfly is a stuffed, er, butterfly that he sleeps with.) I gave it a minute or two, then I went in. Jacob, did you hear something? Yes! What was it? Well, it was a loud thud! I sat straight up like this. [ he demonstrates ] Then I heard ho ho ho ! And milk and cookies ! And I was excited like this! [ more demonstrations ] What do you think happened? I don t know! Dad, what did you do? I told him. It only increased his delight. Did it sound like Santa s sleight landing? (annoyed) No, dad. It sounded like a crash. (brightening) And then Santa coming down the chimney with presents! Oh, it is so exciting! (We don t have a chimney) It was still magical, even though he knew exactly what happened. For his part, Oliver slept through it all. He will still discover the empty plate, empty cup, and slightly less empty area underneath the tree. And neither boy knows about the thank you note from Santa yet. I anticipate smiles in the morning!

4 December 2012

Daniel Kahn Gillmor: Error messages are your friend (postgres is good)

Here is a bit of simple (yet subtly-flawed) sql, which produces different answers on different database engines:
0 dkg@pip:~$ cat test.sql
drop table if exists foo;
create table foo (x int, y int);
insert into foo VALUES (1,3);
insert into foo VALUES (1,5);
select y from foo group by x;
0 dkg@pip:~$ sqlite3 < test.sql
5
0 dkg@pip:~$ mysql -N dkg < test.sql
3
0 dkg@pip:~$ psql -qtA dkg < test.sql
ERROR:  column "foo.y" must appear in the GROUP BY clause or be used in an aggregate function
LINE 1: select y from foo group by x;
               ^
0 dkg@pip:~$ 
are two of the many reasons postgresql is my database engine of choice.Tags: errors, postgresql, sql

8 November 2012

Jo Shields: Evil, or why Douglas Crockford is harmful to Free Software

Yesterday I received a new serious bug report against Mono in Debian. For those not in the know, serious severity is release critical, and can trigger removal of a package from the distribution in order to make a shipping release (e.g. if Debian is in deep freeze, as is the case right now). Bug 692614 relates to a single source file, ./mcs/class/System.Web.Extensions/System.Web.Script.Serialization/JsonDeserializer.cs, which is part of the System.Web.Extensions assembly. This file was written in 2008 by an upstream Mono developer, and is based on some code from json.org specifically JSON_parser.c and JSON_checker.c json.org code carries a non-standard license. Specifically, it s MIT/X11 with an added clause: The Software shall be used for Good, not Evil. I ve had some discussions with Mono upstream, and they believe that they have resolved the matter to their satisfaction their solution will make its way into Debian soon. But in the meantime, let s talk about the clause. The Free Software Foundation s Freedom 0 reads:
A program is free software if the program s users have the four essential freedoms:
  • The freedom to run the program, for any purpose (freedom 0).
The Debian Free Software Guidelines and Open Source Initiative s Open Source Definition clause 6 reads:
No Discrimination Against Fields of Endeavor The license must not restrict anyone from making use of the program in a specific field of endeavor. For example, it may not restrict the program from being used in a business, or from being used for genetic research.
It s fairly clear that the JSON.org license clause goes against both of these, making any piece of software using that license neither Open Source nor Free Software. It is not distributable by any organization which mandates Freedom for its users not in Debian, not in Fedora, not on Google Code. Anybody who cares about their users will reject a clause like this, because it has awful chilling effects. Think it s funny? It really isn t. Who can use the code without being at risk of a lawyer knocking on their doors? Can the Catholic Church? The ACLU? Republicans? Democrats? Military contractors? Genetic engineers? Big pharma? Petrochemicals firms? Communists? Fascists? Mono developers? WikiLeaks? Without a clear definition of good and evil , people need to seriously consider whether they are safe to use the code because if the developer and users interpretations of the terms differ, there could be hell to pay. Would you want to ship a hardware device let s say you re making a smart TV and have some developer of a library send his lawyers around to tell you sorry, TV rots kids brains, it s clearly evil to get your entire distribution channel shut down by injunction? It s not an oversight, kids. Upstream are well aware of the pain this childishness causes anyone who takes software licenses seriously. They giggle about it at conferences, like tweenagers who think they ve one-upped the adults in the room. This non-Free license is intended to mock people who take licensing seriously. In fact, one could easily categorise efforts to pollute the Free Software ecosystem with fake, non-Free software as evil, which would mean all JSON.org code fails to comply with its own license, due to shipping with its license (Inception, anyone?). Before the jokers in the room claim that this kind of problem is deserved by Mono, let s take a look at all the software in Debian which Douglas Crockford endangers with his childishness. PHP? OwnCloud? jQuery? You think Debian serves its users well by pulling jQuery from the next release in order to serve the ego of a man behaving like an eleven year old? It is also interesting to note that the author of this do no evil clause works at PayPal. Read that twice, go and repair your irony meters, then come back. In conclusion: thank you so VERY much Doug Crockford for making the world of Free Software measurably worse.

12 September 2012

Martin F. Krafft: A black day for democracy

Today was a black day for democracy in Germany. The German constitutional court ruled in favour of the European Stability Mechanism. In combination with last week s announcement by the European Central Bank to purchase government bonds without limits (breaking the No-Bail-Out clause at the core of their mandate more obviously and irreversably than ever before), the German people have lost a good deal of democracy today. Why? you may ask because from now on, fiscal and financial policy will be made in Brussels, by people enjoying full immunity, but who are not elected democratically by the European people, let alone the Germans, and they will freely decide over who has to pay and be liable for whom. I am talking about people like Klaus Regling, who was already involved the very first time the Maastricht Criteria were violated. He is now at the front of the largest and most powerful financial weapon ever conceived. With immunity. And people like Mario Draghi, whom I would possibly call the most corrupt person I know. His announcement to save the Euro at whatever cost accidentally came only a day before his motherland Italy had to go to the market for more money and was able to place a bond at such ridiculously low interest rates that anyone who s kept up to speed with Italy s development had to rightfully ask how that was possible. While in the past, for whatever reason, the European people have let the ECB get by saying that they are not bailing out countries when they buy bonds on the secondary market (wtf!), they have finally dropped that restriction (the law). And as of today, the ESM is ready to go, along with the fiscal pact. Germany is now liable for more than quarter of all of the Eurozone s past and future debts. And no citizen will be able to have any more influence in this, or reverse it. Budget, fiscal policy and currency control are forever gone. Not that parliamentarian democracies were ever direct. Yet, in the past, one could at least vote for those people whose promises one was inclined to believe the most. You can still do that in the future, but those people won t be able to influence fiscal or financial policy anymore. There is no way back. The ESM and its employees enjoy full immunity, and the ESM is forever-binding. There is no exit clause. Thanks to the ECB s law breaking and the ESM, which I consider highly unconstitutional, at least in Germany, Eurozone-countries may refinance their debts at interest rates that are in no way related to their ability to pay back loans. All other countries foremost Germany are henceforth liable for others debts. The fundamental rule of the EU that no country would have to stand up for another country, is gone with the wind. Within an hour, the markets reacted. Germany, which previously had to pay negative interest (a sign of stability) saw interests on its bond shoot up. And Spain, Portugal, Greece and others who couldn t previously refinance their old debts, are now getting fresh money cheaper than ever. Spain s president Rajoy today didn t even bother beating around the bush anymore, he s now going to apply for fresh money but won t bother with any saving schemes or other restructurings. Monti in Italy has suggested the same. Wouldn t you take money if you were offered it for free, without the need to pay it back? This is more than inflation, in my opinion. What is currently happening in Europe is active depreciation of individual wealth. Our heads of state are actively working against the people. The Euro has lost all credibility and everyone knows it. It is only a question of time until it will tremble and fall. Meanwhile, the market celebrates and continues their gambles while they still can, on the backs of our currency and our wealth. Most affected are the people who have savings in Euros, whose life insurances are decreasing in worth and who cannot afford to diversify into other asset classes or currencies. On the other hand, those who let their money do the work are being saved. Whoever previously invested into bonds of struggling states, hoping to reap massive interest gains, is now proven right. Brussels has eliminated the risk factor. What kind of message does this send??? Hands up if you thought that our politicians are even interested in closing the rapidly widening gap between rich and poor. Really? That s naive. The Eurozone is corrupt, and our currency has never been as virtual as today. Nobody can say whether saving the Euro at all cost is the right thing and noone knows whether what s currently happening is just bad. I would have wished that our politicians had taken the crisis as an incentive to fix the system in the interest of the people and with a long-term focus: But on the contrary! Europe s policiticans are making it crystal clear that the foundation upon which it was built, the laws and rules, the promises and guarantees, no longer apply. The people were not asked. The promises once made were broken. Our politicians have ruled over our heads. More debts are being made, and more debts to pay off debts, and so on. It s long gotten out of control, now the process is institutionalised. I feel sorry for our kids. I find it irresponsible what is being done to them (in addition to the way we rape the environment). I also feel deeply with the people in the struggling countries who are being screwed by the crisis and are not at fault. What our politicians are doing is unfortunately not going to help long term. The problems are just postponed, and with every day, the inevitable crash will be more painful. I am sorry. Today is a black day for democracy. We have lost souvereignity. We have lost control over our currency. We have lost our budget rights. And I have lost my faith in the last instance of the German government that I trusted. As of today, I know that the German constitutional court is nothing more than a puppet in the hands of the politicians (who are themselves puppets of Brussels and the banks). The limit they imposed (Germany s liability must not increase beyond 190 billion Euros without the federal parliament s consent) is worthless. Soon the politicians will explain to us why it s inevitable that we must raise this limit. Not that the people could prevent it, but still I had hoped for a fundamental ruling. They should not have touched numbers. The EU had a no-bailout-clause from day one. It was conditional from the start. If one of the fundamental principles of a contract is broken, the contract becomes invalid. Not only did I expect the court to rule against socialised debt, I would have wished them to go a step further. The German national bank gave up control over the currency to the ECB only because the ECB incorporated the principles of the German national bank. Once the ECB overturned those principles, Germany should have reclaimed their souvereignity. But noone else in Europe would have wanted that. Merkel became a puppet herself. I am grateful that our daughter has dual citizenship. NP: Porcupine Tree: Live at Atlanta 2010

13 July 2012

Russell Coker: Some Proprietary Platform Issues

Android vs iPad I m currently in discussions with a client about a potential future project which involves a tablet computer talking to some electronic equipment. The options are an Android tablet and an iPad. One advantage of Android is that it runs on devices of all shapes and sizes, so we can choose a device that fits the need rather than designing everything around the iPad. But the real problem with iPad is Apple. To run an app on an iPad you need to submit it to Apple, hopefully get it accepted into the App Market, then install it. This process causes some delay, a minor fee, and has the potential to derail the project if Apple doesn t accept the app on the first try. With Android there is no need to even deal with Google, the app can be installed directly without the Google Play store. I may end up working with an iPad (which admittedly is really nice hardware), but it seems most likely that the project in question will run on Android only. Windows vs Linux and Apple OS/X One of my clients recently paid a web development company to redevelop his web site. I turned out that the web developers in question only knew how to develop for Windows and my client didn t discover this until too late. Now a site that s currently using a small fraction of the resources on a $80 per month Linode instance will run on a Windows virtual server costing $300 per month (which includes SQL server license). The Windows virtual server will probably be managed (because my client uses only Apple and Linux systems and doesn t employ anyone with Windows skills) which adds an extra $100 per month. If the server isn t managed then they will have to hire someone to apply patches and that won t necessarily be cheaper. So using Windows is going to cost my client an extra $400 per month when compared to the possibility of running a Linux system on the existing virtual server. Even if my client had someone with Windows skills to run the server it would still be an extra $300 per month. If the NBN was available then my client could run a Windows server in their office, but it s not yet available in their area. Even for a company that employs people with more Windows skills than Linux skills there are still economic factors in favor of Linux due to smaller hardware requirements and the lack of license fees for all the core software (OS, database server, web server, etc). Summary These anecdotes aren t unusual, it s the sort of thing that happens all the time. Sometimes the result is good (EG avoiding the iPad), sometimes it isn t (being stuck with a proprietary web service). I think I ll have to suggest to my clients that every contract have a no proprietary software clause. Contracts can be amended if there is a reason, but it seems best to make a preemptive strike against companies that sneak proprietary software in and cause significant unexpected expense and difficulty. Related posts:
  1. The Lenovo U1 Hybrid an example of how Proprietary OSs Suck Lenovo have announced their innovative new U1 Hybrid laptop [1]....
  2. My Prediction for the iPhone I have previously written about how I refused an offer...
  3. Liberty and Mobile Phones I own two mobile phones at the moment, I use...

11 July 2012

Paul Tagliamonte: The "hell-server"

I s pose a better name would be How to preform automated deep-source checks I ve pushed the code to paultag/qlight, and it s just for personal use at the moment. It s built to use paultag/sb around sbuild. Basically, it preforms a crudload of tests such as licensecheck, suspicious-source, pyflakes, bashisms, cppcheck, php in linter mode, and a ton of other checks. You can see a large majority of them in the source file. The batch runner is a singleton , so it s safely called more then once. It s running on a 5-minute cron, and I ve configured a dput target to push it into it s queue & run the enqueue script. Here s some example output from a random package I had lying around:
  ===== build status =====
sbo-run-clang  successful
sbo-run-gcc  successful
  ===== global status =====
Ran plugin extract, exit code 0
Ran plugin watch, exit code 0
Ran plugin licensecheck, exit code 0
Ran plugin validate-source, exit code 0
Ran plugin build, exit code 0
Ran plugin descr, exit code 0
Ran plugin lintian, exit code 0
  ===== descr =====
I: PT: Does this sound right? "gbemol is/are a(n) Graphical frontend for the Music Player Daemon (MPD)"
  ===== piuparts =====
   [last 20 lines]
  ===== licenses =====
BSD (3 clause) 
GPL (v2 or later) 
GPL (v2 or later) GENERATED FILE
GPL (v2 or later) (with incorrect FSF address) 
LGPL (v2 or later) (with incorrect FSF address) 
*No copyright* GPL (v2 or later) 
*No copyright* UNKNOWN
  ===== lintian =====
I: gbemol: spelling-error-in-binary usr/bin/gbemol Wheter Whether
W: gbemol: hardening-no-relro usr/bin/gbemol
I: gbemol: spelling-error-in-manpage usr/share/man/man1/gbemol.1.gz allows to allows one to
  ===== watch =====
-- Scanning for watchfiles in .
-- Found watchfile in ./debian
-- In debian/watch, processing watchfile line:
   http://sf.net/gbemol/gbemol-(.*)\.tar.gz
-- Found the following matching hrefs:
     gbemol-0.3.tar.gz
     gbemol-0.3.1.tar.gz
     gbemol-0.3.2.tar.gz
Newest version on remote site is 0.3.2, local version is 0.3.2
 => Package is up to date
-- Scan finished
  ===== source issues =====
suspicious-source:
bashisms:
script ./ltmain.sh does not appear to have a #! interpreter line;
you may get strange results
possible bashism in ./ltmain.sh line 368 ($RANDOM):
        my_tmpdir="$ my_template -$ RANDOM-0 $$"
error: ./ltmain.sh: Unterminated quoted string found, EOF reached. Wanted: <">
pyflakes:
cppcheck:
[src/gbemol-utils.c:167]: (error) Memory leak: path
[src/libmpdclient.c:1654]: (error) Possible null pointer dereference: output - otherwise it is redundant to check if output is null at line 1663
[src/libmpdclient.c:1655]: (error) Possible null pointer dereference: output - otherwise it is redundant to check if output is null at line 1663
[src/libmpdclient.c:1656]: (error) Possible null pointer dereference: output - otherwise it is redundant to check if output is null at line 1663
[src/libmpdclient.c:1664]: (error) Possible null pointer dereference: output - otherwise it is redundant to check if output is null at line 1663
phplint:
desktop validate:
.//.pc/desktop_file.patch/gbemol.desktop: warning: key "Encoding" in group "Desktop Entry" is deprecated
.//.pc/desktop_file.patch/gbemol.desktop: error: (will be fatal in the future): value "gbemol.png" for key "Icon" in group "Desktop Entry" is an icon name with an extension, but there should be no extension as described in the Icon Theme Specification if the value is not an absolute path
.//.pc/desktop_file.patch/gbemol.desktop: warning: value "Application;AudioVideo;Player;" for key "Categories" in group "Desktop Entry" contains a deprecated value "Application"
mp3check:
mp3val:
fontlint:
ftvalid:
msgfmt:
POFileChecker:
POFileSpell:
pngcheck:
jpeginfo:
ogginfo:
oggz-validate:
xmllint:
lacheck:
jslint:
check for environ calls to home:
check for use of /tmp:
./intltool-extract.in:    $OUTFILE     = "./tmp/$OUTFILE.h"
check for use of system():
./aclocal.m4:    lt_ld_extra= awk '/^include /   system(sprintf("cd /etc; cat %s 2>/dev/null", \[$]2)); skip = 1;     if (!skip) print \[$]0; skip = 0;  ' < /etc/ld.so.conf   $SED -e 's/#.*//;/^[  ]*hwcap[    ]/d;s/[:,  ]/ /g;s/=[^=]*$//;s/=[^= ]* / /g;/^$/d'   tr '\n' ' ' 
./intltool-merge.in:    return 1 if ((-x $file) or (system("$file -l >$devnull") == 0));
./intltool-merge.in:    system ("$iconv -f $encoding -t UTF-8 <$devnull 2>$devnull");
./intltool-update.in:    return 1 if ((-x $file) or (system("$file --version >$devnull") == 0));
./intltool-update.in:          system ($EXTRACT, "--update", "--srcdir=$SRCDIR",
./intltool-update.in:          system ($EXTRACT, "--update", "--type=$gettext_type", 
./intltool-update.in:    system ("$MSGMERGE", "-o", $outfile, $infile, "$MODULE.pot");
./intltool-update.in:    system ("$MSGFMT", "-o", "$devnull", "--verbose", $output_file);
./intltool-update.in:   system ("$MSGFMT", "-o", "$devnull", "--verbose", "$SRCDIR/$lang.po");
./configure:    lt_ld_extra= awk '/^include /   system(sprintf("cd /etc; cat %s 2>/dev/null", \$2)); skip = 1;     if (!skip) print \$0; skip = 0;  ' < /etc/ld.so.conf   $SED -e 's/#.*//;/^[   ]*hwcap[    ]/d;s/[:,  ]/ /g;s/=[^=]*$//;s/=[^= ]* / /g;/^$/d'   tr '\n' ' ' 
./intltool-extract.in:        system ("mkdir -p $dirname");
./intltool-extract.in:        system("mkdir tmp/"); 
./debian/gbemol.1:the \fBDebian\fP system (but may be used by others).  Permission is 
./debian/manpage.sgml:      the &debian; system (but may be used by others).  Permission is
  ===== sbo-run-clang.log warnings =====
update-binfmts: warning: unable to close /proc/sys/fs/binfmt_misc/register: Invalid argument
update-binfmts: warning: unable to enable binary format python2.6
dpkg-source: warning: extracting unsigned source package (gbemol_0.3.2-2.dsc)
dpkg-buildpackage: warning: using a gain-root-command while being root
gbemol-preferences.c:483:13: warning: 'gbemol_preferences_load_spin' defined but not used [-Wunused-function]
gbemol-net.c:60:11: warning: variable 'res' set but not used [-Wunused-but-set-variable]
dpkg-shlibdeps: warning: package could avoid a useless dependency if debian/gbemol/usr/bin/gbemol was not linked against libfontconfig.so.1 (it uses none of the library's symbols).
dpkg-shlibdeps: warning: package could avoid a useless dependency if debian/gbemol/usr/bin/gbemol was not linked against libatk-1.0.so.0 (it uses none of the library's symbols).
dpkg-shlibdeps: warning: package could avoid a useless dependency if debian/gbemol/usr/bin/gbemol was not linked against librt.so.1 (it uses none of the library's symbols).
dpkg-shlibdeps: warning: package could avoid a useless dependency if debian/gbemol/usr/bin/gbemol was not linked against libgio-2.0.so.0 (it uses none of the library's symbols).
dpkg-shlibdeps: warning: package could avoid a useless dependency if debian/gbemol/usr/bin/gbemol was not linked against libcairo.so.2 (it uses none of the library's symbols).
dpkg-shlibdeps: warning: package could avoid a useless dependency if debian/gbemol/usr/bin/gbemol was not linked against libpango-1.0.so.0 (it uses none of the library's symbols).
dpkg-shlibdeps: warning: package could avoid a useless dependency if debian/gbemol/usr/bin/gbemol was not linked against libpangocairo-1.0.so.0 (it uses none of the library's symbols).
dpkg-shlibdeps: warning: package could avoid a useless dependency if debian/gbemol/usr/bin/gbemol was not linked against libfreetype.so.6 (it uses none of the library's symbols).
dpkg-shlibdeps: warning: package could avoid a useless dependency if debian/gbemol/usr/bin/gbemol was not linked against libpangoft2-1.0.so.0 (it uses none of the library's symbols).
dpkg-gencontrol: warning: File::FcntlLock not available; using flock which is not NFS-safe
     + add misc-depends to fix lintian warning.
  ===== sbo-run-gcc.log warnings =====
dpkg-source: warning: extracting unsigned source package (gbemol_0.3.2-2.dsc)
dpkg-buildpackage: warning: using a gain-root-command while being root
gbemol-preferences.c:483:13: warning: 'gbemol_preferences_load_spin' defined but not used [-Wunused-function]
gbemol-net.c:60:11: warning: variable 'res' set but not used [-Wunused-but-set-variable]
dpkg-shlibdeps: warning: package could avoid a useless dependency if debian/gbemol/usr/bin/gbemol was not linked against libfontconfig.so.1 (it uses none of the library's symbols).
dpkg-shlibdeps: warning: package could avoid a useless dependency if debian/gbemol/usr/bin/gbemol was not linked against libatk-1.0.so.0 (it uses none of the library's symbols).
dpkg-shlibdeps: warning: package could avoid a useless dependency if debian/gbemol/usr/bin/gbemol was not linked against librt.so.1 (it uses none of the library's symbols).
dpkg-shlibdeps: warning: package could avoid a useless dependency if debian/gbemol/usr/bin/gbemol was not linked against libgio-2.0.so.0 (it uses none of the library's symbols).
dpkg-shlibdeps: warning: package could avoid a useless dependency if debian/gbemol/usr/bin/gbemol was not linked against libcairo.so.2 (it uses none of the library's symbols).
dpkg-shlibdeps: warning: package could avoid a useless dependency if debian/gbemol/usr/bin/gbemol was not linked against libpango-1.0.so.0 (it uses none of the library's symbols).
dpkg-shlibdeps: warning: package could avoid a useless dependency if debian/gbemol/usr/bin/gbemol was not linked against libpangocairo-1.0.so.0 (it uses none of the library's symbols).
dpkg-shlibdeps: warning: package could avoid a useless dependency if debian/gbemol/usr/bin/gbemol was not linked against libfreetype.so.6 (it uses none of the library's symbols).
dpkg-shlibdeps: warning: package could avoid a useless dependency if debian/gbemol/usr/bin/gbemol was not linked against libpangoft2-1.0.so.0 (it uses none of the library's symbols).
dpkg-gencontrol: warning: File::FcntlLock not available; using flock which is not NFS-safe
     + add misc-depends to fix lintian warning.
Not as interesting as it can get, but still pretty cool. Andrea, I swear I m not picking on you :) More to come later (hopefully) :)

22 May 2012

Mark Brown: ASoC updates in 3.4

Linux version 3.4 has been released. This was a very active release for ASoC in framework terms, in addition to the usual bug fixes and so on there were a large number of framework enhancements though most are fairly small or are laying the groundwork for more user visible features like dynamic PCM.

9 March 2012

Tanguy Ortolo: JSON License considered harmful

Summary The JSON License may seem interesting, but it a bad license, both non-free and ambiguous: do not use it. If you are the author of a piece of software that uses JSMin or its PHP port, consider dropping that non-free part or at least rendering it optional.The JSON license Do you know the JSON License? It is a software license inspired by the MIT/Expat license with one additional morality clause:
The Software shall be used for Good, not Evil.
Non-free For novices that clause can be perceived as a good idea serving a noble goal. But the problem here is that is would make your software non-free. Indeed, there are two common definitions of free software: the FSF definition, and the Debian Free Software Guidelines, also known as the Open Source Definition. Basically, both definitions indicate that to be free, a piece of software must be usable with no restrictions, and forbidding evil uses is such a restriction. Specifically, it violates: Ambiguous In addition to these incompatibilities to being free, this morality clause implies additional problems, the most important one being that it is ambiguous, which is very bad for a license clause: who exactly defines what is Good and what is Evil? If it is the author, then this clause could be used for arbitrarily forbidding any use; it it is the licensee, then this clause it as good as void. Do not use it For all these reasons, the JSON License with that infamous morality clause is not suitable for free software, so if you plan to use that license for your work, please consider these suggestions: 1. do not; 2. do not, really; 3. if still in doubt, refer to point 1. There are several standard free software licenses, the closest one being the MIT/Expat one, that you should consider instead. Now, if you really want to be considered as non-free, and rejected by Debian, Red Hat/Fedora, Google Code and more generally any free software distributor, this is the way to go. On thing worth noting is that this morality clause only annoys free software distributors, that are certainly not doing Evil, but will not prevent any evil person to do Evil things with your software. To encourage people to do Good rather than Evil, it would be better to formulate that clause as a suggestion rather than an order:
The Software should rather be used for Good, not Evil.
The JSMin case This infamous JSON License is known to be used by JSMin and its PHP port. If you are a Web application author, be careful when embedded external content, since it can render you whole piece of software non-free! If you are using JSMin, I would suggest that you drop it, or at least make it optional by using a conditional (if JSMin is here, use it, otherwise do nothing) so that distributors such as Debian can expurgate that non-free part.

8 February 2012

Pietro Abate: Dependency order matters !

Recently I've discovered a subtle consequence of how the order in which dependencies are specified in debian actually matters. While re-factoring the code of dose3, I changed the order in which dependencies are considered by our sat solver (of edos-fame) . I witnessed a twofold performance loss just by randomizing how variables were presented to our sat solver. This highlights, on one hand, how our solver is strongly dependent on the structure of the problem and, on the other hand the standard practice of debian maintainers to assign an implicit priority in the disjunctive dependencies where the first is the most preferred packages (and maybe the most tested, at least dependency-wise). The basic idea of distcheck is to encode the dependencies information contained in a Packages file in CNF format and then to feed them to a sat solver to find out if a package has broken dependencies or if its dependencies are such that no matter what, it would be impossible to install this package on a user machine. Conflicts are encoded as binary clauses. So if package A conflicts with package B, I add a constraint they says "not (A and B)" , that is A and B cannot be considered together. The dependencies encoding associates to each disjunction of the depends field a clause that says "A implies B". If a package foo depends on A,B C,D , I'll add "foo implies A and B" & "foo implies C and D" . This encoding is pretty standard and it is easy to understand. The problem is how the sat solver will search for a solution to the problem "Is is possible to install package foo in an empty repository". The solver we use is very efficient and can easily deal with 100K packages or more. But in general is not very good at dealing with random CNF instances. The reason because edos-debcheck is so efficient lies in the way it exploits the structure of the sat problems. The goal of a sat solver is to find a model (that is a variable assignment list) that is compatible with the given set of constraints. So if my encoding of the debian repository is a set of constraints R, the installability problem boils down to add an additional constraint to R imposing that the variable associated to the package foo must be true, and then ask the solver to find a model to make this possible. This installation, in sat terms, would be just an array of variables that must be true in order to satisfy the given set of constraints. If you look at the logic problem as a truth table, the idea is to find a row in this table. This is the solution of your problem. Brute force of course is not an option and modern sat solvers use a number of strategies and heuristic to guide the search in the most intelligent way possible. Some of them try to learn from previous attempts, some of them, when they are lost try to pick a random variable to proceed. If we consider problems that have a lot of structure, award winning sat solver do not back track very much. By exploiting the structure of the problem, their algorithm allows them to considerably narrow down the search only to those variables that are really important to find a solution. All this long introduction was to talk about the solver that is currently used in edos-debcheck and distcheck (that is a rewrite of the edos-debcheck). So why dependency order matters ? If we consider any package, even if the policy does not specify any order in the dependencies, it's common practice to write disjunctive dependencies specifying the most probable and tested alternative first and all other, more esoteric choices later. Moreover real packages are considered *before* virtual packages. Since every developer seems be doing the same, some kind of structure might be hidden in the order in which dependencies are specified. Part of the efficiency of the the solver used in our tools is actually due to the fact that its search strategy is strongly dependent on the order in which literal are specified in each clause. Saying the package foo depends on A and B is "different" then saying it depends on B and A, even if semantically equivalent. In my tests, I found about a twofold performance loss if the order of literals is either randomized or inverted. This is clearly a specific problem related to our solvers, while other solvers might not be susceptible to such small structural changes. Sat competitions often employs some form of obfuscation strategies of well known problems with well known structures in order to make useless to encode a search strategy that exploits the specific structure of a problem. Since here we're not trying to win a sat competition, but to provide tool to solve a specific problem, we are of course very happy to exploit this structure.

8 January 2012

Niels Thykier: Testing migration and package relations

While looking at the dpkg Breaks-field[0]
$ aptitude show dpkg
Package: dpkg
[...]
Version: 1.16.1.2
[...]
Breaks: apt (< 0.7.7), aptitude (< 0.4.7-1), dpkg-dev (< 1.15.8), libdpkg-perl (< 1.15.8), pinfo (< 0.6.9-3.1), tkinfo (< 2.8-3.1)
it occurred to me that most (all?) of these relations were irrelevant to Britney when she migrated dpkg 1.16.1.2 to testing. Right now, at least the APT relations are only relevant if you are doing something like a distribution upgrade from Lenny/Squeeze to Wheezy. Similarly, the version constraints in a lot of dependency relations (e.g. libc6 (>= 2.11) ) are satisfied in testing and unstable at the same time. Removing the version constrains on dependencies is a rather minor thing as it is basically just a minor constant time optimization on each dependency check. However, removing an entire clause slightly reduces the problem size a bit. Particularly, the Conflicts/Breaks relations tends to be expensive for us. The first task was to identify the relations that can (or cannot) be simplified. In a Britney run we at most 4 versions of the same package per architecture, though usually only 1 or 2[1]. I devised a small set of rules to simplify the relations. These rules are applied to each package in the relation (atomic proposition, if you will).
  1. If the relation is versioned and it involves a virtual package in any suite, then do not change the relation in any way. Rationale: A virtual package cannot satisfy any versioned relation (Policy Manual 7.5)
  2. If the relation is a dependency (i.e. Pre-Depends or Depends) and the package is not available in any suite, then do not change the relation in any way.
  3. If the relation is versioned and the relation is satisfied in all suites (where the relationed-on package is available), then remove the version constrain. Rationale: If all (present) versions satisfy the relation, then version constrain does not change the semantics.
  4. If the relation is a conflict (i.e. Conflicts or Breaks) and relation is unsatisfiable in all suites, then remove the relation. Rationale: If none of the (present) versions satisfy the conflict-relation, then there is no conflict[2].
The rules are rather conservative in some cases and there is room for improvements. However, one has to remember that removing too few relations costs a bit in runtime, removing too many breaks testing and possibly a lot. Obviously, I prefer the former to the latter (especially because I will be a part of the clean up -crew). I tested my implementation of those four rules above against the current master branch. In short, it produces the same result as the master branch in all the tests so far. In the hand-made test-suite, the tests generally do not have any superfluous relations. Thus, it is slightly slower, though usually within 0.1 seconds of the master branch. On the other hand, in the live-data samples I have collected so far it does vastly better. For the longest run (sample from 2011-12-13), it reduces the runtime with ~70 seconds (from ~215 to ~145 seconds). In the other runs, it reduces total runtime with ~35 and ~2 seconds, respectively. In these samples, only the amd64 and i386 packages are considered (and human hints are ignored). For those interested, the code is available in my branch. :) [0] There is perfectly valid reason for doing that. I might get back to that in a later post. [1] One in unstable, testing, testing-proposed-updates and proposed-updates. The latter may seem a bit weird, but [0] [2] This is the rule that prune relations like the one in the dpkg Breaks-field. Edit: 2011-01/09, clarified that we have at most 4 versions per architecture.

Next.

Previous.