Search Results: "ana"

30 August 2012

Axel Beckert: deepgrep: grep nested archives with one command

Several months ago, I wrote about grep everything and listed grep-like tools which can grep through compressed files or specific data formats. The blog posting sparked several magazine articles and talks by Frank Hofmann and me. Frank recently noticed that we though missed one more or less mighty tool so far. We missed it, because it s mostly unknown, undocumented and hidden behind a package name which doesn t suggest a real recursive grep everything : deepgrep deepgrep is part of the Debian package strigi-utils, a package which contains utilities related to the KDE desktop search Strigi. deepgrep especially eases the searching through tar balls, even nested ones, but can also search through zip files and OpenOffice.org/LibreOffice documents (which are actually zip files). deepgrep seems to support at least the following archive and compression formats: A search in an archive which is deeply nested looks like this:
$ deepgrep bar foo.ar
foo.ar/foo.tar/foo.tar.gz/foo.zip/foo.tar.bz2/foo.txt.gz/foo.txt:foobar
foo.ar/foo.tar/foo.tar.gz/foo.zip/foo.tar.bz2/foo.txt.gz/foo.txt:bar
deepgrep though neither seems to support any LZMA based compression (lzma, xz, lzip, 7z), nor does it support lzop, rzip, compress (.Z suffix), cab, cpio, xar, or rar. Further current drawbacks of deepgrep: deepfind If you just need the file names of the files in nested archives, the package also contains the tool deepfind which does nothing else than to list all files and directories in a given set of archives or directories:
$ deepfind foo.ar
foo.ar
foo.ar/foo.tar
foo.ar/foo.tar/foo.tar.gz
foo.ar/foo.tar/foo.tar.gz/foo.zip
foo.ar/foo.tar/foo.tar.gz/foo.zip/foo.tar.bz2
foo.ar/foo.tar/foo.tar.gz/foo.zip/foo.tar.bz2/foo.txt.gz
foo.ar/foo.tar/foo.tar.gz/foo.zip/foo.tar.bz2/foo.txt.gz/foo.txt
As with deepgrep, deepfind does not implement any common options of it s normal sister tool find. Dependencies The package strigi-utils doesn t pull in the complete Strigi framework (i.e. no daemon), just a few libraries (libstreams, libstreamanalyzer, and libclucene). On Wheezy it also pulls in some audio/video decoding libraries which may make some server administrators less happy. Conclusion Both tools are quite limited to some basic use cases, but can be worth a fortune if you have to work with nested archives. Nevertheless the claim in the Debian package description of strigi-utils that they re enhanced versions of their well known counterparts is IMHO disproportionate. Most of the missing features and documentation can be explained by the primary purpose of these tools: Being backend for desktop searches. I guess, there wasn t much need for proper commandline usage yet. Until now. ;-) 42.zip And yes, I was curious enough to let deepfind have a look at 42.zip (the one from SecurityFocus, unzip seems not able to unpack 42.zip from unforgettable.dk due a missing version compatibility) and since it just traverses the archive sequentially, it has no problem with that, needing just about 5 MB of RAM and a lot of time:
[ ]
42.zip/lib f.zip/book f.zip/chapter f.zip/doc f.zip/page e.zip
42.zip/lib f.zip/book f.zip/chapter f.zip/doc f.zip/page e.zip/0.dll
42.zip/lib f.zip/book f.zip/chapter f.zip/doc f.zip/page f.zip
42.zip/lib f.zip/book f.zip/chapter f.zip/doc f.zip/page f.zip/0.dll
deepfind 42.zip  11644.12s user 303.89s system 97% cpu 3:24:02.46 total
I though won t try deepgrep on 42.zip. ;-)

28 August 2012

Theppitak Karoonboonyanan: Pango-HarfBuzz Merge and The Effects on Thai Module

One major change in GNOME 3.6 is Pango s shaper engines replacement with HarfBuzz. Only language engines (for word break analysis, for example) are retained. So, I m checking how this affects Thai/Lao rendering and what to do next. Over all, Behdad has put a good effort to make it right. Most Uniscribe behaviors have been achieved for compatibility. He even cares enough to cover some widespread Thai fonts in which the language tag 'latn' is used instead of 'thai', as seen in Mozilla #719366. Unfortunately, this font set has been declared as standard fonts in official documents. The workaround seems inevitable. Supported Fonts In my experiments with some existing Thai OpenType fonts, the new Pango still renders well without regression. Loma font from fonts-tlwg (glyph positioning with GPOS, rearrangement with GSUB): Loma on new Pango Arundina Sans font from Fonts-SIPA-Arundina (positioning by substitution, only GSUB, no GPOS): Arundina Sans on new Pango But for legacy fonts without OpenType features, it renders badly: Non-OpenType on new Pango In addition, according to Behdad, PUA glyphs in legacy fonts are not supported yet. This means there will be regression on fonts designed for Windows XP or below. But modern fonts designed for Windows 7 should be fine. Changes on Bugs The engine replacement from scratch certainly affects existing bugs. Some become obsolete, some still remain. Here are the summary for Thai/Lao engine, as resolved upstream: Closed bugs: Questionable bugs: Remaining bugs:

27 August 2012

Robert Collins: Why platform specific package systems exist and won t go away

A while back mdz blogged about challenges facing Ubuntu and other Linux distributions. He raises the point that runtime libraries for Python / Ruby etc have a unique set of issues because they tend to have their own packaging systems. Merely a month later he attended Debconf 2010 where a presentation was given on the issues that Java packages have on Dpkg based systems. Since then the conversation seems to have dried up. I ve been reminded of it recently in discussions within Canonical looking at how we deploy web services. Matt suggested some ways forward, including: I think its time we revisit and expand on those points. Nothing much has changed in how Ubuntu or other distributions approach integration with other packaging systems but the world has kept evolving. Internet access is growing ever more ubiquitous, more platforms are building packaging systems clojure, scala, node.js, to name but three, and a substantial and ever growing number of products expect to operate in a hybrid fashion with an evolving web service plus a local client which is kept up to date via package updates. Twitter, Facebook and Google Plus are three such products. Android has demonstrated a large scale app store on top of Linux, with its own custom packaging format. In order to expand them, we need some background context on the use cases that these different packaging systems need to support. Platforms such as antivirus scanners, node.js, Python, Clojure and so forth care a great deal about getting their software out to their users. They care about making it extremely easy to get the latest and greatest versions of their libraries. I say this because the evidence is all around us: every successful development community / product has built a targeted package management system which layers on top of Windows, and Mac OSX, and *nux. The only rational explanation I can come up for this behaviour is that the lower level operating system package management tools don t deliver what they need. E.g. this isn t as shallow as wanting a packaging system written in their own language, which would be easy to write off as parochialism rather than a thoughtful solution to their problems. In general packaging systems provide a language for shipping source or binary form, from one or more repositories, to users machines. They may support replications, and they may support multiple operating systems. They generally end up as graph traversal engines, pulling in dependencies of various sorts you can see the DOAP specification for an attempt at generic modelling of this. One problem that turns up rapidly when dealing with Linux distribution package managers is that the versions upstream packages have, and the versions a package has in e.g. Debian, differ. They differ because at some stage, someone will need to do a new package for the distribution when no upstream change has been made. This might be to apply a local patch, or it might be to correct a defect caused by a broken build server. Whatever the cause, there is a many to one relationship between the package versions that end users see via dpkg / rpm etc, and those that upstream ship. It is a near certainty that once this happens to a library package, that comparing package versions across different distribution packages becomes hard. You cannot reliably infer whether a given package version is sufficient as a dependency or not, when comparing binary packages between Red Hat and Debian. Or Debian and Ubuntu. The result of this is that even when the software (e.g. rpm) is available on multiple distributions (say Ubuntu and RHEL), or even on multiple operating systems (say Ubuntu and Windows), that many packages will /have/ to be targeted specifically to build and execute properly. (Obviously, compilation has to proceed separately for different architectures, its more the depedency metadata that says and build with version X of dependency Y that has to be customised). The result of this is that there is to the best of my knowledge no distribution of binary packages that targets Debian/Ubuntu and RHEL and Suse and Windows and Mac OS X, although there are vibrant communities building distributions of and for each in isolation. Some of the ports systems come close, but they are still focused on delivering to a small number of platforms. There s nothing that gives 99% coverage of users. And that means that to reach all their users, they have to write or adopt a new system. For any platform X, there is a strong pressure to have the platform be maintainable by folk that primarily work with X itself, or with the language that X is written in. Consider Python there is strong pressure to use C, or Python, and nothing else, for any tools that is somewhat parochial, but also just good engineering reducing variables and making the system more likely to be well maintained. The closest system I know of Steam is just now porting to Ubuntu (and perhaps Linux in general), and has reached its massive popularity by focusing entirely on applications for Windows, with Mac OSX a recent addition. Systems like pypi which have multi platform eggs do target the wide range of platforms I listed above, but they do so both narrowly and haphazardly: whether a binary or source package is available for a given platform is up to the maintainer of the package, and the packages themselves are dealing with a very narrow subset of the platforms complexity: Python provides compilation logic, they don t create generic C libraries with stable ABI s for use by other programs, they don t have turing complete scripts for dealing with configuration file management and so forth. Anti virus updaters similarly narrow the problem they deal with, and add constraints on latency- updates of anti virus signatures are time sensitive when a new rapidly spreading threat is detected. A minor point, but it adds to the friction of considering a single packaging tool for all needs is the different use cases of low level package management tools like dpkg or rpm vs the use cases that e.g. pypi has. A primary use case for packages on pypi is for them to be used by people that are not machine administrators. They don t have root, and don t want it. Contrast that with dpkg or rpm where the primary use case (to date) is the installation of system wide libraries and tools. Things like man page installation don t make any sense for non-system-wide package systems, whereas they are a primary feature for e.g. dpkg. In short, the per-platform/language tools are (generally):
  1. Written in languages that are familiar to the consumers of the tools.
  2. Targeted at use on top of existing platforms, by non-privileged users, and where temporary breakage is fine.
  3. Intended to get the software packaged in them onto widely disparate operating systems.
  4. Very narrow they make huge assumptions about how things can fit together, which their specific language/toolchain permits, and don t generalise beyond that.
  5. Don t provide for security updates in any specific form: that is left up to folk that ship individual things within the manager.
operating system package managers:
  1. Are written in languages which are very easy to bootstrap onto an architecture, and to deploy onto bare metal (as part of installation).
  2. Designed for delivering system components, and to avoid be able to upgrade the toolchain itself safely.
  3. Originally built to install onto one operating system, ports to other operating systems are usually fragile and only adopted in niche.
  4. Are hugely broad they install data, scripts, binaries, and need to know about late binding, system caches etc for every binary and runtime format the operating system supports
  5. Make special provision to allow security updates to be installed in a low latency fashion, without requiring anything consuming the package that is updated to change [but usually force-uninstalling anything that is super-tightly coupled to a library version].
Anti virus package managers:
  1. Exist to update daemons that run with system wide escalated privileges, or even file system layer drivers.
  2. Update datasets in realtime.
  3. Without permitting updates that are produced by third parties.
Given that, lets look at the routes Matt suggested Decoupling applications from the core as a strategy makes an assumption that the core and applications are partitionable. If they are not, then applications and the core will share common elements that need to be updated together. Consider, for instance, a Python application. If you run with a system installed Python, and it is built without zlib for some reason, but the Python application requires zlib, you have a problem. A classic example of this problem is facing Ubuntu today, with all the system provided tools moving to Python 3, but vast swathes of Python applications still being unported to Python 3 at all. Currently, the Python packaging system virtualenv/buildout + distribute don t provide a way to install the Python runtime itself, but will happily install their own components for everything up the stack from the runtime. Ubuntu makes extensive use of Python for its own tools, so the system Python has a lot of packages installed which buildout etc cannot ignore this often leads to issues with e.g. buildout, when the bootstrap environment has (say) zope.interfaces, but its then not accessible from the built-out environment that disables the standard sys.path (to achieve more robust separation). If we want to pursue decoupling, whether we build a new package manager or use e.g. virtualenv (or gem or npm or ), we ll need to be aware of this issue and perhaps offer, for an extended time, a dedicated no-frills, no-distro-packages install, to avoid it, and to allow an extended supported period for application authors without committing to a massive, distro sponsored porting effort. While its tempting to say we should install pip/npm/lein/maven and other external package systems, this is actually risky: they often evolve sufficiently fast that Ubuntu will be delivering an old, incompatible version of the tool to users well before Ubuntu goes out of support, or even befor the next release of Ubuntu. Treating data as a service. All the cases I ve seen so far of applications grabbing datasets from the web have depended on web infrastructure for validating the dataset. E.g. SSL certificates, or SSL + content checksums. Basically, small self-rolled distribution systems. I m likely ignorant of details here, and I depend on you, dear reader, to edumacate me. There is potential value in having data repackaged, when our packaging system has behind-firewall support, and the adhoc system that (for instance) a virus scanner system has does not. In this case, I specifically mean the problem of updated a machine which has no internet access, not even via a proxy. The challenge I see it is again the cross platform issue: The vendor will be supporting Ubuntu + Debian + RHEL + Suse, and from their perspective its probably cheaper to roll their own solution than to directly support dpkg + rpm + whatever Apple offer + Windows the skills to roll an adhoc distribution tool are more common than the skills to integrate closely with dpkg or rpm What about creating a set of interfaces for talking to dpkg / rpm / the system packagers on Windows and Mac OSX ? Here I think there is some promise, but it needs as Matt said careful thought. PackageKit isn t sufficient, at least today. There are, I think, two specific cases to cater to:
  1. The anti-virus / fresh data set case.
  2. The egg/gem/npm/ specific case.
For the egg/gem/npm case, we would need to support a pretty large set of common functionality, on Windows/Mac OSX / *nux (because otherwise upstream won t adopt what we create: losing 90% of their users (windows) or 5% (mac) isn t going to be well accepted :) . We d need to support multiple installations (because of mutually incompatible dependencies between applications), and we d need to support multiple language bindings in some fashion some approachable fashion where the upstream will feel capable of fixing and tweaking what we offer. We re going to need to support offline updates, replication, local builds, local repositories, and various signing strategies to match the various tradeoffs made by the upstream tools. For the anti-virus / fresh data case, we d need to support a similar set of operating systems, though I strongly suspect that there would be more tolerance for limited support in that most things in that space either have very platform specific code, or they are just a large-scale form of the egg/gem/npm problem, which also wants easy updates. What next? We should validate this discussion with at least two or three upstreams. Find out whats missing I suspect a lot and whats wrong I hope not much :) . Then we ll be in a position to decide if there is a tractable, widespread solution *possible*. Separately, we should stop fighting with upstreams that have their own packaging systems. They are satisfying different use cases than our core distro packaging systems are designed to solve. We should stop mindlessly repackaging things from e.g. eggs to debs, unless we need that specific thing as part of the transitive runtime or buildtime dependencies for the distribution itself. In particular, if us folk that build system packaging tools adopt and use the upstream application packaging tools, we can learn in a deep way the (real) advantages they have, and become more able to reason about how to unify the various engineering efforts going into them and perhaps even eventually satisfy them using dpkg/rpm on our machines.

24 August 2012

Olivier Berger: Generating RDF description of Debian package sources with ADMS.SW

Edit : I ve now managed to roll out my contribution which is now in production on packages.qa.debian.org. See a later post I ve made on the subject, and beware that the generated RDF has changed a bit also. ADMS.SW proposes specifications for description of software present in software catalogues. I ve tried and apply it to the contents of the Debian Package Tracking System (PTS), using transformation of the information known by the PTS to RDF+XML. The result is not yet in production, but here s an example of what can be done, using the Turtle syntax (more readable) :
<style type="text/css"> .T1 color:#000000; font-size:14pt; font-weight:bold; .T2 color:#000000; font-size:12pt; .T3 color:#000000; font-size:8.5pt; .T4 color:#0000ff; font-size:8.5pt; .T5 color:#a020f0; font-size:8.5pt; .T6 color:#228b22; font-size:8.5pt; .T7 color:#b22222; font-size:8.5pt; .T8 color:#008b8b; font-size:8.5pt; .T9 color:#8b2252; font-size:8.5pt; .dp1 .dp2 </style> @base <http://packages.qa.debian.org/> .
@prefix rdf: <http://www.w3.org/1999/02/22 rdf syntax ns#> .
@prefix admssw: <http://purl.org/adms/sw/> .
@prefix doap: <http://usefulinc.com/ns/doap#> .
@prefix dcterms: <http://purl.org/dc/terms/> .
@prefix rdfs: <http://www.w3.org/2000/01/rdf schema#> .
@prefix foaf: <http://xmlns.com/foaf/0.1/> .
@prefix schema: <http://schema.org/> .
@prefix spdx: <http://www.spdx.org/rdf/terms#> .
@prefix : <http://www.w3.org/1999/xhtml> .
@prefix str: <http://exslt.org/strings> . #
# First the packaging project for apache2 in Debian
#
# <http://packages.qa.debian.org/apache2> resource :
<apache2>
a admssw:SoftwareProject ;
doap:name "apache2" ;
doap:description "Debian apache2 source packaging" ;
doap:homepage "http://packages.debian.org/src:apache2" ;
schema:contributor [
a foaf:OnlineAccount ;
foaf:accountName "Debian Apache Maintainers" ;
foaf:accountServiceHomepage <http://qa.debian.org/developer.php?login=debian apache@lists.debian.org>
], [
a foaf:OnlineAccount ;
foaf:accountName "Stefan Fritsch" ;
foaf:accountServiceHomepage <http://qa.debian.org/developer.php?login=sf@debian.org>
], [
a foaf:OnlineAccount ;
foaf:accountName "Steinar H. Gunderson" ;
foaf:accountServiceHomepage <http://qa.debian.org/developer.php?login=sesse@debian.org>
], [
a foaf:OnlineAccount ;
foaf:accountName "Arno T ll" ;
foaf:accountServiceHomepage <http://qa.debian.org/developer.php?login=arno@debian.org>
] ;
# pointer to the release in the different suites :
doap:release <apache2_2.2.16 6+squeeze7>, <apache2_2.2.22 11>, <apache2_2.4.2 2>. #
# Now the different debian package source releases
#
# <http://packages.qa.debian.org/apache2_2.2.16 6+squeeze7> resource
<apache2_2.2.16 6+squeeze7>
a admssw:SoftwareRelease ;
rdfs:label "apache2 2.2.16 6+squeeze7" ;
admssw:project <apache2> ;
dcterms:description "Debian apache2 source package version 2.2.16 6+squeeze7" ;
doap:revision "2.2.16 6+squeeze7" . # This one is the reference version for the PTS as in unstable, so
# contains more details than the others
<apache2_2.2.22 11>
a admssw:SoftwareRelease ;
rdfs:label "apache2 2.2.22 11" ;
admssw:project <apache2> ;
dcterms:description "Debian apache2 source package version 2.2.22 11" ;
doap:revision "2.2.22 11" ;
# this release contains two components
admssw:includedAsset <apache2/apache2_2.2.22 11_debian>, <apache2/apache2_2.2.22_orig>
;
# this release can be downloaded as one package (with dget)
admssw:package <apache2/apache2_2.2.22 11.dsc> ;
# it also has a related release somewhere else
dcterms:relation <https://launchpad.net/ubuntu/+source/apache2/2.2.22 6ubuntu2> .
<apache2_2.4.2 2>
a admssw:SoftwareRelease ;
rdfs:label "apache2 2.4.2 2" ;
dcterms:description "Debian apache2 source package version 2.4.2 2" ; admssw:project <apache2> ;
doap:revision "2.4.2 2" . # Then the .dsc file for the current unstable version
<apache2/apache2_2.2.22 11.dsc>
a admssw:SoftwarePackage ;
dcterms:description "Debian source package descriptor file for apache2 version 2.2.22 11";
schema:downloadUrl "http://cdn.debian.net/debian/pool/main/a/apache2/apache2_2.2.22 11.dsc";
schema:fileSize "2885" ;
spdx:checksum [
a spdx:Checksum ;
spdx:algorithm <apache2#checksumAlgorithm_md5sum> ;
spdx:checksumValue "d7d03719b9f6432beeecd3aa04f7b22c"
] . #
# Then the upstream project
#
<apache2/apache2_orig>
a admssw:SoftwareProject ;
doap:description "The apache2 upstream project" ;
# either its name or homepage can be matched against other ADMS.SW
# or DOAP descriptors
doap:name "apache2" ;
doap:homepage "http://httpd.apache.org/" . # And a upstream release 2.2.22
<apache2/apache2_2.2.22_orig>
a admssw:SoftwareRelease ;
rdfs:label "Upstream apache2 release 2.2.22" ;
dcterms:description "Upstream source release for apache2 version 2.2.22" ;
doap:revision "2.2.22" ;
admssw:project <apache2/apache2_orig> ;
# and a link to the upstream source tarball s description
admssw:package <apache2/apache2_2.2.22.orig.tar.gz> . # now the (potentially re archived) upstream source archive for that release
<apache2/apache2_2.2.22.orig.tar.gz>
a admssw:SoftwarePackage ;
dcterms:description "Upstream source archive for apache2 version 2.2.22 11 (potentially re archived by Debian)";
schema:downloadUrl "http://cdn.debian.net/debian/pool/main/a/apache2/apache2_2.2.22.orig.tar.gz";
# Those 2 bits should help match packagings of the same tarball if needed
schema:fileSize "7200529" ;
spdx:checksum [
a spdx:Checksum ;
spdx:algorithm <apache2#checksumAlgorithm_md5sum> ;
spdx:checksumValue "d77fa5af23df96a8af68ea8114fa6ce1"
] . #
# Now, document specific details of the second component : Debian
# packaging files archive
#
<apache2/apache2_2.2.22 11_debian>
a admssw:SoftwareRelease ;
rdfs:label "Package sources apache2_2.2.22 11" ;
dcterms:description "Debian packaging sources for apache2 version 2.2.22 11" ;
doap:revision "2.2.22 11" ;
admssw:package <apache2/apache2_2.2.22 11.debian.tar.gz> . <apache2/apache2_2.2.22 11.debian.tar.gz>
a admssw:SoftwarePackage ;
dcterms:description "Debian source package files archive for apache2 version 2.2.22 11" ;
schema:downloadUrl "http://cdn.debian.net/debian/pool/main/a/apache2/apache2_2.2.22 11.debian.tar.gz";
schema:fileSize "195980" ;
spdx:checksum [
a spdx:Checksum ;
spdx:algorithm <apache2#checksumAlgorithm_md5sum> ;
spdx:checksumValue "d3d4146ccad51129636b7dbff284a110"
] . #
# Now, we also weave links with the Ubuntu counterparts
#
<https://launchpad.net/ubuntu/+source/apache2>
a admssw:SoftwareProject ;
doap:description "\"apache2\" source package in Ubuntu" ;
# this one promises some trolls and flames
admssw:forkOf <apache2> ;
doap:homepage "https://launchpad.net/ubuntu/+source/apache2" ;
doap:release <https://launchpad.net/ubuntu/+source/apache2/2.2.22 6ubuntu2> . # and its release known by the PTS
<https://launchpad.net/ubuntu/+source/apache2/2.2.22 6ubuntu2>
a admssw:SoftwareRelease ;
rdfs:label "apache2 2.2.22 6ubuntu2" ;
dcterms:description "\"apache2\" 2.2.22 6ubuntu2 source package in Ubuntu" ;
doap:revision "2.2.22 6ubuntu2" ;
admssw:project <https://launchpad.net/ubuntu/+source/apache2> .

Jonathan Carter: Wheezy Theme Updates


Joy Theme When the Joy theme was picked as the chosen theme for Debian 7.0 (Wheezy), I wanted to do what I could to get as much of it in the archive as possible before the freeze that occurred in June. I ve been working with Paul Tagliamonte and Vagrant Cascadian and I m glad that it resulted in some nice things! The Joy theme is modest and beautiful and I think Debian 7.0 is going to be one of the best looking Debian releases so far. Joy Plymouth Theme Plymouth is the part that provides the boot splash. It also takes care of a few other things, like showing prompts and progress indicators for filesystem checks, password prompts for encrypted devices and more so that the splash doesn t need to exit to show those. It s very scriptable, The installers for the Genesi Efika range of devices even use it for it s front-end. I needed to start with a Plymouth theme where I knew the prompts for all the things that happen during boot was at least more or less implemented, so I started off with one that I know works well, the Edubuntu one. So, fun fact: Debian 7.0 s Plymouth theme is actually based on the Edubuntu one. I had to play with the colours a bit to get it right, Edubuntu has a light background so I had to invert contrast for things like the input box, but I think it came out nicely. Splashy isn t in the archives anymore and it s not widely used anymore, so I cleaned it up from the desktop-base package so that there s less clutter shipped with it. Joy LTSP/LDM Theme This screenshot actually looks a bit uglier than it should because I took it on a VM that doesn t have 24bit colour, but we re looking into what we can do to make it look better on 16bit colour as well. During Debconf in Nigaragua, Vagrant and I worked on getting the LDM theme in shape. I have already done what I thought was most of the work by the time Debconf started, but it turns out there was a lot more to it to do it properly. We managed to get rid of the gtk2-engines-murrine dependency, which in itself is quite tiny, but it s a bit backwards in that it depends on the whole rest of the murrine-themes which brings in too much. We took the time to get the dependencies/recommends/etc right and to make it the default theme in Debian without messing it up for any Debian derivatives. It all came together and now when you install LTSP on Debian, you get the Joy theme by default! I m really glad about that because I didn t get the Spacefun LDM theme done in time for the Squeeze release. Enjoy!

22 August 2012

Eugene V. Lyubimkin: Cupt package manager in Debian: from Squeeze to Wheezy

A status update for what changed since Squeeze release and what is the state of Cupt in Wheezy. No new things for those few who follow small blog posts or the changelog, but an overview for, maybe, softly or newly interested.

Cupt, the high-level package manager for Debian with a console interface and therefore APT's competitor, continued its development since the first stable series. In the second major version it's rewritten in C++(11), got many new features such as numerous improvements in the depedency resolver and the dpkg action scheduler, the support of index diffs, the tutorial, wget-based download method, position action override options, logging, colored action preview prompt, treeish detailed error messages if no solutions were found, to name most important. That said, if you want multiarch, CDROM repository support or, say, some exotic download method, -- Cupt won't suit, at least for now. Project-wide support is also still almost completely missing as many developers accept no more than two package managers.

The "bug-freeness" aim still holds, at the moment of writing there is no unfixed runtime bugs of priority normal+. Here I thank again those people who reported bugs, you all definitely made Cupt better. I am sure there are bugs still which wait their time to show up, but that's hardly avoidable.

All in all, 2.x is a big step forward from 1.x. Not a last step -- development continues.

Gunnar Wolf: An industry commits suicide and blames us

[ once again, I am translating somebody else's material In this case, my good Costa Rican friend Carolina Flores. Please excuse my stylistic mistakes My English is far from native as you well know. But this material is worth sharing, and worth investing some tens of minutes doing a quick translation. If you can read Spanish, go read Caro's original entry ] Have you been to a music record store lately? I did so last Saturday, as a mere excercise. I was not planning on buying anything but I wanted to monitor things and confirm my suspicions. What was I suspicious of? First, that I would only find old records. And so it was: The only recent record I found was ...little broken hearts by Norah Jones. the second, that I would only find music for over 50 year old people. so it was: Were I there to look for a present for my father, I would have walked out with 10 good records. Third, that in the store nothing worth commenting would happen. About that last point, I should point out it was around 10 AM and the store had just opened its doors. Lets concede the benefit of doubt. I don't think many of you will remember, but in Barrio La California (where there is now a beauty parlour, almost in front of AM.PM) there was Auco Disco. In Auco Disco there was a guy specialized in rock (Mauricio Alice) and another one specialized in jazz (I don't remember his name). In that record you could always find rare records, but if they were not there, at least you were sure to find somebody to say: "No, we don't have that, but that's an excellent record, it's the best that [insert group here] have ever recorded because just afterwards they switched their guitar player, they had gone a bit south but with that record they are flying. But no, we don't have it; I can recommend you this record by [insert another group] because it has a guitar solo in track six that is amazing". It would happen more or less like that, which means, one would arrive to Auco Disco at 10 AM and leave around 5 PM with three new records, after having listened to a spectacular music selection. What happened to those stores? Were they killed by The Pirate Bay? That's the simplistic answer from the recording industry! The answer is that those stores never got anything from the industry but an invoice. The industry specially in prescindible markets such as ours was limited to hiring artists, taking care of them recording a sellable product, producing the object called record, and that's it. The more commercial radio stations were paid to program those songs as it cannot be casual that "Mosa, mosa" is the summer hit in all of Latin America, can it? but, record stores? Nothing. Lets carry on with that idea: Radio stations are paid to program said music. This idea should not lead us to believe that recording companies are to blame for bad taste. I won't reveal my sources, but I know the success of the "Locura autom tica" song by La Secta was a real example. Nobody paid for it. That song got to the number one because of its own merits(?) (you don't know the effort it took to find that thing, I cannot recommend it to you). Same thing happens with other stations that don't program reggaet n, that try to save the species, and where they play what we do like. But the thing is, everything we like is not available in any record store in this country. Then, even if we wanted to buy a record or give it as a gift to somebody, it is plain impossible. And don't tell me it's the same to present as a gift a link or a CD full of downloaded MP3 as it is to give a record with cover and booklet, wrapped in gift paper. I might be old-school, but the fetish object record still exists, not only because of its cover, but because of its sound. A 3MB MP3 is akin to drinking coffee dripping from a bag that has been used eight times with the same coffee beans. That format is the worst that has ever happened to music, and if we had any bit dignity we would never purchase digital files in Amazon or iTunes safe for MP3 with an acceptable compression level. That, if we could buy them, because not only that is allowed to us. As the musical industry has no interest in resolving ITS problem (that is not our problem, it is those companies') it has not even been resolved how to charge for a MP3 download that includes import fees (well, if downloading from here a MP3 from a USA-based file server can be considered importing goods into Costa Rica!!!) so we don't have to get dizzy entering into the nineties to Titi Online to discover there is nothing by Muse, Andrew Bird, The Killers, Death Cab for Cutie, Paramore, Bj rk... (believe me, I looked them all up, even Norah jones and La Secta. They also were not there). This all leads me to the question, which I present with all due respect (NOT): What the fuck do they expect us to do??? It is outrageous; above all because in the best case they will sell us a watery coffee download that won't allow us to get all the details a vinyl or less compression would give us. In the worst case, post-MP3 groups will end up recording music with no harmonics or hidden sounds, because, what for? Nobody will hear it. They even admit it: "Some musicians and audio engineers say the MP3 format is changing the way studios mix their recordings. They say the MP3 format "flattens" dynamics differences in tone and volume in a song. As a result, a great deal of new music sounds very much alike, and there is nothing as focusing to create a dynamic listening experience. Why working so hard in creating complex sound if nobody can detect it?" (Rolling Stone, The Death of High fidelity, December 26 2007, taken from here). That's why I am not surprised by Adri n's post regarding the sales of old records. The price has nothing to do with it. The causes are related to the fetish object record and what it means or does not mean for people that have never purchased one. Adri n also asks if somebody here keeps buying records. I answered that I would if the stores sold anything I like. I do it even after the nausea I feel while reading "This phonogram is an intellectual work protected in favor of its producer COPYING IT IN WHOLE OR PARTIALLY IS FORBIDDEN " (like that, uppercased, yelling to whoever is only guilty of having bought a record and defending the producer, not the artist). But I am sure that almost nobody buys records because doing so is no longer a gratifying experience; because if buying a record is clicking to wait 15 days for it to reach the mailbox, we prefer clicking on the download link. But there is another reason for people not to buy records anymore. In one of my talks regarding the dictatorship of the all rights reserved, I asked the 30 twenty-something-year-old students if any of them had ever bought a record. One answered he had, because he is an author and performer (cantautor in Spanish) and understands the effort that producing a record entails. The rest of them had never done so. Is it possible that said young people have never listened to real music? Is it possible that, were it not for concerts, what they consider music is a set of washed-out MP3 that are about to fill up 1TB of their computer? Does people no longer buy records because they cannot differentiate one sound from the other? It is not very clear for me where do I want to get to. The recording industry is despicable. An industry that instead of innovating sets its energy on suing adolescents for downloading songs, trying to pass laws restricting our freedoms in Internet, putting up DRMs making us hostages to our devices* and forcing us to listen just the aroma of music, deserves my whole contempt. If we add ot this that said industry won't allow us to legally download their breadcrumbs because it has not understood that Internet does not need a van crossing borders, besides my contempt they deserve my pity and my heartfelt condolence. But the condolence is also for music, real music, that which is not compressed under the shoe using a terrible format. It is also for independent musicians that have not realized that begging for a bit of space to that industry they just add to themselves the "despicable" tag, given they deserve the fruits of their work to enter their bank account. However, there are good things that have come out of this absurdity. Be it for those that have joined projects such as Aut mata (even if it is in MP3) and for dreams come true such as Musopen (that have achieved that the music that's in theory Public Domain becomes so in practice as well). Good for the Electronic Frontier Foundation and the list of lawyers willing to defend people accused of ilegally downloading music in the USA. Good for the Creative Commons licenses that allow free sharing. All those are growing solutions, although none of them allows me to buy the Panamanian Carlos M ndez's record. Thankfully, a friend of mine who knows I will never give a dime to Apple, bought the files for me in iTunes. I thank him deeply, although I would have prefered to go to Auco Disco and have Mauricio tell me that the 2007 EP I have from Carlos is better than the record he did on 2009. * My devices don't have DRM because I use free software. I also use the ogg file format. Image by verbeeldingskr8

Clint Adams: Catching up

Finally, SFLphone 1.2.0 is almost a reasonable Twinkle substitute.

20 August 2012

Anand Kumria: Finding forks.

github has become the place where a lot of code is made available. It turns out that not everyone has the time or inclination to answer queries / questions or apply pull requests. So, whenever I am investigating using something I've found, I always look to see what the forks are; if there are common issues, etc. In the past, I've done this by hand, for small repositories it isn't too time consuming. Today, though, I sat down and wrote find-forks.py which will do this automatically. Now, I find a project, clone it, run ./find-forks.py and settle down with gitk to examine the history and various other forks. <script src="http://gist.github.com/3405534.js"></script> This works well for me, feel free to use or improve as you like. I hereby place this in the Public Domain.

19 August 2012

John Goerzen: Crazy Enough?

So far this year, I ve read somewhere in the neighborhood of 5000 pages. As I ve started to read more, I ve started to watch TV, movies, and Youtube less, because they are simply boring and shallow in comparison. War and Peace, in particular, deeply touched me. Lately I have been reading the Wheel of Time series, which has its own unique characteristics. Whether an epic (or super-epic, such as Wheel of Time) novel, or the Sherlock Holmes series, or nonfiction works, there is something magical about reading a book. We often see characters, real or fictional, that rise from obscurity to do great things for the world. We are transported in time and place to a time or place we will never be able to experience, perhaps because it is long past, or perhaps because it never was. But in any case, we can be inspired. I am reminded of this quote: The people crazy enough to think they can change the world are the ones that do. If someone told me that a street vendor in Tunisia would, in less than a year, cause the overthrow of 4 dictatorships and reform in a handful more, I would have, yes, thought that was crazy. And while Mohamed Bouazizi isn t a household name in much of the world, he managed exactly that. But not just him. It took crazy unarmed people to occupy Tahrir Square, some to die, for progress to be made in Egypt. This story is written all over history. People have done the impossible, have defied all odds, through sheer belief that they could. Civil rights have been granted due to the leaders we all know, but also due to the millions of marchers we don t. Changing the world doesn t have to mean that the world knows you. It just has to mean that you love the world, as Tolstoy pointed out:
Love hinders death. Love is life. All, everything that I understand, I understand only because I love. Everything is, everything exists, only because I love. Everything is united by it alone. Love is God, and to die means that I, a particle of love, shall return to the general and eternal source.
Whatever your stance on religion, this is a powerful quote. Sometimes particles of love might look crazy. But isn t it then that they are the most alive? Isn t it then that they are the greatest hindrance to death and despair?

17 August 2012

Dirk Eddelbuettel: Follow-up to Counting CRAN Package Depends, Imports and LinkingTo

A few days ago, I blogged about visualizing CRAN dependency ranks which turned out to be a somewhat popular post. David Smith followed-up at the REvo blog suggesting to exclude packages already shipping with R (which is indicated by their 'Recommended' priority). Good idea! So here is an updated version, where we limit the display to the top twenty packages counted by reverse 'Depends:', and excluding those already shipping with R such as MASS, lattice, survival, Matrix, or nlme. CRAN package chart of Reverse Depends relations excluding Recommended packages The mvtnorm package is still out by a wide margin, but we can note that (cough, cough) our Rcpp package for seamless R and C++ is now tied for second with the coda package for MCMC analysis. Also of note is the fact that CRAN keeps growing relentlessly and moved from 3969 packages to 3981 packages in the space of these few days... Lastly, I have been asked about the code and/or data behind this. It is really pretty simply as the main data.frame can be had from CRAN (where I also found the initial few lines to load it). After that, one only needs a little bit of subsetting as shown below. I look forward to seeing other people riff on this data set.
#!/usr/bin/r
##
## Initial db downloand from http://developer.r-project.org/CRAN/Scripts/depends.R and adapted
require("tools")
## this function is essentially the same as R Core's from the URL
## http://developer.r-project.org/CRAN/Scripts/depends.R
getDB <- function()  
    contrib.url(getOption("repos")["CRAN"], "source") # trigger chooseCRANmirror() if required
    description <- sprintf("%s/web/packages/packages.rds", getOption("repos")["CRAN"])
    con <- if(substring(description, 1L, 7L) == "file://")  
        file(description, "rb")
      else  
        url(description, "rb")
     
    on.exit(close(con))
    db <- readRDS(gzcon(con))
    rownames(db) <- db[,"Package"]
    db
 
db <- getDB()
## count packages
getCounts <- function(db, col)  
    foo <- sapply(db[,col],
                  function(s)   if (is.na(s)) NA else length(strsplit(s, ",")[[1]])   )
 
## build a data.frame with the number of entries for reverse depends, reverse imports,
## reverse linkingto and reverse suggests; also keep Recommended status
ddall <- data.frame(pkg=db[,1],
                    RDepends=getCounts(db, "Reverse depends"),
                    RImports=getCounts(db, "Reverse imports"),
                    RLinkingTo=getCounts(db, "Reverse linking to"),
                    RSuggests=getCounts(db, "Reverse suggests"),
                    Recommended=db[,"Priority"]=="recommended"
                    )
## Subset to non-Recommended packages as in David Smith's follow-up post
dd <- subset(ddall, is.na(ddall[,"Recommended"])   ddall[,"Recommended"] != TRUE)
labeltxt <- paste("Analysis as of", format(Sys.Date(), "%d %b %Y"),
                  "covering", nrow(db), "total CRAN packages")
cutOff <- 20
doPNG <- TRUE
if (doPNG) png("/tmp/CRAN_ReverseDepends.png", width=600, heigh=600)
z <- dd[head(order(dd[,2], decreasing=TRUE), cutOff),c(1,2)]
dotchart(z[,2], labels=z[,1], cex=1, pch=19,
         main="CRAN Packages sorted by Reverse Depends:",
         sub=paste("Limited to top", cutOff, "packages, excluding 'Recommended' ones shipped with R"),
         xlab=labeltxt)
if (doPNG) dev.off()
if (doPNG) png("/tmp/CRAN_ReverseImports.png", width=600, heigh=600)
z <- dd[head(order(dd[,3], decreasing=TRUE), cutOff),c(1,3)]
dotchart(z[,2], labels=z[,1], cex=1, pch=19,
         main="CRAN Packages sorted by Reverse Imports:",
         sub=paste("Limited to top", cutOff, "packages, excluding 'Recommended' ones shipped with R"),
         xlab=labeltxt)
if (doPNG) dev.off()
# no cutOff but rather a na.omit
if (doPNG) png("/tmp/CRAN_ReverseLinkingTo.png", width=600, heigh=600)
z <- na.omit(dd[head(order(dd[,4], decreasing=TRUE), 30),c(1,4)])
dotchart(z[,2], labels=z[,1], pch=19,
         main="CRAN Packages sorted by Reverse LinkingTo:",
         xlab=labeltxt)
if (doPNG) dev.off()

16 August 2012

Matthew Garrett: Building personal trust with UEFI Secure Boot

The biggest objection that most people have to UEFI Secure Boot as embodied in the Windows 8 certification requirements is the position of Microsoft as the root of trust. But we can go further than that - putting any third party in a position of trust means that there's the risk that your machine will end up trusting code that you don't want it to trust. Can we avoid that?

It turns out that the answer is yes, although perhaps a little more complicated than ideal. The Windows 8 certification requirements insist that (on x86, at least) the key databases be completely modifiable. That means you can delete all the keys provided by your manufacturer, including Microsoft's. However, as far as the spec is concerned, a system without keys is in what's called "Setup Mode", and in this state it'll boot anything - even if it doesn't have a signature.

So, we need to populate the key database. This isn't terribly difficult. James Bottomley has a suite of tools here, including support for generating keys and building an EFI binary that will enrol them into the key databases. So, now you have a bunch of keys and the public half of them is in your platform firmware. What next?

If you've got a system with plug-in graphics hardware, what happens next is that your system no longer has any graphics. The firmware-level drivers for any plug-in hardware also need to be signed, and won't run otherwise. That means no graphics in the firmware. If you're netbooting off a plug-in network card, or booting off a plug-in storage controller, you're going to have similar problems. This is the most awkward part of the entire process. The drivers are all signed with a Microsoft key, so you can't just enrol that without trusting everything else Microsoft have signed. There is a way around this, though. The typical way to use Secure Boot is to provide a list of trusted keys, but you can also provide trusted hashes. Doing this involves reading the device ROM, generating a SHA256 hash of it and then putting that hash in the key database.

This is, obviously, not very practical to do by hand. We intend to provide support for this in Fedora by providing a tool that gets run while the system is in setup mode, reads the ROMs, hashes them and enrols the hashes. We'll probably integrate that into the general key installation tool to make it a one-step procedure. The only remaining problem is that swapping your hardware or updating the firmware will take it back to a broken state, and sadly there's no good answer for that at the moment.

Once you've got a full set of enrolled keys and the hashes of any option ROMs, the only thing to do is to sign your bootloader and kernel. Peter Jones has written a tool to do that, and it's available here. It uses nss and so has support for using any signing hardware that nss supports, which means you can put your private key on a smartcard and sign things using that.

At that point, assuming your machine implements the spec properly everything that it boots will be code that you've explicitly trusted. But how do you know that your firmware is following the spec and hasn't been backdoored? That's tricky. There's a complete open implementation of UEFI here, but it doesn't include the platform setup code that you'd need to run it on any x86 hardware. In theory it'd be possible to run it on top of Coreboot, but right now that's not implemented. There is a complete port to the Beagleboard hardware, and there's instructions for using it here. The remaining issue is that you need a mechanism for securing access to the system flash, since if you can make arbitrary writes to it an attacker can just modify the key store. On x86 this is managed by the flash controller being switched into a mode where all writes have to be carried out from System Management Mode, and the code that runs there verifies that writes are correctly authenticated. I've no idea how you'd implement that on ARM.

There's still some work to be done in order to permit users to verify the entire stack, but Secure Boot does make it possible for the user to have much greater control over what their system runs. The freedom to make decisions about not only what your computer will run but also what it won't is an important one, and we're doing what we can to make sure that users have that freedom.

comment count unavailable comments

Rapha&#235;l Hertzog: Happy Birthday Debian! And memories of an old-timer

For Debian s birthday, Francesca Ciceri of the Debian Publicity team suggested that developers blog about their first experiences with Debian . I found this a good idea so I m going to share my own early experience. It s quite different from what happens nowadays Before speaking of my early Debian experience, I have to set some context. In my youth, I have always been a Windows user and a fan of Bill Gates. That is until I got Internet at home at that point, I got involved in Usenet and made some friends there. One of those made me discover Perl and it has been somewhat of a revelation for me who had only been programming in Visual Basic, Delphi or ObjectPal. Later the same friend explained me that Perl was working much better on Linux and that Debian Linux installs it by default so I should try this one. I had no idea of what Linux was, but given how I loved Perl, I was eager to try his advice. So I got myself a Tri-Linux CD with Debian/RedHat/Slackware on it and started the installation process (which involved preparing boot floppies). But I did not manage to get the graphical interface working despite lots of fiddling with Xfree86 s configuration file. So I ended up installing RedHat and used it for a few months. But since many of the smart guys in my Usenet community were Debian users, I persisted and finally managed to get it to work! After a few months of usage, I was amazed at everything that was available for free and I wanted to give back. I filed my first bug report in July 1998, I created my first Debian packages in August 1998 and I got accepted as an official Debian developer in September 1998 (after a quick chat over the phone with Martin Schulze or James Troup I never understood the name of my interlocutor on the phone and I was so embarassed to have to use my rusty English over the phone that I never asked). That s right, it took me less than 3 months to become a Debian developer (I was 19 years old back then). I learned a lot during those months by reading and interacting with other Debian developers. Many of those went away from Debian in the mean time but some of them are still involved (Joey Hess, Manoj Srivastava, Ian Jackson, Martin Schulze, Steve McIntyre, Bdale Garbee, Adam Heath, John Goerzen, Marco D Itri, Phil Hands, Lars Wirzenius, Santiago Vila, Matthias Klose, Dan Jacobowitz, Michael Meskes, ). My initial Debian work was centered around Perl: I adopted dpkg-ftp (the FTP method for dselect) because it was written in Perl and had lots of outstanding bug reports. But I also got involved in more generic Quality Assurance work and tried to organize the nascent QA team. It was all really a lot of fun, I could take initiatives and it was clear to me that my work was appreciated. I don t know if you find this story interesting but I had some fun time digging through archives to find out the precise dates if you want to learn more about what I did over the following years, I maintain a webpage for this purpose.

One comment Liked this article? Click here. My blog is Flattr-enabled.

Russ Allbery: Review: Getting Started with Dwarf Fortress

Review: Getting Started with Dwarf Fortress, by Peter Tyson
Publisher: O'Reilly
Copyright: May 2012
ISBN: 1-4493-1494-5
Format: Trade paperback
Pages: 219
If you're unfamiliar with Dwarf Fortress, you're in good company. Most people are, even people who play video games regularly. It was heavily inspired by roguelike games: games such as Hack and Rogue and their more recent variations, which have been around since the 1970s and are still frequently played with simple character graphics on 2D maps. Dwarf Fortress even has a roguelike gameplay mode, but the core of the game (and the only part of the game covered in this book) is the fortress mode, where the player controls a band of dwarfs in an attempt to construct and maintain a fortress. All of this is done (by default, at least) in a simulated-text interface filled with a barrage of strange characters and symbols. A Dwarf Fortress game in progress is almost incomprehensible to the uninitiated. (There are graphical tile sets available, but all the illustrations in this book assume the default display mode.) Dwarf Fortress gains its cult popularity (sufficient to support the primary developer solely through voluntary donations; the game is free) from the incredible depth and richness of the world. Almost everything is modeled in startling detail, down to the individual items of clothing that each dwarf wears, their personalities and history, and a considerable part of the industrial lifecycle of any goods that the fortress needs to produce (such as food and all-important alcohol). That's where this book comes in. The game is so massive and involves so many variables, industries, needs, requirements, and perils that the first-time player is likely to see their first games collapse due to resource problems they have no idea how to address. Getting Started with Dwarf Fortress walks the player through the initial fortress setup, the resource cycles, and the fundamentals of the various parts of the game one has to monitor. The fact that this discussion takes over 200 pages should give you an idea of the depth of the game. I should stop here and warn that, similar to my RPG sourcebook reviews, I read this book somewhat against its intended purpose. I haven't played Dwarf Fortress and don't currently intend to. It's not quite my game type: I don't like open world construction games in which it's possible to lose (and thereby lose all the things that one has built). But the idea of the game, the way it's developed, and the fact that O'Reilly would publish a book about it is so delightfully strange that I wanted to get a feel for the game without investing the time required to learn it. Thankfully, while the book is designed to be a tutorial, it also does an excellent job at conveying the feel of a game. Getting Started with Dwarf Fortress is divided into eleven chapters. After a chapter of introduction, it goes over site selection, some of the options for your initial party, and recommended settings for both. It then provides some advice for the initial rooms and initial fortress layout to get started. Subsequent chapters are more topic-oriented, covering food, merchants, resource management (including Strange Moods), digging, and the various chains of industry. The final chapters cover some of the more intricate details of dwarven life, such as justice, vampires, and hospitals, and then provide an introduction to the military and to dwarven engineering. The engineering chapter, in particular, makes it clear that vast complexity and possibility is left unexamined. One drawback of the printed version of the book (I don't know if this also applies to the ebook) is that all of the screen shots here are black and white. This makes Dwarf Fortress's already-forbidding presentation even more difficult to understand. At times, I had a hard time picking out features that were described in the text. A look at on-line screenshots quickly shows that color is massively helpful in making sense of all the little glyphs. I'm sure color printing would have made the printed version prohibitively expensive, but it is a loss. The text, though, is excellent. Not only is it clear and helpful (although with the caveat that I've not personally tested it), it captures much of the humor and feel of the game. Dwarf Fortress incorporates computer-generated storytelling that narrates the events of the game (and the backstory of the world), which has sparked a tradition of delightfully wry commentary on games, game mechanics, and events in the game (particularly the ones that cause the cascading destruction of the whole fortress). Tyson captures that tone brilliantly, weaving it throughout his descriptions. This is a tutorial that's genuinely fun to read. (Although the Tim Denee comics ending chapters sadly didn't work for me.) It's certainly possible to get started with Dwarf Fortress without this book. There are other tutorials on-line, as well as an impressively huge wiki. But this is a great introduction and is entertainment in its own right. If you've thought about playing the game, or if, like me, you want a good feel for what it's like without investing the time in actually playing it, I recommend it. I should probably note that this book covers Dwarf Fortress up to 0.34.07, and the dead tree version that I have obviously won't be further updated. O'Reilly does note on the back cover that if you purchase the ebook edition, it will receive free updates for the "life of the edition" (however long that is). If you're worried about subsequent changes in the game making the book obsolete (given the ongoing active development, this is something of a worry, although this book mostly covers fundamentals), you may want to grab the electronic version. Rating: 8 out of 10

Antoine Beaupr : Getting rid of deb-multimedia and other non-official packages

It's actually quite simple, and this can be used to also track any package that was installed on your system from a different "origin". The "origin" field is defined in the mirror's Release file and represents where the packages come from, be it Debian, Ubuntu, or deb-multimedia (example: here is Debian Wheezy's official Release file). You can check for non official packages in Aptitude using this syntax in the search field, triggered by the / ("slash") key:
!?origin(debian)
en lire plus

15 August 2012

Gintautas Miliauskas: Why Software Engineering Is Not Politics

A recent post by Steve Yegge has made quite a splash. The main thesis of his post is:

Software engineering has its own political axis, ranging from conservative to liberal.

Software engineering, like any other social enterprise, does have a political aspect, but I do not think this thesis is apt. The essential property of politics is that you can not have it both ways. A political problem is a coordination problem, and political decisions by its nature apply for an entire political body. Software engineering as a whole does not have these properties. There is absolutely no reason why any two engineering projects have to follow the same ideals and guidelines. Politics in engineering only appears within boundaries of a project (or, at a slightly higher level, a company), at which point it does not really warrant the name of politics, but rather character traits or personal preferences. If Johnny goes for the puck more often than Louie, that does not really warrant a label of a "liberal".

The fundamental quality that Steve is looking at is actually risk aversiveness, as he noted himself. He should have stopped there. Making that next step of conceptualization in political language does not bring any new insights and just muddles the matter.

It seems that the political labels of "liberal" and "conservative" were picked not for their meaning, but rather for their connotations, as political views are perceived to be 1) very stable, part of a person's self-identity and (therefore) hard to change, 2) inherently polarizing, and 3) difficult or impossible to evaluate objectively. Again, these do not apply to software engineering. For example, probably like many engineers, I am clearly a liberal on personal and small projects, and a conservative on larger and more fundamental ones; so much for the idea of a core identity. As for polarization, it may be prevalent in internet echo chambers, but I have not seen much of that in my professional life, especially when working with experienced engineers. When they do have objections, those objections are grounded in engineering. This leads to the third point, claiming that the approaches can not be compared objectively. This one is the most pernicious of all, because it provides an escape hatch from arguments with substance on what is an engineering decision.

Here is my, admittedly less catchy and impressive, counter thesis: Conservativeness of a software engineering project is an objective decision to be based on circumstances. TL;DR: It depends. (Very original, isn't it?) It is a risk management decision, and those are very tricky, but not subjective.

One thing to note is that the architecture of the system should correspond to the implementation style. Schemaless, heavy on magic, untyped approaches are suited for modular systems where each individual module is of limited complexity, and where the modules are isolated from each other and can recover from failures. The more complex the individual modules (and sometimes the complexity is inherent to the problem), the more benefits from "conservative" techniques. (By the way, this lines up nicely with Steve being liberal and a proponent of service-oriented architectures). If you have the freedom to pick between these two approaches, the decision making does get a bit more ideological. However, neither the "liberal" and "conservative" labels, nor risk averseness by itself is enough for a decision here. Say, if you are developing a service, what is more risky: having a "traditional", monolithic service that is inevitably sensitive to failures and is likely to scale poorly, or a cluster of components that are generally more resistant, but have more complicated aggregate behavior?

Despite the criticisms, I have to give fat credit to Steve. His post was thought-provoking, and it brought attention to the subjective side of engineering, one which is often overlooked. That side should not be left to grow - on the contrary - but the only way to reduce its domain it is to be aware of it. I know I will think twice before making another gut decision where risk averseness plays an important factor.

14 August 2012

Philipp Kern: Debian s390x installation now working

With a current daily build of debian-installer you should now be able to install a VM with Debian s390x. I'm told that even installation within hercules works again. d-i's beta 1 for wheezy has a broken busybox and is hence unusable. But there were more issues:
The resulting installation even boots. But it shares some problems with squeeze's s390 port:
Tip of the day: If you see funny characters in x3270, then you maybe selected the wrong EBCDIC code page. If you use Linux you want to select CP1047 instead of the default CP037 in Options Character Set ( Euro). That's the target of the in-kernel ASCII to EBCDIC converter.

Asheesh Laroia: Zooming in

She zoomed in on the git commits to check that the new contributors were thanked properly. She was not looking for bad programmers or bad community managers. She was looking for the kinds of misses that even excellent programmers and community managers can make under pressure.
A mis-quote of "Can Hospital Chains Improve the Medical Industry?".

13 August 2012

Patrick Matth i: Logging packets with iptables and ULOG

Imagine you have got the following iptables rule set:
*filter
:INPUT ACCEPT [2:130]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [119:14185]
-A INPUT -s 127.0.0.0/8 -j ACCEPT
-A INPUT -p tcp -m tcp dport 22 -j ACCEPT
-A INPUT -p tcp -m tcp dport 80 -j ACCEPT
-A INPUT -p tcp -m tcp tcp-flags FIN,SYN,RST,ACK SYN -j DROP
COMMIT
This would allow all traffic from 127.0.0.0/8, on port 22 and 80. Other (TCP/IP) SYN packages (so on all the other connections) would be dropped.
Now you see, that your counter for the SYN DROP rule is increasing and you want to know what is rejected, but how? The simple answer is ULOG the netfilter userspace logging daemon.
In Debian you have got various implementations/variants of it, the local logging one (which I will use here, just called ulogd) and the -postgres, -mysql and -sqlite3 one (that are not the exact package names), with that you also can log everything to a (remote) database.
An special variant is the -pcap one, it will write the logs in the .pcap format, so you can analyze the full traffic. So for our example it is enough to install the package:
apt-get install ulogd
And then add another rule BEFORE our SYN DROP:
-A INPUT -p tcp -m tcp tcp-flags FIN,SYN,RST,ACK SYN -j ULOG
-A INPUT -p tcp -m tcp tcp-flags FIN,SYN,RST,ACK SYN -j DROP
Now you will find in /var/log/ulog/syslogemu.log a log of all connections, which would be dropped, the log looks like this:
Aug 13 14:42:07 srv1 IN=eth0 OUT= MAC=00:0c:29:8c:2b:6c:00:d0:02:eb:e8:0a:08:00 SRC=75.125.70.194 DST=XXX.XXX.XXX.XXX LEN=40 TOS=00 PREC=0 00 TTL=54 ID=9566 PROTO=TCP SPT=57144 DPT=445 SEQ=2770468863 ACK=0 WINDOW=512 SYN URGP=0
Aug 13 14:45:29 srv1 IN=eth0 OUT= MAC=00:0c:29:8c:2b:6c:00:d0:02:eb:e8:0a:08:00 SRC=75.125.70.194 DST=XXX.XXX.XXX.XXX LEN=40 TOS=00 PREC=0 00 TTL=55 ID=13702 PROTO=TCP SPT=58528 DPT=445 SEQ=1217789951 ACK=0 WINDOW=512 SYN URGP=0
So you have got now the information about the full date, mac address (mostly it will be the one of your gateway), source and destination IP, source and destination port, length, protocol, etc. You also could use it to log outgoing connections to port 80 and the IRC ports:
-A OUTPUT -p tcp -m tcp dport 80 -j ULOG -A OUTPUT -p tcp -m tcp dport 6666:6669 -j ULOG
Whatever you want.

Rapha&#235;l Hertzog: Looking back at 16 years of dpkg history with some figures

With Debian s 19th anniversary approaching, I thought it would be nice to look back at dpkg s history. After all, it s one of the key components of any Debian system. The figures in this article are all based on dpkg s git repository (as of today, commit 9a06920). While the git repository doesn t have all the history, we tried to integrate as much as possible when we created it in 2007. We have data going back to April 1996 In this period between April 1996 and August 2012: Currently the dpkg source tree contains 28303 lines of C, 14956 lines of Perl and 6984 lines of shell (figures generated by David A. Wheeler s SLOCCount ) and is translated in 40 languages (but very few languages managed to translate everything, with all the manual pages there are 3997 strings to translate). The top 5 contributors of all times (in number of commits) is the following (result of git log --pretty='%aN' sort uniq -c sort -k1 -n -r head -n 5):
  1. Guillem Jover with 2663 commits
  2. Rapha l Hertzog with 993 commits
  3. Wichert Akkerman with 682 commits
  4. Christian Perrier with 368 commits
  5. Adam Heath with 342 commits
I would like to point out that those statistics are not entirely representative as people like Ian Jackson (the original author of dpkg s C reimplementation) or Scott James Remnant were important contributors in parts of the history that were recreated by importing tarballs. Each tarball counts for a single commit but usually bundles much more than one change. Also each contributor has its own habits in terms of crafting a work in multiple commits. Last but not least, I have generated this 3 minutes gource visualization of dpkg git s history (I used Planet s head pictures for dpkg maintainers where I could find it). <iframe allowfullscreen="allowfullscreen" frameborder="0" height="281" src="http://www.youtube.com/embed/1x9-Etj1Ew4?fs=1&amp;feature=oembed" width="500"></iframe> Watching this video made me realize that I have been contributing to dpkg for 5 years already. I m looking forward to the next 5 years :-) And what about you? You could be the 147th contributor see this wiki page to learn more about the team and to start contributing.

No comment Liked this article? Click here. My blog is Flattr-enabled.

Next.

Previous.