I just released PyGObject 3.3.91, for GNOME 3.5.91. The big new feature in this release (thanks to the release team for granting an exception) is Simon Feltman s new Signal helper class, which makes defining custom signals a whole lot simpler and more obvious. In the past, you had to do

 class C(GObject.GObject):
    __gsignals__ =  
        'my_signal': (GObject.SIGNAL_RUN_FIRST, GObject.TYPE_NONE,
                      (GObject.TYPE_INT,))
     
    def do_my_signal(self, arg):
        print("my_signal called with %i" % arg)

whereas now this looks like

class C(GObject.GObject):
    @GObject.Signal(arg_types=(int,))
    def my_signal(self, arg):
        print("my_signal called with %i" % arg)

or even more elegantly when using Python 3 and its new type annotations:

class C(GObject.GObject):
    @GObject.Signal
    def my_signal(self, arg:int):
        print("my_signal called with %i" % arg)

Check out the updated example and docstring for other ways how to use it. Overrides can now be in a directory different from the one that pygobject installs itself into. These overrides need to put this into their __init__.py at the top:

from pkgutil import extend_path
__path__ = extend_path(__path__, __name__)

and put themselves somewhere into the default PYTHONPATH. This should make it a lot easier for library packages to ship their own overrides for Python. This new version also comes with a couple of new overrides and bug fixes. See the detailled list of changes below. Thanks to all contributors!

• Fix exception test case for Python 2 (Martin Pitt)
• Bump g-i dependency to >= 1.3.9 (Martin Pitt)
• Show proper exception when trying to allocate a disguised struct (Martin Pitt) (#639972)
• Support marshalling GParamSpec signal arguments (Mark Nauwelaerts) (#683099)
• Add test for a signal that returns a GParamSpec (Martin Pitt) (#683265)
• [API add] Add Signal class for adding and connecting custom signals. (Simon Feltman) (#434924)
• Fix pygtkcompat s Gtk.TreeView.insert_column_with_attributes() (Martin Pitt)
• Add override for Gtk.TreeView.insert_column_with_attributes() (Marta Maria Casetti) (#679415)
• .gitignore: Add missing built files (Martin Pitt)
• Ship tests/gi in tarball (Martin Pitt)
• Split test_overrides.py (Martin Pitt) (#683188)
• _pygi_argument_to_object(): Clean up array unmarshalling (Martin Pitt)
• Fix memory leak in _pygi_argument_to_object() (Alban Browaeys) (#682979)
• Fix setting pointer fields/arguments to NULL using None. (Simon Feltman) (#683150)
• Fix for python 2.6, officially drop support for < 2.6 (Martin Pitt) (#682422)
• Allow overrides in other directories than gi itself (Thibault Saunier) (#680913)
• Clean up sys.path handling in tests (Simon Feltman) (#680913)
• Fix dynamic creation of enum and flag gi types for Python 3.3 (Simon Feltman) (#682323)
• [API add] Override g_menu_item_set_attribute (Paolo Borelli) (#682436)

During lunch the other day, I discussed the shortcomings of the tar file format with friend and co-worker Daniel. The tar file format has a lot of legacy by now, and it's not quite up to date with the latest developments in file systems, such as extended attributes. This makes tar badly suited for things such as backups and other situations where precise reproduction of the input data matters. There are several variants of the tar file format, and various more or less standard extensions to it. GNU tar, for example, added support for pathnames longer than 100 bytes many years ago, and it is now commonly supported. Other problems in the tar file format:

• It has no native support for compression. The Unix Way is to use an external compressor, which is nice, but it makes it necessary to decompress the entire file to get a list of its contents. For large archives, this is very time consuming.
• Even when uncompressed, the file format works badly for some kinds of operations, such as deleting files from the archive, or updating them with new versions.
• The file format is entirely linear. When creating a tar file, it would sometimes be possible to write data from multiple sources at the same time, perhaps compressing them separately, maybe with file type specific compressors. With a linear format, this is not possible without spooling some files into temporary files. An interleaved format, similar to multimedia files, which mix audio and video data into a single stream, would make it possible to be more efficient at writing.
• The supported meta data for files is limited, and it's hard to extend the support without breaking the file format.

This led us to discuss the possibility of a new file format. We had a bit of fun exploring the solution space for a while. However, almost all use of tar these days is for distributing sets of files, where the filename and basic set of file permissions is enough. In other words, for things such as source code, tar is just fine. The archives are small enough, and the other limitations are rarely a problem, but the pain of switching to a new format would be great. Thus, with some reluctance, we concluded that a new format would be a waste of time. But I thought I'd write this up anyway, in case one of my readers wants to start working on this.

Next weekend I will hold an advanced Haskell talk at the Meta Rhein-Main ChaosDays in Darmstadt. It will mostly be a guided tour through the main memory of a running Haskell program, showing how things are represented and how they change over time. For this I cannot avoid to touch the issue of garbage collection, so I created a short animation: (If you do not see it above, then you can try on YouTube.) It is created with Synfig, feel free to use the sources. It seems that Synfig, although it has some features that allow you to link position in various objects, or calculate with them, is not really meant for programmers. Take the arrows: They are all quite similar, and defined by starting point, end point, and how much the initial segment sticks out to the right. I could even create them so that when I move the first or last point, the rest adjusts automatically. So one would expect that I can combine these shapes in one re-usable object that has these three values as parameters, but unfortunately, that is not the case. I guess next time I want to animate something as structured as this, I use either tikz (and generate one page per frame, convert them to an image and then to a video) or a Haskell library like gloss or diagrams. Update: I made some minor improvements to the video. Youtube does not allow me to change the video, so you ll see the new version only here.

Flattr this post

It is a frequently asked Perl question how to write a regular expression that matches anything but something . The usual answer is that it can t be done or that you need to use the !~ operator instead of the =~ operator. But when you are using API s that passes regular expressions around as objects this might not be entirely helpful answers. By digging a bit around in the scarier parts of the perlre manual page I worked out this quite Perl specific solution: qr/(?:(?:something(*COMMIT)(*FAIL)) .)*/. I am not sure that it works in all possible corner cases and I certainly don t want to write the much needed comment that makes the code using the regexp maintainable. In the end it might be possible, but it is just one of those Now you have two problems scenarios.

In late July, the Awesome Foundations invited me to participate in an interesting conversation about open brands at their conference. Awesome is a young collection of organizations struggling with the idea of if, and how, they want to try to control who gets call themselves Awesome. I was asked to talk about how the free software community approaches the issue. Guidance from free software is surprisingly unclear. I have watched and participated in struggles over issues of branding in every successful free software project I've worked in. Many years ago, Greg Pomerantz and I wrote a draft trademark policy for the Debian distribution over a couple beers. Over the last year, I've been working with Debian Project Leader Stefano Zacchiroli and lawyers at the Software Freedom Law Center to help draft a trademark policy for the Debian project. Through that process, I've come up with three principles which I think lead to more clear discussion about whether a free culture or free software should register a trademark and, if they do, how they should think about licensing it. I've listed those principles below in order of importance. 1. We want people to use our brands. Conversation about trademarks seem to turn into an exercise in imagining all the horrible ways in which a brand might be misused. This is silly and wrong. It is worth being extremely clear on this point: Our problem is not that people will misuse our brands. Our problem is that not enough people will use them at all. The most important goal of a trademark policy should be to make legitimate use possible and easy. We want people to make t-shirts with our logos. We want people to write books about our products. We want people to create user groups and hold conferences. We want people to use, talk about, and promote our projects both commercially and non-commercially. Trademarks will limit the diffusion of our brand and, in that way, will hurt our projects. Sometimes, after carefully considering these drawbacks, we think the trade-off is worth making. And sometimes it is. However, projects are generally overly risk averse and, as a result, almost always err on the side of too much control. I am confident that free software and free culture projects' desire to control their brands has done more damage than all brand misuse put together. 2. We want our projects to be able to evolve. The creation of a trademark puts legal power to control a brand in the hands of an individual, firm, or a non-profit. Although it might not seem like such a big deal, this power is, fundamentally, the ability to determine what a project is and is not. By doing this, it creates a single point of failure and a new position of authority and, in that process, limits projects' ability to shift and grow organically over time. I've heard that in US politics, there is no trademark for the terms Republican or Democrat and that you do not need permission to create an organization that claims to be part of either party. And that does not mean that everybody is confused. Through social and organizational structures, it is clear who is in, who is out, and who is on the fringes. More importantly, this structure allows for new branches and groups outside of the orthodoxy to grow and develop on the margins. Both parties have been around since the nineteenth century, have swapped places on the political spectrum on a large number of issues, and have played host to major internal ideological disagreements. Almost any organization should aspire to such longevity, internal debate, and flexibility. 3. We should not confuse our communities. Although they are often abused, trademarks are fundamentally pro-consumer. The point of legally protected brands is to help consumers from being confused as the source of a product or service. Users might love software from the Debian project, or might hate it, but it's nice for them to be able to know that they're getting "Debian Quality" when they download a distribution. Of course, legally protected trademarks aren't the only way to ensure this. Domains names, internal policies, and laws against fraud and misrepresentation all serve this purpose as well. The Open Source Initiative applied for a trademark on the term open source and had their application rejected. The lack of a registered trademark has not kept folks from policing use of the term. Folks try to call their stuff "open source" when it is not and are kept in line by a community of folks who know better. And since lawyers are rarely involved, it is hardly clear that a registered trademark would help in the vast majority of these these situations. It is also the case that most free software/culture organizations lack the money, lawyers, or time, to enforce trademarks in any case. Keeping your communities of users and developers clear on what is, and what isn't, your product and your project is deeply important. But how we choose to do this is something we should never take for granted.

Some months ago I have run into a pretty interesting bug while working on a Ubuntu-based remote desktop system. The whole OS was installed on a server somewhere and users could access their desktop remotely. Some call this stuff Desktop-as-a-Service. The operating system we chose was Ubuntu Oneiric (11.10) and the remote access part was implemented with x2go, which uses nxagent to provide NX transport of X sessions. Users could access their Ubuntu machines remotely, with sound, video, youtube, and all you would expect from a desktop machine. The whole thing was working quite well. Now, as I said that was in May. Ubuntu 12.04 LTS was available, and the choice of upgrading to it sounded pretty much obvious. So we upgraded our test system to Precise and everything seemed to work smoothly. Till we tried to open a PDF document, actually. It took evince about 50 seconds to display a relatively small document. Same story with images opened with the default image viewer, eog. The fix delivered to our users was simple: we have set shotwell-viewer as the default image viewer, and epdfview as the default PDF viewer. Everybody happy. In the meantime, obviously, I was interested in this issue so I ve tried to run run evince from a terminal, getting the following output:

(evince:15833): GRIP-WARNING **: Failed to initialize gesture manager.

Funny. On another test system running Debian Sid (unstable) everything was working smoothly. The diff between Ubuntu s version of evince and Debian s is a 6MB monster. Among other changes, I noticed that Ubuntu s version build-depends on libgrip-dev, which depends on libutouch-geis. Multitouch stuff. Why should multitouch support break my remote session? So on May the 10th I filed a bug on launchpad. How this issue got handled is in my opinion one of the many fine examples of the inherent superiority of free software, coupled with a we won t hide problems mindset. For an example of how bad is the proprietary approach, just check a random bug in Adobe s bug tracking system. But let s go back to the evince bug. Other users reported that their VNC sessions were also affected by the same problem. After a few days it was clear that the culprit was utouch-geis, and a patch appeared. Unfortunately it did not actually address the issue. Somebody else reported that RDP sessions were broken too. At the beginning of June Precise was still affected. Finally, on August the 6th a working patch was submitted by Bradley M. Froehle and included by Chase Douglas (thank you guys). End of August, fixed version of geis accepted into precise-updates, case closed. Now for some considerations. The problem was clearly of a certain importance. A Long Term Support, stable version of Ubuntu shipped with broken PDF and image viewing functionalities. It got fixed properly, even though 3 good months are quite a long time for such a bug to get solved. However, the issue only affected a pretty limited number of users, also certainly not Ubuntu s main target audience. This bug never affected Debian, simply because utouch-geis has not made its way into the archive yet. It takes longer to make changes like this in Debian, but for some categories of users stability is more important than new, cool features. Choice is a good thing.

ELKI is a data mining software project that I have been working on for the last years as part of my PhD research. It is open source (AGPL-3 licensed) and avilable as both a Debian package and Ubuntu package in the official repositories. So a simple aptitude install elki should get you going and give you a menu entry for ELKI. These packages come with the scripts elki to launch the MiniGUI and elki-cli to run from command line. The key feature that sets ELKI apart from existing open source tools used in data mining (e.g. Weka and R) is that it has support for index structures to speed up algorithms, and a very modular architecture that allows various combinations of data types, distance functions, index structures and algorithms. When looking for performance regressions and optimization potential in ELKI, I recently ran some benchmarks on a data set with 110250 images described by 8 dimensional color histograms. This is a decently sized dataset: it takes long enough (usually in the range of 1-10 minutes) to measure true hotspots. When including Weka and R in the comarison I was quite surprised: our k-means implementation runs at the same speed as Rs implementation in C (and around twice that of the more flexible "flexclus" version). For some of the key agorithms (DBSCAN, OPTICS, LOF) we are an order of magnitude faster than Weka and R, and adding index support speeds up the computation by another factor of 5-10x. In the most extreme case - DBSCAN in Weka vs. DBSCAN with R-tree in ELKI - the speedup was a factor of 330x, or 2 minutes (ELKI) as opposed to 11 hours (Weka).
The reason why I was suprised is that I expected ELKI to perform much worse. It is written in Java (as opposed to R's kmeans, which is in C), uses a very flexible architecture which for example does not assume distances to be of type double and just has a lot of glue code inbetween. However, obviously, the Java Hotspot compiler actually lives up to its expectations and manages to inline the whole distance computations into k-means, and then compiles it at a level comparable to C. R executes vectorized operations quite fast, but on non-native code as in the LOF example it can become quite slow, too. (I would not take Weka as reference, in particular with DBSCAN and OPTICS there seems to be something seriously broken. Judging from a quick look at it, the OPTICS implementation actually is not even complete, and both implementations actually copy all data out of Weka into a custom linear database, process it there, then feed back the result into Weka. They should just drop that "extension" altogether. The much newer and Weka-like LOF module is much more comparable.) Note that we also have a different focus than Weka. Weka is really popular for machine learning, in particular for classification. In ELKI, we do not have a single classification algorithm because there is Weka for that. Instead, ELKI focuses on cluster analysis and outlier detection. And ELKI has a lot of algorithms in this domain, I dare to say the largest collection. In particular, they are all in the same framework, so they can be easily compared. R does of course have an impressive collection in CRAN, but in the end they do not really fit together. Anyway, ELKI is a cool research project. It keeps on growing, we have a number of students writing extensions as part of their thesis. It has been extremely helpful for me in my own research, as I could quickly prototype some algorithms, then try different combinations and use my existing evaluation and benchmarking. You need some time to get started (largely because of the modular architecture, Java generics and such hurdles), but then it is a very powerful research tool. But there are just many more algorithms, published sometime, somewhere, but barely with source code available. We'd love to get all these published algorithms into ELKI, so researchers can try them out. And enhance them. And use them for their actual data. So far, ELKI was mostly used for algorithmic research, but it's starting to move out into the "real" world. More and more people that are not computer scientists start using ELKI to analyze their data. Because it has algorithms that no other tools have. I tried to get ELKI into the "Google Summer of Code", but it was not accepted. But I'd really like to see it gain more traction outside the university world. There are a number of cool projects associated with ELKI that I will not be able to do myself the next years, unfortunately.

• A web browser frontend would be cool. Maybe even connected to Google Refine, using Refine for preprocessing the data, then migrating it into ELKI for analysis. The current visualization engine of ELKI is using SVG - this should be fairly easy to port into the web browser. Likely, the web browers will even be faster than the current Apache Batik renderer.
• Visual programming frontend. Weka, RapidMiner, Orange: they all have visual programming style UIs. This seems to work quite well to model the data flow within the analysis. I'd love to see this for ELKI, too.
• Cluster/Cloud backend. ELKI can already handle fairly large data sets on a big enough system. If someone spends extra effort on the index structures, the data won't even need to fit into main memory anymore. Yet, everybody now wants "big data", and parallel computation probably is the future. I'm currently working on some first Hadoop YARN based experiments with ELKI. But this is a huge field, turning this into true "elk yarn". I will likely only lay some foundations (unless I get funding to continue as a PostDoc on this project. I sure hope to get to do at least a few years of postdoc somewhere, as I really enjoy working with students on this kind of project)
• New visualization engine. The current visualization engine, based on Apache Batik and SVG is quite okay. It does what I need, which is to get a quick glance at the results and the ability to export them for publications in print quality. (in particular, I can easily edit the SVG files with Inkscape) But it is not really something fancy (although we have a lot of cool visualizations). And it is slow. I havn't found a portable and fast graphics toolkit for Java yet that can produce SVG files. There is a lot of hype around processing, for example, but it seems to be too much about art for me. In fact, I'd love to use either something like Clutter or Cairo. But getting them to work for Windows and Mac OSX will likely be a pain.
• Human Computer Interaction (HCI). This is in my opinion the biggest challenge we are facing with all the "big data" stuff. If you really go into big data (and not just run Hadoop and Mahout on a single system; yes - a lot of people seem to do this), you will at some point need to go beyond just crunching the numbers. So far, the challenges that we are tackling are largely data summarization and selection. TeraSort is a cool project, and a challenge. Yet, what do you actually get from sorting this large amount of data? What do you get from running k-means on a terabyte? When doing data mining on a small data set, you quickly learn that the main challenge actually is preprocessing the data and choosing parameters the right way so that your result is not bogus. Unless you are doing simple prediction tasks, you often don't have a clearly defined objective. Sure, when predicting churn rates, you can hope to just throw all the data into a big cloud and hope you get some enlightement out. But when you are doing cluster analysis or outlier detection - unsupervised methods - the actual objective by definition cannot be hardcoded into a computer. The key objective then is learn something new on the data set. But if you want to have your user learn something on the data set, you will have to have the user guide the whole process, and you will have to present results to the user. Which gets immensely more difficult with larger data. Big data just does no longer look like this. And neither are the algorithms as simple as k-means or hierarchical clustering. Hierarchical clustering is good for teaching the basic ideas of clustering. But you will not be using a dendrogram for a huge data set. Plus, it has a naive complexity of O(n^3) and for some special cases O(n^2) - too slow for truly big data.
  For the "big data future" once we get over all the excitement of being able to just somehow crunch these numbers we will need to seriously look into what to do with the results (in particular, how to present them to the user), and how to make the algorithms accessible and usable for non-techies. Right now, you cannot expect a social sciences researcher to be able to use a Hadoop cluster. Yet to make sense of the results. But if you are a smart guy to actually solve this, and open up "big data processing" to the average non-IT user, this will be big.
• Oh, and of course there are just hundreds of algorithms not yet available (accessible) as open source. Not in ELKI, and usually not anywhere else either. Just to name a few from my wishlist (I could probably implement many of them in a few hours in ELKI, but I don't have the time to do so myself, plus they are good student or starter project to get used to ELKI): BIRCH, CLARA, CLARANS, CLINK, COBWEB, CURE, DOC, DUSC, EDSC, INSCY, MAFIA, P3C, SCHISM, STATPC, SURFING, ... just to name a few.

If you are a researcher in cluster analysis or outlier detection, consider contributing your algorithms to ELKI. Spend some time optimizing them, adding some documentation. Because, if ELKI keeps on growing and gaining popularity, it will be the future benchmark platform. And this can give you citations, which are somewhat the currency of science these days. Algorithms available in the major toolkits just do get cited more, because people compare to them. See this list for an overview of work cited by ELKI - scientific work that we reimplemented at least to some extend for ELKI. It is one of the services that we provide with ELKI for researchers: not only the algorithm, but also the appropriate citation.

XMPPloit is an exploit tool for a so-called flaw in the XMPP protocol. It has been published recently under the GPLv3 license, and has received much comment. However, it does not seem anybody took the time to study this attack and explain it. Goals XMPPloit is designed to serve as a transparent man-in-the-middle between an XMPP client and its XMPP server, in order to force the client not to encrypt its communications, so that it is possible to read them and modify them on-the-fly. That allows to force the client to use a clear text authentication mechanism, to display its login and password, and to modify any message it sends or receives. Mode of operation If you were expecting a tricky, undetectable attack against the XMPP protocol itself, you will be disappointed. This attack could not be simpler:

1. In XMPP, the clients opens a connection to the server as plain text. In this case, with the attacker between them.
2. The server offers the STARTTLS extension. The attacker blocks it, that is, does not transmit this offer to the client, so the connection stays in plain text.
3. The client authenticate over the plain text connection.

Comments

• This is not at all a flaw of the XMPP protocol. It is applicable to POP, IMAP, SMTP, in fact to any protocol that uses clear text connections and upgrades them using a STARTTLS-like mechanism. This includes HTTP with redirection to HTTPS.
• This is not transparent, at least not with XMPP client that respect the RFC recommendation that Clients SHOULD use TLS to secure the streams prior to attempting the completion of SASL negotiation. In Gajim for instance, the user would be informed that the authentication is going to happen without encryption, and asked for a confirmation.

In fact, this is only an exploit against design flaws of some XMPP clients that would not warn the user that they are about to send their credentials on a clear text channel. Actually, the protocol most vulnerable to this type of attack is HTTP. Many websites that identify their users are made available over clear text HTTP, with an automatic redirection to HTTPS. Unless they use HSTS, this redirection can be blocked by an attacker without being noticed by most users. And, contrary to XMPP, not warn the user of an insecure login is not the fact of some broken clients, but of all the available browsers. I repeat: not a single browser will currently tell the user: Hey, you are about to transmit a password in clear, it could be intercepted! . Yes, there are browser that warn their users this way the first time: Hey, you are about to transmit your answers to a form in clear! Do you wish to do that, and do you want me to warn you next time? but this not quite the same, and it is just badly designed and in practice neither useful nor usable so everybody disables that.

Google are providing some really good employee benefits including benefits to a life partner of a deceased employee [1]. It s not known if all those benefits are available outside the US, in any case the US is the first world country with the least social security so they need it most there. A recent Australian legal case had a father petitioning the court to have his kids take his family name [2]. According to the news report no good reason was given for renaming the kids, merely tradition. The mother won. GlassDoor.com is a site for reviewing companies [3]. It also has job adverts, it seems that they get people in to read the reviews and then advertise jobs. Sarah Resnick interviewed Jacob Appelbaum (of Tor fame) about privacy issues and published the article as Leave Your Cellphone at Home [4]. It s very interesting and references some resources such as riseup.net that I have to try using. Systemd in Fedora 17 has multi-seat support [5]. They support plugging USB terminals in at run-time to dynamically add new consoles for GNOME sessions. The Coding Horror blog has an amusing and informative post about why people shouldn t learn to code [6].

• [1] http://www.news.com.au/technology/death-benefits-google-workers-get-sweet-perks-for-life-and-death/story-e6frfro0-1226447351009
• [2] http://www.news.com.au/national/mum-wins-the-right-to-name-kids/story-fncynjr2-1226447058665
• [3] http://www.glassdoor.com
• [4] http://nplusonemag.com/leave-your-cellphone-at-home
• [5] http://0pointer.de/blog/projects/multi-seat.html
• [6] http://www.codinghorror.com/blog/2012/05/please-dont-learn-to-code.html

Related posts:

1. Links June 2012 This Youtube video is an amusing satire of EULAs and...
2. Links January 2012 Cops in Tennessee routinely steal cash from citizens [1]. They...
3. Links August 2008 Michael Janke is writing a series of posts about estimating...

DPU, as I ve mentioned casually, is a testing framework for Debian tools. I ll expand a bit more about what DPU does, what it s going to be used for, and how it can help you (yes, you!) create tools to work with Debian packages. Firstly, DPU is a test framework. It s sole purpose is only to run tests, and give each test a pass / fail grade. The test syntax will be as declarative and simple as one can keep such things, and tries to do the right thing(tm). It s not currently stable, but it s getting there. DPU aims to replace Lintian s test suite, and clean up the tests to make them a bit more declarative, and less hackish. At it s core, DPU uses a templating system (using Jinja2) to render out a policy-perfect Debian source directory, including (if needed) any number of fake Upstream files, such as Hello World! in Shell (for arch:all), in C (for arch:any), and so forth. DPU tests mangle (slightly) the files before a run simple stuff like removing a manpage, or adding a file where it shouldn t be. After the builder builds the package (with any of the global or test-local builders), any number of checkers (again, global or local) are run, and output compared. This is the part that s currently undergoing work, but the basics are alive, and an example test directory is on my github. Everything s in Python, and new contributors are welcome!

Several months ago, I wrote about grep everything and listed grep-like tools which can grep through compressed files or specific data formats. The blog posting sparked several magazine articles and talks by Frank Hofmann and me. Frank recently noticed that we though missed one more or less mighty tool so far. We missed it, because it s mostly unknown, undocumented and hidden behind a package name which doesn t suggest a real recursive grep everything : deepgrep deepgrep is part of the Debian package strigi-utils, a package which contains utilities related to the KDE desktop search Strigi. deepgrep especially eases the searching through tar balls, even nested ones, but can also search through zip files and OpenOffice.org/LibreOffice documents (which are actually zip files). deepgrep seems to support at least the following archive and compression formats:

• tar
• ar, and hence deb
• rpm (but not cpio)
• gzip/gz
• bzip2/bz2
• zip, and hence jar/war and OpenOffice.org/LibreOffice documents
• MIME messages (i.e. files attached to e-mails)

A search in an archive which is deeply nested looks like this:

$ deepgrep bar foo.ar
foo.ar/foo.tar/foo.tar.gz/foo.zip/foo.tar.bz2/foo.txt.gz/foo.txt:foobar
foo.ar/foo.tar/foo.tar.gz/foo.zip/foo.tar.bz2/foo.txt.gz/foo.txt:bar

deepgrep though neither seems to support any LZMA based compression (lzma, xz, lzip, 7z), nor does it support lzop, rzip, compress (.Z suffix), cab, cpio, xar, or rar. Further current drawbacks of deepgrep:

• Nearly no commandline options, especially none of the common grep options
• No man-page or other documentation
• Exit code not related to search results, you have to check the output to see if something has been found

deepfind If you just need the file names of the files in nested archives, the package also contains the tool deepfind which does nothing else than to list all files and directories in a given set of archives or directories:

$ deepfind foo.ar
foo.ar
foo.ar/foo.tar
foo.ar/foo.tar/foo.tar.gz
foo.ar/foo.tar/foo.tar.gz/foo.zip
foo.ar/foo.tar/foo.tar.gz/foo.zip/foo.tar.bz2
foo.ar/foo.tar/foo.tar.gz/foo.zip/foo.tar.bz2/foo.txt.gz
foo.ar/foo.tar/foo.tar.gz/foo.zip/foo.tar.bz2/foo.txt.gz/foo.txt

As with deepgrep, deepfind does not implement any common options of it s normal sister tool find. Dependencies The package strigi-utils doesn t pull in the complete Strigi framework (i.e. no daemon), just a few libraries (libstreams, libstreamanalyzer, and libclucene). On Wheezy it also pulls in some audio/video decoding libraries which may make some server administrators less happy. Conclusion Both tools are quite limited to some basic use cases, but can be worth a fortune if you have to work with nested archives. Nevertheless the claim in the Debian package description of strigi-utils that they re enhanced versions of their well known counterparts is IMHO disproportionate. Most of the missing features and documentation can be explained by the primary purpose of these tools: Being backend for desktop searches. I guess, there wasn t much need for proper commandline usage yet. Until now. ;-) 42.zip And yes, I was curious enough to let deepfind have a look at 42.zip (the one from SecurityFocus, unzip seems not able to unpack 42.zip from unforgettable.dk due a missing version compatibility) and since it just traverses the archive sequentially, it has no problem with that, needing just about 5 MB of RAM and a lot of time:

[ ]
42.zip/lib f.zip/book f.zip/chapter f.zip/doc f.zip/page e.zip
42.zip/lib f.zip/book f.zip/chapter f.zip/doc f.zip/page e.zip/0.dll
42.zip/lib f.zip/book f.zip/chapter f.zip/doc f.zip/page f.zip
42.zip/lib f.zip/book f.zip/chapter f.zip/doc f.zip/page f.zip/0.dll
deepfind 42.zip  11644.12s user 303.89s system 97% cpu 3:24:02.46 total

I though won t try deepgrep on 42.zip. ;-)

There are so many sites trying to vie for Google results for any and every town in existance in the United States that they are crowding out useful information. They get their list of towns from census data or similar sources, and generate pages for every single entry they find, no matter how small. During the weekend, I was poking around, looking for information on a not-quite-ghost town called Lonerock, in Oregon:
http://en.wikipedia.org/wiki/Lonerock,_Oregon
I love to visit out of the way places like that when I'm back in Oregon, which we hope to do next summer. So I was curious about it - it appears quite isolated, but seems like it might be worth a look. According to the Wikipedia article it is a very small town, with a population of 24 people. And yet, if you look at the Google results for it, you find a few good links at first, and then:

• Any number of results with "city information", which is just census data with a bit of fiddling.
• Current local time. Gee, that's great to know.
• Various 'homes for sale' sites: none listed.
• Horses for sale "in Lonerock", but it turns out they're all in Heppner (30 miles away) or farther.
• Climate statistics.
• Bicycles for sale in Lonerock. Surprisingly, none of those, either.
• Attorneys at law in Lonerock. I think they ran them all out of town: there are none.
• Truck scales and weigh stations in Lonerock. You have to drive at least 50 miles to get to the nearest one.
• Hotels in Lonerock: nope, none of those either.
• Various Cable and Internet offerings (I think in reality they use this: http://en.wikipedia.org/wiki/IP_over_Avian_Carriers )
• Doctors in Lonerock: Neurologists, Gastroenterologists and even Nephrologists. Turns out there aren't any in town.
• Car Rentals in Lonerock. Just in case you fly in to any of the Airports in Lonerock, and want to leave rather than staying in the Hotels in Lonerock.
• Apartments for rent in Lonerock. A town of 24 people in the middle of the wide open west is not the kind of place you're likely to live in an apartment...
• Relocation guides for Lonerock.

And so on and so forth. Scattered in the middle, you can even find a few articles and photo pages by people who actually had something to say or show about the place, but finding them amidst all the crud is not an easy task. To me this seems surprising - it doesn't seem like it'd be that hard to make sites that just happen to have entries for every town in the entire nation rank a bit lower than things written by human beings. Google - have a look at Lonerock, and see if you can use it as a way to seperate the wheat from the chaff!

SSDs have been dropping in price recently so I just bought four Intel 120G devices for $115 each. I installed the first one for my mother in law who had been complaining about system performance. Her system boot time went from 90 seconds to 20 seconds and a KDE login went from about 35 seconds to about 10 seconds. The real problem that she had reported was occasional excessive application delay, while it wasn t possible to diagnose that properly I think it was a combination of her MUA doing synchronous writes while other programs such as Chromium were doing things. To avoid the possibility of a CPU performance problem I replaced her 1.8GHz E4300 system with a 2.66GHz E7300 that I got from a junk pile (it s amazing what s discarded nowadays). I also installed a SSD in my own workstation (a 2.4GHz E4600). The boot time went down from 45s on Ext4 without an encrypted root to 27s with root on BTRFS including the time taken to enter the encryption password (maybe about 23s excluding my typing time). The improvement wasn t as great, but that s because my workstation does some things on bootup that aren t dependent on disk IO such as enabling a bridge with STP (making every workstation a bridge is quieter than using switches). KDE login went from about 27s to about 12s and the time taken to start Chromium and have it be usable (rather than blocking on disk IO) went from 30 seconds to an almost instant response (maybe a few seconds)! Tests on another system indicates that Chromium startup could be improved a lot by purging history, but I don t want to do that. It s unfortunate that Chromium only supports deleting recent history (to remove incriminating entries) but doesn t support deleting ancient history that just isn t useful. I didn t try to seriously benchmark the SSD (changing from Ext4 to BTRFS on my system would significantly reduce the accuracy of the results), I have plans for doing that on more important workloads in the near future. For the moment the most casual tests have shown a significant performance benefit so it s clear that an SSD is the correct storage option for any new workstation which doesn t need more than 120G of storage space. $115 for SSD vs $35 for HDD is a fairly easy choice for a new system. For larger storage the price of hard drives increases more slowly than that of SSD. In spite of the performance benefits I doubt that I will gain a real benefit from this in the next year. The time taken to install the SSD equates to dozens of boot cycles which given a typical workstation uptime in excess of a month is unlikely to happen soon. One minor benefit is that deleting messages in Kmail is an instant operation which saves a little annoyance and there will be other occasional benefits. One significant extra benefit is that an SSD is quiet and dissipates less heat which might allow the system cooling fans to run more slowly. As noisy computers annoy me an SSD is a luxury feature. Also it s good to test new technologies that my clients may need. The next thing on my todo list is to do some tests of ZFS with SSD for L2ARC and ZIL. Related posts:

1. How I Partition Disks Having had a number of hard drives fail over the...
2. Xen and Swap The way Xen works is that the RAM used by...
3. big and cheap USB flash devices It s often the case with technology that serious changes occur...

jenkins-debian-glue is an open source project of mine which recently celebrated its first birthday and it s time to finally write about it. jenkins-debian-glue allows you to build Debian and Ubuntu packages directly from the Jenkins continuous integration system. It retrieves package sources from a version control repository, adjusts debian/changelog (handle version number + mention changes that took place) and builds according source and binary packages out of it. Its lintitan integration provides Q/A reports about the resulting source and binary Debian packages. It started as a small pet project of mine to get integration of Debian packaging inside Jenkins. I mainly had the needs for the Grml project in mind and starting with Grml 2011.12 every release (including also all the daily builds) was built using Jenkins since then. It turned out that my project would also be a perfect match for one of my customers, Sipwise GmbH. In December 2011 the sip:provider 2.4 was the first stable release that was built 100% through Jenkins and featuring jenkins-debian-glue. Thanks to special needs and the open source friendliness of Sipwise I could invest further time into the project. Recently the project got even some further drive thanks to interest by some fellow Debian Developers, most notably are the icinga, nagios-plugins, packages and the PostgreSQL in Debian Hackathon. Now if you re also interested in it: there s an automated deployment procedure available for your service to get started with the whole jenkins-debian-glue and Jenkins stack in less than 10 minutes, working just fine on Debian as well as Ubuntu. For further details please head over to jenkins-debian-glue.org or check out this 6:36min screencast (safe for work, no audio ): <iframe allowfullscreen="allowfullscreen" frameborder="0" height="315" src="http://www.youtube.com/embed/9asp2ZvH90A?autohide=1&amp;showinfo=0" width="560"></iframe> This embedded video doesn t work for you? Try heading over to YouTube.

A while back mdz blogged about challenges facing Ubuntu and other Linux distributions. He raises the point that runtime libraries for Python / Ruby etc have a unique set of issues because they tend to have their own packaging systems. Merely a month later he attended Debconf 2010 where a presentation was given on the issues that Java packages have on Dpkg based systems. Since then the conversation seems to have dried up. I ve been reminded of it recently in discussions within Canonical looking at how we deploy web services. Matt suggested some ways forward, including:

• Decouple applications from the core
• Treat data as a service (rather than packages) get data live from the web rather than going web -> distro-package -> user machines.
• Simplify integration between packaging systems (including non-packaged things)

I think its time we revisit and expand on those points. Nothing much has changed in how Ubuntu or other distributions approach integration with other packaging systems but the world has kept evolving. Internet access is growing ever more ubiquitous, more platforms are building packaging systems clojure, scala, node.js, to name but three, and a substantial and ever growing number of products expect to operate in a hybrid fashion with an evolving web service plus a local client which is kept up to date via package updates. Twitter, Facebook and Google Plus are three such products. Android has demonstrated a large scale app store on top of Linux, with its own custom packaging format. In order to expand them, we need some background context on the use cases that these different packaging systems need to support. Platforms such as antivirus scanners, node.js, Python, Clojure and so forth care a great deal about getting their software out to their users. They care about making it extremely easy to get the latest and greatest versions of their libraries. I say this because the evidence is all around us: every successful development community / product has built a targeted package management system which layers on top of Windows, and Mac OSX, and *nux. The only rational explanation I can come up for this behaviour is that the lower level operating system package management tools don t deliver what they need. E.g. this isn t as shallow as wanting a packaging system written in their own language, which would be easy to write off as parochialism rather than a thoughtful solution to their problems. In general packaging systems provide a language for shipping source or binary form, from one or more repositories, to users machines. They may support replications, and they may support multiple operating systems. They generally end up as graph traversal engines, pulling in dependencies of various sorts you can see the DOAP specification for an attempt at generic modelling of this. One problem that turns up rapidly when dealing with Linux distribution package managers is that the versions upstream packages have, and the versions a package has in e.g. Debian, differ. They differ because at some stage, someone will need to do a new package for the distribution when no upstream change has been made. This might be to apply a local patch, or it might be to correct a defect caused by a broken build server. Whatever the cause, there is a many to one relationship between the package versions that end users see via dpkg / rpm etc, and those that upstream ship. It is a near certainty that once this happens to a library package, that comparing package versions across different distribution packages becomes hard. You cannot reliably infer whether a given package version is sufficient as a dependency or not, when comparing binary packages between Red Hat and Debian. Or Debian and Ubuntu. The result of this is that even when the software (e.g. rpm) is available on multiple distributions (say Ubuntu and RHEL), or even on multiple operating systems (say Ubuntu and Windows), that many packages will /have/ to be targeted specifically to build and execute properly. (Obviously, compilation has to proceed separately for different architectures, its more the depedency metadata that says and build with version X of dependency Y that has to be customised). The result of this is that there is to the best of my knowledge no distribution of binary packages that targets Debian/Ubuntu and RHEL and Suse and Windows and Mac OS X, although there are vibrant communities building distributions of and for each in isolation. Some of the ports systems come close, but they are still focused on delivering to a small number of platforms. There s nothing that gives 99% coverage of users. And that means that to reach all their users, they have to write or adopt a new system. For any platform X, there is a strong pressure to have the platform be maintainable by folk that primarily work with X itself, or with the language that X is written in. Consider Python there is strong pressure to use C, or Python, and nothing else, for any tools that is somewhat parochial, but also just good engineering reducing variables and making the system more likely to be well maintained. The closest system I know of Steam is just now porting to Ubuntu (and perhaps Linux in general), and has reached its massive popularity by focusing entirely on applications for Windows, with Mac OSX a recent addition. Systems like pypi which have multi platform eggs do target the wide range of platforms I listed above, but they do so both narrowly and haphazardly: whether a binary or source package is available for a given platform is up to the maintainer of the package, and the packages themselves are dealing with a very narrow subset of the platforms complexity: Python provides compilation logic, they don t create generic C libraries with stable ABI s for use by other programs, they don t have turing complete scripts for dealing with configuration file management and so forth. Anti virus updaters similarly narrow the problem they deal with, and add constraints on latency- updates of anti virus signatures are time sensitive when a new rapidly spreading threat is detected. A minor point, but it adds to the friction of considering a single packaging tool for all needs is the different use cases of low level package management tools like dpkg or rpm vs the use cases that e.g. pypi has. A primary use case for packages on pypi is for them to be used by people that are not machine administrators. They don t have root, and don t want it. Contrast that with dpkg or rpm where the primary use case (to date) is the installation of system wide libraries and tools. Things like man page installation don t make any sense for non-system-wide package systems, whereas they are a primary feature for e.g. dpkg. In short, the per-platform/language tools are (generally):

1. Written in languages that are familiar to the consumers of the tools.
2. Targeted at use on top of existing platforms, by non-privileged users, and where temporary breakage is fine.
3. Intended to get the software packaged in them onto widely disparate operating systems.
4. Very narrow they make huge assumptions about how things can fit together, which their specific language/toolchain permits, and don t generalise beyond that.
5. Don t provide for security updates in any specific form: that is left up to folk that ship individual things within the manager.

operating system package managers:

1. Are written in languages which are very easy to bootstrap onto an architecture, and to deploy onto bare metal (as part of installation).
2. Designed for delivering system components, and to avoid be able to upgrade the toolchain itself safely.
3. Originally built to install onto one operating system, ports to other operating systems are usually fragile and only adopted in niche.
4. Are hugely broad they install data, scripts, binaries, and need to know about late binding, system caches etc for every binary and runtime format the operating system supports
5. Make special provision to allow security updates to be installed in a low latency fashion, without requiring anything consuming the package that is updated to change [but usually force-uninstalling anything that is super-tightly coupled to a library version].

Anti virus package managers:

1. Exist to update daemons that run with system wide escalated privileges, or even file system layer drivers.
2. Update datasets in realtime.
3. Without permitting updates that are produced by third parties.

Given that, lets look at the routes Matt suggested Decoupling applications from the core as a strategy makes an assumption that the core and applications are partitionable. If they are not, then applications and the core will share common elements that need to be updated together. Consider, for instance, a Python application. If you run with a system installed Python, and it is built without zlib for some reason, but the Python application requires zlib, you have a problem. A classic example of this problem is facing Ubuntu today, with all the system provided tools moving to Python 3, but vast swathes of Python applications still being unported to Python 3 at all. Currently, the Python packaging system virtualenv/buildout + distribute don t provide a way to install the Python runtime itself, but will happily install their own components for everything up the stack from the runtime. Ubuntu makes extensive use of Python for its own tools, so the system Python has a lot of packages installed which buildout etc cannot ignore this often leads to issues with e.g. buildout, when the bootstrap environment has (say) zope.interfaces, but its then not accessible from the built-out environment that disables the standard sys.path (to achieve more robust separation). If we want to pursue decoupling, whether we build a new package manager or use e.g. virtualenv (or gem or npm or ), we ll need to be aware of this issue and perhaps offer, for an extended time, a dedicated no-frills, no-distro-packages install, to avoid it, and to allow an extended supported period for application authors without committing to a massive, distro sponsored porting effort. While its tempting to say we should install pip/npm/lein/maven and other external package systems, this is actually risky: they often evolve sufficiently fast that Ubuntu will be delivering an old, incompatible version of the tool to users well before Ubuntu goes out of support, or even befor the next release of Ubuntu. Treating data as a service. All the cases I ve seen so far of applications grabbing datasets from the web have depended on web infrastructure for validating the dataset. E.g. SSL certificates, or SSL + content checksums. Basically, small self-rolled distribution systems. I m likely ignorant of details here, and I depend on you, dear reader, to edumacate me. There is potential value in having data repackaged, when our packaging system has behind-firewall support, and the adhoc system that (for instance) a virus scanner system has does not. In this case, I specifically mean the problem of updated a machine which has no internet access, not even via a proxy. The challenge I see it is again the cross platform issue: The vendor will be supporting Ubuntu + Debian + RHEL + Suse, and from their perspective its probably cheaper to roll their own solution than to directly support dpkg + rpm + whatever Apple offer + Windows the skills to roll an adhoc distribution tool are more common than the skills to integrate closely with dpkg or rpm What about creating a set of interfaces for talking to dpkg / rpm / the system packagers on Windows and Mac OSX ? Here I think there is some promise, but it needs as Matt said careful thought. PackageKit isn t sufficient, at least today. There are, I think, two specific cases to cater to:

1. The anti-virus / fresh data set case.
2. The egg/gem/npm/ specific case.

For the egg/gem/npm case, we would need to support a pretty large set of common functionality, on Windows/Mac OSX / *nux (because otherwise upstream won t adopt what we create: losing 90% of their users (windows) or 5% (mac) isn t going to be well accepted . We d need to support multiple installations (because of mutually incompatible dependencies between applications), and we d need to support multiple language bindings in some fashion some approachable fashion where the upstream will feel capable of fixing and tweaking what we offer. We re going to need to support offline updates, replication, local builds, local repositories, and various signing strategies to match the various tradeoffs made by the upstream tools. For the anti-virus / fresh data case, we d need to support a similar set of operating systems, though I strongly suspect that there would be more tolerance for limited support in that most things in that space either have very platform specific code, or they are just a large-scale form of the egg/gem/npm problem, which also wants easy updates. What next? We should validate this discussion with at least two or three upstreams. Find out whats missing I suspect a lot and whats wrong I hope not much . Then we ll be in a position to decide if there is a tractable, widespread solution *possible*. Separately, we should stop fighting with upstreams that have their own packaging systems. They are satisfying different use cases than our core distro packaging systems are designed to solve. We should stop mindlessly repackaging things from e.g. eggs to debs, unless we need that specific thing as part of the transitive runtime or buildtime dependencies for the distribution itself. In particular, if us folk that build system packaging tools adopt and use the upstream application packaging tools, we can learn in a deep way the (real) advantages they have, and become more able to reason about how to unify the various engineering efforts going into them and perhaps even eventually satisfy them using dpkg/rpm on our machines.

There are plenty of times where it is nice to have Linux transmit things out a radio. One obvious example is the digital communication modes, where software acts as a sort of modem. A prominent example of this in Debian is fldigi. Sometimes, it is nice to transmit voice instead of a digital signal. This is called voice keying. When operating a contest, for instance, a person might call CQ over and over, with just some brief gaps. Most people that interface a radio with a computer use a sound card interface of some sort. The more modern of these have a simple USB cable that connects to the computer and acts as a USB sound card. So, at a certain level, all that you have to do is play sound out a specific device. But it s not quite so easy, because there is one other wrinkle: you have to engage the radio s transmitter. This is obviously not something that is part of typical sound card APIs. There are all sorts of ways to do it, ranging from dedicated serial or parallel port circuits involving asserting voltage on certain pins, to voice-activated (VOX) circuits. I have used two of these interfaces: the basic Signalink USB and the more powerful RigExpert TI-5. The Signalink USB integrates a VOX circuit and provides cabling to engage the transmitter when VOX is tripped. The TI-5, on the other hand, emulates three USB serial ports, and if you raise RTS on one of them, it will keep the transmitter engaged as long as RTS is high. This is a more accurate and precise approach. VOX-based voice keying with the Signalink USB But let s first look at the Signalink USB case. The problem here is that its VOX circuit is really tuned for digital transmissions, which tend to be either really loud or completely silent. Human speech rises and falls in volume, and it tends to rapidly assert and drop PTT (Push-To-Talk, the name for the control that engages the radio s transmitter) when used with VOX. The solution I hit on was to add a constant, loud tone to the transmitted audio, but one which is outside the range of frequencies that the radio will transmit (which is usually no higher than 3kHz). This can be done using sox and aplay, the ALSA player. Here s my script to call cq with Signalink USB:

#!/bin/bash
# NOTE: use alsamixer and set playback gain to 99
set -e
playcmd ()  
        sox -V0 -m "$1" \
           "  sox -V0 -r 44100 $1 -t wav -c 1 -   synth sine 20000 gain -1" \
            -t wav -   \
           aplay -q  -D default:CARD=CODEC
 
DELAY=$ 1:-1.5 
echo -n "Started at: "
date
STARTTIME= date +%s 
while true; do
        printf "\r"
        echo -n $(( ( date +%s -$STARTTIME) / 60))
        printf "m/$ DELAY s: TRANSMIT"
        playcmd ~/audio/cq/cq.wav
        printf "\r"
        echo -n $(( ( date +%s -$STARTTIME) / 60))
        printf "m/$ DELAY s: off         "
        sleep $DELAY
done

Run this, and it will continuously play your message, with a 1.5s gap in between during which the transmitter is not keyed. The screen will look like this:

Started at: Fri Aug 24 21:17:47 CDT 2012
2m/1.5s: off

The 2m is how long it s been going this time, and the 1.5s shows the configured gap. The sox commands are really two nested ones. The -m causes sox to merge the .wav file in $1 with the 20kHz sine wave being generated, and the entire thing is piped to the ALSA player. Tweaks for RigExpert TI-5 This is actually a much simpler case. We just replace playcmd as follows:

playcmd ()  
        ~/bin/raiserts /dev/ttyUSB1 'aplay -q -D default:CARD=CODEC' < "$1"
 

Where raiserts is a program that simply keeps RTS asserted on the serial port while the given command executes. Here's its source, which I modified a bit from a program I found online:

/* modified from
 * https://www.linuxquestions.org/questions/programming-9/manually-controlling-rts-cts-326590/
 * */
#include <stdio.h>
#include <stdlib.h>
#include <termios.h>
#include <unistd.h>
#include <sys>ioctl.h>
#include <sys< sys="">types.h>
#include <sys>stat.h>
#include <fcntl.h>
static struct termios oldterminfo;
void closeserial(int fd)
 
    tcsetattr(fd, TCSANOW, &oldterminfo);
    if (close(fd) < 0)
        perror("closeserial()");
 
int openserial(char *devicename)
 
    int fd;
    struct termios attr;
    if ((fd = open(devicename, O_RDWR)) == -1)  
        perror("openserial(): open()");
        return 0;
     
    if (tcgetattr(fd, &oldterminfo) == -1)  
        perror("openserial(): tcgetattr()");
        return 0;
     
    attr = oldterminfo;
    attr.c_cflag  = CRTSCTS   CLOCAL;
    attr.c_oflag = 0;
    if (tcflush(fd, TCIOFLUSH) == -1)  
        perror("openserial(): tcflush()");
        return 0;
     
    if (tcsetattr(fd, TCSANOW, &attr) == -1)  
        perror("initserial(): tcsetattr()");
        return 0;
     
    return fd;
 
int setRTS(int fd, int level)
 
    int status;
    if (ioctl(fd, TIOCMGET, &status) == -1)  
        perror("setRTS(): TIOCMGET");
        return 0;
     
    status &= ~TIOCM_DTR;   </sys>* ALWAYS clear DTR */
    if (level)
        status  = TIOCM_RTS;
    else
        status &= ~TIOCM_RTS;
    if (ioctl(fd, TIOCMSET, &status) == -1)  
        perror("setRTS(): TIOCMSET");
        return 0;
     
    return 1;
 
int main(int argc, char *argv[])
 
    int fd, retval;
    char *serialdev;
    if (argc < 3)  
        printf("Syntax: raiserts /dev/ttyname 'command to run while RTS held'\n");
        return 5;
     
    serialdev = argv[1];
    fd = openserial(serialdev);
    if (!fd)  
        fprintf(stderr, "Error while initializing %s.\n", serialdev);
        return 1;
     
    setRTS(fd, 1);
    retval = system(argv[2]);
    setRTS(fd, 0);
    closeserial(fd);
    return retval;
 

This compiles to an executable less than 10K in size. I love it when that happens. So these examples support voice keying both with VOX circuits and with serial-controlled PTT. raiserts.c could be trivially modified to control other serial pins as well, should you have an interface which uses different ones.

Joy Theme When the Joy theme was picked as the chosen theme for Debian 7.0 (Wheezy), I wanted to do what I could to get as much of it in the archive as possible before the freeze that occurred in June. I ve been working with Paul Tagliamonte and Vagrant Cascadian and I m glad that it resulted in some nice things! The Joy theme is modest and beautiful and I think Debian 7.0 is going to be one of the best looking Debian releases so far. Joy Plymouth Theme Plymouth is the part that provides the boot splash. It also takes care of a few other things, like showing prompts and progress indicators for filesystem checks, password prompts for encrypted devices and more so that the splash doesn t need to exit to show those. It s very scriptable, The installers for the Genesi Efika range of devices even use it for it s front-end. I needed to start with a Plymouth theme where I knew the prompts for all the things that happen during boot was at least more or less implemented, so I started off with one that I know works well, the Edubuntu one. So, fun fact: Debian 7.0 s Plymouth theme is actually based on the Edubuntu one. I had to play with the colours a bit to get it right, Edubuntu has a light background so I had to invert contrast for things like the input box, but I think it came out nicely. Splashy isn t in the archives anymore and it s not widely used anymore, so I cleaned it up from the desktop-base package so that there s less clutter shipped with it. Joy LTSP/LDM Theme This screenshot actually looks a bit uglier than it should because I took it on a VM that doesn t have 24bit colour, but we re looking into what we can do to make it look better on 16bit colour as well. During Debconf in Nigaragua, Vagrant and I worked on getting the LDM theme in shape. I have already done what I thought was most of the work by the time Debconf started, but it turns out there was a lot more to it to do it properly. We managed to get rid of the gtk2-engines-murrine dependency, which in itself is quite tiny, but it s a bit backwards in that it depends on the whole rest of the murrine-themes which brings in too much. We took the time to get the dependencies/recommends/etc right and to make it the default theme in Debian without messing it up for any Debian derivatives. It all came together and now when you install LTSP on Debian, you get the Joy theme by default! I m really glad about that because I didn t get the Spacefun LDM theme done in time for the Squeeze release. Enjoy!

Sometimes, very rarely, I am happy that the spam filters in my personal mail server are so far from perfect. Many of us were amazed, some days ago, news broke that Amelia Earhart's plane was found in the Southwestern Pacific. The news is amazing and worthy in itself. But today, I got this mail, proof that Amelia lived through all those years: (all addresses obscured, of course)

Date: Wed, 22 Aug 2012 23:38:59 -0700
From: Amelia Earhart <some-mail@gmail.com>
To: undisclosed-recipients: ;
Subject: Hello
Reply-To: some-other-mail@live.com Hello friend, With due respect to humanity, i am compelled to write to you on a humanitarian ground, however, it has been my wish to execute in life. Please, I am contacting you based on my desire to create a charitable fund in your country with this sum of EURO 3.5 Million which I inherited from my late husband (Dr. Ahrens Earhart). It is my desire to see that this money is invested in any organization of your choice in your country and distributed each year among the charity organizations, the house mother of the child, mosques, churches, schools, support for disadvantaged older men and women or in whatever way you intend helping the less privileged. I took this decision because I was raised out of the house of mother and baby at the moment, i am admitted in a hospital in London were i am taking severe treatment to my cancer illness. Please answer as soon as possible to confirm your agreement with this proposal, so I ll give you all the necessary information that will enable you to have access to this funds my personal accountant will be aware of my arrangements with you. Thank you for your attention and I hope to hear from you soon. Please contact me through my this email- some-other-mail@live.com the nurse helping me will attend to you i have been restricted from touching the computer due to my illness i will direct you through her she is a nice person. Sincerely,
Ms. Amelia Earhart.

[ once again, I am translating somebody else's material In this case, my good Costa Rican friend Carolina Flores. Please excuse my stylistic mistakes My English is far from native as you well know. But this material is worth sharing, and worth investing some tens of minutes doing a quick translation. If you can read Spanish, go read Caro's original entry ] Have you been to a music record store lately? I did so last Saturday, as a mere excercise. I was not planning on buying anything but I wanted to monitor things and confirm my suspicions. What was I suspicious of? First, that I would only find old records. And so it was: The only recent record I found was ...little broken hearts by Norah Jones. the second, that I would only find music for over 50 year old people. so it was: Were I there to look for a present for my father, I would have walked out with 10 good records. Third, that in the store nothing worth commenting would happen. About that last point, I should point out it was around 10 AM and the store had just opened its doors. Lets concede the benefit of doubt. I don't think many of you will remember, but in Barrio La California (where there is now a beauty parlour, almost in front of AM.PM) there was Auco Disco. In Auco Disco there was a guy specialized in rock (Mauricio Alice) and another one specialized in jazz (I don't remember his name). In that record you could always find rare records, but if they were not there, at least you were sure to find somebody to say: "No, we don't have that, but that's an excellent record, it's the best that [insert group here] have ever recorded because just afterwards they switched their guitar player, they had gone a bit south but with that record they are flying. But no, we don't have it; I can recommend you this record by [insert another group] because it has a guitar solo in track six that is amazing". It would happen more or less like that, which means, one would arrive to Auco Disco at 10 AM and leave around 5 PM with three new records, after having listened to a spectacular music selection. What happened to those stores? Were they killed by The Pirate Bay? That's the simplistic answer from the recording industry! The answer is that those stores never got anything from the industry but an invoice. The industry specially in prescindible markets such as ours was limited to hiring artists, taking care of them recording a sellable product, producing the object called record, and that's it. The more commercial radio stations were paid to program those songs as it cannot be casual that "Mosa, mosa" is the summer hit in all of Latin America, can it? but, record stores? Nothing. Lets carry on with that idea: Radio stations are paid to program said music. This idea should not lead us to believe that recording companies are to blame for bad taste. I won't reveal my sources, but I know the success of the "Locura autom tica" song by La Secta was a real example. Nobody paid for it. That song got to the number one because of its own merits(?) (you don't know the effort it took to find that thing, I cannot recommend it to you). Same thing happens with other stations that don't program reggaet n, that try to save the species, and where they play what we do like. But the thing is, everything we like is not available in any record store in this country. Then, even if we wanted to buy a record or give it as a gift to somebody, it is plain impossible. And don't tell me it's the same to present as a gift a link or a CD full of downloaded MP3 as it is to give a record with cover and booklet, wrapped in gift paper. I might be old-school, but the fetish object record still exists, not only because of its cover, but because of its sound. A 3MB MP3 is akin to drinking coffee dripping from a bag that has been used eight times with the same coffee beans. That format is the worst that has ever happened to music, and if we had any bit dignity we would never purchase digital files in Amazon or iTunes safe for MP3 with an acceptable compression level. That, if we could buy them, because not only that is allowed to us. As the musical industry has no interest in resolving ITS problem (that is not our problem, it is those companies') it has not even been resolved how to charge for a MP3 download that includes import fees (well, if downloading from here a MP3 from a USA-based file server can be considered importing goods into Costa Rica!!!) so we don't have to get dizzy entering into the nineties to Titi Online to discover there is nothing by Muse, Andrew Bird, The Killers, Death Cab for Cutie, Paramore, Bj rk... (believe me, I looked them all up, even Norah jones and La Secta. They also were not there). This all leads me to the question, which I present with all due respect (NOT): What the fuck do they expect us to do??? It is outrageous; above all because in the best case they will sell us a watery coffee download that won't allow us to get all the details a vinyl or less compression would give us. In the worst case, post-MP3 groups will end up recording music with no harmonics or hidden sounds, because, what for? Nobody will hear it. They even admit it: "Some musicians and audio engineers say the MP3 format is changing the way studios mix their recordings. They say the MP3 format "flattens" dynamics differences in tone and volume in a song. As a result, a great deal of new music sounds very much alike, and there is nothing as focusing to create a dynamic listening experience. Why working so hard in creating complex sound if nobody can detect it?" (Rolling Stone, The Death of High fidelity, December 26 2007, taken from here). That's why I am not surprised by Adri n's post regarding the sales of old records. The price has nothing to do with it. The causes are related to the fetish object record and what it means or does not mean for people that have never purchased one. Adri n also asks if somebody here keeps buying records. I answered that I would if the stores sold anything I like. I do it even after the nausea I feel while reading "This phonogram is an intellectual work protected in favor of its producer COPYING IT IN WHOLE OR PARTIALLY IS FORBIDDEN " (like that, uppercased, yelling to whoever is only guilty of having bought a record and defending the producer, not the artist). But I am sure that almost nobody buys records because doing so is no longer a gratifying experience; because if buying a record is clicking to wait 15 days for it to reach the mailbox, we prefer clicking on the download link. But there is another reason for people not to buy records anymore. In one of my talks regarding the dictatorship of the all rights reserved, I asked the 30 twenty-something-year-old students if any of them had ever bought a record. One answered he had, because he is an author and performer (cantautor in Spanish) and understands the effort that producing a record entails. The rest of them had never done so. Is it possible that said young people have never listened to real music? Is it possible that, were it not for concerts, what they consider music is a set of washed-out MP3 that are about to fill up 1TB of their computer? Does people no longer buy records because they cannot differentiate one sound from the other? It is not very clear for me where do I want to get to. The recording industry is despicable. An industry that instead of innovating sets its energy on suing adolescents for downloading songs, trying to pass laws restricting our freedoms in Internet, putting up DRMs making us hostages to our devices* and forcing us to listen just the aroma of music, deserves my whole contempt. If we add ot this that said industry won't allow us to legally download their breadcrumbs because it has not understood that Internet does not need a van crossing borders, besides my contempt they deserve my pity and my heartfelt condolence. But the condolence is also for music, real music, that which is not compressed under the shoe using a terrible format. It is also for independent musicians that have not realized that begging for a bit of space to that industry they just add to themselves the "despicable" tag, given they deserve the fruits of their work to enter their bank account. However, there are good things that have come out of this absurdity. Be it for those that have joined projects such as Aut mata (even if it is in MP3) and for dreams come true such as Musopen (that have achieved that the music that's in theory Public Domain becomes so in practice as well). Good for the Electronic Frontier Foundation and the list of lawyers willing to defend people accused of ilegally downloading music in the USA. Good for the Creative Commons licenses that allow free sharing. All those are growing solutions, although none of them allows me to buy the Panamanian Carlos M ndez's record. Thankfully, a friend of mine who knows I will never give a dime to Apple, bought the files for me in iTunes. I thank him deeply, although I would have prefered to go to Auco Disco and have Mauricio tell me that the 2007 EP I have from Carlos is better than the record he did on 2009. * My devices don't have DRM because I use free software. I also use the ogg file format. Image by verbeeldingskr8

Continuing on from my first EFI progress report... My first alpha release of a Wheezy netinst CD for amd64 EFI worked OK for me and a few others, with reasonable feedback. But it was far from complete, with (at least) three outstanding issues:

• It didn't work as a hybrid (CD or USB) image, just working from CD for EFI
• The initial grub display was really ugly, with no graphics
• I hadn't been able to get grub-efi to work, so it only allowed use of elilo.

In the days since then, I've carried on hacking on things. I've not yet sorted out hybrid boot, but I've fixed the other issues. To make graphics work on the grub boot menu was a simple case of loading the right module from grub.cfg ("insmod png"). Getting grub-efi to work took quite a lot more effort, though. I've been looking through Ubuntu's version of debian-installer code and merged a fair amount of their code into my local builds, adding tweaks as necessary. Result? I've got another CD alpha release ready for testing. Yay! Grab the image from http://cdimage.debian.org/cdimage/unofficial/efi-development/upload2/ if you'd like to help test it. Please do, and let me know via debian-boot/debian-cd how you get on. As previously, the "bits" subdirectory contains all the tweaked d-i packages I've played with, in both source and binary form. My debian-cd changes are still in my branch but are just about ready for merging I think. I'll post more d-i patches to the debian-boot list for review shortly.