Compleat is an easy, declarative way to add smart tab-completion for
any command-line program. For a quick description, see the
README. For
more explanation and a brief tutorial, keep reading...
Background
I'm one of those programmers who loves to
carefully tailor my development
environment. I do nearly all of my work at the shell or in a text editor,
and I've spent a dozen years learning and customizing them to work more
quickly and easily.
Most experienced shell users know about programmable completion, which provides
smart tab-completion for for supported programs like ssh and git. (If you are
not familiar it, you
really should install and enable bash-completion, or
the equivalent package for your chosen shell.) You can also add your own
completions for programs that aren't supported but in my experience,
most users never bother.
When I worked at Amazon, everyone used Zsh (which has a very powerful but
especially baroque completion system) and shared the completion scripts they
wrote for our myriad internal tools. Now that I'm in a startup with few other
command line die-hards, I'm on my own when it comes to extending my shell.
So I read the fine manual and started writing my own completions. Over on
GitHub you can see the
script I made for three commands from the Google
Android SDK. It's 200 lines of shell code, fairly straightforward if you
happen to be familiar with the Bash completion API. But as I cranked out more
and more case statements, I felt there must be a better way...
The Idea
It's not hard to describe the usage of a typical command-line program.
There's even a semi-standard format for it, used in man pages and generated by
libraries like
GNU AutoOpt. Here's one for
android, one of the SDK
commands supported by my script:
android [--silent --verbose]
( list [avd target]
create avd ( --target <target> --name <name> --skin <name>
--path <file> --sdcard <file> --force ) ...
move avd (--name <avd> --rename <new> --path <file>) ...
(delete update) avd --name <avd>
create project ( (--package --name --activity --path) <val>
--target <target> ) ...
update project ((--name --path) <val> --target <target>) ...
update adb )
My idea: What if you could teach the shell to complete a program's arguments
just by writing a usage description like this one?
The Solution
With
Compleat, you can add completion for any command just by writing a
usage description and saving it in a configuration folder. The ten-line
description of the
android command above generates the same results as my
76-line bash function, and it's
so much easier to write and understand!
The syntax should be familiar to long-time Unix users. Optional arguments are
enclosed in square brackets; alternate choices are separated by vertical
pipes. An ellipsis following an item means it may be repeated, and
parentheses group several items into one. Words in angle brackets are
parameters for the user to fill in.
Let's look at some more features of the usage format. For programs with
complicated arguments, it can be useful to break them down further. You can
place alternate usages on their own lines separated by semicolons, like this:
android <opts> list [avd target];
android <opts> move avd (--name <avd> --rename <new> --path <file>)...;
android <opts> (delete update) avd --name <avd>;
...and so on. Rather than repeat the common options on every line, I used a
parameter
<opts>. I can define that parameter using the same usage syntax.
opts = [ --silent --verbose ];
For parameters whose values are not fixed but can be computed by another
program, we use a
! symbol followed by a shell command to generate
completions, like this:
avd = ! android list avd grep 'Name:' cut -f2 -d: ;
target = ! android list target grep '^id:' cut -f2 -d' ' ;
And any parameter without a definition will just use the shell's built-in
completion rules, which suggest matching filenames by default.
The
README file has more details of the usage syntax, and instructions
for installing the software. Give it a try, and please send in any usage
files that you want to share! (Questions, bug reports, or patches are also
welcome.)
Future Work
For the next release of Compleat, I would like to make installation easier by
providing better packaging and pre-compiled binaries; support
zsh and other
non-bash shells; and write better documentation.
In the long term, I'm thinking about replacing the usage file interpreter with
a compiler. The compiler would translate the usage file into shell code, or
perhaps another language like C or Haskell. This would potentially improve
performance (although speed isn't an issue right now on my development box),
and would also make it easy for usage files to include logic written in the
target language.
Final Thoughts
Recently I realized that parts of my work are so specialized that my parents
and non-programmer friends will probably never really get them. For example,
Compleat is a program to generate programs to help you... run programs? Sigh.
Well, maybe
someone out there will appreciate it.
Compleat was my weekends/evenings/bus-rides project for the last few weeks (as
you can see in the
GitHub punch card), and my most fun side-project in
quite a while. It's the first "real" program I've written in Haskell, though
I've been experimenting with the language for a while. Now that I'm
comfortable with it, I find that Haskell's particular combination of features
works just right to enable quick exploratory programming, while giving a high
level of confidence in the behavior of the resulting program. Compleat 1.0 is
only 160 lines of Haskell, excluding comments and imports. Every module was
completely rewritten at least once as I tried and compared different
approaches. This is much less daunting when the code in question is only a
couple dozen lines. I don't think this particular program would have been
quite as easy to write at least for me in any of the other
platforms I know (including Ruby, Python, Scheme, and C).
I had the idea for Compleat more than a year ago, but at the time I did not
know how to implement it easily. I quickly realized that what I wanted to
write was a specialized parser generator, and a domain-specific language to go
with it. Unfortunately I never took a compiler-design class in school, and
had forgotten most of what I learned in my programming languages course. So I
began studying parsing algorithms and language implementation, with Compleat
as my ultimate goal.
My good friend Josh and his
Gazelle parser generator helped inspire me
and point me toward other existing work. Compleat actually contains three
parsers. The usage file parser and the input line tokenizer are built on the
excellent
Parsec library. The usage file is then translated into a
parser that's built with my own simple set of parser combinators, which were
inspired both by Parsec and by the original
Monadic Parser Combinators
paper by Graham Hutton and Erik Meijer. The simple evaluator for the usage
DSL applies what I learned from Jonathan Tang's
Write Yourself a Scheme in
48 Hours. And of course
Real World Haskell was an essential
resource for both the nuts and bolts and the design philosophy of Haskell.
So besides producing a tool that will be useful to me and hopefully others, I
also filled in a gap in my CS education, learned some great new languages and
tools, and kindled an interest in several new (to me) research areas. It has
also renewed my belief in the importance of "academic" knowledge to real
engineering problems. (I've already come across at least one problem in my
day job that I was able to solve faster by implementing a simple parser than I
would have a year ago by fumbling with regexes.) And I'll be even happier if
this inspires some friends or strangers to take a closer look at Haskell,
Parsec, or any problem they've thought about and didn't know enough to solve.
Yet.
Just got back from Tokyo a few hours ago: I had an opportunity to give my
'Introduction to High-Performance Computing with R' tutorial / lectures
(for which previous slides can be found here).
This was an all-day talk at the
Institute for Statistical Mathematics
at their new site in Tachikawa in the greater Tokyo area, thanks to an
invitation by Junji Nakano.
Lisa and I turned this into a brief one-week trip to Kyoto and Tokyo, and we
had a truly wonderful time on what was our first visit to Japan. I should blog some
more about it, but now I will give in to the jet lag and catch up on some
sleep...
In fact, this is more useful than one can imagine, because a
considerable amount of time in periodic RC squashing is taken by
triaging the RC bug list to choose your targets. Of course there is
no guarantee that I'll be willing/able to fix handed-over bugs but,
as a RM, Luk has evidently well-trained feelings about how hard
would be to fix a given bug. All in all, for this week it worked
well: 4 of the above 7 came from 
(Context: 
The 
Every now and then
I have a demoralising experience with the quality of Debian packages in sid.
That's to be expected, sid is a development distribution, but it still
sometimes gets on my nerves. I've decided to write about some of my
experiences.
Some rules: I may not rant without
Earlier in the week I was interested in doing some very simple video editing.
A bit of searching and I come across the name of a potentially useful program:
'pitivi'. As luck would have it, there's a Debian package! However, 
Yesterday was the last day of presidential campaign, the most poisonous campaign since 1989, when the communist rule has fallen ... or so says history.
Starting 01/01/2008, Bavaria had introduced a quite hard smoking ban, which
also included bars and restaurants. It however contained a backdoor by
excluding non-public locations, which led to the creation of 'smoker clubs'
where you had to become a member to be admitted. At some point, most clubs
were of this kind.In August 2009, however, the law was changed to exclude beer tents
(Oktoberfest ...) and small bars. Many people belive that this was to get
votes on the elections in september 2009 (which ended up in a minus of 6-7%
compared to the previous election and a historical low for the biggest party).This caused several organizations to call for a public vote on restoring the
smoking ban to the 2008 state (without the 'smokers club' backdoor). In order
to force a public vote on a law (without the governments support!), we need
10% of the voters to register as supporters for the vote. You have to register
at your registered home town. For Bavaria, this means about 940.000 supporters.If you are registered voter in Bavaria, please drop by your municipality
and sign up. You need an ID and 5 Minutes, that's all. 940.000 supporters
is an incredible lot of people to get to the offices, take along your friends!When we get enough supporters, the Bavarian government has two options:
accepting the changes as proposed (and thus making the initative obsolete),
or conducting a public vote on it, offering an alternative (e.g. the current
law, no change) and have the voters decide (which is quite expensive, so if
many many people sign up, they might save that money and just pass the
proposed change themselves).For more information (german only), check the
I was referred to 
I have been reading a great many posts about Android lately, some consisting of criticism, some of praise and some simply addressing issues in the Android community . Let s have a look at those.
Matt Porter s Android Mythbusters presentation and Harald Welte s reaction
3.0 format source packages can now be uploaded to the Debian archive. (Note: despite the title of this post, this is not the "Wig and Pen" format which is 2.0. See dpkg-source(1) for a detailed description of all the source package formats.) I tried converting my libcgi-application-basic-plugin-bundle-perl package with mostly good results.
l-a-b-p-b-p is an aggregation of perl modules (in .tar.gz format) from 
