Apologies in advance for writing a long blog post about the dull and
specialised subject of
double-entry accounting
from the Unix tools perspective, that ends up talking about Monads to boot.
I can't believe I'm about to write such a thing, but I'm finding it an oddly
interesting subject.
double-entry accounting
I've known of, though probably not really understood double entry
accounting for a while, thanks to
GnuCash. I think
GnuCash did something rather special in making such a subject approachable
to the layman, and I've been happy to recommend GnuCash to friends. I was
stoked to find a chapter in my sister Anna's
new book
that happily and plausibly suggests readers use GnuCash.
But for my personal use, GnuCash is clunky. I mean, I'm a programmer,
but I can't bring any of my abilities to bear on it, without a deep dive
into the code. The file format is opaque (and a real pain to keep checked
into git with all the log files); the user interface is often confusing,
but there's no benefit to its learning curve, it never seems to get better
than manually entering data into its ledger, or perhaps importing data from
a bank. I've never found the reports very useful.
I've got perhaps a year of data in GnuCash, but it's fragmented and
incomplete and not something I've been able to motivate myself to keep up
with. So I have less financial data than I'd like. I'm hoping ledger will
change that.
ledger
I've known about ledger for a while, at least since
This Linux Weekly News article.
It's a quintessential Unix tool, that simply processes text files.
The genius of it is the simplicity of the file format, that gets the essence
and full power of double entry bookeeping down to something that approaches a
meme. Once you get the file format stuck in your head, you're done for.
2004/05/27 Book Store
Expenses:Books $20.00
Liabilities:Visa
starting to use hledger
Now as a Haskell guy, I was immediately drawn to
the Haskell clone,
hledger. It's nice that there
are two (mostly) compatable implementations of ledger too. So from here on,
I'll be talking about hledger.
I sat down and built a hledger setup for myself the other day.
I started by converting the GnuCash books I
have been keeping up-to-date,
for a small side business (a rental property). It quickly turned into
something like programming, in the best way, as I used features like:
- Include directives, so I can keep my business data in its own file,
while pulling it into my main one.
- Simple refactorings, like putting "Y 2012" at the top, so I don't have
to write the year in each transaction.
- Account aliases, so I can just type "rent", rather than
"income:rental" and "repairs:contractor" rather than
"expenses:home repair:contractor"
- All the power of my favorite editor.
a modern unix program
While I've been diving into hledger, I've been drawing all kinds of parallels
between it and other modern Unix-friendly programs I use lately. I think
we've gone over a bit of a watershed recently. Unix tools used to be either
very simple and crude (though often quite useful), or really baroque and
complex with far too many options (often due to 10-30 years of evolution).
Or they were a graphical user interface, like GnuCash, and completely
divorced from Unix traditions.
The new unix programs have some commonalities...
- They're a single command, with subcommands. This keeps
the complexity of doing any one thing down, and provides many
opportunities for traditional unix tools philosophy, without
locking the entire program into being a single-purpose unix tool.
hledger's subcommands range from querying and reports, to pipable
print, to a interactive interface.
- They have a simple but powerful idea at their core, that can be
expressed with a phrase like "double entry accounting is a simple
file format" (ledger), or "files, trees, commits" (git).
Building a tool on a central idea is something I strive
to do myself. So the way ledger manages it is particularly interesting to
me.
- They are not afraid to target users who have a rather special kind
of literacy (probably the literacy you need to have gotten to here
in this post). And reward them with a lot of power.
Ledger avoids a lot of the often confusing terminology around accounting,
and assumes a mind primed with the Unix tools philosophy.
- If there's a GUI, it's probably web based. There's little trust in
traditional toolkits having staying power, and the web is where the
momentum is. The GUI is not the main focus, but does offer special
features of its own.
hledger's web UI completely parallels what I've doing with the git-annex
webapp, right down to being implemented using Yesod -- which really needs
to be improved to use some methods I've developed to make it easier to make
webapps that integrate with the desktop and are more secure, if there are
going to be a lot more programs like this using it.
importing data
After manually converting my GnuCash data, I imported all my PayPal history
into hledger. And happily it calculates the same balance Paypal does. It
also tells me I've paid PayPal $180 in transaction fees over the years,
which is something PayPal certianly doesn't tell you on their website.
(However, my current import from PayPal's CSV files is a hackish, and only
handles USD currency, so I miss some currency conversion fees.)
I also imported my Amazon Payments history, which includes all the
Kickstarter transactions. I almost got this to balance, but hledger and
Amazon disagree about how many hundreths of a cent remain in my account.
Still, pretty good, and I know how much I paid Amazon in fees for my
kickstarter, and how much was kicked back to Kickstarter as well.
(Look for a cost breakdown in some future blog post.)
At this point,
hledger stats says I have 3700 transactions on file,
which is not bad for what was just a few hours work yesterday.
One problem is hledger's gotten a little slow with this many transactions. It
takes 5 seconds to get a balance. The
ledger program, written in C, is
reportedly much faster. hledger recently had a
O(n^2) slowdown fixed,
which makes me think it's probably only starting to be optimised. With
Haskell code, you can get lucky and get near C (language, not speed of
light) performace without doing much, or less lucky and get not much better
than python performance until you dive into optimising. So there's hope.
harnessing haskell
If there's one place hledger misses out on being a state of the art modern
Unix program, it's in the rules files that are used to drive CSV
imports. I found these really hard to use; the manual notes that "rules
file parse errors are not the greatest"; and it's just really inflexible.
I think the state of the art would be to use a Domain Specific Language here.
For both my Amazon and PayPal imports I had CVS data something like:
date, description, amount, fees, gross
I want to take the feeds into account, and make a split transaction, like this:
date description
assets:accounts:PayPal $9.90
expenses:transaction fees:PayPal $0.10
income:misc:PayPal -$10.00
This does not seem possible with the rules file. I also wanted to combine
multiple CVS lines, to do with currency conversions, into a single
transaction, and couldn't.
The problem is that the rules file is an ad-hoc format, not a fully
programmable one. If instead, hledger's rules files were compiled into
standalone haskell programs that performed the import, arbitrarily
complicated conversions like the above could be done.
So, I'm thinking about something like this for a DSL.. I doubt I'll get
much further than this, since I have a hacked together multi-pass importer
that meets my basic needs. Still, this would be nice, and being able to
think about adding thing kind of thing to a program cleanly is one of the
reasons I reach for the Haskell version when possible these days.
First, here's part of one of my two paypal import rules files
(the other one extracts the transaction fees):
amount-field 7
date-field 0
description-field %(3) - %(4) - %(11)
base-account assets:accounts:PayPal
Bank Account
assets:accounts:checking
.*
expenses:misc:PayPal
That fills out the basic fields, and makes things with "Bank Account"
in their description be categorised as bank transfers.
Here's how it'd look as Haskell, carefully written to avoid the
$
operator that's more than a little confusing in this context. :)
main :: IO ()
main = convert paypalConveter
paypalConverter :: [CSVLine] -> [Maybe Transaction]
paypalConverter = map convert
where
convert = do
setAmount =<< field 7
setDate =<< field 0
setDescription =<< combine " - " [field 3, field 4, field 11]
defaultAccounts
"assets:accounts:PayPal" ==> "expenses:misc:PayPal"
inDescription "Bank Account" ?
"assets:accounts:PayPal" ==> "assets:accounts:checking"
That seems like something non-haskell people could get their heads around,
especially if they didn't need to write the boilerplate function
definitions and types at the top. But I may be biased. :)
Internally, this seems to be using a combination Reader and Writer monad
that can get at fields from a CSV line and build up a Transaction.
But I really just made up a simple DSL as I went along and thew in
enough syntax to make it seem practical to implement. :)
Of course, a Haskell programmer can skip the monads entirely, or use others
they prefer. And could do arbitrarily complicated stuff during imports,
including building split transactions, and combining together multiple
related CVS lines into a single transaction.
I've recently replaced my laptop's HDD as the S.M.A.R.T. data
suggested a potential failure. I took the opportunity to
try out beta4 of 
My parents will soon be moving to a smaller home, so they are digging up a lot of stuff of my sister and mine past. Among my stuff, they just sent me my first computer (which was, of course, the first computer of my father s company I was using in the spare time). I couldn t believe they kept that. It was an 
Is it just me or does this picture made you nostalgic also?
Update Dec 2012: ResearchGate still keeps on sending me their spam.
Most of the colleagues I had that tried out RG now deleted their
account there, apparently, so the invitation mails become fewer.
Please do not try to push this link on Wikipedia just because
you are also annoyed by their emails. My blog is not a "reliable source" by
Wikipedia standards. It solely reflects my personal view of that web
site, not journalistic or scientific research.
The reason why I call ResearchGate spam is the weasel words they use
to trick authors into sending the invitation spam. Here's the
text coming with the checkbox you need to uncheck (from the
I was a bit busy (mostly in pleasant ways) during the last two weeks, so
this reports covers a longer period of my RC bug activities:
A minor feature release 
The following proposal extends the JSON specification, with the idea
of using JSON as an information interchange format, rather than just a
way of writing certain ECMAscript values. They do not add anything but
only restrict valid JSON content and encoders with some rationale.
First of, I d like to remind everyone, including JSON s author, that
JSON is case-sensitive, except in the four hexdigits
after a backslash-u sequence in a String.
Second, I d like to remind everyone that JSON is not binary-safe. No
way around that, it implements Unicode (actually, 16-bit UCS-2, and it
doesn t guarantee that UTF-16 surrogates are correctly paired) text. I
also consider only UTF- 8, 16,32 B,L E valid encodings for JSON. (No
PDP endian, either. Sorry, guys.)
For my first proposal, I d like to point out CVE-2011-4815 which was
about overflowing hashtables. The obvious fix is to randomise the hash
per hashtable; to ensure this doesn t leak, we sort ASCIIbetically the
keys of an Object in the encoder. (Using Unicode is good here we can
just sort the keys as UTF-8 strings by their uint8_t value or
as Unicode (UCS-2 or even UCS-4 or UTF-16) strings by the codepoints.)
JSON was never preserving the order of elements in an Object anyway so
we make it standardised (we still accept any order, and, when parsing,
in collision cases, the later value wins). This also helps diffs.
For my second proposal, I d like to forbid \u0000, \uFFFE, \uFFFF in
strings. The first because many implementations use C strings, and for
an information interchange format this is better; it also has security
implications to allow NUL in a string. The other two, but not unpaired
UTF-16 surrogates (as ECMAscript uses UCS-2 and got UTF-16 only later)
because they re not valid Unicode; JSON was not binary-safe already so
why bother. Among other benefits, this also helps implementations.
For my third proposal, I d like to agree that implementations should
impose a nesting depth limit that may be user-defined, and in the face
of which, cyclic checking may be ignored by an encoder. I emit nesting
depth overflows as literalnull; might also throw an error. Since I was
asked, the common standard value is to restrict nesting depth is 32,
unless the user specified one. (I also saw 8, but 32 WFM pretty well.)
Most seem to use it even if it may seem low at first. Only specialised
applications probably need more, and they can always pass a value.
For my forth proposal, backslash-escape U+007F U+009F always. It may
upset humans, editors, databases, etc. (This paragraph is newly added,
after some 
Thanks very much to everybody who commented, both publicly and privately, on my previous entry.
To recap: I have three sites that were each generated by slightly different templating software I'd built and tweaked over the years. I was frustrated that the three copies of the generation tool had all drifted and diverged from each other, and was looking to setup a new (static) site.
The obvious conclusion was either:
This is my monthly summary of my free software related activities. If you re among the people who 
Back from the mini-debconf, I have been trying to 
I'm pleased to say that a usable version of shim is now available for 
For many years, I have aspired being a university teacher. I remember asking at different universities as early as ten years ago But I didn't have the needed papers. And yes, I have been a "Licenciado en Ingenier a en Software" for one year already.
Anyway, I won't go into the details on why I didn't do this earlier on. But this time, I did get my act together in time, and by mid October, I contacted Juan Carre n, an enthusiastic teacher I met a long time ago as he helped a lot for the formation and cohesion of the (now defunct?) LIDSOL group (Laboratorio de Investigaci n en Software Libre, Free Software Research Laboratory), a group of very worthy students, mostly from the Engineering careers.
Juan Carre n had long offered me help in getting to the right people in Facultad de Ingenier a as soon as I had my formal requirements ready. I just didn't expect it to be so swift! Within two days of my initial mail, he contacted me back asking me to look at 
Have you ever wondered one of those?
Next release of 
We have a PostgreSQL server with 16 cores that was apparently running well
below its capacity: load something between 3.0 and 4.0, around 200 active
database connections, almost all always being <IDLE>. However, when the
tps count reached 7k transactions per second, things started to throttle, and
pgbouncer (running on the database server) started listing up to half of the
client connections to be in cl_waiting state. Load was still low, but
application performance was bad.
The culprit turned out to be the kernel scheduler, fairly distributing CPU time
among all running processes. There's one single pgbouncer process, but hundreds
of postgres processes.
A simple renice of the pgbouncer process did the trick and gave us another
extra 2k tps.
After seeing some book descriptions recently on planet debian,
let me add some short recommendation, too.
Almost everyone has heard about Gulliver's Travels already,
so usually only very cursory. For example: did you know the book
describes 4 journeys and not only the travel to Lilliput?
Given how influential the book has been, that is even more suprising.
Words like "endian" or "yahoo" originate from it.
My favorite is the third travel, though, especially the acadamy of Lagado,
from which I want to share two gems:
"
His lordship added, 'That he would not, by any further particulars, prevent the pleasure I
should certainly take in viewing the grand academy, whither he was resolved I should go.' He
only desired me to observe a ruined building, upon the side of a mountain about three miles
distant, of which he gave me this account: 'That he had a very convenient mill within half a
mile of his house, turned by a current from a large river, and sufficient for his own family, as
well as a great number of his tenants; that about seven years ago, a club of those projectors
came to him with proposals to destroy this mill, and build another on the side of that mountain,
on the long ridge whereof a long canal must be cut, for a repository of water, to be conveyed up
by pipes and engines to supply the mill, because the wind and air upon a height agitated the
water, and thereby made it fitter for motion, and because the water, descending down a
declivity, would turn the mill with half the current of a river whose course is more upon a
level.' He said, 'that being then not very well with the court, and pressed by many of his
friends, he complied with the proposal; and after employing a hundred men for two years, the
work miscarried, the projectors went off, laying the blame entirely upon him, railing at him
ever since, and putting others upon the same experiment, with equal assurance of success, as
well as equal disappointment.'
"
"I went into another room, where the walls and ceiling were all hung
round with cobwebs, except a narrow passage for the artist to go in and out.
At my entrance, he called aloud to me, 'not to disturb his webs.' He
lamented 'the fatal mistake the world had been so long in, of using
silkworms, while we had such plenty of domestic insects who infinitely excelled
the former, because they understood how to weave, as well as spin.'
And he proposed further, 'that by employing spiders, the charge of dyeing
silks should be wholly saved;' whereof I was fully convinced, when he
showed me a vast number of flies most beautifully coloured, wherewith he fed
his spiders, assuring us 'that the webs would take a tincture from them; and as
he had them of all hues, he hoped to fit everybody s fancy, as soon as he
could find proper food for the flies, of certain gums, oils, and other
glutinous matter, to give a strength and consistence to the threads.'"