Just about three years ago, I created the
automatic rule-based time
tracker (arbtt), a tool that quietly runs in the background, records
your current windows and their titles, and allows you to later process
this data using, well, rules. By now, I have almost half a million
records, spanning 317 days of me working at the computer. Processing
this amount of data has become very memory-hungry: It takes 1G of RAM to
process my data. Other users, with a presumably higher sampling rate,
report even worse performance, which finally made me approach the problem again.
Naturally, one wonders why it is such a memory hog, after all it
takes just one run over the data to produce a report. The data itself is
a lazy list so the garbage collector should the be able to drop the
seen elements immediately.
But arbtt is able to print several reports in one run; there is even
an option (
--each-category) that creates a number of reports based on
the data itself. The code was structured that first a number of values
that are possibly used by several reports are first created one and then
re-used by the reports. And then there is the issue of the filters: The
user specifies a what predicates are interesting (e.g., those where he
was active) and some reports need to know not only which samples were
matched, but also distinguish the consecutive runs of selected samples.
Lots of reasons that made it hard to not use the list of of samples
several times, which causes it to be completely retained in memory. The
idea of completely rewriting everything in a much more imperative way
was not appealing. But luckily Haskell s good abstraction possibilities can help here.
What is the correct way to traverse a lazy list and calculate some information about it? It is a strict left fold, i.e. an algorithm that can be represented as arguments to
foldl' :: (s -> x -> s) -> s -> [x] -> s
For those who are new to folds: The first argument is a function that processes one element from the list, returning the new state of the processing. The second argument is the initial state. And
foldl' can apply such an algorithm to a list to return a final state.
So obviously I had to express my reports as left folds. To think clearer about it, I gave them a name. I also added a finishing step to the mix, which is handy:
data LeftFold x a = forall s. LeftFold
start :: s,
process :: s -> x -> s,
finish :: s -> a
A value of type
LeftFold x a is then an algorithm that traverses a list of
xes and returns something of type
a. The
forall s. means that the user of such a processor must not make any assumptions about the type of the state. I also have a function that runs such a list processor, simply by invoking
foldl' with the arguments stored in the
LeftFold constructor:
runLeftFold :: LeftFold x a -> [x] -> a
runLeftFold (LeftFold st1 p1 f1) xs = f1 (foldl' p1 st1 xs)
Now, as a Haskell programmer, I am searching for monads everywhere, and it really would have been nice if this were a monad, as they are very easy to compose. But unfortunately, they do not form a monad. If they did then I could decide after running one processor and looking at the result how I would want to process the list now, but this obviously contradicts with the desire to traverse the list only once.
But there is the monad s smaller ugly sibling, the applicative functor. One can still somewhat nicely compose them to form more complex algorithms, just with more restrictions. Here is my instance:
instance Functor (LeftFold x) where
fmap f (LeftFold st1 p1 f2) = LeftFold st1 p1 (f . f2)
instance Applicative (LeftFold x) where
pure x = LeftFold () const (const x)
LeftFold st1 p1 f1 <*> LeftFold st2 p2 f2 = LeftFold
start = st1 :!: st2,
process = \(s1 :!: s2) x -> p1 s1 x :!: p2 s2 x,
finish = \(s1 :!: s2) -> f1 s1 (f2 s2)
The
:!: operator is a
strict tuple constructor. This is important, as otherwise the fold would be come a lazy one, I would be building up huge numbers of thunks and performance would be very bad. For the same reason every list processor should make sure that its state does not build thunks once it is in weak head normal form.
Now I am at a abstraction level that I feel comfortable working in, by writing functions that combine the list processors in various ways. Probably the most important one for me is already present in the libraries (here with a specialized type):
sequenceA :: [LeftFold x a] -> LeftFold x [a]
This is the whole trick: Now I can, based on the user input, run any number and selection of reports in one single list traversal. I find it very elegant that under the hood , the strict tuples are nested at precisely the right depth depending on the length of the list.
As hinted at before, I do some interesting stuff that was previously represented as lists of lists of values. Expressing these as left folds was a bit tricky, but again, the complexity is contained in the combinators that I wrote (although it still gets confusing, check out
processIntervalReport in Stats.hs):
monoidFold :: Monoid m => LeftFold m m
mapElems :: LeftFold y a -> (x -> y) -> LeftFold x a
filterWith :: (x -> Bool) -> LeftFold (Bool :!: x) a -> LeftFold x a
onSelected :: LeftFold x a -> LeftFold (Bool :!: x) a
onJusts :: LeftFold x a -> LeftFold (Maybe x) a
onAll :: LeftFold x a -> LeftFold (Bool :!: x) a
runOnGroups :: (x -> x -> Bool) -> LeftFold x y -> LeftFold y z -> LeftFold x z
runOnIntervals :: LeftFold x y -> LeftFold y z -> LeftFold (Bool :!: x) z
lfLength :: LeftFold x Int
lfFirst :: LeftFold x (Maybe x)
lfLast :: LeftFold x (Maybe x)
toList :: LeftFold x [x]
concatFold :: LeftFold [x] [x]
As a result, the program now only requires 10MB and runs 18% faster.
If you wonder why, in the definition of
runLeftFold, I do not apply the finish function strictly: That is intentional; I am actually passing the result of a processor that calculates some often required data (e.g. total time recorded) as an argument to all the list processors. (Near the end of the main function in
stats-main.hs) This loop works fine as long as I am using these only in the
finish functions.
There is one downside to this approach: I cannot easily calculate stuff on demand
and share it between two reports. Previously, I could just bind this to a variable, pass it to all reports, only if at least one is interested it would be evaluated and if several are interested, the result would be shared. I do not have a good solution for this yet.
The code above is in the self-contained module
LeftFold.hs in the arbtt sources.
Several People have had this idea before, here is
one blog post by Max Rabkin.
Flattr this post
A few days ago, Samsung released the source code
of Synaps, an implementation of the
Amazon Web Service CloudWatch API for
OpenStack.
Being a developer on the Ceilometer
project, I've been curious to look on this project and how it could overlap
with Ceilometer or other projects like Heat.
What is CloudWatch?
CloudWatch is a monitoring system provided by Amazon on its Web Services
platform to monitor services. This allows you get notifications and trigger
an action on certain threshold.
For example, this can be used to scale your architecture by monitoring the
number of requests you get on it and its general load by starting new
servers.
Synaps
Synaps is written in around 7k lines of Python (with 28 % of which are
comments), reuses at least one common module of OpenStack
(openstack.common.cfg) and copy some modules from Nova. One thing that
strikes me, is that there seems to be only a few unit tests compared to most
OpenStack projects. Also, many parts of the code and documentation contains
text written in korean, which won't be very helpful for most people! :-)
It uses some external technologies, like Storm,
Cassandra to store its persistent data and
Pandas to do data analysis.
The API server provides an EC2 compatible API only: no OpenStack specific
API. This is probably not a bad thing for now, since I am not aware of any
work in this direction. The API access directly the Cassandra back-end for
read operation, but relies on RPC to do writes. This way, a set of daemon
handles the write using the Storm part of Synaps and do data aggregation.
The authentication only supports LDAP, but it should still be possible to
add a driver for Keystone.
A Java and a Python SDK are provided to record metrics into Synaps, but
there's not enough documentation for it to be useful.
Overlap with Heat
For now, there's not a lot of overlap with Heat, because Heat does not
implement completely the CloudWatch API. Heat actually still misses a lot of
the CloudWatch functions. But as soon as it will implement the CloudWatch
API completely, the overlap will be complete with Synaps in this regard.
One divergence point however, is that Heat uses RPC to access data from the
storage back-end via its engine (the central daemon), whereas Synaps
directly connects to Cassandra. Also, Heat relies on SQLAlchemy, like most
OpenStack projects needing a database.
Overlap with Ceilometer
One of the goal of Ceilometer is to provide data probes and pollsters for
all OpenStack components (Nova, Swift, Quantum ) whereas Synaps let the
OpenStack users to put any kind of metric inside it, and therefore doesn't
provide anything for now.
But the storage of metrics is the main common point between Synaps and
Ceilometer. Synaps chose only one technology, Cassandra, to store its
metrics, whereas Ceilometer took care of building an abstraction layer for
the storage engine. Ceilometer currently allows an operator to use SQL or
MongoDB, but Cassandra could likely be added.
Data metric consolidation is done by Synaps. This makes sense, since Synaps
don't need to have the full data history to trigger alarms. On the opposite,
Ceilometer needs to have a full history to allow things like billing, and
don't do any aggregation on data.
Also, in Synaps, the data analysis is done using Pandas. This means the data
used are retrieved from the Cassandra back-end, and then transformed by
Pandas inside Synaps in something else. It's likely that in such a case,
Synaps should use CQL to achieve that. Ceilometer manipulates the data near
their storage: it means that the computation are done by back-end to be
efficient (SQL, mapreduce ).
Conclusion
Considering Samsung open-sourced Synaps late in the development process, I
don't feel like they aimed to have it becoming a core component. This is
always sad, because the effort put into this implementation are big and it
would have probably little to add some abstraction layers to follow what
other OpenStack projects do. But this takes time and energy, and it's
understandable that Samsung didn't want to achieve this in a short time
frame.
There's a part of the code and architecture that overlaps with Ceilometer
and Heat. Ceilometer is becoming a specialized point to store data metrics
from any source: so it's sad, but understandable, that Synaps did not tried
to reuse it. Fortunately, Heat is working with Ceilometer to achieve exactly
that. This means OpenStack would have only one metrics storage point, used
for billing, for monitoring and alarming.
Therefore, I think Synaps is an implementation of CloudWatch that should be
looked at as an inspiration for Heat and Ceilometer to build a better and
more integrated solution!
Christopher 
Dear Facebook friend,
I like you and I am happy that you take/tag/share photos of me, just not
on Facebook!
I have a Facebook account but 
I am moving into an office. And as any proud husband and father,
I want to have photos of my wife and daughter on my desk. The year
being 2012, I did not hike to the photo shop to get a print framed,
but instead thought to myself that I should put a digital photo
frame onto my table.
This idea exposed me to the ridiculous world of consumer
electronics. It led me to conclude that digital picture frame
designers need to be whacked with cluebats.
Step by step
Once accepted, the thought of a digital photo frame developed
into a product definition along the following lines: my idea photo
frame would connect to my Wifi-network, and obtain the photos
on-the-go from a folder exposed via HTTP or CIFS, and then go on to
display them in random order, incorporating new photos as it
encounters them.
With this in mind, I went to the shops, and since I believe in
specialised retailers and want to support them, my first stop was
For writing tests for GVFS (
Phishing is a fraud technique that consists in reproducing the content of a
targeted website, directing users to it, and recording their credentials when
they log into your website. It is like fishing fish with fake food, only you
phish humans with fake websites.
Well, I have a good news for the phishers: people are using more and more
smartphones, and mobile browsers are making your job really easy.
In fact, many regular smartphone user simply have no way to tell your fake
website apart from the real one.For instance, this is what my bank website looks like in my desktop browser,
and what a corresponding phishing website would look like:
As many of you know, I love Ubuntu. I ve given years of my life for Ubuntu. I ve served as much as anyone s expected to, and long upheld the ideology and technical motivations behind Ubuntu as second to none.
Until recently.
Now, with advertisements in the default DE (and don t tell me they re not, when I search for term and get The Terminator on DVD, it s a ad if you care so much about whatever binding to search applications, why not disable ads by default, and let super-s do your shopping ), and now, begging for a handout when people Download Ubuntu without an option to donate to 
Last weekend I was in Southwest Harbour, Maine again for the annual LTSP hackfest (called
I was about to post a different story but things turned out differently, for good. Now I ll post tips that may save the day in the event of encoding issues when dealing with MySQL table data. Hopefully you find it useful.
Say you have a MySQL table that has collation set to latin1 but you have utf8-encoded data stored. This happened when importing a dump, or because of Murphy s law. When querying the data from your application or from mysql s tools you get awful results for non-ascii characters such tildes. Changing the table s collation definition will only work for new columns and records. Altering the collation for the said column would make things worse. So what to do?
Fortunately MySQL offers a nice way to save the day. You can use a transition conversion and then set the table encoding back to utf8, so it pairs accordingly with the encoding type for the data it has stored.
Direct harassment and overt sexism are certainly a significant part of why the gender ratios in the Linux community[1] are so skewed, but they're not the whole story. Part of feeling comfortable with a community is the knowledge that those in positions of power can be relied upon to engage in appropriate action when undesirable situations arise - no matter how compelling your code of conduct, no matter how detailed your diversity policy, if someone contravenes them and nothing happens, those documents are worthless and your community is unwelcoming. You can't rely on enforcement unless the community can trust its leadership.
This is my monthly summary of my Debian related activities. If you re among the people who 
. To "tag"
your donation as originating from the Debian community, just leave
a message containing the word "debian" at the end of the donation
process.
The work Michael will be doing tomorrow is not entirely
Debian-specific, but Michael is Canonical's responsible for
relationship with upstream. I've worked with him in the past on a
range of topics, and I can testify that part of his work is
generally useful to Debian.
It appears that a 
Thoughts about DRI.Next
On the way to the X Developer s Conference in Nuremberg, Eric and I
chatted about how the DRI2 extension wasn t really doing what we
wanted. We came up with some fairly rough ideas and even held an
I was invited as a panelist for the