Search Results: "reg"

6 September 2013

Sylvain Le Gall: OUnit 2.0 progress, August 2013

After a long pause, I have resumed my work on OUnit2. It is going quite well. The basic features: I still need to do the following: There is still a lot of work, but the current results are already quite good. The speed improvement of the processes runner is a good thing to shorten test timing (HINT: tester needed!). Focus on: the new bracket. In OUnit 1, a bracket was very functional:
 bracket 
    (fun () -> "foo")  (* setup *)
    (fun foo -> ())
    (fun foo -> ())  (* tear down *)
So for common bracket, like bracket_tmpfile
   bracket_tmpfile
       (fun (fn, chn) ->
             (* Do something with chn and fn *)
The problem is that if you were using 2 or 3 temporary files, the level of indentation was high. I have decide to switch to a more imperative approach, registering the tear down function inside the test context:
   let (fn1, chn1) = bracket_tmpfile ctxt in
   let (fn2, chn2) = bracket_tmpfile ctxt in
      ....
This is shorter and clearer (albeit less functional). Focus on: non fatal section Sometimes, you want to verify a set of properties but to have a clear vision of what is going wrong, you need to do more than one assert. In OUnit1, you can do:
 assert_equal exp1 v1;
 assert_equal exp2 v2
But if exp1 <> v1, you quit immediately and you'll never know for exp2 and v2. In OUnit2, you can do:
 non_fatal ctxt (fun ctxt -> assert_equal exp1 v1);
 non_fatal ctxt (fun ctxt -> assert_equal exp2 v2)
In this new version, you will test both equality and the result of the test will be the worst failure you get (or success if both of them succeed). Help wanted If you have a long standing issue with OUnit, this is the time to submit a bug. OUnit BTS If you want to try the dev version of OUnit:
 $> darcs get http://forge.ocamlcore.org/anonscm/darcs/ounit/ounit
Patches always welcome. Special thanks to Thomas Wickham who has entirely written OUnitRunnerThreads and kickstarted the processes runner.

Daniel Pocock: Latest NSA revelations no surprise, but...

The latest NSA revelations confirm the systematic manner in which the NSA exploits known and potentially unknown backdoors to work around cryptography. This is not surprising, I speculated briefly about the OpenBSD and Windows NSA key incidents when Snowden originally came into view. Even two days ago, a detailed piece in Wired Magazine predicted many of the things that are in today's headlines., although Schneier indicates in The Guardian he had seen some of the material before publication. What we really need to know If the NSA and CIA have had these capabilities, just how much did they really know before and during the financial crisis that kicked off in 2007? Could the crisis have been averted? It was already known that the Greek parliament was wired, was the NSA involved in similar shenanigans? This could well be part of the Snowden revelations that we are yet to see. In fact, it is this last issue that could be the most damaging for "US interests". If countries (or their citizens) receive such undeniable evidence of a CIA conspiracy exposed by Snowden in broad daylight and they refuse to pay debts then it could lead to major re-alignment of international politics, alliances and treaties. Syria would be just the tip of the iceberg.

Marko Lalic: PTS User Accounts

PTS User AccountsDuring the last iteration, the PTS rewrite has grown user accounts! This is a major addition, which moves the GSoC project beyond a simple "rewrite".

As always, this feature is available to be tested at the running PTS demo instance found at http://pts.debian.net.

For the moment the features related to user accounts allow for Web-oriented package subscription management. This means that the user can achieve the same functionality as what is offered by the control commands accessible over the control mail interface (control@pts.debian.net).

This includes subscribing to a package directly from the package page by clicking a single button; no confirmation mail required. Users are able to see a list of all their subscriptions and associated keywords, modify their default keywords and subscription-specific keywords all through a single profile page.

Eventually, a user is going to be able to associate multiple email addresses with his account and manage subscriptions for each of them from the same central panel. Actually, all currently existing functionality has already been implemented with this in mind and is backed by extensive Selenium-driven functional tests. This story is found in the backlog and is to be worked on in the last week of GSoC.

Another interesting, albeit Debian-specific, feature that is in the pipeline is allowing Debian Developers to authenticate via sso.debian.org: no PTS-specific registration/passwords required!

Besides package subscription management, introducing user accounts to the PTS opens up interesting possibilities for future development. Perhaps things like personalized package pages, a private stream of notifications in the user's profile... Hopefully, there are members of the community interested in exploring some of them.

PTS Rename?As the scope of the project begins to encompass more and more previously nonexisting functionality and in doing so even starts engulfing some other existing services (an example is that the PTS can now automatically extract and serve some source package files, e.g. debian/copyright)., an idea has been floating around to rename it. The question becomes: what would a good name be? No decision has been made as of yet, so suggestions and opinions are welcome.

5 September 2013

Gergely Nagy: What's cooking in syslog-ng 3.5?

The first alpha of syslog-ng 3.4 was released more than a year ago, and the first stable release earlier this year, development on syslog-ng 3.5 has been going on steadlily since then, with a lot of interesting features and changes pouring in. While the 3.5 release will not bring as many overwhelming changes as the previous two releases did, the sheer number of improvements is still substantial. Our focus was on smaller developments all over the place this time, instead of big sweeping changes. In this post, we will go over most of these features, with use-cases, examples and a few words about why they are so useful and awesome.Multi-line support, new destinations, new template functions, type hinting, unit suffixes, and other things. Read on, and be amazed!These are only bigger features, there has been a lot of small tweaking going on, and of course, all of the fixes that went into prior versions will also be in 3.5 too, and there's still a chance that something else may find its way in before the feature freeze.Multi-line supportOne of the major new features in the 3.5 release will be support for multi-line messages, a feature that has been available in syslog-ng PE for a good while, and which has been ported to and improved upon to the open source edition. Two variants of multi-line are supported, which will be detailed below. Both of them are available for the file() and pipe() sources only.Indented multi-lineThe easiest variant is indented multi-line, where each line can be followed by others, indented by whitespace, and the message continues until the first non-indented line. This is the format used by the Linux kernel too, from version 3.5, for /dev/log. This type of multi-line can be used as follows:
source s_multiline   file("/path/to/file" multi-line-mode(indented));  ;
Consider that /path/to/file has the following content:
First line
 Continuation 1;
 Continuation 2;
Second line
With the indented multi-line-mode() setting, this would turn into two log messages:
First line\n Continuation 1;\n Continuation 2;
Second line
Regexp-based multilineIf multi-line input is not based on indentation, one can use the regexp multi-line-mode() instead, which makes two new settings available: multi-line-prefix() and multi-line-garbage(). These can be used to define the start and the end of a log message: any string between a the beginning matching prefix and a matching garbage will be considered a single message. That is, the prefix will be included, the garbage will not be: it will be discarded.To illustrate:
source s_multiline  
 file("/path/to/file" multi-line-mode(regexp)
                      multi-line-prefix("^prefix")
                      multi-line-garbage(" garbage$"));
 ;
If the source contains these lines:
prefix message
continuing garbage
This will turn into the following log message:
prefix message\ncontinuing
New destinationsIt would be pretty hard to do a syslog-ng release without new destinations, and with 3.5, we will have three of them!STOMPWhile we had AMQP before, we now have a STOMP destination too, so we can stream logs to any STOMP server, with a few simple lines:
stomp(
 host("localhost")
 port(61613)
 destination("/topic/syslog")
 routing_key("")
 persistent(yes)
 ack(no)
 username("someone")
 password("something")
 body("This is the body! YAY! Here's a message: $MESSAGE!\n")
 value-pairs(scope(nv-pairs, syslog-proto, selected-macros))
)
Of course, none of these settings need to be set, you can just use the defaults, and it will just work!RiemannDid you know that syslog-ng is far more than a log collector and processor? No? You do now. When you have access to a lot of logs, and a tremendous amount of power in parsing them, you can use these tools for monitoring easily! And when we're talking monitoring, riemann is a great asset in our toolbox, and with the new destination, we can easily forward metrics to it:
riemann(
 server("localhost")
 port(5555)
 host("$HOST")
 service("$PROGRAM")
 description("$PROGRAM pids")
 metric(int("$PID"))
 ttl(300)
);
Of course, the above is a very silly example, it would make much more sense to extract data from a log message instead, but this will serve as an example.New template functions
$(upper-case TEXT...), $(lower-case TEXT...)
To turn some text into *upper-* or *lower-case*, these two template functions come in handy. They simply turn all their arguments into the respecitve case.
$(delimit DELIMITERS NEW-DELIMITER TEXT)
Sometimes one has a delimited string, where one wishes to replace the delimiters. This function does just that: give it a list of delimiters (you may need to quote it, if it contains whitespace), a replacement, and a text to replace delimiters in, and it does the rest. For example, to replace tabs and spaces with a vertical bar, one could write a template like this:
template("$(delimit \"\t \" \" \" $MESSAGE)\n")
$(env VARIABLE...)
Have you ever felt the need to check an environment variable from within syslog-ng? I did, and now I can.
Type hintingWhile syslog-ng supported sending log messages into various data stores and message queues for a while (SQL at first, MongoDB, JSON and AMQP later), even when those supported different types of data than strings, we could not do anything else. Until now. It is now possible - at places where it makes sense - to annotate templates with type hints, which the destination driver can optionally use. Type hinting is implemented for the mongodb() destination and the $(format-json) template function for now. When no type hint is specified, syslog-ng defaults to string.To add type hints, simply wrap the respective template with the hinted type, like this:
mongodb(
 value-pairs(pair("date", datetime("$UNIXDATE"))
             pair("pid", int64("$PID")))
);
Currently the following type hints exist: boolean (anything that begins with a t or 1 is true, anything that begins with f or 0 is false, everything else is an error), string, literal (same as string, but not quoted if it would be quoted otherwise), int32 (int is an alias for this), int64, and datetime. Only UNIX timestamps can be type-hinted to datetime, anything else will likely result in a casting error.It is also possible to control what happens when type casting fails: syslog-ng can drop the whole message, drop the property, or fall back to string. It can also do all of these silently:
options  
 typecast(on-error(silently-drop-property));
 ;
Using this feature with $(format-json) is very similary too:
$(format-json date=datetime("$UNIXDATE") pid=int64("$PID"))
With this feature in place, you can now store your non-string values with their proper types!Unit suffixesUnit suffixes make it considerably easier to set limits and describe numbers within the syslog-ng configuration. We no longer need to spell out sizes to the byte precious, it is now enough to write: log-fifo-size(200MiB). Now, syslog-ng will understand suffixes for kilo-, mega-, and giga-bytes, (K, M, G, respectively) either in base-10 or base-2 (with an extra i after the suffix). One can also omit the trailing b from the end.So, to set the log-fifo-size() to 2097152 bytes, one can simply use 2MiB. Or, to set it to 2000000, 2Mb. That's a whole lot easier, isn't it? No more counting zeros, no more silly typos in a ten-digit number, no more pain, but easily readable units!Miscellaneous featuresApart from the features above, there have been a lot of other changes and improvements in the code base:

Tanguy Ortolo: PluXml, a file-based, database-free blog

The blog engine PluXml is now available as a Debian package:
Package: pluxml
Version: 5.2-2
Description-en: Light blog engine using XML files
 PluXml is a light and simple blog and CMS engine that uses simple XML
 files to store its data and requires no database. It has all the usual
 features (static pages, comments, categories, tags, medias, RSS
 feed...) and supports multiple users, customizable themes and plugins.
Homepage: http://www.pluxml.org/
Section: web
Priority: extra
PluXml logo: a leaf with its petiole used as one bar of the word  Xml
PluXml is a French blog engine that uses XML files to store its data, thus avoiding the need for an external database system as most blogs do. The name PluXml is a French portmanteau between plume (feather, evoking something light) and XML. Apart from that, although it was started in French only and its official website is still in French, it was internationalized about a year ago. It is designed to be easy enough to use, targeting regular users, but not too fancy: for instance, it provides a web-based administration interface, but content editing uses HTML code unless you install a graphical content editor plugin. That package may be of interest to people like me, that consider the almost systematic use of relational database systems in many web applications as a disease, and more generally to self-hosted people that would rather avoid running an additional database server on their small home servers. The package includes an interactive configuration, that allows to choose the blog language, title, description, admin login and password. Unless debconf was configured not to ask questions, in which case nothing is asked and the blog is left in an barely unusable state, of course.

Keith Packard: Airfest-altimeter-testing

Altimeter Testing at Airfest Bdale and I, along with AJ Towns and Mike Beattie, spent last weekend in Argonia, Kansas, flying rockets with our Kloudbusters friends at Airfest 19. We had a great time! AJ and Mike both arrived a week early at Bdale s to build L3 project airframes, and both flew successful cert flights at Airfest! Airfest was an opportunity for us to test fly prototypes of new flight electronics Bdale and I have spent the last few weeks developing, and I thought I d take a few minutes today to write some notes about what we built and flew. TeleMega We ve been working on TeleMega for quite a while. It s a huge step up in complexity from our original TeleMetrum, as it has a raft of external sensors and six pyro circuits. Bdale flew TeleMega in his new fiberglass 4 airframe on a Loki 75mm blue M demo motor. GPS tracking was excellent; you can see here that GPS altitude tracked the barometric sensor timing exactly: GPS lost lock when the motor lit, but about 3 seconds after motor burnout, it re-acquired the satellite signals and was reporting usable altitude data right away. The GPS reported altitude was higher than the baro sensor, but that can be explained by our approximation of an atmospheric model used to convert pressure into altitude. The rest of the flight was also nominal; TeleMega deployed drogue and main chutes just fine. TeleMetrum We ve redesigned TeleMetrum. The new version uses better sensors (MS5607 baro sensor, MMA6555 accelerometer) and a higher power radio (CC1120 40mW). The board is the same size, all the connectors are in the same places so it s a drop-in replacement, and it s still got two pyro channels and USB for configuration, data download and battery charging. I loaded up my Candy-Cane airframe with a small 5 grain 38mm CTI classic: The flight computer worked perfectly, but GPS reception was not as good as we d like to see: Given how well TeleMega was receiving GPS signals, I m hopeful that we ll be able to tweak TeleMetrum to improve performance. TeleMini We ve also redesigned TeleMini. It s still a two-channel flight computer with logging and telemetry, but we ve replaced the baro sensor with the MS5607, added on-board flash for increased logging space and added on-board screw terminals for an external battery and power switch. You can still use one of our 3.7V batteries, but you can also use another battery providing from 3.7 to 15V. I was hoping to finish up the firmware and fly it, but I ran out of time before the launch. The good news is that all of the components of the board have been tested and work correctly, and the firmware is feature complete , meaning we ve gotten all of the features coded, it s just not quite working yet. EasyMini EasyMini is a new product for us. It s essentially the same as a TeleMini, but without a radio. Two channels, baro-only, with logging. Like TeleMini, it includes an on-board USB connector and can use either one of our 3.7V batteries, or an external battery from 3.7V to 15V. EasyMini and TeleMini are the same size, and have holes in the same places, so you can swap between them easily. I flew EasyMini in my Koala airframe with a 29mm 3 grain CTI blue-streak motor. EasyMini successfully deployed the main chute and logged flight data: We also sent a couple of boards home with Kevin Trojanowski and Greg Rothman for them to play with. TeleGPS TeleGPS is a GPS tracker, incorporating a u-blox Max receiver and a 70cm transmitter. It can send position information via APRS or our usual digital telemetry formats. I was also hoping to have the TeleGPS firmware working, and I spent a couple of nights in the motel coding, but didn t manage to finish up. So, no data from this board either. Production Plans Given the success of the latest TeleMega prototype, we re hoping to have it into production first. We ll do some more RF testing on the bench with the boards to make sure it meets our standards before sending it out for the first production run. The goal is to have TeleMega ready to sell by the end of October. TeleMetrum clearly needs work on the layout to improve GPS RF performance. With the testing equipment that Bdale is in the midst of re-acquiring, it should be possible to finish this up fairly soon. However, the flight firmware looks great, so we re hoping to get these done in time to sell by the end of November. TeleMini is looking great from a hardware perspective, but the firmware needs work. Once the firmware is running, we ll need to make enough test flights to shake out any remaining issues before moving forward with it. EasyMini is also looking finished; I ve got a stack of prototypes and will be getting people to fly them at my local launch in another couple of weeks. The plan here is to build a small batch by hand and get them into the store once we re finished testing, using those to gauge interest before we pay for a larger production run.

4 September 2013

Russell Coker: Ownership of a Club

Last night at the Annual General Meeting we had a motion to disincorporate The Linux Users of Victoria (LUV) [1]. The proposal was for LUV to cease being an incorporated society on condition that Linux Australia (LA) [2] accepts us as a sub-committee. As a sub-committee of LA we would elect our own committee to run things locally but have LA hold the finances, deal with all the paperwork that the government demands, and generally do as many of the non-core tasks associated with running a users group as possible. When we discussed this at the LUV committee meetings it didn t seem like a big deal. But as is often the case with political discussions it turned out to be difficult. There was a lot of discussion about LUV supposedly ceasing to exist, people seem to think that LUV is defined by having an incorporated society. My impression was always that it was defined by a mailing list and having meetings and I was involved in both before there was an incorporated society. Lurkers and Ownership During the discussion we had some input from members who were typically lurkers who seemed to feel that their property rights towards LUV were being infringed, this annoys me. I think that if someone chooses not to be involved in running an organisation then they should choose not to concern themselves with the details of how the organisation is to be run. People who attend the meetings should have a say in how the meetings are run and have reason to be concerned about anything that might affect them and the opinions of speakers also matter. People who are involved with mailing list discussions should have a say in how the lists are run. But people who have never volunteered for a position on the committee shouldn t be greatly concerned about the internal issues of how things are run. Finance Some concern was expressed about the financial situation of LUV and whether we would still get enough donations to keep it running when combined with LA. There was even some FUD suggesting that LA would just take our money (they had assured us that all funds and donations would be ear-marked for us). The current LUV financial situation is that Red Hat pays for the venue for the monthly meetings and the rent for the venue comprises about 2/3 of all donations. The remaining 1/3 comes from one company. So in the current situation if Red Hat ceased donating then we would have 18 months to find another donor or cease holding meetings before our bank balance became unreasonably low. If the company which gives the other significant annual donation was to cease doing so then we could operate for a few years on savings but we would need to find some other source of funding. It seems to me that joining LA would give us more financial security. Then if Red Hat ceased paying for the venue then LA could keep things running until we found another donor, I m confident that LA wouldn t allow LUV to just shut down because of a shortage of donations. If people are really concerned about the financial situation of LUV then they should urgently seek further donations such that if any one donor decided to stop giving then we could still operate as normal. To achieve that goal I think we need at least another $1,000 per annum. This issue of redundancy in donations is something I raise every time that LUV finances are discussed. My conclusion is that people aren t really bothered about the financial security of LUV except when they are looking for reasons to avoid change. Doing New Things During the course of discussion about the future of LUV there were a number of requests for improvement. One significant request was for more support for regional Linux users. Some years ago we held a mini-conference in Ballarat which went well. I think it would be good to do such things again, the cost is not particularly great and I m sure it would be accepted by LA for funding, but we need to organise it. Organising such events is something that anyone can do. Any LUV member can plan an event, get costs for everything that is needed (food, accommodation, travel, etc) and then pitch it to the LUV committee in terms of which things should be paid by LUV and which by the members concerned. We could then work on getting additional funding from LA if necessary. But planning an event takes some effort and it s often effort that can only be done by a local. Finding a suitable venue and getting some assurance that a large enough audience will attend is something that can t be done remotely. I think that the problem for LUV in regard to such things isn t a lack of money or independence. I think that the problem is that the committee spends too much volunteer time on administrative tasks and not enough time directly doing things that benefit members and the community in general. In the past I have declined nomination to the LUV committee because I felt that I could contribute more by giving lectures, finding other speakers, and doing other things to directly improve the group. I was on the committee last year and have now been elected to it again, but I m starting to think that I made a mistake. Maybe I should have declined and let others work on the new model rules and other paperwork. One committee member has claimed that the time taken on administrative tasks isn t taking time away from other LUV related tasks, I invite any committee members who feel that way to address some of the services that members are requesting. Speaking for myself my lack of time directly impacts that I can do for the club. I think that ownership of a club should be related to what people do for the club. If you have a feeling of ownership and lack ideas for how to contribute then you can ask the LUV mailing list, there are lots of people with suggestions for things to do.

2 September 2013

Daniel Pocock: Australia's Labor Party exposed spying on their own members

Australia's Labor Party claims to represent the little guy, advocating the interests of a wide slice of the population over the interests of those who might supposedly exploit them. In this latest blog I'll look at just how the Victorian state branch of the party is emulating the more sinister elements of monitoring and control that you would typically expect from marketers, spammers and the NSA. It makes you wonder, if they do this to their own members, how can Australia's government ever hope to rein in spammers or other undesirable practices? Europe has tried cookie laws and the right to be forgotten, while the ALP has taken these developments almost as a guide to exploiting their own most valuable resource, their membership. The email version of invisible ink Straight out of an email attempting to recruit party members to the federal election campaign, we find that the ALP's logo itself has been turned into a sinister tracking mechanism. Looking underneath the logo, there is a hidden tracking number attached to every message:
http://gallery.mailchimp.com/3021e68df9a7200135725c633/images/Email_Header_ThisisLabor.jpg
There it is, that big long code 3021e68df9a7200135725c633. On it's own, you may think this number can't do much harm. Think again. After all, as the filename says, "ThisisLabor". Reaching the grass roots by long distance email When a party member sees any of the images in the email or clicks any of the links, their email software or web browser sends a HTTP request to the ALP's outsourced tracking server in the United States. Here is an example of what that request may contain:
TCP Headers:
  SOURCE IP ADDRESS: 198.51.100.56
HTTP headers:
  GET /3021e68df9a7200135725c633/images/Email_Header_ThisisLabor.jpg
  HTTP/1.1
  Host: gallery.mailchimp.com
  User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/20130515 Firefox/17.0 Iceweasel/17.0.6
  Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
  Accept-Language: en-US,en;q=0.5
  Accept-Encoding: gzip, deflate
  X-Forwarded-For: 192.168.1.54
  Connection: keep-alive
These codes are all transmitted from the recipient of the email back to the ALP. Notice the tracking number is in the very first line of the HTTP request? These codes send many details to the tracking server. Just what is revealed?
  • What time the message was read
  • How many times the message was opened
  • Whether the message was read on multiple computers (perhaps it was forwarded)
  • The IP addresses reveal which network was used.
    • For people reading email at work or on wifi, the IP address reveals the name of the employer, it can be easily looked up in this public database. Here is an example querying one of the ALPs own IP addresses, 203.24.72.2
    • For people using an internet connection at home, the IP address reveals the service provider. There is more to it though: if one party member regularly visits another party member and uses their wifi, they will both appear to be using the same IP address and the party will be able to deduce that there is a connection between these two people.
  • The web browser name and version and the operating system name and version.
  • The language normally used on the computer
Crunching the numbers You may think that nobody would be interested in all that data or that it is even too much data for anybody to easily study. Think again. The ALP has outsourced processing of this data to an offshore firm that specializes in bulk email, MailChimp. After lecturing everybody about the dangers of outsourcing, the ALP has literally sold the farm (their own membership data) - to a foreign marketing company. Here are some of the claims MailChimp make on their web site:
"Dig even deeper with Subscriber Activity Reports, which give you a real time report for every subscriber s email activities: when they opened, what they clicked, and when they came back for more. You can even generate a list of people who didn t open, and send them a modified campaign."
That's right: if ALP members don't read their emails fast enough, they will send even more of them. Just who has access to this data? Well, the scope is quite broad. According to these rules, it is not only the dubious officials in the party's state office, but Labor Party regulations also list a whole lot of other officials who have access to some subset of the data: branch and FEA district committees, policy committees and maybe some trade union officials too. Once you start to contemplate this, it becomes apparent that the probability of somebody profiling members at an individual level is very real. On the other hand, those rules clearly do not permit the state branch to process the member's data offshore. Double whammy In fact, even MailChimp is further outsourcing the data. When receiving an image from the booby-trapped link in the email, we receive a response like this:
HTTP/1.1 200 OK
Content-Type: image/jpeg
Content-Length: 407474
Connection: keep-alive
Date: Wed, 10 Jul 2013 06:25:55 GMT
Last-Modified: Tue,  03 Sep 2013 04:05:37 GMT
ETag: "df9a7200135725c6331369a22"
Accept-Ranges: bytes
Server: AmazonS3
Age: 62097
Via: 1.0 e71fe855f29513f54a41e213fe653fb1.cloudfront.net (CloudFront)
X-Cache: Hit from cloudfront
X-Amz-Cf-Id: S2V2aW4gUnVkZCBpcyBhIGJhZCBib3kuCg==
which contains more tracking numbers and clearly indicates that the data is being processed in the massive AmazonS3 (a.k.a Amazon online bookstore) cloud platform. Thanks to cloud computing, there is considerable scope for cross-referencing records across other web sites. The IP address becomes a magic key, allowing a range of sinister forces to join the dots at the click of a button and profile the whole private and professional identity of any individual ALP member. Googling ALP members It doesn't stop there however, after following these links a victim will eventually end up on one of the ALP's own web pages. Looking through the HTML code underneath http://www.viclabor.com.au is very easy, in most web browsers, you can just press Control-U to see what is hidden away in the shadows. Buried in the code, I found this little gem:
  var _gaq = _gaq   [];
  _gaq.push(['_setAccount', 'UA-35325645-1']);
  _gaq.push(['_trackPageview']);
  (function()  
    var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true;
    ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js';
    var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s);
   )();
This may be even scarier: it's a JavaScript code that channels all access to the ALP web site into an outsourced tracking system provided by Google. Google can now cross-reference ALP members against their Gmail accounts, their friends and the topics they search for to further profile the type of advertising they should be exposed to. Google, like any private company, has a loyalty to its shareholders and the laws of the country where it is domiciled. Will that coincide with the best interests of ALP members in Victoria? Australia's other parties: samples needed Do you have similar communications from Australia's other political parties? Please send them to me and I'll be happy to examine them for similar shortcomings.

Steve Kemp: So that forum?

So that forum I mentioned? I've setup a test-installation at: What does this forum offer? A cross between hacker news and reddit. If the admin of the forums enables it you can create arbitrary tags, and then view them. For example: It's also very fast, and reasonably easy to customize. Which is good, because the current layout is nasty. Things I like: Not much more to say really; except I've setup a test install and if you wish to login/register and post spam feel free.

B lint R czey: Faketime gets nanosecond timestamps, speeds up games, testing

I was playing with faketime the last few days mainly to implement features needed by ReproducibleBuilds, which is an initiative in Debian for providing binary packages that can be regenerated with the exact same content. Some build steps place timestamps in the resulting binaries, thus we may have to make time perceived by the build process move at a deterministic rate starting from a predetermined point in time.
This is why faketime got support for advancing time with each time(), gettimeofday(), etc. call. Another approach would be recording timestamps perceived by the first build and replaying them in the same order to successive builds. If the build is deterministic apart from the timestamps, this should result in the same binary package for each build.
While playing with faketime I could not resist implementing a few things which may not have been absolutely important for reproducible builds, but were so much fun. The nanosecond resolution of timestamps made games playable at slower or faster speeds making very hard games easier or easy games harder. Jump n'bump just became insanely funny at 200% speed:

jumpnbump-video-screen

faketime -f "+0 x2" jumpnbump Speeding up sleep()-s can also be useful in daily work. If your application calls sleep() often it may significantly slow down testing, but faketime is now able to shorten sleeps, too, speeding up testing such applications!

1 September 2013

Eddy Petri&#537;or: Integrating Beyond Compare with Semanticmerge

Note: This post will probably not be on the liking of those who think free software is always preferable to closed source software, so if you are such a person, please take this article as an invitation to implement better open source alternatives that can realistically compete with the closed source applications I am mentioning here. I am not going to mention here where the open source alternatives are not up to the same level as the commercial tools, I'll leave that for the readers or for another article.



Semanticmerge is a merge tool that attempts to do the right thing when it comes to merging source code. It is language aware and it currently supports Java and C#. Just today the creators of the software have started working on the support for C.

Recently they added Debian packages, so I installed it on my system. For open source development Codice Software, the creators of Semanticmerge, offers free licenses, so I decided to ask for one today, and, although is Sunday, I received an answer and I will get my license on Monday.

When a method is moved from one place to another and changed in a conflicting way in two parallel development lines, Semanticmerge can isolate the offending method and can pass all its incarnations (base, source and destination or, if you prefer, base, mine and theirs) to a text based merge tool to allow the developer to decide how to resolve the merge. On Linux, the Semanticmerge samples are using kdiff3 as the text-based merge tool, which is nice, but I don't use kdiff3, I use Meld, another open source visual tool for merges and comparisons.


OTOH, Beyond Compare is a merge and compare tool made by Scooter Software which provides a very good text based 3-way merge with a 3 sources + 1 result pane, and can compare both files and directories. Two of its killer features is having the ability split differences into important and non-important ones according to the syntax of the compared/merged files, and the ability to easily change or add to the syntax rules in a very user-friendly way. This allows to easily ignore changes in comments, but also basic refactoring such as variable renaming, or other trivial code-wide changes, which allows the developer to focus on the important changes/differences during merges or code reviews.

Syntax support for usual file formats like C, Java, shell, Perl etc. is built in (but can be modified, which is a good thing) and new file types with their syntaxes can be added via the GUI from scratch or based on existing rules.

I evaluated Beyond Compare at my workplace and we decided it would be a good investment to purchases licenses for it for the people in our department.


Having these two software separate is good, but having them integrated with each other would be even better. So I decided I would try to see how it can be done. I installed Beyond compare on my system, too and looked through the examples


The first thing I discovered is that the main assumption of Semanticmerge developers was that the application would be called via the SCM when merges are to be done, so passing lots of parameters would not be problem. I realised that when I saw how one of the samples' starting script invoked semantic merge:
semanticmergetool -s=$sm_dir/src.java -b=$sm_dir/base.java -d=$sm_dir/dst.java -r=/tmp/semanticmergetoolresult.java -edt="kdiff3 \"#sourcefile\" \"#destinationfile\"" -emt="kdiff3 \"#basefile\" \"#sourcefile\" \"#destinationfile\" --L1 \"#basesymbolic\" --L2 \"#sourcesymbolic\" --L3 \"#destinationsymbolic\" -o \"#output\"" -e2mt="kdiff3 \"#sourcefile\" \"#destinationfile\" -o \"#output\""
Can you see the problem? It seems Semanticmerge has no persistent knowledge of the user preferences with regards to the text-based merge tool and exports the issue to the SCM, at the price of overcomplicating the command line. I already mentioned this issue in my license request mail and added the issue and my fix suggestion in their voting system of features to be implemented.

The upside was that by comparing the command line for kdiff3 invocations, the kdiff3 documentation and, by comparison, the Beyond Compare SCM integration information, I could deduce what is the command line necessary for Semanticmerge to use Beyond Compare as an external merge and diff tool.

The -edt, -emt and -e2mt options are the ones which specify how the external diff tool, external 3-way merge tool and external 2-way merge tool is to be called. Once I understood that, I split the problem in its obvious parts, each invocation had to be mapped, from kdiff3 options to beyond compare options, adding the occasional bell and whistle, if possible.

The parts to figure out, ordered by compexity, were:

  1. -edt="kdiff3 \"#sourcefile\" \"#destinationfile\"

  2. -e2mt="kdiff3 \"#sourcefile\" \"#destinationfile\" -o \"#output\""

  3. -emt="kdiff3 \"#basefile\" \"#sourcefile\" \"#destinationfile\" --L1 \"#basesymbolic\" --L2 \"#sourcesymbolic\" --L3 \"#destinationsymbolic\" -o \"#output\""

Semantic merge integrates with kdiff3 in diff mode via the -edt option. This was easy to map to Beyond Compare, I just replaced kdiff3 with bcompare:
-edt="bcompare \"#sourcefile\" \"#destinationfile\""
Integration for 2-way merges was also quite easy, the mapping to Beyond Compare was:
-e2mt="bcompare \"#sourcefile\" \"#destinationfile\" -savetarget=\"#output\""
For the 3-way merge I was a little confused because the Beyond Compare documentation and options were inconsistent between Windows and Linux. On Windows, for some of the SCMs, the options that set the titles for the panes are '/title1', '/title2', '/title3' and '/title4' (way too descriptive for my taste /sarcasm), but for some others are '/lefttitle', '/centertitle', '/righttitle', '/outputtitle', while on Linux the options are the more explicit kind, but with a '-' instead of a '/'.

The basic things were easy, ordering the parameters in the 'source, destination, base, output' instead of kdiff3's 'base, source, destination, -o ouptut', so I wanted to add the bells and whistles, since it really makes more sense for the developer to see something like 'Destination: [method] readOptions' instead of '/tmp/tmp4327687242.tmp', and because that's exactly what is necessary for Semanticmerge when merging methods, since on conflicts the various versions of the functions are placed in temporary files which don't mean anything.

So, after some digging into the examples from Beyond Compare and kdiff3 documentation, I ended up with:
-emt="bcompare \"#sourcefile\" \"#destinationfile\" \"#basefile\" \"#output\" -lefttitle='#sourcesymbolic' -righttitle='#destinationsymbolic' -centertitle='#basesymbolic' -outputtitle='merge result'"

Sadly, I wasn't able to identify the symbolic name for the output, so I added the hard coded 'merge result'. If Codice people would like to help with with this information (or if it exists), I would be more than willing to do update the information and do the necessary changes.

Then I added the bells and whistles for the -edt and -e2mt options, so I ended up with an even more complicated command line. The end result was this monstrosity:
semanticmergetool -s=$sm_dir/src.java -b=$sm_dir/base.java -d=$sm_dir/dst.java -r=/tmp/semanticmergetoolresult.java -edt="bcompare \"#sourcefile\" \"#destinationfile\" -lefttitle='#sourcesymbolic' -righttitle='#destinationsymbolic'" -emt="bcompare \"#sourcefile\" \"#destinationfile\" \"#basefile\" \"#output\" -lefttitle='#sourcesymbolic' -righttitle='#destinationsymbolic' -centertitle='#basesymbolic' -outputtitle='merge result'" -e2mt="bcompare \"#sourcefile\" \"#destinationfile\" -savetarget=\"#output\" -lefttitle='#sourcesymbolic' -righttitle='#destinationsymbolic'"
So when I 3-way merge a function I get something like this (sorry for high resolution, lower resolutions don't do justice to the tools):



I don't expect this post to remain relevant for too much time, because after sending my feedback to Codice, they were open to my suggestion to have persistent settings for the external tool integration, so, in the future, the command line could probably be as simple as:
semanticmergetool -s=$sm_dir/src.java -b=$sm_dir/base.java -d=$sm_dir/dst.java -r=/tmp/semanticmergetoolresult.java
And the integration could be done via the GUI, while the command line can become a way to override the defaults.

Gregor Herrmann: RC bugs 2013/35

& another week of perl (5.18) related bugs has gone by:

Russ Allbery: Review: Redshirts

Review: Redshirts, by John Scalzi
Publisher: Tor
Copyright: June 2012
ISBN: 1-4299-6360-3
Format: Kindle
Pages: 317
The next few reviews deserve a small apology in advance. Normally, I try to write reviews within a few weeks, or at most a month, after reading a book, while it's still fresh in my mind. This is particularly important since I read a lot and sometimes don't have a good memory for specific detail. But this summer has made it completely impossible to keep to any sort of regular writing schedule. As a result, this book, and the subjects of the next few reviews that follow, are books that I read four months ago and am reviewing from memory. Hopefully, I'll still get the details right. Ensign Andrew Dahl is one of the newest crew members of the Universal Union Capital Ship Intrepid. It's his first posting, just out of the Academy. He's one of five new ensigns, expecting a normal tour of duty (albeit with the pride of being on a capital ship of the fleet). But very shortly after arriving on the ship, Andrew and and the other new ensigns discover that something is very... strange. The casualty rate on the ship is surprisingly high (as foreshadowed in a prologue). The other crew members avoid the bridge crew, and particularly away missions, with startling effectiveness (and desperation). And things regularly happen on the ship that, for lack of a better description, simply make no sense. Redshirts starts off as a very obvious Star Trek parody. So obvious, in fact, that it's eyeroll-inducing. The prologue reads exactly like a section of a Star Trek episode written by a third-rate hack, leavened with some of the humor that readers of Scalzi's blog have come to expect. But it's obvious that Scalzi is doing this intentionally, and it doesn't take long before the subject of the book shifts. It's a Star Trek parody that functions much more like a Star Trek parody than any universe should, to the degree that it becomes obvious to the characters. And the characters, despite being straight from central casting, are clearly not expecting their universe to act that way and find this extremely strange. By far the best part of this book is the second quarter: the characters have been introduced, they're starting to figure out what's going on, and they have to do so while dodging away-team duty (or trying to survive it) and participating in the ridiculous plot contrivances that keep occurring. Scalzi mixes a feeling of serious consequences with wry humor and handles the mix deftly. I was both laughing and eagerly reading to see what comes next. The last half of the book, though, has problems. How many problems the reader perceives will, I think, depend on how hard one thinks about the situation Scalzi has set up, or how much one has thought about similar constructs in the past. But I'm afraid the setup does not survive serious scrutiny in ways that, for me at least, seriously damaged my ability to engage with the story. I'm going to avoid saying explicitly where Scalzi goes with this plot, since it is a spoiler (although it's commonly mentioned in discussions of this book on the Internet). But it happens to be an idea that some friends of mine and I pounded on for years in some of our role-playing scenarios, poking at the corners and implications, and subsequently gave up on because it doesn't work for storytelling. The idea is compelling, and on first glance seems like a great story seed. But it's too large. The problem with really big ideas with huge implications is that they can change the nature of a story so much that there's no longer any point of commonality for the reader and the entire nature of story falls apart. And I think that's what happens here. Scalzi handles the deeper implications of the concept by, essentially, ignoring them. He focuses on one narrow set of implications for one specific set of people, largely skips over any discussion of anything larger, and tries a pivot from wry humor (and a bit of slapstick) into serious emotional depth. For me, in part because I'm so familiar with the concept, none of this really worked. The implications of the discovery that drives this story are so much broader and so much more significant than the characters seem to admit that it threw me right out of the story. The ethical implications are absolutely paralyzing, to the point that I don't think you can write a coherent story about them, but ignoring them doesn't work either. It's the kind of discovery that requires action of some kind beyond the attempt by the characters to go back to what they consider a normal life, even if I have a hard time imagining what that action could be. To be fair, he does try to salvage this in the first epilogue, which is all about the implications of his idea and its effects. But it's still too small. That epilogue just barely touches on the implications on a single person, but if one thinks a bit longer, one starts coming up with more and more implications for all of society, for how we talk about story, for how we raise children, for nearly every form of creativity. Implications that involve nasty concepts like slavery and mind control and murder and the nature of consent in ways that are very difficult to avoid. I think Scalzi's attempts at serious emotional connection, including in the other two attached epilogues (which are perspective-shifting close focuses on some of the characters), also suffer from his world setup, but in a different way. The characters are simply too bland to sustain the level of emotional connection he's striving for. They're bland for a very specific reason that is entirely justified by the story that Scalzi is handling, and the implications of that for the primary protagonist were, I thought, one of the best-handled moments of the book. But they're still bland, which undermines the attempted pivot. All that being said, this is not a bad book. Parts of it, particularly the second quarter, are excellent. And even when I wasn't engaging with the characters, Scalzi's writing on a paragraph level is snappy and often quite funny. The first epilogue in particular is hilarious, even if I think it was too limited in the scope of implications that it considers and a bit too meta in where it goes with those implications. And the story is a fun romp, with drama and humor and tension and a fairly satisfying ending. Everyone's a bit too bland and earnest when Scalzi isn't being funny, but he's funny enough that I didn't mind that much. Despite the relatively low rating, I actually recommend this book. If you don't dive into the metaphysical implications, it's a lot of fun, and if you've thought a lot about this particular set of metaphysical implications, it's somewhat satisfying to argue with (and to internally point out all of the hard questions that Scalzi is skipping over). It's particularly fun if you've watched a lot of Star Trek, since Scalzi gets some of the absurdities note-perfect. But be aware that the character reactions to Scalzi's concept don't hold up to much scrutiny. Rating: 7 out of 10

30 August 2013

Joey Hess: best DebConf ever

For me, DebConf13 in Switzerland was a perfect DebConf, and generally an great developer meeting and conference. I've attended several past DebConfs in Canada, Norway, Brazil, Finland, Mexico, Scotland, Spain, NYC, Bosnia, and Nicaragua. I have special memories of each. DebConf13 was ideal in every way. Let's start with the camping. Only a minority of us did it, and my campsite was located in a small field behind a bar that was often noisy late into the night, when I was trying to rest and preserve my European jetlag remedy of inverted sleeping schedule. Still, I love sleeping cocooned in a tent, and going through the routines of camping and spending time alone there was a good counterpoint to DebConf. I generally find myself retreating for a day or two in the middle of a week-long conference, burnt out with social interactions. That didn't happen this time. I need to find way to camp at future DebConfs. (Also special shout out to wonderful people who lent me a warmer sleeping bag, and bought me a pillow while on a trip to town! Upgraded my first backpacking in Europe experience's comfort level a lot!) The next perfect thing for me were the talks I gave at DebConf. I always enjoy presenting at DebConf, and barely remember when I'd get nervous doing it. But I really felt comfortable giving these talks in a way I've not before. I'm pleased with how the Debian Cosmology talk turned out; I'd worried it would be a downer talk, or too light, or not well received, and none of those problems surfaced. I was especially happy with my dh_busfactor session, which I prepared for all of 10 minutes and was structured entirely to benefit me by talking over my thoughts and getting feedback, and in which I probably drove the camerapeople crazy running my own mic into the audience. Despite all that, I think it succeeded being interesting for the participants as well as for me. Hope to follow up on it soon.. But beyond the two talks I submitted, there were a series of ad-hoc scheduled talks. It seemed that every day I spent at least an hour discussing something I'm deeply involved in. This started with the Ikiwiki bof, and then there was an epic series of (unrecorded) BoFs about git stuff, which basically turned into an unofficial git track at DebConf. Much, much discussion of git-annex and other things I'm involved in. BTW, I really like that DebConf, as well as being a formal conference on top, is really an un-conference underneath, with new BoFs and things popping up on the schedule all week long. You can even have completely unscheduled meetings that somehow get just the right mix of people to make something awesome happen.
But it wasn't all about talks. Indeed, I only attended 2 or 3 sessions a day. The DebConf13 venue was controversial, but I found it excellent for getting things done. Rather than one or two massive rooms that get too noisy to think in, this venue had many little nooks and crannies suited for different types of work. There were so many I only discovered some while walking down the hill as I was leaving the venue! Anchoring it all was the excellent view, and I particularly enjoyed the "scenic hacklab". I was particularly impressed with the flexability of the venue's management. Someone asked if we could have cheese for breakfast, and every breakfast thereafter had not one but often three excellent cheeses. One day we'd be complaining that there were not enough vegetables in the food, and the next we'd be eating an excellent vegetarian Swiss lunch. They even apparently relaxed stringent Swiss rules on not allowing tents in sightlines visible from people further up the hill. They were clearly happy to welcome our crazy gathering to Le Camp. They let a DD borrow and use a chain saw and build a bonfire! The venue also encouraged getting out and exploring the surrounding countryside. Too often I end up spending the whole DebConf without seeing much of the area around. This time I at least got down the hill to see the castle in Vaumarcus a few times, and enjoyed many evening walks amoung the grape fields. The conference dinner (on a boat!) tied into this well too. And the daytrip to CERN was a great addition. We got to poke around the control center for a massive particle accellerator! My only regret is we didn't get a chance to actually go inside the LHC tunnels. Back to DebConf, another thing I really liked about this one was that its location allowed several upstream authors of software in Debian (other than the usual suspects who are both upstreams and Debian Developers) to attend. I don't know if Lennart Poettering changed any minds about systemd with his talk, but I was happy for the opportunity to buy him a beer. I didn't get to Monty's talk, but it was excellent to see him attend as well. Finally, I got to meet old friends I'd not seen in years, and new ones I've not met in real life before. We're still not sure where this whole DebConf thing is going -- will it be all about the cheese and wine party and greybeards in 20 years time? But one thing we know is we enjoy our little slice of the year where kilts and coding are equally unremarkable, where problems that seem intractable online somehow solve themselves in person, where we come away with ideas for an entire year and memories for a lifetime.

27 August 2013

Robert Collins: Using vanilla novaclient with Rackspace cloud

The Rackspace docs describe how to use rackspace s custom extensions, but not how to use plain ol nova. Using plain nova is important if you want cloud portability in your scripts. So for future reference these are the settings:

export OS_AUTH_URL=https://identity.api.rackspacecloud.com/v2.0/
export OS_REGION_NAME=DFW
export OS_USERNAME=<username>
export OS_TENANT_NAME=<clientid>
export OS_PASSWORD=<password>
export OS_PROJECT_ID=<clientid>
export OS_NO_CACHE=1
unset NOVA_RAX_AUTH
unset OS_AUTH_SYSTEM

26 August 2013

Jose Luis Rivas: MongoDB: $pull from an embedded document array

TL;DR: db.collection.update( _id: 'x' , $pull: embedded_doc: _id: 'y' ) Queries are wrote for the mongo shell Dot based access If you have been using MongoDB for some time, then there are big chances that you will know the "dot-based access" on the find queries. This is the easier way to write those queries and will probably want to keep using that. Like:
db.collection.find( _id: 'x', 'embedded_doc._id': 'y' )
The thing is that does not work with everything, like: Arrays of embedded documents! Little example schema Let's suppose the following little schema (written for Mongoose): A = new Schema( w: String, x: String ); Document = new Schema( embedded_doc: [A] ); The collection of Document is called documents. Updating arrays If the arrays are not embedded documents I can actually just use a regular update like with any other field combined with a $set and change all the array like:
db.documents.update( _id: 'x',  $set:  array: [
   a: 1 , 
   a: 2 , 
   a: 3 
] )
And using conveniently $pull and $addToSet. But when it comes to embedded documents the history is very different. You can't make a simple $set because it will give you an error related to ModId which you can't change. So you really need to use $push, $addToSet, $pull and it's friends. So all those you can do it as in the manual, but what if you want to make a $pull over an array of objects and w and x are not unique? Well, that's why you probably made them an embedded document! Then you have the _id and you try to use the dot notation like:
db.documents.update( _id: 'x' ,  $pull:  "embedded_doc._id": 'y' )
And it looks good, except that it wont work. You have to use the complete syntax like:
db.documents.update( _id: 'x' ,  $pull:  embedded_doc:  _id: 'y' )

25 August 2013

Gregor Herrmann: RC bugs 2013/31-34

after a break, here's again a report about my work on RC bugs. in this case, all bugs either affect packages in the pkg-perl group and/or packages involved in the Perl 5.18 transition.

Matthias Klumpp: Some Tanglu updates

Tanglu Logo (big)Long time since I published the last update about Tanglu. So here is a short summary about what we did meanwhile (even if you don t hear that much, there is lots of stuff going on!) Better QA in the archive We now use a modified version of Debian s Britney tool to migrate packages from the newly-created staging suite to our current development branch aequorea . This ensures that all packages are completely built on all architectures and don t break other packages. New uploads and syncs/merges now happen through the staging area and can be tested there as well as being blocked on demand, so our current aequorea development branch stays installable and usable for development. People who want the *very* latest stuff, can embed the staging-sources into their sources.list (but we don t encourage you to do that). Improved syncs from Debian The Synchrotron toolset we use to sync packages with Debian recently gained the ability to sync packages using regular expressions. This makes it possible to sync many packages quickly which match a pattern. Tons of infrastructure bugfixes The infrastructure has been tweaked a lot to remove quirks, and it now works quite smoothly. Also, all Debian tools now work flawless in the Tanglu environment. A few issues are remaining, but nothing really important is affected anymore (and some problems are merely cosmetic). Long term we plan to replace the Jenkins build-infrastructure with the software which is running Paul Tagliamontes debuild.me (only the buildd service, the archive will still be managed by dak). This requires lots of work, but will result in software not only usable by Tanglu, but also by Debian itself and everyone who wants a build-service capable of building a Debian-distro. KDE 4.10 and GNOME 3.8 Tanglu now offers KDE 4.10 by default, a sync of GNOME 3.8 is currently in progress. The packages will be updated depending on our resources and otherwise just be synced from Debian unstable/experimental (and made working for Tanglu). systemd 204 & libudev1 Tanglu now offers systemd 204 as default init system, and we transitioned the whole distribution to the latest version of udev. This even highlighted a few issues which could be fixed before the latest systemd reached Debian experimental. The udev transition went nicely, and hopefully Debian will fix bug#717418 too, soon, so both distributions run with the same udev version (which obviously makes things easier for Tanglu ^^) Artwork We now have a plymouth-bootscreen and wallpapers and stuff is in progress :-) Alpha-release & Live-CD ? This is what we are working on we have some issues in creating a working live-cd, since live-build does have some issues with our stack. We are currently resolving issues, but because of the lack of manpower, this is progressing slowly (all contributors also work on other FLOSS projects and of course also have their work :P ) As soon as we have working live-media, we can do a first alpha release and offer installation media. You! Tanglu is a large task. And we need every help we can get, right now especially technical help from people who can build packages (Debian Developers/Maintainers, Ubuntu developers, etc.) We especially need someone to take care of the live-cd. But also the website needs some help, and we need more artwork or improved artwork ;-) In general, if you have an idea how to make Tanglu better (of course, something which matches it s goals :P ) and want to realize it, just come back to us! You can reach us on the mailiglists (tanglu-project for stuff related to the project, tanglu-devel for development of the distro) or in #tanglu / #tanglu-devel (on Freenode).

24 August 2013

Marko Lalic: PTS rewrite: Django Memory Usage


Rewriting the PTS in a fairly large Web framework such as Django is sure to cause larger memory consumption by even the more simple tasks, when compared to a couple of (loosely related) perl scripts. If anything, simply due to the fact that the entire Django machinery needs to be loaded.

However, just how large the memory consumption difference between the two can be seen in the example of the command to dispatch received package emails to users. In the new version, the command is implemented as a Django management command, whereas before it was a perl script dispatch.pl relying on the common.pl script for some additional functions.

The reason this command's memory usage is important is an incident with the current test deployment of the new PTS found at http://pts.debian.net. All package mails that the old PTS receives are also forwarded to the new instance in order to expose the new implementation to real-world mail traffic. At a certain point, over a hundred mails were received instantly causing over a hundred dispatch processes to be launched. This led to the system running out of memory and the kernel had to run the OOM Killer which brought down PostgreSQL server thereby causing the site to become unavailable.

For now, such a problem seems to have been prevented from reoccurring by setting up exim to queue messages when a certain system average load is exceeded, however it is interesting to check how large the memory consumption difference between the old and new dispatch really is.

Measurement setupIn order to measure the memory use, two messages of different sizes were used: 1KB and 1MB.

For the new PTS two cases were considered, when the database used is sqlite3 and PostgreSQL. In both cases, it was simply initialized by syncdb, meaning it contained only the default keywords found in the initial fixture and no registered users and subscriptions. The DEBUG setting was set to False.

For testing the old PTS's dispatch.pl, the database was also initialized empty.

This way, the tests should show the actual difference between the two implementations.

Since the main concern here is the maximum memory usage of the dispatch script, a simple bash script was written which takes a PID of a running process and outputs the maximum memory usage as reported by ps once the process is terminated. The script used is shown below.

#!/usr/bin/env bash

pid=$1

while ps $pid >/dev/null
do
ps -o vsz= $ pid
sleep 0.1
done sort -n tail -n1

Old PTS Measurements
The maximum memory usage of the old PTS was:
  1. 37.8 MB for the 1 KB message
  2. 38.2 MB for the 1 MB message

New PTS Measurements

sqlite3
The maximum memory usage of the new PTS using sqlite3 was:
  1. 91.77 MB for the 1 KB message
  2. 92.59 MB for the 1 MB message

PostgreSQL

The maximum memory usage of the new PTS using PostgreSQL was:
  1. 148 MB for the 1 KB message
  2. 160 MB for the 1 MB message

Comparison and DiscussionWhen comparing the memory used by the old dispatch.pl implementation and the new PTS running sqlite3, the difference does not seem to be too large and is to be expected, since, as mentioned, the whole Django framework is loaded when executing a management command.

However, the huge difference between the two versions of the new PTS: the one using sqlite3 and the one using PostgreSQL is very surprising indeed.

Another interesting measurement is the difference between running a bare Django management command: one including only a sleep statement (in order to allow enough time to measure its memory usage) on PostgreSQL and sqlite3. In both cases, the memory usage was only about 1 MB less than what the respective maximum memory usage when processing the 1 KB message.

All this considered, the logical conclusion seems to be that either psycopg2 package and/or the Django postgresql_psycopg2 database engine use a lot more memory than the corresponding sqlite3 alternatives.

Could anyone shed some more light as to what causes this stark difference between using sqlite3 and Postgres in Django? Anything that could be done to mitigate it?

Marko Lalic: PTS Rewrite Project Status

The rewrite has already achieved some of the major goals which have been set in the beginning, as well as made some changes and improvements along the way. Some of the major points of the new system implemented so far will be presented here.

    Tasks framework
    The current PTS implementation regenerates the HTML pages for all packages a few times per day. A goal of the rewrite was to make smaller incremental updates possible so that the information displayed for each package can be much more dynamic.

    In order to allow incremental updates, it is necessary to consider the situation where updating some information should trigger an update of other information.

    This is where the tasks framework comes in. It allows developers to define a task with a list of "events" that it produces. An event can be triggered by any change to shared information or anything else the task would like to signal. Each event is identified by a simple unique string name. Events can also contain arbitrary (JSON-serializable) objects. Other than events that it produces, a task defines a list of dependencies events it depends on.

    When a single task should be executed, the tasks framework uses this information to build a dependency graph (a DAG) and executes the tasks in topological sort order, making sure each task runs only after all tasks which could raise an event it depends on have finished. It automatically makes the raised events available to the task so that it can access the extra data. Tasks for which no event was raised are skipped.

    Such execution of a task along with all its dependencies is called a Job. A Job's state is persisted to the database after each task completes its operation. This allows Jobs to be continued if there is an unexpected error causing it to terminate.

    So far, the tasks framework has already proven useful. For example, there are tasks which extract some package information to a denormalized model suited for directly displaying the data in the package page after a new package version is detected in a repository. Other examples are tasks which update standards-version warnings (after a new package version is detected) and the news generation task based on various detected changes to the packages (new source version, version removed from repository, version migrated to another repository).

    Vendor-specific customizationAnother goal of the rewrite was to make the PTS less Debian-specific. This is so that derivatives could easily deploy their own versions of the PTS. The idea is to have a number of core PTS services which alternative deployments can customize and augment.

    The simplest way vendors can customize PTS is by changing some of the local settings values. An example of this would be changing the PTS_VENDOR_NAME which leads to this name being displayed in any part of the system.

    This is fairly uninteresting since the information is always static. What is more interesting is when the vendor should provide different information based on some context, e.g. package name, received message, etc. This requirement led to the concept of the vendor app with a rules module.

    Various core parts of the PTS specify hook functions which different vendors can implement to provide customized behavior of the core component. Some examples are:
    • providing URLs to a developer information site (e.g. http://qa.debian.org/developer.php in Debian's case) based on a developer's email
    • providing a URL to a bug tracker for a particular package (based on the package type and bug category type)
    • providing additional rules to tag received package mails with keywords
    • providing additional headers to be injected in the forwarded package mails
    • providing a list of bugs which should be displayed in the bugs panel
    A settings value is set giving the full dotted path to the Python module containing the implementation of these hook functions.

    Vendor-specific featuresA step beyond customizing core features is for vendors to include their own specific functionality. The already described tasks framework allows for implementing custom data update/generation processes (optionally depending on core events). Furthermore, since a vendor-specific Django app is expected, it can contain any vendor-specific models which are managed just as the core models are (syncdb, migrations, etc. all work as expected). The last piece of the puzzle is displaying this information on the package page.

    Currently, all package information is provided in various boxes (general, bugs, links, etc.) which were named panels. The rewrite has made it possible to implement additional panels in a really elegant way.

    It is required to implement a class with some properties giving:
    • the panel's title
    • position in the page (left, center, right)
    • panel importance (higher importance panels are placed above lower importance ones in the same column)
    To generate the HTML for a package, two alternatives are allowed. The first is to provide a property which gives (explicitly marked safe) HTML output which is then included in the correct position verbatim. The second alternative, which is what should cover most needs and should be preferred, is to provide a property giving a template name and a property giving the additional context variables necessary to render this template. By using the power of Python properties, the context can be dynamically generated for each package by the panel. The PTS then makes sure to pass the correct panel's context when rendering its template and to include the rendered result in the final package page.

    Various convenience classes are implemented in the panel API, but those are better left to be read about in the panels API documentation.

    Changes from current PTSSome existing features of PTS were not simply rewritten to fit into a Django project, but also improved upon.

    One of the first such changes was reducing the number of confirmation mails sent by the email control bot to only one per received control mail (as per a wishlist PTS bug).

    Work started by Markus Wanner (see this bug) was expanded on and integrated with the new News model so that any inline GPG signatures found in the content of a news item are automatically extracted and the signers name displayed in the package page.

    The existing TODO and Probems panels have been merged into a single Action Needed panel. However, besides merging their representation in the package page, all the package issues are now represented as an ActionItem Django model which provides additional meta-data (item type, date created/updated, severity level) which could be used in the future to provide additional functionality (such as all packages with a certain issue type).

    The descriptions of the action items were also shortened from their current rendition, with the detailed description being moved to a dedicated item page. If Javascript is enabled in the user's browser, it can also be displayed in a popup.

    GUIThe graphical interface has seen some changes, however I hesitate to claim improvements. In my opinion, some of it is definitely better, more polished and more "modern" (for what it's worth).

    Still, this could benefit from more people getting involved in testing and providing feedback so that the design is polished to suit the largest number of users. For example, the padding between the panels is increased, which to me definitely looks better, however, it is possible that this introduces too much whitespace and reduces the information density below what some may expect.

    RecapOverall, I am quite pleased with where the project is currently. Most of the information provided by the current PTS has been reimplemented, with only a few smaller ones remaining. For now, since we are nearly entering the final month of GSoC, the development will switch focus from porting old features to the new PTS to adding in some new ones. Namely, user registration and account management is in the pipeline for the coming week. It entails features such as managing package subscriptions via the online interface (for multiple emails associated with the account) and sso.debian.org integration for Debian Developers.

    This post aimed to provide the most notable changes and improvements made so far to the PTS. In reality, there has been quite a lot of them and not everything is covered here. For more information, check out the PTS documentation which is updated weekly (corresponding to the weekly deployments) or the source code repository.

    The latest iteration of the PTS can always be found at http://pts.debian.net. The mail control interface bot responds to mails sent to control@pts.debian.net.

    Stay tuned for updates and be sure to check out pts.debian.net on at least a weekly basis to see new deployments in action!

    Next.

    Previous.