A mock webserver to use for unit testing HTTP clients With python -m SimpleHTTPServer it's easy to bring up an HTTP server to use to test HTTP client code, however it only supports GET requests, and I needed to test an HTTP client that needs to perform a file upload. It took way more than I originally expected to put this together, so here it is, hopefully saving other people (including future me) some time:

#!/usr/bin/python3
import http.server
import cgi
import socketserver
import hashlib
import json
PORT = 8081
class Handler(http.server.SimpleHTTPRequestHandler):
    def do_POST(self):
        info =  
            "method": "POST",
            "headers":   k: v for k, v in self.headers.items()  ,
         
        # From https://snipt.net/raw/f8ef141069c3e7ac7e0134c6b58c25bf/?nice
        form = cgi.FieldStorage(
            fp=self.rfile,
            headers=self.headers,
            environ= 'REQUEST_METHOD':'POST',
                     'CONTENT_TYPE':self.headers['Content-Type'],
                      )
        postdata =  
        for k in form.keys():
            if form[k].file:
                buf = form.getvalue(k)
                postdata[k] =  
                    "type": "file",
                    "name": form[k].filename,
                    "size": len(buf),
                    # json.dumps will not serialize a byte() object, so we
                    # return the shasum instead of the file body
                    "sha256": hashlib.sha256(buf).hexdigest(),
                 
            else:
                vals = form.getlist(k)
                if len(vals) == 1:
                    postdata[k] =  
                        "type": "field",
                        "val": vals[0],
                     
                else:
                    postdata[k] =  
                        "type": "multifield",
                        "vals": vals,
                     
        info["postdata"] = postdata
        resbody = json.dumps(info, indent=1)
        print(resbody)
        resbody = resbody.encode("utf-8")
        self.send_response(200)
        self.send_header("Content-type", "application/json")
        self.send_header("Content-Length", str(len(resbody)))
        self.end_headers()
        self.wfile.write(resbody)
class TCPServer(socketserver.TCPServer):
    # Allow to restart the mock server without needing to wait for the socket
    # to end TIME_WAIT: we only listen locally, and we may restart often in
    # some workflows
    allow_reuse_address = True
httpd = TCPServer(("", PORT), Handler)
print("serving at port", PORT)
httpd.serve_forever()

There has been a lot of discussion recently where there is strong disagreement, even about how to discuss the disagreement. Here s a few thoughts on the matter. The thing I personally find the most annoying is when someone thinks what someone else says is inappropriate and says so, it seems like the inevitable response is to scream censorship. When people do that, I m pretty sure they don t know what the word censorship actually means. Debian/Ubuntu/Insert Project Name Here resources are not public spaces and no government is telling people what they can and can t say. When you engage in speech and people respond to that speech, even if you don t feel all warm and fuzzy after reading the response, it s not censorship. It s called discussion. When someone calls out speech that they think is inappropriate, the proper response is not to blame a Code of Conduct or some other set of rules. Projects that have a code, also have a process for dealing with claims the code has been violated. Unless someone invokes that process (which almost never happens), the code is irrelevant. What s relevant is that someone is having a problem with what or how you are saying something and are in some way hurt by it. Let s focus on that. The rules are irrelevant, what matters is working together in a collegial way. I really don t think project members actively want other project members to feel bad/unsafe, but it s hard to get outside ones own defensive reaction to being called out. So please pay less attention to how you re feeling about things and try to see things from the other side. If we can all do a bit more of that, then things can be better for all of us. Final note: If you ve gotten this far and thought Oh, that other person is doing this to me , I have news for you it s not just them.

What the world needs right now is yet another tutorial on R packages and their creation. Luckily, this last Friday and Saturday, I had the opportunity to present in a workshop organized by Frank DiTraglia at Penn's shiny new Warren Center, and held at Wharton. Given the Warren Center's focus, the workshop centered around Big Data and Open Science with R. Yihui Xie and myself alternated on delivering four units on an Introduction to R, Writing R packages, Dynamic Documents with R, and HPC with Rcpp and RcppArmadillo. So I had to come up with a plan for teaching creating R packages -- and decided to do it from the very bottom up, clearly introducing the underlying R CMD ... commands and only then switching to taking advantage of an environment such as the RStudio IDE. The resulting slides are now available on my presentations page. The code examples are in a repo subdirectory on GitHub as well. While both were designed to support the parallel live instruction offered in the workshop, I would be interested in feedback (preferably via email) about how useful the slides are by themselves.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

This is, essentially, a call for help, as I don't really know which program is at a fault here. Given that Emacs's upstream converted their repository from bzr to git, all the commits in mirror repositories became "invalid" in relation to the official repository. What does this mean in practical terms, in my case? Well, bear with me while I try to report my steps.

Noticing a regression and reporting a bug There is a regression with Emacs 24.4 relative to 24.3, which I discovered after Emacs 24.4 became available in Debian's sid. The regression in particular is that Emacs 24.4 doesn't seem to respect my Xresources, while 24.3 does (and this is 100% reproducible: I kept the binary packages of version 24.3 of emacs24 and I can install and reinstall things). When I reported this to upstream, I received a reply that it worked fine with another person that was using XFCE with unstable.

Testing various Desktop environments As I am using the MATE desktop environment, I proceeded to test this assertion by installing XFCE. Emacs 24.4 read my Xresources. I went ahead and installed LXDE. It worked again. I tried once more with GNOME 3, but "regular" GNOME 3 just crashed. I tried with GNOME 3 Classic and Emacs 24.4 just worked again.

Going deep into the rabbit's hole Then, I got more curious and I tried to see why things worked the way that they did and given that there was a mirror of the Emacs repo on github, I cloned it and started to git bisect to find the first problematic commit (I have no idea if bzr even offers something like git's bisect and I wouldn't really know how to do it as quickly as I do with git). To cut short a long story, after many recompiles, many wasted hours, a lot of wasted electrical energy, I found a bad commit and reported it. I received no response after that.

The new repo enters in action Of course, all my hard work bisecting things was completely invalidated after the transition to the new repository went live. To make things relevant again, I used the awesome powers of git, restricting the changes of the newly cloned repository to the e-mail of the committer in question (Chong Yidong) and, from there, I proceeded to another painful process of git bisects. And, sure enough, the first bad commit was the same one that I found with the previous tree. Semi-blindly reverting this commit, and also semi-blindly resolving the conflicts make Emacs's from master work again on my system, but I highly suspect that (given the way that I did it), it would not really be appropriate for upstream. But given also that I failed to receive feedback after my original report, I am not too confident that this bug can be solved soon (even if it doesn't qualify for being fixed in Debian 8). After all this, I don't really know what else to do. I even filed a bug report (more like a request for help) to the Debian MATE maintainers. As a side note, I would have filed a bug to upstream MATE, but it is not really clear what the proper procedure to report bugs to them is---they seem to use github's issues system, but given that they have separate repositories for each component of the project, and that I don't know precisely what repository to report to (or even if it applies to MATE after all), I am more or less paralyzed.

A side note I must say that the conversion was well done by Eric Raymond, because the whole .git repository of the new repo is only about 200MB, with history going back to 1985, while the other repository had about 800GB.

You may have heard that Uber has been under a bit of fire lately for its desires to hire private investigators to dig up dirt on journalists who are critical of Uber. From using users ride data for party entertainment, putting the assistance dogs of blind passengers in the trunk, adding a surcharge to reduce the number of dodgy drivers, or even booking rides with competitors and then cancelling, or using the ride to try and convince the driver to change teams, it s pretty clear that Uber is a pretty good example of how companies are inherently sociopathic. However, most of those examples are internal stupidities that happened to be made public. It s a very rare company that doesn t do all sorts of shady things, on the assumption that the world will never find out about them. Uber goes quite a bit further, though, and is so out-of-touch with the world that it blogs about analysing people s sexual activity for amusement. You ll note that if you follow the above link, it sends you to the Wayback Machine, and not Uber s own site. That s because the original page has recently turned into a 404. Why? Probably because someone at Uber realised that bragging about how Uber employees can amuse themselves by perving on your one night stands might not be a great idea. That still leaves the question open of what sort of a corporate culture makes anyone ever think that inspecting user data for amusement would be a good thing, let alone publicising it? It s horrific. Thankfully, despite Uber s fairly transparent attempt at whitewashing ( clearwashing ?), the good ol Wayback Machine helps us to remember what really went on. It would be amusing if Uber tried to pressure the Internet Archive to remove their copies of this blog post (don t bother, Uber; I ve got a Save As button and I m not afraid to use it). In any event, I ve never used Uber (not that I ve got one-night stands to analyse, anyway), and I ll certainly not be patronising them in the future. If you re not keen on companies amusing themselves with your private data, I suggest you might consider doing the same.

Propellor has supported docker containers for a "long" time, and it works great. This week I've worked on adding more container support. docker containers (revisited) The syntax for docker containers has changed slightly. Here's how it looks now: That makes example.com have a web server in a docker container, as you'd expect, and when propellor is used to deploy the DNS server it'll automatically make www.example.com point to the host (or hosts!) where this container is docked. I use docker a lot, but I have drank little of the Docker KoolAid. I'm not keen on using random blobs created by random third parties using either unreproducible methods, or the weirdly underpowered dockerfiles. (As for vast complicated collections of containers that each run one program and talk to one another etc ... I'll wait and see.) That's why propellor runs inside the docker container and deploys whatever configuration I tell it to, in a way that's both replicatable later and lets me use the full power of Haskell. Which turns out to be useful when moving on from docker containers to something else... systemd-nspawn containers Propellor now supports containers using systemd-nspawn. It looks a lot like the docker example. Notice how I specified the Debian Unstable chroot that forms the basis of this container. Propellor sets up the container by running debootstrap, boots it up using systemd-nspawn, and then runs inside the container to provision it. Unlike docker containers, systemd-nspawn containers use systemd as their init, and it all integrates rather beautifully. You can see the container listed in systemctl status, including the services running inside it, use journalctl to examine its logs, etc. But no, systemd is the devil, and docker is too trendy... chroots Propellor now also supports deploying good old chroots. It looks a lot like the other containers. Rather than repeat myself a third time, and because we don't really run webservers inside chroots much, here's a slightly different example. Again this uses debootstrap to build the chroot, and then it runs propellor inside the chroot to provision it (btw without bothering to install propellor there, thanks to the magic of bind mounts and completely linux distribution-independent packaging). In fact, the systemd-nspawn container code reuses the chroot code, and so turns out to be really rather simple. 132 lines for the chroot support, and 167 lines for the systemd support (which goes somewhat beyond the nspawn containers shown above). Which leads to the hardest part of all this... debootstrap Making a propellor property for debootstrap should be easy. And it was, for Debian systems. However, I have crazy plans that involve running propellor on non-Debian systems, to debootstrap something, and installing debootstrap on an arbitrary linux system is ... too hard. In the end, I needed 253 lines of code to do it, which is barely one magnitude less code than the size of debootstrap itself. I won't go into the ugly details, but this could be made a lot easier if debootstrap catered more to being used outside of Debian. closing Docker and systemd-nspawn have different strengths and weaknesses, and there are sure to be more container systems to come. I'm pleased that Propellor can add support for a new container system in a few hundred lines of code, and that it abstracts away all the unimportant differences between these systems. PS Seems likely that systemd-nspawn containers can be nested to any depth. So, here's a new kind of fork bomb! Strongly typed purely functional container deployment can only protect us against a certian subset of all badly thought out systems. ;)

Almost two months ago I posted our keyring status graphs, showing the progress of the transition to >=2048-bit keys for the different active Debian keyrings. So, here are the new figures. First, the Non-uploading keyring: We were already 100% transitioned. You will only notice a numerical increase: That little bump at the right is our dear friend T ssia finally joining as a Debian Developer. Welcome! \o/ As for the Maintainers keyring: We can see a sharp increase in 4096-bit keys. Four 1024-bit DM keys were migrated to 4096R, but we did have eight new DMs coming in To them, also, welcome \o/. Sadly, we had to remove a 1024-bit key, as Peter Miller sadly passed away. So, in a 234-key universe, 12 new 4096R keys is a large bump! Finally, our current-greatest worry If for nothing else, for the size of the beast: The active Debian Developers keyring. We currently have 983 keys in this keyring, so it takes considerably more effort to change it. But we have managed to push it noticeably. This last upload saw a great deal of movement. We received only one new DD (but hey welcome nonetheless! \o/ ). 13 DD keys were retired; as one of the maintainers of the keyring, of course this makes me sad but then again, in most cases it's rather an acknowledgement of fact: Those keys' holders often state they had long not been really involved in the project, and the decision to retire was in fact timely. But the greatest bulk of movement was the key replacements: A massive 62 1024D keys were replaced with stronger ones. And, yes, the graph changed quite abruptly: We still have a bit over one month to go for our cutoff line, where we will retire all 1024D keys. It is important to say we will not retire the affected accounts, mark them as MIA, nor anything like that. If you are a DD and only have a 1024D key, you will still be a DD, but you will be technically unable to do work directly. You can still upload your packages or send announcements to regulated mailing lists via sponsor requests (although you will be unable to vote). Speaking of votes: We have often said that we believe the bulk of the short keys belong to people not really active in the project anymore. Not all of them, sure, but a big proportion. We just had a big, controversial GR vote with one of the highest voter turnouts in Debian's history. I checked the GR's tally sheet, and the results are interesting: Please excuse my ugly bash, but I'm posting this so you can play with similar runs on different votes and points in time using the public keyring Git repository: So, as of mid-October: 387 out of the 482 votes (80.3%) were cast by developers with >=2048-bit keys, and 95 (19.7%) were cast by short keys. If we were to run the same vote with the new active keyring, 417 votes would have been cast with >=2048-bit keys (87.2%), and 61 with short keys (12.8%). We would have four less votes, as they retired: So, lets hear it for November/December. How much can we push down that pesky yellow line? Disclaimer: Any inaccuracy due to bugs in my code is completely my fault!

With OpenStack embracing the Tooz library more and more over the past year, I think it's a good start to write a bit about it. A bit of history A little more than year ago, with my colleague Yassine Lamgarchal and others at eNovance, we investigated on how to solve a problem often encountered inside OpenStack: synchronization of multiple distributed workers. And while many people in our ecosystem continue to drive development by adding new bells and whistles, we made a point of solving new problems with a generic solution able to address the technical debt at the same time. Yassine wrote the first ideas of what should be the group membership service that was needed for OpenStack, identifying several projects that could make use of this. I've presented this concept during the OpenStack Summit in Hong-Kong during an Oslo session. It turned out that the idea was well-received, and the week following the summit we started the tooz project on StackForge. Goals Tooz is a Python library that provides a coordination API. Its primary goal is to handle groups and membership of these groups in distributed systems. Tooz also provides another useful feature which is distributed locking. This allows distributed nodes to acquire and release locks in order to synchronize themselves (for example to access a shared resource). The architecture If you are familiar with distributed systems, you might be thinking that there are a lot of solutions already available to solve these issues: ZooKeeper, the Raft consensus algorithm or even Redis for example. You'll be thrilled to learn that Tooz is not the result of the NIH syndrome, but is an abstraction layer on top of all these solutions. It uses drivers to provide the real functionalities behind, and does not try to do anything fancy. All the drivers do not have the same amount of functionality of robustness, but depending on your environment, any available driver might be suffice. Like most of OpenStack, we let the deployers/operators/developers chose whichever backend they want to use, informing them of the potential trade-offs they will make. So far, Tooz provides drivers based on:

• Kazoo (ZooKeeper)
• Zake
• memcached
• redis
• SysV IPC (only for distributed locks for now)
• PostgreSQL (only for distributed locks for now)
• MySQL (only for distributed locks for now)

All drivers are distributed across processes. Some can be distributed across the network (ZooKeeper, memcached, redis ) and some are only available on the same host (IPC). Also note that the Tooz API is completely asynchronous, allowing it to be more efficient, and potentially included in an event loop. Features Group membership Tooz provides an API to manage group membership. The basic operations provided are: the creation of a group, the ability to join it, leave it and list its members. It's also possible to be notified as soon as a member joins or leaves a group. Leader election Each group can have a leader elected. Each member can decide if it wants to run for the election. If the leader disappears, another one is elected from the list of current candidates. It's possible to be notified of the election result and to retrieve the leader of a group at any moment. Distributed locking When trying to synchronize several workers in a distributed environment, you may need a way to lock access to some resources. That's what a distributed lock can help you with. Adoption in OpenStack Ceilometer is the first project in OpenStack to use Tooz. It has replaced part of the old alarm distribution system, where RPC was used to detect active alarm evaluator workers. The group membership feature of Tooz was leveraged by Ceilometer to coordinate between alarm evaluator workers. Another new feature part of the Juno release of Ceilometer is the distribution of polling tasks of the central agent among multiple workers. There's again a group membership issue to know which nodes are online and available to receive polling tasks, so Tooz is also being used here. The Oslo team has accepted the adoption of Tooz during this release cycle. That means that it will be maintained by more developers, and will be part of the OpenStack release process. This opens the door to push Tooz further in OpenStack. Our next candidate would be write a service group driver for Nova. The complete documentation for Tooz is available online and has examples for the various features described here, go read it if you're curious and adventurous!

We have had terrible months in Mexico; I don't know how much has appeared about our country in the international media. The last incidents started on the last days of September, when 43 students at a school for rural teachers were forcefully disappeared (in our Latin American countries, this means they were taken by force and no authority can yet prove whether they are alive or dead; forceful disappearance is one of the saddest and most recognized traits of the brutal military dictatorships South America had in the 1970s) in the Iguala region (Guerrero state, South of the country) and three were killed on site. An Army regiment was stationed few blocks from there and refused to help. And yes, we live in a country where (incredibly) this news by themselves would not seem so unheard of... But in this case, there is ample evidence they were taken by the local police forces, not by a gang of (assumed) wrongdoers. And they were handed over to a very violent gang afterwards. Several weeks later, with far from a thorough investigation, we were told they were killed, burnt and thrown to a river. The Iguala city major ran away, and was later captured, but it's not clear why he was captured at two different places. The Guerrero state governor resigned and a new governor was appointed. But this was not the result of a single person behaving far from what their voters would expect It's a symptom of a broken society where policemen will kill when so ordered, where military personnel will look away when pointed out to the obvious, where the drug dealers have captured vast regions of the country where are stronger than the formal powers. And then, instead of dealing with the issue personally as everybody would expect, the president goes on a commercial mission to China. Oh, to fix some issues with a building company. That coincidentally or not was selling a super-luxury house to his wife. A house that she, several days later, decided to sell because it was tarnishing her family's honor and image. And while the president is in China, the person who dealt with the social pressure and told us about the probable (but not proven!) horrible crime where the "bad guys" for some strange and yet unknown reason (even with tens of them captured already) decided to kill and burn and dissolve and disappear 43 future rural teachers presents his version, and finishes his speech saying that "I'm already tired of this topic". Of course, our University is known for its solidarity with social causes; students in our different schools are the first activists in many protests, and we have had a very tense time as the protests are at home here at the university. This last weekend, supposed policemen entered our main campus with a stupid, unbelievable argument (they were looking for a phone reported as stolen three days earlier), get into an argument with some students, and end up firing shots at the students; one of them was wounded in the leg. And the university is now almost under siege: There are policemen surrounding us. We are working as usual, and will most likely finish the semester with normality, but the intimidation (in a country where seeing a policeman is practically never a good sign) is strong. And... Oh, I could go on a lot. Things feel really desperate and out of place. Today I will join probably tens or hundreds of thousands of Mexicans sick of this simulation, sick of this violence, in a demonstration downtown. What will this achieve? Very little, if anything at all. But we cannot just sit here watching how things go from bad to worse. I do not accept to live in a state of exception. So, this picture is just right: A bit over a month ago, two dear friends from Guadalajara city came, and we had a nice walk in the University. Our national university is not only huge, it's also beautiful and loaded with sights. And being so close to home, it's our favorite place to go with friends to show around. This is a fragment of the beautiful mural in the Central Library. And, yes, the University stands for "Viva M xico". And the university stands for "Peace". And we need it all. Desperately.

Apparently (the actual results have not yet been published by the Secretary), the GR is over, and the worst possible option has won. This is an absolutely ambiguous result, while at the same time sending a clear signal that Debian is not to be trusted wrt. investing anything into it, right now. Why is this? Simply: GR not required means that whatever people do is probably right . Besides this, we have one statement from the CTTE ( systemd is default init system for jessie. Period. ) and nothing else. This means that runit, or upstart, or file-rc, or uselessd, can be the default init system for zurg^H^H^H^Hstretch, or even the only one. It also means that the vast majority of Debian Developers are sheeple, neither clearly voting to preserve freedom of choice between init systems for its users, nor clearly voting to unambiguously support systemd and progress over compatibility and choice, nor clearly stating that systemd is important but supporting other init systems is still recommended. (I ll not go into detail on how the proposer of the apparently winning choice recommends others to ignore ftpmaster constraints and licences, and even suggests to run a GR to soften up the DFSG interpretation.) I d have voted this as no, absolutely not if it was possible to do so more strongly. Judging from the statistics, the only thing I voted above NOTA/FD is the one least accepted by DDs, although the only other proposal I considered is the first-rated of them: support for other init systems is recommended but not required. What made me vote it below NOTA/FD was: The Debian Project makes no statement at this time on sysvinit support beyond the jessie release. This sentence made even this proposal unbearable, unacceptable, for people wanting to invest (time, money, etc.) into Debian. Update: Formal result announced. So 358 out of 483 voting DDs decided to be sheeple (if I understand the eMail correctly). We had 1006 DDs with voting rights, which is a bit ashaming as well. That s 48.01% only. I wonder what s worse. This opens up a very hard problem: I m absolutely stunned by this and wondering what to do now. While there is no real alternative to Debian at $dayjob I can always create customised packages in my own APT repository, and while it was great when those were eventually (3.1.17-1) accepted into Debian, even replacing the previous packages completely it is simpler and quicker to not do so. While $dayjob benefits from having packages I work on inside Debian itself, even though I cannot always test all scenarios Debian users would need, some work reduction due to reactions already led to Debian losing out on Mediawiki for jessie and some additional suffering. With my own package repository, I can modulo installing/debootstrap serve my needs for $dayjob much quicker, easily, etc. and only miss out on absolutely delightful user feedback. But then, others could always package software I m upstream of for Debian. Or, if I do not leave the project, continue doing so via QA uploads. I m also disappointed because I have invested quite some effort into trying to make Debian better (my idea to join as DD was if I ve got to use it, it better be damn good! ), into packaging software and convincing people at work that developing software as Debian packages instead of (or not) thinking of packaging later was good. I ve converted our versions of FusionForge and d-push to Debian packages, and it works pretty damn well. Sometimes it needs backports of my own, but that s the corportate world, and no problem to an experienced DD. (I just feel bad we ($orkplace) lost some people, an FTP master along them, before this really gained traction.) I d convert to OpenBSD because, despite MirBSD s history with them, they re the only technically sound alternative, but apparently tedu (whom I respect technically, and who used to offer good advice to even me when asked, and who I think wouldn t choose systemd himself) still (allying with the systemd side (I m not against people being able to choose systemd, for the record, I just don t want to be forced into it myself!)) has some sort of grudge against me. Plus, it d be hard to get customers to follow. So, no alternative right now. But I m used to managing my own forks of software; I m doomed to basically hack and fix anything I use (I recently got someone who owns a licence to an old-enough Visual Studio version to transfer that to me, so I can hack on the Windows Mobile 6 version of Cachebox, to fix bugs in one of the geocaching applications I use. Now I just need to learn C# and the .NET Compact Framework. So I m also used to some amount of pain.) I m still unresolved wrt. the attitude I should show the Debian project now. I had decided to just continue to live on, and work on the things I need done, but that was before this GR non-result. I absolutely cannot recommend anyone to invest into Debian (without sounding hypocriet), but I cannot recommend anything else either. I cannot justify leaving but don t know if I want to stay. I think I should sleep over it. One thing I promised, and thus will do, is to organise a meeting of the Debian/m68k people soonish. But then, major and important and powerful forces inside Debian still insist that Debian-Ports are not part of it [Update: yes, DSA is moving it closer, thanks for that by the way, but that doesn t mean anything to certain maintainers or the Release Team, although, the latter is actually understandable and probably sensible.] yet, all forks of Debian now suffer from the systemd adoption in it instead of having a freedom-of-choice upstream. I ve said, and I still feel that systemd adoption should have done in a Debian downstream / (pure?) blend, and maybe (parts of) GNOME removed from Debian itself for it. (Adding cgroups support to the m68k kernel to support systemd was done. I adviced against it, on the grounds of memory and code size. But no downstream can remove it now.)

Earlier today, Josh send the text below to the R-SIG-Finance list, and I updated the R/Finance website, including its Call for Papers page, accordingly. We are once again very excited about our conference, thrilled about the four confirmed keynotes, and hope that many R / Finance users will not only join us in Chicago in May 2015 -- but also submit an exciting proposal. So read on below, and see you in Chicago in May! Call for Papers: R/Finance 2015: Applied Finance with R
May 29 and 30, 2015
University of Illinois at Chicago, IL, USA
The seventh annual R/Finance conference for applied finance using R will be held on May 29 and 30, 2015 in Chicago, IL, USA at the University of Illinois at Chicago. The conference will cover topics including portfolio management, time series analysis, advanced risk tools, high-performance computing, market microstructure, and econometrics. All will be discussed within the context of using R as a primary tool for financial risk management, portfolio construction, and trading. Over the past six years, R/Finance has included attendees from around the world. It has featured presentations from prominent academics and practitioners, and we anticipate another exciting line-up for 2015. This year will include invited keynote presentations by Emanuel Derman, Louis Marascio, Alexander McNeil, and Rishi Narang. We invite you to submit complete papers in pdf format for consideration. We will also consider one-page abstracts (in txt or pdf format) although more complete papers are preferred. We welcome submissions for both full talks and abbreviated "lightning talks." Both academic and practitioner proposals related to R are encouraged. All slides will be made publicly available at conference time. Presenters are strongly encouraged to provide working R code to accompany the slides. Data sets should also be made public for the purposes of reproducibility (though we realize this may be limited due to contracts with data vendors). Preference may be given to presenters who have released R packages. The conference will award two (or more) $1000 prizes for best papers. A submission must be a full paper to be eligible for a best paper award. Extended abstracts, even if a full paper is provided by conference time, are not eligible for a best paper award. Financial assistance for travel and accommodation may be available to presenters, however requests must be made at the time of submission. Assistance will be granted at the discretion of the conference committee. Please make your submission online at this link. The submission deadline is January 31, 2015. Submitters will be notified via email by February 28, 2015 of acceptance, presentation length, and financial assistance (if requested). Additional details will be announced via the R/Finance conference website as they become available. Information on previous years' presenters and their presentations are also at the conference website. For the program committee:

Gib Bassett, Peter Carl, Dirk Eddelbuettel, Brian Peterson, Dale Rosenthal,
Jeffrey Ryan, Joshua Ulrich

Some days I feel super lazy but I still would like to go on contributing translations to Debian.
Then, I leave the web translations a bit, and change to translate or review Debian package descriptions. It s something that anybody can do without any knowledge of translation tools, since it is a very simple web interface, as you will see. First you need to create a login account, then, login into the system. And then, go to the page of your mother language (in my case, Spanish, es ). You will see some introductory text, and the list of pending translations:

At the end of the page, there is the list of translations pending to review:
We should begin with this, so the work that other people already made arrives quickly its destination. And it s the easiest part, as you will see. Let s pick one of them (libvformat1-dev):
You see the short description in the original English, and the current translation (if there were changes from a former version, they are coloured too). I didn t know what the package libvformat1-dev does, but here s a nice opportunity to learn aobut it a bit :) The short description looks ok for me. Let s go on to the long description: It also looks correct for me. So I leave the text box as is, and go on until the bottom of the page:

and click Accept as is . That s all!! The system brings you back to the page with pending translations and reviews. Let s pick another one: totem

I found a typo and corrected some other words, so I updated the text in the translation box, left a message to the other translators in the comment box, and clicked Accept with changes . And iterate. When 3 translators agree in a translation, it becomes official, and its propagated to apt-cache, aptitude, synaptic, etc., and the website (packages.debian.org). This is the most difficult part (to get 3 reviews for each package description): many language teams are small, and their workforce is spread in many fronts: translations for the website, news and announcements, debconf templates (the messages that are shown to the user when a package is installed), the Debian installer, the documentation, the package descriptions So your help (even when you only review some translations from time to time) will be appreciated, for sure.
Filed under: Tools Tagged: Contributing to libre software, Debian, English, translations

Martin Pitt reported Debian bug #770000 on Tuesday November 18th, against the release.debian.org pseudo-package. Bug #760000 was reported as of August 30th: so there have been 10,000 bugs reported in 3 months minus 12 days. The bug rate increased quite significantly during the last weeks. We can suspect this is related to the release and the freeze (that triggers many unblock requests) I find it interesting that this bug is directly related to the release, directly related to systemd and originated from one of the systemd packages maintainers, if I'm right. So, I'll take this opportunity to publicly thank all people who have brought the systemd packages to what they are now, whether or not they're still maintaining the package. We've all witnessed that Debian if facing a strong social issue nowadays and I'm very deeply sad about this. I hope we'll be able to go through this without losing too many brilliant contributors, as it happened recently. Please prove me right and do The Right Thing for me to be able to continue this silly "round bug number" contest and still believe that, some day, bug #1000000 will really happen and I'm still there to witness it. Ah, and by the way, systemd bloody works on my system. I can't even remember when I switched to it. It Just Worked.

after see attic introduced in a discussion about bup, i figured out i could give it a try. it was answering two of my biggest concerns with bup:

• backup removal
• encryption

and seemed to magically out of nowhere and basically do everything i need, with an inline manual on top of it.

disclaimer Note: this is not a real benchmark! i would probably need to port bup and attic to liw's seivot software to report on this properly (and that would amazing and really interesting, but it's late now). even worse, this was done on a production server with other stuff going on so take results with a grain of salt.

procedure and results Here's what I did. I setup backups of my ridiculously huge ~/src directory on the external hard drive where I usually make my backups. I ran a clean backup with attic, than redid it, then I ran a similar backup with bup, then redid it. Here are the results:

anarcat@marcos:~$ sudo apt-get install attic # this installed 0.13 on debian jessie amd64
[...]
anarcat@marcos:~$ attic init /mnt/attic-test:
Initializing repository at "/media/anarcat/calyx/attic-test"
Encryption NOT enabled.
Use the "--encryption=passphrase keyfile" to enable encryption.
anarcat@marcos:~$ time attic create --stats /mnt/attic-test::src ~/src/
Initializing cache...
------------------------------------------------------------------------------
Archive name: src
Archive fingerprint: 7bdcea8a101dc233d7c122e3f69e67e5b03dbb62596d0b70f5b0759d446d9ed0
Start time: Tue Nov 18 00:42:52 2014
End time: Tue Nov 18 00:54:00 2014
Duration: 11 minutes 8.26 seconds
Number of files: 283910
                       Original size      Compressed size    Deduplicated size
This archive:                6.74 GB              4.27 GB              2.99 GB
All archives:                6.74 GB              4.27 GB              2.99 GB
------------------------------------------------------------------------------
311.60user 68.28system 11:08.49elapsed 56%CPU (0avgtext+0avgdata 122824maxresident)k
15279400inputs+6788816outputs (0major+3258848minor)pagefaults 0swaps
anarcat@marcos:~$ time attic create --stats /mnt/attic-test::src-2014-11-18 ~/src/
------------------------------------------------------------------------------
Archive name: src-2014-11-18
Archive fingerprint: be840f1a49b1deb76aea1cb667d812511943cfb7fee67f0dddc57368bd61c4bf
Start time: Tue Nov 18 00:05:57 2014
End time: Tue Nov 18 00:06:35 2014
Duration: 38.15 seconds
Number of files: 283910
                       Original size      Compressed size    Deduplicated size
This archive:                6.74 GB              4.27 GB            116.63 kB
All archives:               13.47 GB              8.54 GB              3.00 GB
------------------------------------------------------------------------------
30.60user 4.66system 0:38.38elapsed 91%CPU (0avgtext+0avgdata 104688maxresident)k
18264inputs+258696outputs (0major+36892minor)pagefaults 0swaps
anarcat@marcos:~$ sudo apt-get install bup # this installed bup 0.25
anarcat@marcos:~$ free && sync && echo 3   sudo tee /proc/sys/vm/drop_caches && free # flush caches
anarcat@marcos:~$ export BUP_DIR=/mnt/bup-test
anarcat@marcos:~$ bup init
D p t Git vide initialis  dans /mnt/bup-test/
anarcat@marcos:~$ time bup index ~/src
Indexing: 345249, done.
56.57user 14.37system 1:45.29elapsed 67%CPU (0avgtext+0avgdata 85236maxresident)k
699920inputs+104624outputs (4major+25970minor)pagefaults 0swaps
anarcat@marcos:~$ time bup save -n src ~/src
Reading index: 345249, done.
bloom: creating from 1 file (200000 objects).
bloom: adding 1 file (200000 objects).
bloom: creating from 3 files (600000 objects).
Saving: 100.00% (6749592/6749592k, 345249/345249 files), done.
bloom: adding 1 file (126005 objects).
383.08user 61.37system 10:52.68elapsed 68%CPU (0avgtext+0avgdata 194256maxresident)k
14638104inputs+5944384outputs (50major+299868minor)pagefaults 0swaps
anarcat@marcos:attic$ time bup index ~/src
Indexing: 345249, done.
56.13user 13.08system 1:38.65elapsed 70%CPU (0avgtext+0avgdata 133848maxresident)k
806144inputs+104824outputs (137major+38463minor)pagefaults 0swaps
anarcat@marcos:attic$ time bup save -n src2 ~/src
Reading index: 1, done.
Saving: 100.00% (0/0k, 1/1 files), done.
bloom: adding 1 file (1 object).
0.22user 0.05system 0:00.66elapsed 42%CPU (0avgtext+0avgdata 17088maxresident)k
10088inputs+88outputs (39major+15194minor)pagefaults 0swaps

Disk usage is comparable:

anarcat@marcos:attic$ du -sc /mnt/*attic*
2943532K        /mnt/attic-test
2969544K        /mnt/bup-test

People are encouraged to try and reproduce those results, which should be fairly trivial.

Observations Here are interesting things I noted while working with both tools:

• attic is Python3: i could compile it, with dependencies, by doing apt-get build-dep attic and running setup.py - i could also install it with pip if i needed to (but i didn't)
• bup is Python 2, and has a scary makefile
• both have an init command that basically does almost nothing and takes little enough time that i'm ignoring it in the benchmarks
• attic backups are a single command, bup requires me to know that i first want to index and then save, which is a little confusing
• bup has nice progress information, especially during save (because when it loaded the index, it knew how much was remaining) - just because of that, bup "feels" faster
• bup, however, lets me know about its deep internals (like now i know it uses a bloom filter) which is probably barely understandable by most people
• on the contrary, attic gives me useful information about the size of my backups, including the size of the current increment
• it is not possible to get that information from bup, even after the fact - you need to du before and after the backup
• attic modifies the files access times when backing up, while bup is more careful (there's a pull request to fix this in attic, which is how i found out about this)
• both backup systems seem to produce roughly the same data size from the same input

Summary attic and bup are about equally fast. bup took 30 seconds less than attic to save the files, but that's not counting the 1m45s it took indexing them, so on the total run time, bup was actually slower. attic is also (almost) two times faster on the second run as well. but this could be within the margin of error of this very quick experiment, so my provisional verdict for now would be that they are about as fast. bup may be more robust (for example it doesn't modify the atimes), but this has not been extensively tested and is more based with my familiarity with the "conservatism" of the bup team rather than actual tests. considering all the features promised by attic, it makes for a really serious contender to the already amazing bup.

Next steps The properly do this, we would need to:

• include other software (thinking of Zbackup, Burp, ddar, obnam, rdiff-backup and duplicity)
• bench attic with the noatime patch
• bench dev attic vs dev bup
• bench data removal
• bench encryption
• test data recovery
• run multiple backup runs, on different datasets, on a cleaner environment
• ideally, extend seivot to do all of that

Note that the Burp author already did an impressive comparative benchmark of a bunch of those tools for the burp2 design paper, but it unfortunately doesn't include attic or clear ways to reproduce the results.

Over the weekend the NetSurf developers met to make a concentrated effort on improving the browser. This time we were kindly hosted by Codethink in their Manchester office in a pleasant environment with plenty of refreshments.

Five developers managed to attend in person from around the UK: Michael Drake, John-Mark Bell, Daniel Silverstone, Rob Kendrick and Vincent Sanders. We also had Chris Young providing some bug fixes remotely.

We started the weekend by discussing all the thorny core issues that had been put on the agenda and ensuring the outcomes were properly noted. We also held the society AGM which was minuted by Daniel.

The emphasis of this weekend was very much on planning and doing the disruptive changes we had been putting off until we were all together.

John-Mark and myself managed to change the core build system as used by all the libraries to using standard triplets to identify systems and use the gnu autoconf style of naming for parameters (i.e. HOST, BUILD and CC being used correctly).

This was accompanied by improvements and configuration changes to the CI system to accommodate the new usage.

Several issues from the bug tracker were addressed and we put ourselves in a stronger position to address numerous other usability problems in the future.

We managed to pack a great deal into the 20 hours of work on Saturday and Sunday although because we were concentrating much more on planning and infrastructure rather than a release the metrics of commits and files changed were lower than at previous events.