How corporations killed the web I have read with fascination what we would have called before a blog post, except it was featured on The Guardian: Iran's blogfather: Facebook, Instagram and Twitter are killing the web The "blogfather" is Hossein Derakshan or h0d3r, an author from Teheran that was jailed for almost a decade for his blogging. The article is very interesting both because it shows how fast things changed in the last few years, technology-wise, but more importantly, how content-free the web have become, where Facebook's last acquisition, Instagram, is not even censored by Iran. Those platforms have stopped being censored, not because of democratic progress but because they have become totally inoffensive (in the case of Iran) or become a tool of surveillance for the government and targeted advertisement for companies (in the case of, well, most of the world). This struck a chord, personally, at the political level: we are losing control of the internet (if we ever had it). The defeat isn't directly political: we have some institutions like ICANN and the IETF that we can still have an effect on, even if only at the technological level. The defeat is economic, and, of course, through economy comes enormous power. That defeat meant that we have first lost free and open access to the internet (yes, dialup used to be free) and then free hosting of our content (no, Google and Facebook are not free, you are the product). This marked a major change in the way content is treated online. H0d3r explains this as the shift from a link-based internet to a stream-based internet, a "deparure from a books-internet towards a television-internet". I have been warning about this "television-internet" in my talks and conversation for a while and with Netflix taking the crown off Youtube (and making you pay for it, of course), we can assuredly say that H0d3r is right and the television, far from disappearing, is finally being resurrected and taking over the internet.

The Downloadable internet and open standards But I would like to add to that: it is not merely that we had "links" before. We had, and still have, open standards. This made the internet "downloadable" (and by extension, uploadable) and decentralized. (In fact, I still remember my earlier days on the web when I would actually download images (as in "right-click" and "Save as..." images, not just have the browser download and display it on the fly). I would download images because they were big! It could take a minute or sometimes more to download images on older modems. Later, I would do the same with music: I would download WAV files before the rise of the MP3 format, of which I ended up building a significant collection (just fair use copies from friends and owned CDs, of course) and eventually video files.) The downloadable internet is what still allows me to type this article in a text editor, without internet access, while reading H0d3r's blog post on my e-reader, because I downloaded his article off an RSS feed. It is what makes it possible for anyone to download a full copy of this blog post and connected web pages as a git repository and this way get the full history of modifications on all the pages, but also be able to edit it offline and push modifications back in. Wikipedia is downloadable (there are even offline apps for your phone). Open standards like RSS feeds and HTML are downloadable. Heck, even the Internet Archive is downloadable (and I mean, all of it, not just the parts you want), surprisingly enough.

The app-based internet and proprietary software App-based websites like Google Plus and Facebook are not really downloadable. They are meant to be browsed through an app, so what you actually see through your web browser is really more an application, downloaded software than a downloaded piece of content. If you turn off Javascript, you will see that visiting Facebook actually shows no content: everything is downloaded on the fly by an application itself downloaded, on the fly, by your browser. In a way, your browser has become an operating system that runs proprietary, untrusted and unverified applications from the web. (The software is generally completely proprietary, except some frameworks that are published as free software in what looks like the lenient act of a godly king, but is actually more an economic decision of a clever corporation which outsources, for free, R&D and testing to the larger free software community. The real "secret sauce" is basically always proprietary, if only so that we don't freak out on stuff like PRISM that reports everything we do to the government.) Technology is political. This new "app design" is not a simple optimization or an cosmetic accident of a fancy engineer: by moving content through an application, Facebook, Twitter and the like can see exactly what you do on a web page, what you actually read (as opposed to what you click on) and how long. By adding a proprietary interface between you and the content online, the advertisement-surveillance complex can track every move you make online. This is a very fine-tuned surveillance system, and because of the App, you cannot escape it. You cannot share the content outside of Facebook, as you can't download it. Or at least, it's not obvious how you can. Projects like youtube-dl are doing an amazing job reverse-engineering what is becoming the proprietary Youtube streaming protocol, which is constantly changing and is not really documented. But it's a hack: it's a Sisyphus struggle which is bound to fail, and it does, all the time, until we figure out how to either turn those corporations into good netizens respecting and contributing to open standards (unlikely) or destroy those corporations (most likely). You are trapped in their walled garden. No wonder internet.org is Facebook only: for most people nowadays, the internet is the web, and the web is Facebook, Twitter and Google, or an iPad with a bunch of apps, each their own cute little walled garden, crafted just for you. If you think you like the Internet, you should really reconsider what you are watching, what you are consuming, or rather, how it is consuming you. There are alternatives. Facebook is a though nut to crack for free software activists because we lack the critical mass. But Facebook it is also an addiction for a lot of people, and spending less time on that spying machine could be a great improvement for you I am sure. For everything else, we have good free software alternatives and open standards, use them. "Big brother ain't watching you, you're watching him." - CRASS, Nineteen Eighty Bore (audio)

The above sign was at the Melbourne Docks in December 2014 when I was returning from a cruise. I have no idea why there are 3 men and 1 woman on the sign (and a dock worker was also surprised when I explained why I was photographing it). I wonder whether a sign that had 3 women and 1 man would ever have been installed or not noticed if it was installed. At the start of the first day of LCA 2015 the above was displayed at the keynote as a flow-chart for whether someone should ask a question at a lecture. Given that the first real item in the list is that a question should fit in a tweet I think it was inspired by my blog post about the length of conference questions [1]. At the introduction to the Astronomy Miniconf the above slide was displayed. In addition to referencing the flow-chart for asking questions it recommends dimming laptop screens (among other things). The above sign was at a restaurant in Auckland in January 2015. I thought that sort of sexist joke went out of fashion a few decades ago. The above photo is from a Melbourne department store in February 2015. Why gender a nerf gun? That just doesn t make sense. Also it appeared that the only nerf crossbow was the purple/pink one, is a crossbow considered feminine nowadays? The above picture is a screen-shot of one of the Talking Angela series of Android games from March. Appropriating the traditional clothing of marginalised groups is a bad thing. People of Native American heritage who want to wear their traditional clothing face discrimination when they do so, when white people play dress-up in clothing that is a parody of Native American style it s really offensive. The site Racialicious.com has a tag for articles about appropriation [2]. The above was in a library advertising an Ebook reader. In this case they didn t even have pointlessly gendered products they just had pointlessly gendered adverts for the same product. They also perpetuate the myth that only girls read vampire books and only boys read about space. Also why is the girl lying down to read while the boy is sitting up? Above is an Advent calendar on sale in a petrol station. Having end of year holiday presents that have nothing to do with religious festivals makes sense. But Advent is a religious observance. I think this would be a better candidate for war on Christmas paranoia than a coffee cup of the wrong colour. The above photo is of boys and girls pipette suckers. Pointlessly gendered recreational products like Nerf guns is one thing, but I think that doing it to scientific equipment is a bigger problem. Are scientists going to stop work if they can t find a pipette sucker of the desired gender? Is worrying about this going to distract them from their research (really bad if working with infectious or carcinogenic solutions). The Integra advertising claims to be doing this to promote breast cancer research which is also bogus. Here is a Sociological Images article about the problems of using pink to market breast cancer research [3] and the Sociological Images post about pinkwashing (boobies against breast cancer) is also worth reading [4]. As an aside I made a mistake in putting a pipette sucker over the woman s chest in that picture. The way that Integra portreyed her chest is relevant to analysis of this advert. But unfortunately I didn t photograph that. Here is a link to my sociological images post from 2014 [5].

• [1] http://etbe.coker.com.au/2014/01/06/length-conference-questions/
• [2] http://www.racialicious.com/tag/cultural-appropriation/
• [3] http://thesocietypages.org/socimages/2013/10/10/breast-cancer-marketing-has-a-pink-problem/
• [4] http://thesocietypages.org/socimages/2013/10/11/boobies-against-breast-cancer/
• [5] http://etbe.coker.com.au/2014/04/21/sociological-images-2014/

I recently dug through my history of involvement with TeX (Live), and found out that in January there are a lot of anniversaries I should celebrate: 14 years ago I started building binaries for TeX Live, 11 years ago I proposed the packaging TeX Live for Debian, 10 years ago the TeX Live packages entered Debian. There are other things to celebrate next year (2017), namely the 10 year anniversary of the (not so new anymore) infrastructure in short tlmgr of TeX Live packaging, but this will come later. In this blog post I want to concentrate on my involvement in TeX Live and Debian. Those of you not interested in boring and melancholic look-back onto history can safely skip reading this one. For those a bit interested in the history of TeX in Debian, please read on. Debian releases and TeX systems The TeX system of choice has been for long years teTeX, curated by Thomas Esser. Digging through the Debian Archive and combining it with changelog entries as well as personal experiences since I joined Debian, here is a time line of TeX in Debian, all to my best knowledge.

Date	Version	Name	teTeX/TeX Live	Maintainers
1993-96	<1	?	?	Christoph Martin
6/1996	1.1	Buzz	?
12/1996	1.2	Rec	?
6/1997	1.3	Bo	teTeX 0.4
7/1998	2.0	Ham	teTeX 0.9
3/1999	2.1	Slink	teTeX 0.9.9N
8/2000	2.2	Potato	teTeX 1.0
7/2002	3.0	Woody	teTeX 1.0
6/2005	3.1	Sarge	teTeX 2.0	Atsuhito Kohda
4/2007	4.0	Etch	teTeX 3.0, TeX Live 2005	Frank K ster, NP
2/2009	5.0	Lenny	TeX Live 2007	NP
2/2011	6.0	Squeeze	TeX Live 2009
5/2013	7.0	Whezzy	TeX Live 2012
4/2015	8.0	Jessie	TeX Live 2014
???	???	Stretch	TeX Live 2015

The history of TeX in Debian is thus split more or less in 10 years teTeX, and 10 years TeX Live. While I cannot check back to the origins, my guesses are that already in the very first releases (te)TeX was included. The first release I can confirm (via the Debian archive) shipping teTeX is the release Bo (June 1997). Maintainership during the first 10 years showed some fluctuation: The first years/releases (till about 2002) were dominated by Christoph Martin with Adrian Bunk and few others, who did most packaging work on teTeX version 1. After this Atsuhito Kohda with help from Hilmar Preusse and some people brought teTeX up to version 2, and from 2004 to 2007 Frank K ster, again with help of Hilmar Preusse and some other, took over most of the work on teTeX. Other names appearing throughout the changelog are (incomplete list) Julian Gilbey, Ralf Stubner, LaMont Jones, and C.M Connelly (and many more bug-reporters and fixers). Looking at the above table I have to mention the incredible amount of work that both Atsuhito Kohda and Frank K ster have put into the teTeX packages, and many of their contributions have been carried over into the TeX Live packages. While there haven t been many releases during their maintainership, their work has inspired and supported the packaging of TeX Live to a huge extend. Start of TeX Live I got involved in TeX Live back in 2002 when I started building binaries for the alpha-linux architecture. I can t remember when I first had the idea to package TeX Live for Debian, but here is a time line from my first email to the Debian Developers mailing list concerning TeX Live, to the first accepted upload:

Date	Subject/Link	Comment
2005-01-11	binaries for different architectures in debian packages	The first question concerning packaging TeX Live, about including pre-built binaries
2005-01-25	Debian-TeXlive Proposal II	A better proposal, but still including pre-built binaries
2005-05-17	Proposal for a tex-base package	Proposal for tex-base, later tex-common, as basis for both teTeX and TeX live packages
2015-06-10	Bug#312897: ITP: texlive	ITP bug for TeX Live
2005-09-17	Re: Take over of texinfo/info packages	Taking over texinfo which was somehow orphaned started here
2005-11-28	Re: texlive-basic_2005-1_i386.changes REJECTED	My answer to the rejection by ftp-master of the first upload. This email sparked a long discussion about packaging and helped improve the naming of packages (but not really the packaging itself).
2006-01-12	Upload of TeX Live 2005-1 to Debian	The first successful upload
2006-01-22	Accepted texlive-base 2005-1 (source all)	TeX Live packages accepted into Debian/experimental

One can see from the first emails that at that time I didn t have any idea about Debian packaging and proposed to ship the binaries built within the TeX Live system on Debian. What followed was first a long discussion about whether there is any need for just another TeX system. The then maintainer Frank K ster took a clear stance in favor of including TeX Live, and after several rounds of proposals, tests, rejections and improvements, the first successful upload of TeX Live packages to Debian/experimental happened on 12 January 2006, so exactly 10 years ago. Packaging Right from the beginning I used a meta-packaging approach. That is, instead of working directly with the source packages, I wrote (Perl) scripts that generated the source packages from a set of directives. There were several reasons why I choose to introduce this extra layer:

• The original format of the TeX Live packaging information (tpm) were xml files that were parsed with an XML parser (libxml). I guess (from what I have seen over the years) I was the only one ever properly parsing these .tpm files for packaging.
• TeX Live packages were often reshuffled, and Debian package name changed, which would have caused a certain level of pain for the creation of original tar files and packaging.
• Flexibility in creating additional packages and arbitrary dependencies

Till now I am not 100% sure whether it was the best idea, but the scripts remain in place till now, only adapted to the new packaging paradigm in TeX Live (without xml) and adding new functionality. This allows me to just kick off one script that does do all the work, including building .orig.tar.gz, source packages, binary packages. For those interested to follow the frantic activity during the first years, there is a file CHANGES.packaging which for the years from 2005 to 2011 documents very extensively the changes I made in these years. I don t want to count the hours the went into all this Development over the years TeX Live 2005 was just another TeX system but not the preferred one in Debian Etch and beyond. But then in May 2006, Thomas Esser announced the end of development of teTeX, which cleared the path for TeX Live as main TeX system in Debian (and the world!). The next release of Debian, Lenny (1/2009), already carried only TeX Live. Unfortunately it was only TeX Live 2007 and not 2008, mostly due to me having been involved in rewriting the upstream infrastructure based on Debian package files instead of the notorious xml files. This took quite a lot of attention and time from Debian away to upstream development, but this will be discussed in a different post. Similarly, the release of TeX Live included in Debian Squeeze (released 2/2011) was only TeX Live 2009 (instead of 2010), but since then (Wheezy and Jessie) the releases of TeX Live in Debian were always the latest released ones. Current status Since about 2013 I am trying to keep a regular schedule of new TeX Live packages every month. These helps me to keep up with the changes in upstream packaging and reduces the load of packaging a new release of TeX Live. It also bring to users of unstable and testing a very up-to-date TeX system, where packages at most lack 1 month of behind the TeX Live net updates. Future As most of the readers here know, besides caring for TeX (Live) and related packages in Debian, I am also responsible for the TeX Live Manager (tlmgr) and most of upstream s infrastructure including network distribution. Thus, my (spare, outside work) time needs to be distributed between all these projects (and some others) which leaves less and less time for Debian packaging. Fortunately the packaging is in a state that making regular updates once a month is less of a burden, since most steps are automatized. What is still a bit of a struggle is adapting the binary package (src:texlive-bin) to new releases. But also this has become simpler due to less invasive changes over the years. All in all, I don t have many plans for TeX Live in Debian besides keeping the current system running as it is. Search for and advise to future maintainers and collaborators I would be more than happy if new collaborators appear, with fresh ideas and some spare time. Unfortunately, my experience over these 10 years with people showing up and proposing changes (anyone remembers the guy proposing a complete rewrite in ML or so?) is that nobody really wants to invest time and energy, but search for quick solutions. This is not something that will work with a package like TeX Live, sized of several gigabyte (the biggest in the Debian archive), and complicated inner workings. I advise everyone being interested in helping to package TeX Live for Debian, to first install normal TeX Live from TUG, get used to what actions happen during updates (format rebuilds, hyphenation patters, map file updates). One does not need to have a perfect understanding of what exactly happens down there in the guts (I didn t have in the beginning, either), but if you want to help packaging and never heard about what format dumps or map files are, then this might be a slight obstacle. Conclusion TeX Live is the only TeX system in wide use across lots of architectures and operating systems, and the only comparable system, MikTeX, is Windows specific (also there are traces of ports to Unix). Backed by all the big user groups of TeX, TeX Live will remain the prime choice for the foreseeable future, and thus also TeX Live in Debian.

Retweet is self-hosted app coded in Python 3 allowing to retweet all the statuses from a given Twitter account to another one. Lots of filters can be used to retweet only tweets matching given criterias.

• Retweet on Github (stars appreciated if you like it/use it :) )
• Retweet official documentation

Retweet 0.8 is available on the PyPI repository and is already in the official Debian unstable repository. Retweet is in production already for Le Journal Du hacker , a French FOSS community website to share and relay news and LinuxJobs.fr , a job board for the French-speaking FOSS community. The new features of the 0.8 allow Retweet to manage the tweets given how old they are, retweeting only if :

• they are older than a user-specified duration with the parameter older_than
• they are younger than a user-specified duration with the parameter younger_than

Retweet is extensively documented, have a look at the official documentation to understand how to install it, configure it and use it. What about you? does Retweet allow you to develop your Twitter account? Let your comments in this article.

As of today, python3 -> python3.5. There s a bit of a transition, but fortunately because most extensions are packaged to build for all supported python3 versions, we started this transition at about 80% done. Thank you do the maintainers that have done that. It makes these transitions much smoother. As part of getting ready for this transition, I reviewed all the packages that needed to be rebuilt for this stage of the transition to python3.5 and a few common errors stood out:

1. For python3 it s python3:Depends not python:Depends .
2. Do not use python3:Provides . This has never been used for python3 (go read the policy if you doubt me [1]).
3. Almost for sure do not use python:Provides . The only time it should still be used is if some package depends on python2.7-$PACKAGE. It would surprise me if any of these are left in the archive. If so, since python2.7 is the last python2, then they should be adjusted. Work with the maintainer of such an rdepend and once it s removed, then drop the provides.
4. Do not use XB-Python-Version. We no longer use this to manage transitions (there won t be any more python transitions).
5. Do not use XB-Python3-Version. This was never used.

Now that we have robust transition trackers [2], the purpose for which XB-Python-Version is obsolete. In other news, pysupport was recently removed from the archive. This means that, following the previous removal of pycentral, we finally have one and only one python packaging helper (dh-python) that supports both python and python3. Thanks to everyone who made that possible. [1] https://www.debian.org/doc/packaging-manuals/python-policy/ [2] https://release.debian.org/transitions/html/python3.5.html

Last week we finally ended with a proper Debian informal meeting at Seville.

A total amount of 9 people attended, 3 of them DDs (Aurelien Jarno, Guillem Jover, Ana Guerrero) and 1 DM (me).

The meeting started with the usual "personal references" round, and then topics ranged from how to get more people involved with Debian, to GSoC-like programs discussions, and some Debian anecdotes as well.

There were also talks about when and how future meetings should be.

This meeting was hosted by http://www.plan4d.eu/, thanks to Pablo Neira (Netfilter project head).

Some pics of the moment:

The third update in the 0.12.* series of Rcpp arrived on the CRAN network for GNU R earlier today, and has been pushed to Debian. It follows the 0.12.0 release from late July, the 0.12.1 release in September, and the 0.12.2 release in November making it the seventh release at the steady bi-montly release frequency. This release is somewhat more of a maintenance release addressing a number of small bugs and nuisances without adding any new features. Rcpp has become the most popular way of enhancing GNU R with C or C++ code. As of today, 553 packages on CRAN depend on Rcpp for making analytical code go faster and further. That is up by more than fourty packages from the last release in November. Once again, we have new first-time contributors. Kazuki Fukui corrected an issue he encountered when having CLion re-formatted some code for him. Joshua Pritikin corrected a constructor initialization. Of course, we also had several pull reports from regular contributors -- see below for a detailed list of changes extracted from the NEWS file.

Changes in Rcpp version 0.12.3 (2016-01-10)

• Changes in Rcpp API:
  • Const iterators now CharacterVector now behave like regular iterators (PR #404 by Dan fixing #362).
  • Math operators between matrix and scalars type have been added (PR #406 by Qiang fixing #365).
  • A missing std::hash function interface for Rcpp::String has been addded (PR #408 by Qiang fixing #84).
• Changes in Rcpp Attributes:
  • Avoid invalid function names when generating C++ interfaces (PR #403 by JJ fixing #402).
  • Insert additional space around & in function interface (PR #400 by Kazuki Fukui fixing #278).
• Changes in Rcpp Modules:
  • The copy constructor now initialized the base class (PR #411 by Joshua Pritikin fixing #410)
• Changes in Rcpp Repository:
  • Added a file Contributing.md providing some points to potential contributors (PR #414 closing issue #413)

Thanks to CRANberries, you can also look at a diff to the previous release. As always, even fuller details are on the Rcpp Changelog page and the Rcpp page which also leads to the downloads page, the browseable doxygen docs and zip files of doxygen output for the standard formats. A local directory has source and documentation too. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Feed2tweet is a self-hosted Python app to send you RSS feed to Twitter. A long descriptions about why and how to use it is available in my last post about it.

• Feed2tweet on Github (stars appreciated :) )
• The official documentation of Feed2tweet on Readthedocs.

Feed2tweet is in production for Le Journal du hacker, a French Hacker News-style FOSS website. Feed2tweet 0.2 brings a lot of new command line options, contributed by Antoine Beaupr @theanarcat. Taking a short extract of the Feed2tweet 0.2 changelog:

• new command line option -r or dry-run to simulate execution of Feed2tweet
• new command line option -d or debug to increase verbosity of the execution of Feed2tweet
• new command line option -v or verbose to follow the execution of Feed2tweet
• new command line option cachefile to get the path of the cache file
• new command line option hashtaglist to get the path of the hash tag composed by multiple words
• new command line option -r or rss to get the uri of the RSS feed

Lots of issues from the previous project was also fixed. Using Feed2tweet? Send us bug reports/feature requests/push requests/comments about it!

I've been using bridge networking and tap networking for containers and virtual machines on my system. Configuration for bridge network which I use to connect containers was configured using /etc/network/interfaces file as shown below. Basically I setup masquerading and IP forwarding when network comes up using this, so all my containers and virtual machines can access internet. This can be simply done using systemd-networkd with couple of lines, yes couple of lines. For this to work first you need to enable systemd-networkd. Now I need to write 2 configuration file for the above bridge interface under /etc/systemd/network. One file is natbr0.netdev which configures the bridge and the natbr0.network which configures IP address and other stuff for the bridge interface. The IPForward in above configuration is actually redundant, when I set IPMasquerade it automatically enables IPForward. So these configuration is equivalent of what I did in my interfaces file. It also avoids me doing additional iptables usage to add masquerading rules. This pretty much simplifies handling of virtual network devices. There are many other things which can you do with systemd-networkd, like running a DHCPServer on the interface and many other things. I suggest you to read manual pages on systemd.network(5) and systemd.netdev(5). systemd-networkd allows you configure all type of virtual networking devices and actual network interfaces. I've not myself used it to handle actual network interfaces yet.

Holger Levsen writes about the public transport map of Managua, Nicaragua, which is, according to him, the first detailed map of Managua s bus network:

If you haven t been to Managua, you might not be able to immediatly appreciate the usefulness of this. Up until now, there has been no map nor timetable for the bus system, which as you can see now easily and from far away, is actually quite big and is used by 80% of the population in a city, where the streets still have no names.

Having had a look at the map they produced, I have to admit I quite liked it: MapaNica, the community behind said map, are raising funds to make lives of locals easier by publishing a printed version of the map and distributing it. They have already raised more than $3300 of their $7500 goal. Every further donation will help them print more maps. Please go to support.mapanica.net and support their initiative!

Please support the public transportation map of Manugua Some of you might remember DebConf12 in Managua, Nicaragua and the very friendly and helpful locals, who recently contacted me to tell me about their new project, so that I share this on planet Debian: A local community of openstreet map enthusiasts, of which some were involved in DebConf12, has collected for the first time detailed information about Managua's bus network! To bring their efforts further, they will now print these maps on paper, so that even more people can use them in their daily lives. If you haven't been to Managua, you might not be able to immediatly appreciate the usefulness of this. Up until now, there has been no map nor timetable for the bus system, which as you can see now easily and from far away, is actually quite big and is used by 80% of the population in a city, where the streets still have no names. If this made you curious (or just brought back happy memories from 2012) please go to http://support.mapanica.net and donate some money - their campaign is running for 3 more weeks and currently they have already raised 3300 USD, enough to print some maps, but 4200 USD short of their goal. Every further donation will help to print some more maps, even something as little as 20 USD or EUR will help people in their real lives to better understand the beast of Managua's bus route network.

During my MSc program, I was lucky to squeeze into Michael Collin s NLP class. We used his Coursera course as part of the program, which I d highly recommend. Recently I decided to review my NLP studies and I believe the best way to learn or relearn a subject is to teach it. This is one in a series of 4 posts with a walk-through of the algorithms we implemented during the course. I ll provide links to my code hosted on Github. Disclaimer: Before taking this NLP course, the only thing I knew about Python was that it s the one without curly brackets . I learned Python on the go while implementing these algorithms. So if I did anything against Python code conventions or flat-out heinous, I apologize and thank you in advance for your understanding. Feel free to write and let me know.

The Concept To quote Wikipedia, Named-entity recognition (I ve always known it as tagging) is a subtask of information extraction that seeks to locate and classify elements in text into predefined categories such as the names of persons, organizations, locations, expressions of times, quanitites, monetary values, percentages, etc. For example, the algorithm receives as input some text Bill Gates founded Microsoft in 1975. and outputs Bill Gates[person] founded Microsoft[organization] in 1975[date]. Off the top of my head, some useful applications are document matching (ex. a document containing Gates[person] may not be on the same topic as one containing gates[object]) and query searches. I m sure there are lots more, if you check out Collin s Coursera course he may discuss this in greater depth.

The Requirements Development data: The file ner_dev.dat provided by prof. Michael Collins has a series of sentences separated by an empty line, one word per line. Training data: The file ner_train.dat provided by prof. Michael Collins has a series of sentences separated by an empty line, one word and tag per line, speparated by a space. Word-tag count data: The file ner.counts has the format [count] [type of tag] [label] [word]. The tags used are RARE, O, I-MISC, I-PER, I-ORG, I-LOC, B-MISC, B-PER, B-ORG, B-LOC. The tag O means it s not an NE. This file is generated by count_freqs.py, a script provided by prof. Michael Collins. Run count_freqs.py on the training data ner_train.dat

The Algorithm Python code: viterbi.py Usage: python viterbi.py ner.counts ngram.counts [input_file] > [output_file] Summary: The Viterbi algorithm finds the maximum probability path for a series of observations, based on emission and transition probabilities. In a Markov Process, emission is the probability of an output given a state and transition is the probability of transitioning to the state given the previous states. In our case, the emission parameter e(x y) = the probability of the word being x given you attributed tag y. If your training data had 100 counts of person tags, one of which is the word London (I know a guy who named his kid London), e( London person ) = 0.01. Now with 50 counts of location tags, 5 of which are London , e( London location ) = 0.1 which clearly trumps 0.01. The transition parameter q(y_i y_i-1, y_i-2) = the probability of putting tag y in position i given it s two previous tags. This is calculated by Count(trigram)/Count(bigram). For each word in the development data, he Viterbi algorithm will associate a score for a word-tag combo based on the emission and transition parameters it obtained from the training data. It does this for every possible tag and sees which is more likely. Clearly this won t be 100% correct as natural language is unpredictable, but you should get pretty high accuracy. Optional Preprocessing Re-label words in training data with frequency < 5 as RARE - This isn t required, but useful. Re-run count_freqs.py if used. Python code: label_rare.py Usage: python label_rare.py [input_file] Pseudocode:

1. Uses Python Counter to obtain word counts in [input_file]; removes all word-count pairs with count < 5, store remaining pairs in a dictionary named rare_words.
2. Iterates through each line in [input file], checks if word is in rare words dictionary, if so, replaces word with RARE.

Step 1. Get Count(y) and Count(x~y) Python code: emission_counts.py Pseudocode:

1. Iterate through each line in ner.counts file and store each word-label-count combo in a dictionary count_xy and update the dictionary of count_y. For example count_xy[Peter][I-PER] returns the number of times the word Peter was labeled I-PER in the training data and count_y[I-PER] the total number of I-PER tags. The dictionary count_y contains 8 items, one for each label ( RARE , O, I-MISC, I-PER, I-ORG, I-LOC, B-MISC, B-PER, B-ORG, B-LOC);
2. Return count_xy, count_y

Step 2. Get bigram and trigram counts Python code: transition_counts.py Pseudocode:

1. Iterate through each line in the n-gram_counts file
2. If the line contains 2-GRAM add an item to the bigram_counts dictionary using the bigram (two space-separated labels following the tag type 2-gram ) as key, count as value. This dictionary will contain Count(y_i-2,y_i-1).
3. If the line contains 3-GRAM , add an item to the trigram_counts dictionary using the trigram as key, count as value. This dictionary will contain Count(y_i-2, y_i-1, y_i).
4. Return dictionaries of bigram and trigram counts.

Step 3. Viterbi (For each line in the [input_file]):

1. If the word was seen in training data (present in the count_xy dictionary), for each of the possible labels for the word:
2. Calculate emission = count_xy[word][label] / float(count_y[label]
3. Calculate transition = trigram_counts[trigram])/float(bigram_counts[bigram] Note: y_i-2 = *, y_i-1 = * for the first round
4. Set probability = emission x transition
5. Update max(probability) and arg max if needed. 2 If the word was not seen in the training data:
6. Calculate emission = count xy[RARE][label] / float(count y[label].

7. Calculate q(yi

   yi-1, yi-2) = trigram counts[trigram])/float(bigram counts[bigram]. Note: yi-2 = , yi-1 = for the first round

8. Set probability = emission transition
9. Update max(probability) if needed, arg max = RARE
10. Write arg max and log(max(probability)) to output file.
11. Update y_i-2, y_i-1.
12. Update y_i-2, y_i-1.

Evaluation Prof. Michael Collins provided an evaluation script eval_ne_tagger.py to verify the output of your Viterbi implementation. Usage: python eval_ne_tagger.py ner_dev.key [output_file]

List all posts by Authors, nested Categories and Titles is a WordPress Plugin I wrote to fix a menu issue I had during a complex website development. It has been included in the official WordPress Plugin repository. The Plugin is particularly suitable to all multi-nested categories and multi-authors websites handling a large number of posts and complex nested category layout (i.e.: academic papers, newpapers articles, etc). This plugin allows the user to place a shortcode into any page and get rid of a long and nested menu/submenu to show all site s posts. A selector in the page will allow the reader to select grouping by Category/Author/Title. You can also manage to install a tab plugin (i.e.: Tabby Responsive Tabs) and arrange each group on its specific tab. Output grouped by Category will look like:

CAT1
    post1                       AUTHOR
    SUBCAT1
        post2                   AUTHOR
        post3                   AUTHOR
        SUBCAT2
            post4               AUTHOR
            ...
            ...

while in the Author grouping mode, it is:

AUTHOR1
  post1               [CATEGORY]
  post2               [CATEGORY]
AUTHOR2
  post1               [CATEGORY]
  post2               [CATEGORY]
.....

The plugin installs a new menu ACT List Shortcodes in Admin->Tools. The tool is a helper to automatically generate the required shortcode. It will parse the options and display the string to be copied and pasted into any page. The Plugin is holding a GPL2 license and it can be downloaded from its page on WP Plugins.

A new release, now at version number 0.6.9, of the digest package is now on CRAN. I also just prepared the Debian upload. This release is very large part the work of Thierry Onkelinx who added stable sha1 support in a new function sha1(). Here, stable means that numerically equivalent numbers (in the sense of the semi-famous R FAQ entry 7.31) result in identical hashes. This is useful for hashing results from numerical analysis---where the representation may differ bitwise between, say, 32 and 64-bit platforms. We started to write a little more about this in a (at this very point still rather unfinished) little vignette. We also had a nice pull request from fellow Rcpp hacker Qiang Kou who updated the code to use XLENGTH so that large vectors can be supported. CRANberries provides the usual summary of changes to the previous version.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

I was quite busy in December 2015, especially with my new Python projects related to automatize Twitter actions. Have a look at my Github home for more information ! 1. Personal projects

• db2twitter 0.5 released (Github stars appreciated :) )- new option to circle through a ring of the last N tweets in order to resend them on a regular basis
• new project Twitterwatch (Github stars appreciated :) )- find how old is the last sent tweet and triggers an alert if the user-defined duration is exceeded, allowing to know that an automated toolchain of sending tweets is broken
• new project Feed2tweet (Github stars appreciated :) ) send tweets from a RSS feed. It s a fork from rss2twitter, improving hashtag management, configuration options and documentation.

2. Journal du hacker

• setting up the Journal du hacker blog (PluXML)
• feedDiasp allowing RSS feeds to Diaspora* was fixed. The issue was from the diaspy dependency. I could not wait for the upstream (16 days is way too long) and fixed the issue by myself.

That s all folks! See you next month!

The elections for the new OpenStack board are coming closer
and this time the Open Source community has a great
opportunity of representation: Giuseppe Patern is standing as a candidate for the board.
Although Giuseppe is considered by HP and Forrester Research
among the top talented consultants in the world,
Gippa (as he s largely known in the industry) is still one of us ,
a nerd that grew up with a keyboard on his hands.
As he s one of the candidates of the OpenStack board,
Fabio Marzocca wishing to know more has interviewed him.

[FM] The hard question. You re a techie. Why the hell are you running for the board?

[GP] This is indeed a good question It all started as a challenge from some clients and friends that are working in the OpenStack project. The truth is that the board and most of the management of the foundation are from vendors. I m not questioning here if they do a good job or not, it is very likely that they tend to protect their own interests. In my opinion it lacks some community spirit that have fostered Linux development such as Debian and Ubuntu. That s why I m running for it, to bring the community where it should be.

[FM] Back to Debian and Ubuntu, could you tell us your story with Linux?

[GP] I discovered Linux in 1994, but only in 1996 things were serious. By the time I just finished high school and I applied for a job in a local Internet Service Provider. At 15 years I was well known in the local community as I was installing and maintaining several BBSes, so it wasn t hard to get the job. I can say it was love at first sight. I started with Slackware (was the first distro), but I moved into redhat first and then debian. When I was working for the IBM Linux Technology Center, I was in charge of helping porting Linux to PowerPC and backporting LVM to make it similar to AIX. Sun was also a good playground as they acquired Cobalt, a hardware appliance based on debian. Then I shifted more towards Enterprise Linux adoption with 6 years in RedHat and then I went to Canonical. I was happy to go back to Debian and Ubuntu community, because I still believe that Ubuntu Developer Summits (UDS) were the real spirit of a Linux community.

[FM] Another hard question. We know you re somehow involved in the rebellion of Devuan.org. What is your opinion about systemd?

[GP] Let me tell you that it s not totally black/white and let s see the two sides here. Something like systemd was indeed needed. Each distro has its own way of init ing the system and for a package maintainer or commercial software maker, maintaining different init behaviour is insane. And as an init replacement it totally makes sense. However, IMHO systemd went too far away, incorporating into its code something that should not happen. A DHCP client into an init system, seriously? I doubt it was in the spirit of the Unix and Linux system
However, in the real world of pets vs cattle , where application matters more than systems, having a systemd as it is, doesn t change that much.

[FM] OpenStack was incubated in Ubuntu and the roots are quite clear. Is there something else that you would like to see from Debian and Ubuntu in OpenStack?

[GP] Stability, if I can name just one. Currently OpenStack is released every 6 months, which was probably the best choice to speed up the development. However, this is now becoming a weakness, as enterprise customers can t upgrade their critical infrastructures every 6 months. Traditionally Debian is maniacally focused on given a bullet-proof distribution, this is something that in my opinion is missing from OpenStack.

[FM] Gippa, tell us just 2 or 3 topics you will bring to action in case of election

[GP] I d like to introduce an OpenStack LTS process, following the Ubuntu approach: while releasing every 6 months is fair enough for development and testing environments, having a stable release every 2-3 years can give enterprise customers the peace of mind they need while running production environments.
I d also love to see a consolidation of the core (Nova/Neutron/Cinder/Swift): vendors and developers are introducing new features and projects while I d love to see for example- a more stable and scalable Neutron and a more stable connection to Oslo (in particular rabbitmq).
In general, I would encourage more attention to who is actually deploying, integrating and using OpenStack every day. I would also try to foster the ecosystem of ISVs in order to release and certify their software for OpenStack. And last but not least- to see interoperation between regional datacenters: I dream of a world where companies in a given territory can work together , and this is only possible through standards. I hope that OpenStack can represent this standard.

[FM] When are the elections and how can we vote?

[GP] Individual Member Director elections for the 2016 Board will be held online from Monday January 11, 2016 to Friday January 15, 2016. More informations on the website.

During the last weeks I finally got my hands dirty with IPv6. A comment on my blog and an email informed me that my server (hosting this blog) is not reachable via IPv6, albeit it has an IPv6 address. That said, I tried to get that running and fell into several holes, due to firewalls, my home provider not doing IPv6, and not the least due to my own ignorance of IPv6. Fortunately, with the help of some good souls and the technical support I finally got IPv6 connectivity on both my server, as well as at home, up. Server So my server is still running wheezy, which is a bit old, but upgrading with lots of hand-installed stuff is a bit a pain. The network interface was configured via a static IPv4 address, and no traces of IPv6 for now. Simple minded as I am, I assumed that this is the problem and started playing around with all kind of IPv6 configurations in the interfaces file, from using dhcp, and static configuration. The static configuration somehow worked out, but I never got a proper default route set. So I was really puzzled. In addition, at home I don t have IPv6, my provider is not distributing IPv6, so I couldn t test it (but on this a bit later). Fortunately, the good soul from the Netherland Filip B., did a lot of testing and also send permanent suggestions and analysis of the IPv6 situation, I am very much indebted to him. Anyway, his last advice about the problem with the default route was contacting the technical support of my provider, who within a very short time sent me an answer which made it clear to me: My firewall was blocking some ports that are necessary for IPv6 route autonegotiation. As firewall I use ferm, kind of a meta iptables language. In my server s setup the firewall blocks most of the ip traffic outside the expected servers, as well as does rate limit checks on ssh connections (those script kiddies). And while I have had some of the icmpv6 types allowed, it seemed that was insufficient. The solution at the end was to open icmpv6-type s 1-4, 133-137, 141, 142, 148, 149, as well as 130-132 from link-local adresses (fe80::/10). After having changed my ferm.conf, even removing everything related to IPv6 from the interfaces file and rebooting finally provided the server with an IPv6 address and a proper route. Wow, first step done. For Apache, fortunately I am doing already wildcard definitions, so no change was necessary and voil , the server is reachable via IPv6. Home As already mentioned, my provider at home does not provide IPv6 addresses. In addition, the DNS server of the provider is also very, well, peculiar about resolving IPv6 (i.e., AAAA records). Trying to get the AAAA record for ipv6.google.com works without a problem: but trying this with my own server, or for that matter any other server, resulted in no answer (preining.info has no AAAA record). Well, that didn t help. Despite these problems, I tried to get an IPv6 address via 6-in-4 tunneling (tunnelbroker), but that didn t work either, probably due to some reverse lookup and the fact that between the world and me there is a cable modem and a wlan router that all on top of the slightly funny infrastructure here in Japan, where you have to rent lines from NTT, but use a different company as Internet provider. I finally found the solution that worked for me, and that is installing miredo, available in Debian. Installation of the package, and starting the miredo service with systemctl start miredo (or /etc/init.d/miredo start) gave me in an instant a working IPv6 connectivity. ping6 worked on IPv6 addresses. Still, I couldn t fix the DNS lookup problem, so for the time being I have added IPv6 addresses for my own server in /etc/hosts, and by using the SixOrNot Firefox plugin I can actually check that it works. Conclusion It is 2016, and we are still far from end-to-end IPv6, and I have to admit that also my IPv6 knowledge is quite meager, and that I only recently have started to grasp some understanding. The good point was that Debian wheezy already works out of the box with IPv6 (if one is not as clumsy as me and blocks the necessary ports), including web servers and mail servers. I have also contacted my provider (AsahiNet) in the hope that they can update their DNS server, and learned that they are planning to roll out IPv6 addresses in the near future whatever that might mean in Japan, maybe next century?

Smartphones, ownership and hope for the fate of humanity
Do you own your phone or does it own you? DanielPocock.com


I have conflicting opinions about this article.

Usually I carry a dumb phone, so I'm not completely disconnected, but I'm mostly self-limited to "useful" communications by the fact that I have to pay for calls and SMSs. It also has a few useful features like showing the time , an alarm clock and a led torch, but that's it.

I also carry a smartphone, but I've never been able to trust it with my personal data, so there is no email on it and no communication software. It's also always offline to preserve battery, unless I'm actually using it for something (usually maps). It does have an offline wikipedia reader, which is the second thing I use it more often for. About half of the time I try to use it, however, it is off because I forgot to charge it, unless I've planned in advance to use it (which usually means I'm also carrying a laptop and will need to tether it).

So I guess that I should be agreeing with the article that offline life is better, and that we shouldn't depend on phones in our daily life, and mostly I do.

On the other hand, I'm not so sure that all of the people who seem to be interacting with a phone are actually disconnected from the local reality.

More than once I've experienced the use of smartphones as part of a local interaction: one typical case involves people having a conversation IRL and checking some fact on the internet and then sharing the results with the rest of the local group.

Actually, most of the time I've seen a smartphone being used at our table while eating with friends or collegues it was being passed around to show something to the people at the table, or at the very least being read aloud from, so it was part of the local experience, not a way to disconnect from it.

I'm sure that there are cases of abuse, but I still have hope that most of the connected humanity is managing to find a good balance between online and offline.

I don't want to go back carrying a wrist watch. I remember them as something unconfortable that ended up hitting stuff as I moved my hands, and I'd rather have my wrists free while I type, thanks. Pocket watches, OTOH...

the main exception involved one young adult in the middle of significantly older relatives, which is a somewhat different issue, and one that I believe predated smartphones (IIRC in my case similar situations involved trying to be somewhere else by reading a book).

It seems that I did mess it up with last version of Enca and it was not possible to install it without error. Now comes hotfix which fixes tat. If you don't know Enca, it is an Extremely Naive Charset Analyser. It detects character set and encoding of text files and can also convert them to other encodings using either a built-in converter or external libraries and tools like libiconv, librecode, or cstocs. Full list of changes for 1.18 release:

• fix installation of devhelp documentation

Still enca is in maintenance mode only and I have no intentions to write new features. However there is no limitation to other contributors, join the project at GitHub :-). You can download from http://cihar.com/software/enca/.

Filed under: Enca English 0 comments

Downgrading network-manager This morning I woke up. Bad idea. I find in the work mail a compiler error that I cannot reproduce, so I need to log into a machine at work. But #809195. I decided to downgrade network-manager. I recall there was a tool to download packages from snapshots.debian.org, I discussed it recently on IRC, let's sync the IRC logs from my server. Or not (#810212). Never mind, I'll log into the server and grep. Ooh, it's debsnap. However, it doesn't quite do what I hoped (#667712). After some help from #debian-devel (thanks jcristau and LebedevRI), here is how to downgrade network-manager:

# echo "deb http://snapshot.debian.org/archive/debian/20151125T155830Z/ sid main" >> /etc/apt/sources.list.d/tmp-downgrade-nm.list
# apt -o Acquire::Check-Valid-Until=false update
# apt -o Acquire::Check-Valid-Until=false install network-manager=1.0.6-1
# rm /etc/apt/sources.list.d/tmp-downgrade-nm.list
# service network-manager restart

And as user:

$ killall nm-applet
$ nm-applet &

The yak is now nice and shaved, I can now go and see what those compiler errors are all about. Actually, no, there was still an unshaved patch on the yak, and now we have a debcya script.