Search Results: "spa"

24 September 2014

Robert Collins: what-poles-for-the-tent

So Monty and Sean have recently blogged about about the structures (1, 2) they think may work better for OpenStack. I like the thrust of their thinking but had some mumblings of my own to add. Firstly, I very much like the focus on social structure and needs what our users and deployers need from us. That seems entirely right. And I very much like the getting away from TC picking winners and losers. That was never an enjoyable thing when I was on the TC, and I don t think it has made OpenStack better. However, the thing that picking winners and losers did was that it allowed users to pick an API and depend on it. Because it was the X API for OpenStack . If we don t pick winners, then there is no way to say that something is the X API for OpenStack , and that means that there is no forcing function for consistency between different deployer clouds. And so this appears to be why Ring 0 is needed: we think our users want consistency in being able to deploy their application to Rackspace or HP Helion. They want vendor neutrality, and by giving up winners-and-losers we give up vendor neutrality for our users. Thats the only explanation I can come up with for needing a Ring 0 because its still winners and losers (e.g. picking an arbitrary project) keystone, grandfathering it in, if you will. If we really want to get out of the role of selecting projects, I think we need to avoid this. And we need to avoid it without losing vendor neutrality (or we need to give up the idea of vendor neutrality). One might say that we must pick winners for the very core just by its, but I don t think thats true. If the core is small, many people will still want vendor neutrality higher up the stack. If the core is large, then we ll have a larger % of APIs covered and stable granting vendor neutrality. So a core with fixed APIs will be under constant pressure to expand: not just from developers of projects, but from users that want API X to be fixed and guaranteed available and working a particular way at [most] OpenStack clouds. Ring 0 also fulfils a quality aspect we can check that it all works together well in a realistic timeframe with our existing tooling. We are essentially proposing to pick functionality that we guarantee to users; and an API for that which they have everywhere, and the matching implementation we ve tested. To pull from Monty s post: What does a basic end user need to get a compute resource that works and seems like a computer? (end user facet) What does Nova need to count on existing so that it can provide that. He then goes on to list a bunch of things, but most of them are not needed for that: We need Nova (its the only compute API in the project today). We don t need keystone (Nova can run in noauth mode and deployers could just have e.g. Apache auth on top). We don t need Neutron (Nova can do that itself). We don t need cinder (use local volumes). We need Glance. We don t need Designate. We don t need a tonne of stuff that Nova has in it (e.g. quotas) end users kicking off a simple machine have -very- basic needs. Consider the things that used to be in Nova: Deploying containers. Neutron. Cinder. Glance. Ironic. We ve been slowly decomposing Nova (yay!!!) and if we keep doing so we can imagine getting to a point where there truly is a tightly focused code base that just does one thing well. I worry that we won t get there unless we can ensure there is no pressure to be inside Nova to win . So there s a choice between a relatively large set of APIs that make the guaranteed available APIs be comprehensive, or a small set that that will give users what they need just at the beginning but might not be broadly available and we ll be depending on some unspecified process for the deployers to agree and consolidate around what ones they make available consistently. In sort one of the big reasons we were picking winners and losers in the TC was to consolidate effort around a single API not implementation (keystone is already on its second implementation). All the angst about defcore and compatibility testing is going to be multiplied when there is lots of ecosystem choice around APIs above Ring 0, and the only reason that won t be a problem for Ring 0 is that we ll still be picking winners. How might we do this? One way would be to keep picking winners at the API definition level but not the implementation level, and make the competition be able to replace something entirely if they implement the existing API [and win hearts and minds of deployers]. That would open the door to everything being flexible and its happened before with Keystone. Another way would be to not even have a Ring 0. Instead have a project/program that is aimed at delivering the reference API feature-set built out of a single, flat Big Tent and allow that project/program to make localised decisions about what components to use (or not). Testing that all those things work together is not much different than the current approach, but we d have separated out as a single cohesive entity the building of a product (Ring 0 is clearly a product) from the projects that might go into it. Projects that have unstable APIs would clearly be rejected by this team; projects with stable APIs would be considered etc. This team wouldn t be the TC : they too would be subject to the TC s rulings. We could even run multiple such teams as hinted at by Dean Troyer one of the email thread posts. Running with that I d then be suggesting So OpenStack/NaaS would have an API or set of APIs, and they d be responsible for considering maturity, feature set, and so on, but wouldn t own Neutron, or Neutron incubator or any other component they would be a *cross project* team, focused at the product layer, rather than the component layer, which nearly all of our folk end up locked into today. Lastly Sean has also pointed out that we have large N N^2 communication issues I think I m proposing to drive the scope of any one project down to a minimum, which gives us more N, but shrinks the size within any project, so folk don t burn out as easily, *and* so that it is easier to predict the impact of changes clear contracts and APIs help a huge amount there.

Russ Allbery: Review: 2014 Hugos: Short Story Nominees

Review: 2014 Hugos: Short Story Nominees, edited by Loncon 3
Publisher: Loncon 3
Copyright: 2014
Format: Kindle
This is a bit of a weird "book review," since this is not a book. Rather, it's the collection of Hugo-nominated short stories for the 2014 Hugos (given for works published in 2013) at Loncon 3, the 2014 Worldcon. As such, the "editor" is the pool of attendees and supporting members who chose to nominate works, all of which had been previously edited by other editors in their original publication. This is also not something that someone else can acquire; if you were not a supporting or attending member, you didn't get the voting packet. But I believe all of the stories here are available on-line for free in some form, a short search away. "If You Were a Dinosaur, My Love" by Rachel Swirsky: The most common complaint about this story is that it's not really a story, and I have to agree. It's a word image of an alternate world in which the narrator's love is a human-sized dinosaur, starting with some surreal humor and then slowly shifting tone as it reveals the horrible event that's happened to the narrator's actual love, and that's sparked the wish for her love to have claws and teeth. It's reasonably good at what it's trying to do, but I wanted more of a story. The narrator's imagination didn't do much for me. (5) "The Ink Readers of Doi Saket" by Thomas Olde Heuvelt: At least for me, this story suffered from being put in the context of a Hugo nominee. It's an okay enough story about a Thai village downstream from a ritual that involves floating wishes down the river, often with offerings in the improvised small boats. The background of the story is somewhat cynical: the villagers make some of the wishes come true, sort of, while happily collecting the offerings and trying to spread the idea that the wishes with better offerings are more likely to come true. The protagonist follows a familiar twist: he actually can make wishes come true, maybe, but is very innocent about his role in the world. This is not a bad story, although stories written by people with western-sounding names about non-western customs worry me, and there were a few descriptions and approaches here (such as the nickname translations in footnotes and the villager archetypes) that made my teeth itch. But it is not a story that belongs on the Hugo nomination slate, at least in my opinion. It's either cute or mildly irritating, depending on one's mood when one meets it, not horribly original, and very forgettable. (5) "Selkie Stories Are for Losers" by Sofia Samatar: I really liked this story for much of its length. It features a couple of young, blunt, and bitter women, and focuses on the players in the typical selkie story that don't get much attention. The selkie's story is one of captivity or freedom; her lover's story is the inverse, the captor or the lover. But I don't recall a story about the children before, and I think Samatar got the tone right. It has the bitterness of divorce and abandonment mixed with the disillusionment of fantasy turned into pain. My problem with this story is the ending, or rather, the conclusion, since the story doesn't so much end as stop. There's a closing paragraph that gives some hint of the shape to come, but it gave me almost no closure, and it didn't answer any of the emotional questions that the rest of the story raised for me. I wanted something more, some sort of epiphany or clearer determination. (7) "The Water That Falls on You from Nowhere" by John Chu: This was by far my favorite of the nominees, which is convenient since it won. I thought it was the only nominee that felt in the class of stories I would expect to win a Hugo. I think this story needs one important caveat up front. The key conceit of the story is that, in this world, water falls on you out of nowhere if you tell any sort of lie. It does not explore the practical impact on that concept for the broader world. That didn't bother me; for some reason, I wasn't really expecting it to do so. But it did bother several other people I've seen comment on this story. They were quite frustrated that the idea was used primarily to shape a personal and family emotional dilemma, not to explore the impact on the world. So, go into this with the right expectations: if you want world-building or deep exploration of a change in physical laws, you will want a different story. This story, instead, is a beautiful gem about honesty in relationships, about communication about very hard things and very emotional things, about coming out, about trusting people, and about understanding people. I thought it was beautiful. If you read Captain Awkward, or other discussion of how to deal with difficult families and the damage they cause to relationships, seek this one out. It surprised me, and delighted me, and made me cry in places, and I loved the ending. It's more fantasy than science fiction, and it uses the conceit as a trigger for a story about people instead of a story about worlds and technology, but I'm still very happy to see it win. (9) Rating: 7 out of 10

23 September 2014

Dirk Eddelbuettel: RcppArmadillo 0.4.450.1.0

Continuing with his standard pace of approximately one new version per month, Conrad released a new minor release of Armadillo a few days ago. As before, I had created a GitHub-only pre-release which was tested against all eighty-seven (!!) CRAN dependents of our RcppArmadillo package and then uploaded RcppArmadillo 0.4.450.0 to CRAN. The CRAN maintainers pointed out that under the R-development release, a NOTE was issued concerning the C-library's rand() call. This is a pretty new NOTE, but it means using the (sometimes poor quality) rand() generator is now a no-no. Now, Armadillo being as robustly engineered as it is offers a new random number generator based on C++11 as well as a fallback generator for those unfortunate enough to live with an older C++98 compiler. (I would like to note here that I find Conrad's continued support for both C++11, offering very useful modern language idioms, as well as the fallback code for continued deployment and usage by those constrained in their choice of compilers rather exemplary --- because contrary to what some people may claim, it is not a matter of one or the other. C++ always was, and continues to be, a multi-paradigm language which can be supported easily by several standard. But I digress...) In any event, one cannot argue with CRAN about their prescription of a C++98 compiler. So Conrad and I discussed this over email, and came up with a scheme where a user-package (such as RcppArmadillo) can provide an alternate generator which Armadillo then deploys. I implemented a first solution which was then altered / reflected by Conrad in a revised version 4.450.1 of Armadillo. I packaged, and now uploaded, that version as RcppArmadillo 0.4.450.1.0 to both CRAN and into Debian. Besides the RNG change already discussed, this release brings a few smaller changes from the Armadillo side. These are detailed below in the extract from the NEWS file. On the RcppArmadillo side, we now have support for pkgKitten which is both very exciting and likely the topic of another blog post with an example of creating an RcppArmadillo package that purrs. In the process, I overhauled and polished how new packages are created by RcppArmadillo.package.skeleton(). An upcoming blog post may provide an example.
Changes in RcppArmadillo version 0.4.450.1.0 (2014-09-21)
  • Upgraded to Armadillo release Version 4.450.1 (Spring Hill Fort)
    • faster handling of matrix transposes within compound expressions
    • expanded symmatu()/symmatl() to optionally disable taking the complex conjugate of elements
    • expanded sort_index() to handle complex vectors
    • expanded the gmm_diag class with functions to generate random samples
  • A new random-number implementation for Armadillo uses the RNG from R as a fallback (when C++11 is not selected so the C++11-based RNG is unavailable) which avoids using the older C++98-based std::rand
  • The RcppArmadillo.package.skeleton() function was updated to only set an "Imports:" for Rcpp, but not RcppArmadillo which (as a template library) needs only LinkingTo:
  • The RcppArmadillo.package.skeleton() function will now prefer pkgKitten::kitten() over package.skeleton() in order to create a working package which passes R CMD check.
  • The pkgKitten package is now a Suggests:
  • A manual page was added to provide documentation for the functions provided by the skeleton package.
  • A small update was made to the package manual page.
Courtesy of CRANberries, there is also a diffstat report for the most recent release. As always, more detailed information is on the RcppArmadillo page. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

21 September 2014

Dariusz Dwornikowski: statistics of RFS bugs and sponsoring process

For some days I have been working on statistics of the sponsoring process in Debian. I find this to be one of the most important things that Debian has to attract and enable new contributions. It is important to know how this process works, whether we need more sponsors, how effective is the sponsoring and what are the timings connected to it. How I did this ? I have used Debbugs SOAP interface to get all bugs that are filed against sponsorship-requests pseudo package. SOAP gives a little bit of overhead because it needs to download a complete list of bugs for the sponsorship-requests package, and then process them according to given date ranges. The same information can be easily extracted from the UDD database in the future, it will be faster because SQL is better when working with date ranges than python obviously. The most problematic part was getting the "real done date" of a particular bug, and frankly most of my time I have spent on writing a rather dirty and complicated script. The script gets a log for a particular bug number and returns a "real done date". I have published a proof of concept in a previous post.. What I measured ? RFSs is a queue, and in every queue one is interested in a mean time to get processed. In this case I called the metric global MTTGS (mean time to get sponsored). This is a metric that gives the overall performance insight in RFS queue. Time to get sponsored (TTGS) for a bug is a number of days that passed between filing an RFS bug and closing it (bug was sponsored). Mean time to get sponsored is calculated as a sum of TTGSs of all bugs divided by number of bugs (in a given period of time). Global MTTGS is MTTGS calculated for a period of time 2012-1-1 until today(). Besides MTTGS I have also measured typical bug related metrics: Plots and graphs Below is a plot of global MTTGS vs. time (click for a larger image). mttgs plot As you can see, the trend is roughly exponential and MTTGS tends to settle around 60 days at the end of the year 2013. This does not mean that your package will wait 60 days on average nowadays to get sponsored. I remind that this is a global MTTGS, so even if the MTTGS of last month was very low, the global MTTGS would decrease just slightly. It gives, however, a good glance in performance of the process. Even that more packages are filed for sponsoring (see next graphs) now, than in the beginning of the epoch, the sponsoring rate is high enough to flatten the global MTTGS, and with time maybe decrease it. The image below (click for a larger one) shows how many bugs reside in a queue with status open or closed (calculated for each day). For closed we have an almost linear function, so each day more or less the same amount of bugs are closed and they increase the pool of bugs with status closed. For bugs with status open the interesting part begins around May 2012 after the system is saturated or gets popular. It can be interpreted as a plot of how many bugs reside in the queue, the important part is that it is stable and does not show clear increasing trend. open done plot The last plot shows arrival and departure rate of bugs from RFS queue, i.e. how many bugs are opened and closed each day. The interesting part here are the maxima. Let's look at them. opened closed plot Maximal number of opened bugs (21) was on 2012-05-06. As it appears it was a bunch upload of RFSs for tryton-modules-*.. 
  706953  RFS: tryton-modules-account-stock-anglo-saxon/2.8.0-1 
  706954  RFS: tryton-modules-purchase-shipment-cost/2.8.0-1 
  706948  RFS: tryton-modules-production/2.8.0-1 
  706969  RFS: tryton-modules-account-fr/2.8.0-1 
  706946  RFS: tryton-modules-project-invoice/2.8.0-1 
  706950  RFS: tryton-modules-stock-supply-production/2.8.0-1 
  706942  RFS: tryton-modules-product-attribute/2.8.0-1 
  706957  RFS: tryton-modules-stock-lot/2.8.0-1 
  706958  RFS: tryton-modules-carrier-weight/2.8.0-1 
  706941  RFS: tryton-modules-stock-supply-forecast/2.8.0-1 
  706955  RFS: tryton-modules-product-measurements/2.8.0-1 
  706952  RFS: tryton-modules-carrier-percentage/2.8.0-1 
  706949  RFS: tryton-modules-account-asset/2.8.0-1 
  706904  RFS: chinese-checkers/0.4-1 
  706944  RFS: tryton-modules-stock-split/2.8.0-1 
  706981  RFS: distcc/3.1-6 
  706945  RFS: tryton-modules-sale-supply/2.8.0-1 
  706959  RFS: tryton-modules-carrier/2.8.0-1 
  706951  RFS: tryton-modules-sale-shipment-cost/2.8.0-1 
  706943  RFS: tryton-modules-account-stock-continental/2.8.0-1 
  706956  RFS: tryton-modules-sale-supply-drop-shipment/2.8.0-1
Maximum number of closed bugs (18) was on 2013-09-24, and as you probably guessed right also tryton modules had impact on that. 
  706953  RFS: tryton-modules-account-stock-anglo-saxon/2.8.0-1 
  706954  RFS: tryton-modules-purchase-shipment-cost/2.8.0-1 
  706948  RFS: tryton-modules-production/2.8.0-1 
  706969  RFS: tryton-modules-account-fr/2.8.0-1 
  706946  RFS: tryton-modules-project-invoice/2.8.0-1 
  706950  RFS: tryton-modules-stock-supply-production/2.8.0-1 
  706942  RFS: tryton-modules-product-attribute/2.8.0-1 
  706958  RFS: tryton-modules-carrier-weight/2.8.0-1 
  706941  RFS: tryton-modules-stock-supply-forecast/2.8.0-1 
  706955  RFS: tryton-modules-product-measurements/2.8.0-1 
  706952  RFS: tryton-modules-carrier-percentage/2.8.0-1 
  706949  RFS: tryton-modules-account-asset/2.8.0-1 
  706944  RFS: tryton-modules-stock-split/2.8.0-1 
  706959  RFS: tryton-modules-carrier/2.8.0-1 
  723991  RFS: mapserver/6.4.0-2 
  706951  RFS: tryton-modules-sale-shipment-cost/2.8.0-1 
  706943  RFS: tryton-modules-account-stock-continental/2.8.0-1 
  706956  RFS: tryton-modules-sale-supply-drop-shipment/2.8.0-1
The software Most of the software was written in Python. Graphs were generated in R. After a code cleanup I will publish a complete solution on my github account, free to use by everybody. If you would like to see another statistics, please let me know, I can create them if the data provides sufficient information.

20 September 2014

Laura Arjona: Happy Software Freedom Day!

Today we celebrate the day of free software (each year, a saturday around mid-September) More info at softwarefreedomday.org There are no public events in Madrid, but I m going to try to hack and write a bit more this weekend, as my personal celebration. In this blog post you can find some of my very very recent activities on free software, and my plans for this weekend of celebration! Debian Children distros aka Derivatives I had the translation/update of the page www.debian.org/misc/children-distros pending since long time. It s a long page, and I was not sure what was better: if picking up the too-outdated last translation, and review it carefully in order to update it, or starting from scratch. I decided to reuse the last translation (thanks Luis Uribe!) and after some days dedicating my commuting time on it, finally, yesterday evening I finished it at home. Now it s in the review queue, and I hope in 10 days or so it will be uploaded. In the meantime, I have learned a bit about the Debian Derivatives subproject and census, I have watched the Derivatives Panel at DebConf13, and had a look at the bug #723069 about keeping the children-distros page up to date. So now that I m liberated about this translation, I m going to put some time in keeping up to date the original English page (I m part of the www and publicity team, so I think it makes sense). My goal is to review at least one Debian derivative each two days, and when I finish the list, start again. I can update the wiki myself, and for the www, I ll send patches against #723069, unless I m told to do it other way. BTW, wouldn t be nice to mark web/wiki pages as RFH the same as packages?, so other people can easily decide to put some time on them, and make http://www.debian.org even more awesome! Or make them appear in the how-can-i-help reminders :) Mmm maybe it s just a matter of filing a bug and tagging it as gift ? I think no, because nobody has the package www.debian.org installed in their system I ll talk with the maintainer about this. New Member process I promised myself to try to work a bit more in Debian during the summer and September, and if everything goes well, try to apply to the new member process in October. I wanted to read all the documentation first, and one challenge is to review/update the translations of www.debian.org/devel/join folder. This way, both myself and the Spanish speaking community benefit from the effort. Yesterday I translated one of those pending pages and I hope during the weekend I can translate/update the rest. When I finish that, I ll keep reading the other documentation. DebConf15 This summer I was invited to join the DebConf15 organization team and pick up tasks in the publicity area. I was very happy to join, I m not sure at all that I can go to DebConf15 in Heidelberg (Germany), in fact I m quite sure I will not go since mid-August is the only opportunity to visit family who lives far away, but anyway, there are things that we can do before DebConf15 and I can contribute. For now, I attended last Monday to the meeting at IRC, and I m finishing a short blogpost about the DebConf14 talk presenting DebConf15, that will be published in the DebConf15 blog. Android, F-Droid I keep on trying to spread the word about F-Droid and the free software available for Android, last week some of my friends updated Kontalk to the 3.0.b1 version (I had updated at the beginning of September) and they liked that now, the images are sent encrypted as well as the text messages :) Some friends also liked the 2048 game, since it can be played offline, without ads, and so. I decided to spend some time this weekend contributing translations to the Android apps that I use. A long pending issue is to try to put workforce in the F-Droid project itself so apps descriptions are internationalized (the program is fully translatable, but the categories of apps and the descriptions themselves, are not). This is a complicated issue, it requires to take some design decisions, and later, of course, the implementation. I cannot do it alone, and I cannot do it in the short time. But today I have filed a bug report (#35) so maybe I find other people able to help. Jabber/XMPP and the RedesLibres chatroom Since several months I ve been using more often my Jabber/XMPP account to join the chatroom redeslibres@salas.mijabber.es I meet there some people that I follow in Pump.io (for example, the people that write in the Comun cate Libremente or Lignux blogs) and we talk about pump.io, free software, free services, and other things. I feel very comfortable there, it s nice to have a Spanish speaking group inside the Free Software community, and I m also learning a bit about XMPP (I ve tried a lot of desktop and Android clients, just for fun!), free networks, and so. So today I wanted to publicly thank you everybody in that chatroom, that welcomed me so well :) Thank you, free software friends And, by extension, I want to thank you all the people that work and have fun in the Free Software communities, in the projects where I contribute or others. They (we) hack to make the world better, and to allow others join to this beautiful challenge that is making machines do what their (final) users wants. Comments? You can comment on this post in this Pump.io thread.
Filed under: My experiences and opinion Tagged: Android, Communities, Contributing to libre software, Debian, English, F-Droid, federation, Free Software, Freedom, internationalization, libre software, localization, translations

19 September 2014

Dariusz Dwornikowski: getting real "done date" of a bug from Debian BTS

As I wrote in my last post currently, SOAP interface, nor Ultimate Debian Database do not provide a date when a given bug was closed (done date). It is quite hard to calculate statistics on a bug tracker when you do not know when a bug was closed (!!). Done date of bug can be found in its log. The log itself can be downloaded by SOAP method get_bug_log but the processing of it is quite complicated. The same comes to web scrapping of a BTS's web interface. Fortunatelly the web interface gives a possibility to download a log in an mbox format. Below is a script that extracts the done date of a bug from its log in mbox format. It uses requests to download the mbox and caches the result in ~/.cache/rfs_bugs, which you need to create. It performs different checks:
  1. Check existence of a header e.g. Received: (at 657783-done) by bugs.debian.org; 29 Jan 2012 13:27:42 +0000
  2. Check for header CC: NUMBER-close done
  3. Check for header TO: NUMBER-close done
  4. Check for Close: NUMBER in body.
The code is below: 
import requests
from datetime import datetime
import mailbox
import re
import os
import tempfile
def get_done_date(bug_num):
    CACHE_DIR = os.path.expanduser("~") + "/.cache/rfs_bugs/"
    def get_from_cache():
        if os.path.exists(" ".format(CACHE_DIR, bug_num)):
            with open(" ".format(CACHE_DIR, bug_num)) as f:
                return datetime.strptime(f.readlines()[0].rstrip(), "%Y-%m-%d").date()
        else:
            return None
    done_date = get_from_cache()
    if done_date is not None:
        return done_date
    else:
        r = requests.get("https://bugs.debian.org/cgi-bin/bugreport.cgi?mbox=yes;bug= ;mboxstatus=yes".format(self._num))
        d = try_header(r.text)
        if d is None:
            d = try_cc(r.text)
        if d is None:
            d = try_body(r.text)
        if d is not None:
            with open(" ".format(CACHE_DIR, bug_num), "w") as f:
                f.write(" ".format(d.date()))
        else:
            return None
        return d.date()
    def try_body(text):
        reg = "\(at\s.+\)\s+by\sbugs\.debian\.org;\s(\d 1,2 \s\w\w\w\s\d\d\d\d)"
        handle, name = tempfile.mkstemp()
        with open(name, "w") as f:
            f.write(text.encode('latin-1'))
        mbox = mailbox.mbox(name)
        for i in mbox.items():
            if i[1].is_multipart():
                for m in i[1].get_payload():
                    if "close" in str(m) or "done" in str(m):
                        try:
                            result = re.search(reg, i[1]['Received'])
                            return datetime.strptime(result.group(1), "%d %b %Y")
                        except:
                            return None
            else:
                if "close" in i[1].get_payload() or "done" in i[1].get_payload():
                    try:
                        result = re.search(reg, i[1]['Received'])
                        return datetime.strptime(result.group(1), "%d %b %Y")
                    except:
                        return None
        return None
    def try_header(text):
        reg = "Received:\s\(at\s\d\d\d\d\d\d-(close done)\)\s+by.+"
        try:
            result = re.search(reg, r.text)
            line = result.group(0)
            reg2 = "\d 1,2 \s\w\w\w\s\d\d\d\d"
            result = re.search(reg2, line)
            d = datetime.strptime(result.group(0), "%d %b %Y")
            return d
        except:
            return None
    def try_cc(text):
        reg = "\(at\s.+\)\s+by\sbugs\.debian\.org;\s(\d 1,2 \s\w\w\w\s\d\d\d\d)"
        handle, name = tempfile.mkstemp()
        with open(name, "w") as f:
            f.write(text.encode('latin-1'))
        mbox = mailbox.mbox(name)
        for i in mbox.items():
            if ('CC' in i[1] and "done" in i[1]['CC']) or ('To' in i[1] and "done" in i[1]['To']):
                try:
                    result = re.search(reg, i[1]['Received'])
                    return datetime.strptime(result.group(1), "%d %b %Y")
                except:
                    return None
if __name__ == "__main__":
    print get_done_date(752210)
PS: I hope that the script will be not needed in the near future, as Don Armstrong plans a new BTS database, a Debconf14 video is here.

Paul Tagliamonte: Docker PostgreSQL Foreign Data Wrapper

For the tl;dr: Docker FDW is a thing. Star it, hack it, try it out. File bugs, be happy. If you want to see what it's like to read, there's some example SQL down below. The question is first, what the heck is a PostgreSQL Foreign Data Wrapper? PostgreSQL Foreign Data Wrappers are plugins that allow C libraries to provide an adaptor for PostgreSQL to talk to an external database. Some folks have used this to wrap stuff like MongoDB, which I always found to be hilarous (and an epic hack). Enter Multicorn During my time at PyGotham, I saw a talk from Wes Chow about something called Multicorn. He was showing off some really neat plugins, such as the git revision history of CPython, and parsed logfiles from some stuff over at Chartbeat. This basically blew my mind. All throughout the talk I was coming up with all sorts of things that I wanted to do -- this whole library is basically exactly what I've been dreaming about for years. I've always wanted to provide a SQL-like interface into querying API data, joining data cross-API using common crosswalks, such as using Capitol Words to query for Legislators, and use the bioguide ids to JOIN against the congress api to get their Twitter account names. My first shot was to Multicorn the new Open Civic Data API I was working on, chuckled and put it aside as a really awesome hack. Enter Docker It wasn't until tianon connected the dots for me and suggested a Docker FDW did I get really excited. Cue a few hours of hacking, and I'm proud to say -- here's Docker FDW. This lets us ask all sorts of really interesting questions out of the API, and might even help folks writing webapps avoid adding too much Docker-aware logic. Abstractions can be fun! Setting it up I'm going to assume you have a working Multicorn, PostgreSQL and Docker setup (including adding the postgres user to the docker group) So, now let's pop open a psql session. Create a database (I called mine dockerfdw, but it can be anything), and let's create some tables. Before we create the tables, we need to let PostgreSQL know where our objects are. This takes a name for the server, and the Python importable path to our FDW. 
CREATE SERVER docker_containers FOREIGN DATA WRAPPER multicorn options (
    wrapper 'dockerfdw.wrappers.containers.ContainerFdw');
CREATE SERVER docker_image FOREIGN DATA WRAPPER multicorn options (
    wrapper 'dockerfdw.wrappers.images.ImageFdw');
Now that we have the server in place, we can tell PostgreSQL to create a table backed by the FDW by creating a foreign table. I won't go too much into the syntax here, but you might also note that we pass in some options - these are passed to the constructor of the FDW, letting us set stuff like the Docker host. 
CREATE foreign table docker_containers (
    "id"          TEXT,
    "image"       TEXT,
    "name"        TEXT,
    "names"       TEXT[],
    "privileged"  BOOLEAN,
    "ip"          TEXT,
    "bridge"      TEXT,
    "running"     BOOLEAN,
    "pid"         INT,
    "exit_code"   INT,
    "command"     TEXT[]
) server docker_containers options (
    host 'unix:///run/docker.sock'
);
CREATE foreign table docker_images (
    "id"              TEXT,
    "architecture"    TEXT,
    "author"          TEXT,
    "comment"         TEXT,
    "parent"          TEXT,
    "tags"            TEXT[]
) server docker_image options (
    host 'unix:///run/docker.sock'
);
And, now that we have tables in place, we can try to learn something about the Docker containers. Let's start with something fun - a join from containers to images, showing all image tag names, the container names and the ip of the container (if it has one!). 
SELECT docker_containers.ip, docker_containers.names, docker_images.tags
  FROM docker_containers
  RIGHT JOIN docker_images
  ON docker_containers.image=docker_images.id;

     ip                   names                               tags                   
-------------+-----------------------------+-----------------------------------------
                                              ruby:latest 
                                              paultag/vcs-mirror:latest 
                /de-openstates-to-ocd         sunlightlabs/scrapers-us-state:latest 
                /ny-openstates-to-ocd         sunlightlabs/scrapers-us-state:latest 
                /ar-openstates-to-ocd         sunlightlabs/scrapers-us-state:latest 
 172.17.0.47    /ms-openstates-to-ocd         sunlightlabs/scrapers-us-state:latest 
 172.17.0.46    /nc-openstates-to-ocd         sunlightlabs/scrapers-us-state:latest 
                /ia-openstates-to-ocd         sunlightlabs/scrapers-us-state:latest 
                /az-openstates-to-ocd         sunlightlabs/scrapers-us-state:latest 
                /oh-openstates-to-ocd         sunlightlabs/scrapers-us-state:latest 
                /va-openstates-to-ocd         sunlightlabs/scrapers-us-state:latest 
 172.17.0.41    /wa-openstates-to-ocd         sunlightlabs/scrapers-us-state:latest 
                /jovial_poincare              <none>:<none> 
                /jolly_goldstine              <none>:<none> 
                /cranky_torvalds              <none>:<none> 
                /backstabbing_wilson          <none>:<none> 
                /desperate_hoover             <none>:<none> 
                /backstabbing_ardinghelli     <none>:<none> 
                /cocky_feynman                <none>:<none> 
                                              paultag/postgres:latest 
                                              debian:testing 
                                              paultag/crank:latest 
                                              <none>:<none> 
                                              <none>:<none> 
                /stupefied_fermat             hackerschool/doorbot:latest 
                /focused_euclid               debian:unstable 
                /focused_babbage              debian:unstable 
                /clever_torvalds              debian:unstable 
                /stoic_tesla                  debian:unstable 
                /evil_torvalds                debian:unstable 
                /foo                          debian:unstable 
(31 rows)
OK, let's see if we can bring this to the next level now. I finally got around to implementing INSERT and DELETE operations, which turned out to be pretty simple to do. Check this out: 
DELETE FROM docker_containers;

DELETE 1
This will do a stop + kill after a 10 second hang behind the scenes. It's actually a lot of fun to spawn up a container and terminate it from PostgreSQL. 
INSERT INTO docker_containers (name, image) VALUES ('hello', 'debian:unstable') RETURNING id;

                                id                                
------------------------------------------------------------------
 0a903dcf5ae10ee1923064e25ab0f46e0debd513f54860beb44b2a187643ff05
INSERT 0 1
(1 row)
Spawning containers works too - this is still very immature and not super practical, but I figure while I'm showing off, I might as well go all the way. 
SELECT ip FROM docker_containers WHERE id='0a903dcf5ae10ee1923064e25ab0f46e0debd513f54860beb44b2a187643ff05';

     ip      
-------------
 172.17.0.12
(1 row)
Success! This is just a taste of what's to come, so please feel free to hack on Docker FDW, tweet me @paultag, file bugs / feature requests. It's currently a bit of a hack, and it's something that I think has long-term potential after some work goes into making sure that this is a rock solid interface to the Docker API.

18 September 2014

Jaldhar Vyas: Scotland: Vote A DINNAE KEN

From the crack journalists at CNN. [CNN Scottish poll fail] Interesting fact: anyone who wore a kilt at debconf is allowed to vote in the referendum.

Dariusz Dwornikowski: RFS health in Debian

I am working on a small project to create WNPP like statistics for open RFS bugs. I think this could improve a little bit effectiveness of sponsoring new packages by giving insight into bugs that are on their way to being starved (i.e. not ever sponsored, or rotting in a queue). The script attached in this post is written in Python and uses Debbugs SOAP interface to get currently open RFS bugs and calculates their dust and age. The dust factor is calculated as an absolute value of a difference between bugs's age and log_modified. Later I would like to create fully blown stats for an RFS queue, taking into account the whole history (i.e. 2012-1-1 until now), and check its health, calculate MTTGS (mean time to get sponsored). The list looks more or less like this: 
Age  Dust Number  Title
37   0    757966  RFS: lutris/0.3.5-1 [ITP]
1    0    762015  RFS: s3fs-fuse/1.78-1 [ITP #601789] -- FUSE-based file system backed by Amazon S3
81   0    753110  RFS: mrrescue/1.02c-1 [ITP]
456  0    712787  RFS: distkeys/1.0-1 [ITP] -- distribute SSH keys
120  1    748878  RFS: mwc/1.7.2-1 [ITP] -- Powerful website-tracking tool
1    1    762012  RFS: fadecut/0.1.4-1
3    1    761687  RFS: abraca/0.8.0+dfsg-1 -- Simple and powerful graphical client for XMMS2
35   2    758163  RFS: kcm-ufw/0.4.3-1 ITP
3    2    761636  RFS: raceintospace/1.1+dfsg1-1 [ITP]
....
....
The script rfs_health.py can be found below, it uses SOAPpy (only python <3 unfortunately). 
#!/usr/bin/python
import SOAPpy
import time
from datetime import date, timedelta, datetime
url = 'http://bugs.debian.org/cgi-bin/soap.cgi'
namespace = 'Debbugs/SOAP'
server = SOAPpy.SOAPProxy(url, namespace)
class RFS(object):
    def __init__(self, obj):
        self._obj = obj
        self._last_modified = date.fromtimestamp(obj.log_modified)
        self._date = date.fromtimestamp(obj.date)
        if self._obj.pending != 'done':
            self._pending = "pending"
            self._dust = abs(date.today() - self._last_modified).days
        else:
            self._pending = "done"
            self._dust = abs(self._date - self._last_modified).days
        today = date.today()
        self._age = abs(today - self._date).days
    @property
    def status(self):
        return self._pending
    @property
    def date(self):
        return self._date
    @property
    def last_modified(self):
        return self._last_modified
    @property
    def subject(self):
        return self._obj.subject
    @property
    def bug_number(self):
        return self._obj.bug_num
    @property
    def age(self):
        return self._age
    @property
    def dust(self):
        return self._dust
    def __str__(self):
        return "  subject:   age:  dust: ".format(self._obj.bug_num, self._obj.subject, self._age, self._dust)
if __name__ == "__main__":
    bugi = server.get_bugs("package", "sponsorship-requests", "status", "open")
    buglist = [RFS(b.value) for b in server.get_status(bugi).item]
    buglist_sorted_by_dust = sorted(buglist, key=lambda x: x.dust, reverse=False)
    print("Age  Dust Number  Title")
    for i in buglist_sorted_by_dust:
        print(" :<4   :<4   :<7   ".format(i.age, i.dust, i.bug_number, i.subject))

16 September 2014

Hideki Yamane: Intel 910 SSD 400GB - $420


Intel SSD 910 (400GB, SSDPEDOX400G301) is cheaper than ever in Japan - only $420 (and its spec sheet says "Recommended Customer Price BULK: $1929.00", wow).

15 September 2014

Keith Packard: xserver-pending-fixes

A Forest of X Server Changes We've got about another month left in the X server merge window for 1.17 and I've written a small set of fixes which haven't been reviewed yet for merging. I thought I'd advertise them a bit and see if I couldn't encourage a few of you to take a look and see if they're useful, correct and complete. All of these are in my personal X server repository: 
git://people.freedesktop.org/~keithp/xserver.git
Cleaning up the X Registry Branch: registry-fixes I'll bet most of you don't even know about this code. It serves as a database mapping various X enumerations to strings to aid in diagnostics. For the security extensions, SECURITY and XSELinux, it holds names for all of the request, event and errors in the core protocol and all registered extensions. For X-Resource, it has the names of the registered resource types. The X registry gets the request, event and error data from a file, "protocol.txt", which is installed in /usr/lib/xorg/protocol.txt on my machine. It gets the resource names as a part of resource type allocation. So, what's wrong with this? Three basic things:
  1. A simple bug -- protocol.txt is left open while the server runs. This consumes a file descriptor for no good reason.
  2. protocol.txt is read and parsed even if the security extensions aren't available. This wastes time and memory.
  3. The resource names are kept even if X-Resource isn't in use.
The fixes remove the configure options for including the registry code; these functions are only used by the above extensions, so we can tell whether to include the code based solely on whether the extensions are being built. Getting rid of the TCP listener by default Branch: listen-fixes We've had the '-nolisten' option for a while now to disable inbound TCP connections. It's useful for security reasons, but we've never enabled this by default. This patch sequence provides configure options for each of the listen sockets (tcp, unix and local), leaves unix and local enabled by default and disables tcp by default. A new option, '-listen', is added which allows the user to override the -nolisten defaults in case they actually want to use TCP connections to X. Glamor bug fixes branch: glamor-fixes This branch fixes two bugs:
  1. Scale a large pixmap down to a small pixmap. This happens when you display enormous images in a web page. Iceweasel sends the whole huge image to X and uses Render to scale it to the screen. If the image is larger than a single texture, the X server splits it up into tiles, but the code which tries to perform the merged scale is just broken. Five patches fix this.
  2. Shader-based trapezoids. This code uses area coverage to compute trapezoids. That violates the Render spec, which requires point sampling. Further, the performance of these trapezoids is lower than software (by a lot). This one patch removes the code.
Present bug fixes branch: present-fixes A selection of small bug fixes:
  1. Clear pending flips at CloseScreen. This removes a reference to any pending flip pixmap, allowing it to be freed. Otherwise, we'll leak memory across server reset.
  2. Add support for PresentOptionCopy. This has been in the protocol spec for a while, and was completely trivial to implement. However, it never got done. One tiny little patch.
  3. Expose the Present API to drivers via sdksyms.sh. Until now, the present extension APIs have only been available inside the X server. This exposes them to drivers. This took a few cleanup patches first.
Use Present for Glamor XV branch: glamor-present-xv Painting XV to the screen should be done at vblank time to avoid tearing. Present offers vblank synchronized operations. Hooking those two together required a few new present APIs to expose the vblank functionality outside of the present code, then a bit of glamor code to hook up that new API to the XV bits. Switching Glamor to a GL core profile context branch: glamor-core-profile This patch set is still in progress, but demonstrates how close we are. We'll be requiring OpenGL 3.3 for this so that we get texture swizzling, which is required for our single channel objects. The changes present on the branch are:
  1. Switch single channel surfaces from GL_ALPHA to GL_RED.
  2. Use vertex array objects.
  3. Switch ephyr over to using a core 3.3 profile.
Still left to do is
  1. Switch Render code to VBOs
The core code uses VBOs everywhere, but the Render code doesn't. This means that all Render drawing fails, which makes the resulting server not very useful. My main objective for getting this done is to reduce memory usage by about 16MB, which is the space allocated for software rendering in Mesa in case someone does something which the hardware doesn't handle, and that can only with some legacy OpenGL APIs. Please help out! All of these friendly little patches are looking for a bit of review so that they can get merged before the 1.17 window closes.

Julien Danjou: Python bad practice, a concrete case

A lot of people read up on good Python practice, and there's plenty of information about that on the Internet. Many tips are included in the book I wrote this year, The Hacker's Guide to Python. Today I'd like to show a concrete case of code that I don't consider being the state of the art. In my last article where I talked about my new project Gnocchi, I wrote about how I tested, hacked and then ditched whisper out. Here I'm going to explain part of my thought process and a few things that raised my eyebrows when hacking this code. Before I start, please don't get the spirit of this article wrong. It's in no way a personal attack to the authors and contributors (who I don't know). Furthermore, whisper is a piece of code that is in production in thousands of installation, storing metrics for years. While I can argue that I consider the code not to be following best practice, it definitely works well enough and is worthy to a lot of people. Tests The first thing that I noticed when trying to hack on whisper, is the lack of test. There's only one file containing tests, named test_whisper.py, and the coverage it provides is pretty low. One can check that using the coverage tool. 
$ coverage run test_whisper.py
...........
----------------------------------------------------------------------
Ran 11 tests in 0.014s
 
OK
$ coverage report
Name           Stmts   Miss  Cover
----------------------------------
test_whisper     134      4    97%
whisper          584    227    61%
----------------------------------
TOTAL            718    231    67%

While one would think that 61% is "not so bad", taking a quick peak at the actual test code shows that the tests are incomplete. Why I mean by incomplete is that they for example use the library to store values into a database, but they never check if the results can be fetched and if the fetched results are accurate. Here's a good reason one should never blindly trust the test cover percentage as a quality metric. When I tried to modify whisper, as the tests do not check the entire cycle of the values fed into the database, I ended up doing wrong changes but had the tests still pass. No PEP 8, no Python 3 The code doesn't respect PEP 8 . A run of flake8 + hacking shows 732 errors While it does not impact the code itself, it's more painful to hack on it than it is on most Python projects. The hacking tool also shows that the code is not Python 3 ready as there is usage of Python 2 only syntax. A good way to fix that would be to set up tox and adds a few targets for PEP 8 checks and Python 3 tests. Even if the test suite is not complete, starting by having flake8 run without errors and the few unit tests working with Python 3 should put the project in a better light. Not using idiomatic Python A lot of the code could be simplified by using idiomatic Python. Let's take a simple example: 
def fetch(path,fromTime,untilTime=None,now=None):
  fh = None
  try:
    fh = open(path,'rb')
    return file_fetch(fh, fromTime, untilTime, now)
  finally:
    if fh:
      fh.close()

That piece of code could be easily rewritten as: 
def fetch(path,fromTime,untilTime=None,now=None):
  with open(path, 'rb') as fh:
    return file_fetch(fh, fromTime, untilTime, now)

This way, the function looks actually so simple that one can even wonder why it should exists but why not. Usage of loops could also be made more Pythonic: 
for i,archive in enumerate(archiveList):
  if i == len(archiveList) - 1:
    break

could be actually: 
for i, archive in itertools.islice(archiveList, len(archiveList) - 1):

That reduce the code size and makes it easier to read through the code. Wrong abstraction level Also, one thing that I noticed in whisper, is that it abstracts its features at the wrong level. Take the create() function, it's pretty obvious: 
def create(path,archiveList,xFilesFactor=None,aggregationMethod=None,sparse=False,useFallocate=False):
  # Set default params
  if xFilesFactor is None:
    xFilesFactor = 0.5
  if aggregationMethod is None:
    aggregationMethod = 'average'
 
  #Validate archive configurations...
  validateArchiveList(archiveList)
 
  #Looks good, now we create the file and write the header
  if os.path.exists(path):
    raise InvalidConfiguration("File %s already exists!" % path)
  fh = None
  try:
    fh = open(path,'wb')
    if LOCK:
      fcntl.flock( fh.fileno(), fcntl.LOCK_EX )
 
    aggregationType = struct.pack( longFormat, aggregationMethodToType.get(aggregationMethod, 1) )
    oldest = max([secondsPerPoint * points for secondsPerPoint,points in archiveList])
    maxRetention = struct.pack( longFormat, oldest )
    xFilesFactor = struct.pack( floatFormat, float(xFilesFactor) )
    archiveCount = struct.pack(longFormat, len(archiveList))
    packedMetadata = aggregationType + maxRetention + xFilesFactor + archiveCount
    fh.write(packedMetadata)
    headerSize = metadataSize + (archiveInfoSize * len(archiveList))
    archiveOffsetPointer = headerSize
 
    for secondsPerPoint,points in archiveList:
      archiveInfo = struct.pack(archiveInfoFormat, archiveOffsetPointer, secondsPerPoint, points)
      fh.write(archiveInfo)
      archiveOffsetPointer += (points * pointSize)
 
    #If configured to use fallocate and capable of fallocate use that, else
    #attempt sparse if configure or zero pre-allocate if sparse isn't configured.
    if CAN_FALLOCATE and useFallocate:
      remaining = archiveOffsetPointer - headerSize
      fallocate(fh, headerSize, remaining)
    elif sparse:
      fh.seek(archiveOffsetPointer - 1)
      fh.write('\x00')
    else:
      remaining = archiveOffsetPointer - headerSize
      chunksize = 16384
      zeroes = '\x00' * chunksize
      while remaining > chunksize:
        fh.write(zeroes)
        remaining -= chunksize
      fh.write(zeroes[:remaining])
 
    if AUTOFLUSH:
      fh.flush()
      os.fsync(fh.fileno())
  finally:
    if fh:
      fh.close()

The function is doing everything: checking if the file doesn't exist already, opening it, building the structured data, writing this, building more structure, then writing that, etc. That means that the caller has to give a file path, even if it just wants a whipser data structure to store itself elsewhere. StringIO() could be used to fake a file handler, but it will fail if the call to fcntl.flock() is not disabled and it is inefficient anyway. There's a lot of other functions in the code, such as for example setAggregationMethod(), that mixes the handling of the files even doing things like os.fsync() while manipulating structured data. This is definitely not a good design, especially for a library, as it turns out reusing the function in different context is near impossible. Race conditions There are race conditions, for example in create() (see added comment): 
if os.path.exists(path):
  raise InvalidConfiguration("File %s already exists!" % path)
fh = None
try:
  # TOO LATE I ALREADY CREATED THE FILE IN ANOTHER PROCESS YOU ARE GOING TO
  # FAIL WITHOUT GIVING ANY USEFUL INFORMATION TO THE CALLER :-(
  fh = open(path,'wb')

That code should be: 
try:
  fh = os.fdopen(os.open(path, os.O_WRONLY   os.O_CREAT   os.O_EXCL), 'wb')
except OSError as e:
  if e.errno = errno.EEXIST:
    raise InvalidConfiguration("File %s already exists!" % path)

to avoid any race condition. Unwanted optimization We saw earlier the fetch() function that is barely useful, so let's take a look at the file_fetch() function that it's calling. 
def file_fetch(fh, fromTime, untilTime, now = None):
  header = __readHeader(fh)
[...]

The first thing the function does is to read the header from the file handler. Let's take a look at that function: 
def __readHeader(fh):
  info = __headerCache.get(fh.name)
  if info:
    return info
 
  originalOffset = fh.tell()
  fh.seek(0)
  packedMetadata = fh.read(metadataSize)
 
  try:
    (aggregationType,maxRetention,xff,archiveCount) = struct.unpack(metadataFormat,packedMetadata)
  except:
    raise CorruptWhisperFile("Unable to read header", fh.name)
[...]

The first thing the function does is to look into a cache. Why is there a cache? It actually caches the header based with an index based on the file path (fh.name). Except that if one for example decide not to use file and cheat using StringIO, then it does not have any name attribute. So this code path will raise an AttributeError. One has to set a fake name manually on the StringIO instance, and it must be unique so nobody messes with the cache 
import StringIO
 
packedMetadata = <some source>
fh = StringIO.StringIO(packedMetadata)
fh.name = "myfakename"
header = __readHeader(fh)

The cache may actually be useful when accessing files, but it's definitely useless when not using files. But it's not necessarily true that the complexity (even if small) that the cache adds is worth it. I doubt most of whisper based tools are long run processes, so the cache that is really used when accessing the files is the one handled by the operating system kernel, and this one is going to be much more efficient anyway, and shared between processed. There's also no expiry of that cache, which could end up of tons of memory used and wasted. Docstrings None of the docstrings are written in a a parsable syntax like Sphinx. This means you cannot generate any documentation in a nice format that a developer using the library could read easily. The documentation is also not up to date: 
def fetch(path,fromTime,untilTime=None,now=None):
  """fetch(path,fromTime,untilTime=None)
[...]
"""
 
def create(path,archiveList,xFilesFactor=None,aggregationMethod=None,sparse=False,useFallocate=False):
  """create(path,archiveList,xFilesFactor=0.5,aggregationMethod='average')
[...]
"""

This is something that could be avoided if a proper format was picked to write the docstring. A tool cool be used to be noticed when there's a diversion between the actual function signature and the documented one, like missing an argument. Duplicated code Last but not least, there's a lot of code that is duplicated around in the scripts provided by whisper in its bin directory. Theses scripts should be very lightweight and be using the console_scripts facility of setuptools, but they actually contains a lot of (untested) code. Furthermore, some of that code is partially duplicated from the whisper.py library which is against DRY. Conclusion There are a few more things that made me stop considering whisper, but these are part of the whisper features, not necessarily code quality. One can also point out that the code is very condensed and hard to read, and that's a more general problem about how it is organized and abstracted. A lot of these defects are actually points that made me start writing The Hacker's Guide to Python a year ago. Running into this kind of code makes me think it was a really good idea to write a book on advice to write better Python code!
The Hacker's Guide to Python
A book I wrote talking about designing Python applications, state of the art, advice to apply when building your application, various Python tips, etc. Interested? Check it out.

13 September 2014

Laura Arjona: Disabling comments in the blog

I m getting more spam than the amount that I can stand in this blog. Comments are moderated, so the public is not suffering that, only me. From time to time I go to my dashboard and clean the spam. I m afraid that I delete some legit comment in these spam-cleaning-fevers, or, more probably, that a legit comment waits in the queue for several days (weeks?), just because I m lazy to deal with spam and let days pass by (until the fever comes). I think I m going to follow the wisdom of Bradley M. Kuhn and link to a pump.io note for comments on my blog posts (disabling them here in WordPress.com). I usually post a notice when I write something in my blog, so the only task is to update the blog post with the pump.io URL of the thread for comments. While WordPress.com allows to write comments quickly, without need of an account (you write just a name and an email, and the comment), in pump you need to have an account and sign in to comment. That looks as a bad thing, a barrier for people to participate. But of course, it stops spam :) After thinking about it a bit, pump.io it s a federated network, you can choose the pump server that they want, you can create a fake account, you don t need to provide personal information and it s another way to promote one of the social networks where I live. Other systems link to facebook, twitter, or other places, and nobody complains! Even when those services don t have any of the advantages of being in a federated free-software powered social network :) If anybody don t want to use pump.io but wants to comment, other ways to reach me or the related blog post are: So now it s decided, and this is the first post of this new experiment. This text is posted in pump.io too, and you can comment there :)
Filed under: My experiences and opinion, Tools Tagged: Blog, English, federation, pump.io, social networks, Wordpress

12 September 2014

Dariusz Dwornikowski: forwarding messages with attachments in mutt

This is a pain for every mutt user. I do not know why this solution is so hard to find. Just add these two lines to your .muttrc. 
set mime_forward
set mime_forward_rest=yes
This will forward an email with all the attachments, no scripts needed, no fancy tagging or reediting.

11 September 2014

Sylvestre Ledru: Rebuild of Debian using Clang 3.5.0

Clang 3.5.0 has just been released. A new rebuild has been done highlight the progress to get Debian built with clang. tl;dr: Great progress. We decreased from 9.5% to 5.7% of failures. Full results are available on http://clang.debian.net At time of the rebuild with 3.4.2, we had 2040 packages failing to build with clang. With 3.5.0, this dropped to 1261 packages. Fixes With Arthur Marble and Alexander Ovchinnikov, both GSoC students, we worked on various ways to decrease the number of errors. Upstream fixes First, the most obvious way, we fixed programming bugs/mistakes in upstream sources. Basically, we took categories of failure and fixed issues one after the other. We started with simple bugs like 'Wrong main declaration', 'non-void function should return a value' or 'Void function should not return a value'.

They are trivial to fix. We continued with harder fixes like ' Undefined reference' or 'Variable length array for a non POD (plain old data) element'.

So, besides these one, we worked on:
In total, we reported 295 bugs with patches. 85 of them have been fixed (meaning that the Debian maintainer uploaded a new version with the fix).

In parallel, I think that the switch by FreeBSD and Mac OS X to Clang also helped to fix various issues by upstreams. Hacking in clang As a parallel approach, we started to implement a suggestion from Linus Torvalds and a few others. Instead of trying to fix all upstream, where we can, we tried to update clang to improve the gcc compatibility.

gcc has many flags to disable or enable optimizations. Some of them are legacy, others have no sense in clang, etc. Instead of failing in clang with an error, we create a new category of warnings (showing optimization flag '%0' is not supported) and moved all relevant flags into it. Some examples, r212805, r213365, r214906 or r214907

We also updated clang to silent some useless arguments like -finput_charset=UTF-8 (r212110), clang being UTF-8 compliant.

Finally, we worked on the forwarding of linker flags. Clang and gcc have a very different behavior: when gcc does not know an argument, it is going to forward the argument to the linker. Clang, in this case, is going to reject the argument and fail with an error. In clang, we have to explicitly declare which arguments are going to be transfer to the linker. Of course, the correct way to pass arguments to the linker is to use -Xlinker or -Wl but the Debian rebuild proved that these shortcuts are used. Two of these arguments are now forwarded: New errors Just like in other releases, new warnings are added in clang. With (bad) usage of -Werror by upstream software, this causes new build failures: I also took the opportunity to add some further categorizations in the list of errors. Some examples: Next steps The Debile project being close to ready with Cl ment Schreiner's GSoC, we will now have an automatic and transparent way to rebuild packages using clang. Conclusion As stated, we can see a huge drop in term of number of failures over time:
Hopefully, Clang getting better and better, more and more projects adopting it as the default compiler or as a base for plugin/extension developments, this percentage will continue to decrease.
Having some kind of release goal with clang for Jessie+1 can now be considered as potentially reachable. Want to help? There are several things which can be done to help: Acknowledgments Thanks to David Suarez for the rebuilds of the archive, Arthur Marble and Alexander Ovchinnikov for their GSoC works and Nicolas S velin-Radiguet for the few fixes.
Original post blogged on b2evolution.

Matthias Klumpp: Listaller: Back to the future!

Listaller-Logo (with text)It is time for another report on Listaller, the cross-distro 3rd-party package installer, which is now in development for depending how you count 5-6 years. This will become a longer post, so you might grab some coffee or tea ;-) The original idea The Listaller project was initially started with the goal to make application deployment on Linux distributions as simple as possible, by providing a unified package installation format and tools which make building apps for multiple distributions easier and deployment of updates simple. The key ideas were: The current state The current release of Listaller handles all of this with a plugin for PackageKit, the cross-distro package-management abstraction layer. It hooks into PackageKit and reads information passing through to the native distributor backend, and if it encounters Listaller software, it handles it appropriately. It can also inject update information. This results in all Listaller software being shown in any PackageKit frontends, and people can work with it just like if the packages were native packages. Listaller package installations are controlled by a machine policy, so the administrator can decide that e.g. only packages from a trusted source (= GPG signature in trusted database) can be installed. Dependencies can be pulled from the distributor s repositories, or optionally from external sources, like the PyPI. This sounds good on paper, but the current implementation has various problems. The issues The current Listaller approach has some problems. The biggest one lies in the future: Soon, there will be no PackageKit plugins anymore! PackageKit 1.0 will remove support for them, because they appear to be a major source for crashes, even the in-tree plugins cause problems. Also, the PackageKit service itself is currently being trimmed of unneeded features and less-used code. These changes in PackageKit are great and needed for the project (and I support these efforts), but they cause a pretty huge problem for Listaller: The project relies on the PackageKit plugin if used without it, you loose the system-integration part, which is one of the key concepts of Listaller, and a primary goal. But this issue is not the only one. There are more. One huge problem for Listaller is dependency-solving: It needs to know where to get software from in case it isn t installed already. And that has to be done in a cross-distributional way. This is an incredibly complex task, and Listaller contains lots of workarounds for various quirks. It contains so much hacks for distro-specific stuff, that it became really hard to understand. The Listaller dependency model also became very complex, because it tried to handle many corner-cases. This is bad, of course. But the workarounds weren t added for fun, but because it was assumed to be easier than to fixing the root cause, which would have required collaboration between distributors and some changes on the stack, which seemed unlikely to happen at the time the code was written. The systemd effort Also a thing which affects Listaller, is the latest push from the systemd team to allow cross-distro 3rd-party installations to happen. I definitively recommend reading the linked blogpost from Lennart, if you have some spare time! The identified problems are the same as for Listaller, but the solution they propose is completely different, and about three orders of magnitude more invasive than whatever the Listaller project had in mind (I make these numbers up, so don t ask!). There are also a few issues I see with Lennarts approach, I will probably go into detail about that in another blogpost (e.g. it requires multiple copies of a library lying around, where one version might have a security vulnerability, and another one doesn t it s hard to ensure everything is up to date and secure that way, even if you have a top-notch sandbox). I have great respect for the systemd crew and especially Lennart, and I hope them to succeed with their efforts. However, I also think Listaller can achieve a similar things with a less-invasive solution, at least for the 3rd-party app-installations (Listaller is one of the partial-fix solutions with strict focus, so not a direct competitor to the holistic systemd approach. Both solutions could happily live together.) A step into the future Some might have guessed it already: There are some bigger changes coming to Listaller! The most important one is that there will be no Listaller anymore, at least not in its old form. Since the current code relies heavily on the PackageKit plugin, and contains some ugly workarounds, it doesn t make much sense to continue working on it. Instead, I started the Listaller.NEXT project, which is a rewrite of Listaller in C. There are a some goals for the rewrite: I made a last release of the 0.5.x series of Listaller, to work with PackageKit 0.9.x the future lies in the C port. If you are using Listaller (and I know of people who do, for example some deploy statically-linked stuff on internal test-setups with it), stay tuned. The packaging format will stay mostly compatible with the current version, so you will not see many changes there (the plan is to freeze it very soon, so no backwards-incompatible changes are made anymore). The o.5.x series will receive critical bugfixes if necessary. Help needed! As always, there is help needed! Writing C is not that difficult ;-) But user feedback is welcome as well, in case you have an idea. The new code will be hosted on Github in the new listaller-next branch (currently not that much to find there). Long-term, we will completely migrate away from Launchpad. You can expect more blogposts about the Listaller concepts and progress in the next months (as soon as I am done with some AppStream-related things, which take priority).

10 September 2014

Steve Kemp: kvm-hosting will be ceasing, soon.

Seven years ago I wanted to move on from the small virtual machine I had to a larger one. Looking at the the options available it seemed the best approach was to rent a big host, and divide it up into virtual machines myself. Renting a machine with 8Gb of RAM and 500Gb of disk-space, then dividing that into eights would give a decent spec and assuming that I found enough users to pay for the other slots/shares it would be economically viable too. After a few weeks I took the plunge, advertised here, and found users. I had six users: There were some niggles, one user seemed to suffer from connectivity problems more than the others, but on the whole the experiment worked out well. These days, thanks to BigV, Digital Ocean, and all the new-comers there is less need for this kind of thing so last December I announced that the service would cease - and gave all current users 1 year of free service to give them time to migrate away. The service was due to terminate in December, but triggered by some upcoming downtime where our host would have been moved, in the back of a van, from Manchester to York, I've taken the decision to stop it early. It was a fun experiment, it provided me with low cost hosting (subsidized by the other paying users), and provided some other people with hosting of their own that was setup nicely. The only outstanding question is what to do with the domain-names? I could let them expire, I could try to sell them, or I could donate them to other people running hosting setups. If anybody reading this has a use for kvm-hosting.org, kvm-hosting.net, or kvm-hosting.com, then do feel free to get in touch. No promises, obviously, but it'd be a shame for them to end up hosting adverts in a year or twos time..

8 September 2014

Joey Hess: propellor is d-i 2.0

I think I've been writing the second system to replace d-i with in my spare time for a couple months, and never noticed. I'm as suprised as you are, but consider this design: 
main :: IO
main = Installer.main
    & Installer.partition First "/boot" Ext3 (MiB 256)
    & Installer.partition Next "/" Ext4 (GiB 5)
    & Installer.partition Next "/home" Ext4 FreeSpace
    & Installer.grubBoots "hd0"
    & os (System (Debian Stable) "amd64")
    & Apt.stdSourcesList
    & Apt.installed ["task-gnome-desktop"]

main :: IO
main = Installer.main
    & Installer.target CdImage "installer.iso"
    & os (System (Debian Stable) "amd64")
    & Apt.stdSourcesList
    & Apt.installed ["task-xfce-desktop", "ghc", "propellor"]
    & User.autoLogin "root"
    & User.loginStarts "propellor --installer"
Well, now I know where propellor might end up if I felt like spending a month and adding a few thousand lines of code to it.

Jaldhar Vyas: Debconf 14 - Days 1 and 2

Unfortunately I was not able to attend debconf this year but thanks to the awesome video team the all the talks are available for your viewing pleasure. In order to recreate an authentic Portland experience, I took my laptop into the shower along with a vegan donut and had my children stand outside yelling excerpts from salon.com in whiny Canadianesque accents. Here are some notes I took as I watched the talks. Welcome Talk
Debian in the Dark Ages of Free software - Stefan Zacchiroli Weapons of the Geek - Gabriella Coleman bugs.debian.org -- Database Ho! - Don Armstrong Grub Ancient and Modern - Colin and Watson One year of fedmsg in Debian - Nicolas Dandrimont Coming of Age: My Life with Debian - Christine Spang Status report of the Debian Printing Team - Didier Raboud

4 September 2014

Adnan Hodzic: Debian PPA Utility

Debian remains to be my favorite distribution, however there s one thing that s missing, that thing is called PPA. There were numerous discussions on this topic inside of Debian, but AFAIK without any visible movement. Thus, I decided to publish a utility I ve been using for some time now. PPA s Since its introduction, PPA s are exclusively connected to Ubuntu and its derivatives (Mint, Elementary, etc ). But over time, a number of interesting projects appeared whose whole development is happening inside of PPA s. To name few, I m talking about TLP, Geary, Oracle Java Installer, Elementary OS and etc Some of these projects are in WNPP without much happening for a long time, i.e: TLP One option was to repackage these packages and then have them uploaded to Debian, or just go rogue and install them directly from its PPA s. Title of this post might hint which path I took. In theory, adding Ubuntu packages on your Debian system is a bad idea, and adding its PPA s is probably even worse. But, I ve been using couple (TLP, Geary, couple of custom icon sets) of these PPA s on my personal/work boxes, and to be honest, never had a single problem. Also, setting Pinning priority to low for the PPA you added is never a bad idea. Most of the PPA s I use, are usually fairly simple packages with single binary and dependencies which are found in Debian itself. Of course, I don t recommend adding PPA s on production boxes, or even PPA s such as GNOME3 Team PPA s, but rather add Apt Pinning on your system and fetch those packages directly from Debian. Debian PPA Utility Is a very simple utility, which adds add-apt-repository binary script that allows you to add PPA s on Debian. Code is available on GitHub, it s licensed as GPLv3, so feel free to fork it, improve it, use it and abuse it. How to use it? Download/Build package You can download my signed package (source and changes file are in same directory) Or you can build your own by running "dpkg-buildpackage -uc -us" inside of the debian-ppa source directory. Install/Add PPA s After you install the package, you re able to run add-apt-repository and add PPA s. i,e: sudo add-apt-repository ppa:linrunner/tlp debian ppa utility Currently, Debian PPA Utility only works on >= Wheezy. At this point I have no plans to try pushing this utility into Debian, as I m sure even this blog post will be labelled as heresy by many. Update! It was just pointed to me that add-apt-repository is available in software-properties-common package. However, PPA s added by add-apt-repository binary present in this package instead of adding Ubuntu codename s to your list file, will add Debian codenames which without change will make whole PPA entry useless. I believe codename handling is better in Debian PPA Utility . I admit, my only mistake is, instead of fixing things in software-properties-common package, I made a completely new utility which aims to do the same thing. Added: conflicts/replaces: software-properties-common to debian/control file. Anyway, enjoy!

Next.

Previous.