Search Results: "xam"

28 January 2015

Dirk Eddelbuettel: RInside 0.2.12

A new release 0.2.12 of RInside is now on CRAN. RInside provides a set of convenience classes which facilitate embedding of R inside of C++ applications and programs, using the classes and functions provided by the Rcpp integration package. This release adds new examples which were contributed by Christian Authmann, plus some updates and fixes including one requested by the CRAN maintainers regarding GNU extensions to Makefile. The NEWS extract below has more details.
Changes in RInside version 0.2.12 (2015-01-27)
  • Several new examples have been added (with most of the work done by Christian Authmann):
    • standard/rinside_sample15.cpp shows how to create a lattice plot (following a StackOverflow question)
    • standard/rinside_sample16.cpp shows object wrapping, and exposing of C++ functions
    • standard/rinside_sample17.cpp does the same via C++11
    • sandboxed_servers/ adds an entire framework of client/server communication outside the main process (but using a subset of supported types)
  • standard/rinside_module_sample9.cpp was repaired following a fix to InternalFunction in Rcpp
  • For the seven example directories which contain a Makefile, the Makefile was renamed GNUmakefile to please R CMD check as well as the CRAN Maintainers.
CRANberries also provides a short report with changes from the previous release. More information is on the RInside page. Questions, comments etc should go to the rcpp-devel mailing list off the Rcpp R-Forge page.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

27 January 2015

Matthias Klumpp: AppStream 0.8 released!

Yesterday I released version 0.8 of AppStream, the cross-distribution standard for software metadata, that is currently used by GNOME-Software, Muon and Apper in to display rich metadata about applications and other software components. What s new? The new release contains some tweaks on AppStreams documentation, and extends the specification with a few more tags and refinements. For example we now recommend sizes for screenshots. The recommended sizes are the ones GNOME-Software already uses today, and it is a good idea to ship those to make software-centers look great, as others SCs are planning to use them as well. Normal sizes as well as sizes for HiDPI displays are defined. This change affects only the distribution-generated data, the upstream metadata is unaffected by this (the distro-specific metadata generator will resize the screenshots anyway). Another addition to the spec is the introduction of an optional <source_pkgname/> tag, which holds the source package name the packages defined in <pkgname/> tags are built from. This is mainly for internal use by the distributor, e.g. it can decide to use this information to link to internal resources (like bugtrackers, package-watch etc.). It may also be used by software-center applications as additional information to group software components. Furthermore, we introduced a <bundle/> tag for future use with 3rd-party application installation solutions. The tag notifies a software-installer about the presence of a 3rd-party application bundle, and provides the necessary information on how to install it. In order to do that, the software-center needs to support the respective installation solution. Currently, the Limba project and Xdg-App bundles are supported. For software managers, it is a good idea to implement support for 3rd-party app installers, as soon as the solutions are ready. Currently, the projects are worked on heavily. The new tag is currently already used by Limba, which is the reason why it depends on the latest AppStream release. How do I get it? All AppStream libraries, libappstream, libappstream-qt and libappstream-glib, are supporting the 0.8 specification in their latest version so in case you are using one of these, you don t need to do anything. For Debian, the DEP-11 spec is being updated at time, and the changes will land in the DEP-11 tools soon. Improve your metadata! This call goes especilly to many KDE projects! Getting good data is partly a task for the distributor, since packaging issues can result in incorrect or broken data, screenshots need to be properly resized etc. However, flawed upstream data can also prevent software from being shown, since software with broken data or missing data will not be incorporated in the distro XML AppStream data file. Richard Hughes of Fedora has created a nice overview of software failing to be included. You can see the failed-list here the data can be filtered by desktop environment etc. For KDE projects, a Comment= field is often missing in their .desktop files (or a <summary/> tag needs to be added to their AppStream upstream XML file). Keep in mind that you are not only helping Fedora by fixing these issues, but also all other distributions cosuming the metadata you ship upstream. For Debian, we will have a similar overview soon, since it is also a very helpful tool to find packaging issues. If you want to get more information on how to improve your upstream metadata, and how new metadata should look like, take a look at the quickstart guide in the AppStream documentation.

Jingjie Jiang: Yet another post.

In the middle of OPW internship I originally thought taking part in FOSSOPW is a great chance to lift my coding skills and I shouldn t miss it. As time passes by, I now have a new thought towards it. # 1 It do improves your coding skill. Zack, Matthieu and I often have discussions on coding style. For example, once Zack said, For code like this, you should explicitly use an if/else clause, not if-return. I was totally unaware of this sort of issue. Actually I even didn t know how I should call this problem. Matthieu gave me a detailed FYI link on this in no time. Besides, my completeness of thinking is also trained. Recently, I was fixing a HTTP GET Method ?suite=suite-name issue. It s a trivial task. And you know most trivial tasks require lots of scattered modification on the source code base. I did have fixed most places, such as the pages of /src/packagename and /search/ . Zack did a thorough review, and pointed out that the pages rendered by /prefix has some malformed urls in the HTML. Waiiiit, I should have noticed it. But somehow I missed it. Maybe because my mind was wandering at that time? This made me think, I shall have a thorough view of what I should do before getting my hands dirty. Or more preferably, if I could write down what I exactly want to achieve before coding, then silly problems definitely wouldn t occur. This may sound a little bit like TDD. ;). # 2 It makes you look like a (not-that-good) ninja. I use a macbook. It s not my fault! I ve tried several times, but I never successfully find a laptop that is not capable of boiling eggs when running Debian. (and especially KDE+Debian). So I have no way but switched to OSX. The development of Debsources happens on a remote Ubuntu LTS (now Debian SID, haha) virtual machine. Of course I have to install all the dependency on my own, e.g., Postgres, set up port-forwarding, e.g., ssh -D, write automate shell scripts, e.g., dash, but more importantly, I am forced to live under the dark terminal with no GUI. You know the feeling when pain hurts? Yes, exactly! But I survived. How shall I call myself now? A dedicated with-a-lot-of-useless-plugin-installed vimmer? A fond-of-fancy-window tmux-er? Yep, both. I finally found a comfort zone under the black-white-blinking screen. I wonder how people feel when they see a girl hanging out in the library, facing a full-screened black console, typing at a speed of 140wpm (Yeah, I am kidding). I don t know, but please don t call me a geek. Show me your respect, I am a ninja! # 3 It tells you communication is the most important. I bet anyone who has participated in a group-based project would understand what I mean. For one perspective, communication helps to eliminate misunderstanding. So I won t doing some useless stuff for all day and finally find out that it totally doesn t meet the requirement. On the other hand, it speeds up your learning process. I often have problems on git. So in the email I will complain if I mess up with the git repo. After a short while, my dear mentors will reply in detail on how to correctly do the git stuff. My OPW journey is cool! ;).

26 January 2015

Daniel Pocock: Get your Nagios issues as an iCalendar feed

The other day I demonstrated how to get your Github issues/bugs as an iCalendar feed. I'm planning to take this concept further and I just whipped up another Python script, exposing Nagios issues as an iCalendar feed. The script is nagios-icalendar. Usage is explained concisely in the README file, it takes just minutes to get up and running. One interesting feature is that you can append a contact name to the URL and just get the issues for that contact, e.g.: 
http://nagios-server.example.org:5001?contact=daniel
Screenshots Here I demonstrate using Mozilla Lightning / Iceowl-extension to aggregate issues from Nagios, the Fedora instance of Bugzilla and Lumicall's Github issues into a single to-do list.

Francois Marier: Using unattended-upgrades on Rackspace's Debian and Ubuntu servers

I install the unattended-upgrades package on almost all of my Debian and Ubuntu servers in order to ensure that security updates are automatically applied. It works quite well except that I still need to login manually to upgrade my Rackspace servers whenever a new rackspace-monitoring-agent is released because it comes from a separate repository that's not covered by unattended-upgrades. It turns out that unattended-upgrades can be configured to automatically upgrade packages outside of the standard security repositories but it's not very well documented and the few relevant answers you can find online are still using the old whitelist syntax.

Initial setup The first thing to do is to install the package if it's not already done: 
apt-get install unattended-upgrades
and to answer yes to the automatic stable update question. If you don't see the question (because your debconf threshold is too low -- change it with dpkg-reconfigure debconf), you can always trigger the question manually: 
dpkg-reconfigure -plow unattended-upgrades
Once you've got that installed, the configuration file you need to look at is /etc/apt/apt.conf.d/50unattended-upgrades.

Whitelist matching criteria Looking at the unattended-upgrades source code, I found the list of things that can be used to match on in the whitelist:
  • origin (shortcut: o)
  • label (shortcut: l)
  • archive (shortcut: a)
  • suite (which is the same as archive)
  • component (shortcut: c)
  • site (no shortcut)
You can find the value for each of these fields in the appropriate _Release file under /var/lib/apt/lists/. Note that the value of site is the hostname of the package repository, also present in the first part these *_Release filenames (stable.packages.cloudmonitoring.rackspace.com in the example below). In my case, I was looking at the following inside /var/lib/apt/lists/stable.packages.cloudmonitoring.rackspace.com_debian-wheezy-x86%5f64_dists_cloudmonitoring_Release: 
Origin: Rackspace
Codename: cloudmonitoring
Date: Fri, 23 Jan 2015 18:58:49 UTC
Architectures: i386 amd64
Components: main
...
which means that, in addition to site, the only things I could match on were origin and component since there are no Suite or Label fields in the Release file. This is the line I ended up adding to my /etc/apt/apt.conf.d/50unattended-upgrades: 
 Unattended-Upgrade::Origins-Pattern  
         // Archive or Suite based matching:
         // Note that this will silently match a different release after
         // migration to the specified archive (e.g. testing becomes the
         // new stable).
 //      "o=Debian,a=stable";
 //      "o=Debian,a=stable-updates";
 //      "o=Debian,a=proposed-updates";
         "origin=Debian,archive=stable,label=Debian-Security";
         "origin=Debian,archive=oldstable,label=Debian-Security";
+        "origin=Rackspace,component=main";
  ;

Testing To ensure that the config is right and that unattended-upgrades will pick up rackspace-monitoring-agent the next time it runs, I used: 
unattended-upgrade --dry-run --debug
which should output something like this: 
Initial blacklisted packages: 
Starting unattended upgrades script
Allowed origins are: ['origin=Debian,archive=stable,label=Debian-Security', 'origin=Debian,archive=oldstable,label=Debian-Security', 'origin=Rackspace,component=main']
Checking: rackspace-monitoring-agent (["<Origin component:'main' archive:'' origin:'Rackspace' label:'' site:'stable.packages.cloudmonitoring.rackspace.com' isTrusted:True>"])
pkgs that look like they should be upgraded: rackspace-monitoring-agent
...
Option --dry-run given, *not* performing real actions
Packages that are upgraded: rackspace-monitoring-agent

Making sure that automatic updates are happening In order to make sure that all of this is working and that updates are actually happening, I always install apticron on all of the servers I maintain. It runs once a day and emails me a list of packages that need to be updated and it keeps doing that until the system is fully up-to-date. The only thing missing from this is getting a reminder whenever a package update (usually the kernel) requires a reboot to take effect. That's where the update-notifier-common package comes in. Because that package will add a hook that will create the /var/run/reboot-required file whenever a kernel update has been installed, all you need to do is create a cronjob like this in /etc/cron.daily/reboot-required: 
#!/bin/sh
cat /var/run/reboot-required 2> /dev/null   true
assuming of course that you are already receiving emails sent to the root user (if not, add the appropriate alias in /etc/aliases and run newaliases).

25 January 2015

Chris Lamb: Recent Redis hacking

I've done a bunch of hacking on the Redis key/value database server recently:
I also made the following changes to the Debian packaging:

Joey Hess: making propellor safer with GADTs and type families

Since July, I have been aware of an ugly problem with propellor. Certain propellor configurations could have a bug. I've tried to solve the problem at least a half-dozen times without success; it's eaten several weekends. Today I finally managed to fix propellor so it's impossible to write code that has the bug, bending the Haskell type checker to my will with the power of GADTs and type-level functions. the bug Code with the bug looked innocuous enough. Something like this: 
foo :: Property
foo = property "foo" $
    unlessM (liftIO $ doesFileExist "/etc/foo") $ do
        bar <- liftIO $ readFile "/etc/foo.template"
        ensureProperty $ setupFoo bar
The problem comes about because some properties in propellor have Info associated with them. This is used by propellor to introspect over the properties of a host, and do things like set up DNS, or decrypt private data used by the property. At the same time, it's useful to let a Property internally decide to run some other Property. In the example above, that's the ensureProperty line, and the setupFoo Property is run only sometimes, and is passed data that is read from the filesystem. This makes it very hard, indeed probably impossible for Propellor to look inside the monad, realize that setupFoo is being used, and add its Info to the host. Probably, setupFoo doesn't have Info associated with it -- most properties do not. But, it's hard to tell, when writing such a Property if it's safe to use ensureProperty. And worse, setupFoo could later be changed to have Info. Now, in most languages, once this problem was noticed, the solution would probably be to make ensureProperty notice when it's called on a Property that has Info, and print a warning message. That's Good Enough in a sense. But it also really stinks as a solution. It means that building propellor isn't good enough to know you have a working system; you have to let it run on each host, and watch out for warnings. Ugh, no! the solution This screams for GADTs. (Well, it did once I learned how what GADTs are and what they can do.) With GADTs, Property NoInfo and Property HasInfo can be separate data types. Most functions will work on either type (Property i) but ensureProperty can be limited to only accept a Property NoInfo. 
data Property i where
    IProperty :: Desc -> ... -> Info -> Property HasInfo
    SProperty :: Desc -> ... -> Property NoInfo
data HasInfo
data NoInfo
ensureProperty :: Property NoInfo -> Propellor Result
Then the type checker can detect the bug, and refuse to compile it. Yay! Except ... Property combinators There are a lot of Property combinators in propellor. These combine two or more properties in various ways. The most basic one is requires, which only runs the first Property after the second one has successfully been met. So, what's it's type when used with GADT Property? 
requires :: Property i1 -> Property i2 -> Property ???
It seemed I needed some kind of type class, to vary the return type. 
class Combine x y r where
    requires :: x -> y -> r
Now I was able to write 4 instances of Combines, for each combination of 2 Properties with HasInfo or NoInfo. It type checked. But, type inference was busted. A simple expression like 
foo  requires  bar
blew up: 
   No instance for (Requires (Property HasInfo) (Property HasInfo) r0)
      arising from a use of  requires'
    The type variable  r0' is ambiguous
    Possible fix: add a type signature that fixes these type variable(s)
    Note: there is a potential instance available:
      instance Requires
                 (Property HasInfo) (Property HasInfo) (Property HasInfo)
        -- Defined at Propellor/Types.hs:167:10
To avoid that, it needed ":: Property HasInfo" appended -- I didn't want the user to need to write that. I got stuck here for an long time, well over a month. type level programming Finally today I realized that I could fix this with a little type-level programming. 
class Combine x y where
    requires :: x -> y -> CombinedType x y
Here CombinedType is a type-level function, that calculates the type that should be used for a combination of types x and y. This turns out to be really easy to do, once you get your head around type level functions. 
type family CInfo x y
type instance CInfo HasInfo HasInfo = HasInfo
type instance CInfo HasInfo NoInfo = HasInfo
type instance CInfo NoInfo HasInfo = HasInfo
type instance CInfo NoInfo NoInfo = NoInfo
type family CombinedType x y
type instance CombinedType (Property x) (Property y) = Property (CInfo x y)
And, with that change, type inference worked again! \o/ (Bonus: I added some more intances of CombinedType for combining things like RevertableProperties, so propellor's property combinators got more powerful too.) Then I just had to make a massive pass over all of Propellor, fixing the types of each Property to be Property NoInfo or Property HasInfo. I frequently picked the wrong one, but the type checker was able to detect and tell me when I did. A few of the type signatures got slightly complicated, to provide the type checker with sufficient proof to do its thing... 
before :: (IsProp x, Combines y x, IsProp (CombinedType y x)) => x -> y -> CombinedType y x
before x y = (y  requires  x)  describe  (propertyDesc x)
onChange
    :: (Combines (Property x) (Property y))
    => Property x
    => Property y
    => CombinedType (Property x) (Property y)
onChange = -- 6 lines of code omitted
fallback :: (Combines (Property p1) (Property p2)) => Property p1 -> Property p2 -> Property (CInfo p1 p2)
fallback = -- 4 lines of code omitted
.. This mostly happened in property combinators, which is an acceptable tradeoff, when you consider that the type checker is now being used to prove that propellor can't have this bug. Mostly, things went just fine. The only other annoying thing was that some things use a [Property], and since a haskell list can only contain a single type, while Property Info and Property NoInfo are two different types, that needed to be dealt with. Happily, I was able to extend propellor's existing (&) and (!) operators to work in this situation, so a list can be constructed of properties of several different types: 
propertyList "foos" $ props
    & foo
    & foobar
    ! oldfoo
conclusion The resulting 4000 lines of changes will be in the next release of propellor. Just as soon as I test that it always generates the same Info as before, and perhaps works when I run it. (eep) These uses of GADTs and type families are not new; this is merely the first time I used them. It's another Haskell leveling up for me. Anytime you can identify a class of bugs that can impact a complicated code base, and rework the code base to completely avoid that class of bugs, is a time to celebrate!

24 January 2015

Daniel Pocock: Get your Github issues as an iCalendar feed

I've just whipped up a Python script that renders Github issue lists from your favourite projects as an iCalendar feed. The project is called github-icalendar. It uses Python Flask to expose the iCalendar feed over HTTP. It is really easy to get up and running. All the dependencies are available on a modern Linux distribution, for example: 
$ sudo apt-get install python-yaml python-icalendar python-flask python-pygithub
Just create an API token in Github and put it into a configuration file like this: 
api_token: 6b36b3d7579d06c9f8e88bc6fb33864e4765e5fac4a3c2fd1bc33aad
bind_address: ::0
bind_port: 5000
filter: all
Run it from the shell: 
$ ./github_icalendar/main.py github-ics.cfg
and connect to it with your favourite iCalendar client. Consolidating issue lists from Bugzilla, Github, Debian BTS and other sources A single iCalendar client can usually support multiple sources and thereby consolidate lists of issues from multiple bug trackers. This can be much more powerful than combining RSS bug feeds because iCalendar has built-in support for concepts such as priority and deadline. The client can use these to help you identify the most critical issues across all your projects, no matter which bug tracker they use. Bugzilla bugtrackers already expose iCalendar feeds directly, just look for the iCalendar link at the bottom of any search results page. Here is an example URL from the Mozilla instance of Bugzilla. The Ultimate Debian Database consolidates information from the Debian and Ubuntu universe and can already export it as an RSS feed, there is discussion about extrapolating that to an iCalendar feed too. Further possibilities
  • Prioritizing the issues in Github and mapping these priorities to iCalendar priorities
  • Creating tags in Github that allow issues to be ignored/excluded from the feed (e.g. excluding wishlist items)
  • Creating summary entries instead of listing all the issues, e.g. a single task entry with the title Fix 2 critical bugs for project foo
Screenshots The screenshots below are based on the issue list of the Lumicall secure SIP phone for Android. Screenshot - Mozilla Thunderbird/Lightning (Icedove/Iceowl-extension on Debian)

Dirk Eddelbuettel: Rcpp 0.11.4

A new release 0.11.4 of Rcpp is now on the CRAN network for GNU R, and an updated Debian package will be uploaded in due course. Rcpp has become the most popular way of enhancing GNU R with C++ code. As of today, 323 packages on CRAN depend on Rcpp for making analyses go faster and further; BioConductor adds another 41 packages, and casual searches on GitHub suggests dozens mores. This release once again adds a large number of small bug fixes, polishes and enhancements. And like the last time, these changes were made by a group of seven different contributors (counting code commits) plus three more providing concrete suggestions. This shows that the Rcpp development and maintenance rests a large number of (broad) shoulders. See below for a detailed list of changes extracted from the NEWS file.
Changes in Rcpp version 0.11.4 (2015-01-20)
  • Changes in Rcpp API:
    • The ListOf<T> class gains the .attr and .names methods common to other Rcpp vectors.
    • The [dpq]nbinom_mu() scalar functions are now available via the R:: namespace when R 3.1.2 or newer is used.
    • Add an additional test for AIX before attempting to include execinfo.h.
    • Rcpp::stop now supports improved printf-like syntax using the small tinyformat header-only library (following a similar implementation in Rcpp11)
    • Pairlist objects are now protected via an additional Shield<> as suggested by Martin Morgan on the rcpp-devel list.
    • Sorting is now prohibited at compile time for objects of type List, RawVector and ExpressionVector.
    • Vectors now have a Vector::const_iterator that is 'const correct' thanks to fix by Romain following a bug report in rcpp-devel by Martyn Plummer.
    • The mean() sugar function now uses a more robust two-pass method, and new unit tests for mean() were added at the same time.
    • The mean() and var() functions now support all core vector types.
    • The setequal() sugar function has been corrected via suggestion by Qiang Kou following a bug report by S ren H jsgaard.
    • The macros major, minor, and makedev no longer leak in from the (Linux) system header sys/sysmacros.h.
    • The push_front() string function was corrected.
  • Changes in Rcpp Attributes:
    • Only look for plugins in the package's namespace (rather than entire search path).
    • Also scan header files for definitions of functions to be considerd by Attributes.
    • Correct the regular expression for source files which are scanned.
  • Changes in Rcpp unit tests
    • Added a new binary test which will load a pre-built package to ensure that the Application Binary Interface (ABI) did not change; this test will (mostly or) only run at Travis where we have reasonable control over the platform running the test and can provide a binary.
    • New unit tests for sugar functions mean, setequal and var were added as noted above.
  • Changes in Rcpp Examples:
    • For the (old) examples ConvolveBenchmarks and OpenMP, the respective Makefile was renamed to GNUmakefile to please R CMD check as well as the CRAN Maintainers.
Thanks to CRANberries, you can also look at a diff to the previous release As always, even fuller details are on the Rcpp Changelog page and the Rcpp page which also leads to the downloads page, the browseable doxygen docs and zip files of doxygen output for the standard formats. A local directory has source and documentation too. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Diego Escalante Urrelo: Link Pack #05

Lever Rukhin Photographs Los Angeles From His Car
Lever Rukhin shoots the sketchiest parts of Los Angeles from his car, taking a really unique perspective that helps you perceive what LA looks like, if you were in a car An experience that is apparently common to all LA people. People drive too much in the US :-). It s a very interesting interview that goes well with his full site: Lev Rukhin. What I love about this, besides the whole premise, is that Lev went the extra mile and actually hacked his car to make the images he wanted:
Phoblographer: It looks like many of these images have artificial lighting in them. What s your gear setup, and how do you introduce so much light into the scene from your car? Lever: About 9 months ago, I affixed a Mola beauty dish onto the roof rack of my 75 Volvo and juice it with a profoto bi-tube. This takes a bit of practice, as making a turn changes the light completely, which I always try to keep balanced. The Canon 5D3 with a 24mm f1.4 is set up on a tripod. The strobe has allowed me to capture more detail as well as creating a somewhat surreal feel to the sets.
Lev Rukhin Lev Rukhin http://www.levrukhin.com/
The Invisible Woman: A conversation with Bj rk
Bj rk is that Icelandic singer we all hear about but never really pay much attention to because her music is too smart for our simple ears. In this interview she goes over how her latest album is a very personal work, and unexpectedly (?) ends talking about how problematic it s been to be a female auteur in her generation. I have seen the same problem she denounces about people assuming that the male members of a team did all the work while the women just sticked to making coffee and sandwiches. I ve worked with exceptional women that don t get enough credit, but I ve also worked with potentially exceptional women who don t give themselves enough credit. It s a very interesting read, specially since it comes from someone who couldn t be higher in the art food chain. Bj rk is god-damn Bj rk. Only thing that bugs me is that Pitchfork decided to hold back most of the interview for publishing next month. I ll try to go back and read it in full, but I wonder if the technique works for them or if perhaps they are missing the opportunity for a bigger impact. But I digress.
Pitchfork: The world has a difficult time with the female auteur. B: I have nothing against Kanye West. Help me with this I m not dissing him this is about how people talk about him. With the last album he did, he got all the best beatmakers on the planet at the time to make beats for him. A lot of the time, he wasn t even there. Yet no one would question his authorship for a second. If whatever I m saying to you now helps women, I m up for saying it. For example, I did 80% of the beats on Vespertine and it took me three years to work on that album, because it was all microbeats it was like doing a huge embroidery piece. Matmos came in the last two weeks and added percussion on top of the songs, but they didn t do any of the main parts, and they are credited everywhere as having done the whole album. [Matmos ] Drew [Daniel] is a close friend of mine, and in every single interview he did, he corrected it. And they don t even listen to him. It really is strange.
In Defense of the Selfie Stick
Miguel proposes a different take on the consequences of the selfies stick:
When you ask someone to take a picture of you, technically, they are the photographer, and they own the copyright of your picture. ( ) All of a sudden, your backpacking adventure in Europe requires you to pack a stack of legal contracts. Now your exchange goes from Can you take a picture of us? to Can you take a picture of us, making sure that the church is on the top right corner, and also, I am going to need you to sign this paper .
I don t know what s with the selfie stick hate. Let people have fun, it doesn t hurt. If anything, it prevents them from asking you to take their photo, and if we already established you are the kind of people not a big fan of strangers, all the better, right? Why Top Tech CEOs Want Employees With Liberal Arts Degrees
Here s a small extra. When I decided to pursue a humanities/art formal training, I got many naysayers telling me that I was screwing up not specializing even more as a formal (titled) engineer. I argued then, and now, that if I was gonna pay for training, I might as well pay for training outside my comfort zone. The result resonates perfectly with this article. Of course, it s not like the thing is settled, but I can back the various quotes in there. Working with purely technical/engineering types can be an echo chamber, and having trained myself in the humanities and arts I have become incredibly much more sensitive to the human factor of things. I used to think I was already good at this (because we hacker types have lots of confidence), but studying humanities like human communication, social conflict and development, film language, etc; it all has made me a much more capable hacker of things. There s also a nice argument to be made about joining the arts when you are already highly skilled on technical matters. Like Robert Rodr guez s teacher (mentioned in his diary/book Rebel Without a Cause, which I also have to review soon) used to say (generous paraphrasing here): the world is of those who can be their own creative and their own technician.
Both Yi and Sheer recognize that the scientific method is valuable, with its emphasis on logic and reason, especially when dealing with data or engineering problems. But they believe this approach can sometimes be limiting. When I collaborate with people who have a strictly technical background, says Yi, the perspective I find most lacking is an understanding of what motivates people and how to balance multiple factors that are at work outside the realm of technology.
Interesting food for thought, specially if you know an engineer that ditches the arts as of little value for personal growth in their careers/life.
Read more Link Pack, you ll love it

23 January 2015

Chris Lamb: Slack integration for Django

I recently started using the Slack group chat tool in a few teams. Wishing to add some vanity notifications such as sales and user growth milestones from some Django-based projects, I put together an easy-to-use integration between the two called django-slack. Whilst you can use any generic Python-based method of sending messages to Slack, using a Django-specific integration has some advantages:
Here is an example of how to send a message from a Django view: 
from django_slack import slack_message
@login_required
def view(request, item_id):
    item = get_object_or_404(Item, pk=item_id)
    slack_message('items/viewed.slack',  
        'item': item,
        'user': request.user,
     )
    return render(request, 'items/view.html',  
        'item': item,
     )
Where items/viewed.slack (in your templates directory) might contain: 
 % extends django_slack % 
 % block text % 
  user.get_full_name   just viewed   item.title   (  item.content urlize  ).
 % endblock %
.slack files are regular Django templates text is automatically escaped as appropriate and that you can use the regular template filters and tags such as urlize, loops, etc. By default, django-slack posts to the #general channel, but it can be overridden on a per-message basis by specifying a channel block: 
 % block channel % 
#mychannel
 % endblock %
You can also set the icon, URL and emoji in a similar fashion. You can set global defaults for all of these attributes to avoid DRY violations within .slack templates as well. For more information please see the project homepage or read the documentation. Patches and other contributions are welcome via the django-slack GitHub project.

Richard Hartmann: Release Critical Bug report for Week 04

The UDD bugs interface currently knows about the following release critical bugs: >How do we compare to the Squeeze and Wheezy release cycles?
Week Squeeze Wheezy Jessie
43 284 (213+71) 468 (332+136) 319 (240+79)
44 261 (201+60) 408 (265+143) 274 (224+50)
45 261 (205+56) 425 (291+134) 295 (229+66)
46 271 (200+71) 401 (258+143) 427 (313+114)
47 283 (209+74) 366 (221+145) 342 (260+82)
48 256 (177+79) 378 (230+148) 274 (189+85)
49 256 (180+76) 360 (216+155) 226 (147+79)
50 204 (148+56) 339 (195+144) ???
51 178 (124+54) 323 (190+133) 189 (134+55)
52 115 (78+37) 289 (190+99) 147 (112+35)
1 93 (60+33) 287 (171+116) 140 (104+36)
1 93 (60+33) 287 (171+116) 140 (104+36)
2 82 (46+36) 271 (162+109) 157 (124+33)
3 25 (15+10) 249 (165+84) 172 (128+44)
4 14 (8+6) 244 (176+68) 187 (132+55)
5 2 (0+2) 224 (132+92)
6 release! 212 (129+83)
7 release+1 194 (128+66)
8 release+2 206 (144+62)
9 release+3 174 (105+69)
10 release+4 120 (72+48)
11 release+5 115 (74+41)
12 release+6 93 (47+46)
13 release+7 50 (24+26)
14 release+8 51 (32+19)
15 release+9 39 (32+7)
16 release+10 20 (12+8)
17 release+11 24 (19+5)
18 release+12 2 (2+0)
Graphical overview of bug stats thanks to azhag:

Jaldhar Vyas: Mini-Debconf Mumbai 2015

Last weekend I went to Mumbai to attend the Mini-Debconf held at IIT-Bombay. These are my impressions of the trip. Arrival and Impressions of Mumbai Getting there was a quite an adventure in itself. Unlike during my ill-fated attempt to visit a Debian event in Kerala last year when a bureaucratic snafu left me unable to get a visa, the organizers started the process much earlier at their end this time and with proper permissions. Yet in India, the wheels only turn as fast as they want to turn so despite their efforts, it was only literally at the last minute that I actually managed to secure my visa. I should note however that Indian government has done a lot to improve the process compared to the hell I remember from, say, a decade ago. It's fairly straightforward for tourist visas now and I trust they will get around to doing the same for conference visas in the fullness of time. I didn't want to commit to buying a plane ticket until I had the visa so I became concerned that the only flights left would be either really expensive or on the type of airline that flies you over Syria or under the Indian Ocean. I lucked out and got a good price on a Swiss Air flight, not non-stop but you can't have everything. So Thursday afternoon I set off for JFK. With only one small suitcase getting there by subway was no problem and I arrived and checked in with plenty of time. Even TSA passed me through with only a minimal amount of indignity. The first leg of my journey took me to Zurich in about eight hours. We were only in Zurich for an hour and then (by now Friday) it was another 9 hours to Mumbai. Friday was Safala Ekadashi but owing to the necessity of staying hydrated on a long flight I drank a lot of water and ate some fruit which I don't normally do on a fasting day. It was tolerable but not too pleasant; I definitely want to try and make travel plans to avoid such situations in the future. Friday evening local time I got to Mumbai. Chhattrapati Shivaji airport has improved a lot since I saw t last and now has all the amenities an international traveller needs including unrestricted free wifi (Zurich airport are you taking notes?) But here my first ominous piece of bad luck began. No sign of my suitcase. Happily some asking around revealed that it had somehow gotten on some earlier Swiss Air flight instead of the one I was on and was actually waiting for me. I got outside and Debian Developer Praveen Arimbrathodiyil was waiting to pick me up. Normally I don't lke staying in Mumbai very much even though I have relatives there but that's because we usually went during July-August the monsoon season when Mumbai reverts back to the swampy archipelago it was originally built on. This time the weather was nice, cold by local standards, but lovely and spring-like to someone from snowy New Jersey. There have been a lot of improvements to the road infrastructure and people are actually obeying the traffic laws. (Within reason of course. Whether or not a family of six can arrange themselves on one Bajaj scooter is no business of the cops.) The Hotel Tuliip (yes, two i's. Manager didn't know why.) Residency where I was to stay while not quite a five star establishment was adequate for my needs with a bed, hot water shower, and air conditioning. And a TV which to the bellhops great confusion I did not want turned on. (He asked about five times.) There was no Internet access per se but the manager offered to hook up a wireless router to a cable. Which on closer inspection turned out to have been severed at the base. He assured me it would be fixed tomorrow so I didn't complain and decided to do something more productive thank checking my email like sleeping. The next day I woke up in total darkness. Apparently there had been some kind of power problem during the night which tripped a fuse or something. A call to the front desk got them to fix that and then the second piece of bad luck happened. I plugged my Thinkpad in and woke it up from hibernation and a minute later there was a loud pop from the power adapter. Note I have a travel international plug adapter with surge protector so nothing bad ought to have happened but the laptop would on turning on display the message "critical low battery error" and immediately power off. I was unable to google what that meant without Internet access but I decided not to panic and continue getting ready. I would have plenty of opportunity to troubleshoot at the conference venue. Or so I thought... I took an autorickshaw to IIT. There also there have been positive improvements. Being quite obviously a foreigner I was fully prepared to be taken along the "scenic route." But now there are fair zones and the rickshaws all have (tamperproof!) digital fare meters so I was deposited at the main gate without fuss. After reading a board with a scary list of dos and don'ts I presented myself at security only to be inexplicably waved through without a second glance. Later I found out they've abandoned all the security theatre but not got around to updating the signs yet. Mumbai is one of the biggest, densely populated cities in the world but the IIT campus is an oasis of tranquility on the shores of Lake Powai. It's a lot bigger than it looked on the map so I had to wander around a bit before I reached the conference venue but I did make for the official registration time. Registration I was happy to meet several old friends (Such as Kartik Mistry and Kumar Appiah who along with Praveen and myself were the other DDs there,) people who I've corresponded with but never met, and many new people. I'm told 200+ people registered altogether. Most seemed to be students from IIT and elsewhere in Mumbai but there were also some Debian enthusiasts from further afield and most hearteningly some "civilians" who wanted to know what this was all about. With the help of a borrowed Thinkpad adapter I got my laptop running again. (Thankfully, despite the error message, the battery itself was unharmed.) However, my streak of bad luck was not yet over. It was that very weekend that IIT had a freak campus-wide network outage something that had never happened before. And as the presentation for the talk I was to give had apparently been open when I hibernated my laptop the night before, the sudden forced shutdown had trashed the file. (ls showed it as 0 length. An fsck didn't help.) I possibly had a backup on my server but with no Internet access I had no way to retrieve it. I still remained cool. The talk was scheduled for the second day so I could recover it at the hotel. Keynotes Professor Kannan Maudgalya of the FOSSEE (Free and Open Source Software for Education) Project which is part of the central government Ministry for Human Resource Development spoke about various activities of his project. Of particular interest to us are: FOSSEE is well funded, backed by the government and has enthusiastic staff so we should be seeing a lot more from them in the future. Veteran Free Software activist Venky Hariharan spoke about his experiences in lobbying the government on tech issues. He noted that there has been a sea change in attitudes towards Linux and Open source in the bureacracy of late. Several states have been aggressively mandating the use of it as have several national ministries and agencies. We the community can provide a valuable service by helping them in the transition. They also need to be educated on how to work with the community (contributing changes back, not working behind closed doors etc.) Debian History and Debian Cycle Shirish Agarwal spoke about the Debian philosophy and foundational documents such as the social contract and DFSG and how the release cycle works. Nothing new to an experienced user but informative to the newcomers in the audience and sparked some questions and discussion. Keysigning One of my main missions in attending was to help get as many isolated people as possible into the web of trust. Unfortunately the keysigning was not adequately publicized and few people were ready. I would have led them through the process of creating a new key there and then but with the lack of connectivity that idea had to be abandoned. I did manage to sign about 8-10 keys during other times. Future Directions for Debian-IN BOF I led this one. Lots of spirited discussion and I found feedback from new users in particular to be very helpful. Some take aways are: Lil' Debi Kumar Sukhani was a Debian GSoC student and his project which he demonstrated was to be able to install Debian on an Android phone. Why would you want to do this? Apart from the evergreen "Because I can", you can run server software such as sshd on your phone or even use it as an ARM development board. Unfortunately my phone uses Blackberry 10 OS which can run android apps (emulated under QNX) but wouldn't be able to use this. When I get a real Android phone I will try it out. Debian on ARM Siji Sunny gave this talk which was geared more towards hardware types which I am not but one thing I learned was thee difference between all the different ARM subarchitectures. I knew Siji first from a previous incarnation when he worked at CDAC with the late and much lamented Prof. R.K. Joshi. We had a long conversation about those days. Prof. Joshi/CDAC had developed an Indic rendering system called Indix which alas became the Betamax to Pango's VHS but he was also very involved in other Indic computing issues such as working with the Unicode Consortium and the preseration of Sanskrit manuscripts which is also an interest of mine. One good thing that cameout of Indix was some rather nice fonts. I had thought they were still buried in the dungeons of CDAC but apparently they were freed at one point. That's one more thing for me to look into. Evening/Next morning< My cousin met me and we had a leisurely dinner together. It was quite late by the time I got back to the hotel. FOSSEE had kindly lent me one of their tablets (which incidently are powerful enough to run LibreOffice comfortably.) so I thought I might be able to quickly redo my presentation before bedtime. Well, wouldn't you know it the wifi was not fixed. As I should have guessed but all the progress I'd had made me giddily optimistic. There was an option of trying to find an Internet cafe in a commercial area 15-20 minutes walk away. If this had been Gujarat I would have tried it but although I can more or less understand Hindi I can barely put together two sentences and Marathi I don't know at all. So I gave up that idea. I redid the slides from memory as best I could and went to sleep. In the morning I checked out and ferried myself and my suitcase via rickshaw back to the IIT campus. This time I got the driver to take me all the way in to the conference venue. Prof. Maudgalya kindly offered to let me keep the tablet to develop stuff on. I respectfully had to decline because although I love to collect bits of tech the fact it is it would have just gathered dust and ought to go to someone who can make a real contribution with it. I transferred my files to a USB key and borrowed a loaner laptop for my talk. Debian Packaging Workshop While waiting to do my talk I sat in on a workshop Praveen ran taking participants through the whole process of creating a Debian package (a ruby gem was the example.) He's done this before so it was a good presentation and well attended but the lack of connectivity did put a damper on things. Ask Me Anything It turned out the schedule had to be shuffled a bit so my talk was moved later from the announced time. A few people had already showed up so I took some random questions about Debian from them instead. GNOME Shell Accessibility With Orca Krishnakant Mane is remarkable. Although he is blind, he is a developer and a major contributor to Open Source projects. He talked about the Accessibility features of GNOME and compared them (favorably I might add) with proprietary screen readers. Not a subject that's directly useful to me but I found it interesting nonetheless. Rust: The memory safe language Manish Goregaokar talked about one of the new fad programming languages that have gotten a lot of buzz lately. This one is backed by Mozilla and it's interesting enough but I'll stick with C++ and Perl until one of the new ones "wins." Building a Mail Server With Debian Finally I got to give my talk and, yup, the video out on my borrowed laptop was incompatible with the projector. A slight delay to transfer everything to another laptop and I was able to begin. I talked about setting up BIND, postfix, and of course dovecot along with spamassassin, clamav etc. It turned out I had more than enough material and I went atleast 30 minutes over time and even then I had to rush at the end. People said they liked it so I'm happy. The End I gave the concluding remarks. Various people were thanked (including myself) mementos were given and pictures were taken. Despite a few mishaps I enjoyed myself and I am glad I attended. The level of enthusiasm was very high and lessons were learned so the next Debian-IN event should be even better. My departing flight wasn't due to leave until 1:20AM so I killed a few hours with my family before the flight. Once again I was stopping in Zurich, this time for most of a day. The last of my blunders was not to take my coat out of my suitcase and the temperature outside was 29F so I had to spend that whole time enjoing the (not so) many charms of Zurich airport. Atleast the second flight took me to Newark instead of JFK so I was able to get home a little earlier on Monday evening, exhausted but happy I made the trip.

21 January 2015

Enrico Zini: miniscreen

Playing with python, terminfo and command output I am experimenting with showing progress on the terminal for a subcommand that is being run, showing what is happening without scrolling away the output of the main program, and I came out with this little toy. It shows the last X lines of a subcommand output, then gets rid of everything after the command has ended. Usability-wise, it feels like a tease to me: it looks like I'm being shown all sorts of information then they are taken away from me before I managed to make sense of them. However, I find it cute enough to share: 
#!/usr/bin/env python3
#coding: utf-8
# Copyright 2015 Enrico Zini <enrico@enricozini.org>.  Licensed under the terms
# of the GNU General Public License, version 2 or any later version.
import argparse
import fcntl
import select
import curses
import contextlib
import subprocess
import os
import sys
import collections
import shlex
import shutil
import logging
def stream_output(proc):
    """
    Take a subprocess.Popen object and generate its output, line by line,
    annotated with "stdout" or "stderr". At process termination it generates
    one last element: ("result", return_code) with the return code of the
    process.
    """
    fds = [proc.stdout, proc.stderr]
    bufs = [b"", b""]
    types = ["stdout", "stderr"]
    # Set both pipes as non-blocking
    for fd in fds:
        fcntl.fcntl(fd, fcntl.F_SETFL, os.O_NONBLOCK)
    # Multiplex stdout and stderr with different prefixes
    while len(fds) > 0:
        s = select.select(fds, (), ())
        for fd in s[0]:
            idx = fds.index(fd)
            buf = fd.read()
            if len(buf) == 0:
                fds.pop(idx)
                if len(bufs[idx]) != 0:
                    yield types[idx], bufs.pop(idx)
                types.pop(idx)
            else:
                bufs[idx] += buf
                lines = bufs[idx].split(b"\n")
                bufs[idx] = lines.pop()
                for l in lines:
                    yield types[idx], l
    res = proc.wait()
    yield "result", res
@contextlib.contextmanager
def miniscreen(has_fancyterm, name, maxlines=3, silent=False):
    """
    Show the output of a process scrolling in a portion of the screen.

    has_fancyterm: true if the terminal supports fancy features; if false, just
    write lines to standard output

    name: name of the process being run, to use as a header

    maxlines: maximum height of the miniscreen

    silent: do nothing whatsoever, used to disable this without needing to
            change the code structure

    Usage:
        with miniscreen(True, "my process", 5) as print_line:
            for i in range(10):
                print_line(("stdout", "stderr")[i % 2], "Line # ".format(i))
    """
    if not silent and has_fancyterm:
        # Discover all the terminal control sequences that we need
        output_normal = str(curses.tigetstr("sgr0"), "ascii")
        output_up = str(curses.tigetstr("cuu1"), "ascii")
        output_clreol = str(curses.tigetstr("el"), "ascii")
        cols, lines = shutil.get_terminal_size()
        output_width = cols
        fg_color = (curses.tigetstr("setaf") or
                    curses.tigetstr("setf") or "")
        sys.stdout.write(str(curses.tparm(fg_color, 6), "ascii"))
        output_lines = collections.deque(maxlen=maxlines)
        def print_lines():
            """
            Print the lines in our buffer, then move back to the beginning
            """
            sys.stdout.write("  progress:".format(name))
            sys.stdout.write(output_clreol)
            for msg in output_lines:
                sys.stdout.write("\n")
                sys.stdout.write(msg)
                sys.stdout.write(output_clreol)
            sys.stdout.write(output_up * len(output_lines))
            sys.stdout.write("\r")
        try:
            print_lines()
            def _progress_line(type, line):
                """
                Print a new line to the miniscreen
                """
                # Add the new line to our output buffer
                msg = "   ".format("." if type == "stdout" else "!", line)
                if len(msg) > output_width - 4:
                    msg = msg[:output_width - 4] + "..."
                output_lines.append(msg)
                # Update the miniscreen
                print_lines()
            yield _progress_line
            # Clear the miniscreen by filling our ring buffer with empty lines
            # then printing them out
            for i in range(maxlines):
                output_lines.append("")
            print_lines()
        finally:
            sys.stdout.write(output_normal)
    elif not silent:
        def _progress_line(type, line):
            print(" :  ".format(type, line))
        yield _progress_line
    else:
        def _progress_line(type, line):
            pass
        yield _progress_line
def run_command_fancy(name, cmd, env=None, logfd=None, fancy=True, debug=False):
    quoted_cmd = " ".join(shlex.quote(x) for x in cmd)
    log.info("%s running command %s", name, quoted_cmd)
    if logfd: print("runcmd:", quoted_cmd, file=logfd)
    # Run the script itself on an empty environment, so that what was
    # documented is exactly what was run
    proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=env)
    with miniscreen(fancy, name, silent=debug) as progress:
        stderr = []
        for type, val in stream_output(proc):
            if type == "stdout":
                val = val.decode("utf-8")
                if logfd: print("stdout:", val, file=logfd)
                log.debug("%s stdout: %s", name, val)
                progress(type, val)
            elif type == "stderr":
                val = val.decode("utf-8")
                if logfd: print("stderr:", val, file=logfd)
                stderr.append(val)
                log.debug("%s stderr: %s", name, val)
                progress(type, val)
            elif type == "result":
                if logfd: print("retval:", val, file=logfd)
                log.debug("%s retval: %d", name, val)
                retval = val
    if retval != 0:
        lastlines = min(len(stderr), 5)
        log.error("%s exited with code %s", name, retval)
        log.error("Last %d lines of standard error:", lastlines)
        for line in stderr[-lastlines:]:
            log.error("%s: %s", name, line)
    return retval
parser = argparse.ArgumentParser(description="run a command showing only a portion of its output")
parser.add_argument("--logfile", action="store", help="specify a file where the full execution log will be written")
parser.add_argument("--debug", action="store_true", help="debugging output on the terminal")
parser.add_argument("--verbose", action="store_true", help="verbose output on the terminal")
parser.add_argument("command", nargs="*", help="command to run")
args = parser.parse_args()
if args.debug:
    loglevel = logging.DEBUG
elif args.verbose:
    loglevel = logging.INFO
else:
    loglevel = logging.WARN
logging.basicConfig(level=loglevel, stream=sys.stderr)
log = logging.getLogger()
fancy = False
if not args.debug and sys.stdout.isatty():
    curses.setupterm()
    if curses.tigetnum("colors") > 0:
        fancy = True
if args.logfile:
    logfd = open("output.log", "wt")
else:
    logfd = None
retval = run_command_fancy("miniscreen example", args.command, logfd=logfd)
sys.exit(retval)

Jo Shields: mono-project.com Linux packages, January 2015 edition

The latest version of Mono has released (actually, it happened a week ago, but it took me a while to get all sorts of exciting new features bug-checked and shipshape). Stable packages This release covers Mono 3.12, and MonoDevelop 5.7. These are built for all the same targets as last time, with a few caveats (MonoDevelop does not include F# or ASP.NET MVC 4 support). ARM packages will be added in a few weeks time, when I get the new ARM build farm working at Xamarin s Boston office. Ahead-of-time support This probably seems silly since upstream Mono has included it for years, but Mono on Debian has never shipped with AOT d mscorlib.dll or mcs.exe, for awkward package-management reasons. Mono 3.12 fixes this, and will AOT these assemblies optimized for your computer on installation. If you can suggest any other assemblies to add to the list, we now support a simple manifest structure so any assembly can be arbitrarily AOT d on installation. Goodbye Mozroots! I am very pleased to announce that as of this release, Mono users on Linux no longer need to run mozroots to get SSL working. A new command, cert-sync , has been added to this release, which synchronizes the Mono SSL certificate store against your OS certificate store and this tool has been integrated into the packaging system for all mono-project.com packages, so it is automatically used. Just make sure the ca-certificates-mono package is installed on Debian/Ubuntu (it s always bundled on RPM-based) to take advantage! It should be installed on fresh installs by default. If you want to invoke the tool manually (e.g. you installed via make install, not packages) use 
cert-sync /path/to/ca-bundle.crt
On Debian systems, that s 
cert-sync /etc/ssl/certs/ca-certificates.crt
and on Red Hat derivatives it s 
cert-sync /etc/pki/tls/certs/ca-bundle.crt
Your distribution might use a different path, if it s not derived from one of those. Windows installer back from the dead Thanks to help from Alex Koeplinger, I ve brought the Windows installer back from the dead. The last release on the website was for 3.2.3 (it s actually not this version at all it s complicated ), so now the Windows installer has parity with the Linux and OSX versions. The Windows installer (should!) bundles everything the Mac version does F#, PCL facades, IronWhatever, etc, along with Boehm and SGen builds of the Mono runtime done with Visual Studio 2013. An EXPERIMENTAL OH MY GOD DON T USE THIS IN PRODUCTION 64-bit installer is in the works, when I have the time to try and make a 64-build of Gtk#.

19 January 2015

Daniel Pocock: Storage controllers for small Linux NFS networks

While contemplating the disk capacity upgrade for my Microserver at home, I've also been thinking about adding a proper storage controller. Currently I just use the built-in controller in the Microserver. It is an AMD SB820M SATA controller. It is a bottleneck for the SSD IOPS. On the disks, I prefer to use software RAID (such as md or BtrFs) and not become dependent on the metadata format of any specific RAID controller. The RAID controllers don't offer the checksumming feature that is available in BtrFs and ZFS. The use case is NFS for a small number of workstations. NFS synchronous writes block the client while the server ensures data really goes onto the disk. This creates a performance bottleneck. It is actually slower than if clients are writing directly to their local disks through the local OS caches. SSDs on an NFS server offer some benefit because they can complete write operations more quickly and the NFS server can then tell the client the operation is complete. The more performant solution (albeit with a slight risk of data corruption) is to use a storage controller with a non-volatile (battery-backed or flash-protected) write cache. Many RAID controllers have non-volatile write caches. Some online discussions of BtrFs and ZFS have suggested staying away from full RAID controllers though, amongst other things, to avoid the complexities of RAID controllers adding their metadata to the drives. This brings me to the first challenge though: are there suitable storage controllers that have a non-volatile write cache but without having other RAID features? Or a second possibility: out of the various RAID controllers that are available, do any provide first-class JBOD support? Observations I looked at specs and documentation for various RAID controllers and identified some of the following challenges: Next steps Are there other options to look at, for example, alternatives to NFS? If I just add in a non-RAID HBA to enable faster IO to the SSDs will this be enough to make a noticeable difference on the small number of NFS clients I'm using? Or is it inevitable that I will have to go with one of the solutions that involves putting a vendor's volume metadata onto JBOD volumes? If I do go that way, which of the vendors' metadata formats are most likely to be recognized by free software utilities in the future if I ever connect the disk to a generic non-RAID HBA? Thanks to all those people who provided comments about choosing drives for this type of NAS usage. Related reading

18 January 2015

Mark Brown: Heating the Internet of Things

Internet of Things seems to be trendy these days, people like the shiny apps for controlling things and typically there are claims that the devices will perform better than their predecessors by offloading things to the cloud but this makes some people worry that there are potential security issues and it s not always clear that internet usage is actually delivering benefits over something local. One of the more widely deployed applications is smart thermostats for central heating which is something I ve been playing with. I m using Tado, there s also at least Nest and Hive who do similar things, all relying on being connected to the internet for operation. The main thing I ve noticed has been that the temperature regulation in my flat is better, my previous thermostat allowed the temperature to vary by a couple of degrees around the target temperature in winter which got noticeable, with this the temperature generally seems to vary by a fraction of a degree at most. That does use the internet connection to get the temperature outside, though I m fairly sure that most of this is just a better algorithm (the thermostat monitors how quickly the flat heats up when heating and uses this when to turn off rather than waiting for the temperature to hit the target then seeing it rise further as the radiators cool down) and performance would still be substantially improved without it. The other thing that these systems deliver which does benefit much more from the internet connection is that it s easy to control them remotely. This in turn makes it a lot easier to do things like turn the heating off when it s not needed you can do it remotely, and you can turn the heating back on without being in the flat so that you don t need to remember to turn it off before you leave or come home to a cold building. The smarter ones do this automatically based on location detection from smartphones so you don t need to think about it. For example, when I started this post this I was sitting in a coffee shop so the heating had been turned off based on me taking my phone with me and as a result the temperature gone had down a bit. By the time I got home the flat was back up to normal temperature all without any meaningful intervention or visible difference on my part. This is particularly attractive for me given that I work from home I can t easily set a schedule to turn the heating off during the day like someone who works in an office so the heating would be on a lot of the time. Tado and Nest will to varying extents try to do this automatically, I don t know about Hive. The Tado one at least works very well, I can t speak to the others. I ve not had a bill for a full winter yet but I m fairly sure looking at the meter that between the two features I m saving a substantial amount of energy (and hence money and/or the environment depending on what you care about) and I m also seeing a more constant temperature within the flat, my guess would be that most of the saving is coming from the heating being turned off when I leave the flat. For me at least this means that having the thermostat internet connected is worthwhile.

Dirk Eddelbuettel: Running UBSAN tests via clang with Rocker

Every now and then we get reports from CRAN about our packages failing a test there. A challenging one concerns UBSAN, or Undefined Behaviour Sanitizer. For background on UBSAN, see this RedHat blog post for gcc and this one from LLVM about clang. I had written briefly about this before in a blog post introducing the sanitizers package for tests, as well as the corresponding package page for sanitizers, which clearly predates our follow-up Rocker.org repo / project described in this initial announcement and when we became the official R container for Docker. Rocker had support for SAN testing, but UBSAN was not working yet. So following a recent CRAN report against our RcppAnnoy package, I was unable to replicate the error and asked for help on r-devel in this thread. Martyn Plummer and Jan van der Laan kindly sent their configurations in the same thread and off-list; Jeff Horner did so too following an initial tweet offering help. None of these worked for me, but further trials eventually lead me to the (already mentioned above) RedHat blog post with its mention of -fno-sanitize-recover to actually have an error abort a test. Which, coupled with the settings used by Martyn, were what worked for me: clang-3.5 -fsanitize=undefined -fno-sanitize=float-divide-by-zero,vptr,function -fno-sanitize-recover. This is now part of the updated Dockerfile of the R-devel-SAN-Clang repo behind the r-devel-ubsan-clang. It contains these settings, as well a new support script check.r for littler---which enables testing right out the box. Here is a complete example: 
docker                              # run Docker (any recent version, I use 1.2.0)
  run                               # launch a container 
    --rm                            # remove Docker temporary objects when dome
    -ti                             # use a terminal and interactive mode 
    -v $(pwd):/mnt                  # mount the current directory as /mnt in the container
    rocker/r-devel-ubsan-clang      # using the rocker/r-devel-ubsan-clang container
  check.r                           # launch the check.r command from littler (in the container)
    --setwd /mnt                    # with a setwd() to the /mnt directory
    --install-deps                  # installing all package dependencies before the test
    RcppAnnoy_0.0.5.tar.gz          # and test this tarball
I know. It is a mouthful. But it really is merely the standard practice of running Docker to launch a single command. And while I frequently make this the /bin/bash command (hence the -ti options I always use) to work and explore interactively, here we do one better thanks to the (pretty useful so far) check.r script I wrote over the last two days. check.r does about the same as R CMD check. If you look inside check you will see a call to a (non-exported) function from the (R base-internal) tools package. We call the same function here. But to make things more interesting we also first install the package we test to really ensure we have all build-dependencies from CRAN met. (And we plan to extend check.r to support additional apt-get calls in case other libraries etc are needed.) We use the dependencies=TRUE option to have R smartly install Suggests: as well, but only one level (see help(install.packages) for details. With that prerequisite out of the way, the test can proceed as if we had done R CMD check (and additional R CMD INSTALL as well). The result for this (known-bad) package: 
edd@max:~/git$ docker run --rm -ti -v $(pwd):/mnt rocker/r-devel-ubsan-clang check.r --setwd /mnt --install-deps RcppAnnoy_0.0.5.tar.gz 
also installing the dependencies  Rcpp ,  BH ,  RUnit 
trying URL 'http://cran.rstudio.com/src/contrib/Rcpp_0.11.3.tar.gz'
Content type 'application/x-gzip' length 2169583 bytes (2.1 MB)
opened URL
==================================================
downloaded 2.1 MB
trying URL 'http://cran.rstudio.com/src/contrib/BH_1.55.0-3.tar.gz'
Content type 'application/x-gzip' length 7860141 bytes (7.5 MB)
opened URL
==================================================
downloaded 7.5 MB
trying URL 'http://cran.rstudio.com/src/contrib/RUnit_0.4.28.tar.gz'
Content type 'application/x-gzip' length 322486 bytes (314 KB)
opened URL
==================================================
downloaded 314 KB
trying URL 'http://cran.rstudio.com/src/contrib/RcppAnnoy_0.0.4.tar.gz'
Content type 'application/x-gzip' length 25777 bytes (25 KB)
opened URL
==================================================
downloaded 25 KB
* installing *source* package  Rcpp  ...
** package  Rcpp  successfully unpacked and MD5 sums checked
** libs
clang++-3.5 -fsanitize=undefined -fno-sanitize=float-divide-by-zero,vptr,function -fno-sanitize-recover -I/usr/local/lib/R/include -DNDEBUG -I../inst/include/ -I/usr/local/include    -fpic  -pipe -Wall -pedantic -
g  -c Date.cpp -o Date.o
[...]
* checking examples ... OK
* checking for unstated dependencies in  tests  ... OK
* checking tests ...
  Running  runUnitTests.R 
 ERROR
Running the tests in  tests/runUnitTests.R  failed.
Last 13 lines of output:
  +     if (getErrors(tests)$nFail > 0)  
  +         stop("TEST FAILED!")
  +      
  +     if (getErrors(tests)$nErr > 0)  
  +         stop("TEST HAD ERRORS!")
  +      
  +     if (getErrors(tests)$nTestFunc < 1)  
  +         stop("NO TEST FUNCTIONS RUN!")
  +      
  +  
  
  
  Executing test function test01getNNsByVector  ... ../inst/include/annoylib.h:532:40: runtime error: index 3 out of bounds for type 'int const[2]'
* checking PDF version of manual ... OK
* DONE
Status: 1 ERROR, 2 WARNINGs, 1 NOTE
See
   /tmp/RcppAnnoy/..Rcheck/00check.log 
for details.
root@a7687c014e55:/tmp/RcppAnnoy#
The log shows that thanks to check.r, we first download and the install the required packages Rcpp, BH, RUnit and RcppAnnoy itself (in the CRAN release). Rcpp is installed first, we then cut out the middle until we get to ... the failure we set out to confirm. Now having a tool to confirm the error, we can work on improved code. One such fix currently under inspection in a non-release version 0.0.5.1 then passes with the exact same invocation (but pointing at RcppAnnoy_0.0.5.1.tar.gz): 
edd@max:~/git$ docker run --rm -ti -v $(pwd):/mnt rocker/r-devel-ubsan-clang check.r --setwd /mnt --install-deps RcppAnnoy_0.0.5.1.tar.gz
also installing the dependencies  Rcpp ,  BH ,  RUnit 
[...]
* checking examples ... OK
* checking for unstated dependencies in  tests  ... OK
* checking tests ...
  Running  runUnitTests.R 
 OK
* checking PDF version of manual ... OK
* DONE
Status: 1 WARNING
See
   /mnt/RcppAnnoy.Rcheck/00check.log 
for details.
edd@max:~/git$
This proceeds the same way from the same pristine, clean container for testing. It first installs the four required packages, and the proceeds to test the new and improved tarball. Which passes the test which failed above with no issues. Good. So we now have an "appliance" container anybody can download from free from the Docker hub, and deploy as we did here in order to have fully automated, one-command setup for testing for UBSAN errors. UBSAN is a very powerful tool. We are only beginning to deploy it. There are many more useful configuration settings. I would love to hear from anyone who would like to work on building this out via the R-devel-SAN-Clang GitHub repo. Improvements to the littler scripts are similarly welcome (and I plan on releasing an updated littler package "soon").

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

16 January 2015

Richard Hartmann: Release Critical Bug report for Week 03

The UDD bugs interface currently knows about the following release critical bugs: How do we compare to the Squeeze release cycle?
Week Squeeze Wheezy Jessie
43 284 (213+71) 468 (332+136) 319 (240+79)
44 261 (201+60) 408 (265+143) 274 (224+50)
45 261 (205+56) 425 (291+134) 295 (229+66)
46 271 (200+71) 401 (258+143) 427 (313+114)
47 283 (209+74) 366 (221+145) 342 (260+82)
48 256 (177+79) 378 (230+148) 274 (189+85)
49 256 (180+76) 360 (216+155) 226 (147+79)
50 204 (148+56) 339 (195+144) ???
51 178 (124+54) 323 (190+133) 189 (134+55)
52 115 (78+37) 289 (190+99) 147 (112+35)
1 93 (60+33) 287 (171+116) 140 (104+36)
2 82 (46+36) 271 (162+109) 157 (124+33)
3 25 (15+10) 249 (165+84) 172 (128+44)
4 14 (8+6) 244 (176+68)
5 2 (0+2) 224 (132+92)
6 release! 212 (129+83)
7 release+1 194 (128+66)
8 release+2 206 (144+62)
9 release+3 174 (105+69)
10 release+4 120 (72+48)
11 release+5 115 (74+41)
12 release+6 93 (47+46)
13 release+7 50 (24+26)
14 release+8 51 (32+19)
15 release+9 39 (32+7)
16 release+10 20 (12+8)
17 release+11 24 (19+5)
18 release+12 2 (2+0)
Graphical overview of bug stats thanks to azhag:

13 January 2015

Jonathan McDowell: Tracking a ship around the world

I moved back from the California Bay Area to Belfast a while back and for various reasons it looks like I m going to be here a while, so it made sense to have my belongings shipped over here. They haven t quite arrived yet, and I ll do another post about that process once they have, but I ve been doing various tweets prefixed with [shipping] during the process. Various people I ve spoken to (some who should know me better) thought this was happening manually. It wasn t. If you care about how it was done, read on. I d been given details of the ship carrying my container, and searching for that turned up the excellent MarineTraffic which let me see the current location of the ship. Turns out ships broadcast their location using AIS and anyone with a receiver can see the info. Very cool, and I spent some time having a look at various bits of shipping around the UK out of interest. I also found the ship s itinerary which give me some idea of where it would be calling and when. Step one was to start recording this data; it was time sensitive and I wanted to be able to see historical data. I took the easy route and set up a cron job to poll the location and itinerary on an hourly basis, and store the results. That meant I had the data over time, if my parsing turned out to miss something I could easily correct it, and that I wasn t hammering Marine Traffic while writing the parsing code. Next I wanted to parse the results, store them in a more manageable format than the HTML, and alert me when the ship docked somewhere or set off again. I ve been trying to learn more Python rather than doing my default action of turning to Perl for these things, and this seemed like a simple enough project to try out. Beautiful Soup seemed to turn up top for HTML parsing in Python, so that formed the basis. Throwing the info into a database so I could do queries felt like the right move so I used SQLite - if this had been more than a one off I d have considered looking at PostgreSQL and its GIS support. Finally Tweepy made it very easy to tweet from Python in about 4 lines of code. The whole thing weighed in at only 175 lines of code, mostly around pulling the info out of the HTML and then a little to deal with checking for state changes against the current status and the last piece of info in the database. The pieces of information I chose to store were the time of the update (i.e. when the ship sent it, not when my script ran), reported area, reported state, the position + course, reported origin, reported destination and eta. The fact this is all in a database makes it very easy to do a few queries on the data. How fast did the ship go? 
sqlite> SELECT MAX(speed) FROM status;
MAX(speed)
21.9
What areas did it report? 
sqlite> SELECT area FROM status GROUP BY area;
area
-
Atlantic North
California
Caribbean Sea
Celtic Sea
English Channel
Hudson River
Pacific North
Panama Canal
What statuses did we see? 
sqlite> SELECT status FROM status GROUP BY status;
status
At Anchor
Moored
Stopped
Underway
Underway using Engine
Finally having hourly data lets me draw a map of where the ship went. The data isn t complete, because the free AIS info depends on the ship being close enough to a receiving station. That means there were a few days in the North Atlantic without updates, for example. However there s enough to give a good idea of just how well traveled my belongings are, and it gave me an excuse to play with OpenLayers.
(Apologies if the zoom buttons aren t working for you here; I think I need to kick the CSS in some manner I haven t quite figured out yet.)

Next.

Previous.