A new minor / maintenance release RQuantLib 0.3.11 is now on CRAN and in Debian. Like the RcppClassic upload two days ago and the RcppZiggurat and RcppEigen uploads yesterday, this release was motivated at least in part by an upcoming Rcpp release about which I should have more to day soon. A few things got polished and updated, and we added a plugin which should make use via Rcpp Attribute easier. RQuantLib combines (some of) the quantitative analytics of QuantLib with the R statistical computing environment and language. Thanks to CRANberries, there is also a diff to the previous release 0.3.10. Full changelog details, examples and more details about this package are at my RQuantLib page.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

There will be a new option in gcc 4.9 named -fstack-protector-strong , which offers an improved version of -fstack-protector without going all the way to -fstack-protector-all . The stack protector feature itself adds a known canary to the stack during function preamble, and checks it when the function returns. If it changed, there was a stack overflow, and the program aborts. This is fine, but figuring out when to include it is the reason behind the various options. Since traditionally stack overflows happen with string-based manipulations, the default (-fstack-protector), only includes the canary code when a function defines an 8 (--param=ssp-buffer-size=N, N=8 by default) or more byte local character array. This means just a few functions get the checking, but they re probably the most likely to need it, so it s an okay balance. Various distributions ended up lowering their default --param=ssp-buffer-size option down to 4, since there were still cases of functions that should have been protected but the conservative gcc upstream default of 8 wasn t covering them. However, even with the increased function coverage, there are rare cases when a stack overflow happens on other kinds of stack variables. To handle this more paranoid concern, -fstack-protector-all was defined to add the canary to all functions. This results in substantial use of stack space for saving the canary on deep stack users, and measurable (though surprisingly still relatively low) performance hit due to all the saving/checking. For a long time, Chrome OS used this, since we re paranoid. :) In the interest of gaining back some of the lost performance and not hitting our Chrome OS build images with such a giant stack-protector hammer, Han Shen from the Chrome OS compiler team created the new option -fstack-protector-strong, which enables the canary in many more conditions:

• local variable s address used as part of the right hand side of an assignment or function argument
• local variable is an array (or union containing an array), regardless of array type or length
• uses register local variables

This meant we were covering all the more paranoid conditions that might lead to a stack overflow. Chrome OS has been using this option instead of -fstack-protector-all for about 10 months now. As a quick demonstration of the options, you can see this example program under various conditions. It tries to show off an example of shoving serialized data into a non-character variable, like might happen in some network address manipulations or streaming data parsing. Since I m using memcpy here for clarity, the builds will need to turn off FORTIFY_SOURCE, which would also notice the overflow.

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
struct no_chars  
    unsigned int len;
    unsigned int data;
 ;
int main(int argc, char * argv[])
 
    struct no_chars info =    ;
    if (argc < 3)  
        fprintf(stderr, "Usage: %s LENGTH DATA...\n", argv[0]);
        return 1;
     
    info.len = atoi(argv[1]);
    memcpy(&info.data, argv[2], info.len);
    return 0;
 

Built with everything disabled, this faults trying to return to an invalid VMA:

$ gcc -Wall -O2 -U_FORTIFY_SOURCE -fno-stack-protector /tmp/boom.c -o /tmp/boom
$ /tmp/boom 64 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
Segmentation fault (core dumped)

Built with FORTIFY_SOURCE enabled, we see the expected catch of the overflow in memcpy:

$ gcc -Wall -O2 -D_FORTIFY_SOURCE=2 -fno-stack-protector /tmp/boom.c -o /tmp/boom
$ /tmp/boom 64 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
*** buffer overflow detected ***: /tmp/boom terminated
...

So, we ll leave FORTIFY_SOURCE disabled for our comparisons. With pre-4.9 gcc, we can see that -fstack-protector does not get triggered to protect this function:

$ gcc -Wall -O2 -U_FORTIFY_SOURCE -fstack-protector /tmp/boom.c -o /tmp/boom
$ /tmp/boom 64 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
Segmentation fault (core dumped)

However, using -fstack-protector-all does trigger the protection, as expected:

$ gcc -Wall -O2 -U_FORTIFY_SOURCE -fstack-protector-all /tmp/boom.c -o /tmp/boom
$ /tmp/boom 64 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
*** stack smashing detected ***: /tmp/boom terminated
Aborted (core dumped)

And finally, using the gcc snapshot of 4.9, here is -fstack-protector-strong doing its job:

$ /usr/lib/gcc-snapshot/bin/gcc -Wall -O2 -U_FORTIFY_SOURCE -fstack-protector-strong /tmp/boom.c -o /tmp/boom
$ /tmp/boom 64 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
*** stack smashing detected ***: /tmp/boom terminated
Aborted (core dumped)

For Linux 3.14, I ve added support for -fstack-protector-strong via the new CONFIG_CC_STACKPROTECTOR_STRONG option. The old CONFIG_CC_STACKPROTECTOR will be available as CONFIG_CC_STACKPROTECTOR_REGULAR. When comparing the results on builds via size and objdump -d analysis, here s what I found with gcc 4.9: A normal x86_64 defconfig build, without stack protector had a kernel text size of 11430641 bytes with 36110 function bodies. Adding CONFIG_CC_STACKPROTECTOR_REGULAR increased the kernel text size to 11468490 (a +0.33% change), with 1015 of 36110 functions stack-protected (2.81%). Using CONFIG_CC_STACKPROTECTOR_STRONG increased the kernel text size to 11692790 (+2.24%), with 7401 of 36110 functions stack-protected (20.5%). And 20% is a far-cry from 100% if support for -fstack-protector-all was added back to the kernel. The next bit of work will be figuring out the best way to detect the version of gcc in use when doing Debian package builds, and using -fstack-protector-strong instead of -fstack-protector. For Ubuntu, it s much simpler because it ll just be the compiler default.

2014, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.

Data science has been called "the sexiest job" so often, this has recently led to an integer overflow.

---

---

• When you are doing statistics, call it statistics.
• When you are doing unsupervised learning, call it machine learning.
• When your focus is distributed computation, call it distributed computing.
• When you do data management, continute to call it data management and databases.
• When you do data indexing, call it data indexing.
• When you are doing unsupervised data mining, call it cluster analysis, outlier detection, ...
• Whatever it is, try to use a precise term, instead of a buzzword.

I am not (yet?) reporting this as a bug as this happened with a several days old session open, and just while I was upgrading my Sid system, after a long time without doing so (probably since before the vacations started... In December 2013). But I cannot avoid sharing this interesting screenshot. Of course, this does not happen in other browsers. And AFAICT it only happens while reading the Debian Policy (either online or locally, even recoding it to UTF-8). Funniest thing, the Debian policy specifies no Javascript, no stylesheets at all... (Hey, and FWIW... Why is the online copy of the Debian policy still in iso-8859-1 It's not 1995 anymore...) [update] Of course, it's the default font, not only the Debian policy. Just as an example, the following text: Yields the following output: [update 2] And, of course, after finishing the update process... I got a new version of Iceweasel. Restarted it, and everything is back to normal :-

Formally, today is my first day as a student on a formal, scholarized institution Basically for the first time in almost twenty years! Yes, those that know me know that I aspire to live the life of academia. I have worked at public universities for almost all of my adult life (between 1997 and 1999 I worked at a local ISP and at a private school), and have had a minor academic position ( T cnico Acad mico ) for almost ten years. And not having a proper degree limited me from pursuing anything further. Then, in early 2010 I presented the written exam. By late 2010, the corresponding oral exam. That allowed me to get my formal diploma in December 2010. By the end of 2011, I requested to be a teacher in the Engineering Faculty of UNAM, and started teaching Operating Systems a year ago, in January 2012. So, a good advance in the last few years... But I know that if I just sit here, I won't be able to advance my position towards really entering the Sacred Halls of Academia. And there are some rituals I have to comply with. One of those rituals is... Devoting some long time to studying under the formal structures. Ok, so I'm finally a postgraduate student I have enrolled in Especialidad en Seguridad Inform tica y Tecnolog as de la Informaci n, a short (one year) postgraduate program in ESIME Culhuac n, of Instituto Polit cnico Nacional (a small campus of Mexico's second-largest university). Some friends have asked me, why am I starting with a Specialization and not a Masters degree. Some simple reasons: Just as when I went to Tijuana in 2010 to do my written exam, once I got and started with the paperwork, I didn't want to let it go If I postpone it, I will probably lose the push to do it by May-July, when the Masters admission process starts. Also, this specialization can be linked with the masters degree on the same topic given at the same campus. This program is one year long, and the masters two But having them both takes 2.5 years. So, not such a bad deal after all. And finally, because, after such a long time without being scholarized, I fear not having an easy time getting to grips with the discipline. I can commit to overworking myself for a year If it's too much for me, I'll just stay with that degree and give up. I expect to like it and continue... But it's also a safe bet :-) Now, there has to be a downside to picking up this path: Of course, my free time will be harshly reduced. I have reduced my Debian involvement in the last year, as I devoted a huge chunk of my time to teaching and book-writing... This year... We shall see what happens. I can for now only confirm what I have said publicly but inside our team only: I have requested to my peers and to our DPL to step down as a DebConf chair. I love organizing DebConf, but I don't want to be formally committed to a position I just cannot fulfill as I did when I started with it. As for package maintenance, by far most of my packges are team maintained, and those that are not are relatively easy to keep track of. And of course, I'll keep an eye on my keyring-maint duties as well Will even try to link that work with what I do at school! Anyway, lets see what comes now!

Quest for a terminal emulator The requirements I need a terminal emulator. This is a checklist of the features that I need:

• Foreground colour: #aaaaaa.
• Background colour: black.
• "Linux console" colour palette.
• Font: Monospace-10.
• Alt+Fnn to directly switch to one of the first 12 terminals, like in the Linux console.
• F11 to toggle fullscreen.
• Alt+'+', Alt+'-', Alt+'0' to control font sizes.
• All other keybindings disabled, so that terminal programs are free to use them.
• I occasionally need to switch to black-on-white for presentations.
• Tabs.
• When I open a new tab it needs to contain a shell opened inside the same directory of the last tab I was in. The way I normally start working is something like: cd workdir; x-terminal-emulator -e vim todolist and then I start opening new tabs.
• I need -e to work. In fact, I need the terminal to work as x-terminal-emulator -e cmd args...
• I need to conveniently pick a URL in a terminal and open it in Chromium, and not any other browser. Even better, open it in x-www-browser

My experience is that getting all of this to work is not being as easy as it seems, so I'm creating this page to track progress. gnome-terminal I've been happily using this for years, and it did everything I needed, until some months ago it started to open new tabs in the terminal's working directory instead of the last tab's working directory. This is a big point of frustration for me. It also started opening https urls with Firefox, although the preferred browser was Chromium. There seemed to be no way to control it: I looked for firefox or iceweasel in all gconf and dconf settings and found nothing. The browser issue was fixed by accident when I used Xfce4's settings application to change the browser from Chromium to Firefox and then back to Chromium. update, thanks to Mathieu Parent, Josh Triplett, Peter De Wachter, Julien Cristau, and Charles Plessy: It is also possible to restore the "new tab opened inside the same directory of the last tab I was in" behaviour, by enabling "run command as a login shell" so that /etc/profile.d/vte.sh is run (thanks Mathieu Parent for the link). That in turn spawned extra cleanup work in my .bashrc/.bash_profile/.profile setup, which has been randomly evolving since even before my first Debian "buzz" system. I found that it was setting PROMPT_COMMAND to something else to set the terminal title, conflicting with what vte.sh wants to do. With regards to loading /etc/profile.d/vte.sh by default, Peter De Watcher sent pointers to relevant bugs: here, here, and here. An alternative strategy is to work using the prompt rather than PROMPT_COMMAND; an example is in Josh Triplett's .bashrc from git://joshtriplett.org/git/home. Josh Triplett also said:

To fix the browser launched for URLs, you either need to use a desktop environment following GNOME's mechanism for setting the default browser, or edit ~/.local/share/applications/mimeapps.list and make sure x-scheme-handler/http, x-scheme-handler/https, and x-scheme-handler/ftp are set to your preferred browser's desktop file basename under [Added Associations].

All my issues with gnome-terminal are now gone and I'm only too happy to go back to it. rxvt-unicode-256color urxvt took some work. This is where I got with configuration:

URxvt.font: xft:Monospace-10:antialias=true
URxvt.foreground: #aaaaaa
URxvt.background: black
URxvt.scrollBar_right: true
URxvt.cursorBlink: true
URxvt.perl-ext-common: default,matcher,tabbedex
URxvt.url-launcher: /usr/bin/x-www-browser
URxvt.matcher.button: 1
URxvt.perl-lib: /home/enrico/.urxvt/perl
URxvt.color0: black
URxvt.color1: #aa0000
URxvt.color2: #00aa00umask
URxvt.color3: #aa5500
URxvt.color4: #0000aa
URxvt.color5: #aa00aa
URxvt.color6: #00aaaa
URxvt.color7: #aaaaaa
URxvt.color8: #555555
URxvt.color9: #ff5555
URxvt.color10: #55ff55
URxvt.color11: #ffff55
URxvt.color12: #5555ff
URxvt.color13: #ff55ff
URxvt.color14: #55ffff
URxvt.color15: #ffffff

I got all of the tab behaviour that I need by "customizing" the tab script (yuck github ). Missing

• Bold fonts only work if I set TERM=rxvt
• The implementation for opening a new tab in the directory of the current tab is rather fragile.
• Resizing the terminal window causes an annoying flicker

sakura Configuration is in .config/sakura/sakura.conf and these bits help:

colorset1_fore=rgb(170,170,170)
colorset1_back=rgb(0,0,0)
colorset1_opacity=99
colorset2_fore=rgb(0,0,0)
colorset2_back=rgb(254,254,254)
colorset2_opacity=99
font=Monospace 10
show_always_first_tab=No
scrollbar=false
fullscreen_key=F11
palette=linux

Missing

• Alt+Fnn to directly switch to one of the first 12 terminals, like in the Linux console. Sakura hardcodes using numbers for switching, and only allows to change the modifier.
• Alt+'+', Alt+'-', Alt+'0' to control font sizes.
• Tabs are huge and take way too much screen space.

lxterminal Configuration is in .config/lxterminal/lxterminal.conf and this is relevant to me:

[general]
fontname=DejaVu Sans Mono 10
fgcolor=#aaaaaaaaaaaa
disallowbold=false
cursorblinks=true
tabpos=top
hidescrollbar=false
hidemenubar=true
hideclosebutton=true
disablef10=true
disablealt=true

Also, to open a url directly you control+click it. Missing

• Alt+Fnn to directly switch to one of the first 12 terminals, like in the Linux console.
• F11 to toggle fullscreen.
• Alt+'+', Alt+'-', Alt+'0' to control font sizes.
• I occasionally need to switch to black-on-white for presentations: this is inconvenient, as it requires to set foreground and background separately each time.

terminator Configuration is in .config/terminator/config and this is relevant to me:

[global_config]
  use_custom_url_handler = True
  custom_url_handler = x-www-browser
  inactive_color_offset = 1.0
[keybindings]
  close_term = None
  close_window = None
  copy = None
  cycle_next = None
  cycle_prev = None
  go_down = None
  go_next = None
  go_prev = None
  go_up = None
  group_all = None
  group_tab = None
  hide_window = None
  move_tab_left = None
  move_tab_right = None
  new_tab = None
  new_terminator = None
  new_window = None
  next_tab = None
  paste = None
  prev_tab = None
  reset_clear = None
  reset = None
  resize_down = None
  resize_left = None
  resize_right = None
  resize_up = None
  rotate_ccw = None
  rotate_cw = None
  scaled_zoom = None
  search = None
  split_horiz = None
  split_vert = None
  switch_to_tab_1 = <Alt>F1
  switch_to_tab_2 = <Alt>F2
  switch_to_tab_3 = <Alt>F3
  switch_to_tab_4 = <Alt>F4
  switch_to_tab_5 = <Alt>F5
  switch_to_tab_6 = <Alt>F6
  switch_to_tab_7 = <Alt>F7
  switch_to_tab_8 = <Alt>F8
  switch_to_tab_9 = <Alt>F9
  switch_to_tab_10 = <Alt>F10
  toggle_scrollbar = None
  toggle_zoom = None
  ungroup_all = None
  ungroup_tab = None
[profiles]
  <span class="createlink">default</span>
    palette = "#000000:#aa0000:#00aa00:#aa5500:#0000aa:#aa00aa:#00aaaa:#aaaaaa:#555555:#ff5555:#55ff55:#ffff55:#5555ff:#ff55ff:#55ffff:#ffffff"
    copy_on_selection = True
    icon_bell = False
    background_image = None
    show_titlebar = False

Missing

• I need -e to work. In fact, I need the terminal to work as x-terminal-emulator -e cmd args.... #734655 but should be fixed soon
• New windows start with the title bar visible, even if I configured it not to show. It disappears as soon as the window gets focused, causing the terminal contents to resize. It is just a glitch, but oh so annoying!

update: Richard Hartmann pointed out that terminator's upstream maintainer now changed after the old one didn't have time any more, and it should have a release with a ton of improvements anytime soon. xfce4-terminal Configuration is in .config/xfce4/terminal, and this is relevant to me: terminalrc:

[Configuration]
FontName=Monospace 10
MiscAlwaysShowTabs=FALSE
MiscBell=FALSE
MiscBordersDefault=TRUE
MiscCursorBlinks=FALSE
MiscCursorShape=TERMINAL_CURSOR_SHAPE_BLOCK
MiscDefaultGeometry=80x24
MiscInheritGeometry=FALSE
MiscMenubarDefault=FALSE
MiscMouseAutohide=FALSE
MiscToolbarDefault=FALSE
MiscConfirmClose=TRUE
MiscCycleTabs=TRUE
MiscTabCloseButtons=TRUE
MiscTabCloseMiddleClick=TRUE
MiscTabPosition=GTK_POS_TOP
MiscHighlightUrls=TRUE
ShortcutsNoMenukey=TRUE
ShortcutsNoMnemonics=TRUE
ColorForeground=#aaaaaaaaaaaa

accels.scm:

(gtk_accel_path "<Actions>/terminal-window/goto-tab-1" "<Alt>F1")
(gtk_accel_path "<Actions>/terminal-window/goto-tab-2" "<Alt>F2")
(gtk_accel_path "<Actions>/terminal-window/goto-tab-3" "<Alt>F3")
(gtk_accel_path "<Actions>/terminal-window/goto-tab-4" "<Alt>F4")
(gtk_accel_path "<Actions>/terminal-window/goto-tab-5" "<Alt>F5")
(gtk_accel_path "<Actions>/terminal-window/goto-tab-6" "<Alt>F6")
(gtk_accel_path "<Actions>/terminal-window/goto-tab-7" "<Alt>F7")
(gtk_accel_path "<Actions>/terminal-window/goto-tab-8" "<Alt>F8")
(gtk_accel_path "<Actions>/terminal-window/goto-tab-9" "<Alt>F9")
(gtk_accel_path "<Actions>/terminal-window/goto-tab-10" "<Alt>F10")
(gtk_accel_path "<Actions>/terminal-window/goto-tab-11" "<Alt>F11")
(gtk_accel_path "<Actions>/terminal-window/goto-tab-12" "<Alt>F12")

update: Yves-Alexis Perez points out that to disable the F1 for help in the terminal, you need to remove the accelerator. I tried this and this and didn't have success, but I confess I did not dig too much into it. Although xfce4-terminal -e does not work as I expect, xfce4-terminal registers a wrapper for x-terminal-emulator that does the right thing with respect to -e (also thanks Yves-Alexis Perez). Missing

• Alt+'+', Alt+'-', Alt+'0' to control font sizes.
• All other keybindings disabled, so that terminal programs are free to use them. I could not find a way to disable F1 for help.
• I occasionally need to switch to black-on-white for presentations. I can switch to a "Black on white" preset, but there is no "Gray on black" to switch back to. I can switch back to "White on black", but then I need to manually set the text colour to #aaaaaa. update: Yves-Alexis Perez found that you can add presets to .local/share/colorthemes taking examples from /usr/share/xfce4/terminal/colorschemes; next time I consider switching to xfce4-terminal I'll try that.

roxterm Configuration is in .config/roxterm.sourceforge.net/ split in several files corresponding to profiles. This is a reasonable starting point for me: Profiles/Default:

[roxterm profile]
colour_scheme=Default
disable_menu_access=1
disable_menu_shortcuts=1
disable_tab_menu_shortcuts=0
tab_close_btn=0
hide_menubar=1
always_show_tabs=0

Colours/Default:

[roxterm colour scheme]
0=#000000000000
1=#aaaa00000000
2=#0000aaaa0000
3=#aaaa55550000
4=#00000000aaaa
5=#aaaa0000aaaa
6=#0000aaaaaaaa
7=#aaaaaaaaaaaa
8=#555555555555
9=#ffff55555555
10=#5555ffff5555
11=#ffffffff5555
12=#55555555ffff
13=#ffff5555ffff
14=#5555ffffffff
15=#ffffffffffff
palette_size=16
foreground=#aaaaaaaaaaaa
background=#000000000000
cursor=#cccccccccccc
bold=
dim=

Shortcuts/Default:

[roxterm shortcuts scheme]
File/New Window=
File/New Tab=
File/Close Window=
File/Close Tab=
Tabs/Previous Tab=
Tabs/Next Tab=
Edit/Copy=
Edit/Paste=
View/Zoom In=<Control>plus
View/Zoom Out=<Control>minus
View/Normal Size=<Control>0
View/Full Screen=F11
View/Scroll Up One Line=
View/Scroll Down One Line=
Help/Help=
Edit/Copy & Paste=
Search/Find...=
Search/Find Next=
Search/Find Previous=
File/New Window With Profile/Default=
File/New Tab With Profile/Default=
Tabs/Select_Tab_0=<Alt>F1
Tabs/Select_Tab_1=<Alt>F2
Tabs/Select_Tab_2=<Alt>F3
Tabs/Select_Tab_3=<Alt>F4
Tabs/Select_Tab_4=<Alt>F5
Tabs/Select_Tab_5=<Alt>F6
Tabs/Select_Tab_6=<Alt>F7
Tabs/Select_Tab_7=<Alt>F8
Tabs/Select_Tab_8=<Alt>F9
Tabs/Select_Tab_9=<Alt>F9
Tabs/Select_Tab_10=<Alt>F10
Tabs/Select_Tab_11=<Alt>F11
Tabs/Select_Tab_12=<Alt>F12

Global:

[roxterm options]
edit_shortcuts=0
prefer_dark_theme=1
colour_scheme=Default
warn_close=1

Missing Nothing of my initial requirements seems to be missing, really, so I'm sticking to it for a while to see what happens. The first itch to scratch is that when the menubar is hidden, the popup menu becomes the entire menubar contents, which does not fit the general use case to have a contextual menu with the most common shortcuts. I'll just declare it useless and get myself used to some new hotkey for starting a new terminal. update: after fixing my issues with gnome-terminal I've switched back to gnome-terminal: its interface feels less clunky as I'm already used to it. Other references Guillem Jover made a similar analysis in 2009, it can be found here. Thomas Koch mentioned that termit should be able to do all I need, and is scriptable in Lua. I like the sound of that, and it's definitely one I should look next time I find myself shopping for terminal emulators.

A new major release 0.4.0 of RProtoBuf, is now on CRAN. RProtoBuf provides GNU R bindings for the Google Protocol Buffers ("Protobuf") data encoding library used and released by Google, and deployed as a language and operating-system agnostic protocol by numerous projects. With this release, we are welcoming Jeroen Ooms to the team. Jeroen had already worked on some RProtoBuf extensions in the context of his OpenCPU project; we have now integrated those Protocol Buffers functions. And Jeroen pushed all the right buttons to finally get RProtoBuf built on everybody's least-favourite operating system by provding a static library for use by win-builder and CRAN. Murray once again did a lot of work on internals. His use of the LLVM tool llvm-format was particular helpful to make our coding style a little more consistent. The complete NEWS file entry for this release follows:

Changes in RProtoBuf version 0.4.0 (2014-01-14)

• Changes to support CRAN builds for MS Windows.
• Added functions serialize_pb, unserialize_pb, and can_serialize_pb plus documentation from Jeroen Ooms RProtoBufUtils package.
• New dir inst/python with some Python examples.
• Added Jeroen Ooms as author.
• Vignettes have been converted to the R 3.0.0 or later use of external vignette builders, no longer need a Makefile
• Added missing methods to dollar completion list for Message, Descriptor, EnumValueDescriptor, and FileDescriptor classes.
• Add missing export for .DollarNames EnumValueDescriptor to allow completion on that class.
• Add more than 15 additional pages to the main Intro vignette documenting better all of the S4 classes implemented by RProtoBuf, updating the type mapping tables to take into account 64-bit support, and documenting advanced features such as Extensions.
• Added better error checking in EnumDescriptors to catch the case when wrong types are provided.
• Updated the FileDescriptor name() method to accept a boolean for full paths just like the generic name() method.
• Correct a bug that incorrectly dispatched as.character() when as.list() was called on Descriptor objects.
• Update FileDescriptor $ dispatch to work properly for the names of fields defined in the FileDescriptor, instead of just returning NULL even for types returned by $ completion.
• Added a reservation for extension fields in the example tutorial.Person schema.
• Support setting int32 fields with character representations and raise an R-level stop() error if the provided string can not be parsed as a 32-bit integer, rather than crashing the R instance.
• Update the project TODO file.
• Add better documentation and tests for all of the above.
• Corrected the handling of uint32 and fixed32 types in protocol buffers to ensure that they work with numbers as large as 2^32 - 1, which is larger than an integer can hold in R since R does not have an unsigned integer class. These values are stored as doubles internally now to avoid losing precision.
• Added unit tests to verify behavior of RProtoBuf with extreme values for uint32 types.
• Removed old exception handling code and instead rely on the more modern Rcpp::stop method maintained in Rcpp.
• Add better error messages when setting a repeated field of messages to inform the user which element index was of the wrong type and what the expected type was.
• Add an optional 'partial' argument to readASCII allowing uninitialized message fragments to be read in.
• (internal) Added const qualifiers in more places throughout the C++ code for type safety.
• (internal) Standardize coding conventions of the C++ files and run them through clang-format for consistency. A STYLE file has been submitted to R-Forge with details about the coding standards and how they are enforced with Emacs and clang-format.
• Applied changes suggested by Kevin Ushey to the S4 class handling to support both the currently released Rcpp as well as the currently pending next version.

CRANberries also provides a diff to the previous release 0.3.2. More information is at the RProtoBuf page which has a draft package vignette, a 'quick' overview vignette and a unit test summary vignette. Questions, comments etc should go to the rprotobuf mailing list off the RProtoBuf page at R-Forge.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

I was back at my parents' over Christmas, like usual. Before I got back my Dad had mentioned they'd been having ADSL stability issues. Previously I'd noticed some issues with keeping a connection up for more than a couple of days, but it had got so bad he was noticing problems during the day. The eventual resolution isn't going to surprise anyone who's dealt with these things before, but I went through a number of steps to try and improve things. Firstly, I arranged for a new router to be delivered before I got back. My old Netgear DG834G was still in use and while it didn't seem to have been the problem I'd been meaning to get something with 802.11n instead of the 802.11g it supports for a while. I ended up with a TP-Link TD-W8980, which has dual band wifi, ADSL2+, GigE switch and looked to have some basic OpenWRT support in case I want to play with that in the future. Switching over was relatively simple and as part of that procedure I also switched the ADSL microfilter in use (I've seen these fail before with no apparent cause). Once the new router was up I looked at trying to get some line statistics from it. Unfortunately although it supports SNMP I found it didn't provide the ADSL MIB, meaning I ended up doing some web scraping to get the upstream/downstream sync rates/SNR/attenuation details. Examination of these over the first day indicated an excessive amount of noise on the line. The ISP offer the ability in their web interface to change the target SNR for the line. I increased this from 6db to 9db in the hope of some extra stability. This resulted in a 2Mb/s drop in the sync speed for the line, but as this brought it down to 18Mb/s I wasn't too worried about that. Watching the stats for a further few days indicated that there were still regular periods of excessive noise, so I removed the faceplate from the NTE5 master BT socket, removing all extensions from the line. This resulted in regaining the 2Mb/s that had been lost from increasing the SNR target, and after watching the line for a few days confirmed that it had significantly decreased the noise levels. It turned out that the old external ringer that was already present on the line when my parents' moved in was still connected, although it had stopped working some time previously. Also there was an unused and much spliced extension in place. Removed both of these and replacing the NTE5 faceplate led to a line that was still stable. At the time of writing the connection has been up since before the new year, significantly longer than it had managed for some time. As I said at the start I doubt this comes as a surprise to anyone who's dealt with this sort of line issue before. It wasn't particularly surprising to me (other than the level of the noise present), but I went through each of the steps to try and be sure that I had isolated the root cause and could be sure things were actually better. It turned out that doing the screen scraping and graphing the results was a good way to verify this. Observe: The blue/red lines indicate the SNR for the upstream and downstream links - the initial lower area is when this was set to a 6db target, then later is a 9db target. Green are the forward error correction errors divided by 100 (to make everything fit better on the same graph). These are correctable, but still indicate issues. Yellow are CRC errors, indicating something that actually caused a problem. They can be clearly seen to correlate with the FEC errors, which makes sense. Notice the huge difference removing the extensions makes to both of these numbers. Also notice just how clear graphing the data makes things - it was easy to show my parents' the graph and indicate how things had been improved and should thus be better.

Somalia's chief exports appear to be morally-ambiguous Salon articles about piracy and sophomoric evidence against libertarianism. However, it is the former topic that Captain Phillips concerns itself with, inspired by the hijacking of the Maersk Alabama container ship in 2009. What is truth? In the end, Captain Phillips does not rise above Pontius Pilate in providing an answer, but it certainly tries using more refined instruments than irony or leaden sarcasm. This motif pervades the film. Obviously, it is based on a "true story" and brings aboard that baggage, but it also permeates the plot in a much deeper sense. For example, Phillips and the US Navy lie almost compulsively to the pirates, whilst the pirates only really lie once (where they put Phillips in greater danger). Notice further that Phillips only starts to tell the truth when he thinks all hope is lost. These telling observations become even more fascinating when you realise that they must be based on the testimony of the, well, liars. Clearly, deception is a weapon to be monopolised and there are few limits on what good guys can or should lie about if they believe they can save lives. Futhermore, there is an utterly disarming epilogue where Phillips is being treated for shock by clinical efficient medical staff. Not only will this scuttle any "blanket around the shoulders" clich but is probably a highly accurate portrayal of what actually happens post-trauma. This echoes the kind of truth Werner Herzog aims for in his filmmaking as well his guilt-inducing duality between uncomfortable lingering and compulsive viewing. Lastly, a starter for a meta-discussion: can a film based on real-world events even be "spoilered"? Hearing headlines on the radio before you read your newspaper hardly robs you of a literary journey... Captain Phillips does have some quotidian problems. Firstly, the only tool for ratcheting up tension is for the Somalians to launch verbal broadsides at the Americans, with each compromise somehow escalating the situation. This technique is effective but well before the climatic rescue scene where it is really needed it has been subject to the most extreme diminishing returns. (I cannot be the first to notice the "Africans festooned with guns shouting incomphensively" trope I hope it is based on a Babel-esque mechanism of disorientation from miscommunication rather than anything more unsavoury.) Secondly, the US Navy acts like a teacher with an Ofsted inspector observing quietly from the corner of the classroom; far too well-behaved it suspends belief, with no post-kill gloating or even the tiniest of post-arrest congratulations. Whilst nobody wants to see the Navy overreact badly to other military branches getting all the glory, nobody wants to see a suspiciously bland recruitment vehicle either. Paradoxically, this hermetic treatment made me unduly fascinated by them as if they were part of some military "uncanny valley". Two quick observations:

• All US Somali interactions are recorded by a naval officer. No doubt a value-for-money defense against a USS Abu Ghraib, but knowing the plot is based on factual events, it was perhaps a little too Baudrillardian to ponder how the presence of the Navy's cameras in a scene actually lent weight to the film's version of events, crucially without me even knowing whether the parallel "real-life" footage is verifiable or not.
• The navigational computers not only seem to require lines to drawn repeatedly between points of interest, but the Maersk Alabama's arbitrary relabelling as MOTHERSHIP seems to imply that an officer could humourously rename a radar contact to something unbecoming of a 12A classification.

Finally, despite the title, the film is actually about two captains; the skillful liar Phillips and ... well, that's the real problem. Whilst Captain Muse is certainly no caricatured Hook, we are offered little depth beyond a "You're not just a fisherman" faux-revelation that leads nowhere. I was left inventing reasons for his akrasia so that he made any sense whatsoever. One could charitably argue that the film attempts to stay objective on Muse, but the inability for the film to take any obvious ethical stance actually seems to confuse and then compromise the narrative. What deeper truth is actually being revealed? Is this film or documentary? Worse still, the moral vacuum is invariably filled by the viewer's existing political outlook: are Somali pirates victims of circumstance who are forced into (alas, regrettable) swashbuckling adventures to pacify plank-threatening warlords? Or are they violent and dangerous criminals who habour an irrational resentment against the West, flimsily represented by material goods in shipping containers? Your improvised answer to this Rorschach test will always sit more haphazardly in the film than any pre-constructed treatment ever could. 6/10

Somalia's chief exports appear to be morally-ambiguous Salon articles about piracy and sophomoric evidence against libertarianism. However, it is the former topic that Captain Phillips concerns itself with, inspired by the hijacking of the Maersk Alabama container ship in 2009. What is truth? In the end, Captain Phillips does not rise above Pontius Pilate in providing an answer, but it certainly tries using more refined instruments than irony or leaden sarcasm. This motif pervades the film. Obviously, it is based on a "true story" and brings aboard all that well-travelled baggage, but it also permeates the plot in a much deeper sense. For example, Phillips and the US Navy lie almost compulsively to the pirates, whilst the pirates only really lie once where they put Phillips in greater danger. Notice further that Phillips only starts to tell the truth when he thinks all hope is lost. These telling observations become even more fascinating when you realise that they must be based on the testimony of the, well, liars. Clearly, deception is a weapon to be monopolised and there are few limits on what good guys can or should lie about if they believe they can save lives. Lastly, there is an utterly disarming epilogue where Phillips is being treated for shock by clinical efficient medical staff. Not only will it scuttle any "blanket around the shoulders" clich but is probably a highly accurate portrayal of what actually happens post-trauma. This echoes the kind of truth Werner Herzog often aims for in his filmmaking as well his guilt-inducing duality between uncomfortable lingering and compulsive viewing. Another angle worthy of discussion: can a film based on real-world events even be "spoilered"? Hearing headlines on the before you read the newspaper hardly robs you of a literary journey... Captain Phillips does have some quotidian problems. Firstly, the only tool for ratcheting up tension is for the Somalians to launch verbal broadsides at the Americans, with each compromise somehow escalating the situation further. This technique is genuinely effective but well before the climatic rescue scene where it is really needed it has been subject to the most extreme diminishing returns. (I cannot be the first to notice the "Africans festooned with guns shouting incomphensively" trope I hope it is based on a Babel-esque mechanism of disorientation and miscommunication rather than anything, frankly, unsavoury.) Secondly, the US Navy acts like a teacher with an Ofsted inspector observing quietly from the corner of the classroom; far too well-behaved it suspends belief, with no post-kill gloating or even the tiniest of post-arrest congratulations. Whilst nobody wants to see the Navy overreact badly to other military branches getting all the glory, nobody wants to see a suspiciously bland recruitment vehicle either. Paradoxically, this hermetic treatment made me unduly fascinated by them, as if they were part of some military "uncanny valley". Two quick observations:

• All US Somali interactions are recorded by a naval officer. No doubt a value-for-money defense against a USS Abu Ghraib, but knowing the plot is based on a factual events, it was perhaps a little too Baudrillardian to ponder how the presence of the Navy's cameras in a scene actually lent weight to the film's version of events, crucially without me even knowing whether the parallel "real life" footage is verifiable or not.
• The navigational computers not only seem to require lines to drawn repeatedly between points of interest, but the Maersk Alabama's arbitrary relabelling as MOTHERSHIP seems to imply that an officer could humourously rename a contact to something unbecoming of a 12A classification.

Finally, despite the title, the film is actually about two captains; the skillful liar Phillips and ... well, that's the real problem. Whilst Captain Muse is certainly no caricatured Hook, we are offered little depth beyond a "You're not just a fisherman" faux-revelation that leads nowhere. I was left inventing reasons for his akrasia so that he made any sense whatsoever. One could charitably argue that the film attempts to stay objective on Muse, but the inability for the film to take any obvious ethical stance actually seems to confuse and then compromise the narrative. What deeper truth is actually being revealed? Is this film or documentary? Worse still, the moral vacuum is invariably filled by the viewer's existing political outlook: are Somali pirates victims of circumstance who are forced into (alas, regrettable) swashbuckling adventures to pacify plank-threatening warlords? Or are they violent and dangerous criminals who habour an irrational resentment against the West, flimsily represented by material goods in shipping containers? Your improvised answer to this Rorschach test will always sit more haphazardly in the film than any pre-constructed treatment ever could. 6/10

One of my pet projects in Debian is the Debian Maintainer Dashboard. Built on top of UDD, DMD provides a maintainer-centric view to answer the I have a few hours for Debian, what should I do now? question (see example). Christophe Siraut did a lot of great work recently on DMD, rewriting much of the internals. As a result, he also added a RSS feed feature: you can now get notified of new TODO list items by subscribing to that feed. If you have suggestions or comments, please use the debian-qa@ list (see this thread). Thanks, Christophe!

I haven't been doing Debian work for a while now, due to various reasons. However, I try to keep reading (but not posting to) the mailing list, such as to keep in touch (in a virtual way). What surprises me is how the community evolves: in some ways not at all (was there an init-system flamewar this week already or not?), in some ways a lot (mailing list bans!). Since I don't post to the mailing lists, consider this post as my +1 towards the listmaster's work: yes, we should limit the damage that abusive people do to the project. I'm surprised at how some people call list bans as censorship (it's not, IMHO), and how they treat abuse as something light. Another thing that surprises me is how abuse and arrogance go hand in hand. In the explanations for bans, I've seen not just one example where the abusive person is not only, well, abusive, but also considers themselves superior (morally, technically, etc.). If you're already so much superior, why hit on people? If you already know so much, why the personal attacks? And no, being immature is not the whole or even the only explanation This page has a nice collection of quotes around the same motive: how you treat other people, especially people whom you regard less knowledgeable than you, tells a lot about your character. One quote is nice in how succinct and how applicable it is to the kind of abuse I've seen:

To insult or to abuse those who cannot resist, or dare not resent the injury, is a sure mark of cowardice, [ ].

From 1852 September 30, Farmer s Cabinet, Courtesy to Inferiors, Page 1, Column 2, Volume 51, Number 8, Amherst, New Hampshire. The rest of the quote also contains a somewhat sexist remark, so I've left that part out (but I'm surprised they still talked about swords in the 1850s). And yes, I hope that this spring I'll be able to get back to my Debian work.

We just released LLVM & Clang 3.4 (already available in Debian Jessie and Ubuntu Trusty). The clang new release introduces some more warning and error detections. The point of the blog post is not to detail the changes (LLVM / Clang) but more to take a step back on the LLVM and Clang toolchains, usages and dissemination. For the last couple years, I have been presenting our work on the Debian side to make Clang part of the Debian infrastructure. I add the chance to speak at many events (Debconf 12 & 13, FOSDEM 2013, Linux Plumbers 2013, Distro recipes, etc). I even had Linus Torvalds attending to one of these talks (and proposing new approach). I had many feedback from the Debian/Ubuntu communities but also from other communities.
More and more projects are using Clang as part of their workflows, in term of development, production and QA. For example, some of them are running continuous integration using clang instead of gcc, others are running scan-build (Wireshark or LLVM itself) or Address Sanitizer (asan) (Firefox for example), etc.
More and more actors are moving from gcc to clang for their developments (Sony with their PS4, Chrome under Mac OS X, Apple, the Linux Kernel with the LLVMLinux project, FreeBSD, etc).
I stopped counting the number of times when people answered me "Yeh, I am now using Clang" when I said I am involved upstream. In parallel, more and more projects are using LLVM as a backend or computing engine and, in the meantime, getting more and more attraction (not because they are based on LLVM but just because they are great projects). For example, Mesa (llvmpipe), Julia, Rust, Emscripten or Native Client (NaCL) are excellent examples of this. The two last are excellent proofs that LLVM based technologies have an impact which goes further than geek-tools and will be used on a daily basis by millions of users. I also saw an interesting and growing number of new projects using libclang and libTooling. My feeling is that the lack of plugin capabilities in gcc for a long time and, now, its current complexity limited the innovation in term of compilation capabilities, analysis and fun tools. The features and the quality of this library are enabling new developments and I am sure we will see more and more excellent tools based on this library during 2014.
For example, the LLVM toolchain ships Clang modernize and clang format and some cool projects like include-what-you-use, DXR, OCLint or creduce are emerging. Thanks to this great competition, gcc is also improving (ASAN, JIT, better warnings, etc), remaining still relevant as before. No doubt that 2014 will be a great year for compilers.

Last night it seems the so called 'Nagios Plugins' project was cut off. The story started a decade ago when development of the plugins compatible with Nagios was taken over by an independend group of developers. Some time later, the domain of the 'Nagios Plugins' project was handed over to 'Nagios Enterprises, LLC.' due trademark reasons. To get an idea about that, I suggest to read on the 'Nagios Trademark Truth' and 'Nagios Trademark Triumph Provides Promise To Open Source Developers, Shows Power of Community'. In the latter Mr. Galstad is cited with "This violation took more than four years and thousands in legal fees before it was finally resolved. My hope is that this can serve as an example to Open Source developers worldwide that they can overcome infringements and protect their brands if they are persistent and engage their respective communities.". Yesterday in the evening the project members recognised that the DNS of nagios-plugins.org was moved to a different location. Now there seems to be a hosted (some may call that 'hijacked' or 'pirated') 'mirror' of the old site. Anyhow ... the content is already different from the original one or maybe changed even more in the future. From my point of view this can serve as an example to Open Source developers worldwide that they can be obstructed by (trademark holding) companies even if those companies profit from the work of them. So please don't use downloads from there, even the release tarballs maybe modified. The news all about that: 'Nagios Plugins are dead, we now have Monitoring Plugins in place!" Be sure you download your tarballs from https://www.monitoring-plugins.org/download.html and verify your checksums! You can follow the Monitoring Plugins Development Team also via Twitter.

Follow me on Identi.ca or Twitter One of my resolutions for 2014 is to keep trying harder to talk about my Debian contributions. So here it is, on a monthly basis this time I hope, quite short this month because I m leaving for holidays at the end of the week and I think I won t have time to contribute more this month. Debian packages Below are some packages I updated recently: 1. Brebis, the fully automated backup checker I successively packaged Brebis, the fully automated backup checker, versions 0.6 to 0.9 (the latter is today in Debian Sid and Jessie) since my last Debian activities blog posts (in french). 2. Pycallgraph, a Python library that creates call graphs for Python programs Pycallgraph is one of the first Debian packages I have been maintaining. So when I noticed it was upgraded after so much time I was really eager to package the new version 1.0.1. Now available in Debian Sid! Just belown an example of the generated graph for the application Belier, a sysadmin tool. 3. Belier, the SSH connection generation tool Nothing really new for Belier, the SSH connection generation tool but I updated the package in order to update the configuration of the Debian package and get rid of some warning messages. Kind of maintenance job I kept avoiding and avoiding, until now. Bug report I d like to take a few seconds to talk about an interesting bug report about the need for a nagios3-dev package I created asking for a new nagios3-dev or nagios3-headers package to offer a simple access to the headers of Nagios for the developers of Nagios external modules. In Nagios, some headers are generated after the ./configure, meaning it may be platform dependent. I (and not only me, the same request was active already by someone other Nagios module developers) thought it was simple to ask the Nagios3 Debian package maintainer to offer these files in a dedicated package. It seems until now people just add the missing files in tarball of their app or in their Debian packages, even if these Nagios headers are not really part of this application. In my opinion, that s why Build-Depends packages are for, don t you think? It seems it is not so simple. I could not understand why it was more important to prevent Debian users who need these files to access these files in a convenient way than letting them to put theses files in there own source tarball/repository. The maintainer told me it could break things. Sure. But we all know an external module of any app often relies on a really specific version of this app. That s nothing new, thats how external modules work. That s not every apps in the world which could provide a stable API to ease the development of external modules. But at least they give access to their dev files or headers when they are FOSS. But feel free to explain to me. After all, the bug reports are still not tagged as "won t fix" Your turn now I d be delighted to have your opinions in the comments of this blog post.

I recently found, that the automount feature of my recent Debian Sid / GNOME 3 system didn't work anymore. For example: I plugged in an USB stick. Nothing happened, although dmesg clearly showed, that the system recognized it. After manually mounting and unmounting the device, nautilus automatically offered to automount it. But I always had to manually mount and unmount it first. Seems the culprit was a pmount installation. JFTR: In some cases also usbmount might cause trouble.