Search Results: "reg"

2 August 2016

Dirk Eddelbuettel: RcppGetconf 0.0.2

A first update for the recent RcppGetconf package for reading system configuration --- not unlike getconf from the libc library --- is now out. Almost immediately after I tweeted / blogged asking for help with OS X builds, fellow Rcpp hacker Qiang Kou created a clever pull request allowing for exactly that. So now we cover two POSIX systems that matter most these days --- Linux and OS X --- but as there are more out there, please do try, test and send those pull requests. We also added a new function getConfig() retrieving a single (given) value complementing the earlier catch-all function getAll(). You can find out more about RcppGetconf from the local RcppGetconf page and the GitHub repo. Changes in this release are as follows:
Changes in inline version 0.0.2 (2016-08-01)
  • A new function getConfig for single values was added.
  • The struct vars is now defined more portable allowing compilation on OS X (PR #1 by Qiang Kou).
Courtesy of CRANberries, there is a diffstat report. More about the package is at the local RcppGetconf page and the GitHub repo.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

1 August 2016

Ben Armstrong: Retiring as a Debian developer

This is a repost and update of my retirement letter sent privately to Debian last month, July 10, 2016. At that time I received many notes of appreciation and good wishes which I treasure. Now, I d like to say goodbye to the broader Debian community and, as well, indicate which of the cleanup items have since been addressed in strikethrough style and with annotations. Also, I d like to stay in touch with many of you, so I have added some comments oriented towards those of you who are interested in doing that after the letter.
When in 1995, on a tip from a friend, I installed Debian on my 386 at work and was enthralled with the results, I could not have foreseen that two years later, friends I had made on channel #debian would nudge me to become a Debian developer. Nor when that happened did I have any idea that twenty years later, I d consider Debian to be like family, the greatest free software community in the world, and would still be promoting it and helping people with it whenever I could. Debian quietly, unexpectedly became a part of what defines me. My priorities in life have changed over that time, though. I have shifted my attention to things that are more important to me in life, such as my family, my health and well-being physically and spiritually, and bringing all I can to bear on the task of preserving our local wilderness areas and trails. In the latter area, I m now bringing all of what Debian has helped shaped me to be to the table, launching some ambitious projects I hope will bear fruit in the coming years, and make a measurable contribution to help us hang onto our precious natural preserves where I live. Unfortunately, as I ve poured more time and energy into these things, I ve increasingly not been giving my packages the care they need. Nor do I have any roles or goals now for any of the Debian projects I was previously involved in. So, after much careful deliberation, and as much as it pains me to say it, it s time to retire as a Debian developer. It has been a great privilege to work with you, and to meet many of you in New York at Debconf 10. I plan to be around online, and will continue to take an interest in Debian, lending a hand when I can. Thanks for all of the fun times, for all that I ve learned, and for the privilege to make awesome things with you. I ll treasure this forever. So much for the soppy bits.  Now, business. These things remain to clean up upon my departure, and I d appreciate help from QA, and anyone else who can lend a hand. My packages are effectively orphaned, but I haven t the time to do any of the cleanup myself, so please speak up if you can help.
  1. Debian Jr.
    • O: junior-doc. The junior-doc package has been awaiting an overhaul by whoever revives the project since I gave it up years ago. I m still listed as maintainer and that should be changed to Debian Junior Maintainers <debianjr-maintainers@lists.alioth.debian.org> if they want it. Otherwise, it is orphaned.
    • I should also be dropped from Uploaders from debian-junior, the metapackages source. Fixed in git.
  2. Tux Paint. This is a very special package that deserves to go to someone who will love it and care for it well. There are three source packages in all:
    • O: tuxpaint
    • O: tuxpaint-config
    • O: tuxpaint-stamps
  3. O: xletters. This is a cute little typing practice game and needs a new maintainer.
  4. XPilot is co-maintained by Phil Brooke <pjb@debian.org>, so he should replace me as Maintainer. Phil said he ll pick up xpilot-ng and will also look at xpilot-extra.
    • xpilot-ng
    • O: xpilot-extra (recently removed from testing due to my neglect, and not co-maintained by Phil; it s unclear if anyone really uses this anymore)
  5. GTypist is co-maintained by Daniel Leidert <dleidert@debian.org> and should replace me as Maintainer.
  6. My ruby packages. A group of packages that I brought into Debian as dependencies of taskwarrior-web, which I never completed. Maybe they ll be useful in and of themselves, and maybe not. In any case, they are maintained by pkg-ruby-extras-maintainers, but I m the sole developer in Uploaders and should be removed: Fixed in git.
    • ruby-blockenspiel
    • ruby-parseconfig
    • ruby-rack-flash3
    • ruby-simple-navigation
    • ruby-sinatra-simple-navigation
    • ruby-term-ansicolor
    • ruby-versionomy
  7. Debian Live stuff: I am listed in Uploaders for live-manual (fixed in git) and debian-installer-launcher (fixed in git) and need to be removed.
  8. O: eeepc-acpi-scripts. The defunct Debian EeePC project has just this one package. Recently, the mailing list was asked about its status, and it was recently NMU d. To my knowledge, nobody from the original team remains to take care of it, so it needs a new maintainer. I should be removed from Uploaders, and since the Debian Eee PC Team no longer exists, it should be removed as maintainer. It is effectively orphaned unless someone speaks up.
There are also some Alioth projects / lists that are defunct that I ll need to talk to the Alioth admins about cleaning up in the coming days. One of these is <debian-eeepc-devel@lists.alioth.debian.org> and since it is still listed as the maintainer of eeepc-acpi-scripts, that needs to be sorted out before the list can be closed. Thanks again, and see you around!
Ben

Stay in touch For those of you who would like to stay in touch, here are some ways to do that:
  • Follow my blog: http://syn.theti.ca
    If you already do that, great! If not, welcome to my blog! For the past couple of years you may have noticed a decrease in technical content and increase in local trails and conservation oriented posts. You can expect more of the latter.
  • Say hi to me on irc: SynrG (also SynrGy) on irc.oftc.net (irc.debian.org) or irc.freenode.net.
    I still intend to hang out and offer support when I can, just no longer as a developer. Channel #debian-offtopic on either network is a good place to catch up with me socially.
  • Follow me on Facebook: https://www.facebook.com/SynrG
    For better or worse, a lot of the trails and conservation folks hang out here, and many of you in the Debian community are already my Facebook friends.
  • Look for my Bluff Trail posts on their site: https://wrweo.ca
    Providing tech support to this organization is where much of my time and energy is going these days. I post here once in a while, but do most of my work behind the scenes as a volunteer and, newly this year, as a board member.
Facebooktwittergoogle_plusredditpinterestlinkedinmail

Petter Reinholdtsen: Techno TV broadcasting live across Norway and the Internet (#debconf16, #nuug) on @frikanalen

Did you know there is a TV channel broadcasting talks from DebConf 16 across an entire country? Or that there is a TV channel broadcasting talks by or about Linus Torvalds, Tor, OpenID, Common Lisp, Civic Tech, EFF founder John Barlow, how to make 3D printer electronics and many more fascinating topics? It works using only free software (all of it available from Github), and is administrated using a web browser and a web API. The TV channel is the Norwegian open channel Frikanalen, and I am involved via the NUUG member association in running and developing the software for the channel. The channel is organised as a member organisation where its members can upload and broadcast what they want (think of it as Youtube for national broadcasting television). Individuals can broadcast too. The time slots are handled on a first come, first serve basis. Because the channel have almost no viewers and very few active members, we can experiment with TV technology without too much flack when we make mistakes. And thanks to the few active members, most of the slots on the schedule are free. I see this as an opportunity to spread knowledge about technology and free software, and have a script I run regularly to fill up all the open slots the next few days with technology related video. The end result is a channel I like to describe as Techno TV - filled with interesting talks and presentations. It is available on channel 50 on the Norwegian national digital TV network (RiksTV). It is also available as a multicast stream on Uninett. And finally, it is available as a WebM unicast stream from Frikanalen and NUUG. Check it out. :)

Chris Lamb: Free software activities in July 2016

Here is my monthly update covering a large part of what I have been doing in the free software world (previously):



Debian
  • Created a proof-of-concept wrapper for pymysql to reduce the diff between Ubuntu and Debian's packaging of python-django. (tree)
  • Improved the NEW queue HTML report to display absolute timestamps when placing the cursor over relative times as well as to tidy the underlying HTML generation.
  • Tidied and pushed for the adoption of a patch against dak to also send mails to the signer of an uploaded package on security-master. (#796784)

LTS

This month I have been paid to work 14 hours on Debian Long Term Support (LTS). In that time I did the following:
  • "Frontdesk" duties, triaging CVEs, etc.
  • Improved the bin/lts-cve-triage.py script to ignore packages that have been marked as unsupported.
  • Improved the bin/contact-maintainers script to print a nicer error message if you mistype the package name.
  • Issued the following advisories:
    • DLA 541-1 for libvirt making the password policy consistent across the QEMU and VNC backends with respect to empty passwords.
    • DLA 574-1 for graphicsmagick fixing two denial-of-service vulnerabilities.
    • DLA 548-1 and DLA 550-1 for drupal7 fixing an open HTTP redirect vulnerability and a privilege escalation issue respectfully.
    • DLA 557-1 for dietlibc removing the current directory from the current path.
    • DLA 577-1 for redis preventing the redis-cli tool creating world-readable history files.

Uploads
  • redis:
    • 3.2.1-2 Avoiding race conditions in upstream test suite.
    • 3.2.1-3 Correcting world_readable ~/.rediscli_history files.
    • 3.2.1-4 Preventing a race condition in the previous upload's patch.
    • 3.2.2-1 New upstream release.
    • 3.2.1-4~bpo8+1 Backport to jessie-backports.
  • strip-nondeterminism:
    • 0.020-1 Improved the PNG handler to not blindly trust chunk sizes, rewriting most of the existing code.
    • 0.021-1 Correcting a regression in the PNG handler where it would leave temporary files in the generated binaries.
    • 0.022-1 Correcting a further regression in the PNG handler with respect to IEND chunk detection.
  • python-redis (2.10.5-1~bpo8+1) Backport to jessie-backports.
  • reprotest (0.2) Sponsored upload.

Patches contributed


I submitted patches to fix faulty initscripts in lm-sensors, rsync, sane-backends & vsftpd.

In addition, I submitted 7 patches to fix typos in debian/rules against cme:, gnugk: incorrect reference to dh_install_init, php-sql-formatter, python-django-crispy-forms, libhook-lexwrap-perl, mknbi & ruby-unf-ext.

I also submitted 6 patches to fix reproducible toolchain issues (ie. ensuring the output is reproducible rather than the package itself) against libextutils-parsexs-perl: Please make the output reproducible, perl, naturaldocs, python-docutils, ruby-ronn & txt2tags.

Lastly, I submitted 65 patches to fix specific reproducibility issues in amanda, boolector, borgbackup, cc1111, cfingerd, check-all-the-things, cobbler, ctop, cvs2svn, eb, eurephia, ezstream, feh, fonts-noto, fspy, ftplib, fvwm, gearmand, gngb, golang-github-miekg-pkcs11, gpick, gretl, hibernate, hmmer, hocr, idjc, ifmail, ironic, irsim, lacheck, libmemcached-libmemcached-perl, libmongoc, libwebsockets, minidlna, mknbi, nbc, neat, nfstrace, nmh, ntopng, pagekite, pavuk, proftpd-dfsg, pxlib, pysal, python-kinterbasdb, python-mkdocs, sa-exim, speech-tools, stressapptest, tcpflow, tcpreen, ui-auto, uisp, uswsusp, vtun, vtwm, why3, wit, wordgrinder, xloadimage, xmlcopyeditor, xorp, xserver-xorg-video-openchrome & yersinia.

RC bugs

I also filed 68 RC bugs for packages that access the internet during build against betamax, curl, django-localflavor, django-polymorphic, dnspython, docker-registry, elasticsearch-curator, elib.intl, elib.intl, elib.intl, fabulous, flask-restful, flask-restful, flask-restful, foolscap, gnucash-docs, golang-github-azure-go-autorest, golang-github-fluent-fluent-logger-golang, golang-github-franela-goreq, golang-github-mesos-mesos-go, golang-github-shopify-sarama, golang-github-unknwon-com, golang-github-xeipuuv-gojsonschema, htsjdk, lemonldap-ng, libanyevent-http-perl, libcommons-codec-java, libfurl-perl, libgravatar-url-perl, libgravatar-url-perl, libgravatar-url-perl, libgravatar-url-perl, libgravatar-url-perl, libhttp-async-perl, libhttp-oai-perl, libhttp-proxy-perl, libpoe-component-client-http-perl, libuv, libuv1, licenseutils, licenseutils, licenseutils, musicbrainzngs, node-oauth, node-redis, nodejs, pycurl, pytest, python-aiohttp, python-asyncssh, python-future, python-guacamole, python-latexcodec, python-pysnmp4, python-qtawesome, python-simpy, python-social-auth, python-structlog, python-sunlight, python-webob, python-werkzeug, python-ws4py, testpath, traitlets, urlgrabber, varnish-modules, webtest & zurl.


Finally, I filed 100 FTBFS bugs against abind, backup-manager, boot, bzr-git, cfengine3, chron, cloud-sptheme, cookiecutter, date, django-uwsgi, djangorestframework, docker-swarm, ekg2, evil-el, fasianoptions, fassets, fastinfoset, fest-assert, fimport, ftrading, gdnsd, ghc-testsuite, golang-github-magiconair-properties, golang-github-mattn-go-shellwords, golang-github-mitchellh-go-homedir, gplots, gregmisc, highlight.js, influxdb, jersey1, jflex, jhdf, kimwitu, libapache-htpasswd-perl, libconfig-model-itself-perl, libhtml-tidy-perl, liblinux-prctl-perl, libmoox-options-perl, libmousex-getopt-perl, libparanamer-java, librevenge, libvirt-python, license-reconcile, louie, mako, mate-indicator-applet, maven-compiler-plugin, mgt, mgt, mgt, misc3d, mnormt, nbd, ngetty, node-xmpp, nomad, perforate, pyoperators, pyqi, python-activipy, python-bioblend, python-cement, python-gevent, python-pydot-ng, python-requests-toolbelt, python-ruffus, python-scrapy, r-cran-digest, r-cran-getopt, r-cran-lpsolve, r-cran-rms, r-cran-timedate, resteasy, ruby-berkshelf-api-client, ruby-fog-libvirt, ruby-grape-msgpack, ruby-jquery-rails, ruby-kramdown-rfc2629, ruby-moneta, ruby-parser, ruby-puppet-forge, ruby-rbvmomi, ruby-redis-actionpack, ruby-unindent, ruby-web-console, scalapack-doc, scannotation, snow, sorl-thumbnail, svgwrite, systemd-docker, tiles-request, torcs, utf8proc, vagrant-libvirt, voms-api-java, wcwidth, xdffileio, xmlgraphics-commons & yorick.

FTP Team

As a Debian FTP assistant I ACCEPTed 114 packages: apertium-isl-eng, apertium-mk-bg, apertium-urd-hin, apprecommender, auto-apt-proxy, beast-mcmc, caffe, caffe-contrib, debian-edu, dh-make-perl, django-notification, dpkg-cross, elisp-slime-nav, evil-el, fig2dev, file, flightgear-phi, friendly-recovery, fwupd, gcc-5-cross, gdbm, gnustep-gui, golang-github-cznic-lldb, golang-github-dghubble-sling, golang-github-docker-leadership, golang-github-rogpeppe-fastuuid, golang-github-skarademir-naturalsort, golang-glide, gtk+2.0, gtranscribe, kdepim4, kitchen, lepton, libcgi-github-webhook-perl, libcypher-parser, libimporter-perl, liblist-someutils-perl, liblouis, liblouisutdml, libneo4j-client, libosinfo, libsys-cpuaffinity-perl, libtest2-suite-perl, linux, linux-grsec, lua-basexx, lua-compat53, lua-fifo, lua-http, lua-lpeg-patterns, lua-mmdb, lua-openssl, mash, mysql-5.7, node-quickselect, nsntrace, nvidia-graphics-drivers, nvidia-graphics-drivers-legacy-304xx, nvidia-graphics-drivers-legacy-340xx, openorienteering-mapper, oslo-sphinx, p4est, patator, petsc, php-mailparse, php-yaml, pykdtree, pypass, python-bioblend, python-cotyledon, python-jack-client, python-mido, python-openid-cla, python-os-api-ref, python-pydotplus, python-qtconsole, python-repoze.sphinx.autointerface, python-vispy, python-zenoss, r-cran-bbmle, r-cran-corpcor, r-cran-ellipse, r-cran-minpack.lm, r-cran-rglwidget, r-cran-rngtools, r-cran-scatterd3, r-cran-shinybs, r-cran-tibble, reproject, retext, ring, ruby-github-api, ruby-rails-assets-jquery-ui, ruby-swd, ruby-url-safe-base64, ruby-vmstat, ruby-webfinger, rustc, shadowsocks-libev, slepc, staticsite, steam, straight.plugin, svgwrite, tasksh, u-msgpack-python, ufo2otf, user-mode-linux, utf8proc, vizigrep, volk, wchartype, websockify & wireguard.

31 July 2016

Joey Hess: PocketCHIP quick review

PocketCHIP is the pocket sized Linux terminal I always used to want. Which is to say, it runs (nearly) stock Debian, X, etc, it has a physical keyboard, and the hardware and software is (nearly) non-proprietary and very hackable. Best of all, it's fun and it encourages playful learning. It's also clunky and flawed and constructed out of cheap components. This keeps it from being something I'd actually carry around in my pocket and use regularly. The smart thing they've done though is embrace these limitations, targeting it at the hobbiest, and not trying to compete with smart phones. The PocketCHIP is its own little device in its own little niche. PocketCHIP front, side, and back views Unless you're into hardware hacking and want to hook wires up to the GPIO pins, the best hardware feature is the complete keyboard, with even Escape and Control and arrow keys. You can ssh around and run vi on it, run your favorite REPL (I use ghci) to do quick programming, etc. The keyboard is small and a little strange, but you get used to it quickly; your QWERTY muscle memory is transferrable to it. I had fun installing nethack on it and handing it to my sister who had never played nethack before, to watch her learn to play. The screen resolution is 480x272, which is pretty tiny. And, it's a cheap resistive touchscreen, with a bezil around it. This makes it very hard to use scroll bars and icons near the edge of the screen. The customized interface that ships with it avoids these problems, and so I've been using that for now. When I have time, I plan to put a fullscreen window manager on it, and write a pdmenu menu configuration for it, so everything can be driven using the keyboard. I also have not installed Debian from scratch on it yet. This would be tricky because it uses a somewhat patched kernel (to support the display and wifi). The shipped distribution is sadly not entirely free software. There are some nonfree drivers and firmwares. And, they included a non-free gaming environment on it (a very nice one for part of the audience, that allows editing the games, but non-free nevertheless). They did do a good job of packaging up all the custom software they include on it, although they don't seem to have published source packages for everything. (They might be infringing my GPL copyright of flash-kernel by distributing a modified version without source. I say "might" because flash-kernel is a pile of shell scripts, so you could probably extract the (probably trivial) modifications. Still.. Also, they seem to have patched network-manager in some way and I wasn't able to find the corresponding source.) The battery life is around 5 hours. Unfortunately the "sleep" mode only turns off the backlight and maybe wifi, and leaves the rest of the system running. This and the slightly awkward form factor too big to really comfortably fit in a pocket limit the use of PocketCHIP quite a bit. Perhaps the sleeping will get sorted out, and perhaps I'll delete the GPIO breakout board from the top of mine to make it more pocket sized.

Ritesh Raj Sarraf: User Mode Linux

Recently, we had the User-Mode Linux suite out of Debian, which included user-mode-linux, user-mode-linux-doc and uml-utilities package. We are happy that we were able to bring it back into the archvie quick, and hope to maintain it active. For many who may not know about UML, here's a discription from its website:
User-Mode Linux is a safe, secure way of running Linux versions and Linux processes. Run buggy software, experiment with new Linux kernels or distributions, and poke around in the internals of Linux, all without risking your main Linux setup.
User-Mode Linux gives you a virtual machine that may have more hardware and software virtual resources than your actual, physical computer. Disk storage for the virtual machine is entirely contained inside a single file on your physical machine. You can assign your virtual machine only the hardware access you want it to have. With properly limited access, nothing you do on the virtual machine can change or damage your real computer, or its software.
Most of the use cases mentioned here are achievable with Containers today. The big difference UML provides in, is with a separate kernel. UML is an implementation of Linux as an architecture of Linux itself. It supports x86 and x86_64 architecture. And given that it is the port of the kernel, you can do many of the tests and experiments of the regular kernel, safely inside a confined UML environment. As with other virtualization implementations, the limitation comes in if you are working on physical hardware. With its re-entry in Debian, I wanted to revive my local setup. First is the packaging structure and the second is its integration with current standard tools
  • Packaging: For packaging UML in Debian, we rely on the packaged linux-source package provided by the kernel team.
    • linux-source package: We build UML from the linux sources that are provided by the debian kernel team. This works fine for now. Whenever there's a kernel vulnerability, there'll be an updated source package, to which we'll rebuild the UML package.
    • Merge with debian-kernel: In the longer run, we'd like to push UML package into the debian kernel team. UML is a component of the Linux kernel, and that is where it should be built from.
  • Integration: It works fairly well right now. On modern systems with systemd, where (nspawn) containers can easily have a network interface bound to it, UML lacks a bit behind. It'd be nice if we could see some UML integration with systemd.
    • Networking under systemd: Setting up networking, for UML, under systemd is fairly straight. In fact, with systemd, it is much simpler. Below is the host network (tuntap) setup, to which UML can bind for all its network needs.
rrs@learner:~/tidBits (master)$ cat /etc/systemd/network/tap.netdev 
[NetDev]
Name=tap0
Kind=tap
[TAP]
Group=uml-net
User=uml-net
2016-08-01 / 15:41:40       
rrs@learner:~/tidBits (master)$ cat /etc/systemd/network/tap.network 
[Match]
Name=tap0
[Network]
DHCPServer=yes
IPForward=yes
IPMasquerade=yes
Address=172.16.20.2
LLMNR=yes
MulticastDNS=yes
DNS=172.16.20.1
2016-08-01 / 15:41:43       

systemd allows for defining user/group ownership in its file. With this setup, and uml-utilities running, one can simply fire a UML instance as below:

rrs@learner:~/rrs-home/Community/Packaging/user-mode-linux (master)$ linux ubd0=~/rrs-home/NoTrack/uml.img eth0=daemon mem=1024M rw  
Core dump limits :
    soft - 0
    hard - NONE
Checking that ptrace can change system call numbers...OK
Checking syscall emulation patch for ptrace...OK
Checking advanced syscall emulation patch for ptrace...OK
Checking environment variables for a tempdir...none found
Checking if /dev/shm is on tmpfs...OK
Checking PROT_EXEC mmap in /dev/shm...OK
Adding 23609344 bytes to physical memory to account for exec-shield gap
Linux version 4.6.3 (root@chutzpah) (gcc version 5.4.0 20160609 (Debian 5.4.0-6) ) #2 Sat Jul 16 16:22:22 UTC 2016
Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 263721
Kernel command line: ubd0=/home/rrs/rrs-home/NoTrack/uml.img eth0=daemon mem=1024M rw root=98:0
PID hash table entries: 4096 (order: 3, 32768 bytes)
Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes)
Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes)
Memory: 1020852K/1071632K available (4803K kernel code, 1207K rwdata, 1340K rodata, 157K init, 217K bss, 50780K reserved, 0K cma-reserved)
SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
NR_IRQS:15
.....snipped.......
root@uml:~# cat /proc/cpuinfo 
processor    : 0
vendor_id    : User Mode Linux
model name    : UML
mode        : skas
host        : Linux learner 4.6.0-1-amd64 #1 SMP Debian 4.6.4-1 (2016-07-18) x86_64
bogomips    : 6048.97
root@uml:~# ping www.debian.org
PING www.debian.org (130.89.148.14) 56(84) bytes of data.
64 bytes from klecker4.snt.utwente.nl (130.89.148.14): icmp_seq=1 ttl=46 time=372 ms
64 bytes from klecker4.snt.utwente.nl (130.89.148.14): icmp_seq=2 ttl=46 time=395 ms
64 bytes from klecker4.snt.utwente.nl (130.89.148.14): icmp_seq=3 ttl=46 time=315 ms
^C
--- www.debian.org ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 315.550/361.064/395.440/33.556 ms

And here are some (incomplete and non-conclusive) performance numbers

root@uml:~# dd if=/dev/zero of=foo.img bs=1M count=2500 conv=fsync
2500+0 records in
2500+0 records out
2621440000 bytes (2.6 GB, 2.4 GiB) copied, 39.4876 s, 66.4 MB/s
vs
rrs@learner:/var/tmp/Debian-Build/Result$ dd if=/dev/zero of=foo.img bs=1M count=2500 conv=fsync
2500+0 records in
2500+0 records out
2621440000 bytes (2.6 GB, 2.4 GiB) copied, 41.2126 s, 63.6 MB/s
2016-08-01 / 15:59:15       

Categories:

Keywords:

Like:

29 July 2016

Dirk Eddelbuettel: RcppGetconf 0.0.1

A new package RcppGetconf to read system configuration --- not unlike getconf from the libc library. Now R can read what system calls sysconf, pathconf and confstr have to say. The package is still pretty green, and now on CRAN in a very first version, corresponding to a very first (and single !) commit. Right now, the CRAN just has one function getAll() similar to getconf -a. A first example shows how it provides all values which can be retried -- currently 320 on my systems.
R> res <- getAll()
R> head(res)
               key value type
1         LINK_MAX 65000 path
2  _POSIX_LINK_MAX 65000 path
3        MAX_CANON   255 path
4 _POSIX_MAX_CANON   255 path
5        MAX_INPUT   255 path
6 _POSIX_MAX_INPUT   255 path
R> tail(res)
                      key  value type
315    LEVEL4_CACHE_ASSOC      0  sys
316 LEVEL4_CACHE_LINESIZE      0  sys
317                  IPV6 200809  sys
318           RAW_SOCKETS 200809  sys
319           _POSIX_IPV6 200809  sys
320    _POSIX_RAW_SOCKETS 200809  sys
R> 
Earlier this evening I added a second function to the GitHub repo which can access indivial values. But right now, the biggest need is really for someone with some systems skills---and an OS X machine---to look at the code, and maybe the getconf.c from the C library, in order to make this build OS X. If you can help, please get in touch. More about the package is at the local RcppGetconf page and the GitHub repo.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

26 July 2016

Norbert Preining: TUG 2016 Day 1 Routers and Reading

The first day of the real conference started with an excellent overview of what one can do with TeX, spanning from traditional scientific journal styles to generating router configuration for cruising ships.
tug2016-color All this was crowned with an invited talk my Kevin Larson from Microsoft s typography department on how to support reading comprehension. Pavneet Aurora Opening: Passport to the TeX canvas Pavneet, our never-sleeping host and master of organization, opened the conference with a very philosophical introduction, touching upon a wide range of topics ranging from Microsoft, Twitter to the beauty of books, pages, and type. I think at some point he even mentioned TeX, but I can t remember for sure. His words put up a very nice and all-inclusive stage, a community that is open to all kind of influences with any disregard or prejudice. Let us hope that is reflects reality. Thanks Pavneet. Geoffrey Poore Advances in PythonTeX Our first regular talk was by Geoffrey reporting on recent advances in PythonTeX, a package that allows including python code in your TeX document. Starting with an introduction to PythonTeX, Geoggrey reports about an improved verbatim environment, fvextra, which patches fancyvrb, and improved interaction between tikz and PythonTeX. As I am a heavy user of listings for my teaching on algebraic specification languages, I will surely take a look at this package and see how it compares to listings. Stefan Kottwitz TeX in industry I: Programming Cisco network switches using TeX Next was Stefan from Lufthansa Industry Solutions, who reported first about his working environment, Cruise Ships with a very demanding IT infrastructure he has to design and implement. Then he introduced us to his way of generating IP configurations for all the devices using TeX. The reason he chose this method is that it allows him to generate at the same time proper documentation. It was surprising for me to hear that by using TeX he could far more efficiently and quicker produce well designed and easily accessible documentation, which helped both the company as well as made the clients happy! Stefan Kottwitz TeX in industry II: Designing converged network solutions After a coffee break, Stefan continued his exploration into industrial usage of TeX, this time about using tikz to generate graphics representing the network topology on the ships. Boris Veytsman Making ACM LaTeX styles Next up was Boris which brought us back to traditional realms of TeX when he guided us into the abyss of ACM LaTeX styles he tried to maintain for some time, until he plunged into a complete rewrite of the styles. Frank Mittelbach Alice goes floating global optimized pagination including picture placements The last talk before lunch (probably a strategic placement, otherwise Frank would continue for hours and hours) was Frank on global optimization of page breaks. Frank showed us what can and can not be done with current LaTeX, and how to play around with global optimization of pagination, using Alice in Wonderland as running example. We can only hope that his package is soon available in an easily consumable version to play around. Thai lunch Pavneet has organized three different lunch-styles for the three days of the conference, today s was Thai with spring rools, fried noodles, one kind of very interesting orange noodles, and chicken something. Michael Doob baseball rules summary After lunch Michael gave us an accessible explanation of the most arcane rules a game can have the baseball rules by using pseudo code. I think the total number of loc needed to explain the overall rules would fill more pages than the New York phonebook, so I am deeply impressed by all those who can understand these rules. Some of us even wandered off in the late afternoon to see a match with life explanations of Michael. Amartyo Banerjee, S.K. Venkatesan A Telegram bot for printing LaTeX files Next up was Amartyo who showed a Telegram (as in messenger application) bot running on a Raspberry Pi, that receives (La)TeX files and sends back compiled PDF files. While it is not ready for consumption (If you sneeze the bot will crash!), it looks like a promising application. Furthermore, it is nice to see how open APIs (like Telegram) can spur development of useful tools, while closed APIs (including threatening users, like WhatApp) hinders it. Norbert Preining Security improvements in the TeX Live Manager and installer Next up was my own talk about beefing up the security of TeX Live by providing integrity and authenticity checks via GnuPG, a feature that has been introduced with the recent release of TeX Live 2016. The following discussion gave me several good idea on how to further improve security and usability. Arthur Reutenauer -The TeX Live M sub-project (and open discussion) Arthur presented the TeX Live M (where the M stands supposedly for Mojca, who couldn t attend unfortunately) project: Their aim is to provide a curated and quality verified sub-part of TeX Live that is sufficiently complete for many applications, and easier for distributors and packagers. We had a lively discussion after Arthur s short presentation, mostly about why TeX Live does not have a on-the-fly installation like MikTeX. I insisted that this is already possible, using the tex-on-the-fly package which uses the mktextex infrastructure, but also caution against using it by default due to delays induced by repeatably reading the TeX Live database. I think this is a worth-while project for someone interested in learning the internals of TeX Live, but I am not sure whether I want to invest time into this feature. Another discussion point was about a testing infrastructure, which I am currently working on. This is in fact high on my list, to have some automatic minimal functionality testing a LaTeX package should at least load! Kevin Larson Reading between the lines: Improving comprehension for students Having a guest from Microsoft is rather rare in our quite Unix-centered environment, so big thanks to Pavneet again for setting up this contact, and big thanks to Kevin for coming. Kevin gave us a profound introduction to reading disabilities and how to improve reading comprehension. Starting with an excursion into what makes a font readable and how Microsoft develops optimally readable fonts, he than turned to reading disabilities like dyslexia, and how markup of text can increase students comprehension rate. He also toppled my long-term believe that dyslexia is connected to the similar shape of letters which are somehow visually malprocessed this was the scientific status from the 1920ies till the 70ies, but since then all researchers have abandoned this interpretation and dyslexia is now linked to problems linking shape to phonems. Kevin did an excellent job with a slightly difficult audience some people picking about grammer differences between British and US English and permanently derailing the discussion, and even more the high percentage of typographically somehow specially tasted participants. After the talk I had a lengthy discussion with Kevin about if/how this research can be carried over to non-Roman writing systems, in particular Kanji/Hanzi based writing systems, where dyslexia shows itself probably in different context. Kevin also mentioned that they want to add interword space to Chinese to help learners of Chinese (children, foreigners) to better parse, and studies showed that this helps a lot in comprehension. On a meta-level, this talk bracketed with the morning introduction by Pavneet, describing an open environment with stimulus back and forth in all directions. I am very happy that Kevin took the pain to come in his tight schedule, and I hope that the future will bring better cooperation at the end we are all working somehow on the same front only the the tools differ.
izakaya-sake-partyAfter the closing of the session, one part of our group went off to the baseball match, while another group dived into a Japanese-style Izakaya where we managed to kill huge amounts of sake and quite an amount of food. The photo shows me after the first bottle of sake, while just seeping on an intermediate small amount of genshu (kind of strong undiluted sake) before continuing to the next bottle. An interesting and stimulating first day of TUG, and I am sure that everyone was looking forward to day 2.

25 July 2016

Norbert Preining: TUG 2016 Day 0 Books and Beers

The second pre-conference day was dedicated to books and beers, with a visit to an exquisite print studio, and a beer tasting session at one of the craft breweries in Canada. In addition we could grab a view into the Canadian lifestyle by visiting Pavneet s beautiful house in the countryside, as well as enjoying traditional style pastries from a bakery.
Heidelberg printing machine at Porcupine's quilt
In short, a perfect combination for us typography and beer savvy freaks! This morning we had somehow an early start from the hotel. Soon the bus left downtown Toronto and entered countryside Ontario, large landscapes filled with huge (for my Japanese feeling) estates and houses, separated by fields, forests and wild landscape. Very beautiful and inviting to live there. On our way to the printing workshop we stopped at Pavneet s house for a very short visit of the exterior, which includes mathematics in the bricking. According to Pavneet, his kids hate to see math on the wall I would be proud to have it.
Pavneet's house is hiding some mathematics
A bit further on we entered into Erin, where the Porcupine s Quill is located. A small building along the street, one could easily oversee that rare jewel! In addition considering that according to the owners, Google Maps has a bad error which would lead you to a completely different location. This printing workshop, led by Tim and Elke Inkster, is producing books in a traditional style using an old Heidelberg Offset printing machine.
Entrance to the Porcupine's Quill, a local bookshop doing excellent printing
Elke introduced us to the sewing of folded signatures together with a lovely old sewing machine. It was the first time I actually saw one in action.
Sewn signatures
Tim, the head master of the printing shop, first entertained us with stories about Chinese publishers visiting them in the old cold-war times, before diving into explanations of the actual machines around, like the Heidelberg offset printing machine.
Master Tim is showing us offset technique
In the back of the basement of the little studio there is also a huge folding machine, which cuts up the big signatures of 16 pages and folds them into bundles. An impressive example of tricky engineering.
The folding machine creates from a printed signature 16 pages in proper order.
Due to the small size of the printing studio, our groups were actually split into two groups, and while the other group got its guided tour, we grabbed coffee and traditional cookies and pastries from the nearby Holtom s bakery. Loads of nice pastries with various filling, my favorite being the slightly salty cherry pie, and above all the rhubarb-raspberry pie.
Nearby old-style bakery, selling Viennese-style "Kaisersemmel", calling them "Kaiser buns". There must be an Austrian hiding somewhere around.
To my absolute astonishment I also found there a Viennese Kaisersemmel , called Kaiser bun here, but keeping the shape and the idea (but unfortunately not the crispy cracky quality of the original in Vienna). Of course I got two of them, together with a nice jam from the region, and enjoyed these Viennese breakfast the next day morning.
Viennese breakfast from the Bakery near Porcupine's Quill
Leaving the Quill we headed for a lunch in a nice pizzeria (I got Pizza Toscana) which also served excellent local beer how would I like to have something like this over in Japan! Our last stop on this excursion was Stone Hammer Brewery, ne of the most famous craft breweries in Canada.
One of the top craft breweries in Canada, the Stone Hammer
keep-calmAlthough they wont win a prize for typography (besides one page of a coaster there that carried a nice pun), their beers are exquisite. We got five different beers to taste, plus extensive explanations on brewing methods and differences. Now I finally understand why most of the new craft breweries in Japan are making Ales: Ales don t need a long process and are ready for sale in rather short time, compared to e.g. lagers.)
Explanations of the the secrets of beer brewing
Nothing to add to this poster found in the Stone Hammer Brewery!
Also at the Stone Hammer Brewery I spotted this very nice poster on the wall of the toilet. And I cannot agree more, everything can easily be discussed over a good beer it calms down aversions, makes even the worst enemies friends, and is healthy for both the mind and body. Filled with excellent beer, some of us (notably an unnamed US TeXnician and politician), stacked up on beers to carry home. I was very tempted to get a huge batch, but putting cans into an airplane seems to be not the optimal idea. Since we are talking cans, I was surprised to hear that many craft beer brewers nowadays prefer cans due to their better protection of the beer from light and oxygen, both killers of good beer. Before leaving we took a last look at the Periodic Table of Beer Types, which left me in awe about how much I don t know and I probably never will know. In particular, I heard the first time of a Vienna style beer Vienna is not really famous for beer, better to say, it is infamous. So maybe this is a different Vienna than my home town that is meant here.
Lots to study here, the Periodic Table of Beers
Another two hour bus ride brought us back to Toronto, where we met with other participants at the reception in a restaurant of Mediterranean cuisine, where I could enjoy for the first time in years a good Tahina and Humus. All around another excellent day, now I only would like to have two days of holidays, guess I need to relax in the lectures starting from tomorrow.

24 July 2016

Tom Marble: jai-gagne-le-tour-de-crosstown-2016

J'ai gagn le Tour de Crosstown 2016! Everyone knows that today the finish line for Le Tour de France was crossed on Les Champs- lys es in Paris... And if you haven't seen some of the videos I highly recommend checking out the onboard camera views and the landscapes! Quel beau pays I'm happy to let you know that today I won the Tour de Crosstown 2016 which is the cycling competition at Lifetime Crosstown inspired by and concurrent to Le Tour de France. There were about twenty cyclists competing to see who could earn the most points -- by attending cycling class bien s r. I earned the maillot jaune with 23 points and my next closest competitor had 16 points (with the peloton far behind). But that's just part of the story.

Tour de Crosstown 2016
For some time I've been coming to Life Time Fitness at Crosstown for yoga (in Josefina's class) and playing racquetball with my friend David. The cycling studio is right next to the racquetball courts and there's been a class on Saturday's at the same time we usually play. I told David that it looked like fun and he said, having tried it, that it is fun (and a big workout). In early June David got busy and then had an injury that has kept him off the court ever since. So one Saturday morning I decided to try cycling. I borrowed a heart rate monitor (but had no idea what it was for) and tried to bike along in my regular gym shorts, shoes and a t-shirt. Despite being a cycling newbie I was immediately captured by Alison's music and enthusiasm. She's dancing on her bike and you can't help but lock in the beat. Of course that's just after she tells you to dial up the resistance... and the sweat just pours out! I admit that workout hit me pretty hard, but I had to come back and try the 5:45 am Wednesday EDGE cycle class (gulp). Despite what sounds like a crazy impossible time to get out and on a bike it actually works out super well. This plan requires one to up-level one's organization and after the workout I can assure you that you're fully awake and charged for the day! Soon I invested in my own heart rate monitor. Then I realized it would work so much better if I had a metabolic assessment to tune my aerobic and anaerobic training zones. While I signed up for the assessment I decided to work with May as my personal trainer. In addition to helping me with my upper body (complementing the cycling) May is a nutritionist and has helped me dial in this critical facet of training. Even though I'm still working to tune my diet around my workouts, I've already learned a lot by using My Fitness Pal and, most importantly, I have a whole new attitude about food. Pour les curieux, la nutritioniste maison s'est absent e en France pendant le mois de juillet. Soon I would invest in bike shoes, jerseys and shorts and begin to push myself into the proper zones during workouts and fuel my body properly afterwords. All these changes have led to dramatic weight loss \o/ A few of you know that the past two years have involved a lot of personal hardship. Upon reflection I have come to appreciate that things in my life that I can actually control are a massive opportunity. I decided that fixing my exercise and nutrition were the opportunities I want to focus on. A note for for my Debian friends... I'm sorry to have missed you in Cape Town, but I hope to join you in Montr al next year. So when the Tour de Crosstown started in July I decided this was the time for me to get serious. I want to thank all the instructors for the great workouts (and for all the calories I've left on the bike): Alison, Kristine, Olivia, Tasha, and Caroline! The result of my lifestyle changes are hard to describe.. I feel an amazing amount of energy every day. The impact of prior back injury is now almost non-existent. And what range of motion I hadn't recovered from the previous summer's being "washing machined" by a 3 meter wave while body surfing at the beach in Hossegor is now fully working. Now I'm thinking it's time to treat myself to a new bike :) I'm looking at large touring frames and am currently thinking of the Surly Disc Trucker. In terms of bike shops I've had a good experience with One on One and Grand Performance has come highly recommended. If anyone has suggestions for bikes, bike features, or good shops please let me know! I would encourage everyone here in Minneapolis to join me as guest for a Wed morning 5:45am EDGE cycle class. I'm betting you'll have as much fun as a I do.. and I guarantee you will sweat! The challenge in waking up will pay off handsomely in making you energized for the whole day. Let's bike allons-y!

Dirk Eddelbuettel: RcppArmadillo 0.7.200.2.0

armadillo image The second Armadillo release of the 7.* series came out a few weeks ago: version 7.200.2. And RcppArmadillo version 0.7.200.2.0 is now on CRAN and uploaded to Debian. This followed the usual thorough reverse-dependecy checking of by now over 240 packages using it. For once, I let it simmer a little preparing only a package update via the GitHub repo without preparing a CRAN upload to lower the update frequency a little. Seeing that Conrad has started to release 7.300.0 tarballs, the time for a (final) 7.200.2 upload was now right. Just like the previous, it now requires a recent enough compiler. As g++ is so common, we explicitly test for version 4.6 or newer. So if you happen to be on an older RHEL or CentOS release, you may need to get yourself a more modern compiler. R on Windows is now at 4.9.3 which is decent (yet stable) choice; the 4.8 series of g++ will also do. For reference, the current LTS of Ubuntu is at 5.4.0, and we have g++ 6.1 available in Debian testing. This new upstream release adds new indexing helpers, additional return codes on some matrix transformations, increased speed for compound expressions via vectorise, corrects some LAPACK feature detections (affecting principally complex number use under OS X), and a rewritten sample() function thanks to James Balamuta. Armadillo is a powerful and expressive C++ template library for linear algebra aiming towards a good balance between speed and ease of use with a syntax deliberately close to a Matlab. Changes in this release (and the preceding GitHub-only release 0.7.200.1.0 are as follows:
Changes in RcppArmadillo version 0.7.200.2.0 (2016-07-22)
  • Upgraded to Armadillo release 7.200.2
  • The sampling extension was rewritten to use Armadillo vector types instead of Rcpp types (PR #101 by James Balamuta)
Changes in RcppArmadillo version 0.7.200.1.0 (2016-06-06)
  • Upgraded to Armadillo release 7.200.1
    • added .index_min() and .index_max()
    • expanded ind2sub() to handle vectors of indices
    • expanded sub2ind() to handle matrix of subscripts
    • expanded expmat(), logmat() and sqrtmat() to optionally return a bool indicating success
    • faster handling of compound expressions by vectorise()
  • The configure code now (once again) sets the values for the LAPACK feature #define correctly.
Courtesy of CRANberries, there is a diffstat report. More detailed information is on the RcppArmadillo page. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Gregor Herrmann: RC bugs 2016/01-29

seems I've neglected both my blog & my RC bug fixing activities in the last months. anyway, since I still keep track of RC bugs I worked on, I thought I might as well publish the list:

Russ Allbery: Review: The Run of His Life

Review: The Run of His Life, by Jeffrey Toobin
Publisher: Random House
Copyright: 1996, 1997
Printing: 2015
ISBN: 0-307-82916-2
Format: Kindle
Pages: 498
The O.J. Simpson trial needs little introduction to anyone who lived through it in the United States, but a brief summary for those who didn't. O.J. Simpson is a Hall of Fame football player and one of the best running backs to ever play the game. He's also black, which is very relevant much of what later happened. After he retired from professional play, he became a television football commentator and a spokesperson for various companies (particularly Hertz, a car rental business). In 1994, he was arrested for the murder of two people: his ex-wife, Nicole Brown Simpson, and Ron Goldman (a friend of Nicole's). The arrest happened after a bizarre low-speed police chase across Los Angeles in a white Bronco that was broadcast live on network television. The media turned the resulting criminal trial into a reality TV show, with live cable television broadcasts of all of the court proceedings. After nearly a full year of trial (with the jury sequestered for nine months more on that later), a mostly black jury returned a verdict of not guilty after a mere four hours of deliberation. Following the criminal trial, in an extremely unusual legal proceeding, Simpson was found civilly liable for Ron Goldman's death in a lawsuit brought by his family. Bizarre events surrounding the case continued long afterwards. A book titled If I Did It (with "if" in very tiny letters on the cover) was published, ghost-written but allegedly with Simpson's input and cooperation, and was widely considered a confession. Another legal judgment let the Goldman family get all the profits from that book's publication. In an unrelated (but also bizarre) incident in Las Vegas, Simpson was later arrested for kidnapping and armed robbery and is currently in prison until at least 2017. It is almost impossible to have lived through the O.J. Simpson trial in the United States and not have formed some opinion on it. I was in college and without a TV set at the time, and even I watched some of the live trial coverage. Reactions to the trial were extremely racially polarized, as you might have guessed. A lot of black people believed at the time that Simpson was innocent (probably fewer now, given subsequent events). A lot of white people thought he was obviously guilty and was let off by a black jury for racial reasons. My personal opinion, prior to reading this book, was a common "third way" among white liberals: Simpson almost certainly committed the murders, but the racist Los Angeles police department decided to frame him for a crime that he did commit by trying to make the evidence stronger. That's a legitimate reason in the US justice system for finding someone innocent: the state has an obligation to follow correct procedure and treat the defendant fairly in order to get a conviction. I have a strong bias towards trusting juries; frequently, it seems that the media second-guesses the outcome of a case that makes perfect sense as soon as you see all the information the jury had (or didn't have). Toobin's book changed my mind. Perhaps because I wasn't watching all of the coverage, I was greatly underestimating the level of incompetence and bad decision-making by everyone involved: the prosecution, the defense, the police, the jury, and the judge. This court case was a disaster from start to finish; no one involved comes away looking good. Simpson was clearly guilty given the evidence presented, but the case was so badly mishandled that it gave the jury room to reach the wrong verdict. (It's telling that, in the far better managed subsequent civil case, the jury had no trouble reaching a guilty verdict.) The Run of His Life is a very detailed examination of the entire Simpson case, from the night of the murder through the end of the trial and (in an epilogue) the civil case. Toobin was himself involved in the media firestorm, breaking some early news of the defense's decision to focus on race in The New Yorker and then involved throughout the trial as a legal analyst, and he makes it clear when he becomes part of the story. But despite that, this book felt objective to me. There are tons of direct quotes, lots of clear description of the evidence, underlying interviews with many of the people involved to source statements in the book, and a comprehensive approach to the facts. I think Toobin is a bit baffled by the black reaction to the case, and that felt like a gap in the comprehensiveness and the one place where he might be accused of falling back on stereotypes and easy judgments. But other than hole, Toobin applies his criticism even-handedly and devastatingly to all parties. I won't go into all the details of how Toobin changed my mind. It was a cumulative effect across the whole book, and if you're curious, I do recommend reading it. A lot was the detailed discussion of the forensic evidence, which was undermined for the jury at trial but looks very solid outside the hothouse of the case. But there is one critical piece that I would hope would be handled differently today, twenty years later, than it was by the prosecutors in that case: Simpson's history of domestic violence against Nicole. With what we now know about patterns of domestic abuse, the escalation to murder looks entirely unsurprising. And that history of domestic abuse was exceedingly well-documented: multiple external witnesses, police reports, and one actual prior conviction for spousal abuse (for which Simpson did "community service" that was basically a joke). The prosecution did a very poor job of establishing this history and the jury discounted it. That was a huge mistake by both parties. I'll mention one other critical collection of facts that Toobin explains well and that contradicted my previous impression of the case: the relationship between Simpson and the police. Today, in the era of Black Lives Matter, the routine abuse of black Americans by the police is more widely known. At the time of the murders, it was less recognized among white Americans, although black Americans certainly knew about it. But even in 1994, the Los Angeles police department was notorious as one of the most corrupt and racist big-city police departments in the United States. This is the police department that beat Rodney King. Mark Fuhrman, one of the police officers involved in the case (although not that significantly, despite his role at the trial), was clearly racist and had no business being a police officer. It was therefore entirely believable that these people would have decided to frame a black man for a murder he actually committed. What Toobin argues, quite persuasively and with quite a lot of evidence, is that this analysis may make sense given the racial tensions in Los Angeles but ignores another critical characteristic of Los Angeles politics, namely a deference to celebrity. Prior to this trial, O.J. Simpson largely followed the path of many black athletes who become broadly popular in white America: underplaying race and focusing on his personal celebrity and connections. (Toobin records a quote from Simpson earlier in his life that perfectly captures this approach: "I'm not black, I'm O.J.") Simpson spent more time with white businessmen than the black inhabitants of central Los Angeles. And, more to the point, the police treated him as a celebrity, not as a black man. Toobin takes some time to chronicle the remarkable history of deference and familiarity that the police showed Simpson. He regularly invited police officers to his house for parties. The police had a long history of largely ignoring or downplaying his abuse of his wife, including not arresting him in situations that clearly seemed to call for that, showing a remarkable amount of deference to his side of the story, not pursuing clear violations of the court judgment after his one conviction for spousal abuse, and never showing much inclination to believe or protect Nicole. Even on the night of the murder, they started following a standard playbook for giving a celebrity advance warning of investigations that might involve them before the news media found out about them. It seems clear, given the evidence that Toobin collected, that the racist Los Angeles police didn't focus that animus at Simpson, a wealthy celebrity living in Brentwood. He wasn't a black man in their eyes; he was a rich Hall of Fame football player and a friend. This obviously raises the question of how the jury could return an innocent verdict. Toobin provides plenty of material to analyze that question from multiple angles in his detailed account of the case, but I can tell you my conclusion: Judge Lance Ito did a horrifically incompetent job of managing the case. He let the lawyers wander all over the case, interacted bizarrely with the media coverage (and was part of letting the media turn it into a daytime drama), was not crisp or clear about his standards of evidence and admissibility, and, perhaps worst of all, let the case meander on at incredible length. With a fully sequestered jury allowed only brief conjugal visits and no media contact (not even bookstore shopping!). Quite a lot of anger was focused on the jury after the acquittal, and I do think they reached the wrong conclusion and had all the information they would have needed to reach the correct one. But Toobin touches on something that I think would be very hard to comprehend without having lived through it. The jury and alternate pool essentially lived in prison for nine months, with guards and very strict rules about contact with the outside world, in a country where compensation for jury duty is almost nonexistent. There were a lot of other factors behind their decision, including racial tensions and the sheer pressure from judging a celebrity case about which everyone has an opinion, but I think it's nearly impossible to underestimate the psychological tension and stress from being locked up with random other people under armed guard for three quarters of a year. It's hard for jury members to do an exhaustive and careful deliberation in a typical trial that takes a week and doesn't involve sequestration. People want to get back to their lives and families. I can only imagine the state I would be in after nine months of this, or how poor psychological shape I would be in to make a careful and considered decision. Similarly, for those who condemned the jury for profiting via books and media appearances after the trial, the current compensation for jurors is $15 per day (not hour). I believe at the time it was around $5 per day. There are a few employers who will pay full salary for the entire jury service, but only a few; many cap the length at a few weeks, and some employers treat all jury duty as unpaid leave. The only legal requirement for employers in the United States is that employees that serve on a jury have their job held for them to return to, but compensation is pathetic, not tied to minimum wage, and employers do not have to supplement it. I'm much less inclined to blame the jurors than the system that badly mistreated them. As you can probably tell from the length of this review, I found The Run of His Life fascinating. If I had followed the whole saga more closely at the time, this may have been old material, but I think my vague impressions and patchwork of assumptions were more typical than not among people who were around for the trial but didn't invest a lot of effort into following it. If you are like me, and you have any interest in the case or the details of how the US criminal justice system works, this is a fascinating case study, and Toobin does a great job with it. Recommended. Rating: 8 out of 10

22 July 2016

Norbert Preining: Yukio Mishima The Temple of the Golden Pavilion

A masterpiece of modern Japanese literature: Yukio Mishima ( ) The Temple of the Golden Pavilion ( ). The fictional story about the very real arson attack that destroyed the Golden Pavilion in 1950.
mishima-golden-temple A bit different treatise on beauty and ugliness!
How shall I put it? Beauty yes, beauty is like a decayed tooth. It rubs against one s tongue, it hangs there, hurting one, insisting on its own existence, finally [ ] the tooth extracted. Then, as one looks at the small, dirty, brown, blood-stained tooth lying in one s hand, one s thoughts are likely to be as follows: Is this it?
Mizoguchi, a stutterer, is from young years on taken by a near mystical imagination of the Golden Pavilion, influence by his father who considers it the most beautiful object in the world. After his father s death he moves to Kyoto and becomes acolyte in the temple. He develops a friendship with Kashiwagi, who uses his clubfeet to make women feel sorry for him and make them fall in love with his clubfeet, as he puts it. Kashiwagi also puts Mizoguchi onto the first tracks of amorous experiences, but Mizoguchi invariably turns out to be impotent but not due to his stuttering, but due to the image of the Golden Pavilion appearing in the essential moment and destroying every chance.
Yes, this was really the coast of the Sea of Japan! Here was the source of all my unhappiness, of all my gloomy thoughts, the origin of all my ugliness and all my strength. It was a wild sea.
Mizoguchi is getting more and more mentally about his relation with the head monk, neglects his studies, and after a stark reprimand he escapes to the north coast, from where he is brought back by police to the temple. He decides to burn down the Golden Pavilion, which has taken more and more command of his thinking and doing. He carries out the deed with the aim to burn himself in the top floor, but escapes in the last second to retreat into the hills to watch the spectacle.
Closely based on the true story of the arsonist of the Golden Pavilion, whom Mishima even visited in prison, the book is a treatise about beauty and ugly.
At his trial he [the real arsonist] said: I hate myself, my evil, ugly, stammering self. Yet he also said that he did not in any way regret having burned down the Kinkakuji.
Nancy Wilson Ross in the preface
Mishima is master in showing these two extremes by contrasting the refined qualities of Japanese culture flower arrangement, playing the shakuhachi, with immediate outburst of contrasting behavior: cold and brutal recklessness. Take for example the scene were Kashiwagi is arranging flowers, stolen by Mizoguchi from the temple grounds, while Mizoguchi is playing the flute. They also discuss koans and various interpretations. Enters the Ikebana teacher, and mistress of Kashiwagi. She congratulates Kashiwagi to his excellent arrangement, which he answers coldly by quitting their relationship, both as teacher as well as mistress, and telling her not to see him again in a formal style. She, still ceremonially kneeling, suddenly destroys the flower arrangement, only to be beaten and thrown out by Kashiwagi. And the beauty and harmony has turned to ugliness and hate in seconds. Beauty and Ugliness, two sides of the same medal, or inherently the same, because it is only up to the point of view. Mishima ingeniously plays with this duality, and leads us through the slow and painful development of Mizoguchi to the bitter end, which finally gives him freedom, freedom from the force of beauty. Sometimes seeing how our society is obsessed with beauty I cannot get rid of the feeling that there are far more Mizoguchis at heart.

Dirk Eddelbuettel: RcppCCTZ 0.0.5

Version 0.0.5 of RcppCCTZ arrived on CRAN a couple of days ago. It reflects an upstream fixed made a few weeks ago. CRAN tests revealed that g++-6 was tripping over one missing #define; this was added upstream and I subsequently synchronized with upstream. At the same time the set of examples was extended (see below). Somehow useR! 2016 got in the way and while working on the then-incomplete examples during the traveling I forgot to release this until CRAN reminded me that their tests still failed. I promptly prepared the 0.0.5 release but somehow failed to update NEWS files etc. They are correct in the repo but not in the shipped package. Oh well. CCTZ is a C++ library for translating between absolute and civil times using the rules of a time zone. In fact, it is two libraries. One for dealing with civil time: human-readable dates and times, and one for converting between between absolute and civil times via time zones. It requires only a proper C++11 compiler and the standard IANA time zone data base which standard Unix, Linux, OS X, ... computers tend to have in /usr/share/zoneinfo. RcppCCTZ connects this library to R by relying on Rcpp. Two good examples are now included, and shown here. The first one tabulates the time difference between New York and London (at a weekly level for compactness):
R> example(tzDiff)
tzDiffR> # simple call: difference now
tzDiffR> tzDiff("America/New_York", "Europe/London", Sys.time())
[1] 5
tzDiffR> # tabulate difference for every week of the year
tzDiffR> table(sapply(0:52, function(d) tzDiff("America/New_York", "Europe/London",
tzDiff+                                       as.POSIXct(as.Date("2016-01-01") + d*7))))
 4  5 
 3 50 
R> 
Because the two continents happen to spring forward and fall backwards between regular and daylight savings times, there are, respectively, two and one week periods where the difference is one hour less than usual. A second example shifts the time to a different time zone:
R> example(toTz)
toTzR> toTz(Sys.time(), "America/New_York", "Europe/London")
[1] "2016-07-14 10:28:39.91740 CDT"
R> 
Note that because we return a POSIXct object, it is printed by R with the default (local) TZ attribute (for "America/Chicago" in my case). A more direct example asks what time it is in my time zone when it is midnight in Tokyo:
R> toTz(ISOdatetime(2016,7,15,0,0,0), "Japan", "America/Chicago")
[1] "2016-07-14 15:00:00 CDT"
R>
More changes will come in 0.0.6 as soon as I find time to translate the nice time_tool (command-line) example into an R function. Changes in this version are summarized here:
Changes in version 0.0.5 (2016-07-09)
  • New utility example functions toTz() and tzDiff
  • Synchronized with small upstream change for additional #ifdef for compiler differentiation
We also have a diff to the previous version thanks to CRANberries. More details are at the RcppCCTZ page; code, issue tickets etc at the GitHub repository.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

21 July 2016

Reproducible builds folks: Reproducible builds: week 62 in Stretch cycle

What happened in the Reproducible Builds effort between June 26th and July 2nd 2016: Read on to find out why we're lagging some weeks behind ! GSoC and Outreachy updates Toolchain fixes With the doxygen upload we are now down to only 2 modified packages in our repository: dpkg and rdfind. Weekly reports delay and the future of statistics To catch up with our backlog of weekly reports we have decided to skip some of the statistics for this week. We might publish them in a future report, or we might switch to a format where we summarize them more (and which we can create (even) more automatically), we'll see. We are doing these weekly statistics because we believe it's appropriate and useful to credit people's work and make it more visible. What do you think? We would love to hear your thoughts on this matter! Do you read these statistics? Somewhat? Actually, thanks to the power of notmuch, Holger came up with what you can see below, so what's missing for this week are the uploads fixing irreprodubilities. Which we really would like to show for the reasons stated above and because we really really need these uploads to happen ;-) But then we also like to confirm the bugs are really gone, which (atm) requires manual checking, and to look for the words "reproducible" and "deterministic" (and spelling variations) in debian/changelogs of all uploads, to spot reproducible work not tracked via the BTS. And we still need to catch up on the backlog of weekly reports. Bugs submitted with reproducible usertags It seems DebCamp in Cape Town was hugely successful and made some people get a lot of work done: 61 bugs have been filed with reproducible builds usertags and 60 of them had patches: Package reviews 437 new reviews have been added (though most of them were just linking the bug, "only" 56 new issues in packages were found), an unknown number has been been updated and 60 have been removed in this week, adding to our knowledge about identified issues. 4 new issue types have been found: Weekly QA work 98 FTBFS bugs have been reported by Chris Lamb and Santiago Vila. diffoscope development strip-nondeterminism development tests.reproducible-builds.org Misc. This week's edition was written by Mattia Rizzolo, Reiner Herrmann, Ceridwen and Holger Levsen and reviewed by a bunch of Reproducible builds folks on IRC.

20 July 2016

Shirish Agarwal: Debconf 16 and My Experience with Debian

It has been often said that you should continually try new things in life so that a. Unlike the fish you do not mistake the pond to be the sea. b. You see other people, other types and ways of living and being which you normally won t in your day-to-day existence. With both of those as mantras I decided to take a leap into the unknown. I was unsure both about the visa process as well as the travel bit as I was traveling to an unknown place and although I had done some research about the place I was unsure about the authenticity of whatever is/was shared on the web. During the whole journey both to and fro, I couldn t sleep a wink. The Doha airport is huge. There are 5 Concourses, A, B , C, D, E and around 30+ gates in each Concourse. The ambition of the small state is something to be reckoned with. Almost 95% of the blue workers in the entire airport were of Asian sub-continent. While the Qatari Rial is 19 times stronger to us, the workers I suspect are worse-off than people doing similar things back home. Add to that the sharia law, even for all the money in the world, I wouldn t want to settle therein. Anyways, during the journey, a small surprise awaited me, Ritesh Raj Saraff, a DD was also traveling to Debconf. We bumped into each other while going to see the Doha City, courtesy Al-Hamad International Airport. I would probably share a bit more about Doha and my experiences with the city in upcoming posts. Cut to Cape Town, South Africa, we landed in the city half an hour after our scheduled time and then we sped along to University of Cape Town (UCT) which was to become our home for the next 13 odd days. The first few days were a whirlwind as there were new people to meet, old people whom I knew only as an e-mail id or an IRC nickname turned out to be real people and you have to try to articulate yourself in English, which is not a native language of mine. During Debcamp I was fortunate to be able visit some of the places and the wiki page had a lot of places which I knew I wouldn t be able to complete unless I had 15 days unlimited time and money to go around so didn t even try. I had gone with few goals in mind :- a. Do some documentation of the event In this I failed completely as just the walk from the venue to where the talks were energy-draining for me. Apart from that, you get swept in meeting new people and talking about one of million topics in Debian which interest you or the other person and while they are fulfilling, it is and was both physically and emotionally draining for me (in a good way). Bernelle (one of the organizers) had warned us of this phenomenon but you disregard it as you know you have a limited time-frame in which to meet and greet people and it is all a over-whelming experience. b. Another goal was to meet my Indian brethren who had left the country around 60~100 years mostly as slaves of East India company In this I was partially successful. I met a couple of beautiful ladies who had either a father or a mother who was Indian while the other was of African heritage. It seemed in them a yearning to know the culture but from what little they had, only Bollywood and Indian cuisine was what they could make of Indian culture. One of the girls, ummm women to be more truer, shared a somewhat grim tale. She had both an African boyfriend as well as Indian boyfriend in her life and in both cases, she was rejected by the boy s parents because she wasn t pure enough. This was deja vu all over again as the same thing can be seen here happening in casteism so there wasn t any advice I could give but just nod in empathy. What was sort of relevation was when their parents or grandparents came, the name and surnames were thrown off and the surname was just the place from where they belong. From the discussions it emerged that there were also lot of cases of forced conversions to Christianity during that era as well as temptations of a better life. As shared, this goal succeeded partially, as I was actually interested in their parents or grand-parents to know the events that shaped the Indian diaspora over there. While the children know only of today, yester-years could only be known by those people who made the unwilling perilous journey to Africa. I had also wanted to know more about Gandhiji s role in that era but alas, that part of history would have to wait for another day as I guess, both those goals would only have met had I visited Durban but that was not to be. I had applied for one talk My Experience with Debian and one workshop for Installation of Debian on systems. The My Experience with Debian was aimed at newbies and I had thought of using show-and-tell to share the differences between proprietary Operating Systems and a FOSS distribution such as Debian. I was going to take simple things such as changelogs, apt-listbugs, real-time knowledge of updates and upgrades as well as /etc/apt/sources.list to share both the versatility of the Debian desktop and real improvements than what proprietary Operating Systems had to offer. But I found myself engaging with Debian Developers (DD s) rather than the newbies so had to change the orientation and fundamentals of the talk on the fly. I knew or suspected rather that the old idea would not work as it would just be repeating to the choir. With that in the back of mind, and the idea that perhaps they would not be so aware of the politics and events which happened in India over the last couple of decades, I tried to share what little I was able to recollect what little I was able to remember about those times. Apart from that, I was also highly conscious that I had been given just the before lunch slot aka You are in the way of my lunch slot. So I knew I had to speak my piece as quickly as possible being as clear as can be. Later, I did get feedback that I was fast and seeing it through couple of times, do agree that I could have done a better job. What s done is done and the only thing I could do to salvage it a bit is to make a presentation which I am sharing as below. my_experience_with_debian Would be nice if somebody could come up with a lighter template for presentations. For reference the template I have taken it from is shared at https://wiki.debian.org/Presentations . Some pictures from the presentation. vlcsnap-00004 me_sharing my_experience_with_debian You can find the video at http://meetings-archive.debian.net/pub/debian-meetings/2016/debconf16/My_Experience_with_Debian.webm This is by no means the end of the Debconf16 experience, but actually the starting. I hope to share more of my thoughts, ideas and get as much feedback from all the wonderful people I met during Debconf.
Filed under: Miscellenous Tagged: #Debconf16, Doha, My talk, Qatar

Steinar H. Gunderson: Solskogen 2016 videos

I just published the videos from Solskogen 2016 on Youtube; you can find them all in this playlist. The are basically exactly what was being sent out on the live stream, frame for frame, except that the audio for the live shader compos has been remastered, and of course a lot of dead time has been cut out (the stream was sending over several days, but most of the time, only the information loop from the bigscreen). YouTube doesn't really support the variable 50/60 Hz frame rate we've been using well as far as I can tell, but mostly it seems to go to some 60 Hz upconversion, which is okay enough, because the rest of your setup most likely isn't free-framerate anyway. Solskogen is interesting in that we're trying to do a high-quality stream with essentially zero money allocated to it; where something like Debconf can use 2500 for renting and transporting equipment (granted, for two or three rooms and not our single stream), we're largely dependent on personal equipment as well as borrowing things here and there. (I think we borrowed stuff from more or less ten distinct places.) Furthermore, we're nowhere near the situation of two cameras, a laptop, perhaps a few microphones ; not only do you expect to run full 1080p60 to the bigscreen and switch between that and information slides for each production, but an Amiga 500 doesn't really have an HDMI port, and Commodore 64 delivers an infamously broken 50.12 Hz signal that you really need to deal with carefully if you want it to not look like crap. These two factors together lead to a rather eclectic setup; here, visualized beautifully from my ASCII art by ditaa: Solskogen 2016 A/V setup diagram Of course, for me, the really interesting part here is near the end of the chain, with Nageru, my live video mixer, doing the stream mixing and encoding. (There's also Cubemap, the video reflector, but honestly, I never worry about that anymore. Serving 150 simultaneous clients is just not something to write home about anymore; the only adjustment I would want to make would probably be some WebSockets support to be able to deal with iOS without having to use a secondary HLS stream.) Of course, to make things even more complicated, the live shader compo needs two different inputs (the two coders' laptops) live on the bigscreen, which was done with two video capture cards, text chroma-keyed on top from Chroma, and OBS, because the guy controlling the bigscreen has different preferences from me. I would take his screen in as a dirty feed and then put my own stuff around it, like this: Solskogen 2016 shader compo screenshot (Unfortunately, I forgot to take a screenshot of Nageru itself during this run.) Solskogen was the first time I'd really used Nageru in production, and despite super-extensive testing, there's always something that can go wrong. And indeed there was: First of all, we discovered that the local Internet line was reduced from 30/10 to 5/0.5 (which is, frankly, unusable for streaming video), and after we'd half-way fixed that (we got it to 25/4 or so by prodding the ISP, of which we could reserve about 2 for video demoscene content is really hard to encode, so I'd prefer a lot more) Nageru started crashing. It wasn't even crashes I understood anything of. Generally it seemed like the NVIDIA drivers were returning GL_OUT_OF_MEMORY on things like creating mipmaps; it's logical that they'd be allocating memory, but we had 6 GB of GPU memory and 16 GB of CPU memory, and lots of it was free. (The PC we used for encoding was much, much faster than what you need to run Nageru smoothly, so we had plenty of CPU power left to run x264 in, although you can of course always want more.) It seemed to be mostly related to zoom transitions, so I generally avoided those and ran that night's compos in a more static fashion. It wasn't until later that night (or morning, if you will) that I actually understood the bug (through the godsend of the NVX_gpu_memory_info extension, which gave me enough information about the GPU memory state that I understood I wasn't leaking GPU memory at all); I had set Nageru to lock all of its memory used in RAM, so that it would never ever get swapped out and lose frames for that reason. I had set the limit for lockable RAM based on my test setup, with 4 GB of RAM, but this setup had much more RAM, a 1080p60 input (which uses more RAM, of course) and a second camera, all of which I hadn't been able to test before, since I simply didn't have the hardware available. So I wasn't hitting the available RAM, but I was hitting the amount of RAM that Linux was willing to lock into memory for me, and at that point, it'd rather return errors on memory allocations (including the allocations the driver needed to make for its texture memory backings) than to violate the never swap contract. Once I fixed this (by simply increasing the amount of lockable memory in limits.conf), everything was rock-stable, just like it should be, and I could turn my attention to the actual production. Often during compos, I don't really need the mixing power of Nageru (it just shows a single input, albeit scaled using high-quality Lanczos3 scaling on the GPU to get it down from 1080p60 to 720p60), but since entries come in using different sound levels (I wanted the stream to conform to EBU R128, which it generally did) and different platforms expect different audio work (e.g., you wouldn't put a compressor on an MP3 track that was already mastered, but we did that on e.g. SID tracks since they have nearly zero ability to control the overall volume), there was a fair bit of manual audio tweaking during some of the compos. That, and of course, the live 50/60 Hz switches were a lot of fun: If an Amiga entry was coming up, we'd 1. fade to a camera, 2. fade in an overlay saying we were switching to 50 Hz so have patience, 3. set the camera as master clock (because the bigscreen's clock is going to go away soon), 4. change the scaler from 60 Hz to 50 Hz (takes two clicks and a bit of waiting), 5. change the scaler input in Nageru from 1080p60 to 1080p50, 6. steps 3,2,1 in reverse. Next time, I'll try to make that slightly smoother, especially as the lack of audio during the switch (it comes in on the bigscreen SDI feed) tended to confuse viewers. So, well, that was a lot of fun, and it certainly validated that you can do a pretty complicated real-life stream with Nageru. I have a long list of small tweaks I want to make, though; nothing beats actual experience when it comes to improving processes. :-)

Daniel Stender: Theano in Debian: maintenance, BLAS and CUDA

I'm glad to announce that we have the current release of Theano (0.8.2) in Debian unstable now, it's on its way into the testing branch and the Debian derivatives, heading for Debian 9. The Debian package is maintained in behalf of the Debian Science Team. We have a binary package with the modules in the Python 2.7 import path (python-theano), if you want or need to stick to that branch a little longer (as a matter of fact, in the current popcon stats it's the most installed package), and a package running on the default Python 3 version (python3-theano). The comprehensive documentation is available for offline usage in another binary package (theano-doc). Although Theano builds its extensions on run time and therefore all binary packages contain the same code, the source package generates arch specific packages1 for the reason that the exhaustive test suite could run over all the architectures to detect if there are problems somewhere (#824116). what's this? In a nutshell, Theano is a computer algebra system (CAS) and expression compiler, which is implemented in Python as a library. It is named after a Classical Greek female mathematician and it's developed at the LISA lab (located at MILA, the Montreal Institute for Learning Algorithms) at the Universit de Montr al. Theano tightly integrates multi-dimensional arrays (N-dimensional, ND-array) from NumPy (numpy.ndarray), which are broadly used in Scientific Python for the representation of numeric data. It features a declarative Python based language with symbolic operations for the functional definition of mathematical expressions, which allows to create functions that compute values for them. Internally the expressions are represented as directed graphs with nodes for variables and operations. The internal compiler then optimizes those graphs for stability and speed and then generates high-performance native machine code to evaluate resp. compute these mathematical expressions2. One of the main features of Theano is that it's capable to compute also on GPU processors (graphical processor unit), like on custom graphic cards (e.g. the developers are using a GeForce GTX Titan X for benchmarks). Today's GPUs became very powerful parallel floating point devices which can be employed also for scientific computations instead of 3D video games3. The acronym "GPGPU" (general purpose graphical processor unit) refers to special cards like NVIDIA's Tesla4, which could be used alike (more on that below). Thus, Theano is a high-performance number cruncher with an own computing engine which could be used for large-scale scientific computations. If you haven't came across Theano as a Pythonistic professional mathematician, it's also one of the most prevalent frameworks for implementing deep learning applications (training multi-layered, "deep" artificial neural networks, DNN) around5, and has been developed with a focus on machine learning from the ground up. There are several higher level user interfaces build in the top of Theano (for DNN, Keras, Lasagne, Blocks, and others, or for Python probalistic programming, PyMC3). I'll seek for some of them also becoming available in Debian, too. helper scripts Both binary packages ship three convenience scripts, theano-cache, theano-test, and theano-nose. Instead of them being copied into /usr/bin, which would result into a binaries-have-conflict violation, the scripts are to be found in /usr/share/python-theano (python3-theano respectively), so that both module packages of Theano can be installed at the same time. The scripts could be run directly from these folders, e.g. do $ python /usr/share/python-theano/theano-nose to achieve that. If you're going to heavy use them, you could add the directory of the flavour you prefer (Python 2 or Python 3) to the $PATH environment variable manually by either typing e.g. $ export PATH=/usr/share/python-theano:$PATH on the prompt, or save that line into ~/.bashrc. Manpages aren't available for these little helper scripts6, but you could always get info on what they do and which arguments they accept by invoking them with the -h (for theano-nose) resp. help flag (for theano-cache). running the tests On some occasions you might want to run the testsuite of the installed library, like to check over if everything runs fine on your GPU hardware. There are two different ways to run the tests (anyway you need to have python ,3 -nose installed). One is, you could launch the test suite by doing $ python -c 'import theano; theano.test() (or the same with python3 to test the other flavour), that's the same what the helper script theano-test does. However, by doing it that way some particular tests might fail by raising errors also for the group of known failures. Known failures are excluded from being errors if you run the tests by theano-nose, which is a wrapper around nosetests, so this might be always the better choice. You can run this convenience script with the option --theano on the installed library, or from the source package root, which you could pull by $ sudo apt-get source theano (there you have also the option to use bin/theano-nose). The script accept options for nosetests, so you might run it with -v to increase verbosity. For the tests the configuration switch config.device must be set to cpu. This will also include the GPU tests when a proper accessible device is detected, so that's a little misleading in the sense of it doesn't mean "run everything on the CPU". You're on the safe side if you run it always like this: $ THEANO_FLAGS=device=cpu theano-nose, if you've set config.device to gpu in your ~/.theanorc. Depending on the available hardware and the used BLAS implementation (see below) it could take quite a long time to run the whole test suite through, on the Core-i5 in my laptop that takes around an hour even excluded the GPU related tests (which perform pretty fast, though). Theano features a couple of switches to manipulate the default configuration for optimization and compilation. There is a rivalry between optimization and compilation costs against performance of the test suite, and it turned out the test suite performs a quicker with lesser graph optimization. There are two different switches available to control config.optimizer, the fast_run toggles maximal optimization, while fast_compile runs only a minimal set of graph optimization features. These settings are used by the general mode switches for config.mode, which is either FAST_RUN by default, or FAST_COMPILE. The default mode FAST_RUN (optimizer=fast_run, linker=cvm) needs around 72 minutes on my lower mid-level machine (on un-optimized BLAS). To set mode=FAST_COMPILE (optimizer=fast_compile, linker=py) brings some boost for the performance of the test suite because it runs the whole suite in 46 minutes. The downside of that is that C code compilation is disabled in this mode by using the linker py, and also the GPU related tests are not included. I've played around with using the optimizer fast_compile with some of the other linkers (c py and cvm, and their versions without garbage collection) as alternative to FAST_COMPILE with minimal optimization but also machine code compilation incl. GPU testing. But to my experience, fast_compile without another than the linker py results in some new errors and failures of some tests on amd64, and this might the case also on other architectures, too. By the way, another useful feature is DebugMode for config.mode, which verifies the correctness of all optimizations and compares the C to Python results. If you want to have detailed info on the configuration settings of Theano, do $ python -c 'import theano; print theano.config' less, and check out the chapter config in the library documentation in the documentation. cache maintenance Theano isn't a JIT (just-in-time) compiler like Numba, which generates native machine code in the memory and executes it immediately, but it saves the generated native machine code into compiledirs. The reason for doing it that way is quite practical like the docs explain, the persistent cache on disk makes it possible to avoid generating code for the same operation, and to avoid compiling again when different operations generate the same code. The compiledirs by default are located within $(HOME)/.theano/. After some time the folder becomes quite large, and might look something like this:
$ ls ~/.theano
compiledir_Linux-4.5--amd64-x86_64-with-debian-stretch-sid--2.7.11+-64
compiledir_Linux-4.5--amd64-x86_64-with-debian-stretch-sid--2.7.12-64
compiledir_Linux-4.5--amd64-x86_64-with-debian-stretch-sid--2.7.12rc1-64
compiledir_Linux-4.5--amd64-x86_64-with-debian-stretch-sid--3.5.1+-64
compiledir_Linux-4.5--amd64-x86_64-with-debian-stretch-sid--3.5.2-64
compiledir_Linux-4.5--amd64-x86_64-with-debian-stretch-sid--3.5.2rc1-64
If the used Python version changed like in this example you might to want to purge obsolete cache. For working with the cache resp. the compiledirs, the helper theano-cache comes in handy. If you invoke it without any arguments the current cache location is put out like ~/.theano/compiledir_Linux-4.5--amd64-x86_64-with-debian-stretch-sid--2.7.12-64 (the script is run from /usr/share/python-theano). So, the compiledirs for the old Python versions in this example (11+ and 12rc1) can be removed to free the space they occupy. All compiledirs resp. cache directories meaning the whole cache could be erased by $ theano-cache basecompiledir purge, the effect is the same as by performing $ rm -rf ~/.theano. You might want to do that e.g. if you're using different hardware, like when you got yourself another graphics card. Or habitual from time to time when the compiledirs fill up so much that it slows down processing with the harddisk being very busy all the time, if you don't have an SSD drive available. For example, the disk space of build chroots carrying (mainly) the tests completely compiled through on default Python 2 and Python 3 consumes around 1.3 GB (see here). BLAS implementations Theano needs a level 3 implementation of BLAS (Basic Linear Algebra Subprograms) for operations between vectors (one-dimensional mathematical objects) and matrices (two-dimensional objects) carried out on the CPU. NumPy is already build on BLAS and pulls the standard implementation (libblas3, soure package: lapack), but Theano links directly to it instead of using NumPy as intermediate layer to reduce the computational overhead. For this, Theano needs development headers and the binary packages pull libblas-dev by default, if any other development package of another BLAS implementation (like OpenBLAS or ATLAS) isn't already installed, or pulled with them (providing the virtual package libblas.so). The linker flags could be manipulated directly through the configuration switch config.blas.ldflags, which is by default set to -L/usr/lib -lblas -lblas. By the way, if you set it to an empty value, Theano falls back to using BLAS through NumPy, if you want to have that for some reason. On Debian, there is a very convenient way to switch between BLAS implementations by the alternatives mechanism. If you have several alternative implementations installed at the same time, you can switch from one to another easily by just doing:
$ sudo update-alternatives --config libblas.so
There are 3 choices for the alternative libblas.so (providing /usr/lib/libblas.so).
  Selection    Path                                  Priority   Status
------------------------------------------------------------
* 0            /usr/lib/openblas-base/libblas.so      40        auto mode
  1            /usr/lib/atlas-base/atlas/libblas.so   35        manual mode
  2            /usr/lib/libblas/libblas.so            10        manual mode
  3            /usr/lib/openblas-base/libblas.so      40        manual mode
Press <enter> to keep the current choice[*], or type selection number:
The implementations are performing differently on different hardware, so you might want to take the time to compare which one does it best on your processor (the other packages are libatlas-base-dev and libopenblas-dev), and choose that to optimize your system. If you want to squeeze out all which is in there for carrying out Theano's computations on the CPU, another option is to compile an optimized version of a BLAS library especially for your processor. I'm going to write another blog posting on this issue. The binary packages of Theano ship the script check_blas.py to check over how well a BLAS implementation performs with it, and if everything works right. That script is located in the misc subfolder of the library, you could locate it by doing $ dpkg -L python-theano grep check_blas (or for the package python3-theano accordingly), and run it with the Python interpreter. By default the scripts puts out a lot of info like a huge perfomance comparison reference table, the current setting of blas.ldflags, the compiledir, the setting of floatX, OS information, the GCC version, the current NumPy config towards BLAS, NumPy location and version, if Theano linked directly or has used the NumPy binding, and finally and most important, the execution time. If just the execution time for quick perfomance comparisons is needed this script could be invoked with -q. Theano on CUDA The function compiler of Theano works with alternative backends to carry out the computations, like the ones for graphics cards. Currently, there are two different backends for GPU processing available, one docks onto NVIDIA's CUDA (Compute Unified Device Architecture) technology7, and another one for libgpuarray, which is also developed by the Theano developers in parallel. The libgpuarray library is an interesting alternative for Theano, it's a GPU tensor (multi-dimensional mathematical object) array written in C with Python bindings based on Cython, which has the advantage of running also on OpenCL8. OpenCL, unlike CUDA9, is full free software, vendor neutral and overcomes the limitation of the CUDA toolkit being only available for amd64 and the ppc64el port (see here). I've opened an ITP on libgpuarray and we'll see if and how this works out. Another reason for it would be great to have it available is that it looks like CUDA currently runs into problems with GCC 610. More on that, soon. Here's a litle checklist for setting up your CUDA device so that you don't have to experience something like this:
$ THEANO_FLAGS=device=gpu,floatX=float32 python ./cat_dog_classifier.py 
WARNING (theano.sandbox.cuda): CUDA is installed, but device gpu is not available (error: Unable to get the number of gpus available: no CUDA-capable device is detected)
hardware check For running Theano on CUDA you need an NVIDIA graphics card which is capable of doing that. You can recheck if your device is supported by CUDA here. When the hardware isn't too old (CUDA support started with GeForce 8 and Quadro X series) or too strange I think it isn't working only in exceptional cases. You can check your model and if the device is present in the system on the bare hardware level by doing this:
$ lspci   grep -i nvidia
04:00.0 3D controller: NVIDIA Corporation GM108M [GeForce 940M] (rev a2)
If a line like this doesn't get returned, your device most probably is broken, or not properly connected (ouch). If rev ff appears at the end of the line that means the device is off meaning powered down. This might be happening if you have a laptop with Optimus graphics hardware, and the related drivers have switched off the unoccupied device to safe energy11. kernel module Running CUDA applications requires the proprietary NVIDIA driver kernel module to be loaded into the kernel and working. If you haven't already installed it for another purpose, the NVIDIA driver and the CUDA toolkit are both in the non-free section of the Debian archive, which is not enabled by default. To get non-free packages you have to add non-free (and it's better to do so, also contrib) to your package source in /etc/apt/sources.list, which might then look like this:
deb http://httpredir.debian.org/debian/ testing main contrib non-free
After doing that, perform $ apt-cache update to update the package lists, and there you go with the non-free packages. The headers of the running kernel are needed to compile modules, you can get them together with the NVIDIA kernel module package by running:
$ sudo apt-get install linux-headers-$(uname -r) nvidia-kernel-dkms build-essential
DKMS will then build the NVIDIA module for the kernel and does some other things on the system. When the installation has finished, it's generally advised to reboot the system completely. troubleshooting If you have problems with the CUDA device, it's advised to verify if the following things concerning the NVIDIA driver resp. kernel module are in order: blacklist nouveau Check if the default Nouveau kernel module driver (which blocks the NVIDIA module) for some reason still gets loaded by doing $ lsmod grep nouveau. If nothing gets returned, that's right. If it's still in the kernel, just add blacklist nouveau to /etc/modprobe.d/blacklist.conf, and update the booting ramdisk with sudo update-initramfs -u afterwards. Then reboot once more, this shouldn't be the case then anymore. rebuild kernel module To fix it when the module haven't been properly compiled for some reason you could trigger a rebuild of the NVIDIA kernel module with $ sudo dpkg-reconfigure nvidia-kernel-dkms. When you're about to send your hardware in to repair because everything looks all right but the device just isn't working, that really could help (own experience). After the rebuild of the module or modules (if you have a few kernel packages installed) has completed, you could recheck if the module really is available by running:
$ sudo modinfo nvidia-current
filename:       /lib/modules/4.4.0-1-amd64/updates/dkms/nvidia-current.ko
alias:          char-major-195-*
version:        352.79
supported:      external
license:        NVIDIA
alias:          pci:v000010DEd00000E00sv*sd*bc04sc80i00*
alias:          pci:v000010DEd*sv*sd*bc03sc02i00*
alias:          pci:v000010DEd*sv*sd*bc03sc00i00*
depends:        drm
vermagic:       4.4.0-1-amd64 SMP mod_unload modversions 
parm:           NVreg_Mobile:int
It should be something similiar to this when everything is all right. reload kernel module When there are problems with the GPU, maybe the kernel module isn't properly loaded. You could recheck if the module has been properly loaded by doing
$ lsmod   grep nvidia
nvidia_uvm             73728  0
nvidia               8540160  1 nvidia_uvm
drm                   356352  7 i915,drm_kms_helper,nvidia
The kernel module could be loaded resp. reloaded with $ sudo nvidia-modprobe (that tool is from the package nvidia-modprobe). unsupported graphics card Be sure that you graphics cards is supported by the current driver kernel module. If you have bought new hardware, that's quite possible to come out being a problem. You can get the version of the current NVIDIA driver with:
$ cat /proc/driver/nvidia/version 
NVRM version: NVIDIA UNIX x86_64 Kernel Module 352.79  Wed Jan 13 16:17:53 PST 2016
GCC version:  gcc version 5.3.1 20160528 (Debian 5.3.1-21)
Then, google the version number like nvidia 352.79, this should get you onto an official driver download page like this. There, check for what's to be found under "Supported Products". I you're stuck with that there are two options, to wait until the driver in Debian got updated, or replace it with the latest driver package from NVIDIA. That's possible to do, but something more for experienced users. occupied graphics card The CUDA driver cannot work while the graphical interface is busy like by processing the graphical display of your X.Org server. Which kernel driver actually is used to process the desktop could be examined by this command:12
$ grep '(II).*([0-9]):' /var/log/Xorg.0.log
[    37.700] (II) intel(0): Using Kernel Mode Setting driver: i915, version 1.6.0 20150522
[    37.700] (II) intel(0): SNA compiled: xserver-xorg-video-intel 2:2.99.917-2 (Vincent Cheng <vcheng@debian.org>)
 ... 
[    39.808] (II) intel(0): switch to mode 1920x1080@60.0 on eDP1 using pipe 0, position (0, 0), rotation normal, reflection none
[    39.810] (II) intel(0): Setting screen physical size to 508 x 285
[    67.576] (II) intel(0): EDID vendor "CMN", prod id 5941
[    67.576] (II) intel(0): Printing DDC gathered Modelines:
[    67.576] (II) intel(0): Modeline "1920x1080"x0.0  152.84  1920 1968 2000 2250  1080 1083 1088 1132 -hsync -vsync (67.9 kHz eP)
This example shows that the rendering of the desktop is performed by the graphical device of the Intel CPU, which is just like it's needed for running CUDA applications on your NVIDIA graphics card, if you don't have another one. nvidia-cuda-toolkit With the Debian package of the CUDA toolkit everything pretty much runs out of the box for Theano. Just install it with apt-get, and you're ready to go, the CUDA backend is the default one. Pycuda is also a suggested dependency of the binary packages, it could be pulled together with the CUDA toolkit. The up-to-date CUDA release 7.5 is of course available, with that you have Maxwell architecture support so that you can run Theano on e.g. a GeForce GTX Titan X with 6,2 TFLOPS on single precision13 at an affordable price. CUDA 814 is around the corner with support for the new Pascal architecture15. Like the GeForce GTX 1080 high-end gaming graphics card already has 8,23 TFLOPS16. When it comes to professional GPGPU hardware like the Tesla P100 there is much more computational power available, scalable by multiplication of cores resp. cards up to genuine little supercomputers which fit on a desk, like the DGX-117. Theano can use multiple GPUs for calculations to work with highly scaled hardware, I'll write another blog post on this issue. Theano on the GPU It's not difficult to run Theano on the GPU. Only single precision floating point numbers (float32) are supported on the GPU, but that is sufficient for deep learning applications. Theano uses double precision floats (float64) by default, so you have to set the configuration variable config.floatX to float32, like written on above, either with the THEANO_FLAGS environment variable or better in your .theanorc file, if you're going to use the GPU a lot. Switching to the GPU actually happens with the config.device configuration variable, which must be set to either gpu or gpu0, gpu1 etc., to choose a particular one if multiple devices are available. Here's is a little test script check1.py, it's taken from the docs and slightly altered. You can run that script either with python or python3 (there was a single test failure on the Python 3 package, so the Python 2 library might be a little more stable currently). For comparison, here's an example on how it perfoms on my hardware, one time on the CPU, one more time on the GPU:
$ THEANO_FLAGS=floatX=float32 python ./check1.py 
[Elemwise exp,no_inplace (<TensorType(float32, vector)>)]
Looping 1000 times took 4.481719 seconds
Result is [ 1.23178029  1.61879337  1.52278066 ...,  2.20771813  2.29967761
  1.62323284]
Used the cpu
$ THEANO_FLAGS=floatX=float32,device=gpu python ./check1.py 
Using gpu device 0: GeForce 940M (CNMeM is disabled, cuDNN not available)
[GpuElemwise exp,no_inplace (<CudaNdarrayType(float32, vector)>), HostFromGpu(GpuElemwise exp,no_inplace .0)]
Looping 1000 times took 1.164906 seconds
Result is [ 1.23178029  1.61879349  1.52278066 ...,  2.20771813  2.29967761
  1.62323296]
Used the gpu
If you got a result like this you're ready to go with Theano on Debian, training computer vision classifiers or whatever you want to do with it. I'll write more on for what Theano could be used, soon.

  1. Some ports are disabled because they are currently not supported by Theano. There are NotImplementedErrors and other errors in the tests on the numpy.ndarray object being not aligned. The developers commented on that, see here. And on some ports the build flags -m32 resp. -m64 of Theano aren't supported by g++, the build flags can't be manipulated easily.
  2. Theano Development Team: "Theano: a Python framework for fast computation of mathematical expressions"
  3. Marc Couture: "Today's high-powered GPUs: strong for graphics and for maths". In: RTC magazine June 2015, pp. 22 25
  4. Ogier Maitre: "Understanding NVIDIA GPGPU hardware". In: Tsutsui/Collet (eds.): Massively parallel evolutionary computation on GPGPUs. Berlin, Heidelberg: Springer 2013, pp. 15-34
  5. Geoffrey French: "Deep learing tutorial: advanved techniques". PyData London 2016 presentation
  6. Like the description of the Lintian tag binary-without-manpage says, that's not needed for them being in /usr/share.
  7. Tom. R. Halfhill: "Parallel processing with CUDA: Nvidia's high-performance computing platform uses massive multithreading". In: Microprocessor Report January 28, 2008
  8. Faber et.al: "Parallelwelten: GPU-Programmierung mit OpenCL". In: C't 26/2014, pp. 160-165
  9. For comparison, see: Valentine Sinitsyn: "Feel the taste of GPU programming". In: Linux Voice February 2015, pp. 106-109
  10. https://lists.debian.org/debian-devel/2016/07/msg00004.html
  11. If Optimus (hybrid) graphics hardware is present (like commonly today on PC laptops), Debian launches the X-server on the graphics processing unit of the CPU, which is ideal for CUDA. The problem with Optimus actually is the graphics processing on the dedicated GPU. If you are using Bumblebee, the Python interpreter which you want to run Theano on has be to be started with the launcher primusrun, because Bumblebee powers the GPU down with the tool bbswitch every time it isn't used, and I think also the kernel module of the driver is dynamically loaded.
  12. Thorsten Leemhuis: "Treiberreviere. Probleme mit Grafiktreibern f r Linux l sen": In: C't Nr.2/2013, pp. 156-161
  13. Martin Fischer: "4K-Rakete: Die schnellste Single-GPU-Grafikkarte der Welt". In C't 13/2015, pp. 60-61
  14. http://www.heise.de/developer/meldung/Nvidia-CUDA-8-bringt-Optimierungen-fuer-die-Pascal-Architektur-3164254.html
  15. Martin Fischer: "All In: Nvidia enth llt die GPU-Architektur 'Pascal'". In: C't 9/2016, pp. 30-31
  16. Martin Fischer: "Turbo-Pascal: High-End-Grafikkarte f r Spieler: GeForce GTX 1080". In: C't 13/2016, pp. 100-103
  17. http://www.golem.de/news/dgx-1-nvidias-supercomputerchen-mit-8x-tesla-p100-1604-120155.html

19 July 2016

Michael Prokop: DebConf16 in Capetown/South Africa: Lessons learnt

DebConf 16 in Capetown/South Africa was fantastic for many reasons. My Capetown/South Africa/Culture/Flight related lessons: My technical lessons from DebConf16: BTW, thanks to the video team the recordings from the sessions are available online.

Next.

Previous.