Search Results: "reg"

18 January 2015

Mark Brown: Heating the Internet of Things

Internet of Things seems to be trendy these days, people like the shiny apps for controlling things and typically there are claims that the devices will perform better than their predecessors by offloading things to the cloud but this makes some people worry that there are potential security issues and it s not always clear that internet usage is actually delivering benefits over something local. One of the more widely deployed applications is smart thermostats for central heating which is something I ve been playing with. I m using Tado, there s also at least Nest and Hive who do similar things, all relying on being connected to the internet for operation. The main thing I ve noticed has been that the temperature regulation in my flat is better, my previous thermostat allowed the temperature to vary by a couple of degrees around the target temperature in winter which got noticeable, with this the temperature generally seems to vary by a fraction of a degree at most. That does use the internet connection to get the temperature outside, though I m fairly sure that most of this is just a better algorithm (the thermostat monitors how quickly the flat heats up when heating and uses this when to turn off rather than waiting for the temperature to hit the target then seeing it rise further as the radiators cool down) and performance would still be substantially improved without it. The other thing that these systems deliver which does benefit much more from the internet connection is that it s easy to control them remotely. This in turn makes it a lot easier to do things like turn the heating off when it s not needed you can do it remotely, and you can turn the heating back on without being in the flat so that you don t need to remember to turn it off before you leave or come home to a cold building. The smarter ones do this automatically based on location detection from smartphones so you don t need to think about it. For example, when I started this post this I was sitting in a coffee shop so the heating had been turned off based on me taking my phone with me and as a result the temperature gone had down a bit. By the time I got home the flat was back up to normal temperature all without any meaningful intervention or visible difference on my part. This is particularly attractive for me given that I work from home I can t easily set a schedule to turn the heating off during the day like someone who works in an office so the heating would be on a lot of the time. Tado and Nest will to varying extents try to do this automatically, I don t know about Hive. The Tado one at least works very well, I can t speak to the others. I ve not had a bill for a full winter yet but I m fairly sure looking at the meter that between the two features I m saving a substantial amount of energy (and hence money and/or the environment depending on what you care about) and I m also seeing a more constant temperature within the flat, my guess would be that most of the saving is coming from the heating being turned off when I leave the flat. For me at least this means that having the thermostat internet connected is worthwhile.

Dirk Eddelbuettel: Running UBSAN tests via clang with Rocker

Every now and then we get reports from CRAN about our packages failing a test there. A challenging one concerns UBSAN, or Undefined Behaviour Sanitizer. For background on UBSAN, see this RedHat blog post for gcc and this one from LLVM about clang. I had written briefly about this before in a blog post introducing the sanitizers package for tests, as well as the corresponding package page for sanitizers, which clearly predates our follow-up Rocker.org repo / project described in this initial announcement and when we became the official R container for Docker. Rocker had support for SAN testing, but UBSAN was not working yet. So following a recent CRAN report against our RcppAnnoy package, I was unable to replicate the error and asked for help on r-devel in this thread. Martyn Plummer and Jan van der Laan kindly sent their configurations in the same thread and off-list; Jeff Horner did so too following an initial tweet offering help. None of these worked for me, but further trials eventually lead me to the (already mentioned above) RedHat blog post with its mention of -fno-sanitize-recover to actually have an error abort a test. Which, coupled with the settings used by Martyn, were what worked for me: clang-3.5 -fsanitize=undefined -fno-sanitize=float-divide-by-zero,vptr,function -fno-sanitize-recover. This is now part of the updated Dockerfile of the R-devel-SAN-Clang repo behind the r-devel-ubsan-clang. It contains these settings, as well a new support script check.r for littler---which enables testing right out the box. Here is a complete example:

docker                              # run Docker (any recent version, I use 1.2.0)
  run                               # launch a container 
    --rm                            # remove Docker temporary objects when dome
    -ti                             # use a terminal and interactive mode 
    -v $(pwd):/mnt                  # mount the current directory as /mnt in the container
    rocker/r-devel-ubsan-clang      # using the rocker/r-devel-ubsan-clang container
  check.r                           # launch the check.r command from littler (in the container)
    --setwd /mnt                    # with a setwd() to the /mnt directory
    --install-deps                  # installing all package dependencies before the test
    RcppAnnoy_0.0.5.tar.gz          # and test this tarball

I know. It is a mouthful. But it really is merely the standard practice of running Docker to launch a single command. And while I frequently make this the /bin/bash command (hence the -ti options I always use) to work and explore interactively, here we do one better thanks to the (pretty useful so far) check.r script I wrote over the last two days. check.r does about the same as R CMD check. If you look inside check you will see a call to a (non-exported) function from the (R base-internal) tools package. We call the same function here. But to make things more interesting we also first install the package we test to really ensure we have all build-dependencies from CRAN met. (And we plan to extend check.r to support additional apt-get calls in case other libraries etc are needed.) We use the dependencies=TRUE option to have R smartly install Suggests: as well, but only one level (see help(install.packages) for details. With that prerequisite out of the way, the test can proceed as if we had done R CMD check (and additional R CMD INSTALL as well). The result for this (known-bad) package:

edd@max:~/git$ docker run --rm -ti -v $(pwd):/mnt rocker/r-devel-ubsan-clang check.r --setwd /mnt --install-deps RcppAnnoy_0.0.5.tar.gz 
also installing the dependencies  Rcpp ,  BH ,  RUnit 
trying URL 'http://cran.rstudio.com/src/contrib/Rcpp_0.11.3.tar.gz'
Content type 'application/x-gzip' length 2169583 bytes (2.1 MB)
opened URL
==================================================
downloaded 2.1 MB
trying URL 'http://cran.rstudio.com/src/contrib/BH_1.55.0-3.tar.gz'
Content type 'application/x-gzip' length 7860141 bytes (7.5 MB)
opened URL
==================================================
downloaded 7.5 MB
trying URL 'http://cran.rstudio.com/src/contrib/RUnit_0.4.28.tar.gz'
Content type 'application/x-gzip' length 322486 bytes (314 KB)
opened URL
==================================================
downloaded 314 KB
trying URL 'http://cran.rstudio.com/src/contrib/RcppAnnoy_0.0.4.tar.gz'
Content type 'application/x-gzip' length 25777 bytes (25 KB)
opened URL
==================================================
downloaded 25 KB
* installing *source* package  Rcpp  ...
** package  Rcpp  successfully unpacked and MD5 sums checked
** libs
clang++-3.5 -fsanitize=undefined -fno-sanitize=float-divide-by-zero,vptr,function -fno-sanitize-recover -I/usr/local/lib/R/include -DNDEBUG -I../inst/include/ -I/usr/local/include    -fpic  -pipe -Wall -pedantic -
g  -c Date.cpp -o Date.o
[...]
* checking examples ... OK
* checking for unstated dependencies in  tests  ... OK
* checking tests ...
  Running  runUnitTests.R 
 ERROR
Running the tests in  tests/runUnitTests.R  failed.
Last 13 lines of output:
  +     if (getErrors(tests)$nFail > 0)  
  +         stop("TEST FAILED!")
  +      
  +     if (getErrors(tests)$nErr > 0)  
  +         stop("TEST HAD ERRORS!")
  +      
  +     if (getErrors(tests)$nTestFunc < 1)  
  +         stop("NO TEST FUNCTIONS RUN!")
  +      
  +  
  
  
  Executing test function test01getNNsByVector  ... ../inst/include/annoylib.h:532:40: runtime error: index 3 out of bounds for type 'int const[2]'
* checking PDF version of manual ... OK
* DONE
Status: 1 ERROR, 2 WARNINGs, 1 NOTE
See
   /tmp/RcppAnnoy/..Rcheck/00check.log 
for details.
root@a7687c014e55:/tmp/RcppAnnoy#

The log shows that thanks to check.r, we first download and the install the required packages Rcpp, BH, RUnit and RcppAnnoy itself (in the CRAN release). Rcpp is installed first, we then cut out the middle until we get to ... the failure we set out to confirm. Now having a tool to confirm the error, we can work on improved code. One such fix currently under inspection in a non-release version 0.0.5.1 then passes with the exact same invocation (but pointing at RcppAnnoy_0.0.5.1.tar.gz):

edd@max:~/git$ docker run --rm -ti -v $(pwd):/mnt rocker/r-devel-ubsan-clang check.r --setwd /mnt --install-deps RcppAnnoy_0.0.5.1.tar.gz
also installing the dependencies  Rcpp ,  BH ,  RUnit 
[...]
* checking examples ... OK
* checking for unstated dependencies in  tests  ... OK
* checking tests ...
  Running  runUnitTests.R 
 OK
* checking PDF version of manual ... OK
* DONE
Status: 1 WARNING
See
   /mnt/RcppAnnoy.Rcheck/00check.log 
for details.
edd@max:~/git$

This proceeds the same way from the same pristine, clean container for testing. It first installs the four required packages, and the proceeds to test the new and improved tarball. Which passes the test which failed above with no issues. Good. So we now have an "appliance" container anybody can download from free from the Docker hub, and deploy as we did here in order to have fully automated, one-command setup for testing for UBSAN errors. UBSAN is a very powerful tool. We are only beginning to deploy it. There are many more useful configuration settings. I would love to hear from anyone who would like to work on building this out via the R-devel-SAN-Clang GitHub repo. Improvements to the littler scripts are similarly welcome (and I plan on releasing an updated littler package "soon").

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Chris Lamb: Adjusting a backing track with SoX

Earlier today I came across some classical sheet music that included a "playalong" CD, just like a regular recording except it omits the solo cello part. After a quick listen it became clear there were two problems:

The recording was made at A=442, rather than the more standard A=440.
The tempi of the movements was not to my taste, either too fast or too slow.

SoX, the "Swiss Army knife of sound processing programs", can easily adjust the latter, but to remedy the former it must be provided with a dimensionless "cent" unit ie. 1/100th of a semitone rather than the 442Hz and 440Hz reference frequencies. First, we calculate the cent difference with:

https://d1icoid1cnixnp.cloudfront.net/yadt/blog.Image/image/original/24.jpeg

Next, we rip the material from the CD:

$ sudo apt-get install ripit flac
[..]
$ ripit --coder 2 --eject --nointeraction
[..]

And finally we adjust the tempo and pitch:

$ apt-get install sox libsox-fmt-mp3
[..]
$ sox 01.flac 01.mp3 pitch -7.85 tempo 1.00 # (Tuning notes)
$ sox 02.flac 02.mp3 pitch -7.85 tempo 0.95 # Too fast!
$ sox 03.flac 03.mp3 pitch -7.85 tempo 1.01 # Close..
$ sox 04.flac 04.mp3 pitch -7.85 tempo 1.03 # Too slow!

(I'm converting to MP3 at the same time it'll be more convenient on my phone.)

Rog rio Brito: Uploading SICP to Youtube

Intro I am not alone in considering Harold Abelson and Gerald Jay Sussman's recorded lectures based on their book "Structure and Interpretation of Computer Programs" is a masterpiece. There are many things to like about the content of the lectures, beginning with some pearls and wisdom about the craft of writing software (even though this is not really a "software enginneering" book), the clarity with which the concepts are described, the Freedom-friendly aspects of the authors regarding the material that they produced and much, the breadth of the subjects covered and much more.

The videos, their length, and splitting them The course consists of 20 video files and they are all uploaded on Youtube already. There is one thing, though: while the lectures are naturally divided into segments (the instructors took a break in after every 30 minutes or so worth of lectures), the videos corresponding to each lecture have all the segments concatenated. To better watch them, accounting for the easier possibility to put a few of the lectures in a mobile device or to avoid fast forwarding long videos from my NAS when I am watching them on my TV (and some other factors), I decided to sit down, take notes for each video of where the breaks where, and write a simple Python script to help split the videos in segments, and, then, reencode the segments. I decided not to take the videos from Youtube to perform my splitting activities, but, instead, to operate on one of the "sources" that the authors once had in their homepage (videos encoded in DivX and audio in MP3). The videos are still available as a torrent file (with a magnet link for the hash 650704e4439d7857a33fe4e32bcfdc2cb1db34db), with some very good souls still seeding it (I can seed it too, if desired). Alas, I have not found a source for the higher quality MPEG1 videos, but I think that the videos are legible enough to avoid bothering with a larger download. I soon found out that there are some beneficial side-effects of splitting the videos, like not having to edit/equalize the entire audio of the videos when only a segment was bad (which is understandable, as these lectures were recorded almost 30 years ago and technology was not as advanced as things are today). So, since I already have the split videos lying around here, I figured out that, perhaps, other people may want to download them, as they may be more convenient to watch (say, during commutes or whatever/whenever/wherever it best suits them). Of course, uploading all the videos is going to take a while and I would only do it if people would really benefit from them. If you think so, let me know here (or if you know someone who would like the split version of the videos, spread the word).

16 January 2015

EvolvisForge blog: Debian/m68k hacking weekend commencing soonish

As I said, I did not certain events that begun with lea and end with ing prevent me from organising a Debian/m68k hack weekend. Well, that weekend is now. I m too unorganised, and I spent too much time in the last few evenings to organise things so I built up a sleep deficit already and the feedback was slow. (But so are the computers.) And someone I d have loved to come was hurt and can t come. On the plus side, several people I ve long wanted to meet IRL are coming, either already today or tomorrow. I hope we all will have a lot of fun. Legal disclaimer: Debian/m68k is a port of Debian to m68k. It used to be official, but now isn t. It belongs to debian-ports.org, which may run on DSA hardware, but is not acknowledged by Debian at large, unfortunately. Debian is a registered trademark owned by Software in the Public Interest, Inc.

15 January 2015

Lunar: 80%

Unfortunately I could not go on stage at the 31st Chaos Communication Congress to present reproducible builds in Debian alongside Mike Perry from the Tor Project and Seth Schoen from the Electronic Frontier Foundation. I've tried to make it up for it, though and we have made amazing progress. Wiki reorganization What was a massive and frightening wiki page now looks really more welcoming:

Screenshot of ReproducibleBuilds on Debian wiki

Depending on what one is looking for, it should be much easier to find. There's now a high-level status overview given on the landing page, maintainers can learn how to make their packages reproducible, enthusiasts can more easily find what can help the project, and we have even started writing some history. .buildinfo for all packages New year's eve saw me hacking Perl to write dpkg-genbuildinfo. Similar to dpkg-genchanges, it's run by dpkg-buildpackage to produce .buildinfo control files. This is where the build environment, and hash of source and binary packages are recorded. This script, integrated with dpkg, replace the previous debhelper interim solution written by Niko Tyni. We used to fix mtimes in control.tar and data.tar using a specific addition to debhelper named dh_fixmtimes. To better support the ALWAYS_EXCLUDE environment variable and for pragramtic reasons, we moved the process in dh_builddeb. Both changes were quickly pushed to our continuous integration platform. Before, only packages using dh would create a .buildinfo and thus eventually be considered reproducible. With these modifications, many more packages had their chance and this shows:

Growing amount of packages considered reproducible

Yes, with our experimental toolchain we are now at more than eighty percent! That's more than 17200 source packages! srebuild Another big item on the todo-list was crossed over by Johannes Schauer. srebuild is a wrapper around sbuild:

Given a .buildinfo file, it first finds a timestamp of Debian Sid from snapshot.debian.org which contains the requested packages in their exact versions. It then runs sbuild with the right architecture as given by the .buildinfo file and the right base system to upgrade from, as given by the version of the base-files package version in the .buildinfo file. Using two hooks it will install the right package versions and verify that the installed packages are in the right version before the build starts.

Understanding problems Over 1700 packages have now been reviewed to understand why build results could not be reproduced on our experimental platform. The variations between the two builds are currently limited to time and file ordering, but this still has uncovered many problems. There are still toolchain fixes to be made (more than 180 packages for the PHP registry) which can make many packages reproducible at once, but others like C pre-processor macros will require many individual changes. debbindiff, the main tool used to understand differences, has gained support for .udeb, TrueType and OpenType fonts, PNG and PDF files. It's less likely to crash on problems with encoding or external tool. But most importantly for large package, it has been made a lot faster, thanks to Reiner Herrmann and Helmut Grohne. Helmut has also been able to spot cross-compilation issues by using debbindiff! Targeting our efforts It gives warm fuzzy feelings to hit the 80% mark, but it would be a bit irrelevant if this would not concern packages that matter. Thankfully, Holger worked on producing statistics for more specific package sets. Mattia Rizzolo has also done great work to improve the scripts generating the various pages visible on reproducible.debian.net. All essential and build-esential packages, except gcc and bash, are considered reproducible or have patches ready. After some lengthy builds, I also managed to come up with a patch to make linux build reproducibly. Miscellaneous After my initial attempt to modify r-base to remove a timestamp in R packages, Dirk Eddelbuettel discussed the issue with upstream and came up with a better patch. The latter has already been merged upstream! Dirk's solution is to allow timestamps to be set using an external environment variable. This is also how I modified FontForge to make it possible to reproduce fonts. Identifiers generated by xsltproc have also been an issue. After reviewing my initial patch, Andrew Awyer came up with a much nicer solution. Its potential performance implications need to be evaluated before submission, though. Chris West has been working on packages built with Maven amongst other things. PDF generated by GhostScript, another painful source of troubles, is being worked on by Peter De Wachter. Holger got X.509 certificates signed by the CA cartel for jenkins.debian.net and reproducible.debian.net. No more scary security messages now. Let's hope next year we will be able to get certificates through Let's Encrypt! Let's make a difference together As you can imagine with all that happened in the past weeks, the #debian-reproducible IRC channel has been a cool place to hang out. It's very energizing to get together and share contributions, exchange tips and discuss hardest points. Mandatory quote:

* h01ger is very happy to see again and again how this is a nice
         learning circle...! i've learned a whole lot here too... in
         just 3 months... and its going on...!

Reproducible builds are not going to change anything for most of our users. They simply don't care how they get software on their computer. But they care to get the right software without having to worry about it. That's our responsibility, as developers. Enabling users to trust their software is important and a major contribution, we as Debian, can make to the wider free software movement. Once Jessie is released, we should make a collective effort to make reproducible builds an highlight of our next release.

Tim Retout: Docker London Meetup - January 2015

Last week, I visited London for the January Docker meetup, which was the first time I'd attended this group. It was a talk-oriented format, with around 200 attendees packed into Shoreditch Village Hall; free pizza and beer was provided thanks to the sponsors, which was awesome (and makes logistics easier when you're travelling there from work). There were three talks. First, Andrew Martin from British Gas spoke about how they use Docker for testing and continuous deployment of their Node.js microservices - buzzword bingo! But it's helpful to see how companies approach these things. Second, Johan Euphrosine from Google gave a short demo of Google Cloud Platform for running Docker containers (mostly around Container Engine, but also briefly App Engine). This was relevant to my interests, but I'd already seen this sort of talk online. Third, Dan Williams presented his holiday photos featuring a journey on a container ship, which wins points from me for liberal interpretation of the meetup topic, and was genuinely very entertaining/interesting - I just regret having to leave to catch a train halfway through. In summary, this was worth attending, but as someone just getting started with containers I'd love some sort of smaller meetings with opportunities for interaction/activity. There's such a variety of people/use cases for Docker that I'm not sure how much everyone had in common with each other; it would be interesting to find out.

13 January 2015

Dirk Eddelbuettel: RcppGSL 0.2.3

A new version of RcppGSL is now on CRAN today. This package provides an interface from R to the GNU GSL using our Rcpp. Similar to the recent RcppClassic release, this update was triggered by the CRAN maintainers desire to keep the Makefile free of GNU make extensions. At the same time we simplified the configure file a little, and refreshed the look and feel of the vignette. No user-facing changes were made. The NEWS file entries follows below:

Changes in version 0.2.3 (2015-01-10)

The src/Makevars.in was pruned of GNU make features at the request of the CRAN Maintainers

configure.ac and configure were updated, and shortened

The RcppGSL-intro.Rnw vignette was updated for its look and feel.

Courtesy of CRANberries, a summary of changes to the most recent release is available. More information is on the RcppGSL page. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

12 January 2015

Russell Coker: Systemd Notes

A few months ago I gave a lecture about systemd for the Linux Users of Victoria. Here are some of my notes reformatted as a blog post: Scripts in /etc/init.d can still be used, they work the same way as they do under sysvinit for the user. You type the same commands to start and stop daemons. To get a result similar to changing runlevel use the systemctl isolate command. Runlevels were never really supported in Debian (unlike Red Hat where they were used for starting and stopping the X server) so for Debian users there s no change here. The command systemctl with no params shows a list of loaded services and highlights failed units. The command journalctl -u UNIT-PATTERN shows journal entries for the unit(s) in question. The pattern uses wildcards not regexs. The systemd journal includes the stdout and stderr of all daemons. This solves the problem of daemons that don t log all errors to syslog and leave the sysadmin wondering why they don t work. The command systemctl status UNIT gives the status and last log entries for the unit in question. A program can use ioctl(fd, TIOCSTI, ) to push characters into a tty buffer. If the sysadmin runs an untrusted program with the same controlling tty then it can cause the sysadmin shell to run hostile commands. The system call setsid() to create a new terminal session is one solution but managing which daemons can be started with it is difficult. The way that systemd manages start/stop of all daemons solves this. I am glad to be rid of the run_init program we used to use on SE Linux systems to deal with this. Systemd has a mechanism to ask for passwords for SSL keys and encrypted filesystems etc. There have been problems with that in the past but I think they are all fixed now. While there is some difficulty during development the end result of having one consistent way of managing this will be better than having multiple daemons doing it in different ways. The commands systemctl enable and systemctl disable enable/disable daemon start at boot which is easier than the SysVinit alternative of update-rc.d in Debian. Systemd has built in seat management, which is not more complex than consolekit which it replaces. Consolekit was installed automatically without controversy so I don t think there should be controversy about systemd replacing consolekit. Systemd improves performance by parallel start and autofs style fsck. The command systemd-cgtop shows resource use for cgroups it creates. The command systemd-analyze blame shows what delayed the boot process and
systemd-analyze critical-chain shows the critical path in boot delays. Sysremd also has security features such as service private /tmp and restricting service access to directory trees. Conclusion For basic use things just work, you don t need to learn anything new to use systemd. It provides significant benefits for boot speed and potentially security. It doesn t seem more complex than other alternative solutions to the same problems. https://wiki.debian.org/systemd http://freedesktop.org/wiki/Software/systemd/Optimizations/ http://0pointer.de/blog/projects/security.html

11 January 2015

Dirk Eddelbuettel: rfoaas 0.1.1

A brand new and shiny version of rfoaas is now on CRAN. The rfoaas package provides an interface for R to the most excellent FOAAS service--which provides a modern, scalable and RESTful web service for the frequent need to tell someone to f$#@ off. There are two (internal) changes of merit in this version. First off, as FOAAS was refactored upstream, we are now forced to supply an accept: text/plain http request header. Which, sadly enough, is not something the url() function in R can do---so we brought in more cavalry and now depend on the httr package by Hadley, and use its GET() method. A second change is that we added a (simple but effective enough) regression test which simply calls all foaas entry points available throug rfoaas, and compares this to the anticipated result. To run it, you need to set an environment variable RunFOAASTests=yes as eg out Travis script does. Finally, we aligned the version number with upstream to signal that we cover all available entry points of that version. As usual, CRANberries provides a diff to the previous release. Questions, comments etc should go to the GitHub issue tracker.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Ben Hutchings: Linux suspend/resume regression in Debian 7.8

There was a regression in Linux 3.2.65, which unfortunately was included in this weekend's Debian stable point release (7.8) as I didn't point out the bug reports to the stable release team. At least some systems are now failing to resume after suspending to RAM; instead they reboot. I have tracked down the change that caused this, and it should be fixed as part of a security update soon. The change is in code specific to 64-bit x86 (i.e. the Debian amd64 architecture). If you need suspend/resume to work, you might wish to avoid upgrading the linux-image-3.2.0-4-amd64 package until that future update. Update: I have uploaded packages with the problematic changes reverted to people.debian.org. The version number is 3.2.65-1+deb7u1~test (lower than the next security update will be). Mail 774436@bugs.debian.org if this doesn't work for you.

Steve McIntyre: UEFI Debian installer work for Jessie, part 5

Time for another update on my work for UEFI improvements in Jessie! I've spent more time on the integration of 32-bit grub-efi with a 64-bit Debian system, and just published a new test image on pettersson. I've added:

a patch to the Linux kernel to add a new /sys file which exposes the size of the underlying UEFI platform (32- or 64-bit).
a patch to grub2 to read that new /sys file in grub-install to determine the right version of grub-efi to install by default
a patch to grub-installer to do similar

These remove the manual steps that were necessary for a 64-bit installation with the previous build. I've just used this exact image (and a network mirror) to install a fully-functional 64-bit Gnome system on the X205TA, simply by selecting "64-bit install" from the GRUB menu and following prompts. Yay! Visit http://cdimage.debian.org/cdimage/unofficial/efi-development/jessie-upload3/ to download and test the image. Now, there's no guarantee that the kernel patch I've submitted to the linux-efi folks will be accepted in its current form, and even if it is I'll have to get it and the other code I've written accepted into the various packages and then into Jessie! But for now this image should work just fine for Bay Trail folks I hope! WARNING: this CD is provided for testing only. Use at your own risk! If you have appropriate (U)EFI hardware, please try this image and let me know how you get on, via the debian-cd and debian-boot mailing lists. For now, I'm going to pause development here. The core code I'm using to make these images is all in the debian-cd and d-i repos, and I'll push the other patches once I know they'll work with the kernel. But I've got a slew of other things that I need to work on in the next few weeks, in no particular order:

RC bugs filed against abcde
Sorting out Mac-only 32-bit netinst images (only EFI boot? without EFI?)
Regular openstack image generation for Jessie
Regular debian-live image generation for Jessie
...

I'm currently not planning to make all of Debian's amd64 images bootable using 32-bit UEFI like this image - I'm happy to leave this as just an option for our multi-arch i386/amd64 images (netinst or DVD only). I think that's a reasonable compromise here, and it's also the easiest thing for me to do with the current debian-cd build system. Finally, apologies if you've asked me questions about the earlier images in this series and I've not responded yet. Fixing that ASAP!

10 January 2015

Dirk Eddelbuettel: RcppClassic 0.9.6

A maintenance release of RcppClassic, now at version 0.9.6, went out to CRAN today. This package provides a maintained version of the otherwise deprecated first Rcpp API; no new projects should use it. No changes were in user-facing code. The Makevars file was change to accomodate a request by the CRAN Maintainer to keep it free of GNU Make extensions. At the same time, we overhauled the look and feel of the (very short) vignette. Build instructions were updated both in the vignette and in the included example package. Other accumulated changes since the last release were updates to the DESCRIPTION and NAMESPACE file as well two namespace-related R code updates. Courtesy of CRANberries, there is the set of changes relative to the previous release. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

9 January 2015

Diego Escalante Urrelo: Link Pack #02

First sequel to my Link Pack series (I ll remove the quotes when it s LP#05): Link Pack #01. This time I m going for fewer articles, to try to keep things less overwhelming. There s no special theme, and I m actually leaving out some nice things I read recently. On the plus side, that means I have good material for a Link Pack #03. Also, I m gonna stick with Link Pack as a name, because it s good enough :-).

A Teenager s View on Social Media: Written by an actual teen
A well thought and realistic take on how social media is being used nowadays by teenagers. I have seen the patterns the author describes, and actually follow many of them. Does that mean I m still a teenager? It s interesting that the messaging and group-messaging part of the article is very US centric, or at least very US centric from my point of view. WhatsApp is the default messenger application south of the states, and fills the role of somewhere you can chat with people without having to give them your full personal information , that is, a place where you can chat with someone without running out of SMS and without adding them on Facebook (which would open them to stalk your whole profile and other friends). Some carriers in South Am rica offer unlimited plans for specific applications like WhatsApp. What Would Jesus Buy? (2007) Full movie
Reverend Billy Talen from the Church of Stop Shopping Gospel is trying to prevent the Shopocalypse from happening. It s an entertaining story of a group of funny guys and girls trying to share a message with comedy (that means A+ on my list). Simple and independent, a nice film. 13 Nutrition Lies That Made The World Sick And Fat
A pet peeve of mine. Nutrition is not really that complicated, but unfortunately there are a lot of myths that make people take really bad decisions. If you only read one thing in ~~2014~~ 2015, read this.

Bottom Line: The low-fat, high-carb diet recommended by the mainstream nutrition organizations is a miserable failure and has been repeatedly proven to be ineffective. ( ) Bottom Line: Low-carb diets are the easiest, healthiest and most effective way to lose weight and reverse metabolic disease. It is pretty much a scientific fact at this point.

GM s hit and run: How a lawyer, mechanic, and engineer blew open the worst auto scandal in history
Cars are so complex nowadays, and dependent on electronics, that I m honestly afraid of them. I have made software for many years and I know how hard, impossible, it is to get things perfect . I can t imagine how hard it is for something so critical as brakes, steering wheels, etc. Even cameras can t get focus right some times, and it s been many many years.

Countless articles have been written about General Motors and its massive recalls earlier this year. What hasn t been fully told is how GM might have gotten away with multiple counts of consumercide were it not for the efforts of three men: a Georgia lawyer, a Mississippi mechanic, and a Florida engineer. ( ) Brooke Melton needn t have died that night. She was killed by a corporation s callous disregard for the safety of its customers, made worse by a regulatory agency reluctant to regulate.

The Long Game: Part 1 and The Long Game: Part 2
Two very short (less than 5 minutes) video essays about how notable people in the story of creativity are always celebrated without mentioning the boring years when they were nothing but losers. It s a fun little video, worth a watch for the idea and the interesting editing. It feels like someone really wanted to create these.

Dirk Eddelbuettel: random 0.2.3

A new release of my random package for truly (hardware-based) random numbers as provided by random.org is now on CRAN. The main change is a switch to the curl() function from the eponymous package by Jeroen Ooms. This was caused by random.org now using https instead of http, annd the fact that te url() function from R does not cope well with the redirect. Besides this (enforced) change, everything else remains the same. Courtesy of CRANberries comes a diffstat report for this release. Current and previous releases are available here as well as on CRAN.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

8 January 2015

Russell Coker: Conference Suggestions

LCA 2015 is next week so it seems like a good time to offer some suggestions for other delegates based on observations of past LCAs. There s nothing LCA specific about the advice, but everything is based on events that happened at past LCAs. Don t Oppose a Lecture Question time at the end of a lecture isn t the time to demonstrate that you oppose everything about the lecture. Discussion time between talks at a mini-conf isn t a time to demonstrate that you oppose the entire mini-conf. If you think a lecture or mini-conf is entirely wrong then you shouldn t attend. The conference organisers decide which lectures and mini-confs are worthy of inclusion and the large number of people who attend the conference are signalling their support for the judgement of the conference organisers. The people who attend the lectures and mini-confs in question want to learn about the topics in question and people who object should be silent. If someone gives a lecture about technology which appears to have a flaw then it might be OK to ask one single question about how that issue is resolved, apart from that the lecture hall is for the lecturer to describe their vision. The worst example of this was between talks at the Haecksen mini-conf last year when an elderly man tried at great length to convince me that everything about feminism is wrong. I m not sure to what degree the Haecksen mini-conf is supposed to be a feminist event, but I think it s quite obviously connected to feminism which is of course was why he wanted to pull that stunt. After he discovered that I was not going to be convinced and that I wasn t at all interested in the discussion he went to the front of the room to make a sexist joke and left. Consider Your Share of Conference Resources I ve previously written about the length of conference questions [1]. Question time after a lecture is a resource that is shared among all delegates. Consider whether you are asking more questions than the other delegates and whether the questions are adding benefit to other people. If not then send email to the speaker or talk to them after their lecture. Note that good questions can add significant value to the experience of most delegates. For example when a lecturer appears to be having difficulty in describing their ideas to the audience then good questions can make a real difference, but it takes significant skill to ask such questions. Dorm Walls Are Thin LCA is one of many conferences that is typically held at a university with dorm rooms offered for delegates. Dorm rooms tend to have thinner walls than hotel rooms so it s good to avoid needless noise at night. If one of your devices is going to make sounds at night please check the volume settings before you start it. At one LCA I was startled at about 2AM but the sound of a very loud porn video from a nearby dorm room, the volume was reduced within a few seconds, but it s difficult to get to sleep quickly after that sort of surprise. If you set an alarm then try to avoid waking other people. If you set an early alarm and then just get up then other people will get back to sleep, but pressing snooze repeatedly for several hours (as has been done in the past) is anti-social. Generally I think that an alarm should be at a low volume unless it is set for less than an hour before the first lecture in which case waking people in other dorm rooms might be doing them a favor. Phones in Lectures Do I need to write about this? Apparently I do because people keep doing it! Phones can be easily turned to vibrate mode, most people who I ve observed taking calls in LCA lectures have managed this but it s worth noting for those who don t. There are very few good reasons for actually taking a call when in a lecture. If the hospital calls to tell you that they have found a matching organ donor then it s a good reason to take the call, but I can t think of any other good example. Many LCA delegates do system administration work and get calls at all times of the day and night when servers have problems. But that isn t an excuse for having a conversation in the middle of the lecture hall while the lecture is in progress (as has been done). If you press the green button on a phone you can then walk out of the lecture hall before talking, it s expected that mobile phone calls sometimes have signal problems at the start of the call so no-one is going to be particularly surprised if it takes 10 seconds before you say hello. As an aside, I think that the requirement for not disturbing other people depends on the number of people who are there to be disturbed. In tutorials there are fewer people and the requirements for avoiding phone calls are less strict. In BoFs the requirements are less strict again. But the above is based on behaviour I ve witnessed in mini-confs and main lectures. Smoking It is the responsibility of people who consume substances to ensure that their actions don t affect others. For smokers that means smoking far enough away from lecture halls that it s possible for other delegates to attend the lecture without breathing in smoke. Don t smoke in the lecture halls or near the doorways. Also using an e-cigarette is still smoking, don t do it in a lecture hall. Photography Unwanted photography can be harassment. I don t think there s a need to ask for permission to photograp people who harass others or break the law. But photographing people who break the social agreement as to what should be done in a lecture probably isn t. At a previous LCA a man wanted to ask so many questions at a keynote lecture that he had a page of written notes (seriously), that was obviously outside the expected range of behaviour but probably didn t justify the many people who photographed him. A Final Note I don t think that LCA is in any way different from other conferences in this regard. Also I don t think that there s much that conference organisers can or should do about such things.

[1] http://etbe.coker.com.au/2014/01/06/length-conference-questions/

MJ Ray: Social Network Wishlist

All I want for 2015 is a Free/Open Source Software social network which is:

easy to register on (no reCaptcha disability-discriminator or similar, a simple openID, activation emails that actually arrive);
has an email help address or online support or phone number or something other than the website which can be used if the registration system causes a problem;
can email when things happen that I might be interested in;
can email me summaries of what s happened last week/month in case they don t know what they re interested in;
doesn t email me too much (but this is rare);
interacts well with other websites (allows long-term members to post links, sends trackbacks or pingbacks to let the remote site know we re talking about them, makes it easy for us to dent/tweet/link to the forum nicely, and so on);
isn t full of spam (has limits on link-posting, moderators are contactable/accountable and so on, and the software gives them decent anti-spam tools);
lets me back up my data;
is friendly and welcoming and trolls are kept in check.

Is this too much to ask for? Does it exist already?

7 January 2015

Chris Lamb: Web scraping: Let's move on

Every few days, someone publishes a new guide, tutorial, library or framework about web scraping, the practice of extracting information from websites where an API is either not provided or is otherwise incomplete. However, I find these resources fundamentally deceptive the arduous parts of "real world" scraping simply aren't in the parsing and extraction of data from the target page, the typical focus of these articles. The difficulties are invariably in "post-processing"; working around incomplete data on the page, handling errors gracefully and retrying in some (but not all) situations, keeping on top of layout/URL/data changes to the target site, not hitting your target site too often, logging into the target site if necessary and rotating credentials and IP addresses, respecting robots.txt, target site being utterly braindead, keeping users meaningfully informed of scraping progress if they are waiting of it, target site adding and removing data resulting in a null-leaning database schema, sane parallelisation in the presence of prioritisation of important requests, difficulties in monitoring a scraping system due to its implicitly non-deterministic nature, and general problems associated with long-running background processes in web stacks. Et cetera. In other words, extracting the right text on the page is the easiest and trivial part by far, with little practical difference between an admittedly cute jQuery-esque parsing library or even just using a blunt regular expression. It would be quixotic to simply retort that sites should provide "proper" APIs but I would love to see more attempts at solutions that go beyond the superficial.

Dirk Eddelbuettel: RcppCNPy 0.2.4

A new release of the RcppCNPy package is now on CRAN. This release mostly solidifies and fixes things. Support for saving integer objects, which was expanded in release 0.2.3, was not entirely correct. Operations on big-endian systems were not up to snuff either. Wush Wu helped in getting this right with very diligent testing and patching particularly on big-endian hardware. We also got a pull request from Romain to reflect better const correctness at the Rcpp side of things. Last but not least we obliged by the CRAN Maintainers to not assume one could call gzip from system() call because, well, you guessed it.

Changes in version 0.2.4 (2015-01-05)

Support for saving integer objects was not correct and has been fixed.

Support for loading and saving on 'big endian' systems was incomplete, has been greatly expanded and corrected, thanks in large part to very diligent testing as well as patching by Wush Wu.

The implementation now uses const iterators, thanks to a pull request by Romain Francois.

The vignette no longer assumes that one can call gzip via system as the world's leading consumer OS may disagree.

CRANberries also provides a diffstat report for the latest release. As always, feedback is welcome and the rcpp-devel mailing list off the R-Forge page for Rcpp is may be the best place to start a discussion. GitHub issue tickets are also welcome.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

6 January 2015

Holger Levsen: 20150106-lts-december-2014

My LTS December In December 2014 I spent 11h on Debian LTS work and managed to get six DLAs released and another one almost done... I did:

Release DLA 103-1 which was previously prepared by Ben, Raphael and myself. So while for this release in December I only had to review one patch, I also had to build the package, provide prelimary .debs, ask for feedback, do some final smoke tests, write the announcement and do the upload. In total this still took 2.5h to "just release it"...
Doing DLA 114-1 for bind9 was rather straightforward,
As was DLA 116-1 for ntp, which I managed to release within one hour after the DSA for wheezy, despite me having to make the patch apply cleanly due to some openssl differences...
I mentioned the bit about openssl because noone ever made a mistake with such patches. Seriously, I mean: I would welcome a public review system for security fixes. We are all humans and we all make mistakes. I do think my ntp patching was safe, but... mistakes happen.
DLA 118-1 was basically "just" a new 2.6.32.65 kernel update, which I almost released on my own, until (thankfully) Ben helped me wih one patch from .65 not applying (a fix for a wrong fix which Debian already had correctly fixed), which was due to a patch not correctly removed due to linenumber changes. And while I was still wrapping my head around applying+deapplying these very similar looking patches, Ben had already commited the fix. I'm quite happy with this sharing the work, due to the following benefits: a.) Ben can spend more time on important tasks and b.) the LTS user get more kernel security fixes faster.
DLA 119-1 for subversion was a rather straight forward take from the wheez DSAs again, I just had to make sure to also include the 2nd regression fixing DSA.
And then, I failed to finish my work on a jqueryui update before 31c3 started. And 31c3 really only ended yesterday when I helped putting stuff on trucks and cleaned the big hall... So that's also why I'm only writing this blog post now, and not two weeks ago, like I probably better had. Anyway, according to the security-tracker jqueryui is affected by two CVEs and that's wrong: CVE-2012-6662 does not affect the squeeze version. CVE-2010-5312 on the other hand affects the squeeze version, I know how to fix it, I just lacked a quiet moment to prepare my fix properly and test it, and so I've rather postponed doing so during 31c3... so, expect a DLA for jqeuryui very soon now!

Thanks to everyone who is supporting Squeeze LTS in whatever form! Even just expressing that you or the company or project you're working with is using LTS, is useful, as it's always nice to hear once work is used and appreciated. If you can contribute more, please do so. If you can't, that's also fine. It's free software after all :-)

Next.

Previous.