Search Results: "ag"

25 November 2016

Iain R. Learmonth: vmdebootstrap Sprint Report

This is now a little overdue, but here it is. On the 10th and 11th of November, the second vmdebootstrap sprint took place. Lars Wirzenius (liw), Ana Custura (ana_c) and myself were present. liw focussed on the core of vmdebootstrap, where he sketched out what the future of vmdebootstrap may look like. He documented this in a mailing list post and also presented (video). Ana and myself worked on live-wrapper, which uses vmdebootstrap internally for the squashfs generation. I worked on improving logging, using a better method for getting paths within the image, enabling generation of Packages and Release files for the image archive and also made the images installable (live-wrapper 0.5 onwards will include an installer by default). Ana worked on the inclusion of HDT and memtest86+ in the live images and enabled both ISOLINUX (for BIOS boot) and GRUB (for EFI boot) to boot the text-mode and graphical installers. live-wrapper 0.5 was released on the 16th November with these fixes included. You can find live-wrapper documentation at https://live-wrapper.readthedocs.io/en/latest/. (The documentation still needs some work, some options may be incorrectly described). Thanks to the sponsors that made this work possible. You re awesome. (:

Michael Stapelberg: Debian package build tools

Personally, I find the packaging tools which are available in Debian far too complex. To better understand the options we have, I created a diagram of tools which are frequently used, only covering the build step (i.e. no post-build quality assurance checks or packaging-time helpers): debian package build tools When I was first introduced to Debian packaging, people recommended I use pbuilder. Given how complex the toolchain is in the pbuilder case, I don t understand why that is (was?) a common recommendation. Back in August 2015, so well over a year ago, I switched to sbuild, motivated by how much simpler it was to implement ratt (rebuilds reverse build dependencies) using sbuild, and I have not looked back. Are there people who do not use sbuild for reasons other than familiarity? If so, please let me know, I d like to understand. I also made a version of the diagram above, colored by the programming languages in which the tools are implemented. The chosen colors are heavily biased :-). debian package build tools, by language To me, the diagram above means: if you want to make substantial changes to the Debian build tool infrastructure, you need to become an expert in all of Python, Perl, Bash, C and Make. I know that this is not true for every change, but it still irks me that there might be changes for which it is required. I propose to eliminate complexity in Debian by deprecating the pbuilder toolchain in favor of sbuild.

24 November 2016

Dirk Eddelbuettel: RcppExamples 0.1.8

A new version of the RcppExamples package is now on CRAN. The RcppExamples package provides a handful of short examples detailing by concrete working examples how to set up basic R data structures in C++. This version takes advantage of the updated date and datetime classes in Rcpp 0.12.8 (which are optional for now and being phased in while we deprecate the old ones). A NEWS extract follows:

Changes in RcppExamples version 0.1.8 (2016-11-24)
  • Updated DateExample to show vector addition available under Rcpp 0.12.8 when the (currently still phased in and optional) new Date(time) classes are used via the define in src/Makevars,.win; with fallback code for older versions
  • Other minor edits to DESCRIPTION and README.md

Courtesy of CRANberries, there is also a diffstat report for the most recent release.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Sven Hoexter: first ditch effort - LyX 2.2.2 in unstable build with Qt5

No, not about the latest NOFX record, though it's a great one. Buy it. ;) Took me a hell of a long time to get my head out of my arse and dive again into some Debian related work. Thanks to Nik for pushing me from time to time. So I've taken the time to upload LyX 2.2.2 to unstable and it's now build with Qt5. Afterall the package is still missing a lot of love, but I hope we've once again something for the upcoming stable release, that is close to the latest upstream stable release. If you use LyX please give it a try. For myself it's now the 6th year that I stopped using LyX after maintaining it for five years. And still I'm sponsoring the uploads and try to keep it at least functional. Strange how we sometimes take care of stuff even if we no longer have an active use for them.

Ritesh Raj Sarraf: LIO -fb in Debian

LIO -fb is the new SCSI Target for Debian. Previously, we maintained the LIO tools from the pre-fork upstream branch. But, with good reasons, we've now moved to the newer -fb (Free Branch). As the maintainer for those pacakges, I have a local LIO setup. Overy the years, I've been tuning and using this setup with a bunch of SCSI clients. Now with the new -fb packages it was worrisome for me, on how to migrate (Note: migration is not supported by the Debian packages) my old setup to the new one. Thanks to Andy Grover for mentioning it, migrating your configuration is doable. With some minor intervention, I was able to switch my config from old LIO setup to the new LIO -fb pacakges. As you can see from the output below, both the outputs look the same, which is a good thing. LIO reads its configuration from /etc/target/ and passes it into the kernel. The kernel loads the config. The real time config is present in configfs, within the kernel. Users willing for such migration need to ensure that the loaded config data remains in configfs. And then, using the new -fb tools (targetctl), the configuration data needs to be read and written to a new format in /etc/.
/> ls
o- / ......................................................................................................................... [...]
  o- backstores .............................................................................................................. [...]
    o- fileio ................................................................................................... [0 Storage Object]
    o- iblock .................................................................................................. [4 Storage Objects]
      o- CENTOS ................................................................................................. [/dev/vdd, in use]
      o- SAN1 ................................................................................................... [/dev/vdb, in use]
      o- SAN2 ................................................................................................... [/dev/vdc, in use]
      o- SANROOT .............................................. [/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi0-0-0-0, in use]
    o- pscsi .................................................................................................... [0 Storage Object]
    o- rd_mcp ................................................................................................... [0 Storage Object]
  o- ib_srpt ........................................................................................................... [0 Targets]
  o- iscsi ............................................................................................................. [3 Targets]
    o- iqn.1994-05.com.redhat:23d8eb7fa1fc ................................................................................. [1 TPG]
      o- tpg1 ............................................................................................................ [enabled]
        o- acls ............................................................................................................ [1 ACL]
          o- iqn.1994-05.com.redhat:23d8eb7fa1fc .................................................................... [1 Mapped LUN]
            o- mapped_lun0 ............................................................................................. [lun0 (rw)]
        o- luns ............................................................................................................ [1 LUN]
          o- lun0 ....................................................................................... [iblock/CENTOS (/dev/vdd)]
        o- portals ..................................................................................................... [4 Portals]
          o- 172.16.20.40:3260 ................................................................................. [OK, iser disabled]
          o- 172.16.20.41:3260 ................................................................................. [OK, iser disabled]
          o- 172.16.20.42:3260 ................................................................................. [OK, iser disabled]
          o- 172.16.20.43:3260 ................................................................................. [OK, iser disabled]
    o- iqn.2003-01.org.linux-iscsi.debian.sanboot .......................................................................... [1 TPG]
      o- tpg1 ............................................................................................................ [enabled]
        o- acls ............................................................................................................ [1 ACL]
          o- iqn.2005-03.org.open-iscsi:fd2bc2f4652a-sanboot ........................................................ [1 Mapped LUN]
            o- mapped_lun0 ............................................................................................. [lun0 (rw)]
        o- luns ............................................................................................................ [1 LUN]
          o- lun0 .................................... [iblock/SANROOT (/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi0-0-0-0)]
        o- portals ..................................................................................................... [4 Portals]
          o- 172.16.20.40:3260 ................................................................................. [OK, iser disabled]
          o- 172.16.20.41:3260 ................................................................................. [OK, iser disabled]
          o- 172.16.20.42:3260 ................................................................................. [OK, iser disabled]
          o- 172.16.20.43:3260 ................................................................................. [OK, iser disabled]
    o- iqn.2003-01.org.linux-iscsi.debian.x8664 ............................................................................ [1 TPG]
      o- tpg1 ............................................................................................................ [enabled]
        o- acls ............................................................................................................ [1 ACL]
          o- iqn.1993-08.org.debian:01:2972f6b5fc7 ................................................................. [2 Mapped LUNs]
            o- mapped_lun0 ............................................................................................. [lun0 (rw)]
            o- mapped_lun1 ............................................................................................. [lun1 (rw)]
        o- luns ........................................................................................................... [2 LUNs]
          o- lun0 ......................................................................................... [iblock/SAN1 (/dev/vdb)]
          o- lun1 ......................................................................................... [iblock/SAN2 (/dev/vdc)]
        o- portals ..................................................................................................... [4 Portals]
          o- 172.16.20.40:3260 ................................................................................. [OK, iser disabled]
          o- 172.16.20.41:3260 ................................................................................. [OK, iser disabled]
          o- 172.16.20.42:3260 ................................................................................. [OK, iser disabled]
          o- 172.16.20.43:3260 ................................................................................. [OK, iser disabled]
  o- loopback .......................................................................................................... [0 Targets]
  o- qla2xxx ........................................................................................................... [0 Targets]
  o- tcm_fc ............................................................................................................ [0 Targets]
  o- vhost ............................................................................................................. [0 Targets]
/> 
/> ls
o- / ......................................................................................................................... [...]
  o- backstores .............................................................................................................. [...]
    o- block .................................................................................................. [Storage Objects: 4]
      o- CENTOS ........................................................................... [/dev/vdd (2.0GiB) write-thru activated]
      o- SAN1 ............................................................................. [/dev/vdb (1.0GiB) write-thru activated]
      o- SAN2 ............................................................................. [/dev/vdc (1.0GiB) write-thru activated]
      o- SANROOT ........................ [/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi0-0-0-0 (8.0GiB) write-thru activated]
    o- fileio ................................................................................................. [Storage Objects: 0]
    o- pscsi .................................................................................................. [Storage Objects: 0]
    o- ramdisk ................................................................................................ [Storage Objects: 0]
  o- iscsi ............................................................................................................ [Targets: 3]
    o- iqn.1994-05.com.redhat:23d8eb7fa1fc ............................................................................... [TPGs: 1]
      o- tpg1 ............................................................................................... [no-gen-acls, no-auth]
        o- acls .......................................................................................................... [ACLs: 1]
          o- iqn.1994-05.com.redhat:23d8eb7fa1fc .................................................................. [Mapped LUNs: 1]
            o- mapped_lun0 ................................................................................ [lun0 block/CENTOS (rw)]
        o- luns .......................................................................................................... [LUNs: 1]
          o- lun0 ........................................................................................ [block/CENTOS (/dev/vdd)]
        o- portals .................................................................................................... [Portals: 4]
          o- 172.16.20.40:3260 ................................................................................................ [OK]
          o- 172.16.20.41:3260 ................................................................................................ [OK]
          o- 172.16.20.42:3260 ................................................................................................ [OK]
          o- 172.16.20.43:3260 ................................................................................................ [OK]
    o- iqn.2003-01.org.linux-iscsi.debian.sanboot ........................................................................ [TPGs: 1]
      o- tpg1 ............................................................................................... [no-gen-acls, no-auth]
        o- acls .......................................................................................................... [ACLs: 1]
          o- iqn.2005-03.org.open-iscsi:fd2bc2f4652a-sanboot ...................................................... [Mapped LUNs: 1]
            o- mapped_lun0 ............................................................................... [lun0 block/SANROOT (rw)]
        o- luns .......................................................................................................... [LUNs: 1]
          o- lun0 ..................................... [block/SANROOT (/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi0-0-0-0)]
        o- portals .................................................................................................... [Portals: 4]
          o- 172.16.20.40:3260 ................................................................................................ [OK]
          o- 172.16.20.41:3260 ................................................................................................ [OK]
          o- 172.16.20.42:3260 ................................................................................................ [OK]
          o- 172.16.20.43:3260 ................................................................................................ [OK]
    o- iqn.2003-01.org.linux-iscsi.debian.x8664 .......................................................................... [TPGs: 1]
      o- tpg1 ............................................................................................... [no-gen-acls, no-auth]
        o- acls .......................................................................................................... [ACLs: 1]
          o- iqn.1993-08.org.debian:01:2972f6b5fc7 ................................................................ [Mapped LUNs: 2]
            o- mapped_lun0 .................................................................................. [lun0 block/SAN1 (rw)]
            o- mapped_lun1 .................................................................................. [lun1 block/SAN2 (rw)]
        o- luns .......................................................................................................... [LUNs: 2]
          o- lun0 .......................................................................................... [block/SAN1 (/dev/vdb)]
          o- lun1 .......................................................................................... [block/SAN2 (/dev/vdc)]
        o- portals .................................................................................................... [Portals: 4]
          o- 172.16.20.40:3260 ................................................................................................ [OK]
          o- 172.16.20.41:3260 ................................................................................................ [OK]
          o- 172.16.20.42:3260 ................................................................................................ [OK]
          o- 172.16.20.43:3260 ................................................................................................ [OK]
  o- loopback ......................................................................................................... [Targets: 0]
  o- vhost ............................................................................................................ [Targets: 0]
/> 

Categories:

Keywords:

Like:

Ritesh Raj Sarraf: SAN Updates for Debian Stretch

Now that we prepare for the next Debian Stable release (Stretch), it is time to provide some updates on what the current state of some of the (storage related) packages in Debian is. This is not an update on the complete list of packages related to storage, but it does cover some of them. REMOVALS
  • iscsitarget - The iscsitarget stood as a great SCSI target for the Linux kernel. It seems to have had a good user base not just in Linux but also with VMWare users. But this storage target was always out-of-tree. With LIO having gotten merged as the default in-kernel SCSI Target, development on iscsitarget seems to have stalled. In Debian, for Stretch, there will be no iscsitarget. The package is already removed from Debian Testing and Debian Unstable, and nobody has volunteered to take over it.
  • system-storage-manager - This tool intended to be a simple unified storage tool, through which one could work with various storage technologies like LVM, BTRFS, cryptsetup, SCSI etc. But the upstream development hasn't really been much lately. For Debian Stable, it shouldn't be part of it, given it has some bugs.
  • libstoragemgmt - libstoragemgmt is a universal storage client-side library to talk to remote Storage Arrays. The project is active upstream. For Debian, the package is out-of-date and, now, also needs a maintainer. Unless someone picks up this package, it will not be part of Debian Stretch.
UPDATES
  • open-iscsi - This is the default iSCSI Initiator for Linux distributions. After a long slow development, upstream recently did a new release. This new release accomplished an important milestone; Hardware Offloading for QLogic cards. A special thanks to Frank Fegert, who helped with many aspects of the new iscsiuio package. And thanks to Christian Seiler, who is now co-maintaining the package, it is in great shape. We have fixed some long outstanding bugs and open-iscsi now has much much better integration with the whole system. For Jessie too, we have the up-to-date open-iscsi pacakges (including the new iscsiuio package, with iSCSI Offload) available through jessie-packports
  • open-isns - iSNS is the Naming Service for Storage. This is a new package in Debian Stretch. For users on Debian Jessie, Christian's efforts have made the open-isns package available in jessie-backports too.
  • multipath-tools - After years of slow development, multipath-tools too saw some active development this year, thanks to Xose and Christophe. The Debian version is up-to-date with the latest upstream release. For Debian Stretch, multipath-tools should have good integration with systemd.
  • sg3-utils - sg3 provides simple tools to query, using SCSI commands. The package is up-to-date and in good shape for Debian Stretch.
  • LIO Target - This is going to be the big entry for Debian Stretch. LIO is the in-kernel SCSI Target for Linux. For various reasons, we did not have LIO in Jessie. For Stretch, thanks to Christian Seiler and Christophe Vu-Brugier, we now have the well maintained -fb fork into Debian, which will replace the initial packages from the pre-fork upstream. The -fb fork is maintained by Andy Grover, and now, seems to have users from many other distributions and the kernel community. And given that LIO -fb branch is also part of the RHEL product family, we hope to see a well maintained project and an active upstream. The older packages: targetcli, python-rtslib and python-configshell shall be removed from the archive soon.
Debian users and derivatives, using these storage tools, may want to test/report now. Because once Stretch is released, getting new fixes in may not be easy enough. So please, if you have reliance on these tools, please test and report bugs, now.

Categories:

Keywords:

Like:

Michael Stapelberg: Debian stretch on the Raspberry Pi 3

The last couple of days, I worked on getting Debian to run on the Raspberry Pi 3. Thanks to the work of many talented people, the Linux kernel in version 4.8 is _almost_ ready to run on the Raspberry Pi 3. The only missing thing is the bcm2835 MMC driver, which is required to read the root file system from the SD card. I ve asked our maintainers to include the patch for the time being. Aside from the kernel, one also needs a working bootloader, hence I used Ubuntu s linux-firmware-raspi2 package and uploaded the linux-firmware-raspi3 package to Debian. The package is currently in the NEW queue and needs to be accepted by ftp-master before entering Debian. The most popular method of providing a Linux distribution for the Raspberry Pi is to provide an image that can be written to an SD card. I made two little changes to vmdebootstrap (#845439, #845526) which make it easier to create such an image. The Debian wiki page https://wiki.debian.org/RaspberryPi3 describes the current state of affairs and should be updated, as this blog post will not be updated. As a preview version (i.e. unofficial, unsupported, etc.) until all the necessary bits and pieces are in place to build images in a proper place in Debian, I built and uploaded the resulting image. Find it at https://people.debian.org/~stapelberg/raspberrypi3/. To install the image, insert the SD card into your computer (I m assuming it s available as /dev/sdb) and copy the image onto it:
$ wget https://people.debian.org/~stapelberg/raspberrypi3/2016-11-24-raspberry-pi-3-stretch-PREVIEW.img
$ sudo dd if=2016-11-24-raspberry-pi-3-stretch-PREVIEW.img of=/dev/sdb bs=5M
I hope this initial work on getting Debian booted will motivate other people to contribute little improvements here and there. A list of current limitations and potential improvements can be found on the RaspberryPi3 Debian wiki page.

23 November 2016

Joachim Breitner: microG on Jolla

I am a incorrigibly in picking non-mainstream, open smartphones, and then struggling hard. Back then in 2008, I tried to use the OpenMoko FreeRunner, but eventually gave up because of hardware glitches and reverted to my good old Siemens S35. It was not that I would not be willing to put up with inconveniences, but as soon as it makes live more difficult for the people I communicate with, it becomes hard to sustain. Two years ago I tried again, and got myself a Jolla phone, running Sailfish OS. Things are much nicer now: The hardware is mature, battery live is good, and the Android compatibility layer enables me to run many important apps that are hard to replace, especially the Deutsche Bahn Navigator and various messengers, namely Telegram, Facebook Messenger, Threema and GroupMe. Some apps that require Google Play Services, which provides a bunch of common tasks and usually comes with the Google Play store would not run on my phone, as Google Play is not supported on Sailfish OS. So far, the most annoying ones of that sort were Uber and Lyft, making me pay for expensive taxis when others would ride cheaper, but I can live with that. I tried to install Google Play Services from shady sources, but it would regularly crash.

Signal on Jolla Now in Philadelphia, people urged me to use the Signal messenger, and I was convinced by its support for good end-to-end crypto, while still supporting offline messages and allowing me to switch from my phone to my desktop and back during a conversation. The official Signal app uses Google Cloud Messaging (GCM, part of Google Play Services) to get push updates about new posts, and while I do not oppose this use of Google services (it really is just a ping without any metadata), this is a problem on Sailfish OS. Luckily, the Signal client is open source, and someone created a LibreSignal edition that replaced the use of GCM with websockets, and indeed, this worked on my phone, and I could communicate. Things were not ideal, though: I would often have to restart the app to get newly received messages; messages that I send via Signal Desktop would often not show up on the phone and, most severe, basically after every three messages, sending more messages from Desktop would stop working for my correspondents, which freaked them out. (Strangely it continued working from their phone app, so we coped for a while.) So again, my choice of non-standard devices causes inconveniences to others. This, and the fact that the original authors of Signal and the maintainers of LibreSignal got into a fight that ended LibreSignal discontinued, meant that I have to change something about this situation. I was almost ready to give in and get myself a Samsung S7 or something boring of the sort, but then I decided to tackle this issue once more, following some of the more obscure instructions out there, trying to get vanilla Signal working on my phone. About a day later, I got it, and this is how I did it.

microG So I need Google Play Services somehow, but installing the real thing did not seem to be very promising (I tried, and regularly got pop-ups telling me that Play Services has crashed.) But I found some references to a project called microG , which is an independent re-implementation of (some of) of the play services, in particular including GCM. Installing microG itself was easy, as you can add their repository to F-Droid. I installed the core services, the services framework and the fake store apps. If this had been all that was to do, things would be easy!

Play Store detection work arounds But Signal would still complain about the lack of Google Play Services. It asks Android if an app with a certain name is installed, and would refuse to work if this app does not exist. For some reason, the microG apps cannot just have the names of the real Google apps. There seem to be two ways of working around this: Patching Signal, or enabling Signature Spoofing. The initially most promising instructions (which are in a README in a tarball on a fishy file hoster linked from an answer on the Jolla support forum ) suggested patching Signal, and actually came both with a version of an app called Lucky Patcher as well as a patched Android package, but both about two years old. I tried a recent version of the Lucky Patcher, but it failed to patch the current version of Signal.

Signature Spoofing So on to Signature Spoofing. This is a feature of some non-standard Android builds that allow apps (such as microG) to fake the existence of other apps (the Play Store), and is recommended by the microG project. Sailfish OS s Android compatibility layer Alien Dalvik does not support it out of the box, but there is a tool tingle that adds this feature to existing Android systems. One just has to get the /system/framework/framework.jar file, put it into the input folder of this project, run python main.py, select 2, and copy the framework.jar from output/ back. Great.

Deodexing Alien Dalvik Only that it only works on deodexed files. I did not know anything about odexed Android Java classes (and did not really want to know), but there was not way around. Following this explanation I gathered that one finds files foo.odex in the Android system folder, runs some tool on them to create a classes.dex file, and adds that to the corresponding foo.jar or foo.apk file, copies this back to the phone and deletes the foo.odex file. The annoying this is that one does not only have to do it for framework.jar in order to please tingle, because if one does it to one odex file, one has to do to all! It seems that for people using Windows, the Universal Deodexer V5 seems to be a convenient tool, but I had to go more manually. So I first fetched smali , compiled it using ./gradlew build. Then I fetched the folders /opt/alien/system/framework and /opt/alien/system/app from the phone (e.g. using scp). Keep a backup of these in case something breaks. Then I ran these commands (disclaimer: I fetched these from my bash history and slightly cleaned them up. This is not a fire-and-forget script! Use it when you know what it and you are doing):
cd framework
for file in *.odex
do
  java -jar ~/build/smali/baksmali/build/libs/baksmali.jar deodex $file -o out
  java -jar ~/build/smali/smali/build/libs/smali.jar a out -o classes.dex
  zip -u $(basename $file .odex).jar classes.dex
  rm -rf out classes.dex $file
done
cd ..
cd app
for file in *.odex
do
  java -jar ~/build/smali/baksmali/build/libs/baksmali.jar deodex -d ../framework $file -o out
  java -jar ~/build/smali/smali/build/libs/smali.jar a out -o classes.dex
  zip -u $(basename $file .odex).apk classes.dex
  rm -rf out classes.dex $file
done
cd ..
The resulting framework.jar can now be patched with tingle:
mv framework/framework.jar ~/build/tingle/input
cd ~/build/tingle
./main.py
# select 2
cd -
mv ~/build/tingle/output/framework.jar framework/framework.jar
Now I copy these framework and app folders back on my phone, and restart Dalvik:
devel-su systemctl restart aliendalvik.service
It might start a bit slower than usually, but eventually, all the Android apps should work as before. The final bit that was missing in my case was that I had to reinstall Signal: If it is installed before microG is installed, it does not get permission to use GCM, and when it tries (while registering: After generating the keys) it just crashes. I copied /data/data/org.thoughtcrime.secretsms/ before removing Signal and moved it back after (with cp -a to preserve permissions) so that I could keep my history. And now, it seems, vanilla Signal is working just fine on my Jolla phone!

What s missing Am I completely happy with Signal? No! An important feature that it is lacking is a way to get out all data (message history including media files) in a file format that can be read without Signal; e.g. YAML files or clean HTML code. I do want to be able to re-read some of the more interesting conversations when I am 74 or 75, and I doubt that there will be a Signal App, or even Android, then. I hope that this becomes available in time, maybe in the Desktop version. I would also hope that pidgin gets support to the Signal protocol, so that I conveniently use one program for all my messaging needs on the desktop. Finally it would be nice if my Signal identity was less tied to one phone number. I have a German and a US phone number, and would want to be reachable under both on all my clients. (If you want to contact me on Signal, use my US phone number.)

Alternatives Could I have avoided this hassle by simply convincing people to use something other than Signal? Tricky, at the moment. Telegram (which works super reliable for me, and has a pidgin plugin) has dubious crypto and does not support crypto while using multiple clients. Threema has no desktop client that I know of. OTR on top of Jabber does not support offline messages. So nothing great seems to exist right now. In the long run, the best bet seems to be OMEMO (which is, in essence, the Signal protocol) on top of Jabber. It is currently supported by one Android Jabber client (Conversations) and one Desktop application (gajim, via a plugin). I should keep an eye on pidgin support for OMEMO and other development around this.

Tanguy Ortolo: Generate man pages for awscli

No man pages, but almost The AWS Command Line Interface, which is available in Debian, provides no man page. Instead, that tool has an integrated help system, which allows you to run commands such as aws rds help, that, for what I have seen, generates some reStructuredText, then converts it to a man page in troff format, then calls troff to convert it to text with basic formatting, and eventually passes it to a pager. Since this is close to what man does, the result looks like a degraded man page, with some features missing such as the adaptation to the terminal width. Well, this is better than nothing, and better than what many under-documented tools can offer, but for several reasons, it still sucks: most importantly, it does not respect administrators' habits and it does not integrate with the system man database. You it does not allow you to use commands such as apropos, and you will get no man page name auto-completion from your shell since there is no man page.Generate the man pages Now, since the integrated help system does generate a man page internally, we can hack it to output it, and save it to a file:
Description: Enable a mode to generate troff man pages
 The awscli help system internally uses man pages, but only to convert
 them to text and show them with the pager. This patch enables a mode
 that prints the troff code so the user can save the man page.
 .
 To use that mode, run the help commands with an environment variable
 OUTPUT set to 'troff', for instance:
     OUTPUT='troff' aws rds help
Forwarded: no
Author: Tanguy Ortolo <tanguy+debian@ortolo.eu>
Last-Update: 2016-11-22
Index: /usr/lib/python3/dist-packages/awscli/help.py
===================================================================
--- /usr/lib/python3/dist-packages/awscli/help.py       2016-11-21 12:14:22.236254730 +0100
+++ /usr/lib/python3/dist-packages/awscli/help.py       2016-11-21 12:14:22.236254730 +0100
@@ -49,6 +49,8 @@
     Return the appropriate HelpRenderer implementation for the
     current platform.
     """
+    if 'OUTPUT' in os.environ and os.environ['OUTPUT'] == 'troff':
+        return TroffHelpRenderer()
     if platform.system() == 'Windows':
         return WindowsHelpRenderer()
     else:
@@ -97,6 +99,15 @@
         return contents
+class TroffHelpRenderer(object):
+    """
+    Render help content as troff code.
+    """
+
+    def render(self, contents):
+        sys.stdout.buffer.write(publish_string(contents, writer=manpage.Writer()))
+
+
 class PosixHelpRenderer(PagingHelpRenderer):
     """
     Render help content on a Posix-like system.  This includes
This patch must be applied from the root directory with patch -p0, otherwise GNU patch will not accept to work on files with absolute names. With that patch, you can run help commands with an environment variable OUTPUT='troff' to get the man page to use it as you like, for instance:
% OUTPUT='troff' aws rds help > aws_rds.1
% man -lt aws_rds.1   lp
Generate all the man pages Now that we are able to generate the man page of any aws command, all we need to generate all of them is a list of all the available commands. This is not that easy, because the commands are somehow derived from functions provided by a Python library named botocore, which are derived from a bunch of configuration files, and some of them are added, removed or renamed. Anyway, I have been able to write a Python script that does that, but it includes a static list of these modifications:
#! /usr/bin/python3
import subprocess
import awscli.clidriver
def write_manpage(command):
    manpage = open('%s.1' % '_'.join(command), 'w')
    command.append('help')
    process = subprocess.Popen(command,
            env= 'OUTPUT': 'troff' ,
            stdout=manpage)
    process.wait()
    manpage.close()
driver = awscli.clidriver.CLIDriver()
command_table = driver._get_command_table()
renamed_commands = \
     
        'config': 'configservice',
        'codedeploy': 'deploy',
        's3': 's3api'
     
added_commands = \
     
        's3': ['cp', 'ls', 'mb', 'mv', 'presign', 'rb', 'rm', 'sync',
               'website']
     
removed_subcommands = \
     
        'ses': ['delete-verified-email-address',
                'list-verified-email-addresses',
                'verify-email-address'],
        'ec2': ['import-instance', 'import-volume'],
        'emr': ['run-job-flow', 'describe-job-flows',
                'add-job-flow-steps', 'terminate-job-flows',
                'list-bootstrap-actions', 'list-instance-groups',
                'set-termination-protection',
                'set-visible-to-all-users'],
        'rds': ['modify-option-group']
     
added_subcommands = \
     
        'rds': ['add-option-to-option-group',
                'remove-option-from-option-group']
     
# Build a dictionary of real commands, including renames, additions and
# removals.
real_commands =  
for command in command_table:
    subcommands = []
    subcommand_table = command_table[command]._get_command_table()
    for subcommand in subcommand_table:
        # Skip removed subcommands
        if command in removed_subcommands \
                and subcommand in removed_subcommands[command]:
            continue
        subcommands.append(subcommand)
    # Add added subcommands
    if command in added_subcommands:
        for subcommand in added_subcommands[command]:
            subcommands.append(subcommand)
    # Directly add non-renamed commands
    if command not in renamed_commands:
        real_commands[command] = subcommands
    # Add renamed commands
    else:
        real_commands[renamed_commands[command]] = subcommands
# Add added commands
for command in added_commands:
    real_commands[command] = added_commands[command]
# For each real command and subcommand, generate a manpage
write_manpage(['aws'])
for command in real_commands:
    write_manpage(['aws', command])
    for subcommand in real_commands[command]:
        write_manpage(['aws', command, subcommand])
                         'sync', 'website'] 
This script will generate more than 2,000 man page files in the current directory; you will then be able to move them to /usr/local/share/man/man1. Since this is a lot of man pages, it may be appropriate to concatenate them by major command, for instance all the aws rds together

22 November 2016

Lars Wirzenius: Debian miniconf in Cambridge

I spent a few days in Cambridge for a minidebconf. This is a tiny version of the full annual Debconf. We had a couple of days for hacking, and another two days for talks. I spent my hacking time on thinking about vmdebootstrap (my tool for generating disk images with an installed Debian), and came to the conclusion I need to atone my sins for writing such crappy code by rewriting it from scratch to be nicer to use. I gave a talk about this, too. The mailing list post has the important parts, and meetings-archive has a video. I haven't started the rewrite, and it's not going to make it for stretch. I also gave two other talks, on the early days of Linux, and Qvarn, the latter being what I do at work. Thank you to ARM, for sponsoring the location, and the other sponsors for sponsoring food. These in-real-life meetings between developers are important for the productivity and social cohesion of Debian.

Steve Langasek: A new chapter

I don't often write on this blog, and when I do, it's either tech related, or light life stuff. Over the next few weeks, it's going to get a lot more political. If you currently follow this blog for its technical content, you may be tempted to tune out. I would encourage you to stay and listen. I'm passionate about the technology that I work on; but the greatest problems facing our world today are not ones that will be solved with software. American democracy is in bad shape, and it's because of what we're doing to it. This is not a problem of the Right or of the Left; it is not a problem that began with the election of Donald Trump, and it's not a problem that will go away at the end of his term. It is partly a structural problem with the way our elections work, but more than that it's a problem of how we're splitting into separate tribes, isolating ourselves from those who don't agree with us. As Russ Allbery wrote the morning after the election, everything about how we organize ourselves online today - and how we let Facebook and Twitter organize us - leads us to surround ourselves with people who already think the same way we do. That leaves all of us with huge blind spots for other people in our country, and it stifles the free exchange of ideas that is so essential for a healthy democracy. We need leaders who will work to make America a better and more just place for all our neighbors, not just a two-party system that plays tug-of-war using two different sets of voters that feel shut out. And the way we organize ourselves today (online and off) does not let us recognize those leaders. There's a lot of talk now about Facebook changing how it decides what to show people; and maybe they can manage to help everyone's online experience be a little less of a bubble. But part of the change needs to come from us. We need to be willing to engage, civilly, with people whose perspective is different from ours, and make the effort to understand where the other is coming from. So for the next few weeks, I'm going to talk. And I'm going to listen. I have no unique qualifications to speak about the country's issues. But I do have a perspective of my own, which might be different enough from yours to be useful. I was born and raised in Iowa, and graduated from college there. This election cycle, I learned that Iowa holds the distinction of being the state with the lowest percentage of college-educated whites. I'm part of that statistic, because a few years after graduating I moved to Portland, Oregon - a place that's notoriously so far to the left of what we think of as the middle, that it actually has anarchists who would shamefully use a peaceful protest as cover to commit property crime. So I know a few things about the people in each state, that I think the other should hear. I'm also that rarest of creatures, a Portlander who goes to church (Catholic). But I still choose as my neighbors the weird, wonderful, and welcoming community that we have here, whatever Glenn Beck might think. I have a son, and I worry about what kind of world he'll grow up to live in. I work in software, which means I'm doing a lot better than a lot of people in the country right now; it also means that from where I sit, I see trends already in progress that will have an effect on the working class and the middle class that makes NAFTA look like a gnat's fart in comparison. And so I worry for what kind of world we will all live in, if we don't make some changes fast. Let's have a conversation. No comments enabled on this blog, but you can find me on G+ or on Facebook.

21 November 2016

Mike Gabriel: Please Welcome D0n1elT to the FLOSS World

TL;DR; If you run a FLOSS development project and you notice D0n1elT appearing on your IRC channel, please give him a warm welcome. D0n1elT is a young man highy talented in various FLOSS related topics already. He probably needs some guidance at the beginning and I hope he won't be too shy to ask for it. But you can be sure: your channel has been joined by someone you should consider as a future resource. The Long Story During the last two weeks I had the great pleasure of supervising a fine young man (very young, still, indeed) in all sorts of IT topics. This young man turned out to be so skilled and interested in various FLOSS related areas, I really want to introduce him to all of you. The young man's real name is Daniel Teichmann. On IRC he may appear under his nick: D0n1elT. His GnuPG Fingerprint is: 6C6E 7F8F F7E8 B22E FC76 E9F7 8A79 028F DA56 7C6C. Daniel goes to a local school here in Nothern Germany, near where I live. He attends the 9th grade at his school, and as common for students of his age and grade, practical training was scheduled for the last two weeks. Daniel had originally applied for practical training at some other business near his place of living (which is quite far off from the school, actually). However, that company cancelled his training position two work days before the training was supposed to start. Daniel's teacher rang me up and asked for help. He advertised Daniel as someone who is far advanced in IT topics compared to his co-students. "He even writes his own programs (in Java and C++)." Spontaneously, Andreas Buchholz (CEO of LOGO EDV-Systeme GmbH) and I decided to accept Daniel as a trainee. Without having met him, with no application interview beforehand. The deal was: Daniel comes to Andreas business location in Kiel (40-50km away from Daniel's place of living) and I (working as freelancer for LOGO on a regular basis) do the supervising part. On day one and two, as a warm-up, Daniel installed a Debian Edu Main Server, worked himself through GOsa, LDAP, SSH, GnuPG, Jabber and IRC and configured two routers. All topics were new to him and I could hardly think of new tasks to give to him. As means of communication we set up a Jabber account, then an IRC account (as backup). However, it turned out that Daniel really got a hang of IRC over the next couple of days, so we used that as primary communication channel. Daniel had already programmed various projects in Java (whereas I have never touched Java, so far :-( ). He has written plugins for Minecraft servers. He knows well how to implement object oriented coding models. His coding style looks very good and clean (esp. for someone who has never head a nitpicking code reviewer). He started coding at the age of 9. Instead of diving into Java (where I would not have been of much help, anyway) I decided to provide him with some really basic and Unix-like knowledge: Bash scripting. I wanted to see how he handles another "language" and how he applies his Java knowledge to a lower level, syntactically weaker language. Guess what, he managed that assignment very well. Working on Impressive Display At Daniel's school we run substitute teacher info screens based on a fancy Bash script, named impressive-display, and the impressive PDF viewer. The Impressive Display tool is available in Debian testing/unstable under the same name. So over the next couple of days we worked on Impressive Display. Daniel contributed so many new code passages, conceptual ideas and security concerns, that I decided to make him co-copyright holder. Every change contributed by him received intensive testing before committing to Git. While working on Impressive Display, collaborating with Daniel via Git was a mere pleasure. In his spare time Daniel likes watching Github tutorials. Quite extraordinary. The result is a new major release of Impressive Display: Version 0.3.1 (bumped up from 0.2.3). We added the feature of handling info screen farms based on PXE boot images. It is now possible to configure as many different info screens as needed within the same PXE bootable chroot. Furthermore, Impressive Display now has a PDF presentation (written in LaTeX Beamer) that documents how to setup your own info screens. The PDF presentation is the default PDF that comes up when you start Impressive Display directly after installation. Investigating other Realms We also took a deeper look at remote desktop stuff, one of my most favourite topics. By that impulse Daniel set up his first Vserver machine at some hosting provider. He figured out how to run X2Go Server on that machine with an XFCE desktop. Next step was to run the irssi instance from his notebook inside a screen session on the Vserver. Some days later, Daniel PM'ed me: "I have an IRC bouncer now...". Quintessence It was a great pleasure meeting this young, highly curious and already highly skilled young man over the past two weeks. Daniel, it was an asset to me working with you. You are such a fast learner when it comes to getting accustomed to new working environments, it is amazing. I cannot deny having observed the tendency of preferring rather geeky tools. I was highly delighted, that What's-That and Facebook are nothing that rocks you so much. Unfortunately, all of the above makes you quite unique and non-mainstream among people of your age. My wish for you (and the FLOSS world) is that you start getting in touch with other (FLOSS) developers, maybe of your age, maybe older, and that you (if this is what you want) become an asset to the world of Free Software. The Free Software world can be a world where technical, political and spiritual work become one with friendship among people. Take care and farewell! I am sure, we will meet again. light+love Mike Gabriel (aka sunweaver on IRC and debian.org)

Gustavo Noronha Silva: A tale of cylinders and shadows

Like I wrote before, we at Collabora have been working on improving WebKitGTK+ performance for customer projects, such as Apertis. We took the opportunity brought by recent improvements to WebKitGTK+ and GTK+ itself to make the final leg of drawing contents to screen as efficient as possible. And then we went on investigating why so much CPU was still being used in some of our test cases. The first weird thing we noticed is performance was actually degraded on Wayland compared to running under X11. After some investigation we found a lot of time was being spent inside GTK+, painting the window s background. Here s the thing: the problem only showed under Wayland because in that case GTK+ is responsible for painting the window decorations, whereas in the X11 case the window manager does it. That means all of that expensive blurring and rendering of shadows fell on GTK+ s lap. During the web engines hackfest, a couple of months ago, I delved deeper into the problem and noticed, with Carlos Garcia s help, that it was even worse when HiDPI displays were thrown into the mix. The scaling made things unbearably slower. You might also be wondering why would painting of window decorations be such a problem, anyway? They should only be repainted when a window changes size or state anyway, which should be pretty rare, right? Right, that is one of the reasons why we had to make it fast, though: the resizing experience was pretty terrible. But we ll get back to that later. So I dug into that, made a few tries at understanding the issue and came up with a patch showing how applying the blur was being way too expensive. After a bit of discussion with our own Pekka Paalanen and Benjamin Otte we found the root cause: a fast path was not being hit by pixman due to the difference in scale factors on the shadow mask and the target surface. We made the shadow mask scale the same as the surface s and voil , sane performance. I keep talking about this being a performance problem, but how bad was it? In the following video you can see how huge the impact in performance of this problem was on my very recent laptop with a HiDPI display. The video starts with an Epiphany window running with a patched GTK+ showing a nice demo the WebKit folks cooked for CSS animations and 3D transforms. After a few seconds I quickly alt-tab to the version running with unpatched GTK+ I made the window the exact size and position of the other one, so that it is under the same conditions and the difference can be seen more easily. It is massive. Yes, all of that slow down was caused by repainting window shadows! OK, so that solved the problem for HiDPI displays, made resizing saner, great! But why is GTK+ repainting the window even if only the contents are changing, anyway? Well, that turned out to be an off-by-one bug in the code that checks whether the invalidated area includes part of the window decorations. If the area being changed spanned the whole window width, say, it would always cause the shadows to be repainted. By fixing that, we now avoid all of the shadow drawing code when we are running full-window animations such as the CSS poster circle or gtk3-demo s pixbufs demo. As you can see in the video below, the gtk3-demo running with the patched GTK+ (the one on the right) is using a lot less CPU and has smoother animation than the one running with the unpatched GTK+ (left). Pretty much all of the overhead caused by window decorations is gone in the patched version. It is still using quite a bit of CPU to animate those pixbufs, though, so some work still remains. Also, the overhead added to integrate cairo and GL rendering in GTK+ is pretty significant in the WebKitGTK+ CSS animation case. Hopefully that ll get much better from GTK+ 4 onwards.

Reproducible builds folks: Reproducible Builds: week 82 in Stretch cycle

What happened in the Reproducible Builds effort between Sunday November 13 and Saturday November 19 2016: Media coverage Elsewhere in Debian Documentation update Packages reviewed and fixed, and bugs filed Reviews of unreproducible packages 43 package reviews have been added, 4 have been updated and 12 have been removed in this week, adding to our knowledge about identified issues. 2 issue types have been updated: 4 issue types have been added: Weekly QA work During our reproducibility testing, some FTBFS bugs have been detected and reported by: strip-nondeterminism development disorderfs development debrebuild development debrebuild is new tool proposed by HW42 and josch (see #774415: "From srebuild sbuild-wrapper to debrebuild"). debrepatch development debrepatch is a set of scripts that we're currently developing to make it easier to track unapplied patches. We have a lot of those and we're not always sure if they still work. The plan is to set up jobs to automatically apply old reproducibility patches to newer versions of packages and notify the right people if they don't apply and/or no longer make the package reproducible. debpatch is a component of debrepatch that applies debdiffs to Debian source packages. In other words, it is to debdiff(1) what patch(1) is to diff(1). It is a general tool that is not specific to Reproducible Builds. This week, Ximin Luo worked on making it more "production-ready" and will soon submit it for inclusion in devscripts. reprotest development Ximin Luo significantly improved reprotest, adding presets and auto-detection of which preset to use. One can now run e.g. reprotest auto . or reprotest auto $pkg_$ver.dsc instead of the long command lines that were needed before. He also made it easier to set up build dependencies inside the virtual server and made it possible to specify pre-build dependencies that reprotest itself needs to set up the variations. Previously one had to manually edit the virtual server to do that, which was not very usable to humans without an in-depth knowledge of the building process. These changes will be tested some more and then released in the near future as reprotest 0.4. tests.reproducible-builds.org Misc. This week's edition was written by Chris Lamb, Holger Levsen, Ximin Luo and reviewed by a bunch of Reproducible Builds folks on IRC.

20 November 2016

Steinar H. Gunderson: Nageru documentation

Even though the World Chess Championship takes up a lot of time these days, I've still found some time for Nageru, my live video mixer. But this time it doesn't come in form of code; rather, I've spent my time writing documentation. I spent some time fretting over what technical solution I wanted. I explicitly wanted end-user documentation, not developer documentation I rarely find HTML-rendered versions of every member function in a class the best way to understand a program anyway. Actually, on the contrary: Having all sorts of syntax interwoven in class comments tends to be more distracting than anything else. Eventually I settled on Sphinx, not because I found it fantastic (in particular, ReST is a pain with its bizarre variable punctuation-based syntax), but because I'm convinced it has all the momentum right now. Just like git did back in the day, the fact that the Linux kernel has chosen it means it will inevitably grow a quite large ecosystem, and I won't be ending up having to maintain it anytime soon. I tried finding a balance between spending time on installation/setup (only really useful for first-time users, and even then, only a subset of them), concept documentation (how to deal with live video in general, and how Nageru fits into a larger ecosystem of software and equipment) and more concrete documentation of all the various features and quirks of Nageru itself. Hopefully, most people will find at least something that's not already obvious to them, without drowning in detail. You can read the documentation at https://nageru.sesse.net/doc/, or if you want to send patches, the right place to patch is the git repository.

19 November 2016

Dirk Eddelbuettel: BH 1.62.0-1

The BH package on CRAN was updated to version 1.62.0. BH provides a large part of the Boost C++ libraries as a set of template headers for use by R, possibly with Rcpp as well as other packages. This release upgrades the version of Boost to the upstream version Boost 1.62.0, and adds three new libraries as shown in the brief summary of changes from the NEWS file which follows below. Special thanks to Kurt Hornik and Duncan Murdoch for help tracking down one abort() call which was seeping into R package builds, and then (re-)testing the proposed fix. We are now modifying one more file ever so slightly to use ::Rf_error(...) instead.

Changes in version 1.62.0-1 (2016-11-15)
  • Upgraded to Boost 1.62 installed directly from upstream source
  • Added Boost property_tree as requested in #29 by Aydin Demircioglu
  • Added Boost scope_exit as requested in #30 by Kirill Mueller
  • Added Boost atomic which we had informally added since 1.58.0

Courtesy of CRANberries, there is also a diffstat report for the most recent release. Comments and suggestions are welcome via the mailing list or the issue tracker at the GitHubGitHub repo.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Keith Packard: AltOS-Lisp-2

Updates to Altos Lisp I wrote a few days ago about a tiny lisp interpreter I wrote for AltOS Really, it's almost "done" now, I just wanted to make a few improvements Incremental Collection I was on a walk on Wednesday when I figured out that I didn't need to do a full collection every time; a partial collection that only scanned the upper portion of memory would often find plenty of free space to keep working for a while. To recap, the heap is in two pieces; the ROM piece and the RAM piece. The ROM piece is generated during the build process and never changes afterwards (hence the name), so the only piece which is collected is the RAM piece. Collection works like:
chunk_low = heap base
new_top = heap base
For all of the heap
    Find the first 64 live objects above chunk_low
    Compact them all to new_top
    Rewrite references in the whole heap for them
    Set new_top above the new locations
    Set chunk_low above the old locations
top = new_top
The trick is to realize that there's really no need to start at the bottom of the heap; you can start anywhere you like and compact stuff, possibly leaving holes below that location in the heap. As the heap tends to have long-lived objects slowly sift down to the beginning, it's useful to compact objects higher than that, skipping the compaction process for the more stable area in memory. Each time the whole heap is scanned, the top location is recorded. After that, incremental collects happen starting at that location, and when that doesn't produce enough free space, a full collect is done. The collector now runs a bunch faster on average now. Binary Searches I stuck some linear searches in a few places in the code, the first was in the collector when looking to see where an object had moved to. As there are 64 entries, the search is reduced from 32 to 6 compares on average. The second place was in the frame objects, which hold the list of atom/value bindings for each lexical scope (including the global scope). These aren't terribly large, but a binary search is still a fine plan. I wanted to write down here the basic pattern I'm using for binary searches these days, which avoids some of the boundary conditions I've managed to generate in the past:
int find (needle)  
    int l = 0;
    int r = count - 1;
    while (l <= r)  
        int m = (l + r) >> 1;
        if (haystack[m] < needle)
            l = m + 1;
        else
            r = m - 1;
     
    return l;
 
With this version, the caller can then check to see if there's an exact match, and if not, then the returned value is the location in the array where the value should be inserted. If the needle is known to not be in the haystack, and if the haystack is large enough to accept the new value:
void insert(needle)  
    int l = find(needle);
    memmove(&haystack[l+1],
        &haystack[l],
        (num - l) * sizeof (haystack[0]));
    haystack[l] = needle;
 
Similarly, if the caller just wants to know if the value is in the array:
bool exists(needle)  
    int l = find(needle);
    return (l < count && haystack[l] == needle);
 
Call with Current Continuation Because the execution stack is managed on the heap, it's completely trivial to provide the scheme-like call with current continuation, which constructs an object which can be 'called' to transfer control to a saved location:
> (+ "hello " (call/cc (lambda (return) (setq boo return) (return "foo "))) "world")
"hello foo world"
> (boo "bar ")
"hello bar world"
> (boo "yikes ")
"hello yikes world"
One thing I'd done previously is dump the entire state of the interpreter on any error, and that included a full stack trace. I adopted that code for printing of these continuation objects:
boo
[
    expr:   (call/cc (lambda (return) (set (quote boo) return) (return "foo ")))
    state:  val
    values: (call/cc
             [recurse...]
             )
    sexprs: ()
    frame:   
]
[
    expr:   (+ "hello " (call/cc (lambda (return) (set (quote boo) return) (return "foo "))) "world")
    state:  formal
    values: (+
             "hello "
             )
    sexprs: ((call/cc (lambda (return) (set (quote boo) return) (return "foo ")))
             "world"
             )
    frame:   
]
The top stack frame is about to return from the call/cc spot with a value; supply a value to 'boo' and that's where you start. The next frame is in the middle of computing formals for the + s-expression. It's found the + function, and the "hello " string and has yet to get the value from call/cc or the value of the "world" string. Once the call/cc "returns", that value will get moved to the values list and the sexpr list will move forward one spot to compute the "world" value. Implementing this whole mechanism took only a few dozen lines of code as the existing stack contexts were already a continuation in effect. The hardest piece was figuring out that I needed to copy the entire stack each time the continuation was created or executed as it is effectively destroyed in the process of evaluation. I haven't implemented dynamic-wind yet; when I did that for nickle, it was a bit of a pain threading execution through the unwind paths. Re-using Frames I decided to try and re-use frames (those objects which hold atom/value bindings for each lexical scope). It wasn't that hard; the only trick was to mark frames which have been referenced from elsewhere as not-for-reuse and then avoid sticking those in the re-use queue. This reduced allocations even further so that for simple looping or tail-calling code, the allocator may never end up being called. How Big Is It? I've managed to squeeze the interpreter and all of the rest of the AltOS system into 25kB of Cortex-M0 code. That leaves space for the 4kB boot loader and 3kB of flash to save/restore the 3kB heap across resets. Adding builtins to control timers and GPIOs would make this a reasonable software load for an Arduino; offering a rather different programming model for those with a taste for adventure. Modern ARM-based Arduino boards have plenty of flash and ram for this. It might be interesting to get this running on the Arduino Zero; there's no real reason to replace the OS either; porting the lisp interpreter into the bare Arduino environment wouldn't take long.

18 November 2016

Dirk Eddelbuettel: Rcpp 0.12.8: And more goodies

Yesterday the eighth update in the 0.12.* series of Rcpp made it to the CRAN network for GNU R where the Windows binary has by now been generated too; the Debian package is on its way as well. This 0.12.8 release follows the 0.12.0 release from late July, the 0.12.1 release in September, the 0.12.2 release in November, the 0.12.3 release in January, the 0.12.4 release in March, the 0.12.5 release in May, the 0.12.6 release in July, and the 0.12.7 release in September --- making it the twelveth release at the steady bi-montly release frequency. While we are keeping with the pattern, we have managed to include quite a lot of nice stuff in this release. None of it is a major feauture, though, and so we have not increased the middle number. Among the changes this release are (once again) much improved exception handling (thanks chiefly to Jim Hester), better large vector support (by Qiang), a number of Sugar extensions (mostly Nathan, Qiang and Dan) and beginnings of new DateVector and DatetimeVectors classes, and other changes detailed below. We plan to properly phase in the new date(time) classes. For now, you have to use a #define such as this one in Rcpp.h which remains commented-out for now. We plan to switch this on as the new default no earlier than twelve months from now. Rcpp has become the most popular way of enhancing GNU R with C or C++ code. As of today, 843 packages on CRAN depend on Rcpp for making analytical code go faster and further. That is up by eightyfour packages, or a full ten percent, just since the last release in early September! Again, we are lucky to have such a large group of contributors. Among them, we have invited Nathan Russell to the Rcpp Core team given his consistently excellent pull requests (as well as many outstanding Stackoverflow answers for Rcpp). More details on changes are below.

Changes in Rcpp version 0.12.8 (2016-11-16)
  • Changes in Rcpp API:
    • String and vector elements now use extended R_xlen_t indices (Qiang in PR #560)
    • Hashing functions now return unsigned int (Qiang in PR #561)
    • Added static methods eye(), ones(), and zeros() for select matrix types (Nathan Russell in PR #569)
    • The exception call stack is again correctly reported; print methods and tests added too (Jim Hester in PR #582 fixing #579)
    • Variatic macros no longer use a GNU extensions (Nathan in PR #575)
    • Hash index functions were standardized on returning unsigned integers (Also PR #575)
  • Changes in Rcpp Sugar:
    • Added new Sugar functions rowSums(), colSums(), rowMeans(), colMeans() (PR #551 by Nathan Russell fixing #549)
    • Range Sugar now used R_xlen_t type for start/end (PR #568 by Qiang Kou)
    • Defining RCPP_NO_SUGAR no longer breaks the build. (PR #585 by Daniel C. Dillon)
  • Changes in Rcpp unit tests
    • A test for expression vectors was corrected.
    • The constructor test for datetime vectors reflects the new classes which treats Inf correctly (and still as a non-finite value)
  • Changes in Rcpp Attributes
    • An 'empty' return was corrected (PR #589 fixing issue #588, and with thanks to Duncan Murdoch for the heads-up)
  • Updated Date and Datetime vector classes:
    • The DateVector and DatetimeVector classes were renamed with a prefix old; they are currently typedef'ed to the existing name (#557)
    • New variants newDateVector and newDatetimeVector were added based on NumericVector (also #557, #577, #581, #587)
    • By defining RCPP_NEW_DATE_DATETIME_VECTORS the new classes can activated. We intend to make the new classes the default no sooner than twelve months from this release.
    • The capabilities() function can also be used for presence of this feature

Thanks to CRANberries, you can also look at a diff to the previous release. As always, even fuller details are on the Rcpp Changelog page and the Rcpp page which also leads to the downloads page, the browseable doxygen docs and zip files of doxygen output for the standard formats. A local directory has source and documentation too. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

17 November 2016

Reproducible builds folks: Reproducible Builds: week 81 in Stretch cycle

What happened in the Reproducible Builds effort between Sunday November 6 and Saturday November 12 2016: Media coverage Matthew Garrett blogged about Tor, TPMs and service integrity attestation and how reproducible builds are the base for systems integrity. The Linux Foundation announced renewed funding for us as part of the Core Infrastructure Initiative. Thank you! Outreachy updates Maria Glukhova has been accepted into the Outreachy winter internship and will work with us the Debian reproducible builds team. To quote her words
siamezzze: I've been accepted to #outreachy winter internship - going to
work with Debian reproducible builds team. So excited about that! <3
Debian
Toolchain development and fixes dpkg: debrebuild: Bugs filed Chris Lamb: Daniel Shahaf: Niko Tyni: Reiner Herrman: Reviews of unreproducible packages 136 package reviews have been added, 5 have been updated and 7 have been removed in this week, adding to our knowledge about identified issues. 3 issue types have been updated: Weekly QA work During of reproducibility testing, some FTBFS bugs have been detected and reported by: diffoscope development A new version of diffoscope 62~bpo8+1 was uploaded to jessie-backports by Mattia Rizzolo. Meanwhile in git, Ximin Luo greatly improved speed by fixing a O(n2) lookup which was causing diffs of large packages such as GCC and glibc to take many more hours than was necessary. When this commit is released, we should hopefully see full diffs for such packages again. Currently we have 197 source packages which - when built - diffoscope fails to analyse. buildinfo.debian.net development tests.reproducible-builds.org Debian: reproducible-builds.org website F-Droid was finally added to our list of partner projects. (This was an oversight and they had already been working with us for some time.) Misc. This week's edition was written by Ximin Luo and Holger Levsen and reviewed by a bunch of Reproducible Builds folks on IRC.

Urvika Gola: Reaching out to Outreachy

The past few weeks have been a whirlwind of work with my application process for Outreachy taking my full attention. When I got to know about Outreachy, I was intrigued as well as dubious. I had many unanswered questions in my mind. Honestly, I had that fear of failure which prevented me from submitting my partially filled application form. I kept contemplating if the application was good enough , if my answers were perfect and had the right balance of du-uh and oh-ah!
In moments of doubt, it is important to surround yourself with people who believe in you more than you believe in yourself. I ve been fortunate enough to have two amazing people, my sister Anjali, who is an engineer at Intel and my friend, Pranav Jain who completed his GSoC 16 with Debian.
They believed in me when I sat staring at my application and encouraged me to click that final button. When I initially applied for Outreachy, I was given a task for building Lumicall and subsequent task was to examine a BASH script which solves the DNS-01 Challenge.
I deployed the DNS-01 challenge in Java and tested my solution against a server.
Within a limited time frame, I figured things out and wrote my solution in Java and then eagerly waited for the results to come out. Going through a full cycle of : lifecycle.JPG I was elated with joy when I got to know I ve been selected for Outreachy to work with Debian. I was excited about open source & found the idea of working on the project open source fun because of the numerous possibilities of contributing towards a voice video and chat communication software.
My project mentor, Daniel Pocock, played a pivotal role in the time after I had submitted my application. Like a true mentor, he replied to my queries promptly and guided me towards finding the solutions to problems on my own. He exemplified how to feel comfortable with developing on open source. I felt inspired and encouraged to move along in my work. Beyond him, The MiniDebConf was when I was finally introduced to the Debian community. It was an overwhelming experience and I felt proud to have come so far.. It was pretty cool to see JitsiMeet being used for this video call. I was also introduced to two of my mentors , Juliana Louback & Bruno Magalh es . I am very excited to learn from them.

I am glad I applied for Outreachy which helped me identify my strengths and I am totally excited to be working with Debian on the project and learn as much as I can throughout the period. I am not a blog person, this is my first blog ever! I would love to share my experience with you all in the hopes of inspiring someone else who is afraid of clicking that final button!

Next.

Previous.