Search Results: "rmh"

Reproducible builds folks: Reproducible Builds: week 82 in Stretch cycle

What happened in the Reproducible Builds effort between Sunday November 13 and Saturday November 19 2016: Media coverage

Chris Lamb and Holger Levsen presented a status update at MiniDebConf Cambridge. (Video, Slides).

Elsewhere in Debian

dpkg 1.18.14 has migrated to stretch.
Chris Lamb filed #844431 ("packages should build reproducibly") against debian-policy.
Ximin worked on glibc reproducibility this week, catching some bugs in disorderfs, FUSE, as well as glibc itself.

Documentation update

Webconverger was added to our list of projects working on reproducible builds..

Packages reviewed and fixed, and bugs filed

#844491 filed against linux by Chris Lamb.
#844992 filed against perlbrew by Chris Lamb.
#844993 filed against qsynth by Chris Lamb.
#845034 filed against initramfs-tools by Chris Lamb.
#845055 filed against markdown by Chris Lamb.
#844752 filed against perl by Niko Tyni.
#845007 filed against qsampler by Reiner Herrmann.
#845008 filed against qmidinet by Reiner Herrmann.
#845010 filed against qxgedit by Reiner Herrmann.

Reviews of unreproducible packages 43 package reviews have been added, 4 have been updated and 12 have been removed in this week, adding to our knowledge about identified issues. 2 issue types have been updated:

4 issue types have been added:

Weekly QA work During our reproducibility testing, some FTBFS bugs have been detected and reported by:

Chris Lamb (26)
Daniel Stender (1)
Filip Pytloun (1)
Lucas Nussbaum (28)
Michael Biebl (1)

strip-nondeterminism development

Don't make tests rely on Debian::Debhelper::Dh_Lib. (Chris Lamb)

disorderfs development

#844498 ("disorderfs: using it for building kills the host")

debrebuild development debrebuild is new tool proposed by HW42 and josch (see #774415: "From srebuild sbuild-wrapper to debrebuild"). debrepatch development debrepatch is a set of scripts that we're currently developing to make it easier to track unapplied patches. We have a lot of those and we're not always sure if they still work. The plan is to set up jobs to automatically apply old reproducibility patches to newer versions of packages and notify the right people if they don't apply and/or no longer make the package reproducible. debpatch is a component of debrepatch that applies debdiffs to Debian source packages. In other words, it is to debdiff(1) what patch(1) is to diff(1). It is a general tool that is not specific to Reproducible Builds. This week, Ximin Luo worked on making it more "production-ready" and will soon submit it for inclusion in devscripts. reprotest development Ximin Luo significantly improved reprotest, adding presets and auto-detection of which preset to use. One can now run e.g. reprotest auto . or

reprotest
auto $pkg_$ver.dsc

instead of the long command lines that were needed before. He also made it easier to set up build dependencies inside the virtual server and made it possible to specify pre-build dependencies that reprotest itself needs to set up the variations. Previously one had to manually edit the virtual server to do that, which was not very usable to humans without an in-depth knowledge of the building process. These changes will be tested some more and then released in the near future as reprotest 0.4.

Ximin Luo:
- Set uname to two explicitly different values
- Add a --no-diffoscope option. (Closes: #844512)
- Add "auto" presets and --testbed-init
- Add --testbed-pre and --testbed-init arguments
- Fix main script so that build commands starting with ENVVAR=val work after append_command()
- Dramatically expand documentation (many commits)

tests.reproducible-builds.org

Debian:
- An index of our usertagged bugs page was added by Holger after a Q+A session in Cambridge.
- Holger also setup two new i386 builders, build12+16, for >50% increased build performance. For this, we went from 18+17 cores on two 48GB machines to 10+10+9+9 cores on four 36GB ram machines and from 16 to 24 builder jobs. Thanks to Profitbricks for providing us with all these ressources once more!
- h01ger also tried to enable disorderfs again, but hit #844498, which brought down the i386 builders, so he disabled it again. Next will be trying disorderfs on armhf or amd64, to see whether this bug also manifests there.

Misc. This week's edition was written by Chris Lamb, Holger Levsen, Ximin Luo and reviewed by a bunch of Reproducible Builds folks on IRC.

Uwe Kleine-K nig: Installing Debian Stretch on a Turris Omnia

Recently I got "my" Turris Omnia and it didn't take long to replace the original firmware with Debian. If you want to reproduce, here is what you have to do: Open the case of the Turris Omnia, connect the hacker pack (or an RS232-to-TTL adapter) to access the U-Boot prompt (see Turris Omnia: How to use the "Hacker pack"). Then download the installer and device tree:

# cd /srv/tftp
# wget https://d-i.debian.org/daily-images/armhf/daily/netboot/vmlinuz
# wget https://d-i.debian.org/daily-images/armhf/daily/netboot/initrd.gz
# wget https://www.kleine-koenig.org/tmp/armada-385-turris-omnia.dtb

(The latter is not included yet in Debian, but I'm working on that.) and after connecting the Turris Omnia's WAN to a dhcp managed network start it in U-Boot:

dhcp
setenv serverip 192.168.1.17
tftpboot 0x01000000 vmlinuz
tftpboot 0x02000000 armada-385-turris-omnia.dtb
tftpboot 0x03000000 initrd.gz
bootz 0x01000000 0x03000000:$filesize 0x02000000

With 192.168.1.17 being the IPv4 of the machine you have the tftp server running. I suggest to use btrfs as rootfs because that works well with U-Boot. Before finishing the installation put the dtb in the rootfs as /boot/dtb. To then boot into Debian do in U-Boot:

setenv mmcboot=btrload mmc 0 0x01000000 boot/vmlinuz\; btrload mmc 0 0x02000000 boot/dtb\; btrload mmc 0 0x03000000 boot/initrd.img\; bootz 0x01000000 0x03000000:$filesize 0x02000000
setenv bootargs console=ttyS0,115200 rootfstype=btrfs rootdelay=2 root=/dev/mmcblk0p1 rootflags=commit=5 rw
saveenv
boot

Known issues:

rtc doesn't work (workaround: mw 0xf10184a0 0xfd4d4cfa in U-Boot)
SFP and switch don't work, MAC addresses are random
wifi fails to probe

If you have problems, don't hesitate to contact me. Also check the Debian Wiki for further details.

Dimitri John Ledkov: /boot less LVM rootfs in Zesty

On Ubuntu many of the default boot loaders support booting kernels located on LVM volumes. This includes following platforms

i686, x86_64 bios grub2
arm64, armhf, i686, x86_64 UEFI grub2
PReP partitions on IBM PowerPC
zipl on IBM zSystems

For all of the above the d-i has been modified in Zesty to create LVM based installations without a dedicated /boot partition. We shall celebrate this achievement. Hopefully this means one doesn't need to remove kernels as much, or care about sizing /boot volume appropriately any more.

If there are more bootloaders in Ubuntu that support booting off LVM, please do get in touch with me. I'm interested if I can safely enable following platforms as well:

armhf with u-boot
arm64 with u-boot
ppc64el with PReP volume

ps. boots pic is from here

Hideki Yamane: LTS for PowerPC?

Debian9 "Stretch" drops powerpc as release architecture, it means Debian based powerpc box would need more effort to maintain in the future.

If you use Debian for powerpc box in production environment, it maybe better to consider to join LTS funding and ask LTS team to add it to Jessie LTS architecture. I'm not sure they accept it or not, but if it's okay (as they did for armel and armhf), it may be worth to reduce your trouble.

Uwe Kleine-K nig: Installing Debian Stretch on an Omnia Turris

Recently I got "my" Omnia Turris and it didn't take long to replace the original firmware with Debian. If you want to reproduce, here is what you have to do: Open the case of the Omnia Turris, connect the hacker pack (or an RS232-to-TTL adapter) to access the U-Boot prompt (see Turris Omnia: How to use the "Hacker pack"). Then download the installer and device tree:

# cd /srv/tftp
# wget https://d-i.debian.org/daily-images/armhf/daily/netboot/vmlinuz
# wget https://d-i.debian.org/daily-images/armhf/daily/netboot/initrd.gz
# wget https://www.kleine-koenig.org/tmp/armada-385-turris-omnia.dtb

(The latter is not included yet in Debian, but I'm working on that.) and after connecting the Omnia Turris's WAN to a dhcp managed network start it in U-Boot:

dhcp
setenv serverip 192.168.1.17
tftpboot 0x01000000 vmlinuz
tftpboot 0x02000000 armada-385-turris-omnia.dtb
tftpboot 0x03000000 initrd.gz
bootz 0x01000000 0x03000000:$filesize 0x02000000

With 192.168.1.17 being the IPv4 of the machine you have the tftp server running. I suggest to use btrfs as rootfs because that works well with U-Boot. Before finishing the installation put the dtb in the rootfs as /boot/dtb. To then boot into Debian do in U-Boot:

setenv mmcboot=btrload mmc 0 0x01000000 boot/vmlinuz\; btrload mmc 0 0x02000000 boot/dtb\; btrload mmc 0 0x03000000 boot/initrd.img\; bootz 0x01000000 0x03000000:$filesize 0x02000000
setenv bootargs console=ttyS0,115200 rootfstype=btrfs rootdelay=2 root=/dev/mmcblk0p1 rootflags=commit=5 rw
saveenv
boot

Known issues:

rtc doesn't work (workaround: mw 0xf10184a0 0xfd4d4cfa in U-Boot)
SFP and switch don't work, MAC addresses are random
wifi fails to probe

If you have problems, don't hesitate to contact me.

Christoph Egger: Running Debian on the ClearFog

Back in August, I was looking for a Homeserver replacement. During FrOSCon I was then reminded of the Turris Omnia project by NIC.cz. The basic SoC (Marvel Armada 38x) seemed to be nice hand have decent mainline support (and, with the turris, users interested in keeping it working). Only I don't want any WIFI and I wasn't sure the standard case would be all that usefully. Fortunately, there's also a simple board available with the same SoC called ClearFog and so I got one of these (the Base version). With shipping and the SSD (the only 2242 M.2 SSD with 250 GiB I could find, a ADATA SP600) it slightly exceeds the budget but well. ClearFog with SSD

When installing the machine, the obvious goal was to use mainline FOSS components only if possible. Fortunately there's mainline kernel support for the device as well as mainline U-Boot. First attempts to boot from a micro SD card did not work out at all, both with mainline U-Boot and the vendor version though. Turns out the eMMC version of the board does not support any micro SD cards at all, a fact that is documented but others failed to notice as well. U-Boot As the board does not come with any loader on eMMC and booting directly from M.2 requires removing some resistors from the board, the easiest way is using UART for booting. The vendor wiki has some shell script wrapping an included C fragment to feed U-Boot to the device but all that is really needed is U-Boot's kwboot utility. For some reason the SPL didn't properly detect UART booting on my device (wrong magic number) but patching the if (in arch-mvebu's spl.c) and always assume UART boot is an easy way around. The plan then was to boot a Debian armhf rootfs with a defconfig kernel from USB stick. and install U-Boot and the rootfs to eMMC from within that system. Unfortunately U-Boot seems to be unable to talk to the USB3 port so no kernel loading from there. One could probably make UART loading work but switching between screen for serial console and xmodem seemed somewhat fragile and I never got it working. However ethernet can be made to work, though you need to set eth1addr to eth3addr (or just the right one of these) in U-Boot, saveenv and reboot. After that TFTP works (but is somewhat slow). eMMC There's one last step required to allow U-Boot and Linux to access the eMMC. eMMC is wired to the same PINs as the SD card would be. However the SD card has an additional indicator pin showing whether a card is present. You might be lucky inserting a dummy card into the slot or go the clean route and remove the pin specification from the device tree.

--- a/arch/arm/dts/armada-388-clearfog.dts
+++ b/arch/arm/dts/armada-388-clearfog.dts
@@ -306,7 +307,6 @@
                        sdhci@d8000  
                                bus-width = <4>;
-                               cd-gpios = <&gpio0 20 GPIO_ACTIVE_LOW>;
                                no-1-8-v;
                                pinctrl-0 = <&clearfog_sdhci_pins
                                             &clearfog_sdhci_cd_pins>;

Next Up is flashing the U-Boot to eMMC. This seems to work with the vendor U-Boot but proves to be tricky with mainline. The fun part boils down to the fact that the boot firmware reads the first block from eMMC, but the second from SD card. If you write the mainline U-Boot, which was written and tested for SD card, to eMMC the SPL will try to load the main U-Boot starting from it's second sector from flash -- obviously resulting in garbage. This one took me several tries to figure out and made me read most of the SPL code for the device. The fix however is trivial (apart from the question on how to support all different variants from one codebase, which I'll leave to the U-Boot developers):

--- a/include/configs/clearfog.h
+++ b/include/configs/clearfog.h
@@ -143,8 +143,7 @@
 #define CONFIG_SPL_LIBDISK_SUPPORT
 #define CONFIG_SYS_MMC_U_BOOT_OFFS             (160 << 10)
 #define CONFIG_SYS_U_BOOT_OFFS                 CONFIG_SYS_MMC_U_BOOT_OFFS
-#define CONFIG_SYS_MMCSD_RAW_MODE_U_BOOT_SECTOR        ((CONFIG_SYS_U_BOOT_OFFS / 512)\
-                                                + 1)
+#define CONFIG_SYS_MMCSD_RAW_MODE_U_BOOT_SECTOR        (CONFIG_SYS_U_BOOT_OFFS / 512)
 #define CONFIG_SYS_U_BOOT_MAX_SIZE_SECTORS     ((512 << 10) / 512) /* 512KiB */
 #ifdef CONFIG_SPL_BUILD
 #define CONFIG_FIXED_SDHCI_ALIGNED_BUFFER      0x00180000      /* in SDRAM */

Linux Now we have a System booting from eMMC with mainline U-Boot (which is a most welcome speedup compared to the UART and TFTP combination from the beginning). Getting to fine-tune linux on the device -- we want to install the armmp Debian kernel and have it work. As all the drivers are build as modules for that kernel this also means initrd support. Funnily U-Boots bootz allows booting a plain vmlinux kernel but I couldn't get it to boot a plain initrd. Passing a uImage initrd and a normal kernel however works pretty well. Back when I first tried there were some modules missing and ethernet didn't work with the PHY driver built as a module. In the meantime the PHY problem was fixed in the Debian kernel and almost all modules already added. Ben then only added the USB3 module on my suggestion and as a result, unstable's armhf armmp kernel should work perfectly well on the device (you still need to patch the device tree similar to the patch above). Still missing is an updated flash-kernel to automatically generate the initrd uImage which is work in progress but got stalled until I fixed the U-Boot on eMMC problem and everything should be fine -- maybe get debian u-boot builds for that board. Pro versus Base The main difference so far between the Pro and the Base version of the ClearFog is the switch chip which is included on the Pro. The Base instead "just" has two gigabit ethernet ports and a SFP. Both, linux' and U-Boot's device tree are intended for the Pro version which makes on of the ethernet ports unusable (it tries to find the switch behind the ethernet port which isn't there). To get both ports working (or the one you settled on earlier) there's a second patch to the device tree (my version might be sub-optimal but works), U-Boot -- the linux-kernel version is a trivial adaption:

--- a/arch/arm/dts/armada-388-clearfog.dts
+++ b/arch/arm/dts/armada-388-clearfog.dts
@@ -89,13 +89,10 @@
                internal-regs  
                        ethernet@30000  
                                mac-address = [00 50 43 02 02 02];
+                               managed = "in-band-status";
+                               phy = <&phy1>;
                                phy-mode = "sgmii";
                                status = "okay";
-
-                               fixed-link  
-                                       speed = <1000>;
-                                       full-duplex;
-                                ;
                         ;
                        ethernet@34000  
@@ -227,6 +224,10 @@
                                pinctrl-0 = <&mdio_pins>;
                                pinctrl-names = "default";
+                               phy1: ethernet-phy@1   /* Marvell 88E1512 */
+                                    reg = <1>;
+                                ;
+
                                phy_dedicated: ethernet-phy@0  
                                        /*
                                         * Annoyingly, the marvell phy driver
@@ -386,62 +386,6 @@
                tx-fault-gpio = <&expander0 13 GPIO_ACTIVE_HIGH>;
         ;
-       dsa@0  
-               compatible = "marvell,dsa";
-               dsa,ethernet = <&eth1>;
-               dsa,mii-bus = <&mdio>;
-               pinctrl-0 = <&clearfog_dsa0_clk_pins &clearfog_dsa0_pins>;
-               pinctrl-names = "default";
-               #address-cells = <2>;
-               #size-cells = <0>;
-
-               switch@0  
-                       #address-cells = <1>;
-                       #size-cells = <0>;
-                       reg = <4 0>;
-
-                       port@0  
-                               reg = <0>;
-                               label = "lan1";
-                        ;
-
-                       port@1  
-                               reg = <1>;
-                               label = "lan2";
-                        ;
-
-                       port@2  
-                               reg = <2>;
-                               label = "lan3";
-                        ;
-
-                       port@3  
-                               reg = <3>;
-                               label = "lan4";
-                        ;
-
-                       port@4  
-                               reg = <4>;
-                               label = "lan5";
-                        ;
-
-                       port@5  
-                               reg = <5>;
-                               label = "cpu";
-                        ;
-
-                       port@6  
-                               /* 88E1512 external phy */
-                               reg = <6>;
-                               label = "lan6";
-                               fixed-link  
-                                       speed = <1000>;
-                                       full-duplex;
-                                ;
-                        ;
-                ;
-        ;
-
        gpio-keys  
                compatible = "gpio-keys";
                pinctrl-0 = <&rear_button_pins>;

Conclusion Apart from the mess with eMMC this seems to be a pretty nice device. It's now happily running with a M.2 SSD providing enough storage for now and still has a mSATA/mPCIe plug left for future journeys. It seems to be drawing around 5.5 Watts with SSD and one Ethernet connected while mostly idle and can feed around 500 Mb/s from disk over an encrypted ethernet connection which is, I guess, not too bad. My plans now include helping to finish flash-kernel support, creating a nice case and probably get it deployed. I might bring it to FOSDEM first though. Working on it was really quite some fun (apart from the frustrating parts finding the one-block-offset ..) and people were really helpful. Big thanks here to Debian's arm folks, Ben Hutchings the kernel maintainer and U-Boot upstream (especially Tom Rini and Stefan Roese)

Reproducible builds folks: Reproducible Builds: week 77 in Stretch cycle

What happened in the Reproducible Builds effort between Sunday October 9 and Saturday October 15 2016: Media coverage

despinosa wrote a blog post on Vala and reproducibility
h01ger and lynxis gave a talk called "From Reproducible Debian builds to Reproducible OpenWrt, LEDE" (video, slides) at the OpenWrt Summit 2016 held in Berlin, together with ELCE, held by the Linux Foundation.
A discussion on debian-devel@ resulted in a nice quotable comment from Paul Wise: "(Reproducible) builds from source (with continuous rechecking) is the only way to have enough confidence that a Debian user has the freedoms promised to them by the Debian social contract."
Chris Lamb will present a talk at Software Freedom Kosovo on reproducible builds on Saturday 22nd October.

Documentation update After discussions with HW42, Steven Chamberlain, Vagrant Cascadian, Daniel Shahaf, Christopher Berg, Daniel Kahn Gillmor and others, Ximin Luo has started writing up more concrete and detailed design plans for setting SOURCE_ROOT_DIR for reproducible debugging symbols, buildinfo security semantics and buildinfo security infrastructure. Toolchain development and fixes Dmitry Shachnev noted that our patch for #831779 has been temporarily rejected by docutils upstream; we are trying to persuade them again. Tony Mancill uploaded javatools/0.59 to unstable containing original patch by Chris Lamb. This fixed an issue where documentation Recommends: substvars would not be reproducible. Ximin Luo filed bug 77985 to GCC as a pre-requisite for future patches to make debugging symbols reproducible. Packages reviewed and fixed, and bugs filed The following updated packages have become reproducible - in our current test setup - after being fixed:

cobbler/2.6.6+dfsg1-13 by Thomas Goirand, original patch by Chris Lamb.
collectd/5.6.1-1 by Marc Fournier.
fonts-tiresias/0.1-3 by G rkan Myczko, original patch by Chris Lamb.
fntsample/4.0-2 by , original patch by Chris Lamb.
fpga-icestorm/0~20160913git266e758-2 by Ruben Undheim, original patch by Chris Lamb.
frog/0.13.5-1 by Maarten van Gompel, original patch by Chris Lamb.
lambda-align/1.0.0-2 by Sascha Steinbiss, original patch by Chris Lamb.
pleiades/1.7.0-2 by Hideki Yamane, original patch by Chris Lamb.
sweethome3d/5.2+dfsg-1 by Markus Koschany, original fix by Gabriele Giacone.
trac-subtickets/0.2.0-2 by W. Martin Borgert.

The following updated packages appear to be reproducible now, for reasons we were not able to figure out. (Relevant changelogs did not mention reproducible builds.)

aodh/3.0.0-2 by Thomas Goirand.
eog-plugins/3.16.5-1 by Michael Biebl.
flam3/3.0.1-5 by Daniele Adriana Goulart Lopes.
hyphy/2.2.7+dfsg-1 by Andreas Tille.
libbson/1.4.1-1 by A. Jesse Jiryu Davis.
libmongoc/1.4.1-1 by A. Jesse Jiryu Davis.
lxc/1:2.0.5-1 by Evgeni Golov.
spice-gtk/0.33-1 by Liang Guo.
spice-vdagent/0.17.0-1 by Liang Guo.
tnef/1.4.12-1 by Kevin Coyner.

Some uploads have addressed some reproducibility issues, but not all of them:

chktex/1.7.6-1 by Thorsten Alteholz, original patch by Sascha Steinbiss.
dbus/1.10.12-1 by Simon McVittie.
doomsday/1.15.8-3 by Markus Koschany, #839338 by Lucas Nussbaum.
emacs25/25.1+1-1 by Rob Browning.
gpgme1.0/1.7.0-3 by Daniel Kahn Gillmor.
monkeysign/2.2.0 by Antoine Beaupr .
python-attrs/16.2.0-1 by Tristan Seligmann, original patch by Chris Lamb.
shotwell/0.24.0-1 by J rg Frings-F rst, original patch by Alexis Bienven e.
supple/1.0.6-2 by Daniel Silverstone.
why/2.36-1 by Ralf Treinen, original patch by Valentin Lorentz.

Some uploads have addressed nearly all reproducibility issues, except for build path issues:

palo/1.96 by Helge Deller, #778437 by Chris Lamb.
rbdoom3bfg/1.1.0~preview3+dfsg+git20160807-1 by Tobias Frost.
singular/4.0.3-p3+ds-1 by Jerome Benoit.
varnish/5.0.0-3 by Stig Sandbeck Mathisen, original patch by Chris Lamb.
yaml-cpp/0.5.2-4 by Paul Novotny, original patch by Reiner Herrmann.

Patches submitted that have not made their way to the archive yet:

#840741 filed against http-icons by Chris Lamb.
#840177 filed against qconf by Chris Lamb.
#840845 filed against python-pygraphviz by Chris Lamb.
#840346 filed against qjoypad by Chris Lamb.

Reviews of unreproducible packages 101 package reviews have been added, 49 have been updated and 4 have been removed in this week, adding to our knowledge about identified issues. 3 issue types have been updated:

Added max_output_size_reached, ftbfs_due_to_jenkins_semaphore_setup, and build_id_differences_only.

Weekly QA work During of reproducibility testing, some FTBFS bugs have been detected and reported by:

Anders Kaseorg (1)
Chris Lamb (18)

tests.reproducible-builds.org Debian:

h01ger has turned off the "Scheduled in testing+unstable+experimental" regular IRC notifications and turned them into emails to those running jenkins.d.n.
Re-add opi2a armhf node and 3 new builder jobs for a total of 60 build jobs for armhf. (h01ger and vagrant)
vagrant suggested to add a variation of init systems effecting the build, and h01ger added it to the TODO list.
Steven Chamberlain submitted a patch so that now all buildinfo files are collected (unsigned yet) at submit@buildinfo.kfreebsd.eu.
Holger enabled CPU type variation (Intel Haswell or AMD Opteron 62xx) for i386. Thanks to Profitbricks.com for their great and continued support!

Openwrt/LEDE/NetBSD/coreboot/Fedora/archlinux:

Increase memory on the 2 build nodes from 12 to 16gb, thanks to profitbricks.com

Misc. We are running a poll to find a good time for an IRC meeting. This week's edition was written by Ximin Luo, Holger Levsen & Chris Lamb and reviewed by a bunch of Reproducible Builds folks on IRC.

Vincent Sanders: The pine stays green in winter... wisdom in hardship.

In December 2015 I saw the kickstarter for the Pine64. The project seemed to have a viable hardware design and after my experience with the hikey I decided it could not be a great deal worse.

The system I acquired comprises of:

Quad core Allwinner A64 processor clocked at 1.2GHz
2 Gigabytes of DDR3 memory
Gigabit Ethernet
two 480Mbit USB 2.0 ports
HDMI type A
micro SD card for storage.

Hardware based kickstarter projects are susceptible to several issues and the usual suspects occurred causing delays:

Inability to scale, several thousand backers instead of the hundred they were aiming for
Issues with production
Issues with shipping

My personal view is that PINE 64 inc. handled it pretty well, much better than several other projects I have backed and as my Norman Douglas quotation suggests I think they have gained some wisdom from this.

I received my hardware at the beginning of April only a couple of months after their initial estimated shipping date which as these things go is not a huge delay. I understand some people who had slightly more complex orders were just receiving their orders in late June which is perhaps unfortunate but still well within kickstarter project norms.

As an aside: I fear that many people simply misunderstand the crowdfunding model for hardware projects and fail to understand that they are not buying a finished product, on the other side of the debate I think many projects need to learn expectation management much better than they do. Hyping the product to get interest is obviously the point of the crowdfunding platform, but over promising and under delivering always causes unhappy customers.

Despite the delays in production and shipping the information available for the board was (and sadly remains) inadequate. As usual I wanted to case my board and there were no useful dimension drawings so I had to make my own from direct measurements together with a STL 3D model.

Also a mental sigh for "yet another poor form factor decision" so another special case size and design. After putting together a design and fabricating with the laser cutter I moved on to the software.

Once more this is where, once again, the story turns bleak. We find a very pretty website but no obvious link to the software (hint scroll to the bottom and find the "support" wiki link) once you find the wiki you will eventually discover that the provided software is either an Android 5.1.1 image (which failed to start on my board) or relies on some random guy from the forums who has put together his own OS images using a hacked up Allwinner Board Support Package (BSP) kernel.

Now please do not misunderstand me, I think the work by Simon Eisenmann (longsleep) to get a working kernel and Lenny Raposo to get viable OS images is outstanding and useful. I just feel that Allwinner and vendors like Pine64 Inc. should have provided something much, much better than they have. Even the efforts to get mainline support for this hardware are all completely volunteer community efforts and are are making slow progress as a result.

Assuming I wanted to run a useful OS on this hardware and not just use it as a modern work of art I installed a basic Debian arm64 using Lenny Raposo's pine64 pro site downloads. I was going to use the system for compiling and builds so used the "Debian Base" image to get a minimal setup. After generating unique ssh keys, renaming the default user and checking all the passwords and permissions I convinced myself the system was reasonably trustworthy.

The standard Debian Jessie OS runs as expected with few surprises. The main concern I have is that there are a number of unpackaged scripts installed (prefixed with pine64_) which perform several operations from reporting system health (using sysfs entries) to upgrading the kernel and bootloader.

While I understand these scripts have been provided for the novice users to reduce support burden, doing even more of the vendors job, I would much rather have had proper packages for these scripts, kernel and bootloader which apt could manage. This would have reduced image creation to a simple debootstrap giving much greater confidence in the images provenance.

The 3.10 based kernel is three years old at the time of writing and lacks a great number of features for the aarch64 ARM processors introduced since release. However I was pleasantly surprised at kvm apparently being available.

# dmesg grep -i kvm
[    7.592896] kvm [1]: Using HYP init bounce page @b87c4000
[    7.593361] kvm [1]: interrupt-controller@1c84000 IRQ25
[    7.593778] kvm [1]: timer IRQ27
[    7.593801] kvm [1]: Hyp mode initialized successfully

I installed the libvirt packages (and hence all their dependencies like qemu) and created a bridge ready for the virtual machines.

I needed access to storage for the host disc images and while I could have gone the route of using USB attached SATA as with the hikey I decided to try and use network attached storage instead. Initially I investigated iSCSI but it seems the Linux target (iSCSI uses initiator for client and target for server) support is either old, broken or unpackaged.

I turned to network block device (nbd) which is packaged and seems to have reasonable stability out of the box on modern distributions. This appeared to work well, indeed over the gigabit Ethernet interface I managed to get a sustained 40 megabytes a second read and write rate in basic testing. This is better performance than a USB 2.0 attached SSD on the hikey

I fired up the guest and perhaps I should have known better than to expect a 3.10 vendor kernel to cope. The immediate hard crashes despite tuning many variables convinced me that virtualisation was not viable with this kernel.

So abandoning that approach I attempted to run the CI workload directly on the system. To my dismay this also proved problematic. The processor has the bad habit of throttling due to thermal issues (despite a substantial heatsink) and because the storage is network attached throttling the CPU also massively impacts I/O.

The limitations meant that the workload caused the system to move between high performance and almost no progress on a roughly ten second cycle. This caused a simple NetSurf recompile CI job to take over fifteen minutes. For comparison the same task takes the armhf builder (CubieTruck) four minutes and a 64 bit x86 build which takes around a minute.

If the workload is tuned to a single core which does not trip thermal throttling the build took seven minutes. which is almost identical to the existing single core virtual machine instance running on the hikey.

In conclusion the Pine64 is an interesting bit of hardware with fatally flawed software offering. Without Simon and Lenny providing their builds to the community the device would be practically useless rather than just performing poorly. There appears to have been no progress whatsoever on the software offering from Pine64 in the six months since I received the device and no prospect of mainline Allwinner support for the SoC either.

Effectively I have spent around 50usd (40 for the board and 10 for the enclosure) on a failed experiment. Perhaps in the future the software will improve sufficiently for it to become useful but I do not hold out much hope that this will come from Pine64 themselves.

Reproducible builds folks: Reproducible Builds: week 76 in Stretch cycle

What happened in the Reproducible Builds effort between Sunday October 2 and Saturday October 8 2016: Media coverage

There was an interesting discussion on Hacker News about diverse double-compilation that touched on reproducible builds.

Events

Vagrant Cascadian gave an impromptu talk about reproducible builds at CAT Barcamp on 8th October.
Holger discussed Reproducible coreboot at coreboot.berlin. Unlike other projects, coreboot doesn't do binary releases because there have been many instances of people taking some incorrect coreboot binary, flashed it and bricked their machines The end idea is that coreboot will simply release .buildinfo files (and still no binaries) instead.

Packages reviewed and fixed, and bugs filed

#839633 filed against alabaster by Chris Lamb.
#840156 filed against lambda-align by Chris Lamb.
#840140 filed against fastx-toolkit by Chris Lamb.
#840111 filed against stumpwm by Chris Lamb.
#839631 filed against playitslowly by Chris Lamb.
#840155 filed against xotcl by Chris Lamb.
#840100 filed against wims-help by Chris Lamb.
#840156 filed against lambda-align by Chris Lamb.
#839633 filed against alabaster by Chris Lamb.
#839587 filed against sympa by Chris Lamb.
#840098 filed against fpga-icestorm by Chris Lamb.
#840143 filed against mypy by Chris Lamb.
#839977 filed against apache2 by Daniel Shahaf.
#839647 filed against dmucs by Reiner Herrmann.

Reviews of unreproducible packages 31 package reviews have been added, 27 have been updated and over 20 have been removed in this week, adding to our knowledge about identified issues. 3 issue types have been addded:

1 issue type has been updated:

Add fix for dict_ordering_in_python_alabaster_sphinx_theme_extra_nav_links toolchain issue.

Weekly QA work During of reproducibility testing, some FTBFS bugs have been detected and reported by:

Chris Lamb (12)

tests.reproducible-builds.org Debian:

The data in reproducible-tracker.json (which is fed to tracker.d.o and DDPO) has been changed to contain data from testing as the build path variations we introduced for unstable are not yet ready for wider consumption. For testing/stretch we recomment to create reproducible packages by rebuilding in the same path. (h01ger)
Various reproducibility statistics for testing/stretch have been added to the dashboard view. (h01ger)
The repository comparison page has been improved to only show obsolete packages if they exist (which they currently don't as we have rebuilt everything from the plain Debian repos, except for our modified dpkg due to #138409 and #787980). (h01ger)
All armhf boards are now using Linux kernels provided by Debian. (vagrant)

Misc. This week's edition was written by Chris Lamb, Holger Levsen & Vagrant Cascadian and reviewed by a bunch of Reproducible Builds folks on IRC.

Reproducible builds folks: Reproducible Builds: week 74 in Stretch cycle

Here is what happened in the Reproducible Builds effort between Sunday September 18 and Saturday September 24 2016: Outreachy We intend to participate in Outreachy Round 13 and look forward for new enthusiastic applications to contribute to reproducible builds. We're offering four different areas to work on:

Improve test and debugging tools.
Improving reproducibility of Debian packages.
Improving Debian infrastructure.
Help collaboration across distributions.

Reproducible Builds World summit #2 We are planning e a similar event to our Athens 2015 summit and expect to reveal more information soon. If you haven't been contacted yet but would like to attend, please contact holger. Toolchain development and fixes Mattia uploaded dpkg/1.18.10.0~reproducible1 to our experimental repository. and covered the details for the upload in a mailing list post. The most important change is the incorporation of improvements made by Guillem Jover (dpkg maintainer) to the .buildinfo generator. This is also in the hope that it will speed up the merge in the upstream. One of the other relevant changes from before is that .buildinfo files generated from binary-only builds will no longer include the hash of the .dsc file in Checksums-Sha256 as documented in the specification. Even if it was considered important to include a checksum of the source package in .buildinfo, storing it that way breaks other assumptions (eg. that Checksums-Sha256 contains only files part of that are part of a single upload, wheras the .dsc might not be part of that upload), thus we look forward for another solution to store the source checksum in .buildinfo. Bugs filed

#838713 filed against python-xlib by Chris Lamb.
#838754 filed against golang-google-grpc by Chris Lamb.
#838188 filed against ocaml by Johannes Schauer.
#838785 filed against funnelweb by Reiner Herrmann.

Reviews of unreproducible packages 250 package reviews have been added, 4 have been updated and 4 have been removed in this week, adding to our knowledge about identified issues. 4 issue types have been added:

3 issue types have been updated:

Weekly QA work FTBFS bugs have been reported by:

Chris Lamb (11)
Santiago Vila (2)

Documentation updates h01ger created a new Jenkins job so that every commit pushed to the master branch for the website will update reproducible-builds.org. diffoscope development

strip-nondeterminism development

Chris Lamb:

reprotest development

Ximin Luo uploaded reprotest 0.3 and 0.3.1 to unstable with these changes:

tests.reproducible-builds.org

The full rebuild of all packages in unstable (for all tested archs) with the new build path variation has been completed. This has had the result that we are down to ~75% reproducible packages in unstable now. In comparison, for testing (where we don't vary the build path) we are still at ~90%. IRC notifications for unstable have been enabled again. (Holger)
Make the notes job robust about bad data (see #833695 and #833738). (Holger)
Setup profitbricks-build7 running stretch as testing reproducible builds of F-Droid need to use a newer version of vagrant in order to support running vagrant VMs with kvm on kvm. (Holger)
The misbehaving 'opi2a' armhf node has been replaced with a Jetson-TK1 board kindly donated by NVidia. This machine is using an NVIDIA tegra-k1 (cortex-a15) quad-core board. (vagrant and Holger)

Misc. This week's edition was written by Chris Lamb, Holger Levsen and Mattia Rizzolo and reviewed by a bunch of Reproducible Builds folks on IRC.

Julian Andres Klode: Introducing TrieHash, a order-preserving minimal perfect hash function generator for C(++)

Abstract I introduce TrieHash an algorithm for constructing perfect hash functions from tries. The generated hash functions are pure C code, minimal, order-preserving and outperform existing alternatives. Together with the generated header files,they can also be used as a generic string to enumeration mapper (enums are created by the tool). Introduction APT (and dpkg) spend a lot of time in parsing various files, especially Packages files. APT currently uses a function called AlphaHash which hashes the last 8 bytes of a word in a case-insensitive manner to hash fields in those files (dpkg just compares strings in an array of structs). There is one obvious drawback to using a normal hash function: When we want to access the data in the hash table, we have to hash the key again, causing us to hash every accessed key at least twice. It turned out that this affects something like 5 to 10% of the cache generation performance. Enter perfect hash functions: A perfect hash function matches a set of words to constant values without collisions. You can thus just use the index to index into your hash table directly, and do not have to hash again (if you generate the function at compile time and store key constants) or handle collision resolution. As #debian-apt people know, I happened to play a bit around with tries this week before guillem suggested perfect hashing. Let me tell you one thing: My trie implementation was very naive, that did not really improve things a lot Enter TrieHash Now, how is this related to hashing? The answer is simple: I wrote a perfect hash function generator that is based on tries. You give it a list of words, it puts them in a trie, and generates C code out of it, using recursive switch statements (see code generation below). The function achieves competitive performance with other hash functions, it even usually outperforms them. Given a dictionary, it generates an enumeration (a C enum or C++ enum class) of all words in the dictionary, with the values corresponding to the order in the dictionary (the order-preserving property), and a function mapping strings to members of that enumeration. By default, the first word is considered to be 0 and each word increases a counter by one (that is, it generates a minimal hash function). You can tweak that however:

= 0
WordLabel ~ Word
OtherWord = 9

will return 0 for an unknown value, map Word to the enum member WordLabel and map OtherWord to 9. That is, the input list functions like the body of a C enumeration. If no label is specified for a word, it will be generated from the word. For more details see the documentation C code generation

switch(string[0]   32)  
case 't':
    switch(string[1]   32)  
    case 'a':
        switch(string[2]   32)  
        case 'g':
            return Tag;
         
     
 
return Unknown;

Yes, really recursive switches they directly represent the trie. Now, we did not really do a straightforward translation, there are some optimisations to make the whole thing faster and easier to look at: First of all, the 32 you see is used to make the check case insensitive in case all cases of the switch body are alphabetical characters. If there are non-alphabetical characters, it will generate two cases per character, one upper case and one lowercase (with one break in it). I did not know that lowercase and uppercase characters differed by only one bit before, thanks to the clang compiler for pointing that out in its generated assembler code! Secondly, we only insert breaks only between cases. Initially, each case ended with a return Unknown, but guillem (the dpkg developer) suggested it might be faster to let them fallthrough where possible. Turns out it was not faster on a good compiler, but it s still more readable anywhere. Finally, we build one trie per word length, and switch by the word length first. Like the 32 trick, his gives a huge improvement in performance. Digging into the assembler code The whole code translates to roughly 4 instructions per byte:

A memory load,
an or with 32
a comparison, and
a conditional jump.

(On x86, the case sensitive version actually only has a cmp-with-memory and a conditional jump). Due to https://gcc.gnu.org/bugzilla/show_bug.cgi?id=77729 this may be one instruction more: On some architectures an unneeded zero-extend-byte instruction is inserted this causes a 20% performance loss. Performance evaluation I run the hash against all 82 words understood by APT in Packages and Sources files, 1,000,000 times for each word, and summed up the average run-time:

host	arch	Trie	TrieCase	GPerfCase	GPerf	DJB
plummer	ppc64el	540	601	1914	2000	1345
eller	mipsel	4728	5255	12018	7837	4087
asachi	arm64	1000	1603	4333	2401	1625
asachi	armhf	1230	1350	5593	5002	1784
barriere	amd64	689	950	3218	1982	1776
x230	amd64	465	504	1200	837	693

Suffice to say, GPerf does not really come close. All hosts except the x230 are Debian porterboxes. The x230 is my laptop with a a Core i5-3320M, barriere has an Opteron 23xx. I included the DJB hash function for another reference. Source code The generator is written in Perl, licensed under the MIT license and available from https://github.com/julian-klode/triehash I initially prototyped it in Python, but guillem complained that this would add new build dependencies to dpkg, so I rewrote it in Perl. Benchmark is available from https://github.com/julian-klode/hashbench Usage See the script for POD documentation.
Filed under: General

Chris Lamb: Free software activities in August 2016

Here is my monthly update covering what I have been doing in the free software world (previously):

Worked on nsntrace, a userspace tool to perform network traces on processes using kernel namespaces:
- Overhauled error handling to ensure the return code of the wrapped process is returned to the surrounding environment. (#10).
- Permit the -u argument to also accept uids as well as usernames. (#16).
- Always kill the (hard-looping) udp_send utility, even on test failures. (#13).
- Updated configure.ac to look for iptables in /sbin & /usr/sbin (#11) and to raise an error if pcap.h is missing (#15).
- Drop bashisms in #!/bin/sh script (#14) and ignore the generated manpage in the Git repository (#12).
Independently discovered an regression in the Django web development framework where field__isnull=False filters were not working with some foreign keys, resulting in extending the testsuite and release documentation. (#7104).
Proposed a change to django-enumfield (a custom field for type-safe constants) to ensure passing a string type to Enum.get returned None on error to match the documentation. (#36).
Fixed an issue in the Mopidy music player's podcast extension where the testsuite was failing tests in extreme timezones. (#40).
Proposed changes to make various upstream's reproducible:
- botan, a crypto/TLS library for C++11. (#587).
- cookiecutter, a project template generator, removing nondeterministic keyword arguments from appearing in the documentation. (#800).
- pyicu, a Python wraper for the IBM Unicode library. (#27).
Integrated a number of issues raised by @piotr1212 to python-fadvise, my Python interface to posix_fadvise(2), where the API was not being applied to open file descriptors (#1) and moving the .so to a module directory (#2).
Various improvements to try.diffoscope.org, a hosted version of the diffoscope in-depth and content-aware diff utility, including introducing an HTTP API (#21), updating the SSL certificate and correcting a logic issue where errors in diffoscope itself were not being detected correctly (b0ff49). Continued thanks to Bytemark for sponsoring the hardware.
Fixed a bug in django-slack, my library to easily post messages to the Slack group-messaging utility, correcting an EncodeError exception under Python 3 (#53) and updated the minimum required version of Django to 1.7 (#54).
Various updates to tickle-me-email, my Getting Things Done-inspired email toolbox, to also match / in IMAP's LIST separators (#6) and to encode the folder list as UTF-7 (#7). Thanks to @resiak.
Clarified the documentation for travis.debian.net my hosted script to easily test and build Debian packages on the Travis CI continuous integration platform regarding how to integrate with Github (#20).

Reproducible builds

Whilst anyone can inspect the source code of free software for malicious flaws, most Linux distributions provide binary (or "compiled") packages to end users. The motivation behind the Reproducible Builds effort is to allow verification that no flaws have been introduced either maliciously and accidentally during this compilation process by promising identical binary packages are always generated from a given source.

Toolchain issues I submitted the following patches to fix reproducibility-related toolchain issues:

My work in the Reproducible Builds project was also covered in our weekly reports. (#67, #68, #69, #70).

Diffoscope diffoscope is our "diff on steroids" that will not only recursively unpack archives but will transform binary formats into human-readable forms in order to compare them:

Added a command-line interface to the try.diffoscope.org web service.
Added a JSON comparator.
In the HTML output, highlight lines when hovering to make it easier to visually track.
Ensure that we pass str types to our Difference class, otherwise we can't be sure we can render them later.
Testsuite improvements:
- Generate test coverage reports.
- Add tests for Haskell and GitIndex comparators.
- Completely refactored all of the comparator tests, extracting out commonly-used routines.
- Confirm rendering of text and HTML presenters when checking non-existing files.
- Dropped a squashfs test as it was simply too unreliable and/or has too many requirements to satisfy.
A large number of miscellaneous cleanups, including:
- Reworking the comparator setup/preference internals by dynamically importing classes via a single list.
- Split exceptions out into dedicated diffoscope.exc module.
- Tidying the PROVIDERS dict in diffoscope/__init__.py.
- Use html.escape over xml.sax.saxutils.escape, cgi.escape, etc.
- Removing hard-coding of manual page targets names in debian/rules.
- Specify all string format arguments as logging function parameters, not using interpolation.
- Tidying imports, correcting indentation levels and drop unnecessary whitespace.

disorderfs disorderfs is our FUSE filesystem that deliberately introduces nondeterminism in system calls such as readdir(3).

Added a testsuite to prevent regressions. (f124965)
Added a --sort-dirents=yes no option for forcing deterministic ordering. (2aae325)

Other

Improved strip-nondeterminism, our tool to remove specific nondeterministic information after a build:
- Match more styles of Java .properties files.
- Remove hyphen from "non-determinism" and "non-deterministic" throughout package for consistency.
Improvements to our testing infrastucture:
- Improve the top-level navigation so that we can always get back to "home" of a package.
- Give expandable elements cursor: pointer CSS styling to highlight they are clickable.
- Drop various trailing underlined whitespaces after links.
- Explicitly log that build was successful or not.
- Various code-quality improvements, including prefering str.format over concatentation.
Miscellaneous updates to our filter-packages internal tool:
- Add --random=N and --url options.
- Add support for --show=comments.
- Correct ordering so that --show-version runs after --filter-ftbfs.
- Rename --show-ftbfs to --filter-ftbfs and --show-version to --show=version.
Created a proof-of-concept reproducible-utils package to contain commonly-used snippets aimed at developers wishing to make their packages reproducible.

I also submitted 92 patches to fix specific reproducibility issues in advi, amora-server, apt-cacher-ng, ara, argyll, audiotools, bam, bedtools, binutils-m68hc1x, botan1.10, broccoli, congress, cookiecutter, dacs, dapl, dateutils, ddd, dicom3tools, dispcalgui, dnssec-trigger, echoping, eekboek, emacspeak, eyed3, fdroidserver, flashrom, fntsample, forkstat, gkrellm, gkrellm, gnunet-gtk, handbrake, hardinfo, ircd-irc2, ircd-ircu, jack-audio-connection-kit, jpy, kxmlgui, libbson, libdc0, libdevel-cover-perl, libfm, libpam-ldap, libquvi, librep, lilyterm, mozvoikko, mp4h, mp4v2, myghty, n2n, nagios-nrpe, nikwi, nmh, nsnake, openhackware, pd-pdstring, phpab, phpdox, phpldapadmin, pixelmed-codec, pleiades, pybit, pygtksourceview, pyicu, python-attrs, python-gflags, quvi, radare2, rc, rest2web, roaraudio, rt-extension-customfieldsonupdate, ruby-compass, ruby-pg, sheepdog, tf5, ttf-tiresias, ttf-tiresias, tuxpaint, tuxpaint-config, twitter-bootstrap3, udpcast, uhub, valknut, varnish, vips, vit, wims, winswitch, wmweather+ & xshisen.

Debian GNU/Linux

Patches contributed

Lintian:
devscripts: Add an "--extra-packages" option to install arbitrary extra packages
dh-python: Emit invokation and command-line arguments of "after" and "before" commands by default
debsources: Retain line context on 404s by appending the hash for specific links
python-debian: Improve message when passing incorrect type to iter_paragraphs
autopkgtest: Does not run pyflakes3 tests
python3-debian: Include examples in package
snapshot.debian.org: Correct link to API page in sidebar
dh-python: Misparses d/control and copies build profiles into substvars
auto-apt-proxy: FTBFS: In POSIX sh, 'local' is undefined

I also submitted 22 patches to fix typos in debian/rules files against ctsim, f2c, fonts-elusive-icons, ifrit, ldapscripts, libss7, libvmime, link-grammar, menulibre, mit-scheme, mugshot, nlopt, nunit, proftpd-mod-autohost, proftpd-mod-clamav, rabbyt, radvd, ruby-image-science, snmpsim, speech-tools, varscan & whatmaps.

Debian LTS

This month I have been paid to work 15 hours on Debian Long Term Support (LTS). In that time I did the following:

"Frontdesk" duties, triaging CVEs, etc.
Authored the patch & issued DLA 596-1 for extplorer, a web-based file manager, fixing an archive traversal exploit.
Issued DLA 598-1 for suckless-tools, fixing a segmentation fault in the slock screen locking tool.
Issued DLA 599-1 for cracklib2, a pro-active password checker library, fixing a stack-based buffer overflow when parsing large GECOS fields.
Improved the find-work internal tool adding optional colour highlighting and migrating it to Python 3.
Wrote an lts-missing-uploads tool to find mistakes where there was no correponding package in the archive after an announcement.
Added optional colour highlighting to the lts-cve-triage tool.

Uploads

redis 2:3.2.3-1 New upstream release, move to the DEP-5 debian/copyright format, ensure that we are running as root in LSB initscripts and add a README.Source regarding our local copies of redis.conf and sentinel.conf.
python-django:
- 1:1.10-1 New upstream release.
- 1:1.10-2 Fix test failures due to mishandled upstream translation updates.

gunicorn:
- 19.6.0-2 Reload logrotate in the postrotate action to avoid processes writing to the old files and move to DEP-5 debian/copyright format.
- 19.6.0-3 Drop our /usr/sbin/gunicorn ,3 -debian and related Debian-specific machinery to be more like upstream.
- 19.6.0-4 Drop "template" systemd .service files and point towards examples and documentation instead.

adminer:
- 4.2.5-1 Take over package maintenance, completely overhauling the packaging with a new upstream version, move to virtual-mysql-server to support MariaDB, updating package names of dependencies and fix the outdated Apache configuration.
- 4.2.5-2 Correct the php5 package names.

Bugs filed (without patches)

python3-debian: Can't parse input of bytes under Python 3
devscripts: Please warn if DEBUILD_DPKG_BUILDPACKAGE_OPTS contains "-i -I"
gbrowse: Ships a predictable OpenID constumer secret
Popcon, etc:
- python-popcon: Must not crash on invalid data
- popularity-contest: all-popcon-results.txt contains invalid data
congress: CongressParser.py not generated during build
libfuzzer-3.8-dev:
ceph: Maintainer fields points to a moderated mailing list
nsntrace: Please package a new upstream snapshot and please upload to jessie-backports.
safecat: Homepage field refers to missing URL
foxeye: Please make the the package autoreconfable

RC bugs

I filed 3 RC bugs with patches:

gnome-packagekit: Missing Build-Depends on 'sp'
python-popcon: Assumes all-popcon-results.txt.gz file is UTF-8
python3-popcon: Missing Depends on 'python3-xdg'

I additionally filed 8 RC bugs for packages that access the internet during build against autopkgtest, golang-github-xenolf-lego, pam-python, pexpect, python-certbot, python-glanceclient, python-pykka & python-tornado.

I also filed 74 FTBFS bugs against airlift-airline, airlift-slice, alter-sequence-alignment, apktool, atril, auto-apt-proxy, bookkeeper, bristol, btfs, caja-extensions, ccbuild, cinder, clustalo, colorhug-client, cpp-netlib, dimbl, edk2, elasticsearch, ganv, git-remote-hg, golang-codegangsta-cli, golang-goyaml, gr-radar, imagevis3d, jacktrip, jalv, kdepim, kiriki, konversation, libabw, libcereal, libdancer-plugin-database-perl, libdist-zilla-plugins-cjm-perl, libfreemarker-java, libgraph-writer-dsm-perl, libmail-gnupg-perl, libminc, libsmi, linthesia, lv2-c++-tools, lvtk, mate-power-manager, mcmcpack, mopidy-podcast, nageru, nfstrace, nova, nurpawiki, open-gram, php-crypt-gpg, picmi, projectl, pygpgme, python-apt, python-django-bootstrap-form, python-django-navtag, python-oslo.config, qmmp, qsapecng, r-cran-sem, rocs, ruby-mini-magick, seahorse-nautilus, shiro, snap, tcpcopy, tiledarray, triggerhappy, ucto, urdfdom, vmmlib, yara-python, yi & z3.

FTP Team As a Debian FTP assistant I ACCEPTed 90 packages: android-platform-external-jsilver, android-platform-frameworks-data-binding, camlpdf, consolation, dfwinreg, diffoscope, django-restricted-resource, django-testproject, django-testscenarios, gitlab-ci-multi-runner, gnome-shell-extension-taskbar, golang-github-flynn-archive-go-shlex, golang-github-jamesclonk-vultr, golang-github-weppos-dnsimple-go, golang-golang-x-time, google-android-ndk-installer, haskell-expiring-cache-map, haskell-hclip, haskell-hdbc-session, haskell-microlens-ghc, haskell-names-th, haskell-persistable-record, haskell-should-not-typecheck, haskell-soap, haskell-soap-tls, haskell-th-reify-compat, haskell-with-location, haskell-wreq, kbtin, libclipboard-perl, libgtk3-simplelist-perl, libjs-jquery-selectize.js, liblemon, libplack-middleware-header-perl, libreoffice, libreswan, libtest-deep-json-perl, libtest-timer-perl, linux, linux-signed, live-tasks, llvm-toolchain-3.8, llvm-toolchain-snapshot, lua-luv, lua-torch-image, lua-torch-nn, magic-wormhole, mini-buildd, ncbi-vdb, node-ast-util, node-es6-module-transpiler, node-es6-promise, node-inline-source-map, node-number-is-nan, node-object-assign, nvidia-graphics-drivers, openhft-chronicle-bytes, openhft-chronicle-core, openhft-chronicle-network, openhft-chronicle-threads, openhft-chronicle-wire, pycodestyle, python-aptly, python-atomicwrites, python-click-log, python-django-casclient, python-git-os-job, python-hypothesis, python-nosehtmloutput, python-overpy, python-parsel, python-prov, python-py, python-schema, python-tackerclient, python-tornado, pyvo, r-cran-cairo, r-cran-mi, r-cran-rcppgsl, r-cran-sem, ruby-curses, ruby-fog-rackspace, ruby-mixlib-archive, ruby-tzinfo-data, salt-formula-swift, scapy3k, self-destructing-cookies, trollius-redis & websploit.

Reproducible builds folks: Reproducible builds: week 70 in Stretch cycle

What happened in the Reproducible Builds effort between Sunday August 21 and Saturday August 27 2016: GSoC and Outreachy updates

Satyam Zode's final report for GSOC 2016

Packages reviewed and fixed, and bugs filed

#834976 filed against auto-apt-proxy by Chris Lamb.
#835637 filed against myghty by Chris Lamb.
#835061 filed against varnish by Chris Lamb.
#835633 filed against pleiades by Chris Lamb.
#835625 filed against nikwi by Chris Lamb.
#835263 filed against binutils-m68hc1x by Chris Lamb.
#835448 filed against eekboek by Chris Lamb.
#835143 filed against ttf-tiresias by Chris Lamb.
#835637 filed against myghty by Chris Lamb.
#835495 filed against broccoli by Chris Lamb.
#835129 filed against dateutils by Chris Lamb.
#835051 filed against sheepdog by Chris Lamb.
#835145 filed against udpcast by Chris Lamb.
#834983 filed against eyed3 by Chris Lamb.
#835617 filed against congress by Chris Lamb.
#835376 filed against lilyterm by Chris Lamb.
#835130 filed against ircd-ircu by Chris Lamb.
#835262 filed against radare2 by Chris Lamb.
#835193 filed against phpdox by Chris Lamb.
#835265 filed against argyll by Chris Lamb.
#835259 filed against quvi by Chris Lamb.
#835371 filed against dispcalgui by Chris Lamb.
#835147 filed against javatools by Chris Lamb.
#834988 filed against twitter-bootstrap3 by Chris Lamb.
#835463 filed against fdroidserver by Chris Lamb.
#835646 filed against dh-lua by Chris Lamb.
#835447 filed against libmodule-build-withxspp-perl by Chris Lamb.
#835418 filed against libfm by Chris Lamb.
#835143 filed against ttf-tiresias by Chris Lamb.
#834993 filed against oss4 by Reiner Herrmann.

Reviews of unreproducible packages 10 package reviews have been added and 6 have been updated this week, adding to our knowledge about identified issues. A large number of issue types have been updated:

Weekly QA work 29 FTBFS bugs have been reported by:

Chris Lamb (27)
Daniel Stender (1)
Santiago Vila (1)

diffoscope development

Holger also created another test job for diffoscope on jenkins.debian.net, so that now also all commits to branches other than master are being tested. strip-nondeterminism development strip-nondeterminism 0.023-1 was uploaded by Chris Lamb:

 * Support Android .apk files with the JAR normalizer.
 * handlers/png.pm: Drop unused Archive::Zip import
 * Remove hyphen from non-determinism and non-deterministic.
 * javaproperties.pm: Match more styles of .properties and loosen filename matching.
 * Improve tests:
   - Make fixture runner generic to all normalizer types.
   - Replace (single) pearregistry test with a fixture.
   - Set a canonical time for fixture tests.
   - Add gzip testcase fixture.
   - Replace t/javadoc.t with fixture
   - Replace t/ar.t with a fixture.
   - t/javaproperties: move pom.properties and version.properties tests to fixtures
   - t/fixtures.t: move to using subtests
   - t/fixtures.t: Explicitly test that we can find a normalizer
   - t/fixtures.t: Don't run normalizer if we didn't find one.

strip-nondeterminism 0.023-2 uploaded by Mattia Rizzolo to allow stderr in autopkgtest. disorderfs development

Chris Lamb:
- Add --sort-dirents=yes no option for forcing deterministic.
- Testsuite:

tests.reproducible-builds.org Debian:

Since we introduced build path variations for unstable and experimental last week, our IRC channel has been flooded with notifcations about packages becoming unreproducible - and you might have noticed some of your packages having become unreproducible recently too. To make our IRC more bearable again, notifications for status changes on i386 and armhf have been disabled, so that now we only get notifications for status changes in unstable. (h01ger)
Link to jenkins documentation in every page (h01ger)
The "pre build" check, whether a node is up, now also detects if a node has a read-only filesystem, which sometimes happens on some broken armhf nodes. (h01ger)
To further improve monitoring of those armhf nodes Work to make them send mails (through an ISP which is blocking outgoing mails) has been started and should be finished next week. (h01ger)
As one of the armhf nodes (opi2a) is acting strange, a workaround has been added to make it's deployment work despite that. (h01ger)
Collapse whitespace to avoid ugly "trailing underlines" in hyperlinks for diffoscope results and pkg sets (Chris Lamb)
Give details HTML elements "cursor: pointer" CSS property to highlight they are clickable. (Chris Lamb)
The db connection timeout has been raised to a minute when using SQLAlchemy too. (h01ger).

Somewhat related to reproducible builds there has been a first Debian jenkins team maintainance meeting on the #debian-qa IRC channel, to discuss current issues with the setup and to start the work of migrating jenkins.debian.net to jenkins.debian.org. The next meeting will take place on September 28th 2016 at 19 UTC. Misc. This week's edition was written by Chris Lamb and Holger Levsen and reviewed by a bunch of Reproducible Builds folks on IRC.

Reproducible builds folks: Reproducible Builds: week 69 in Stretch cycle

What happened in the Reproducible Builds effort between Sunday August 14 and Saturday August 20 2016: Fasten your seatbelts Important note: we enabled build path variation for unstable now, so your package(s) might become unreproducible, while previously it was said to be reproducible given a specific build path it probably still is reproducible but read on for the details below in the tests.reproducible-builds.org section! As said many times: this is still research and we are working to make it reality. Media coverage Daniel Stender blogged about python packaging and explained some caveats regarding reproducible builds. Toolchain developments Thomas Schmitt uploaded xorriso which now obeys SOURCE_DATE_EPOCH. As stated in its man pages:

ENVIRONMENT
[...]
SOURCE_DATE_EPOCH  belongs to the specs of reproducible-builds.org.  It
is supposed to be either undefined or to contain a decimal number which
tells the seconds since january 1st 1970. If it contains a number, then
it is used as time value to set the  default  of  --modification-date=,
--gpt_disk_guid,  and  --set_all_file_dates.  Startup files and program
options can override the effect of SOURCE_DATE_EPOCH.

Packages reviewed and fixed, and bugs filed The following packages have become reproducible after being fixed:

cadencii/3.3.9+svn20110818.r1732-5 by Ying-Chun Liu, original patch by Chris Lamb.
cfengine3/3.7.4-3 by Antonio Radici.
findbugs/3.1.0~preview2-1 by Emmanuel Bourg.
gcpegg/5.1-14 by Bdale Garbee, original patch by Chris Lamb.
gnunet-gtk/0.10.1-4 by Bertrand Marc, original patch by Chris Lamb.
konwert/1.8-12 by Yann Dirson, original patch by Chris Lamb.
link-grammar/5.3.8-2 by Jeremy Bicha, original patch by Chris Lamb.
metastudent-data/2.0.1-1 by Sascha Steinbiss.
pixmap/2.6pl4-20 by Paul Slootman, original patch by ~~Chris Lamb~~ Maria Valentina Marin a.k.a. Akira.
python-afl/0.5.4-2 by Daniel Stender.
tf5/5.0beta8-6 by Russ Allbery, original patch by Chris Lamb.
triplane/1.0.8-2 by Markus Koschanyover, original patch by Chris Lamb.
vips/8.3.3-1 by Laszlo Boszormenyi, original patch by Chris Lamb.
xzgv/0.9.1-4 by Theodore Y. Ts'o, original patch by Chris Lamb.

The following updated packages appear to be reproducible now, for reasons we were not able to figure out. (Relevant changelogs did not mention reproducible builds.)

vulkan/1.0.21.0+dfsg1-1 by Timo Aaltonen.

The following 2 packages were not changed, but have become reproducible due to changes in their build-dependencies: tagsoup tclx8.4. Some uploads have addressed some reproducibility issues, but not all of them:

cappuccino/0.5.1-4 by Breno Leitao, original patch by Chris Lamb.
darkice/1.3-0.1 by Nicolas Bouleng.
flask-restful/0.3.5-1 by Ond ej Nov , original patch by Chris Lamb.
kodi/16.1+dfsg1-2 by Balint Reczey, original patch by Lukas Rechberger.
libaws/3.3.2-3 by Nicolas Bouleng.
liblog4ada/1.3-1 by Nicolas Bouleng.
libxmlezout/1.06.1-8 by Nicolas Bouleng.
ruby-rmagick/2.15.4+dfsg-2 by Antonio Terceir, original patch from Chris Lamb.
taggrepper/0.05-2 by Kumar Appaiah, original patch and original patch from Maria Valentina Marin.
xotcl/1.6.8-2 by Stefan Sobernig, original patch by Chris Lamb.

Patches submitted that have not made their way to the archive yet:

#834755 against apt-cacher-ng by Chris Lamb.
#834976 against auto-apt-proxy by Chris Lamb.
#834549 against botan1.10 by Chris Lamb.
#834861 against cookiecutter by Chris Lamb.
#834779 against dicom3tools by Chris Lamb.
#834859 against echoping by Chris Lamb.
#834983 against eyed3 by Chris Lamb.
#834735 against filepp by Chris Lamb.
#834957 against flashrom by Chris Lamb.
#834773 against gkrellm by Chris Lamb.
#834292 against gr-radar by Chris Lamb.
#834956 against ircd-irc2 by Chris Lamb.
#834897 against jpy by Chris Lamb.
#834452 against libdc0 by Chris Lamb.
#834324 against libnss-ldap by Reiner Herrmann.
#834945 against libquvi by Chris Lamb.
#834534 against librep by Chris Lamb.
#834537 against mozvoikko by Chris Lamb.
#834857 against nagios-nrpe by Chris Lamb.
#834316 against openocd by Reiner Herrmann.
#834993 against oss4 by Reiner Herrmann.
#834302 against pd-flite by Reiner Herrmann.
#834754 against phpab by Chris Lamb.
#834279 against phpldapadmin by Chris Lamb.
#834277 against pybit by Chris Lamb.
#834553 against pyicu by Chris Lamb.
#834541 against ruby-pg by Chris Lamb.
#835051 against sheepdog by Chris Lamb.
#834896 against ttf-tiresias by Chris Lamb.
#834780 against tuxpaint-config by Chris Lamb.
#834988 against twitter-bootstrap3 by Chris Lamb.
#834776 against uhub by Chris Lamb.
#835061 against varnish by Chris Lamb.

Bug tracker house keeping:

Chris Lamb pinged 164 bugs he filed more than 90 days ago which have a patch and had no maintainer reaction.

Reviews of unreproducible packages 55 package reviews have been added, 161 have been updated and 136 have been removed in this week, adding to our knowledge about identified issues. 2 issue types have been updated:

Added user_in_documentation_generated_by_gsdoc
Added locale_differences_in_pom

Weekly QA work FTBFS bugs have been reported by:

Chris Lamb (16)
Santiago Vila (2)

diffoscope development Chris Lamb, Holger Levsen and Mattia Rizzolo worked on diffoscope this week. Improvements were made to SquashFS and JSON comparison, the https://try.diffoscope.org/ web service, documentation, packaging, and general code quality. diffoscope 57, 58, and 59 were uploaded to unstable by Chris Lamb. Versions 57 and 58 were both broken, so Holger set up a job on jenkins.debian.net to test diffoscope on each git commit. He also wrote a CONTRIBUTING document to help prevent this from happening in future. From these efforts, we were also able to learn that diffoscope is now reproducible even when built across multiple architectures:

< h01ger>   https://tests.reproducible-builds.org/debian/rb-pkg/unstable/amd64/diffoscope.html shows these packages were built on amd64:
< h01ger>    bd21db708fe91c01ba1c9cb35b9d41a7c9b0db2b 62288 diffoscope_59_all.deb
< h01ger>    366200bf2841136a4c8f8c30bdc87057d59a4cdd 20146 trydiffoscope_59_all.deb
< h01ger>   and on i386:
< h01ger>    bd21db708fe91c01ba1c9cb35b9d41a7c9b0db2b 62288 diffoscope_59_all.deb
< h01ger>    366200bf2841136a4c8f8c30bdc87057d59a4cdd 20146 trydiffoscope_59_all.deb
< h01ger>   and on armhf:
< h01ger>    bd21db708fe91c01ba1c9cb35b9d41a7c9b0db2b 62288 diffoscope_59_all.deb
< h01ger>    366200bf2841136a4c8f8c30bdc87057d59a4cdd 20146 trydiffoscope_59_all.deb

And those also match the binaries uploaded by Chris in his diffoscope 59 binary upload to ftp.debian.org, yay! Eating our own dogfood and enjoying it! tests.reproducible-builds.org Debian related:

show percentage of results in the last 24/48h (h01ger)
switch python database backend to SQLAlchemy (Valerie)
vary build path varitation for unstable and experimental for all architectures. (h01ger)

The last change probably will have an impact you will see: your package might become unreproducible in unstable and this will be shown on tracker.debian.org, while it will still be reproducible in testing. We've done this, because we think reproducible builds are possible with arbitrary build paths. But: we don't think those are a realistic goal for stretch, where we still recommend to use .buildinfo to record the build patch and then do rebuilds using that path. We are doing this, because besides doing theoretical groundwork we also have a practical goal: enable users to independently verify builds. And if they only can do this with a fixed path, so be it. For now

To be clear: for Stretch we recommend that reproducible builds are done in the same build path as the "original" build. Finally, and just for our future references, when we enabled build path variation on Saturday, August 20th 2016, the numbers for unstable were:

suite	all	reproducible	unreproducible	ftbfs	depwait	not for this arch	blacklisted
unstable/amd64	24693	21794 (88.2%)	1753 (7.1%)	972 (3.9%)	65 (0.2%)	95 (0.3%)	10 (0.0%)
unstable/i386	24693	21182 (85.7%)	2349 (9.5%)	972 (3.9%)	76 (0.3%)	103 (0.4%)	10 (0.0%)
unstable/armhf	24693	20889 (84.6%)	2050 (8.3%)	1126 (4.5%)	199 (0.8%)	296 (1.1%)	129 (0.5%)

Misc. Ximin Luo updated our git setup scripts to make it easier for people to write proper descriptions for our repositories. This week's edition was written by Ximin Luo and Holger Levsen and reviewed by a bunch of Reproducible Builds folks on IRC.

Shirish Agarwal: The road to TOR

Happy Independence Day to all. I had been looking forward to this day so I can use to share with my brothers and sisters what little I know about TOR . Independence means so many things to many people. For me, it means having freedom, valuing it and using it to benefit not just to ourselves but to people at large. And for that to happen, at least on the web, it has to rise above censorship if we are to get there at all. I am 40 years old, and if I can t read whatever I want to read without asking the state-military-Corporate trinity than be damned with that. Debconf was instrumental as I was able to understand and share many of the privacy concerns that we all have. This blog post is partly a tribute to being part of a community and being part of Debconf16. So, in that search for privacy couple of years ago, I came across TOR . TOR stands for The Onion Router project. Explaining tor is simple. Let us take the standard way in which we approach the website using a browser or any other means. a. We type out a site name, say debian.org in the URL/URI bar .
b. Now the first thing the browser would do is look into its DNS Cache to see if the name/URL has been used before. If it is something like debian.org which has been used before and is *fresh* and there is content already it would serve the content from the cache there itself.
c. In case, if it s not or the content is stale or something, it would generate a DNS lookup through the various routing tables till the DNS IP Address is found and information relayed to the browser.
d. The browser takes the IP Address and opens a TCP connection to the server, you have the handshake happen and after that it s business as usual.
e. In case if it doesn t work, you could get errors like Could not connect to server xyz or some special errors with error codes. This is a much simplified version of what happens or goes through normally with most/all of the browsers. One good way to see how the whole thing happens is to use traceroute and use the whois service. For e.g. [$] traceroute debian.org and then

[$] whois 5.153.231.4   grep inetnum

inetnum:        5.153.231.0 - 5.153.231.255

Just using whois IP Address gives much more. I just shared a short version because I find it interesting that Debian has booked all 255 possible IP Addresses but speculating on that would be probably be a job for a different day. Now the difference when using TOR are two things a. The conversation is encrypted (somewhat like using https but encrypted through the relays)
b. The conversation is relayed over 2-3 relays and it will give a somewhat different identification to the DNS server at the other end.
c. It is only at the end-points that the conversation will be in plain text. For e.g. the TOR connection I m using atm is from me France (relay) Switzerland (relay) Germany (relay) WordPress.com . So wordpress thinks that all the connection is happening via Germany while I m here in India. It would also tells that I m running MS-Windows some version and a different browser while I m from somewhere in India, on Debian, using another browser altogether

There are various motivations for doing that. For myself, I m just a private person and do not need or want that any other person/s or even the State should be looking over my shoulder as to what I m doing. And the argument that we need to spy on citizens because Terrorists are there doesn t hold water over me. There are many ways in which they can pass messages even without tor or web. The Government-Corporate-Military just get more powerful if and when they know what common people think, do, eat etc. So the question is how does you install tor if you a private sort of person . If you are on a Debian machine, you are one step closer to doing that. So the first thing that you need to do is install the following $ sudo aptitude install ooniprobe python-certifi tor tor-geoipdb torsocks torbrowser-launcher Once the above is done, then run torbrowser-launcher. This is how it would work out the first time it is run [$] torbrowser-launcher

Tor Browser Launcher

By Micah Lee, licensed under MIT

version 0.2.6

https://github.com/micahflee/torbrowser-launcher

Creating GnuPG homedir /home/shirish/.local/share/torbrowser/gnupg_homedir

Downloading and installing Tor Browser for the first time.

Downloading https://dist.torproject.org/torbrowser/update_2/release/Linux_x86_64-gcc3/x/en-US

Latest version: 6.0.3

Downloading https://dist.torproject.org/torbrowser/6.0.3/tor-browser-linux64-6.0.3_en-US.tar.xz.asc

Downloading https://dist.torproject.org/torbrowser/6.0.3/tor-browser-linux64-6.0.3_en-US.tar.xz

Verifying signature

Extracting tor-browser-linux64-6.0.3_en-US.tar.xz

Running /home/shirish/.local/share/torbrowser/tbb/x86_64/tor-browser_en-US/start-tor-browser.desktop

Launching './Browser/start-tor-browser --detach'...

As can be seen above, you basically download the tor browser remotely from the website. Obviously, for this port 80 needs to be opened. One of the more interesting things is that it tells you where it installs the browser. /home/shirish/.local/share/torbrowser/tbb/x86_64/tor-browser_en-US/Browser/start-tor-browser and then detaches. The first time the TOR browser actually runs it looks something similar to this

Torbrowser picture

Additionally it would give you 4 choices. Depending on your need for safety, security and convenience you make a choice and live with it. Now the only thing remaining to do is have an alias for your torbrowser. So I made [$] alias tor

tor=/home/shirish/.local/share/torbrowser/tbb/x86_64/tor-browser_en-US/Browser/start-tor-browser It is suggested that you do not use the same usernames on the onion network. Also apart from the regular URL addresses such as flossexperiences.wordpress.com you will also see sites such as https://www.abc12defgh3ijkl.onion.to (fictional address) Now there would be others who would want to use the same/similar settings say as there are in their Mozilla Firefox installation. To do that do the following steps a. First close down both Torbrowser and Mozilla Firefox .
b. Open your file browser and go to where your mozilla profile details are. In typical Debian installations it is at ~/.mozilla/firefox/5r7t1r92.default In the next tab, navigate to ~/.local/share/torbrowser/tbb/x86_64/tor-browser_en-US/Browser/TorBrowser/Data/Browser/profile.default c. Now copy the following files over from your mozilla profile to your tor browser profile and you can resume where you left off.

and the following folders/directories

Once the above is done, fire up your torbrowser with the alias shared. This is usually put it in your .bashrc file or depending on whatever terminal interpreter you use, wherever the config file will be. Welcome to the world of TOR. Now, after a time if you benefit from tor and would like to give back to the tor community, you should look up tor bridges and relay. As the blog post has become long enough, I would end it now and hopefully we can talk about tor bridges and relay some other day.
Filed under: Miscellenous Tagged: #anonymity, #Debconf16, #debian, #tor, #torbrowser, GNU, Linux, Privacy

Reproducible builds folks: Reproducible Builds: week 68 in Stretch cycle

What happened in the Reproducible Builds effort between Sunday August 7 and Saturday August 13 2016: GSoC and Outreachy updates

spectranaut blogged about a database migration plan
ceridwen provided a weekly update of reprotest development

Reproducible work in other projects Thomas Schmitt scdbackup@gmx.net implemented a new -as mkisofs option:

--set_all_file_dates timestring
Set mtime, atime, and ctime of all files and directories to  the
given time.
Valid  timestring  formats  are:  'Nov  8  14:51:13  CET  2007',
110814512007.13, 2007110814511300. See also --modification-date=
and man xorriso, Examples of input timestrings.
This  action  stays  delayed until mkisofs emulation ends. Up to
then it  can  be  revoked  by  --set_all_file_dates  with  empty
timestring.   In  any  case  files  which get into the ISO after
mkisofs emulation ended will not  be  affected,  unless  another
mkisofs emulation applies --set_all_file_date again.

LEDE developer Jonas Gorski submitted a patch to fix build times in their kernel:

kernel: allow reproducable builds
Similar how we fix the file times in the filesystems, fix the build time
of the kernel, and make the build number static. This should allow the
kernel build to be reproducable when combined with setting the
KERNEL\_BUILD\_USER and \_DOMAIN in case of different machines.
The reproducability only applies to non-initramfs kernels, those still
require additional changes.
Signed-off-by: Jonas Gorski <jonas.gorski@gmail.com>

Packages reviewed and fixed, and bugs filed Patches have been submitted by:

Package reviews 28 reviews have been added, 4 have been updated and 7 have been removed in this week, adding to our knowledge about identified issues. Issue types have been added/updated:

Weekly QA work FTBFS bugs have been reported by:

Chris Lamb (23)
shirish (1)

diffoscope development

strip-nondeterminism development

Chris Lamb:
- Remove hyphen from non-determinism and non-deterministic.

tests.reproducible-builds.org

schedule testing/i386 more often than unstable+experimental, in order to see the results of building with build path variation. (h01ger)
spectranaut wrote a patch for using sqlalchemy which has yet to be merged.
Our build path variation tests on testing/i386 have brought the first results: the build essential package set is now 43% unreproducible compared to "only" 26% on amd64. So another conclusion from this is probably that the build essential maintainers should merge our patches, the other is that build path variation is still a goal far way, which also can be seen "nicely" now in the general suite graph showing the impact of build path variation introduced last week. (h0lger)
Chris Lamb wrote a patch to improve the top-level navigation so that we can always get back to "home" of a package.
Chris Lamb also wrote a patch to explicitly log when a build was successful instead of it being implicit.

Misc. Chris started to ping old bugs with patches and no maintainer reaction so far. This week's edition was written by Chris Lamb and Holger Levsen and reviewed by a bunch of Reproducible Builds folks on IRC.

Michael Prokop: systemd backport of v230 available for Debian/jessie

At DebConf 16 I was working on a systemd backport for Debian/jessie. Results are officially available via the Debian archive now. In Debian jessie we have systemd v215 (which originally dates back to 2014-07-03 upstream-wise, plus changes + fixes from pkg-systemd folks of course). Now via Debian backports you have the option to update systemd to a very recent version: v230. If you have jessie-backports enabled it s just an apt install systemd -t jessie-backports away. For the upstream changes between v215 and v230 see upstream s NEWS file for list of changes. (Actually the systemd backport is available since 2016-07-19 for amd64, arm64 + armhf, though for mips, mipsel, powerpc, ppc64el + s390x we had to fight against GCC ICEs when compiling on/for Debian/jessie and for i386 architecture the systemd test-suite identified broken O_TMPFILE permission handling.) Thanks to the Alexander Wirt from the backports team for accepting my backport, thanks to intrigeri for the related apparmor backport, Guus Sliepen for the related ifupdown backport and Didier Raboud for the related usb-modeswitch/usb-modeswitch-data backports. Thanks to everyone testing my systemd backport and reporting feedback. Thanks a lot to Felipe Sateler and Martin Pitt for reviews, feedback and cooperation. And special thanks to Michael Biebl for all his feedback, reviews and help with the systemd backport from its very beginnings until the latest upload. PS: I cannot stress this enough how fantastic Debian s pkg-systemd team is. Responsive, friendly, helpful, dedicated and skilled folks, thanks folks!

Martin Michlmayr: Debian on Seagate Personal Cloud and Seagate NAS

The majority of NAS devices supported in Debian are based on Marvell's Kirkwood platform. This platform is quite dated now and can only run Debian's armel port. Debian now supports the Seagate Personal Cloud and Seagate NAS devices. They are based on Marvell's Armada 370, a platform which can run Debian's armhf port. Unfortunately, even the Armada 370 is a bit dated now, so I would not recommend these devices for new purchases. If you have one already, however, you now have the option to run native Debian. There are some features I like about the Seagate NAS devices:

Network console: you can connect to the boot loader via the network. This is useful to load Debian or to run recovery commands if needed.
Mainline support: the devices are supported in the mainline kernel.
Good contacts: Seagate engineer Simon Guinot is interested in Debian support and is a joy to work with. There's also a community for LaCie NAS devices (Seagate acquired LaCie).

If you have a Seagate Personal Cloud and Seagate NAS, you can follow the instructions on the Debian wiki. If Seagate releases more NAS devices on Marvell's Armada platform, I intend to add Debian support.

Reproducible builds folks: Reproducible builds: week 62 in Stretch cycle

What happened in the Reproducible Builds effort between June 26th and July 2nd 2016: Read on to find out why we're lagging some weeks behind ! GSoC and Outreachy updates

Ceridwen described using autopkgtest code to communicate with containers and how to test the container handling.
reprotest 0.1 has been accepted into Debian unstable, and any user reports, bug reports, feature requests, etc. would be appreciated. This is still an alpha release, and nothing is set in stone.

Toolchain fixes

Matthias Klose uploaded doxygen/1.8.11-3 to Debian unstable (closing #792201) with the upstream patch improving SOURCE_DATE_EPOCH support by using UTC as timezone when parsing the value. This was the last patch we were carrying in our repository, thus this upload obsoletes the version in our experimental repository.
cmake/3.5.2-2 was uploaded by Felix Geyer, which sorts file lists obtained with file(GLOB).
Dmitry Shachnev uploaded sphinx/1.4.4-2, which fixes a timezone related issue when SOURCE_DATE_EPOCH is set.

With the doxygen upload we are now down to only 2 modified packages in our repository: dpkg and rdfind. Weekly reports delay and the future of statistics To catch up with our backlog of weekly reports we have decided to skip some of the statistics for this week. We might publish them in a future report, or we might switch to a format where we summarize them more (and which we can create (even) more automatically), we'll see. We are doing these weekly statistics because we believe it's appropriate and useful to credit people's work and make it more visible. What do you think? We would love to hear your thoughts on this matter! Do you read these statistics? Somewhat? Actually, thanks to the power of notmuch, Holger came up with what you can see below, so what's missing for this week are the uploads fixing irreprodubilities. Which we really would like to show for the reasons stated above and because we really really need these uploads to happen ;-)

But then we also like to confirm the bugs are really gone, which (atm) requires manual checking, and to look for the words "reproducible" and "deterministic" (and spelling variations) in debian/changelogs of all uploads, to spot reproducible work not tracked via the BTS. And we still need to catch up on the backlog of weekly reports. Bugs submitted with reproducible usertags It seems DebCamp in Cape Town was hugely successful and made some people get a lot of work done: 61 bugs have been filed with reproducible builds usertags and 60 of them had patches:

#829365 against pdl by Reiner Herrmann: please make the build reproducible.
#829362 against ruby-ronn by Chris Lamb: please make the output reproducible.
#829357 against ctop by Chris Lamb: please make the build reproducible.
#829325 against txt2tags by Chris Lamb: please make the output reproducible.
#829323 against pdl by Reiner Herrmann: please generated sorted output.
#829322 against libpdl-netcdf-perl by Reiner Herrmann: please make the build reproducible.
#829320 against libpdl-io-hdf5-perl by Reiner Herrmann: please make the build reproducible.
#829297 against check-all-the-things by Chris Lamb: please make the build reproducible.
#829296 against perl by Chris Lamb: please make the output of ParseXS.pm reproducible.
#829295 against libextutils-parsexs-perl by Chris Lamb: please make the output reproducible.
#829270 against tomsfastmath by Reiner Herrmann: please make the build reproducible.
#829263 against libmemcached-libmemcached-perl by Chris Lamb: please make the build reproducible.
#829262 against slashem by Reiner Herrmann: please make the build reproducible.
#829261 against passwordsafe by Reiner Herrmann: please make the build reproducible.
#829249 against ncftp by Reiner Herrmann: please make the build reproducible.
#829133 against icon by Reiner Herrmann: please make the build reproducible.
#829129 against ayttm by Reiner Herrmann: please make the build reproducible.
#829011 against link-grammar by Jeremy Bicha and Chris Lamb: please make the build reproducible.
#829000 against fracplanet by Reiner Herrmann: please make the build reproducible.
#828994 against syncthing by Dhole: please make the build reproducible.
#828993 against openttd by Reiner Herrmann: please make the build reproducible.
#828989 against ntp by Reiner Herrmann: please make the build reproducible.
#828977 against faust by Reiner Herrmann: please make the build reproducible.
#828971 against clasp by Reiner Herrmann: please make the build reproducible.
#828969 against dogecoin by Reiner Herrmann: please make the build reproducible.
#828909 against qtruby by Reiner Herrmann: please make the build reproducible.
#828906 against mailfilter by Reiner Herrmann: please make the build reproducible.
#828898 against lordsawar by Reiner Herrmann: please make the build reproducible.
#828891 against bbdb by Reiner Herrmann: please make the build reproducible.
#828890 against libsdl2-gfx by Reiner Herrmann: please make the build reproducible.
#828888 against aspell-en by Reiner Herrmann: please make the build reproducible.
#828876 against ario by Reiner Herrmann: please make the build reproducible.
#828867 against zephyr by Reiner Herrmann: please make the build reproducible.
#828856 against lrzsz by Reiner Herrmann: please make the build reproducible.
#828855 against wmweather by Reiner Herrmann: please make the build reproducible.
#828852 against moblin-icon-theme by Reiner Herrmann: please make the build reproducible.
#828810 against fakeroot by Juan Picca: [PATCH] Make the build reproducible.
#828793 against minicom by Reiner Herrmann: please make the build reproducible.
#828791 against fio by Reiner Herrmann: please make the build reproducible.
#828790 against debiandoc-sgml-doc by Dhole: please make the build reproducible.
#828788 against pyparted by Reiner Herrmann: please make the build reproducible.
#828786 against tcptraceroute by Reiner Herrmann: please make the build reproducible.
#828780 against iptraf by Reiner Herrmann: please make the build reproducible.
#828778 against module-assistant by Reiner Herrmann: please make the build reproducible.
#828766 against liblucy-perl by Reiner Herrmann: please make the build reproducible.
#828762 against slang2 by Reiner Herrmann: please make the build reproducible.
#828756 against python-reportlab by Reiner Herrmann: please make the build reproducible.
#828752 against mod-dnssd by Reiner Herrmann: please make the build reproducible.
#828749 against telepathy-salut by Reiner Herrmann: please make the build reproducible.
#828748 against libphonenumber by Reiner Herrmann: please make the build reproducible.
#828745 against directfb by Reiner Herrmann: please make the build reproducible.
#828683 against mc by Reiner Herrmann and Yury V. Zaytsev: please make the build reproducible.
#828681 against keyutils by Reiner Herrmann: please make the build reproducible.
#828680 against debiandoc-sgml-doc-pt-br by Dhole: please make the build reproducible.
#828639 against libmarpa-r2-perl by Reiner Herrmann: please make the build reproducible.
#828636 against libembperl-perl by Axel Beckert, Reiner Herrmann and gregor herrmann: please make the build reproducible.
#828635 against libnet-tclink-perl by Reiner Herrmann: please make the build reproducible.
#828628 against media-player-info by Reiner Herrmann: please make the build reproducible.
#828627 against uim by Alexis Bienven e: please make the build reproducible (locale).
#828226 against tiger by Alexis Bienven e: please make the build reproducible (environment, locale).
#828222 against pygoocanvas by Chris Lamb: please make the build reproducible.
#828216 against pyinfra by Daniel Stender: assertion error with timestamps.

Package reviews 437 new reviews have been added (though most of them were just linking the bug, "only" 56 new issues in packages were found), an unknown number has been been updated and 60 have been removed in this week, adding to our knowledge about identified issues. 4 new issue types have been found:

Weekly QA work 98 FTBFS bugs have been reported by Chris Lamb and Santiago Vila. diffoscope development

Reiner Herrmann clarified the help text for the input arguments.
#829115 against diffoscope by Axel Beckert and Mattia Rizzolo:: /comparators/ps.py: TypeError: cannot use a string pattern on a bytes-like object.

strip-nondeterminism development

Chris Lamb made sure that .zhfst files are treated as ZIP files.

tests.reproducible-builds.org

Mattia Rizzolo uploaded pbuilder/0.225.1~bpo8+1 to jessie-backports and it has been installed on all build nodes. As a consequence all armhf and i386 builds will be done with eatmydata; this will hopefully cut down the build time by a noticable factor.

Misc. This week's edition was written by Mattia Rizzolo, Reiner Herrmann, Ceridwen and Holger Levsen and reviewed by a bunch of Reproducible builds folks on IRC.

Reproducible builds folks: Reproducible builds: week 60 in Stretch cycle

What happened in the Reproducible Builds effort between June 12th and June 18th 2016: Media coverage

Sune Vuorela blogged about making Qt documentation reproducible while spending good times in the Alps.

GSoC and Outreachy updates Weekly reports by our participants:

Toolchain fixes

texlive-bin/2016.20160513.41080-3 has been uploaded to unstable, featuring support for FORCE_SOURCE_DATE. See the last post for details on it.
doxygen/1.4.4-1 has been uploaded to unstable, fixing #822197 (upstream bug), which caused some generated html file to contain unreproducible memory addresses of Python objects used at build time.
debhelper/9.20160618 has been uploaded to unstable, fixing #824490, which instructs ant to not save the username of the build user in the generated files. Original patch by Emmanuel Bourg.
HW42 reported a long-known (although only internally) bug in our dpkg-buildinfo. this particular bug doesn't affect our current infrastructure, but it's a blocker for having .buildinfo support merged upstream.
epydoc/3.0.1+dfsg-13 and 3.0.1+dfsg-14 have been uploaded by Kenneth J. Pronovici which fixes nondeterministic ordering issues in generated documentation and removes memory addresses. Original patches (#825968 and #827416) by Sascha Steinbiss.

With this upload of texlive-bin we decided to stop keeping our patched fork of as most of the patches for SOURCE_DATE_EPOCH support had been integrated upstream already, and the last one (making FORCE_SOURCE_DATE default to 1) had been refused. So, we are now going to let the archive be rebuilt against unstable's texlive-bin and see how many packages will become unreproducible with this change; once enough data will be collected we will ponder whether FORCE_SOURCE_DATE should be exported by helper tools (such as debhelper) or manually exported by every package that needs it. (For those wondering: we still recommend to follow SOURCE_DATE_EPOCH always and don't recommend other projects to implement FORCE_SOURCE_DATE ) With the drop of texlive-bin we now have only three modified packages in our experimental repository. Reproducible work in other projects

Ed Maste sent a patch to have ELF Tool Chain's elfcopy support SOURCE_DATE_EPOCH.
Ed Maste changed FreeBSD's ar(1) to have a reproducible output by default when invoked with -s.
Ed Maste merged changes from NetBSD's makefs to FreeBSD's to add a command line option for setting the timestamp.
Ed Maste reported on FreeBSD package reproducibility details from the investigation for his BSDCan talk, including diffoscope results for the non-reproducible packages.
Holger Levsen spent some time thinking and documenting how to store package notes and issues in a multi-distribution friendly manner, so that we can share these issues and notes between reproducible efforts across different projects. Thoughts we had about this topic during the Athens meeting last December are included in the updated README.

Packages fixed The following 12 packages have become reproducible due to changes in their build dependencies: django-floppyforms flask-restful hy jets3t kombu llvm-toolchain-3.8 moap python-bottle python-debtcollector python-django-debug-toolbar python-osprofiler stevedore The following packages have become reproducible after being fixed:

adios/1.9.0-10 by Alastair McKinstry.
airstrike/0.99+1.0pre6a-8 by Markus Koschany, original patch by Reiner Herrmann.
apt-dater/1.0.3-1 by Patrick Matth i, original patch by Chris Lamb.
bitlbee/3.4.2-1 by Jelmer Vernoo .
dar/2.5.5-1 by Laszlo Boszormenyi.
fastjet/3.0.6+dfsg-2 by Mattia Rizzolo, original patch by Maria Valentina Marin.
libretro-beetle-pce-fast/0.9.38.7+git20160609-1 by S rgio Benjamim, original patch by Reiner Herrmann.
libretro-beetle-psx/0.9.38.6+git20151019-2 by S rgio Benjamim, original patch by Reiner Herrmann.
mx/1.99.4-1 by Ying-Chun Liu.
python-coverage/4.1+dfsg.1-1 by Ben Finney.
shiro/1.2.5-1 by Tony Mancill, original patch by Chris Lamb.
splix/2.0.0+svn315-5 by Didier Raboud.
u-boot/2016.07~rc1+dfsg1-3 by Vagrant Cascadian, debugging and patch by HW42, patches submitted upstream.

Some uploads have fixed some reproducibility issues, but not all of them:

gcc-mingw-w64/18 by Stephen Kitt.
gnuplot/5.0.3+dfsg3-6 by Anton Gladky, original patch by Alexis Bienven e.

Uploads with reproducibility fixes that currently fail to build:

ruby2.3/2.3.1-3 by Christian Hofstaedtler, avoids unreproducible rbconfig.rb files by always using bash for building.

Patches submitted that have not made their way to the archive yet:

#827109 against asciijump by Reiner Herrmann: sort source files for deterministic linking order.
#827112 against boswars by Reiner Herrmann: sort source files for deterministic linking order.
#827114 against overgod by Reiner Herrmann: use C locale for sorting source files.
#827115 against netpbm by Alexis Bienven e: honour SOURCE_DATE_EPOCH while generating output.
#827124 against funguloids by Reiner Herrmann: use C locale for sorting files.
#827145 against scummvm by Reiner Herrmann: don't embed extra fields in zip archive and build with proper host architecture.
#827150 against netpanzer by Reiner Herrmann: sort source files for deterministic linking order.
#827172 against reaver by Alexis Bienven e: sort object files for deterministic linking order.
#827187 against latex2html by Alexis Bienven e: iterate deterministic over Perl hashes; honour SOURCE_DATE_EPOCH for output; strip username from output; sort index keys.
#827313 against cherrypy3 by Sascha Steinbiss: prevent memory addresses in output.
#827361 against matplotlib by Alexis Bienven e: honour SOURCE_DATE_EPOCH in output and sort keys while iterating over dict.
#827382 against dwarfutils by Reiner Herrmann: fix array size, which caused memory from outside a table to be embedded into output.
#827384 against skytools3 by Sascha Steinbiss: use stable sorting order and remove timestamps from documentation.
#827419 against ldaptor by Sascha Steinbiss: sort list of input files and prevent home directory from leaking into documentation.
#827546 against git-buildpackage by Sascha Steinbiss: replace timestamps in documentation with changelog date; prevent temporary paths in documentation.
#827572 against xprobe by Reiner Herrmann: sort list of object files in static library archives.

Package reviews 36 reviews have been added, 12 have been updated and 31 have been removed in this week. 17 FTBFS bugs have been reported by Chris Lamb, Santiago Vila and Dominic Hargreaves. diffoscope development Satyam worked on argument completion (#826711) for diffoscope. strip-nondeterminism development Mattia Rizzolo uploaded strip-nondeterminism 0.019-1~bpo8+1 to jessie-backports. reprotest development Ceridwen filed an Intent To Package (ITP) bug for reprotest as #827293. tests.reproducible-builds.org

Mattia Rizzolo uploaded pbuilder 0.225 to unstable, providing built-in support for eatmydata. We're planning to use it in armhf and i386 builders where we don't build in tmpfs, to increase the build speed some more.
Valery Young reworked the appearance of the package page, hopefully making them more intuitive and usable. In the process she changed the script generating them to use a real templating system, thus improving maintenance for the future.
Holger adjusted the scheduler to reschedule packages in state 'depwait' after two days instead of three.
Mattia added the bug title next to the bug numbers in the notes.

Misc. This week's edition was written by Mattia Rizzolo, Reiner Herrmann, Ed Maste and Holger Levsen and reviewed by a bunch of Reproducible builds folks on IRC.

Search Results: "rmh"

21 November 2016

17 November 2016

14 November 2016

13 November 2016

8 November 2016

22 October 2016

19 October 2016

11 October 2016

10 October 2016

26 September 2016

25 September 2016

31 August 2016

28 August 2016

23 August 2016

15 August 2016

14 August 2016

28 July 2016

22 July 2016

21 July 2016

21 June 2016