Whack-A-Mole machines from <https://commons.wikimedia.org/wiki/File:Whac-A-Mole_Cedar_Point.jpg>

This is part of a series of posts on compiling a custom version of Qt5 in order to develop for both amd64 and a Raspberry Pi. Now that I have a sysroot, I try to use it to build Qt5 with QtWebEngine. Nothing seems to work straightforwardly with Qt5's build system, and hit an endless series of significant blockers to try and work around.

Problem in wayland code QtWayland's source currently does not compile:

../../../hardwareintegration/client/brcm-egl/qwaylandbrcmeglwindow.cpp: In constructor  QtWaylandClient::QWaylandBrcmEglWindow::QWaylandBrcmEglWindow(QWindow*) :
../../../hardwareintegration/client/brcm-egl/qwaylandbrcmeglwindow.cpp:131:67: error: no matching function for call to  QtWaylandClient::QWaylandWindow::QWaylandWindow(QWindow*&) 
     , m_eventQueue(wl_display_create_queue(mDisplay->wl_display()))
                                                                   ^
In file included from ../../../../include/QtWaylandClient/5.15.0/QtWaylandClient/private/qwaylandwindow_p.h:1,
                 from ../../../hardwareintegration/client/brcm-egl/qwaylandbrcmeglwindow.h:43,
                 from ../../../hardwareintegration/client/brcm-egl/qwaylandbrcmeglwindow.cpp:40:
../../../../include/QtWaylandClient/5.15.0/QtWaylandClient/private/../../../../../src/client/qwaylandwindow_p.h:97:5: note: candidate:  QtWaylandClient::QWaylandWindow::QWaylandWindow(QWindow*, QtWayland
Client::QWaylandDisplay*) 
     QWaylandWindow(QWindow *window, QWaylandDisplay *display);
     ^~~~~~~~~~~~~~
../../../../include/QtWaylandClient/5.15.0/QtWaylandClient/private/../../../../../src/client/qwaylandwindow_p.h:97:5: note:   candidate expects 2 arguments, 1 provided
make[5]: Leaving directory '/home/build/armhf/qt-everywhere-src-5.15.0/qttools/src/qdoc'

I am not trying to debug here. I understand that Wayland support is not a requirement, and I'm adding -skip wayland to Qt5's configure options. Next round. nss not found Qt5 embeds Chrome's sources. Chrome's sources require libnss3-dev to be available for both host and target architectures. Although I now have it installed both on the build system and in the sysroot, the pkg-config wrapper that Qt5 hooks into its Chrome's sources, failes to find it:

Command: /usr/bin/python2 /home/build/armhf/qt-everywhere-src-5.15.0/qtwebengine/src/3rdparty/chromium/build/config/linux/pkg-config.py -s /home/build/sysroot/ -a arm -p /usr/bin/arm-linux-gnueabihf-pkg-config --system_libdir lib nss -v -lssl3
Returned 1.
stderr:
Package nss was not found in the pkg-config search path.
Perhaps you should add the directory containing  nss.pc'
to the PKG_CONFIG_PATH environment variable
No package 'nss' found
Traceback (most recent call last):
  File "/home/build/armhf/qt-everywhere-src-5.15.0/qtwebengine/src/3rdparty/chromium/build/config/linux/pkg-config.py", line 248, in <module>
    sys.exit(main())
  File "/home/build/armhf/qt-everywhere-src-5.15.0/qtwebengine/src/3rdparty/chromium/build/config/linux/pkg-config.py", line 143, in main
    prefix = GetPkgConfigPrefixToStrip(options, args)
  File "/home/build/armhf/qt-everywhere-src-5.15.0/qtwebengine/src/3rdparty/chromium/build/config/linux/pkg-config.py", line 82, in GetPkgConfigPrefixToStrip
    "--variable=prefix"] + args, env=os.environ).decode('utf-8')
  File "/usr/lib/python2.7/subprocess.py", line 223, in check_output
    raise CalledProcessError(retcode, cmd, output=output)
subprocess.CalledProcessError: Command '['/usr/bin/arm-linux-gnueabihf-pkg-config', '--variable=prefix', 'nss']' returned non-zero exit status 1
See //build/config/linux/nss/BUILD.gn:15:3: whence it was called.
  pkg_config("system_nss_no_ssl_config")  
  ^---------------------------------------
See //crypto/BUILD.gn:218:25: which caused the file to be included.
    public_configs += [ "//build/config/linux/nss:system_nss_no_ssl_config" ]
                        ^--------------------------------------------------
Project ERROR: GN run error!

It's trying to look into $SYSROOT/usr/lib/pkgconfig, while it should be $SYSROOT//usr/lib/arm-linux-gnueabihf/pkgconfig. I worked around this this patch to qtwebengine/src/3rdparty/chromium/build/config/linux/pkg-config.py:

--- pkg-config.py.orig  2020-07-16 11:46:21.005373002 +0200
+++ pkg-config.py   2020-07-16 11:46:02.605296967 +0200
@@ -61,6 +61,7 @@
   libdir = sysroot + '/usr/' + options.system_libdir + '/pkgconfig'
   libdir += ':' + sysroot + '/usr/share/pkgconfig'
+  libdir += ':' + sysroot + '/usr/lib/arm-linux-gnueabihf/pkgconfig'
   os.environ['PKG_CONFIG_LIBDIR'] = libdir
   return libdir

Next round. g++ 8.3.0 Internal Compiler Error Qt5's sources embed Chrome's sources that embed the skia library sources. One of the skia library sources, when cross-compiled to ARM with -O1 or -O2 with g++ 8.3.0, produces an Internal Compiler Error:

/usr/bin/arm-linux-gnueabihf-g++ -MMD -MF obj/skia/skcms/skcms.o.d -DUSE_UDEV -DUSE_AURA=1 -DUSE_NSS_CERTS=1 -DUSE_OZONE=1 -DOFFICIAL_BUILD -DTOOLKIT_QT -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -DNO_UNWIND_TABLES -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -DCR_SYSROOT_HASH=76e6068f9f6954e2ab1ff98ce5fa236d3d85bcbd -DNDEBUG -DNVALGRIND -DDYNAMIC_ANNOTATIONS_ENABLED=0 -I../../3rdparty/chromium/third_party/skia/include/third_party/skcms -Igen -I../../3rdparty/chromium -w -std=c11 -mfp16-format=ieee -fno-strict-aliasing --param=ssp-buffer-size=4 -fstack-protector -fno-unwind-tables -fno-asynchronous-unwind-tables -fPIC -pipe -pthread -march=armv7-a -mfloat-abi=hard -mtune=generic-armv7-a -mfpu=vfpv3-d16 -mthumb -Wall -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -Wno-psabi -Wno-unused-local-typedefs -Wno-maybe-uninitialized -Wno-deprecated-declarations -fno-delete-null-pointer-checks -Wno-comments -Wno-packed-not-aligned -Wno-dangling-else -Wno-missing-field-initializers -Wno-unused-parameter -O2 -fno-ident -fdata-sections -ffunction-sections -fno-omit-frame-pointer -g0 -fvisibility=hidden -std=gnu++14 -Wno-narrowing -Wno-class-memaccess -Wno-attributes -Wno-class-memaccess -Wno-subobject-linkage -Wno-invalid-offsetof -Wno-return-type -Wno-deprecated-copy -fno-exceptions -fno-rtti --sysroot=../../../../../../sysroot/ -fvisibility-inlines-hidden -c ../../3rdparty/chromium/third_party/skia/third_party/skcms/skcms.cc -o obj/skia/skcms/skcms.o
during RTL pass: expand
In file included from ../../3rdparty/chromium/third_party/skia/third_party/skcms/skcms.cc:2053:
../../3rdparty/chromium/third_party/skia/third_party/skcms/src/Transform_inl.h: In function  void baseline::exec_ops(const Op*, const void**, const char*, char*, int) :
../../3rdparty/chromium/third_party/skia/third_party/skcms/src/Transform_inl.h:766:13: internal compiler error: in convert_move, at expr.c:218
 static void exec_ops(const Op* ops, const void** args,
             ^~~~~~~~
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-8/README.Bugs> for instructions.

I reported the bug at https://gcc.gnu.org/bugzilla/show_bug.cgi?id=96206 Since this source compiles with -O0, I attempted to fix this by editing qtwebkit/src/3rdparty/chromium/build/config/compiler/BUILD.gn and replacing instances of -O1 and -O2 with -O0. Spoiler: wrong attempt. We'll see it in the next round. Impossible constraint in asm Qt5's sources embed Chrome's sources that embed the ffmpeg library sources. Even if ffmpeg's development libraries are present both in the host and in the target system, the build system insists in compiling and using the bundled version. Unfortunately, using -O0 breaks the build of ffmpeg:

/usr/bin/arm-linux-gnueabihf-gcc -MMD -MF obj/third_party/ffmpeg/ffmpeg_internal/opus.o.d -DHAVE_AV_CONFIG_H -D_POSIX_C_SOURCE=200112 -D_XOPEN_SOURCE=600 -DPIC -DFFMPEG_CONFIGURATION=NULL -DCHROMIUM_NO_LOGGING -D_ISOC99_SOURCE -D_LARGEFILE_SOURCE -DUSE_UDEV -DUSE_AURA=1 -DUSE_NSS_CERTS=1 -DUSE_OZONE=1 -DOFFICIAL_BUILD -DTOOLKIT_QT -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -DNO_UNWIND_TABLES -DCR_SYSROOT_HASH=76e6068f9f6954e2ab1ff98ce5fa236d3d85bcbd -DNDEBUG -DNVALGRIND -DDYNAMIC_ANNOTATIONS_ENABLED=0 -DOPUS_FIXED_POINT -I../../3rdparty/chromium/third_party/ffmpeg/chromium/config/Chromium/linux/arm -I../../3rdparty/chromium/third_party/ffmpeg -I../../3rdparty/chromium/third_party/ffmpeg/compat/atomics/gcc -Igen -I../../3rdparty/chromium -I../../3rdparty/chromium/third_party/opus/src/include -fPIC -Wno-deprecated-declarations -fomit-frame-pointer -w -std=c99 -pthread -fno-math-errno -fno-signed-zeros -fno-tree-vectorize -fno-strict-aliasing --param=ssp-buffer-size=4 -fstack-protector -fno-unwind-tables -fno-asynchronous-unwind-tables -fPIC -pipe -pthread -march=armv7-a -mfloat-abi=hard -mtune=generic-armv7-a -mfpu=vfpv3-d16 -mthumb -g0 -fvisibility=hidden -Wno-psabi -Wno-unused-local-typedefs -Wno-maybe-uninitialized -Wno-deprecated-declarations -fno-delete-null-pointer-checks -Wno-comments -Wno-packed-not-aligned -Wno-dangling-else -Wno-missing-field-initializers -Wno-unused-parameter -O0 -fno-ident -fdata-sections -ffunction-sections -std=gnu11 --sysroot=../../../../../../sysroot/ -c ../../3rdparty/chromium/third_party/ffmpeg/libavcodec/opus.c -o obj/third_party/ffmpeg/ffmpeg_internal/opus.o
In file included from ../../3rdparty/chromium/third_party/ffmpeg/libavutil/intmath.h:30,
                 from ../../3rdparty/chromium/third_party/ffmpeg/libavutil/common.h:106,
                 from ../../3rdparty/chromium/third_party/ffmpeg/libavutil/avutil.h:296,
                 from ../../3rdparty/chromium/third_party/ffmpeg/libavutil/audio_fifo.h:30,
                 from ../../3rdparty/chromium/third_party/ffmpeg/libavcodec/opus.h:28,
                 from ../../3rdparty/chromium/third_party/ffmpeg/libavcodec/opus_celt.h:29,
                 from ../../3rdparty/chromium/third_party/ffmpeg/libavcodec/opus.c:32:
../../3rdparty/chromium/third_party/ffmpeg/libavcodec/opus.c: In function  ff_celt_quant_bands :
../../3rdparty/chromium/third_party/ffmpeg/libavutil/arm/intmath.h:77:5: error: impossible constraint in  asm 
     __asm__ ("usat %0, %2, %1" : "=r"(x) : "r"(a), "i"(p));
     ^~~~~~~

The same source compiles with using -O2 instead of -O0. I worked around this by undoing the previous change, and limiting -O0 to just the source that causes the Internal Compiler Error. I edited qtwebengine/src/3rdparty/chromium/third_party/skia/third_party/skcms/skcms.cc to prepend:

#pragma GCC push_options
#pragma GCC optimize ("O0")

and append:

#pragma GCC pop_options

Next round. Missing build-deps for i386 code Qt5's sources embed Chrome's sources that embed the V8 library sources. For some reason, torque, that is part of V8, wants to build some of its sources into 32 bit code with -m32, and I did not have i386 cross-compilation libraries installed:

/usr/bin/g++ -MMD -MF v8_snapshot/obj/v8/torque_base/csa-generator.o.d -DUSE_UDEV -DUSE_AURA=1 -DUSE_NSS_CERTS=1 -DUSE_OZONE=1 -DOFFICIAL_BUILD -DTOOLKIT_QT -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -DNO_UNWIND_TABLES -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -DNDEBUG -DNVALGRIND -DDYNAMIC_ANNOTATIONS_ENABLED=0 -DV8_TYPED_ARRAY_MAX_SIZE_IN_HEAP=64 -DENABLE_MINOR_MC -DV8_INTL_SUPPORT -DV8_CONCURRENT_MARKING -DV8_ENABLE_LAZY_SOURCE_POSITIONS -DV8_EMBEDDED_BUILTINS -DV8_SHARED_RO_HEAP -DV8_WIN64_UNWINDING_INFO -DV8_ENABLE_REGEXP_INTERPRETER_THREADED_DISPATCH -DV8_31BIT_SMIS_ON_64BIT_ARCH -DV8_DEPRECATION_WARNINGS -DV8_TARGET_ARCH_ARM -DCAN_USE_ARMV7_INSTRUCTIONS -DCAN_USE_VFP3_INSTRUCTIONS -DUSE_EABI_HARDFLOAT=1 -DV8_HAVE_TARGET_OS -DV8_TARGET_OS_LINUX -DDISABLE_UNTRUSTED_CODE_MITIGATIONS -DV8_31BIT_SMIS_ON_64BIT_ARCH -DV8_DEPRECATION_WARNINGS -Iv8_snapshot/gen -I../../3rdparty/chromium -I../../3rdparty/chromium/v8 -Iv8_snapshot/gen/v8 -fno-strict-aliasing --param=ssp-buffer-size=4 -fstack-protector -fno-unwind-tables -fno-asynchronous-unwind-tables -fPIC -pipe -pthread -m32 -msse2 -mfpmath=sse -mmmx -Wall -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -Wno-unused-local-typedefs -Wno-maybe-uninitialized -Wno-deprecated-declarations -fno-delete-null-pointer-checks -Wno-comments -Wno-packed-not-aligned -Wno-dangling-else -Wno-missing-field-initializers -Wno-unused-parameter -fno-omit-frame-pointer -g0 -fvisibility=hidden -Wno-strict-overflow -Wno-return-type -O3 -fno-ident -fdata-sections -ffunction-sections -std=gnu++14 -Wno-narrowing -Wno-class-memaccess -Wno-attributes -Wno-class-memaccess -Wno-subobject-linkage -Wno-invalid-offsetof -Wno-return-type -Wno-deprecated-copy -fvisibility-inlines-hidden -fexceptions -frtti -c ../../3rdparty/chromium/v8/src/torque/csa-generator.cc -o v8_snapshot/obj/v8/torque_base/csa-generator.o
In file included from ../../3rdparty/chromium/v8/src/torque/csa-generator.h:8,
                 from ../../3rdparty/chromium/v8/src/torque/csa-generator.cc:5:
/usr/include/c++/8/iostream:38:10: fatal error: bits/c++config.h: No such file or directory
 #include <bits/c++config.h>
          ^~~~~~~~~~~~~~~~~~
compilation terminated.

New build dependencies needed:

apt install lib32stdc++-8-dev
apt install libc6-dev-i386
dpkg --add-architecture i386
apt install linux-libc-dev:i386

Next round. OpenGL build issues Next bump are OpenGL related compiler issues:

/usr/bin/arm-linux-gnueabihf-g++ -MMD -MF obj/QtWebEngineCore/gl_ozone_glx_qt.o.d -DCHROMIUM_VERSION=\"80.0.3987.163\" -DUSE_UDEV -DUSE_AURA=1 -DUSE_NSS_CERTS=1 -DUSE_OZONE=1 -DOFFICIAL_BUILD -DTOOLKIT_QT -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -DNO_UNWIND_TABLES -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -DCR_SYSROOT_HASH=76e6068f9f6954e2ab1ff98ce5fa236d3d85bcbd -DNDEBUG -DNVALGRIND -DDYNAMIC_ANNOTATIONS_ENABLED=0 -DQT_NO_LINKED_LIST -DQT_NO_KEYWORDS -DQT_USE_QSTRINGBUILDER -DQ_FORWARD_DECLARE_OBJC_CLASS=QT_FORWARD_DECLARE_CLASS -DQTWEBENGINECORE_VERSION_STR=\"5.15.0\" -DQTWEBENGINEPROCESS_NAME=\"QtWebEngineProcess\" -DBUILDING_CHROMIUM -DQTWEBENGINE_EMBEDDED_SWITCHES -DQT_NO_EXCEPTIONS -D_LARGEFILE64_SOURCE -D_LARGEFILE_SOURCE -DQT_NO_DEBUG -DQT_QUICK_LIB -DQT_GUI_LIB -DQT_QMLMODELS_LIB -DQT_WEBCHANNEL_LIB -DQT_QML_LIB -DQT_NETWORK_LIB -DQT_POSITIONING_LIB -DQT_CORE_LIB -DQT_WEBENGINECOREHEADERS_LIB -DVK_NO_PROTOTYPES -DGL_GLEXT_PROTOTYPES -DUSE_GLX -DUSE_EGL -DGOOGLE_PROTOBUF_NO_RTTI -DGOOGLE_PROTOBUF_NO_STATIC_INITIALIZER -DHAVE_PTHREAD -DU_USING_ICU_NAMESPACE=0 -DU_ENABLE_DYLOAD=0 -DUSE_CHROMIUM_ICU=1 -DU_STATIC_IMPLEMENTATION -DICU_UTIL_DATA_IMPL=ICU_UTIL_DATA_FILE -DUCHAR_TYPE=uint16_t -DWEBRTC_NON_STATIC_TRACE_EVENT_HANDLERS=0 -DWEBRTC_CHROMIUM_BUILD -DWEBRTC_POSIX -DWEBRTC_LINUX -DABSL_ALLOCATOR_NOTHROW=1 -DWEBRTC_USE_BUILTIN_ISAC_FIX=1 -DWEBRTC_USE_BUILTIN_ISAC_FLOAT=0 -DHAVE_SCTP -DNO_MAIN_THREAD_WRAPPING -DSK_HAS_PNG_LIBRARY -DSK_HAS_WEBP_LIBRARY -DSK_USER_CONFIG_HEADER=\"../../skia/config/SkUserConfig.h\" -DSK_GL -DSK_HAS_JPEG_LIBRARY -DSK_USE_LIBGIFCODEC -DSK_VULKAN_HEADER=\"../../skia/config/SkVulkanConfig.h\" -DSK_VULKAN=1 -DSK_SUPPORT_GPU=1 -DSK_GPU_WORKAROUNDS_HEADER=\"gpu/config/gpu_driver_bug_workaround_autogen.h\" -DVK_NO_PROTOTYPES -DLEVELDB_PLATFORM_CHROMIUM=1 -DLEVELDB_PLATFORM_CHROMIUM=1 -DV8_31BIT_SMIS_ON_64BIT_ARCH -DV8_DEPRECATION_WARNINGS -I../../3rdparty/chromium/skia/config -I../../3rdparty/chromium/third_party -I../../3rdparty/chromium/third_party/boringssl/src/include -I../../3rdparty/chromium/third_party/skia/include/core -Igen -I../../3rdparty/chromium -I/home/build/armhf/qt-everywhere-src-5.15.0/qtwebengine/src/core -I/home/build/armhf/qt-everywhere-src-5.15.0/qtwebengine/src/core/api -I/home/build/armhf/qt-everywhere-src-5.15.0/qtdeclarative/include/QtQuick/5.15.0 -I/home/build/armhf/qt-everywhere-src-5.15.0/qtdeclarative/include/QtQuick/5.15.0/QtQuick -I/home/build/armhf/qt-everywhere-src-5.15.0/qtbase/include/QtGui/5.15.0 -I/home/build/armhf/qt-everywhere-src-5.15.0/qtbase/include/QtGui/5.15.0/QtGui -I/home/build/armhf/qt-everywhere-src-5.15.0/qtdeclarative/include -I/home/build/armhf/qt-everywhere-src-5.15.0/qtdeclarative/include/QtQuick -I/home/build/armhf/qt-everywhere-src-5.15.0/qtbase/include -I/home/build/armhf/qt-everywhere-src-5.15.0/qtbase/include/QtGui -I/home/build/armhf/qt-everywhere-src-5.15.0/qtdeclarative/include/QtQmlModels/5.15.0 -I/home/build/armhf/qt-everywhere-src-5.15.0/qtdeclarative/include/QtQmlModels/5.15.0/QtQmlModels -I/home/build/armhf/qt-everywhere-src-5.15.0/qtdeclarative/include/QtQml/5.15.0 -I/home/build/armhf/qt-everywhere-src-5.15.0/qtdeclarative/include/QtQml/5.15.0/QtQml -I/home/build/armhf/qt-everywhere-src-5.15.0/qtbase/include/QtCore/5.15.0 -I/home/build/armhf/qt-everywhere-src-5.15.0/qtbase/include/QtCore/5.15.0/QtCore -I/home/build/armhf/qt-everywhere-src-5.15.0/qtdeclarative/include/QtQmlModels -I/home/build/armhf/qt-everywhere-src-5.15.0/qtwebchannel/include -I/home/build/armhf/qt-everywhere-src-5.15.0/qtwebchannel/include/QtWebChannel -I/home/build/armhf/qt-everywhere-src-5.15.0/qtdeclarative/include/QtQml -I/home/build/armhf/qt-everywhere-src-5.15.0/qtbase/include/QtNetwork -I/home/build/armhf/qt-everywhere-src-5.15.0/qtlocation/include -I/home/build/armhf/qt-everywhere-src-5.15.0/qtlocation/include/QtPositioning -I/home/build/armhf/qt-everywhere-src-5.15.0/qtbase/include/QtCore -I/home/build/armhf/qt-everywhere-src-5.15.0/qtwebengine/include -I/home/build/armhf/qt-everywhere-src-5.15.0/qtwebengine/include/QtWebEngineCore -I/home/build/armhf/qt-everywhere-src-5.15.0/qtwebengine/include/QtWebEngineCore/5.15.0 -I/home/build/armhf/qt-everywhere-src-5.15.0/qtwebengine/include/QtWebEngineCore/5.15.0/QtWebEngineCore -I.moc -I/home/build/sysroot/opt/vc/include -I/home/build/sysroot/opt/vc/include/interface/vcos/pthreads -I/home/build/sysroot/opt/vc/include/interface/vmcs_host/linux -Igen/.moc -I/home/build/armhf/qt-everywhere-src-5.15.0/qtbase/mkspecs/devices/linux-rasp-pi2-g++ -Igen -Igen -I../../3rdparty/chromium/third_party/libyuv/include -Igen -I../../3rdparty/chromium/third_party/jsoncpp/source/include -I../../3rdparty/chromium/third_party/jsoncpp/generated -Igen -Igen -I../../3rdparty/chromium/third_party/khronos -I../../3rdparty/chromium/gpu -I../../3rdparty/chromium/third_party/vulkan/include -I../../3rdparty/chromium/third_party/perfetto/include -Igen/third_party/perfetto/build_config -Igen -Igen -Igen/third_party/dawn/src/include -I../../3rdparty/chromium/third_party/dawn/src/include -Igen -I../../3rdparty/chromium/third_party/boringssl/src/include -I../../3rdparty/chromium/third_party/protobuf/src -Igen/protoc_out -I../../3rdparty/chromium/third_party/protobuf/src -I../../3rdparty/chromium/third_party/ced/src -I../../3rdparty/chromium/third_party/icu/source/common -I../../3rdparty/chromium/third_party/icu/source/i18n -I../../3rdparty/chromium/third_party/webrtc_overrides -I../../3rdparty/chromium/third_party/webrtc -Igen/third_party/webrtc -I../../3rdparty/chromium/third_party/abseil-cpp -I../../3rdparty/chromium/third_party/skia -I../../3rdparty/chromium/third_party/libgifcodec -I../../3rdparty/chromium/third_party/vulkan/include -I../../3rdparty/chromium/third_party/skia/third_party/vulkanmemoryallocator -I../../3rdparty/chromium/third_party/vulkan/include -Igen/third_party/perfetto -Igen/third_party/perfetto -Igen/third_party/perfetto -Igen/third_party/perfetto -Igen/third_party/perfetto -Igen/third_party/perfetto -Igen/third_party/perfetto -Igen/third_party/perfetto -Igen/third_party/perfetto -Igen/third_party/perfetto -I../../3rdparty/chromium/third_party/crashpad/crashpad -I../../3rdparty/chromium/third_party/crashpad/crashpad/compat/non_mac -I../../3rdparty/chromium/third_party/crashpad/crashpad/compat/linux -I../../3rdparty/chromium/third_party/crashpad/crashpad/compat/non_win -I../../3rdparty/chromium/third_party/libwebm/source -I../../3rdparty/chromium/third_party/leveldatabase -I../../3rdparty/chromium/third_party/leveldatabase/src -I../../3rdparty/chromium/third_party/leveldatabase/src/include -I../../3rdparty/chromium/v8/include -Igen/v8/include -I../../3rdparty/chromium/third_party/mesa_headers -fno-strict-aliasing --param=ssp-buffer-size=4 -fstack-protector -fno-unwind-tables -fno-asynchronous-unwind-tables -fPIC -pipe -pthread -march=armv7-a -mfloat-abi=hard -mtune=generic-armv7-a -mfpu=vfpv3-d16 -mthumb -Wall -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -Wno-psabi -Wno-unused-local-typedefs -Wno-maybe-uninitialized -Wno-deprecated-declarations -fno-delete-null-pointer-checks -Wno-comments -Wno-packed-not-aligned -Wno-dangling-else -Wno-missing-field-initializers -Wno-unused-parameter -O2 -fno-ident -fdata-sections -ffunction-sections -fno-omit-frame-pointer -g0 -fvisibility=hidden -g -O2 -fdebug-prefix-map=/home/build/armhf/qt-everywhere-src-5.15.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -O2 -fno-exceptions -Wall -Wextra -D_REENTRANT -I/home/build/sysroot/usr/include/nss -I/home/build/sysroot/usr/include/nspr -std=gnu++14 -Wno-narrowing -Wno-class-memaccess -Wno-attributes -Wno-class-memaccess -Wno-subobject-linkage -Wno-invalid-offsetof -Wno-return-type -Wno-deprecated-copy -fno-exceptions -fno-rtti --sysroot=../../../../../../sysroot/ -fvisibility-inlines-hidden -g -O2 -fdebug-prefix-map=/home/build/armhf/qt-everywhere-src-5.15.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -O2 -std=gnu++1y -fno-exceptions -Wall -Wextra -D_REENTRANT -Wno-unused-parameter -Wno-unused-variable -Wno-deprecated-declarations -c /home/build/armhf/qt-everywhere-src-5.15.0/qtwebengine/src/core/ozone/gl_ozone_glx_qt.cpp -o obj/QtWebEngineCore/gl_ozone_glx_qt.o
In file included from ../../3rdparty/chromium/ui/gl/gl_bindings.h:497,
                 from ../../3rdparty/chromium/ui/gl/gl_gl_api_implementation.h:12,
                 from /home/build/armhf/qt-everywhere-src-5.15.0/qtwebengine/src/core/ozone/gl_ozone_glx_qt.cpp:49:
../../3rdparty/chromium/ui/gl/gl_bindings_autogen_egl.h:227:5: error:  EGLSetBlobFuncANDROID  has not been declared
     EGLSetBlobFuncANDROID set,
     ^~~~~~~~~~~~~~~~~~~~~
../../3rdparty/chromium/ui/gl/gl_bindings_autogen_egl.h:228:5: error:  EGLGetBlobFuncANDROID  has not been declared
     EGLGetBlobFuncANDROID get);
     ^~~~~~~~~~~~~~~~~~~~~
../../3rdparty/chromium/ui/gl/gl_bindings_autogen_egl.h:571:46: error:  EGLSetBlobFuncANDROID  has not been declared
                                              EGLSetBlobFuncANDROID set,
                                              ^~~~~~~~~~~~~~~~~~~~~
../../3rdparty/chromium/ui/gl/gl_bindings_autogen_egl.h:572:46: error:  EGLGetBlobFuncANDROID  has not been declared
                                              EGLGetBlobFuncANDROID get) = 0;
                                              ^~~~~~~~~~~~~~~~~~~~~
cc1plus: warning: unrecognized command line option  -Wno-deprecated-copy

/usr/bin/arm-linux-gnueabihf-g++ -MMD -MF obj/QtWebEngineCore/display_gl_output_surface.o.d -DCHROMIUM_VERSION=\"80.0.3987.163\" -DUSE_UDEV -DUSE_AURA=1 -DUSE_NSS_CERTS=1 -DUSE_OZONE=1 -DOFFICIAL_BUILD -DTOOLKIT_QT -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -DNO_UNWIND_TABLES -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -DCR_SYSROOT_HASH=76e6068f9f6954e2ab1ff98ce5fa236d3d85bcbd -DNDEBUG -DNVALGRIND -DDYNAMIC_ANNOTATIONS_ENABLED=0 -DQT_NO_LINKED_LIST -DQT_NO_KEYWORDS -DQT_USE_QSTRINGBUILDER -DQ_FORWARD_DECLARE_OBJC_CLASS=QT_FORWARD_DECLARE_CLASS -DQTWEBENGINECORE_VERSION_STR=\"5.15.0\" -DQTWEBENGINEPROCESS_NAME=\"QtWebEngineProcess\" -DBUILDING_CHROMIUM -DQTWEBENGINE_EMBEDDED_SWITCHES -DQT_NO_EXCEPTIONS -D_LARGEFILE64_SOURCE -D_LARGEFILE_SOURCE -DQT_NO_DEBUG -DQT_QUICK_LIB -DQT_GUI_LIB -DQT_QMLMODELS_LIB -DQT_WEBCHANNEL_LIB -DQT_QML_LIB -DQT_NETWORK_LIB -DQT_POSITIONING_LIB -DQT_CORE_LIB -DQT_WEBENGINECOREHEADERS_LIB -DVK_NO_PROTOTYPES -DGL_GLEXT_PROTOTYPES -DUSE_GLX -DUSE_EGL -DGOOGLE_PROTOBUF_NO_RTTI -DGOOGLE_PROTOBUF_NO_STATIC_INITIALIZER -DHAVE_PTHREAD -DU_USING_ICU_NAMESPACE=0 -DU_ENABLE_DYLOAD=0 -DUSE_CHROMIUM_ICU=1 -DU_STATIC_IMPLEMENTATION -DICU_UTIL_DATA_IMPL=ICU_UTIL_DATA_FILE -DUCHAR_TYPE=uint16_t -DWEBRTC_NON_STATIC_TRACE_EVENT_HANDLERS=0 -DWEBRTC_CHROMIUM_BUILD -DWEBRTC_POSIX -DWEBRTC_LINUX -DABSL_ALLOCATOR_NOTHROW=1 -DWEBRTC_USE_BUILTIN_ISAC_FIX=1 -DWEBRTC_USE_BUILTIN_ISAC_FLOAT=0 -DHAVE_SCTP -DNO_MAIN_THREAD_WRAPPING -DSK_HAS_PNG_LIBRARY -DSK_HAS_WEBP_LIBRARY -DSK_USER_CONFIG_HEADER=\"../../skia/config/SkUserConfig.h\" -DSK_GL -DSK_HAS_JPEG_LIBRARY -DSK_USE_LIBGIFCODEC -DSK_VULKAN_HEADER=\"../../skia/config/SkVulkanConfig.h\" -DSK_VULKAN=1 -DSK_SUPPORT_GPU=1 -DSK_GPU_WORKAROUNDS_HEADER=\"gpu/config/gpu_driver_bug_workaround_autogen.h\" -DVK_NO_PROTOTYPES -DLEVELDB_PLATFORM_CHROMIUM=1 -DLEVELDB_PLATFORM_CHROMIUM=1 -DV8_31BIT_SMIS_ON_64BIT_ARCH -DV8_DEPRECATION_WARNINGS -I../../3rdparty/chromium/skia/config -I../../3rdparty/chromium/third_party -I../../3rdparty/chromium/third_party/boringssl/src/include -I../../3rdparty/chromium/third_party/skia/include/core -Igen -I../../3rdparty/chromium -I/home/build/armhf/qt-everywhere-src-5.15.0/qtwebengine/src/core -I/home/build/armhf/qt-everywhere-src-5.15.0/qtwebengine/src/core/api -I/home/build/armhf/qt-everywhere-src-5.15.0/qtdeclarative/include/QtQuick/5.15.0 -I/home/build/armhf/qt-everywhere-src-5.15.0/qtdeclarative/include/QtQuick/5.15.0/QtQuick -I/home/build/armhf/qt-everywhere-src-5.15.0/qtbase/include/QtGui/5.15.0 -I/home/build/armhf/qt-everywhere-src-5.15.0/qtbase/include/QtGui/5.15.0/QtGui -I/home/build/armhf/qt-everywhere-src-5.15.0/qtdeclarative/include -I/home/build/armhf/qt-everywhere-src-5.15.0/qtdeclarative/include/QtQuick -I/home/build/armhf/qt-everywhere-src-5.15.0/qtbase/include -I/home/build/armhf/qt-everywhere-src-5.15.0/qtbase/include/QtGui -I/home/build/armhf/qt-everywhere-src-5.15.0/qtdeclarative/include/QtQmlModels/5.15.0 -I/home/build/armhf/qt-everywhere-src-5.15.0/qtdeclarative/include/QtQmlModels/5.15.0/QtQmlModels -I/home/build/armhf/qt-everywhere-src-5.15.0/qtdeclarative/include/QtQml/5.15.0 -I/home/build/armhf/qt-everywhere-src-5.15.0/qtdeclarative/include/QtQml/5.15.0/QtQml -I/home/build/armhf/qt-everywhere-src-5.15.0/qtbase/include/QtCore/5.15.0 -I/home/build/armhf/qt-everywhere-src-5.15.0/qtbase/include/QtCore/5.15.0/QtCore -I/home/build/armhf/qt-everywhere-src-5.15.0/qtdeclarative/include/QtQmlModels -I/home/build/armhf/qt-everywhere-src-5.15.0/qtwebchannel/include -I/home/build/armhf/qt-everywhere-src-5.15.0/qtwebchannel/include/QtWebChannel -I/home/build/armhf/qt-everywhere-src-5.15.0/qtdeclarative/include/QtQml -I/home/build/armhf/qt-everywhere-src-5.15.0/qtbase/include/QtNetwork -I/home/build/armhf/qt-everywhere-src-5.15.0/qtlocation/include -I/home/build/armhf/qt-everywhere-src-5.15.0/qtlocation/include/QtPositioning -I/home/build/armhf/qt-everywhere-src-5.15.0/qtbase/include/QtCore -I/home/build/armhf/qt-everywhere-src-5.15.0/qtwebengine/include -I/home/build/armhf/qt-everywhere-src-5.15.0/qtwebengine/include/QtWebEngineCore -I/home/build/armhf/qt-everywhere-src-5.15.0/qtwebengine/include/QtWebEngineCore/5.15.0 -I/home/build/armhf/qt-everywhere-src-5.15.0/qtwebengine/include/QtWebEngineCore/5.15.0/QtWebEngineCore -I.moc -I/home/build/sysroot/opt/vc/include -I/home/build/sysroot/opt/vc/include/interface/vcos/pthreads -I/home/build/sysroot/opt/vc/include/interface/vmcs_host/linux -Igen/.moc -I/home/build/armhf/qt-everywhere-src-5.15.0/qtbase/mkspecs/devices/linux-rasp-pi2-g++ -Igen -Igen -I../../3rdparty/chromium/third_party/libyuv/include -Igen -I../../3rdparty/chromium/third_party/jsoncpp/source/include -I../../3rdparty/chromium/third_party/jsoncpp/generated -Igen -Igen -I../../3rdparty/chromium/third_party/khronos -I../../3rdparty/chromium/gpu -I../../3rdparty/chromium/third_party/vulkan/include -I../../3rdparty/chromium/third_party/perfetto/include -Igen/third_party/perfetto/build_config -Igen -Igen -Igen/third_party/dawn/src/include -I../../3rdparty/chromium/third_party/dawn/src/include -Igen -I../../3rdparty/chromium/third_party/boringssl/src/include -I../../3rdparty/chromium/third_party/protobuf/src -Igen/protoc_out -I../../3rdparty/chromium/third_party/protobuf/src -I../../3rdparty/chromium/third_party/ced/src -I../../3rdparty/chromium/third_party/icu/source/common -I../../3rdparty/chromium/third_party/icu/source/i18n -I../../3rdparty/chromium/third_party/webrtc_overrides -I../../3rdparty/chromium/third_party/webrtc -Igen/third_party/webrtc -I../../3rdparty/chromium/third_party/abseil-cpp -I../../3rdparty/chromium/third_party/skia -I../../3rdparty/chromium/third_party/libgifcodec -I../../3rdparty/chromium/third_party/vulkan/include -I../../3rdparty/chromium/third_party/skia/third_party/vulkanmemoryallocator -I../../3rdparty/chromium/third_party/vulkan/include -Igen/third_party/perfetto -Igen/third_party/perfetto -Igen/third_party/perfetto -Igen/third_party/perfetto -Igen/third_party/perfetto -Igen/third_party/perfetto -Igen/third_party/perfetto -Igen/third_party/perfetto -Igen/third_party/perfetto -Igen/third_party/perfetto -I../../3rdparty/chromium/third_party/crashpad/crashpad -I../../3rdparty/chromium/third_party/crashpad/crashpad/compat/non_mac -I../../3rdparty/chromium/third_party/crashpad/crashpad/compat/linux -I../../3rdparty/chromium/third_party/crashpad/crashpad/compat/non_win -I../../3rdparty/chromium/third_party/libwebm/source -I../../3rdparty/chromium/third_party/leveldatabase -I../../3rdparty/chromium/third_party/leveldatabase/src -I../../3rdparty/chromium/third_party/leveldatabase/src/include -I../../3rdparty/chromium/v8/include -Igen/v8/include -I../../3rdparty/chromium/third_party/mesa_headers -fno-strict-aliasing --param=ssp-buffer-size=4 -fstack-protector -fno-unwind-tables -fno-asynchronous-unwind-tables -fPIC -pipe -pthread -march=armv7-a -mfloat-abi=hard -mtune=generic-armv7-a -mfpu=vfpv3-d16 -mthumb -Wall -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -Wno-psabi -Wno-unused-local-typedefs -Wno-maybe-uninitialized -Wno-deprecated-declarations -fno-delete-null-pointer-checks -Wno-comments -Wno-packed-not-aligned -Wno-dangling-else -Wno-missing-field-initializers -Wno-unused-parameter -O2 -fno-ident -fdata-sections -ffunction-sections -fno-omit-frame-pointer -g0 -fvisibility=hidden -g -O2 -fdebug-prefix-map=/home/build/armhf/qt-everywhere-src-5.15.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -O2 -fno-exceptions -Wall -Wextra -D_REENTRANT -I/home/build/sysroot/usr/include/nss -I/home/build/sysroot/usr/include/nspr -std=gnu++14 -Wno-narrowing -Wno-class-memaccess -Wno-attributes -Wno-class-memaccess -Wno-subobject-linkage -Wno-invalid-offsetof -Wno-return-type -Wno-deprecated-copy -fno-exceptions -fno-rtti --sysroot=../../../../../../sysroot/ -fvisibility-inlines-hidden -g -O2 -fdebug-prefix-map=/home/build/armhf/qt-everywhere-src-5.15.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -O2 -std=gnu++1y -fno-exceptions -Wall -Wextra -D_REENTRANT -Wno-unused-parameter -Wno-unused-variable -Wno-deprecated-declarations -c /home/build/armhf/qt-everywhere-src-5.15.0/qtwebengine/src/core/compositor/display_gl_output_surface.cpp -o obj/QtWebEngineCore/display_gl_output_surface.o
In file included from ../../3rdparty/chromium/gpu/command_buffer/client/gles2_interface.h:8,
                 from ../../3rdparty/chromium/gpu/command_buffer/client/client_transfer_cache.h:15,
                 from ../../3rdparty/chromium/gpu/command_buffer/client/gles2_implementation.h:28,
                 from /home/build/armhf/qt-everywhere-src-5.15.0/qtwebengine/src/core/compositor/display_gl_output_surface.cpp:47:
/home/build/sysroot/opt/vc/include/GLES2/gl2.h:78: warning: "GL_FALSE" redefined
 #define GL_FALSE                          (GLboolean)0
In file included from ../../3rdparty/chromium/gpu/command_buffer/client/client_context_state.h:10,
                 from ../../3rdparty/chromium/gpu/command_buffer/client/gles2_implementation.h:27,
                 from /home/build/armhf/qt-everywhere-src-5.15.0/qtwebengine/src/core/compositor/display_gl_output_surface.cpp:47:
../../3rdparty/chromium/third_party/khronos/GLES3/gl3.h:85: note: this is the location of the previous definition
 #define GL_FALSE                          0
In file included from ../../3rdparty/chromium/gpu/command_buffer/client/gles2_interface.h:8,
                 from ../../3rdparty/chromium/gpu/command_buffer/client/client_transfer_cache.h:15,
                 from ../../3rdparty/chromium/gpu/command_buffer/client/gles2_implementation.h:28,
                 from /home/build/armhf/qt-everywhere-src-5.15.0/qtwebengine/src/core/compositor/display_gl_output_surface.cpp:47:
/home/build/sysroot/opt/vc/include/GLES2/gl2.h:79: warning: "GL_TRUE" redefined
 #define GL_TRUE                           (GLboolean)1
In file included from ../../3rdparty/chromium/gpu/command_buffer/client/client_context_state.h:10,
                 from ../../3rdparty/chromium/gpu/command_buffer/client/gles2_implementation.h:27,
                 from /home/build/armhf/qt-everywhere-src-5.15.0/qtwebengine/src/core/compositor/display_gl_output_surface.cpp:47:
../../3rdparty/chromium/third_party/khronos/GLES3/gl3.h:86: note: this is the location of the previous definition
 #define GL_TRUE                           1
In file included from ../../3rdparty/chromium/gpu/command_buffer/client/gles2_interface.h:8,
                 from ../../3rdparty/chromium/gpu/command_buffer/client/client_transfer_cache.h:15,
                 from ../../3rdparty/chromium/gpu/command_buffer/client/gles2_implementation.h:28,
                 from /home/build/armhf/qt-everywhere-src-5.15.0/qtwebengine/src/core/compositor/display_gl_output_surface.cpp:47:
/home/build/sysroot/opt/vc/include/GLES2/gl2.h:600:37: error: conflicting declaration of C function  void glShaderSource(GLuint, GLsizei, const GLchar**, const GLint*) 
 GL_APICALL void         GL_APIENTRY glShaderSource (GLuint shader, GLsizei count, const GLchar** string, const GLint* length);
                                     ^~~~~~~~~~~~~~
In file included from ../../3rdparty/chromium/gpu/command_buffer/client/client_context_state.h:10,
                 from ../../3rdparty/chromium/gpu/command_buffer/client/gles2_implementation.h:27,
                 from /home/build/armhf/qt-everywhere-src-5.15.0/qtwebengine/src/core/compositor/display_gl_output_surface.cpp:47:
../../3rdparty/chromium/third_party/khronos/GLES3/gl3.h:624:29: note: previous declaration  void glShaderSource(GLuint, GLsizei, const GLchar* const*, const GLint*) 
 GL_APICALL void GL_APIENTRY glShaderSource (GLuint shader, GLsizei count, const GLchar *const*string, const GLint *length);
                             ^~~~~~~~~~~~~~
cc1plus: warning: unrecognized command line option  -Wno-deprecated-copy

I'm out of the allocated hour budget, and I'll stop here for now. Building Qt5 has been providing some of the most nightmarish work time in my entire professional life. If my daily job became being required to deal with this kind of insanity, I would strongly invest in a change of career. Update Andreas Gruber wrote:

Long story short, a fast solution for the issue with EGLSetBlobFuncANDROID is to remove libraspberrypi-dev from your sysroot and do a full rebuild. There will be some changes to the configure results, so please review them - if they are relevant for you - before proceeding with your work.

And thanks to Andreas, the story can continue...

This is part of a series of posts on compiling a custom version of Qt5 in order to develop for both amd64 and a Raspberry Pi. As an attempt to get webview to compile, I'm reattempting to build a Qt5 cross-compiling environment using a raspbian sysroot, instead of having dependencies for both arm and amd64 installed in the build system. Using dependencies installed in a straightforward way in the build system has failed because of issues like #963136, where some of the build dependencies are needed for both architectures, but the corresponding Debian -dev packages are not yet coinstallable. This is something that causes many people much pain.

Start from a clean sysroot Looking for a Raspbian image, I found out that it has been renamed to "Raspberry Pi OS". I realised that software names are like underwear: as soon as they become well used, they need to be changed. I downloaded ~~Raspbian~~Raspberry Pi OS Lite from https://www.raspberrypi.org/downloads/raspberry-pi-os/ to start with something minimal. It came out as something like 1.5G uncompressed, which wasn't as minimal as I would have hoped, but that'll be what I'll have to work with. Adding build dependencies I have acquired significant experience manipulating ~~Raspbian~~Raspberry Pi OS images from working with Himblick. This time I'm working with the disk image directly, instead of an SD card, since I will be needing it as a sysroot during the build, and I won't need to actually boot it on real hardware. The trick is to work with kpartx to make the partitions in the image available as loopback block devices. I have extracted a lot of relevant code from Himblick into a Python library I called Transilience The result is this provisioning script, which is able to take a ~~Raspbian~~Raspberry Pi OS image, enlarge it, and install Debian packages into it. I find this script pretty cool, also in the way it embeds quite a bit of experience gathered on the field. I can also be integrated in a fully automated setup and provisioning system. The next step will be to use the result as a sysroot to build Qt.

Here are some things that you need to do to get Windows 10 running on a Debian host under KVM. UEFI Booting UEFI is big and complex, but most of what it does isn t needed at all. If all you want to do is boot from an image of a disk with a GPT partition table then you just install the package ovmf and add something like the following to your KVM start script:

UEFI"-drive if=pflash,format=raw,readonly,file=/usr/share/OVMF/OVMF_CODE.fd -drive if=pflash,format=raw,readonly,file=/usr/share/OVMF/OVMF_VARS.fd"

Note that some of the documentation on this doesn t have the OVMF_VARS.fd file set to readonly. Allowing writes to that file means that the VM boot process (and maybe later) can change EFI variables that affect later boots and other VMs if they all share the same file. For a basic boot you don t need to change variables so you want it read-only. Also having it read-only is necessary if you want to run KVM as non-root. As an experiment I tried booting without the OVMF_VARS.fd file, it didn t boot and then even after configuring it to use the OVMF_VARS.fd file again Windows gave a boot error about the boot configuration data file that required booting from recovery media. Apparently configuration mistakes with EFI can mess up the Windows installation, so be careful and backup the Windows installation regularly! Linux can boot from EFI but you generally don t want to unless the boot device is larger than 2TB. It s relatively easy to convert a Linux installation on a GPT disk to a virtual image on a DOS partition table disk or on block devices without partition tables and that gives a faster boot. If the same person runs the host hardware and the VMs then the best choice for Linux is to have no partition tables just one filesystem per block device (which makes resizing much easier) and have the kernel passed as a parameter to kvm. So booting a VM from EFI is probably only useful for booting Windows VMs and for Linux boot loader development and testing. As an aside, the Debian Wiki page about Secure Boot on a VM [4] was useful for this. It s unfortunate that it and so much of the documentation about UEFI is about secure boot which isn t so useful if you just want to boot a system without regard to the secure boot features. Emulated IDE Disks Debian kernels (and probably kernels from many other distributions) are compiled with the paravirtualised storage device drivers. Windows by default doesn t support such devices so you need to emulate an IDE/SATA disk so you can boot Windows and install the paravirtualised storage driver. The following configuration snippet has a commented line for paravirtualised IO (which is fast) and an uncommented line for a virtual IDE/SATA disk that will allow an unmodified Windows 10 installation to boot.

#DRIVE="-drive format=raw,file=/home/kvm/windows10,if=virtio"
DRIVE="-drive id=disk,format=raw,file=/home/kvm/windows10,if=none -device ahci,id=ahci -device ide-drive,drive=disk,bus=ahci.0"

Spice Video Spice is an alternative to VNC, Here is the main web site for Spice [1]. Spice has many features that could be really useful for some people, like audio, sharing USB devices from the client, and streaming video support. I don t have a need for those features right now but it s handy to have options. My main reason for choosing Spice over VNC is that the mouse cursor in the ssvnc doesn t follow the actual mouse and can be difficult or impossible to click on items near edges of the screen. The following configuration will make the QEMU code listen with SSL on port 1234 on all IPv4 addresses. Note that this exposes the Spice password to anyone who can run ps on the KVM server, I ve filed Debian bug #965061 requesting the option of a password file to address this. Also note that the qxl virtual video hardware is VGA compatible and can be expected to work with OS images that haven t been modified for virtualisation, but that they work better with special video drivers.

KEYDIR=/etc/letsencrypt/live/kvm.example.com-0001
-spice password=xxxxxxxx,x509-cacert-file=$KEYDIR/chain.pem,x509-key-file=$KEYDIR/privkey.pem,x509-cert-file=$KEYDIR/cert.pem,tls-port=1234,tls-channel=main -vga qxl

To connect to the Spice server I installed the spice-client-gtk package in Debian and ran the following command:

spicy -h kvm.example.com -s 1234 -w xxxxxxxx

Note that this exposes the Spice password to anyone who can run ps on the system used as a client for Spice, I ve filed Debian bug #965060 requesting the option of a password file to address this. This configuration with an unmodified Windows 10 image only supported 800*600 resolution VGA display. Networking To setup bridged networking as non-root you need to do something like the following as root:

chgrp kvm /usr/lib/qemu/qemu-bridge-helper
setcap cap_net_admin+ep /usr/lib/qemu/qemu-bridge-helper
mkdir -p /etc/qemu
echo "allow all" > /etc/qemu/bridge.conf
chgrp kvm /etc/qemu/bridge.conf
chmod 640 /etc/qemu/bridge.conf

Windows 10 supports the emulated Intel E1000 network card. Configuration like the following configures networking on a bridge named br0 with an emulated E1000 card. MAC addresses that have a 1 in the second least significant bit of the first octet are locally administered (like IPv4 addresses starting with 10. ), see the Wikipedia page about MAC Address for details. The following is an example of network configuration where $ID is an ID number for the virtual machine. So far I haven t come close to 256 VMs on one network so I ve only needed one octet.

NET="-device e1000,netdev=net0,mac=02:00:00:00:01:$ID -netdev tap,id=net0,helper=/usr/lib/qemu/qemu-bridge-helper,br=br0"

Final KVM Settings

KEYDIR=/etc/letsencrypt/live/kvm.example.com-0001
SPICE="-spice password=xxxxxxxx,x509-cacert-file=$KEYDIR/chain.pem,x509-key-file=$KEYDIR/privkey.pem,x509-cert-file=$KEYDIR/cert.pem,tls-port=1234,tls-channel=main -vga qxl"
UEFI="-drive if=pflash,format=raw,readonly,file=/usr/share/OVMF/OVMF_CODE.fd -drive if=pflash,format=raw,readonly,file=/usr/share/OVMF/OVMF_VARS.fd"
DRIVE="-drive format=raw,file=/home/kvm/windows10,if=virtio"
NET="-device e1000,netdev=net0,mac=02:00:00:00:01:$ID -netdev tap,id=net0,helper=/usr/lib/qemu/qemu-bridge-helper,br=br0"
kvm -m 4000 -smp 2 $SPICE $UEFI $DRIVE $NET

Windows Settings The Spice Download page has a link for spice-guest-tools that has the QNX video driver among other things [2]. This seems to be needed for resolutions greater than 800*600. The Virt-Manager Download page has a link for virt-viewer which is the Spice client for Windows systems [3], they have MSI files for both i386 and AMD64 Windows. It s probably a good idea to set display and system to sleep after never (I haven t tested what happens if you don t do that, but there s no benefit in sleeping). Before uploading an image I disabled the pagefile and set the partition to the minimum size so I had less data to upload. Problems Here are some things I haven t solved yet. The aSpice Android client for the Spice protocol fails to connect with the QEMU code at the server giving the following message on stderr: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca:../ssl/record/rec_layer_s3.c:1544:SSL alert number 48 . Spice is supposed to support dynamic changes to screen resolution on the VM to match the window size at the client, this doesn t work for me, not even with the Red Hat QNX drivers installed. The Windows Spice client doesn t seem to support TLS, I guess running some sort of proxy for TLS would work but I haven t tried that yet.

tl;dr: Use MessagePack, rather than CBOR. Introduction I recently wanted to choose a binary encoding. This was for a project using Rust serde, so I looked at the list of formats there. I ended up reading about CBOR and MessagePack. Both of these are binary formats for a JSON-like data model. Both of them are "schemaless", meaning you can decode them without knowing the structure. (This also provides some forwards compatibility.) They are, in fact, quite similar (although they are totally incompatible). This is no accident: CBOR is, effectively, a fork of MessagePack. Both formats continue to exist and both are being used in new programs. I needed to make a choice but lacked enough information. I thought I would try to examine the reasons and nature of the split, and to make some kind of judgement about the situation. So I did a lot of reading [11]. Here are my conclusions. History and politics Between about 2010 and 2013 there was only MessagePack. Unfortunately, MessagePack had some problems. The biggest of these was that it lacked a separate string type. Strings were to be encoded simply as byte blocks. This caused serious problems for many MessagePack library implementors: for example, when decoding a MessagePack file the Python library wouldn't know whether to produce a Python bytes object, or a string. Straightforward data structures wouldn't round trip through MessagePack. [1] [2] It seems that in late 2012 this came to the attention to someone with an IETF background. According to them, after unsatisfactory conversations with MessagePack upstream, they decided they would have to fork. They submitted an Internet Draft for a partially-incompatible protocol [3] [4]. Little seemed to happen in the IETF until soon before the Orlando in-person IETF meeting in February 2013.[5] These conversations sparked some discussion in the MessagePack issue tracker. There were long threads including about process [1,2,4 ibid]. But there was also a useful technical discussion, about proposed backward compatible improves to the MessagePack spec.[5] The prominent IETF contributor provided some helpful input in these discussions in the MessagePack community - but also pushed quite hard for a "tagging" system, which suggestion was not accepted (see my technical analysis, below). An improved MessagePack spec resulted, with string support, developed largely by the MessagePack community. It seems to have been available in useable form since mid-2013 and was officially published as canonical in August 2013. Meanwhile a parallel process was pursued in the IETF, based on the IETF contributor's fork, with 11 Internet-Drafts from February[7] to September[8]. This seems to have continued even though the original technical reason for the fork - lack of string vs binary distinction - no longer applied. The IETF proponent expressed unhappiness about MessagePack's stewardship and process as much as they did about the technical details [4, ibid]. The IETF process culminated in the CBOR RFC[9]. The discussion on process questions between the IETF proponent and MessagePack upstream, in the MessagePack issue tracker [4, ibid] should make uncomfortable reading for IETF members. The IETF acceptance of CBOR despite clear and fundamental objections from MessagePack upstream[13] and indeed other respected IETF members[14], does not reflect well on the IETF. The much vaunted openness of the IETF process seems to have been rather one-sided. The IETF proponent here was an IETF Chair. Certainly the CBOR author was very well-spoken and constantly talks about politeness and cooperation and process; but what they actually did was very hostile. They accused the MessagePack community of an "us and them" attitude while simultaneously pursuing a forked specification! The CBOR RFC does mention MessagePack in Appendix E.2. But not to acknowledge that CBOR was inspired by MessagePack. Rather, it does so to make a set of tendentious criticisms of MessagePack. Perhaps these criticisms were true when they were first written in an I-D but they were certainly false by the time the RFC was actually published, which occurred after the MessagePack improvement process was completely concluded, with a formal spec issued. Since then both formats have existed in parallel. Occasionally people discuss which one is better, and sometimes it is alleged that "yes CBOR is the successor to MessagePack", which is not really fair.[9][10] Technical differences The two formats have a similar arrangement: initial byte which can encode small integers, or type and length, or type and specify a longer length encoding. But there are important differences. Overall, MessagePack is very significantly simpler. Floating point CBOR supports five floating point formats! Not only three sizes of IEEE754, but also decimal floating point, and bigfloats. This seems astonishing for a supposedly-simple format. (Some of these are supported via the semi-optional tag mechanism - see below.) Indefinite strings and arrays Like MessagePack, CBOR mostly precedes items with their length. But CBOR also supports "indefinite" strings, arrays, and so on, where the length is not specified at the beginning. The object (array, string, whatever) is terminated by a special "break" item. This seems to me to be a mistake. If you wanted the kind of application where MessagePack or CBOR would be useful, streaming sub-objects of unknown length is not that important. This possibility considerably complicates decoders. CBOR tagging system CBOR has a second layer of sort-of-type which can be attached to each data item. The set of possible tags is open-ended and extensible, but the CBOR spec itself gives tag values for: two kinds of date format; positive and negative bignums; decimal floats (see above); binary but expected to be encoded if converted to JSON (in base64url, base64, or base16); nestedly encoded CBOR; URIs; base64 data (two formats); regexps; MIME messages; and a special tag to make file(1) work. In practice it is not clear how many of these are used, but a decoder must be prepared to at least discard them. The amount of additional spec complexity here is quite astonishing. IMO binary formats like this will (just like JSON) be used by a next layer which always has an idea of what the data means, including (where the data is a binary blob) what encoding it is in etc. So these tags are not useful. These tags might look like a middle way between (i) extending the binary protocol with a whole new type such as an extension type (incompatible with old readers) and encoding your new kind data in a existing type (leaving all readers who don't know the schema to print it as just integers or bytes or string). But I think they are more trouble than they are worth. The tags are uncomfortably similar to the ASN.1 tag system, which is widely regarded as one of ASN.1's unfortunate complexities. MessagePack extension mechanism MessagePack explicitly reserves some encoding space for users and for future extensions: there is an "extension type". The payload is an extension type byte plus some more data bytes; the data bytes are in a format to be defined by the extension type byte. Half of the possible extension byte values are reserved for future specification, and half are designated for application use. This is pleasingly straightforward. (There is also one unused primary initial byte value, but that would be rejected by existing decoders and doesn't seem like a likely direction for future expansion.) Minor other differences in integer encoding The encodings of integers differ. In MessagePack, signed and unsigned integers have different typecodes. In CBOR, signed and unsigned positive integers have the same typecodes; negative integers have a different set of typecodes. This means that a CBOR reader which knows it is expecting a signed value will have to do a top-bit-set check on the actual data value! And a CBOR writer must check the value to choose a typecode. MessagePack reserves fewer shortcodes for small negative integers, than for small positive integers. Conclusions and lessons MessagePack seems to have been prompted into fixing the missing string type problem, but only by the threat of a fork. However, this fork went ahead even after MessagePack clearly accepted the need for a string type. MessagePack had a fixed protocol spec before the IETF did. The continued pursuit of the IETF fork was ostensibly been motivated by a disapproval of the development process and in particular a sense that the IETF process was superior. However, it seems to me that the IETF process was abused by CBOR's proponent, who just wanted things their own way. I have seen claims by IETF proponents that the open decisionmaking system inherently produces superior results. However, in this case the IETF process produced a bad specification. To the extent that other IETF contributors had influence over the ultimate CBOR RFC, I don't think they significantly improved it. CBOR has been described as MessagePack bikeshedded by the IETF. That would have been bad enough, but I think it's worse than that. To a large extent CBOR is one person's NIH-induced bad design rubber stamped by the IETF. CBOR's problems are not simply matters of taste: it's significantly overcomplicated. One lesson for the rest of us is that although being the upstream and nominally in charge of a project seems to give us a lot of power, it's wise to listen carefully to one's users and downstreams. Once people are annoyed enough to fork, the fork will have a life of its own. Another lesson is that many of us should be much warier of the supposed moral authority of the IETF. Many IETF standards are awful (Oauth 2 [12]; IKE; DNSSEC; the list goes on). Sometimes (especially when network adoption effects are weak, as with MessagePack vs CBOR) better results can be obtained from a smaller group, or even an individual, who simply need the thing for their own uses. Finally, governance systems of public institutions like the IETF need to be robust in defending the interests of outsiders (and hence of society at large) against eloquent insiders who know how to work the process machinery. Any institution which nominally serves the public good faces a constant risk of devolving into self-servingness. This risk gets worse the more powerful and respected the institution becomes. References

~~#13~~: First-class string type in serialization specification (MessagePack issue tracker, June 2010 - August 2013)
~~#121~~: Msgpack can't differentiate between raw binary data and text strings (MessagePack issue tracker, November 2012 - February 2013)
draft-bormann-apparea-bpack-00: The binarypack JSON-like representation format (IETF Internet-Draft, October 2012)
~~#129~~: MessagePack should be developed in an open process (MessagePack issue tracker, February 2013 - March 2013)
Re: JSON mailing list and BoF (IETF apps-discuss mailing list message from Carsten Bormann, 18 February 2013)
~~#128~~: Discussions on the upcoming MessagePack spec that adds the string type to the protocol (MessagePack issue tracker, February 2013 - August 2013)
draft-bormann-apparea-bpack-01: The binarypack JSON-like representation format (IETF Internet-Draft, February 2013)
draft-bormann-cbor: Concise Binary Object Representation (CBOR) (IETF Internet-Drafts, May 2013 - September 2013)
RFC 7049: Concise Binary Object Representation (CBOR) (October 2013)
"MessagePack should be replaced with [CBOR] everywhere ..." (floatboth on Hacker News, 8th April 2017)
Discussion with very useful set of history links (camgunz on Hacker News, 9th April 2017)
OAuth 2.0 and the Road to Hell (Eran Hammer, blog posting from 2012, via Wayback Machine)
Re: [apps-discuss] [Json] msgpack/binarypack (Re: JSON mailing list and BoF) (IETF list message from Sadyuki Furuhashi, 4th March 2013)
"no apologies for complaining about this farce" (IETF list message from Phillip Hallam-Baker, 15th August 2013)
Edited 2020-07-14 18:55 to fix a minor formatting issue, and 2020-07-14 22:54 to fix two typos

comments

Ansible Molecule is a toolkit for testing Ansible roles. It allows for easy execution and verification of your roles and also manages the environment (container, VM, etc) in which those are executed. In the Foreman project we have a collection of Ansible roles to setup Foreman instances called forklift. The roles vary from configuring Libvirt and Vagrant for our CI to deploying full fledged Foreman and Katello setups with Proxies and everything. The repository also contains a dynamic Vagrant file that can generate Foreman and Katello installations on all supported Debian, Ubuntu and CentOS platforms using the previously mentioned roles. This feature is super helpful when you need to debug something specific to an OS/version combination. Up until recently, all those roles didn't have any tests. We would run ansible-lint on them, but that was it. As I am planning to do some heavier work on some of the roles to enhance our upgrade testing, I decided to add some tests first. Using Molecule, of course. Adding Molecule to an existing role is easy: molecule init scenario -r my-role-name will add all the necessary files/examples for you. It's left as an exercise to the reader how to actually test the role properly as this is not what this post is about. Executing the tests with Molecule is also easy: molecule test. And there are also examples how to integrate the test execution with the common CI systems. But what happens if you have more than one role in the repository? Molecule has support for monorepos, however that is rather limited: it will detect the role path correctly, so roles can depend on other roles from the same repository, but it won't find and execute tests for roles if you run it from the repository root. There is an undocumented way to set MOLECULE_GLOB so that Molecule would detect test scenarios in different paths, but I couldn't get it to work nicely for executing tests of multiple roles and upstream currently does not plan to implement this. Well, bash to the rescue!

for roledir in roles/*/molecule; do
    pushd $(dirname $roledir)
    molecule test
    popd
done

Add that to your CI and be happy! The CI will execute all available tests and you can still execute those for the role you're hacking on by just calling molecule test as you're used to. However, we can do even better. When you initialize a role with Molecule or add Molecule to an existing role, there are quite a lot of files added in the molecule directory plus an yamllint configuration in the role root. If you have many roles, you will notice that especially the molecule.yml and .yamllint files look very similar for each role. It would be much nicer if we could keep those in a shared place. Molecule supports a "base config": a configuration file that gets merged with the molecule.yml of your project. By default, that's ~/.config/molecule/config.yml, but Molecule will actually look for a .config/molecule/config.yml in two places: the root of the VCS repository and your HOME. And guess what? The one in the repository wins (that's not yet well documented). So by adding a .config/molecule/config.yml to the repository, we can place all shared configuration there and don't have to duplicate it in every role. And that .yamllint file? We can also move that to the repository root and add the following to Molecule's (now shared) configuration:

lint: yamllint --config-file $ MOLECULE_PROJECT_DIRECTORY /../../.yamllint --format parsable .

This will define the lint action as calling yamllint with the configuration stored in the repository root instead of the project directory, assuming you store your roles as roles/<rolename>/ in the repository. And that's it. We now have a central place for our Molecule and yamllint configurations and only need to place role-specific data into the role directory.

The Linux community is a fascinating and powerful space.When I joined the Ubuntu project approximately five years ago, I (vaguely at the time) understood that there was a profound sense of community and passion everywhere that is difficult to find in other spaces. My involvement has increased, and so has my understanding. I had thought of starting a blog as a means of conveying the information that I stumbled across, but my writing skills were very crude and regrettable, being in my early teenage years.I have finally decided to take the leap. In this blog, I would like to occasionally provide updates on my work, either through focused deep dives on a particular topic, or broad updates on low hanging fruit that has been eliminated. While the articles may be somewhat spontaneous, I decided that an initial post was in order to explain my goals. Feel free to subscribe for more detailed posts in the future, as there are many more to come.

In my last post I wrote:

The sleep 0.3s is needed because xdg-open exits right after starting the program, and when invoked by mutt it means that mutt could delete the attachment before evince has a chance to open it. I had to use the same workaround for sensible-browser, since the same happens when a browser opens a document in an existing tab. I feel like writing some wrapper about all this that forks the viewer, then waits for an IN_OPEN event on its argument via inotify before exiting.

I wrote it: https://github.com/spanezz/waitused/

$ ./waitused --help
usage: waitused [-h] path ...
Run a command exiting only after it quits and a given file has been opened and
closed
positional arguments:
  path        file to monitor
  command     command to run
optional arguments:
  -h, --help  show this help message and exit

This works around situations like mutt deleting the temporary attachment file after run-mailcap is run, while run-mailcap runs a program that backgrounds before opening its input file. Example

waitused file.pdf xdg-open file.pdf
waitused file.pdf run-mailcap file.pdf

Example ~/.mailcap entry

application/pdf; waitused -- %s xdg-open %s; test=test -n "$DISPLAY"

Update: Teddy Hogeborn pointed out that the initial mailcap entry would fail on files starting with a dash. I added -- for waitused, but unfortunately there seems to be no way at the moment to have xdg-open open files starting with a dash (see: #964949

The last step of my laptop migration was to fix mime type associations, that seem to associate opening file depending on whatever application was installed last, phases of the moon, and what option is the most annoying. The state of my system after a fresh install, is that, for application/pdf, xdg-open (used for example by pcmanfm) runs inkscape, and run-mailcap (used for example by neomutt) runs the calibre ebook viewer. It looks like there are at least two systems to understand, debug and fix, instead of one. xdg-open This comes from package xdg-utils, and works using .desktop files:

# This runs inkscape
$ xdg-open file.pdf

There is a tool called xdg-mime that queries what .desktop file is associated with a given mime type:

$ xdg-mime query default application/pdf
inkscape.desktop

You can use xdg-mime default to change an association, and it works nicely:

$ xdg-mime default org.gnome.Evince.desktop application/pdf
$ xdg-mime query default application/pdf
org.gnome.Evince.desktop

However, if you accidentally mistype the name of the .desktop file, it won't complain and it will silently reset the association to the arbitrary default:

$ xdg-mime default org.gnome.Evince.desktop application/pdf
$ xdg-mime query default application/pdf
org.gnome.Evince.desktop
$ xdg-mime default evince.desktop application/pdf
$ echo $?
0
$ xdg-mime query default application/pdf
inkscape.desktop

You can use a GUI like xfce4-mime-settings from the xfce4-settings package to perform the same kind of changes avoiding typing mistakes. The associations seem to be saved in ~/.config/mimeapps.list run-mailcap This comes from the package mime-support You can test things by running it using --norun:

$ run-mailcap --norun file.pdf
ebook-viewer file.pdf

run-mailcap uses the ~/.mailcap and /etc/mailcap to map mime types to commands. This is what's in the system default:

$ grep application/pdf /etc/mailcap
application/pdf; ebook-viewer %s; test=test -n "$DISPLAY"
application/pdf; calibre %s; test=test -n "$DISPLAY"
application/pdf; gimp-2.10 %s; test=test -n "$DISPLAY"
application/pdf; evince %s; test=test -n "$DISPLAY"

To fix this, I copypasted the evince line into ~/.mailcap, and indeed it gets used:

$ run-mailcap --norun file.pdf
evince file.pdf

There is a /etc/mailcap.order file providing a limited way to order entries in /etc/mailcap, but it can only be manipulated system-wide, and cannot be used for user preferences. Sadly, this means that if a package changes its mailcap invocation because of, say, a security issue in the former one, the local override will never get fixed. I am really not comfortable about that. As a workaround, I put this in my ~/.mailcap:

application/pdf; xdg-open %s && sleep 0.3s; test=test -n "$DISPLAY"

The sleep 0.3s is needed because xdg-open exits right after starting the program, and when invoked by mutt it means that mutt could delete the attachment before evince has a chance to open it. I had to use the same workaround for sensible-browser, since the same happens when a browser opens a document in an existing tab. I feel like writing some wrapper about all this that forks the viewer, then waits for an IN_OPEN event on its argument via inotify before exiting. I wonder if there is any reason run-mailcap could not be implemented as a wrapper to xdg-open. I reported #964723 elaborating on these thoughts.

My laptop battery started to explode in slow motion. HP requires 10 business days to repair my laptop under warranty, and I cannot afford that length of downtime. Alternatively, HP quoted me 375 + VAT for on-site repairs, which I tought was very funny. For 376.55 + VAT, which is pretty much exactly the same amount, I bought instead a refurbished ThinkPad X240 with a dual-core I5, 8G of RAM, 250G SSD, and a 1920x1080 IPS display, to use as a spare while my laptop is being repaired. I'd like to thank HP for giving me the opportunity to own a ThinkPad. Since I'm migrating all my system to the spare and then (hopefully) back, I'm documenting what I need to be fully productive on new hardware. Install Debian A basic Debian netinst with no tasks selected is good enough to get going. Note that if wifi worked in Debian Installer, it doesn't mean that it will work in the minimal system it installed. See here for instructions on quickly bringing up wifi on a newly installed minimal system. Copy /home A simple tar of /home is all I needed to copy my data over. A neat way to do it was connecting the two laptops with an ethernet cable, and using netcat:

# On the source
tar -C / -zcf - home   nc -l -p 12345 -N
# On the target
nc 10.0.0.1 12345   tar -C / -zxf -

Since the data travel unencrypted in this way, don't do it over wifi. Install packages I maintain a few simple local metapackages that depend on the packages I usually used. I could just install those and let apt bring in their dependencies. For the build dependencies of the programs I develop, I use mk-build-deps from the devscripts package to create metapackages that make sure they are installed. Here's an extract from debian/control of the metapackage:

Source: enrico
Section: admin
Priority: optional
Maintainer: Enrico Zini <enrico@debian.org>
Build-Depends: debhelper (>= 11)
Standards-Version: 3.7.2.1
Package: enrico
Section: admin
Architecture: all
Depends:
  mc, mmv, moreutils, powertop, syncmaildir, notmuch,
  ncdu, vcsh, ddate, jq, git-annex, eatmydata,
  vdirsyncer, khal, etckeeper, moc, pwgen
Description: Enrico's working environment
Package: enrico-devel
Section: devel
Architecture: all
Depends:
  git, python3-git, git-svn, gitk, ansible, fabric,
  valgrind, kcachegrind, zeal, meld, d-feet, flake8, mypy, ipython3,
  strace, ltrace
Description: Enrico's development environment
Package: enrico-gui
Section: x11
Architecture: all
Depends:
  xclip, gnome-terminal, qalculate-gtk, liferea, gajim,
  mumble, sm, syncthing, virt-manager
Recommends: k3b
Description: Enrico's GUI environment
Package: enrico-sanity
Section: admin
Architecture: all
Conflicts: libapache2-mod-php, libapache2-mod-php5, php5, php5-cgi, php5-fpm, libapache2-mod-php7.0, php7.0, libphp7.0-embed, libphp-embed, libphp5-embed
Description: Enrico's sanity
 Metapackage with a list of packages that I do not want anywhere near my
 system.

System-wide customizations I tend to avoid changing system-wide configuration as much as possible, so copying over /home and installing packages takes care of 99% of my needs. There are a few system-wide tweaks I cannot do without:

setup postfix to send mail using my mail server
copy Network Manager system connections files in /etc/NetworkManager/system-connections/
update-alternatives --config editor

For postfix, I have a little ansible playbook that takes care of it. Network Manager system connections need to be copied manually: a plain copy and a systemctl restart network-manager are enough. Note that Network Manager will ignore the files unless their owner and permissions are what it expects. Fine tuning Comparing the output of dpkg --get-selections between the old and the new system might highlight packages manually installed in a hurry and not added to the metapackages. Finally, what remains is fixing the sad state of mimetype associations, which seem to associate opening file depending on whatever application was installed last, phases of the moon, and what option is the most annoying. Currently on my system, PDFs are opened in inkscape by xdg-open and in calibre by run-mailcap. Let's see how long it takes to figure this one out.

Right on the heels of the news of 2000 CRAN packages using Rcpp (and also hitting 12.5 of CRAN package, or one in eight), we are happy to announce release 1.0.5 of Rcpp. Since the ten-year anniversary and the 1.0.0 release release in November 2018, we have been sticking to a four-month release cycle. The last release has, however, left us with a particularly bad taste due to some rather peculiar interactions with a very small (but ever so vocal) portion of the user base. So going forward, we will change two things. First off, we reiterate that we have already made rolling releases. Each minor snapshot of the main git branch gets a point releases. Between release 1.0.4 and this 1.0.5 release, there were in fact twelve of those. Each and every one of these was made available via the drat repo, and we will continue to do so going forward. Releases to CRAN, however, are real work. If they then end up with as much nonsense as the last release 1.0.4, we think it is appropriate to slow things down some more so we intend to now switch to a six-months cycle. As mentioned, interim releases are always just one install.packages() call with a properly set repos argument away. Rcpp has become the most popular way of enhancing R with C or C++ code. As of today, 2002 packages on CRAN depend on Rcpp for making analytical code go faster and further, along with 203 in BioConductor. And per the (partial) logs of CRAN downloads, we are running steady at around one millions downloads per month. This release features again a number of different pull requests by different contributors covering the full range of API improvements, attributes enhancements, changes to Sugar and helper functions, extended documentation as well as continuous integration deplayment. See the list below for details.

Changes in Rcpp patch release version 1.0.5 (2020-07-01)

Changes in Rcpp API:

The exception handler code in #1043 was updated to ensure proper include behavior (Kevin in #1047 fixing #1046).

A missing Rcpp_list6 definition was added to support R 3.3.* builds (Davis Vaughan in #1049 fixing #1048).

Missing Rcpp_list 2,3,4,5 definition were added to the Rcpp namespace (Dirk in #1054 fixing #1053).

A further updated corrected the header include and provided a missing else branch (Mattias Ellert in #1055).

Two more assignments are protected with Rcpp::Shield (Dirk in #1059).

One call to abs is now properly namespaced with std:: (Uwe Korn in #1069).

String object memory preservation was corrected/simplified (Kevin in #1082).

Changes in Rcpp Attributes:

Empty strings are not passed to R CMD SHLIB which was seen with R 4.0.0 on Windows (Kevin in #1062 fixing #1061).

The short_file_name() helper function is safer with respect to temporaries (Kevin in #1067 fixing #1066, and #1071 fixing #1070).

Changes in Rcpp Sugar:

Two sample() objects are now standard vectors and not R_alloc created (Dirk in #1075 fixing #1074).

Changes in Rcpp support functions:

Rcpp.package.skeleton() adjusts for a (documented) change in R 4.0.0 (Dirk in #1088 fixing #1087).

Changes in Rcpp Documentation:

The pdf file of the earlier introduction is again typeset with bibliographic information (Dirk).

A new vignette describing how to package C++ libraries has been added (Dirk in #1078 fixing #1077).

Changes in Rcpp Deployment:

Travis CI unit tests now run a matrix over the versions of R also tested at CRAN (rel/dev/oldrel/oldoldrel), and coverage runs in parallel for a net speed-up (Dirk in #1056 and #1057).

The exceptions test is now partially skipped on Solaris as it already is on Windows (Dirk in #1065).

The default CI runner was upgraded to R 4.0.0 (Dirk).

The CI matrix spans R 3.5, 3.6, r-release and r-devel (Dirk).

Thanks to CRANberries, you can also look at a diff to the previous release. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page. Bugs reports are welcome at the GitHub issue tracker as well (where one can also search among open or closed issues); questions are also welcome under rcpp tag at StackOverflow which also allows searching among the (currently) 2455 previous questions. If you like this or other open-source work I do, you can now sponsor me at GitHub. For the first year, GitHub will match your contributions.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Welcome to the June 2020 report from the Reproducible Builds project. In these reports we outline the most important things that we and the rest of the community have been up to over the past month.

What are reproducible builds? One of the original promises of open source software is that distributed peer review and transparency of process results in enhanced end-user security. But whilst anyone may inspect the source code of free and open source software for malicious flaws, almost all software today is distributed as pre-compiled binaries. This allows nefarious third-parties to compromise systems by injecting malicious code into seemingly secure software during the various compilation and distribution processes.

News The GitHub Security Lab published a long article on the discovery of a piece of malware designed to backdoor open source projects that used the build process and its resulting artifacts to spread itself. In the course of their analysis and investigation, the GitHub team uncovered 26 open source projects that were backdoored by this malware and were actively serving malicious code. (Full article) Carl Dong from Chaincode Labs uploaded a presentation on Bitcoin Build System Security and reproducible builds to YouTube: The app intended to trace infection chains of Covid-19 in Switzerland published information on how to perform a reproducible build. The Reproducible Builds project has received funding in the past from the Open Technology Fund (OTF) to reach specific technical goals, as well as to enable the project to meet in-person at our summits. The OTF has actually also assisted countless other organisations that promote transparent, civil society as well as those that provide tools to circumvent censorship and repressive surveillance. However, the OTF has now been threatened with closure. (More info) It was noticed that Reproducible Builds was mentioned in the book End-user Computer Security by Mark Fernandes (published by WikiBooks) in the section titled Detection of malware in software. Lastly, reproducible builds and other ideas around software supply chain were mentioned in a recent episode of the Ubuntu Podcast in a wider discussion about the Snap and application stores (at approx 16:00).

Distribution work In the ArchLinux distribution, a goal to remove `.doctrees` from installed files was created via Arch s TODO list mechanism. These `.doctree` files are caches generated by the Sphinx documentation generator when developing documentation so that Sphinx does not have to reparse all input files across runs. They should not be packaged, especially as they lead to the package being unreproducible as their pickled format contains unreproducible data. Jelle van der Waa and Eli Schwartz submitted various upstream patches to fix projects that install these by default. Dimitry Andric was able to determine why the reproducibility status of FreeBSD s `base.txz` depended on the number of CPU cores, attributing it to an optimisation made to the Clang C compiler [ ]. After further detailed discussion on the FreeBSD bug it was possible to get the binaries reproducible again [ ]. For the GNU Guix operating system, Vagrant Cascadian started a thread about collecting reproducibility metrics and Jan janneke Nieuwenhuizen posted that they had further reduced their bootstrap seed to 25% which is intended to reduce the amount of code to be audited to avoid potential compiler backdoors. In openSUSE, Bernhard M. Wiedemann published his monthly Reproducible Builds status update as well as made the following changes within the distribution itself:

`autogen` (Date issue)

`carla` (Timestamp in Windows Portable Executable executables)

`fonttosfnt/xorg-x11-fonts` (Address space layout randomization issue)

`fossil` (Date issue)

`gcc10 C++` (Link-time optimisation issue)

`grep` (Profile-guided optimisation issue)

`kubernetes1.18` (Remove Go build identifier)

`libjcat` (Remove certificate)

`lifelines` (Date issue)

`miredo` (Drop hostname)

`stressapptest` (Override date, user & host)

Debian Holger Levsen filed three bugs (#961857, #961858 & #961859) against the `reproducible-check` tool that reports on the reproducible status of installed packages on a running Debian system. They were subsequently all fixed by Chris Lamb [ ][ ][ ]. Timo R hling filed a wishlist bug against the `debhelper` build tool impacting the reproducibility status of 100s of packages that use the CMake build system which led to a number of tests and next steps. [ ] Chris Lamb contributed to a conversation regarding the nondeterministic execution of order of Debian maintainer scripts that results in the arbitrary allocation of UNIX group IDs, referencing the Tails operating system s approach this [ ]. Vagrant Cascadian also added to a discussion regarding verification formats for reproducible builds. 47 reviews of Debian packages were added, 37 were updated and 69 were removed this month adding to our knowledge about identified issues. Chris Lamb identified and classified a new `uids_gids_in_tarballs_generated_by_cmake_kde_package_app_templates` issue [ ] and updated the `paths_vary_due_to_usrmerge as deterministic` issue, and Vagrant Cascadian updated the `cmake_rpath_contains_build_path` and `gcc_captures_build_path` issues. [ ][ ][ ]. Lastly, Debian Developer Bill Allombert started a mailing list thread regarding setting the `-fdebug-prefix-map` command-line argument via an environment variable and Holger Levsen also filed three bugs against the `debrebuild` Debian package rebuilder tool (#961861, #961862 & #961864).

Development On our website this month, Arnout Engelen added a link to our Mastodon account [ ] and moved the `SOURCE_DATE_EPOCH` `git log` example to another section [ ]. Chris Lamb also limited the number of news posts to avoid showing items from (for example) 2017 [ ]. strip-nondeterminism is our tool to remove specific non-deterministic results from a completed build. It is used automatically in most Debian package builds. This month, Mattia Rizzolo bumped the `debhelper` compatibility level to 13 [ ] and adjusted a related dependency to avoid potential circular dependency [ ].

Upstream work The Reproducible Builds project attempts to fix unreproducible packages and we try to to send all of our patches upstream. This month, we wrote a large number of such patches including:

Andreas Schleifer:

`conky`

`kismet`

`nitrokey-app`

Bernhard M. Wiedemann:

`autogen` (race condition)

`cockpit` (date)

`fossil` (date)

`libnvidia-container` (date)

`libv3270` ( date)

Chris Lamb:

#962401 filed against `netcdf-fortran`.

#962589 filed against `seqtools`.

#962702 filed against `python-pauvre`.

#963119 filed against `petitboot`.

#963120 filed against `fonts-anonymous-pro`.

#963124 filed against `python-pyqtgraph` (forwarded upstream)

#963485 filed against `libqmi`.

#963486 filed against `tkabber-plugins`.

#963533 filed against `python-stem`.

#963537 filed against `golang-v2ray-core`.

#963600 filed against `critcl`.

#963602 filed against `gftl`.

#963603 filed against `libmbim`.

#963688 filed against `neovim-qt`.

#963740 filed against `golang-github-viant-toolbox`.

Hendrik Meyer:

`gitality`

`lsof`

Nick Wellnhofer:

`libxml2` (random data corruption)

Jelle van der Waa:

`gnome-builder`

`PHP`

Eli Schwartz:

`ghc`

`vigra`

Vagrant Cascadian:

#961954 filed against `lirc`.

#962021 filed against `graphviz`.

#962305 filed against `libtommath`.

#963466, #963467, #963470 and #963472 filed against `qemu`.

#963518 filed against `source-highlight`.

Bernhard M. Wiedemann also filed reports for `frr` (build fails on single-processor machines), `ghc-yesod-static/git-annex` (a filesystem ordering issue) and `ooRexx` (ASLR-related issue).

diffoscope diffoscope is our in-depth diff-on-steroids utility which helps us diagnose reproducibility issues in packages. It does not define reproducibility, but rather provides a helpful and human-readable guidance for packages that are not reproducible, rather than relying essentially-useless binary diffs. This month, Chris Lamb uploaded versions `147`, `148` and `149` to Debian and made the following changes:

New features:

Add output from `strings(1)` to ELF binaries. (#148)

Dump `PE32+` executables (such as EFI applications) using `objdump(1)`. (#181)

Add support for Zsh shell completion. (#158)

Bug fixes:

Prevent a traceback when comparing PDF documents that did not contain metadata (ie. a PDF `/Info` stanza). (#150)

Fix compatibility with `jsondiff` version 1.2.0. (#159)

Fix an issue in GnuPG keybox file handling that left filenames in the diff. [ ]

Correct detection of JSON files due to missing call to `File.recognizes` that checks candidates against `file(1)`. [ ]

Output improvements:

Use the CSS `word-break` property over manually adding `U+200B` zero-width spaces as these were making copy-pasting cumbersome. (!53)

Downgrade the `tlsh` warning message to an info level warning. (#29)

Logging improvements:

Log calls to `subprocess.check_output` by using a wrapper. (#151)

Clarify that we are generating presenter formats in a debug-level message. [ ]

Log the version of jsondiff used. [ ]

Testsuite improvements:

Update tests for `file(1)` version 5.39. (#179)

Drop accidentally-duplicated copy of the `--diff-mask` tests. [ ]

Don t mask an existing test. [ ]

Codebase improvements:

Replace obscure references to WF with Wagner-Fischer for clarity. [ ]

Use a semantic `AbstractMissingType` type instead of remembering to check for both types of missing files. [ ]

Add a comment regarding potential security issue in the `.changes`, `.dsc` and `.buildinfo` comparators. [ ]

Drop a large number of unused imports. [ ][ ][ ][ ][ ]

Make many code sections more Pythonic. [ ][ ][ ][ ]

Prevent some variable aliasing issues. [ ][ ][ ]

Use some tactical `f-strings` to tidy up code [ ][ ] and remove explicit `u"unicode"` strings [ ].

Refactor a large number of routines for clarity. [ ][ ][ ][ ]

trydiffoscope is the web-based version of diffoscope. This month, Chris Lamb also corrected the location for the celerybeat scheduler to ensure that the clean/tidy tasks are actually called which had caused an accidental resource exhaustion. (#12) In addition Jean-Romain Garnier made the following changes:

Fix the `--new-file` option when comparing directories by merging `DirectoryContainer.compare` and `Container.compare`. (#180)

Allow user to mask/filter diff output via `--diff-mask=REGEX`. (!51)

Make child pages open in new window in the `--html-dir` presenter format. [ ]

Improve the diffs in the `--html-dir` format. [ ][ ]

Lastly, Daniel Fullmer fixed the Coreboot filesystem comparator [ ] and Mattia Rizzolo prevented warnings from the `tlsh` fuzzy-matching library during tests [ ] and tweaked the build system to remove an unwanted `.build` directory [ ]. For the GNU Guix distribution Vagrant Cascadian updated the version of diffoscope to version 147 [ ] and later 148 [ ].

Testing framework We operate a large and many-featured Jenkins-based testing framework that powers `tests.reproducible-builds.org`. Amongst many other tasks, this tracks the status of our reproducibility efforts across many distributions as well as identifies any regressions that have been introduced. This month, Holger Levsen made the following changes:

Debian-related changes:

Prevent bogus failure emails from `rsync2buildinfos.debian.net` every night. [ ]

Merge a fix from David Bremner s database of `.buildinfo` files to include a fix regarding comparing source vs. binary package versions. [ ]

Only run the Debian package rebuilder job twice per day. [ ]

Increase bullseye scheduling. [ ]

System health status page:

Add a note displaying whether a node needs to be rebooted for a kernel upgrade. [ ]

Fix sorting order of failed jobs. [ ]

Expand footer to link to the related Jenkins job. [ ]

Add `archlinux_html_pages`, `openwrt_rebuilder_today` and `openwrt_rebuilder_future` to known broken jobs. [ ]

Add HTML `<meta>` header to refresh the page every 5 minutes. [ ]

Count the number of ignored jobs [ ], ignore permanently known broken jobs [ ] and jobs on known offline nodes [ ].

Only consider the known offline status from Git. [ ]

Various output improvements. [ ][ ]

Tools:

Switch URLs for the Grml Live Linux and PureOS package sets. [ ][ ]

Don t try to build a disorderfs Debian source package. [ ][ ][ ]

Stop building diffoscope as we are moving this to Salsa. [ ][ ]

Merge several is diffoscope up-to-date on every platform? test jobs into one [ ] and fail less noisily if the version in Debian cannot be determined [ ].

In addition: Marcus Hoffmann was added as a maintainer of the F-Droid reproducible checking components [ ], Jelle van der Waa updated the is diffoscope up-to-date in every platform check for Arch Linux and diffoscope [ ], Mattia Rizzolo backed up a copy of a remove script run on the Codethink-hosted jump server [ ] and Vagrant Cascadian temporarily disabled the `fixfilepath` on bullseye, to get better data about the `ftbfs_due_to_f-file-prefix-map` categorised issue. Lastly, the usual build node maintenance was performed by Holger Levsen [ ][ ], Mattia Rizzolo [ ] and Vagrant Cascadian [ ][ ][ ][ ][ ].

If you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:

IRC: `#reproducible-builds` on `irc.oftc.net`.

Twitter: @ReproBuilds

Mastodon: @reproducible_builds@fosstodon.org

Reddit: /r/ReproducibleBuilds

Mailing list: `rb-general@lists.reproducible-builds.org`

This month s report was written by Bernhard M. Wiedemann, Chris Lamb, Eli Schwartz, Holger Levsen, Jelle van der Waa and Vagrant Cascadian. It was subsequently reviewed by a bunch of Reproducible Builds folks on IRC and the mailing list.

Astroturfing: How To Spot A Fake Movement

capitalism covid19 news politics Crowds on Demand - Protests, Rallies and Advocacy archive.org

2020-07-06

If the Reopen America protests seem a little off to you, that's because they are. In this video we're going to talk about astroturfing and how insidious it i...

Volunteers 3D-Print Unobtainable $11,000 Valve For $1 To Keep Covid-19 Patients Alive; Original Manufacturer Threatens To Sue

capitalism covid19 health news Volunteers produce 3D-printed valves for life-saving coronavirus treatments archive.org

2020-07-06

Techdirt has just written about the extraordinary legal action taken against a company producing Covid-19 tests. Sadly, it's not the only example of some individuals putting profits before people. Here's a story from Italy, which is...

Germany tries to stop US from luring away firm seeking coronavirus vaccine

capitalism covid19 health news archive.org

2020-07-06

Berlin is trying to stop Washington from persuading a German company seeking a coronavirus vaccine to move its research to the United States.

He Has 17,700 Bottles of Hand Sanitizer and Nowhere to Sell Them

capitalism covid19 news archive.org

2020-07-06

Amazon cracked down on coronavirus price gouging. Now, while the rest of the world searches, some sellers are holding stockpiles of sanitizer and masks.

Theranos vampire lives on: Owner of failed blood-testing biz's patents sues maker of actual COVID-19-testing kit

capitalism covid19 news archive.org

2020-07-06

And 3D-printed valve for breathing machine sparks legal threat

How an Austrian ski paradise became a COVID-19 hotspot

capitalism covid19 news archive.org

2020-07-06

Ischgl, an Austrian ski resort, has achieved tragic international fame: hundreds of tourists are believed to have contracted the coronavirus there and taken it home with them. The Tyrolean state government is now facing serious criticism. EURACTIV Germany reports.

Hospitals Need to Repair Ventilators. Manufacturers Are Making That Impossible

capitalism covid19 health news archive.org

2020-07-06

We are seeing how the monopolistic repair and lobbying practices of medical device companies are making our response to the coronavirus pandemic harder.

Homeless people in Las Vegas sleep 6 feet apart in parking lot as thousands of hotel rooms sit empty

capitalism covid19 news privilege archive.org

2020-07-06

Las Vegas, Nevada has come under criticism after reportedly setting up a temporary homeless shelter in a parking lot complete with social distancing barriers.

Review: The Light Brigade, by Kameron Hurley

Publisher:	Saga
Copyright:	2019
ISBN:	1-4814-4798-X
Format:	Kindle
Pages:	355

In the wake of the Blink, which left a giant crater where S o Paulo was, Dietz signed up for the military. To be a hero. To satisfy an oath of vengeance. To kill aliens. Corporations have consumed the governments that used to run Earth and have divided the world between them. Dietz's family, before the Blink, were ghouls in Tene-Silva territory, non-citizens who scavenged a precarious life on the margins. Citizenship is a reward for loyalty and a mechanism of control. The only people who don't fit into the corporate framework are the Martians, former colonists who went dark for ten years and re-emerged as a splinter group offering to use their superior technology to repair environmental damage to the northern hemisphere caused by corporate wars. When the Blink happens, apparently done with technology far beyond what the corporations have, corporate war with the Martians is the unsurprising result. Long-time SF readers will immediately recognize The Light Brigade as a response to Starship Troopers with far more cynical world-building. For the first few chapters, the parallelism is very strong, down to the destruction of a large South American city (S o Paulo instead of Buenos Aires), a naive military volunteer, and horrific basic training. But, rather than dropships, the soldiers in Dietz's world are sent into battle via, essentially, Star Trek transporters. These still very experimental transporters send Dietz to a different mission than the one in the briefing. Advance warning that I'm going to talk about what's happening with Dietz's drops below. It's a spoiler, but you would find out not far into the book and I don't think it ruins anything important. (On the contrary, it may give you an incentive to stick through the slow and unappealing first few chapters.) I had so many suspension of disbelief problems with this book. So many. This starts with the technology. The core piece of world-building is Star Trek transporters, so fine, we're not talking about hard physics. Every SF story gets one or two free bits of impossible technology, and Hurley does a good job showing the transporters through a jaundiced military eye. But, late in the book, this technology devolves into one of my least-favorite bits of SF hand-waving that, for me, destroyed that gritty edge. Technology problems go beyond the transporters. One of the bits of horror in basic training is, essentially, torture simulators, whose goal is apparently to teach soldiers to dissociate (not that the book calls it that). One problem is that I never understood why a military would want to teach dissociation to so many people, but a deeper problem is that the mechanics of this simulation made no sense. Dietz's training in this simulator is a significant ongoing plot point, and it kept feeling like it was cribbed from The Matrix rather than something translatable into how computers work. Technology was the more minor suspension of disbelief problem, though. The larger problem was the political and social world-building. Hurley constructs a grim, totalitarian future, which is a fine world-building choice although I think it robs some nuance from the story she is telling about how militaries lie to soldiers. But the totalitarian model she uses is one of near-total information control. People believe what the corporations tell them to believe, or at least are indifferent to it. Huge world events (with major plot significance) are distorted or outright lies, and those lies are apparently believed by everyone. The skepticism that exists is limited to grumbling about leadership competence and cynicism about motives, not disagreement with the provided history. This is critical to the story; it's a driver behind Dietz's character growth and is required to set up the story's conclusion. This is a model of totalitarianism that's familiar from Orwell's Nineteen Eighty-Four. The problem: The Internet broke this model. You now need North Korean levels of isolation to pull off total message control, which is incompatible with the social structure or technology level that Hurley shows. You may be objecting that the modern world is full of people who believe outrageous propaganda against all evidence. But the world-building problem is not that some people believe the corporate propaganda. It's that everyone does. Modern totalitarians have stopped trying to achieve uniformity (because it stopped working) and instead make the disagreement part of the appeal. You no longer get half a country to believe a lie by ensuring they never hear the truth. Instead, you equate belief in the lie with loyalty to a social or political group, and belief in the truth with affiliation with some enemy. This goes hand in hand with "flooding the zone" with disinformation and fakes and wild stories until people's belief in the accessibility of objective truth is worn down and all facts become ideological statements. This does work, all too well, but it relies on more information, not less. (See Zeynep Tufekci's excellent Twitter and Tear Gas if you're unfamiliar with this analysis.) In that world, Dietz would have heard the official history, the true history, and all sorts of wild alternative histories, making correct belief a matter of political loyalty. There is no sign of that. Hurley does gesture towards some technology to try to explain this surprising corporate effectiveness. All the soldiers have implants, and military censors can supposedly listen in at any time. But, in the story, this censorship is primarily aimed at grumbling and local disloyalty. There is no sign that it's being used to keep knowledge of significant facts from spreading, nor is there any sign of the same control among the general population. It's stated in the story that the censors can't even keep up with soldiers; one would have to get unlucky to be caught. And yet the corporation maintains preternatural information control. The place this bugged me the most is around knowledge of the current date. For reasons that will be obvious in a moment, Dietz has reasons to badly want to know what month and year it is and is unable to find this information anywhere. This appears to be intentional; Tene-Silva has a good (albeit not that urgent) reason to keep soldiers from knowing the date. But I don't think Hurley realizes just how hard that is. Take a look around the computer you're using to read this and think about how many places the date shows up. Apart from the ubiquitous clock and calendar app, there are dates on every file, dates on every news story, dates on search results, dates in instant messages, dates on email messages and voice mail... they're everywhere. And it's not just the computer. The soldiers can easily smuggle prohibited outside goods into the base; knowledge of the date would be much easier. And even if Dietz doesn't want to ask anyone, there are opportunities to go off base during missions. Somehow every newspaper and every news bulletin has its dates suppressed? It's not credible, and it threw me straight out of the story. These world-building problems are unfortunate, since at the heart of The Light Brigade is a (spoiler alert) well-constructed time travel story that I would have otherwise enjoyed. Dietz is being tossed around in time with each jump. And, unlike some of these stories, Hurley does not take the escape hatch of alternate worlds or possible futures. There is a single coherent timeline that Dietz and the reader experience in one order and the rest of the world experiences in a different order. The construction of this timeline is incredibly well-done. Time can only disconnect at jump and return points, and Hurley maintains tight control over the number of unresolved connections. At every point in the story, I could list all of the unresolved discontinuities and enjoy their complexity and implications without feeling overwhelmed by them. Dietz gains some foreknowledge, but in a way that's wildly erratic and hard to piece together fast enough for a single soldier to do anything about the plot. The world spins out of control with foreshadowing of grimmer and grimmer events, and then Hurley pulls it back together in a thoroughly satisfying interweaving of long-anticipated scenes and major surprises. I'm not usually a fan of time travel stories, but this is one of the best I've read. It also has a satisfying emotional conclusion (albeit marred for me by some unbelievable mystical technobabble), which is impressive given how awful and nasty Hurley makes this world. Dietz is a great first-person narrator, believably naive and cynical by turns, and piecing together the story structure alongside the protagonist built my emotional attachment to Dietz's character arc. Hurley writes the emotional dynamics of soldiers thoughtfully and well: shit-talking, fights, sudden moments of connection, shared cynicism over degenerating conditions, and the underlying growth of squad loyalty that takes over other motivations and becomes the reason to keep on fighting. Hurley also pulled off a neat homage to (and improvement on) Starship Troopers that caught me entirely by surprise and that I've hopefully not spoiled. This is a solid science fiction novel if you can handle the world-building. I couldn't, but I understand why it was nominated for the Hugo and Clarke awards. Recommended if you're less picky about technological and social believability than I am, although content warning for a lot of bloody violence and death (including against children) and a horrifically depressing world. Rating: 6 out of 10

Welcome to the 28th post in the relatively random R recommendations series, or R⁴ for short. Our last post was a double entry in this R⁴ series and the newer T⁴ video series and covered a topic touched upon in this R⁴ series multiple times: easy binary install, especially on Ubuntu. That post already previewed the newest kid on the block: RStudio s RSPM, now formally announced. In the post we were only able to show Ubuntu 18.04 aka bionic. With the formal release of RSPM support has been added for Ubuntu 20.04 aka focal and we are happy to announce that of course we added a corresponding Rocker r-rspm container. So you can now take full advantage of RSPM either via docker pull rocker/r-rspm:18.04 or via docker pull rocker/r-rspm:20.04 covering the two most recent LTS releases. RSPM is a nice accomplishment. Covering multiple Linux distributions is an excellent achievement. Allowing users to reason in terms of the CRAN packages (i.e. installing xml2, not r-cran-xml2) eases use. Doing it from via the standard R command install.packages() (or wrapper around it like our install.r from littler package) is very good too and an excellent technical achievement. There is, as best as I can tell, only one shortcoming, along with one small bit of false advertising. The shortcoming is technical. By bringing the package installation into the user application domain, it is separated from the system and lacks integration with system libraries. What do I mean here? If you were to add R to a plain Ubuntu container, say 18.04 or 20.04, then added the few lines to support RSPM and install xml2 it would install. And fail. Why? Because the system library libxml2 does not get installed with the RSPM package whereas the .deb from the distribution or PPAs does. So to help with some popular packages I added libxml2, libunits and a few more for geospatial work to the rocker/r-rspm containers. Being already present ensures packages xml2 and units can run immediately. Please file issue tickets at the Rocker repo if you come across other missing libraries we could preload. (A related minor nag is incomplete coverage. At least one of my CRAN packages does not (yet?) come as a RSPM binary. Then again, CRAN has 16k packages, and the RSPM coverage is much wider than the PPA one. But completeness would be neat. The final nag is lack of Debian support which seems, well, odd.) So what about the small bit of false advertising? Well it is claimed that RSPM makes installation so much faster on Linux . True, faster than the slowest possible installation from source. Also easier. But we had numerous posts on this blog showing other speed gains: Using ccache. And, of course, using binaries. And as the initial video mentioned above showed, installing from the PPAs is also faster than via RSPM. That is easy to replicate. Just set up the rocker/r-ubuntu:20.04 (or 18.04) container alongside the rocker/r-rspm:20.04 (or also 18.04) container. And then time install.r rstan (or install.r tinyverse) in the RSPM one against apt -y update; apt install -y r-cran-rstan (or ... r-cran-tinyverse). In every case I tried, the installation using binaries from the PPA was still faster by a few seconds. Not that it matters greatly: both are very, very quick compared to source installation (as e.g. shown here in 2017 (!!)) but the standard Ubuntu .deb installation is simply faster than using RSPM. (Likely due to better CDN usage so this may change over time. Neither method appears to do downloads in parallel so there is scope for both for doing better.) So in sum: Welcome to RSPM, and nice new tool and feel free to drive it using rocker/r-rspm:18.04 or rocker/r-rspm:20.04. If you like this or other open-source work I do, you can now sponsor me at GitHub. For the first year, GitHub will match your contributions.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

It seems GitHub is planning to rename the default brach for newly created repositories from "master" to "main". It's incredible how much positive PR you can get with a one line configuration change, while still working together with the ICE. However, this post is not about bashing GitHub. Changing the default branch for newly created repositories is good. And you also should do that for the ones you create with git init locally. But what about all the repositories out there? GitHub surely won't force-rename those branches, but we can! Ian will do this as he touches the individual repositories, but I tend to forget things unless I do them immediately Oh, so this is another "automate everything with an API" post? Yes, yes it is! And yes, I am going to use GitHub here, but something similar should be implementable on any git hosting platform that has an API. Of course, if you have SSH access to the repositories, you can also just edit HEAD in an for loop in bash, but that would be boring ;-) I'm going with devel btw, as I'm already used to develop in the Foreman project and devel in Ansible. acquire credentials My GitHub account is 2FA enabled, so I can't just use my username and password in a basic HTTP API client. So the first step is to acquire a personal access token, that can be used instead. Of course I could also have implemented OAuth2 in my lousy script, but ain't nobody have time for that. The token will require the "repo" permission to be able to change repositories. And we'll need some boilerplate code (I'm using Python3 and requests, but anything else will work too):

#!/usr/bin/env python3
import requests
BASE='https://api.github.com'
USER='evgeni'
TOKEN='abcdef'
headers =  'User-Agent': '@ '.format(USER) 
auth = (USER, TOKEN)
session = requests.Session()
session.auth = auth
session.headers.update(headers)
session.verify = True

This will store our username, token, and create a requests.Session so that we don't have to pass the same data all the time. get a list of repositories to change I want to change all my own repos that are not archived, not forks, and actually have the default branch set to master, YMMV. As we're authenticated, we can just list the repositories of the currently authenticated user, and limit them to "owner" only. GitHub uses pagination for their API, so we'll have to loop until we get to the end of the repository list.

repos_to_change = []
url = ' /user/repos?type=owner'.format(BASE)
while url:
    r = session.get(url)
    if r.ok:
        repos = r.json()
        for repo in repos:
            if not repo['archived'] and not repo['fork'] and repo['default_branch'] == 'master':
                repos_to_change.append(repo['name'])
        if 'next' in r.links:
            url = r.links['next']['url']
        else:
            url = None
    else:
        url = None

create a new devel branch and mark it as default Now that we know which repos to change, we need to fetch the SHA of the current master, create a new devel branch pointing at the same commit and then set that new branch as the default branch.

for repo in repos_to_change:
    master_data = session.get(' /repos/evgeni/ /git/ref/heads/master'.format(BASE, repo)).json()
    data =  'ref': 'refs/heads/devel', 'sha': master_data['object']['sha'] 
    session.post(' /repos/ / /git/refs'.format(BASE, USER, repo), json=data)
    default_branch_data =  'default_branch': 'devel' 
    session.patch(' /repos/ / '.format(BASE, USER, repo), json=default_branch_data)
    session.delete(' /repos/ / /git/refs/heads/ '.format(BASE, USER, repo, 'master'))

I've also opted in to actually delete the old master, as I think that's the safest way to let the users know that it's gone. Letting it rot in the repository would mean people can still pull and won't notice that there are no changes anymore as the default branch moved to devel. So announcement I've updated all my (those in the evgeni namespace) non-archived repositories to have devel instead of master as the default branch. Have fun updating! code

#!/usr/bin/env python3
import requests
BASE='https://api.github.com'
USER='evgeni'
TOKEN='abcd'
headers =  'User-Agent': '@ '.format(USER) 
auth = (USER, TOKEN)
session = requests.Session()
session.auth = auth
session.headers.update(headers)
session.verify = True
repos_to_change = []
url = ' /user/repos?type=owner'.format(BASE)
while url:
    r = session.get(url)
    if r.ok:
        repos = r.json()
        for repo in repos:
            if not repo['archived'] and not repo['fork'] and repo['default_branch'] == 'master':
                repos_to_change.append(repo['name'])
        if 'next' in r.links:
            url = r.links['next']['url']
        else:
            url = None
    else:
        url = None
for repo in repos_to_change:
    master_data = session.get(' /repos/evgeni/ /git/ref/heads/master'.format(BASE, repo)).json()
    data =  'ref': 'refs/heads/devel', 'sha': master_data['object']['sha'] 
    session.post(' /repos/ / /git/refs'.format(BASE, USER, repo), json=data)
    default_branch_data =  'default_branch': 'devel' 
    session.patch(' /repos/ / '.format(BASE, USER, repo), json=default_branch_data)
    session.delete(' /repos/ / /git/refs/heads/ '.format(BASE, USER, repo, 'master'))

If you have multiple PHP web sites on a server in a default configuration they will all be able to read each other s files in a default configuration. If you have multiple PHP web sites that have stored data or passwords for databases in configuration files then there are significant problems if they aren t all trusted. Even if the sites are all trusted (IE the same person configures them all) if there is a security problem in one site it s ideal to prevent that being used to immediately attack all sites. mpm_itk The first thing I tried was mpm_itk [1]. This is a version of the traditional prefork module for Apache that has one process for each HTTP connection. When it s installed you just put the directive AssignUserID USER GROUP in your VirtualHost section and that virtual host runs as the user:group in question. It will work with any Apache module that works with mpm_prefork. In my experiment with mpm_itk I first tried running with a different UID for each site, but that conflicted with the pagespeed module [2]. The pagespeed module optimises HTML and CSS files to improve performance and it has a directory tree where it stores cached versions of some of the files. It doesn t like working with copies of itself under different UIDs writing to that tree. This isn t a real problem, setting up the different PHP files with database passwords to be read by the desired group is easy enough. So I just ran each site with a different GID but used the same UID for all of them. The first problem with mpm_itk is that the mpm_prefork code that it s based on is the slowest mpm that is available and which is also incompatible with HTTP/2. A minor issue of mpm_itk is that it makes Apache take ages to stop or restart, I don t know why and can t be certain it s not a configuration error on my part. As an aside here is a site for testing your server s support for HTTP/2 [3]. To enable HTTP/2 you have to be running mpm_event and enable the http2 module. Then for every virtual host that is to support it (generally all https virtual hosts) put the line Protocols h2 h2c http/1.1 in the virtual host configuration. A good feature of mpm_itk is that it has everything for the site running under the same UID, all Apache modules and Apache itself. So there s no issue of one thing getting access to a file and another not getting access. After a trial I decided not to keep using mpm_itk because I want HTTP/2 support. php-fpm Pools The Apache PHP module depends on mpm_prefork so it also has the issues of not working with HTTP/2 and of causing the web server to be slow. The solution is php-fpm, a separate server for running PHP code that uses the fastcgi protocol to talk to Apache. Here s a link to the upstream documentation for php-fpm [4]. In Debian this is in the php7.3-fpm package. In Debian the directory /etc/php/7.3/fpm/pool.d has the configuration for pools . Below is an example of a configuration file for a pool:

# cat /etc/php/7.3/fpm/pool.d/example.com.conf
[example.com]
user = example.com
group = example.com
listen = /run/php/php7.3-example.com.sock
listen.owner = www-data
listen.group = www-data
pm = dynamic
pm.max_children = 5
pm.start_servers = 2
pm.min_spare_servers = 1
pm.max_spare_servers = 3

Here is the upstream documentation for fpm configuration [5]. Then for the Apache configuration for the site in question you could have something like the following:

ProxyPassMatch "^/(.*\.php(/.*)?)$" "unix:/run/php/php7.3-example.com.sock fcgi://localhost/usr/share/wordpress/"

The fcgi://localhost part is just part of the way of specifying a Unix domain socket. From the Apache Wiki it appears that the method for configuring the TCP connections is more obvious [6]. I chose Unix domain sockets because it allows putting the domain name in the socket address. Matching domains for the web server to port numbers is something that s likely to be error prone while matching based on domain names is easier to check and also easier to put in Apache configuration macros. There was some additional hassle with getting Apache to read the files created by PHP processes (the options include running PHP scripts with the www-data group, having SETGID directories for storing files, and having world-readable files). But this got things basically working. Nginx My Google searches for running multiple PHP sites under different UIDs didn t turn up any good hits. It was only after I found the DigitalOcean page on doing this with Nginx [7] that I knew what to search for to find the way of doing it in Apache.

Here is my transparent report for my work on the Debian Long Term Support (LTS) and Debian Extended Long Term Support (ELTS), which extend the security support for past Debian releases, as a paid contributor. In June, the monthly sponsored hours were split evenly among contributors depending on their max availability - I was assigned 30h for LTS (out of 30 max; all done) and 5.25h for ELTS (out of 20 max; all done). While LTS is part of the Debian project, fellow contributors sometimes surprise me: suggestion to vote for sponsors-funded projects with concorcet was only met with overhead concerns, and there were requests for executive / business owner decisions (we're currently heading towards consultative vote); I heard concerns about discussing non-technical issues publicly (IRC team meetings are public though); the private mail infrastructure was moved from self-hosting straight to Google; when some got an issue with Debian Social for our first video conference, there were immediate suggestions to move to Zoom...
Well, we do need some people to make those LTS firmware updates in non-free

Also this was the last month before shifting suites: goodbye to Jessie LTS and Wheezy ELTS, welcome Stretch LTS and Jessie ELTS. ELTS - Wheezy

mysql-connector-java: improve testsuite setup; prepare wheezy/jessie/stretch triple builds; coordinate versioning scheme with security-team; security upload ELA 234-1
ntp: wheezy+jessie triage: 1 ignored (too intrusive to backport); 1 postponed (hard to exploit, no patch)
Clean-up (ditch) wheezy VMs

LTS - Jessie

mysql-connector-java: see common work in ELTS
mysql-connector-java: security uploads DLA 2245-1 (LTS) and DSA 4703 (oldstable)
ntp: wheezy+jessie triage (see ELTS)
rails: global triage, backport 2 patches, security upload DLA 2251-1
rails: global security: prepare stretch/oldstable update
rails: new important CVE on unmaintained 4.x, fixes introduce several regressions, propose new fix to upstream, update stretch proposed update [and jessie, but rails will turn out unsupported in ELTS]
python3.4: prepare update to fix all pending non-criticial issues, 5/6 ready
private video^W^Wpublic IRC team meeting

Documentation/Scripts

LTS/TestsSuites/mysql-connector-java: improve testsuite setup for better coverage
LTS/TestSuites/tiff: document package maintainer's (extensive) tests
LTS/TestSuites/rails: first version
LTS/TestSuites/python: how to run individual test
LTS/Development: clarifications on grouping fixes and validating patches
internal discussion on (not) capping LTS-funded hours
discussion on unbound and freerdp EOL
tzdata, libdatetime-timezone-perl: check and explain delayed update workflow
ELTS: update new tracker URL in documentation

Focus This month I didn't have any particular focus. I just worked on issues in my info bubble.

Changes

git-imerge: use python3 by default

youtube-dl: add RTRFM radio station extractor

ifenslave: cleanup, fix infinite recursion, fix missing variables

check-all-the-things: update URLs, add many sources of and ideas for new checks

how-can-i-help: ignore debhelper cruft, build manual page from source, prevent printing autorm header unless needed

apt-listchanges: cleanups (1 2 3 4), remove hardcoding, add latest option, syslog/log frontends and title disable option, configuration directory

lintian: use UTF-8 aware spelling sorting, fix wording (1 2)

spelling dictionaries: codespell (1 2 3 4 5 6 7 8), lintian (1 2 3 4 5 6 7 8 9)

Debian usertags QA: drop hardcoding, ignore cruft, refactor, add rename/transfer/copy functions, debug mode (1 2)

Debian package uploads: ifenslave (1 2), lxml

Debian wiki pages: AutomatedInstallation, Cloud/AmazonEC2Image, DebianAlternatives, DebianEvents/internet/2020/MiniDebConfOnline, DebianInstaller/Bugs, DebianMentors (Faq), DebianNetDomains, DebianTesting, Derivatives/Census (EmmabuntusDE, Template), DeveloperNews, Firmware/Open, GetInvolved-Beta, HelpDebian, InstallingDebianOn (Dell, Lemote/Loongson2K1000, Marvell, Router, Turris, TUXEDO (NOTEBOOKS/InfinityBookS14v5)), IRC, Keysigning/Offers, LTS, MemberBenefits, MIPS/LoongsonPi, Mobile, Multiarch/Tuples, PanfrostLima, PinePhone, Ports/ia64, PortTemplate, ProgrammingLanguage, Schroot, snd-emu10k1, SponsoredMaintainer, SystemBuildTools, Teams (3D-printer, Cloud, Debbugs/ArchitectureTags, DPL/OfficialImages)

FOSSjobs wiki pages: resources

Issues

Crashes in nouveau libraries (reported on IRC), webkit2gtk

Move conserver from Debian non-free to main

File conflict in brz

Outdated description in hub

Strange dpkg-parsechangelog use in fritzing, fritzing-parts

Patch merging needed in apt-listchanges

Missing link in password-manager-resources

Conffile removal needed in bsdmainutils

Features in lintian

Missing quoting in wifi-qr

Review

Spam: reported 174 Debian mailing list posts

Debian wiki: RecentChanges for the month

Debian screenshots:

approved bashtop buildbot buildstream deepin-movie deepin-terminal ding drawing dte firefox-locale-gu font-manager fonts-hermit fonts-proggy fonts-terminus gimp-gutenprint gnome-builder gnome-shell-extension-arc-menu gztool kdevelop lilypond lxhotkey-gtk mkdocs-nature ncdu nvidia-legacy-340xx-driver obs-studio opencubicplayer openshot-qt packetsender partitionmanager plakativ poezio pulseeffects pyzo roundcube scalpel scrcpy tesseract-ocr thunar tilde tracetuner verbiste-gtk zytrax

rejected packetsender (help output), pyzo/sasm/gztool/herculesstudio/posterazor (not Debian), minetest-mod-character-creator (not minetest)

rejected deletion of partimage/flashrom/apache2 (spam)

Administration

Debian BTS: usertags QA

Debian IRC channels: fixed a channel mode lock

Debian wiki: unblock IP addresses, approve accounts, ping folks with bouncing email

Communication

Respond to queries from Debian users and developers on the mailing lists and IRC

Sponsors The ifenslave and apt-listchanges work was sponsored by my employer. All other work was done on a volunteer basis.

Here is my monthly update covering what I have been doing in the free software world during June 2020 (previous month):

Opened two pull requests against the Ghostwriter distraction-free Markdown editor to:
- Persist whether "focus mode" is enabled across between sessions. (#522)
- Correct the ordering of the MarkdownAST::toString() debugging output. (#520)
Will McGugan's "Rich" is a Python library to output formatted text, tables, syntax etc. to the terminal. I filed a pull request in order to allow for easy enabling and disabling of displaying the file path in Rich's logging handler. (#115)
As part of my duties of being on the board of directors of the Open Source Initiative and Software in the Public Interest I attended their respective monthly meetings and participated in various licensing and other discussions occurring on the internet, as well as the usual internal discussions regarding logistics and policy etc.
Filed a pull request against the PyQtGraph Scientific Graphics and graphical user interface library to make the documentation build reproducibly. (#1265)
Reviewed and merged a large number of changes by Pavel Dolecek to my Strava Enhancement Suite, a Chrome extension to improve the user experience on the Strava athletic tracker.

For Lintian, the static analysis tool for Debian packages:

Don't emit breakout-link for architecture-independent .jar files under /usr/lib. (#963939)
Correct a reference to override_dh_ in the long description of the excessive-debhelper-overrides tag. [...]
Update data/fields/perl-provides for Perl 5.030003. [...]
Check for execute_after and execute_before spelling mistakes just like override_*. [...]

Reproducible Builds One of the original promises of open source software is that distributed peer review and transparency of process results in enhanced end-user security. However, whilst anyone may inspect the source code of free and open source software for malicious flaws, almost all software today is distributed as pre-compiled binaries. This allows nefarious third-parties to compromise systems by injecting malicious code into ostensibly secure software during the various compilation and distribution processes. The motivation behind the Reproducible Builds effort is to ensure no flaws have been introduced during this compilation process by promising identical results are always generated from a given source, thus allowing multiple third-parties to come to a consensus on whether a build was compromised. The project is proud to be a member project of the Software Freedom Conservancy. Conservancy acts as a corporate umbrella allowing projects to operate as non-profit initiatives without managing their own corporate structure. If you like the work of the Conservancy or the Reproducible Builds project, please consider becoming an official supporter. This month, I:

Filed a pull request against the PyQtGraph to make the documentation build reproducibly. (#1265)
Addressed a regression in the octave-queueing package in the handling of absolute paths in Texinfo files. (#962187)
Submitted the following patches to fix reproducibility-related toolchain issues within Debian:
- critcl: Please make the teapot.txt files reproducible. (#963600)
- sasmodels: Regression in debian/patches/reproducible-c-models.patch. (#962050)
Contributed to a discussion regarding improving the reproducibility of packages that use the CMake build system. [...]
Added to a conversation regarding the nondeterministic execution of order of Debian maintainer scripts that results in the arbitrary allocation of UNIX group IDs referencing the Tails operating system. [...]
I also submitted 14 patches to fix specific reproducibility issues in fonts-anonymous-pro, gftl, golang-github-viant-toolbox, golang-v2ray-core, libmbim, libqmi, neovim-qt, netcdf-fortran, petitboot, python-pauvre, python-pyqtgraph, python-stem, seqtools & tkabber-plugins.
Kept isdebianreproducibleyet.com up to date. [...][...]
Categorised a large number of packages and issues in the Reproducible Builds "notes" repository.
Updated the main Reproducible Builds website and documentation to limit the number of "news" posts to, for example, avoid showing items from 2017. [...]
Drafted, published and publicised our monthly report.

Elsewhere in our tooling, I made the following changes to diffoscope including preparing and uploading versions 147, 148 and 149 to Debian:

New features:
- Add output from strings(1) to ELF binaries. (#148)
- Allow user to mask/filter diff output via --diff-mask=REGEX. (!51)
- Dump PE32+ executables (such as EFI applications) using objdump(1). (#181)
- Add support for Zsh shell completion. (#158)
Bug fixes:
- Prevent a traceback when comparing PDF documents that did not contain metadata (ie. a PDF /Info stanza). (#150)
- Fix compatibility with jsondiff version 1.2.0. (#159)
- Fix an issue in GnuPG keybox file handling that left filenames in the diff. [...]
- Correct detection of JSON files due to missing call to File.recognizes that checks candidates against file(1). [...]
Output improvements:
- Use the CSS word-break property over manually adding U+200B zero-width spaces as these were making copy-pasting cumbersome. (!53)
- Downgrade the tlsh warning message to an "info" level warning. (#29)
Logging improvements:
- Log calls to subprocess.check_output by using a wrapper. (#151)
- Clarify that we are generating presenter formats in a debug-level message. [...]
- Log the version of jsondiff used. [...]
Testsuite improvements:
- Update tests for file(1) version 5.39. (#179)
- Drop accidentally-duplicated copy of the --diff-mask tests. [...]
- Don't mask an existing test. [...]
Codebase improvements:
- Replace obscure references to "WF" with "Wagner-Fischer" for clarity. [...]
- Use a semantic AbstractMissingType type instead of remembering to check for both types of "missing" files. [...]
- Add a comment regarding potential security issue in the .changes, .dsc and .buildinfo comparators. [...]
- Drop a large number of unused imports. [...][...][...][...][...]
- Make many code sections more Pythonic. [...][...][...][...]
- Prevent some variable aliasing issues. [...][...][...]
- Use some tactical f-strings to tidy up code [...][...] and remove explicit u"unicode" strings [...].
- Refactor a large number of routines for clarity. [...][...][...][...]

trydiffoscope is the web-based version of diffoscope. This month, I specified a location for the celerybeat scheduler to ensure that the clean/tidy tasks are actually called which had caused an accidental resource exhaustion. (#12)

Debian I filed three bugs against:

cmark: Please update the homepage URI. (#962576)
petitboot: Please update Vcs-Git urls. (#963123)
python-pauvre: FTBFS if the DISPLAY environment variable is exported. (#962698)

Debian LTS This month I have worked 18 hours on Debian Long Term Support (LTS) and 5 hours on its sister Extended LTS project.

Investigated and triaged angular.js [...],icinga2 [...], intel-microcode [...], jquery [...], pdns-recursor [...], unbound [...] & wordpress [...].
Frontdesk duties, including responding to user/developer questions, reviewing others' packages, participating in mailing list discussions as well as attending our contributor meeting.
Issued DLA 2233-1 to fix two issues in the Django web development framework in order to fix a potential data leakage via malformed memcached keys (CVE-2020-13254) and to prevent a cross-site scripting attack in the Django administration system (CVE-2020-13596). This was followed by DLA 2233-2 to address a regression as well as uploads to Debian stretch (1.10.7-2+deb9u9) and buster (1.11.29-1~deb10u1). (More info)
Issued DLA 2235-1 to prevent a file descriptor leak in the D-Bus message bus (CVE-2020-12049).
Issued DLA 2239-1 for a security module for using the TACACS+ authentication service to prevent an issue where shared secrets such as private server keys were being added in plaintext to various logs.
Issued DLA 2244-1 to address an escaping issue in PHPMailer, an email generation utility class for the PHP programming language.
Issued DLA 2252-1 for the ngircd IRC server as it was discovered that there was an out-of-bounds access vulnerability in the server-to-server protocol.
Issued DLA 2253-1 to resolve a vulnerability in the Lynis a security auditing tool because a shared secret could be obtained by simple observation of the process list when a data upload is being performed.

You can find out more about the project via the following video:

Uploads

redis (6.0.4-1 & 6.0.5-1) New upstream releases.
python-django:
- 2.2.13-1 New upstream security release followed by a fix for an upstream regression 2.2.13-2
- 3.0.7-1 New upstream security release followed by a fix for an upstream regression in 3.0.7-2
- 3.1~beta1-1 New upstream beta release.

I can t believe I spent all these years using only grep. Most of us know how to use awk to print the nth column of a file:

$ awk ' print $1 ' /etc/hosts

will print all IP addresses from /etc/hosts But you can also do filtering before printing the chosen column:

$ awk '$5 >= 2  print $2 ' /path/to/file

will print the second column of all lines, where the 5th column is greater than 2. That would have been hard with grep. Now I can use that to find out all deployments on my openshift cluster, where the number of current replicas is greater than 2.

$ oc get deployments --all-namespaces   awk '$5 >= 2  print $2 '
NAME
oauth-openshift
console
downloads
router-default
etcd-quorum-guard
prometheus-adapter
thanos-querier
packageserver

I know that openshift/kubernetes both have a powerful query selector syntax, but for the moment awk will do.

Search Results: "spa"

16 July 2020

Enrico Zini: Build Qt5 cross-builder with raspbian sysroot: compiling with the sysroot

Enrico Zini: Build Qt5 cross-builder with raspbian sysroot: building the sysroot

Russell Coker: Windows 10 on Debian under KVM

14 July 2020

Ian Jackson: MessagePack vs CBOR (RFC7049)

12 July 2020

Evgeni Golov: Using Ansible Molecule to test roles in monorepos

11 July 2020

Simon Quigley: Adventures in Writing

10 July 2020

Enrico Zini: Wait until a command opened a file

9 July 2020

Enrico Zini: Mime type associations

Enrico Zini: Laptop migration

6 July 2020

Dirk Eddelbuettel: Rcpp 1.0.5: Several Updates

Reproducible Builds: Reproducible Builds in June 2020

5 July 2020

Enrico Zini: COVID-19 and Capitalism

4 July 2020

Russ Allbery: Review: The Light Brigade

3 July 2020

Dirk Eddelbuettel: #28: Welcome RSPM and test-drive with Bionic and Focal

2 July 2020

Evgeni Golov: Automatically renaming the default git branch to "devel"

Russell Coker: Isolating PHP Web Sites

1 July 2020

Sylvain Beucler: Debian LTS and ELTS - June 2020

Paul Wise: FLOSS Activities June 2020

Focus This month I didn't have any particular focus. I just worked on issues in my info bubble.

Administration

Debian BTS: usertags QA

Debian IRC channels: fixed a channel mode lock

Debian wiki: unblock IP addresses, approve accounts, ping folks with bouncing email

Communication

Respond to queries from Debian users and developers on the mailing lists and IRC

Sponsors The ifenslave and apt-listchanges work was sponsored by my employer. All other work was done on a volunteer basis.

30 June 2020

Chris Lamb: Free software activities in June 2020

Emmanuel Kasper: Learning openshift: a good moment to revisit awk too

Search Results: "spa"

16 July 2020

14 July 2020

12 July 2020

11 July 2020

10 July 2020

9 July 2020

6 July 2020

5 July 2020

4 July 2020

3 July 2020

2 July 2020

1 July 2020

Focus This month I didn't have any particular focus. I just worked on issues in my info bubble.

Administration Debian BTS: usertags QA Debian IRC channels: fixed a channel mode lock Debian wiki: unblock IP addresses, approve accounts, ping folks with bouncing email

Communication Respond to queries from Debian users and developers on the mailing lists and IRC

Sponsors The ifenslave and apt-listchanges work was sponsored by my employer. All other work was done on a volunteer basis.

30 June 2020

Administration

Debian BTS: usertags QA

Debian IRC channels: fixed a channel mode lock

Debian wiki: unblock IP addresses, approve accounts, ping folks with bouncing email

Communication

Respond to queries from Debian users and developers on the mailing lists and IRC