kitty
comes with a thing to transfer that definition, but it
breaks if the remote host is running tcsh
(don't ask). Similary the
one liner for alacritty
on the arch wiki seems to assume the remote
shell is bash. Forthwith, a dumb shell script that works to send the
terminfo of the current terminal emulator to the remote host.
EDIT: Jakub Wilk worked out this can be replaced with the oneliner
infocmp ssh $host tic -x -
#!/bin/sh
if [ "$#" -ne 1 ]; then
printf "usage: sendterminfo host\n"
exit 1
fi
host="$1"
filename=$(mktemp terminfoXXXXXX)
cleanup ()
rm "$filename"
trap cleanup EXIT
infocmp > "$filename"
remotefile=$(ssh "$host" mktemp)
scp -q "$filename" "$host:$remotefile"
ssh "$host" "tic -x \"$remotefile\""
ssh "$host" rm "$remotefile"
kitty
comes with a thing to transfer that definition, but it
breaks if the remote host is running tcsh
(don't ask). Similary the
one liner for alacritty
on the arch wiki seems to assume the remote
shell is bash. Forthwith, a dumb shell script that works to send the
terminfo of the current terminal emulator to the remote host.
EDIT: Jakub Wilk worked out this can be replaced with the oneliner
infocmp ssh $host tic -x -
#!/bin/sh
if [ "$#" -ne 1 ]; then
printf "usage: sendterminfo host\n"
exit 1
fi
host="$1"
filename=$(mktemp terminfoXXXXXX)
cleanup ()
rm "$filename"
trap cleanup EXIT
infocmp > "$filename"
remotefile=$(ssh "$host" mktemp)
scp -q "$filename" "$host:$remotefile"
ssh "$host" "tic -x \"$remotefile\""
ssh "$host" rm "$remotefile"
setup.py
.
I could do git grep -h import sort -u
, then review the output by hand, but
I lacked the motivation for it. Much better to take a stab at solving the
general problem
The result is at https://github.com/spanezz/python-devel-tools.
One fun part is scanning a directory tree, using ast
to find import
statements scattered around the code:
class Scanner:
def __init__(self):
self.names: Set[str] = set()
def scan_dir(self, root: str):
for dirpath, dirnames, filenames, dir_fd in os.fwalk(root):
for fn in filenames:
if fn.endswith(".py"):
with dirfd_open(fn, dir_fd=dir_fd) as fd:
self.scan_file(fd, os.path.join(dirpath, fn))
st = os.stat(fn, dir_fd=dir_fd)
if st.st_mode & (stat.S_IXUSR stat.S_IXGRP stat.S_IXOTH):
with dirfd_open(fn, dir_fd=dir_fd) as fd:
try:
lead = fd.readline()
except UnicodeDecodeError:
continue
if re_python_shebang.match(lead):
fd.seek(0)
self.scan_file(fd, os.path.join(dirpath, fn))
def scan_file(self, fd: TextIO, pathname: str):
log.info("Reading file %s", pathname)
try:
tree = ast.parse(fd.read(), pathname)
except SyntaxError as e:
log.warning("%s: file cannot be parsed", pathname, exc_info=e)
return
self.scan_tree(tree)
def scan_tree(self, tree: ast.AST):
for stm in tree.body:
if isinstance(stm, ast.Import):
for alias in stm.names:
if not isinstance(alias.name, str):
print("NAME", repr(alias.name), stm)
self.names.add(alias.name)
elif isinstance(stm, ast.ImportFrom):
if stm.module is not None:
self.names.add(stm.module)
elif hasattr(stm, "body"):
self.scan_tree(stm)
scanner = Scanner()
scanner.scan_dir(args.dir)
sys.path.append(args.dir)
by_sys_path: Dict[str, List[str]] = collections.defaultdict(list)
for name in sorted(scanner.names):
spec = importlib.util.find_spec(name)
if spec is None or spec.origin is None:
by_sys_path[""].append(name)
else:
for sp in sys.path:
if spec.origin.startswith(sp):
by_sys_path[sp].append(name)
break
else:
by_sys_path[spec.origin].append(name)
for sys_path, names in sorted(by_sys_path.items()):
print(f" sys_path or 'unidentified' :")
for name in names:
print(f" name ")
$ ./scan-imports /himblick
unidentified:
changemonitor
chroot
cmdline
mediadir
player
server
settings
static
syncer
utils
/himblick:
himblib.cmdline
himblib.host_setup
himblib.player
himblib.sd
/usr/lib/python3.9:
__future__
argparse
asyncio
collections
configparser
contextlib
datetime
io
json
logging
mimetypes
os
pathlib
re
secrets
shlex
shutil
signal
subprocess
tempfile
textwrap
typing
/usr/lib/python3/dist-packages:
asyncssh
parted
progressbar
pyinotify
setuptools
tornado
tornado.escape
tornado.httpserver
tornado.ioloop
tornado.netutil
tornado.web
tornado.websocket
yaml
built-in:
sys
time
Build-Arch-
Conflicts
,Depends
to build-rdeps. (adc87981)remote.<name>.push-url
when using debcheckout to clone a git repository. (Debian bug #753838)gpg --with-colons
, instead of manually parsing gpg -K
output.
Aside from being the Right Way to get machine parseable information out of gpg, it fixed completion when gpg
is a 2.x version. (Debian bug #837380)DEB_BUILD_PROFILES
and using a separate, minimal docker image for running autopkgtests.
unibilium
reproducible-check is our script to determine which packages actually installed on your system are reproducible or not.
diffoscope is our in-depth and content-aware diff utility that can locate and diagnose reproducibility issues.
strip-nondeterminism is our tool to remove specific non-deterministic results from a completed build.
disorderfs is our FUSE-based filesystem that deliberately introduces non-determinism into directory system calls in order to flush out reproducibility issues.
defer_pwchange
support in Heimdal, and I spent
some time on that and tracked it down to an upstream bug in Heimdal as
well as a few issues in pam-krb5. The pam-krb5 issues are now fixed in
Git, but I haven't gotten any response upstream from the Heimdal bug
report. I also dusted off three old Heimdal patches and submitted them as
upstream merge requests and reported some more deficiencies I found in
FAST support. On the pam-krb5 front, I updated the test suite for the
current version of Heimdal (which changed some of the prompting) and
updated the portability support code, but haven't yet pulled the trigger
on a new release.
Other Software
I merged a couple of pull requests in podlators, one to fix various typos
(thanks, Jakub Wilk) and one to change the formatting of man page
references and function names to match the current Linux manual page
standard (thanks, Guillem Jover). I also documented a bad interaction
with line-buffered output in the Term::ANSIColor man page. Neither of
these have seen a new release yet.
py3versions to list versions in a consistent order
. Issue reported by Santiago Vila with a tentative patch by Chris Lamb. Sadly, it appears the problem is not entirely solved.Pod::Man
. This would complement or replace the previously implemented POD_MAN_DATE
environment variable in a more generic way.
Niko Tyni proposed a fix to prevent mtime variation in directories due to debhelper usage of cp --parents -p
.
Packages fixed
The following 119 packages became reproducible due to changes in their
build dependencies:
aac-tactics,
aafigure,
apgdiff,
bin-prot,
boxbackup,
calendar,
camlmix,
cconv,
cdist,
cl-asdf,
cli-common,
cluster-glue,
cppo,
cvs,
esdl,
ess,
faucc,
fauhdlc,
fbcat,
flex-old,
freetennis,
ftgl,
gap,
ghc,
git-cola,
globus-authz-callout-error,
globus-authz,
globus-callout,
globus-common,
globus-ftp-client,
globus-ftp-control,
globus-gass-cache,
globus-gass-copy,
globus-gass-transfer,
globus-gram-client,
globus-gram-job-manager-callout-error,
globus-gram-protocol,
globus-gridmap-callout-error,
globus-gsi-callback,
globus-gsi-cert-utils,
globus-gsi-credential,
globus-gsi-openssl-error,
globus-gsi-proxy-core,
globus-gsi-proxy-ssl,
globus-gsi-sysconfig,
globus-gss-assist,
globus-gssapi-error,
globus-gssapi-gsi,
globus-net-manager,
globus-openssl-module,
globus-rsl,
globus-scheduler-event-generator,
globus-xio-gridftp-driver,
globus-xio-gsi-driver,
globus-xio,
gnome-control-center,
grml2usb,
grub,
guilt,
hgview,
htmlcxx,
hwloc,
imms,
kde-l10n,
keystone,
kimwitu++,
kimwitu-doc,
kmod,
krb5,
laby,
ledger,
libcrypto++,
libopendbx,
libsyncml,
libwps,
lprng-doc,
madwimax,
maria,
mediawiki-math,
menhir,
misery,
monotone-viz,
morse,
mpfr4,
obus,
ocaml-csv,
ocaml-reins,
ocamldsort,
ocp-indent,
openscenegraph,
opensp,
optcomp,
opus,
otags,
pa-bench,
pa-ounit,
pa-test,
parmap,
pcaputils,
perl-cross-debian,
prooftree,
pyfits,
pywavelets,
pywbem,
rpy,
signify,
siscone,
swtchart,
tipa,
typerep,
tyxml,
unison2.32.52,
unison2.40.102,
unison,
uuidm,
variantslib,
zipios++,
zlibc,
zope-maildrophost.
The following packages became reproducible after getting fixed:
reproducible.debian.net
).PPPort_xs.PL
. Forwarded upstream.amd64
packages has been lowered from 14 to 7 days, thanks to the increase of hardware resources sponsored by ProfitBricks last week. (h01ger)
diffoscope development
diffoscope version 37 has been released on October 15th. It adds support for two new file formats (CBFS images and Debian .dsc
files). After proposing the required changes to TLSH, fuzzy hashes are now computed incrementally. This will avoid reading entire files in memory which caused problems for large packages.
New tests have been added for the command-line interface. More character encoding issues have been fixed. Malformed md5sums
will now be compared as binary files instead of making diffoscope crash amongst several other minor fixes.
Version 38 was released two days later to fix the versioned dependency on python3-tlsh.
strip-nondeterminism development
strip-nondeterminism version 0.013-1 has been uploaded to the archive. It fixes an issue with nonconformant PNG files with trailing garbage reported by Roland Rosenfeld.
disorderfs development
disorderfs version 0.4.1-1 is a stop-gap release that will disable lock propagation, unless --share-locks=yes
is specified, as it still is affected by unidentified issues.
Documentation update
Lunar has been busy creating a proper website for reproducible-builds.org
that would be a common location for news, documentation, and tools for all free software projects working on reproducible builds. It's not yet ready to be published, but it's surely getting there.
ibus-table-createdb
deterministic.
Niko Tyni wrote a patch to make libmodule-build-perl linking order deterministic.
Santiago Vila has been leading discussions on the best way to fix timestamps coming from Gettext POT files.
Packages fixed
The following 35 packages became reproducible due to changes in their
build dependencies:
apache-log4j2,
dctrl-tools,
dms,
gitit,
gnubik,
isrcsubmit,
mailutils,
normaliz,
oaklisp,
octave-fpl,
octave-specfun,
octave-vrml,
opencolorio,
openvdb,
pescetti,
php-guzzlehttp,
proofgeneral,
pyblosxom,
pyopencl,
pyqi,
python-expyriment,
python-flask-httpauth,
python-mzml,
python-simpy,
python-tidylib,
reactive-streams,
scmxx,
shared-mime-info,
sikuli,
siproxd,
srtp,
tachyon,
tcltk-defaults,
urjtag,
velvet.
The following packages became reproducible after getting fixed:
C
when sorting source file list.debian/changelog
entry in build string..pyc
files..pyc
files.debian/changelog
entry.debian/changelog
entry as build time.debian/changelog
entry as build time.Build.PL
.debian/changelog
entry as build time.--fuzzy-threshold
option to specify the TLSH score used as cut-off
for fuzzy matching. Specifying 0
will disable fuzzy-matching entirely.
Suggested by Jakub Wilk.--new-file
option to treat absent files as empty. This make diffoscope a great
tool to look at the content of an archive at once by comparing it with a non-existent
file (example).
Suggested by Jakub Wilk.--help
..file
assembler directive can help with random filenames in debug symbols.
Package reviews
235 reviews have
been removed, 84 added and 277 updated this week.
29 new FTBFS bugs were filled by Chris Lamb, Chris West (Faux), Daniel Stender, and Niko Tyni.
New issues identified this week: random_order_in_ibus_table_createdb_output, random_order_in_antlr_output, nondetermistic_link_order_in_module_build, and timestamps_in_tex_documents.
Misc.
Thanks to Dhole and Thomas Vincent, the talk held at DebConf15 now has subtitles!
Void Linux started to merge changes to make packages produced by xbps reproducible.
Week 1 | Week 2 | Week 3 | Week 4 | Week 5 | Week 6 | Week 7 | |
---|---|---|---|---|---|---|---|
# Packages | 10 | 15 | 10 | 14 | 10 | 9 | 21 |
Total | 10 | 25 | 35 | 49 | 59 | 68 | 89 |
jessie
release I spent most of my Debian time on work
in the Debian Perl
Group. we tried to get down the list of new upstream releases (from over
500 to currently 379; unfortunately the CPAN never sleeps), we were &
still are busy preparing for the Perl
5.22 transition (e.g. we uploaded something between 300 & 400
packages to deal with Module::Build & CGI.pm being removed from perl
core; only team-maintained packages so far), & we had a pleasant &
productive sprint
in Barcelona in May. & I also tried to fix some of the RC bugs
in our packages which popped up over the previous months.
yesterday & today I finally found some time to help with the GCC 5 transition, mostly by making
QA or Non-Maintainer Uploads with patches that already were in the BTS.
a big thanks especially to the team at HP which provided a couple
dozens patches!
& here's the list of RC bugs I've worked on in the last 3 months:
Debian is undertaking a huge effort to develop a reproducible builds system. I'd like to thank you for that. This could be Debian's most important project, with how badly computer security has been going.
PerniciousPunk in Reddit's Ask me anything! to Neil McGovern, DPL. What happened in the reproducible builds effort this week: Toolchain fixes More tools are getting patched to use the value of the SOURCE_DATE_EPOCH environment variable as the current time:
SOURCE_DATE_EPOCH
to the time of the latest debian/changelog
entry when exporting build flags, patch sent as #791823 (Dhole),texlive-bin
(akira) and libxslt
(Dhole) with the aforementioned support for SOURCE_DATE_EPOCH
.debhelper
exported TZ=UTC
and this made packages capturing the current date (without the time) reproducible in the current test environment.
The following packages became reproducible after getting fixed:
debian/changelog
date in the manpage.debian/changelog
date as build date and use debian
as the builder hostname.debian/changelog
date as bui
ld date.reproducible.debian.net
.
--clamp-mtime
option to tar.
Patches have been submitted to add support for SOURCE_DATE_EPOCH to txt2man (Reiner Herrmann), epydoc (Reiner Herrmann), GCC (Dhole), and Doxygen (akira).
Dhole uploaded a new experimental debhelper to the reproducible repository which exports SOURCE_DATE_EPOCH. As part of the experiment, the patch also sets TZ
to UTC
which should help with most timezone issues. It might still be problematic for some packages which would change their settings based on this.
Mattia Rizzolo sent upstream a patch originally written by Lunar to make the generate-id()
function be deterministic in libxslt. While that patch was quickly rejected by upstream, Andrew Ayer came up with a much better one which sadly could have some performance impact. Daniel Veillard replied with another patch that should be deterministic in most cases without needing extra data structures. It's impact is currently being investigated by retesting packages on reproducible.debian.net
.
akira added a new option to sbuild for configuring the path in which packages are built. This will be needed for the srebuild script.
Niko Tyni asked Perl upstream about it using the __DATE__
and __TIME__
C processor macros.
Packages fixed
The following 143 packages became reproducible due to changes in their
build dependencies:
alot,
argvalidate,
astroquery,
blender,
bpython,
brian,
calibre,
cfourcc,
chaussette,
checkbox-ng,
cloc,
configshell,
daisy-player,
dipy,
dnsruby,
dput-ng,
dsc-statistics,
eliom,
emacspeak,
freeipmi,
geant321,
gpick,
grapefruit,
heat-cfntools,
imagetooth,
jansson,
jmapviewer,
lava-tool,
libhtml-lint-perl,
libtime-y2038-perl,
lift,
lua-ldoc,
luarocks,
mailman-api,
matroxset,
maven-hpi-plugin,
mknbi,
mpi4py,
mpmath,
msnlib,
munkres,
musicbrainzngs,
nova,
pecomato,
pgrouting,
pngcheck,
powerline,
profitbricks-client,
pyepr,
pylibssh2,
pylogsparser,
pystemmer,
pytest,
python-amqp,
python-apt,
python-carrot,
python-crypto,
python-darts.lib.utils.lru,
python-demgengeo,
python-graph,
python-mock,
python-musicbrainz2,
python-pathtools,
python-pskc,
python-psutil,
python-pypump,
python-repoze.sphinx.autointerface,
python-repoze.tm2,
python-repoze.what-plugins,
python-repoze.what,
python-repoze.who-plugins,
python-xstatic-term.js,
reclass,
resource-agents,
rgain,
rttool,
ruby-aggregate,
ruby-archive-tar-minitar,
ruby-bcat,
ruby-blankslate,
ruby-coffee-script,
ruby-colored,
ruby-dbd-mysql,
ruby-dbd-odbc,
ruby-dbd-pg,
ruby-dbd-sqlite3,
ruby-dbi,
ruby-dirty-memoize,
ruby-encryptor,
ruby-erubis,
ruby-fast-xs,
ruby-fusefs,
ruby-gd,
ruby-git,
ruby-globalhotkeys,
ruby-god,
ruby-hike,
ruby-hmac,
ruby-integration,
ruby-ipaddress,
ruby-jnunemaker-matchy,
ruby-memoize,
ruby-merb-core,
ruby-merb-haml,
ruby-merb-helpers,
ruby-metaid,
ruby-mina,
ruby-net-irc,
ruby-net-netrc,
ruby-odbc,
ruby-packet,
ruby-parseconfig,
ruby-platform,
ruby-plist,
ruby-popen4,
ruby-rchardet,
ruby-romkan,
ruby-rubyforge,
ruby-rubytorrent,
ruby-samuel,
ruby-shoulda-matchers,
ruby-sourcify,
ruby-test-spec,
ruby-validatable,
ruby-wirble,
ruby-xml-simple,
ruby-zoom,
ryu,
simplejson,
spamassassin-heatu,
speaklater,
stompserver,
syncevolution,
syncmaildir,
thin,
ticgit,
tox,
transmissionrpc,
vdr-plugin-xine,
waitress,
whereami,
xlsx2csv,
zathura.
The following packages became reproducible after getting fixed:
debian/changelog
entry as build datedebian/changelog
entry.#define
s..-i/--issues
: schedule all packages affected by the given issue.-r/--status
: schedule all packages with the given status.-b/--before
: schedule all packages built before the given date-t/--after
: schedule all packages built after the given date.--noisy
: notify the IRC channel also when the build starts, with a URL to watch it in real time..deb
stable.
akira filled #789843 to make tex4ht stop printing timestamps in its HTML output by default.
Dhole wrote a patch for xutils-dev to prevent timestamps when creating gzip compresed files.
Reiner Herrmann sent a follow-up patch for wheel to use UTC as timezone when outputing timestamps.
Mattia Rizzolo started a discussion regarding the failure to build from source of subversion when -Wdate-time
is added to CPPFLAGS
which happens when asking dpkg-buildflags
to use the reproducible
profile. SWIG errors out because it doesn't recognize the aforementioned flag.
Trying to get the .buildinfo specification to more definitive state, Lunar started a discussion on storing the checksums of the binary package used in dpkg
status database.
akira discovered while proposing a fix for simgrid that CMake internal command to create tarballs would record a timestamp in the gzip header. A way to prevent it is to use the GZIP
environment variable to ask gzip
not to store timestamps, but this will soon become unsupported. It's up for discussion if the best place to fix the problem would be to fix it for all CMake users at once.
Infrastructure-related work
Andreas Henriksson did a delayed NMU upload of pbuilder which adds minimal support for build profiles and includes several fixes from Mattia Rizzolo affecting reproducibility tests.
Neils Thykier uploaded lintian which both raises the severity of package-contains-timestamped-gzip and avoids false positives for this tag (thanks to Tomasz Buchert).
Petter Reinholdtsen filled #789761 suggesting that how-can-i-help should prompt its users about fixing reproducibility issues.
Packages fixed
The following packages became reproducible due to changes in their
build dependencies:
autorun4linuxcd,
libwildmagic,
lifelines,
plexus-i18n,
texlive-base,
texlive-extra,
texlive-lang.
The following packages became reproducible after getting fixed:
debian/changelog
entry.CMakeLists.txt
to give GZIP=-n for tar.__DATE__
and __TIME__
macros.HTML_TIMESTAMP=NO
in Doxygen configuration.HTML_TIMESTAMP=NO
to Doxygen.$datetime
from footer.html
used by Doxygen.HTML_TIMESTAMP=NO
in Doxygen configuration.$Storable::canonical = 1
to make space_groups.db.PL
output deterministic.HTML_TIMESTAMP=NO
in Doxygen configuration.HTML_TIMESTAMP=NO
in Doxygen configuration.HTML_TIMESTAMP=NO
in Doxygen configuration.HTML_TIMESTAMP=NO
in Doxygen configuration.HTML_TIMESTAMP=NO
in Doxygen configuration.$datetime
from the footer.HTML_TIMESTAMP=NO
in Doxygen configuration.LC_ALL
to C
before sorting.debian/changelog
as build datejavap
, readelf
, objdump
, zipinfo
, unsqusahfs
; useless MD5 checksum and last modified date in javap
output; bad handling of charsets in PO files; the destination path for gzip compressed files not ending in .gz
; only metadata of cpio
archives were actually compared. stat
output was further trimmed to make directory comparison more useful.
Having the test suite enabled a refactoring of how comparators were written, switching from a forest of differences to a single tree. This helped removing dust from the oldest parts of the code.
Together with some other small changes, version 25 was released on June 27th. A follow up release was made the next day to fix a hole in the test suite and the resulting unidentified leftover from the comparator refactoring. (Lunar)
Documentation update
Ximin Luo improved code examples for some proposed environment variables for reference timestamps. Dhole added an example on how to fix timestamps C pre-processor macros by adding a way to set the build date externally. akira documented her fix for tex4ht timestamps.
Package reviews
94 obsolete
reviews have
been removed, 330 added and 153 updated this week.
Hats off for Chris West (Faux) who investigated many fail to build from source issues and reported the relevant bugs.
Slight improvements were made to the scripts for editing the review database, edit-notes and clean-notes. (Mattia Rizzolo)
Meetings
A meeting was held on June 23rd. Minutes are available.
The next meeting will happen on Tuesday 2015-07-07 at 17:00 UTC.
Misc.
The Linux Foundation announced that it was funding the work of Lunar and h01ger on reproducible builds in Debian and other distributions. This was further relayed in a Bits from Debian blog post.
Next.