Search Results: "ed"

6 November 2021

Evgeni Golov: I just want to run this one Python script

So I couldn't sleep the other night, and my brain wanted to think about odd problems Ever had a script that's compatible with both, Python 2 and 3, but you didn't want to bother the user to know which interpreter to call? Maybe because the script is often used in environments where only one Python is available (as either /usr/bin/python OR /usr/bin/python3) and users just expect things to work? And it's only that one script file, no package, no additional wrapper script, nothing. Yes, this is a rather odd scenario. And yes, using Python doesn't make it easier, but trust me, you wouldn't want to implement the same in bash. Nothing that you will read from here on should ever be actually implemented, it will summon dragons and kill kittens. But it was a fun midnight thought, and I like to share nightmares! The nice thing about Python is it supports docstrings, essentially strings you can put inside your code which are kind of comments, but without being hidden inside commnent blocks. These are often used for documentation that you can reach using Python's help() function. (Did I mention I love help()?) Bash on the other hand, does not support docstrings. Even better, it doesn't give a damn whether you quote commands or not. You can call "ls" and you'll get your directory listing the same way as with ls. Now, nobody would under normal circumstances quote ls. Parameters to it, sure, those can contain special characters, but ls?! Another nice thing about Python: it doesn't do any weird string interpolation by default (ssssh, f-strings are cool, but not default). So "$(ls)" is exactly that, a string containing a Dollar sign, an open parenthesis, the characters "l" and "s" and a closing parenthesis. Bash, well, Bash will run ls, right? If you don't yet know where this is going, you have a clean mind, enjoy it while it lasts! So another thing that Bash has is exec: "replace[s] the shell without creating a new process". That means that if you write exec python in your script, the process will be replaced with Python, and when the Python process ends, your script ends, as Bash isn't running anymore. Well, technically, your script ended the moment exec was called, as at that point there was no more Bash process to execute the script further. Using exec is a pretty common technique if you want to setup the environment for a program and then call it, without Bash hanging around any longer as it's not needed. So we could have a script like this:
#!/bin/bash
exec python myscript.py "$@"
Here "$@" essentially means "pass all parameters that were passed to this Bash script to the Python call" (see parameter expansion). We could also write it like this:
#!/bin/bash
"exec" "python" "myscript.py" "$@"
As far as Bash is concerned, this is the same script. But it just became valid Python, as for Python those are just docstrings. So, given this is a valid Bash script and a valid Python script now, can we make it do something useful in the Python part? Sure!
#!/bin/bash
"exec" "python" "myscript.py" "$@"
print("lol")
If we call this using Bash, it never gets further than the exec line, and when called using Python it will print lol as that's the only effective Python statement in that file. Okay, but what if this script would be called myscript.py? Exactly, calling it with Python would print lol and calling it with Bash would end up printing lol too (because it gets re-executed with Python). We can even make it name-agnostic, as Bash knows the name of the script we called:
#!/bin/bash
"exec" "python" "$0" "$@"
print("lol")
But this is still calling python, and it could be python3 on the target (or even something worse, but we're not writing a Halloween story here!). Enter another Bash command: command (SCNR!), especially "The -v option causes a single word indicating the command or file name used to invoke command to be displayed". It will also exit non-zero if the command is not found, so we can do things like $(command -v python3 command -v python) to find a Python on the system.
#!/bin/bash
"exec" "$(command -v python3   command -v python)" "$0" "$@"
print("lol")
Not well readable, huh? Variables help!
#!/bin/bash
__PYTHON="$(command -v python3   command -v python)"
"exec" "$ __PYTHON " "$0" "$@"
print("lol")
For Python the variable assignment is just a var with a weird string, for Bash it gets executed and we store the result. Nice! Now we have a Bash header that will find a working Python and then re-execute itself using said Python, allowing us to use some proper scripting language. If you're worried that $0 won't point at the right file, just wrap it with some readlink -f .

Reproducible Builds: Reproducible Builds in October 2021

Welcome to the October 2021 report from the Reproducible Builds project!
This month Samanta Navarro posted to the oss-security security mailing on a novel category of exploit in the .tar archive format, where a single .tar file contains different contents depending on the tar utility being used. Naturally, this has consequences for reproducible builds as Samanta goes onto reply:

Arch Linux uses libarchive (bsdtar) in its build environment. The default tar program installed is GNU tar. It is possible to create a source distribution which leads to different files seen by the build environment than compared to a careful reviewer and other Linux distributions.
Samanta notes that addressing the tar utilities themselves will not be a sufficient fix:
I have submitted bug reports and patches to some projects but eventually I had to conclude that the problem itself cannot be fixed by these implementations alone. The best choice for these tools would be to only allow archives which are fully compatible to standards but this in turn would render a lot of archives broken.
Reproducible builds, with its twin ideas of reaching consensus on the build outputs as well as precisely recording and describing the build environment, would help address this problem at a higher level.
Codethink announced that they had achieved ISO-26262 ASIL D Tool Certification, a way of determining specific safety standards for software. Codethink used open source tooling to achieve this, but they also leverage:
Reproducibility, repeatability and traceability of builds, drawing heavily on best-practices championed by the Reproducible Builds project.

Elsewhere on the internet, according to a comment on Hacker News, Microsoft are now comparing NPM Javascript packages with their original source repositories:
I got a PR in my repository a few days ago leading back to a team trying to make it easier for packages to be reproducible from source.

Lastly, Martin Monperrus started an interesting thread on our mailing list about Github, specifically that their autogenerated release tarballs are not deterministic . The thread generated a significant number of replies that are worth reading.

Events and presentations

Community news On our mailing list this month:
There were quite a few changes to the Reproducible Builds website and documentation this month as well, including Feng Chai updating some links on our publications page [ ] and marco updated our project metadata around the Bitcoin Core building guide [ ].
Lastly, we ran another productive meeting on IRC during October. A full set of notes from the meeting is available to view.

Distribution work Qubes was heavily featured in the latest edition of Linux Weekly News, and a significant section was dedicated to discussing reproducibility. For example, it was mentioned that the Qubes project has been working on incorporating reproducible builds into its continuous integration (CI) infrastructure . But the LWN article goes on to describe that:
The current goal is to be able to build the Qubes OS Debian templates solely from packages that can be built reproducibly. Templates in Qubes OS are VM images that can be used to start an application qube quickly based on the template. The qube will have read-only access to the root filesystem of the template, so that the same root filesystem can be shared with multiple application qubes. There are official templates for several variants of both Fedora and Debian, as well as community maintained templates for several other distributions.
You can view the whole article on LWN, and Fr d ric also published a lengthy summary about their work on reproducible builds in Qubes as well for those wishing to learn more.
In Debian this month, 133 reviews of Debian packages were added, 81 were updated and 24 were removed this month, adding to Debian s ever-growing knowledge about identified issues. A number of issues were categorised and added by Chris Lamb and Vagrant Cascadian too [ ][ ][ ]. In addition, work on alternative snapshot service has made progress by Fr d ric Pierret and Holger Levsen this month, including moving from the existing host (snapshot.notset.fr) to snapshot.reproducible-builds.org (more info) thanks to OSUOSL for the machine and hosting and Debian for the disks.
Finally, Bernhard M. Wiedemann posted his monthly reproducible builds status report.

diffoscope diffoscope is our in-depth and content-aware diff utility. Not only can it locate and diagnose reproducibility issues, it can provide human-readable diffs from many kinds of binary formats. This month, Chris Lamb made the following changes, including preparing and uploading versions 186, 187, 188 and 189 to Debian
  • New features:
    • Add support for Python Sphinx inventory files (usually named objects.inv on-disk). [ ]
    • Add support for comparing .pyc files. Thanks to Sergei Trofimovich for the inspiration. [ ]
    • Try some alternative suffixes (e.g. .py) to support distributions that strip or retain them. [ ][ ]
  • Bug fixes:
    • Fix Python decompilation tests under Python 3.10+ [ ] and for Python 3.7 [ ].
    • Don t raise a traceback if we cannot unmarshal Python bytecode. This is in order to support Python 3.7 failing to load .pyc files generated with newer versions of Python. [ ]
    • Skip Python bytecode testing where we do not have an expected diff. [ ]
  • Codebase improvements:
    • Use our file_version_is_lt utility instead of accepting both versions of uImage expected diff. [ ]
    • Split out a custom call to assert_diff for a .startswith equivalent. [ ]
    • Use skipif instead of manual conditionals in some tests. [ ]
In addition, Jelle van der Waa added external tool references for Arch Linux for ocamlobjinfo, openssl and ffmpeg [ ][ ][ ] and added Arch Linux as a Continuous Integration (CI) test target. [ ] and Vagrant Cascadian updated the testsuite to skip Python bytecode comparisons when file(1) is older than 5.39. [ ] as well as added external tool references for the Guix distribution for dumppdf and ppudump. [ ][ ]. Vagrant Cascadian also updated the diffoscope package in GNU Guix [ ][ ]. Lastly, Guangyuan Yang updated the FreeBSD package name on the website [ ], Mattia Rizzolo made a change to override a new Lintian warning due to the new test files [ ], Roland Clobus added support to detect and log if the GNU_BUILD_ID field in an ELF binary been modified [ ], Sandro J ckel updated a number of helpful links on the website [ ] and Sergei Trofimovich made the uImage test output support file() version 5.41 [ ].

reprotest reprotest is the Reproducible Build s project end-user tool to build same source code twice in widely differing environments, checking the binaries produced by the builds for any differences. This month, reprotest version 0.7.18 was uploaded to Debian unstable by Holger Levsen, which also included a change by Holger to clarify that Python 3.9 is used nowadays [ ], but it also included two changes by Vasyl Gello to implement realistic CPU architecture shuffling [ ] and to log the selected variations when the verbosity is configured at a sufficiently high level [ ]. Finally, Vagrant Cascadian updated reprotest to version 0.7.18 in GNU Guix.

Upstream patches The Reproducible Builds project detects, dissects and attempts to fix unreproducible packages. We try to send all of our patches upstream where appropriate. We authored a large number of such patches this month, including:

Testing framework The Reproducible Builds project runs a testing framework at tests.reproducible-builds.org, to check packages and other artifacts for reproducibility. This month, the following changes were made:
  • Holger Levsen:
    • Debian-related changes:
      • Incorporate a fix from bremner into builtin-pho related to binary-NMUs. [ ]
      • Keep bullseye environments around longer, in an attempt to fix a Jenkins issue. [ ]
      • Improve the documentation of buildinfos.debian.net. [ ]
      • Improve documentation for the builtin-pho setup. [ ][ ]
    • OpenWrt-related changes:
      • Also use -j1 for better debugging. [ ]
      • Document that that Python 3.x is now used. [ ]
      • Enable further debugging for the toolchain build. [ ]
    • New snapshot.reproducible-builds.org service:
      • Actually add new node. [ ][ ]
      • Install xfsprogs on snapshot.reproducible-builds.org. [ ]
      • Create account for fpierret on new node. [ ]
      • Run node_health_check job on new node too. [ ]
  • Mattia Rizzolo:
    • Debian-related changes:
      • Handle schroot errors when invoking diffoscope instead of masking them. [ ][ ]
      • Declare and define some variables separately to avoid masking the subshell return code. [ ]
      • Fix variable name. [ ]
      • Improve log reporting. [ ]
      • Execute apt-get update with the -q argument to get more decent logs. [ ]
      • Set the Debian HTTP mirror and proxy for snapshot.reproducible-builds.org. [ ]
      • Install the libarchive-tools package (instead of bsdtar) when updating Jenkins nodes. [ ]
    • Be stricter about errors when starting the node agent [ ] and don t overwrite NODE_NAME so that we can expect Jenkins to properly set for us [ ].
    • Explicitly warn if the NODE_NAME is not a fully-qualified domain name (FQDN). [ ]
    • Document whether a node runs in the future. [ ]
    • Disable postgresql_autodoc as it not available in bullseye. [ ]
    • Don t be so eager when deleting schroot internals, call to schroot -e to terminate the schroots instead. [ ]
    • Only consider schroot underlays for deletion that are over a month old. [ ][ ]
    • Only try to unmount /proc if it s actually mounted. [ ]
    • Move the db_backup task to its own Jenkins job. [ ]
Lastly, Vasyl Gello added usage information to the reproducible_build.sh script [ ].

Contributing If you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:

Vincent Bernat: How to rsync files between two remotes?

scp -3 can copy files between two remote hosts through localhost. This comes in handy when the two servers cannot communicate directly or if they are unable to authenticate one to the other.1 Unfortunately, rsync does not support such a feature. Here is a trick to emulate the behavior of scp -3 with SSH tunnels. When syncing with a remote host, rsync invokes ssh to spawn a remote rsync --server process. It interacts with it through its standard input and output. The idea is to recreate the same setup using SSH tunnels and socat, a versatile tool to establish bidirectional data transfers. The first step is to connect to the source server and ask rsync the command-line to spawn the remote rsync --server process. The -e flag overrides the command to use to get a remote shell: instead of ssh, we use echo.
$ ssh web04
$ rsync -e 'sh -c ">&2 echo $@" echo' -aLv /data/. web05:/data/.
web05 rsync --server -vlogDtpre.iLsfxCIvu . /data/.
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(228) [sender=3.2.3]
The second step is to connect to the destination server with local port forwarding. When connecting to the local port 5000, the TCP connection is forwarded through SSH to the remote port 5000 and handled by socat. When receiving the connection, socat spawns the rsync --server command we got at the previous step and connects its standard input and output to the incoming TCP socket.
$ ssh -L 127.0.0.1:5000:127.0.0.1:5000 web05
$ socat tcp-listen:5000,reuseaddr exec:"rsync --server -vlogDtpre.iLsfxCIvu . /data/."
The last step is to connect to the source with remote port forwarding. socat is used in place of a regular SSH connection and connects its standard input and output to a TCP socket connected to the remote port 5000. Thanks to the remote port forwarding, SSH forwards the data to the local port 5000. From there, it is relayed back to the destination, as described in the previous step.
$ ssh -R 127.0.0.1:5000:127.0.0.1:5000 web04
$ rsync -e 'sh -c "socat stdio tcp-connect:127.0.0.1:5000"' -aLv /data/. remote:/data/.
sending incremental file list
haproxy.debian.net/
haproxy.debian.net/dists/buster-backports-1.8/Contents-amd64.bz2
haproxy.debian.net/dists/buster-backports-1.8/Contents-i386.bz2
[ ]
media.luffy.cx/videos/2021-frnog34-jerikan/progressive.mp4
sent 921,719,453 bytes  received 26,939 bytes  7,229,383.47 bytes/sec
total size is 7,526,872,300  speedup is 8.17
This little diagram may help understand how everything fits together:
Diagram showing how all processes are connected together: rsync, socat and ssh
How each process is connected together. Arrows labeled stdio are implemented as two pipes connecting the process to the left to the standard input and output of the process to the right. Don't be fooled by the apparent symmetry!
The rsync manual page prohibits the use of --server. Use this hack at your own risk!
The options --server and --sender are used internally by rsync, and should never be typed by a user under normal circumstances. Some awareness of these options may be needed in certain scenarios, such as when setting up a login that can only run an rsync command. For instance, the support directory of the rsync distribution has an example script named rrsync (for restricted rsync) that can be used with a restricted ssh login.

Addendum I was hoping to get something similar with a one-liner. But this does not work!
$ socat \
>  exec:"ssh web04 rsync --server --sender -vlLogDtpre.iLsfxCIvu . /data/." \
>  exec:"ssh web05 rsync --server -vlogDtpre.iLsfxCIvu /data/. /data/." \
over-long vstring received (511 > 255)
over-long vstring received (511 > 255)
rsync error: requested action not supported (code 4) at compat.c(387) [sender=3.2.3]
rsync error: requested action not supported (code 4) at compat.c(387) [Receiver=3.2.3]
socat[878291] E waitpid(): child 878292 exited with status 4
socat[878291] E waitpid(): child 878293 exited with status 4

  1. And SSH agent forwarding is dangerous. Don t use it if you can.

Jaldhar Vyas: Sal Mubarak 2078!

Diwali Fireworks Wishing everyone in Debian good health and prosperity in the Gujarati New Year Vikram Samvat 2078 called Pramadi.

5 November 2021

Jonathan Dowland: 25 things I would like to 3D print

Last year I started collecting ideas of things I would like to 3D print one day, on Twitter. Twitter is fundamentally ephemeral, so I'll collect it here instead. I got up to 14 items on Twitter, and now I'm up to 25. I don't own a 3D printer, but I have access to one at the work office. Perhaps this list is my subconcious trying to convince me to buy one. What am I missing? What else should I be thinking of printing? Let me know!
  1. Some kind of 45 leaning prong to dry bottles and flasks on
  2. A tea tray and coasters
  3. a replacement prop arm/foot for my computer keyboard (something like this but for the Lenovo Ultranav)
  4. some attempted representation of Borges Library of Babel, a la @jwz The Library of Babel, again
  5. an exploration of the geometry of Susanna Clarke s Piranesi
  6. further iterations of my castle
  7. Small tins to keep loose-leaf tea in
  8. Who am I kidding, bound to be a map from DOOM. E1M1 perhaps, or something more regular (MAP07? E2M8?) See also this amazing print of Quake 3: Arena's "Camping Grounds"
  9. replacement bits for kids toy sets, e.g. a bolt with long shank from Early Learning Centre Build It Deluxe Set, without all 4 of which you can't build much of anything
  10. A stand for a decorative Christmas bauble (kid's hand print on it) A roll of cellotape works pretty well in the mean time
  11. DIY bits-and-bobs sorter/storage (nuts and bolts etc)
  12. A space ship from Elite/Frontier. Probably a Cobra mk3 or maybe a Viper mk2. In a glow-in-the-dark PLA which i d overpaint with gunmetal except for the fuselage
  13. A watch stand/holder/storage thing Except it would look nicer in wood (And I m more inclined to get rid of all but one watch instead)
  14. Little tabbed 7 dividers for vinyl records, with A-Z cut into the tabs 12 ones might be a stretch (something like this)
  15. A low-profile custom trackpoint cover like the ones by SaotoTech (e.g.)
  16. A vinyl record. (Not sure that any 3D printer I would have access to would have the necessary resolution. I haven't done any research yet.)
  17. A free-standing inclined vinyl record display stand (e.g.)
  18. A "Settlers of Catan" set. I've got the travel edition which is great but it would be nicer to have a larger-sized set. There are some things I really like about the travel set that the full-size set lacks; so designing and printing a larger set myself could incorporate them. Also I don't feel inclined to buy the full-size set for 50 or so to end up with essentially the same game I already bought. No doubt I'd spent at least that much in PLA.
  19. Little kids trinkets. Pacman ghosts, that sort-of thing. Whatever my daughters come up with next.
  20. Lego storage/sorters.
  21. Some kind of lenticular picture. Perhaps a gift or Christmas card combined in one.
  22. A bracket to install a Gotek drive in my Amiga 500 (e.g.). I've managed without but the fit isn't great.
  23. An attempt at using the 3D printer for 2D drawing. I would never get the same kind of quality results you can get from a proper plotter, but still Take a look at some proper plotter art!
  24. Garden decorations. I like the idea of porous geometric shapes that you can plant mosses or ferns into, but also things which might be taken over and "used" by nature in ways I hadn't thought of.
  25. Floor plans / 3D plans of my house (including variations if I remodelled)

Molly de Blanc: Tea(GL) Sandwiches

For SeaGL 2021 I m making some tea sandwiches. I know nothing about tea sandwiches, but read a Wikipedia article. If you want to make them along with me, I recommend getting the ingredients ahead of time. Smoked Not-salmon Sandwiches I guess you could make these with smoked salmon, though I think it s cleverer this way. Ingredients Procedure Apple and Brie Sandwiches This sounded good to me. Let s see how it goes. Ingredients Procedure

Jonathan Dowland: Mastodon again

I've been an occasional Mastodon user for (apparently) four years. I experimented with running my own instance for a while before giving that up, then I shopped around for an interesting little instance and settled on one, which was lovely until the maintainer decided (quite reasonably) it wasn't worth the trouble any more and turned it off. So I'm in need of a new instance! The first account I had was @Jmtd@mastodon.cloud, but that instance was too busy for my liking. I'd like to find something with small but vibrant local traffic, ideally matching to my interests. In fact one of the problems with my last instance1 was it was too quiet, and the local timeline was mostly empty. So something between "empty" and "overwhelmingly busy" as mastodon.cloud is, or was. Recommendations welcome! I'd set @jmtd@mastodon.cloud to forward to my instance for most of the last four years. I've turned that off now, so I am (at least temporarily) back to that instance. That's got me thinking: perhaps it's less onerous to run a local instance that only forwards, and to use that as a sort-of "permalink", allowing me to migrate from real instance-to-instance when required, but giving me a Mastodon "perma-link". Something to look at. (It's also worth considering that you should back up the list of accounts you are following off whichever instance you are on. I'm starting again with almost a clean slate.)

  1. I'm deliberately omitting the name of my last instance to give the former maintainer a break.

Reproducible Builds (diffoscope): diffoscope 190 released

The diffoscope maintainers are pleased to announce the release of diffoscope version 190. This version includes the following changes:
[ Chris Lamb ]
* Don't raise a traceback if we cannot de-marshal Python bytecode to support
  Python 3.7 loading newer .pyc files.
  (Closes: reproducible-builds/diffoscope#284)
* Fix Python tests under Python 3.7 with file 5.39+.
[ Vagrant Cascadian ]
* Skip Python bytecode testing when "file" is older than 5.39.
[ Roland Clobus ]
* Detect whether the GNU_BUILD_ID field has been modified.
You find out more by visiting the project homepage.

4 November 2021

Antoine Beaupr : A Python contextmanager gotcha

Dear lazy web... I've had this code sitting around as a wtf.py for a while. I've been meaning to understand what's going on and write a blog post about it for a while, but I'm lacking the time. Now that I have a few minutes, I actually sat down to look at it and I think I figured it out:
from contextlib import contextmanager
@contextmanager
def bad():
    print(&aposin the context manager&apos)
    try:
        print("yielding value")
        yield &aposvalue&apos
    finally:
        return print(&aposcleaning up&apos)
@contextmanager
def good():
    print(&aposin the context manager&apos)
    try:
        print("yielding value")
        yield &aposvalue&apos
    finally:
        print(&aposcleaning up&apos)
with bad() as v:
    print(&aposgot v = %s&apos % v)
    raise Exception(&aposexception not raised!&apos)  # SILENCED!
print("this code is reached")
with good() as v:
    print(&aposgot v = %s&apos % v)
    raise Exception(&aposexpection normally raised&apos)
print("NOT REACHED (expected)")
For those, like me, who need a walkthrough, here's what the above does:
  1. define a bad context manager (the things you use with with statements) with contextlib.contextmanager) which:
    1. prints a debug statement
    2. return a value
    3. then returns and prints a debug statement
  2. define a good context manager in much the same way, except it doesn't return, it just prints statement
  3. use the bad context manager to show how it bypasses an exception
  4. use the good context manager to show how it correctly raises the exception
The output of this code (in Debian 11 bullseye, Python 3.9.2) is:
in the context manager
yielding value
got v = value
cleaning up
this code is reached
in the context manager
yielding value
got v = value
cleaning up
Traceback (most recent call last):
  File "/home/anarcat/wikis/anarc.at/wtf.py", line 31, in <module>
    raise Exception('expection normally raised')
Exception: expection normally raised
What is surprising to me, with this code, is not only does the exception not get raised, but also the return statement doesn't seem to actually execute, or at least not in the parent scope: if it would, this code is reached wouldn't be printed and the rest of the code wouldn't run either. So what's going on here? Now I know that I should be careful with return in my context manager, but why? And why is it silencing the exception? The reason why it's being silenced is this little chunk in the with documentation:
If the suite was exited due to an exception, and the return value from the exit() method was false, the exception is reraised. If the return value was true, the exception is suppressed, and execution continues with the statement following the with statement.
This feels a little too magic. If you write a context manager with __exit__(), you're kind of forced to lookup again what that API is. But the contextmanager decorator hides that away and it's easy to make that mistake... Credits to the Python tips book for teaching me about that trick in the first place.

Mike Gabriel: Call for Translations: Ayatana Indicators 0.9.x Release Series

We (Robert Tari, the UBports developers team, myself) are very close to releasing Ayatana Indicators 0.9.x. The work on Ayatana Indicators is currently nearly completed funded by the UBports Foundation and over the past half year, many many changes, improvements and clean-ups have been added to the code. Ayatana Indicators 0.9.x will be the first release series to be in the development tree of Ubuntu Touch 20.04 (which is currently under very heavy development). Ayatana Indicators 0.9.x will also be used in various other desktop environments available in upcoming Ubuntu 22.04 LTS, such as Ubuntu MATE, Xubuntu, (optionally in) Ubuntu Budgie (please correct my wording, if you know better), (send me a note, if I forgot your desktop env), etc. So, to all Ubuntu Touch, Ubuntu MATE, Xubuntu, etc. users. If you not already are a translator of Ayatana Indicators and you are good in English and fluent in at least one other language, please consider helping out with translating or improving translations of Ayatana Indicators. The translation work needs to be done on Hosted Weblate [1], please sign up for an account (if you haven't done so, yet) and chime in. Thanks so much for your contributions! light+love
Mike https://hosted.weblate.org/projects/ayatana-indicators/

Sandro Tosi: Python: send emails with embedded images

to send emails with images you need to use MIMEMultipart, but the basic approach:
import smtplib

from email.mime.multipart import MIMEMultipart
from email.mime.image import MIMEImage

msg = MIMEMultipart('alternative')
msg['Subject'] = "subject"
msg['From'] = from_addr
msg['To'] = to_addr

part = MIMEImage(open('/path/to/image', 'rb').read())

s = smtplib.SMTP('localhost')
s.sendmail(from_addr, to_addr, msg.as_string())
s.quit()

will produce an email with empty body and the image as an attachment.The better way, ie to have the image as part of the body of the email, requires to write an HTML body that refers to that image:
import smtplib

from email.mime.multipart import MIMEMultipart
from email.mime.text import MIMEText
from email.mime.image import MIMEImage

msg = MIMEMultipart('alternative')
msg['Subject'] = "subject
msg['From'] = from_addr
msg['To'] = to_addr

text = MIMEText('<img src="cid:image1">', 'html')
msg.attach(text)

image = MIMEImage(open('/path/to/image', 'rb').read())

# Define the image's ID as referenced in the HTML body above
image.add_header('Content-ID', '<image1>')
msg.attach(image)

s = smtplib.SMTP('localhost')
s.sendmail(from_addr, to_addr, msg.as_string())
s.quit()

The trick is to define an image with a specific Content-ID and make that the only item in an HTML body: now you have an email with contains that specific image as the only content of the body, embedded in it.

Bonus point: if you want to take a snapshot of a webpage (which is kinda the reason i needed the code above) i found it extremely useful to use the Google PageSpeed Insights API; a good description on how to use that API with Python is available at this StackOverflow answer.

UPDATE (2020-12-26): I was made aware via email that some mail providers may not display images inline when the Content-ID value is too short (say, for example, Content-ID: 1). A solution that seems to work on most of the providers is using a sequence of random chars prefixed with a dot and suffixed with a valid mail domain.

Sandro Tosi: Python: send emails with embedded images

to send emails with images you need to use MIMEMultipart, but the basic approach:
import smtplib

from email.mime.multipart import MIMEMultipart
from email.mime.image import MIMEImage

msg = MIMEMultipart('alternative')
msg['Subject'] = "subject"
msg['From'] = from_addr
msg['To'] = to_addr

part = MIMEImage(open('/path/to/image', 'rb').read())

s = smtplib.SMTP('localhost')
s.sendmail(from_addr, to_addr, msg.as_string())
s.quit()

will produce an email with empty body and the image as an attachment.The better way, ie to have the image as part of the body of the email, requires to write an HTML body that refers to that image:
import smtplib

from email.mime.multipart import MIMEMultipart
from email.mime.text import MIMEText
from email.mime.image import MIMEImage

msg = MIMEMultipart('alternative')
msg['Subject'] = "subject
msg['From'] = from_addr
msg['To'] = to_addr

text = MIMEText('<img src="cid:image1">', 'html')
msg.attach(text)

image = MIMEImage(open('/path/to/image', 'rb').read())

# Define the image's ID as referenced in the HTML body above
image.add_header('Content-ID', '<image1>')
msg.attach(image)

s = smtplib.SMTP('localhost')
s.sendmail(from_addr, to_addr, msg.as_string())
s.quit()

The trick is to define an image with a specific Content-ID and make that the only item in an HTML body: now you have an email with contains that specific image as the only content of the body, embedded in it.

Bonus point: if you want to take a snapshot of a webpage (which is kinda the reason i needed the code above) i found it extremely useful to use the Google PageSpeed Insights API; a good description on how to use that API with Python is available at this StackOverflow answer.

UPDATE (2020-12-26): I was made aware via email that some mail providers may not display images inline when the Content-ID value is too short (say, for example, Content-ID: 1). A solution that seems to work on most of the providers is using a sequence of random chars prefixed with a dot and suffixed with a valid mail domain.

Russell Coker: USB Microphones

The Situation I bought myself some USB microphones over ebay, I couldn t see any with USB type A connectors (the original USB connectors) and bought ones with USB-C connectors. I thought it would be good to have microphones that could work with recent mobile phones and with PCs, because surely it wouldn t be difficult to get an adaptor. I tested one of the microphones, it worked well on a phone. I bought a pair of adaptors for USB A ports on a PC or laptop to USB-C (here s the link to where I bought them). I used one of the adaptors with a USB-C HDMI device which gave the following line from lsusb, I didn t try using a HDMI monitor on my laptop, having the device recognised was enough.
Bus 003 Device 002: ID 2109:0100 VIA Labs, Inc. USB 2.0 BILLBOARD
I tried connecting a USB-C microphone and Linux didn t recognise the existence of a USB device, I tried that on a PC and a laptop on multiple ports. I wondered whether the description of the VIA BILLBOARD device as USB 2.0 was relevant to my problem. According to Big Mess O Wires USB-C has separate wires for USB 3.1 and USB 2 [1]. So someone could make a device that converts USB-A to USB-C with only USB-2 wires in place. I tested the USB-A to USB-C adaptor with the HDMI device in a USB SuperSpeed (IE 3.x) port and it still identified as USB 2.0. I suspect that the USB-C HDMI device is using all the high speed wires for DisplayPort data (with a conversion to HDMI) and therefore looks like a USB 2.0 device. The Problem I want to install a microphone in my workstation for long Zoom training sessions (7 hours in a day) that otherwise require me to use multiple Android devices as I don t have a device that will do 7 hours of Zoom without running out of battery. A new workstation with USB-C is unreasonably expensive. A PCIe USB-C card would give me the port at the back of the machine, I can t have the back of the machine near the microphone because it s too noisy. If I could have a USB-C hub with reasonable length cables (the 1M cables typical for USB 2.0 hubs would be fine) connected to a USB-C port at the back of my workstation that would work. But there seems to be a great lack of USB-C hubs. NewBeDev has an informative post about the lack of USB-C hubs that have multiple USB-C ports [2]. There also seems to be a lack of USB-C hubs with cables longer than 20cm. The Solution I ended up ordering a Sades Wand gaming headset [3], that has over-ear headphones and an attached microphone which connects to the computer via USB 2.0. I gave the URL for the sades.com.au web site for reference but you will get a significantly better price by buying on ebay ($39+postage vs about $30 including postage). I guess I won t be using my new USB-C microphones for a while.

3 November 2021

Benjamin Mako Hill: Q&A about doing a PhD with my research group

Ever considered doing research about online communities, free culture/software, and peer production full time? It s PhD admission season and my research group the Community Data Science Collective is doing an open-to-anyone Q&A about PhD admissions this Friday November 5th. We ve got room in the session and its not too late to sign up to join us! The session will be a good opportunity to hear from and talk to faculty recruiting students to our various programs at the University of Washington, Purdue, and Northwestern and to talk with current and previous students in the group. I am hoping to admit at least one new PhD advisee to the Department of Communication at UW this year (maybe more) and am currently co-advising (and/or have previously co-advised) students in UW s Allen School of Computer Science & Engineering, Department of Human-Centered Design & Engineering, and Information School. One thing to keep in mind is that my primary/home department Communication has a deadline for PhD applications of November 15th this year. The registration deadline for the Q&A session is listed as today but we ll do what we can to sneak you in even if you register late. That said, please do register ASAP so we can get you the link to the session!

Dirk Eddelbuettel: RDieHarder 0.2.2 on CRAN: Simpler Build, Fixes

An updated version 0.2.2 of the random-number generator tester RDieHarder (based on the DieHarder suite developed / maintained by Robert Brown with contributions by David Bauer and myself) is now on CRAN. I should dub this the due to Brian Ripley release. He sent me a detailed five-point email a few days ago which detailed a change I could not have tested ( no access ), a change I would not have known ( somewhat obscure C language bit-level manipulation ), a change I had missed (how my build setup failed for M1mac), another advanced C level fix, and one more simple fix I actually knew. Speechless. The man (I presume) does not sleep and is just so generous with his time and expertise. So based on the input I rejigged the package over the weekend and made two more (substantial) changes. First, extending on what 0.2.0 brought, I will no longer attempt to use an external libdieharder library (or build one on the fly) that was issue one. Now we just declare all C files as dependents of the package shared library, and things are simpler and more consistent. Sadly, that also implies everything is in the package so I had to edit out a metric ton of stdout or exit() reference with the appropriate R C API hooks to appease the CRAN Policy deities. Win some, loose some. But the package is now simpler, and cleaner, and should be in good standing. (Or so one hopes. Earlier today, and within hours of it hitting CRAN, I got an issue ticket from a motivated user about yet another ( mostly harmless in the Douglas Adams sense) compiler warning Good now too.) Thanks to CRANberries, you can also look at the most recent diff. If you like this or other open-source work I do, you can now sponsor me at GitHub.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

2 November 2021

Ben Hutchings: Debian LTS work, October 2021

In October I was assigned 1.25 hours of work by Freexian's Debian LTS initiative and carried over 28.75 hours from earlier months. I worked 14.75 hours and will carry over the remainder. I updated the linux-4.19 package to merge in the changes in buster release 10.11, uploaded that and issued DLA-2785-1. I continued work on an update to the linux (4.9) package, but did not make an upload this month.

Enrico Zini: help2man and subcommands

help2man is quite nice for autogenerating manpages from command line help, making sure that they stay up to date as command line options evolve. It works quite well, except for commands with subcommands, like Python programs that use argparse's add_subparser. So, here's a quick hack that calls help2man for each subcommand, and stitches everything together in a simple manpage.
#!/usr/bin/python3
import re
import shutil
import sys
import subprocess
import tempfile
# TODO: move to argparse
command = sys.argv[1]
# Use setup.py to get the program version
res = subprocess.run([sys.executable, "setup.py", "--version"], stdout=subprocess.PIPE, text=True, check=True)
version = res.stdout.strip()
# Call the main commandline help to get a list of subcommands
res = subprocess.run([sys.executable, command, "--help"], stdout=subprocess.PIPE, text=True, check=True)
subcommands = re.sub(r'^.+\ (.+)\ .+$', r'\1', res.stdout, flags=re.DOTALL).split(',')
# Generate a help2man --include file with an extra section for each subcommand
with tempfile.NamedTemporaryFile("wt") as tf:
    print("[>DESCRIPTION]", file=tf)
    for subcommand in subcommands:
        res = subprocess.run(
                ["help2man", f"--name= command ", "--section=1",
                 "--no-info", "--version-string=dummy", f"./ command   subcommand "],
                stdout=subprocess.PIPE, text=True, check=True)
        subcommand_doc = re.sub(r'^.+.SH DESCRIPTION', '', res.stdout, flags=re.DOTALL)
        print(".SH ", subcommand.upper(), " SUBCOMMAND", file=tf)
        tf.write(subcommand_doc)
    with open(f" command .1.in", "rt") as fd:
        shutil.copyfileobj(fd, tf)
    tf.flush()
    # Call help2man on the main command line help, with the extra include file
    # we just generated
    subprocess.run(
            ["help2man", f"--include= tf.name ", f"--name= command ",
             "--section=1", "--no-info", f"--version-string= version ",
             "--output=arkimaps.1", "./arkimaps"],
            check=True)

1 November 2021

Russell Coker: Talking to Criminals

I think most people and everyone who reads my blog is familiar with the phone support scams that are common nowadays. There s the we are Microsoft support and have found a problem with your PC , the we are from your ISP and want to warn you that your Internet access will be cut off , and the here s the bill for something expensive and we need you to confirm whether you want to pay . Most people hang up when scammers call them and don t call them back. But I like to talk to them. I review the quality of their criminal enterprise and tell them that I expect better quality criminals to call me. I ask them if they are proud to be criminals and if their parents would be proud of them. I ask them if they are paid well to be a criminal. Usually they just hang up and on one occasion the criminal told me to get lost before hanging up. Today I got a spam message telling me to phone +61-2-8006-7237 about an invoice for Norton Software Enhancer and Firewall Defender if I wanted to dispute it. It was interesting that they had an invoice number in the email which they asked me for when I called, at the time I didn t think to make up an invoice number with the same format to determine if they were actually looking it up, in retrospect I should have used a random 9 digit number to determine if they had a database for this. On the first call they just hung up on me. The second call they told me you won t save anyone before hanging up. The third call I got on to a friendly and talkative guy who told me that he was making good money being a criminal. I asked if he was in India or Australia (both guys had accents from the Indian subcontinent), he said he was in Pakistan. He said that he made good money by Pakistani standards as $1 Australian is over 100 Pakistani Rupees. He asked me if I d like to work for him, I said that I make good money doing legal things, he said that if I have so much money I could send him some. ;) He also offered to take me on a tour of Islamabad if I visited, this could have been a genuine offer to have a friendly meeting with someone from the opposite site of computer security or an attempt at kidnap for ransom. He didn t address my question about whether the local authorities would be interested in his work, presumably he thinks that a combination of local authorities not caring much and the difficulty of tracking international crime makes him safe. It was an interesting conversation, I encourage everyone to chat to such criminals. They are right that you won t save anyone. But you can have some fun and occasionally learn some interesting things.

Andreas R nnquist: Debian packages, version numbers and pre-release versions

Getting the latest version of a package into Debian involves checking when there are new versions available fortunately (and not surprisingly) Debian has tools to make this simpler. I have recently ran into the problem when an upstream beta version sorts higher than a newer non-beta version. Which is problematic, of course. This is due to Debian sorting something like 1.0b as later than a pure 1.0 version.
gusnan@debian-i7:~ > dpkg --compare-versions 1.0b lt 1.0 && echo true
gusnan@debian-i7:~ > dpkg --compare-versions 1.0b gt 1.0 && echo true
true
But there s a solution name the beta versions something like 1.0~beta. And you don t need to force upstream to make any changes either. You can use uscan and the watch file to make it interpret an upstream 1.0b version as 1.0~beta in Debian. This is done by using a line like
uversionmangle=s/(\d)[\_\.\-\+]?((RC rc pre dev beta alpha b a)\d*)$/$1~$2/;s/\~b/\~beta/;,\
in uversionmangle in your debian/watch file. In this case i have added on the end something to make the ending ~b into ~beta instead. Full version of the watch file available here.

Thorsten Alteholz: My Debian Activities in October 2021

FTP master This month I accepted 341 and rejected 46 packages. The rejection is as high as last month. I hope everybody is aware that pressing just one key when accepting a package is much faster than writing an explanation why a package has to be rejected. Anyway, the overall number of packages that got accepted was 355. Debian LTS This was my eighty-eighth month that I did some work for the Debian LTS initiative, started by Raphael Hertzog at Freexian. This month my all in all workload has been 28.5h. During that time I did LTS and normal security uploads of: I also continued to work on exiv2. Last but not least I did some days of frontdesk duties. Debian ELTS This month was the fortieth ELTS month. During my allocated time I uploaded: Last but not least I did some days of frontdesk duties. Debian Printing I improved packaging or fixed bugs or uploaded a new version of: Last but not least I looked at some old bugs and checked whether they could be closed. Debian Astro Though being a silent member of Debian Astro for a long time, I am now going to be more active now. Most of the time I will be focused on packages for telescope control, but of course I won t stay away from other topics. So I uploaded: If you know of other missing packages, don t hesitate to tell me! Other stuff On my neverending golang challenge I again uploaded some packages either for NEW or as source upload. I uploaded new upstream versions of: I improved packaging or fixed bugs of:

Next.

Previous.