Enrico Zini: Building a Transilience playbook in a zipapp
This is part of a series of posts on ideas for an ansible-like provisioning
system, implemented in Transilience.
Mitogen is a great library, but scarily
complicated, and I've been wondering how hard it would be to make alternative
connection methods for Transilience.
Here's a wild idea: can I package a whole Transilience playbook, plus
dependencies, in a zipapp,
then send the zipapp to the machine to be provisioned, and run it locally?
It turns out I can.
Creating the zipapp
This is somewhat hackish, but until I can rely on Python 3.9's improved
Since the zipapp contains not just the playbook, the roles, and the roles'
assets, but also Transilience and Jinja2, it can run on any system that has a
Python 3.7+ interpreter, and nothing else!
I added it to the standard set of playbook command line options, so any
Transilience playbook can turn itself into a self-contained zipapp:
Loading assets from the zipapp
I had to create ZipFile varieties of some bits of infrastructure in
Transilience, to load templates, files, and Ansible yaml files from zip files.
You can see above a way to detect if a module is loaded from a zipfile: check
if the module's
I also created a
An interesting side effect of having smarter local file accessors is that I can
cache the contents of small files and transmit them to the remote host together
with the other action parameters, saving a potential network round trip for
each
And on the remote:
Compare with a Mitogen run:
From a single test run, not a good benchmark, it's
importlib.resources
module, I cannot think of a better way:
def zipapp(self, target: str, interpreter=None): """ Bundle this playbook into a self-contained zipapp """ import zipapp import jinja2 import transilience if interpreter is None: interpreter = sys.executable if getattr(transilience.__loader__, "archive", None): # Recursively iterating module directories requires Python 3.9+ raise NotImplementedError("Cannot currently create a zipapp from a zipapp") with tempfile.TemporaryDirectory() as workdir: # Copy transilience shutil.copytree(os.path.dirname(__file__), os.path.join(workdir, "transilience")) # Copy jinja2 shutil.copytree(os.path.dirname(jinja2.__file__), os.path.join(workdir, "jinja2")) # Copy argv[0] as __main__.py shutil.copy(sys.argv[0], os.path.join(workdir, "__main__.py")) # Copy argv[0]/roles role_dir = os.path.join(os.path.dirname(sys.argv[0]), "roles") if os.path.isdir(role_dir): shutil.copytree(role_dir, os.path.join(workdir, "roles")) # Turn everything into a zipapp zipapp.create_archive(workdir, target, interpreter=interpreter, compressed=True)
$ ./provision --help usage: provision [-h] [-v] [--debug] [-C] [--local LOCAL] [--ansible-to-python role --ansible-to-ast role --zipapp file.pyz] [...] --zipapp file.pyz bundle this playbook in a self-contained executable python zipapp
__loader__ attribute has an archive attribute.
Here's a Jinja2 template loader that looks into a zip:
class ZipLoader(jinja2.BaseLoader): def __init__(self, archive: zipfile.ZipFile, root: str): self.zipfile = archive self.root = root def get_source(self, environment: jinja2.Environment, template: str): path = os.path.join(self.root, template) with self.zipfile.open(path, "r") as fd: source = fd.read().decode() return source, None, lambda: True
FileAsset
abstract interface to represent a local file, and had Role.lookup_file return
an appropriate instance:
def lookup_file(self, path: str) -> str: """ Resolve a pathname inside the place where the role assets are stored. Returns a pathname to the file """ if self.role_assets_zipfile is not None: return ZipFileAsset(self.role_assets_zipfile, os.path.join(self.role_assets_root, path)) else: return LocalFileAsset(os.path.join(self.role_assets_root, path))
builtin.copy action that has a small source.
The result
The result is kind of fun:
$ time ./provision --zipapp test.pyz real 0m0.203s user 0m0.174s sys 0m0.029s $ time scp test.pyz root@test: test.pyz 100% 528KB 388.9KB/s 00:01 real 0m1.576s user 0m0.010s sys 0m0.007s
# time ./test.pyz --local=test 2021-06-29 18:05:41,546 test: [connected 0.000s] [...] 2021-06-29 18:12:31,555 test: 88 total actions in 0.00ms: 87 unchanged, 0 changed, 1 skipped, 0 failed, 0 not executed. real 0m0.979s user 0m0.783s sys 0m0.172s
$ time PYTHONPATH=../transilience/ ./provision 2021-06-29 18:13:44 test: [connected 0.427s] [...] 2021-06-29 18:13:46 test: 88 total actions in 2.50s: 87 unchanged, 0 changed, 1 skipped, 0 failed, 0 not executed. real 0m2.697s user 0m0.856s sys 0m0.042s
0.203 + 1.576 + 0.979 =
2.758s with the zipapp and 2.697s with Mitogen. Even if I've been lucky,
it's a similar order of magnitude.
What can I use this for?
This was mostly a fun hack.
It could however be the basis for a Fabric-based
connector, or a clusterssh-based
connector, or for bundling a Transilience playbook into an installation image,
or to add a provisioning script to the boot partition of a Raspberry Pi. It
looks like an interesting trick to have up one's sleeve.
One could even build an Ansible-based connector(!) in which a simple Ansible
playbook, with no facts gathering, is used to build the zipapp, push it to
remote systems and run it. That would be the wackiest way of speeding up
Ansible, ever!
Next: using Systemd containers with unittest,
for Transilience's test suite.
In the last few months I presented several talks. Topics ranged from a round table on free
software, to sharing some of my work as SRE in the Cloud Services team at the Wikimedia Foundation.
For some of them the videos are now published, so I would like to write a reference here, mostly as
a way to collect such events for my own record. Isn t that what
I had been meaning to write on the above topic for almost a couple of months now but just kept procrastinating about it. That push came to a shove when Sucheta Dalal and Debasis Basu shared their understanding, wisdom, and all in the new book called
The psyche of the Bengali and the Bhadralok has gone through enormous shifts. I have met quite a few and do see the guilt they feel. If one wonders as to how socialist parties are able to hold power in Bengal, look no further than
I've been using
I even tried the same with xvkbd but no avail. I do use mate as my desktop-manager so maybe the instructions need some refinement ????
$ cat /etc/lightdm/lightdm-gtk-greeter.conf grep keyboard
If you re writing a lot of Common Lisp and you want to follow the convention
of using all uppercase to refer to symbols in docstrings, comments etc., you
really need something better than the shift key. Similarly if you re writing
C and you have
"Again, we see there is nothing you can possess which I cannot take away."
The imagery of the ark appears to be borrowed from James Tissot's The Ark Passes Over the Jordan, part of the fin de siecle fascination with the occult and (ironically enough given the background of Raiders' director), a French Catholic revival.

Every couple of months, one of my users falls prey to phishing
attacks, and send their login/password data to an unknown somebody
who poses as Well, as me, their always-friendly and always-helpful
systems administrator.
What follows is, of course, me spending a week trying to get our
systems out of all of the RBLs/DNSBLs. But, no matter how fast I act,
there s always distruption and lost mails (bounced or classified as
spam) for my users.
Most of my users use the Webmail I have configured on our institute s
servers,
