The problem I had - Mason, so, sadly, FastCGI
Since the update to current Debian stable, the website for
YARRG, (a play-aid for Puzzle Pirates which I wrote some years ago), started to occasionally return Internal Server Error , apparently due to
bug(s) in some FastCGI libraries.
I was using FastCGI because the website is written in
Mason, a Perl web framework, and I found that Mason CGI calls were slow. I m using CGI - yes, trad CGI - via userv-cgi. Running Mason this way would compile the template for each HTTP request just when it was rendered, and then throw the compiled version away. The more modern approach of an application server doesn t scale well to a system which has many web applications most of which are very small. The admin overhead of maintaining a daemon, and corresponding webserver config, for each such service would be prohibitive, even with some kind of autoprovisioning setup. FastCGI has an interpreter wrapper which seemed like it ought to solve this problem, but it s quite inconvenient, and often flaky.
I decided I could do better, and set out to eliminate FastCGI from my setup. The result seems to be a success; once I d done all the hard work of writing
prefork-interp, I found the result very straightforward to deploy.
prefork-interp
prefork-interp is a small C program which wraps a script, plus a scripting language library to cooperate with the wrapper program. Together they achieve the following:
- Startup cost of the script (loading modules it uses, precompuations, loading and processing of data files, etc.) is paid once, and reused for subsequent invocations of the same script.
- Minimal intervention to the script source code:
- one new library to import
- one new call to make from that library, right after the script intialisation is complete
- change to the
#! line.
- The new initialisation complete call turns the program into a little server (a daemon), and then returns once for each actual invocation, each time in a fresh grandchild process.
Features:
- Seamless reloading on changes to the script source code (automatic, and configurable).
- Concurrency limiting.
- Options for distinguishing different configurations of the same script so that they get a server each.
- You can run the same script standalone, as a one-off execution, as well as under
prefork-interp.
- Currently, a script-side library is provided for Perl. I m pretty sure Python would be fairly straightforward.
Important properties not always satisfied by competing approaches:
- Error output (stderr) and exit status from both phases of the script code execution faithfully reproduced to the calling context. Environment, arguments, and stdin/stdout/stderr descriptors, passed through to each invocation.
- No polling, other than a long-term idle timeout, so good on laptops (or phones).
- Automatic lifetime management of the per-script server, including startup and cleanup. No integration needed with system startup machinery: No explicit management of daemons, init scripts, systemd units, cron jobs, etc.
- Useable right away without fuss for CGI programs but also for other kinds of program invocation.
- (I believe) reliable handling of unusual states arising from failed invocations or races.
Swans paddling furiously
The implementation is much more complicated than the (apparent) interface.
I won t go into all the details here (there are some
terrifying diagrams in the source code if you really want), but some highlights:
We use an
AF_UNIX socket (hopefully in
/run/user/UID, but in
~ if not) for rendezvous. We can try to connect without locking, but we must protect the socket with a separate lockfile to avoid two concurrent restart attempts.
We want stderr from the script setup (pre-initialisation) to be delivered to the caller, so the script ought to inherit our stderr and then will need to replace it later. Twice, in fact, because the daemonic server process can t have a stderr.
When a script is restarted for any reason, any old socket will be removed. We want the old server process to detect that and quit. (If hung about, it would wait for the idle timeout; if this happened a lot - eg, a constantly changing set of services - we might end up running out of pids or something.) Spotting the socket disappearing, without polling, involves use of a library capable of using
inotify (or the equivalent elsewhere). Choosing a C library to do this is not so hard, but portable interfaces to this functionality can be hard to find in scripting languages, and also we don t want every language binding to have to reimplement these checks. So for this purpose there s a little watcher process, and associated IPC.
When an invoking instance of
prefork-interp is killed, we must arrange for the executing service instance to stop reading from its stdin (and, ideally, writing its stdout). Otherwise it s stealing input from
prefork-interp s successors (maybe the user s shell)!
Cleanup ought not to depend on positive actions by failing processes, so each element of the system has to detect failures of its peers by means such as EOF on sockets/pipes.
Obtaining prefork-interp
I put this new tool in my chiark-utils package, which is a collection of useful miscellany. It s available
from git.
Currently I make releases by
uploading to Debian, where prefork-interp has just hit Debian unstable, in chiark-utils 7.0.0.
Support for other scripting languages
I would love Python to be supported. If any pythonistas reading this think you might like to help out, please get in touch. The specification for the protocol, and what the script library needs to do, is
documented in the source code
Future plans for chiark-utils
chiark-utils as a whole is in need of some tidying up of its build system and packaging.
I intend to try to do some reorganisation. Currently I think it would be better to organising the source tree more strictly with a directory for each included facility, rather than grouping compiled and scripts together.
The Debian binary packages should be reorganised more fully according to their dependencies, so that installing a program will ensure that it works.
I should probably move the official git repo from my own git+gitweb to a forge (so we can have MRs and issues and so on).
And there should be a lot more testing, including Debian autopkgtests.
edited 2022-08-23 10:30 +01:00 to improve the formatting
comments
It's time for another update on Debian's firmware GR. I wrote about
the
problem back
in April and about the vote
itself a
few days back.
Voting closed last night and we have
a result!
This is unofficial so far - the official result will
follow shortly when the Project Secretary sends a signed mail to
confirm it. But that's normally just a formality at this point.
A Result!
The headline result is: Choice 5 / Proposal E
won: Change
SC for non-free firmware in installer, one installer. I'm
happy with this - it's the option that I voted highest, after
all. More importantly, however, it's a clear choice
of direction, as I was hoping for. Of the 366 Debian Developers who
voted, 289 of them voted this option above NOTA and 63 below, so it
also meets the 3:1 super-majority requirement for amending a Debian
Foundation Document (Constitution
section 4.1.3).
So, what happens next?
We have quite a few things to do now, ideally before the freeze for
Debian 12 (bookworm),
due January
2023. This list of work items is almost
definitely not complete, and Cyril and I are aiming
to get together this week and do more detailed planning for the d-i
pieces. Off the top of my head I can think of the following:
This post describes how I ve put together a simple static content server for
kubernetes clusters using a Pod with a persistent volume and multiple
containers: an sftp server to manage contents, a web server to publish them
with optional access control and another one to run scripts which need access
to the volume filesystem.
The sftp server runs using
(The picture is of the previous edition.)
Almost two years after the previous Norwegian Bokm l translation of
the "
The seventeenth release of 