Enrico Zini: html5-sse
HTML5 Server-sent events
I have a Django view that runs a slow script server-side, and streams the
script output to Javascript. This is the bit of code that runs the script and
turns the output into a stream of events:
I used to just serialize its output and stream it to JavaScript, then monitor
And this is the template that renders it:
It's simple enough, it seems reasonably well
supported
besides needing a polyfill for IE
and, astonishingly, it even works!
def stream_output(proc):
'''
Take a subprocess.Popen object and generate its output, line by line,
annotated with "stdout" or "stderr". At process termination it generates
one last element: ("result", return_code) with the return code of the
process.
'''
fds = [proc.stdout, proc.stderr]
bufs = [b"", b""]
types = ["stdout", "stderr"]
# Set both pipes as non-blocking
for fd in fds:
fcntl.fcntl(fd, fcntl.F_SETFL, os.O_NONBLOCK)
# Multiplex stdout and stderr with different prefixes
while len(fds) > 0:
s = select.select(fds, (), ())
for fd in s[0]:
idx = fds.index(fd)
buf = fd.read()
if len(buf) == 0:
fds.pop(idx)
if len(bufs[idx]) != 0:
yield types[idx], bufs.pop(idx)
types.pop(idx)
else:
bufs[idx] += buf
lines = bufs[idx].split(b"\n")
bufs[idx] = lines.pop()
for l in lines:
yield types[idx], l
res = proc.wait()
yield "result", res
onreadystatechange on the XMLHttpRequest object browser-side, but then it
started failing on Chrome, which won't trigger onreadystatechange until
something like a kilobyte of data has been received.
I didn't want to stream a kilobyte of padding just to work-around this, so it
was time to try out Server-sent events.
See also this.
This is the Django view that sends the events:
class HookRun(View):
def get(self, request):
proc = run_script(request)
def make_events():
for evtype, data in utils.stream_output(proc):
if evtype == "result":
yield "event: \ndata: \n\n".format(evtype, data)
else:
yield "event: \ndata: \n\n".format(evtype, data.decode("utf-8", "replace"))
return http.StreamingHttpResponse(make_events(), content_type='text/event-stream')
@method_decorator(never_cache)
def dispatch(self, *args, **kwargs):
return super().dispatch(*args, **kwargs)
% extends "base.html" %
% load i18n %
% block head_resources %
block.super
<style type="text/css">
.out
font-family: monospace;
padding: 0;
margin: 0;
.stdout
.stderr color: red;
.result
.ok color: green;
.ko color: red;
</style>
# Polyfill for IE, typical... https://github.com/remy/polyfills/blob/master/EventSource.js #
<script src=" STATIC_URL js/EventSource.js"></script>
<script type="text/javascript">
$(function()
// Manage spinners and other ajax-related feedback
$(document).nav();
$(document).nav("ajax_start");
var out = $("#output");
var event_source = new EventSource(" % url 'session_hookrun' name=name % ");
event_source.addEventListener("open", function(e)
//console.log("EventSource open:", arguments);
);
event_source.addEventListener("stdout", function(e)
out.append($("<p>").attr("class", "out stdout").text(e.data));
);
event_source.addEventListener("stderr", function(e)
out.append($("<p>").attr("class", "out stderr").text(e.data));
);
event_source.addEventListener("result", function(e)
if (+e.data == 0)
out.append($("<p>").attr("class", "result ok").text(" % trans 'Success' % "));
else
out.append($("<p>").attr("class", "result ko").text(" % trans 'Script failed with code' % " + e.data));
event_source.close();
$(document).nav("ajax_end");
);
event_source.addEventListener("error", function(e)
// There is an annoyance here: e does not contain any kind of error
// message.
out.append($("<p>").attr("class", "result ko").text(" % trans 'Error receiving script output from the server' % "));
console.error("EventSource error:", arguments);
event_source.close();
$(document).nav("ajax_end");
);
);
</script>
% endblock %
% block content %
<h1> % trans "Processing..." % </h1>
<div id="output">
</div>
% endblock %
Present and Compositors
The current Present extension is pretty unfriendly to compositing
managers, causing an extra frame of latency between the applications
operation and the scanout buffer. Here's how I'm fixing that.
An extra frame of lag
When an application uses PresentPixmap, that operation is generally
delayed until the next vblank interval. When using X without
composting, this ensures that the operation will get started in the
vblank interval, and, if the rendering operation is quick enough,
you'll get the frame presented without any tearing.
When using a compositing manager, the operation is still delayed until
the vblank interval. That means that the CopyArea and subsequent
Damage event generation don't occur until the display has already
started the next frame. The compositing manager receives the damage
event and constructs a new frame, but it also wants to avoid tearing,
so that frame won't get displayed immediately, instead it'll get
delayed until the next frame, introducing the lag.
Copy now, complete later
While away from the keyboard this morning, I had a sudden idea -- what
if we performed the CopyArea and generated Damage right when the
PresentPixmap request was executed but delayed the PresentComplete
event until vblank happened.
With the contents updated and damage delivered, the compositing
manager can immediately start constructing a new scene for the
upcoming frame. When that is complete, it can also use PresentPixmap
(either directly or through OpenGL) to queue the screen update.
If it's fast enough, that will all happen before vblank and the
application contents will actually appear at the desired time.
Now, at the appointed vblank time, the PresentComplete event will get
delivered to the client, telling it that the operation has finished
and that its contents are now on the screen. If the compositing
manager was quick, this event won't even be a lie.
We'll be lying less often
Right now, the CopyArea, Damage and PresentComplete operations all
happen after the vblank has passed. As the compositing manager delays
the screen update until the next vblank, then every single
PresentComplete event will have the wrong UST/MSC values in it.
With the CopyArea happening immediately, we've a pretty good chance
that the compositing manager will get the application contents up on
the screen at the target time. When this happens, the PresentComplete
event will have the correct values in it.
How can we do better?
The only way to do better is to have the PresentComplete event
generated when the compositing manager displays the frame. I've talked
about how that should work, but it's a bit twisty, and will require
changes in the compositing manager to report the association between
their PresentPixmap request and the applications' PresentPixmap
requests.
Where's the code
I've got a set of three patches, two of which restructure the existing
code without changing any behavior and a final patch which adds this
improvement. Comments and review are encouraged, as always!
A surprise to see my box booting up with the default GRUB 2.x menu, followed by cannot find a working init .
What happened?
Well, grub:i386 and grub:x32 are distinct packages, so APT helpfully decided to purge the GRUB config. OK. Manual boot menu entry editing later, re-adding GRUB_DISABLE_SUBMENU=y and GRUB_CMDLINE_LINUX= syscall.x32=y to /etc/default/grub, removing quiet again from GRUB_CMDLINE_LINUX_DEFAULT, and uncommenting GRUB_TERMINAL=console and don t forget to sudo update-grub . There. This should work.
On the plus side, nvidia-driver:i386 seems to work but not with boinc-client:x32 (why, again? I swear, its GPU detection has been driving me nuts on > of all systems I installed it on, already!).
On the minus side, I now have to figure out why
tglase@tglase:~ $ sudo ifup -v tap1
Inspired by "
Like 
On this day last year 
Instead of a calculator, I tend to use 
Munin is a great tool. If you can script it, you can monitor it with
munin. Unfortunately, however, munin is slow; that is, it will take
snapshots once every five minutes, and not look at systems in between.
If you have a short load spike that takes just a few seconds, chances
are pretty high munin missed it. It also comes with a great
webinterfacefrontendthing that allows you to dig deep in the history of
what you've been monitoring.
By the time munin tells you that your Kerberos KDCs are all down, you've
probably had each of your users call you several times to tell you that
they can't log in. You could use nagios or one of its brethren, but it
takes about a minute before such tools will notice these things, too.
Maybe use CollectD then? Rather than check once every several minutes,
CollectD will collect information every few seconds. Unfortunately,
however, due to the performance requirements to accomplish that (without
causing undue server load), writing scripts for CollectD is not as easy
as it is for Munin. In addition, webinterfacefrontendthings aren't
really part of the CollectD code (there are several, but most that I've
looked at are lacking in some respect), so usually if you're using
CollectD, you're missing out some.
And collectd doesn't do the nagios thing of actually telling you when
things go down.
So what if you could see it when things go bad?
At one customer, I came in contact with 
Streamable archive formats
I was recently asked about some construct in a shell script starting with a colon(:), leading me into a long monologue about it. Afterwards I realized I had forgotten to mention half of the nice things. So here for your amusement some usage for the colon in the shell:
To find the meaning of ":" in the bash manpage[1], you have to look at the start of the SHELL BUILTIN COMMANDS section. There you find:
Zoe's Kindergarten encourages parents to come in and spend some time with the
kids. I've heard reports of other parents coming in and doing baking with the
kids or other activities at various times throughout the year.
Zoe and I had both wanted me to come in for something, but it had taken me
until the last few weeks of the year to get my act together and do something.
I'd thought about coming in and doing some baking, but that seemed rather done
to death already, and it's not like baking is really my thing, so I thought I'd
do something technological. I just wracked my brains for something low effort
and Kindergarten-age friendly.
The Kindergarten has a couple of 
Rhoda is guarded. She is secretly in love with her brother. When
he gets a girlfriend, she finds a boyfriend. She tells no one what
she's really thinking.
Rhoda likes to seize the day. In the midst of evening conversation,
she will excuse herself to use the bathroom or come right back ,
then, within literally two minutes, she will go home with an
acquaintance or stranger. Often these encounters or the aftermaths
thereof do not go according to her liking, and she will make veiled
remarks of hostility, should she see those people again.
Rhoda was unhappy so she changed everything in her life. Though
multiple things remained constant, she concluded that she was, in
fact, the problem.
Rhoda does not make enough money on which to live. She works
part-time, and turns down all other job offers. Other people
make up for her financial shortcomings so that she is not
homeless and starving.
Rhoda is certain that it is more difficult to be female than male.
I do hang out in #debian-women on IRC, which shouldn't be much of a surprise after my last blog entry about my 