One of the loveliest things about Unix is the select() function (or
its replacement, poll()), and the way it lets a single thread handle a
host of concurrent tasks efficiently by just using file descriptors as
work queues.
Unfortunately, it can be a nuisance to use – you end up having to
structure your program as a state machine around the select() invocation,
rather than the actual procedure you want to have happen. You can avoid that
by not using select() and instead just having a separate thread/process for
every task you want to do – but that creates a bunch of tedious overhead
for the OS (and admin) to worry about.
But magically making state machines is what Python’s generators are
all about; so for my little pet project that involves forking a bunch
of subprocesses to do the interesting computational work my python program
wants done, I thought I’d see if I could use that to make my code more
obvious.
What I want to achieve is to have a bunch of subprocesses accepting
some setup data, then a bunch of two byte ids, terminated by two bytes
of 0xFF, and for each of the two byte inputs to output a line of text
giving the calculation result. For the time being at least, I want the
IO to be asynchronous: so I’ll give it as many inputs as I can, rather
than waiting for the result before sending the next input.
So basically, I want to write something like:
def send_inputs(f, s, n):
f.write(s) # write setup data
for i in xrange(n):
f.write(struct.pack("!H", i))
f.write(struct.pack("!H", 0xFFFF))
def read_output(f):
for line in f:
if is_interesting(line):
print line
Except of course, that doesn’t work directly because writing some data
or reading a line can block, and when it does, I want it to be doing
something else (reading instead of writing or vice-versa, or paying
attention to another process).
Generators are the way to do that in Python, with the “yield” keyword
passing control flow and some information back somewhere else, so adopting
the theory that: (a) I’ll only resume from a “yield” when it’s okay to
write some more data, (b) if I “yield None” there’s probably no point
coming back to me unless you’ve got some more data for me to read, and
(c) I’ll provide a single parameter which is an iterator that will give me
input when it’s available and None when it’s not, I can code the above as:
def send_inputs(_):
# s, n declared in enclosing scope
yield s
for i in xrange(n):
yield struct.pack("!H", i))
yield struct.pack("!H", 0xFFFF)
def read_output(f):
for line in f:
if line is None: yield None; continue
if is_interesting(line):
print line
There’s a few complications there. For one, I could be yielding more
data than can actually be written, so I might want to buffer there to
avoid blocking. (I haven’t bothered; just as I haven’t worried about
“print” possibly blocking) Likewise, I might only receive part of a line,
or I might receive more than one line at once, and afaics a buffer there
is unavoidable. If I were doing fixed size reads (instead of line at a
time), that might be different.
So far, the above seems pretty pleasant to me – those functions
describe what I want to have happen in a nice procedural manner (almost
as if they had a thread all to themselves) with the only extra bit the
“None, None, continue” line, which I’m willing to accept in order not to
use threads.
Making that actually function does need a little grunging around, but
happily we can hide that away in a module – so my API looks like:
p = subprocess.Popen(["./helper"], stdin=PIPE, stdout=PIPE, close_fds=True)
comm = communicate.Communication()
comm.add(send_inputs, p.stdin, None)
comm.add(read_output, None, p.stdout, communicate.ByLine())
comm.communicate()
The comm.add() function takes a generator function, an output fd
(ie, the subprocess’s stdin), an input fd (the subprocess’s output),
and an (optional) iterator. The generator gets created when communication
starts, with the iterator passed as the argument. The iterator needs to
have an “add” function (which gets given the bytes received), a “waiting”
function, which returns True or False depending on whether it can provide
any more input for the generator, and a “finish” function that gets called
once EOF is hit on the input. (Actually, it doesn’t strictly need to be
an iterator, though it’s convenient for the generator if it is)
The generator functions once “executed” return an object with a next()
method that’ll run the function you defined until the next “yield”
(in which case next() will return the value yielded), or a “return”
is hit (in which case the StopIteration exception is raised).
So what we then want to do to have this all work then, is this: (a)
do a select() on all the files we’ve been given; (b) for the ones we
can read from, read them and add() to the corresponding iterators; (c)
for the generators that don’t have an output file, or whose output file
we can write to, invoke next() until either: they raise StopIteration,
they yield a value for us to output, or they yield None and their iterator
reports that it’s waiting. Add in some code to ensure that reads from the
file descriptors don’t block, and you get:
def communicate(self):
readable, writable = [], []
for g,o,i,iter in self.coroutines:
if i is not None:
fcntl.fcntl(i, fcntl.F_SETFL,
fcntl.fcntl(i, fcntl.F_GETFL) os.O_NONBLOCK)
readable.append(i)
if o is not None:
writable.append(o)
while readable != [] or writable != []:
read, write, exc = select.select(readable, writable, [])
for g,o,i,iter in self.coroutines:
if i in read:
x = i.read()
if x == "": # eof
iter.finish()
readable.remove(i)
else:
iter.add(x)
if o is None or o in write:
x = None
try:
while x is None and not iter.waiting():
x = g.next()
if x is not None:
o.write(x)
except StopIteration:
if o is not None:
writable.remove(o)
return
You can break it by: (a) yielding more than you can write without
blocking (it’ll block rather than buffer, and you might get a deadlock),
(b) yielding a value from a generator that doesn’t have a file associated
with it (None.write(x) won’t work), (c) having generators that don’t
actually yield, and (d) probably some other ways. And it would’ve been
nice if I could have somehow moved the “yield None” into the iterator
so that it was implicit in the “for line in f”, rather than explicit.
But even so, I quite like it.
I'm currently part way through testing Chronicle and I'm probably going to change the address of my blog (with Redirects) as part of that. I don't much like having blog in the address and some research seems to agree with me. I've lived with this mistake for four years now, through various software upgrades, so I'll take the chance to change. But what should I change the path to? /writing/reflections/ /news/ /notes/ /articles/ or something else? Vote now by leaving a comment! Oh and about the debian project leader vote: reluctant candidates are good, teams are good, people who already have had high office are less good. Go vote on that too if you can.
Where, oh, where do I start with this one??
To take part in the 
So calling me Mika is perfectly fine because it’s a kind of uniqueness for me. There might be just too many Michaels around and at least in Austria my nickname is everything but common. Quoting the marketing department: Great trademark! 
The new release from the 
The default EEE PC setup consists of a (read-only) system partition and a user partition overlay. It should be simple to download the 16 Mb D-I boot image, save it to the system partition and write a boot loader that would allow the choice between normal boot, EEE PC rescue boot and Debian installation boot. With a bit of pre-seeding we could have the d-i install a Debian instance in place of the user partition unless the user overrides that.
People who have little knowledge tend to think that they know more than they do, while others who have much more knowledge tend to think that they know less. That’s what 
Installing Debian on your Eee PC is even easier now, as the option to 
I thought the spanish region of Extremadura had 4 million inhabitants and 200000 computers running GNU/Linex, a custom Debian distribution. 
Mutt
I'm currently looking for a printer. I want a Laser printer,
black & white should be enough, but color may be ok too if not
too expensive (especially toner). I guess I prefer a network one,
even if I don't really understand the way everything should
work.
As I understand it, a network printer just listens on a defined
port (9100/jetdirect or ipp/63), and prints jobs sent by a spooler.
There is no integrated spooler, so there are two solutions:
Are you rebasing or merging? seems to be the 64 thousand
dollar question over in 
...Christian, but you can use 'bubulle' if you like, provided you
pronounce it properly, ie the French way (b b l'). This is actually what most French(-speaking) DDs do.
There is a special exception for Amaya, who uses "boubouy " in her Sprenchglish.
I am proud of "bubulle", indeed. The nickname was given to me in the old days of BBS and *very* early Linux development in France, by 
In my 
Obviously any human being can recognize satire. There is obvious motivation on behalf of SCO’s PR department (correct me if I am wrong on where this comment originated; I did just some basic Google lookups, and also, this is a PR tactic known as “dead agenting”) to ignore the fact that this was clearly a satirical review. I would say, “perhaps SCO doesn’t understand why it is criticized,” but I believe it is obvious that they know why people are being critical and satirical of their company and their efforts.
I would like for everyone to think, for a moment, about how things could be if SCO didn’t take the litigious strategy that they have taken over the past 6 years. SCO UnixWare could have remained a successful product, and could have in reality become a major asset to the open source world. SCO could have been the good guys here, but instead they chose a path of litigation instead of innovation.
In the Open Source and Free Software world, innovation is what is seen as important. Companies may be in some sense rival to each other, but when it comes down to it, everyone appreciates and respects the work that everyone else contributes to the ecosystem. When people do not contribute in a positive way, or worse, betray their peers like SCO has done, it opens them up for criticism.
As I see the situation, I think that SCO’s current tactics as a company represent those of someone who is paranoid. People who are paranoid attack first, instead of defend their position. So, while I am sure this is a pointless question, I believe it must be asked anyway: “What would SCO UnixWare have been like if they did not take a litigious strategy against Linux users?”
If you look at it like this, it doesn’t really matter if we’re right, or SCO is right on who has the basic cases (well, it does, but we all know how that’s going to come out anyway).
Why does SCO keep pursuing a strategy of litigation when the feasibility of this strategy is at best, unknown? To some extent, I think it is now a matter of morale. After all, it’s bad if you have to admit that the strategy you have taken for the last six years, was wrong. In some ways, you could say that SCO’s current strategy is cult-like.
Furthermore, if any SCO people are interested in doing an interview about this whole matter, I encourage you to comment back, and I’ll be happy to publicize SCO’s viewpoint of this. It’s only fair.