Francois Marier: Sample Python application using Libgearman
Gearman is a distributed queue with several language bindings.While Gearman has a nice Python implementation (python-gearman) of the client and worker, I chose to use the libgearman bindings (python-libgearman) directly since they are already packaged for Debian (as python-gearman.libgearman).
Unfortunately, these bindings are not very well documented, so here's the sample application I wished I had seen before I started.
Using the command-line toolsBefore diving into the Python bindings, you should make sure that you can get a quick application working on the command line (using the gearman-tools package).
Here's a very simple worker which returns verbatim the input it receives:
gearman -w -f myfunction catgearman -f myfunction 'test'Using the Python libgearman bindingsOnce your gearman setup is working (debugging is easier with the command-line tools), you can roll the gearman connection code into your application.
Here's a simple Python worker which returns what it receives:
and a matching client:#!/usr/bin/python
from gearman import libgearman
def work(job):
workload = job.get_workload()
return workload
gm_worker = libgearman.Worker()
gm_worker.add_server('localhost')
gm_worker.add_function('myfunction', work)
while True:
gm_worker.work()
This should behave in exactly the same way as the command-line examples above.#!/usr/bin/python
from gearman import libgearman
gm_client = libgearman.Client()
gm_client.add_server('localhost')
result = gm_client.do('myfunction', 'test')
print result
Returning job errorsIf you want to expose to the client errors in the processing done by the worker, modify the worker like this:
and the client this way:#!/usr/bin/python
from gearman import libgearman
def work(job):
workload = job.get_workload()
if workload == 'fail':
job.send_fail()
return workload
gm_worker = libgearman.Worker()
gm_worker.add_server('localhost')
gm_worker.add_function('myfunction', work)
while True:
gm_worker.work()
LicenseThe above source code is released under the following terms:#!/usr/bin/python
from gearman import libgearman
gm_client = libgearman.Client()
gm_client.add_server('localhost')
result = gm_client.do('myfunction', 'fail')
print result
To the extent possible under law, Francois Marier has waived all copyright and related or neighboring rights to this sample libgearman Python application. This work is published from: New Zealand.
I was laid off a couple of months ago, and have been thinking about
what I want to do next. Actually, I've mostly been writing code for
my pet projects, but earning a living again is occasionally on my mind.
I would, very much, like to earn a living while deveoping interesting
free software that I would like to use myself.
There's a bunch of options available:
Just 
I just uploaded gdebi_0.8~exp1 into experimental.
It s been a few months since 
Since my laptop features 
We downloaded a total of 8 
Anyone who's been following anything I've written lately may be under the impression that I dislike EFI. They'd be entirely correct. It's an awful thing and I've lost far too much of my life to it. It complicates the process of booting for no real benefit to the OS. The only real advantage we've seen so far is that we can configure boot devices in a vaguely vendor-neutral manner without having to care about BIOS drive numbers. Woo.
Whenever someone asked me about a calendar application, especially for the textmode, I always encouraged them to give 
As usual many people requested travel sponsorship to attend DebConf 11 and
are now waiting eagerly on the outcome of their request. We are sorry,
that we can t give you much details yet. To give you an idea of the
time line, here is a small explanation of the process.
About 100 people requested travel sponsorship. In total that would about
50 000 Euro. Problem is currently we don t exactly know, how much money we
may spend on travel sponsorship ourselves (By the way: If a potential
sponsor is reading this, please 
DECLARATIVE DIVERSIONS
======================
Introduction
A diversion is when it is possible to have dpkg not overwrite a file when it
reinstalls the package it belongs to, and to have it put the file from the
package somewhere else instead.
Declarative diversions involves a new control file with a declarative
syntax which dpkg will parse and process directly as part of the package unpack
and removal phases, eliminating the problems resulting from non-atomic
handling of diversions.
Topics
There are a number of topics involved in implementing this kind of project
* What syntax do we use for the new control file?
* What dpkg does with the control file
* How do we handle diversions to a non-existant directory?
* How do we order unpacking a new package which adds a diversion?
* How do we order unpacking a new package that removes a diversion?
* How do we order removing a package which had a diversion?
* How do we handle errors?
* What happens to dpkg-divert?
Details - Control File Syntax
-
It will conform to RFC2822 style with the following format:
* Divert-From:
* Divert-To:
* Blank lines and lines beginning with # will be comments
Divert-To will be optional and if it is ommitted then files being diverted
will have their filename changed to file.distrib
The above style has it s advantages, one of the main ones being that there is
no need to escape whitespace within filenames. It also allows for more commands
to be added at a later date.
Details - Control File Handling
-
Within control.tar.gz the file should be named diversions
This file is then copied to /var/lib/info/$package.diversions
We need to store the control file for the currently-installed package so that
we can detect declarative diversions that have been removed from the control
file between versions (On-disk version and the to-be-unpacked version).
If there is no previous version of a control file then all current diversions
within the package are left in place and any new ones declared within the
control file are added.
If there is a previous version of the control file then the diversions
listed within the file are compared with the diversions currently in place.
If a diversion exists that is no longer listed in the control file then it is
removed. If there is a diversion listed in the control file that doesn t
currently exist then it is added.
Details - Handling Diversions to non-existant directories
-
Diverting files to directories that don t exist can cause a number of problems.
If the package does not own the directory it may be left orphaned on
removal of the package
The package is responsible for ensuring the availability of the target
directory in the unpack phase.
If a declarative diversion attempts to divert a file to a folder that neither
exists already on the filesystem at unpack time, nor is shipped within
the package, dpkg will return an error at unpack time.
Details - Ordering Requirements
-
=>Unpacking a new package that adds a diversion
*1. Check if package has a previous diversion control file
*2. Copy control file to tmp directory
*3. Calculate diversions to be added
*4. Unpack folders
*5. Add diversions
*6. Unpack files
*7. Copy control file to /var/lib/dpkg/info/$package.diversion
*Performing diversions this way avoids trouble with creating folders and
accidentally overwriting the wrong version of files.
*Another major problem would be extracting a new version of the file and
overwriting the old one.
*This might cause the wrong version of the file to be diverted and the
package to break.
=>Unpacking a new package that removes a diversion
*1. Check if package has a previous diversion control file
*2. Copy control file to tmp directory
*3. Calculate diversions to be removed
*4. Check if dropped diversion results in a file conflict and abort the
unpack if it does.
*5. Unpack folders
*6. Unpack files
*7. Remove old files no longer needed in the package
*8. Remove diversions
*9. Copy control file to /var/lib/dpkg/info/$package.diversion
*If a package has no previous diversion control file then no diversions
will be removed and existing diversions can only be removed using
dpkg-divert.
*If removing a diversion results in a file conflict then dpkg will return
an error at unpack time before anything is unpacked and the unpack
is aborted.
=>Removing a package which had a diversion
*1. Remove files
*2. Remove diversions
*3. Remove folders
*4. Remove control file at /var/lib/dpkg/info/$package.diversion
*This ensures that all files, diversions and folders are removed correctly
Details - Error handling
Errors in diversions will have to handled with a great deal of care due to
the fact that if they are not the package could be broken.
This means that a great deal of checks must be done to ensure that all the files
can be diverted properly before any actual diverting takes place. If they can t
the package installation/update must be stopped and rolled back to avoid the
package being installed incorrectly or broken.
Details - dpkg-divert
-
When we impliment the new diversion method we should keep the current
dpkg-divert. This allows maintainers to catch up with the new method without
breaking their packages. It also allows maintainers to perform some operations
that aren t support by the new method.
All current functionality of dpkg-divert will be left intact for use by
sysadmins and for maintainers to catch up with the new system. The two
systems will work together without directly conflicting with each other.
Example Usage #1
The file to be diverted is /usr/share/foo
It needs to be moved to /usr/share/bar
The syntax of the control file would be:
#Start File
Divert-From: /usr/share/foo
Divert-To: /usr/share/bar
#End File
Example Usage #2
In this example the maintainer doesn t want to move the file to any specific
folder
The syntax of the control file would be:
#Start File
Divert-From: /usr/share/foo
#End File
This would divert the file to /usr/share/foo.distrib
Footnotes
-
RFC2822 Guide:
* 
(First posting to Pl net Commandline! Tag: pcli)
Vutral asked in IRC how to synchronise two shells environment
while they re running. As you may know, POSIX systems cannot change
a process environment vector after it has been started, only the
process itself can. Well, 
When soliciting for opinions, recently, I discovered that the python-based 
* We re doing heavy uploading and fixing in Debian-IN repository nowadays. Thanks to Vasudev for taking initiative on Git migration + various nice stuffs pending from long time.
* Gwibber is till mess. Help wanted as usual!
* And, MiniDebConf India at Pune is planned at Pune again. Unless, it is colliding heavily with my wife s plan, I ll be there as last time 
