Embedding Python into other applications to provide a scripting mechanism is a popular practice.
Ganglia can run user-supplied Python scripts for metric collection and the
Blender project does it too, allowing users to develop custom tools and use script code to direct their animations.
There are various reasons people choose Python:
- Modern, object orientated style of programming
- Interpreted language (no need to compile things, just edit and run the code)
- Python has a vast array of modules providing many features such as database and network access, data processing, etc
The bottom line is that the application developer who chooses to embed Python in their existing application can benefit from the product of all this existing code multiplied by the imagination of their users.
Enter
repro
repro is the SIP proxy of the
reSIProcate project. reSIProcate is an advanced SIP implementation developed in C++.
repro is a multi-threaded process.
repro's most serious competitor is the
Kamailio SIP proxy. Kamailio has its own bespoke scripting language that it has inherited from the
SIP Express Router (SER) family of projects.
repro has always been far more rigid in its capabilities than Kamailio. On the other hand, while Kamailio has given users great flexibility, it has also come at a cost: users can easily build configurations that are not valid or may not do what they really intend if they don't understand the intricacies of the SIP protocol.
Here is an example of the Kamailio configuration script (from Daniel's excellent blog about building
a Skype-like service in less than an hour)
Kamailio also has a wide array of plugins for things like database and LDAP access.
repro only had embedded
bdb and
MySQL support.
Embedding Python into
repro appears to be a quick way to fill many of these gaps and allow users to combine the power of the reSIProcate stack with their own custom routing logic. On the other hand, it is not simply copying the Kamailio scripting solution: rather, it provides a distinctive alternative.
Starting the integration
Embedding Python is such a popular practice that there is even
dedicated documentation on the subject. As well as looking there, I also looked over the example provided by the
embedded Python module for Ganglia.
Looking over the Ganglia
mod_python code I noticed a lot of boilerplate code for reference counting and other tedious activities. Given that reSIProcate is C++ code, I thought I would look for a C++ solution to this and I came across
PyCXX.
PyCXX is licensed under BSD-like terms similar to reSIProcate itself so it is a good fit. There is also
the alternative Boost.Python API, however, reSIProcate has been built without Boost dependencies so I decided to stick with PyCXX.
I looked over the
PyCXX examples and the documentation and was able to complete a first cut of the embedded Python scripting feature very quickly.
Using PyCXX
One unusual thing I noticed about PyCXX is that the Debian package,
python-cxx-dev does not provide any shared library. Instead, some uncompiled source files are provided and each project using PyCXX must compile them and link them statically itself. Here is how I do that in the
Makefile.am for
pyroute in
repro:
AM_CXXFLAGS = -I $(top_srcdir)
reproplugin_LTLIBRARIES = libpyroute.la
libpyroute_la_SOURCES = PyRoutePlugin.cxx
libpyroute_la_SOURCES += PyRouteWorker.cxx
libpyroute_la_SOURCES += $(PYCXX_SRCDIR)/cxxextensions.c
libpyroute_la_SOURCES += $(PYCXX_SRCDIR)/cxx_extensions.cxx
libpyroute_la_SOURCES += $(PYCXX_SRCDIR)/cxxsupport.cxx
libpyroute_la_SOURCES += $(PYCXX_SRCDIR)/../IndirectPythonInterface.cxx
libpyroute_la_CPPFLAGS = $(DEPS_PYTHON_CFLAGS)
libpyroute_la_LDFLAGS = -module -avoid-version
libpyroute_la_LDFLAGS += $(DEPS_PYTHON_LIBS)
EXTRA_DIST = example.py
noinst_HEADERS = PyRouteWorker.hxx
noinst_HEADERS += PyThreadSupport.hxx
The value
PYCXX_SRCDIR must be provided on the
configure command line. On Debian, it is
/usr/share/python2.7/CXX/Python2
Going multi-threaded
My initial implementation simply invoked the Python method from the main routing thread of the
repro SIP proxy. This meant that it would only be suitable for executing functions that complete quickly, ruling out the use of any Python scripts that talk to network servers or other slow activities.
When the proxy becomes heavily loaded, it is important that it can complete many tasks asynchronously, such as forwarding chat messages between users in real-time.
Therefore, it was essential to extend the solution to run the Python scripts in a pool of worker threads.
At this point, I had an initial feeling that there may be danger in just calling the Python methods from some other random threads started by my own code. I went to see the manual and
I came across this specific documentation about the subject.
It looks quite easy, just wrap the call to the user-supplied Python code in something like this:
PyGILState_STATE gstate;
gstate = PyGILState_Ensure();
/* Perform Python actions here. */
result = CallSomeFunction();
/* evaluate result or handle exception */
/* Release the thread. No Python API allowed beyond this point. */
PyGILState_Release(gstate);
Unfortunately, I found that this would not work and that one of two problems occur when using this code:
- The thread blocks on the call to PyGILState_Ensure()
- The program crashes with a segmentation fault when the call to a Python method was invoked
Exactly which of these outcomes I experienced seemed to depend on whether I tried to explicitly call
PyEval_ReleaseThread() from the main thread after doing the
Py_Initialize() and other setup tasks.
I tried various permutations of using
PyGILState_Ensure()PyGILState_Release() and/or
PyEval_SaveThread()/PyEval_ReleaseThread() but I always had one of the same problems.
The next thing that occurred to me is that maybe PyCXX provides some framework for thread integration: I had a look through the code and couldn't find any reference to the threading functionality from the C API.
I went looking for more articles and mailing list discussions and found implementation notes such as
this one in Linux Journal and
this wiki from the Blender developers. Most of them just appeared to be repeating what was in the manual, with a few subtle differences, but none of this provided an immediate solution.
Eventually, I discovered this other blog
about concurrency with embedded Python and it suggests something not highlighted in any of the other resources: calling
PyThreadState_New(m_interpreterState) in each thread after it starts and before it does anything else. Combining this with the use of
PyEval_SaveThread()/PyEval_ReleaseThread() fixed the problem: the use of
PyThreadState_New() was not otherwise mentioned in the relevant section of the Python guide.
I decided to take this solution a step further and create a convenient C++ class to encapsulate the logic, you can see this in
PyThreadSupport.hxx:
class PyExternalUser
public:
PyExternalUser(PyInterpreterState* interpreterState)
: mInterpreterState(interpreterState),
mThreadState(PyThreadState_New(mInterpreterState)) ;
class Use
public:
Use(PyExternalUser& user)
: mUser(user)
PyEval_RestoreThread(mUser.getThreadState()); ;
~Use() mUser.setThreadState(PyEval_SaveThread()); ;
private:
PyExternalUser& mUser;
;
friend class Use;
protected:
PyThreadState* getThreadState() return mThreadState; ;
void setThreadState(PyThreadState* threadState) mThreadState = threadState; ;
private:
PyInterpreterState* mInterpreterState;
PyThreadState* mThreadState;
;
and the way to use it is demonstrated in the
PyRouteWorker class. Observe how
PyExternalUser::Use is instantiated in the
PyRouteWorker::process() method: when it goes out of scope (either due to a normal return, an error or an exception) the necessary call to
PyEval_SaveThread() is made in the
PyExternalUser::Use::~Use() destructor.
Using other Python modules and DSO problems
All of the above worked for basic Python such as this trivial example script:
def on_load():
'''Do initialisation when module loads'''
print 'example: on_load invoked'
def provide_route(method, request_uri, headers):
'''Process a request URI and return the target URI(s)'''
print 'example: method = ' + method
print 'example: request_uri = ' + request_uri
print 'example: From = ' + headers["From"]
print 'example: To = ' + headers["To"]
routes = list()
routes.append('sip:bob@example.org')
routes.append('sip:alice@example.org')
return routes
However, it needs a more credible and useful test: using the
python-ldap module to try and query an LDAP server appears like a good choice.
Upon trying to use
import ldap in the Python script,
repro would refuse to load the Python script, choking on an error like this:
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/ldap/__init__.py", line 22, in
import _ldap
ImportError: /usr/lib/python2.7/dist-packages/_ldap.so: undefined symbol: PyExc_SystemError
I looked at the file
_ldap.so and discovered that it is linked with the LDAP libraries but not explicitly linked to any version of the Python runtime libraries. It expects the application hosting it to provide the Python symbols globally.
In my own implementation, my embedded Python encapsulation code is provide as a DSO plugin, similar to the way plugins are loaded in
Ganglia or Apache. The DSO links to Python: the DSO is loaded by a
dlopen() call from the main process. The main
repro binary has no direct link to Python libraries.
Adding
RTLD_GLOBAL to the top-level
dlopen() call for loading the plugin is one way to ensure the Python symbols are made available to the Python modules loaded indirectly by the Python interpreter. This solution may be suitable for applications that don't mix and match many different components.
Doing something useful with it
Now it was all working nicely, I took a boilerplate LDAP Python example and used it for making a trivial script that converts
sip:user@example.org to something like
sip:9001@pbx.example.org, assuming that
9001 is the
telephoneNumber associated with the
user@ email address in LDAP.
It is surprisingly simple and easily adaptable to local requirements depending upon the local LDAP structures:
import ldap
from urlparse import urlparse
def on_load():
'''Do initialisation when module loads'''
#print 'ldap router: on_load invoked'
def provide_route(method, request_uri, headers):
'''Process a request URI and return the target URI(s)'''
#print 'ldap router: request_uri = ' + request_uri
_request_uri = urlparse(request_uri)
routes = list()
# Basic LDAP server parameters:
server_uri = 'ldaps://ldap.example.org'
base_dn = "dc=example,dc=org"
# this domain will be appended to the phone numbers when creating
# the target URI:
phone_domain = 'pbx.example.org'
# urlparse is not great for "sip:" URIs,
# the user@host portion is in the 'path' element:
filter = "(&(objectClass=inetOrgPerson)(mail=%s))" % _request_uri.path
#print "Using filter: %s" % filter
try:
con = ldap.initialize(server_uri)
scope = ldap.SCOPE_SUBTREE
retrieve_attributes = None
result_id = con.search(base_dn, scope, filter, retrieve_attributes)
result_set = []
while 1:
timeout = 1
result_type, result_data = con.result(result_id, 0, None)
if (result_data == []):
break
else:
if result_type == ldap.RES_SEARCH_ENTRY:
result_set.append(result_data)
if len(result_set) == 0:
#print "No Results."
return routes
for i in range(len(result_set)):
for entry in result_set[i]:
if entry[1].has_key('telephoneNumber'):
phone = entry[1]['telephoneNumber'][0]
routes.append('sip:' + phone + '@' + phone_domain)
except ldap.LDAPError, error_message:
print "Couldn't Connect. %s " % error_message
return routes
Embedded Python opens up a world of possibilities
After Ganglia 3.1.0 introduced an embedded Python scripting facility,
dozens of new modules started appearing in github. Python scripting lowers the barrier for new contributors to a project and makes it much easier to fine tune free software projects to meet local requirements: hopefully we will see similar trends with the
repro SIP proxy and other projects that choose Python.
The code is committed
here in the reSIProcate repository. These features will appear in the next beta release of reSIProcate and Debian packages will be available in unstable in a few days.
Object Lifetimes under Present Redirection
Present extension redirection passes responsibility for showing
application contents from the X server to the compositing
manager. This eliminates an extra copy to the composite extension
window buffer and also allows the application contents to be shown
in the right frame.
(Currently, the copy from application buffer to window buffer is
synchronized to the application-specified frame, and then a Damage
event is delivered to the compositing manager which constructs the
screen image using the window buffer and presents that to the screen
at least one frame later, which generally adds another frame of delay
for the application.)
The redirection operation itself is simple just wrap the
PresentPixmap request up into an event and send it to the compositing
manager. However, the pixmap to be presented is allocated by the
application, and hence may disappear at any time. We need to carefully
control the lifetime of the pixmap ID and the specific frame contents
of the pixmap so that the compositing manager can reliably construct
complete frames.
We ll separately discuss the lifetime of the specific frame contents
from that of the pixmap itself. By the frame contents , I mean the
image, as a set of pixel values in the pixmap, to be presented for a
specific frame.
Present Pixmap contents lifetime
After the application is finished constructing a particular frame in a
pixmap, it passes the pixmap to the X server with PresentPixmap. With
non-redirected Present, the X server is responsible for generating a
PresentIdleNotify event once the server is finished using the
contents. There are three different cases that the server handles,
matching the three different PresentCompleteModes:
Fighting with passwords have always been one of my favorite battles in the past and unfortunately the former always won. I never liked using the root user that much for administering a machine and made a massive use of sudo, I won t list all the benefits of using sudo, but the 
Over the last few weeks I have been working on getting the
Greetings from the FAU Security Team (FAUST), the 
If you maintain software in debian, one of your regular maintenance tasks is checking for new upstream versions, reviewing them, and preparing them for debian if appropriate. One of those steps is often to verify the cryptographic signature on the upstream source archive. At the moment, most maintainers do the cryptographic check manually, or maybe even don't bother to do it at all. For the common case of detached OpenPGP signatures, though, 