Embedding Python into other applications to provide a scripting mechanism is a popular practice.
Ganglia can run user-supplied Python scripts for metric collection and the
Blender project does it too, allowing users to develop custom tools and use script code to direct their animations.
There are various reasons people choose Python:
- Modern, object orientated style of programming
- Interpreted language (no need to compile things, just edit and run the code)
- Python has a vast array of modules providing many features such as database and network access, data processing, etc
The bottom line is that the application developer who chooses to embed Python in their existing application can benefit from the product of all this existing code multiplied by the imagination of their users.
Enter
repro
repro is the SIP proxy of the
reSIProcate project. reSIProcate is an advanced SIP implementation developed in C++.
repro is a multi-threaded process.
repro's most serious competitor is the
Kamailio SIP proxy. Kamailio has its own bespoke scripting language that it has inherited from the
SIP Express Router (SER) family of projects.
repro has always been far more rigid in its capabilities than Kamailio. On the other hand, while Kamailio has given users great flexibility, it has also come at a cost: users can easily build configurations that are not valid or may not do what they really intend if they don't understand the intricacies of the SIP protocol.
Here is an example of the Kamailio configuration script (from Daniel's excellent blog about building
a Skype-like service in less than an hour)
Kamailio also has a wide array of plugins for things like database and LDAP access.
repro only had embedded
bdb and
MySQL support.
Embedding Python into
repro appears to be a quick way to fill many of these gaps and allow users to combine the power of the reSIProcate stack with their own custom routing logic. On the other hand, it is not simply copying the Kamailio scripting solution: rather, it provides a distinctive alternative.
Starting the integration
Embedding Python is such a popular practice that there is even
dedicated documentation on the subject. As well as looking there, I also looked over the example provided by the
embedded Python module for Ganglia.
Looking over the Ganglia
mod_python code I noticed a lot of boilerplate code for reference counting and other tedious activities. Given that reSIProcate is C++ code, I thought I would look for a C++ solution to this and I came across
PyCXX.
PyCXX is licensed under BSD-like terms similar to reSIProcate itself so it is a good fit. There is also
the alternative Boost.Python API, however, reSIProcate has been built without Boost dependencies so I decided to stick with PyCXX.
I looked over the
PyCXX examples and the documentation and was able to complete a first cut of the embedded Python scripting feature very quickly.
Using PyCXX
One unusual thing I noticed about PyCXX is that the Debian package,
python-cxx-dev does not provide any shared library. Instead, some uncompiled source files are provided and each project using PyCXX must compile them and link them statically itself. Here is how I do that in the
Makefile.am for
pyroute in
repro:
AM_CXXFLAGS = -I $(top_srcdir)
reproplugin_LTLIBRARIES = libpyroute.la
libpyroute_la_SOURCES = PyRoutePlugin.cxx
libpyroute_la_SOURCES += PyRouteWorker.cxx
libpyroute_la_SOURCES += $(PYCXX_SRCDIR)/cxxextensions.c
libpyroute_la_SOURCES += $(PYCXX_SRCDIR)/cxx_extensions.cxx
libpyroute_la_SOURCES += $(PYCXX_SRCDIR)/cxxsupport.cxx
libpyroute_la_SOURCES += $(PYCXX_SRCDIR)/../IndirectPythonInterface.cxx
libpyroute_la_CPPFLAGS = $(DEPS_PYTHON_CFLAGS)
libpyroute_la_LDFLAGS = -module -avoid-version
libpyroute_la_LDFLAGS += $(DEPS_PYTHON_LIBS)
EXTRA_DIST = example.py
noinst_HEADERS = PyRouteWorker.hxx
noinst_HEADERS += PyThreadSupport.hxx
The value
PYCXX_SRCDIR must be provided on the
configure command line. On Debian, it is
/usr/share/python2.7/CXX/Python2
Going multi-threaded
My initial implementation simply invoked the Python method from the main routing thread of the
repro SIP proxy. This meant that it would only be suitable for executing functions that complete quickly, ruling out the use of any Python scripts that talk to network servers or other slow activities.
When the proxy becomes heavily loaded, it is important that it can complete many tasks asynchronously, such as forwarding chat messages between users in real-time.
Therefore, it was essential to extend the solution to run the Python scripts in a pool of worker threads.
At this point, I had an initial feeling that there may be danger in just calling the Python methods from some other random threads started by my own code. I went to see the manual and
I came across this specific documentation about the subject.
It looks quite easy, just wrap the call to the user-supplied Python code in something like this:
PyGILState_STATE gstate;
gstate = PyGILState_Ensure();
/* Perform Python actions here. */
result = CallSomeFunction();
/* evaluate result or handle exception */
/* Release the thread. No Python API allowed beyond this point. */
PyGILState_Release(gstate);
Unfortunately, I found that this would not work and that one of two problems occur when using this code:
- The thread blocks on the call to PyGILState_Ensure()
- The program crashes with a segmentation fault when the call to a Python method was invoked
Exactly which of these outcomes I experienced seemed to depend on whether I tried to explicitly call
PyEval_ReleaseThread() from the main thread after doing the
Py_Initialize() and other setup tasks.
I tried various permutations of using
PyGILState_Ensure()PyGILState_Release() and/or
PyEval_SaveThread()/PyEval_ReleaseThread() but I always had one of the same problems.
The next thing that occurred to me is that maybe PyCXX provides some framework for thread integration: I had a look through the code and couldn't find any reference to the threading functionality from the C API.
I went looking for more articles and mailing list discussions and found implementation notes such as
this one in Linux Journal and
this wiki from the Blender developers. Most of them just appeared to be repeating what was in the manual, with a few subtle differences, but none of this provided an immediate solution.
Eventually, I discovered this other blog
about concurrency with embedded Python and it suggests something not highlighted in any of the other resources: calling
PyThreadState_New(m_interpreterState) in each thread after it starts and before it does anything else. Combining this with the use of
PyEval_SaveThread()/PyEval_ReleaseThread() fixed the problem: the use of
PyThreadState_New() was not otherwise mentioned in the relevant section of the Python guide.
I decided to take this solution a step further and create a convenient C++ class to encapsulate the logic, you can see this in
PyThreadSupport.hxx:
class PyExternalUser
public:
PyExternalUser(PyInterpreterState* interpreterState)
: mInterpreterState(interpreterState),
mThreadState(PyThreadState_New(mInterpreterState)) ;
class Use
public:
Use(PyExternalUser& user)
: mUser(user)
PyEval_RestoreThread(mUser.getThreadState()); ;
~Use() mUser.setThreadState(PyEval_SaveThread()); ;
private:
PyExternalUser& mUser;
;
friend class Use;
protected:
PyThreadState* getThreadState() return mThreadState; ;
void setThreadState(PyThreadState* threadState) mThreadState = threadState; ;
private:
PyInterpreterState* mInterpreterState;
PyThreadState* mThreadState;
;
and the way to use it is demonstrated in the
PyRouteWorker class. Observe how
PyExternalUser::Use is instantiated in the
PyRouteWorker::process() method: when it goes out of scope (either due to a normal return, an error or an exception) the necessary call to
PyEval_SaveThread() is made in the
PyExternalUser::Use::~Use() destructor.
Using other Python modules and DSO problems
All of the above worked for basic Python such as this trivial example script:
def on_load():
'''Do initialisation when module loads'''
print 'example: on_load invoked'
def provide_route(method, request_uri, headers):
'''Process a request URI and return the target URI(s)'''
print 'example: method = ' + method
print 'example: request_uri = ' + request_uri
print 'example: From = ' + headers["From"]
print 'example: To = ' + headers["To"]
routes = list()
routes.append('sip:bob@example.org')
routes.append('sip:alice@example.org')
return routes
However, it needs a more credible and useful test: using the
python-ldap module to try and query an LDAP server appears like a good choice.
Upon trying to use
import ldap in the Python script,
repro would refuse to load the Python script, choking on an error like this:
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/ldap/__init__.py", line 22, in
import _ldap
ImportError: /usr/lib/python2.7/dist-packages/_ldap.so: undefined symbol: PyExc_SystemError
I looked at the file
_ldap.so and discovered that it is linked with the LDAP libraries but not explicitly linked to any version of the Python runtime libraries. It expects the application hosting it to provide the Python symbols globally.
In my own implementation, my embedded Python encapsulation code is provide as a DSO plugin, similar to the way plugins are loaded in
Ganglia or Apache. The DSO links to Python: the DSO is loaded by a
dlopen() call from the main process. The main
repro binary has no direct link to Python libraries.
Adding
RTLD_GLOBAL to the top-level
dlopen() call for loading the plugin is one way to ensure the Python symbols are made available to the Python modules loaded indirectly by the Python interpreter. This solution may be suitable for applications that don't mix and match many different components.
Doing something useful with it
Now it was all working nicely, I took a boilerplate LDAP Python example and used it for making a trivial script that converts
sip:user@example.org to something like
sip:9001@pbx.example.org, assuming that
9001 is the
telephoneNumber associated with the
user@ email address in LDAP.
It is surprisingly simple and easily adaptable to local requirements depending upon the local LDAP structures:
import ldap
from urlparse import urlparse
def on_load():
'''Do initialisation when module loads'''
#print 'ldap router: on_load invoked'
def provide_route(method, request_uri, headers):
'''Process a request URI and return the target URI(s)'''
#print 'ldap router: request_uri = ' + request_uri
_request_uri = urlparse(request_uri)
routes = list()
# Basic LDAP server parameters:
server_uri = 'ldaps://ldap.example.org'
base_dn = "dc=example,dc=org"
# this domain will be appended to the phone numbers when creating
# the target URI:
phone_domain = 'pbx.example.org'
# urlparse is not great for "sip:" URIs,
# the user@host portion is in the 'path' element:
filter = "(&(objectClass=inetOrgPerson)(mail=%s))" % _request_uri.path
#print "Using filter: %s" % filter
try:
con = ldap.initialize(server_uri)
scope = ldap.SCOPE_SUBTREE
retrieve_attributes = None
result_id = con.search(base_dn, scope, filter, retrieve_attributes)
result_set = []
while 1:
timeout = 1
result_type, result_data = con.result(result_id, 0, None)
if (result_data == []):
break
else:
if result_type == ldap.RES_SEARCH_ENTRY:
result_set.append(result_data)
if len(result_set) == 0:
#print "No Results."
return routes
for i in range(len(result_set)):
for entry in result_set[i]:
if entry[1].has_key('telephoneNumber'):
phone = entry[1]['telephoneNumber'][0]
routes.append('sip:' + phone + '@' + phone_domain)
except ldap.LDAPError, error_message:
print "Couldn't Connect. %s " % error_message
return routes
Embedded Python opens up a world of possibilities
After Ganglia 3.1.0 introduced an embedded Python scripting facility,
dozens of new modules started appearing in github. Python scripting lowers the barrier for new contributors to a project and makes it much easier to fine tune free software projects to meet local requirements: hopefully we will see similar trends with the
repro SIP proxy and other projects that choose Python.
The code is committed
here in the reSIProcate repository. These features will appear in the next beta release of reSIProcate and Debian packages will be available in unstable in a few days.
I've said recently that pervasive surveillance is wrong. I don't think anyone from the NSA should have a leadership position in the development or deployment of Internet communications, because their interests are at odds with the interest of the rest of the Internet. But someone at the NSA is in exactly such a position. They ought to step down. Here's the background: The Internet Research Task Force (IRTF) is a body tasked with research into underlying concepts, themes, and technologies related to the Internet as a whole. They act as a research organization that cooperates and complements the engineering and standards-setting activities of the Internet Engineering Task Force (IETF). The IRTF is divided into issue-specific research groups, each of which has a Chair or Co-Chairs who have "wide discretion in the conduct of Research Group business", and are tasked with organizing the research and discussion, ensuring that the group makes progress on the relevant issues, and communicating the general sense of the results back to the rest of the IRTF and the IETF. One of the IRTF's research groups specializes in cryptography: the Crypto Forum Research Group (CFRG). There are two current chairs of the CFRG: David McGrew <mcgrew@cisco.com> and Kevin M. Igoe <kmigoe@nsa.gov>. As you can see from his e-mail address, Kevin M. Igoe is affiliated with the National Security Agency (NSA). The NSA itself actively tries to weaken cryptography on the Internet so that they can improve their surveillance, and one of the ways they try to do so is to "influence policies, standards, and specifications". On the CFRG list yesterday, Trevor Perrin requested the removal of Kevin M. Igoe from his position as Co-chair of the CFRG. Trevor's specific arguments rest heavily on the technical merits of a proposed cryptographic mechanism called Dragonfly key exchange, but I think the focus on Dragonfly itself is the least of the concerns for the IRTF. I've seconded Trevor's proposal, and asked Kevin directly to step down and to provide us with information about any attempts by the NSA to interfere with or subvert recommendations coming from these standards bodies. Below is my letter in full: 
I recently got my self and 
Greetings from the FAU Security Team (FAUST), the 
In a 
Cleaning up X server warnings
So I was sitting in the Narita airport with a couple of other free
software developers merging X server patches. One of the developers
was looking over my shoulder while the X server was building and
casually commented on the number of warnings generated by the
compiler.
I felt like I had invited someone into my house without cleaning for
months embarrassed and ashamed that we d let the code devolve into
this state.
Of course, we ve got excuses the X server code base is one of the
oldest pieces of regularly used free software in existence. It was
started before ANSI-C was codified. No function prototypes, no
const , no void * , no enums or stdint.h. There may be a few
developers out there who remember those days (fondly, of course), but
I think most of us are glad that our favorite systems language has
gained a lot of compile-time checking in the last 25 years.
We ve spent time in the past adding function prototypes and cleaning
up other warnings, but there was never a point at which the X server
compiled without any warnings. More recently, we ve added a pile of
new warning flags when compiling the X server which only served to
increase the number of warnings dramatically.
The current situation
With the master branch of the X server and released versions of the
dependencies, we generate 1047 warnings in the default build.
-Wcast-qual considered chatty
The GCC flag, -Wcast-qual, complains when you cast a pointer and
change the const qualifier status. A very common thing for the X
server to do is declare pointers as const to mark them as immutable
once assigned. Often, the relevant data is actually constructed once
at startup in allocated memory and stored to the data
structure. During server reset, that needs to be freed, but free
doesn t take a const pointer, so we cast to (void *), which
-Wcast-qual then complains about. Loudly.
Of the 1047 warnings, 380 of them are generated by this one warning
flag. I ve gone ahead and just disabled it in util/macros for now.
String constants are a pain
The X server uses string constants to initialize defaults for font
paths, configuration options, font names along with a host of other
things. These end up getting stored in variables that can also take
allocated storage. I ve gone ahead and declared the relevant objects
as const and then fixed the code to suit.
I don t have a count of the number of warnings these changes fixed;
they were scattered across dozens of X server directories, and I was
fixing one directory at a time, but probably more than half of the
remaining warnings were of this form.
And a host of other warnings
Fixing the rest of the warnings was mostly a matter of stepping
through them one at a time and actually adjusting the code. Shadowed
declarations, unused values, redundant declarations and missing printf
attributes were probably the bulk of them though.
Changes to external modules
Instead of just hacking the X server code, I ve created patches for
other modules where necessary to fix the problems in the right
place.
A nice set of recent posts have done a great job detailing the remaining ways that a root user can get at kernel memory. Part of this is driven by the ideas behind UEFI Secure Boot, but they come from the same goal: making sure that the root user cannot directly subvert the running kernel. My perspective on this is toward making sure that an attacker who has gained access and then gained root privileges can t continue to elevate their access and install invisible kernel rootkits.
An outline for possible attack vectors is spelled out by Matthew Gerrett s continuing 
(Yes, yes... Maybe I should post in Spanish.. But hey, gotta keep consistecy in my blog!)
General, public, open invitation
Are you in Mexico City, or do you plan to be next Wednesday (December 11)?
Are you interested in video edition? In Free Software?
I will have the pleasure to host at home the great Chema Serralde, a good friend, and a multifacetic guru both in the technical and musical areas. He will present a workshop: Video editing from the command line.
I asked Chema for an outline of his talk, but given he is a busy guy, I will basically translate 
Today I should have been heading down to York, to attend the Bytemark Christmas party. Instead I'm here in Edinburgh, because wind/storms basically shutdown the rail network in Scotland for the morning.
Technically I could have probably made it, but only belatedly and only at a huge cost to my sanity. The train-station was insane with stranded people, and there seemed no guarantee the recently-revived service would continue.
So instead I'm sulking at home.
I had a lot of other things scheduled to do in York/London today/tomorrow, for reasons that will become apparent next week, so to say I'm annoyed is an understatement.
In happier news I'm not dead.
Replacing my phone
My first Android phone was
a 
I recently bought a Cubieboard to replace my old Sheevaplug which has finally blown a power supply capacitor (this appears to be a common defect of Sheevaplugs), so I am publishing these instructions which show how to install Debian on