In this article, I'm going to explain you how you can extend
Swift, the
OpenStack
Object Storage project, so it performs extra action on files at upload or at
download time.
We're going to build an anti-virus filter inside Swift. The goal is to
refuse uploaded data if they contain a virus. To help us with virus
analyses, we'll use
ClamAV.
WSGI, paste and middleware
In a pipeline, a black cat can become a white cat with the help of
some middleware.
To do our content analysis, the best place to hook in the Swift architecture
is at the beginning of every request, on swift-proxy, before the file is
actually stored on the cluster. Swift proxy uses, like many other OpenStack
projects,
paste to build his HTTP architecture.
Paste uses WSGI and provides an architecture based on a pipeline. The
pipeline is composed of a succession of middleware, ending with one
application. Each middleware has the chance to look at the request or at the
response, can modify it, and then pass it to the following middleware. The
latest component of the pipeline is the real application, and in this case,
the Swift proxy server.
If you've already deployed Swift, you encountered a default pipeline in the
swift-proxy.conf configuration file:
[pipeline:main]
pipeline = catch_errors healthcheck cache ratelimit tempauth proxy-logging proxy-server
This is a really basic pipeline with a few middleware. The first one catches
error, the second one is in charge to return
200 OK response if you send a
GET /healthcheck request on your proxy server. The third one is in charge
of caching, the fourth one is used for rate limiting, the fifth for
authentication, the sixth one for logging, and the final one is the actual
proxy server, in charge of proxying the request to the account, container,
or object servers (the others components of Swift). Of course, we could
remove or add any of the middleware here at our convenience.
Be aware that the order matters: for example, if you put
healthcheck after
tempauth, you won't be able to access the
/healthcheck URL without being
authenticated!
ClamAV
If you don't know
ClamAV, it's an antivirus engine
designed for detecting trojans, viruses, malware and other malicious
threats. Wwe're going to use it to scan every incoming file. To build the
middleware, we'll use the Python binding
pyclamd. The API is quite simple, see:
>>> import pyclamd
>>> pyclamd.init_unix_socket('/var/run/clamav/clamd.ctl')
>>> print pyclamd.scan_stream(pyclamd.EICAR)
'stream': 'Eicar-Test-Signature(44d88612fea8a8f36de82e1278abb02f:68)'
>>> print pyclamd.scan_stream("safe!")
None
Anatomy of a WSGI middleware
Your WSGI middleware should consist of a callable object. Usually this is
done with a class implementing the
__call__ method. Here's a basic
boilerplate:
class SwiftClamavMiddleware(object):
"""Middleware doing virus scan for Swift."""
def __init__(self, app, conf):
# app is the final application
self.app = app
def __call__(self, env, start_response):
return self.app(env, start_response)
def filter_factory(global_conf, **local_conf):
conf = global_conf.copy()
conf.update(local_conf)
def clamav_filter(app):
return SwiftClamavMiddleware(app, conf)
return clamav_filter
I'm not going to expand more on why this is built this way, but if you want
to have more info on this kind of filter middleware, you can read
their documentation on Paste.
This middleware will just do nothing as it is. It's going to simply pass all
requests it receives to the final application, and returns the result.
Testing our basic middleware
Now is a really good time to add unit tests. I hope you didn't think we were
going to write code without some tests, right? It's really easy to test a
middleware, as we're going to use
WebOb for that.
import unittest
from webob import Request
class FakeApp(object):
def __call__(self, env, start_response):
return Response(body="FAKE APP")(env, start_response)
class TestSwiftClamavMiddleware(unittest.TestCase):
def setUp(self):
self.app = SwiftClamavMiddleware(FakeApp(), )
def test_simple_request(self):
resp = Request.blank('/',
environ=
'REQUEST_METHOD': 'GET',
).get_response(self.app)
self.assertEqual(resp.body, "FAKE APP")
We create a FakeApp class, that represents a fake WSGI application. You
could also use a real application, or write a fake application looking like
the one you want to test. It'll require more time, but your tests will be
closer to the reality.
Here we write the simplest test we can for our middleware. We're just
sending a
GET / request to it, so it passes the request to the final
application and returns the result. It is transparent, it does nothing.
Now, with that solid base we'll able to add more features and test these
features incrementally.
Plugging ClamAV in
With our base ready, we can start thinking about how to plug ClamAV in. What
we want to check here, is the content of the file when it's uploaded.
If we refer to the
OpenStack object storage API
, a file upload is done via a
PUT request, so we're going to limit the
check to that kind of requests. Obviously, more checks could be added, but
we'll keep things simple here for the sake of comprehensibility.
With WSGI, the content of the request is available in
env['wsgi.input'] as
an object implementing a file interface. We'll scan that stream with ClamAV
to check for viruses.
import pyclamd
from webob import Response
class SwiftClamavMiddleware(object):
"""Middleware doing virus scan for Swift."""
def __init__(self, app, conf):
pyclamd.init_unix_socket('/var/run/clamav/clamd.ctl')
# app is the final application
self.app = app
def __call__(self, env, start_response):
if env['REQUEST_METHOD'] == "PUT":
# We have to read the whole content in memory because pyclamd
# forces us to, but this is a bad idea if the file is huge.
scan = pyclamd.scan_stream(env['wsgi.input'].read())
if scan:
return Response(status=403,
body="Virus %s detected" % scan['stream'],
content_type="text/plain")(env, start_response)
return self.app(env, start_response)
def filter_factory(global_conf, **local_conf):
conf = global_conf.copy()
conf.update(local_conf)
def clamav_filter(app):
return SwiftClamavMiddleware(app, conf)
return clamav_filter
That's it. We only check for
PUT requests and if there's a virus in the
file, we return a 403 Forbidden error with the name of the detected virus,
bypassing entirely the rest of the middleware chain and the application
handling.
Then, we can simply test it.
import unittest
from cStringIO import StringIO
from webob import Request, Response
class FakeApp(object):
def __call__(self, env, start_response):
return Response(body="FAKE APP")(env, start_response)
class TestSwiftClamavMiddleware(unittest.TestCase):
def setUp(self):
self.app = SwiftClamavMiddleware(FakeApp(), )
def test_put_empty(self):
resp = Request.blank('/v1/account/container/object',
environ=
'REQUEST_METHOD': 'PUT',
).get_response(self.app)
self.assertEqual(resp.body, "FAKE APP")
def test_put_no_virus(self):
resp = Request.blank('/v1/account/container/object',
environ=
'REQUEST_METHOD': 'PUT',
'wsgi.input': StringIO('foobar')
).get_response(self.app)
self.assertEqual(resp.body, "FAKE APP")
def test_put_virus(self):
resp = Request.blank('/v1/account/container/object',
environ=
'REQUEST_METHOD': 'PUT',
'wsgi.input': StringIO(pyclamd.EICAR)
).get_response(self.app)
self.assertEqual(resp.status_code, 403)
The first test
test_put_empty simulates an empty
PUT request. The second
one,
test_put_no_virus simulates a regular
PUT request but with a simple
file containing no virus.
Finally, the third and last test simulates the upload of a virus using the
EICAR test file. This is a special test file that
is recognized as a virus, even if it's not real one. Very handy for testing
virus detection software!
Configuring Swift proxy
Our middleware is ready! We can configure Swift's proxy server to use it. We
need to add the following lines to our
swift-proxy.conf to teach it how to
load the filter:
[filter:clamav]
paste.filter_factory = swiftclamav:clamav_filter
We'll assume that our Python modules is named
swiftclamava here. Now that
we've defined our filter and how to load it, we can use it in our pipeline:
[pipeline:main]
pipeline = catch_errors healthcheck cache ratelimit tempauth clamav proxy-logging proxy-server
Just before reaching the
proxy-server, and after the user being
authenticated, the content will be scanned for viruses. It's important here
to put this after authentication for example, because otherwise we may scan
content that will get rejected by the authtemp module, thus scanning for
nothing!
Beyond scanning
And voil , we now have a simple middleware testing uploaded content and
refusing infected files. We could enhance it with various other things, like
configuration handling, but I'll let that as an exercise for the interested
readers.
We didn't exploited it here, but note that you can also manipulate request
headers and modify them if needed. For example, we could have added a header
X-Object-Meta-Scanned-By: ClamAV to indicates that the file has been
scanned by ClamAV.
You should now be able to build your own middleware doing whatever you want
with uploaded data. Happy hacking!
Most OpenPGP User IDs look like this: 
I always looked at Github from a distant eye, mainly because I always
disliked their pull-request handling, and saw no value in the social hype it
brings. Why?
One click away isn't one click effort
The pull-request system looks like an incredible easy way to contribute to
any project hosted on Github. You're a click away to send your contribution
to any software. But the problem is that any worthy contribution isn't an
effort of a single click.
Doing any proper and useful contribution to a software is never done right
the first time. There's a dance you will have to play. A slowly rhythmed
back and forth between you and the software maintainer or team. You'll have
to dance it until your contribution is correct and can be merged.
But as a software maintainer, not everybody is going to follow you on this
choregraphy, and you'll end up with pull-request you'll never get finished
unless you wrap things up yourself. So the gain in pull-requests here, isn't
really bigger than a good bug report in most cases.
This is where the social argument of Github isn't anymore. As soon as you're
talking about projects bigger than a color theme for your favorite text
editor, this feature is overrated.
Contribution rework
If you're lucky enough, your contributor will play along and follow you on
this pull-request review process. You'll make suggestions, he will listen
and will modify his pull-request to follow your advice.
At this point, there's two technics he can use to please you.
Technic #1: the Topping
Github's pull-requests invite you to send an entire branch, eclipsing the
fact that it is composed of several commits. The problem is that a lot of
one-click-away contributors do not masterize Git and/or do not make efforts
to build a logical patchset, and nothing warns them that their branch
history is wrong. So they tend to change stuff around, commit, make a
mistake, commit, fix this mistake, commit, etc. This kind of branch is
composed of the whole brain's construction process of your contributor, and
is a real pain to review. To the point I quite often give up.
<figure>
<figcaption>
A typical case: 3 commits to build a 4 lines long file.
</figcaption>
</figure>
Without Github, the old method that all software used, and that many
software still use (e.g. Linux), is to send a patch set over e-mail (or any
other medium like Gerrit). This method has one positive effect, that it
forces the contributor to acknowledge the list of commits he is going to
publish. So, if the contributor he has fixup commits in his history, they
are going to be seen as first class citizen. And nobody is going to want to
see that, neither your contributor, nor the software maintainers. Therefore,
such a system tend to push contributors to write atomic, logical and
self-contained patchset that can be more easily reviewed.
Technic #2: the History Rewriter
This is actually the good way to build a working and logical patchset using
Git. Rewriting history and amending problematic patches using the famous
git rebase --interactive trick.
The problem is that if your contributor does this and then repush the branch
composing your pull-request to Github, you will both lose the previous
review done, each time. There's no history on the different versions of the
branch that has been pushed.
In the old alternative system like e-mail, no information is lost when
reworked patches are resent, obviously. This is far better because it eases
the following of the iterative discussions that the patch triggered.
Of course, it would be possible for Github to enhance this and fix it, but
currently it doesn't handle this use case correctly..
<figure>
It's not a secret for anyone that I've been contributing to OpenStack as a
daily routine for the last 18 months. The more I contribute, the more I like
the contribution workflow and process. It's already
The summit has been wonderful. It was my first OpenStack design summit --
even more as a PTL -- and bumping into various people I've never met so far
and worked with online only was a real pleasure!
<figure>
<figcaption>Me and Nick ready to talk about Ceilometer new features.</figcaption>
</figure>
<figcaption>Doug explaining Ceilometer architecture.</figcaption>
</figure>
The next session was about getting feedbacks from our users. We had a lot of
surprise to discover wonderful real use-cases and deployments, like the CERN
using Ceilometer and generating 2 GB of data per day!
The following sessions ran on Thursday and were much more about new features
discussion. A lot ot already existing blueprints were discussed and quickly
validated during the first morning session. Then,
Anyone that has a fondness for good ol RSpec knows that there s a fair
number of matchers and predicates and whatnot involved. Life isn t helped
by the recent (as of 2.11) decision to switch to using 
I ve been using this process for debugging the installer while launched from a Debian Live desktop. Having a full live environment at your disposal during install is a boon to debugging because you don t have to work within the constraints normally imposed upon you by the limited debian-installer environment. Caveat: running debian-installer from the debian-installer-launcher still has one or two unresolved issues, so unless you re specifically working on live integration issues like I am, compare with a conventional d-i image install to gain the added confidence the install is the same in both environments.
There are no longer active maintainers working on 
Debian-installer is a great piece of software, which allows one to
install the system in a variety of situations: manually, on a serial
console, graphically, automatically, there are a number of scenarios it
handles well.
However, there is one issue: when installing Debian to a machine that
requires firmware just to get at the network, getting things to work can
be a hassle. There is a 
My 1Tb laptop drive started misbehaving a few weeks ago, just spending a lot of time spinning when it should have been reading frequently changed files (like the browser cache). I was tempted to blame the browser at that point as no other packages appeared affected.
