Continuing my post series on the tools I use these days in Python, this time I
would like to talk about a library I really like, named
voluptuous.
It's no secret that most of the time, when a program receives data from the
outside, it's a big deal to handle it. Indeed, most of the time your program
has no guarantee that the stream is valid and that it contains what is
expected.
The
robustness principle
says you should be liberal in what you accept, though
that is not always a good idea
neither. Whatever policy you chose, you need to process those data and
implement a policy that will work lax or not.
That means that the program need to look into the data received, check that it
finds everything it needs, complete what might be missing (e.g. set some
default), transform some data, and maybe reject those data in the end.
Data validation
The first step is to validate the data, which means checking all the fields are
there and all the types are right or understandable (parseable).
Voluptuous
provides a single interface for all that called a
Schema.
>>> from voluptuous import Schema
>>> s = Schema(
... 'q': str,
... 'per_page': int,
... 'page': int,
... )
>>> s( "q": "hello" )
'q': 'hello'
>>> s( "q": "hello", "page": "world" )
voluptuous.MultipleInvalid: expected int for dictionary value @ data['page']
>>> s( "q": "hello", "unknown": "key" )
voluptuous.MultipleInvalid: extra keys not allowed @ data['unknown']
The argument to
voluptuous.Schema should be the data structure that you
expect.
Voluptuous accepts any kind of data structure, so it could also be a
simple string or an array of dict of array of integer. You get it.
Here it's a
dict with a few keys that if present should be validated as
certain types. By default,
Voluptuous does not raise an error if some keys
are missing. However, it is invalid to have extra keys in a dict by default. If
you want to allow extra keys, it is possible to specify it.
>>> from voluptuous import Schema
>>> s = Schema( "foo": str , extra=True)
>>> s( "bar": 2 )
"bar": 2
It is also possible to make some keys mandatory.
>>> from voluptuous import Schema, Required
>>> s = Schema( Required("foo"): str )
>>> s( )
voluptuous.MultipleInvalid: required key not provided @ data['foo']
You can create custom data type very easily.
Voluptuous data types are
actually just functions that are called with one argument, the value, and that
should either return the value or raise an
Invalid or
ValueError exception.
>>> from voluptuous import Schema, Invalid
>>> def StringWithLength5(value):
... if isinstance(value, str) and len(value) == 5:
... return value
... raise Invalid("Not a string with 5 chars")
...
>>> s = Schema(StringWithLength5)
>>> s("hello")
'hello'
>>> s("hello world")
voluptuous.MultipleInvalid: Not a string with 5 chars
Most of the time though, there is no need to create your own data types.
Voluptuous provides logical operators that can, combined with a few others
provided primitives such as
voluptuous.Length or
voluptuous.Range, create a
large range of validation scheme.
>>> from voluptuous import Schema, Length, All
>>> s = Schema(All(str, Length(min=3, max=5)))
>>> s("hello")
"hello"
>>> s("hello world")
voluptuous.MultipleInvalid: length of value must be at most 5
The
voluptuous documentation has a
good set of examples that you can check to have a good overview of what you can
do.
Data transformation
What's important to remember, is that each data type that you use is a function
that is called and returns a value, if the value is considered valid. That
value returned is what is actually used and returned after the schema
validation:
>>> import uuid
>>> from voluptuous import Schema
>>> def UUID(value):
... return uuid.UUID(value)
...
>>> s = Schema( "foo": UUID )
>>> data_converted = s( "foo": "uuid?" )
voluptuous.MultipleInvalid: not a valid value for dictionary value @ data['foo']
>>> data_converted = s( "foo": "8B7BA51C-DFF5-45DD-B28C-6911A2317D1D" )
>>> data_converted
'foo': UUID('8b7ba51c-dff5-45dd-b28c-6911a2317d1d')
By defining a custom
UUID function that converts a value to a UUID, the
schema converts the string passed in the data to a Python UUID object
validating the format at the same time.
Note a little trick here: it's not possible to use directly
uuid.UUID in the
schema, otherwise
Voluptuous would check that the data is actually an
instance of
uuid.UUID:
>>> from voluptuous import Schema
>>> s = Schema( "foo": uuid.UUID )
>>> s( "foo": "8B7BA51C-DFF5-45DD-B28C-6911A2317D1D" )
voluptuous.MultipleInvalid: expected UUID for dictionary value @ data['foo']
>>> s( "foo": uuid.uuid4() )
'foo': UUID('60b6d6c4-e719-47a7-8e2e-b4a4a30631ed')
And that's not what is wanted here.
That mechanism is really neat to transform, for example, strings to timestamps.
>>> import datetime
>>> from voluptuous import Schema
>>> def Timestamp(value):
... return datetime.datetime.strptime(value, "%Y-%m-%dT%H:%M:%S")
...
>>> s = Schema( "foo": Timestamp )
>>> s( "foo": '2015-03-03T12:12:12' )
'foo': datetime.datetime(2015, 3, 3, 12, 12, 12)
>>> s( "foo": '2015-03-03T12:12' )
voluptuous.MultipleInvalid: not a valid value for dictionary value @ data['foo']
Recursive schemas
So far,
Voluptuous has one limitation so far: the ability to have recursive
schemas. The simplest way to circumvent it is by using another function as an
indirection.
>>> from voluptuous import Schema, Any
>>> def _MySchema(value):
... return MySchema(value)
...
>>> from voluptuous import Any
>>> MySchema = Schema( "foo": Any("bar", _MySchema) )
>>> MySchema( "foo": "foo": "bar" )
'foo': 'foo': 'bar'
>>> MySchema( "foo": "foo": "baz" )
voluptuous.MultipleInvalid: not a valid value for dictionary value @ data['foo']['foo']
Usage in REST API
I started to use
Voluptuous to validate data in a the REST API provided by
Gnocchi. So far it has been a really good tool,
and we've been able to
create a complete REST API
that is very easy to validate on the server side. I would definitely recommend
it for that. It blends with any Web framework easily.
One of the upside compared to solution like
JSON Schema, is the ability to create or re-use your
own custom data types while converting values at validation time. It is also
very Pythonic, and extensible it's pretty great to use for all of that. It's
also not tied to any serialization format.
On the other hand, JSON Schema is language agnostic and is serializable itself
as JSON. That makes it easy to be exported and provided to a consumer so it can
understand the API and validate the data potentially on its side.
I knew it was about this time of the year 10 years ago when my Debian account
was created, but I couldn't remember the exact date until I looked it up
earlier this evening: today :). Rene Engelhard had been my advocate, and Marc
Brockschmidt my AM. Thanks guys!
A lot of time has passed since then, and I've worked in various parts of the
project. I became an application manager almost immediately, and quickly got
into the NM front desk as well, revamping parts of the NM process which had
become pretty bureaucratic (I think we are now, 10 years later, back where we
should be, thanks to almost all of the paperwork being automated, thanks
Enrico!). I've processed 37 NMs, most of them between 2005 and 2008, later I
was only active as front desk and eventually Debian account manager. I've
recently picked up AMing again, which I still find quite refreshing as the AM
will always also learn new things.
Quality Assurance was and is the other big field. Starting by doing QA uploads
of orphaned packages, I attended some QA meetings around Germany, and picked up
maintenance of the DDPO pages, which I still maintain. The link between QA and
NM is the MIA team where I was active for some years until they kindly kicked
me out because I was MIA there myself. I'm glad they are still using some of
the scripts I was writing to automate some things.
My favorite MUA is mutt, of which I became co-maintainer in 2007, and later
maintainer. I'm still listed in the uploaders field, but admittedly I haven't
really done anything there lately.
Also in 2007 I started working at credativ, after having been a research
assistant at the university, which meant making my Debian work professional. Of
course it also meant more real work and less time for the hobby part, but I was
still very active around that time. Later in 2010 I was marrying, and we got
two kids, at which point family was of course much more important, so my Debian
involvement dropped to a minimum. (Mostly lurking on IRC ;)
Being a PostgreSQL consultant at work, it was natural to start looking into the
packaging, so I started submitting patches to postgresql-common in 2011, and
became a co-maintainer in 2012. Since then, I've mostly been working on
PostgreSQL-related packages, of which far too many have my (co-)maintainer
stamp on them. To link the Debian and PostgreSQL worlds together, we started an
external repository (apt.postgresql.org) that contains packages for the
PostgreSQL major releases that Debian doesn't ship. Most of my open source time
at the moment is spent on getting all PostgreSQL packages in shape for Debian
and this repository.
According to minechangelogs, currently 844 changelog entries in Debian mention
my name, or were authored by me. Scrolling back yields memories of packages
that are long gone again from unstable, or I passed on to other maintainers.
There are way too many people in Debian that I enjoy(ed) working with to list
them here, and many of them are my friends. Debian is really the extended
family on the internet. My last DebConf before this year had been in Mar del
Plata - I had met some people at other conferences like FOSDEM, but meeting
(almost) everyone again in Heidelberg was very nice. I even remembered all
basic Mao rules :D.
So, thanks to everyone out there for making Debian such a wonderful place to
be!
So quite some time ago, I filed 
The C1 Server
Debian LTS
August was the fourth month I contributed to 
Since quite some time I am using 
So I decided to rename all the local folders to the scheme described in the above guide. At the end I want to have a folder layout as follows:
Once again time for the monthly upstream 
I am not quite sure about the text on the back, but guess it will
do. I picked three quotes from the official site for the book, and
hope it will work to trigger the interest of potential readers. The
Norwegian cover will look the same, but with the texts and bar code
replaced with the Norwegian version.
The book is very close to being ready for publication, and I expect
to upload the final draft to Lulu in the next few days and order a
final proof reading copy to verify that everything look like it should
before allowing everyone to order their own copy of Free Culture, in
English or Norwegian Bokm l. I'm waiting to give the the productive
proof readers a chance to complete their work.
What happened in the 
My monthly report covers a large part of what I have been doing in the free software world. I write it for 
For the concrete work I did during DebConf, I can only claim two schroot uploads to fix the lack of support of the new overlay filesystem that replaces aufs in the official Debian kernel, and some Distro Tracker work (fixing an issue that some people had when they were logged in via Debian s SSO).
While the numerous discussions I had during DebConf can t be qualified as work , they certainly contribute to build up work plans for the future:
As a Kali developer, I attended multiple sessions related to derivatives (notably the 
The new features are:
Selection of images: Up to now images for the widget were selected at random. The current version allows selecting images either at random (the default as before), but also in ascending or descending order by various criteria (uploaded, availability time, id, name, etc). With this change it is now possible to always display the most recent image(s) from a gallery.
