Julien Danjou: Data validation in Python with voluptuous
Continuing my post series on the tools I use these days in Python, this time I
would like to talk about a library I really like, named
voluptuous.
It's no secret that most of the time, when a program receives data from the
outside, it's a big deal to handle it. Indeed, most of the time your program
has no guarantee that the stream is valid and that it contains what is
expected.
The robustness principle
says you should be liberal in what you accept, though
that is not always a good idea
neither. Whatever policy you chose, you need to process those data and
implement a policy that will work lax or not.
That means that the program need to look into the data received, check that it
finds everything it needs, complete what might be missing (e.g. set some
default), transform some data, and maybe reject those data in the end.
Schema.
>>> from voluptuous import Schema
>>> s = Schema(
... 'q': str,
... 'per_page': int,
... 'page': int,
... )
>>> s( "q": "hello" )
'q': 'hello'
>>> s( "q": "hello", "page": "world" )
voluptuous.MultipleInvalid: expected int for dictionary value @ data['page']
>>> s( "q": "hello", "unknown": "key" )
voluptuous.MultipleInvalid: extra keys not allowed @ data['unknown']
The argument to
voluptuous.Schema should be the data structure that you
expect. Voluptuous accepts any kind of data structure, so it could also be a
simple string or an array of dict of array of integer. You get it.
Here it's a dict with a few keys that if present should be validated as
certain types. By default, Voluptuous does not raise an error if some keys
are missing. However, it is invalid to have extra keys in a dict by default. If
you want to allow extra keys, it is possible to specify it.
>>> from voluptuous import Schema
>>> s = Schema( "foo": str , extra=True)
>>> s( "bar": 2 )
"bar": 2
It is also possible to make some keys mandatory.
>>> from voluptuous import Schema, Required
>>> s = Schema( Required("foo"): str )
>>> s( )
voluptuous.MultipleInvalid: required key not provided @ data['foo']
You can create custom data type very easily. Voluptuous data types are actually just functions that are called with one argument, the value, and that should either return the value or raise an
Invalid or ValueError exception.
>>> from voluptuous import Schema, Invalid
>>> def StringWithLength5(value):
... if isinstance(value, str) and len(value) == 5:
... return value
... raise Invalid("Not a string with 5 chars")
...
>>> s = Schema(StringWithLength5)
>>> s("hello")
'hello'
>>> s("hello world")
voluptuous.MultipleInvalid: Not a string with 5 chars
Most of the time though, there is no need to create your own data types. Voluptuous provides logical operators that can, combined with a few others provided primitives such as
voluptuous.Length or voluptuous.Range, create a
large range of validation scheme.
>>> from voluptuous import Schema, Length, All
>>> s = Schema(All(str, Length(min=3, max=5)))
>>> s("hello")
"hello"
>>> s("hello world")
voluptuous.MultipleInvalid: length of value must be at most 5
The voluptuous documentation has a good set of examples that you can check to have a good overview of what you can do. Data transformation
>>> import uuid
>>> from voluptuous import Schema
>>> def UUID(value):
... return uuid.UUID(value)
...
>>> s = Schema( "foo": UUID )
>>> data_converted = s( "foo": "uuid?" )
voluptuous.MultipleInvalid: not a valid value for dictionary value @ data['foo']
>>> data_converted = s( "foo": "8B7BA51C-DFF5-45DD-B28C-6911A2317D1D" )
>>> data_converted
'foo': UUID('8b7ba51c-dff5-45dd-b28c-6911a2317d1d')
By defining a custom
UUID function that converts a value to a UUID, the
schema converts the string passed in the data to a Python UUID object
validating the format at the same time.
Note a little trick here: it's not possible to use directly uuid.UUID in the
schema, otherwise Voluptuous would check that the data is actually an
instance of uuid.UUID:
>>> from voluptuous import Schema
>>> s = Schema( "foo": uuid.UUID )
>>> s( "foo": "8B7BA51C-DFF5-45DD-B28C-6911A2317D1D" )
voluptuous.MultipleInvalid: expected UUID for dictionary value @ data['foo']
>>> s( "foo": uuid.uuid4() )
'foo': UUID('60b6d6c4-e719-47a7-8e2e-b4a4a30631ed')
And that's not what is wanted here. That mechanism is really neat to transform, for example, strings to timestamps.
>>> import datetime
>>> from voluptuous import Schema
>>> def Timestamp(value):
... return datetime.datetime.strptime(value, "%Y-%m-%dT%H:%M:%S")
...
>>> s = Schema( "foo": Timestamp )
>>> s( "foo": '2015-03-03T12:12:12' )
'foo': datetime.datetime(2015, 3, 3, 12, 12, 12)
>>> s( "foo": '2015-03-03T12:12' )
voluptuous.MultipleInvalid: not a valid value for dictionary value @ data['foo']
Recursive schemas So far, Voluptuous has one limitation so far: the ability to have recursive schemas. The simplest way to circumvent it is by using another function as an indirection.
>>> from voluptuous import Schema, Any
>>> def _MySchema(value):
... return MySchema(value)
...
>>> from voluptuous import Any
>>> MySchema = Schema( "foo": Any("bar", _MySchema) )
>>> MySchema( "foo": "foo": "bar" )
'foo': 'foo': 'bar'
>>> MySchema( "foo": "foo": "baz" )
voluptuous.MultipleInvalid: not a valid value for dictionary value @ data['foo']['foo']
Usage in REST API
Since quite some time I am using 
So I decided to rename all the local folders to the scheme described in the above guide. At the end I want to have a folder layout as follows:
What happened in the 
My monthly report covers a large part of what I have been doing in the free software world. I write it for 
For the concrete work I did during DebConf, I can only claim two schroot uploads to fix the lack of support of the new overlay filesystem that replaces aufs in the official Debian kernel, and some Distro Tracker work (fixing an issue that some people had when they were logged in via Debian s SSO).
While the numerous discussions I had during DebConf can t be qualified as work , they certainly contribute to build up work plans for the future:
As a Kali developer, I attended multiple sessions related to derivatives (notably the 
The 
It's been over 2 years since I 
A week ago, we concluded our biggest DebConf ever! It was a huge success.
Following Ivans 
Regarding Internet Explorer it jumped in my face with a strange error message that recommended to enable
TLS 1.0, 1.1 and 1.2 in the options menu. Of course that's already enable. I'll try to take a look at the
handshake next week, but I bet we've to accept for the moment that IE will not work with so many SANs.
Would be interesting to try out Windows 10 with Spartan, but well I'm not that interested in Windows to
invest more time on that front. Other TLS implementations, like Java, would be also interesting to test.
Sven Hoexter replied to 
In order to 
180 degrees people say their life has changed by 180 whenever something alters their priorities, their viewpoints, their targets in life.
In our case, it's been 
I arrived to Heidelberg Main Train Station around 9am on 15th August and almost right away found Debian people so it made my trip to hostel easier. After arrival I checked in but needed to wait for 3 hours to get the key (it seems that SA will not have that problem at all, which is already an improvement). Although waiting was 3 hours long, it wasn't actually difficult at all as I started hugging and saying hi to many old (the super old super friend of mine - moray, or how I call him, "doc") and new friends. I just must say - if you know or don't know Rhonda, try to get somehow into her hugs. With her hug I acknowledged that I really did arrive to reunion.