Continuing my post series on the tools I use these days in Python, this time I
would like to talk about a library I really like, named
voluptuous.
It's no secret that most of the time, when a program receives data from the
outside, it's a big deal to handle it. Indeed, most of the time your program
has no guarantee that the stream is valid and that it contains what is
expected.
The
robustness principle
says you should be liberal in what you accept, though
that is not always a good idea
neither. Whatever policy you chose, you need to process those data and
implement a policy that will work lax or not.
That means that the program need to look into the data received, check that it
finds everything it needs, complete what might be missing (e.g. set some
default), transform some data, and maybe reject those data in the end.
Data validation
The first step is to validate the data, which means checking all the fields are
there and all the types are right or understandable (parseable).
Voluptuous
provides a single interface for all that called a
Schema.
>>> from voluptuous import Schema
>>> s = Schema(
... 'q': str,
... 'per_page': int,
... 'page': int,
... )
>>> s( "q": "hello" )
'q': 'hello'
>>> s( "q": "hello", "page": "world" )
voluptuous.MultipleInvalid: expected int for dictionary value @ data['page']
>>> s( "q": "hello", "unknown": "key" )
voluptuous.MultipleInvalid: extra keys not allowed @ data['unknown']
The argument to
voluptuous.Schema should be the data structure that you
expect.
Voluptuous accepts any kind of data structure, so it could also be a
simple string or an array of dict of array of integer. You get it.
Here it's a
dict with a few keys that if present should be validated as
certain types. By default,
Voluptuous does not raise an error if some keys
are missing. However, it is invalid to have extra keys in a dict by default. If
you want to allow extra keys, it is possible to specify it.
>>> from voluptuous import Schema
>>> s = Schema( "foo": str , extra=True)
>>> s( "bar": 2 )
"bar": 2
It is also possible to make some keys mandatory.
>>> from voluptuous import Schema, Required
>>> s = Schema( Required("foo"): str )
>>> s( )
voluptuous.MultipleInvalid: required key not provided @ data['foo']
You can create custom data type very easily.
Voluptuous data types are
actually just functions that are called with one argument, the value, and that
should either return the value or raise an
Invalid or
ValueError exception.
>>> from voluptuous import Schema, Invalid
>>> def StringWithLength5(value):
... if isinstance(value, str) and len(value) == 5:
... return value
... raise Invalid("Not a string with 5 chars")
...
>>> s = Schema(StringWithLength5)
>>> s("hello")
'hello'
>>> s("hello world")
voluptuous.MultipleInvalid: Not a string with 5 chars
Most of the time though, there is no need to create your own data types.
Voluptuous provides logical operators that can, combined with a few others
provided primitives such as
voluptuous.Length or
voluptuous.Range, create a
large range of validation scheme.
>>> from voluptuous import Schema, Length, All
>>> s = Schema(All(str, Length(min=3, max=5)))
>>> s("hello")
"hello"
>>> s("hello world")
voluptuous.MultipleInvalid: length of value must be at most 5
The
voluptuous documentation has a
good set of examples that you can check to have a good overview of what you can
do.
Data transformation
What's important to remember, is that each data type that you use is a function
that is called and returns a value, if the value is considered valid. That
value returned is what is actually used and returned after the schema
validation:
>>> import uuid
>>> from voluptuous import Schema
>>> def UUID(value):
... return uuid.UUID(value)
...
>>> s = Schema( "foo": UUID )
>>> data_converted = s( "foo": "uuid?" )
voluptuous.MultipleInvalid: not a valid value for dictionary value @ data['foo']
>>> data_converted = s( "foo": "8B7BA51C-DFF5-45DD-B28C-6911A2317D1D" )
>>> data_converted
'foo': UUID('8b7ba51c-dff5-45dd-b28c-6911a2317d1d')
By defining a custom
UUID function that converts a value to a UUID, the
schema converts the string passed in the data to a Python UUID object
validating the format at the same time.
Note a little trick here: it's not possible to use directly
uuid.UUID in the
schema, otherwise
Voluptuous would check that the data is actually an
instance of
uuid.UUID:
>>> from voluptuous import Schema
>>> s = Schema( "foo": uuid.UUID )
>>> s( "foo": "8B7BA51C-DFF5-45DD-B28C-6911A2317D1D" )
voluptuous.MultipleInvalid: expected UUID for dictionary value @ data['foo']
>>> s( "foo": uuid.uuid4() )
'foo': UUID('60b6d6c4-e719-47a7-8e2e-b4a4a30631ed')
And that's not what is wanted here.
That mechanism is really neat to transform, for example, strings to timestamps.
>>> import datetime
>>> from voluptuous import Schema
>>> def Timestamp(value):
... return datetime.datetime.strptime(value, "%Y-%m-%dT%H:%M:%S")
...
>>> s = Schema( "foo": Timestamp )
>>> s( "foo": '2015-03-03T12:12:12' )
'foo': datetime.datetime(2015, 3, 3, 12, 12, 12)
>>> s( "foo": '2015-03-03T12:12' )
voluptuous.MultipleInvalid: not a valid value for dictionary value @ data['foo']
Recursive schemas
So far,
Voluptuous has one limitation so far: the ability to have recursive
schemas. The simplest way to circumvent it is by using another function as an
indirection.
>>> from voluptuous import Schema, Any
>>> def _MySchema(value):
... return MySchema(value)
...
>>> from voluptuous import Any
>>> MySchema = Schema( "foo": Any("bar", _MySchema) )
>>> MySchema( "foo": "foo": "bar" )
'foo': 'foo': 'bar'
>>> MySchema( "foo": "foo": "baz" )
voluptuous.MultipleInvalid: not a valid value for dictionary value @ data['foo']['foo']
Usage in REST API
I started to use
Voluptuous to validate data in a the REST API provided by
Gnocchi. So far it has been a really good tool,
and we've been able to
create a complete REST API
that is very easy to validate on the server side. I would definitely recommend
it for that. It blends with any Web framework easily.
One of the upside compared to solution like
JSON Schema, is the ability to create or re-use your
own custom data types while converting values at validation time. It is also
very Pythonic, and extensible it's pretty great to use for all of that. It's
also not tied to any serialization format.
On the other hand, JSON Schema is language agnostic and is serializable itself
as JSON. That makes it easy to be exported and provided to a consumer so it can
understand the API and validate the data potentially on its side.
The C1 Server
Scaleway started selling ARM based hosted server in April. I've intended to blog about this for a while, since it was time to upgrade from wheezy to jessie was timely, why not switch provider from an X86 based to ARM one at the same time? In many ways scaleway node is opposite to what "Enterprise ARM" people are working on. Each server is based on an oldish ARMv7 Quad-Core Marvell Armada XP, instead of a brand new 64-bit ARMv8 cpu. There is no UEFI, ACPI or any other "industry standards" involved, just a smooth web interface and a command line tool to manage your node(s). And the node is yours, it's not shared with others with virtualization. The picture above is a single node, which is stacked with 911 other nodes into a single rack. This week, the C1 price was dropped to a very reasonable 2.99 per month, or 0.006 per hour. Software runs on hardware, news at 11The performance is more than enough for my needs - shell, email and light web serving. dovecot, postfix, irssi and apache2 are just an apt-get away. Anyone who says you need x86 for Linux servers is forgetting that Linux software is open source, and if not already available, can be compiled to any architecture with little effort. Thus the migration pains were only because I chose to modernize configuration of dovecot and friends. Details of the new setup shall be left for another post.
Debian LTS
August was the fourth month I contributed to 
Since quite some time I am using 
So I decided to rename all the local folders to the scheme described in the above guide. At the end I want to have a folder layout as follows:
Once again time for the monthly upstream 
I am not quite sure about the text on the back, but guess it will
do. I picked three quotes from the official site for the book, and
hope it will work to trigger the interest of potential readers. The
Norwegian cover will look the same, but with the texts and bar code
replaced with the Norwegian version.
The book is very close to being ready for publication, and I expect
to upload the final draft to Lulu in the next few days and order a
final proof reading copy to verify that everything look like it should
before allowing everyone to order their own copy of Free Culture, in
English or Norwegian Bokm l. I'm waiting to give the the productive
proof readers a chance to complete their work.
I m receiving more requests for upload accounts to the 
This is probably because schroot spawns a login shell, and systemd considers it belonging to a different process group. So, I had to launch the command systemctl set-property user.slice CPUShares=512, so all processes belonging to the user.slice will receive the same share of the debomatic ones. I consider this a workaround, I m open to suggestions how to properly solve this issue 
What happened in the 
My monthly report covers a large part of what I have been doing in the free software world. I write it for 
For the concrete work I did during DebConf, I can only claim two schroot uploads to fix the lack of support of the new overlay filesystem that replaces aufs in the official Debian kernel, and some Distro Tracker work (fixing an issue that some people had when they were logged in via Debian s SSO).
While the numerous discussions I had during DebConf can t be qualified as work , they certainly contribute to build up work plans for the future:
As a Kali developer, I attended multiple sessions related to derivatives (notably the 
The new features are:
Selection of images: Up to now images for the widget were selected at random. The current version allows selecting images either at random (the default as before), but also in ascending or descending order by various criteria (uploaded, availability time, id, name, etc). With this change it is now possible to always display the most recent image(s) from a gallery.
I've been writing ELF parser for fun using C++ templates to see how much I can simplify. I've been reading bionic loader code enough these days that I already know how it would look like in C and gradually converted into C++ but it's nevertheless fun to have a pet project that kind of grows.
The 
It's been over 2 years since I 
