Streaming JSON objects with python
Dear Lazyweb,
Suppose I'd like to create an HTTP service that gives you a looong list of JSON
objects. For example,
one JSON object for every package in
Debian.
One way I could do that is to just transfer a big, dozens of megabytes JSON
array with one element per package. With the standard
json module in python,
this requires the client to buffer all the data in memory for decoding, as the
JSON result would be something like this:
[
info for package 1 ,
info for package 2 ,
...dozens of megabytes...
info for package 30000 ,
...
]
However, it would be reasonable to engineer things so that the client can
process packages one at a time, as they are produced. This can dramatically
reduce client memory usage, as well as remove a large latency between the
start of the request and the start of processing.
One trivial idea would be to just send JSON objects one after the other, and
sort of call load multiple
times.
It doesn't work:
import json
from cStringIO import StringIO
a = StringIO()
json.dump( 1:3 , a)
json.dump( 2:4 , a)
a.reset()
print a.getvalue()
print json.load(a)
gives:
$ python z1.py
"1": 3 "2": 4
Traceback (most recent call last):
File "z1.py", line 8, in <module>
print json.load(a)
File "/usr/lib/python2.6/json/__init__.py", line 267, in load
parse_constant=parse_constant, **kw)
File "/usr/lib/python2.6/json/__init__.py", line 307, in loads
return _default_decoder.decode(s)
File "/usr/lib/python2.6/json/decoder.py", line 322, in decode
raise ValueError(errmsg("Extra data", s, end, len(s)))
ValueError: Extra data: line 1 column 8 - line 1 column 16 (char 8 - 16)
And the
json module documentation
does not provide any way of telling the "stream" decoder to just stop parsing
after the first valid object.
An alternative can be to use
chunked transfer
encoding and transfer
each JSON object in a separate HTTP chunk. It would add the overhead of a
little HTTP header per JSON object, which could be significant when generating
lots of small JSON objects, but it would be an implementation that strictly
follows HTTP and JSON standards to the letter in a clean way.
Except:
- How do you stream HTTP chunks in WSGI? The WSGI
specification's only mention
of chunks is that if you produce a result as a generator of "chunks", they
are just sent one after the other (effectively, concatenated) in the
response stream.
- How do you read and process one chunk at a time with urllib/urllib2?
I searched on Google but I only found people in tears.
Another alternative is to wrap the JSON objects in some custom container
protocol, but then we are not talking standards anymore.
Is this all you can do with the standard python library? Can this only be done
by adding a dependency on
a C library with two
obscure
separately
maintained bindings, or by writing my own
JSON parser?
Ideas? Please mail enrico@enricozini.org and I'll add them here.
Updates
Wouter van Heyst mentions that he heard of
ijson, which indeed has
my exact use case in the synopsis, but after some research looks like a
third
binding for
yajl.
Aigars Mahinovs suggests using
YAML
instead of JSON. Indeed YAML supports streaming very well (it's even already
supported in
DDE, but last time I tried to do
anything serious with it,
PyYaml was
slooow. Of course,
this being the Python ecosystem, There Are Lots Other Ways To Do It And You
Shall Be Patronised For Not Chosing The Proper One, but Debian only has PyYaml.
Another itch with YAML is that there is no YAML parser in the standard python
library, which requires clients to add an extra dependency.
Jyrki Pulliainen has so far mailed my favourite solution: disable formatting in
the encoder and separate JSON objects with newlines:
we built a streaming JSON app as a part of our platform. However, we
didn't stream a single JSON object as that was a bit cumbersome. When
having syntax like:
[
<json-obj1>,
<json-obj2>,
....
<json-objN>
]
The reader has some problems. Should it read the whole stream to find
the closing bracket ] even if the stream would me tens of megabytes?
Should it try to split on , and parse single objects? What if the
objects contain , so should there be some regexp magic?
We ended up streaming separate JSON objects separated by newlines
(\n). This way the reader can read until hitting \n or end of stream
and then split the result on newlines. The new stream looked like
this:
<json-obj1>\n
<json-obj2>\n
...
<json-objN>\n
This way we could efficiently stream hundreds of thousands of JSON
objects from server to client without reserving a load of memory. For
the WSGI part how to do this, the best would probably to do a
generator that would yield a single JSON object with the newline (we
didn't use WSGI back then). Reading can be done using any reader
capable of handling streams. We used pycurl for reading, but I can't
really recommend it as it is way too complicated for a simple task
like this :p
Hopefully this helps!
This means that if the client has no access to a JSON reader that supports a
stream of concatenated toplevel objects, they can still trivially code it on
top of any JSON library. This means that working clients can be written based
just on Squeeze's python standard library. That's what I'm going to do; I'll
probably implement it as a separate 'jsons' output format for DDE, because
producing a JSON response with multiple toplevel obejcts somehow breaks the
JSON standard.
Thanks to all who provided this excellent information!
A friend just pointed me to Liberating Knowledge: A Librarian's Manifesto for Change by Barbara Fister, an academic librarian who also happens to write mystery novels. She has an excellent perspective on the meaning of libraries, and the tradeoffs involved with the current societal trend toward privatizing knowledge through so-called "intellectual property" regulations. In a great critique of the passivity of academia and libraries in the face of attempts at intellectual enclosure by private corporations, she writes: 
A couple of days ago, an email arrived in which I am kindly asked to
provide my PayPal coordinates. And before you dear reader start to laugh,
it was actually perfectly genuine and came from a very happy user of the
the release team has 
I have finally reached a decision regarding my involvement in the Debian Ruby packaging efforts. I have decided to stop. This has been a very hard decision to make. I have invested huge amounts of time in that work over the years. I still love the language, and will continue to use it on a daily basis for my own developments. I still hope that it will succeed. I know that some people will be disappointed by that decision (and that others will think your work was useless anyway, people should use RVM and rubygems ).
But I also know that I won t be able to push for all the required changes alone. I just don t have the time, nor the motivation. For the record, here are the changes I would have liked to see in the Ruby community.
The core Ruby development community should mature.
It would be really neat if there were some tool which recorded which dotfiles an application read, used, or created.
As an example emacs uses .emacs, but won't create it. However firefox will create and fill ~/.mozilla if it isn't present, and links will create ~/.links2.
What would we do with that data? I'm not sure off the top of my head, but I think it is interesting to collect regardless. Perhaps a simple tool such as apt-file to download the data and let you search:
Last night was the last time I came back from a pub with my clothes
stinking due to tobacco smoke. The Spanish congress has finally approved a
real anti-smoking law which will ban smoking on public areas, with no
exceptions or ways to workaround the ban. Starting on January 2nd, the
Spanish state will be a smoke-free region (or mostly, it seems it'll be
permitted in open-air events like football stadiums or bullrings, and I
don't think that will be a great problem for me, specially the latter).
For years, my intolerance to smoke has been increasing and I'm really
expectant to see the benefits of this law in my habits. After more than 30
years dealing with smoke around us, it's our turn now. I've been speaking
to a few barmans. In general they seem worried this will affect their
business, but I can't see how it will. Spain has a big culture for having
mid-morning almuerzos in bars, and people are not going to give
that up due to not being able to smoke. They will just do it after they
get out, not during the coffee, and that's it. The barmans of the two bars
I visit most are non-smokers, but have to breath the smoke of hundreds of
cigarretes every day, and can't do anything about it. Until tomorrow, when
this will end and everyone will have a right to breath better air. I hope
this kind of legislation continues to be adopted throughout Europe, because
the FOSDEM welcome party is probably the next smoke horror I'll have to face
soonish. :)
I ve been mostly active in the OCaml team where we designed a tool to compute automatically the dependencies between OCaml packages, called 
But, since I m not a magician yet, I d stick to more realistic plans:
In such cases, most people use to say "never saw this" and such common
sentences.
However, really, this time this is true: it's now 30 years since I live
in the Paris area, a place where climate is not known to be a snowy
one: Paris and its region are not that far from the Channel and the
influence of Atlantic Ocean is supposedly important. And, really, I
never saw this.
For such reasons, here, winters are not really "continental" and we
often have few snow during an entire winter, if any. Indeed, during
the 200* years, we had several winters without any snow at all.
Last year was already different: we had 3 snow episodes. A quite big
one on Dec 17th (about 15cm in the western area of the region),
another in mid-January where snow lasted for about one week because of
very low temperatures (below -10 C) and a final one in February.
Still, the 2010-2011 winter is something I never saw before. In less
than one month, we had 6 different snow episodes: Nov 28th (roughly),
Dec 8th, Dec 17th, Dec 18th, Dec 20th and now Dec 22th. My area (the
western part of the region) was hardly hit with 20cm on Dec 8th and
about 15cm on Dec 20th and 22th....up to a cumulated 80cm height.
Really something unusual. And what that happens in a region where
about 12M people live, sometimes at rush hours, you can expect a giant
mess (and a giant mess it was on Dec 8th).
During these weeks, I personnally experimented my new way of
commuting: run/train/run, which means running the 3km desert country road
from my house to the train station, catching a train, then running the
final 4km in suburbs close to Paris. Both runs do actually replace a
standard bus ride. This prove to be convenient as buses are often
stopped during snow episodes (hey, there are hills in the Paris
area!).
That made things that I'll really remember for the future: this
morning's run in the country with 15cm of fresh snow, without any
single step mark (except the rabbit or wild pig ones), with the light
of my head lamp, no noise and an entirely white landscape....is
special and nearly unique when you live only 30km away from the town.
Nevertheless, that might prefigure what we will experience in the
upcoming years or dozen of years: a generally much colder winter over
western Europe, much closer to what Central Europe always
had. Personnally, I like that, but we probably need to prepare our
facilities to such events: last week-end in UK, Belgium, France,
Germany, etc. was really a giant mess (Heathrow airport closed, Roissy
nearly closed, Brussels flooded with derouted planes, Frankfurt stuck,
Eurostars stopped, gee). Is that climate change (as a consequence of
Gulf Stream vanishing) or just a collection of random events? We'll
see.
Cube-mate and friend, 
(This is part of a series of posts on 
A few months ago, 