For quite some time now, I've been following
Rust, a new(ish) programming language by Mozilla. It's been around for a while, but recently it's been attracting a lot of buzz. Around the new year, I finally decided to take a deeper look into the language. I find that the best way to learn a new programming language is to simply dive in and try to build something nontrivial. At that point I'd also been interested in learning more about the specifics behind the new HTTP/2 spec, so I decided to combine the two... The result is a (work in progress) client library for HTTP/2 written in Rust, which can be found at this GitHub repository:
https://github.com/mlalic/solicit What follows is a description of what I aim to achieve with the library, as well as an overview of its current design. If that doesn't interest you, you can just check out the GitHub repo for some examples or the source code itself.
Goals Given that there are already some fairly popular HTTP libraries for Rust (
hyper leading the way), I decided that I wouldn't aim to replace them. Rather, the idea is to focus on the lower levels of the HTTP/2 protocol what is essentially another transport protocol and expose an API that allows those libraries to use an HTTP/2 connection to send requests and receive responses. HTTP/2 doesn't change the semantics of the protocol, so once any HTTP/1.1 library gets the raw headers and body, it should be able to represent the response regardless of the fact that it came on an HTTP/2 connection. The same holds for sending requests: only the representation of the request changes, not its content. There are some minor differences, such as HTTP/2 doing away with the reason phrases being part of the response, but those were only meant to be human readable and "MUST NOT" have had any influence on the semantics of the response. Therefore, the primary goal for the library is to provide an API for using an HTTP/2 connection on the lowest level, but making it easy for clients to customize or augment any particular step of handling a response, without having to know the details of functionality below the layer they want to customize. As an example, if one wanted to build a client that sends out progress notifications for downloads, it wouldn't be good if they needed to muck about with raw HTTP/2 frames ideally, they'd only need to implement a new way to handle incoming data chunks for streams. The same applies for implementing a client that streams the download into a file (instead of saving it in memory) or one that sends the chunks to a different component for processing as soon as they're received (e.g. parsing HTML). Since the Rust standard library only exposes blocking IO primitives, I decided to base this library on blocking IO, as well. However, the goal was to make it flexible enough so that clients that would like to allow multiple requests to be sent concurrently on the same HTTP/2 connection can be built, but without the library itself spawning any threads. Finally, on a less conceptual level (and more of an implementation level), the code should have extensive test coverage.
Design The library is split into several layers, each of which provides services to the one above it. A brief description of each layer follows.
Transport Layer This layer provides the API for raw byte stream communication with the peer. Essentially, it's a wrapper around the particular transport protocol that is used for the HTTP/2 connection. As the HTTP/2 spec allows the transport protocol to be either a cleartext TCP connection or a TLS-protected one, those are the ones that need to be implemented at this level. The API of the layer is represented by a single trait
TransportStream that simply requires
io::Read and
io::Write to be implemented and (for now) provides a single convenience method based on the underlying
io::Read implementation. For a clear-text TCP connection, the only thing that was required to implement it was
one line of code:
impl TransportStream for TcpStream
HTTP/2 Connection Layer The connection layer provides an API for reading and writing to a raw HTTP/2 connection. This means reading and writing HTTP/2 frames, not arbitrary bytes. Each HTTP/2 frame has to follow the same
basic structure(a 9-byte header followed by a sequence of byte representing the payload of the frame). This is represented by the
RawFramestruct. Each HTTP/2 frame type can have its own specific format for the content of the payload and assign meaning to the flags found in the header. Therefore, each frame needs to have its own implementation that can parse a
RawFrame or return an error if the raw frame does not represent a valid frame of that type. Some common methods that each frame type implementation needs to support are given by the
Frametrait that must be implemented by the implementations. Finally, the
HttpConnectionstruct handles reading the byte stream from the underlying
TransportStream instance, correctly delineating frames and parsing them into the appropriate frame type implementation (by delegating to the correct
Frame implementation based on the frame type found in the header). Obviously, varying the implementation of the transport layer is a matter of choosing the appropriate
TransportStream implementation. This includes using mock implementations for testing the
HttpConnection itself.
HTTP/2 Client Connection The
ClientConnectionbuilds upon the
HttpConnection by using it to expose functionality that is specific to the client side of an HTTP/2 connection. This includes sending requests and reading responses. The struct takes care of creating frames that need to be sent for a given request (including HPACK header encoding) and shipping them off to the server (using the underlying
HttpConnection instance). As for handling a response, the
ClientConnection expects to get an instance of a struct implementing the
Sessiontrait when constructed. This trait essentially defines a number of callbacks: methods that will be called on this instance when the corresponding event occurs in the
ClientConnection. Therefore, the
ClientConnection understands the semantics of the particular frames that it reads (e.g. performs the decoding of the
HEADERS frames' payloads, passing the raw data received in a
DATA frame to the session, etc.). However, it does
not know anything about streams; it only passes the stream ID up to the session layer, without thinking about whether the stream is open, valid, etc.
HTTP/2 Session As mentioned in the previous section, each
ClientConnection requires an instance of a struct implementing the
Sessiontrait to be passed to it at construction time. The
Session trait defines a number of callbacks that are invoked in the appropriate moments during the processing of frames read on a client connection. For example,
new_headers (takes a list of heaers and a stream on which they were received) or
new_data_chunk (a buffer and a stream on which the data arrived). The session is in charge of handling these events as they arise. It is up to it to decide what (if anything) should be done with the data that it receives. There is a default implementation of the
Session trait (unimaginatively named
DefaultSession), which provides an implementation that tracks which streams are currently open and delegates the processing of new events on particular streams to instances of a
Streamtrait. This allows for client implementations to simply vary the implementation of a
Stream that they provide to the
DefaultSession in order to change how a particular part of a response is handled, but let the
DefaultSession take care of tracking which streams are open and valid. Different implementations of a
Session still can exist, if the particular client needs more fine-grained customization over how these events are handled, but for most purposes the default session, along with
Stream customization should probably be enough.
HTTP/2 Stream Again, as mentioned previously, the
DefaultSessionimplementation of the
Session trait relies on a
Streamtrait providing the handling for events arising on individual streams. This level should be the one that gets customized most often in alternate implementations. A default stream implementation
DefaultStreamhandles the messages by building up a response in-memory. Different implementations could instead send progress notifications on an associated channel, for instance.
HTTP/2 Client The
final layeris that of a client that exposes methods that would usually be expected for a client HTTP library: sending requests, obtaining responses. This layer is not the primary focus, though, and it only exposes two simple implementations that showcase how the lower level APIs can be used to build a client. The second purpose it fulfills is as a "sanity check" that the APIs actually do allow the goals to be met. Therefore, one of the clients allows for requests to be sent from concurrently running threads and the responses to those requests to be obtained asynchronously (in the form of a
Future<Response>). However, the responses are basic and very raw: they are stored fully in memory (not available until they've been read until the end) and both the body, as well as the headers, exposed as raw byte sequences. As an example how
this clientcan be used (as well as the simpler
single-threaded version), check out the
Examplessection in the GitHub repository's README file.
Future Work
Refactoring APIs As the project was started as a Rust learning exercise, the initial APIs are slightly awkward and fairly inefficient to work with. For example, the
solicit::hpackmodule requires headers to be represented as a tuple of
Vec<u8>, which means that whenever we need to encode a header list, even though their values might be (statically known) byte sequence literals, such as
(b":method", b"GET"), we need to make a bunch of
to_vec calls on the
&[u8]s before passing them to the encoder, leading not only to a really ugly and tedious way of obtaining the encoded representation of the headers, but also to a lot of unnecessary allocations. Another API that will see some refactoring and improvement is the
TransportStream, as well as the
HttpConnection in order to make it possible to have separate threads perform reads and writes on the connection. Currently, the socket is owned by the
HttpConnection instance and all writes and reads require a
&mut reference, making this impossible. This change would tackle this by embracing Rust's new
try_clone method on the
TcpStream into the
TransportStream and refactoring the
HttpConnection to allow splitting it into a write and read handle.
HTTP/2 Features As already mentioned, only a small subset of the full HTTP/2 spec is currently implemented, so, obviously, implementing the rest of it falls under the most important future work. Some near- to mid-term priorities that will be tackled are:
- TLS-backed connections: a new
TransportStream implementation that wraps a TCP socket into a TLS wrapper. This will probably rely on OpenSSL bindings. - Support for all HTTP/2 frames: only the three vital frames are implemented so far; other frames are a prerequisite to implementing properly closing streams, handling closed streams etc.
- Signaling stream- and connection-level errors to the peer
- Flow control: right now this aspect is completely ignored
- Server push: the idea is that multithreaded clients could handle pushed responses as soon as they arive, whereas single-threaded ones would be allowed to fetch a list of them every so often.
Obviously, those aren't all the features of HTTP/2, but it's a rough list of things I'm planning on working on myself as the next steps.
Integration With an (existing?) HTTP Library I would like it if in the end the library or at least some of its pieces ended up being used in the scope of one of the fully featured HTTP libraries. If there's any interest in this, I wouldn't mind putting in some extra legwork to help make it happen.
Conclusion While I'm certain that there can be better, more efficient, more elegant implementations, I've had a lot of fun working on this so far (learning new things is always fun!) and if nothing else I'll continue in order to gain the deepest possible understanding of HTTP/2. For anyone that might have read all the way through, thanks for reading and I'm open for any sort of feedback!
Like each month, here comes a report about the work of paid contributors to Debian LTS.
Individual reports
In February, 58 work hours have been equally split among 4 paid contributors. Their reports are available:
Thanks to our sponsors
The new sponsors of the month are in bold.
I'm using 
A new minor release of 
ApologyAs a twisted turn of fate, I need to start my platform with an apology. Last year, I