Paul Tagliamonte: designing arf, an sdr iq encoding format
hz.tools will be tagged
#hztools.Want to jump right to the draft? I'll be maintaining ARF going forward at /draft-tagliamonte-arf-00.txt.
ARF (Archive of RF)
A few years ago, I needed to finally solve some of these shortcomings and tried
to see if a new format would stick. I sat down and wrote out my design goals
before I started figuring out what it looked like.
First, whatever I come up with must be capable of being streamed and processed
while being streamed. This includes streaming across the network or merely
written to disk as it s being created. No post-processing required. This is
mostly an artifact of how I ve built all my tools and how I intereact with my
SDRs. I use them extensively over the network (both locally, as well
as remotely by friends across my wider
lan). This decision sometimes even
prompts me to do some crazy things from time
to time.
I need actual, real support for multiple IQ channels from my multi-channel SDRs
(Ettus, Kerberos/Kracken SDR, etc) for playing with things like
beamforming.
My new format must be capable of storing
multiple streams in a single capture file, rather than a pile of files in
a directory (and hope they re aligned).
Finally, metadata must be capable of being stored in-band. The initial set of
metadata I needed to formalize in-stream were Frequency Changes and
Discontinuities. Since then, ARF has grown a few more.
After getting all that down, I opted to start at what I thought the simplest
container would look like,
TLV
(tag-length-value) encoded packets. This is a fairly well trodden path,
and used by a bunch of existing protocols
we
all
know
and
love.
Each ARF file (or stream) was a set of
encoded packets (sometimes called data units in other specs). This means that
unknown packet types may be skipped (since the length is included) and
additional data can be added after the existing fields without breaking
existing decoders.
Heads up!
Once this is posted, I'm not super likely to update this page. Once this
goes out, the latest stable copy of the ARF spec is maintained at
draft-tagliamonte-arf-00.txt.
This page may quickly become out of date, so if you're actually interested in
implementing this, I've put a lot of effort into making the draft
comprehensive, and I plan to maintain it as I edit the format.
Unlike a traditional TLV structure, I opted to add flags to the top-level
packet. This gives me a bit of wiggle room down the line, and gives me a
feature that I like from ASN.1 a critical bit. The critical bit indicates
that the packet must be understood fully by implementers, which allows future
backward incompatible changes by marking a new packet type as critical. This
would only really be done if something meaningfully changed the interpretation
of the backwards compatible data to follow.
| Flag | Description |
| 0x01 | Critical (tag must be understood) |
tag field. This tag indicates how the contents of the
value field should be interpreted.
| Tag ID | Description |
| 0x01 | Header |
| 0x02 | Stream Header |
| 0x03 | Samples |
| 0x04 | Frequency Change |
| 0x05 | Timing |
| 0x06 | Discontinuity |
| 0x07 | Location |
| 0xFE | Vendor Extension |
00, // tag (0; no subpacket is 0 yet)
00, // flags (0; no flags)
00, 00 // length (0; no data)
// data would go here, but there is none
UUID
This field represents a globally unique idenfifer, as defined by RFC 9562, as
16 raw bytes.
Frequency
Data encoded in a Frequency field is stored as microhz (1 Hz is stored as
1000000, 2 Hz is stored as 2000000) as an unsigned 64 bit integer. This has a
minimum value of 0 Hz, and a maximum value of 18446744073709551615 uHz, or just
above 18.4 THz. This is a bit of a tradeoff, but it s a set of issues that I
would gladly contend with rather than deal with the related issues with storing
frequency data as a floating point value downstream. Not a huge factor, but as
an aside, this is also how my current generation SDR processing code (sparky)
stores Frequency data internally, which makes conversion between the two
natural.
IQ samples
ARF supports IQ samples in a number of different formats. Part of the idea here
is I want it to be easy for capturing programs to encode ARF for a specific
radio without mandating a single iq format representation. For IQ types with
a scalar value which takes more than a single byte, this is always paired
with a Byte Order field, to indicate if the IQ scalar values are little or
big endian.
sparky)
stores Frequency data internally, which makes conversion between the two
natural.
IQ samples ARF supports IQ samples in a number of different formats. Part of the idea here is I want it to be easy for capturing programs to encode ARF for a specific radio without mandating a single iq format representation. For IQ types with a scalar value which takes more than a single byte, this is always paired with a Byte Order field, to indicate if the IQ scalar values are little or big endian.
| ID | Name | Description |
| 0x01 | f32 | interleaved 32 bit floating point scalar values |
| 0x02 | i8 | interleaved 8 bit signed integer scalar values |
| 0x03 | i16 | interleaved 16 bit signed integer scalar values |
| 0x04 | u8 | interleaved 8 bit unsigned integer scalar values |
| 0x05 | f64 | interleaved 64 bit floating point scalar values |
| 0x06 | f16 | interleaved 16 bit floating point scalar values |
Header
Each ARF file must start with a specific Header packet. The header contains
information about the ARF stream writ large to follow. Header packets are
always marked as critical .
magic
flags
start
guid
site guid
#st
In order to help with checking the basic parsing and encoding of this format,
the following is an example header subpacket (when encoded or decoded this
will be found inside an ARF packet as described above) which should parse
without error, with known values.
00, 00, 00, fa, de, dc, ab, 1e, // magic
00, 00, 00, 00, 00, 00, 00, 00, // flags
18, 27, a6, c0, b5, 3b, 06, 07, // start time (1740543127)
// guid (fb47f2f0-957f-4545-94b3-75bc4018dd4b)
fb, 47, f2, f0, 95, 7f, 45, 45,
94, b3, 75, bc, 40, 18, dd, 4b,
// site_id (ba07c5ce-352b-4b20-a8ac-782628e805ca)
ba, 07, c5, ce, 35, 2b, 4b, 20,
a8, ac, 78, 26, 28, e8, 05, ca
Stream Header
Immediately after the arf Header, some number of Stream Headers
follow. There must be exactly the same number of Stream Header packets as are
indicated by the num streams field of the Header. This has the nice effect of
enabling clients to read all the stream headers without requiring buffering of
unread packets from the stream.
id
flags
fmt
bo
rate
freq
guid
site
In order to help with checking the basic parsing and encoding of this format,
the following is an example stream header subpacket (when encoded or decoded
this will be found inside an ARF packet as described above) which should parse
without error, with known values.
00, 01, // id (1)
00, 00, 00, 00, 00, 00, 00, 00, // flags
01, // format (float32)
01, // byte order (Little Endian)
00, 00, 01, d1, a9, 4a, 20, 00, // rate (2 MHz)
00, 00, 5a, f3, 10, 7a, 40, 00, // frequency (100 MHz)
// guid (7b98019d-694e-417a-8f18-167e2052be4d)
7b, 98, 01, 9d, 69, 4e, 41, 7a,
8f, 18, 16, 7e, 20, 52, be, 4d,
// site_id (98c98dc7-c3c6-47fe-bc05-05fb37b2e0db)
98, c9, 8d, c7, c3, c6, 47, fe,
bc, 05, 05, fb, 37, b2, e0, db,
Samples
Block of IQ samples in the format indicated by this stream s format and
byte_order field sent in the related Stream Header.
id
iq samples
In order to help with checking the basic parsing and encoding of this format,
the following is an samples subpacket (when encoded or decoded
this will be found inside an ARF packet as described above). The IQ values
here are notional (and are either 2 8 bit samples, or 1 16 bit sample,
depending on what the related Stream Header was).
01, // id
ab, cd, ab, cd, // iq samples
Frequency Change
The center frequency of the IQ stream has changed since the
Stream Header or last Frequency Change
has been sent. This is useful to capture IQ streams that are jumping
around in frequency during the duration of the capture, rather than
starting and stopping them.
id
frequency
In order to help with checking the basic parsing and encoding of this format,
the following is a frequency change subpacket (when encoded or decoded
this will be found inside an ARF packet as described above).
01, // id
00, 00, b5, e6, 20, f4, 80, 00 // frequency (200 MHz)
Discontinuity
Since the last Samples packet for this stream, samples have been dropped
or not encoded to this stream. This can be used for a stream that has
dropped samples for some reason, a large gap (radio was needed for something
else), or communicating iq snippits .
id
In order to help with checking the basic parsing and encoding of this format,
the following is a discontinuity subpacket (when encoded or decoded this will
be found inside an ARF packet as described above).
01, // id
Location
Up-to-date location as of this moment of the IQ stream, usually from a GPS.
This allows for in-band geospatial information to be marked in the IQ stream.
This can be used for all sorts of things (detected IQ packet snippits aligned
with a time and location or a survey of rf noise in an area)
The sys field indicates the Geodetic system to be used for the provided
latitude, longitude and elevation fields. The full list of supported
geodetic systems is currently just WGS84, but in case something meaningfully
changes in the future, it d be nice to migrate forward.
Unfortunately, being a bit of a coward here, the accuracy field is a bit of a
cop-out. I d really rather it be what we see out of kinematic state estimation
tools like a kalman filter, or at minimum, some sort of ellipsoid. This is
neither of those - it s a perfect sphere of error where we pick the largest
error in any direction and use that. Truthfully, I can t be bothered to model
this accurately, and I don t want to contort myself into half-assing something
I know I will half-ass just because I know better.
00, 00, 00, fa, de, dc, ab, 1e, // magic
00, 00, 00, 00, 00, 00, 00, 00, // flags
18, 27, a6, c0, b5, 3b, 06, 07, // start time (1740543127)
// guid (fb47f2f0-957f-4545-94b3-75bc4018dd4b)
fb, 47, f2, f0, 95, 7f, 45, 45,
94, b3, 75, bc, 40, 18, dd, 4b,
// site_id (ba07c5ce-352b-4b20-a8ac-782628e805ca)
ba, 07, c5, ce, 35, 2b, 4b, 20,
a8, ac, 78, 26, 28, e8, 05, ca
num streams field of the Header. This has the nice effect of
enabling clients to read all the stream headers without requiring buffering of
unread packets from the stream.
00, 01, // id (1)
00, 00, 00, 00, 00, 00, 00, 00, // flags
01, // format (float32)
01, // byte order (Little Endian)
00, 00, 01, d1, a9, 4a, 20, 00, // rate (2 MHz)
00, 00, 5a, f3, 10, 7a, 40, 00, // frequency (100 MHz)
// guid (7b98019d-694e-417a-8f18-167e2052be4d)
7b, 98, 01, 9d, 69, 4e, 41, 7a,
8f, 18, 16, 7e, 20, 52, be, 4d,
// site_id (98c98dc7-c3c6-47fe-bc05-05fb37b2e0db)
98, c9, 8d, c7, c3, c6, 47, fe,
bc, 05, 05, fb, 37, b2, e0, db,
Samples
Block of IQ samples in the format indicated by this stream s format and
byte_order field sent in the related Stream Header.
id
iq samples
In order to help with checking the basic parsing and encoding of this format,
the following is an samples subpacket (when encoded or decoded
this will be found inside an ARF packet as described above). The IQ values
here are notional (and are either 2 8 bit samples, or 1 16 bit sample,
depending on what the related Stream Header was).
01, // id
ab, cd, ab, cd, // iq samples
Frequency Change
The center frequency of the IQ stream has changed since the
Stream Header or last Frequency Change
has been sent. This is useful to capture IQ streams that are jumping
around in frequency during the duration of the capture, rather than
starting and stopping them.
id
frequency
In order to help with checking the basic parsing and encoding of this format,
the following is a frequency change subpacket (when encoded or decoded
this will be found inside an ARF packet as described above).
01, // id
00, 00, b5, e6, 20, f4, 80, 00 // frequency (200 MHz)
Discontinuity
Since the last Samples packet for this stream, samples have been dropped
or not encoded to this stream. This can be used for a stream that has
dropped samples for some reason, a large gap (radio was needed for something
else), or communicating iq snippits .
id
In order to help with checking the basic parsing and encoding of this format,
the following is a discontinuity subpacket (when encoded or decoded this will
be found inside an ARF packet as described above).
01, // id
Location
Up-to-date location as of this moment of the IQ stream, usually from a GPS.
This allows for in-band geospatial information to be marked in the IQ stream.
This can be used for all sorts of things (detected IQ packet snippits aligned
with a time and location or a survey of rf noise in an area)
The sys field indicates the Geodetic system to be used for the provided
latitude, longitude and elevation fields. The full list of supported
geodetic systems is currently just WGS84, but in case something meaningfully
changes in the future, it d be nice to migrate forward.
Unfortunately, being a bit of a coward here, the accuracy field is a bit of a
cop-out. I d really rather it be what we see out of kinematic state estimation
tools like a kalman filter, or at minimum, some sort of ellipsoid. This is
neither of those - it s a perfect sphere of error where we pick the largest
error in any direction and use that. Truthfully, I can t be bothered to model
this accurately, and I don t want to contort myself into half-assing something
I know I will half-ass just because I know better.
01, // id
ab, cd, ab, cd, // iq samples
01, // id
00, 00, b5, e6, 20, f4, 80, 00 // frequency (200 MHz)
Discontinuity
Since the last Samples packet for this stream, samples have been dropped
or not encoded to this stream. This can be used for a stream that has
dropped samples for some reason, a large gap (radio was needed for something
else), or communicating iq snippits .
id
In order to help with checking the basic parsing and encoding of this format,
the following is a discontinuity subpacket (when encoded or decoded this will
be found inside an ARF packet as described above).
01, // id
Location
Up-to-date location as of this moment of the IQ stream, usually from a GPS.
This allows for in-band geospatial information to be marked in the IQ stream.
This can be used for all sorts of things (detected IQ packet snippits aligned
with a time and location or a survey of rf noise in an area)
The sys field indicates the Geodetic system to be used for the provided
latitude, longitude and elevation fields. The full list of supported
geodetic systems is currently just WGS84, but in case something meaningfully
changes in the future, it d be nice to migrate forward.
Unfortunately, being a bit of a coward here, the accuracy field is a bit of a
cop-out. I d really rather it be what we see out of kinematic state estimation
tools like a kalman filter, or at minimum, some sort of ellipsoid. This is
neither of those - it s a perfect sphere of error where we pick the largest
error in any direction and use that. Truthfully, I can t be bothered to model
this accurately, and I don t want to contort myself into half-assing something
I know I will half-ass just because I know better.
01, // id
sys field indicates the Geodetic system to be used for the provided
latitude, longitude and elevation fields. The full list of supported
geodetic systems is currently just WGS84, but in case something meaningfully
changes in the future, it d be nice to migrate forward.
Unfortunately, being a bit of a coward here, the accuracy field is a bit of a
cop-out. I d really rather it be what we see out of kinematic state estimation
tools like a kalman filter, or at minimum, some sort of ellipsoid. This is
neither of those - it s a perfect sphere of error where we pick the largest
error in any direction and use that. Truthfully, I can t be bothered to model
this accurately, and I don t want to contort myself into half-assing something
I know I will half-ass just because I know better.
| System | Description |
| 0x01 | WGS84 - World Geodetic System 1984 |
00, 00, 00, 00, 00, 00, 00, 00, // flags
01, // system (wgs84)
3f, f3, be, 76, c8, b4, 39, 58, // latitude (1.234)
40, 02, c2, 8f, 5c, 28, f5, c3, // longitude (2.345)
40, 59, 00, 00, 00, 00, 00, 00, // elevation (100)
40, 24, 00, 00, 00, 00, 00, 00 // accuracy (10)
Vendor Extension
In addition to the fields I put in the spec, I expect that I may need custom
packet types I can t think of now. There s all sorts of useful data that could
be encoded into the stream, so I d rather there be an officially sanctioned
mechanism that allows future work on the spec without constraining myself.
Just an example, I ve used a custom subpacket to create test vectors, the data
is encoded into a Vendor Extension, followed by the IQ for the modulated
packet. If the demodulated data and in-band original data don t match, we ve
regressed. You could imagine in-band speech-to-text, antenna rotator azimuth
information, or demodulated digital sideband data (like FM HDR data) too. Or
even things I can t even think of!
id
data
In order to help with checking the basic parsing and encoding of this format,
the following is a vendor extension subpacket (when encoded or decoded this
will be found inside an ARF packet as described above).
// extension id (b24305f6-ff73-4b7a-ae99-7a6b37a5d5cd)
b2, 43, 05, f6, ff, 73, 4b, 7a,
ae, 99, 7a, 6b, 37, a5, d5, cd,
// data (0x01, 0x02, 0x03, 0x04, 0x05)
01, 02, 03, 04, 05
Tradeoffs
The biggest tradeoff that I m not entirely happy with is limiting the length
of a packet to u16 65535 bytes. Given the u8 sample header, this limits us
to 8191 32 bit sample pairs at a time. I wound up believing that the overhead in
terms of additional packet framing is worth it because always encoding 4
byte lengths felt like overkill, and a dynamic length scheme ballooned
codepaths in the decoder that I was trying to keep as easy to change as
possible as I worked with the format.
// extension id (b24305f6-ff73-4b7a-ae99-7a6b37a5d5cd)
b2, 43, 05, f6, ff, 73, 4b, 7a,
ae, 99, 7a, 6b, 37, a5, d5, cd,
// data (0x01, 0x02, 0x03, 0x04, 0x05)
01, 02, 03, 04, 05
u16 65535 bytes. Given the u8 sample header, this limits us
to 8191 32 bit sample pairs at a time. I wound up believing that the overhead in
terms of additional packet framing is worth it because always encoding 4
byte lengths felt like overkill, and a dynamic length scheme ballooned
codepaths in the decoder that I was trying to keep as easy to change as
possible as I worked with the format.
A maintenance release 0.3.13 of the 
The twentyfourth release of 








The
The next step, already in January, was mentioning in a fountain-pen
enthusiasts forum that I planned such a test, and asking if people were
interested in having me buy a few samples of more inks when I was
buying my next pen.
The word enthusiasts is probably a hint of the reason why soon
afterwards I received a package with the pen I had planned to buy, its
converter, and a couple dozens ink samples.
And then a couple envelopes with additional samples of inks that weren t
available on the shops, from said enthusiasts.
Added to the inks I already had acquired since the last lightfastness
test, it meant that they couldn t all fit in one single page, and thus I
had some room to add some inks I had already tested: some were requests,
and for others I tried to select ones that felt relevant.
Since I m changing the test setup, I ve decided I should probably keep
doing this until I ve tested again all of the inks I still have
available.
For the paper, I ve used A4 sheets of
While I was filling the sheet for the lightfastness tests, I decided to
also prepare a second set of sheet, for a liquid resistance drop test.
On each line, beside the name of the ink, I added five sets of crossing
parallel lines, and let everything dry for a few days.
Then I used a syringe to put a drop of a liquid on each set of lines,
waited for it to be absorbed into the paper and to dry, at least
overnight, but sometimes also for a day or two (life happened), and then
looked at the results and did the next test.
The first liquid was water, with the usual wild difference between
washable and permanent inks, and all of the intermediate possibilities.
The second liquid was isopropyl alcohol, and I was surprised to see
that, with very few exceptions, most inks didn t change at all. I
wonder whether that s related to the fact that instead of forming a drop
it was absorbed almost immediately into the paper, and dried in a very
short time.
The third liquid was hydrogen peroxide: beside the individual results I
noticed that its column yellowed visibly; I wonder whether that means
that the paper I used has optical brighteners, and it will also yellow
under the sun: that wouldn t be ideal, but it would also be a surprise,
for paper that is acid free and sold for arts.
The fourth liquid was citric acid, by mixing a bit less than a teaspoon
of citric acid granules in just enough very warm water (heated to 70 C,
i.e. the lowest temperature available on my kettle) to dissolve most of
the acid. I forgot that I had some old PH strips until one hour after
I ve put the drop on the paper, and I don t know whether something had
changed, but when I did remember about them it showed a deep red between
1 and 2. I don t think I can trust those strips too much, however.
This backfired badly: the drop of citric acid never dried out, but
formed a sticky paste that prevented me from scanning the results,
and I m not sure whether I ll do the last test, which was supposed to be
household bleach.
Luckily I had scanned the partial results, and they are shown here.
After one full day with plenty of sun, nothing really had changed,
except possibly for a vague hint that the Herbin Bleu Myosotis may have
have been a bit lighter than it started, but it may also have been a
suggestion.
After three days, however, some results started to show, with the most
fugitive inks starting to be visibly changed, becoming either paler or
in some case duller.
And the full week showed more of that, with a few more inks starting to
show visible change.
After two weeks the paper had significantly yellowed, something I did
not expect from drawing paper (and which means that I will probably use
a different paper when making similar tests in the future).
As for the inks, there were a couple more inks with visible changes, but
mostly it was more of the same as seen in the previous week.
Three weeks started to show changes in the black and most irongall inks, and of course more changes in the even less resistant inks.
Week four saw a bit more clouds and rain than the first few weeks, and
there weren t big changes, but mostly more of what had already started
to happen earlier.
A month didn t change much compared to the four weeks, but I did the
scans for completeness, and from now on I m going to update monthly.
I was reading