Paul Tagliamonte: Announcing hz.tools
Interested in future updates? Follow me on mastodon at
@paul@soylent.green. Posts about
If you're on the Fediverse, I'd very much appreciate boosts on my announcement toot!
Ever since 2019, I ve been learning about how radios work, and trying to learn
about using them the hard way by writing as much of the stack as is
practical (for some value of practical) myself. I wrote my first Hello World
in 2018, which was a simple FM radio player, which used hz.tools will be tagged
#hztools.If you're on the Fediverse, I'd very much appreciate boosts on my announcement toot!
librtlsdr to read in
an IQ stream, did some filtering, and played the real valued audio stream via
pulseaudio. Over 4 years this has slowly grown through persistence, lots of
questions to too many friends to thank (although I will try), and the eternal
patience of my wife hearing about radios nonstop for years into a number
of Go repos that can do quite a bit, and support a handful of radios.
I ve resisted making the repos public not out of embarrassment or a desire to
keep secrets, but rather, an attempt to keep myself free of any maintenance
obligations to users so that I could freely break my own API, add and remove
API surface as I saw fit. The worst case was to have this project feel like
work, and I can t imagine that will happen if I feel frustrated by PRs
that are getting ahead of me solving problems I didn t yet know about, or
bugs I didn t understand the fix for.
As my rate of changes to the most central dependencies has slowed, i ve begun
to entertain the idea of publishing them. After a bit of back and forth, I ve
decided
it s time to make a number of them public,
and to start working on them in the open, as I ve built up a bit of knowledge
in the space, and I and feel confident that the repo doesn t contain overt
lies. That s not to say it doesn t contain lies, but those lies are likely
hidden and lurking in the dark. Beware.
That being said, it shouldn t be a surprise to say I ve not published
everything yet for the same reasons as above. I plan to open repos as the
rate of changes slows and I understand the problems the library solves well
enough or if the project dead ends and I ve stopped learning.
Intention behind hz.tools
It s my sincere hope that my repos help to make Software Defined Radio (SDR)
code a bit easier to understand, and serves as an understandable framework to
learn with. It s a large codebase, but one that is possible to sit down and
understand because, well, it was written by a single person. Frankly, I m also
not productive enough in my free time in the middle of the night and on
weekends and holidays to create a codebase that s too large to understand, I
hope!
I remain wary of this project turning into work, so my goal is to be very
upfront about my boundaries, and the limits of what classes of contributions
i m interested in seeing.
Here s some goals of open sourcing these repos:
- I do want this library to be used to learn with. Please go through it
all and use it to learn about radios and how software can control them!
- I am interested in bugs if there s a problem you discover. Such bugs
are likely a great chance for me to fix something I ve misunderstood or
typoed.
- I am interested in PRs fixing bugs you find. I may need a bit of a
back and forth to fully understand the problem if I do not understand
the bug and fix yet. I hope you may have some grace if it s taking a long
time.
Here s a list of some anti-goals of open sourcing these repos.
- I do not want this library to become a critical dependency of an
important project, since I do not have the time to deal with the maintenance
burden. Putting me in that position is going to make me very uncomfortable.
- I am not interested in feature requests, the features have grown as
I ve hit problems, I m not interested in building or maintaining features
for features sake. The API surface should be exposed enough to allow others
to experiment with such things out-of-tree.
- I m not interested in clever code replacing clear code without a very
compelling reason.
- I use GNU/Linux (specifically Debian
),
and from time-to-time I ve made sure that my code runs on OpenBSD
too.
Platforms beyond that will likely not be supported at the expense of either
of those two. I ll take fixes for bugs that fix a problem on another
platform, but not damage the code to work around issues / lack of features
on other platforms (like Windows).
I m not saying all this to be a jerk, I do it to
make sure I can continue on my journey to learn about how radios work without
my full time job becoming maintaining a radio framework single-handedly for
other people to use even if it means I need to close PRs or bugs without
merging it or fixing the issue.
With all that out of the way, I m very happy to announce that the repos are now
public under github.com/hztools.
Should you use this?
Probably not. The intent here is not to provide a general purpose Go SDR
framework for everyone to build on, although I am keenly aware it looks and
feels like it, since that what it is to me. This is a learning project, so for
any use beyond joining me in learning should use something like
GNU Radio
or a similar framework that has a community behind it.
In fact, I suspect most contributors ought to be contributing to GNU Radio, and
not this project. If I can encourage people to do so,
contribute to GNU Radio!
Nothing makes me happier than seeing GNU Radio continue to be the go-to, and
well supported. Consider
donating to GNU Radio!
hz.tools/rf - Frequency types
The hz.tools/rf library contains the abstract concept of frequency, and
some very basic helpers to interact with frequency ranges (such as helpers
to deal with frequency ranges, or frequency range math) as well as frequencies
and some very basic conversions (to meters, etc) and parsers (to parse values
like 10MHz). This ensures that all the hz.tools libraries have a shared
understanding of Frequencies, a standard way of representing ranges of
Frequencies, and the ability to handle the IO boundary with things like CLI
arguments, JSON or YAML.
The git repo can be found at
github.com/hztools/go-rf, and is
importable as hz.tools/rf.
// Parse a frequency using hz.tools/rf.ParseHz, and print it to stdout.
freq := rf.MustParseHz("-10kHz")
fmt.Printf("Frequency: %s\n", freq+rf.MHz)
// Prints: 'Frequency: 990kHz'
// Return the Intersection between two RF ranges, and print
// it to stdout.
r1 := rf.Range rf.KHz, rf.MHz
r2 := rf.Range rf.Hz(10), rf.KHz * 100
fmt.Printf("Range: %s\n", r1.Intersection(r2))
// Prints: Range: 1000Hz->100kHz
These can be used to represent tons of things - ranges can be used for things
like the tunable range of an SDR, the bandpass of a filter or the frequencies
that correspond to a bin of an FFT, while frequencies can be used for things
such as frequency offsets or the tuned center frequency.
hz.tools/sdr - SDR I/O and IQ Types
This is the big one. This library represents the majority of the shared
types and bindings, and is likely the most useful place to look at when
learning about the IO boundary between a program and an SDR.
The git repo can be found at
github.com/hztools/go-sdr, and is
importable as hz.tools/sdr.
This library is designed to look (and in some cases, mirror) the Go io idioms
so that this library feels as idiomatic as it can, so that Go builtins interact
with IQ in a way that s possible to reason about, and to avoid reinventing the
wheel by designing new API surface. While some of the API looks (and is even
called) the same thing as a similar function in io, the implementation is
usually a lot more naive, and may have unexpected sharp edges such as
concurrency issues or performance problems.
The following IQ types are implemented using the sdr.Samples interface. The
hz.tools/sdr package contains helpers for conversion between types, and some
basic manipulation of IQ streams.
),
and from time-to-time I ve made sure that my code runs on OpenBSD
too.
Platforms beyond that will likely not be supported at the expense of either
of those two. I ll take fixes for bugs that fix a problem on another
platform, but not damage the code to work around issues / lack of features
on other platforms (like Windows).hz.tools/rf - Frequency types
The hz.tools/rf library contains the abstract concept of frequency, and
some very basic helpers to interact with frequency ranges (such as helpers
to deal with frequency ranges, or frequency range math) as well as frequencies
and some very basic conversions (to meters, etc) and parsers (to parse values
like 10MHz). This ensures that all the hz.tools libraries have a shared
understanding of Frequencies, a standard way of representing ranges of
Frequencies, and the ability to handle the IO boundary with things like CLI
arguments, JSON or YAML.
The git repo can be found at
github.com/hztools/go-rf, and is
importable as hz.tools/rf.
// Parse a frequency using hz.tools/rf.ParseHz, and print it to stdout.
freq := rf.MustParseHz("-10kHz")
fmt.Printf("Frequency: %s\n", freq+rf.MHz)
// Prints: 'Frequency: 990kHz'
// Return the Intersection between two RF ranges, and print
// it to stdout.
r1 := rf.Range rf.KHz, rf.MHz
r2 := rf.Range rf.Hz(10), rf.KHz * 100
fmt.Printf("Range: %s\n", r1.Intersection(r2))
// Prints: Range: 1000Hz->100kHz
These can be used to represent tons of things - ranges can be used for things
like the tunable range of an SDR, the bandpass of a filter or the frequencies
that correspond to a bin of an FFT, while frequencies can be used for things
such as frequency offsets or the tuned center frequency.
hz.tools/sdr - SDR I/O and IQ Types
This is the big one. This library represents the majority of the shared
types and bindings, and is likely the most useful place to look at when
learning about the IO boundary between a program and an SDR.
The git repo can be found at
github.com/hztools/go-sdr, and is
importable as hz.tools/sdr.
This library is designed to look (and in some cases, mirror) the Go io idioms
so that this library feels as idiomatic as it can, so that Go builtins interact
with IQ in a way that s possible to reason about, and to avoid reinventing the
wheel by designing new API surface. While some of the API looks (and is even
called) the same thing as a similar function in io, the implementation is
usually a lot more naive, and may have unexpected sharp edges such as
concurrency issues or performance problems.
The following IQ types are implemented using the sdr.Samples interface. The
hz.tools/sdr package contains helpers for conversion between types, and some
basic manipulation of IQ streams.
// Parse a frequency using hz.tools/rf.ParseHz, and print it to stdout.
freq := rf.MustParseHz("-10kHz")
fmt.Printf("Frequency: %s\n", freq+rf.MHz)
// Prints: 'Frequency: 990kHz'
// Return the Intersection between two RF ranges, and print
// it to stdout.
r1 := rf.Range rf.KHz, rf.MHz
r2 := rf.Range rf.Hz(10), rf.KHz * 100
fmt.Printf("Range: %s\n", r1.Intersection(r2))
// Prints: Range: 1000Hz->100kHz
io idioms
so that this library feels as idiomatic as it can, so that Go builtins interact
with IQ in a way that s possible to reason about, and to avoid reinventing the
wheel by designing new API surface. While some of the API looks (and is even
called) the same thing as a similar function in io, the implementation is
usually a lot more naive, and may have unexpected sharp edges such as
concurrency issues or performance problems.
The following IQ types are implemented using the sdr.Samples interface. The
hz.tools/sdr package contains helpers for conversion between types, and some
basic manipulation of IQ streams.
| IQ Format | hz.tools Name | Underlying Go Type |
|---|---|---|
| Interleaved uint8 (rtl-sdr) | sdr.SamplesU8 |
[][2]uint8 |
| Interleaved int8 (hackrf, uhd) | sdr.SamplesI8 |
[][2]int8 |
| Interleaved int16 (pluto, uhd) | sdr.SamplesI16 |
[][2]int16 |
| Interleaved float32 (airspy, uhd) | sdr.SamplesC64 |
[]complex64 |
| SDR | Format | RX/TX | State |
|---|---|---|---|
| rtl | u8 | RX | Good |
| HackRF | i8 | RX/TX | Good |
| PlutoSDR | i16 | RX/TX | Good |
| rtl kerberos | u8 | RX | Old |
| uhd | i16/c64/i8 | RX/TX | Good |
| airspyhf | c64 | RX | Exp |
| Import | What is it? |
|---|---|
| hz.tools/sdr | Core IQ types, supporting types and implementations that interact with the byte boundary |
| hz.tools/sdr/rtl | sdr.Receiver implementation using librtlsdr. |
| hz.tools/sdr/rtl/kerberos | Helpers to enable coherent RX using the Kerberos SDR. |
| hz.tools/sdr/rtl/e4k | Helpers to interact with the E4000 RTL-SDR dongle. |
| hz.tools/sdr/fft | Interfaces for performing an FFT, which are implemented by other packages. |
| hz.tools/sdr/rtltcp | sdr.Receiver implementation for rtl_tcp servers. |
| hz.tools/sdr/pluto | sdr.Transceiver implementation for the PlutoSDR using libiio. |
| hz.tools/sdr/uhd | sdr.Transceiver implementation for UHD radios, specifically the B210 and B200mini |
| hz.tools/sdr/hackrf | sdr.Transceiver implementation for the HackRF using libhackrf. |
| hz.tools/sdr/mock | Mock SDR for testing purposes. |
| hz.tools/sdr/airspyhf | sdr.Receiver implementation for the AirspyHF+ Discovery with libairspyhf. |
| hz.tools/sdr/internal/simd | SIMD helpers for IQ operations, written in Go ASM. This isn t the best to learn from, and it contains pure go implemtnations alongside. |
| hz.tools/sdr/stream | Common Reader/Writer helpers that operate on IQ streams. |
hz.tools/fftw - hz.tools/sdr/fft implementation
The hz.tools/fftw package contains bindings to libfftw3 to implement
the hz.tools/sdr/fft.Planner type to transform between the time and
frequency domain.
The git repo can be found at
github.com/hztools/go-fftw, and is
importable as hz.tools/fftw.
This is the default throughout most of my codebase, although that default is
only expressed at the leaf package libraries should not be hardcoding the
use of this library in favor of taking an fft.Planner, unless it s used as
part of testing. There are a bunch of ways to do an FFT out there, things like
clFFT or a pure-go FFT implementation could be plugged in depending on what s
being solved for.
hz.tools/ fm,am - analog audio demodulation and modulation
The hz.tools/fm and hz.tools/am packages contain demodulators for
AM analog radio, and FM analog radio. This code is a bit old, so it has
a lot of room for cleanup, but it ll do a very basic demodulation of IQ
to audio.
The git repos can be found at
github.com/hztools/go-fm and
github.com/hztools/go-am,
and are importable as
hz.tools/fm and
hz.tools/am.
As a bonus, the hz.tools/fm package also contains a modulator, which has been
tested on the air and with some of my handheld radios. This code is a bit
old, since the hz.tools/fm code is effectively the first IQ processing code
I d ever written, but it still runs and I run it from time to time.
// Basic sketch for playing FM radio using a reader stream from
// an SDR or other IQ stream.
bandwidth := 150*rf.KHz
reader, err = stream.ConvertReader(reader, sdr.SampleFormatC64)
if err != nil
...
demod, err := fm.Demodulate(reader, fm.DemodulatorConfig
Deviation: bandwidth / 2,
Downsample: 8, // some value here depending on sample rate
Planner: fftw.Plan,
)
if err != nil
...
speaker, err := pulseaudio.NewWriter(pulseaudio.Config
Format: pulseaudio.SampleFormatFloat32NE,
Rate: demod.SampleRate(),
AppName: "rf",
StreamName: "fm",
Channels: 1,
SinkName: "",
)
if err != nil
...
buf := make([]float32, 1024*64)
for
i, err := demod.Read(buf)
if err != nil
...
if i == 0
panic("...")
if err := speaker.Write(buf[:i]); err != nil
...
hz.tools/rfcap - byte serialization for IQ data
The hz.tools/rfcap package is the reference implementation of the
rfcap spec , and is how I store IQ captures
locally, and how I send them across a byte boundary.
The git repo can be found at
github.com/hztools/go-rfcap, and is
importable as hz.tools/rfcap.
If you re interested in storing IQ in a way others can use, the better approach
is to use SigMF rfcap exists for cases
like using UNIX pipes to move IQ around, through APIs, or when I send
IQ data through an OS socket, to ensure the sample format (and other metadata)
is communicated with it.
rfcap has a number of limitations, for instance, it can not express a change
in frequency or sample rate during the capture, since the header is fixed at
the beginning of the file.
hz.tools/fm and hz.tools/am packages contain demodulators for
AM analog radio, and FM analog radio. This code is a bit old, so it has
a lot of room for cleanup, but it ll do a very basic demodulation of IQ
to audio.
The git repos can be found at
github.com/hztools/go-fm and
github.com/hztools/go-am,
and are importable as
hz.tools/fm and
hz.tools/am.
As a bonus, the hz.tools/fm package also contains a modulator, which has been
tested on the air and with some of my handheld radios. This code is a bit
old, since the hz.tools/fm code is effectively the first IQ processing code
I d ever written, but it still runs and I run it from time to time.
// Basic sketch for playing FM radio using a reader stream from
// an SDR or other IQ stream.
bandwidth := 150*rf.KHz
reader, err = stream.ConvertReader(reader, sdr.SampleFormatC64)
if err != nil
...
demod, err := fm.Demodulate(reader, fm.DemodulatorConfig
Deviation: bandwidth / 2,
Downsample: 8, // some value here depending on sample rate
Planner: fftw.Plan,
)
if err != nil
...
speaker, err := pulseaudio.NewWriter(pulseaudio.Config
Format: pulseaudio.SampleFormatFloat32NE,
Rate: demod.SampleRate(),
AppName: "rf",
StreamName: "fm",
Channels: 1,
SinkName: "",
)
if err != nil
...
buf := make([]float32, 1024*64)
for
i, err := demod.Read(buf)
if err != nil
...
if i == 0
panic("...")
if err := speaker.Write(buf[:i]); err != nil
...
Like each month, have a look at the work funded by 
It was brought to my attention recently that the mobile viewing experience of this blog was not exactly what I d hope for. In my poor defence I proof read on my desktop and the only time I see my posts on mobile is via 
Back in September last year I chose to back the 
I haven t seen any actual stability problems, but I wouldn t want to run for any length of time like that. I ve ordered a heatsink and also realised that the board supports a Raspberry Pi style PoE Hat , so I ve got one of those that includes a fan ordered (I am a complete convert to PoE especially for small systems like this).
With the desk fan setup I ve been able to run the board for extended periods under load (I did a full recompile of the Debian 6.1.12-1 kernel package and it took about 10 hours). The M.2 slot is unfortunately only a single PCIe v2 lane, and my testing topped out at about 180MB/s. IIRC that is about half what the slot should be capable of, and less than a 10th of what the SSD can do. Ethernet testing with 
In case you want to know how to configure IPv6 using CenturyLink s 6rd tunneling service.
There are four files to update with my script:
To achieve my