Paul Tagliamonte: Open to work!

hz.tools
will be tagged
#hztools.librtlsdr
to read in
an IQ stream, did some filtering, and played the real valued audio stream via
pulseaudio
. Over 4 years this has slowly grown through persistence, lots of
questions to too many friends to thank (although I will try), and the eternal
patience of my wife hearing about radios nonstop for years into a number
of Go repos that can do quite a bit, and support a handful of radios.
I ve resisted making the repos public not out of embarrassment or a desire to
keep secrets, but rather, an attempt to keep myself free of any maintenance
obligations to users so that I could freely break my own API, add and remove
API surface as I saw fit. The worst case was to have this project feel like
work, and I can t imagine that will happen if I feel frustrated by PRs
that are getting ahead of me solving problems I didn t yet know about, or
bugs I didn t understand the fix for.
As my rate of changes to the most central dependencies has slowed, i ve begun
to entertain the idea of publishing them. After a bit of back and forth, I ve
decided
it s time to make a number of them public,
and to start working on them in the open, as I ve built up a bit of knowledge
in the space, and I and feel confident that the repo doesn t contain overt
lies. That s not to say it doesn t contain lies, but those lies are likely
hidden and lurking in the dark. Beware.
That being said, it shouldn t be a surprise to say I ve not published
everything yet for the same reasons as above. I plan to open repos as the
rate of changes slows and I understand the problems the library solves well
enough or if the project dead ends and I ve stopped learning.
hz.tools/rf
library contains the abstract concept of frequency, and
some very basic helpers to interact with frequency ranges (such as helpers
to deal with frequency ranges, or frequency range math) as well as frequencies
and some very basic conversions (to meters, etc) and parsers (to parse values
like 10MHz
). This ensures that all the hz.tools
libraries have a shared
understanding of Frequencies, a standard way of representing ranges of
Frequencies, and the ability to handle the IO boundary with things like CLI
arguments, JSON or YAML.
The git repo can be found at
github.com/hztools/go-rf, and is
importable as hz.tools/rf.
// Parse a frequency using hz.tools/rf.ParseHz, and print it to stdout.
freq := rf.MustParseHz("-10kHz")
fmt.Printf("Frequency: %s\n", freq+rf.MHz)
// Prints: 'Frequency: 990kHz'
// Return the Intersection between two RF ranges, and print
// it to stdout.
r1 := rf.Range rf.KHz, rf.MHz
r2 := rf.Range rf.Hz(10), rf.KHz * 100
fmt.Printf("Range: %s\n", r1.Intersection(r2))
// Prints: Range: 1000Hz->100kHz
io
idioms
so that this library feels as idiomatic as it can, so that Go builtins interact
with IQ in a way that s possible to reason about, and to avoid reinventing the
wheel by designing new API surface. While some of the API looks (and is even
called) the same thing as a similar function in io
, the implementation is
usually a lot more naive, and may have unexpected sharp edges such as
concurrency issues or performance problems.
The following IQ types are implemented using the sdr.Samples
interface. The
hz.tools/sdr
package contains helpers for conversion between types, and some
basic manipulation of IQ streams.
IQ Format | hz.tools Name | Underlying Go Type |
---|---|---|
Interleaved uint8 (rtl-sdr) | sdr.SamplesU8 |
[][2]uint8 |
Interleaved int8 (hackrf, uhd) | sdr.SamplesI8 |
[][2]int8 |
Interleaved int16 (pluto, uhd) | sdr.SamplesI16 |
[][2]int16 |
Interleaved float32 (airspy, uhd) | sdr.SamplesC64 |
[]complex64 |
SDR | Format | RX/TX | State |
---|---|---|---|
rtl | u8 | RX | Good |
HackRF | i8 | RX/TX | Good |
PlutoSDR | i16 | RX/TX | Good |
rtl kerberos | u8 | RX | Old |
uhd | i16/c64/i8 | RX/TX | Good |
airspyhf | c64 | RX | Exp |
Import | What is it? |
---|---|
hz.tools/sdr | Core IQ types, supporting types and implementations that interact with the byte boundary |
hz.tools/sdr/rtl | sdr.Receiver implementation using librtlsdr . |
hz.tools/sdr/rtl/kerberos | Helpers to enable coherent RX using the Kerberos SDR. |
hz.tools/sdr/rtl/e4k | Helpers to interact with the E4000 RTL-SDR dongle. |
hz.tools/sdr/fft | Interfaces for performing an FFT, which are implemented by other packages. |
hz.tools/sdr/rtltcp | sdr.Receiver implementation for rtl_tcp servers. |
hz.tools/sdr/pluto | sdr.Transceiver implementation for the PlutoSDR using libiio . |
hz.tools/sdr/uhd | sdr.Transceiver implementation for UHD radios, specifically the B210 and B200mini |
hz.tools/sdr/hackrf | sdr.Transceiver implementation for the HackRF using libhackrf . |
hz.tools/sdr/mock | Mock SDR for testing purposes. |
hz.tools/sdr/airspyhf | sdr.Receiver implementation for the AirspyHF+ Discovery with libairspyhf . |
hz.tools/sdr/internal/simd | SIMD helpers for IQ operations, written in Go ASM. This isn t the best to learn from, and it contains pure go implemtnations alongside. |
hz.tools/sdr/stream | Common Reader/Writer helpers that operate on IQ streams. |
hz.tools/fftw
package contains bindings to libfftw3
to implement
the hz.tools/sdr/fft.Planner
type to transform between the time and
frequency domain.
The git repo can be found at
github.com/hztools/go-fftw, and is
importable as hz.tools/fftw.
This is the default throughout most of my codebase, although that default is
only expressed at the leaf package libraries should not be hardcoding the
use of this library in favor of taking an fft.Planner
, unless it s used as
part of testing. There are a bunch of ways to do an FFT out there, things like
clFFT
or a pure-go FFT implementation could be plugged in depending on what s
being solved for.
hz.tools/fm
and hz.tools/am
packages contain demodulators for
AM analog radio, and FM analog radio. This code is a bit old, so it has
a lot of room for cleanup, but it ll do a very basic demodulation of IQ
to audio.
The git repos can be found at
github.com/hztools/go-fm and
github.com/hztools/go-am,
and are importable as
hz.tools/fm and
hz.tools/am.
As a bonus, the hz.tools/fm
package also contains a modulator, which has been
tested on the air and with some of my handheld radios. This code is a bit
old, since the hz.tools/fm
code is effectively the first IQ processing code
I d ever written, but it still runs and I run it from time to time.
// Basic sketch for playing FM radio using a reader stream from
// an SDR or other IQ stream.
bandwidth := 150*rf.KHz
reader, err = stream.ConvertReader(reader, sdr.SampleFormatC64)
if err != nil
...
demod, err := fm.Demodulate(reader, fm.DemodulatorConfig
Deviation: bandwidth / 2,
Downsample: 8, // some value here depending on sample rate
Planner: fftw.Plan,
)
if err != nil
...
speaker, err := pulseaudio.NewWriter(pulseaudio.Config
Format: pulseaudio.SampleFormatFloat32NE,
Rate: demod.SampleRate(),
AppName: "rf",
StreamName: "fm",
Channels: 1,
SinkName: "",
)
if err != nil
...
buf := make([]float32, 1024*64)
for
i, err := demod.Read(buf)
if err != nil
...
if i == 0
panic("...")
if err := speaker.Write(buf[:i]); err != nil
...
hz.tools/rfcap
package is the reference implementation of the
rfcap spec , and is how I store IQ captures
locally, and how I send them across a byte boundary.
The git repo can be found at
github.com/hztools/go-rfcap, and is
importable as hz.tools/rfcap.
If you re interested in storing IQ in a way others can use, the better approach
is to use SigMF rfcap
exists for cases
like using UNIX pipes to move IQ around, through APIs, or when I send
IQ data through an OS socket, to ensure the sample format (and other metadata)
is communicated with it.
rfcap
has a number of limitations, for instance, it can not express a change
in frequency or sample rate during the capture, since the header is fixed at
the beginning of the file.
H
(check) matrix
of width N
, you can check your message vector (msg
) of length N
by
multipling H
and msg
, and checking if the output vector is all zero.
// scheme contains our G (generator) and
// H (check) matrices.
scheme := G: Matrix ... , H: Matrix ...
// msg contains our LDPC message (data and
// check bits).
msg := Vector ...
// N is also the length of the encoded
// msg vector after check bits have been
// added.
N := scheme.G.Width
// Now, let's generate our 'check' vector.
ch := Multiply(scheme.H, msg)
// if the ch vector is all zeros, we know
// that the message is valid, and we don't
// need to do anything.
if ch.IsZero()
// handle the case where the message
// is fine as-is.
return ...
// Expensive decode here
g
(generator) matrix out, building a bipartite graph, and
iteratively reprocessing the bit values, until constraints are satisfied and
the message has been corrected.
This got me thinking - what is the output vector when it s not all zeros?
Since 1
values in the output vector indicates consistency problems in the
message bits as they relate to the check bits, I wondered if this could be used
to speed up my LDPC decoder. It appears to work, so this post is half an attempt
to document this technique before I put it in my hot path, and half a plea for
those who do like to talk about FEC to tell me what name this technique
actually is.
// for clarity's sake, the Vector
// type is being used as the lookup
// key here, even though it may
// need to be a hash or string in
// some cases.
idx := map[Vector]int
for i := 0; i < N; i++
// Create a vector of length N
v := Vector
v.FlipBit(i)
// Now, let's use the generator matrix to encode
// the data with checksums, and then use the
// check matrix on the message to figure out what
// bit pattern results
ev := Multiply(scheme.H, Multiply(v, scheme.G))
idx[ev] = i
idx
mapping, we can now
go back to the hot path on Checking the incoming message data:
// if the ch vector is all zeros, we know
// that the message is valid, and we don't
// need to do anything.
if ch.IsZero()
// handle the case where the message
// is fine as-is.
return ...
errIdx, ok := idx[ch]
if ok
msg.FlipBit(errIdx)
// Verify the LDPC message using
// H again here.
return ...
// Expensive decode here
802.3an-2006
. Even if I was to find a collision for a
higher-order k-Bit value, I m tempted to continue with this approach, and treat
each set of bits in the Vector s bin (like a hash-table), checking the LDPC
validity after each bit set in the bin. As long as the collision rate is small
enough, it should be possible to correct k-Bits of error faster than the more
expensive Belief Propagation approach. That being said, I m not entirely
convinced collisions will be very common, but it ll take a bit more time
working through the math to say that with any confidence.
Have you seen this approach called something official in publications? See
an obvious flaw in the system? Send me a tip, please!
pattyd
) via a UNIX named socket.
That daemon will communicate with a particular radio using a
KISS TNC serial device.
The Go bindings implement as many standard Go library interfaces as is
practical, allowing for the plug and play use of patty
(and AX.25) in
places where you would expect a network socket (such as TCP) to work, such
as Go s http library.
package main
import (
"fmt"
"log"
"net"
"os"
"time"
"k3xec.com/patty"
)
func main()
callsign := "N0CALL-10"
client, err := patty.Open("patty.sock")
if err != nil
panic(err)
l, err := client.Listen("ax25", callsign)
if err != nil
panic(err)
for
log.Printf("Listening for requests to %s", l.Addr())
conn, err := l.Accept()
if err != nil
log.Printf("Error accepting: %s", err)
continue
go handle(conn)
func handle(c net.Conn) error
defer c.Close()
log.Printf("New connection from %s (local: %s)", c.RemoteAddr(), c.LocalAddr())
fmt.Fprintf(c,
Hello! This is Paul's experimental %s node. Feel free
to poke around. Let me know if you spot anything funny.
Five pings are to follow!
, c.LocalAddr())
for i := 0; i < 5; i++
time.Sleep(time.Second * 5)
fmt.Fprintf(c, "Ping!\n")
return nil
Callsign
), that will need to be explicitly passed
in when turning an Ethernet frame into a PACKRAT Frame.
...
// ToPackrat will create a packrat frame from an Ethernet frame.
func ToPackrat(callsign [8]byte, frame *ethernet.Frame) (*packrat.Frame, error)
var frameType packrat.FrameType
switch frame.EtherType
case ethernet.EtherTypeIPv4:
frameType = packrat.FrameTypeIPv4
default:
return nil, fmt.Errorf("ethernet: unsupported ethernet type %x", frame.EtherType)
return &packrat.Frame
Destination: frame.Destination,
Source: frame.Source,
Type: frameType,
Callsign: callsign,
Payload: frame.Payload,
, nil
// FromPackrat will create an Ethernet frame from a Packrat frame.
func FromPackrat(frame *packrat.Frame) (*ethernet.Frame, error)
var etherType ethernet.EtherType
switch frame.Type
case packrat.FrameTypeRaw:
return nil, fmt.Errorf("ethernet: unsupported packrat type 'raw'")
case packrat.FrameTypeIPv4:
etherType = ethernet.EtherTypeIPv4
default:
return nil, fmt.Errorf("ethernet: unknown packrat type %x", frame.Type)
// We lose the Callsign here, which is sad.
return ðernet.Frame
Destination: frame.Destination,
Source: frame.Source,
EtherType: etherType,
Payload: frame.Payload,
, nil
ToPackrat
and FromPackrat
can now be used to transmorgify
PACKRAT into Ethernet, or Ethernet into PACKRAT. Let s put them into use!
...
import (
"net"
"github.com/mdlayher/ethernet"
"github.com/songgao/water"
"github.com/vishvananda/netlink"
)
...
config := water.Config DeviceType: water.TAP
config.Name = "rat0"
iface, err := water.New(config)
...
netIface, err := netlink.LinkByName("rat0")
...
// Pick a range here that works for you!
//
// For my local network, I'm using some IPs
// that AMPR (ampr.org) was nice enough to
// allocate to me for ham radio use. Thanks,
// AMPR!
//
// Let's just use 10.* here, though.
//
ip, cidr, err := net.ParseCIDR("10.0.0.1/24")
...
cidr.IP = ip
err = netlink.AddrAdd(netIface, &netlink.Addr
IPNet: cidr,
Peer: cidr,
)
...
// Add all our neighbors to the ARP table
for _, neighbor := range neighbors
netlink.NeighAdd(&netlink.Neigh
LinkIndex: netIface.Attrs().Index,
Type: netlink.FAMILY_V4,
State: netlink.NUD_PERMANENT,
IP: neighbor.IP,
HardwareAddr: neighbor.MAC,
)
// Pick a MAC that is globally unique here, this is
// just used as an example!
addr, err := net.ParseMAC("FA:DE:DC:AB:LE:01")
...
netlink.LinkSetHardwareAddr(netIface, addr)
...
err = netlink.LinkSetUp(netIface)
var frame = ðernet.Frame
var buf = make([]byte, 1500)
for
n, err := iface.Read(buf)
...
err = frame.UnmarshalBinary(buf[:n])
...
// process frame here (to come)
...
ip neigh
according to our pre-defined neighbors), and send that IP packet to our daemon,
it s now on us to send IPv4 data over the airwaves. Here, we re going to take
packets coming in from our TAP interface, and marshal the Ethernet frame into a
PACKRAT Frame and transmit it. As with the rest of the RF code, we ll leave
that up to the implementer, of course, using what was built during Part 2:
Transmitting BPSK symbols and Part 4: Framing
data.
...
for
// continued from above
n, err := iface.Read(buf)
...
err = frame.UnmarshalBinary(buf[:n])
...
switch frame.EtherType
case 0x0800:
// ipv4 packet
pack, err := ToPackrat(
// Add my callsign to all Frames, for now
[8]byte 'K', '3', 'X', 'E', 'C' ,
frame,
)
...
err = transmitPacket(pack)
...
...
packratReader.Next
in the code below, and the exact way that works
is up to the implementer.
...
for
// pull the next packrat frame from
// the symbol stream as we did in the
// last post
packet, err := packratReader.Next()
...
// check for CRC errors and drop invalid
// packets
err = packet.Check()
...
if bytes.Equal(packet.Source, addr)
// if we've heard ourself transmitting
// let's avoid looping back
continue
// create an ethernet frame
frame, err := FromPackrat(packet)
...
buf, err := frame.MarshalBinary()
...
// and inject it into the tap
err = iface.Write(buf)
...
...
/dev/udp
and an Ettus B210, sending packets into the TAP interface.
EtherType
field to indicate the Payload type. I also stole the idea of
using a CRC at the end of the Frame to check for corruption, as well as the
specific CRC method (crc32
using 0xedb88320
as the polynomial).
Lastly, I added a callsign
field to make life easier on ham radio frequencies
if I was ever to seriously attempt to use a variant of this protocol over the
air with multiple users. However, given this scheme is not a commonly used
scheme, it s best practice to use a nearby radio to identify your transmissions
on the same frequency while testing or use a Faraday box to test without
transmitting over the airwaves. I added the callsign field in an effort to lean
into the spirit of the Part 97 regulations, even if I relied on a phone
emission to identify the Frames.
As an aside, I asked the ARRL for input here, and their stance to me over email
was I d be OK according to the regs if I were to stick to UHF and put my
callsign into the BPSK stream using a widely understood encoding (even with no
knowledge of PACKRAT, the callsign is ASCII over BPSK and should be easily
demodulatable for followup with me). Even with all this, I opted to use FM
phone to transmit my callsign when I was active on the air (specifically, using
an SDR and a small bash script to automate transmission while I watched for
interference or other band users).
Right, back to the Frame:
type FrameType [2]byte
type Frame struct
Destination net.HardwareAddr
Source net.HardwareAddr
Callsign [8]byte
Type FrameType
Payload []byte
CRC uint32
sync
sequence, which the sender
will transmit before the Frame, while the receiver listens for that
sequence to know when it s in byte alignment with the symbol stream.
My sync
sequence is [3]byte 'U', 'f', '~'
which works out to be a
very pleasant bit sequence of 01010101 01100110 01111110
. It s important
to have soothing preambles for your Frames. We need all the good energy
we can get at this point.
var (
FrameStart = [3]byte 'U', 'f', '~'
FrameMaxPayloadSize = 1500
)
FrameType
values for the type
field,
which I can use to determine what is done with that data next,
something Ethernet was originally missing, but has since grown
to depend on (who needs Length anyway? Not me. See below!)
FrameType | Description | Bytes |
Raw | Bytes in the Payload field are opaque and not to be parsed. | [2]byte 0x00, 0x01 |
IPv4 | Bytes in the Payload field are an IPv4 packet. | [2]byte 0x00, 0x02 |
var (
FrameTypeRaw = FrameType 0, 1
FrameTypeIPv4 = FrameType 0, 2
)
sync
bit pattern in the symbols we re
receiving from our BPSK demodulator. There s some smart ways to do this, but
given that I m not much of a smart man, I again decided to go for simple
instead. Given our incoming vector of symbols (which are still float
values)
prepend one at a time to a vector of floats that is the same length as the
sync
phrase, and compare against the sync
phrase, to determine if we re in
sync with the byte boundary within the symbol stream.
The only trick here is that because we re using BPSK to modulate and demodulate
the data, post phaselock we can be 180 degrees out of alignment (such that a +1
is demodulated as -1, or vice versa). To deal with that, I check against both
the sync
phrase as well as the inverse of the sync
phrase (both [1, -1, 1]
as well as [-1, 1, -1]
) where if the inverse sync is matched, all symbols
to follow will be inverted as well. This effectively turns our symbols back
into bits, even if we re flipped out of phase. Other techniques like
NRZI will represent a 0 or
1 by a change in phase state which is great, but can often cascade into long
runs of bit errors, and is generally more complex to implement. That
representation isn t ambiguous, given you look for a phase change, not the
absolute phase value, which is incredibly compelling.
Here s a notional example of how I ve been thinking about the phrase sliding
window and how I ve been thinking of the checks. Each row is a new symbol
taken from the BPSK receiver, and pushed to the head of the sliding window,
moving all symbols back in the vector by one.
var (
sync = []float ...
buf = make([]float, len(sync))
incomingSymbols = []float ...
)
for _, el := range incomingSymbols
copy(buf, buf[1:])
buf[len(buf)-1] = el
if compare(sync, buf)
// we're synced!
break
Buffer | Sync | Inverse Sync |
[ ]float 0, ,0 | [ ]float -1, ,-1 | [ ]float 1, ,1 |
[ ]float 0, ,1 | [ ]float -1, ,-1 | [ ]float 1, ,1 |
[more bits in] | [ ]float -1, ,-1 | [ ]float 1, ,1 |
[ ]float 1, ,1 | [ ]float -1, ,-1 | [ ]float 1, ,1 |
tau/2
(specifically, tau/(2^1)
for BPSK, 2^2
for QPSK) angles, and given that squaring has the effect of
doubling the angle, and angles are all mod tau
, this will drive our wave
comprised of two opposite phases back into a continuous wave effectively
removing our BPSK modulation, making it much easier to detect in the
frequency domain. Thanks to Tom Bereknyei
for helping me with that!
...
var iq []complex
var freq []complex
for i := range iq
iq[i] = iq[i] * iq[i]
// perform an fft, computing the frequency
// domain vector in freq given the iq data
// contained in iq .
fft(iq, freq)
// get the array index of the max value in the
// freq array given the magnitude value of the
// complex numbers.
var binIdx = max(abs(freq))
...
...
var sampleRate = 2,621,440
// bandwidth is the range of frequencies
// contained inside a single FFT bin,
// measured in Hz.
var bandwidth = sampleRate/len(freq)
...
...
// binIdx is the index into the freq slice
// containing the frequency domain data.
var binIdx = 0
// binFreq is the frequency of the bin
// denoted by binIdx
var binFreq = 0
if binIdx > len(freq)/2
// This branch covers the case where the bin
// is past the middle point - which is to say,
// if this is a negative frequency.
binFreq = bandwidth * (binIdx - len(freq))
else
// This branch covers the case where the bin
// is in the first half of the frequency array,
// which is to say - if this frequency is
// a positive frequency.
binFreq = bandwidth * binIdx
...
...
var binFreq = 0
...
// [compute the binFreq as above]
...
// Adjust for the squaring of our IQ data
binFreq = binFreq / 2
...
binFreq
by generating a carrier wave at a specific frequency and rotating every
sample by our carrier wave so that a wave at the same frequency will
slow down (or stand still!) as it approaches 0Hz relative to the carrier
wave.
var tau = pi * 2
// ts tracks where in time we are (basically: phase)
var ts float
// inc is the amount we step forward in time (seconds)
// each sample.
var inc float = (1 / sampleRate)
// amount to shift frequencies, in Hz,
// in this case, shift +12 kHz to 0Hz
var shift = -12,000
for i := range iq
ts += inc
if ts > tau
// not actually needed, but keeps ts within
// 0 to 2*pi (since it is modulus 2*pi anyway)
ts -= tau
// Here, we're going to create a carrier wave
// at the provided frequency (in this case,
// -12kHz)
cwIq = complex(cos(tau*shift*ts), sin(tau*shift*ts))
iq[i] = iq[i] * cwIq
// Generate sinc taps
func sinc(x float) float
if x == 0
return 1
var v = pi * x
return sin(v) / v
...
var dst []float
var length = float(len(dst))
if int(length)%2 == 0
length++
for j := range dst
i := float(j)
dst[j] = sinc(2 * cutoff * (i - (length-1)/2))
...
...
// Apply sinc taps to an IQ stream
var iq []complex
// taps as created in dst above
var taps []float
var delay = make([]complex, len(taps))
for i := range iq
// let's shift the next sample into
// the delay buffer
copy(delay[1:], delay)
delay[0] = iq[i]
var phasor complex
for j := range delay
// for each sample in the buffer, let's
// weight them by the tap values, and
// create a new complex number based on
// filtering the real and imag values.
phasor += complex(
taps[j] * real(delay[j]),
taps[j] * imag(delay[j]),
)
// now that we've run this sample
// through the filter, we can go ahead
// and scale it back (since we multiply
// above) and drop it back into the iq
// buffer.
iq[i] = complex(
real(phasor) / len(taps),
imag(phasor) / len(taps),
)
...
var alpha = 0.1
var beta = (alpha * alpha) / 2
var phase = 0.0
var frequency = 0.0
...
for i := range iq
predicted = complex(cos(phase), sin(phase))
sample = iq[i] * conj(predicted)
delta = phase(sample)
predicted2 = complex(cos(phase+pi), sin(phase+pi))
sample2 = iq[i] * conj(predicted2)
delta2 = phase(sample2)
if abs(delta2) < abs(delta)
// note that we do not update 'sample'.
delta = delta2
phase += alpha * delta
frequency += beta * delta
// adjust the iq sample to the PLL rotated
// sample.
iq[i] = sample
...
var idleThreshold
var thresholdFactor = 10
...
// sigThreshold is used to determine if the symbol
// is -1, +1 or 0. It's 1.3 times the idle signal
// threshold.
var sigThreshold = (idleThreshold * 0.3) + idleThreshold
// iq contains a single symbol's worth of IQ samples.
// clock alignment isn't really considered; so we'll
// get a bad packet if we have a symbol transition
// in the middle of this buffer. No attempt is made
// to correct for this yet.
var iq []complex
// avg is used to average a chunk of samples in the
// symbol buffer.
var avg float
var mid = len(iq) / 2
// midNum is used to determine how many symbols to
// average at the middle of the symbol.
var midNum = len(iq) / 50
for j := mid; j < mid+midNum; j++
avg += real(iq[j])
avg /= midNum
var symbol float
switch
case avg > sigThreshold:
symbol = 1
case avg < -sigThreshold:
symbol = -1
default:
symbol = 0
// update the idleThreshold using the thresholdFactor
// to average the idleThreshold over more samples to
// get a better idea of average noise.
idleThreshold = (
(idleThreshold*(thresholdFactor-1) + symbol) \
/ thresholdFactor
)
// write symbol to output somewhere
...
+1
, -1
or 0
, we
can frame / unframe the data contained in the stream, and decode Packets
contained inside, coming next in Part 4!
1+0i
, which will transmit a pure sine wave at
exactly the center frequency of the radio.
var sine []complex
for i := range sine
sine[i] = complex(1.0, 0.0)
var sine []complex
for i := range sine
sine[i] = complex(-1.0, 0.0)
// angle is in radians - here we have
// 1.5 Pi (3 Tau) or 270 degrees.
var angle = pi * 1.5
// amplitude controls the transmitted
// strength of the carrier wave.
var amplitude = 1.0
// output buffer as above
var sine []complex
for i := range sine
sine[i] = complex(
amplitude*cos(angle),
amplitude*sin(angle),
)
var sampleRate = 2,621,440
var baudRate = 1024
// This represents the number of IQ samples
// required to send a single symbol at the
// provided baud and sample rate. I picked
// two numbers in order to avoid half samples.
// We will transmit each symbol in blocks of
// this size.
var samplesPerSymbol = sampleRate / baudRate
var samples = make([]complex, samplesPerSymbol)
// symbol is one of 1, -1 or 0.
for each symbol in symbols
for i := range samples
samples[i] = complex(symbol, 0)
// write the samples out to an output file
// or radio.
write(samples)
1+1i
is represented as 1.0 1.0
and the complex number
-1-1i
is represented as -1.0 -1.0
. Unless otherwise specified, all the
IQ samples and pseudocode to follow assumes interleaved float32 IQ data
streams.
Example interleaved float32 file (10Hz Wave at 1024 Samples per Second)
1+1i
is represented as 0xFF 0xFF
and the complex number
-1-1i
is represented as 0x00 0x00
. The complex number 0+0i
is not easily
representable since half of 0xFF
is 127.5
.
Complex Number | Representation |
1+1i | []uint8 0xFF, 0xFF |
-1+1i | []uint8 0x00, 0xFF |
-1-1i | []uint8 0x00, 0x00 |
0+0i | []uint8 0x80, 0x80 or []uint8 0x7F, 0x7F |
...
in = []uint8 0x7F, 0x7F
real = (float(iq[0])-127.5)/127.5
imag = (float(iq[1])-127.5)/127.5
out = complex(real, imag)
....
int8
values can
range between -128
to 127
, which means there s bit of ambiguity in
how +1, 0 and -1 are represented. Either you can create perfectly symmetric
ranges of values between +1 and -1, but 0 is not representable, have more
possible values in the negative range, or allow values above (or just below)
the maximum in the range to be allowed.
Within my implementation, my approach has been to scale based on the max
integer value of the type, so the lowest possible signed value is actually
slightly smaller than -1
. Generally, if your code is seeing values that low
the difference in step between -1 and slightly less than -1 isn t very
significant, even with only 8 bits. Just a curiosity to be aware of.
Complex Number | Representation |
1+1i | []int8 127, 127 |
-1+1i | []int8 -128, 127 |
-1-1i | []int8 -128, -128 |
0+0i | []int8 0, 0 |
...
in = []int8 -5, 112
real = (float(in[0]))/127
imag = (float(in[1]))/127
out = complex(real, imag)
....
Complex Number | Representation |
1+1i | []int16 32767, 32767 |
-1+1i | []int16 -32768, 32767 |
-1-1i | []int16 -32768, -32768 |
0+0i | []int16 0, 0 |
...
in = []int16 -15072, 496
// shift left 4 bits (16 bits - 12 bits = 4 bits)
// to move from LSB aligned to MSB aligned.
in[0] = in[0] << 4
in[1] = in[1] << 4
real = (float(in[0]))/32767
imag = (float(in[1]))/32767
out = complex(real, imag)
....
$ curl http://44.127.0.8:8000/
* Connected to 44.127.0.8 (44.127.0.8) port 8000 (#0)
> GET / HTTP/1.1
> Host: localhost:1313
> User-Agent: curl/7.79.1
> Accept: */*
>
* Mark bundle as not supporting multiuse
* HTTP/1.0, assume close after body
< HTTP/1.0 200 OK
< Content-Length: 236
<
____ _ ____ _ ______ _ _____
_ \ / \ / ___ / / _ \ / \ _ _
_) / _ \ ' / _) / _ \
__/ ___ \ ___ . \ _ < / ___ \
_ /_/ \_\____ _ \_\_ \_\/_/ \_\_
* Closing connection 0
$ ip link add vevx0a type veth peer name vevx0z $ ip addr add 169.254.0.2/31 dev vevx0a $ ip addr add 169.254.0.3/31 dev vevx0z $ ip link add vxlan0 type vxlan id 42 \ local 169.254.0.2 dev vevx0a dstport 4789 $ # Note the above 'dev' and 'local' ip are set here $ ip addr add 10.10.10.1/24 dev vxlan0results in vxlan0 listening on all interfaces, not just
vevx0z
or vevx0a
. To prove it to myself, I spun up a docker container (using a completely different network bridge with no connection to any of the interfaces above), and ran a Go program to send VXLAN UDP packets to my bridge host:
$ docker run -it --rm -v $(pwd):/mnt debian:unstable /mnt/spam 172.17.0.1:4789 $which results in packets getting injected into my vxlan interface
$ sudo tcpdump -e -i vxlan0 tcpdump: verbose output suppressed, use -v[v]... for full protocol decode listening on vxlan0, link-type EN10MB (Ethernet), snapshot length 262144 bytes 21:30:15.746754 de:ad:be:ef:00:01 (oui Unknown) > Broadcast, ethertype IPv4 (0x0800), length 64: truncated-ip - 27706 bytes missing! 33.0.0.0 > localhost: ip-proto-114 21:30:15.746773 de:ad:be:ef:00:01 (oui Unknown) > Broadcast, ethertype IPv4 (0x0800), length 64: truncated-ip - 27706 bytes missing! 33.0.0.0 > localhost: ip-proto-114 21:30:15.746787 de:ad:be:ef:00:01 (oui Unknown) > Broadcast, ethertype IPv4 (0x0800), length 64: truncated-ip - 27706 bytes missing! 33.0.0.0 > localhost: ip-proto-114 21:30:15.746801 de:ad:be:ef:00:01 (oui Unknown) > Broadcast, ethertype IPv4 (0x0800), length 64: truncated-ip - 27706 bytes missing! 33.0.0.0 > localhost: ip-proto-114 21:30:15.746815 de:ad:be:ef:00:01 (oui Unknown) > Broadcast, ethertype IPv4 (0x0800), length 64: truncated-ip - 27706 bytes missing! 33.0.0.0 > localhost: ip-proto-114 21:30:15.746827 de:ad:be:ef:00:01 (oui Unknown) > Broadcast, ethertype IPv4 (0x0800), length 64: truncated-ip - 27706 bytes missing! 33.0.0.0 > localhost: ip-proto-114 21:30:15.746870 de:ad:be:ef:00:01 (oui Unknown) > Broadcast, ethertype IPv4 (0x0800), length 64: truncated-ip - 27706 bytes missing! 33.0.0.0 > localhost: ip-proto-114 21:30:15.746885 de:ad:be:ef:00:01 (oui Unknown) > Broadcast, ethertype IPv4 (0x0800), length 64: truncated-ip - 27706 bytes missing! 33.0.0.0 > localhost: ip-proto-114 21:30:15.746899 de:ad:be:ef:00:01 (oui Unknown) > Broadcast, ethertype IPv4 (0x0800), length 64: truncated-ip - 27706 bytes missing! 33.0.0.0 > localhost: ip-proto-114 21:30:15.746913 de:ad:be:ef:00:01 (oui Unknown) > Broadcast, ethertype IPv4 (0x0800), length 64: truncated-ip - 27706 bytes missing! 33.0.0.0 > localhost: ip-proto-114 10 packets captured 10 packets received by filter 0 packets dropped by kernel(the program in question is the following:)
package main import ( "net" "os" "github.com/mdlayher/ethernet" "github.com/mdlayher/vxlan" ) func main() conn, err := net.Dial("udp", os.Args[1]) if err != nil panic(err) for i := 0; i < 10; i++ vxf := &vxlan.Frame VNI: vxlan.VNI(42), Ethernet: ðernet.Frame Source: net.HardwareAddr 0xDE, 0xAD, 0xBE, 0xEF, 0x00, 0x01 , Destination: net.HardwareAddr 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF , EtherType: ethernet.EtherTypeIPv4, Payload: []byte("Hello, World!"), , frb, err := vxf.MarshalBinary() if err != nil panic(err) _, err = conn.Write(frb) if err != nil panic(err)When using vxlan, be absolutely sure all hosts that can address any interface on the host are authorized to send arbitrary packets into any VLAN that box can send to, or there s very careful and specific controls and firewalling. Note this includes public interfaces (e.g., dual-homed private network / internet boxes), or any type of dual-homing (VPNs, etc).
Name | Max dBm | stdev dBm | stdev min dBm | stdev max dBm |
---|---|---|---|---|
HackRF | +12.6 | +/-2.0 | +/-0.8 | +/-3.0 |
PlutoSDR | +3.3 | +/-2.0 | +/-0.9 | +/-3.7 |
B210 | +18.3 | +/-2.6 | +/-1.4 | +/-6.0 |
433.05MHz
up to 434.79MHz
. I fired up my trusty waterfall tuned to a
center frequency of 433.92MHz
(since it s right in the middle of the band, and
it let me see far enough up and down the band to spot the remote) and pressed
a few buttons. Imagine my surprise when I realize the operational frequency of
this device is 433.920MHz
, exactly dead center. Weird, but lucky!
1000
or a 1110
which caused me to
realize this was encoded using something I saw documented elsewhere, where the
0 is a short pulse, and a 1 is a long pulse, not unlike morse code, but
where each symbol takes up a fixed length of time (monospace morse code?).
Working on that assumption, I changed my inspectrum symbol width, and
demodulated a few more by hand. This wound up demodulating nicely (and the
preamble / clock sync could be represented as repeating 0
s, which is handy!)
and gave us a symbol rate of 612(ish) symbols per second a lot closer to
what I was expecting.
0000000000110101100100010
(treat a short pulse as a 0, and
a long pulse as a 1). If you re interested in following along at home, click on
the inspectrum image, and write down the bits you see, and compare it to what
I have!
Right, so it looks like from what we can tell so far that the packet looks
something like this:
Button | Demod'd Bits |
On | 0000000000110101100100010 |
Off | 00000000001101011001010000 |
Dim Up | 0000000000110101100110100 |
Dim Down | 0000000000110101100100100 |
Timer 1h | 0000000000110101100110010 |
Timer 2h | 0000000000110101100100110 |
Timer 4h | 0000000000110101100100000 |
Dim 100% | 0000000000110101000101010 |
Dim 75% | 00000000001101010001001100 |
Dim 50% | 00000000001101010001001000 |
Dim 25% | 0000000000110101000100000 |
1
as 1110
, and a 0
as 1000
. This let me allocate IQ
space for the symbol, break the bit into 4 symbols, and if that symbol is 1,
write out values from a carrier wave (cos
in the real
values, and sin
in
the imaginary
values) to the buffer. Now that I can go from bits to IQ data,
I can transmit that IQ data using my PlutoSDR or HackRF and try and control my
tree. I gave it a try, and the tree blinked off!
Success!
But wait that s not enough for me I know I can t just demodulate bits and
try and replay the bits forever there s stuff like addresses and keys and
stuff, and I want to get a second one of these working. Let s take a look at
the bits to see if we spot anything fun & interesting.
At first glance, a few things jumped out at me as being weird? First is
that the preamble is 10 bits long (fine, let s move along - maybe it
just needs 8 in a row and there s two to ensure clocks sync?). Next is that
the messages are not all the same length. I double (and triple!) checked
the messages, and it s true, the messages are not all the same length. Adding
an extra bit at the end didn t break anything, but I wonder if that s just due
to the implementation rather than the protocol.
But, good news, it looks like we have a stable prefix to the messages from the
remote must be my device s address! The stable 6 bits that jump out right
away are 110101
. Something seems weird, though, 6 bits is a bit awkward, even
for a bit limited embedded device. Why 6? But hey, wait, we had 10 bits in the
preamble, what if we have an 8 bit address meaning my device is 00110101
,
and the preamble is 8 0
symbols! Those are numbers that someone working on
an 8 bit aligned platform would pick! To test this, I added a 0
to the
preamble to see if the message starts at the first 1
, or if it requires all
the bits to be fully decoded, and lo and behold, the tree did not turn on or
off. This would seem to me to confirm that the 0s are part of the address,
and I can assume we have two 8 bit aligned bytes in the prefix of the message.
1001
or 0001
, but other than that, there s a lot of chaos. This is where
things get really squishy. I needed more information to try and figure this out,
but no matter how many times I sent a command it was always the same bits (so,
no counters), and things feel very opaque still.
The only way I was going to make any progress is to get another switch and see
how the messages from the remote change. Off to Amazon I went, and ordered
another switch from the same page, and eagerly waited its arrival.
[]byte 0x00, 0x35
, that means that the vast majority of bits sent are always
going to be the same for any button press on any remote made by this vendor.
Seems like a waste of bits to me, but hey, what do I know.
Additionally, this also tells us the trailing zeros are not part of the data
encoding scheme, which is progress!
Button | Scancode Bits | Integer |
On | 10010001 | 145 / 0x91 |
Off | 10010100 | 148 / 0x94 |
Dim Up | 10011010 | 154 / 0x9A |
Dim Down | 10010010 | 146 / 0x92 |
Timer 1h | 10011001 | 154 / 0x99 |
Timer 2h | 10010011 | 147 / 0x93 |
Timer 4h | 10010000 | 144 / 0x90 |
Dim 100% | 00010101 | 21 / 0x15 |
Dim 75% | 00010011 | 19 / 0x13 |
Dim 50% | 00010010 | 18 / 0x12 |
Dim 25% | 00010000 | 16 / 0x10 |
0x05
is 0x03
+ 0x02
or if it s 0x01
+ 0x04
. On the other hand, treating it as two 4-bit
integers won t work for 0x10
to 0x15
(since they need 5 bits to
represent). It s also likely the most significant bit is a combo indicator,
which only leaves 7 bits for the actual keypress data. Stuffing 10 bits of data
into 7 bits is likely resulting in some really intricate bit work.
On a last ditch whim, I tried to XOR the math into working, but some initial
brute forcing to make the math work given the provided examples did not result
in anything. It could be a bitpacked field that I don t understand, but I don t
think I can make progress on that without inside knowledge and much more work.
Here s the table containing the numbers I was working off of:
Keys | Key Codes | Scancode |
S3 + S9 | 0x01 + 0x03 | 0x96 |
S6 + S12 | 0x07 + 0x09 | 0x94 |
S22 + S10 | 0x0D + 0x0F | 0x3F |
ab-cd-ef-ab-cd-ef.by-mac.paultag.house.
, which is harder to accedentally collide.
dnsmasq.leases, watch inotify
for IN_MODIFY
signals, and sync the records to AWS Route 53.
I pushed it up to my GitHub as DNSync.
PRs welcome!
@WMATA
, where it returns the closest station if you Yo it your location. For hilarity, feel free to Yo
it from outside DC.
For added fun, and puns, I wrote a dbus
proxy for the API as weel, at wmata-dbus, so you can query the next train over dbus. One thought was to make a GNOME Shell extension to tell me when the next train is. I d love help with this (or pointers on how to learn how to do this right).
minica tag@domain.tls domain.tld
will issue two TLS certs (one with a Client EKU, and one server) issued from a single CA.
Next time you re in need of a few TLS keys (without having to worry about stuff like revocation or anything), this might be the quickest way out!
libpam-pkcs11
is a pretty
easy to use module that will let you log into your system locally using a
PKCS#11 token locally.
One of the least documented things, though, was how to use an OpenSC PKCS#11
token in Chrome. First, close all web browsers you have open.
sudo apt-get install libnss3-tools
certutil -U -d sql:$HOME/.pki/nssdb
modutil -add "OpenSC" -libfile /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so -dbdir sql:$HOME/.pki/nssdb
modutil -list "OpenSC" -dbdir sql:$HOME/.pki/nssdb
modutil -enable "OpenSC" -dbdir sql:$HOME/.pki/nssdb
nss
to use, so let's double
check that the tokens are registered:
certutil -U -d sql:$HOME/.pki/nssdb
certutil -L -h "OpenSC" -d sql:$HOME/.pki/nssdb
modutil -delete "OpenSC" -dbdir sql:$HOME/.pki/nssdb
0x06 0x14 0x00 0x04 0x00 0x34 0x11 0x00 0x00 0x5D
Some were 10 bytes, other were 11, and most started with similar looking
things. The first byte was usually a 0x06
or 0x07
, followed by two
bytes 0x14 0x00
, and either a 0x04
or 0x05
. Since the first few bytes
were similarly structured, I assumed the first octet (either 0x06
or 0x07
)
was actually a length, since the first 4 octets seemed always present.
So, my best guess is that we have a Length byte at index 0, followed by
two bytes for the Protocol, a flag for if you're Reading or Writing (best
guess on that one), and opaque data following that. Sometimes it's a const
of sorts, and sometimes an octet (either little or big endian, confusingly).
Length
Read / Write
Protocol Data
---- ------------------------
0x06 0x14 0x00 0x04 0x00 0x34 0x11 0x00 0x00 0x5D
(setv *commands*
; function type family control
'((power-on nil nil (0x06 0x14 0x00 0x04 0x00 0x34 0x11 0x00 0x00 0x5D))
(power-off nil nil (0x06 0x14 0x00 0x04 0x00 0x34 0x11 0x01 0x00 0x5E))
(power-status const power (0x07 0x14 0x00 0x05 0x00 0x34 0x00 0x00 0x11 0x00 0x5E))
(reset nil nil (0x06 0x14 0x00 0x04 0x00 0x34 0x11 0x02 0x00 0x5F))
...
(setv *consts*
'((power ((on (0x00 0x00 0x01))
(off (0x00 0x00 0x00))))
(freeze ((on (0x00 0x00 0x01))
(off (0x00 0x00 0x00))))
...
(defn make-api-function [function type family data]
(defn ~function [serial]
(import [PJD5132.dsl [interpret-response]]
[PJD5132.serial [read-response/raw]])
(serial.write (bytearray [~@data]))
(interpret-response ~(str type) ~(str family) (read-response/raw serial))))
(import [PJD5132.commands [*commands*]]
[PJD5132.dsl [make-api-function]])
(list (map (fn [(, function type family command)]
(make-api-function function type family command)) *commands*)))
power-on
from *commands*
which
takes a single argument (serial
) for the serial port, and it'll send a
command, and return the response. The best part about all this is you only
have to define the data once in a list, and the rest comes for free.
Finally, I do want to be able to turn my projector on and off over the network
so I went ahead and make a Flask "API" on top of this. First, let's define
a macro to define Flask routes:
(defmacro defroute [name root &rest methods]
(import os.path)
(defn generate-method [path method status]
(with-decorator (app.route ~path) (fn []
(import [PJD5132.api [~method ~(if status status method)]])
(try (do (setv ret (~method serial-line))
~(if status (setv ret (~status serial-line)))
(json.dumps ret))
(except [e ValueError]
(setv response (make-response (.format "Fatal Error: ValueError: " (str e))))
(setv response.status-code 500)
response)))))
(setv path (.format "/projector/ " name))
(setv actions (dict methods))
(do ~(generate-method path root nil)
~@(list-comp (generate-method (os.path.join path method-path) method root)
[(, method-path method) methods])))
power
route, which will expand out into the Flask route code above.
(defroute power
power-status
("on" power-on)
("off" power-off))
$ curl http://192.168.1.50/projector/power
"off"
$ curl http://192.168.1.50/projector/power/on
"on"
$ curl http://192.168.1.50/projector/power
"on"
$ curl 192.168.1.50/projector/volume
10
$ curl 192.168.1.50/projector/volume/decrease
9
$ curl 192.168.1.50/projector/volume/decrease
8
$ curl 192.168.1.50/projector/volume/decrease
7
$ curl 192.168.1.50/projector/volume/increase
8
$ curl 192.168.1.50/projector/volume/increase
9
$ curl 192.168.1.50/projector/volume/increase
10
Next.