Paul Tagliamonte: wrapping it all up (Part 8/12)
If you're looking for it, the intro to the PHY (layer 1) that this belongs at a high level and listing of the parts that make it up (including this one) is on the phy post.
Encoder), and something that
takes chunks of IQ samples in, writing out decoded bytes (which we ll call the
Decoder).
Encoder
Let s begin with the Encoder, since it s slightly less involved. I ve tried
to make this a bit more accessible by drawing a diagram out before describing
the order of operations, so that it s possible to follow along visually.
While the process here can look like a lot, it s really not that bad.
We begin by taking the incoming bytes, converting the bytes into bits, and chunk
those bits into parts which are sized to fit completely within an LDPC message. We will then
encode incoming data into LDPC messages, using our
configured LDPC Matrix (the table we appropriated from 802.3an). Next, we
apply whitning over all the bits in our
encoded (and packed) LDPC messages, using our configured whitening constant. In
the case of QPSK, pairs of bits will then be
modulated into a QAM subcarrier, where each
QAM point represents a range of bits in the message. We ll go through each of
those modulated IQ subcarriers, and
set each corresponding data subcarrier
in order, for each OFDM symbol contained in the pigeon Burst. The preamble configuration
is then used to generate (or, more likely, can be used at startup to precompute)
the Schmidl-Cox preamble, which is written to the
first IQ samples in our output IQ buffer. Finally, we will do a series of
inverse FFT operations to convert each
OFDM symbol to the time domain, including their cyclic prefix.
Let s take a look at doing that, but in code this time now:
// (lightly edited for clarity)
impl Encoder
..
/// Encode the provided bits into the output time-domain IQ samples.
fn encode(
&mut self,
dst: &mut [IQ],
src: &Vector,
) -> Result<Burst, Error>
let src =
let mut raw = Vector::new(self.fec.message_len());
// Set raw 's data bits, compute and set LDPC
// checkbits.
self.fec.add(&mut raw, src);
// Apply whitening, and return
raw.xor(&self.whitening)
;
// copy in the precomputed schmidl-cox preamble to dst
let preamble_len = self.preamble_iq.len();
dst[..preamble_len].copy_from_slice(&self.preamble_iq);
// allocate a new (frequency domain) 'Burst' container.
let mut burst = Burst::new(
&self.mode.ofdm.plan,
self.mode.ofdm.symbols
);
// modulate bits from 'src' as iq, and set each
// data subcarrier for each ofdm symbol in the
// burst.
self.burst_encoder.multiplex(&mut burst, &src);
// convert from frequency-domain data into time
// domain iq samples, writing out ofdm symbols
// and cyclic prefixes to dst .
self.burst_encoder
.transform(&mut dst[preamble_len..], &burst)?;
// normalize all IQ samples; the maximum magnitude
// in the IQ buffer may be very small, which weakens
// our transmitted signal. Scale all IQ samples such
// that the maximum IQ sample magnitude will be '1.0'.
dst.norm();
Ok(burst)
Using the Encoder should hopefully be fairly straightforward we ll
give it a bag of bytes, and get back some IQ samples that we can ask our
nearest SDR to transmit.
As for what happens on the other end?
Decoder
Next up is the mirror image of our Encoder the, imaginatively named,
Decoder. The Decoder is slightly more involved (since it has to find the
packet in the IQ stream, as well as correct for channel error(s)), so we ll do
the same thing as above start with a diagram. My hope is going over the
Encoder first helps us only really focus on the new stuff, otherwise it
should feel like running the Encoder backwards.
Here, we start with an incoming stream of IQ, where we will process
scan detections as they come in
from our Schmidl-Cox detector and burst Scanner. This will give us a snippit
of IQ, sized to exactly our Burst. We ll begin to correct our IQ samples by
first doing
frequency estimation and correction
in the time domain using our preamble and ofdm configuration. We ll then do
a series of inverse FFTs to extract each OFDM symbol
in our Burst, where we can then do
channel estimation and correction.
With the OFDM symbols (hopefully) good enough, we can now
map each data subcarrier back to bits, and
unapply whitning. The resulting
bits are then chunked back up into LDPC messages,
which are then checked, and concatanated data extracted. Finally the bits
are turned back into bytes, which are written to our output buffer.
However, before we get into the code to do this there s one last detail. We
know bursts won t overlap (if they do, it s likely not possible to recover
right now even though other PHYs can and do), so any time we see something we believe
to be a burst, we can skip ahead by the burst s (constant) length within
the IQ, and avoid trying to decode anything else in there.
The nice side-effect here is this also gives us an interesting property for the
Decoder namely, we know the maximum number of Burst detections we can
get for a given block of incoming IQ data if they were packed end-to-end and
we can pre-allocate the memory we need, avoiding allocations for every
demodulation attempt (which may or may not even be a valid Burst).
This pre-allocated block of memory to hold the burst s data is something that
I ve called a frame buffer internally. Each frame buffer contains exactly
sized buffers to hold decoded information from the burst an iq buffer
that is exactly the same number of samples required to encode the preamble
and data, exactly the number of bits needed to store pre and post FEC data,
pre-allocated byte array, etc.
Not shockingly, the code looks like this:
#[derive(Clone)]
pub struct FrameBuffer
/// Corrected IQ samples
pub samples: Samples,
/// post-correction OFDM burst
pub burst: Burst,
/// demodulated bits from the OFDM burst
pub bits: Vector,
/// demodulated bits from the OFDM burst,
/// after FEC, and cleaned
pub raw_bits: Vector,
/// Layer 2 contents of the Frame
pub contents: Vec<u8>,
Of course, that alone is handy but we need to use them. So let s go ahead
and do what we promised above each Decoder uses a fixed number of
pre-allocated FrameBuffers to store packets in-flight, packed into what
is, creatively, called FrameBuffers within my code.
As an aside, I likely should have called this a
Memory Pool, since that s the
common
and accepted
name for this design pattern but being stuck with unfortunate names is
the burden of those of us who stumble into sensible ideas over time. The only
nuance here is that I use the pools strictly sequentially we only save
the FrameBuffer if the LDPC checksum is correct, allowing us to only keep
track of how many successful packets we have and being able to get the valid
FrameBuffers, rather than storing a handle to each FrameBuffer as we go
a promise that most memory pools do not make, since blocks can usually be
taken and returned in any order.
Let s go ahead and do the whole Decoder dance now:
// (lightly edited for clarity)
impl Decoder
..
/// Process incoming IQ for Pigeon Bursts, and
/// demodulate them.
pub fn decode(
&mut self,
buf: &[IQ],
) -> Result<Vec<(Detection, &FrameBuffer)>, Error>
let mut ret = Vec::new();
let mode = self.scanner.mode().clone();
// reset the "valid frame buffer count" back to 0
self.frames.reset();
// call the scanner and get scan detections for
// this block of iq ( buf )
for detection in self.scanner.scan(buf)
// for each detection, we're (only) going to process
// the iq snippit, but pass along the metadata
// such as SNR.
let ScanDetection
snippit,
snr,
range,
m,
= detection;
// grab the next free frame buffer to work within.
let frame_buffer = self.frames.next_mut();
// Copy the snippit into the frame buffer (a mutable
// location)
frame_buffer.samples.copy_from_slice(snippit);
// estimate the frequency offset based on the
// Burst's Schmidl-Cox preamble.
let preamble_fo = preamble::estimate_frequency_offset(
&mode.preamble,
mode.rate,
&frame_buffer.samples[..mode.preamble.samples()],
);
// Shift the IQ stream by the estimated frequency
// offset -- hopefully we're closer to 0Hz
frame_buffer.samples.shift(mode.rate, preamble_fo);
// estimate the frequency offset based on the
// each burst's **cyclic prefix** -- exactly like
// we did with the Burst Schmidl-Cox preamble,
// but this time on each OFDM symbol.
let ofdm_fo = ofdm::estimate_frequency_offset(
&mode.ofdm,
mode.rate,
&frame_buffer.samples[mode.preamble.samples()..],
);
// Shift the IQ stream closer yet; hopefully this
// is a very small nudge even closer still to 0Hz.
frame_buffer.samples.shift(mode.rate, ofdm_fo);
// Do a bunch of inverse fft operations for each
// OFDM symbol, filling the frequency-domain Symbol
// structs in frame_buffer.burst (Burst) struct.
//
// this will also do channel estimation and
// correction before returning.
self
.decoder
.transform(
&mut frame_buffer.burst,
&frame_buffer.samples[mode.preamble.samples()..],
)?;
// "demultiplex" each data subcarrier's frequency-domain
// IQ constellation point, setting the correct bit range.
self.decoder.demultiplex(
&mut frame_buffer.raw_bits,
&frame_buffer.burst
);
// unapply whitening by XOR-ing the buffer with
// the well-known whitening vector.
frame_buffer.raw_bits = frame_buffer.raw_bits.xor(
&self.whitening);
// verify that the LDPC message(s) are all correct,
// and if so, concatanate the the data bits (no check
// bits) to the bits vector.
if self.fec.decode(
&mut frame_buffer.bits,
&frame_buffer.raw_bits
).is_err()
// this is where invalid packets fail. we gave it a good go.
// next packet please.
continue;
// copy the raw bits out, as bytes, to the contents buffer.
frame_buffer.bits.copy_as_bytes(&mut frame_buffer.contents);
// store metadata/metrics on the demodulation.
ret.push(Detection
m,
snr,
index: range.start,
);
// save the contents of this frame buffer (don't
// reuse this buffer next go-around).
self.frames.save();
// We're going to take out the borrow on the frame at
// the end since we don't want to deal with telling the
// compiler via code gymnastics that the mut and non-mut
// borrows are OK since they're non-overlapping.
Ok(ret.into_iter().zip(self.frames.iter()).collect())
Phew. That was kinda a lot. In fact it s basically the whole thing. This
function is as close to how do you read an OFDM packet as it gets, and
perhaps the most important part of this whole series. Beyond that, though, this
is a huge conceptual unlock. This means we now have an incredibly powerful
primitive; the ability to take bytes and go to/from IQ samples over the air.
Let s talk about pigeon modes
// (lightly edited for clarity)
impl Encoder
..
/// Encode the provided bits into the output time-domain IQ samples.
fn encode(
&mut self,
dst: &mut [IQ],
src: &Vector,
) -> Result<Burst, Error>
let src =
let mut raw = Vector::new(self.fec.message_len());
// Set raw 's data bits, compute and set LDPC
// checkbits.
self.fec.add(&mut raw, src);
// Apply whitening, and return
raw.xor(&self.whitening)
;
// copy in the precomputed schmidl-cox preamble to dst
let preamble_len = self.preamble_iq.len();
dst[..preamble_len].copy_from_slice(&self.preamble_iq);
// allocate a new (frequency domain) 'Burst' container.
let mut burst = Burst::new(
&self.mode.ofdm.plan,
self.mode.ofdm.symbols
);
// modulate bits from 'src' as iq, and set each
// data subcarrier for each ofdm symbol in the
// burst.
self.burst_encoder.multiplex(&mut burst, &src);
// convert from frequency-domain data into time
// domain iq samples, writing out ofdm symbols
// and cyclic prefixes to dst .
self.burst_encoder
.transform(&mut dst[preamble_len..], &burst)?;
// normalize all IQ samples; the maximum magnitude
// in the IQ buffer may be very small, which weakens
// our transmitted signal. Scale all IQ samples such
// that the maximum IQ sample magnitude will be '1.0'.
dst.norm();
Ok(burst)
Encoder the, imaginatively named,
Decoder. The Decoder is slightly more involved (since it has to find the
packet in the IQ stream, as well as correct for channel error(s)), so we ll do
the same thing as above start with a diagram. My hope is going over the
Encoder first helps us only really focus on the new stuff, otherwise it
should feel like running the Encoder backwards.
Here, we start with an incoming stream of IQ, where we will process
scan detections as they come in
from our Schmidl-Cox detector and burst Scanner. This will give us a snippit
of IQ, sized to exactly our Burst. We ll begin to correct our IQ samples by
first doing
frequency estimation and correction
in the time domain using our preamble and ofdm configuration. We ll then do
a series of inverse FFTs to extract each OFDM symbol
in our Burst, where we can then do
channel estimation and correction.
With the OFDM symbols (hopefully) good enough, we can now
map each data subcarrier back to bits, and
unapply whitning. The resulting
bits are then chunked back up into LDPC messages,
which are then checked, and concatanated data extracted. Finally the bits
are turned back into bytes, which are written to our output buffer.
However, before we get into the code to do this there s one last detail. We
know bursts won t overlap (if they do, it s likely not possible to recover
right now even though other PHYs can and do), so any time we see something we believe
to be a burst, we can skip ahead by the burst s (constant) length within
the IQ, and avoid trying to decode anything else in there.
The nice side-effect here is this also gives us an interesting property for the
Decoder namely, we know the maximum number of Burst detections we can
get for a given block of incoming IQ data if they were packed end-to-end and
we can pre-allocate the memory we need, avoiding allocations for every
demodulation attempt (which may or may not even be a valid Burst).
This pre-allocated block of memory to hold the burst s data is something that
I ve called a frame buffer internally. Each frame buffer contains exactly
sized buffers to hold decoded information from the burst an iq buffer
that is exactly the same number of samples required to encode the preamble
and data, exactly the number of bits needed to store pre and post FEC data,
pre-allocated byte array, etc.
Not shockingly, the code looks like this:
#[derive(Clone)]
pub struct FrameBuffer
/// Corrected IQ samples
pub samples: Samples,
/// post-correction OFDM burst
pub burst: Burst,
/// demodulated bits from the OFDM burst
pub bits: Vector,
/// demodulated bits from the OFDM burst,
/// after FEC, and cleaned
pub raw_bits: Vector,
/// Layer 2 contents of the Frame
pub contents: Vec<u8>,
Decoder uses a fixed number of
pre-allocated FrameBuffers to store packets in-flight, packed into what
is, creatively, called FrameBuffers within my code.
As an aside, I likely should have called this a
Memory Pool, since that s the
common
and accepted
name for this design pattern but being stuck with unfortunate names is
the burden of those of us who stumble into sensible ideas over time. The only
nuance here is that I use the pools strictly sequentially we only save
the FrameBuffer if the LDPC checksum is correct, allowing us to only keep
track of how many successful packets we have and being able to get the valid
FrameBuffers, rather than storing a handle to each FrameBuffer as we go
a promise that most memory pools do not make, since blocks can usually be
taken and returned in any order.
Let s go ahead and do the whole Decoder dance now:
// (lightly edited for clarity)
impl Decoder
..
/// Process incoming IQ for Pigeon Bursts, and
/// demodulate them.
pub fn decode(
&mut self,
buf: &[IQ],
) -> Result<Vec<(Detection, &FrameBuffer)>, Error>
let mut ret = Vec::new();
let mode = self.scanner.mode().clone();
// reset the "valid frame buffer count" back to 0
self.frames.reset();
// call the scanner and get scan detections for
// this block of iq ( buf )
for detection in self.scanner.scan(buf)
// for each detection, we're (only) going to process
// the iq snippit, but pass along the metadata
// such as SNR.
let ScanDetection
snippit,
snr,
range,
m,
= detection;
// grab the next free frame buffer to work within.
let frame_buffer = self.frames.next_mut();
// Copy the snippit into the frame buffer (a mutable
// location)
frame_buffer.samples.copy_from_slice(snippit);
// estimate the frequency offset based on the
// Burst's Schmidl-Cox preamble.
let preamble_fo = preamble::estimate_frequency_offset(
&mode.preamble,
mode.rate,
&frame_buffer.samples[..mode.preamble.samples()],
);
// Shift the IQ stream by the estimated frequency
// offset -- hopefully we're closer to 0Hz
frame_buffer.samples.shift(mode.rate, preamble_fo);
// estimate the frequency offset based on the
// each burst's **cyclic prefix** -- exactly like
// we did with the Burst Schmidl-Cox preamble,
// but this time on each OFDM symbol.
let ofdm_fo = ofdm::estimate_frequency_offset(
&mode.ofdm,
mode.rate,
&frame_buffer.samples[mode.preamble.samples()..],
);
// Shift the IQ stream closer yet; hopefully this
// is a very small nudge even closer still to 0Hz.
frame_buffer.samples.shift(mode.rate, ofdm_fo);
// Do a bunch of inverse fft operations for each
// OFDM symbol, filling the frequency-domain Symbol
// structs in frame_buffer.burst (Burst) struct.
//
// this will also do channel estimation and
// correction before returning.
self
.decoder
.transform(
&mut frame_buffer.burst,
&frame_buffer.samples[mode.preamble.samples()..],
)?;
// "demultiplex" each data subcarrier's frequency-domain
// IQ constellation point, setting the correct bit range.
self.decoder.demultiplex(
&mut frame_buffer.raw_bits,
&frame_buffer.burst
);
// unapply whitening by XOR-ing the buffer with
// the well-known whitening vector.
frame_buffer.raw_bits = frame_buffer.raw_bits.xor(
&self.whitening);
// verify that the LDPC message(s) are all correct,
// and if so, concatanate the the data bits (no check
// bits) to the bits vector.
if self.fec.decode(
&mut frame_buffer.bits,
&frame_buffer.raw_bits
).is_err()
// this is where invalid packets fail. we gave it a good go.
// next packet please.
continue;
// copy the raw bits out, as bytes, to the contents buffer.
frame_buffer.bits.copy_as_bytes(&mut frame_buffer.contents);
// store metadata/metrics on the demodulation.
ret.push(Detection
m,
snr,
index: range.start,
);
// save the contents of this frame buffer (don't
// reuse this buffer next go-around).
self.frames.save();
// We're going to take out the borrow on the frame at
// the end since we don't want to deal with telling the
// compiler via code gymnastics that the mut and non-mut
// borrows are OK since they're non-overlapping.
Ok(ret.into_iter().zip(self.frames.iter()).collect())
My Debian contributions this month were all 


One of those arguments became its own fix. My wording implied symbols were
unpacked inside the isolated sbuild environment (the consequence was I was
handed a
Every merge request above landed with unit tests, but those only tell you
that the pieces behave correctly. What Colin and I wanted was a real gdb
fetching real symbols from a real instance, so
Welcome to post 59 in the
I spent 6 hours working with an AI to write a bash script that would show me both active & inactive reserved DHCP addresses, as well as active leases from pool. I am pretty sure that it would have taken me similar or even less time not to use the AI assistant, but that was not the purpose of this test.
Many years ago, before I had my sewing pattern website, I made myself a
simple corset cover according to
It worked, I wore it. Years later I saw
Then, after finishing my vampire shirt, I noticed that I had just enough
fabric left for a corset cover, and by just enough I really mean just
enough, as I discovered when laying the pattern on the fabric.
So I dug in my files to get the original pattern I used, brought it up
to date, and added the missing details such as the pleating guides that
I had skipped when making the pattern just for myself. Doing so I
realized that on my old cover I had done the fake pleat in the front
wrong, making just a single pleat instead of a box pleat. Also, I
originally directly gathered the sleeves in the armscyes, but watching
the book again I realized that the sleeves were made up of a gathered
ruffle plus a straight band.
Both issues were fixed and I could cut the fabric and start sewing. By
machine, including using a narrow hem foot instead of sewing rolled hems
by hand as my instinct kept reminding me would have looked neater.
But this is a garment from a sewing machine time, and probably one that
in many cases would have been bought from a mass producer, and it s
underwear, so there is no real need for the hems to be perfect, as it s
going to be hidden anyway. But most importantly, I wanted to write
instructions for machine sewing, for a change, and so I had to machine
sew all steps that I had to take pictures of.
I did do the buttonholes by hand, because I hate the buttonhole
attachment on my machine, and the buttonhole attachment hates me.
I used a lighter weight fabric for the sleeve ruffles, both because I
didn t have a big enough piece of main fabric not to have to piece them,
and because I felt that it looks better, as it s the same voile I used
for the ruffles on the vampire shirt.
When it came to the beading lace, I had two that I had bought more or
less thinking about this project: the earlier one was narrow and
suitable to do its job, but the one I had bought more recently was
taller, with an edge that made it suitable to give more fullness to the
bust when gathered up.
I contemplated for a short while, and then decided to go for fullness
and use the taller border for the top edge, but the smaller one at the
waist, where fullness is not wanted.
The book claimed that this pattern required little labour, and indeed it
did: even when taking step by step pictures it only took a few hours
spread over a week, plus the time to make buttonholes by hand over the
next week.
And then the reason for the whole project: I published my
Remember the
For the face I decided to make holes in the cardboard and fill them with
hot glue to make a sort of light pipe, with the LEDs pressed against
them on the inside. It s not perfect, but it mostly works.
And then everything stopped: while I waited for the PVA to dry I started
doing something else, and then there were other projects, and other, and
the clock lingered in the Pile. There was a brief interruption as I
started to paint the first coat of brown, and then I moved back to the
other projects.
Until, months later, I decided it was time to finish using the brown and
white tubes of paint that I had on my desktop, so I could put them away
However, the raspberry had quite a lot of pins, and it felt wasteful to
use it on something that basically needs one. On the other hand, I had
recently bought a few
The next step involved dealing with the temporary connections to make
them a bit more permanent: I have been using LibrePCB for that other
project, so of course what I did was grabbing a bit of perfboard and
YOLO a growing rat nest of cables over it, without bothering with
drawing any kind of schematics in advance. And having to desolder stuff
and solder it again a couple of times, because I had issues with the
difference between left and right, and with the concept of rotations in
3D space.
Everything was brought back into the case, in a mostly stable
configuration with an usb cable coming out of a hole in the back for
power and surprisingly it works.
Or at least, 95% of the issues it still has are software, plus I still
need to add a few features, so right now it lives above my desktop, with
the cable dangling close to an USB port, so that I can continue working
on that in the next few weeks.
The external look is not going to change, so there will be changes on
I ve been a