Francesco Paolo Lovergine: Terramaster NASes with Debian, take three: scaling up.
As is often the case in scientific environments, IT solutions grow chaotically,
often guided by available funds at the wrong moments, or transient choices for
transient goals. Here at work, we are not an exception. Historically, since the
birth of the Copernicus European project, the Earth Observation domain has
become progressively more demanding about storage, due to the frequency and size
of satellite acquisitions, available free of charge for research groups, free to
access, but not without maintenance and processing costs, of course.As a result, we have progressively explored low-cost solutions for downloading
and processing Sentinel constellation time series. Over a ten-year window, our
needs moved from dozens of terabytes to hundreds of terabytes. We started with
simple Synology-based solutions, very optimistically, only to discover that the
optimism was not justified. When storage needs to scale up, there are no easy
solutions.The only comparable IT environments are HPC clusters, which operate at a
completely different order of magnitude in terms of funding. In that context,
the Petabyte order of magnitude is generally achievable by using
proprietary/FOSS distributed POSIX filesystems
such as GPFS, Lustre, or even
Ceph. None of them are exempt from problems from time to time. The sad reality
is that maintaining a very large POSIX filesystem requires compromises, and no
single solution fits all requirements, starting from cost to flexibility,
scalability, and reliability.
I wrote about the hidden costs of maintaining self-hosted storage solutions
hereAlmost ten years ago, we chose GlusterFS as an acceptable compromise for a
distributed POSIX filesystem. It is not object storage, but a pure POSIX
filesystem with a simple architecture, which fits our environment, where most
people rely on the folders-and-files abstraction to organize their data. All
nodes and their bricks are equal (no dedicated metadata nodes) and can scale
without major limitations with a reasonable number of nodes. In our experience,
the real bottleneck is network speed, which can introduce high latency and cause
cache issues.We are now finally moving our old Synology-based distributed filesystem to a
pool of TerraMaster U12-500 Plus units, installed with Debian 13. Compared with
TerraMaster SOHO products, these rack-mount units can run without any non-free
firmware. You must manually enable the usual 9000 MTU jumbo frame for the 10Gb
network, which is generally required on all connected hosts and also the switch
(in our case, a MikroTik box) to properly manage full bandwidth. We measured
average performance around 200MB/sec, which aligns with the average sequential
speed of 6Gb SATA HDDs.Regarding using GlusterFS, the most obvious choice is to use it with XFS as the
filesystem for the NAS bricks; this proved in practice to be the optimal
configuration, after not-too-nice experiences with Btrfs on Synology DSM. Still,
it requires some tuning for periodic checks across the various layers of the
storage system. One is about short/long HDD testing, which I already covered in
another post. A trickier aspect is managing possible bit rot in the brick
filesystems from time to time. While Btrfs, with its COW management, can run
low-impact scrubs on both metadata and data, XFS can only partially do the same
for metadata through xfs_scrub, which is also, as clearly stated, an
experimental feature to use at your own risk. In that regard, it is much better
to run GlusterFS scrubbing instead.While xfs_scrub is as light as the SMART short self-test for HDDs, both RAID
checks and GlusterFS bitrot scrub are heavy operations that should run rarely to
avoid stressing HDDs and systems, and never at the same time. It is mandatory to
avoid overlapping all such operations, and that s a good reason to use systemd
timers and run the heavy checks in alternating months.One suboptimal aspect of our solution is the absence of direct LED control for
HDDs, which is unfortunately available only under TOS with a proprietary driver.
This is obviously required for disk replacement from time to time, when needed.
On the other hand, software support is extended to the lifetime of the Debian
distribution, with major upgrades every 3-4 years on average. Also, this is much
more flexible and efficient than using pre-built custom OSes. Of course, if you
prefer dashboards and the usual GUI candy, this isn't a viable choice; a better
option would be TrueNAS, which has a much higher memory footprint,
especially when paired with OpenZFS.
The MX record and some TXT records configured for
However, when playing it, it becomes very obvious (as you can likely see) that
there s a fair amount of packet loss (understandable) and probably some packet
collisions taking place.
After tweaking the local routing tables (and restarting
Yes! I forgot to make this one a video, so no gif. I did wind up having a bit
if trouble with a few gopher clients and IPv6 support I may send some
patches if I can find the time.
Given i m going to stub most of that out, let s take a look at what some
similar Layer 2 protocols use things like Ethernet or WiFi frames. Both
contain structured information regarding the transmitter, desired recipient,
type of data, and the higher-level data itself (such as IP packets). As a
result of attempting to learn from others, the Pigeon Layer 2 (called, simply,
I gave up the center bin (0 Hz) and 8 of the 64 bits on each side (1/4 of the
signal!) to give myself a bit of elbow room. This is perhaps definitely a bit
overkill, but it s been an extremely robust choice. If you multiply that
through, this accounts for 312.5 kHz of frequency domain padding at the high
and low end of the bandwidth, or 625.0 kHz of bandwidth which is not to be used.
My intent in placement was to pick out roughly even stretches of data bins
bracketed between pilots, with as few data subcarriers as practical outside
of a pilot (using extrapolation). I ve played a bit with my AGWN simulator(s),
as well as logging errors between two SDRs, and the configuration I have this
in has been fairly resillant (for whatever reason), and withstood a few rounds
of tweaking.
This puts the total number of data bins at 43, which, since Mode A carries data
in QPSK/QAM-4 (two bits per data subcarrier), means we can carry 86 bits of
data per OFDM symbol. That fairly modest capacity is largely due to the
modulation scheme (or fft size, but increasing that has been fraught) we re
using for Mode A but I ve made up for it by including
While the process here can look like a lot, it s really not that bad.
We begin by taking the incoming bytes, converting the bytes into bits, and chunk
those bits into parts which are sized to fit completely within an LDPC message. We will then
Here, we start with an incoming stream of IQ, where we will process
Not shockingly, the code looks like this:
As an aside, I likely should have called this a
It is widely accepted that there is an AI bubble in the financial markets at the moment. The moderate opinion is however that LLMs are constantly improving and will eventually take over more and more tasks from humans and increase productivity. But are LLMs actually getting smarter, or just better at fooling us?
There is a growing faction of technical experts that argue that LLMs are actually so bad for real progress, that they are banning their use and requiring human-only work to ensure quality and efficient use of humans time. A recent
I believe that, in the context of Debian voting, we are better off when we
know the opinion of our peers, however, since
Ranking this ballot was a mess due to the complex array of options it
encodes. I warmly thank Lucas Nussbaum for coming up with the