History, Setup
So for quite some time I have a QNAP TS-873x here, equipped with 8
Western Digital Red 10 TB disks, plus 2 WD Blue 500G M2 SSDs. The QNAP
itself has an AMD Embedded R-Series RX-421MD with 4 cores and was
equipped with 48G RAM.
Initially I had been quite happy, the system is nice. It was fast, it
was easy to get to run and the setup of things I wanted was simple
enough. All in a web interface that tries to imitate a kind of
workstation feeling and also tries to hide that it is actually a
webinterface.
Natually with that amount of disks I had a RAID6 for the disks, plus
RAID1 for the SSDs. And then configured as a big storage pool with the
RAID1 as cache. Below the hood QNAP uses MDADM Raid and LVM (if you
want, with thin provisioning), in some form of emdedded linux. The
interface allows for regular snapshots of your storage with flexible
enough schedules to create them, so it all appears pretty good.
QNAP slow
Fast forward some time and it gets annoying. First off you really
should have regular raid resyncs scheduled, and while you can set
priorities on them and have them low priority, they make the whole
system feel very sluggish, quite annoying. And sure, power failure
(rare, but can happen) means another full resync run. Also, it appears
all of the snapshots are always mounted to some
/mnt/snapshot/something place (df on the system gets quite unusable).
Second, the reboot times. QNAP seems to be affected by the more
features, fuck performance virus, and bloat their OS with more and
more features while completly ignoring the performance. Everytime they
do an upgrade it feels worse. Lately reboot times went up to 10 to
15 minutes - and then it still hadn t started the virtual machines /
docker containers one might run on. Another 5 to 10 minutes for those.
Opening the file explorer - ages on calculating what to show. Trying
to get the storage setup shown? Go get a coffee, but please fetch the
beans directly from the plantation, or you are too fast.
Annoying it was. And no, no broken disks or fan or anything, it all
checks out fine.
Replace QNAPs QTS system
So I started looking around what to do. More RAM may help a little
bit, but I already had 48G, the system itself appears to only do 64G
maximum, so not much chance of it helping enough. Hardware is all fine
and working, so software needs to be changed. Sounds hard, but turns
out, it is not.
TrueNAS
And I found that multiple people replaced the QNAPs own system with a
TrueNAS installation and generally had been happy. Looking further I
found that TrueNAS has a variant called Scale - which is based on
Debian. Doubly good, that, so I went off checking what I may need for
it.
Requirements
Heck, that was a step back. To install TrueNAS you need an HDMI out
and a disk to put it on. The one that QTS uses is too small, so no
option.
QNAPs original internal
USB drive, DOM
So either use one of the SSDs that played cache (and
should do so again in TrueNAS, or get the QNAP original replaced.
HDMI out is simple, get a cheap card and put it into one of the two
PCIe-4x slots, done. The disk thing looked more complicated, as QNAP
uses some internal usb stick thing . Turns out it is just a USB
stick that has an 8+1pin connector. Couldn t find anything nice as
replacement, but hey, there are 9-pin to USB-A adapters.
a 9pin to USB A adapter
With that adapter, one can take some random M2 SSD and an M2-to-USB
case, plus some cabling, and voila, we have a nice system disk.
9pin adapter to USB-A connected with some
more cable
Obviously there isn t a good place to put this SSD case and cable, but the
QNAP case is large enough to find space and use some cable ties to store it
safely. Space enough to get the cable from the side, where the
mainboard is to the place I mounted it, so all fine.
Mounted SSD in its external case
The next best M2 SSD was a Western Digital Red with 500G - and while
this is WAY too much for TrueNAS, it works. And hey, only using a tiny
fraction? Oh so much more cells available internally to use when
others break. Or something
Together with the Asus card mounted I was able to install TrueNAS.
Which is simple, their installer is easy enough to follow, just make
sure to select the right disk to put it on.
Preserving data during the move
Switching from QNAP QTS to TrueNAS Scale means changing from MDADM
Raid with LVM and ext4 on top to ZFS and as such all data on it gets
erased. So a backup first is helpful, and I got myself two external
Seagate USB Disks of 6TB each - enough for the data I wanted to keep.
Copying things all over took ages, especially as the QNAP backup
thingie sucks, it was breaking quite often. Also, for some reason I
did not investigate, the performance of it was real bad. It started at
a maximum of 50MB/s, but the last terabyte of data was copied
at MUCH less than that, and so it took much longer than I anticipated.
Copying back was slow too, but much less so. Of course reading things
usually is faster than writing, with it going around 100MB/s most of
the time, which is quite a bit more - still not what USB3 can actually
do, but I guess the AMD chip doesn t want to go that fast.
TrueNAS experience
The installation went mostly smooth, the only real trouble had been on
my side. Turns out that a bad network cable does NOT help the network
setup, who would have thought. Other than that it is the usual set of
questions you would expect, a reboot, and then some webinterface.
And here the differences start. The whole system boots up much faster.
Not even a third of the time compared to QTS.
One important thing: As TrueNAS scale is Debian based, and hence a
linux kernel, it automatically detects and assembles the old RAID
arrays that QTS put on. Which TrueNAS can do nothing with, so it helps
to manually stop them and wipe the disks.
Afterwards I put ZFS on the disks, with a similar setup to what I had
before. The spinning rust are the data disks in a RAIDZ2 setup, the
two SSDs are added as cache devices. Unlike MDADM, ZFS does not have a
long sync process. Also unlike the MDADM/LVM/EXT4 setup from before,
ZFS works different. It manages the raid thing but it also does the
volume and filesystem parts. Quite different handling, and I m still
getting used to it, so no, I won t write some ZFS introduction now.
Features
The two systems can not be compared completly, they are having a
pretty different target audience. QNAP is more for the user that wants
some network storage that offers a ton of extra features easily
available via a clickable interface. While TrueNAS appears more
oriented to people that want a fast but reliable storage system.
TrueNAS does not offer all the extra bloat the QNAP delivers. Still,
you have the ability to run virtual machines and it seems it comes
with Rancher, so some kubernetes/container ability is there. It lacks
essential features like assigning PCI devices to virtual machines, so
is not useful right now, but I assume that will come in a future
version.
I am still exploring it all, but I like what I have right now. Still
rebuilding my setup to have all shares exported and used again, but
the most important are working already.
My first Rust crate: munin-plugin
Sooo, some time ago I had to rewrite a munin plugin from
Shell to Rust, due to the shell version going crazy after some runtime
and using up a CPU all for its own. Sure, it only did that on Systems
with Oracle Database installed, so that monster seems to be bad (who
would have guessed?), but somehow I had to fixup this plugin and
wasn t allowed to drop that wannabe-database.
A while later I wrote a plugin to graph Fibre Channel Host data, and
then Network interface statistics, all with a one-second resolution
for the graphs, to allow one to zoom in and see every spike. Not
have RRD round of the interesting parts.
As one can imagine, that turns out to be a lot of very similar code -
after all, most of the difference is in the graph config statements
and actual data gathering, but the rest of code is just the same.
As I already know there are more plugins (hello rsyslog statistics) I
have to (sometimes re-)write in Rust, I took some time and wrote me a
Rust library to make writing munin-plugins in Rust easier. Yay, my
first crate on crates.io (and
wrote lots of docs for it).
By now I made my 1 second resolution CPU load
plugin and the 1 second
resolution Network interface
plugin use this lib already. To
test less complicated plugins with the lib, I took the munin default
plugin load (Linux variant) and made a Rust version from it, but
mostly to see that something as simple as that is also easy to
implement: Munin load
I got some idea on how to provide a useful default implementation of
the fetch function, so one can write even less code, when using this
library.
It is my first library in Rust, so if you see something bad or missing
in there, feel free to open issues or pull requests.
Now, having done this, one thing missing: Someone to (re)write munin
itself in something that is actually fast Not munin-node, but
munin. Or maybe the RRD usage, but with a few hundred nodes in it,
with loads of graphs, we had to adjust munin code and change some
timeout or it would commit suicide regularly. And some other code
change for not always checking for a rename, or something like it. And
only run parts of the default cronjob once an hour, not on every
update run. And switch to fetching data over ssh (and munin-async on
the nodes). And rrdcached with loads of caching for the trillions of
files (currently amounts to ~800G of data).. And it still needs way
more CPU than it should. Soo, lots of possible optimizations hidden in
there. Though I bet a non-scripting language rewrite might gain the
most. (Except, of course, someone needs to do it :) )
Shell? Rust!
Not the first shell script I took and made a rust version of, but
probably my largest yet. This time I took my little tm (tmux
helper) tool which is (well, was) a
bit more than 600 lines of shell, and converted it to
Rust.
I got most of the functionality done now, only one major part is
missing.
What s tm?
tm started as a tiny shell script to make handling
tmux easier. The first commit in
git was in July 2013, but I started writing and using it in 2011. It
started out as a kind-of wrapper around ssh, opening tmux windows with
an ssh session on some other hosts. It quickly gained support to open
multiple ssh sessions in one window, telling tmux to synchronize input
(send input to all targets at once), which is great when you have a
set of machines that ought to get the same commands.
tm vs clusterssh / mussh
In spirit it is similar to clusterssh or mussh, allowing to run the
same command on many hosts at the same time. clusterssh sets out to
open new terminals (xterm) per host and gives you an input line, that
it sends everywhere. mussh appears to take your command and then send
it to all the hosts. Both have disadvantages in my opinion: clusterssh
opens lots of xterm windows, and you can not easily switch between
multiple sessions, mussh just seems to send things over ssh and be
done.
tm instead just creates a tmux session, telling it to ssh to the
targets, possibly setting the tmux option to send input to all panes.
And leaves all the rest of the handling to tmux. So you can
detach a session and reattach later easily,
use tmux great builtin support for copy/paste,
see all output, modify things even for one machine only,
zoom in to one machine that needs just ONE bit different (cssh can
do this too),
let colleagues also connect to your tmux session, when needed,
easily add more machines to the mix, if needed,
and all the other extra features tmux brings.
More tm
tm also supports just attaching to existing sessions as well as
killing sessions, mostly for lazyness (less to type than using tmux
directly).
At some point tm gained support for setting up sessions according to
some session file . It knows two formats now, one is simple and
mostly a list of hostnames to open synchronized sessions for. This may
contain LIST commands, which let tm execute that command, expected
output is list of hostnames (or more LIST commands) for the session.
That, combined with the replacement part, lets us have one config file
that opens a set of VMs based on tags our
Ganeti runs, based on tags. It is simply a LIST
command asking for VMs tagged with the replacement arg and up. Very
handy. Or also all VMs on host X .
The second format is basically free form tmux commands . Mostly
commandline tmux call, just drop the tmux in front collection.
Both of them supporting a crude variable replacement.
Conversion to Rust
Some while ago I started playing with Rust and it somehow clicked , I
do like it. My local git tells me, that I tried starting off with
go in 2017, but that appearently did not work out.
Fun, everywhere I can read says that Rust ought to be harder to learn.
So by now I have most of the functionality implemented in the Rust
version, even if I am sure that the code isn t a good Rust example.
I m learning, after all, and already have adjusted big parts of it,
multiple times, whenever I learn (and understand) something more - and
am also sure that this will happen again
Compatibility with old tm
It turns out that my goal of staying compatible with the behaviour of
the old shell script does make some things rather complicated. For
example, the LIST commands in session config files - in shell I just
execute them commands, and shell deals with variable/parameter
expansion, I just set IFS to newline only and read in what I get
back. Simple. Because shell is doing a lot of things for me.
Now, in Rust, it is a different thing at all:
Properly splitting the line into shell words, taking care of quoting
(can t simply take whitespace) (there is shlex)
Expanding specials like ~ and $HOME (there is
home_dir).
Supporting environment variables in general, tm has some that adjust
behaviour of it. Which shell can use globally. Used
lazy_static for a
similar effect - they aren t going to change at runtime ever,
anyways.
Properly supporting the commandline arguments also turned out to be a
bit more work. Rust appearently has multiple crates supporting this, I
settled on clap, but as tm
supports getopts -style as well as free-form arguments (subcommands
in clap), it takes a bit to get that interpreted right.
Speed
Most of the time entirely unimportant in the tool that tm is (open a
tmux with one to some ssh connections to some places is not exactly
hard or time consuming), there are situations, where one can notice
that it s calling out to tmux over and over again, for every single
bit to do, and that just takes time: Configurations that open sessions
to 20 and more hosts at the same time especially lag in setup time.
(My largest setup goes to 443 panes in one window). The compiled Rust
version is so much faster there, it s just great. Nice side effect,
that is. And yes, in the end it is also only driving tmux, still, it
takes less than half the time to do so.
Code, Fun parts
As this is still me learning to write Rust, I am sure the code has
lots to improve. Some of which I will sure find on my own, but if you
have time, I love PRs (or just mails with hints).
Github
Also the first time I used Github Actions to see how it goes. Letting
it build, test, run
clippy and also run a code
coverage tool (Yay, more than
50% covered ) on it. Unsure my tests are good, I am not used to
writing tests for code, but hey, coverage!
Up next
I do have to implement the last missing feature, which is reading the
other config file format. A little scared, as that means somehow
translating those lines into correct calls within the
tmux_interface
I am using, not sure that is easy. I could be bad and just shell
out to tmux on it all the time, but somehow I don t like the thought
of doing that. Maybe (ab)using the control mode, but then, why would I
use tmux_interface, so trying to handle it with that first.
Afterwards I want to gain a new command, to save existing sessions and
be able to recreate them easily. Shouldn t be too hard, tmux has a way
to get at that info, somewhere.
SSH private key scanner (keys without passphrase)
So for policy reasons, customer wanted to ensure that every SSH
private key in use by a human on their systems has a passphrase set.
And asked us to make sure this is the case.
There is no way in SSH to check this during connection, so client side
needs to be looked at. Which means looking at actual files on the
system.
Turns out there are multiple formats for the private keys - and I
really do not want to implement something able to deal with that on my
own.
OpenSSH to the rescue, it ships a little tool ssh-keygen, most
commonly known for its ability to generate SSH keys. But it can do
much more with keys. One action is interesting here for our case: The
ability to print out the public key to a given private key. For a key
that is unprotected, this will just work. A key with a passphrase
instead leads to it asking you for one.
So we have our way to check if a key is protected by a passphrase. Now
we only need to find all possible keys (note, the requirement is not
keys in .ssh/ , but all possible, so we need to scan for them.
But we do not want to run ssh-keygen on just any file, we would like
to do it when we are halfway sure, that it is actually a key. Well,
turns out, even though SSH has multiple formats, they all appear to
have the string PRIVATE KEY somewhere very early (usually first
line). And they are tiny - even a 16384bit RSA key is just above 12000
bytes long.
Lets find every file thats less then 13000 bytes and has the magic
string in it, and throw it at ssh-keygen - if we get a public key
back, flag it. Also, we supply a random (ohwell, hardcoded)
passphrase, to avoid it prompting for any.
Scanning the whole system, one will find quite a surprising number of
unprotected SSH keys. Well, better description possibly Unprotected
RSA private keys , so the output does need to be checked by a human.
This, of course, can be done in shell, quite simple. So i wrote some
Rust code instead, as I am still on my task to try and learn more of
it. If you are interested, you can find sshprivscanner and play with
it, patches/fixes/whatever
welcome.
Munin plugin and it s CPU usage (and a rewrite in rust)
With my last blog on the Munin plugins CPU usage I complained about Oracle
Linux doing something really weird, driving up CPU usage when running
a fairly simple Shell script with a loop in.
Turns out, I was wrong. It is not OL7 that makes this problem show
up. It appears to be something from the Oracle Enterprise Database
installed on the system, that makes it go this crazy. I ve now had
this show up on RedHat7 systems too, and the only thing that singles
them out is that overpriced index card system on it.
I still don t know what the actual reason for this is, and honestly,
don t have enough time to dig deep into it. It is not something that a
bit of debugging/tracing finds - especially as it does start out all
nice, and accumulates more CPU usage over time. Which would suggest
some kind of leak leading to more processing needed, or so - but then
it is only CPU affected, not memory, and ONLY on systems with that
database on. Meh.
Well, I recently (December vacation) got me to look deeper into
learning Rust. My first project with that was a multi-threaded milter
to do some TLS checks on outgoing mails (kind of fun customer
requirements there), and heck, Rust did make that a surprisingly easy
task in the end. (Comparing the old, single-threaded C code with my
multi-threaded Rust version, a third of the code length doing more,
and being way easier to extend with wanted new features is nice).
So my second project was Replace this shell script with a Rust binary
doing the same . Hell yeah. Didn t take that long and looks good
(well, the result. Not sure about the code. People knowing rust may
possibly scratch out eyes when looking at it). Not
yet running for that long, but even compared to the shell on systems
that did not show the above mentioned bugs (read: Debian, without
Oracle foo), uses WAY less CPU (again, mentioned by highly accurate
outputs of the top command). So longer term I hope this version won t
run into the same problems as the shell one. Time will tell.
If you are interested in the code, go find it
here, and if you happen to know
rust and not run away screaming, I m happy for tips and code fixes,
I m sure this can be improved lots. (At least cargo clippy is happy,
so basics are done )
Update: According to munin, the rust version creates 14
forks/second less than the shell one. And the fork rate change is same
on machines with/without the database. That 14 is more than I would
have guessed. CPU usage as expected: only on the problem hosts with
Oracle Database installed you can see a huge difference, otherwise it
is not an easily noticable difference. That is, on an otherwise idle
host (munin graph shows average use of low one-digit numbers), one can
see a drop of around 1% in the CPU usage graph from munin. Ohwell,
poor Shell.
Munin plugin and it s CPU usage (shell fixup)
So at work we do have a munin server running, and one of
the graphs we do for every system is a network statistics one with a
resolution of 1 second. That s a simple enough script to have, and it
is working nicely - on 98% of our machines. You just don t notice the
data gatherer at all, so that we also have some other graphs done with
a 1 second resolution. For some, this really helps.
Basics
The basic code for this is simple. There is a bunch of stuff to start
the background gathering, some to print out the config, and some to
hand out the data when munin wants it. Plenty standard.
The interesting bit that goes wrong and uses too much CPU on one
Linux Distribution is this:
run_acquire()echo"$$">$ pidfilewhile :;do
TSTAMP=$(date +%s)echo$ IFACE_tx.value $ TSTAMP:$(cat /sys/class/net/$ IFACE/statistics/tx_bytes )>>$ cacheecho$ IFACE_rx.value $ TSTAMP:$(cat /sys/class/net/$ IFACE/statistics/rx_bytes )>>$ cache# Sleep for the rest of the secondsleep 0.$(printf'%04d'$((10000-10#$(date +%4N))))done
That code works, and none of Debian wheezy, stretch and buster as well
as RedHat 6 or 7 shows anything, it just works, no noticable load generated.
Now, Oracle Linux 7 thinks differently. The above code run there
generates between 8 and 15% CPU usage (on fairly recent Intel CPUs,
but that shouldn t matter). (CPU usage measured with the highly
accurate use of top and looking what it tells )
Whyever.
Fixing
Ok, well, the code above isn t all the nicest shell, actually. There
is room for improvement. But beware, the older the bash, the less one
can fix it.
So, first of, there are two useless uses of cat. Bash can do that
for us, just use the $(< /PATH/TO/FILE ) way.
Oh, Bash5 knows the epoch directly, we can replace the date call for
the timestamp and use $ EPOCHSECONDS
Too bad Bash4 can t do that. But hey, it s builtin printf can help
out, a nice TSTAMP=$(printf %(%s)T\n -1) works.
Unfortunately, Bash4.2 and later, not 4.1, and meh, we have a 4.1
system, so that has to stay with the date call there.
Taking that, we end up with 3 different possible versions, depending
on the Bash on the system.
obtain5()## Purest bash version, Bash can tell us epochs directlyecho$ IFACE_tx.value $ EPOCHSECONDS:$(</sys/class/net/$ IFACE/statistics/tx_bytes)>>$ cacheecho$ IFACE_rx.value $ EPOCHSECONDS:$(</sys/class/net/$ IFACE/statistics/rx_bytes)>>$ cache# Sleep for the rest of the secondsleep 0.$(printf'%04d'$((10000-10#$(date +%4N))))
obtain42()## Bash cant tell us epochs directly, but the builtin printf canTSTAMP=$(printf'%(%s)T\n'-1)echo$ IFACE_tx.value $ TSTAMP:$(</sys/class/net/$ IFACE/statistics/tx_bytes)>>$ cacheecho$ IFACE_rx.value $ TSTAMP:$(</sys/class/net/$ IFACE/statistics/rx_bytes)>>$ cache# Sleep for the rest of the secondsleep 0.$(printf'%04d'$((10000-10#$(date +%4N))))
obtain41()## Bash needs help from a tool to get epoch, means one exec() all the timeTSTAMP=$(date +%s)echo$ IFACE_tx.value $ TSTAMP:$(</sys/class/net/$ IFACE/statistics/tx_bytes)>>$ cacheecho$ IFACE_rx.value $ TSTAMP:$(</sys/class/net/$ IFACE/statistics/rx_bytes)>>$ cache# Sleep for the rest of the secondsleep 0.$(printf'%04d'$((10000-10#$(date +%4N))))
run_acquire()echo"$$">$ pidfilecase$ BASH_VERSINFO[0]in
5)while :;do
obtain5
done;;
4)if[[$ BASHVERSION[1]-ge 2 ]];then
while :;do
obtain42
done
else
while :;do
obtain41
done
fi;;esac
Does it help?
Oh yes, it does. Oracle Linux 7 appears to use Bash 4.2, so uses
obtain42 and hey, removing one date and two cat calls, and it has a
sane CPU usage of 0 (again, highly accurate number generated from
top ). Appears OL7 is doing heck-what-do-i-know extra, when calling
other tools, for whatever gains, removing that does help (who would
have thought).
(None of RedHat or Oracle Linux has SELinux turned on, so that one
shouldn t bite. But it is clear OL7 doing something extra for
everything that bash spawns.)
Debian NEW Queue
So for some reason I got myself motivated again to deal with some
packages in Debians NEW Queue.
We had 420 source packages waiting for some kind of processing when I
started, now we are down to something around
10. (Silly, people keep
uploading stuff )
That s not entirely my own work, others from the team have been active
too, but for those few days I went through a lot of stuff waiting. And
must say it still feels mostly like it did when I somehow stopped
doing much in NEW.
Except - well, I feel that maintainers are much better in preparing
their packages, especially that dreaded task of getting the copyright
file written seems to be one that is handled much better. Now, thats
not supported by any real numbers, just a feeling, but a good one, I
think.
Rust
Dealing with NEW meant I got in contact with one part that currently
generates some friction between the FTP Team and one group of package
maintainers - the Rust team.
Note: this is, of course, entirely written from my point of view.
Though with the intention of presenting it as objective as possible.
Also, I know what rust is, and have tried a Hello world in it, but
that s about my deep knowledge of it
The problem
Libraries in rust are bundled/shipped/whatever in something called
crates, and you manage what your stuff needs and provides with a tool
called cargo.
A library (one per crate) can provide multiple features, say a TLS lib
can link against gnutls or openssl or some other random
implementation. Such features may even be combinable in various
different ways, so one can have a high number of possible feature
combinations for one crate.
There is a tool called debcargo which helps creating a
Debian package out of a crate. And that tool generates so-called
feature-packages, one per feature / combination thereof.
Those feature packages are empty packages, only containing a symlink
for their /usr/share/doc/ directory, so their size is smaller
than the metadata they will produce. Inside the archive and the files
generated by it, stuff that every user everywhere has to download
and their apt has to process. Additionally, any change of those feature
sets means one round through NEW, which is also not ideal.
So, naturally, the FTP Team dislikes those empty feature packages.
Really, a lot.
There appears to be a different way. Not having the feature packages,
but putting all the combinations into a Provides header. That
sometimes works, but has two problems:
It can generate really long Provides: lines. I mean, REALLY
REALLY REALLY long. Somewhat around 250kb is the current
record. Thats long enough that a tool (not dak itself) broke on it.
Sure, that tool needs to be fixed, but still, that s not nice.
Currently preferred from us, though.
Some of the features may need different dependencies (say, gnutls
vs openssl), should those conflict with each other, you can not
combine them into one package.
Solutions
Currently we do not have a good one. The rust maintainers and the ftp
team are talking, exploring various ideas, we will see what will come
out.
Devel archive / Component
One of the possible solutions for the feature package problem would be
something that another set of packages could also make good use of, I
think. The introduction of a new archive or component, meant only for
packages that are needed to build something, but where users are
discouraged from ever using them.
What?
Well, take golang as an example. While we have a load of
golang-something packages in Debian, and they are used for building
applications written in go - none of those golang-something are meant
to be installed by users. If you use the language and develop in it,
the go get way is the one you are expected to use.
So having an archive (or maybe component like main or contrib) that,
by default, won t be activated for users, but only for things like
buildds or archive rebuilds, will make one problem (hated metadata
bloat) be evaluated wildly different.
It may also allow a more relaxed processing of binary-NEW (easier
additions of new feature packages).
But but but
Yes, it is not the most perfect solution. Without taking much energy
to think about, it requires
an adjustment in how main is handled. Right now we have the golden
rule that main is self contained, that is, things in it may not
need anything outside it for building or running. That would need
to be adjusted for building. (Go as well as currently rust are
always building static binaries, so no library dependencies there).
It would need handling for the release, that is, the release team
would need to deal with that archive/component too. We haven t,
yet, talked to them (still, slowly, discussing inside FTP Team).
So, no idea how many rusty knives they want to sink into our nice
bodies for that idea
Final
Well, it is still very much open. Had an IRC meeting with the rust
people, will have another end of November, it will slowly go forward.
And maybe someone comes up with an entire new idea that we all love.
Don t know, time will tell.
Seems my blog lately just consist of updates to my automated login
script for the ICE wifi But I do hate the entirely useless Click a
button crap, every day, twice. I ve seen it once, now leave me alone,
please.
Updated script:
#!/bin/bash# (Some) docs at# https://wiki.ubuntuusers.de/NetworkManager/Dispatcher/IFACE=$ 1:-"none"ACTION=$ 2:-"up"TMPDIR=$ TMPDIR:-"/tmp"WGET="/usr/bin/wget"TIMEOUT="/usr/bin/timeout -k 20 15"case$ ACTIONin
up)CONID=$ CONNECTION_ID:-$(iwconfig $IFACE grep ESSID cut -d":" -f2 sed 's/^[^"]*"\ "[^"]*$//g') if[[$ CONID== WIFIonICE ]]; then
REFERER="http://www.wifionice.de/de/"LOGIN="http://www.wifionice.de/de/"COOKIETMP=$(mktemp -p $ TMPDIR nmwifionice.XXXXXXXXX)trap"rm -f $ COOKIETMP" EXIT TERM HUP INT QUIT
csrftoken=$($ TIMEOUT$ WGET -q -O - --keep-session-cookies --save-cookies=$ COOKIETMP --referer $ REFERER$ LOGIN grep -oP 'CSRFToken"\ value="\K[0-9a-z]+')if[[ -z $ csrftoken]]; then
echo"CSRFToken is empty"exit 0
fi
sleep 1
$ TIMEOUT$ WGET -q -O - --load-cookies=$ COOKIETMP --post-data="login=true&connect=connect&CSRFToken=$ csrftoken" --referer $ REFERER$ LOGIN >/dev/null
fi;;*)# We are not interested in this
:
;;esac
With recent changes the automated login script
for WifiOnICE stopped working. Fortunately a fix is easy, it is enough
to add a referrer header to the call and have de/ added to the url.
Updated script:
#!/bin/bash# (Some) docs at# https://wiki.ubuntuusers.de/NetworkManager/Dispatcher/IFACE=$ 1:-"none"ACTION=$ 2:-"up"case$ ACTIONin
up)CONID=$ CONNECTION_ID:-$(iwgetid "$ IFACE" -r) if[[$ CONID== WIFIonICE ]]; then
/usr/bin/timeout -k 20 15 /usr/bin/wget -q -O - --referer http://www.wifionice.de/de/ http://www.wifionice.de/de/?login > /dev/null
fi;;*)# We are not interested in this
:
;;esac
If you have the fortune to need to follow some silly Login button
for some wifi, regularly, the following little script may help you
avoid this idiotic (and useless) task.
This example uses the WIFIonICE, the free wifi on german ICE trains,
simply as I have it twice a day, and got annoyed by the pointless
Login button. A friend pointed me at just wget-ting the login page, so
I made Network-Manager do this for me. Should work for anything
similar that doesn t need some elaborate webform filled out.
#!/bin/bash# (Some) docs at# https://wiki.ubuntuusers.de/NetworkManager/Dispatcher/IFACE=$ 1:-"none"ACTION=$ 2:-"up"case$ ACTIONin
up)CONID=$ CONNECTION_ID:-$(iwconfig $IFACE grep ESSID cut -d":" -f2 sed 's/^[^"]*"\ "[^"]*$//g') if[[$ CONID== WIFIonICE ]]; then
/usr/bin/timeout -k 20 15 /usr/bin/wget -q -O - http://www.wifionice.de/?login > /dev/null
fi;;*)# We are not interested in this
:
;;esac
This script needs to be put into /etc/NetworkManager/dispatcher.d
and made executable, owned by the root user. It will run on every
connection change, thats why the ACTION is checked. The case may be a
bit much here, but it could be easily extended to do a lot more.
Yay, no more silly Open this webpage and press login crap.
In case you just upgraded to the latest gnupg-agent and
used gnupg-agent as your ssh-agent you may find that ssh refuses to
work with a simple but not helpful
sign_and_send_pubkey: signing failed: agent refused operation
This seems to come from systemd starting the agent, no
longer a script at the start of the X session. And so it ends up with
either no or an unusable tty. A simple
gpg-connect-agent updatestartuptty /bye
updates that and voila, ssh agent functionality is back in.
Note: This assumes you have enable-ssh-support in your ~/.gnupg/gpg-agent.conf
About 10 months ago, we enabled an auto-decrufter in dak. Then after 3 months it had become the top 11th remover . Today, there are only 3 humans left that have removed more packages than the auto-decrufter impressively enough, one of them is not even an active FTP-master (anymore). The current score board:
5371 Luca Falavigna
5121 Alexander Reichle-Schmehl
4401 Ansgar Burchardt
3928 DAK's auto-decrufter
3257 Scott Kitterman
2225 Joerg Jaspert
1983 James Troup
1793 Torsten Werner
1025 Jeroen van Wolffelaar
763 Ryan Murray
For comparison, here is the number removals by year for the past 6 years:
5103 2011
2765 2012
3342 2013
3394 2014
3766 2015 (1842 removed by auto-decrufter)
2845 2016 (2086 removed by auto-decrufter)
Which tells us that in 2015, the FTP masters and the decrufter performed on average over 10 removals a day. And by the looks of it, 2016 will surpass that. Of course, the auto-decrufter has a tendency to increase the number of removed items since it is an advocate of remove early, remove often! .
Data is from https://ftp-master.debian.org/removals-full.txt. Scoreboard computed as:
And as just announced on d-d-a, I m trying to break all the tools
dealing with the (Debian) archive. Or something like it. But its about
time to get rid of MD5Sum checksums, and SHA1 can go with it directly.
As it is only in experimental for now, we can test and see what still
breaks. I hope it won t be too much, so we can get it over all the
archive (minus the stable stuff, of course).
For some reason, what I like most in this change is the following
python code that ended up in our InRelease file generation tool:
importapt_pkg# Note: suite.checksums being an array with possible values of md5sum, sha1, sha256hashfuncs=dict(zip([x.upper().replace('UM','um')forxinsuite.checksums],[getattr(apt_pkg,"%s"%(x))forxin[x.replace("sum","")+"sum"forxinsuite.checksums]]))
Though I m sure this can be done in much more readable ways, it s
doing what we need, but heck, took me a while to get it.
But probably the change to get rid of gzip will be much more
challenging / hard to get through. Lets see, in the few minutes after
my mail, I already got some notices about possible breakage here.
Fortunately those indices and the release file stuff are nicely
seperated settings, so if it turns out we only take the checksums drop
for now into the other suites, thats doable.
While writing this, the last bits of squeeze that haven t been on
archive.debian.org yet are moved over there, that is security and the
lts suite. As soon as that script done, some tiny archive magic will
happen and over the next few dinstall runs, the rest of squeeze that
still was there will no longer be on our regular mirror network
anymore.
If you still use it, now is the time to upgrade.
Update: Forgot to mention, but yes, squeeze-backports(-sloppy) is
also archived on archive.debian.org.
It took nearly a year, but today a new ftpsync version got released.
Most of the work for this release was done by weasel, with one new
feature submitted by waldi, my work was mostly style fixes and a bit
of documentation. And of course the release now.
If you run a mirror, you will find the new version at the usual place,
that is the project/ftpsync/ subdirectory. You may also want to
subscribe to the debian mirrors mailinglist, as the mirror team will
post more information about changes in ftpsync there.
I'm pleased to say that I have joined the Debian FTPTeam as of the Friday before last. See Joerg Jaspert's announcement on debian-devel-announce.
The FTPTeam is responsible for maintaining the Debian software archive, and ensures that new software in Debian is high-quality and compliant with our policies.
As an "ftpassistant", I (along with Paul, Scott, Gergely, and others) will be helping to process the NEW queue, which is currently at a whopping 297 packages. Here's hoping we'll be able to get that number down over the coming weeks!
Wohoo, one more in that
littleseriesof mine,
another update to tm.
This time it gained getopts style options - while retaining the old
non-getopts style. It now can open more than one multisession to the
same set of hosts, so you can connect multiple times to host1 host2
at the same time. No matter if you open them using the ms subcommand
or the pre-defined sessions from the .tmux.d directory.
I also took a bit and wrote some zsh completion for it. I m pretty
sure this can be done better and am happy for patches, but it does
work for me, so for now I m happy.
You can, as usual, grab tm from my misc git
repository,
the zsh completion is in my zsh config
repository
- also at Github.
There was Lenny, there was Squeeze, now there is Wheezy.
Another major release of Debian where I had the pleasure to do the ftpmaster work for the release. Like the last times, Mark joined to help with the work. But 2 FTPMasters aren t enough for one Wheezy, or so, so Ansgar had his first run aside from point releases.
With the whole load of work the release team did in the past to prepare this, combined with all the changes in dak we had since Squeeze, it turned out to be rather easy for us.
Again, a few moments and not a detailed log (times in CEST):
[10:45] Let s prepare. Cronjobs go off on ftp-master and security, shared tmux session setup, including a live pipe into a file so other team members can follow using less +F -R , a titanpad page opened to share in writing the commands to process later.
[11:18] One last package update done manually. The latest and greatest of installation-guide, 20130503, really wants to join the rest in wheezy.
[12:00] While I did a database backup and started preparing the set of commands for later, Mark and Ansgar prepared our code to deal with the changes we are about to make.
[12:33] I start executing the commands (shell and sql), so somewhen after here we have Squeeze as oldstable (with all its extra suites also renamed), Wheezy as stable (taking its own suites with it, as well as gaining the -updates).
And Jessie starts its life.
[12:45] Heissa, wouldn t be fun if we wouldn t have some bugs, so we just created a load of directories not needed or wanted. Not getting Jessies either. Ohwell.
[13:00] Jessie gets a copy of Wheezy to start with. Made the mistake to try that with our usual tool, control-suite. Err. Wrong. This is doing a set of checks on each package inserted. Would take ages. While one SQL to bypass the checks was a minute or so.
[13:xx] Got annoyed by our concept of consistency. Its great. Every tool takes suite names/codenames in a different way. Seperated with spaces, with commas, with codenames, with suitenames, we mix it in all possible combinations.
[14:00] Ansgar finished updating our security master host to cope with the changes for the release.
[14:39] Squeeze Release files got regenerated and signed again.
[14:54] Wheezy Release files got generated and signed.
[15:05] The cd build machine has a synced mirror tree, so they can start building our CD/DVD images
[16:25] A manual dinstall finished. And while it had some small problems (including nearly overwriting something inside the wheezy dists/ tree), it went pretty smoothly.
[21:00] After having kept the archive turned off for the whole day, we have 90 uploads with ~3Gb of uploads waiting. As it is now unlikely that we need larger archive changes, we turn (most of) our cronjobs back on, to get them processed.
Right now (that is, 21:08 CEST) I am mostly waiting for the CD build to finish. Current schedule seems to have that in some two and a half hours, after which we can push the mirrors and our press people announce it. Except if Murphy wants to show up, but lets hope that not.
Thanks go out to everyone involved in preparing this release, be it in the past by fixing bugs, uploading packages, doing whatever was needed, as well as doing the work today.
Just did a small update to my tm helper script.
There is now a new variable, TMSESSHOST. If that is set to false, the hostname
of the current machine won t be prepended to the generated session name. true is the default, as that was the behaviour in the past.
It now can attach to any session, not just the ones it created itself, thus nearly entirely removing my need of ever calling tmux directly.
And cosmetically - window names for sessions defined in the .tmux.d directory are no longer plain Multisession , but the session name as defined in the file.
If you are interested, you can get it over here from my git repository of misc stuff
QNAPs original internal
USB drive, DOM
a 9pin to USB A adapter
9pin adapter to USB-A connected with some
more cable
Mounted SSD in its external case
QNAPs original internal
USB drive, DOM
a 9pin to USB A adapter
9pin adapter to USB-A connected with some
more cable
Mounted SSD in its external case
Data is from 
I'm pleased to say that I have joined the Debian