Sergio Talens-Oliag: New Blog

last_job_sync
to synchronize job dependencies of the previous
submission. Although the DRM scheduler guarantees the order of starting to
execute a job in the same queue in the kernel space, the order of completion
isn t predictable. On the other hand, we still needed to use syncobjs to follow
job completion since we have event threads on the CPU side. Therefore, a more
accurate implementation requires last_job syncobjs to track when each engine
(CL, TFU, and CSD) is idle. We also needed to keep the driver working on
previous versions of v3d kernel-driver with single semaphores, then we kept
tracking ANY last_job_sync
to preserve the previous implementation.
This was waiting for multisync support in the v3d kernel, which is already available. Exposing this feature however enabled a few more CTS tests that exposed pre-existing bugs in the user-space driver so we fix those here before exposing the feature.
This should give you emulated timeline semaphores for free and kernel-assisted sharable timeline semaphores for cheap once you have the kernel interface wired in.
apt
way to install those packages. And downloading those one by one and then installing them sounds like too much work.
Step Zero: Prerequisites
If you are an Ubuntu (or Debian!) developer, you might already have ubuntu-dev-tools installed. If not, it has some really useful tools!
$ sudo apt install ubuntu-dev-tools
$ mkdir mesa-downgrade; cd mesa-downgrade
pull-lp-debs
. The first argument is the source package name. In this case, I next need to specify what version I want; otherwise it will give me the latest version which isn t helpful. I could specify a series codename like jammy or impish but that won t give me what I want this time.
$ pull-lp-debs mesa 21.3.5-1ubuntu2
$ sudo apt install --only-upgrade --mark-auto ./*.deb
--only-upgrade
tells apt
to only install packages that are already installed. I don t actually need all 24 packages installed; I just want to change the versions for the stuff I already have.
--mark-auto
tells apt
to keep these packages marked in dpkg
as automatically installed. This allows any of these packages to be suggested for removal once there isn t anything else depending on them. That s useful if you don t want to have old libraries installed on your system in case you do manual installation like this frequently.
Finally, the apt
install syntax has a quirk: It needs a path to a file because it wants an easy way to distinguish from a package name. So adding ./
before filenames works.
I guess this is a bug. apt
should be taught that libegl-mesa0_21.3.5-1ubuntu2_amd64.deb is a file name not a package name.
Step Four: Cleanup
Let s assume that you installed old versions. To get back to the current package versions, you can just upgrade like normal.
$ sudo apt dist-upgrade
$ sudo apt-mark hold
apt-mark list
and apt-mark unhold
to see what packages you have held and release the holds. Remember you won t get security updates or other bug fixes for held packages!
And when you re done with the debs we download, you can remove all the files:
$ cd .. ; rm -ri mesa-downgrade
/jammy
suffix on the package name.
$ sudo apt install libegl-mesa0/jammy
apt list
Here s one suggested way to find them:
$ apt list --installed --all-versions grep local] --after-context 1
apt
is designed to upgrade packages not downgrade them. You can break things by downgrading. For instance, a database could upgrade its format to a new version but I wouldn t expect it to be able to reverse that just because you attempt to install an older version.
/usr/share/metainfo
, which was very inefficient.
To shorten a long story, the old caching code was rewritten with the new concepts of caches not necessarily being system-wide and caches existing for more fine-grained groups of files in mind. The new caching code uses Richard Hughes excellent libxmlb internally for memory-mapped data storage. Unlike LMDB, libxmlb knows about the XML document model, so queries can be much more powerful and we do not need to build indices manually. The library is also already used by GNOME Software and fwupd for parsing of (refined) AppStream metadata, so it works quite well for that usecase. As a result, search queries via libappstream are now a bit slower (very much depends on the query, roughly 20% on average), but can be mmuch more powerful. The caching code is a lot more robust, which should speed up startup time of applications. And in addition to all of that, the AsPool
class has gained a flag to allow it to monitor AppStream source data for changes and refresh the cache fully automatically and transparently in the background.
All software written against the previous version of the libappstream library should continue to work with the new caching code, but to make use of some of the new features, software using it may need adjustments. A lot of methods have been deprecated too now.
2. Experimental compose support
Compiling MetaInfo and other metadata into AppStream collection metadata, extracting icons, language information, refining data and caching media is an involved process. The appstream-generator tool does this very well for data from Linux distribution sources, but the tool is also pretty heavyweight with lots of knobs to adjust, an underlying database and a complex algorithm for icon extraction. Embedding it into other tools via anything else but its command-line API is also not easy (due to D s GC initialization, and because it was never written with that feature in mind). Sometimes a simpler tool is all you need, so the libappstream-compose library as well as appstreamcli compose
are being developed at the moment. The library contains building blocks for developing a tool like appstream-generator while the cli tool allows to simply extract metadata from any directory tree, which can be used by e.g. Flatpak. For this to work well, a lot of appstream-generator s D code is translated into plain C, so the implementation stays identical but the language changes.
Ultimately, the generator tool will use libappstream-compose for any general data refinement, and only implement things necessary to extract data from the archive of distributions. New applications (e.g. for new bundling systems and other purposes) can then use the same building blocks to implement new data generators similar to appstream-generator with ease, sharing much of the code that would be identical between implementations anyway.
2. Supporting user input controls
Want to advertise that your application supports touch input? Keyboard input? Has support for graphics tablets? Gamepads? Sure, nothing is easier than that with the new control
relation item and supports
relation kind (since 0.12.11 / 0.15.0, details):
<supports>
<control>pointing</control>
<control>keyboard</control>
<control>touch</control>
<control>tablet</control>
</supports>
display_length
relation item to require or recommend a minimum (or maximum) display size that the described GUI application can work with. For example:
<requires>
<display_length compare="ge">360</display_length>
</requires>
display_length
value will be checked against the longest edge of a display by default (by explicitly specifying the shorter edge, this can be changed).
This feature is available since 0.13.0, details. See also Tobias Bernard s blog entry on this topic.
4. Tags
This is a feature that was originally requested for the LVFS/fwupd, but one of the great things about AppStream is that we can take very project-specific ideas and generalize them so something comes out of them that is useful for many. The new tags
tag allows people to tag components with an arbitrary namespaced string. This can be useful for project-internal organization of applications, as well as to convey certain additional properties to a software center, e.g. an application could mark itself as featured in a specific software center only. Metadata generators may also add their own tags to components to improve organization. AppStream gives no recommendations as to how these tags are to be interpreted except for them being a strictly optional feature. So any meaning is something clients and metadata authors need to negotiate. It therefore is a more specialized usecase of the already existing custom
tag, and I expect it to be primarily useful within larger organizations that produce a lot of software components that need sorting. For example:
<tags>
<tag namespace="lvfs">vendor-2021q1</tag>
<tag namespace="plasma">featured</tag>
</tags>
control
and display_length
tags, resolved a few minor issues and also added a button to instantly copy the generated output to clipboard so people can paste it into their project. If you want to create a new MetaInfo file, this tool is the best way to do it!
The creator tool will also not transfer any data out of your webbrowser, it is strictly a client-side application.
And that is about it for the most notable changes in AppStream land! Of course there is a lot more, additional tags for the LVFS and content rating have been added, lots of bugs have been squashed, the documentation has been refined a lot and the library has gained a lot of new API to make building software centers easier. Still, there is a lot to do and quite a few open feature requests too. Onwards to 1.0!
mbsync
).
But I also evaluated OfflineIMAP which was resurrected from the Python
2 apocalypse, and because I had used it before, for a long time.
Read on for the details.
$ notmuch count --exclude=false
372758
$ du -sh --exclude xapian Maildir
13G Maildir
The baseline we are comparing against is SMD (syncmaildir) which
performs the sync in about 7-8 seconds locally (3.5 seconds for each
push/pull command) and about 10-12 seconds remotely.
Anything close to that or better is good enough. I do not have recent
numbers for a SMD full sync baseline, but the setup
documentation
mentions 20 minutes for a full sync. That was a few years ago, and the
spool has obviously grown since then, so that is not a reliable
baseline.
A baseline for a full sync might be also set with rsync, which copies
files at nearly 40MB/s, or 317Mb/s!
anarcat@angela:tmp(main)$ time rsync -a --info=progress2 --exclude xapian shell.anarc.at:Maildir/ Maildir/
12,647,814,731 100% 37.85MB/s 0:05:18 (xfr#394981, to-chk=0/395815)
72.38user 106.10system 5:19.59elapsed 55%CPU (0avgtext+0avgdata 15988maxresident)k
8816inputs+26305112outputs (0major+50953minor)pagefaults 0swaps
That is 5 minutes to transfer the entire spool. Incremental syncs are
obviously pretty fast too:
anarcat@angela:tmp(main)$ time rsync -a --info=progress2 --exclude xapian shell.anarc.at:Maildir/ Maildir/
0 0% 0.00kB/s 0:00:00 (xfr#0, to-chk=0/395815)
1.42user 0.81system 0:03.31elapsed 67%CPU (0avgtext+0avgdata 14100maxresident)k
120inputs+0outputs (3major+12709minor)pagefaults 0swaps
As an extra curiosity, here's the performance with tar
, pretty
similar with rsync
, minus incremental which I cannot be bothered to
figure out right now:
anarcat@angela:tmp(main)$ time ssh shell.anarc.at tar --exclude xapian -cf - Maildir/ pv -s 13G tar xf -
56.68user 58.86system 5:17.08elapsed 36%CPU (0avgtext+0avgdata 8764maxresident)k
0inputs+0outputs (0major+7266minor)pagefaults 0swaps
12,1GiO 0:05:17 [39,0MiB/s] [===================================================================> ] 92%
Interesting that rsync
manages to almost beat a plain tar
on file
transfer, I'm actually surprised by how well it performs here,
considering there are many little files to transfer.
(But then again, this maybe is exactly where rsync
shines: while
tar
needs to glue all those little files together, rsync
can just
directly talk to the other side and tell it to do live
changes. Something to look at in another article maybe?)
Since both ends are NVMe drives, those should easily saturate a
gigabit link. And in fact, a backup of the server mail spool achieves
much faster transfer rate on disks:
anarcat@marcos:~$ tar fc - Maildir pv -s 13G > Maildir.tar
15,0GiO 0:01:57 [ 131MiB/s] [===================================] 115%
That's 131Mibyyte per second, vastly faster than the gigabit
link. The client has similar performance:
anarcat@angela:~(main)$ tar fc - Maildir pv -s 17G > Maildir.tar
16,2GiO 0:02:22 [ 116MiB/s] [==================================] 95%
So those disks should be able to saturate a gigabit link, and they are
not the bottleneck on fast links. Which begs the question of what is
blocking performance of a similar transfer over the gigabit link, but
that's another question altogether, because no sync program ever
reaches the above performance anyways.
Finally, note that when I migrated to SMD, I wrote a small
performance
comparison that
could be interesting here. It show SMD to be faster than OfflineIMAP,
but not as much as we see here. In fact, it looks like OfflineIMAP
slowed down significantly since then (May 2018), but this could be due
to my larger mail spool as well.
mbsync
) project is written in C and supports
syncing Maildir and IMAP folders, with possibly multiple replicas. I
haven't tested this but I suspect it might be possible to sync between
two IMAP servers as well. It supports partial mirorrs, message flags,
full folder support, and "trash" functionality.
.mbsyncrc
configuration file:
SyncState *
Sync New ReNew Flags
IMAPAccount anarcat
Host imap.anarc.at
User anarcat
PassCmd "pass imap.anarc.at"
SSLType IMAPS
CertificateFile /etc/ssl/certs/ca-certificates.crt
IMAPStore anarcat-remote
Account anarcat
MaildirStore anarcat-local
# Maildir/top/sub/sub
#SubFolders Verbatim
# Maildir/.top.sub.sub
SubFolders Maildir++
# Maildir/top/.sub/.sub
# SubFolders legacy
# The trailing "/" is important
#Path ~/Maildir-mbsync/
Inbox ~/Maildir-mbsync/
Channel anarcat
# AKA Far, convert when all clients are 1.4+
Master :anarcat-remote:
# AKA Near
Slave :anarcat-local:
# Exclude everything under the internal [Gmail] folder, except the interesting folders
#Patterns * ![Gmail]* "[Gmail]/Sent Mail" "[Gmail]/Starred" "[Gmail]/All Mail"
# Or include everything
Patterns *
# Automatically create missing mailboxes, both locally and on the server
#Create Both
Create slave
# Sync the movement of messages between folders and deletions, add after making sure the sync works
#Expunge Both
Long gone are the days where I would spend a long time reading a
manual page to figure out the meaning of every option. If that's your
thing, you might like this one. But I'm more of a "EXAMPLES
section" kind of person now, and I somehow couldn't find a sample file
on the website. I started from the Arch wiki one but it's
actually not great because it's made for Gmail (which is not a usual
Dovecot server). So a sample config file in the manpage would be a
great addition. Thankfully, the Debian packages ships one in
/usr/share/doc/isync/examples/mbsyncrc.sample
but I only found that
after I wrote my configuration. It was still useful and I recommend
people take a look if they want to understand the syntax.
Also, that syntax is a little overly complicated. For example, Far
needs colons, like:
Far :anarcat-remote:
Why? That seems just too complicated. I also found that sections are
not clearly identified: IMAPAccount
and Channel
mark section
beginnings, for example, which is not at all obvious until you learn
about mbsync
's internals. There are also weird ordering issues: the
SyncState
option needs to be before IMAPAccount
, presumably
because it's global.
Using a more standard format like .INI or TOML could improve that
situation.
rsync
.
The incremental runs are roughly 2 seconds, which is even more
impressive, as that's actually faster than rsync
:
===> multitime results
1: mbsync -a
Mean Std.Dev. Min Median Max
real 2.015 0.052 1.930 2.029 2.105
user 0.660 0.040 0.592 0.661 0.722
sys 0.338 0.033 0.268 0.341 0.387
Those tests were performed with isync 1.3.0-2.2 on Debian
bullseye. Tests with a newer isync release originally failed because
of a corrupted message that triggered bug 999804 (see
below). Running 1.4.3 under valgrind works around the bug, but adds a
50% performance cost, the full sync running in 1h35m.
Once the upstream patch is applied, performance with 1.4.3 is fairly
similar, considering that the new sync included the register
folder
with 4000 messages:
120.74user 213.19system 59:47.69elapsed 9%CPU (0avgtext+0avgdata 105420maxresident)k
29128inputs+28284376outputs (0major+45711minor)pagefaults 0swaps
That is ~13GB in ~60 minutes, which gives us 28.3Mbps. Incrementals
are also pretty similar to 1.3.x, again considering the double-connect
cost:
===> multitime results
1: mbsync -a
Mean Std.Dev. Min Median Max
real 2.500 0.087 2.340 2.491 2.629
user 0.718 0.037 0.679 0.711 0.793
sys 0.322 0.024 0.284 0.320 0.365
Those tests were all done on a Gigabit link, but what happens on a
slower link? My server uplink is slow: 25 Mbps down, 6 Mbps up. There
mbsync
is worse than the SMD baseline:
===> multitime results
1: mbsync -a
Mean Std.Dev. Min Median Max
real 31.531 0.724 30.764 31.271 33.100
user 1.858 0.125 1.721 1.818 2.131
sys 0.610 0.063 0.506 0.600 0.695
That's 30 seconds for a sync, which is an order of magnitude slower
than SMD.
mbsync
UI is kind of neat:
anarcat@angela:~(main)$ mbsync -a
Notice: Master/Slave are deprecated; use Far/Near instead.
C: 1/2 B: 204/205 F: +0/0 *0/0 #0/0 N: +1/200 *0/0 #0/0
(Note that nice switch away from slavery-related terms too.)
The display is minimal, and yet informative. It's not obvious what
does mean at first glance, but the manpage is useful at least for
clarifying that:
This represents the cumulative progress over channels, boxes, and messages affected on the far and near side, respectively. The message counts represent added messages, messages with updated flags, and trashed messages, respectively. No attempt is made to calculate the totals in advance, so they grow over time as more information is gathered. (Emphasis mine).In other words:
C 2/2
: channels done/total (2 done out of 2)B 204/205
: mailboxes done/total (204 out of 205)F
: changes on the far sideN: +10/200 *0/0 #0/0
: changes on the "near" side:
+10/200
: 10 out of 200 messages downloaded*0/0
: no flag changed#0/0
: no message deletedS
to "mark spam", which basically assigns the tag spam
to the
message and removes a bunch of others. Then I have a
notmuch-purge script which moves that message to the spam folder,
for training purposes. It basically does this:
notmuch search --output=files --format=text0 "$search_spam" \
xargs -r -0 mv -t "$HOME/Maildir/$ PREFIX junk/cur/"
This method, which worked fine in SMD (and also OfflineIMAP) created
this error on sync:
Maildir error: duplicate UID 37578.
And indeed, there are now two messages with that UID in the mailbox:
anarcat@angela:~(main)$ find Maildir/.junk/ -name '*U=37578*'
Maildir/.junk/cur/1637427889.134334_2.angela,U=37578:2,S
Maildir/.junk/cur/1637348602.2492889_221804.angela,U=37578:2,S
This is actually a known limitation or, as mbsync(1) calls it, a
"RECOMMENDATION":
When using the more efficient default UID mapping scheme, it is important that the MUA renames files when moving them between Maildir fold ers. Mutt always does that, while mu4e needs to be configured to do it:So it seems I would need to fix my script. It's unclear how the paths should be renamed, which is unfortunate, because I would need to change my script to adapt to(setq mu4e-change-filenames-when-moving t)
mbsync
, but I can't tell how just from
reading the above.
(A manual fix is actually to rename the file to remove the U=
field:
mbsync
will generate a new one and then sync correctly.)
Fortunately, someone else already fixed that issue: afew, a
notmuch tagging script (much puns, such hurt), has a move mode
that can rename files correctly, specifically designed to deal
with mbsync
. I had already been told about afew, but it's one more
reason to standardize my notmuch hooks on that project, it looks like.
Update: I have tried to use afew and found it has significant
performance issues. It also has a completely different paradigm
to what I am used to: it assumes all incoming mail has a new
and
lays its own tags on top of that (inbox
, sent
, etc). It can only
move files from one folder at a time (see this bug) which
breaks my spam training workflow. In general, I sync my tags into
folders (e.g. ham
, spam
, sent
) and message flags (e.g. inbox
is F
, unread
is "not S
", etc), and afew is not well suited for
this (although there are hacks that try to fix this). I have
worked hard to make my tagging scripts idempotent, and it's something
afew doesn't currently have. Still, it would be better to have
that code in Python than bash, so maybe I should consider my options
here.
mbsync
is really fast, but the downside of that is that
it's written in C, and with that comes a whole set of security
issues. The Debian security tracker has only three CVEs on
isync, but the above issues show there could be many more.
Reading the source code certainly did not make me very comfortable
with trusting it with untrusted data. I considered sandboxing it with
systemd (below) but having systemd run as a --user
process makes
that difficult. I also considered using an apparmor profile but
that is not trivial because we need to allow SSH and only some parts
of it...
Thankfully, upstream has been diligent at addressing the issues I have
found. They provided a patch within a few days which did fix the sync
issues.
Update: upstream actually took the issue very seriously. They not only
got CVE-2021-44143 assigned for my bug report, they also
audited the code and found several more issues collectively identified
as CVE-2021-3657, which actually also affect 1.3 (ie. Debian
11/bullseye/stable). Somehow my corpus doesn't trigger that issue, but
it was still considered serious enough to warrant a CVE. So one the
one hand: excellent response from upstream; but on the other hand: how
many more of those could there be in there?
mbsync
as a
systemd service. It suggests using the --verbose
(-V
) flag which
is a little intense here, as it outputs 1444 lines of messages.
I have used the following .service
file:
[Unit]
Description=Mailbox synchronization service
ConditionHost=!marcos
Wants=network-online.target
After=network-online.target
Before=notmuch-new.service
[Service]
Type=oneshot
ExecStart=/usr/bin/mbsync -a
Nice=10
IOSchedulingClass=idle
NoNewPrivileges=true
[Install]
WantedBy=default.target
And the following .timer
:
[Unit]
Description=Mailbox synchronization timer
ConditionHost=!marcos
[Timer]
OnBootSec=2m
OnUnitActiveSec=5m
Unit=mbsync.service
[Install]
WantedBy=timers.target
Note that we trigger notmuch
through systemd, with the Before
and
also by adding mbsync.service
to the notmuch-new.service
file:
[Unit]
Description=notmuch new
After=mbsync.service
[Service]
Type=oneshot
Nice=10
ExecStart=/usr/bin/notmuch new
[Install]
WantedBy=mbsync.service
An improvement over polling repeatedly with a .timer
would be to
wake up only on IMAP notify, but neither imapnotify nor
goimapnotify seem to be packaged in Debian. It would also not
cover for the "sent folder" use case, where we need to wake up on
local changes.
IMAPStore remote
Tunnel "ssh -q host.remote.com /usr/sbin/imapd"
Add BatchMode
, restrict to IdentitiesOnly
, provide a password-less
key just for this, add compression (-C
), find the Dovecot imap
binary, and you get this:
IMAPAccount anarcat-tunnel
Tunnel "ssh -o BatchMode=yes -o IdentitiesOnly=yes -i ~/.ssh/id_ed25519_mbsync -o HostKeyAlias=shell.anarc.at -C anarcat@imap.anarc.at /usr/lib/dovecot/imap"
And it actually seems to work:
$ mbsync -a
Notice: Master/Slave are deprecated; use Far/Near instead.
C: 0/2 B: 0/1 F: +0/0 *0/0 #0/0 N: +0/0 *0/0 #0/0imap(anarcat): Error: net_connect_unix(/run/dovecot/stats-writer) failed: Permission denied
C: 2/2 B: 205/205 F: +0/0 *0/0 #0/0 N: +1/1 *3/3 #0/0imap(anarcat)<1611280><90uUOuyElmEQlhgAFjQyWQ>: Info: Logged out in=10808 out=15396642 deleted=0 expunged=0 trashed=0 hdr_count=0 hdr_bytes=0 body_count=1 body_bytes=8087
It's a bit noisy, however. dovecot/imap
doesn't have a "usage" to
speak of, but even the source code doesn't hint at a way to disable
that Error
message, so that's unfortunate. That socket is owned by
root:dovecot
so presumably Dovecot runs the imap
process as
$user:dovecot
, which we can't do here. Oh well?
Interestingly, the SSH setup is not faster than IMAP.
With IMAP:
===> multitime results
1: mbsync -a
Mean Std.Dev. Min Median Max
real 2.367 0.065 2.220 2.376 2.458
user 0.793 0.047 0.731 0.776 0.871
sys 0.426 0.040 0.364 0.434 0.476
With SSH:
===> multitime results
1: mbsync -a
Mean Std.Dev. Min Median Max
real 2.515 0.088 2.274 2.532 2.594
user 0.753 0.043 0.645 0.766 0.804
sys 0.328 0.045 0.212 0.340 0.393
Basically: 200ms slower. Tolerable.
mbsync
on my first workstation. The
work on the second one was more streamlined, especially since the
corruption on mailboxes was fixed:
dpkg -i isync_1.4.3-1.1~_amd64.deb
rsync -a --info=progress2 angela:Maildir/ Maildir-mbsync/
find Maildir-mbsync/ -type f -name '*.angela,*' -print0 rename -0 's/\.angela,/\.curie,/'
rm -rf Maildir-mbsync/.notmuch/xapian/
systemctl --user --now disable smd-pull.service smd-pull.timer smd-push.service smd-push.timer notmuch-new.service notmuch-new.timer
smd-pull --show-tags ; smd-push --show-tags ; notmuch new ; notmuch-sync-flagged -v
notmuch dump pv > notmuch.dump
cp -al Maildir Maildir-bak
ssh-keygen -t ed25519 -f .ssh/id_ed25519_mbsync
cat .ssh/id_ed25519_mbsync.pub
.ssh/authorized_keys
on the server, like this:
command="/usr/lib/dovecot/imap",restrict ssh-ed25519 AAAAC...mv Maildir Maildir-smd
mv Maildir-mbsync Maildir
notmuch new
pv notmuch.dump notmuch restore
smd-*
services:
systemctl --user enable mbsync.timer notmuch-new.service
systemctl --user start mbsync.timer
rm ~/.config/systemd/user/smd*
systemctl daemon-reload[...]
Warning: cannot apply tags to missing message: CAN6gO7_QgCaiDFvpG3AXHi6fW12qaN286+2a7ERQ2CQtzjSEPw@mail.gmail.com
Warning: cannot apply tags to missing message: CAPTU9Wmp0yAmaxO+qo8CegzRQZhCP853TWQ_Ne-YF94MDUZ+Dw@mail.gmail.com
Warning: cannot apply tags to missing message: F5086003-2917-4659-B7D2-66C62FCD4128@gmail.com
[...]
Warning: cannot apply tags to missing message: mailman.2.1316793601.53477.sage-members@mailman.sage.org
Warning: cannot apply tags to missing message: mailman.7.1317646801.26891.outages-discussion@outages.org
Warning: cannot apply tags to missing message: notmuch-sha1-000458df6e48d4857187a000d643ac971deeef47
Warning: cannot apply tags to missing message: notmuch-sha1-0079d8e0c3340e6f88c66f4c49fca758ea71d06d
Warning: cannot apply tags to missing message: notmuch-sha1-0194baa4cfb6d39bc9e4d8c049adaccaa777467d
Warning: cannot apply tags to missing message: notmuch-sha1-02aede494fc3f9e9f060cfd7c044d6d724ad287c
Warning: cannot apply tags to missing message: notmuch-sha1-06606c625d3b3445420e737afd9a245ae66e5562
Warning: cannot apply tags to missing message: notmuch-sha1-0747b020f7551415b9bf5059c58e0a637ba53b13
[...]
As detailed in the crash report,
all of those were actually innocuous and could be ignored.
Also note that we completely trash the notmuch
database because it's
actually faster to reindex from scratch than let notmuch
slowly
figure out that all mails are new and all the old mails are
gone. The fresh indexing took:
nov 19 15:08:54 angela notmuch[2521117]: Processed 384679 total files in 23m 41s (270 files/sec.).
nov 19 15:08:54 angela notmuch[2521117]: Added 372610 new messages to the database.
While a reindexing on top of an existing database was going twice as
slow, at about 120 files/sec.
SyncState *
Sync All
# IMAP side, AKA "Far"
IMAPAccount anarcat-imap
Host imap.anarc.at
User anarcat
PassCmd "pass imap.anarc.at"
SSLType IMAPS
CertificateFile /etc/ssl/certs/ca-certificates.crt
IMAPAccount anarcat-tunnel
Tunnel "ssh -o BatchMode=yes -o IdentitiesOnly=yes -i ~/.ssh/id_ed25519_mbsync -o HostKeyAlias=shell.anarc.at -C anarcat@imap.anarc.at /usr/lib/dovecot/imap"
IMAPStore anarcat-remote
Account anarcat-tunnel
# Maildir side, AKA "Near"
MaildirStore anarcat-local
# Maildir/top/sub/sub
#SubFolders Verbatim
# Maildir/.top.sub.sub
SubFolders Maildir++
# Maildir/top/.sub/.sub
# SubFolders legacy
# The trailing "/" is important
#Path ~/Maildir-mbsync/
Inbox ~/Maildir/
# what binds Maildir and IMAP
Channel anarcat
Far :anarcat-remote:
Near :anarcat-local:
# Exclude everything under the internal [Gmail] folder, except the interesting folders
#Patterns * ![Gmail]* "[Gmail]/Sent Mail" "[Gmail]/Starred" "[Gmail]/All Mail"
# Or include everything
#Patterns *
Patterns * !register !.register
# Automatically create missing mailboxes, both locally and on the server
Create Both
#Create Near
# Sync the movement of messages between folders and deletions, add after making sure the sync works
Expunge Both
# Propagate mailbox deletion
Remove both
IMAPAccount anarcat-register-imap
Host imap.anarc.at
User register
PassCmd "pass imap.anarc.at-register"
SSLType IMAPS
CertificateFile /etc/ssl/certs/ca-certificates.crt
IMAPAccount anarcat-register-tunnel
Tunnel "ssh -o BatchMode=yes -o IdentitiesOnly=yes -i ~/.ssh/id_ed25519_mbsync -o HostKeyAlias=shell.anarc.at -C register@imap.anarc.at /usr/lib/dovecot/imap"
IMAPStore anarcat-register-remote
Account anarcat-register-tunnel
MaildirStore anarcat-register-local
SubFolders Maildir++
Inbox ~/Maildir/.register/
Channel anarcat-register
Far :anarcat-register-remote:
Near :anarcat-register-local:
Create Both
Expunge Both
Remove both
Note that it may be out of sync with my live (and private)
configuration file, as I do not publish my "dotfiles" repository
publicly for security reasons.
notmuch
, and
would sometimes crash mysteriously. It's been a while, so my memory is
hazy on that.
It also kind of died in a fire when Python 2 stop being
maintained. The main author moved on to a different project,
imapfw which could serve as a framework to build IMAP clients,
but never seemed to implement all of the OfflineIMAP features and
certainly not configuration file compatibility. Thankfully, a new team
of volunteers ported OfflineIMAP to Python 3 and we can now test that
new version to see if it is an improvement over mbsync
.
Copy message from RemoteAnarcat:junk:
ERROR: Copying message 30624 [acc: Anarcat]
decoding with 'X-EUC-TW' codec failed (AttributeError: 'memoryview' object has no attribute 'decode')
Thread 'Copy message from RemoteAnarcat:junk' terminated with exception:
Traceback (most recent call last):
File "/usr/share/offlineimap3/offlineimap/imaputil.py", line 406, in utf7m_decode
for c in binary.decode():
AttributeError: 'memoryview' object has no attribute 'decode'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/share/offlineimap3/offlineimap/threadutil.py", line 146, in run
Thread.run(self)
File "/usr/lib/python3.9/threading.py", line 892, in run
self._target(*self._args, **self._kwargs)
File "/usr/share/offlineimap3/offlineimap/folder/Base.py", line 802, in copymessageto
message = self.getmessage(uid)
File "/usr/share/offlineimap3/offlineimap/folder/IMAP.py", line 342, in getmessage
data = self._fetch_from_imap(str(uid), self.retrycount)
File "/usr/share/offlineimap3/offlineimap/folder/IMAP.py", line 908, in _fetch_from_imap
ndata1 = self.parser['8bit-RFC'].parsebytes(data[0][1])
File "/usr/lib/python3.9/email/parser.py", line 123, in parsebytes
return self.parser.parsestr(text, headersonly)
File "/usr/lib/python3.9/email/parser.py", line 67, in parsestr
return self.parse(StringIO(text), headersonly=headersonly)
File "/usr/lib/python3.9/email/parser.py", line 56, in parse
feedparser.feed(data)
File "/usr/lib/python3.9/email/feedparser.py", line 176, in feed
self._call_parse()
File "/usr/lib/python3.9/email/feedparser.py", line 180, in _call_parse
self._parse()
File "/usr/lib/python3.9/email/feedparser.py", line 385, in _parsegen
for retval in self._parsegen():
File "/usr/lib/python3.9/email/feedparser.py", line 298, in _parsegen
for retval in self._parsegen():
File "/usr/lib/python3.9/email/feedparser.py", line 385, in _parsegen
for retval in self._parsegen():
File "/usr/lib/python3.9/email/feedparser.py", line 256, in _parsegen
if self._cur.get_content_type() == 'message/delivery-status':
File "/usr/lib/python3.9/email/message.py", line 578, in get_content_type
value = self.get('content-type', missing)
File "/usr/lib/python3.9/email/message.py", line 471, in get
return self.policy.header_fetch_parse(k, v)
File "/usr/lib/python3.9/email/policy.py", line 163, in header_fetch_parse
return self.header_factory(name, value)
File "/usr/lib/python3.9/email/headerregistry.py", line 601, in __call__
return self[name](name, value)
File "/usr/lib/python3.9/email/headerregistry.py", line 196, in __new__
cls.parse(value, kwds)
File "/usr/lib/python3.9/email/headerregistry.py", line 445, in parse
kwds['parse_tree'] = parse_tree = cls.value_parser(value)
File "/usr/lib/python3.9/email/_header_value_parser.py", line 2675, in parse_content_type_header
ctype.append(parse_mime_parameters(value[1:]))
File "/usr/lib/python3.9/email/_header_value_parser.py", line 2569, in parse_mime_parameters
token, value = get_parameter(value)
File "/usr/lib/python3.9/email/_header_value_parser.py", line 2492, in get_parameter
token, value = get_value(value)
File "/usr/lib/python3.9/email/_header_value_parser.py", line 2403, in get_value
token, value = get_quoted_string(value)
File "/usr/lib/python3.9/email/_header_value_parser.py", line 1294, in get_quoted_string
token, value = get_bare_quoted_string(value)
File "/usr/lib/python3.9/email/_header_value_parser.py", line 1223, in get_bare_quoted_string
token, value = get_encoded_word(value)
File "/usr/lib/python3.9/email/_header_value_parser.py", line 1064, in get_encoded_word
text, charset, lang, defects = _ew.decode('=?' + tok + '?=')
File "/usr/lib/python3.9/email/_encoded_words.py", line 181, in decode
string = bstring.decode(charset)
AttributeError: decoding with 'X-EUC-TW' codec failed (AttributeError: 'memoryview' object has no attribute 'decode')
Last 1 debug messages logged for Copy message from RemoteAnarcat:junk prior to exception:
thread: Register new thread 'Copy message from RemoteAnarcat:junk' (account 'Anarcat')
ERROR: Exceptions occurred during the run!
ERROR: Copying message 30624 [acc: Anarcat]
decoding with 'X-EUC-TW' codec failed (AttributeError: 'memoryview' object has no attribute 'decode')
Traceback:
File "/usr/share/offlineimap3/offlineimap/folder/Base.py", line 802, in copymessageto
message = self.getmessage(uid)
File "/usr/share/offlineimap3/offlineimap/folder/IMAP.py", line 342, in getmessage
data = self._fetch_from_imap(str(uid), self.retrycount)
File "/usr/share/offlineimap3/offlineimap/folder/IMAP.py", line 908, in _fetch_from_imap
ndata1 = self.parser['8bit-RFC'].parsebytes(data[0][1])
File "/usr/lib/python3.9/email/parser.py", line 123, in parsebytes
return self.parser.parsestr(text, headersonly)
File "/usr/lib/python3.9/email/parser.py", line 67, in parsestr
return self.parse(StringIO(text), headersonly=headersonly)
File "/usr/lib/python3.9/email/parser.py", line 56, in parse
feedparser.feed(data)
File "/usr/lib/python3.9/email/feedparser.py", line 176, in feed
self._call_parse()
File "/usr/lib/python3.9/email/feedparser.py", line 180, in _call_parse
self._parse()
File "/usr/lib/python3.9/email/feedparser.py", line 385, in _parsegen
for retval in self._parsegen():
File "/usr/lib/python3.9/email/feedparser.py", line 298, in _parsegen
for retval in self._parsegen():
File "/usr/lib/python3.9/email/feedparser.py", line 385, in _parsegen
for retval in self._parsegen():
File "/usr/lib/python3.9/email/feedparser.py", line 256, in _parsegen
if self._cur.get_content_type() == 'message/delivery-status':
File "/usr/lib/python3.9/email/message.py", line 578, in get_content_type
value = self.get('content-type', missing)
File "/usr/lib/python3.9/email/message.py", line 471, in get
return self.policy.header_fetch_parse(k, v)
File "/usr/lib/python3.9/email/policy.py", line 163, in header_fetch_parse
return self.header_factory(name, value)
File "/usr/lib/python3.9/email/headerregistry.py", line 601, in __call__
return self[name](name, value)
File "/usr/lib/python3.9/email/headerregistry.py", line 196, in __new__
cls.parse(value, kwds)
File "/usr/lib/python3.9/email/headerregistry.py", line 445, in parse
kwds['parse_tree'] = parse_tree = cls.value_parser(value)
File "/usr/lib/python3.9/email/_header_value_parser.py", line 2675, in parse_content_type_header
ctype.append(parse_mime_parameters(value[1:]))
File "/usr/lib/python3.9/email/_header_value_parser.py", line 2569, in parse_mime_parameters
token, value = get_parameter(value)
File "/usr/lib/python3.9/email/_header_value_parser.py", line 2492, in get_parameter
token, value = get_value(value)
File "/usr/lib/python3.9/email/_header_value_parser.py", line 2403, in get_value
token, value = get_quoted_string(value)
File "/usr/lib/python3.9/email/_header_value_parser.py", line 1294, in get_quoted_string
token, value = get_bare_quoted_string(value)
File "/usr/lib/python3.9/email/_header_value_parser.py", line 1223, in get_bare_quoted_string
token, value = get_encoded_word(value)
File "/usr/lib/python3.9/email/_header_value_parser.py", line 1064, in get_encoded_word
text, charset, lang, defects = _ew.decode('=?' + tok + '?=')
File "/usr/lib/python3.9/email/_encoded_words.py", line 181, in decode
string = bstring.decode(charset)
Folder junk [acc: Anarcat]:
Copy message UID 30626 (29008/49310) RemoteAnarcat:junk -> LocalAnarcat:junk
Command exited with non-zero status 100
5252.91user 535.86system 3:21:00elapsed 47%CPU (0avgtext+0avgdata 846304maxresident)k
96344inputs+26563792outputs (1189major+2155815minor)pagefaults 0swaps
That only transferred about 8GB of mail, which gives us a transfer
rate of 5.3Mbit/s, more than 5 times slower than mbsync
. This bug is
possibly limited to the bullseye
version of offlineimap3
(the
lovely 0.0~git20210225.1e7ef9e+dfsg-4
), while the current sid
version (the equally gorgeous 0.0~git20211018.e64c254+dfsg-1
) seems
unaffected.
*** Finished account 'Anarcat' in 511:12
ERROR: Exceptions occurred during the run!
ERROR: Exception parsing message with ID (<20190619152034.BFB8810E07A@marcos.anarc.at>) from imaplib (response type: bytes).
AttributeError: decoding with 'X-EUC-TW' codec failed (AttributeError: 'memoryview' object has no attribute 'decode')
Traceback:
File "/usr/share/offlineimap3/offlineimap/folder/Base.py", line 810, in copymessageto
message = self.getmessage(uid)
File "/usr/share/offlineimap3/offlineimap/folder/IMAP.py", line 343, in getmessage
data = self._fetch_from_imap(str(uid), self.retrycount)
File "/usr/share/offlineimap3/offlineimap/folder/IMAP.py", line 910, in _fetch_from_imap
raise OfflineImapError(
ERROR: Exception parsing message with ID (<40A270DB.9090609@alternatives.ca>) from imaplib (response type: bytes).
AttributeError: decoding with 'x-mac-roman' codec failed (AttributeError: 'memoryview' object has no attribute 'decode')
Traceback:
File "/usr/share/offlineimap3/offlineimap/folder/Base.py", line 810, in copymessageto
message = self.getmessage(uid)
File "/usr/share/offlineimap3/offlineimap/folder/IMAP.py", line 343, in getmessage
data = self._fetch_from_imap(str(uid), self.retrycount)
File "/usr/share/offlineimap3/offlineimap/folder/IMAP.py", line 910, in _fetch_from_imap
raise OfflineImapError(
ERROR: IMAP server 'RemoteAnarcat' does not have a message with UID '32686'
Traceback:
File "/usr/share/offlineimap3/offlineimap/folder/Base.py", line 810, in copymessageto
message = self.getmessage(uid)
File "/usr/share/offlineimap3/offlineimap/folder/IMAP.py", line 343, in getmessage
data = self._fetch_from_imap(str(uid), self.retrycount)
File "/usr/share/offlineimap3/offlineimap/folder/IMAP.py", line 889, in _fetch_from_imap
raise OfflineImapError(reason, severity)
Command exited with non-zero status 1
8273.52user 983.80system 8:31:12elapsed 30%CPU (0avgtext+0avgdata 841936maxresident)k
56376inputs+43247608outputs (811major+4972914minor)pagefaults 0swaps
"offlineimap -o " took 8 hours 31 mins 15 secs
This is 8h31m for transferring 12G, which is around 3.1Mbit/s. That is
nine times slower than mbsync
, almost an order of magnitude!
Now that we have a full sync, we can test incremental
synchronization. That is also much slower:
===> multitime results
1: sh -c "offlineimap -o true"
Mean Std.Dev. Min Median Max
real 24.639 0.513 23.946 24.526 25.708
user 23.912 0.473 23.404 23.795 24.947
sys 1.743 0.105 1.607 1.729 2.002
That is also an order of magnitude slower than mbsync
, and
significantly slower than what you'd expect from a sync process. ~30
seconds is long enough to make me impatient and distracted; 3 seconds,
less so: I can wait and see the results almost immediately.
mbsync
over a slow
link, but I Haven't tested that theory.
The OfflineIMAP mail spool is missing quite a few messages as well:
anarcat@angela:~(main)$ find Maildir-offlineimap -type f -type f -a \! -name '.*' wc -l
381463
anarcat@angela:~(main)$ find Maildir -type f -type f -a \! -name '.*' wc -l
385247
... although that's probably all either new messages or the
register
folder, so OfflineIMAP might actually be in a better
position there. But digging in more, it seems like the actual
per-folder diff is fairly similar to mbsync
: a few messages missing
here and there. Considering OfflineIMAP's instability and poor
performance, I have not looked any deeper in those discrepancies.
doveadm
and offlineimap
, but requires running an IMAP server
locally, Perlrsmtp
which is a nice name
for rsendmail
. not evaluated because it seems awfully complex to
setup, Haskellmbsync
to sync my mail. I'm a little disappointed by
the synchronisation times over the slow link, but I guess that's on
par for the course if we use IMAP. We are bound by the network speed
much more than with custom protocols. I'm also worried about the C
implementation and the crashes I have witnessed, but I am encouraged
by the fast upstream response.
Time will tell if I will stick with that setup. I'm certainly curious
about the promises of interimap and mail-sync, but I have ran out of
time on this project.
gusnan@debian-i7:~ > dpkg --compare-versions 1.0b lt 1.0 && echo true gusnan@debian-i7:~ > dpkg --compare-versions 1.0b gt 1.0 && echo true trueBut there s a solution name the beta versions something like 1.0~beta. And you don t need to force upstream to make any changes either. You can use uscan and the watch file to make it interpret an upstream 1.0b version as 1.0~beta in Debian. This is done by using a line like
uversionmangle=s/(\d)[\_\.\-\+]?((RC rc pre dev beta alpha b a)\d*)$/$1~$2/;s/\~b/\~beta/;,\
in uversionmangle in your debian/watch file. In this case i have added on the end something to make the ending ~b into ~beta instead. Full version of the watch file available here.
run()
method that performs the
action:
@dataclass class Action: """ Base class for all action implementations. An Action is the equivalent of an ansible module: a declarative representation of an idempotent operation on a system. An Action can be run immediately, or serialized, sent to a remote system, run, and sent back with its results. """ uuid: str = field(default_factory=lambda: str(uuid.uuid4())) result: Result = field(default_factory=Result) def summary(self): """ Return a short text description of this action """ return self.__class__.__name__ def run(self, system: transilience.system.System): """ Perform the action """ self.result.state = ResultState.NOOP
summary()
method
so that trivial tasks like that can take care of naming themselves.
Dataclasses allow to introspect fields and annotate them with extra metadata,
and together with docstrings, I can make actions reasonably self-documeting.
I ported some of Ansible's modules over: see complete list in the git
repository.
Running Actions in a script
With a bit of glue code
I can now run Ansible-style functions from a plain Python script:
#!/usr/bin/python3 from transilience.runner import Script script = Script() for i in range(10): script.builtin.file(state="touch", path=f"/tmp/test i ")
#!/usr/bin/python3 from transilience.runner import Script from transilience.system import Mitogen script = Script(system=Mitogen("my server", "ssh", hostname="machine.example.org", username="user")) for i in range(10): script.builtin.file(state="touch", path=f"/tmp/test i ")
$ time ansible-playbook test.yaml [...] real 0m15.232s user 0m4.033s sys 0m1.336s $ time ./test_script real 0m4.934s user 0m0.547s sys 0m0.049s
builtin.file
sources for, uhm, reasons,
and the use of follow
stood out to my eyes. Reading on, not only that. I feel
like the ansible codebase needs a serious review, at least in essential core
modules like this one.
In the file module documentation
it says:
This flag indicates that filesystem links, if they exist, should be followed.In the recursive_set_attributes implementation instead, follow means "follow symlinks to directories", but if a symlink to a file is found, it does not get followed, kind of. What happens is that ansible will try to change the mode of the symlink, which makes sense on some operating systems. And it does try to use
lchmod
if
present. Buf if not, this happens:
# Attempt to set the perms of the symlink but be # careful not to change the perms of the underlying # file while trying underlying_stat = os.stat(b_path) os.chmod(b_path, mode) new_underlying_stat = os.stat(b_path) if underlying_stat.st_mode != new_underlying_stat.st_mode: os.chmod(b_path, stat.S_IMODE(underlying_stat.st_mode))
follow
following links during recursion: how does it avoid loops?
I don't see a cache of (device, inode)
pairs visited. Let's try:
fatal: [localhost]: FAILED! => "changed": false, "details": "maximum recursion depth exceeded", "gid": 1000, "group": "enrico", "mode": "0755", "msg": "mode must be in octal or symbolic form", "owner": "enrico", "path": "/tmp/test/test1", "size": 0, "state": "directory", "uid": 1000
ln -s .. foo
in a directory that gets recursed will always fail the
task. Fun!
More quirks
Turning a symlink into a hardlink is considered a noop
if the symlink points to the same file:
--- - hosts: localhost tasks: - name: create test file file: path: /tmp/testfile state: touch - name: create test link file: path: /tmp/testlink state: link src: /tmp/testfile - name: turn it into a hard link file: path: /tmp/testlink state: hard src: /tmp/testfile
$ ansible-playbook test3.yaml [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' PLAY [localhost] ************************************************************************************************************************************************************************************************************ TASK [Gathering Facts] ****************************************************************************************************************************************************************************************************** ok: [localhost] TASK [create test file] ***************************************************************************************************************************************************************************************************** changed: [localhost] TASK [create test link] ***************************************************************************************************************************************************************************************************** changed: [localhost] TASK [turn it into a hard link] ********************************************************************************************************************************************************************************************* ok: [localhost] PLAY RECAP ****************************************************************************************************************************************************************************************************************** localhost : ok=4 changed=2 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
--- - hosts: localhost tasks: - name: create test dir file: path: /tmp/testdir state: directory - name: turn it into a symlink file: path: /tmp/testdir state: hard src: /tmp/ force: yes
$ ansible-playbook test4.yaml [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' PLAY [localhost] ************************************************************************************************************************************************************************************************************ TASK [Gathering Facts] ****************************************************************************************************************************************************************************************************** ok: [localhost] TASK [create test dir] ****************************************************************************************************************************************************************************************************** changed: [localhost] TASK [turn it into a symlink] *********************************************************************************************************************************************************************************************** fatal: [localhost]: FAILED! => "changed": false, "gid": 1000, "group": "enrico", "mode": "0755", "msg": "Error while replacing: [Errno 21] Is a directory: b'/tmp/testdir'", "owner": "enrico", "path": "/tmp/testdir", "size": 0, "state": "directory", "uid": 1000 PLAY RECAP ****************************************************************************************************************************************************************************************************************** localhost : ok=2 changed=1 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0
if (st1.st_dev, st1.st_ino) == (st2.st_dev, st2.st_ino):
10.*
and my host s network is a 192.168/24
. I understand each pod gets its own virtual ethernet interface and associated IP address, and these are generally private within the cluster (and firewalled out other than for exposed services). What does that actually look like?
$ ip route
default via 192.168.53.1 dev enx00e04c6851de
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown
192.168.0.0/24 dev weave proto kernel scope link src 192.168.0.1
192.168.53.0/24 dev enx00e04c6851de proto kernel scope link src 192.168.53.147
10.107.66.138
(the IP my echoserver from the previous post is available on directly from the host). What about network interfaces? (under the cut because it s lengthy)
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
2: enx00e04c6851de: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:e0:4c:68:51:de brd ff:ff:ff:ff:ff:ff
inet 192.168.53.147/24 brd 192.168.53.255 scope global dynamic enx00e04c6851de
valid_lft 41571sec preferred_lft 41571sec
3: wlp1s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 74:d8:3e:70:3b:18 brd ff:ff:ff:ff:ff:ff
4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
link/ether 02:42:18:04:9e:08 brd ff:ff:ff:ff:ff:ff
inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
valid_lft forever preferred_lft forever
5: datapath: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1376 qdisc noqueue state UNKNOWN group default qlen 1000
link/ether d2:5a:fd:c1:56:23 brd ff:ff:ff:ff:ff:ff
7: weave: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1376 qdisc noqueue state UP group default qlen 1000
link/ether 12:82:8f:ed:c7:bf brd ff:ff:ff:ff:ff:ff
inet 192.168.0.1/24 brd 192.168.0.255 scope global weave
valid_lft forever preferred_lft forever
9: vethwe-datapath@vethwe-bridge: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1376 qdisc noqueue master datapath state UP group default
link/ether b6:49:88:d6:6d:84 brd ff:ff:ff:ff:ff:ff
10: vethwe-bridge@vethwe-datapath: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1376 qdisc noqueue master weave state UP group default
link/ether 6e:6c:03:1d:e5:0e brd ff:ff:ff:ff:ff:ff
11: vxlan-6784: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65535 qdisc noqueue master datapath state UNKNOWN group default qlen 1000
link/ether 9a:af:c5:0a:b3:fd brd ff:ff:ff:ff:ff:ff
13: vethwepl534c0a6@if12: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1376 qdisc noqueue master weave state UP group default
link/ether 1e:ac:f1:85:61:9a brd ff:ff:ff:ff:ff:ff link-netnsid 0
15: vethwepl9ffd6b6@if14: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1376 qdisc noqueue master weave state UP group default
link/ether 56:ca:71:2a:ab:39 brd ff:ff:ff:ff:ff:ff link-netnsid 1
17: vethwepl62b369d@if16: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1376 qdisc noqueue master weave state UP group default
link/ether e2:a0:bb:ee:fc:73 brd ff:ff:ff:ff:ff:ff link-netnsid 2
23: vethwepl6669168@if22: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1376 qdisc noqueue master weave state UP group default
link/ether f2:e7:e6:95:e0:61 brd ff:ff:ff:ff:ff:ff link-netnsid 3
weave
interface (the master weave
bit). Still no clues about the 10.*
range. What about ARP?
192.168.53.1 dev enx00e04c6851de lladdr e4:8d:8c:35:98:d5 DELAY
192.168.0.4 dev datapath lladdr da:22:06:96:50:cb STALE
192.168.0.2 dev weave lladdr 66:eb:ce:16:3c:62 REACHABLE
192.168.53.136 dev enx00e04c6851de lladdr 00:e0:4c:39:f2:54 REACHABLE
192.168.0.6 dev weave lladdr 56:a9:f0:d2:9e:f3 STALE
192.168.0.3 dev datapath lladdr f2:42:c9:c3:08:71 STALE
192.168.0.3 dev weave lladdr f2:42:c9:c3:08:71 REACHABLE
192.168.0.2 dev datapath lladdr 66:eb:ce:16:3c:62 STALE
192.168.0.6 dev datapath lladdr 56:a9:f0:d2:9e:f3 STALE
192.168.0.4 dev weave lladdr da:22:06:96:50:cb STALE
192.168.0.5 dev datapath lladdr fe:6f:1b:14:56:5a STALE
192.168.0.5 dev weave lladdr fe:6f:1b:14:56:5a REACHABLE
table ip nat
chain DOCKER
iifname "docker0" counter packets 0 bytes 0 return
chain POSTROUTING
type nat hook postrouting priority srcnat; policy accept;
counter packets 531750 bytes 31913539 jump KUBE-POSTROUTING
oifname != "docker0" ip saddr 172.17.0.0/16 counter packets 1 bytes 84 masquerade
counter packets 525600 bytes 31544134 jump WEAVE
chain PREROUTING
type nat hook prerouting priority dstnat; policy accept;
counter packets 180 bytes 12525 jump KUBE-SERVICES
fib daddr type local counter packets 23 bytes 1380 jump DOCKER
chain OUTPUT
type nat hook output priority -100; policy accept;
counter packets 527005 bytes 31628455 jump KUBE-SERVICES
ip daddr != 127.0.0.0/8 fib daddr type local counter packets 285425 bytes 17125524 jump DOCKER
chain KUBE-MARK-DROP
counter packets 0 bytes 0 meta mark set mark or 0x8000
chain KUBE-MARK-MASQ
counter packets 0 bytes 0 meta mark set mark or 0x4000
chain KUBE-POSTROUTING
mark and 0x4000 != 0x4000 counter packets 4622 bytes 277720 return
counter packets 0 bytes 0 meta mark set mark xor 0x4000
counter packets 0 bytes 0 masquerade
chain KUBE-KUBELET-CANARY
chain INPUT
type nat hook input priority 100; policy accept;
chain KUBE-PROXY-CANARY
chain KUBE-SERVICES
meta l4proto tcp ip daddr 10.96.0.10 tcp dport 9153 counter packets 0 bytes 0 jump KUBE-SVC-JD5MR3NA4I4DYORP
meta l4proto tcp ip daddr 10.107.66.138 tcp dport 8080 counter packets 1 bytes 60 jump KUBE-SVC-666FUMINWJLRRQPD
meta l4proto tcp ip daddr 10.111.16.129 tcp dport 443 counter packets 0 bytes 0 jump KUBE-SVC-EZYNCFY2F7N6OQA2
meta l4proto tcp ip daddr 10.96.9.41 tcp dport 443 counter packets 0 bytes 0 jump KUBE-SVC-EDNDUDH2C75GIR6O
meta l4proto tcp ip daddr 192.168.53.147 tcp dport 443 counter packets 0 bytes 0 jump KUBE-XLB-EDNDUDH2C75GIR6O
meta l4proto tcp ip daddr 10.96.9.41 tcp dport 80 counter packets 0 bytes 0 jump KUBE-SVC-CG5I4G2RS3ZVWGLK
meta l4proto tcp ip daddr 192.168.53.147 tcp dport 80 counter packets 0 bytes 0 jump KUBE-XLB-CG5I4G2RS3ZVWGLK
meta l4proto tcp ip daddr 10.96.0.1 tcp dport 443 counter packets 0 bytes 0 jump KUBE-SVC-NPX46M4PTMTKRN6Y
meta l4proto udp ip daddr 10.96.0.10 udp dport 53 counter packets 0 bytes 0 jump KUBE-SVC-TCOU7JCQXEZGVUNU
meta l4proto tcp ip daddr 10.96.0.10 tcp dport 53 counter packets 0 bytes 0 jump KUBE-SVC-ERIFXISQEP7F7OF4
fib daddr type local counter packets 3312 bytes 198720 jump KUBE-NODEPORTS
chain KUBE-NODEPORTS
meta l4proto tcp tcp dport 31529 counter packets 0 bytes 0 jump KUBE-MARK-MASQ
meta l4proto tcp tcp dport 31529 counter packets 0 bytes 0 jump KUBE-SVC-666FUMINWJLRRQPD
meta l4proto tcp ip saddr 127.0.0.0/8 tcp dport 30894 counter packets 0 bytes 0 jump KUBE-MARK-MASQ
meta l4proto tcp tcp dport 30894 counter packets 0 bytes 0 jump KUBE-XLB-EDNDUDH2C75GIR6O
meta l4proto tcp ip saddr 127.0.0.0/8 tcp dport 32740 counter packets 0 bytes 0 jump KUBE-MARK-MASQ
meta l4proto tcp tcp dport 32740 counter packets 0 bytes 0 jump KUBE-XLB-CG5I4G2RS3ZVWGLK
chain KUBE-SVC-NPX46M4PTMTKRN6Y
counter packets 0 bytes 0 jump KUBE-SEP-Y6PHKONXBG3JINP2
chain KUBE-SEP-Y6PHKONXBG3JINP2
ip saddr 192.168.53.147 counter packets 0 bytes 0 jump KUBE-MARK-MASQ
meta l4proto tcp counter packets 0 bytes 0 dnat to 192.168.53.147:6443
chain WEAVE
# match-set weaver-no-masq-local dst counter packets 135966 bytes 8160820 return
ip saddr 192.168.0.0/24 ip daddr 224.0.0.0/4 counter packets 0 bytes 0 return
ip saddr != 192.168.0.0/24 ip daddr 192.168.0.0/24 counter packets 0 bytes 0 masquerade
ip saddr 192.168.0.0/24 ip daddr != 192.168.0.0/24 counter packets 33 bytes 2941 masquerade
chain WEAVE-CANARY
chain KUBE-SVC-JD5MR3NA4I4DYORP
counter packets 0 bytes 0 jump KUBE-SEP-6JI23ZDEH4VLR5EN
counter packets 0 bytes 0 jump KUBE-SEP-FATPLMAF37ZNQP5P
chain KUBE-SEP-6JI23ZDEH4VLR5EN
ip saddr 192.168.0.2 counter packets 0 bytes 0 jump KUBE-MARK-MASQ
meta l4proto tcp counter packets 0 bytes 0 dnat to 192.168.0.2:9153
chain KUBE-SVC-TCOU7JCQXEZGVUNU
counter packets 0 bytes 0 jump KUBE-SEP-JTN4UBVS7OG5RONX
counter packets 0 bytes 0 jump KUBE-SEP-4TCKAEJ6POVEFPVW
chain KUBE-SEP-JTN4UBVS7OG5RONX
ip saddr 192.168.0.2 counter packets 0 bytes 0 jump KUBE-MARK-MASQ
meta l4proto udp counter packets 0 bytes 0 dnat to 192.168.0.2:53
chain KUBE-SVC-ERIFXISQEP7F7OF4
counter packets 0 bytes 0 jump KUBE-SEP-UPZX2EM3TRFH2ASL
counter packets 0 bytes 0 jump KUBE-SEP-KPHYKKPVMB473Z76
chain KUBE-SEP-UPZX2EM3TRFH2ASL
ip saddr 192.168.0.2 counter packets 0 bytes 0 jump KUBE-MARK-MASQ
meta l4proto tcp counter packets 0 bytes 0 dnat to 192.168.0.2:53
chain KUBE-SEP-4TCKAEJ6POVEFPVW
ip saddr 192.168.0.3 counter packets 0 bytes 0 jump KUBE-MARK-MASQ
meta l4proto udp counter packets 0 bytes 0 dnat to 192.168.0.3:53
chain KUBE-SEP-KPHYKKPVMB473Z76
ip saddr 192.168.0.3 counter packets 0 bytes 0 jump KUBE-MARK-MASQ
meta l4proto tcp counter packets 0 bytes 0 dnat to 192.168.0.3:53
chain KUBE-SEP-FATPLMAF37ZNQP5P
ip saddr 192.168.0.3 counter packets 0 bytes 0 jump KUBE-MARK-MASQ
meta l4proto tcp counter packets 0 bytes 0 dnat to 192.168.0.3:9153
chain KUBE-SVC-666FUMINWJLRRQPD
counter packets 1 bytes 60 jump KUBE-SEP-LYLDBZYLHY4MT3AQ
chain KUBE-SEP-LYLDBZYLHY4MT3AQ
ip saddr 192.168.0.4 counter packets 0 bytes 0 jump KUBE-MARK-MASQ
meta l4proto tcp counter packets 1 bytes 60 dnat to 192.168.0.4:8080
chain KUBE-XLB-EDNDUDH2C75GIR6O
fib saddr type local counter packets 0 bytes 0 jump KUBE-MARK-MASQ
fib saddr type local counter packets 0 bytes 0 jump KUBE-SVC-EDNDUDH2C75GIR6O
counter packets 0 bytes 0 jump KUBE-SEP-BLQHCYCSXY3NRKLC
chain KUBE-XLB-CG5I4G2RS3ZVWGLK
fib saddr type local counter packets 0 bytes 0 jump KUBE-MARK-MASQ
fib saddr type local counter packets 0 bytes 0 jump KUBE-SVC-CG5I4G2RS3ZVWGLK
counter packets 0 bytes 0 jump KUBE-SEP-5XVRKWM672JGTWXH
chain KUBE-SVC-EDNDUDH2C75GIR6O
counter packets 0 bytes 0 jump KUBE-SEP-BLQHCYCSXY3NRKLC
chain KUBE-SEP-BLQHCYCSXY3NRKLC
ip saddr 192.168.0.5 counter packets 0 bytes 0 jump KUBE-MARK-MASQ
meta l4proto tcp counter packets 0 bytes 0 dnat to 192.168.0.5:443
chain KUBE-SVC-CG5I4G2RS3ZVWGLK
counter packets 0 bytes 0 jump KUBE-SEP-5XVRKWM672JGTWXH
chain KUBE-SEP-5XVRKWM672JGTWXH
ip saddr 192.168.0.5 counter packets 0 bytes 0 jump KUBE-MARK-MASQ
meta l4proto tcp counter packets 0 bytes 0 dnat to 192.168.0.5:80
chain KUBE-SVC-EZYNCFY2F7N6OQA2
counter packets 0 bytes 0 jump KUBE-SEP-JYW326XAJ4KK7QPG
chain KUBE-SEP-JYW326XAJ4KK7QPG
ip saddr 192.168.0.5 counter packets 0 bytes 0 jump KUBE-MARK-MASQ
meta l4proto tcp counter packets 0 bytes 0 dnat to 192.168.0.5:8443
table ip filter
chain DOCKER
chain DOCKER-ISOLATION-STAGE-1
iifname "docker0" oifname != "docker0" counter packets 0 bytes 0 jump DOCKER-ISOLATION-STAGE-2
counter packets 0 bytes 0 return
chain DOCKER-ISOLATION-STAGE-2
oifname "docker0" counter packets 0 bytes 0 drop
counter packets 0 bytes 0 return
chain FORWARD
type filter hook forward priority filter; policy drop;
iifname "weave" counter packets 213 bytes 54014 jump WEAVE-NPC-EGRESS
oifname "weave" counter packets 150 bytes 30038 jump WEAVE-NPC
oifname "weave" ct state new counter packets 0 bytes 0 log group 86
oifname "weave" counter packets 0 bytes 0 drop
iifname "weave" oifname != "weave" counter packets 33 bytes 2941 accept
oifname "weave" ct state related,established counter packets 0 bytes 0 accept
counter packets 0 bytes 0 jump KUBE-FORWARD
ct state new counter packets 0 bytes 0 jump KUBE-SERVICES
ct state new counter packets 0 bytes 0 jump KUBE-EXTERNAL-SERVICES
counter packets 0 bytes 0 jump DOCKER-USER
counter packets 0 bytes 0 jump DOCKER-ISOLATION-STAGE-1
oifname "docker0" ct state related,established counter packets 0 bytes 0 accept
oifname "docker0" counter packets 0 bytes 0 jump DOCKER
iifname "docker0" oifname != "docker0" counter packets 0 bytes 0 accept
iifname "docker0" oifname "docker0" counter packets 0 bytes 0 accept
chain DOCKER-USER
counter packets 0 bytes 0 return
chain KUBE-FIREWALL
mark and 0x8000 == 0x8000 counter packets 0 bytes 0 drop
ip saddr != 127.0.0.0/8 ip daddr 127.0.0.0/8 ct status dnat counter packets 0 bytes 0 drop
chain OUTPUT
type filter hook output priority filter; policy accept;
ct state new counter packets 527014 bytes 31628984 jump KUBE-SERVICES
counter packets 36324809 bytes 6021214027 jump KUBE-FIREWALL
meta l4proto != esp mark and 0x20000 == 0x20000 counter packets 0 bytes 0 drop
chain INPUT
type filter hook input priority filter; policy accept;
counter packets 35869492 bytes 5971008896 jump KUBE-NODEPORTS
ct state new counter packets 390938 bytes 23457377 jump KUBE-EXTERNAL-SERVICES
counter packets 36249774 bytes 6030068622 jump KUBE-FIREWALL
meta l4proto tcp ip daddr 127.0.0.1 tcp dport 6784 fib saddr type != local ct state != related,established counter packets 0 bytes 0 drop
iifname "weave" counter packets 907273 bytes 88697229 jump WEAVE-NPC-EGRESS
counter packets 34809601 bytes 5818213726 jump WEAVE-IPSEC-IN
chain KUBE-KUBELET-CANARY
chain KUBE-PROXY-CANARY
chain KUBE-EXTERNAL-SERVICES
chain KUBE-NODEPORTS
meta l4proto tcp tcp dport 32196 counter packets 0 bytes 0 accept
meta l4proto tcp tcp dport 32196 counter packets 0 bytes 0 accept
chain KUBE-SERVICES
chain KUBE-FORWARD
ct state invalid counter packets 0 bytes 0 drop
mark and 0x4000 == 0x4000 counter packets 0 bytes 0 accept
ct state related,established counter packets 0 bytes 0 accept
ct state related,established counter packets 0 bytes 0 accept
chain WEAVE-NPC-INGRESS
chain WEAVE-NPC-DEFAULT
# match-set weave-;rGqyMIl1HN^cfDki~Z$3]6!N dst counter packets 14 bytes 840 accept
# match-set weave-P.B !ZhkAr5q=XZ?3 tMBA+0 dst counter packets 0 bytes 0 accept
# match-set weave-Rzff h:=]JaaJl/G;(XJpGjZ[ dst counter packets 0 bytes 0 accept
# match-set weave-]B*(W?)t*z5O17G044[gUo#$l dst counter packets 0 bytes 0 accept
# match-set weave-iLgO^ o=U/*%KE[@=W:l~ 9T dst counter packets 9 bytes 540 accept
chain WEAVE-NPC
ct state related,established counter packets 124 bytes 28478 accept
ip daddr 224.0.0.0/4 counter packets 0 bytes 0 accept
# PHYSDEV match --physdev-out vethwe-bridge --physdev-is-bridged counter packets 3 bytes 180 accept
ct state new counter packets 23 bytes 1380 jump WEAVE-NPC-DEFAULT
ct state new counter packets 0 bytes 0 jump WEAVE-NPC-INGRESS
chain WEAVE-NPC-EGRESS-ACCEPT
counter packets 48 bytes 3769 meta mark set mark or 0x40000
chain WEAVE-NPC-EGRESS-CUSTOM
chain WEAVE-NPC-EGRESS-DEFAULT
# match-set weave-s_+ChJId4Uy_$ G;WdH ~TK)I src counter packets 0 bytes 0 jump WEAVE-NPC-EGRESS-ACCEPT
# match-set weave-s_+ChJId4Uy_$ G;WdH ~TK)I src counter packets 0 bytes 0 return
# match-set weave-E1ney4o[ojNrLk.6rOHi;7MPE src counter packets 31 bytes 2749 jump WEAVE-NPC-EGRESS-ACCEPT
# match-set weave-E1ney4o[ojNrLk.6rOHi;7MPE src counter packets 31 bytes 2749 return
# match-set weave-41s)5vQ^o/xWGz6a20N:~?# E src counter packets 0 bytes 0 jump WEAVE-NPC-EGRESS-ACCEPT
# match-set weave-41s)5vQ^o/xWGz6a20N:~?# E src counter packets 0 bytes 0 return
# match-set weave-sui%__gZ kX~oZgI_Ttqp=Dp src counter packets 0 bytes 0 jump WEAVE-NPC-EGRESS-ACCEPT
# match-set weave-sui%__gZ kX~oZgI_Ttqp=Dp src counter packets 0 bytes 0 return
# match-set weave-nmMUaDKV*YkQcP5s?Q[R54Ep3 src counter packets 17 bytes 1020 jump WEAVE-NPC-EGRESS-ACCEPT
# match-set weave-nmMUaDKV*YkQcP5s?Q[R54Ep3 src counter packets 17 bytes 1020 return
chain WEAVE-NPC-EGRESS
ct state related,established counter packets 907425 bytes 88746642 accept
# PHYSDEV match --physdev-in vethwe-bridge --physdev-is-bridged counter packets 0 bytes 0 return
fib daddr type local counter packets 11 bytes 640 return
ip daddr 224.0.0.0/4 counter packets 0 bytes 0 return
ct state new counter packets 50 bytes 3961 jump WEAVE-NPC-EGRESS-DEFAULT
ct state new mark and 0x40000 != 0x40000 counter packets 2 bytes 192 jump WEAVE-NPC-EGRESS-CUSTOM
chain WEAVE-IPSEC-IN
chain WEAVE-CANARY
table ip mangle
chain KUBE-KUBELET-CANARY
chain PREROUTING
type filter hook prerouting priority mangle; policy accept;
chain INPUT
type filter hook input priority mangle; policy accept;
counter packets 35716863 bytes 5906910315 jump WEAVE-IPSEC-IN
chain FORWARD
type filter hook forward priority mangle; policy accept;
chain OUTPUT
type route hook output priority mangle; policy accept;
counter packets 35804064 bytes 5938944956 jump WEAVE-IPSEC-OUT
chain POSTROUTING
type filter hook postrouting priority mangle; policy accept;
chain KUBE-PROXY-CANARY
chain WEAVE-IPSEC-IN
chain WEAVE-IPSEC-IN-MARK
counter packets 0 bytes 0 meta mark set mark or 0x20000
chain WEAVE-IPSEC-OUT
chain WEAVE-IPSEC-OUT-MARK
counter packets 0 bytes 0 meta mark set mark or 0x20000
chain WEAVE-CANARY
nftables
entries, but it explains what s going on. We have a nat entry for:
meta l4proto tcp ip daddr 10.107.66.138 tcp dport 8080 counter packets 1 bytes 60 jump KUBE-SVC-666FUMINWJLRRQPD
KUBE-SEP-LYLDBZYLHY4MT3AQ
and:
meta l4proto tcp counter packets 1 bytes 60 dnat to 192.168.0.4:8080
192.168.0.4
. Which we can see in our routing table via the weave
interface. Mystery explained. We can see the ingress for the externally visible HTTP service as well:
meta l4proto tcp ip daddr 192.168.33.147 tcp dport 80 counter packets 0 bytes 0 jump KUBE-XLB-CG5I4G2RS3ZVWGLK
meta l4proto tcp counter packets 0 bytes 0 dnat to 192.168.0.5:80
echoserver
pod to be 192.168.0.4
and the IP address instead our nginx ingress pod to be 192.168.0.5
. Let s look:
root@udon:/# docker ps grep echoserver
7cbb177bee18 k8s.gcr.io/echoserver "/usr/local/bin/run. " 3 days ago Up 3 days k8s_echoserver_hello-node-59bffcc9fd-8hkgb_default_c7111c9e-7131-40e0-876d-be89d5ca1812_0
root@udon:/# docker exec -it 7cbb177bee18 /bin/bash
root@hello-node-59bffcc9fd-8hkgb:/# awk '/32 host/ print f f=$2 ' <<< "$(</proc/net/fib_trie)" sort -u
127.0.0.1
192.168.0.4
192.168.0.4
address.
I ve touched here upon the ability to actually enter a container and have a poke around its running environment by using docker
directly. Next step is to use that to investigate what containers have actually been spun up and what they re doing. I ll also revisit networking when I get to the point of building a multi-node cluster, to examine how the bridging between different hosts is done.
Kiran s Interview in Times of India (TOI) There isn t much to say apart from I haven t used it. I just didn t want to. It just is unethical. Hopefully, in the coming days GOI does something better. That is the only thing we are surviving on, hope.
Host foo.company.com
Port 22
Host *.company.com
Port 10022
With this example, ssh connection to foo.company.com is done using port 22 and connection to bar.company.com with port 10022.
If the Host sections are written back in reverse order:
Host *.company.com
Port 10022
Host foo.company.com
Port 22
Then, ssh would be happy to use the first matching section for foo.company.com , i.e. *.company.com and would use the wrong port (10022)
This is now fixed with Config::Model::OpenSsh 2.8.4.3 which is available on cpan and in Debian/experimental.
While I was at it, I ve also updated Managing OpenSsh configuration with cme wiki page.
All the best
Interestingly as well, while in the vaccine issue, Brazil Anvisa doesn t know what they are doing or the regulator just isn t knowledgeable etc. (statements by various people in GOI, when it comes to testing kits, the same is an approver.)
ICMR/DGCI approving internationally validated kits, Press release.
Next.