This post serves as a report from my attendance to Kubecon and CloudNativeCon 2023 Europe that took place in
Amsterdam in April 2023. It was my second time physically attending this conference, the first one was in
Austin, Texas (USA) in 2017. I also attended once in a virtual fashion.
The content here is mostly generated for the sake of my own recollection and learnings, and is written from
the notes I took during the event.
The very first session was the opening keynote, which reunited the whole crowd to bootstrap the event and
share the excitement about the days ahead. Some astonishing numbers were announced: there were more than
10.000 people attending, and apparently it could confidently be said that it was the largest open source
technology conference taking place in Europe in recent times.
It was also communicated that the next couple iteration of the event will be run in China in September 2023
and Paris in March 2024.
More numbers, the CNCF was hosting about 159 projects, involving 1300 maintainers and about 200.000
contributors. The cloud-native community is ever-increasing, and there seems to be a strong trend in the
industry for cloud-native technology adoption and all-things related to PaaS and IaaS.
The event program had different tracks, and in each one there was an interesting mix of low-level and higher
level talks for a variety of audience. On many occasions I found that reading the talk title alone was not
enough to know in advance if a talk was a 101 kind of thing or for experienced engineers. But unlike in
previous editions, I didn t have the feeling that the purpose of the conference was to try selling me
anything. Obviously, speakers would make sure to mention, or highlight in a subtle way, the involvement of a
given company in a given solution or piece of the ecosystem. But it was non-invasive and fair enough for me.
On a different note, I found the breakout rooms to be often small. I think there were only a couple of rooms
that could accommodate more than 500 people, which is a fairly small allowance for 10k attendees. I realized
with frustration that the more interesting talks were immediately fully booked, with people waiting in line
some 45 minutes before the session time. Because of this, I missed a few important sessions that I ll
hopefully watch online later.
Finally, on a more technical side, I ve learned many things, that instead of grouping by session I ll group
by topic, given how some subjects were mentioned in several talks.
On gitops and CI/CD pipelines
Most of the mentions went to
FluxCD and
ArgoCD. At
that point there were no doubts that gitops was a mature approach and both flux and argoCD could do an
excellent job. ArgoCD seemed a bit more over-engineered to be a more general purpose CD pipeline, and flux
felt a bit more tailored for simpler gitops setups. I discovered that both have nice web user interfaces that
I wasn t previously familiar with.
However, in two different talks I got the impression that the initial setup of them was simple, but migrating
your current workflow to gitops could result in a bumpy ride. This is, the challenge is not deploying
flux/argo itself, but moving everything into a state that both humans and flux/argo can understand. I also
saw some curious mentions to the config drifts that can happen in some cases, even if the goal of gitops is
precisely for that to never happen. Such mentions were usually accompanied by some hints on how to operate
the situation by hand.
Worth mentioning, I missed any practical information about one of the key pieces to this whole gitops story:
building container images. Most of the showcased scenarios were using pre-built container images, so in that
sense they were simple. Building and pushing to an image registry is one of the two key points we would need
to solve in Toolforge Kubernetes if adopting gitops.
In general, even if gitops were already in our radar for
Toolforge Kubernetes,
I think it climbed a few steps in my priority list after the conference.
Another learning was this site:
https://opengitops.dev/.
On etcd, performance and resource management
I attended a talk focused on etcd performance tuning that was very encouraging. They were basically talking
about the
exact
same problems we
have had in Toolforge Kubernetes, like api-server and etcd failure modes, and how sensitive etcd is to
disk
latency, IO pressure and network throughput. Even though
Toolforge Kubernetes scale is small compared to other Kubernetes deployments out there, I found it very
interesting to see other s approaches to the same set of challenges.
I learned how most Kubernetes components and apps can overload the api-server. Because even the api-server
talks to itself. Simple things like
kubectl may have a completely different impact on the API depending on
usage, for example when listing the whole list of objects (very expensive) vs a single object.
The conclusion was to try avoiding hitting the api-server with LIST calls, and use ResourceVersion which
avoids full-dumps from etcd (which, by the way, is the default when using bare
kubectl get calls). I
already knew some of this, and for example the
jobs-framework-emailer was already making use of this
ResourceVersion functionality.
There have been a lot of improvements in the performance side of Kubernetes in recent times, or more
specifically, in how resources are managed and used by the system. I saw a review of resource management from
the perspective of the container runtime and kubelet, and plans to support fancy things like topology-aware
scheduling decisions and dynamic resource claims (changing the pod resource claims without
re-defining/re-starting the pods).
On cluster management, bootstrapping and multi-tenancy
I attended a couple of talks that mentioned kubeadm, and one in particular was from the maintainers
themselves. This was of interest to me because as of today
we use it for
Toolforge. They shared all
the latest developments and improvements, and the plans and roadmap for the future, with a special mention to
something they called kubeadm operator , apparently capable of auto-upgrading the cluster, auto-renewing
certificates and such.
I also saw a comparison between the different cluster bootstrappers, which to me confirmed that kubeadm was
the best, from the point of view of being a well established and well-known workflow, plus having a very
active contributor base. The kubeadm developers invited the audience to submit feature requests,
so I did.
The different talks confirmed that the basic unit for multi-tenancy in kubernetes is the namespace. Any
serious multi-tenant usage should leverage this. There were some ongoing conversations, in official sessions
and in the hallway, about the right tool to implement K8s-whitin-K8s, and
vcluster
was mentioned enough times for me to be convinced it was the right candidate. This was despite of my impression
that multiclusters / multicloud are regarded as hard topics in the general community. I definitely would like to play
with it sometime down the road.
On networking
I attended a couple of basic sessions that served really well to understand how Kubernetes instrumented the
network to achieve its goal. The conference program had sessions to cover topics ranging from network
debugging recommendations, CNI implementations, to IPv6 support. Also, one of the keynote sessions had a
reference to how kube-proxy is not able to perform NAT for SIP connections, which is interesting because I
believe Netfilter Conntrack could do it if properly configured. One of the conclusions on the CNI front was
that Calico has a massive community adoption (in Netfilter mode), which is reassuring, especially considering
it is
the one we use for Toolforge Kubernetes.
On jobs
I attended a couple of talks that were related to HPC/grid-like usages of Kubernetes. I was truly impressed
by some folks out there who were using Kubernetes Jobs on massive scales, such as to train machine learning
models and other fancy AI projects.
It is acknowledged in the community that the early implementation of things like Jobs and CronJobs had some
limitations that are now gone, or at least greatly improved. Some new functionalities have been added as
well. Indexed Jobs, for example, enables each Job to have a number (index) and process a chunk of a larger
batch of data based on that index. It would allow for full grid-like features like sequential (or again,
indexed) processing, coordination between Job and more graceful Job restarts. My first reaction was: Is that
something we would like to enable in
Toolforge Jobs Framework?
On policy and security
A surprisingly good amount of sessions covered interesting topics related to policy and security. It was nice
to learn two realities:
- kubernetes is capable of doing pretty much anything security-wise and create
greatly secured environments.
- it does not by default. The defaults are not security-strict on purpose.
It kind of made sense to me: Kubernetes was used for a wide range of use cases, and developers didn t know
beforehand to which particular setup they should accommodate the default security levels.
One session in particular covered the most basic security features that should be enabled for any Kubernetes
system that would get exposed to random end users. In my opinion, the Toolforge Kubernetes setup was already
doing a good job in that regard. To my joy, some sessions referred to the Pod Security Admission mechanism,
which is one of the key security features we re about to adopt (when migrating away from
Pod Security Policy).
I also learned a bit more about Secret resources, their current implementation and how to leverage a
combo of CSI and RBAC for a more secure usage of external secrets.
Finally, one of the major takeaways from the conference was learning about
kyverno and
kubeaudit. I was previously aware of the
OPA Gatekeeper. From the several demos I saw, it
was to me that kyverno should help us make Toolforge Kubernetes more sustainable by replacing all of our
custom admission controllers
with it. I already opened a ticket to
track this idea, which I ll be
proposing to my team soon.
Final notes
In general, I believe I learned many things, and perhaps even more importantly I re-learned some stuff I had
forgotten because of lack of daily exposure. I m really happy that the cloud native way of thinking was
reinforced in me, which I still need because most of my muscle memory to approach systems architecture and
engineering is from the old pre-cloud days.
List of sessions I attended on the first day:
- Keynote
- Node Resource Management: The Big Picture - Sascha Grunert & Swati Sehgal, Red Hat; Alexander Kanevskiy, Intel; Evan Lezar, NVIDIA; David Porter, Google.
- How We Securely Scaled Multi-Tenancy with VCluster, Crossplane, and Argo CD - Ilia Medvedev & Kostis Kapelonis, Codefresh. (Couldn t really attend, room full)
- Flux Beyond Git: Harnessing the Power of OCI - Stefan Prodan & Hidde Beydals, Weaveworks. (Couldn t really attend, room full)
- Tutorial: Measure Twice, Cut Once: Dive Into Network Foundations the Right Way! - Marino Wijay & Jason Skrzypek, Solo.io
- Argo CD Core - A Pure GitOps Agent for Kubernetes - Alexander Matyushentsev, Akuity & Leonardo Luz Almeida, Intuit
- Kubeadm Deep Dive - Rohit Anand, NEC & Paco Xu, Dao
- Cloud Operate Multi-Tenancy Service Mesh with ArgoCD in Production - Lin Sun, Solo.io & Faseela K, Ericsson Software Technology
List of sessions I attended on the second day:
List of sessions I attended on third day:
- Keynote
- Prevent Embarrassing Cluster Takeovers with This One Simple Trick! - Daniele de Araujo dos Santos & Shane Lawrence, Shopify
- Hacking and Defending Kubernetes Clusters: We ll Do It LIVE!!! - Fabian Kammel & James Cleverley-Prance, ControlPlane
- Painless Multi-Cloud to the Edge Powered by NATS & Kubernetes - Tomasz Pietrek & David Gee, Synadia
- Demystifing IPv6 Kubernetes - Antonio Jose Ojea Garcia, Google & Fernando Gont, Yalo
- Open Policy Agent. (OPA) Intro & Deep Dive - Charlie Egan, Styra, Inc. (Couldn t really attend, room full)
- Practical Challenges with Pod Security Admission - V K rbes & Christian Schlotter, VMware
- Enabling HPC and ML Workloads with the Latest Kubernetes Job Features - Micha Wo niak, Google & Vanessa Sochat, Lawrence Livermore National Laboratory
- Can You Keep a Secret? on Secret Management in Kubernetes - Liav Yona & Gal Cohen, Firefly
The videos have been
published on Youtube.
Debian v12 with codename bookworm was released as new stable release on 10th of June 2023. Similar to what we had with #newinbullseye and previous releases, now it s time for #newinbookworm!
I was the driving force at several of my customers to be well prepared for bookworm. As usual with major upgrades, there are some things to be aware of, and hereby I m starting my public notes on bookworm that might be worth also for other folks. My focus is primarily on server systems and looking at things from a sysadmin perspective.
Further readings
As usual start at the official Debian release notes, make sure to especially go through What s new in Debian 12 + Issues to be aware of for bookworm.
Package versions
As a starting point, let s look at some selected packages and their versions in bullseye vs. bookworm as of 2023-02-10 (mainly having amd64 in mind):
Just a few days back we came to know about the horrific Train Crash that happened in Odisha (Orissa). There are some things that are known and somethings that can be inferred by observance. Sadly, it seems the incident is going to be covered up 
. Some of the facts that have not been contested in the public domain are that there were three lines. One loop line on which the Goods Train was standing and there was an up and a down line. So three lines were there. Apparently, the signalling system and the inter-locking system had issues as highlighted by an official about a month 
A
I write letters. The kind that are written on paper with a dip pen 
So I headed over to the obvious program anybody would use for these things (XeLaTeX, of course) and quickly designed a page with fields for the basic thinks I want to record; it was a bit hurried, and I may improve on it the next time I make one, but I expect this one to last me two or three years, and it is good enough.
I ve decided to make it A6 sized, so that it doesn t require a lot of space on my busy desktop, and it could be carried inside a portable desktop, if I ever decide to finish the one for which I ve made a mockup years ago :)
I ve also added a few pages for the addresses of my correspondents (and an index of the letters I ve exchanged with them), and a few empty pages for other notes.
Then I ve used my 
After printing (an older version where some of the pages are repeated. whoops, but it only happened 4 times, and it s not a big deal), it was time for binding this into a book.
I ve opted for Coptic stitch, so that the book will open completely flat and writing on it will be easier and the covers are 2 mm cardboard covered in linen-look bookbinding paper (sadly I no longer have a source for bookbinding cloth made from actual cloth).
I tried to screenprint a simple design on the cover: the first attempt was unusable (the paper was smaller than the screen, so I couldn t keep it in the right place and moved as I was screenprinting); on the second attempt I used some masking tape to keep the paper in place, and they were a bit better, but I need more practice with the technique.
Finally, I decided that for such a Victorian thing I will use an Iron-gall ink, but it s Rohrer & Knlingner Scabiosa, with a purple undertone, because life s too short to use blue-black ink :D
And now, I m off to write an actual letter, rather than writing online about things that are related to letter writing.
The following contributors got their Debian Developer accounts in the last two months:
Considering I ve created my own private cloud in my home as part of: wp-k8s: WordPress on privately hosted Kubernetes cluster (Raspberry Pi 4 + Synology)....
The post 
A first follow-up to the 
This isn't a book review, although the reason that I am typing this now is
because of a book, You Are Not Alone: from the creator and host of
Griefcast, Cariad Lloyd, ISBN: 978-1526621870 and I include a handful of
quotes from Cariad where there is really no better way of describing things.
Many people experience death for the first time as a child, often relating to a
family pet. Death is universal but every experience of death is unique. One of
the myths of grief is the idea of the Five Stages but this is a
misinterpretation. Denial, Anger, Bargaining, Depression and Acceptance
represent the five stage model of death and have nothing to do with
grief. The five stages were developed from studying those who are
terminally ill, the dying, not those who then grieve for the dead person and
have to go on living without them. Grief is for those who loved the person who
has died and it varies between each of those people just as people vary in how
they love someone. The Five Stages end at the moment of death, grief is what
comes next and most people do not grieve in stages, it can be more like a
tangled knot.
Death has a date and time, so that is why the last stage of the model is
Acceptance. Grief has no timetable, those who grieve will carry that grief for
the rest of their lives. Death starts the process of grief in those who go on
living just as it ends the life of the person who is loved. "Grief eases and
changes and returns but it never disappears.".
I suspect many will have already stopped reading by this point. People do not
talk about death and grief enough and this only adds to the burden of those who
carry their grief. It can be of enormous comfort to those who have carried
grief for some time to talk directly about the dead, not in vague pleasantries
but with specific and strong memories. Find a safe place without distractions
and talk with the person grieving face to face. Name the dead person. Go to
places with strong memories and be there alongside. Talk about the times with
that person before their death. Early on, everything about grief is painful and
sad. It does ease but it remains unpredictable. Closing it away in a box inside
your head (as I did at one point) is like cutting off a damaged limb but
keeping the pain in a box on the shelf. You still miss the limb and eventually,
the box starts leaking.
For me, there were family pets which died but my first job out of university
was to work in hospitals, helping the nurses manage the medication regimen and
providing specialist advice as a pharmacist. It will not be long in that
environment before everyone on the ward gets direct experience of the death of
a person. In some ways, this helped me to separate the process of death from
the process of grief. I cared for these people as patients but these were not
my loved ones. Later, I worked in specialist terminal care units, including
providing potential treatments as part of clinical trials. Here, it was not
expected for any patient to be discharged alive. The more aggressive
chemotherapies had already been tried and had failed, this was about pain
relief, symptom management and helping the loved ones. Palliative care is not
just about the patient, it involves helping the loved ones to accept what is
happening as this provides comfort to the patient by closing the loop.
Grief is stressful. One of the most common 
After my father passed away, I brought home most of the personal items
he had, both at home and at his office. Among many, many (many, many,
many) other things, I brought two of his personal treasures: His photo
collection and a box with the 8mm movies he shot approximately between
1956 and 1989, when he was forced into modernity and got a portable
videocassette recorder.
I have talked with several friends, as I really want to get it all in
a digital format, and while I ve been making slow but steady advances
scanning the photo reels, I was particularly dismayed (even though it
was most expected most personal electronic devices aren t meant to
last over 50 years) to find out the 8mm projector was no longer in
working conditions; the lamp and the fans work, but the spindles won t
spin. Of course, it is quite likely it is easy to fix, but it is
beyond my tinkering abilities and finding photographic equipment
repair shops is no longer easy. Anyway, even if I got it fixed,
filming a movie from a screen, even with a decent camera, is a lousy
way to get it digitized.
But almost by mere chance, I got in contact with my cousin Daniel, ho
came to Mexico to visit his parents, and had precisely brought with
him a 8mm/Super8 movie scanner! It is a much simpler piece of
equipment than I had expected, and while it does present some minor
glitches (i.e. the vertical framing slightly loses alignment over the
course of a medium-length film scanning session, and no adjustment is
possible while the scan is ongoing), this is something
that can be decently fixed in post-processing, and a scanning session
can be split with no ill effects. Anyway, it is quite uncommon a
mid-length (5min) film can be done without interrupting i.e. to join a
splice, mostly given my father didn t just film, but also edited a lot
(this is, it s not just family pictures, but all different kinds of
fiction and documentary work he did).
Draw graph of number of commits. I thought I had lots of commits this month for Chrome OS.
To count those, all merged commits have committer Chromeos LUCI.
To get the timestamp of the commits,
git log --committer="Chromeos LUCI" --pretty=%ct . gives me the list of timestamps.
UNIX timestamp can be parsed with datetime.fromtimestamp, and then that array can be processed with density graph plot or histogram plotting tool, such as plt.hist.
Way back at