First off, my members-only post is much happier. Members-only posts continue until open source Medium becomes available. Don t worry, if you write to me privately, I can give you a copy on request.Here are a few donuts for everyone:https://medium.com/media/48d02097292c207c14cfa0203f840bc2/hrefhttps://medium.com/media/092204ce4dd42849ab42c7dae9172c8a/hrefhttps://medium.com/media/28bd9e3206e39011fab17e7d7159c374/hrefGood.
Hey folks! Ope, you might need the Midwest Voice Translator on this one. Read this one in a heavy Midwest accent, and tell your folks I
Don t knock the fence down before you know why it s up. I repeat this phrase over and over again, yet the (metaphorical) Homeowner s Association still decides my fence is the wrong color.Well, now you get to know why the fence is up. If anyone s actually willing to challenge me on this level, I d welcome it.The four ideas I d like to discuss are this: quantum physics, Lutheranism, mental resilience, and psychology. I ve been studying these topics intensely for the past decade as a passion project. I m just going to let my thoughts flow, but I d like to hear other opinions on this.Can the mysteries of the mind, the subatomic world, and faith converge to reveal deeper truths?When it comes to self-taught knowledge on analysis, I m mostly learned on Freud, with some hints of Jung and Peterson. I ve read much of the original source material, and watched countless presentations on it. This all being said, I m both learned on Rothbard and Marx, so if there is a major flaw in the way of Freud is frowned upon, I d genuinely like to know so I can update my research and juxtapose the two schools of thought.Alongside this, although probably not directly relevant, I m learned on John Locke and transcendentalism. What I d like to focus on here is this the Id.The Id is the pleasure-seeking, instinctual part of the psyche. Jung further extends this into the idea of the shadow self, and Peterson maps the meanings of these texts into a combined work (at least in my rudimentary understanding).In my research, the Id represents the part of your psyche that deals with religious values. As an example, if you re an impulsive person, turning to a spiritual or religious outlet can be highly beneficial. I ve been using references from the foundational text of the Judaeo-Christian value system this entire time, feel free to re-read my other blog posts (instead of claiming they don t exist).Let s tie this into quantum physics. This is the part where I ll struggle most. I ve watched several movies about this, read several books, and even learned about it academically, but quantum physics is likely to be my weak spot here.I did some research, and here are the elements I m looking for: uncertainty principle, wave-particle duality, quantum entanglement, and the observer effect.I already know about the cat in the box. And the Cat in the Hat, for that matter. I know about wave-particle duality from an incredibly intelligent high school physics teacher of mine. I know about the uncertainty principle purely in a colloquial sense. The remaining element I need to wrap my head around is quantum entanglement, but it feels like I m almost there.These concepts do actually challenge the idea of pure free will. It s almost like we re coming full circle. Some theologians (including myself, if you can call me a self-taught one) do believe the idea of quantum indeterminacy can be a space where divine action may take place. You could also liken the unpredictable nature of the Id to quantum indeterminacy as well. These are ones to think about, because in all reality, they re subjective opinions. I do believe they re interconnected.In terms of Lutheranism, I ll be short on this one. Please do go read the full history behind Martin Luther and his turbulent relationship with Catholicism. I m not a Bible thumper, and I actually think this is the first time I ve mentioned religion publicly at all. This being said, now I m actually ready to defend the points on an academic level.The Id represents hidden psychological forces, quantum physics reveals subatomic mysteries, and Lutheranism emphasizes faith in the unseen God. Okay, so we have the baseline. Now, time for some mental resilience. When I think of mental resilience, the first people I think of are David Goggins and Jocko Willink. I ve also enjoyed Dr. Andrew Huberman s podcast.The idea there is simple if you understand exactly how to learn, you know your fundamentals well enough to draw them and explain them vividly on a whiteboard, and you can make it a habit, at that point you re ready to work on your mental resilience. Little by little, gradually, how far can you push the bar towards the ceiling?There s obviously limits. People sometimes get scared when I mention mental resilience, but obviously that s a bit of a catch 22. There are plenty of satirical videos out there, and of course, I don t believe in Goggins or Jocko wholeheartedly. They re just tools in the toolbox when times get tough.I wish you all well, and I hope this gets you thinking about those people who just insist there is no God or higher being, and think you re stupid for believing there is one. Those people obviously haven t read analysis, in my own opinion.Have a great night!
Edit: for the people calling this unhinged, you don t have the full context. I know it s not a good look. Just stop and think for a second, though.I d like to start this post by saying that I am indeed well. I ve thought so from the very beginning, and it s been confirmed by professionals as such. That being said, there is still this perception that people are still believing the other person that needs help.See, when you re deeply involved in this space for ten years and start a blog to share all the cool things you ve learned, you don t expect people to spread the rumor that you re crazy. And you especially don t expect them to resort to a legal bluff and then brag about it in a private IRC channel (yes, I have the proof.)I d like my apology letter, please. And I d like you all to stop using Orwellian tactics to silence me by spreading those rumors, please. I know the source of the rumor-mill, and it s being dealt with. There has been progress, but one thing hasn t happened yet I haven t received my apology letter. I ll even volunteer, out of my own good spirit, to remove this post and all BwE parts of my first post if you send me a full apology letter in the next 24 hours. I d do that, just to get you to open your eyes and smile.Silencing people, or asking them to remove parts of a post you don t like, is never okay. You could choose whether to publish it on your platform, sure, but you certainly don t get to pick and choose what I put in my personal blog posts.I am a United States citizen with a 1st amendment right that shall be respected. I ve also studied and enjoyed writing for years. I m sick and tired of the back and forth on this. I ve wanted to move on and you are actively not letting me.I have the full IRC logs. Several other people have attested they also have them. What would you like to do next?I m not sure about you, but I m sitting here eating a donut. This isn t harrassment, this is self defense. At some point, you have to get sick and tired of issues happening, and you have to stand up for yourself, on your own footing.It isn t easy. I probably won t look back favorably on this post and thought process in a few years. That being said, it s necessary to defend myself on someone who has mistreated me personally on so many grave levels. You ll note I still haven t made the name public.They want to make you out to look like a loose cannon. They want to push your buttons and make you write a rant blog post telling you how much they hate them. I m not going to dignify that, because I know a specific electrician who wouldn t be happy if I had to write him a dissent.I m done. Don t defame me anymore.With donuts, A popular topic of public conversation in 2025 is balance. How do we balance budgets, how do we balance entities, and how do we balance perspectives? How do we balance the right of free expression with our ability to effectively convey a message?Here s another popular topic of conversation AI. What is it? What does it do?I m going to give you some resources, as someone who first learned the inner workings of AI about ten years ago.I ll start with the presentation I gave in middle school. Our objective was to give a presentation on a topic of our choice, and we would be graded on our ability to give a presentation. Instead of talking about specific things or events, I talked about the broader idea of fully establishing an artificial form of intelligence.This is the video I used as a basis for that presentation:https://medium.com/media/21d2427a502b7c7cb669220e2e3478c8/hrefNot only did I explain exactly how this specific video game worked, it helped me understand machine learning and genetic algorithms. If I m recalling correctly, the actual title of my presentation had to do with genetic algorithms specifically.In the presentation, I specifically tied in Darwin s readings on evolution (of course, I had to keep it secular ), directly relating the information I learned about evolution in science class into a presentation about what would become AI. But Simon, the title of that video says Machine Learning. Do you have your glasses on?!? Yes, yes I do. It took me a few years to watch this space evolve, as I focused on other portions of the open source world. This changed when I attended SCaLE 21x. At that conference, the product manager for AI at Canonical (apologies if I m misquoting your exact title) gave a presentation on how she sees this space evolving. It s a must watch, in my opinion:https://medium.com/media/a13b2e46fc8acaa3bebf01a7f7bdeebb/hrefThis comprehensive presentation really covers the entire space, and does an excellent job at giving the whole picture.The short of it is this calling everything AI is inaccurate. Using AI for everything under the sun also isn t accurate. Speaking of the sun, it will get us if we don t find a sustainable way to get all that energy we ll need.I also read a paper on this issue, which I believe ties it together nicely. Published in June 2024, it s titled Situational Awareness The Decade Ahead and does an excellent job in predicting how this space will evolve. So far, it s been very accurate.The reason I m explaining this is fairly simple. In 2025, I still don t think many people have taken the time to dig into the content. From many conversations I ve heard, including one I took notes on in an entirely personal capacity, I m finding that not many people have a decent idea for where this space is going.It s been researched! :)If someone can provide a dissent for this view of the artificial intelligence space in the comments, I d be more than happy to hear it. Here s where I think this connects to the average person Many of the open source companies right now, without naming names, are focusing too much on the corporate benefits of AI. Yes, AI will be an incredibly useful tool for large organizations, and it will have a great benefit for how we conduct business over the course of the next decade. But do we have enough balance?Before you go all-in on AI, just do your research, please. Take a look at your options, and choose one that is correctly calibrated with the space as you see it.Lastly, when I talk about AI, I always bring up Orwell. I m a very firm, strong believer in free speech. AI must not be used to censor content, and the people who design your AI stack are very important. Look at which one of the options, as a company, enforces a diversity policy that is consistent with your values. The values of that company will carry over into its product. If you think I m wrong about this point, seriously, go read 1984 by George Orwell a few times over. You ll get the picture on what we re looking to avoid.In short, there s no need to over-complicate AI to those who don t understand it. Use the video game example. It s simple, and it works. Try using that same sentiment in your messaging, too. Appealing to both companies and individual users, together, should be important for open source companies, especially those with a large user base.I wish you all well. If you re getting to the end of this post and you re mad at me, sorry about that. Go re-read 1984 just one more time, please. ;)
In my home state of Wisconsin, there is an incredibly popular gas station called Kwik Trip. (Not to be confused with Quik Trip.) It is legitimately one of the best gas stations I ve ever been to, and I m a frequent customer.What makes it that great?Well, everything about it. The store is clean, the lights work, the staff are always friendly (and encourage you to come back next time), there s usually bakery on sale (just depends on location etc), and the list goes on.There s even a light-switch in the bathroom of a large amount of locations that you can flip if a janitor needs to attend to things. It actually does set off an alarm in the back room.A dear friend of mine from Wisconsin once told me something along the lines of, it s inaccurate to call Kwik Trip a gas station, because in all reality, it s a five star restaurant. (M , I hope you re well.)In my own opinion, they have an espresso machine. That s what really matters. ;)I mentioned the discount bakery. In reality, it s a pretty great system. To my limited understanding, the bakery that is older than standard but younger than expiry are set to half price and put towards the front of the store. In my personal experience, the vast majority of the time, the quality is still amazing. In fact, even if it isn t, the people working at Kwik Trip seem to genuinely enjoy their job.When you re looking at that discount rack of bakery, what do you choose? A personal favorite of mine is the banana nut bread with frosting on top. (To the non-Americans, yes, it does taste like it s homemade, it doesn t taste like something made in a factory.)Everyone chooses different bakery items. And honestly, there could be different discount items out depending on the time. You take what you can get, but you still have your own preferences. You like a specific type of donut (custard-filled, or maybe jelly-filled). Frosting, sprinkles there are so many ways to make different bakery items.It s not only art, it s kind of a science too.Is there a Kwik Trip that you ve called a gas station instead of a five star restaurant? Do you also want to tell people about your gas station? Do you only pick certain bakery items off the discount rack, or maybe ignore it completely? (And yes, there would be good reason to ignore the bakery in favor of the Hot Spot, I d consider that acceptable in my personal opinion.)Remember, sometimes you just have to like donuts.https://medium.com/media/73f78efd7bd6bb9ce495c2f08428c7d3/hrefHave a sweet day. :)
This morning, I went to make my usual cup of coffee. I was given an espresso machine for Christmas, and I ve developed this technique for making a warm drink that hits the spot every time.I ll start by turning on my espresso machine and starting a single shot of espresso. It dispenses and drips while I m working on the other parts.I then grab a coffee cup. Usually one of the taller ones. For maybe the bottom inch or two of the cup, that gets sugar and chocolate milk. Microwave for 45 seconds, pour in the espresso, then wash out the actual espresso from the metal cup with milk. Pour all of that in, another 45 seconds in the microwave, a few quick stirs, and you re all set.To the actual baristas out there, that probably sounds horrible. It probably sounds like the worst possible recommendation for a morning coffee ever.But, you know what? It works.So, I went to put my coffee into the microwave today, and I realized that someone else had put the glass plate for the microwave into the sink after accidentally spilling their breakfast on it.Instead of saying, well, I m not going to have my coffee this morning, I grabbed a large plate. I remembered the physics of levers from high school, and I understood that if I balanced everything just right, it would heat my coffee up.And well, here I am. With an un-spilled coffee and a story to tell.My point here is actually pretty simple, and this is before I even read any messages for the day. People with much more formal educations sometimes look at the guy engineering coffee with his microwave and think, what is this guy doing?!? All I m doing is making a really good cup of coffee. And to be honest, it tastes amazing.That s all. Have a wonderful day.
Toolboxes and Hammers Be YouEveryone has a story. We all started from somewhere, and we re all going somewhere.Ten years ago this summer, I first heard of Ubuntu. It took me time to learn how to properly pronounce the word, although I m glad I learned that early on. I was less fortunate when it came to the pronunciation of the acronym for the Ubuntu Code of Conduct. I had spent time and time again breaking my computer, and I d wanted to start fresh.I ve actually talked about this in an interview before, which you can find here (skip to 5:02 6:12 for my short explanation, I m in orange):https://medium.com/media/ad59becdbd06d230b875fb1512df1921/hrefI ve also done a few interviews over the years, here are some of the more recent ones:https://medium.com/media/83bda448d5f2a979f848e17f04376aa6/hrefAsk Noah Show 377Lastly, I did a few talks at SCaLE 21x (which the Ubuntu community donation funds helped me attend, thank you for that!):https://medium.com/media/0fbde7ef0ed83c2272a8653a5ea38b67/hrefhttps://medium.com/media/4d18f1770dc7eed6c7a9d711ff6a6e89/hrefMy story is fairly simple to summarize, if you don t have the time to go through all the clips.I started in the Ubuntu project at 13 years old, as a middle school student living in Green Bay, WI. I m now 23 years old, still living in Green Bay, but I became an Ubuntu Core Developer, Lubuntu s Release Manager, and worked up to a very great and comfortable spot.So, Simon, what advice would you give to someone at 13 who wants to do the same thing? Here are a few tips * Don t be afraid to be yourself. If you put on a mask, it hinders your growth, and you ll end up paying for it later anyway.
After more than ten years on my trusted X250, and with a lot of financial help for Debian (which I really thank, more on that later), I finally jumped on a new ThinkPad, an X13 Gen 5.
The migration path was really easy: I'm doing daily backups with borg of the whole filesystems on an encrypted USB drive, so I just had to boot a live USB key on the new laptop, plug the USB drive, create the partitioning (encryption, LVM etc.) and then run borg extract. Since I'm using LABEL in the various fstab I didn't have much to change.
I actually had a small hiccup because my daily backup scripts used ProtectKernelModules and besides preventing loading modules into the running kernel, it also prevents access to /usr/lib/modules. So when restoring I didn't have any modules for the installed kernels. No big deal, I reinstalled the kernel package from the chroot and it did work just fine.
All in all it was pretty smooth.
I've started a similar page as the X250 for the X13G5 but honestly I don't think I'll have to document a lot of stuff because everything basically works out of the box. It's not really a surprise because we went a long way since 2015 and Linux kernels are really tested on a lot of hardware, including laptops these days, and Intel laptops are the most standard stuff you can find. I guess it's still rocky for ARM64 laptops (and especially Apple hardware) but the point was less to do porting work for Debian and rather beeing more efficient for the current stuff I maintain (and sometimes struggle with).
As said above, the laptop has been funded by Debian and I really thank the DPL and the Debian France treasurer for authorizing it and beeing really fast on the reimbursement.
I had already posted a long time ago about hardware funding for Debian developers. It took me quite a while but I finally managed to ask for help because I couldn't afford the hardware at this point and it was becoming problematic. This is not something which should be done lightly (Debian wouldn't have the funds) but this is definitely something which should be done if needed. Don't hesitate to ask your fellow Debian developpers about advice on this.
After my previous posts related to Argo CD (one about
argocd-autopilot and another with some
usage examples) I started to look into
Kluctl (I also plan to review Flux, but I m more interested on the kluctl
approach right now).
While reading an entry on the project blog about Cluster API
somehow I ended up on the vCluster site and decided to give it a try, as it can be a valid
way of providing developers with on demand clusters for debugging or run CI/CD tests before deploying things on common
clusters or even to have multiple debugging virtual clusters on a local machine with only one of them running at any
given time.
On this post I will deploy a vcluster using the k3d_argocd kubernetes cluster (the one we created on the posts about
argocd) as the host and will show how to:
traefik) to access the API of the virtual one (removes the need of having to use the
vcluster connect command to access it with kubectl),sealed-secrets of the host cluster to manage the virtual cluster secrets.vcluster applicationTo create the virtual clusters we need the vcluster command, we can install it with arkade:
arkade get vclustervcluster.yaml fileTo create the cluster we are going to use the following vcluster.yaml file (you can find the documentation about all
its options here):
controlPlane:
proxy:
# Extra hostnames to sign the vCluster proxy certificate for
extraSANs:
- my-vcluster-api.lo.mixinet.net
exportKubeConfig:
context: my-vcluster_k3d-argocd
server: https://my-vcluster-api.lo.mixinet.net:8443
secret:
name: my-vcluster-kubeconfig
sync:
toHost:
ingresses:
enabled: true
serviceAccounts:
enabled: true
fromHost:
ingressClasses:
enabled: true
nodes:
enabled: true
clearImageStatus: true
secrets:
enabled: true
mappings:
byName:
# sync all Secrets from the 'my-vcluster-default' namespace to the
# virtual "default" namespace.
"my-vcluster-default/*": "default/*"
# We could add other namespace mappings if needed, i.e.:
# "my-vcluster-kube-system/*": "kube-system/*"controlPlane section we ve added the proxy.extraSANs entry to add an extra host name to make sure it is added
to the cluster certificates if we use it from an ingress.
The exportKubeConfig section creates a kubeconfig secret on the virtual cluster namespace using the provided
host name; the secret can be used by GitOps tools or we can dump it to a file to connect from our machine.
On the sync section we enable the synchronization of Ingress objects and ServiceAccounts from the virtual to the
host cluster:
IngressClass objects, to be able to use the host ingress server(s).Nodes (we are not using the info right now, but it could be interesting if we want to have the real information
of the nodes running pods of the virtual cluster).Secrets from the my-vcluster-default host namespace to the default of the virtual cluster; that
synchronization allows us to deploy SealedSecrets on the host that generate secrets that are copied
automatically to the virtual one. Initially we only copy secrets for one namespace but if the virtual cluster needs
others we can add namespaces on the host and their mappings to the virtual one on the vcluster.yaml file.vcluster create my-vcluster --namespace my-vcluster --upgrade --connect=false \
--values vcluster.yamlmy-vcluster namespace using the vcluster.yaml file shown before without
connecting to the cluster from our local machine (if we don t pass that option the command adds an entry on our
kubeconfig and launches a proxy to connect to the virtual cluster that we don t plan to use).IngressTcpRoute object to be able to connect to the vcluster API, we use the
following definition:
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRouteTCP
metadata:
name: my-vcluster-api
namespace: my-vcluster
spec:
entryPoints:
- websecure
routes:
- match: HostSNI( my-vcluster-api.lo.mixinet.net )
services:
- name: my-vcluster
port: 443
tls:
passthrough: truevcluster.yaml file, as explained before).my-vcluster-kubeconfig secret on its
namespace on the host cluster.
To dump it to the ~/.kube/my-vcluster-config we can do the following:
kubectl get -n my-vcluster secret/my-vcluster-kubeconfig \
--template=" .data.config " base64 -d > ~/.kube/my-vcluster-configvkubectl alias to adjust the KUBECONFIG variable to access it:
alias vkubectl="KUBECONFIG=~/.kube/my-vcluster-config kubectl"kubectx or a similar tool to change
the context (for our vcluster the context will be my-vcluster_k3d-argocd). If the KUBECONFIG variable is defined and
only has the PATH to a single file the merge can be done running the following:
KUBECONFIG="$KUBECONFIG:~/.kube/my-vcluster-config" kubectl config view \
--flatten >"$KUBECONFIG.new"
mv "$KUBECONFIG.new" "$KUBECONFIG"vkubectl alias when connecting to the virtual cluster, i.e. to check that it
works we can run the cluster-info subcommand:
vkubectl cluster-info
Kubernetes control plane is running at https://my-vcluster-api.lo.mixinet.net:8443
CoreDNS is running at https://my-vcluster-api.lo.mixinet.net:8443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.dummyhttpd applicationTo test the virtual cluster we are going to install the dummyhttpd application using the following
kustomization.yaml file:
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- https://forgejo.mixinet.net/blogops/argocd-applications.git//dummyhttp/?ref=dummyhttp-v1.0.0
# Add the config map
configMapGenerator:
- name: dummyhttp-configmap
literals:
- CM_VAR="Vcluster Test Value"
behavior: create
options:
disableNameSuffixHash: true
patches:
# Change the ingress host name
- target:
kind: Ingress
name: dummyhttp
patch: -
- op: replace
path: /spec/rules/0/host
value: vcluster-dummyhttp.lo.mixinet.net
# Add reloader annotations -- it will only work if we install reloader on the
# virtual cluster, as the one on the host cluster doesn't see the vcluster
# deployment objects
- target:
kind: Deployment
name: dummyhttp
patch: -
- op: add
path: /metadata/annotations
value:
reloader.stakater.com/auto: "true"
reloader.stakater.com/rollout-strategy: "restart"kustomize and vkubectl:
kustomize build . vkubectl apply -f -
configmap/dummyhttp-configmap created
service/dummyhttp created
deployment.apps/dummyhttp created
ingress.networking.k8s.io/dummyhttp createdcurl:
curl -s https://vcluster-dummyhttp.lo.mixinet.net:8443/ jq -cM .
"c": "Vcluster Test Value","s": "" vcluster now are:
vkubectl get all,configmap,ingress
NAME READY STATUS RESTARTS AGE
pod/dummyhttp-55569589bc-9zl7t 1/1 Running 0 24s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/dummyhttp ClusterIP 10.43.51.39 <none> 80/TCP 24s
service/kubernetes ClusterIP 10.43.153.12 <none> 443/TCP 14m
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/dummyhttp 1/1 1 1 24s
NAME DESIRED CURRENT READY AGE
replicaset.apps/dummyhttp-55569589bc 1 1 1 24s
NAME DATA AGE
configmap/dummyhttp-configmap 1 24s
configmap/kube-root-ca.crt 1 14m
NAME CLASS HOSTS ADDRESS PORTS AGE
ingress.networking.k8s.io/dummyhttp traefik vcluster-dummyhttp.lo.mixinet.net 172.20.0.2,172.20.0.3,172.20.0.4 80 24smy-vcluster namespace of the host cluster:
kubectl get all,configmap,ingress -n my-vcluster
NAME READY STATUS RESTARTS AGE
pod/coredns-bbb5b66cc-snwpn-x-kube-system-x-my-vcluster 1/1 Running 0 18m
pod/dummyhttp-55569589bc-9zl7t-x-default-x-my-vcluster 1/1 Running 0 45s
pod/my-vcluster-0 1/1 Running 0 19m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/dummyhttp-x-default-x-my-vcluster ClusterIP 10.43.51.39 <none> 80/TCP 45s
service/kube-dns-x-kube-system-x-my-vcluster ClusterIP 10.43.91.198 <none> 53/UDP,53/TCP,9153/TCP 18m
service/my-vcluster ClusterIP 10.43.153.12 <none> 443/TCP,10250/TCP 19m
service/my-vcluster-headless ClusterIP None <none> 443/TCP 19m
service/my-vcluster-node-k3d-argocd-agent-1 ClusterIP 10.43.189.188 <none> 10250/TCP 18m
NAME READY AGE
statefulset.apps/my-vcluster 1/1 19m
NAME DATA AGE
configmap/coredns-x-kube-system-x-my-vcluster 2 18m
configmap/dummyhttp-configmap-x-default-x-my-vcluster 1 45s
configmap/kube-root-ca.crt 1 19m
configmap/kube-root-ca.crt-x-default-x-my-vcluster 1 11m
configmap/kube-root-ca.crt-x-kube-system-x-my-vcluster 1 18m
configmap/vc-coredns-my-vcluster 1 19m
NAME CLASS HOSTS ADDRESS PORTS AGE
ingress.networking.k8s.io/dummyhttp-x-default-x-my-vcluster traefik vcluster-dummyhttp.lo.mixinet.net 172.20.0.2,172.20.0.3,172.20.0.4 80 45sService, Pod, Configmap and Ingress objects, but there is no copy of the
Deployment or ReplicaSet.
dummyhttpdTo use the hosts sealed secrets controller with the virtual cluster we will create the my-vcluster-default namespace
and add there the sealed secrets we want to have available as secrets on the default namespace of the virtual cluster:
kubectl create namespace my-vcluster-default
echo -n "Vcluster Boo" kubectl create secret generic "dummyhttp-secret" \
--namespace "my-vcluster-default" --dry-run=client \
--from-file=SECRET_VAR=/dev/stdin -o yaml >dummyhttp-secret.yaml
kubeseal -f dummyhttp-secret.yaml -w dummyhttp-sealed-secret.yaml
kubectl apply -f dummyhttp-sealed-secret.yaml
rm -f dummyhttp-secret.yaml dummyhttp-sealed-secret.yaml kubectl get sealedsecrets.bitnami.com,secrets -n my-vcluster-default
NAME STATUS SYNCED AGE
sealedsecret.bitnami.com/dummyhttp-secret True 34s
NAME TYPE DATA AGE
secret/dummyhttp-secret Opaque 1 34s vkubectl get secrets
NAME TYPE DATA AGE
dummyhttp-secret Opaque 1 34s
vkubectl get secret/dummyhttp-secret --template=" .data.SECRET_VAR " \
base64 -d
Vcluster Boocurl command has not changed because, although we have the reloader controller deployed on the
host cluster, it does not see the Deployment object of the virtual one and the pods are not touched:
curl -s https://vcluster-dummyhttp.lo.mixinet.net:8443/ jq -cM .
"c": "Vcluster Test Value","s": "" reloader applicationTo make reloader work on the virtual cluster we just need to install it as we did on the host using the following
kustomization.yaml file:
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: kube-system
resources:
- github.com/stakater/Reloader/deployments/kubernetes/?ref=v1.4.2
patches:
# Add flags to reload workloads when ConfigMaps or Secrets are created or deleted
- target:
kind: Deployment
name: reloader-reloader
patch: -
- op: add
path: /spec/template/spec/containers/0/args
value:
- '--reload-on-create=true'
- '--reload-on-delete=true'
- '--reload-strategy=annotations'kustomize and vkubectl:
kustomize build . vkubectl apply -f -
serviceaccount/reloader-reloader created
clusterrole.rbac.authorization.k8s.io/reloader-reloader-role created
clusterrolebinding.rbac.authorization.k8s.io/reloader-reloader-role-binding created
deployment.apps/reloader-reloader createdDeployment are not updated, but
we can force things removing the secret on the host system; after we do that the secret is re-created from the sealed
version and copied to the virtual cluster where the reloader controller updates the pod and the curl command shows the
new output:
kubectl delete -n my-vcluster-default secrets dummyhttp-secret
secret "dummyhttp-secret" deleted
sleep 2
vkubectl get pods
NAME READY STATUS RESTARTS AGE
dummyhttp-78bf5fb885-fmsvs 1/1 Terminating 0 6m33s
dummyhttp-c68684bbf-nx8f9 1/1 Running 0 6s
curl -s https://vcluster-dummyhttp.lo.mixinet.net:8443/ jq -cM .
"c":"Vcluster Test Value","s":"Vcluster Boo" echo -n "New secret" kubectl create secret generic "dummyhttp-secret" \
--namespace "my-vcluster-default" --dry-run=client \
--from-file=SECRET_VAR=/dev/stdin -o yaml >dummyhttp-secret.yaml
kubeseal -f dummyhttp-secret.yaml -w dummyhttp-sealed-secret.yaml
kubectl apply -f dummyhttp-sealed-secret.yaml
rm -f dummyhttp-secret.yaml dummyhttp-sealed-secret.yaml
curl -s https://vcluster-dummyhttp.lo.mixinet.net:8443/ jq -cM .
"c":"Vcluster Test Value","s":"New secret" kubectl:
kubectl get pods,statefulsets -n my-vcluster
NAME READY STATUS RESTARTS AGE
pod/coredns-bbb5b66cc-snwpn-x-kube-system-x-my-vcluster 1/1 Running 0 127m
pod/dummyhttp-587c7855d7-pt9b8-x-default-x-my-vcluster 1/1 Running 0 4m39s
pod/my-vcluster-0 1/1 Running 0 128m
pod/reloader-reloader-7f56c54d75-544gd-x-kube-system-x-my-vcluster 1/1 Running 0 60m
NAME READY AGE
statefulset.apps/my-vcluster 1/1 128mPods are gone because
the statefulSet is scaled down to 0 (note that other resources like volumes are not removed, but all the objects that
have to be scheduled and consume CPU cycles are not running, which can translate in a lot of savings when running on
clusters from cloud platforms or, in a local cluster like the one we are using, frees resources like CPU and memory that
now can be used for other things):
vcluster pause my-vcluster
11:20:47 info Scale down statefulSet my-vcluster/my-vcluster...
11:20:48 done Successfully paused vcluster my-vcluster/my-vcluster
kubectl get pods,statefulsets -n my-vcluster
NAME READY AGE
statefulset.apps/my-vcluster 0/0 130mcurl command fails:
curl -s https://vcluster-dummyhttp.localhost.mixinet.net:8443
404 page not found404 because there is no pod behind the service):
kubectl get ingress -n my-vcluster
NAME CLASS HOSTS ADDRESS PORTS AGE
dummyhttp-x-default-x-my-vcluster traefik vcluster-dummyhttp.lo.mixinet.net 172.20.0.2,172.20.0.3,172.20.0.4 80 120mvcluster API; the error shown by kubectl is related
to the TLS certificate because the 404 page uses the wildcard certificate instead of the self signed one:
vkubectl get pods
Unable to connect to the server: tls: failed to verify certificate: x509: certificate signed by unknown authority
curl -s https://my-vcluster-api.lo.mixinet.net:8443/api/v1/
404 page not found
curl -v -s https://my-vcluster-api.lo.mixinet.net:8443/api/v1/ 2>&1 grep subject
* subject: CN=lo.mixinet.net
* subjectAltName: host "my-vcluster-api.lo.mixinet.net" matched cert's "*.lo.mixinet.net"resume command:
vcluster resume my-vcluster
12:03:14 done Successfully resumed vcluster my-vcluster in namespace my-vclusterdelete command:
vcluster delete my-vcluster
12:09:18 info Delete vcluster my-vcluster...
12:09:18 done Successfully deleted virtual cluster my-vcluster in namespace my-vcluster
12:09:18 done Successfully deleted virtual cluster namespace my-vcluster
12:09:18 info Waiting for virtual cluster to be deleted...
12:09:50 done Virtual Cluster is deletedmy-vcluster-default namespace because it was created by us.
If we delete the namespace all the secrets and sealed secrets on it are also removed:
kubectl delete namespace my-vcluster-default
namespace "my-vcluster-default" deletedConfigMap used by the
argocd-server (the resource was updated but the application Pod was not because there was no change on the
argocd-server deployment); our original fix was to kill the pod manually, but the manual operation is something we
want to avoid.
The proposed solution to this kind of issues on the
helm documentation is to add
annotations to the Deployments with values that are a hash of the ConfigMaps or Secrets used by them, this way if
a file is updated the annotation is also updated and when the Deployment changes are applied a roll out of the pods is
triggered.
On this post we will install a couple of controllers and an application to show how we can handle Secrets with
argocd and solve the issue with updates on ConfigMaps and Secrets, to do it we will execute the following tasks:
ConfigMaps and Secrets and does rolling upgrades on the Pods that use them from Deployment,
StatefulSet, DaemonSet or DeploymentConfig objects when they are updated (by default we have to add some
annotations to the objects to make things work).ConfigMaps and Secrets and test that the Reloader controller does its
job when we add or update a ConfigMap.test project for argocd-autopilotAs we did our installation using argocd-autopilot we will use its structure to manage the applications.
The first thing to do is to create a project (we will name it test) as follows:
argocd-autopilot project create test
INFO cloning git repository: https://forgejo.mixinet.net/blogops/argocd.git
Enumerating objects: 18, done.
Counting objects: 100% (18/18), done.
Compressing objects: 100% (16/16), done.
Total 18 (delta 1), reused 0 (delta 0), pack-reused 0
INFO using revision: "", installation path: "/"
INFO pushing new project manifest to repo
INFO project created: 'test'test project is available we will use it on our argocd-autopilot invocations when creating
applications.reloader controllerTo add the reloader application to the test project as a kustomize application and deploy it on the tools
namespace with argocd-autopilot we do the following:
argocd-autopilot app create reloader \
--app 'github.com/stakater/Reloader/deployments/kubernetes/?ref=v1.4.2' \
--project test --type kustomize --dest-namespace tools
INFO cloning git repository: https://forgejo.mixinet.net/blogops/argocd.git
Enumerating objects: 19, done.
Counting objects: 100% (19/19), done.
Compressing objects: 100% (18/18), done.
Total 19 (delta 2), reused 0 (delta 0), pack-reused 0
INFO using revision: "", installation path: "/"
INFO created 'application namespace' file at '/bootstrap/cluster-resources/in-cluster/tools-ns.yaml'
INFO committing changes to gitops repo...
INFO installed application: reloaderargocd repository:
tools namespace:bootstrap/cluster-resources/in-cluster/tools-ns.yaml
apiVersion: v1
kind: Namespace
metadata:
annotations:
argocd.argoproj.io/sync-options: Prune=false
creationTimestamp: null
name: tools
spec:
status: reloader base application from the upstream repository:apps/reloader/base/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- github.com/stakater/Reloader/deployments/kubernetes/?ref=v1.4.2kustomization.yaml file for the test project (by default it includes the same configuration used on the
base definition, but we could make other changes if needed):apps/reloader/overlays/test/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: tools
resources:
- ../../baseconfig.json file used to define the application on argocd for the test project (it points to the folder
that includes the previous kustomization.yaml file):apps/reloader/overlays/test/config.json
"appName": "reloader",
"userGivenName": "reloader",
"destNamespace": "tools",
"destServer": "https://kubernetes.default.svc",
"srcPath": "apps/reloader/overlays/test",
"srcRepoURL": "https://forgejo.mixinet.net/blogops/argocd.git",
"srcTargetRevision": "",
"labels": null,
"annotations": null
argocd command line application:
argocd app get argocd/test-reloader -o tree
Name: argocd/test-reloader
Project: test
Server: https://kubernetes.default.svc
Namespace: tools
URL: https://argocd.lo.mixinet.net:8443/applications/test-reloader
Source:
- Repo: https://forgejo.mixinet.net/blogops/argocd.git
Target:
Path: apps/reloader/overlays/test
SyncWindow: Sync Allowed
Sync Policy: Automated (Prune)
Sync Status: Synced to (2893b56)
Health Status: Healthy
KIND/NAME STATUS HEALTH MESSAGE
ClusterRole/reloader-reloader-role Synced
ClusterRoleBinding/reloader-reloader-role-binding Synced
ServiceAccount/reloader-reloader Synced serviceaccount/reloader-reloader created
Deployment/reloader-reloader Synced Healthy deployment.apps/reloader-reloader created
ReplicaSet/reloader-reloader-5b6dcc7b6f Healthy
Pod/reloader-reloader-5b6dcc7b6f-vwjcx Healthyreloader serverThe runtime configuration flags for the reloader server are described on the project
README.md
file, in our case we want to adjust three values:
ConfigMap or Secret is created,ConfigMap or Secret is deleted,annotations strategy for reloads, as it is the recommended mode of operation when using argocd.apps/reloader/overlays/test/kustomization.yaml file to patch the pod container template, the
text added is the following:
patches:
# Add flags to reload workloads when ConfigMaps or Secrets are created or deleted
- target:
kind: Deployment
name: reloader-reloader
patch: -
- op: add
path: /spec/template/spec/containers/0/args
value:
- '--reload-on-create=true'
- '--reload-on-delete=true'
- '--reload-strategy=annotations'dummyhttp applicationTo do a quick test we are going to deploy the dummyhttp web server using an
image generated using the following Dockerfile:
# Image to run the dummyhttp application <https://github.com/svenstaro/dummyhttp>
# This arg could be passed by the container build command (used with mirrors)
ARG OCI_REGISTRY_PREFIX
# Latest tested version of alpine
FROM $ OCI_REGISTRY_PREFIX alpine:3.21.3
# Tool versions
ARG DUMMYHTTP_VERS=1.1.1
# Download binary
RUN ARCH="$(apk --print-arch)" && \
VERS="$DUMMYHTTP_VERS" && \
URL="https://github.com/svenstaro/dummyhttp/releases/download/v$VERS/dummyhttp-$VERS-$ARCH-unknown-linux-musl" && \
wget "$URL" -O "/tmp/dummyhttp" && \
install /tmp/dummyhttp /usr/local/bin && \
rm -f /tmp/dummyhttp
# Set the entrypoint to /usr/local/bin/dummyhttp
ENTRYPOINT [ "/usr/local/bin/dummyhttp" ]kustomize base application is available on a monorepo that contains the following files:
Deployment definition that uses the previous image but uses /bin/sh -c as its entrypoint (command in the
k8s Pod terminology) and passes as its argument a string that runs the eval command to be able to expand
environment variables passed to the pod (the definition includes two optional variables, one taken from a
ConfigMap and another one from a Secret):apiVersion: apps/v1
kind: Deployment
metadata:
name: dummyhttp
labels:
app: dummyhttp
spec:
selector:
matchLabels:
app: dummyhttp
template:
metadata:
labels:
app: dummyhttp
spec:
containers:
- name: dummyhttp
image: forgejo.mixinet.net/oci/dummyhttp:1.0.0
command: [ "/bin/sh", "-c" ]
args:
- 'eval dummyhttp -b \" \\\"c\\\": \\\"$CM_VAR\\\", \\\"s\\\": \\\"$SECRET_VAR\\\" \"'
ports:
- containerPort: 8080
env:
- name: CM_VAR
valueFrom:
configMapKeyRef:
name: dummyhttp-configmap
key: CM_VAR
optional: true
- name: SECRET_VAR
valueFrom:
secretKeyRef:
name: dummyhttp-secret
key: SECRET_VAR
optional: trueService that publishes the previous Deployment (the only relevant thing to mention is that the web server uses
the port 8080 by default):apiVersion: v1
kind: Service
metadata:
name: dummyhttp
spec:
selector:
app: dummyhttp
ports:
- name: http
port: 80
targetPort: 8080Ingress definition to allow access to the application from the outside:apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: dummyhttp
annotations:
traefik.ingress.kubernetes.io/router.tls: "true"
spec:
rules:
- host: dummyhttp.localhost.mixinet.net
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: dummyhttp
port:
number: 80kustomization.yaml file that includes the previous files:apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- deployment.yaml
- service.yaml
- ingress.yamldummyhttp application from argocdWe could create the dummyhttp application using the argocd-autopilot command as we ve done on the reloader case,
but we are going to do it manually to show how simple it is.
First we ve created the apps/dummyhttp/base/kustomization.yaml file to include the application from the previous
repository:
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- https://forgejo.mixinet.net/blogops/argocd-applications.git//dummyhttp/?ref=dummyhttp-v1.0.0apps/dummyhttp/overlays/test/kustomization.yaml file to include the previous file:
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../../baseapps/dummyhttp/overlays/test/config.json file to configure the application as the
ApplicationSet defined by argocd-autopilot expects:
"appName": "dummyhttp",
"userGivenName": "dummyhttp",
"destNamespace": "default",
"destServer": "https://kubernetes.default.svc",
"srcPath": "apps/dummyhttp/overlays/test",
"srcRepoURL": "https://forgejo.mixinet.net/blogops/argocd.git",
"srcTargetRevision": "",
"labels": null,
"annotations": null
argocd deploys the application; we can check that
things are working using curl:
curl -s https://dummyhttp.lo.mixinet.net:8443/ jq -M .
"c": "",
"s": ""
apps/dummyhttp/overlays/test/kustomization.yaml file:
reloader (one to enable it and another one to set the roll out strategy to restart to
avoid touching the deployments, as that can generate issues with argocd).hostname (not really needed, but something quite reasonable for a specific project).--- a/apps/dummyhttp/overlays/test/kustomization.yaml
+++ b/apps/dummyhttp/overlays/test/kustomization.yaml
@@ -2,3 +2,22 @@ apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../../base
+patches:
+# Add reloader annotations
+- target:
+ kind: Deployment
+ name: dummyhttp
+ patch: -
+ - op: add
+ path: /metadata/annotations
+ value:
+ reloader.stakater.com/auto: "true"
+ reloader.stakater.com/rollout-strategy: "restart"
+# Change the ingress host name
+- target:
+ kind: Ingress
+ name: dummyhttp
+ patch: -
+ - op: replace
+ path: /spec/rules/0/host
+ value: test-dummyhttp.lo.mixinet.netargocd cli to check the status of the application:
argocd app get argocd/test-dummyhttp -o tree
Name: argocd/test-dummyhttp
Project: test
Server: https://kubernetes.default.svc
Namespace: default
URL: https://argocd.lo.mixinet.net:8443/applications/test-dummyhttp
Source:
- Repo: https://forgejo.mixinet.net/blogops/argocd.git
Target:
Path: apps/dummyhttp/overlays/test
SyncWindow: Sync Allowed
Sync Policy: Automated (Prune)
Sync Status: Synced to (fbc6031)
Health Status: Healthy
KIND/NAME STATUS HEALTH MESSAGE
Deployment/dummyhttp Synced Healthy deployment.apps/dummyhttp configured
ReplicaSet/dummyhttp-55569589bc Healthy
Pod/dummyhttp-55569589bc-qhnfk Healthy
Ingress/dummyhttp Synced Healthy ingress.networking.k8s.io/dummyhttp configured
Service/dummyhttp Synced Healthy service/dummyhttp unchanged
Endpoints/dummyhttp
EndpointSlice/dummyhttp-x57blDeployment and Ingress where updated, but the Service is unchanged.
To validate that the ingress is using the new hostname we can use curl:
curl -s https://dummyhttp.lo.mixinet.net:8443/
404 page not found
curl -s https://test-dummyhttp.lo.mixinet.net:8443/
"c": "", "s": "" ConfigMapNow that the system is adjusted to reload the application when the ConfigMap or Secret is created, deleted or
updated we are ready to add one file and see how the system reacts.
We modify the apps/dummyhttp/overlays/test/kustomization.yaml file to create the ConfigMap using the
configMapGenerator as follows:
--- a/apps/dummyhttp/overlays/test/kustomization.yaml
+++ b/apps/dummyhttp/overlays/test/kustomization.yaml
@@ -2,6 +2,14 @@ apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../../base
+# Add the config map
+configMapGenerator:
+- name: dummyhttp-configmap
+ literals:
+ - CM_VAR="Default Test Value"
+ behavior: create
+ options:
+ disableNameSuffixHash: true
patches:
# Add reloader annotations
- target:ConfigMap is available, the pod has been deleted and
started again and the curl output includes the new value:
kubectl get configmaps,pods
NAME READY STATUS RESTARTS AGE
configmap/dummyhttp-configmap 1 11s
configmap/kube-root-ca.crt 1 4d7h
NAME DATA AGE
pod/dummyhttp-779c96c44b-pjq4d 1/1 Running 0 11s
pod/dummyhttp-fc964557f-jvpkx 1/1 Terminating 0 2m42s
curl -s https://test-dummyhttp.lo.mixinet.net:8443 jq -M .
"c": "Default Test Value",
"s": ""
helm with argocd-autopilotRight now there is no direct support in argocd-autopilot to manage applications using helm (see the issue
#38 on the project), but we want to use a chart in our
next example.
There are multiple ways to add the support, but the simplest one that allows us to keep using argocd-autopilot is to
use kustomize applications that call helm as described
here.
The only thing needed before being able to use the approach is to add the kustomize.buildOptions flag to the
argocd-cm on the bootstrap/argo-cd/kustomization.yaml file, its contents now are follows:
apiVersion: kustomize.config.k8s.io/v1beta1
configMapGenerator:
- behavior: merge
literals:
# Enable helm usage from kustomize (see https://github.com/argoproj/argo-cd/issues/2789#issuecomment-960271294)
- kustomize.buildOptions="--enable-helm"
-
repository.credentials=- passwordSecret:
key: git_token
name: autopilot-secret
url: https://forgejo.mixinet.net/
usernameSecret:
key: git_username
name: autopilot-secret
name: argocd-cm
# Disable TLS for the Argo Server (see https://argo-cd.readthedocs.io/en/stable/operator-manual/ingress/#traefik-v30)
- behavior: merge
literals:
- "server.insecure=true"
name: argocd-cmd-params-cm
kind: Kustomization
namespace: argocd
resources:
- github.com/argoproj-labs/argocd-autopilot/manifests/base?ref=v0.4.19
- ingress_route.yamlsealed-secrets controllerTo manage secrets in our cluster we are going to use the
sealed-secrets controller and to install it we are going to use its
chart.
As we mentioned on the previous section, the idea is to create a kustomize application and use that to deploy the
chart, but we are going to create the files manually, as we are not going import the base kustomization files from a
remote repository.
As there is no clear way to override helm Chart values using
overlays we are going to use a generator to create the helm configuration from an external resource and include it
from our overlays (the idea has been taken from this repository,
which was referenced from a comment
on the kustomize issue #38 mentioned earlier).
sealed-secrets applicationWe have created the following files and folders manually:
apps/sealed-secrets/
helm
chart.yaml
kustomization.yaml
overlays
test
config.json
kustomization.yaml
values.yamlhelm folder contains the generator template that will be included from our overlays.
The kustomization.yaml includes the chart.yaml as a resource:
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- chart.yamlchart.yaml file defines the HelmChartInflationGenerator:
apiVersion: builtin
kind: HelmChartInflationGenerator
metadata:
name: sealed-secrets
releaseName: sealed-secrets
name: sealed-secrets
namespace: kube-system
repo: https://bitnami-labs.github.io/sealed-secrets
version: 2.17.2
includeCRDs: true
# Add common values to all argo-cd projects inline
valuesInline:
fullnameOverride: sealed-secrets-controller
# Load a values.yaml file from the same directory that uses this generator
valuesFile: values.yamlnamespace to kube-system and adds the fullnameOverride on the
valuesInline key because we want to use those settings on all the projects (they are the values expected by the
kubeseal command line application, so we adjust them to avoid the need to add additional parameters to it).
We adjust global values as inline to be able to use a the valuesFile from our overlays; as we are using a generator
the path is relative to the folder that contains the kustomization.yaml file that calls it, in our case we will need
to have a values.yaml file on each overlay folder (if we don t want to overwrite any values for a project we can
create an empty file, but it has to exist).
Finally, our overlay folder contains three files, a kustomization.yaml file that includes the generator from the
helm folder, the values.yaml file needed by the chart and the config.json file used by argocd-autopilot to
install the application.
The kustomization.yaml file contents are:
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
# Uncomment if you want to add additional resources using kustomize
#resources:
#- ../../base
generators:
- ../../helmvalues.yaml file enables the ingress for the application and adjusts its hostname:
ingress:
enabled: true
hostname: test-sealed-secrets.lo.mixinet.netconfig.json file is similar to the ones used with the other applications we have installed:
"appName": "sealed-secrets",
"userGivenName": "sealed-secrets",
"destNamespace": "kube-system",
"destServer": "https://kubernetes.default.svc",
"srcPath": "apps/sealed-secrets/overlays/test",
"srcRepoURL": "https://forgejo.mixinet.net/blogops/argocd.git",
"srcTargetRevision": "",
"labels": null,
"annotations": null
sealed-secrets application is installed in our cluster, we can check it using
curl to get the public certificate used by it:
curl -s https://test-sealed-secrets.lo.mixinet.net:8443/v1/cert.pem
-----BEGIN CERTIFICATE-----
[...]
-----END CERTIFICATE-----dummyhttp-secretTo create sealed secrets we need to install the kubeseal tool:
arkade get kubesealdummyhttp-secret that contains some value on the SECRET_VAR key (the easiest
way for doing it is to use kubectl):
echo -n "Boo" kubectl create secret generic dummyhttp-secret \
--dry-run=client --from-file=SECRET_VAR=/dev/stdin -o yaml \
>/tmp/dummyhttp-secret.yamlapiVersion: v1
data:
SECRET_VAR: Qm9v
kind: Secret
metadata:
creationTimestamp: null
name: dummyhttp-secretkubeseal tool we can do the following:
kubeseal -f /tmp/dummyhttp-secret.yaml -w /tmp/dummyhttp-sealed-secret.yamlkube-system namespace and set the controller name to sealed-secrets-controller as the tool expects.
If we need to create the secrets without credentials we can connect to the ingress address we added to retrieve the
public key:
kubeseal -f /tmp/dummyhttp-secret.yaml -w /tmp/dummyhttp-sealed-secret.yaml \
--cert https://test-sealed-secrets.lo.mixinet.net:8443/v1/cert.pemapiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
creationTimestamp: null
name: dummyhttp-secret
namespace: default
spec:
encryptedData:
SECRET_VAR: [...]
template:
metadata:
creationTimestamp: null
name: dummyhttp-secret
namespace: defaultargocd application),
but before doing that we are going to check the output of our dummyhttp service and get the list of Secrets and
SealedSecrets in the default namespace:
curl -s https://test-dummyhttp.lo.mixinet.net:8443 jq -M .
"c": "Default Test Value",
"s": ""
kubectl get sealedsecrets,secrets
No resources found in default namespace.SealedSecret to the dummyapp copying the file and adding it to the kustomization.yaml file:
--- a/apps/dummyhttp/overlays/test/kustomization.yaml
+++ b/apps/dummyhttp/overlays/test/kustomization.yaml
@@ -2,6 +2,7 @@ apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../../base
+- dummyhttp-sealed-secret.yaml
# Create the config map value
configMapGenerator:
- name: dummyhttp-configmapSealedSecret and the controller generates the Secret:
kubectl apply -f /tmp/dummyhttp-sealed-secret.yaml
sealedsecret.bitnami.com/dummyhttp-secret created
kubectl get sealedsecrets,secrets
NAME STATUS SYNCED AGE
sealedsecret.bitnami.com/dummyhttp-secret True 3s
NAME TYPE DATA AGE
secret/dummyhttp-secret Opaque 1 3s curl -s https://test-dummyhttp.lo.mixinet.net:8443 jq -M .
"c": "Default Test Value",
"s": "Boo"
sealed-secrets in production clustersIf you plan to use sealed-secrets look into its
documentation to understand how it manages the
private keys, how to backup things and keep in mind that, as the documentation
explains, you can rotate
your sealed version of the secrets, but that doesn t change the actual secrets.
If you want to rotate your secrets you have to update them and commit the sealed version of the updates (as the
controller also rotates the encryption keys your new sealed version will also be using a newer key, so you will be doing
both things at the same time).argocd-autopilot model, including the use of helm
charts inside kustomize applications and how to install and use the sealed-secrets controller.
It has been interesting and I ve learnt a lot about argocd in the process, but I believe that if I ever want to use it
in production I will also review the native helm support in argocd using a separate repository to manage the
applications, at least to be able to compare it to the model explained here. For a long time I ve been wanting to try GitOps tools, but I haven t had the chance to try them for real on the projects
I was working on.
As now I have some spare time I ve decided I m going to play a little with Argo CD,
Flux and Kluctl to test them and be able to use one of them in a real project
in the future if it looks appropriate.
On this post I will use Argo-CD Autopilot to install argocd on a
k3d local cluster installed using OpenTofu to test the autopilot approach of
managing argocd and test the tool (as it manages argocd using a git repository it can be used to test argocd as
well).
arkadeRecently I ve been using the arkade tool to install kubernetes related
applications on Linux servers and containers, I usually get the applications with it and install them on the
/usr/local/bin folder.
For this post I ve created a simple script that checks if the tools I ll be using are available and installs them on the
$HOME/.arkade/bin folder if missing (I m assuming that docker is already available, as it is not installable with
arkade):
#!/bin/sh
# TOOLS LIST
ARKADE_APPS="argocd argocd-autopilot k3d kubectl sops tofu"
# Add the arkade binary directory to the path if missing
case ":$ PATH :" in
*:"$ HOME /.arkade/bin":*) ;;
*) export PATH="$ PATH :$ HOME /.arkade/bin" ;;
esac
# Install or update arkade
if command -v arkade >/dev/null; then
echo "Trying to update the arkade application"
sudo arkade update
else
echo "Installing the arkade application"
curl -sLS https://get.arkade.dev sudo sh
fi
echo ""
echo "Installing tools with arkade"
echo ""
for app in $ARKADE_APPS; do
app_path="$(command -v $app)" true
if [ "$app_path" ]; then
echo "The application '$app' already available on '$app_path'"
else
arkade get "$app"
fi
done
cat <<EOF
Add the ~/.arkade/bin directory to your PATH if tools have been installed there
EOFopentofuAlthough using k3d directly will be a good choice for the creation of the cluster, I m using tofu to do it because
that will probably be the tool used to do it if we were working with Cloud Platforms like AWS or Google.
The main.tf file is as follows:
terraform
required_providers
k3d =
source = "moio/k3d"
version = "0.0.12"
sops =
source = "carlpett/sops"
version = "1.2.0"
data "sops_file" "secrets"
source_file = "secrets.yaml"
resource "k3d_cluster" "argocd_cluster"
name = "argocd"
servers = 1
agents = 2
image = "rancher/k3s:v1.31.5-k3s1"
network = "argocd"
token = data.sops_file.secrets.data["token"]
port
host_port = 8443
container_port = 443
node_filters = [
"loadbalancer",
]
k3d
disable_load_balancer = false
disable_image_volume = false
kubeconfig
update_default_kubeconfig = true
switch_current_context = true
runtime
gpu_request = "all"
k3d configuration is quite simple, as I plan to use the default traefik ingress controller with TLS I publish
the 443 port on the hosts 8443 port, I ll explain how I add a valid certificate on the next step.
I ve prepared the following script to initialize and apply the changes:
#!/bin/sh
set -e
# VARIABLES
# Default token for the argocd cluster
K3D_CLUSTER_TOKEN="argocdToken"
# Relative PATH to install the k3d cluster using terr-iaform
K3D_TF_RELPATH="k3d-tf"
# Secrets yaml file
SECRETS_YAML="secrets.yaml"
# Relative PATH to the workdir from the script directory
WORK_DIR_RELPATH=".."
# Compute WORKDIR
SCRIPT="$(readlink -f "$0")"
SCRIPT_DIR="$(dirname "$SCRIPT")"
WORK_DIR="$(readlink -f "$SCRIPT_DIR/$WORK_DIR_RELPATH")"
# Update the PATH to add the arkade bin directory
# Add the arkade binary directory to the path if missing
case ":$ PATH :" in
*:"$ HOME /.arkade/bin":*) ;;
*) export PATH="$ PATH :$ HOME /.arkade/bin" ;;
esac
# Go to the k3d-tf dir
cd "$WORK_DIR/$K3D_TF_RELPATH" exit 1
# Create secrets.yaml file and encode it with sops if missing
if [ ! -f "$SECRETS_YAML" ]; then
echo "token: $K3D_CLUSTER_TOKEN" >"$SECRETS_YAML"
sops encrypt -i "$SECRETS_YAML"
fi
# Initialize terraform
tofu init
# Apply the configuration
tofu applyk3d ingressAs an optional step, after creating the k3d cluster I m going to add a default wildcard certificate for the traefik
ingress server to be able to use everything with HTTPS without certificate issues.
As I manage my own DNS domain I ve created the lo.mixinet.net and *.lo.mixinet.net DNS entries on my public and
private DNS servers (both return 127.0.0.1 and ::1) and I ve created a TLS certificate for both entries using
Let s Encrypt with Certbot.
The certificate is updated automatically on one of my servers and when I need it I copy the contents of the
fullchain.pem and privkey.pem files from the /etc/letsencrypt/live/lo.mixinet.net server directory to the local
files lo.mixinet.net.crt and lo.mixinet.net.key.
After copying the files I run the following file to install or update the certificate and configure it as the default
for traefik:
#!/bin/sh
# Script to update the
secret="lo-mixinet-net-ingress-cert"
cert="$ 1:-lo.mixinet.net.crt "
key="$ 2:-lo.mixinet.net.key "
if [ -f "$cert" ] && [ -f "$key" ]; then
kubectl -n kube-system create secret tls $secret \
--key=$key \
--cert=$cert \
--dry-run=client --save-config -o yaml kubectl apply -f -
kubectl apply -f - << EOF
apiVersion: traefik.containo.us/v1alpha1
kind: TLSStore
metadata:
name: default
namespace: kube-system
spec:
defaultCertificate:
secretName: $secret
EOF
else
cat <<EOF
To add or update the traefik TLS certificate the following files are needed:
- cert: '$cert'
- key: '$key'
Note: you can pass the paths as arguments to this script.
EOF
fiargocd with argocd-autopilotautopilotI ll be using a project on my forgejo instance to manage argocd, the repository I ve created is on the URL
https://forgejo.mixinet.net/blogops/argocd and I ve created a private user named argocd that only has write access to
that repository.
Logging as the argocd user on forgejo I ve created a token with permission to read and write repositories that I ve
saved on my pass password store on the mixinet.net/argocd@forgejo/repository-write
entry.GIT_REPO and GIT_TOKEN values):
#!/bin/sh
set -e
# VARIABLES
# Relative PATH to the workdir from the script directory
WORK_DIR_RELPATH=".."
# Compute WORKDIR
SCRIPT="$(readlink -f "$0")"
SCRIPT_DIR="$(dirname "$SCRIPT")"
WORK_DIR="$(readlink -f "$SCRIPT_DIR/$WORK_DIR_RELPATH")"
# Update the PATH to add the arkade bin directory
# Add the arkade binary directory to the path if missing
case ":$ PATH :" in
*:"$ HOME /.arkade/bin":*) ;;
*) export PATH="$ PATH :$ HOME /.arkade/bin" ;;
esac
# Go to the working directory
cd "$WORK_DIR" exit 1
# Set GIT variables
if [ -z "$GIT_REPO" ]; then
export GIT_REPO="https://forgejo.mixinet.net/blogops/argocd.git"
fi
if [ -z "$GIT_TOKEN" ]; then
GIT_TOKEN="$(pass mixinet.net/argocd@forgejo/repository-write)"
export GIT_TOKEN
fi
argocd-autopilot repo bootstrap --provider gitea bin/argocd-bootstrap.sh
INFO cloning repo: https://forgejo.mixinet.net/blogops/argocd.git
INFO empty repository, initializing a new one with specified remote
INFO using revision: "", installation path: ""
INFO using context: "k3d-argocd", namespace: "argocd"
INFO applying bootstrap manifests to cluster...
namespace/argocd created
customresourcedefinition.apiextensions.k8s.io/applications.argoproj.io created
customresourcedefinition.apiextensions.k8s.io/applicationsets.argoproj.io created
customresourcedefinition.apiextensions.k8s.io/appprojects.argoproj.io created
serviceaccount/argocd-application-controller created
serviceaccount/argocd-applicationset-controller created
serviceaccount/argocd-dex-server created
serviceaccount/argocd-notifications-controller created
serviceaccount/argocd-redis created
serviceaccount/argocd-repo-server created
serviceaccount/argocd-server created
role.rbac.authorization.k8s.io/argocd-application-controller created
role.rbac.authorization.k8s.io/argocd-applicationset-controller created
role.rbac.authorization.k8s.io/argocd-dex-server created
role.rbac.authorization.k8s.io/argocd-notifications-controller created
role.rbac.authorization.k8s.io/argocd-redis created
role.rbac.authorization.k8s.io/argocd-server created
clusterrole.rbac.authorization.k8s.io/argocd-application-controller created
clusterrole.rbac.authorization.k8s.io/argocd-applicationset-controller created
clusterrole.rbac.authorization.k8s.io/argocd-server created
rolebinding.rbac.authorization.k8s.io/argocd-application-controller created
rolebinding.rbac.authorization.k8s.io/argocd-applicationset-controller created
rolebinding.rbac.authorization.k8s.io/argocd-dex-server created
rolebinding.rbac.authorization.k8s.io/argocd-notifications-controller created
rolebinding.rbac.authorization.k8s.io/argocd-redis created
rolebinding.rbac.authorization.k8s.io/argocd-server created
clusterrolebinding.rbac.authorization.k8s.io/argocd-application-controller created
clusterrolebinding.rbac.authorization.k8s.io/argocd-applicationset-controller created
clusterrolebinding.rbac.authorization.k8s.io/argocd-server created
configmap/argocd-cm created
configmap/argocd-cmd-params-cm created
configmap/argocd-gpg-keys-cm created
configmap/argocd-notifications-cm created
configmap/argocd-rbac-cm created
configmap/argocd-ssh-known-hosts-cm created
configmap/argocd-tls-certs-cm created
secret/argocd-notifications-secret created
secret/argocd-secret created
service/argocd-applicationset-controller created
service/argocd-dex-server created
service/argocd-metrics created
service/argocd-notifications-controller-metrics created
service/argocd-redis created
service/argocd-repo-server created
service/argocd-server created
service/argocd-server-metrics created
deployment.apps/argocd-applicationset-controller created
deployment.apps/argocd-dex-server created
deployment.apps/argocd-notifications-controller created
deployment.apps/argocd-redis created
deployment.apps/argocd-repo-server created
deployment.apps/argocd-server created
statefulset.apps/argocd-application-controller created
networkpolicy.networking.k8s.io/argocd-application-controller-network-policy created
networkpolicy.networking.k8s.io/argocd-applicationset-controller-network-policy created
networkpolicy.networking.k8s.io/argocd-dex-server-network-policy created
networkpolicy.networking.k8s.io/argocd-notifications-controller-network-policy created
networkpolicy.networking.k8s.io/argocd-redis-network-policy created
networkpolicy.networking.k8s.io/argocd-repo-server-network-policy created
networkpolicy.networking.k8s.io/argocd-server-network-policy created
secret/autopilot-secret created
INFO pushing bootstrap manifests to repo
INFO applying argo-cd bootstrap application
INFO pushing bootstrap manifests to repo
INFO applying argo-cd bootstrap application
application.argoproj.io/autopilot-bootstrap created
INFO running argocd login to initialize argocd config
Context 'autopilot' updated
INFO argocd initialized. password: XXXXXXX-XXXXXXXX
INFO run:
kubectl port-forward -n argocd svc/argocd-server 8080:80argocd installed and running, it can be checked using the port-forward and connecting to
https://localhost:8080/ (the certificate will be wrong, we are going to fix that in the next step).argocd installation in gitNow that we have the application deployed we can clone the argocd repository and edit the deployment to disable TLS
for the argocd server (we are going to use TLS termination with traefik and that needs the server running as insecure,
see the Argo CD documentation)
ssh clone ssh://git@forgejo.mixinet.net/blogops/argocd.git
cd argocd
edit bootstrap/argo-cd/kustomization.yaml
git commit -m 'Disable TLS for the argocd-server'kustomization.yaml file are the following:
--- a/bootstrap/argo-cd/kustomization.yaml
+++ b/bootstrap/argo-cd/kustomization.yaml
@@ -11,6 +11,11 @@ configMapGenerator:
key: git_username
name: autopilot-secret
name: argocd-cm
+ # Disable TLS for the Argo Server (see https://argo-cd.readthedocs.io/en/stable/operator-manual/ingress/#traefik-v30)
+- behavior: merge
+ literals:
+ - "server.insecure=true"
+ name: argocd-cmd-params-cm
kind: Kustomization
namespace: argocd
resources:argo-cd application manually to make sure they are applied:
argocd-cmd-params-cm ConfigMap to make sure everything is OK:
apiVersion: v1
data:
server.insecure: "true"
kind: ConfigMap
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration:
"apiVersion":"v1","data": "server.insecure":"true" ,"kind":"ConfigMap","metadata": "annotations": ,"labels": "app.kubernetes.io/instance":"argo-cd","app.kubernetes.io/name":"argocd-cmd-params-cm","app.kubernetes.io/part-of":"argocd" ,"name":"argocd-cmd-params-cm","namespace":"argocd"
creationTimestamp: "2025-04-27T17:31:54Z"
labels:
app.kubernetes.io/instance: argo-cd
app.kubernetes.io/name: argocd-cmd-params-cm
app.kubernetes.io/part-of: argocd
name: argocd-cmd-params-cm
namespace: argocd
resourceVersion: "16731"
uid: a460638f-1d82-47f6-982c-3017699d5f14ConfigMap we have to restart the argocd-server to read it again, to do it we delete the
server pods so they are re-created using the updated resource:
kubectl delete pods -n argocd -l app.kubernetes.io/name=argocd-serverport-forward command is killed automatically, if we run it again the connection to get to the
argocd-server has to be done using HTTP instead of HTTPS.
Instead of testing that we are going to add an ingress definition to be able to connect to the server using HTTPS and
GRPC against the address argocd.lo.mixinet.net using the wildcard TLS certificate we installed earlier.
To do it we to edit the bootstrap/argo-cd/kustomization.yaml file to add the ingress_route.yaml file to the
deployment:
--- a/bootstrap/argo-cd/kustomization.yaml
+++ b/bootstrap/argo-cd/kustomization.yaml
@@ -20,3 +20,4 @@ kind: Kustomization
namespace: argocd
resources:
- github.com/argoproj-labs/argocd-autopilot/manifests/base?ref=v0.4.19
+- ingress_route.yamlingress_route.yaml file contents are the following:
apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
name: argocd-server
namespace: argocd
spec:
entryPoints:
- websecure
routes:
- kind: Rule
match: Host( argocd.lo.mixinet.net )
priority: 10
services:
- name: argocd-server
port: 80
- kind: Rule
match: Host( argocd.lo.mixinet.net ) && Header( Content-Type , application/grpc )
priority: 11
services:
- name: argocd-server
port: 80
scheme: h2c
tls:
certResolver: default argocd --grpc-web login argocd.lo.mixinet.net:8443
Username: admin
Password:
'admin:login' logged in successfully
Context 'argocd.lo.mixinet.net:8443' updated
argocd app list -o name
argocd/argo-cd
argocd/autopilot-bootstrap
argocd/cluster-resources-in-cluster
argocd/rootriscv64 CPU architecture is useful for testing things. The HiFive Premier P550 seems to be a common hardware choice. The P550 is possible to purchase online. You also need a (mini-)ATX chassi, power supply (~500W is more than sufficient), PCI-to-M2 converter and a NVMe storage device. Total cost per machine was around $8k/ 8k for me. Assembly was simple: bolt everything, connect ATX power, connect cables for the front-panel, USB and and Audio. Be sure to toggle the physical power switch on the P550 before you close the box. Front-panel power button will start your machine. There is a P550 user manual available.
Below I will guide you to install the GitLab Runner on the pre-installed Ubuntu 24.04 that ships with the P550, and configure it to use Podman in root-less mode and without the --privileged flag, without any additional capabilities like SYS_ADMIN. Presumably you want to migrate to some other OS instead; hey Trisquel 13 riscv64 I m waiting for you! I wouldn t recommend using this machine for anything sensitive, there is an awful lot of non-free and/or vendor-specific software installed, and the hardware itself is young. I am not aware of any riscv64 hardware that can run a libre OS, all of them appear to require non-free blobs and usually a non-mainline kernel.
sudo -i and change the hostname:echo jas-p550-01 > /etc/hostnameapt-get update && apt-get dist-upgrade -u.apt-get install minicom
minicom -o -D /dev/ttyUSB3
#cmd: ifconfig
inet 192.168.0.2 netmask: 255.255.240.0
gatway 192.168.0.1
SOM_Mac0: 8c:00:00:00:00:00
SOM_Mac1: 8c:00:00:00:00:00
MCU_Mac: 8c:00:00:00:00:00
#cmd: setmac 0 CA:FE:42:17:23:00
The MAC setting will be valid after rebooting the carrier board!!!
MAC[0] addr set to CA:FE:42:17:23:00(ca:fe:42:17:23:0)
#cmd: setmac 1 CA:FE:42:17:23:01
The MAC setting will be valid after rebooting the carrier board!!!
MAC[1] addr set to CA:FE:42:17:23:01(ca:fe:42:17:23:1)
#cmd: setmac 2 CA:FE:42:17:23:02
The MAC setting will be valid after rebooting the carrier board!!!
MAC[2] addr set to CA:FE:42:17:23:02(ca:fe:42:17:23:2)
#cmd:
apt-get install openocd
wget https://raw.githubusercontent.com/sifiveinc/hifive-premier-p550-tools/refs/heads/master/mcu-firmware/stm32_openocd.cfg
echo 'acc115d283ff8533d6ae5226565478d0128923c8a479a768d806487378c5f6c3 stm32_openocd.cfg' sha256sum -c
openocd -f stm32_openocd.cfg &
telnet localhost 4444
...
echo 'ssh-ed25519 AAA...' > ~/.ssh/authorized_keys
sed -i 's;^#PasswordAuthentication.*;PasswordAuthentication no;' /etc/ssh/sshd_config
service ssh restart
parted /dev/nvme0n1 printNow with a reasonable setup ready, let s install the GitLab Runner. The following is adapted from gitlab-runner s official installation instructions documentation. The normal installation flow doesn t work because they don t publish
blkdiscard /dev/nvme0n1
parted /dev/nvme0n1 mklabel gpt
parted /dev/nvme0n1 mkpart jas-p550-nvm-02 ext2 1MiB 100% align-check optimal 1
parted /dev/nvme0n1 set 1 lvm on
partprobe /dev/nvme0n1
pvcreate /dev/nvme0n1p1
vgcreate vg0 /dev/nvme0n1p1
lvcreate -L 400G -n glr vg0
mkfs.ext4 -L glr /dev/mapper/vg0-glr
riscv64 apt repositories, so you will have to perform upgrades manually.
# wget https://s3.dualstack.us-east-1.amazonaws.com/gitlab-runner-downloads/latest/deb/gitlab-runner_riscv64.debRemember the NVMe device? Let s not forget to use it, to avoid wear and tear of the internal MMC root disk. Do this now before any files in
# wget https://s3.dualstack.us-east-1.amazonaws.com/gitlab-runner-downloads/latest/deb/gitlab-runner-helper-images.deb
wget https://gitlab-runner-downloads.s3.amazonaws.com/v17.11.0/deb/gitlab-runner_riscv64.deb
wget https://gitlab-runner-downloads.s3.amazonaws.com/v17.11.0/deb/gitlab-runner-helper-images.deb
echo '68a4c2a4b5988a5a5bae019c8b82b6e340376c1b2190228df657164c534bc3c3 gitlab-runner-helper-images.deb' sha256sum -c
echo 'ee37dc76d3c5b52e4ba35cf8703813f54f536f75cfc208387f5aa1686add7a8c gitlab-runner_riscv64.deb' sha256sum -c
dpkg -i gitlab-runner-helper-images.deb gitlab-runner_riscv64.deb
/home/gitlab-runner appears, or you have to move them manually.
gitlab-runner stopNext install
echo 'LABEL=glr /home/gitlab-runner ext4 defaults,noatime 0 1' >> /etc/fstab
systemctl daemon-reload
mount /home/gitlab-runner
gitlab-runner and configure it. Replace token glrt-REPLACEME below with the registration token you get from your GitLab project s Settings -> CI/CD -> Runners -> New project runner. I used the tags riscv64 and a runner description of the hostname.
gitlab-runner register --non-interactive --url https://gitlab.com --token glrt-REPLACEME --name $(hostname) --executor docker --docker-image debian:stableWe install and configure gitlab-runner to use podman, and to use non-root user.
apt-get install podmanYou need to run some commands as the
gitlab-runner stop
usermod --add-subuids 100000-165535 --add-subgids 100000-165535 gitlab-runner
gitlab-runner user, but unfortunately some interaction between sudo/su and pam_systemd makes this harder than it should be. So you have to setup SSH for the user and login via SSH to run the commands. Does anyone know of a better way to do this?
# on the p550:We modify
cp -a /root/.ssh/ /home/gitlab-runner/
chown -R gitlab-runner:gitlab-runner /home/gitlab-runner/.ssh/
# on your laptop:
ssh gitlab-runner@jas-p550-01
systemctl --user --now enable podman.socket
systemctl --user --now start podman.socket
loginctl enable-linger gitlab-runner gitlab-runner
systemctl status --user podman.socket
/etc/gitlab-runner/config.toml as follows, replace 997 with the user id shown by systemctl status above. See feature flags documentation for more documentation.
...Note that unlike the documentation I do not add the privileged = true parameter here. I will come back to this later. Restart the system to confirm that pushing a
[[runners]]
environment = ["FF_NETWORK_PER_BUILD=1", "FF_USE_FASTZIP=1"]
...
[runners.docker]
host = "unix:///run/user/997/podman/podman.sock"
.gitlab-ci.yml with a job that uses the riscv64 tag like the following works properly.
dump-env-details-riscv64:Your gitlab-runner should now be receiving jobs and running them in rootless podman. You may view the log using
stage: build
image: riscv64/debian:testing
tags: [ riscv64 ]
script:
- set
journalctl as follows:
journalctl --follow _SYSTEMD_UNIT=gitlab-runner.serviceTo stop the graphical environment and disable some unnecessary services, you can use:
systemctl set-default multi-user.targetAt this point, things were working fine and I was running many successful builds. Now starts the fun part with operational aspects! I had a problem when running buildah to build a new container from within a job, and noticed that aardvark-dns was crashing. You can use the Debian aardvark-dns binary instead.
systemctl disable openvpn cups cups-browsed sssd colord
wget http://ftp.de.debian.org/debian/pool/main/a/aardvark-dns/aardvark-dns_1.14.0-3_riscv64.debMy setup uses podman in rootless mode without passing the privileged parameter or any add-cap parameters to add non-default capabilities. This is sufficient for most builds. However if you try to create container using buildah from within a job, you may see errors like this:
echo 'df33117b6069ac84d3e97dba2c59ba53775207dbaa1b123c3f87b3f312d2f87a aardvark-dns_1.14.0-3_riscv64.deb' sha256sum -c
mkdir t
cd t
dpkg -x ../aardvark-dns_1.14.0-3_riscv64.deb .
mv /usr/lib/podman/aardvark-dns /usr/lib/podman/aardvark-dns.ubuntu
mv usr/lib/podman/aardvark-dns /usr/lib/podman/aardvark-dns.debian
Writing manifest to image destinationAccording to GitLab runner security considerations, you should not enable the privileged = true parameter, and the alternative appears to run Podman as root with privileged=false. Indeed setting privileged=true as in the following example solves the problem, and I suppose running podman as root would too.
Error: mounting new container: mounting build container "8bf1ec03d967eae87095906d8544f51309363ddf28c60462d16d73a0a7279ce1": creating overlay mount to /var/lib/containers/storage/overlay/23785e20a8bac468dbf028bf524274c91fbd70dae195a6cdb10241c345346e6f/merged, mount_data="lowerdir=/var/lib/containers/storage/overlay/l/I3TWYVYTRZ4KVYCT6FJKHR3WHW,upperdir=/var/lib/containers/storage/overlay/23785e20a8bac468dbf028bf524274c91fbd70dae195a6cdb10241c345346e6f/diff,workdir=/var/lib/containers/storage/overlay/23785e20a8bac468dbf028bf524274c91fbd70dae195a6cdb10241c345346e6f/work,volatile": using mount program /usr/bin/fuse-overlayfs: unknown argument ignored: lazytime
fuse: device not found, try 'modprobe fuse' first
fuse-overlayfs: cannot mount: No such file or directory
: exit status 1
[[runners]]Can we do better? After some experimentation, and reading open issues with suggested capabilities and configuration snippets, I ended up with the following configuration. It runs podman in rootless mode (as the
[runners.docker]
privileged = true
gitlab-runner user) without --privileged, but add the CAP_SYS_ADMIN capability and exposes the /dev/fuse device. Still, this is running as non-root user on the machine, so I think it is an improvement compared to using --privileged and also compared to running podman as root.
[[runners]]Still I worry about the security properties of such a setup, so I only enable these settings for a separately configured runner instance that I use when I need this docker-in-docker (oh, I meant buildah-in-podman) functionality. I found one article discussing Rootless Podman without the privileged flag that suggest isolation=chroot but I have yet to make this work. Suggestions for improvement are welcome. Happy Riscv64 Building! Update 2025-05-05: I was able to make it work without the SYS_ADMIN capability too, with a GitLab
[runners.docker]
privileged = false
cap_add = ["SYS_ADMIN"]
devices = ["/dev/fuse"]
/etc/gitlab-runner/config.toml like the following:
[[runners]]
[runners.docker]
privileged = false
devices = ["/dev/fuse"]--isolation chroot to Buildah like this:
buildah build --isolation chroot -t $CI_REGISTRY_IMAGE:name image/ppc64el platform too. Remaining loop-holes are escaping from the chroot into the non-root gitlab-runner user, and escalating that privilege to root. The /dev/fuse and sub-uid/gid may be privilege escalation vectors here, otherwise I believe you ve found a serious software security issue rather than a configuration mistake.
Back when I setup my home automation I ended up with one piece that used an external service: Amazon Alexa. I d rather not have done this, but voice control is extremely convenient, both for us, and guests. Since then Home Assistant has done a lot of work in developing the capability of a local voice assistant - 2023 was their Year of Voice. I ve had brief looks at this in the past, but never quite had the time to dig into setting it up, and was put off by the fact a lot of the setup instructions were just Download our prebuilt components . While I admire the efforts to get Home Assistant fully packaged for Debian I accept that s a tricky proposition, and settle for running it in a venv on a Debian stable container. Voice requires a lot more binary components, and I want to have voice satellites in more than one location, so I set about trying to understand a bit better what I was deploying, and actually building the binary bits myself.
This is the start of a write-up of that. I ll break it into a bunch of posts, trying to cover one bit in each, because otherwise this will get massive. Let s start with some requirements:
$ diff -u m5stack-atom-echo.yaml assistant.yaml
--- m5stack-atom-echo.yaml 2025-04-18 13:41:21.812766112 +0100
+++ assistant.yaml 2025-01-20 17:33:24.918585244 +0000
@@ -1,7 +1,7 @@
substitutions:
- name: m5stack-atom-echo
+ name: study-atom-echo
friendly_name: M5Stack Atom Echo
- micro_wake_word_model: okay_nabu # alexa, hey_jarvis, hey_mycroft are also supported
+ micro_wake_word_model: hey_jarvis # alexa, hey_jarvis, hey_mycroft are also supported
esphome:
name: $ name
@@ -16,15 +16,26 @@
version: 4.4.8
platform_version: 5.4.0
+# Enable logging
logger:
+
+# Enable Home Assistant API
api:
+ encryption:
+ key: "TGlrZVRoaXNJc1JlYWxseUl0Rm9vbGlzaFBlb3BsZSE="
ota:
- platform: esphome
- id: ota_esphome
+ password: "itsnotarealthing"
wifi:
+ ssid: "My Wifi Goes Here"
+ password: "AndThePasswordGoesHere"
+
+ # Enable fallback hotspot (captive portal) in case wifi connection fails
ap:
+ ssid: "Study-Atom-Echo Fallback Hotspot"
+ password: "ThisIsRandom"
captive_portal:
venv and get it to build + flash the image for you:
noodles@sevai:~$ python3 -m venv esphome-atom-echo
noodles@sevai:~$ . esphome-atom-echo/bin/activate
(esphome-atom-echo) noodles@sevai:~$ cd esphome-atom-echo/
(esphome-atom-echo) noodles@sevai:~/esphome-atom-echo$ pip install esphome==2024.12.4
Collecting esphome==2024.12.4
Using cached esphome-2024.12.4-py3-none-any.whl (4.1 MB)
Successfully installed FontTools-4.57.0 PyYAML-6.0.2 appdirs-1.4.4 attrs-25.3.0 bottle-0.13.2 defcon-0.12.1 esphome-2024.12.4 esphome-dashboard-20241217.1 freetype-py-2.5.1 fs-2.4.16 gflanguages-0.7.3 glyphsLib-6.10.1 glyphsets-1.0.0 openstep-plist-0.5.0 pillow-10.4.0 platformio-6.1.16 protobuf-3.20.3 puremagic-1.27 ufoLib2-0.17.1 unicodedata2-16.0.0
(esphome-atom-echo) noodles@sevai:~/esphome-atom-echo$ esphome compile assistant.yaml
INFO ESPHome 2024.12.4
INFO Reading configuration assistant.yaml...
INFO Updating https://github.com/esphome/esphome.git@pull/5230/head
INFO Updating https://github.com/jesserockz/esphome-components.git@None
Linking .pioenvs/study-atom-echo/firmware.elf
/home/noodles/.platformio/packages/toolchain-xtensa-esp32@8.4.0+2021r2-patch5/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld: missing --end-group; added as last command line option
RAM: [= ] 10.6% (used 34632 bytes from 327680 bytes)
Flash: [======== ] 79.8% (used 1463813 bytes from 1835008 bytes)
Building .pioenvs/study-atom-echo/firmware.bin
Creating esp32 image...
Successfully created esp32 image.
esp32_create_combined_bin([".pioenvs/study-atom-echo/firmware.bin"], [".pioenvs/study-atom-echo/firmware.elf"])
Wrote 0x176fb0 bytes to file /home/noodles/esphome-atom-echo/.esphome/build/study-atom-echo/.pioenvs/study-atom-echo/firmware.factory.bin, ready to flash to offset 0x0
esp32_copy_ota_bin([".pioenvs/study-atom-echo/firmware.bin"], [".pioenvs/study-atom-echo/firmware.elf"])
==================================================================================== [SUCCESS] Took 130.57 seconds ====================================================================================
INFO Successfully compiled program.
(esphome-atom-echo) noodles@sevai:~/esphome-atom-echo$ esphome upload --device /dev/serial/by-id/usb-Hades2001_M5stack_9552AF8367-if00-port0 assistant.yaml
INFO ESPHome 2024.12.4
INFO Reading configuration assistant.yaml...
INFO Updating https://github.com/esphome/esphome.git@pull/5230/head
INFO Updating https://github.com/jesserockz/esphome-components.git@None
INFO Upload with baud rate 460800 failed. Trying again with baud rate 115200.
esptool.py v4.7.0
Serial port /dev/serial/by-id/usb-Hades2001_M5stack_9552AF8367-if00-port0
Connecting....
Chip is ESP32-PICO-D4 (revision v1.1)
Features: WiFi, BT, Dual Core, 240MHz, Embedded Flash, VRef calibration in efuse, Coding Scheme None
Crystal is 40MHz
MAC: 64:b7:08:8a:1b:c0
Uploading stub...
Running stub...
Stub running...
Configuring flash size...
Auto-detected Flash size: 4MB
Flash will be erased from 0x00010000 to 0x00176fff...
Flash will be erased from 0x00001000 to 0x00007fff...
Flash will be erased from 0x00008000 to 0x00008fff...
Flash will be erased from 0x00009000 to 0x0000afff...
Compressed 1470384 bytes to 914252...
Wrote 1470384 bytes (914252 compressed) at 0x00010000 in 82.0 seconds (effective 143.5 kbit/s)...
Hash of data verified.
Compressed 25632 bytes to 16088...
Wrote 25632 bytes (16088 compressed) at 0x00001000 in 1.8 seconds (effective 113.1 kbit/s)...
Hash of data verified.
Compressed 3072 bytes to 134...
Wrote 3072 bytes (134 compressed) at 0x00008000 in 0.1 seconds (effective 383.7 kbit/s)...
Hash of data verified.
Compressed 8192 bytes to 31...
Wrote 8192 bytes (31 compressed) at 0x00009000 in 0.1 seconds (effective 813.5 kbit/s)...
Hash of data verified.
Leaving...
Hard resetting via RTS pin...
INFO Successfully uploaded program.
$ picocom --quiet --imap lfcrlf --baud 115200 /dev/serial/by-id/usb-Hades2001_M5stack_9552AF8367-if00-port0
I (29) boot: ESP-IDF 4.4.8 2nd stage bootloader
I (29) boot: compile time 17:31:08
I (29) boot: Multicore bootloader
I (32) boot: chip revision: v1.1
I (36) boot.esp32: SPI Speed : 40MHz
I (40) boot.esp32: SPI Mode : DIO
I (45) boot.esp32: SPI Flash Size : 4MB
I (49) boot: Enabling RNG early entropy source...
I (55) boot: Partition Table:
I (58) boot: ## Label Usage Type ST Offset Length
I (66) boot: 0 otadata OTA data 01 00 00009000 00002000
I (73) boot: 1 phy_init RF data 01 01 0000b000 00001000
I (81) boot: 2 app0 OTA app 00 10 00010000 001c0000
I (88) boot: 3 app1 OTA app 00 11 001d0000 001c0000
I (96) boot: 4 nvs WiFi data 01 02 00390000 0006d000
I (103) boot: End of partition table
I (107) esp_image: segment 0: paddr=00010020 vaddr=3f400020 size=58974h (362868) map
I (247) esp_image: segment 1: paddr=0006899c vaddr=3ffb0000 size=03400h ( 13312) load
I (253) esp_image: segment 2: paddr=0006bda4 vaddr=40080000 size=04274h ( 17012) load
I (260) esp_image: segment 3: paddr=00070020 vaddr=400d0020 size=f5cb8h (1006776) map
I (626) esp_image: segment 4: paddr=00165ce0 vaddr=40084274 size=112ach ( 70316) load
I (665) boot: Loaded app from partition at offset 0x10000
I (665) boot: Disabling RNG early entropy source...
I (677) cpu_start: Multicore app
I (677) cpu_start: Pro cpu up.
I (677) cpu_start: Starting app cpu, entry point is 0x400825c8
I (0) cpu_start: App cpu up.
I (695) cpu_start: Pro cpu start user code
I (695) cpu_start: cpu freq: 160000000
I (695) cpu_start: Application information:
I (700) cpu_start: Project name: study-atom-echo
I (705) cpu_start: App version: 2024.12.4
I (710) cpu_start: Compile time: Apr 18 2025 17:29:39
I (716) cpu_start: ELF file SHA256: 1db4989a56c6c930...
I (722) cpu_start: ESP-IDF: 4.4.8
I (727) cpu_start: Min chip rev: v0.0
I (732) cpu_start: Max chip rev: v3.99
I (737) cpu_start: Chip rev: v1.1
I (742) heap_init: Initializing. RAM available for dynamic allocation:
I (749) heap_init: At 3FFAE6E0 len 00001920 (6 KiB): DRAM
I (755) heap_init: At 3FFB8748 len 000278B8 (158 KiB): DRAM
I (761) heap_init: At 3FFE0440 len 00003AE0 (14 KiB): D/IRAM
I (767) heap_init: At 3FFE4350 len 0001BCB0 (111 KiB): D/IRAM
I (774) heap_init: At 40095520 len 0000AAE0 (42 KiB): IRAM
I (781) spi_flash: detected chip: gd
I (784) spi_flash: flash io: dio
I (790) cpu_start: Starting scheduler on PRO CPU.
I (0) cpu_start: Starting scheduler on APP CPU.
[I][logger:171]: Log initialized
[C][safe_mode:079]: There have been 0 suspected unsuccessful boot attempts
[D][esp32.preferences:114]: Saving 1 preferences to flash...
[D][esp32.preferences:143]: Saving 1 preferences to flash: 0 cached, 1 written, 0 failed
[I][app:029]: Running through setup()...
[C][esp32_rmt_led_strip:021]: Setting up ESP32 LED Strip...
[D][template.select:014]: Setting up Template Select
[D][template.select:023]: State from initial (could not load stored index): On device
[D][select:015]: 'Wake word engine location': Sending state On device (index 1)
[D][esp-idf:000]: I (100) gpio: GPIO[39] InputEn: 1 OutputEn: 0 OpenDrain: 0 Pullup: 0 Pulldown: 0 Intr:0
[D][binary_sensor:034]: 'Button': Sending initial state OFF
[C][light:021]: Setting up light 'M5Stack Atom Echo 8a1bc0'...
[D][light:036]: 'M5Stack Atom Echo 8a1bc0' Setting:
[D][light:041]: Color mode: RGB
[D][template.switch:046]: Restored state ON
[D][switch:012]: 'Use listen light' Turning ON.
[D][switch:055]: 'Use listen light': Sending state ON
[D][light:036]: 'M5Stack Atom Echo 8a1bc0' Setting:
[D][light:047]: State: ON
[D][light:051]: Brightness: 60%
[D][light:059]: Red: 100%, Green: 89%, Blue: 71%
[D][template.switch:046]: Restored state OFF
[D][switch:016]: 'timer_ringing' Turning OFF.
[D][switch:055]: 'timer_ringing': Sending state OFF
[C][i2s_audio:028]: Setting up I2S Audio...
[C][i2s_audio.microphone:018]: Setting up I2S Audio Microphone...
[C][i2s_audio.speaker:096]: Setting up I2S Audio Speaker...
[C][wifi:048]: Setting up WiFi...
[D][esp-idf:000]: I (206) wifi:
[D][esp-idf:000]: wifi driver task: 3ffc8544, prio:23, stack:6656, core=0
[D][esp-idf:000]:
[D][esp-idf:000][wifi]: I (1238) system_api: Base MAC address is not set
[D][esp-idf:000][wifi]: I (1239) system_api: read default base MAC address from EFUSE
[D][esp-idf:000][wifi]: I (1274) wifi:
[D][esp-idf:000][wifi]: wifi firmware version: ff661c3
[D][esp-idf:000][wifi]:
[D][esp-idf:000][wifi]: I (1274) wifi:
[D][esp-idf:000][wifi]: wifi certification version: v7.0
[D][esp-idf:000][wifi]:
[D][esp-idf:000][wifi]: I (1286) wifi:
[D][esp-idf:000][wifi]: config NVS flash: enabled
[D][esp-idf:000][wifi]:
[D][esp-idf:000][wifi]: I (1297) wifi:
[D][esp-idf:000][wifi]: config nano formating: disabled
[D][esp-idf:000][wifi]:
[D][esp-idf:000][wifi]: I (1317) wifi:
[D][esp-idf:000][wifi]: Init data frame dynamic rx buffer num: 32
[D][esp-idf:000][wifi]:
[D][esp-idf:000][wifi]: I (1338) wifi:
[D][esp-idf:000][wifi]: Init static rx mgmt buffer num: 5
[D][esp-idf:000][wifi]:
[D][esp-idf:000][wifi]: I (1348) wifi:
[D][esp-idf:000][wifi]: Init management short buffer num: 32
[D][esp-idf:000][wifi]:
[D][esp-idf:000][wifi]: I (1368) wifi:
[D][esp-idf:000][wifi]: Init dynamic tx buffer num: 32
[D][esp-idf:000][wifi]:
[D][esp-idf:000][wifi]: I (1389) wifi:
[D][esp-idf:000][wifi]: Init static rx buffer size: 1600
[D][esp-idf:000][wifi]:
[D][esp-idf:000][wifi]: I (1399) wifi:
[D][esp-idf:000][wifi]: Init static rx buffer num: 10
[D][esp-idf:000][wifi]:
[D][esp-idf:000][wifi]: I (1419) wifi:
[D][esp-idf:000][wifi]: Init dynamic rx buffer num: 32
[D][esp-idf:000][wifi]:
[D][esp-idf:000]: I (1441) wifi_init: rx ba win: 6
[D][esp-idf:000]: I (1441) wifi_init: tcpip mbox: 32
[D][esp-idf:000]: I (1450) wifi_init: udp mbox: 6
[D][esp-idf:000]: I (1450) wifi_init: tcp mbox: 6
[D][esp-idf:000]: I (1460) wifi_init: tcp tx win: 5760
[D][esp-idf:000]: I (1471) wifi_init: tcp rx win: 5760
[D][esp-idf:000]: I (1481) wifi_init: tcp mss: 1440
[D][esp-idf:000]: I (1481) wifi_init: WiFi IRAM OP enabled
[D][esp-idf:000]: I (1491) wifi_init: WiFi RX IRAM OP enabled
[C][wifi:061]: Starting WiFi...
[C][wifi:062]: Local MAC: 64:B7:08:8A:1B:C0
[D][esp-idf:000][wifi]: I (1513) phy_init: phy_version 4791,2c4672b,Dec 20 2023,16:06:06
[D][esp-idf:000][wifi]: I (1599) wifi:
[D][esp-idf:000][wifi]: mode : sta (64:b7:08:8a:1b:c0)
[D][esp-idf:000][wifi]:
[D][esp-idf:000][wifi]: I (1600) wifi:
[D][esp-idf:000][wifi]: enable tsf
[D][esp-idf:000][wifi]:
[D][esp-idf:000][wifi]: I (1605) wifi:
[D][esp-idf:000][wifi]: Set ps type: 1
[D][esp-idf:000][wifi]:
[D][wifi:482]: Starting scan...
[D][esp32.preferences:114]: Saving 1 preferences to flash...
[D][esp32.preferences:143]: Saving 1 preferences to flash: 1 cached, 0 written, 0 failed
[W][micro_wake_word:151]: Wake word detection can't start as the component hasn't been setup yet
[D][esp-idf:000][wifi]: I (1646) wifi:
[D][esp-idf:000][wifi]: Set ps type: 1
[D][esp-idf:000][wifi]:
[W][component:157]: Component wifi set Warning flag: scanning for networks
[I][wifi:617]: WiFi Connected!
[D][wifi:626]: Disabling AP...
[C][api:026]: Setting up Home Assistant API server...
[C][micro_wake_word:062]: Setting up microWakeWord...
[C][micro_wake_word:069]: Micro Wake Word initialized
[I][app:062]: setup() finished successfully!
[W][component:170]: Component wifi cleared Warning flag
[W][component:157]: Component api set Warning flag: unspecified
[I][app:100]: ESPHome version 2024.12.4 compiled on Apr 18 2025, 17:29:39
[C][logger:185]: Logger:
[C][logger:186]: Level: DEBUG
[C][logger:188]: Log Baud Rate: 115200
[C][logger:189]: Hardware UART: UART0
[C][esp32_rmt_led_strip:187]: ESP32 RMT LED Strip:
[C][esp32_rmt_led_strip:188]: Pin: 27
[C][esp32_rmt_led_strip:189]: Channel: 0
[C][esp32_rmt_led_strip:214]: RGB Order: GRB
[C][esp32_rmt_led_strip:215]: Max refresh rate: 0
[C][esp32_rmt_led_strip:216]: Number of LEDs: 1
[C][template.select:065]: Template Select 'Wake word engine location'
[C][template.select:066]: Update Interval: 60.0s
[C][template.select:069]: Optimistic: YES
[C][template.select:070]: Initial Option: On device
[C][template.select:071]: Restore Value: YES
[C][gpio.binary_sensor:015]: GPIO Binary Sensor 'Button'
[C][gpio.binary_sensor:016]: Pin: GPIO39
[C][light:092]: Light 'M5Stack Atom Echo 8a1bc0'
[C][light:094]: Default Transition Length: 0.0s
[C][light:095]: Gamma Correct: 2.80
[C][template.switch:068]: Template Switch 'Use listen light'
[C][template.switch:091]: Restore Mode: restore defaults to ON
[C][template.switch:057]: Optimistic: YES
[C][template.switch:068]: Template Switch 'timer_ringing'
[C][template.switch:091]: Restore Mode: always OFF
[C][template.switch:057]: Optimistic: YES
[C][factory_reset.button:011]: Factory Reset Button 'Factory reset'
[C][factory_reset.button:011]: Icon: 'mdi:restart-alert'
[C][captive_portal:089]: Captive Portal:
[C][mdns:116]: mDNS:
[C][mdns:117]: Hostname: study-atom-echo-8a1bc0
[C][esphome.ota:073]: Over-The-Air updates:
[C][esphome.ota:074]: Address: study-atom-echo.local:3232
[C][esphome.ota:075]: Version: 2
[C][esphome.ota:078]: Password configured
[C][safe_mode:018]: Safe Mode:
[C][safe_mode:020]: Boot considered successful after 60 seconds
[C][safe_mode:021]: Invoke after 10 boot attempts
[C][safe_mode:023]: Remain in safe mode for 300 seconds
[C][api:140]: API Server:
[C][api:141]: Address: study-atom-echo.local:6053
[C][api:143]: Using noise encryption: YES
[C][micro_wake_word:051]: microWakeWord:
[C][micro_wake_word:052]: models:
[C][micro_wake_word:015]: - Wake Word: Hey Jarvis
[C][micro_wake_word:016]: Probability cutoff: 0.970
[C][micro_wake_word:017]: Sliding window size: 5
[C][micro_wake_word:021]: - VAD Model
[C][micro_wake_word:022]: Probability cutoff: 0.500
[C][micro_wake_word:023]: Sliding window size: 5
[D][api:103]: Accepted 192.168.39.6
[W][component:170]: Component api cleared Warning flag
[W][component:237]: Component api took a long time for an operation (58 ms).
[W][component:238]: Components should block for at most 30 ms.
[D][api.connection:1446]: Home Assistant 2024.3.3 (192.168.39.6): Connected successfully
[D][ring_buffer:034]: Created ring buffer with size 2048
[D][micro_wake_word:399]: Resetting buffers and probabilities
[D][micro_wake_word:195]: State changed from IDLE to START_MICROPHONE
[D][micro_wake_word:107]: Starting Microphone
[D][micro_wake_word:195]: State changed from START_MICROPHONE to STARTING_MICROPHONE
[D][esp-idf:000]: I (11279) I2S: DMA Malloc info, datalen=blocksize=1024, dma_buf_count=4
[D][micro_wake_word:195]: State changed from STARTING_MICROPHONE to DETECTING_WAKE_WORD
noise_psk key you use the same string as I have under api/encryption/key in my diff above (obviously do your own, I used dd if=/dev/urandom bs=32 count=1 base64 to generate mine).
If you re like me and a compulsive VLANer and firewaller even within your own network then you need to allow Home Assistant to connect on TCP port 6053 to the ATOM Echo, and also allow access to/from UDP port 6055 on the Echo (it ll send audio from that port to Home Assistant, then receive back audio to the same port).
At this point you can now shout Hey Jarvis, what time is it? at the Echo, and the white light will turn flashing blue (indicating it s heard the wake word). Which means we re ready to teach Home Assistant how to do something with the incoming audio.
How long has it been since you last saw a conversation over different blogs
syndicated at the same planet? Well, it s one of the good memories of the
early 2010s. And there is an opportunity to re-engage!
I came across Evgeni s post naming things is
hard in Planet
Debian. So, what names have I given my
computers?
I have had many since the mid-1990s I also had several during the decade
before that, but before Linux, my computers didn t hve a formal
name. Naming my computers something nice Linux gave me.
I have forgotten many. Some of the names I have used:
campus and tlali, and one computer pending installation, ollin. The
credit for their names is not mine.
campus: A mighty SPARCstation
5! Because it was the
main (and for some time, the only!) server in our campus.tlali: A regular PC used as a Linux server. Tlali means something
like lands in n huatl, the prehispanic language spoken in central
Mexico. My workplace was Iztacala, which translates as the place
where there are white houses ; tlali and cali are related words.ollin: was a big IBM RS/6000 system running AIX. It came to us,
probably already obsolete, as a (useless) donation from Fundaci n
UNAM; I don t recall the exact model, but it looked very much like
this one. Ran on
AIX. We had no software for it, and frankly never really got it to
be productive. Funnily, its name Ollin means movement in N huatl.
I added some servers to the lineup during the two years I was in Iztacala:tlamantli: An Alpha
21164 server that doubled
as my desktop. Given the tradition in Iztacala of naming things in
N huatl, but trying to be somewhat funny, tlamantli just means a
thing; I understand the word is usually bound to a quantifier.tepancuate: A regular PC system we set up with OpenBSD as a
firewall. It means wall in N huatl.ajusco, managed by
V ctor, named after the mountain on whose
slopes our campus was). But my only computer there was:
shmate: , meaning old rag in yiddish. The word shmate is used like
thingy, although it would usually mean old and slightly worn-out
thingy. It was a quite nice machine, though. I had a Pentium 4 with
512MB RAM, not bad for 2003!mosca: is my desktop. It has changed hardware several times (of
course) over the years, but it s still the same Debian Sid install I
did in January 2005 (I must have reinstalled once, when I got it
replaced by an AMD64). Its name is the Spanish name for the common
fly. I have often used it to describe my work, since I got in the early
1990s an automated bilingual translator called TRANSLATE; it came on
seven 5.25 floppies. As a teenager, I somehow got my hands on a copy,
and installed it in my 80386SX. Fed it its own README to see how it
fared. And the first sentence made me burst in laughter: TRANSLATE
performs on the fly translation TRADUCE realiza traducci n sobre la
mosca . Starting then, I always think of on the fly as sobre la
mosca . As Groucho said, I guess Time flies like an arrow, but
fruit flies like a banana.lafa When I got there, we didn t have any servers; for some time, I
took one of the computer lab s systems to serve our web page and
receive mail. But when we got some budget approved, we bought a
fsckin-big server. Big as in four-rack-units. Double CPUs (not
multicore, but two independent early Xeon CPUs, if I m not
mistaken. Still, it was still a 32 bits system). (lafa) is a
big, more flexible kind of Arab bread than pita; I loved it when I
lived in Israel. And there is an album (and song) by
Teapacks, an Israeli group I am very
fond of, hajaim shelja belafa (your life in a lafa), saying, hey,
brother! Your life is in a lafa. You throw everything in a big
pita. You didn t have time to chew, you already swallowed it .joma: Our firewall. means wall in Hebrew.baktun: lafa was great, but over the years, it got old. After many
years, I finally got the Institute to buy a second server. We got it in
December 2012. There was a lot of noise around then because the world
was supposed to end on 2012.12.21, as the Mayan calendar reached a
full long cycle. This long cycle is called /baktun/. So, it was fitting
as the name of the new server.teom: As lafa was almost immediately decomissioned and turned into
a virtual machine in the much bigger baktun,, I wanted to split
services, make off-hardware backups, and such. Almost two years later,
my request was approved and we bought a second server. But instead of
buying it from a regular provider, we got it off a stash of machines
bought by our university s central IT
entity. To my surprise, it had the exact same
hardware configuration as baktun, bought two years earlier. Even the
serial number was absurdly close. So, I had it as baktun s long-lost
twin. Hence, (transliterated as teom), the Hebrew word for
twin. About a year after teom arrived to my life, my twin children
were also born, but their naming followed a completely different logic
process than my computers pato: The earliest system I had that I remember giving a name to. I
built a 80386SX in 1991, buying each component separately. The box had
a 1-inch square for integrators to put their branding And after some
time, I carefully printed and applied a label that said Catarm quina
PATO (the first word, very small). Pato (duck) is how we d call a
no-brand system. Catarm quina because it was the system where I ran
my BBS, CatarSYS (1992-1994).malenkaya: In 2008 I got a 9 Acer Aspire One
netbook
(Atom N270 i386, 1GB RAM). I really loved that machine! Although it was
quite limited, it was my main computer while on the road for almost
five years. malenkaya means small (for female) in Russian.matlalli: After malenkaya started being too limited for my regular
use, I bought its successor Acer Aspire One
model. This
one was way larger (10.1 inches screen) and I wasn t too happy about
it at the beginning, but I ended up loving it. So much, in fact, that
we bought at least four very similar such computers for us and our
family members. This computer was dirt cheap, and endured five further
years of lugging everywhere. matlalli is due to its turquoise color:
it is the N huatl word for blue or green.cajita: In 2014 I got a beautiful Cubox i4
Pro
computer. It took me some time to get it to boot and be generally
useful, but it ended up being my home server for many years, until I
had a power supply malfunction which bricked it. cajita means little
box in Spanish.pitentzin: Another 10.1 Acer Aspire One (the last in the lineup; the
CPU is a Celeron 877, so it does run AMD64, and it supports up to 16GB
RAM, I think I have it with 12). We originally bought it for my family
in Argentina, but they didn t really use it much, and after a couple of
years we got it back. We decided it would be the computer for the kids,
at least for the time being. And although it is a 2013 laptop, it s
still our everyday media station driver. Oh, and the name pitentzin?
N huatl for /children/.tliltik: In 2018, I bought a second-hand Thinkpad X230. It was my
daily driver for about three years. I reflashed its firmware with
CoreBoot, and repeated the experience for seven people IIRC in
DebConf18. With it, I learned to love the Thinkpad keyboard. Naturally
for a thinkpad, tliltik means black in N huatl.uesebe: When COVID struck, we were all sent home, and my university
lent me a nice recently bought Intel i7 HP laptop. At first, I didn t
want to mess up its Windows install (so I set up a USB-drive-based
installation, hence the name uesebe); when it was clear the lockdown
was going to be long (and that tliltik had too many aches to be used
for my daily work), I transferred the install to its HDD and used it
throughout the pandemic, until mid 2022.bolex: I bought this computer for my father in 2020. After he passed
away in May 2022, I took his computer, and named it bolex because
that s the brand of the 8mm cinema camera he loved and had since 1955,
and with which he created most of his
films. It is really an entry-level machine,
though (a single-core, dual-threaded Celeron), and it was too limited
when I started distance-teaching again, so I had to store it as an
emergency system.yogurtu: During the pandemics, I spent quite a bit of time fiddling
with the Raspberry Pi family. But all in all, while they are nice
machines for many uses, they are too limited to be daily drivers. Or
even enough for taking i.e. to Debconf and have them be my conference
computer. I bought an almost-new-but-used ( 2 year old) Yoga C630 ARM
laptop. I
often brag about my happy experience with it, and how it brings a
reasonably powerful ARM Linux system to my everyday life. In our last
DebConf, I didn t even pick up my USB-C power connector every day; the
battery just lasts over ten hours of active work. But I m not here
doing ads, right? yogurtu naturally is derived from the Yoga brand
it has, but is taken from Yogurtu
Ngh , a fictional
character by the Argentinian comical-musical group Les
Luthiers, that has marked my life.misnenet: Towards mid 2023, when it was clear that bolex would not
be a good daily driver, and considering we would be spending six months
in Argentina, I bought a new desktop system. It seems I have something
for small computers: I decided for a refurbished HP EliteDesk 800 G5
Mini
i7
system. I picked it because, at close to 18 18 3.5cm it perfectly fits
in my DebConf18 bag. A laptop, it is clearly not, but it can easily
travel with me when needed. Oh, and the name? Because for this model,
HP uses different enclosures based on the kind of processor: The i3
model has a flat, black aluminum top But mine has lots of tiny
holes, covering two areas of roughly 15 7cm, with a tiny hole every
~2mm, and with a solid strip between them. Of course,
(misnenet, in Hebrew) means strainer.
Most of my Debian contributions this month were
sponsored by Freexian.
You can also support my work directly via
Liberapay.
OpenSSH
Changes in dropbear 2025.87 broke OpenSSH s regression
tests. I cherry-picked the fix.
I reviewed and merged patches from Luca
Boccassi to
send and accept the COLORTERM and NO_COLOR environment variables.
Python team
Following up on last month, I fixed some
more uscan errors:
Are you a student aspiring to participate in the Google Summer of Code 2025? Would you like to improve the continuous integration pipeline used at salsa.debian.org, the Debian GitLab instance, to help improve the quality of tens of thousands of software packages in Debian?
This summer 2025, I and Emmanuel Arias will be participating as mentors in the GSoC program. We are available to mentor students who propose and develop improvements to the Salsa CI pipeline, as we are members of the Debian team that maintains it.
A post by Santiago Ruano Rinc n in the GitLab blog explains what Salsa CI is and its short history since inception in 2018. At the time of the article in fall 2023 there were 9000+ source packages in Debian using Salsa CI. Now in 2025 there are over 27,000 source packages in Debian using it, and since summer 2024 some Ubuntu developers have started using it for enhanced quality assurance of packaging changes before uploading new package revisions to Ubuntu. Personally, I have been using Salsa CI since its inception, and contributing as a team member since 2019. See my blog post about GitLab CI for MariaDB in Debian for a description of an advanced and extensive use case.
Helping Salsa CI is a great way to make a global impact, as it will help avoid regressions and improve the quality of Debian packages. The benefits reach far beyond just Debian, as it will also help hundreds of Debian derivatives, such as Ubuntu, Linux Mint, Tails, Purism PureOS, Pop!_OS, Zorin OS, Raspberry Pi OS, a large portion of Docker containers, and even the Windows Subsystem for Linux.
As promised on my previous post, on this entry I ll explain how I ve set up forgejo
actions on the source repository of this site to build it using a runner instead of doing it on the public server using
a webhook to trigger the operation.
curl to send a notification to an instance of the webhook server installed on the
remote server that triggers a script that updates the site using the git branch.webhook serviceOn the server machine we have installed and configured the webhook service to run a script that updates the site.
To install the application and setup the configuration we have used the following script:
#!/bin/sh
set -e
# ---------
# VARIABLES
# ---------
ARCH="$(dpkg --print-architecture)"
WEBHOOK_VERSION="2.8.2"
DOWNLOAD_URL="https://github.com/adnanh/webhook/releases/download"
WEBHOOK_TGZ_URL="$DOWNLOAD_URL/$WEBHOOK_VERSION/webhook-linux-$ARCH.tar.gz"
WEBHOOK_SERVICE_NAME="webhook"
# Files
WEBHOOK_SERVICE_FILE="/etc/systemd/system/$WEBHOOK_SERVICE_NAME.service"
WEBHOOK_SOCKET_FILE="/etc/systemd/system/$WEBHOOK_SERVICE_NAME.socket"
WEBHOOK_TML_TEMPLATE="/srv/blogops/action/webhook.yml.envsubst"
WEBHOOK_YML="/etc/webhook.yml"
# Config file values
WEBHOOK_USER="$(id -u)"
WEBHOOK_GROUP="$(id -g)"
WEBHOOK_LISTEN_STREAM="172.31.31.1:4444"
# ----
# MAIN
# ----
# Install binary from releases (on Debian only version 2.8.0 is available, but
# I need the 2.8.2 version to support the systemd activation mode).
curl -fsSL -o "/tmp/webhook.tgz" "$WEBHOOK_TGZ_URL"
tar -C /tmp -xzf /tmp/webhook.tgz
sudo install -m 755 "/tmp/webhook-linux-$ARCH/webhook" /usr/local/bin/webhook
rm -rf "/tmp/webhook-linux-$ARCH" /tmp/webhook.tgz
# Service file
sudo sh -c "cat >'$WEBHOOK_SERVICE_FILE'" <<EOF
[Unit]
Description=Webhook server
[Service]
Type=exec
ExecStart=webhook -nopanic -hooks $WEBHOOK_YML
User=$WEBHOOK_USER
Group=$WEBHOOK_GROUP
EOF
# Socket config
sudo sh -c "cat >'$WEBHOOK_SOCKET_FILE'" <<EOF
[Unit]
Description=Webhook server socket
[Socket]
# Set FreeBind to listen on missing addresses (the VPN can be down sometimes)
FreeBind=true
# Set ListenStream to the IP and port you want to listen on
ListenStream=$WEBHOOK_LISTEN_STREAM
[Install]
WantedBy=multi-user.target
EOF
# Config file
BLOGOPS_TOKEN="$(uuid)" \
envsubst <"$WEBHOOK_TML_TEMPLATE" sudo sh -c "cat >$WEBHOOK_YML"
chmod 0640 "$WEBHOOK_YML"
chwon "$WEBHOOK_USER:$WEBHOOK_GROUP" "$WEBHOOK_YML"
# Restart and enable service
sudo systemctl daemon-reload
sudo systemctl stop "$WEBHOOK_SERVICE_NAME.socket"
sudo systemctl start "$WEBHOOK_SERVICE_NAME.socket"
sudo systemctl enable "$WEBHOOK_SERVICE_NAME.socket"
# ----
# vim: ts=2:sw=2:et:ai:sts=2systemd with socket activation.
The configuration file template is the following one:
- id: "update-blogops"
execute-command: "/srv/blogops/action/bin/update-blogops.sh"
command-working-directory: "/srv/blogops"
trigger-rule:
match:
type: "value"
value: "$BLOGOPS_TOKEN"
parameter:
source: "header"
name: "X-Blogops-Token"/etc/webhook.yml has the BLOGOPS_TOKEN adjusted to a random value that has to exported as a secret on
the forgejo project (see later).
Once the service is started each time the action is executed the webhook daemon will get a notification and will run
the following update-blogops.sh script to publish the updated version of the site:
#!/bin/sh
set -e
# ---------
# VARIABLES
# ---------
# Values
REPO_URL="ssh://git@forgejo.mixinet.net/mixinet/blogops.git"
REPO_BRANCH="html"
REPO_DIR="public"
MAIL_PREFIX="[BLOGOPS-UPDATE-ACTION] "
# Address that gets all messages, leave it empty if not wanted
MAIL_TO_ADDR="blogops@mixinet.net"
# Directories
BASE_DIR="/srv/blogops"
PUBLIC_DIR="$BASE_DIR/$REPO_DIR"
NGINX_BASE_DIR="$BASE_DIR/nginx"
PUBLIC_HTML_DIR="$NGINX_BASE_DIR/public_html"
ACTION_BASE_DIR="$BASE_DIR/action"
ACTION_LOG_DIR="$ACTION_BASE_DIR/log"
# Files
OUTPUT_BASENAME="$(date +%Y%m%d-%H%M%S.%N)"
ACTION_LOGFILE_PATH="$ACTION_LOG_DIR/$OUTPUT_BASENAME.log"
# ---------
# Functions
# ---------
action_log()
echo "$(date -R) $*" >>"$ACTION_LOGFILE_PATH"
action_check_directories()
for _d in "$ACTION_BASE_DIR" "$ACTION_LOG_DIR"; do
[ -d "$_d" ] mkdir "$_d"
done
action_clean_directories()
# Try to remove empty dirs
for _d in "$ACTION_LOG_DIR" "$ACTION_BASE_DIR"; do
if [ -d "$_d" ]; then
rmdir "$_d" 2>/dev/null true
fi
done
mail_success()
to_addr="$MAIL_TO_ADDR"
if [ "$to_addr" ]; then
subject="OK - updated blogops site"
mail -s "$ MAIL_PREFIX $ subject " "$to_addr" <"$ACTION_LOGFILE_PATH"
fi
mail_failure()
to_addr="$MAIL_TO_ADDR"
if [ "$to_addr" ]; then
subject="KO - failed to update blogops site"
mail -s "$ MAIL_PREFIX $ subject " "$to_addr" <"$ACTION_LOGFILE_PATH"
fi
exit 1
# ----
# MAIN
# ----
ret="0"
# Check directories
action_check_directories
# Go to the base directory
cd "$BASE_DIR"
# Remove the old build dir if present
if [ -d "$PUBLIC_DIR" ]; then
rm -rf "$PUBLIC_DIR"
fi
# Update the repository checkout
action_log "Updating the repository checkout"
git fetch --all >>"$ACTION_LOGFILE_PATH" 2>&1 ret="$?"
if [ "$ret" -ne "0" ]; then
action_log "Failed to update the repository checkout"
mail_failure
fi
# Get it from the repo branch & extract it
action_log "Downloading and extracting last site version using 'git archive'"
git archive --remote="$REPO_URL" "$REPO_BRANCH" "$REPO_DIR" \
tar xf - >>"$ACTION_LOGFILE_PATH" 2>&1 ret="$?"
# Fail if public dir was missing
if [ "$ret" -ne "0" ] [ ! -d "$PUBLIC_DIR" ]; then
action_log "Failed to download or extract site"
mail_failure
fi
# Remove old public_html copies
action_log 'Removing old site versions, if present'
find $NGINX_BASE_DIR -mindepth 1 -maxdepth 1 -name 'public_html-*' -type d \
-exec rm -rf \; >>"$ACTION_LOGFILE_PATH" 2>&1 ret="$?"
if [ "$ret" -ne "0" ]; then
action_log "Removal of old site versions failed"
mail_failure
fi
# Switch site directory
TS="$(date +%Y%m%d-%H%M%S)"
if [ -d "$PUBLIC_HTML_DIR" ]; then
action_log "Moving '$PUBLIC_HTML_DIR' to '$PUBLIC_HTML_DIR-$TS'"
mv "$PUBLIC_HTML_DIR" "$PUBLIC_HTML_DIR-$TS" >>"$ACTION_LOGFILE_PATH" 2>&1
ret="$?"
fi
if [ "$ret" -eq "0" ]; then
action_log "Moving '$PUBLIC_DIR' to '$PUBLIC_HTML_DIR'"
mv "$PUBLIC_DIR" "$PUBLIC_HTML_DIR" >>"$ACTION_LOGFILE_PATH" 2>&1
ret="$?"
fi
if [ "$ret" -ne "0" ]; then
action_log "Site switch failed"
mail_failure
else
action_log "Site updated successfully"
mail_success
fi
# ----
# vim: ts=2:sw=2:et:ai:sts=2hugo-adoc workflowThe workflow is defined in the .forgejo/workflows/hugo-adoc.yml file and looks like this:
name: hugo-adoc
# Run this job on push events to the main branch
on:
push:
branches:
- 'main'
jobs:
build-and-push:
if: $ vars.BLOGOPS_WEBHOOK_URL != '' && secrets.BLOGOPS_TOKEN != ''
runs-on: docker
container:
image: forgejo.mixinet.net/oci/hugo-adoc:latest
# Allow the job to write to the repository (not really needed on forgejo)
permissions:
contents: write
steps:
- name: Checkout the repo
uses: actions/checkout@v4
with:
submodules: 'true'
- name: Build the site
shell: sh
run:
rm -rf public
hugo
- name: Push compiled site to html branch
shell: sh
run:
# Set the git user
git config --global user.email "blogops@mixinet.net"
git config --global user.name "BlogOps"
# Create a new orphan branch called html (it was not pulled by the
# checkout step)
git switch --orphan html
# Add the public directory to the branch
git add public
# Commit the changes
git commit --quiet -m "Updated site @ $(date -R)" public
# Push the changes to the html branch
git push origin html --force
# Switch back to the main branch
git switch main
- name: Call the blogops update webhook endpoint
shell: sh
run:
HEADER="X-Blogops-Token: $ secrets.BLOGOPS_TOKEN "
curl --fail -k -H "$HEADER" $ vars.BLOGOPS_WEBHOOK_URL BLOGOPS_TOKEN variable to the project secrets (its value is the one
included on the /etc/webhook.yml file created when installing the webhook service) and the BLOGOPS_WEBHOOK_URL
project variable (its value is the URL of the webhook server, in my case
http://172.31.31.1:4444/hooks/update-blogops); note that the job includes the -k flag on the curl command just in
case I end up using TLS on the webhook server in the future, as discussed previously.webhook server which, IMHO, is a more secure setup. Last week I decided I wanted to try out forgejo actions to build this blog instead of using
webhooks, so I looked the documentation and started playing with it until I had it working as I wanted.
This post is to describe how I ve installed and configured a forgejo runner, how I ve added an
oci organization to my instance to build, publish and mirror container images and added a couple of
additional organizations (actions and docker for now) to mirror interesting
actions.
The changes made to build the site using actions will be documented on a separate post, as I ll be using this entry to
test the new setup on the blog project.
$ cd /srv
$ git clone https://forgejo.mixinet.net/blogops/forgejo-runner.git
$ cd forgejo-runner
$ sh ./bin/setup-runner.shsetup-runner.sh script does multiple things:
forgejo-runner user and group.runner file with a predefined secret and the docker labelsetup-runner.sh code is available here.
After running the script the runner has to be registered with the forgejo server, it can be done using the following
command:
$ forgejo forgejo-cli actions register --name "$RUNNER_NAME" \
--secret "$FORGEJO_SECRET"RUNNER_NAME variable is defined on the setup-runner.sh script and the FORGEJO_SECRET must match the value used
on the .runner file.
docker-composeTo launch the runner I m going to use a docker-compose.yml file that starts two containers, a docker in docker
service to run the containers used by the workflow jobs and another one that runs the forgejo-runner itself.
The initial version used a TCP port to communicate with the dockerd server from the runner, but when I tried to build
images from a workflow I noticed that the containers launched by the runner were not going to be able to execute
another dockerd inside the dind one and, even if they were, it was going to be expensive computationally.
To avoid the issue I modified the dind service to use a unix socket on a shared volume that can be used by the
runner service to communicate with the daemon and also re-shared with the job containers so the dockerd server can
be used from them to build images.
dind service, but just in case I want to run
other containers on the host I prefer to keep the one used for the runner isolated from it).
For those concerned about sharing the same server an alternative would be to launch a second dockerd only for the jobs
(i.e. actions-dind) using the same approach (the volume with its socket will have to be shared with the runner
service so it can be re-shared, but the runner does not need to use it).docker-compose.yaml file is as follows:
services:
dind:
image: docker:dind
container_name: 'dind'
privileged: 'true'
command: ['dockerd', '-H', 'unix:///dind/docker.sock', '-G', '$RUNNER_GID']
restart: 'unless-stopped'
volumes:
- ./dind:/dind
runner:
image: 'data.forgejo.org/forgejo/runner:6.2.2'
links:
- dind
depends_on:
dind:
condition: service_started
container_name: 'runner'
environment:
DOCKER_HOST: 'unix:///dind/docker.sock'
user: $RUNNER_UID:$RUNNER_GID
volumes:
- ./config.yaml:/config.yaml
- ./data:/data
- ./dind:/dind
restart: 'unless-stopped'
command: '/bin/sh -c "sleep 5; forgejo-runner daemon -c /config.yaml"'dockerd server is started with the -H unix:///dind/docker.sock flag to use the unix socket to communicate
with the daemon instead of using a TCP port (as said, it is faster and allows us to share the socket with the
containers started by the runner).dockerd daemon with the RUNNER_GID group so the runner can communicate with it (the socket
gets that group which is the same used by the runner).data directory, the dind folder where docker creates the unix
socket and a config.yaml file used by us to change the default runner configuration.config.yaml file was originally created using the forgejo-runner:
$ docker run --rm data.forgejo.org/forgejo/runner:6.2.2 \
forgejo-runner generate-config > config.yamlcapacity has been increased to 2 (that allows it to run two jobs at the
same time) and the /dind/docker.sock value has been added to the valid_volumes key to allow the containers launched
by the runner to mount it when needed; the diff against the default version is as follows:
@@ -13,7 +13,8 @@
# Where to store the registration result.
file: .runner
# Execute how many tasks concurrently at the same time.
- capacity: 1
+ # STO: Allow 2 concurrent tasks
+ capacity: 2
# Extra environment variables to run jobs.
envs:
A_TEST_ENV_NAME_1: a_test_env_value_1
@@ -87,7 +88,9 @@
# If you want to allow any volume, please use the following configuration:
# valid_volumes:
# - '**'
- valid_volumes: []
+ # STO: Allow to mount the /dind/docker.sock on the containers
+ valid_volumes:
+ - /dind/docker.sock
# overrides the docker client host with the specified one.
# If "-" or "", an available docker host will automatically be found.
# If "automount", an available docker host will automatically be found and ...RUNNER_UID and RUNNER_GID variables and call docker-compose up to start the
containers on the background:
$ RUNNER_UID="$(id -u forgejo-runner)" RUNNER_GID="$(id -g forgejo-runner)" \
docker compose up -dcode one for them):
uses keyword) we have added the
following section to the app.ini file of our forgejo server:
[actions]
ENABLED = true
DEFAULT_ACTIONS_URL = https://forgejo.mixinet.netoci organization I ve created a token with package:write permission for my own
user because I m a member of the organization and I m authorized to publish packages on it (a different user could be
created, but as I said this is for personal use, so there is no need to complicate things for now).
To allow the use of those credentials on the actions I have added a secret (REGISTRY_PASS) and a variable
(REGISTRY_USER) to the oci organization to allow the actions to use them.
I ve also logged myself on my local docker client to be able to push images to the oci group by hand, as I it is
needed for bootstrapping the system (as I m using local images on the worflows I need to push them to the server before
running the ones that are used to build the images).oci/images projectThe images project is a monorepo that contains the source files for the images we are going to build and a couple of
actions.
The image sources are on sub directories of the repository, to be considered an image the folder has to contain a
Dockerfile that will be used to build the image.
The repository has two workflows:
build-image-from-tag: Workflow to build, tag and push an image to the oci organizationmulti-semantic-release: Workflow to create tags for the images using the multi-semantic-release tool.registry="forgejo.mixinet.net/oci"
for img in alpine-mixinet node-mixinet multi-semantic-release; do
docker build -t $registry/$img:1.0.0 $img
docker tag $registry/$img:1.0.0 $registry/$img:latest
docker push $registry/$img:1.0.0
docker push $registry/$img:latest
donebuild-image-from-tag workflowThis workflow uses a docker client to build an image from a tag on the repository with the format
image-name-v[0-9].[0-9].[0-9]+.
As the runner is executed on a container (instead of using lxc) it seemed unreasonable to run another dind
container from that one, that is why, after some tests, I decided to share the dind service server socket with the
runner container and enabled the option to mount it also on the containers launched by the runner when needed (I only
do it on the build-image-from-tag action for now).
The action was configured to run using a trigger or when new tags with the right format were created, but when the tag
is created by multi-semantic-release the trigger does not work for some reason, so now it only runs the job on
triggers and checks if it is launched for a tag with the right format on the job itself.
The source code of the action is as follows:
name: build-image-from-tag
on:
workflow_dispatch:
jobs:
build:
# Don't build the image if the registry credentials are not set, the ref is not a tag or it doesn't contain '-v'
if: $ vars.REGISTRY_USER != '' && secrets.REGISTRY_PASS != '' && startsWith(github.ref, 'refs/tags/') && contains(github.ref, '-v')
runs-on: docker
container:
image: forgejo.mixinet.net/oci/node-mixinet:latest
# Mount the dind socket on the container at the default location
options: -v /dind/docker.sock:/var/run/docker.sock
steps:
- name: Extract image name and tag from git and get registry name from env
id: job_data
run:
echo "::set-output name=img_name::$ GITHUB_REF_NAME%%-v* "
echo "::set-output name=img_tag::$ GITHUB_REF_NAME##*-v "
echo "::set-output name=registry::$(
echo "$ github.server_url " sed -e 's%https://%%'
)"
echo "::set-output name=oci_registry_prefix::$(
echo "$ github.server_url /oci" sed -e 's%https://%%'
)"
- name: Checkout the repo
uses: actions/checkout@v4
- name: Export build dir and Dockerfile
id: build_data
run:
img="$ steps.job_data.outputs.img_name "
build_dir="$(pwd)/$ img "
dockerfile="$ build_dir /Dockerfile"
if [ -f "$dockerfile" ]; then
echo "::set-output name=build_dir::$build_dir"
echo "::set-output name=dockerfile::$dockerfile"
else
echo "Couldn't find the Dockerfile for the '$img' image"
exit 1
fi
- name: Login to the Container Registry
uses: docker/login-action@v3
with:
registry: $ steps.job_data.outputs.registry
username: $ vars.REGISTRY_USER
password: $ secrets.REGISTRY_PASS
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: Build and Push
uses: docker/build-push-action@v6
with:
push: true
tags:
$ steps.job_data.outputs.oci_registry_prefix /$ steps.job_data.outputs.img_name :$ steps.job_data.outputs.img_tag
$ steps.job_data.outputs.oci_registry_prefix /$ steps.job_data.outputs.img_name :latest
context: $ steps.build_data.outputs.build_dir
file: $ steps.build_data.outputs.dockerfile
build-args:
OCI_REGISTRY_PREFIX=$ steps.job_data.outputs.oci_registry_prefix /if condition of the build job is not perfect, but it is good enough to avoid wrong uses as long as nobody
uses manual tags with the wrong format and expects things to work (it checks if the REGISTRY_USER and
REGISTRY_PASS variables are set, if the ref is a tag and if it contains the -v string).dind socket we mount it on the container using the options key on the container section
of the job (this only works if supported by the runner configuration as explained before).job_data step to get information about the image from the tag and the registry URL from the environment
variables, it is executed first because all the information is available without checking out the repository.job_data step to get the build dir and Dockerfile paths from the repository (right now we are
assuming fixed paths and checking if the Dockerfile exists, but in the future we could use a configuration file to
get them, if needed).OCI_REGISTRY_PREFIX build argument to the Dockerfile to be able to use it
on the FROM instruction (we are using it in our images).multi-semantic-release workflowThis workflow is used to run the multi-semantic-release tool on pushes to the main branch.
It is configured to create the configuration files on the fly (it prepares things to tag the folders that contain a
Dockerfile using a couple of template files available on the repository s .forgejo directory) and run the
multi-semantic-release tool to create tags and push them to the repository if new versions are to be built.
Initially we assumed that the tag creation pushed by multi-semantic-release would be enough to run the
build-tagged-image-task action, but as it didn t work we removed the rule to run the action on tag creation and added
code to trigger the action using an api call for the newly created tags (we get them from the output of the
multi-semantic-release execution).
The source code of the action is as follows:
name: multi-semantic-release
on:
push:
branches:
- 'main'
jobs:
multi-semantic-release:
runs-on: docker
container:
image: forgejo.mixinet.net/oci/multi-semantic-release:latest
steps:
- name: Checkout the repo
uses: actions/checkout@v4
- name: Generate multi-semantic-release configuration
shell: sh
run:
# Get the list of images to work with (the folders that have a Dockerfile)
images="$(for img in */Dockerfile; do dirname "$img"; done)"
# Generate a values.yaml file for the main packages.json file
package_json_values_yaml=".package.json-values.yaml"
echo "images:" >"$package_json_values_yaml"
for img in $images; do
echo " - $img" >>"$package_json_values_yaml"
done
echo "::group::Generated values.yaml for the project"
cat "$package_json_values_yaml"
echo "::endgroup::"
# Generate the package.json file validating that is a good json file with jq
tmpl -f "$package_json_values_yaml" ".forgejo/package.json.tmpl" jq . > "package.json"
echo "::group::Generated package.json for the project"
cat "package.json"
echo "::endgroup::"
# Remove the temporary values file
rm -f "$package_json_values_yaml"
# Generate the package.json file for each image
for img in $images; do
tmpl -v "img_name=$img" -v "img_path=$img" ".forgejo/ws-package.json.tmpl" jq . > "$img/package.json"
echo "::group::Generated package.json for the '$img' image"
cat "$img/package.json"
echo "::endgroup::"
done
- name: Run multi-semantic-release
shell: sh
run:
multi-semantic-release tee .multi-semantic-release.log
- name: Trigger builds
shell: sh
run:
# Get the list of tags published on the previous steps
tags="$(
sed -n -e 's/^\[.*\] \[\(.*\)\] .* Published release \([0-9]\+\.[0-9]\+\.[0-9]\+\) on .*$/\1-v\2/p' \
.multi-semantic-release.log
)"
rm -f .multi-semantic-release.log
if [ "$tags" ]; then
# Prepare the url for building the images
workflow="build-image-from-tag.yaml"
dispatch_url="$ github.api_url /repos/$ github.repository /actions/workflows/$workflow/dispatches"
echo "$tags" while read -r tag; do
echo "Triggering build for tag '$tag'"
curl \
-H "Content-Type:application/json" \
-H "Authorization: token $ secrets.GITHUB_TOKEN " \
-d " \"ref\":\"$tag\" " "$dispatch_url"
done
fitmpl tool to process the multi-semantic-release configuration templates comes from previous uses,
but on this case we could use a different approach (i.e. envsubst could be used) but we left it because it keeps
things simple and can be useful in the future if we want to do more complex things with the template files.tee to show and dump to a file the output of the multi-semantic-release execution.tags using sed against the output of the multi-semantic-release execution and for
each one found we use curl to call the forgejo API to trigger the build job; as the call is against the same
project we can use the GITHUB_TOKEN generated for the workflow to do it, without creating a user token that has to
be shared as a secret..forgejo/package.json.tmpl file is the following one:
"name": "multi-semantic-release",
"version": "0.0.0-semantically-released",
"private": true,
"multi-release":
"tagFormat": "$ name -v$ version "
,
"workspaces": .images toJson
.forgejo/ws-package.json.tmpl file is the following one:
"name": " .img_name ",
"license": "UNLICENSED",
"release":
"plugins": [
[
"@semantic-release/commit-analyzer",
"preset": "conventionalcommits",
"releaseRules": [
"breaking": true, "release": "major" ,
"revert": true, "release": "patch" ,
"type": "feat", "release": "minor" ,
"type": "fix", "release": "patch" ,
"type": "perf", "release": "patch"
]
],
[
"semantic-release-replace-plugin",
"replacements": [
"files": [ " .img_path /msr.yaml" ],
"from": "^version:.*$",
"to": "version: $ nextRelease.version ",
"allowEmptyPaths": true
]
],
[
"@semantic-release/git",
"assets": [ "msr.yaml" ],
"message": "ci(release): .img_name -v$ nextRelease.version \n\n$ nextRelease.notes "
]
],
"branches": [ "main" ]
oci/mirrors projectThe repository contains a template for the configuration file we are going to use with regsync
(regsync.envsubst.yml) to mirror images from remote registries using a workflow that generates a configuration file
from the template and runs the tool.
The initial version of the regsync.envsubst.yml file is prepared to mirror alpine containers from version 3.21 to
3.29 (we explicitly remove version 3.20) and needs the forgejo.mixinet.net/oci/node-mixinet:latest image to run
(as explained before it was pushed manually to the server):
version: 1
creds:
- registry: "$REGISTRY"
user: "$REGISTRY_USER"
pass: "$REGISTRY_PASS"
sync:
- source: alpine
target: $REGISTRY/oci/alpine
type: repository
tags:
allow:
- "latest"
- "3\\.2\\d+"
- "3\\.2\\d+.\\d+"
deny:
- "3\\.20"
- "3\\.20.\\d+"mirror workflowThe mirror workflow creates a configuration file replacing the value of the REGISTRY environment variable (computed
by removing the protocol from the server_url), the REGISTRY_USER organization value and the REGISTRY_PASS secret
using the envsubst command and running the regsync tool to mirror the images using the configuration file.
The action is configured to run daily, on push events when the regsync.envsubst.yml file is modified on the main
branch and can also be triggered manually.
The source code of the action is as follows:
name: mirror
on:
schedule:
- cron: '@daily'
push:
branches:
- main
paths:
- 'regsync.envsubst.yml'
workflow_dispatch:
jobs:
mirror:
if: $ vars.REGISTRY_USER != '' && secrets.REGISTRY_PASS != ''
runs-on: docker
container:
image: forgejo.mixinet.net/oci/node-mixinet:latest
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Sync images
run:
REGISTRY="$(echo "$ github.server_url " sed -e 's%https://%%')" \
REGISTRY_USER="$ vars.REGISTRY_USER " \
REGISTRY_PASS="$ secrets.REGISTRY_PASS " \
envsubst <regsync.envsubst.yml >.regsync.yml
regsync --config .regsync.yml once
rm -f .regsync.ymlforgejo-runner and configured it to run actions for our own server and things are working fine.
This approach allows us to have a powerful CI/CD system on a modest home server, something very useful for maintaining
personal projects and playing with things without needing SaaS platforms like github or
gitlab.Next.