argparse subcommands
are great, but they have a quirk in which options are only available right
after the subcommand that define them.
So, if you for example add the --verbose / -v argument to your main parser,
and you have subcommands, you need to give the -v option before the
subcommand name. For example, given this script:
$ ./mycmd test
False
$ ./mycmd -v test
True
$ ./mycmd test -v
usage: mycmd [-h][-v]test ...
mycmd: error: unrecognized arguments: -v
This sometimes makes sense, and many other times it's really annoying, since
the user has to remember at which level an option was defined.
Last night some pieces clicked in my head, and I created a not-too-dirty
ArgumentParser subclass
that adds a shared option to arguments, that propagates them to subparsers:
Hey,
If you enjoy using upstream Linux kernel in your Raspberry Pi system or just
want to give a try in the freshest kernel graphics drivers there, the good news
is that now you can compile and boot the V3D driver from the mainline in your
Raspberry Pi 4. Thanks to the work of Stefan, Peter and Nicolas
[1][2],
the V3D enablement reached the Linux kernel mainline. That means hacking and
using new features available in the upstream V3D driver directly from the
source.
However, even for those used to compiling and installing a custom kernel in the
Raspberry Pi, there are some quirks to getting the mainline v3d module
available in 32-bit and 64-bit systems. I ve quickly summarized how to compile
and install upstream kernel versions (>=6.0) in this short blog post.
Note: V3D driver is not present in Raspberry Pi models 0-3.
First, I m taking into account that you already know how to cross-compile a
custom kernel to your system. If it is not your case, a good tutorial is
already available in the Raspberry Pi
documentation,
but it targets the kernel in the rpi-linux repository (downstream kernel).
From this documentation, the main differences in the upstream kernel are
presented below:
Raspberry Pi 4 64-bit (arm64)
Diff short summary:
instead of getting the .config file from bcm2711_defconfig, get it by running make ARCH=arm64 defconfig
compile and install the kernel image and modules as usual, but just copy the dtb file arch/arm64/boot/dts/broadcom/bcm2711-rpi-4-b.dtb to the /boot of your target machine (no /overlays directory to copy)
change /boot/config.txt:
comment/remove the dt_overlay=vc4-kms-v3d entry
add a device_tree=bcm2711-rpi-4-b.dtb entry
Raspberry Pi 4 32-bits (arm)
Diff short summary:
get the .config file by running make ARCH=arm multi_v7_defconfig
using make ARCH=arm menuconfig or a similar tool, enable CONFIG_ARM_LPAE=y
compile and install the kernel image and modules as usual, but just copy the dtb file arch/arm/boot/dts/bcm2711-rpi-4-b.dtb to the /boot of your target machine (no /overlays directory to copy)
change /boot/config.txt:
comment/remove the dt_overlay=vc4-kms-v3d entry
add a device_tree=bcm2711-rpi-4-b.dtb entry
Step-by-step for remote deployment:
Set variables
Raspberry Pi 4 64-bit (arm64)
cd <path-to-upstream-linux-directory>
KERNEL= make kernelrelease
ARCH="arm64"
CROSS_COMPILE="aarch64-linux-gnu-"
DEFCONFIG=defconfig
IMAGE=Image
DTB_PATH="broadcom/bcm2711-rpi-4-b.dtb"
RPI4=<ip>
TMP= mktemp -d
Raspberry Pi 4 32-bits (arm)
cd <path-to-upstream-linux-directory>
KERNEL= make kernelrelease
ARCH="arm"
CROSS_COMPILE="arm-linux-gnueabihf-"
DEFCONFIG=multi_v7_defconfig
IMAGE=zImage
DTB_PATH="bcm2711-rpi-4-b.dtb"
RPI4=<ip>
TMP= mktemp -d
Get default .config file
make ARCH=$ARCH CROSS_COMPILE=$CROSS_COMPILE $DEFCONFIG
Raspberry Pi 4 32-bit (arm)
Additional step for 32-bit system.
Enable CONFIG_ARM_LPAE=y using make ARCH=arm menuconfig
Cross-compile the mainline kernel
make ARCH=$ARCH CROSS_COMPILE=$CROSS_COMPILE $IMAGE modules dtbs
Install modules to send
make ARCH=$ARCH CROSS_COMPILE=$CROSS_COMPILE INSTALL_MOD_PATH=$TMP modules_install
Copy kernel image, modules and the dtb to your remote system
Set config.txt of you RPi 4
In your Raspberry Pi 4, open the config file /boot/config.txt
comment/remove the dt_overlay=vc4-kms-v3d entry
add a device_tree=bcm2711-rpi-4-b.dtb entry
add a kernel=<image-name> entry
Why not Kworkflow?
You can safely use the steps above, but if you are hacking the kernel and need
to repeat this compiling and installing steps repeatedly, why don t try the
Kworkflow?
Kworkflow is a set of scripts to
synthesize all steps to have a custom kernel compiled and installed in local
and remote machines and it supports kernel building and deployment to Raspberry
Pi machines for Raspbian 32 bit and 64 bit.
After learning the kernel compilation and installation step by step, you can
simply use kw bd command and have a custom kernel installed in your Raspberry
Pi 4.
Note: the userspace 3D acceleration (glx/mesa) is working as expected on arm64,
but the driver is not loaded yet for arm. Besides that, a bunch of pte invalid
errors may appear when using 3D acceleration, it s a known issue that are still
under investigation.
If you ve done anything in the Kubernetes space in recent years, you ve most likely come across the words Service Mesh . It s backed by a set of mature technologies that provides cross-cutting networking, security, infrastructure capabilities to be used by workloads running in Kubernetes in a manner that is transparent to the actual workload. This abstraction enables application developers to not worry about building in otherwise sophisticated capabilities for networking, routing, circuit-breaking and security, and simply rely on the services offered by the service mesh.In this post, I ll be covering Linkerd, which is an alternative to Istio. It has gone through a significant re-write when it transitioned from the JVM to a Go-based Control Plane and a Rust-based Data Plane a few years back and is now a part of the CNCF and is backed by Buoyant. It has proven itself widely for use in production workloads and has a healthy community and release cadence.It achieves this with a side-car container that communicates with a Linkerd control plane that allows central management of policy, telemetry, mutual TLS, traffic routing, shaping, retries, load balancing, circuit-breaking and other cross-cutting concerns before the traffic hits the container. This has made the task of implementing the application services much simpler as it is managed by container orchestrator and service mesh. I covered Istio in a prior post a few years back, and much of the content is still applicable for this post, if you d like to have a look.Here are the broad architectural components of Linkerd:The components are separated into the control plane and the data plane.The control plane components live in its own namespace and consists of a controller that the Linkerd CLI interacts with via the Kubernetes API. The destination service is used for service discovery, TLS identity, policy on access control for inter-service communication and service profile information on routing, retries, timeouts. The identity service acts as the Certificate Authority which responds to Certificate Signing Requests (CSRs) from proxies for initialization and for service-to-service encrypted traffic. The proxy injector is an admission webhook that injects the Linkerd proxy side car and the init container automatically into a pod when the linkerd.io/inject: enabled is available on the namespace or workload.On the data plane side are two components. First, the init container, which is responsible for automatically forwarding incoming and outgoing traffic through the Linkerd proxy via iptables rules. Second, the Linkerd proxy, which is a lightweight micro-proxy written in Rust, is the data plane itself.I will be walking you through the setup of Linkerd (2.12.2 at the time of writing) on a Kubernetes cluster.Let s see what s running on the cluster currently. This assumes you have a cluster running and kubectl is installed and available on the PATH.
On most systems, this should be sufficient to setup the CLI. You may need to restart your terminal to load the updated paths. If you have a non-standard configuration and linkerd is not found after the installation, add the following to your PATH to be able to find the cli:
export PATH=$PATH:~/.linkerd2/bin/
At this point, checking the version would give you the following:
$ linkerd version Client version: stable-2.12.2 Server version: unavailable
Setting up Linkerd Control PlaneBefore installing Linkerd on the cluster, run the following step to check the cluster for pre-requisites:
kubernetes-api -------------- can initialize the client can query the Kubernetes API
kubernetes-version ------------------ is running the minimum Kubernetes API version is running the minimum kubectl version
pre-kubernetes-setup -------------------- control plane namespace does not already exist can create non-namespaced resources can create ServiceAccounts can create Services can create Deployments can create CronJobs can create ConfigMaps can create Secrets can read Secrets can read extension-apiserver-authentication configmap no clock skew detected
linkerd-version --------------- can determine the latest version cli is up-to-date
Status check results are
All the pre-requisites appear to be good right now, and so installation can proceed.The first step of the installation is to setup the Custom Resource Definitions (CRDs) that Linkerd requires. The linkerd cli only prints the resource YAMLs to standard output and does not create them directly in Kubernetes, so you would need to pipe the output to kubectl apply to create the resources in the cluster that you re working with.
$ linkerd install --crds kubectl apply -f - Rendering Linkerd CRDs... Next, run linkerd install kubectl apply -f - to install the control plane.
customresourcedefinition.apiextensions.k8s.io/authorizationpolicies.policy.linkerd.io created customresourcedefinition.apiextensions.k8s.io/httproutes.policy.linkerd.io created customresourcedefinition.apiextensions.k8s.io/meshtlsauthentications.policy.linkerd.io created customresourcedefinition.apiextensions.k8s.io/networkauthentications.policy.linkerd.io created customresourcedefinition.apiextensions.k8s.io/serverauthorizations.policy.linkerd.io created customresourcedefinition.apiextensions.k8s.io/servers.policy.linkerd.io created customresourcedefinition.apiextensions.k8s.io/serviceprofiles.linkerd.io created
Next, install the Linkerd control plane components in the same manner, this time without the crds switch:
$ linkerd install kubectl apply -f - namespace/linkerd created clusterrole.rbac.authorization.k8s.io/linkerd-linkerd-identity created clusterrolebinding.rbac.authorization.k8s.io/linkerd-linkerd-identity created serviceaccount/linkerd-identity created clusterrole.rbac.authorization.k8s.io/linkerd-linkerd-destination created clusterrolebinding.rbac.authorization.k8s.io/linkerd-linkerd-destination created serviceaccount/linkerd-destination created secret/linkerd-sp-validator-k8s-tls created validatingwebhookconfiguration.admissionregistration.k8s.io/linkerd-sp-validator-webhook-config created secret/linkerd-policy-validator-k8s-tls created validatingwebhookconfiguration.admissionregistration.k8s.io/linkerd-policy-validator-webhook-config created clusterrole.rbac.authorization.k8s.io/linkerd-policy created clusterrolebinding.rbac.authorization.k8s.io/linkerd-destination-policy created role.rbac.authorization.k8s.io/linkerd-heartbeat created rolebinding.rbac.authorization.k8s.io/linkerd-heartbeat created clusterrole.rbac.authorization.k8s.io/linkerd-heartbeat created clusterrolebinding.rbac.authorization.k8s.io/linkerd-heartbeat created serviceaccount/linkerd-heartbeat created clusterrole.rbac.authorization.k8s.io/linkerd-linkerd-proxy-injector created clusterrolebinding.rbac.authorization.k8s.io/linkerd-linkerd-proxy-injector created serviceaccount/linkerd-proxy-injector created secret/linkerd-proxy-injector-k8s-tls created mutatingwebhookconfiguration.admissionregistration.k8s.io/linkerd-proxy-injector-webhook-config created configmap/linkerd-config created secret/linkerd-identity-issuer created configmap/linkerd-identity-trust-roots created service/linkerd-identity created service/linkerd-identity-headless created deployment.apps/linkerd-identity created service/linkerd-dst created service/linkerd-dst-headless created service/linkerd-sp-validator created service/linkerd-policy created service/linkerd-policy-validator created deployment.apps/linkerd-destination created cronjob.batch/linkerd-heartbeat created deployment.apps/linkerd-proxy-injector created service/linkerd-proxy-injector created secret/linkerd-config-overrides created
Kubernetes will start spinning up the data plane components and you should see the following when you list the pods:
kubernetes-api -------------- can initialize the client can query the Kubernetes API
kubernetes-version ------------------ is running the minimum Kubernetes API version is running the minimum kubectl version
linkerd-existence ----------------- 'linkerd-config' config map exists heartbeat ServiceAccount exist control plane replica sets are ready no unschedulable pods control plane pods are ready cluster networks contains all pods cluster networks contains all services
linkerd-config -------------- control plane Namespace exists control plane ClusterRoles exist control plane ClusterRoleBindings exist control plane ServiceAccounts exist control plane CustomResourceDefinitions exist control plane MutatingWebhookConfigurations exist control plane ValidatingWebhookConfigurations exist proxy-init container runs as root user if docker container runtime is used
linkerd-identity ---------------- certificate config is valid trust anchors are using supported crypto algorithm trust anchors are within their validity period trust anchors are valid for at least 60 days issuer cert is using supported crypto algorithm issuer cert is within its validity period issuer cert is valid for at least 60 days issuer cert is issued by the trust anchor
linkerd-webhooks-and-apisvc-tls ------------------------------- proxy-injector webhook has valid cert proxy-injector cert is valid for at least 60 days sp-validator webhook has valid cert sp-validator cert is valid for at least 60 days policy-validator webhook has valid cert policy-validator cert is valid for at least 60 days
linkerd-version --------------- can determine the latest version cli is up-to-date
control-plane-version --------------------- can retrieve the control plane version control plane is up-to-date control plane and cli versions match
linkerd-control-plane-proxy --------------------------- control plane proxies are healthy control plane proxies are up-to-date control plane proxies and cli versions match
Status check results are
Everything looks good.Setting up the Viz ExtensionAt this point, the required components for the service mesh are setup, but let s also install the viz extension, which provides a good visualization capabilities that will come in handy subsequently. Once again, linkerd uses the same pattern for installing the extension.
$ linkerd viz install kubectl apply -f - namespace/linkerd-viz created clusterrole.rbac.authorization.k8s.io/linkerd-linkerd-viz-metrics-api created clusterrolebinding.rbac.authorization.k8s.io/linkerd-linkerd-viz-metrics-api created serviceaccount/metrics-api created clusterrole.rbac.authorization.k8s.io/linkerd-linkerd-viz-prometheus created clusterrolebinding.rbac.authorization.k8s.io/linkerd-linkerd-viz-prometheus created serviceaccount/prometheus created clusterrole.rbac.authorization.k8s.io/linkerd-linkerd-viz-tap created clusterrole.rbac.authorization.k8s.io/linkerd-linkerd-viz-tap-admin created clusterrolebinding.rbac.authorization.k8s.io/linkerd-linkerd-viz-tap created clusterrolebinding.rbac.authorization.k8s.io/linkerd-linkerd-viz-tap-auth-delegator created serviceaccount/tap created rolebinding.rbac.authorization.k8s.io/linkerd-linkerd-viz-tap-auth-reader created secret/tap-k8s-tls created apiservice.apiregistration.k8s.io/v1alpha1.tap.linkerd.io created role.rbac.authorization.k8s.io/web created rolebinding.rbac.authorization.k8s.io/web created clusterrole.rbac.authorization.k8s.io/linkerd-linkerd-viz-web-check created clusterrolebinding.rbac.authorization.k8s.io/linkerd-linkerd-viz-web-check created clusterrolebinding.rbac.authorization.k8s.io/linkerd-linkerd-viz-web-admin created clusterrole.rbac.authorization.k8s.io/linkerd-linkerd-viz-web-api created clusterrolebinding.rbac.authorization.k8s.io/linkerd-linkerd-viz-web-api created serviceaccount/web created server.policy.linkerd.io/admin created authorizationpolicy.policy.linkerd.io/admin created networkauthentication.policy.linkerd.io/kubelet created server.policy.linkerd.io/proxy-admin created authorizationpolicy.policy.linkerd.io/proxy-admin created service/metrics-api created deployment.apps/metrics-api created server.policy.linkerd.io/metrics-api created authorizationpolicy.policy.linkerd.io/metrics-api created meshtlsauthentication.policy.linkerd.io/metrics-api-web created configmap/prometheus-config created service/prometheus created deployment.apps/prometheus created service/tap created deployment.apps/tap created server.policy.linkerd.io/tap-api created authorizationpolicy.policy.linkerd.io/tap created clusterrole.rbac.authorization.k8s.io/linkerd-tap-injector created clusterrolebinding.rbac.authorization.k8s.io/linkerd-tap-injector created serviceaccount/tap-injector created secret/tap-injector-k8s-tls created mutatingwebhookconfiguration.admissionregistration.k8s.io/linkerd-tap-injector-webhook-config created service/tap-injector created deployment.apps/tap-injector created server.policy.linkerd.io/tap-injector-webhook created authorizationpolicy.policy.linkerd.io/tap-injector created networkauthentication.policy.linkerd.io/kube-api-server created service/web created deployment.apps/web created serviceprofile.linkerd.io/metrics-api.linkerd-viz.svc.cluster.local created serviceprofile.linkerd.io/prometheus.linkerd-viz.svc.cluster.local created
A few seconds later, you should see the following in your pod list:
The viz components live in the linkerd-viz namespace.You can now checkout the viz dashboard:
$ linkerd viz dashboard Linkerd dashboard available at: http://localhost:50750 Grafana dashboard available at: http://localhost:50750/grafana Opening Linkerd dashboard in the default browser Opening in existing browser session.
The Meshed column indicates the workload that is currently integrated with the Linkerd control plane. As you can see, there are no application deployments right now that are running.Injecting the Linkerd Data Plane componentsThere are two ways to integrate Linkerd to the application containers:1 by manually injecting the Linkerd data plane components 2 by instructing Kubernetes to automatically inject the data plane componentsInject Linkerd data plane manuallyLet s try the first option. Below is a simple nginx-app that I will deploy into the cluster:
Back in the viz dashboard, I do see the workload deployed, but it isn t currently communicating with the Linkerd control plane, and so doesn t show any metrics, and the Meshed count is 0:Looking at the Pod s deployment YAML, I can see that it only includes the nginx container:
Let s directly inject the linkerd data plane into this running container. We do this by retrieving the YAML of the deployment, piping it to linkerd cli to inject the necessary components and then piping to kubectl apply the changed resources.
Back in the viz dashboard, the workload now is integrated into Linkerd control plane.Looking at the updated Pod definition, we see a number of changes that the linkerd has injected that allows it to integrate with the control plane. Let s have a look:
At this point, the necessary components are setup for you to explore Linkerd further. You can also try out the jaeger and multicluster extensions, similar to the process of installing and using the viz extension and try out their capabilities.Inject Linkerd data plane automaticallyIn this approach, we shall we how to instruct Kubernetes to automatically inject the Linkerd data plane to workloads at deployment time.We can achieve this by adding the linkerd.io/inject annotation to the deployment descriptor which causes the proxy injector admission hook to execute and inject linkerd data plane components automatically at the time of deployment.
This annotation can also be specified at the namespace level to affect all the workloads within the namespace. Note that any resources created before the annotation was added to the namespace will require a rollout restart to trigger the injection of the Linkerd components.Uninstalling LinkerdNow that we have walked through the installation and setup process of Linkerd, let s also cover how to remove it from the infrastructure and go back to the state prior to its installation.The first step would be to remove extensions, such as viz.
Carpe Jugulum is the 23rd Discworld novel and the 6th witches
novel. I would not recommend reading it before Maskerade, which introduces Agnes.
There are some spoilers for Wyrd
Sisters, Lords and Ladies, and
Maskerade in the setup here and hence in the plot description
below. I don't think they matter that much, but if you're avoiding all
spoilers for earlier books, you may want to skip over this one. (You're
unlikely to want to read it before those books anyway.)
It is time to name the child of the king of Lancre, and in a gesture of
good will and modernization, he has invited his neighbors in Uberwald to
attend. Given that those neighbors are vampires, an open invitation was
perhaps not the wisest choice.
Meanwhile, Granny Weatherwax's invitation has gone missing. On the plus
side, that meant she was home to be summoned to the bedside of a pregnant
woman who was kicked by a cow, where she makes the type of hard decision
that Granny has been making throughout the series. On the minus side, the
apparent snub seems to send her into a spiral of anger at the lack of
appreciation.
Points off right from the start for a plot based on a misunderstanding and
a subsequent refusal of people to simply talk to each other. It is partly
engineered, but still, it's a cheap and irritating plot.
This is an odd book.
The vampires (or vampyres, as the Count wants to use) think of themselves
as modern and sophisticated, making a break from the past by attempting to
overcome such traditional problems as burning up in the sunlight and fear
of religious symbols and garlic. The Count has put his family through
rigorous training and desensitization, deciding such traditional
vulnerabilities are outdated things of the past. He has, however, kept
the belief that vampires are at the top of a natural chain of being,
humans are essentially cattle, and vampires naturally should rule and feed
on the population. Lancre is an attractive new food source. Vampires
also have mind control powers, control the weather, and can put their
minds into magpies.
They are, in short, enemies designed for Granny Weatherwax, the witch
expert in headology. A shame that Granny is apparently off sulking.
Nanny and Agnes may have to handle the vampires on their own, with the
help of Magrat.
One of the things that makes this book odd is that it seemed like
Pratchett was setting up some character growth, giving Agnes a chance to
shine, and giving Nanny Ogg a challenge that she didn't want. This sort
of happens, but then nothing much comes of it. Most of the book is the
vampires preening about how powerful they are and easily conquering
Lancre, while everyone else flails ineffectively. Pratchett does pull
together an ending with some nice set pieces, but that ending doesn't
deliver on any of the changes or developments it felt like the story was
setting up.
We do get a lot of Granny, along with an amusingly earnest priest of Om
(lots of references to Small Gods here,
while firmly establishing it as long-ago history). Granny is one of my
favorite Discworld characters, so I don't mind that, but we've seen Granny
solve a lot of problems before. I wanted to see more of Agnes, who is the
interesting new character and whose dynamic with her inner voice feels
like it has a great deal of unrealized potential.
There is a sharp and condensed version of comparative religion from
Granny, which is probably the strongest part of the book and includes one
of those Discworld quotes that has been widely repeated out of context:
"And sin, young man, is when you treat people as things. Including
yourself. That's what sin is."
"It's a lot more complicated than that "
"No. It ain't. When people say things are a lot more complicated
than that, they means they're getting worried that they won t like the
truth. People as things, that's where it starts."
This loses a bit in context because this book is literally about treating
people as things, and thus the observation feels more obvious when it
arrives in this book than when you encounter it on its own, but it's still
a great quote.
Sadly, I found a lot of this book annoying. One of those annoyances is a
pet peeve that others may or may not share: I have very little patience
for dialogue in phonetically-spelled dialect, and there are two
substantial cases of that here. One is a servant named Igor who speaks
with an affected lisp represented by replacing every ess sound with th,
resulting in lots of this:
"No, my Uncle Igor thtill workth for him. Been thtruck by lightning
three hundred timeth and thtill putth in a full night'th work."
I like Igor as a character (he's essentially a refugee from The
Addams Family, which adds a good counterpoint to the malicious and
arrogant evil of the vampires), but my brain stumbles over words like
"thtill" every time. It's not that I can't decipher it; it's that the
deciphering breaks the flow of reading in a way that I found not at all
fun. It bugged me enough that I started skipping his lines if I couldn't
work them out right away.
The other example is the Nac Mac Feegles, who are... well, in the book,
they're Pictsies and a type of fairy, but they're Scottish Smurfs, right
down to only having one female (at least in this book). They're
entertainingly homicidal, but they all talk like this:
"Ach, hins tak yar scaggie, yer dank yowl callyake!"
I'm from the US and bad with accents and even worse with accents
reproduced in weird spellings, and I'm afraid that I found 95% of
everything said by Nac Mac Feegles completely incomprehensible to the
point where I gave up even trying to read it. (I'm now rather worried
about the Tiffany Aching books and am hoping Pratchett toned the dialect
down a lot, because I'm not sure I can deal with more of this.)
But even apart from the dialect, I thought something was off about the
plot structure of this book. There's a lot of focus on characters who
don't seem to contribute much to the plot resolution. I wanted more of
the varied strengths of Lancre coming together, rather than the focus on
Granny. And the vampires are absurdly powerful, unflappable, smarmy, and
contemptuous of everyone, which makes for threatening villains but also
means spending a lot of narrative time with a Discworld version of
Jacob Rees-Mogg. I
feel like there's enough of that in the news already.
Also, while I will avoid saying too much about the plot, I get very
suspicious when older forms of oppression are presented as good
alternatives to modernizing, rationalist spins on exploitation. I see
what Pratchett was trying to do, and there is an interesting point here
about everyone having personal relationships and knowing their roles (a
long-standing theme of the Lancre Discworld stories). But I think the
reason why there is some nostalgia for older autocracy is that we only
hear about it from stories, and the process of storytelling often creates
emotional distance and a patina of adventure and happy outcomes. Maybe
you can make an argument that classic British imperialism is superior to
smug neoliberalism, but both of them are quite bad and I don't want either
of them.
On a similar note, Nanny Ogg's tyranny over her entire extended clan
continues to be played for laughs, but it's rather unappealing and seems
more abusive the more one thinks about it. I realize the witches are not
intended to be wholly good or uncomplicated moral figures, but I want to
like Nanny, and Pratchett seems to be writing her as likable, even though
she has an astonishing lack of respect for all the people she's related
to. One might even say that she treats them like things.
There are some great bits in this book, and I suspect there are many
people who liked it more than I did. I wouldn't be surprised if it was
someone's favorite Discworld novel. But there were enough bits that
didn't work for me that I thought it averaged out to a middle-of-the-road
entry.
Followed by The Fifth Elephant in publication order. This is the
last regular witches novel, but some of the thematic thread is picked up
by The Wee Free Men, the first Tiffany Aching novel.
Rating: 7 out of 10
Review: What Makes This Book So Great, by Jo Walton
Publisher:
Tor
Copyright:
January 2014
ISBN:
0-7653-3193-4
Format:
Hardcover
Pages:
447
Jo Walton, in addition to being an excellent science fiction and fantasy
writer, is a prodigious reader and frequent participant in on-line SFF
book discussion going back to the Usenet days. This book is a collection
of short essays previously published on Tor.com between July 2008 and February 2011. The unifying theme is that
Walton regularly re-reads her favorite books, and each essay (apart from
some general essays on related topics) is about why this specific book is
one that she re-reads, and (as the title says) what makes it so great.
Searching for the title of one of the essays turns it up on Tor.com still,
so this is one of those collections that you don't have to buy since you
can read its contents on-line for free. That said, it looks like these
essays were from before Tor.com started classifying posts into series, so
it's going to be challenging to track them down in the huge number of
other articles Walton has written for the site. (That said, you can't go
far wrong by reading any of her essays at random.)
I read these essays as they were originally published, so this was also a
re-read for me, but it had been a while. I'm happy to report that they
were just as much fun the second time.
In the introduction and in the final essay of this collection, Walton
draws a distinction between what she's doing, criticism, and reviewing.
As someone else who writes about books (in a far more amateur fashion), I
liked this distinction.
The way I'd characterize it is that criticism is primarily about the work:
taking it apart to see what makes it tick, looking for symbolism and
hidden meanings, and comparing and contrasting other works that are
tackling similar themes. I've often finished a work of criticism and
still had no idea if the author enjoyed reading the work being criticized
or not, since that isn't the point.
Reviewing is assistance to consumers and focuses more on the reader: would
you enjoy this book? Is it enjoyable to read? Does it say something new?
What genre and style is it in, so that you can match that to your tastes?
Talking about books is neither of those things, although it's a bit closer
to reviewing. But the emphasis is on one's personal enjoyment instead of
attempting to review a product for others. When I talk about books with
friends, I talk primarily about what bits I liked, what bits I didn't
like, where the emotional beats were for me, and what interesting things
the book did that surprised me or caught my attention. One can find a
review in there, and sometimes even criticism, but the focus and the
formality is different. (And, to be honest, my reviews are more on the
"talking about the book" side than fully proper reviews.)
These essays are indeed talking about books. They're all re-reads; in
some cases the first re-read, but more frequently the latest of many
re-reads. There are lots of spoilers, which makes for bad reviews (the
target audience of a review hasn't read the book yet) but good fodder for
conversations about books. (The spoilers are mostly marked, but if you're
particularly averse to spoilers, you'll need to read carefully.) Most of
the essays are about a single book, but there are a few on more general
topics, such as Walton's bafflement that anyone would skim a novel.
Since these are re-reads, and the essays collected here are more than a
decade old, the focus is on older books. Some of them are famous: Vinge's
A Fire Upon the Deep and
A Deepness in the Sky, early Le Guin,
Samuel Delaney's SF novels, Salmon Rushdie's Midnight's Children.
Some of them are more obscure. C.J. Cherryh, for example, is a writer who
never seems to get much on-line attention, but who is one of Walton's
favorites.
Most of the essays stand alone or come in small clusters about a writer,
often sprinkled through the book instead of clumped together. (The book
publishes the essays in the same order they originally appeared on
Tor.com.) The two largest groups of essays are re-readings of every book
in Steven Brust's Vlad Taltos universe (including
Brokedown Palace and the
Paarfi books) up to Jhegaala, and every book in Lois McMaster Bujold's Miles
Vorkosigan series up to Diplomatic
Immunity. This is fitting: those are two of the great series of science
fiction, but don't seem to be written about nearly as much as I would
expect.
There are over 130 essays in a 447 page book, so there's a lot of material
here and none of them outlive their welcome. Walton has a comfortable,
approachable style that bubbles with delight and appreciation for books.
I think it's impossible to read this collection without wanting to read
more, and without adding several more books to the ever-teetering to-read
pile.
This is perhaps not the best source of reading recommendations if you
dislike spoilers, although it can be used for that if you read carefully.
But if you love listening to conversations about the genre and talking
about how books bounce off each other, and particularly if you have read
most of these books already or don't mind spoilers, this collection is a
delight. If you're the type of SFF reader who likes reading the reviews
in Locus or is already reading Tor.com, highly recommended.
Rating: 8 out of 10
As usual with these every-two-year posts, probably of direct interest only
to California residents. Maybe the more obscure things we're voting on
will be a minor curiosity to people elsewhere.
Apologies to Planet Debian readers for the explicitly political post
because I'm too lazy to change my blog software to do more fine-grained
post classification. For what it's worth, most of the discussion here
will be about the more fiddly and nuanced things we vote on, not on the
major hot-button proposition.
As in 2020, I'm only going to cover the ballot propositions, as all of the
state-wide and most of the district races are both obvious to me and
boring to talk about. The hyperlocal races are more interesting this
year, but the number of people who would care and who are also reading
this blog is essentially nonexistent, so I won't bother writing them up.
This year, everything except Proposition 1 is an initiative (not put on
the ballot by the legislature), which means I default to voting against
them because they're usually poorly-written.
Proposition 1: YES. Adds reproductive rights to the California
state constitution. I'm fairly sure everyone reading this has already
made up their mind on this topic and certainly nothing will ever change my
mind, so I'll leave it at that.
Proposition 26: YES. This mushes two different things together
in an unhelpful way: allowing sports betting at some racetracks, and
allowing a wider variety of gambling on tribal lands.
I have no strong opinion about the former (I'll get into that more with
the next proposition). For the latter, my starting point is that Native
American tribes are and should be treated like independent governments
with their own laws (which is what we promised them by treaty and then
have systematically and maliciously betrayed ever since). I am not a
citizen of any of the tribes and therefore fundamentally I should not get
a say on this. I'm not a big fan of gambling or of the companies they're
likely to hire to run casinos, but it should be their land and their
decision.
Proposition 27: NO. This, on the other hand, is about on-line
sports betting outside of tribal lands, and looks to be a lot more about
corruption and corporate greed.
I am fairly dubious that outlawing gambling in general is that good of an
idea. I think the harms are overstated given the existence of even wilder
forms of gambling (crypto and financial derivatives) that are perfectly
legal, and I'm always suspicious of attempting to solve social problems
with police and prohibition systems. If there were a ballot proposition
to simply legalize gambling in California, I'd have to think hard about
that.
But this is not that. This requires companies that want to offer on-line
gambling to pay substantial up-front costs (which will restrict this to
only huge gambling companies). In return, they are allowed entry into
what is essentially a state-constructed partial monopoly. As usual,
there's a typical vice tax deal attached where those companies are taxed
to fund some program (in this case, homeless services and mental health
treatment), but these sorts of taxes tend to be regressive in effect. We
could just tax richer people like me to pay for those services instead.
I'm also dubious that the money for homelessness will be used to build
housing, which is what we need to do to address the problem.
Proposition 28: YES. Sets aside money for art and music funding
in public and charter schools. This is a reluctant yes because this sort
of law should not be done via proposition; it should be done through a
normal legislative process that balances all of the priorities for school
funding. But despite the broken process by which this was put forward, it
seems like a reasonable law and no one is opposing it, so okay, fine.
Proposition 29: NO. The attempt to force all dialysis clinics to
have licensed doctors on site is back again.
Everything about the way dialysis health care is provided in California
makes me angry. We should have a state health care system similar to the
NHS. We should open dialysis clinics based on the number of people
requiring dialysis in that area. Every one of them should be unionized.
We absolutely should not allow for-profit companies to have primary
responsibility for basic life-saving medical care like dialysis.
But this proposition does not solve any of those problems, and what it
claims to do is false. It claims that by setting credential requirements
on who has to be on-site at a dialysis clinic, the clinics will become
safer. This is simply not true, for all of the reasons discussed in
Still Not Safe.
This is not how safety works.
The safest person to do dialysis is someone with extensive experience in
performing dialysis, who has seen all the problems and has an intuition
for what to watch out for. That has less to do with credentials than with
good training specifically in dialysis, apprenticeship, and
practice, not to mention reasonable hours and good pay so that the workers
are not stressed. Do I think the private dialysis clinics are likely
doing a good job with this? Hah. (Do I think dialysis clinics run by
large medical non-profits would do a good job with this? Also hah.) But
this would enshrine into law a fundamentally incorrect solution to the
problem that makes dialysis more expensive without addressing any of the
other problems with the system.
It's the same tactic that was used on abortion clinics, with the same
bogus argument that having people with specific credentials on-site would
make them safer. It was false then and it's still false now. I would
agree with better regulation of dialysis clinics, but this specific
regulation is entirely wrong-headed.
Also, while this isn't an overriding factor, I get annoyed when the same
proposition shows up again without substantial changes. For matters of
fundamental rights, okay, sure. But for technical regulation fixes like
this one, the proponents should consider taking no for an answer and
trying a different approach. Like going to the legislature, which is
where this kind of regulation should be designed anyway.
Proposition 30: YES. Raises taxes on the personal income of
extremely rich Californians (over $2 million in income in one year) to
fund various climate change mitigation programs. This is another
reluctant yes vote, because once again this shouldn't be done by
initiative and should be written properly by the legislature. I also
don't like restricting tax revenue to particular programs, which reduce
budget flexibilty to no real purpose. It's not important to me that these
revenues go to these specific programs, although the programs seem like
good ones to fund.
But the reality remains that wealthy Silicon Valley executives are
undertaxed and the only way we can ever manage to raise taxes is through
voting for things like this, so fine.
Proposition 31: NO. The Calfornia legislature banned the sale of
flavored vape products. If NO beats YES on this proposition, that ban
will be overturned.
Drug prohibition has never, ever worked, and yet we keep trying it over
and over again in the hope that this time we'll get a different outcome.
As usual, the pitch in favor of this is all about the children,
specifically the claim that flavored tobacco products are only about
increasing their appeal to kids because... kids like candy? Or something?
I am extremely dubious of this argument; it's obvious to me from walking
around city streets that adults prefer the flavored products as well and
sale to kids is already prohibited and unchanged by this proposition.
I don't like vaping. I wish people would stop, at least around me,
because the scent is obnoxious and the flavored stuff is even more
obnoxious, even apart from whatever health problems it causes. But I'm
never going to vote for drug prohibition because drug prohibition doesn't
work. It just creates a black market and organized crime and makes
society overall worse. Yes, the tobacco companies are some of the worst
corporations on the planet, and I hope they get sued into oblivion (and
ideally prosecuted) for all the lying they do, but I'm still not going to
vote for prohibition. Even the best kind of prohibition that only outlaws
sale and not possession.
Also, secondarily but still significant, bans like this just frustrate a
bunch of people and burn good will and political capital, which we should
be trying to preserve to tackle far more important problems. The politics
of outlawing people's pleasures for their own good are not great. We have
a lot of serious problems to deal with; maybe let's not pick fights we
don't have to.
Pune Rains
Lemme start with a slightly funny picture that tells as much about Pune, my city as anything else does.
Pune- Leave your attitude behind, we have our own
This and similar tags, puns and whatnot you will find if you are entering Pune from the road highway. You can also find similar similar symbols and Puns all over the city and they are partly sarcasm and ironic and partly the truth. Puneities work from the attitude that they know everything rather than nothing, including yours truly . What is the basis of that or why is there such kind of confidence I have no clue or idea, it is what it is. Approximately 24 hrs. ago, apparently we had a cloudburst. What I came to know later is that we got 100 mm of rain. Sharing from local news site. Much more interesting was a thread made on Reddit where many people half-seriously asked where they can buy a boat. One of the reasons being even if it s October, in fact, we passed middle of October and it s still raining. Even today in the evening, it rained for quite a while. As I had shared in a few blog posts before, June where rains should have started, it didn t, it actually started late July or even August, so something has shifted. The current leadership does not believe in Anthropogenic Climate Change or human activity induced climate change even though that is a reality. I could share many links and even using the term above should give links to various studies. Most of the people who are opposed to it are either misinformed or influenced from the fossil fuel industry. Again, could share many links, but will share just one atm. I have talked to quite a few people about it but nobody has ever been able to give a convincing answer as to why GM had to crush the cars. Let s even take the argument that it was the worst manufactured car in history and there have been quite a few, have the others been crushed? If not, then the reason shared or given by most people sounds hollow. And if you look into it, they had an opportunity that they let go, and now most of them are scrambling and yet most of the legacy auto manufacturers will be out of existence if they don t get back into the game in the next 2-3 years. There have been a bunch of announcements but we are yet to see. The Chinese though have moved far ahead, although one has to remark that they have been doing that for the last decade, so they have a 10-year head start, hardly surprising then.
But I need to get back to the subject, another gentleman on Reddit remarked that if you start to use boat, and others start to use boat, then the Govt. will tax it. In fact, somebody had shared the below the other day
Different types of taxes collected by GOI
Many of the taxes that I have shared above are by the Modi Govt. who came on the platform, manifesto that once they come to power they will reduce taxes for the common man, they have reduced taxes but only for the Corporates. For the common man, the taxes have only gone up, both direct tax and indirect tax. Any reference to the Tory party who have also done similar things and have also shared that it is labor who had done large expenditures even though they have been 8 years in power, I am sure for most is purely coincidental. Incidentally, that is the same tack that was taken even by the Republican party. They all like to give tax benefits to the 1% while for the rest is austerity claiming some reason, even if it has been proven to be false.
Corporate Tax Rate, Revenue Loss to Govt.
The figures mentioned above are findings of parliamentary panel so nobody can accuse of anybody having a bias. Also, I probably had shared this but still feel the need to re-share it as people still believe that 2G scam happened even though there are plenty of arguments I can share to prove how it was all a fabricated lie.
Vinod Rai Mafinama in Uttarakhand High Court.Part 2 of the same Mafinama.
How pathetic Mr. Rai s understanding of economics is can be gauged from the fact that he was made Chairman of IDFC and subsequently had to be thrown out. That whole lie was engineered to throw UPA out and it worked. There are and have been many such coincidences happening over the last 8 years, parallel stories happening in India and UK.
This was just yesterday, about a year back Air India was given back to the Tatas, There was controversy about the supposed auctions as only Indigo was the only other party allowed to be at auction but not allowed to buy but more as a spectator as they already have 60% of the Indian civil aviation market. And there was lot of cheering from the Govt. side that finally Air India has been bought home to its true owners, the Tatas. The Tatas too started cheering and sharing how they will take down all the workers, worker unions and everything will be happy glory within a year. In fact, just couple of days back they shared new plans. Btw for the takeover of Air India, they had bought loans from the Banks and they are in the category of too big to fail. As I have shared couple of times before, RBI has not shared any inspection reports of nationalized or private banks after 2013/14. While by law, RBI is supposed to do inspection reports every 3 months and share them in the Public domain. And if you ask any of their supporter, for everything they will say UPA did x or y, which only goes to show morally bankrupt the present Govt. is. Coming back to the topic, before I forget, the idea of sharing their plans is so that they can again borrow money from the banks.
But that is not the only story. Just one day back, Smita Prakash, one of the biggest cheerleaders of the present Govt. (she is the boss at Asia News International (ANI)) posted how Air India had treated her sister and other 21 passengers. Basically, they had bought business tickets but the whole cabin was dirty, they complained and they were forced to sit in economy class, not just her cousin sister but the other 21 odd passengers too. Of course Ms Smita became calm as her sister was given free air tickets on Vistara and other goodies. Of course, after that she didn t post anything about the other 21 odd passengers after that. And yes, I understand she is supposed to be a reporter but as can be seen from the twitter thread, there is or was no follow up. Incidentally, she is one of many who has calling others about Revdi culture (freebies to the masses.) but guess that only applies to other people not her or her sister. Again, if there are any coincidences of similar nature in the UK or when Trump was P.M. of the U.S. they are just coincidental .
The Uncosted budget
India and the UK have many parallels, it s mind boggling. Before we get into the nitty-gritties, saw something that would be of some interest to the people here.
For those who might not be able to see above, apparently there is place in UK called Tufton Street where there are quite a few organizations that are shadowy and whose finances are not known as to how they are financed. Ms. Truss and quite a few of the people in the cabinet are from the same shadowy organizations. Mr. Kwasi Kwarteng, the just-departed chancellor is and was part of the same group.
Now even for me it was a new term to learn and understand what is an uncosted budget is. To make it much more easier I share the example using a common person who goes to the bank for a loan
Now M/S X wants a loan of say Rs. 1000/- for whatever reason. He/she/they go to the bank and asks give me a loan of say INR 1000/- The banks asks them to produce a statement of accounts to show what their financial position is. They produce a somewhat half-filled statement of accounts In which all liabilities are shown, but incomes are not. The bank says you already have so much liabilities, how are you gonna pay those, accounts have to be matched otherwise you are not solvent. M/s X adamantly refuses to do any changes citing that they don t need to.
At this point, M/s X credit rating goes down and nobody in the market will give them a loan. At the same time, the assets they had held, their value also depreciates because it became known that they can t act responsibly. So whose to say whether or not M/s X has those number of assets and priced them accurately. But the drama doesn t end there, M/s X says this is the responsibility of actually Mr. Z ( cue The Bank of England) as they are my accountant/lawyer etc. M/s Z says as any lawyer/accountant should. This is not under my remit. If the clients either gives incomplete information or false information or whatever then it is their responsibility not mine. And in fact, the Chancellor is supposed to be the one who is given the responsibility of making the budget.
The Chancellor is very similar to our Finance Minister. Because the UK has constitutional monarchy, I am guessing the terms are slightly different, otherwise the functionality seems to be the same. For two weeks, there was lot of chaos, lot of pension funds lost quite a bit in the market and in the end Mr. Kwasi Kwarteng was ousted out of the job. Incredibly, the same media and newspapers who had praised Mr. Kwarteng just few weeks back as the best Tory budget, they couldn t wait to bury him. And while I have attempted to simplify what happened, the best explanation of what has happened can be found in an article from the guardian. Speculation is rife in the UK as to who s ruling atm as the new Chancellor has reversed almost all the policies that Ms. Truss had bought and she is now more or less a figurehead. Mr. Hunt, the new chancellor doesn t have anybody behind him. Apparently, the gentleman wanted to throw his hat the ring in the Tory leadership contest that was held about a month back and he couldn t get 20 MP s to support him. Another thing that is different between UK and India is that in UK by law the PM has to answer questions put up to him or her by the opposition leaders. That is the way accountability is measured there. This is known as PMQ s or Prime Minister Questions and Answers. One can just go to YouTube or any streaming service and give Liz Truss and PMQ s and if they are interested of a certain date, give a date and they can see how she answered the questions thrown at her. Unfortunately, all she could do in both times were non-answers. In fact, the Tories seem to be using some of Labor s policies after they had bad-mouthed the same policies. Politics of right-wing both in the UK and the US seems so out of touch with the people whom they are supposed to protect and administer. An article about cyclists which is sort of half-truth, half irony shows how screwed up the policies are of the RW (right-wing). Now they are questions about the pensions triple-lock. Sadly, it is the working class who would suffer the most, most of the rich have already moved their money abroad several years ago. The Financial Times, did share a video about how things have been unfolding
Seems Ms. Truss forgot to add Financial Times in the list of anti-growth coalition she is so fond of. Also, the Tory party seems to want to create more tax havens in the UK and calling them investment zones. Of course, most of the top tax havens are situated around the UK itself. I wouldn t go more into as that would probably require its own article, although most of that information is all in public domain.
Fracking
I don t really want to take much time as the blog post has become long. There have been many articles written why Fracking is bad and that is why even the Tories had put in their Manifesto that they won t allow Fracking but apparently, today they are trying to reopen Fracking. And again how bad it is and can be found out by the article in Guardian.
I make a lot of plots, and the fragmentation of tools in this space really
bugs me. People writing Python code mostly use matplotlib, R people use
ggplot2. MS people use the internal Excel thing. I've seen people use
gtkdatabox for GTK widgets, rrdtool for logging, qcustomplot for qt. And
so on. This is really unhelpful, and it would benefit everybody if there was a
single solid plotting backend with lots of bindings to different languages and
tools.
For my own usage, I've been fighting this quixotic battle, using gnuplot as
the plotting backend for all my use cases. gnuplot is
very mature
stable
fast
powerful
supported on every (with reason) platform
supports lots and lots of output backends
There are some things it can't do, but those can be added, and I haven't felt it
to be limiting in over 20 years of using it.
I rarely use it directly, and usually interact with it through one of
I wrote all of these, although the Perl library was taken over by others long
ago.
Recently I needed a plotting widget for an FLTK program written in Python. It
would be great if there was a C++ class deriving from Fl_Widget that would be
wrapped by pyfltk, but there isn't.
But it turns out that I already had all the tools to quickly hack together
something that mostly works. This is a not-ready-for-primetime hack, but it
works so well, I'd like to write it up. Hopefully this will be done "properly"
someday.
Approach
Alright. So here I'm trying to tie together a Python program, gnuplot output
and an FLTK widget. This is a Python program, I can use gnuplotlib to talk
to the gnuplot backend. In a perfect world, gnuplot would ship a backend
interfacing to FLTK. But it doesn't. What it does do is to ship an x11
backend that makes plots with X11 commands, and it allows these commands to be
directed to an arbitrary X11 window. So we
Make an FLTK widget that simply creates an X11 window, and never actually
draws into it
Tell gnuplot to plot into this window
Demo
This is really simple, and works shockingly well. Here's my Fl_gnuplotlib
widget:
#!/usr/bin/python3import sys
import gnuplotlib as gp
import fltk
classFl_Gnuplotlib_Window(fltk.Fl_Window):
def__init__(self, x,y,w,h, **plot_options):
super().__init__(x,y,w,h)
self.end()
self._plot = Noneself._delayed_plot_options = Noneself.init_plot(**plot_options)
definit_plot(self, **plot_options):
if'terminal'in plot_options:
raiseException("Fl_Gnuplotlib_Window needs control of the terminal, but the user asked for a specific 'terminal'")
ifself._plot isnotNone:
self._plot = Noneself._delayed_plot_options = Nonexid = fltk.fl_xid(self)
if xid == 0:
# I don't have an xid (yet?), so I delay the initself._delayed_plot_options = plot_options
return# will barf if we already have a terminal
gp.add_plot_option(plot_options,
terminal = f'x11 window "0x xid:x "')
self._plot = gp.gnuplotlib(**plot_options)
defplot(self, *args, **kwargs):
ifself._plot isNone:
ifself._delayed_plot_options isNone:
raiseException("plot has not been initialized")
self.init_plot(**self._delayed_plot_options)
ifself._plot isNone:
raiseException("plot has not been initialized. Delayed initialization failed")
self._plot.plot(*args, **kwargs)
Clearly it's simply making an Fl_Window, and pointing gnuplotlib at it. And
a sample application that uses this widget:
This is nice and simple. Exactly what a program using a widget to make a plot
(while being oblivious to the details) should look like. It creates a window,
places the one plotting widget into it, and cycles the plot inside it at 1Hz
(cycling between a parabola, a sinusoid and a line). Clearly we could place
other UI elements around it, or add more plots, or whatever.
The output looks like this:
To run you need to apt install python3-numpysane python3-gnuplotlib
python3-fltk. If running an older distro on a non-Debian-based distro, you
should grab those from source.
Discussion
This works. But it's a hack. Some issues:
This plotting widget currently can output only. It can make whatever plot we
like, but it cannot accept UI input from the container program in any way
More than that, when focused it completely replaces the FLTK event logic for
that window. So all keyboard input is swallowed, including the keys to access
FLTK menus, to exit the application, etc, etc.
This approach requires us to use the x11 gnuplot terminal. This works, but
it's no longer the terminal preferred by the gnuplot devs, and it it's
maintained as vigilantly as the others.
And it has bugs. For instance, asking to plot into a window that doesn't yet
exist, causes it to create a new window. This breaks FLTK applications that
start up and create a plot immediately. Here's a mailing list thread
discussing these issues.
So this is a very functional hack, but it's still hack. And it feels like
making this solid will take a lot of work. Maybe. I'll push more on this as I
need it. Stay tuned!
The purpose of this post is to demonstrate a first approach to the analysis of multiwavelength kinetic data, like those obtained using stopped-flow data. To practice, we will use data that were acquired during the stopped flow practicals of the MetBio summer school from the FrenchBIC. During the practicals, the student monitored the reaction of myoglobin (in its Fe(III) state) with azide, which yields a fast and strong change in the absorbance spectrum of the protein, which was monitored using a diode array. The data is publicly available on zenodo.
Aims of this tutorial
The purpose of this tutorial is to teach you to use the free softwareQSoas to run a simple, multiwavelength exponential fit on the data, and to look at the results. This is not a kinetics lecture, so that it will not go in depth about the use of the exponential fit and its meaning.
Getting started: loading the file
First, make sure you have a working version of QSoas, you can download them (for free) there. Then download the data files from zenodo. We will work only on the data file Azide-1.25mm_001.dat, but of course, the purpose of this tutorial is to enable you to work on all of them. The data files contain the time evolution of the absorbance for all wavelengths, in a matrix format, in which each row correpond to a time point and each column to a wavelength.
Start QSoas, and launch the command:
QSoas> load /comments='"'
Then, choose the Azide-1.25mm_001.dat data file. This should bring up a horizontal red line at the bottom of the data display, with X values between about 0 and 2.5. If you zoom on the red line with the mouse wheel, you'll realize it is data. The /comments='"' part is very important since it allows the extraction of the wavelength from the data. We will look at what it means another day. At this stage, you can look at the loaded data using the command:
QSoas> edit
You should have a window looking like this:
The rows each correspond to a data point displayed on the window below. The first column correspond to the X values, the second the Y values, and all the other ones to extra Y columns (they are not displayed by default). What is especially interesting is the first row, which contains a nan as the X value and what is obviously the wavelength for all the Y values. To tell that QSoas should take this line as the wavelength (which will be the perpendicular coordinate, the coordinate of the other direction of the matrix), first close the edit window and run:
QSoas> set-perp /from-row=0
Splitting and fitting
Now, we have a single dataset containing a lot of Y columns. We want to fit all of them simultaneously with a (mono) exponential fit. For that, we first need to split the big matrix into a series of X,Y datasets (because fitting only works on the first Y). This is possible by running:
QSoas> expand /style=red-to-blue /flags=kinetics
Your screen should now look like this:
You're looking at the kinetics at all wavelengths at the same time (this may take some time to display on your computer, it is after all a rather large number of data points). The /style=red-to-blue is not strictly necessary, but it gives the red to blue color gradient which makes things easier to look at (and cooler !). The /flags=kinetics is there to attach a label (a flag) to the newly created datasets so we can easily manipulate all of them at the same time. Then it's time to fit, with the following command:
QSoas> mfit-exponential-decay flagged:kinetics
This should bring up a new window. After resizing it, you should have something that looks like this:
The bottom of the fit window is taken by the parameters, each with two checkboxes on the right to set them fixed (i.e. not determined by the fitting mechanism) and/or global (i.e. with a single value for all the datasets, here all the wavelengths). The top shows the current dataset along with the corresponding fit (in green), and, below, the residuals. You can change the dataset by clicking on the horizontal arrows or using Ctrl+PgUp or Ctrl+PgDown (keep holding it to scan fast). See the Z = 728.15 showing that QSoas has recognized that the currently displayed dataset corresponds to the wavelength 728.15. The equation fitted to the data is: $$y(x) = A_\infty + A_1 \times \exp -(x - x_0)/\tau_1$$
In this case, while the \(A_1\) and \(A_\infty\) parameters clearly depend on the wavelength, the time constant of evolution should be independent of wavelength (the process happens at a certain rate regardless of the wavelength we're analyzing), so that the \(\tau_1\) parameter should be common for all the datasets/wavelengths. Just click on the global checkbox at the right of the tau_1 parameter, make sure it is checked, and hit the Fit button...
The fit should not take long (less than a minute), and then you end up with the results of the fits: all the parameters. The best way to look at the non global parameters like \(A_1\) and \(A_\infty\) is to use the Show Parameters item from the Parameters menu. Using it and clicking on A_inf too should give you a display like this one:
The A_inf parameter corresponds to the spectum at infinite time (of azide-bound heme), while the A_1 parameter corresponds to the difference spectrum between the initial (azide-free) and final (azide-bound) states.
Now, the fit is finished, you can save the parameters if you want to reload them in a later fit by using the Parameters/Save menu item or export them in a form more suitable for plotting using Parameters/Export (although QSoas can also display and the parameters saved using Save). This concludes this first approach to fitting the data. What you can do is
look at the depence of the tau_1 parameter as a function of the azide concentration;
try fitting more than one exponential, using for instance:
How to read the code above
All the lines starting by QSoas> in the code areas above are meant to be typed into the QSoas command line (at the bottom of the window), and started by pressing enter at the end. You must remove the QSoas> bit. The other lines (when applicable) show you the response of QSoas, in the terminal just above the command-line. You may want to play with the QSoas tutorial to learn more about how to interact with QSoas.
About QSoas
QSoas is a powerful open source data analysis program that focuses on flexibility and powerful fitting capacities. It is released under the GNU General Public License. It is described in Fourmond, Anal. Chem., 2016, 88 (10), pp 5050 5052. Current version is 3.1. You can freely (and at no cost) download its source code or precompiled versions for MacOS and Windows there. Alternatively, you can clone from the GitHub repository.
Contact: find my email address there, or contact me on LinkedIn.
Dowry
Few days back, had posted about the movie Raksha Bandhan and whatever I felt about it. Sadly, just couple of days back, somebody shared this link. Part of me was shocked and part of me was not. Couple of acquaintances of mine in the past had said the same thing for their daughters. And in such situations you are generally left speechless because you don t know what the right thing to do is. If he has shared it with you being an outsider, how many times he must have told the same to their wife and daughters? And from what little I have gathered in life, many people have justified it on similar lines. And while the protests were there, sadly the book was not removed. Now if nurses are reading such literature, how their thought process might be forming, you can tell :(. And these are the ones whom we call for when we are sick and tired :(. And I have not taken into account how the girls/women themselves might be feeling. There are similar things in another country but probably not the same, nor the same motivations though although feeling helplessness in both would be a common thing.
But such statements are not alone. Another gentleman in slightly different context shared this as well
The above is a statement shared in a book recommended for CTET (Central Teacher s Eligibility Test that became mandatory to be taken as the RTE (Right To Education) Act came in.). The statement says People from cold places are white, beautiful, well-built, healthy and wise. And people from hot places are black, irritable and of violent nature.
Now while I can agree with one part of the statement that people residing in colder regions are more fair than others but there are loads of other factors that determine fairness or skin color/skin pigmentation. After a bit of search came to know that this and similar articulation have been made in an idea/work called Environmental Determinism . Now if you look at that page, you would realize this was what colonialism is and was all about. The idea that the white man had god-given right to rule over others. Similarly, if you are fair, you can lord over others. Seems simplistic, but yet it has a powerful hold on many people in India. Forget the common man, this thinking is and was applicable to some of our better-known Freedom fighters. Pune s own Bal Gangadhar Tilak The Artic Home to the Vedas. It sort of talks about Aryans and how they invaded India and became settled here. I haven t read or have access to the book so have to rely on third-party sources. The reason I m sharing all this is that the right-wing has been doing this myth-making for sometime now and unless and until you put a light on it, it will continue to perpetuate . For those who have read this blog, do know that India is and has been in casteism from ever. They even took the fair comment and applied it to all Brahmins. According to them, all Brahmins are fair and hence have god-given right to lord over others. What is called the Eton boy s network serves the same in this casteism. The only solution is those idea under limelight and investigate. To take the above, how does one prove that all fair people are wise and peaceful while all people black and brown are violent. If that is so, how does one count for Mahatma Gandhi, Martin Luther King Junior, Nelson Mandela, Michael Jackson the list is probably endless. And not to forget that when Mahatma Gandhiji did his nonviolent movements either in India or in South Africa, both black and brown people in millions took part. Similar examples of Martin Luther King Jr. I know and read of so many non-violent civl movements that took place in the U.S. For e.g. Rosa Parks and the Montgomery Bus Boycott. So just based on these examples, one can conclude that at least the part about the fair having exclusive rights to being fair and noble is not correct.
Now as far as violence goes, while every race, every community has had done violence in the past or been a victim of the same. So no one is and can be blameless, although in light of the above statement, the question can argumentated as to who were the Vikings? Both popular imagination and serious history shares stories about Vikings. The Vikings were somewhat nomadic in nature even though they had permanent settlements but even then they went on raids, raped women, captured both men and women and sold them at slaves. So they are what pirates came to be, but not the kind Hollywood romanticizes about. Europe in itself has been a tale in conflict since time immemorial. It is only after the formation of EU that most of these countries stopped fighting each other From a historical point perspective, it is too new. So even the part of fair being non-violent dies in face of this evidence. I could go on but this is enough on that topic.
Railways and Industrial Action around the World.
While I have shared about Railways so many times on this blog, it continues to fascinate me that how people don t understand the first things about Railways. For e.g. Railways is a natural monopoly. What that means is and you can look at all and any type of privatization around the world, you will see it is a monopoly. Unlike the road or Skies, Railways is and would always be limited by infrastructure and the ability to have new infrastructure. Unlike in road or Skies (even they have their limits) you cannot run train services on a whim. At any particular point in time, only a single train could and should occupy a stretch of Railway network. You could have more trains on one line, but then the likelihood of front or rear-end collisions becomes a real possibility. You also need all sorts of good and reliable communications, redundant infrastructure so if one thing fails then you have something in place. The reason being a single train can carry anywhere from 2000 to 5000 passengers or more. While this is true of Indian Railways, Railways around the world would probably have some sort of similar numbers.It is in this light that I share the below videos.
To be more precise, see the fuller video
Now to give context to the recording above, Mike Lynch is the general secretary at RMT. For those who came in late, both UK and the U.S. have been threatened by railway strikes. And the reason for the strikes or threat of strikes is similar. Now from the company perspective, all they care is to invest less and make the most profits that can be given to equity shareholders. At the same time, they have freezed the salaries of railway workers for the last 3 years. While the politicians who were asking the questions, apparently gave themselves raise twice this year. They are asking them to negotiate at 8% while inflation in the UK has been 12.3% and projected to go higher. And it is not only the money. Since the 1980s when UK privatized the Railways, they stopped investing in the infrastructure. And that meant that the UK Railway infrastructure over period of time started getting behind and is even behind say Indian Railways which used to provide most bang for the buck. And Indian Railways is far from ideal. Ironically, most of the operators on UK are nationalized Railways of France, Germany etc. but after the hard Brexit, they too are mulling to cut their operations short, they have too There is also the EU Entry/Exit system that would come next year.
Why am I sharing about what is happening in UK Rail, because the Indian Government wants to follow the same thing, and fooling the public into saying we would do it better. What inevitably will happen is that ticket prices go up, people no longer use the service, the number of services go down and eventually they are cancelled. This has happened both in Indian Railways as well as Airlines. In fact, GOI just recently announced a credit scheme just a few days back to help Airlines stay afloat. I was chatting with a friend who had come down to Pune from Chennai and the round-trip cost him INR 15k/- on that single trip alone. We reminisced how a few years ago, 8 years to be precise, we could buy an Air ticket for 2.5k/- just a few days before the trip and did it. I remember doing/experiencing at least a dozen odd trips via air in the years before 2014. My friend used to come to Pune, almost every weekend because he could afford it, now he can t do that. And these are people who are in the above 5-10% of the population. And this is not just in UK, but also in the United States. There is one big difference though, the U.S. is mainly a freight carrier while the UK Railway Operations are mostly passenger based. What was and is interesting that Scotland had to nationalize their services as they realized the Operators cannot or will not function when they were most needed. Most of the public even in the UK seem to want a nationalized rail service, at least their polls say so. So, it would definitely be interesting to see what happens in the UK next year.
In the end, I know I promised to share about books, but the above incidents have just been too fascinating to not just share the news but also share what I think about them. Free markets function good where there is competition, for example what is and has been happening in China for EV s but not where you have natural monopolies. In all Railway privatization, you have to handover the area to one person, then they have no motivation. If you have multiple operators, then there would always be haggling as to who will run the train and at what time. In either scenario, it doesn t work and raises prices while not delivering anything better
I do take examples from UK because lot of things are India are still the legacy of the British. The whole civil department that was created in 1953 is/was a copy of the British civil department at that time and it is to this day.
P.S. Just came to know that the UK Chancellor Kwasi Kwarteng was just sacked as UK Chancellor. I do commend Truss for facing the press even though she might be dumped a week later unlike our PM who hasn t faced a single press conference in the last 8 odd years.
https://www.youtube.com/watch?v=oTP6ogBqU7of
The difference in Indian and UK politics seems to be that the English are now asking questions while here in India, most people are still sleeping without a care in the world.
Another thing to note Minidebconf Palakkad is gonna happen 12-13th November 2022. I am probably not gonna go but would request everyone who wants to do something in free software to attend it. I am not sure whether I would be of any use like this and also when I get back, it would be an empty house. But for people young and old, who want to do anything with free/open source software it is a chance not to be missed. Registration of the same closes on 1st of November 2022. All the best, break a leg
Just read this, beautifully done.
The Guile bindings for GnuTLS has been part of GnuTLS since spring 2007 when Ludovic Court s contributed it after some initial discussion. I have been looking into getting back to do GnuTLS coding, and during a recent GnuTLS meeting one topic was Guile bindings. It seemed like a fairly self-contained project to pick up on. It is interesting to re-read the old thread when this work was included: some of the concerns brought up there now have track record to be evaluated on. My opinion that the cost of introducing a new project per language binding today is smaller than the cost of maintaining language bindings as part of the core project. I believe the cost/benefit ratio has changed during the past 15 years: introducing a new project used to come with a significant cost but this is no longer the case, as tooling and processes for packaging have improved. I have had similar experience with Java, C# and Emacs Lisp bindings for GNU Libidn as well, where maintaining them centralized slow down the pace of updates. Andreas Metzler pointed to a similar conclusion reached by Russ Allbery.
There are many ways to separate a project into two projects; just copying the files into a new git repository would have been the simplest and was my original plan. However Ludo mentioned git-filter-branch in an email, and the idea of keeping all git history for some of the relevant files seemed worth pursuing to me. I quickly found git-filter-repo which appears to be the recommend approach, and experimenting with it I found a way to filter out the GnuTLS repo into a small git repository that Guile-GnuTLS could be based on. The commands I used were the following, if you want to reproduce things.
I debated with myself back and forth whether to include some files that would be named the same in the new repository but would share little to no similar lines, for example configure.ac, Makefile.am not to mention README and NEWS. Initially I thought it would be nice to preserve the history for all lines that went into the new project, but this is a subjective judgement call. What brought me over to a more minimal approach was that the contributor history and attribution would be quite strange for the new repository: Should Guile-GnuTLS attribute the work of the thousands of commits to configure.ac which had nothing to do with Guile? Should the people who wrote that be mentioned as contributor of Guile-GnuTLS? I think not.
The next step was to get a reasonable GitLab CI/CD pipeline up, to make sure the project builds on some free GNU/Linux distributions like Trisquel and PureOS as well as the usual non-free distributions like Debian and Fedora to have coverage of dpkg and rpm based distributions. I included builds on Alpine and ArchLinux as well, because they tend to trigger other portability issues. I wish there were GNU Guix docker images available for easy testing on that platform as well. The GitLab CI/CD rules for a project like this are fairly simple.
To get things out of the door, I tagged the result as v3.7.9 and published a GitLab release page for Guile-GnuTLS that includes OpenPGP-signed source tarballs manually uploaded built on my laptop. The URLs for these tarballs are not very pleasant to work with, and discovering new releases automatically appears unreliable, but I don t know of a better approach.
To finish this project, I have proposed a GnuTLS merge request to remove all Guile-related parts from the GnuTLS core.
Doing some GnuTLS-related work again felt nice, it was quite some time ago so thank you for giving me this opportunity. Thoughts or comments? Happy hacking!
Version 0.3.14 of the Rblpapi package arrived on CRAN earlier today. Rblpapi provides a direct interface between R and the Bloomberg Terminal via the C++ API provided by Bloomberg (but note that a valid Bloomberg license and installation is required).
This is the fourteenth release since the package first appeared on CRAN in 2016. It comprises a nice PR from Robert Harlow extending support to B-PIPE authentication (for those who have it) along with a few fixes made since the last release in January. The last one provided from a kind assist by Tomas Kalibera who pointed out how to overcome an absolute rpath dynamic linker instruction (and as I noticed noticed something I already did in another package ah well) so that we no longer require StagedInstall: yes.
The detailed list of changes follow below.
Changes in Rblpapi version 0.3.14 (2022-10-05)
Build configuration was generalized to consider local copies of library and headers (Dirk in #362)
A ticker symbol was corrected (Dirk in #368 addressing an issue #366 and #367)
Support for B-PIPE was added (Robert Harlow in #369 closing #342)
The package no longer requires staged installation thanks to an assist from Tomas Kalibera (Dirk in #373)
The retired package fts is no longer suggested (Dirk in #374 closing #372)
Courtesy of my CRANberries, there is also a diffstat report for the this release. As always, more detailed information is on the Rblpapi page. Questions, comments etc should go to the issue tickets system at the GitHub repo.
If you like this or other open-source work I do, you can now sponsor me at GitHub.
Fast laptops are expensive, cheap laptops are slow. But even a fast laptop is slower than a decent workstation, and if your developers want a local build environment they're probably going to want a decent workstation. They'll want a fast (and expensive) laptop as well, though, because they're not going to carry their workstation home with them and obviously you expect them to be able to work from home. And in two or three years they'll probably want a new laptop and a new workstation, and that's even more money. Not to mention the risks associated with them doing development work on their laptop and then drunkenly leaving it in a bar or having it stolen or the contents being copied off it while they're passing through immigration at an airport. Surely there's a better way?
This is the thinking that leads to "Let's give developers a Chromebook and a VM running in the cloud". And it's an appealing option! You spend far less on the laptop, and the VM is probably cheaper than the workstation - you can shut it down when it's idle, you can upgrade it to have more CPUs and RAM as necessary, and you get to impose all sorts of additional neat security policies because you have full control over the network. You can run a full desktop environment on the VM, stream it to a cheap laptop, and get the fast workstation experience on something that weighs about a kilogram. Your developers get the benefit of a fast machine wherever they are, and everyone's happy.
But having worked at more than one company that's tried this approach, my experience is that very few people end up happy. I'm going to give a few reasons here, but I can't guarantee that they cover everything - and, to be clear, many (possibly most) of the reasons I'm going to describe aren't impossible to fix, they're simply not priorities. I'm also going to restrict this discussion to the case of "We run a full graphical environment on the VM, and stream that to the laptop" - an approach that only offers SSH access is much more manageable, but also significantly more restricted in certain ways. With those details mentioned, let's begin.
The first thing to note is that the overall experience is heavily tied to the protocol you use for the remote display. Chrome Remote Desktop is extremely appealing from a simplicity perspective, but is also lacking some extremely key features (eg, letting you use multiple displays on the local system), so from a developer perspective it's suboptimal. If you read the rest of this post and want to try this anyway, spend some time working with your users to find out what their requirements are and figure out which technology best suits them.
Second, let's talk about GPUs. Trying to run a modern desktop environment without any GPU acceleration is going to be a miserable experience. Sure, throwing enough CPU at the problem will get you past the worst of this, but you're still going to end up with users who need to do 3D visualisation, or who are doing VR development, or who expect WebGL to work without burning up every single one of the CPU cores you so graciously allocated to their VM. Cloud providers will happily give you GPU instances, but that's going to cost more and you're going to need to re-run your numbers to verify that this is still a financial win. "But most of my users don't need that!" you might say, and we'll get to that later on.
Next! Video quality! This seems like a trivial point, but if you're giving your users a VM as their primary interface, then they're going to do things like try to use Youtube inside it because there's a conference presentation that's relevant to their interests. The obvious solution here is "Do your video streaming in a browser on the local system, not on the VM" but from personal experience that's a super awkward pain point! If I click on a link inside the VM it's going to open a browser there, and now I have a browser in the VM and a local browser and which of them contains the tab I'm looking for WHO CAN SAY. So your users are going to watch stuff inside their VM, and re-compressing decompressed video is going to look like shit unless you're throwing a huge amount of bandwidth at the problem. And this is ignoring the additional irritation of your browser being unreadable while you're rapidly scrolling through pages, or terminal output from build processes being a muddy blur of artifacts, or the corner case of "I work for Youtube and I need to be able to examine 4K streams to determine whether changes have resulted in a degraded experience" which is a very real job and one that becomes impossible when you pass their lovingly crafted optimisations through whatever codec your remote desktop protocol has decided to pick based on some random guesses about the local network, and look everyone is going to have a bad time.
The browser experience. As mentioned before, you'll have local browsers and remote browsers. Do they have the same security policy? Who knows! Are all the third party services you depend on going to be ok with the same user being logged in from two different IPs simultaneously because they lost track of which browser they had an open session in? Who knows! Are your users going to become frustrated? Who knows oh wait no I know the answer to this one, it's "yes".
Accessibility! More of your users than you expect rely on various accessibility interfaces, be those mechanisms for increasing contrast, screen magnifiers, text-to-speech, speech-to-text, alternative input mechanisms and so on. And you probably don't know this, but most of these mechanisms involve having accessibility software be able to introspect the UI of applications in order to provide appropriate input or expose available options and the like. So, I'm running a local text-to-speech agent. How does it know what's happening in the remote VM? It doesn't because it's just getting an a/v stream, so you need to run another accessibility stack inside the remote VM and the two of them are unaware of each others existence and this works just as badly as you'd think. Alternative input mechanism? Good fucking luck with that, you're at best going to fall back to "Send synthesized keyboard inputs" and that is nowhere near as good as "Set the contents of this text box to this unicode string" and yeah I used to work on accessibility software maybe you can tell. And how is the VM going to send data to a braille output device? Anyway, good luck with the lawsuits over arbitrarily making life harder for a bunch of members of a protected class.
One of the benefits here is supposed to be a security improvement, so let's talk about WebAuthn. I'm a big fan of WebAuthn, given that it's a multi-factor authentication mechanism that actually does a good job of protecting against phishing, but if my users are running stuff inside a VM, how do I use it? If you work at Google there's a solution, but that does mean limiting yourself to Chrome Remote Desktop (there are extremely good reasons why this isn't generally available). Microsoft have apparently just specced a mechanism for doing this over RDP, but otherwise you're left doing stuff like forwarding USB over IP, and that means that your USB WebAuthn no longer works locally. It also doesn't work for any other type of WebAuthn token, such as a bluetooth device, or an Apple TouchID sensor, or any of the Windows Hello support. If you're planning on moving to WebAuthn and also planning on moving to remote VM desktops, you're going to have a bad time.
That's the stuff that comes to mind immediately. And sure, maybe each of these issues is irrelevant to most of your users. But the actual question you need to ask is what percentage of your users will hit one or more of these, because if that's more than an insignificant percentage you'll still be staffing all the teams that dealt with hardware, handling local OS installs, worrying about lost or stolen devices, and the glorious future of just being able to stop worrying about this is going to be gone and the financial benefits you promised would appear are probably not going to work out in the same way.
A lot of this falls back to the usual story of corporate IT - understand the needs of your users and whether what you're proposing actually meets them. Almost everything I've described here is a corner case, but if your company is larger than about 20 people there's a high probability that at least one person is going to fall into at least one of these corner cases. You're going to need to spend a lot of time understanding your user population to have a real understanding of what the actual costs are here, and I haven't seen anyone do that work before trying to launch this and (inevitably) going back to just giving people actual computers.
There are alternatives! Modern IDEs tend to support SSHing out to remote hosts to perform builds there, so as long as you're ok with source code being visible on laptops you can at least shift the "I need a workstation with a bunch of CPU" problem out to the cloud. The laptops are going to need to be more expensive because they're also going to need to run more software locally, but it wouldn't surprise me if this ends up being cheaper than the full-on cloud desktop experience in most cases.
Overall, the most important thing to take into account here is that your users almost certainly have more use cases than you expect, and this sort of change is going to have direct impact on the workflow of every single one of your users. Make sure you know how much that's going to be, and take that into consideration when suggesting it'll save you money.
Almost to the week one year since the last maintenance release, we can announce another maintenance release, now at version 0.0.17, of the RVowpalWabbit package. The CRAN maintainers kindly and politly pointed out that I was (cough, cough) apparently the last maintainer who had packages that set StagedInstall: no. Guilty as charged.
RVowpalWabbit is one the two packages; the other one will hopefully follow shortly . And while I long suspected linking aspects to drive this (this is an old package, newer R packaging of the awesome VowpalWabbit is in rvw, I was plain wrong here. The cause was an absolute path to an included dataset, computed in an example, which then gets serialized. As Tomas Kalibera suggested, we can replace the constant with a function and all is well. So here is 0.0.17.
As noted before, there is a newer package rvw based on the excellent GSoC 2018 and beyond work by Ivan Pavlov (mentored by James and myself) so if you are into VowpalWabbit from R go check it out.
CRANberries provides a summary of changes to the previous version. More information is on the RVowpalWabbit page. Issues and bugreports should go to the GitHub issue tracker.
If you like this or other open-source work I do, you can now sponsor me at GitHub.
Death Certificate
After waiting for almost two, two, and a half months, I finally got mum s death certificate last week. A part of me was saddened as it felt like I was nailing her or putting nails to the coffin or whatever it is, (even though I m an Agarwal) I just felt sad and awful. I was told just get a death certificate and your problems will be over. Some people wanted me to give some amount under the table or something which I didn t want to party of and because of that perhaps it took a month, month and a half more as I came to know later that it had been issued almost a month and a half back. The inflation over the last 8 years of the present Govt. has made the corrupt even more corrupt, all the while projecting and telling others that the others are corrupt. There had been also a few politicians who were caught red-handed but then pieces of evidence & witnesses vanish overnight. I don t really wanna go in that direction as it would make for an unpleasant reading with no solutions at all unless the present Central Govt. goes out.
Intestate and Will
I came to know the word Intestate. This was a new word/term for me. A lookup told me that intestate means a person dying without putting a will. That legal term comes from U.K. law. I had read a long long time back that almost all our laws have and were made or taken from U.K. law. IIRC, massive sections of the CRPC Act even today have that colonial legacy. While in its (BJP) manifesto that had been shared with the public at the time of the election, they had shared that they will remove a whole swathe of laws that don t make sense in today s environment. But when hard and good questions were asked, they trimmed a few, modified a few, and left most of them as it is. Sadly, most of the laws that they did modify increased Government control over people instead of decreasing, It s been 8 years and yet we still don t have a Privacy law. They had made something but it was too vague and would have invited suits from day 1 so pretty much on backburner :(. A good insight into what I mean is an article in the Hindu I read a few days back. Once you read that article, I am sure you will have as many questions as I have but sadly no answers.
Law is not supposed to be partisan but today it is. I could cite examples from both the U.S. and UK courts about progressive judgments or the way they go about it, but then again when our people think they know better But this again does not help me apart from setting some kind of background of where we are.) I have on this blog also shared how Africans have been setting new records in transparency and they did it almost 5 years back. For those new to the blog, African countries have been now broadcasting proceedings of their SC for almost 5 years now. I noticed it when privacy law was being debated and a few handles that I follow on Twitter and elsewhere had gone and given their submission in their SC. It was fascinating to not only hear but also read about the case from multiple viewpoints.
And just to remind people, I am sharing all of this from Pune, Maharashtra which is the second-biggest city in Maharashtra that has something like six million people and probably a million or more transitory students, and casual laborers but then again that doesn t help me other than providing a kind of context to what I m sharing..
Now a couple of years back or more I had asked mum to make a will. If she wanted to bequeath something to somebody else she could do that, had shared about that. There was some article in Indian Express or elsewhere that told people what they should be doing, especially if they had cost the barrier of age 60. Now for reasons best known to her, she refused and now I have to figure out what is the right way to go about doing things.
Twitter Experiences
Now before Twitter, a few people had been asking me about having a legal heir certificate, while others are asking about a succession certificate and some claim a Death Certificate is enough. Now I asked the same question on Twitter hoping at the max of 5-10 responses but was overwhelmed by the response. I got something like 50-60 odd replies. Probably, one of the better responses was given by Dr. Paras Jain who shared the following
Answer is qualified Movable assets nothing required Bank LIC flat with society nomination done nothing required except death certificate. However, each will insist on a notarized indemnity bond If the nomination is not done. Depends on whims & fancy of each mind legal heir certificate,+ all Dr. Paras Jain. (cleared up the grammar a little, otherwise, views are of Dr. Paras.)
What was interesting for me is that most people just didn t give me advice, many of them also shared their own experiences or what they did or went through. I was surprised to learn e.g. that a succession certificate can take up to 6 months or more. Part of me isn t surprised to learn that as do know we have a huge pendency of cases in High Courts, District Courts leading all the way to the Supreme Court. India Today shared a brief article sharing the same and similar issues. Such delays have become far too common now
Supertech Demolition and Others
Over the last couple of months, a number of high-profile demolitions have taken place and in most cases, the loss has been of homebuyers. See for e.g. the case of Supertech. A much more detailed article was penned by Moneylife. There were a few Muslims whose homes were demolished just a couple of months back that were being celebrated, but now just 2-3 days back a politician by the name of Shrikant Tyagi, a BJP leader, his flat was partly demolished and there was a lot of hue and cry. Although we shouldn t be discussing on the basis of religion but legality, somehow the idea has been put that there are two kinds of laws, one for the majority, the other for the minority. And this has been going on for the last 8 odd years, hence you see different reactions to the same incidents instead of similar reactions. In all the cases, no strictures are passed either against the Municipality or against lenders.
The most obvious question, let s say for argument s sake, I was a homeowner in Supertech. I bought a flat for say 10 lakhs in 2012. According to the courts, today I am supposed to get 22 lakhs at 12% simple interest for 10 years. Let s say even if the builder was in a position and does honor the order, the homeowner will not get a house in the same area as the circle rate would probably have quadrupled by then at the very least. The circle rate alone might be the above amount. The reason is very simple, a builder buys land on the cheap when there is no development around. His/her/their whole idea is once development happens due to other builders also building flats, the whole area gets developed and they are able to sell the flats at a premium. Even Circle rates get affected as the builder pays below the table and asks the officers of the municipal authority to hike the circle rate every few months. Again, wouldn t go into much depth as the whole thing is rotten to the core. There are many such projects. I have shared Krishnaraj Rao s videos on this blog a few times. I am sure there are a few good men like him. At the end, sadly this is where we are
P.S. I haven t shared any book reviews this week as this post itself has become too long. I probably may blog about a couple of books in the next couple of days, till later.
This is the 21st Discworld novel and relies on the previous Watch novels
for characterization and cast development. I would not start here.
In the middle of the Circle Sea, the body of water between Ankh-Morpork
and the desert empire of Klatch, a territorial squabble between one
fishing family from Ankh-Morpork and one from Klatch is interrupted by a
weathercock rising dramatically from the sea. When the weathercock is
shortly followed by the city to which it is attached and the island on
which that city is resting, it's justification for more than a fishing
squabble. It's a good reason for a war over new territory.
The start of hostilities is an assassination attempt on a prince of
Klatch. Vimes and the Watch start investigating, but politics outraces
police work. Wars are a matter for the nobility and their armies, not for
normal civilian leadership. Lord Vetinari resigns, leaving the city under
the command of Lord Rust, who is eager for a glorious military victory
against their long-term rivals. The Klatchians seem equally eager to
oblige.
One of the useful properties of a long series is that you build up a cast
of characters you can throw at a plot, and if you can assume the reader
has read enough of the previous books, you don't have to spend a lot of
time on establishing characterization and can get straight to the story.
Pratchett uses that here. You could read this cold, I suppose, because
most of the Watch are obvious enough types that the bits of
characterization they get are enough, but it works best with the nuance
and layers of the previous books. Of course Colon is the most susceptible
to the jingoism that prompts the book's title, and of course Angua's
abilities make her the best detective. The familiar characters let
Pratchett dive right in to the political machinations.
Everyone plays to type here: Vetinari is deftly maneuvering everyone into
place to make the situation work out the way he wants, Vimes is stubborn
and ethical and needs Vetinari to push him in the right direction, and
Carrot is sensible and effortlessly charismatic. Colon and Nobby are, as
usual, comic relief of a sort, spending much of the book with Vetinari
while not understanding what he's up to. But Nobby gets an interesting
bit of characterization in the form of an extended turn as a spy that
starts as cross-dressing and becomes an understated sort of gender
exploration hidden behind humor that's less mocking than one might expect.
Pratchett has been slowly playing more with gender in this series, and
while it's simple and a bit deemphasized, I like it.
I think the best part of this book, thematically, is the contrast between
Carrot's and Vimes's reactions to the war. Carrot is a paragon of a
certain type of ethics in Watch novels, but a war is one of the things
that plays to his weaknesses. Carrot follows rules, and wars have rules
of a type. You can potentially draw Carrot into them. But Vimes, despite
being someone who enforces rules professionally, is deeply suspicious of
them, which makes him harder to fool. Pratchett uses one of the Klatchian
characters to hold a mirror up to Vimes in ways that are minor spoilers,
but that I quite liked.
The argument of jingoism, made by both Lord Rust and by the Klatchian
prince, is that wars are something special, outside the normal rules of
justice. Vimes absolutely refuses this position. As someone from the US,
his reaction to Lord Rust's attempted militarization of the Watch was one
of the best moments of the book.
Not a muscle moved on Rust's face. There was a clink as
Vimes's badge was set neatly on the table.
"I don't have to take this," Vimes said calmly.
"Oh, so you'd rather be a civilian, would you?"
"A watchman is a civilian, you inbred streak of pus!"
Vimes is also willing to think of a war as a possible crime, which may not
be as effective as Vetinari's tricky scheming but which is very
emotionally satisfying.
As with most Pratchett books, the moral underpinnings of the story aren't
that elaborate: people are people despite cultural differences, wars are
bad, and people are too ready to believe the worst of their neighbors.
The story arc is not going to provide great insights into human character
that the reader did not already have. But watching Vimes stubbornly
attempt to do the right thing regardless of the rule book is wholly
satisfying, and watching Vetinari at work is equally, if differently,
enjoyable.
Not the best Discworld novel, but one of the better ones.
Followed by The Last Continent in publication order, and by
The Fifth Elephant thematically.
Rating: 8 out of 10
This post describes how I ve put together a simple static content server for
kubernetes clusters using a Pod with a persistent volume and multiple
containers: an sftp server to manage contents, a web server to publish them
with optional access control and another one to run scripts which need access
to the volume filesystem.
The sftp server runs using
MySecureShell, the web
server is nginx and the script runner uses the
webhook tool to publish endpoints to call
them (the calls will come from other Pods that run backend servers or are
executed from Jobs or CronJobs).
HistoryThe system was developed because we had a NodeJS API with endpoints to upload
files and store them on S3 compatible services that were later accessed via
HTTPS, but the requirements changed and we needed to be able to publish folders
instead of individual files using their original names and apply access
restrictions using our API.
Thinking about our requirements the use of a regular filesystem to keep the
files and folders was a good option, as uploading and serving files is simple.
For the upload I decided to use the sftp protocol, mainly because I already
had an sftp container image based on
mysecureshell prepared; once
we settled on that we added sftp support to the API server and configured it
to upload the files to our server instead of using S3 buckets.
To publish the files we added a nginx container configured
to work as a reverse proxy that uses the
ngx_http_auth_request_module
to validate access to the files (the sub request is configurable, in our
deployment we have configured it to call our API to check if the user can
access a given URL).
Finally we added a third container when we needed to execute some tasks
directly on the filesystem (using kubectl exec with the existing containers
did not seem a good idea, as that is not supported by CronJobs objects, for
example).
The solution we found avoiding the NIH Syndrome (i.e. write our own tool) was
to use the webhook tool to provide the
endpoints to call the scripts; for now we have three:
one to get the disc usage of a PATH,
one to hardlink all the files that are identical on the filesystem,
one to copy files and folders from S3 buckets to our filesystem.
Container definitions
mysecureshellThe mysecureshell container can be used to provide an sftp service with
multiple users (although the files are owned by the same UID and GID) using
standalone containers (launched with docker or podman) or in an
orchestration system like kubernetes, as we are going to do here.
The image is generated using the following Dockerfile:
The /etc/sftp_config file is used to
configure
the mysecureshell server to have all the user homes under /sftp/data, only
allow them to see the files under their home directories as if it were at the
root of the server and close idle connections after 5m of inactivity:
etc/sftp_config
# Default mysecureshell configuration<Default>
# All users will have access their home directory under /sftp/data
Home /sftp/data/$USER
# Log to a file inside /sftp/logs/ (only works when the directory exists)
LogFile /sftp/logs/mysecureshell.log
# Force users to stay in their home directory
StayAtHome true
# Hide Home PATH, it will be shown as /
VirtualChroot true
# Hide real file/directory owner (just change displayed permissions)
DirFakeUser true
# Hide real file/directory group (just change displayed permissions)
DirFakeGroup true
# We do not want users to keep forever their idle connection
IdleTimeOut 5m
</Default>
# vim: ts=2:sw=2:et
The entrypoint.sh script is the one responsible to prepare the container for
the users included on the /secrets/user_pass.txt file (creates the users with
their HOME directories under /sftp/data and a /bin/false shell and
creates the key files from /secrets/user_keys.txt if available).
The script expects a couple of environment variables:
SFTP_UID: UID used to run the daemon and for all the files, it has to be
different than 0 (all the files managed by this daemon are going to be
owned by the same user and group, even if the remote users are different).
SFTP_GID: GID used to run the daemon and for all the files, it has to be
different than 0.
And can use the SSH_PORT and SSH_PARAMS values if present.
It also requires the following files (they can be mounted as secrets in
kubernetes):
/secrets/host_keys.txt: Text file containing the ssh server keys in mime
format; the file is processed using the reformime utility (the one included
on busybox) and can be generated using the
gen-host-keys script included on the container (it uses ssh-keygen and
makemime).
/secrets/user_pass.txt: Text file containing lines of the form
username:password_in_clear_text (only the users included on this file are
available on the sftp server, in fact in our deployment we use only the
scs user for everything).
And optionally can use another one:
/secrets/user_keys.txt: Text file that contains lines of the form
username:public_ssh_ed25519_or_rsa_key; the public keys are installed on
the server and can be used to log into the sftp server if the username
exists on the user_pass.txt file.
The contents of the entrypoint.sh script are:
entrypoint.sh
#!/bin/shset-e# ---------# VARIABLES# ---------# Expects SSH_UID & SSH_GID on the environment and uses the value of the# SSH_PORT & SSH_PARAMS variables if present# SSH_PARAMSSSH_PARAMS="-D -e -p $ SSH_PORT:=22$ SSH_PARAMS"# Fixed values# DIRECTORIESHOME_DIR="/sftp/data"CONF_FILES_DIR="/secrets"AUTH_KEYS_PATH="/etc/ssh/auth_keys"# FILESHOST_KEYS="$CONF_FILES_DIR/host_keys.txt"USER_KEYS="$CONF_FILES_DIR/user_keys.txt"USER_PASS="$CONF_FILES_DIR/user_pass.txt"USER_SHELL_CMD="/usr/bin/mysecureshell"# TYPESHOST_KEY_TYPES="dsa ecdsa ed25519 rsa"# ---------# FUNCTIONS# ---------# Validate HOST_KEYS, USER_PASS, SFTP_UID and SFTP_GID
_check_environment()# Check the ssh server keys ... we don't boot if we don't have themif[!-f"$HOST_KEYS"];then
cat<<EOF
We need the host keys on the '$HOST_KEYS' file to proceed.
Call the 'gen-host-keys' script to create and export them on a mime file.
EOF
exit 1
fi# Check that we have users ... if we don't we can't continueif[!-f"$USER_PASS"];then
cat<<EOF
We need at least the '$USER_PASS' file to provision users.
Call the 'gen-users-tar' script to create a tar file to create an archive that
contains public and private keys for users, a 'user_keys.txt' with the public
keys of the users and a 'user_pass.txt' file with random passwords for them
(pass the list of usernames to it).
EOF
exit 1
fi# Check SFTP_UIDif[-z"$SFTP_UID"];then
echo"The 'SFTP_UID' can't be empty, pass a 'GID'."exit 1
fi
if["$SFTP_UID"-eq"0"];then
echo"The 'SFTP_UID' can't be 0, use a different 'UID'"exit 1
fi# Check SFTP_GIDif[-z"$SFTP_GID"];then
echo"The 'SFTP_GID' can't be empty, pass a 'GID'."exit 1
fi
if["$SFTP_GID"-eq"0"];then
echo"The 'SFTP_GID' can't be 0, use a different 'GID'"exit 1
fi# Adjust ssh host keys
_setup_host_keys()opwd="$(pwd)"tmpdir="$(mktemp-d)"cd"$tmpdir"ret="0"
reformime <"$HOST_KEYS"ret="1"for kt in$HOST_KEY_TYPES;do
key="ssh_host_$ kt_key"pub="ssh_host_$ kt_key.pub"if[!-f"$key"];then
echo"Missing '$key' file"ret="1"fi
if[!-f"$pub"];then
echo"Missing '$pub' file"ret="1"fi
if["$ret"-ne"0"];then
continue
fi
cat"$key">"/etc/ssh/$key"chmod 0600 "/etc/ssh/$key"chown root:root "/etc/ssh/$key"cat"$pub">"/etc/ssh/$pub"chmod 0600 "/etc/ssh/$pub"chown root:root "/etc/ssh/$pub"done
cd"$opwd"rm-rf"$tmpdir"return"$ret"# Create users
_setup_user_pass()opwd="$(pwd)"tmpdir="$(mktemp-d)"cd"$tmpdir"ret="0"[-d"$HOME_DIR"]mkdir"$HOME_DIR"# Make sure the data dir can be managed by the sftp userchown"$SFTP_UID:$SFTP_GID""$HOME_DIR"# Allow the user (and root) to create directories inside the $HOME_DIR, if# we don't allow it the directory creation fails on EFS (AWS)chmod 0755 "$HOME_DIR"# Create usersecho"sftp:sftp:$SFTP_UID:$SFTP_GID:::/bin/false">"newusers.txt"sed-n"/^[^#]/ s/:/ /p ""$USER_PASS"while read-r _u _p;do
echo"$_u:$_p:$SFTP_UID:$SFTP_GID::$HOME_DIR/$_u:$USER_SHELL_CMD"done>>"newusers.txt"
newusers --badnames newusers.txt
# Disable write permission on the directory to forbid remote sftp users to# remove their own root dir (they have already done it); we adjust that# here to avoid issues with EFS (see before)chmod 0555 "$HOME_DIR"# Clean up the tmpdircd"$opwd"rm-rf"$tmpdir"return"$ret"# Adjust user keys
_setup_user_keys()if[-f"$USER_KEYS"];then
sed-n"/^[^#]/ s/:/ /p ""$USER_KEYS"while read-r _u _k;do
echo"$_k">>"$AUTH_KEYS_PATH/$_u"done
fi# Main function
exec_sshd()
_check_environment
_setup_host_keys
_setup_user_pass
_setup_user_keys
echo"Running: /usr/sbin/sshd $SSH_PARAMS"# shellcheck disable=SC2086exec /usr/sbin/sshd -D$SSH_PARAMS# ----# MAIN# ----case"$1"in"server") exec_sshd ;;*)exec"$@";;esac# vim: ts=2:sw=2:et
The container also includes a couple of auxiliary scripts, the first one can be
used to generate the host_keys.txt file as follows:
$docker run --rm stodh/mysecureshell gen-host-keys > host_keys.txt
Where the script is as simple as:
bin/gen-host-keys
#!/bin/shset-e# Generate new host keys
ssh-keygen -A>/dev/null
# Replace hostnamesed-i-e's/@.*$/@mysecureshell/' /etc/ssh/ssh_host_*_key.pub
# Print in mime format (stdout)
makemime /etc/ssh/ssh_host_*# vim: ts=2:sw=2:et
And there is another script to generate a .tar file that contains auth data
for the list of usernames passed to it (the file contains a user_pass.txt
file with random passwords for the users, public and private ssh keys for them
and the user_keys.txt file that matches the generated keys).
To generate a tar file for the user scs we can execute the following:
$docker run --rm stodh/mysecureshell gen-users-tar scs > /tmp/scs-users.tar
To see the contents and the text inside the user_pass.txt file we can do:
Basically we are removing the existing docker-entrypoint.d scripts from the
standard image and adding a new one that configures the web server as we want
using a couple of environment variables:
AUTH_REQUEST_URI: URL to use for the auth_request, if the variable is not
found on the environment auth_request is not used.
HTML_ROOT: Base directory of the web server, if not passed the default
/usr/share/nginx/html is used.
Note that if we don t pass the variables everything works as if we were using
the original nginx image.
The contents of the configuration script are:
docker-entrypoint.d/10-update-default-conf.sh
As we will see later the idea is to use the /sftp/data or /sftp/data/scs
folder as the root of the web published by this container and create an
Ingress object to provide access to it outside of our kubernetes cluster.
webhook-scsThe webhook-scs container is generated using the following Dockerfile:
Again, we use a multi-stage build because in production we wanted to support a
functionality that is not already on the official versions (streaming the
command output as a response instead of waiting until the execution ends); this
time we build the image applying the PATCH included on this
pull request against a released
version of the source instead of creating a fork.
The entrypoint.sh script is used to generate the webhook configuration file
for the existing hooks using environment variables (basically the
WEBHOOK_WORKDIR and the *_TOKEN variables) and launch the webhook
service:
entrypoint.sh
#!/bin/shset-e# ---------# VARIABLES# ---------WEBHOOK_BIN="$ WEBHOOK_BIN:-/webhook/hooks"WEBHOOK_YML="$ WEBHOOK_YML:-/webhook/scs.yml"WEBHOOK_OPTS="$ WEBHOOK_OPTS:--verbose"# ---------# FUNCTIONS# ---------
print_du_yml()cat<<EOF
- id: du
execute-command: '$WEBHOOK_BIN/du.sh'
command-working-directory: '$WORKDIR'
response-headers:
- name: 'Content-Type'
value: 'application/json'
http-methods: ['GET']
include-command-output-in-response: true
include-command-output-in-response-on-error: true
pass-arguments-to-command:
- source: 'url'
name: 'path'
pass-environment-to-command:
- source: 'string'
envname: 'OUTPUT_FORMAT'
name: 'json'
EOF
print_hardlink_yml()cat<<EOF
- id: hardlink
execute-command: '$WEBHOOK_BIN/hardlink.sh'
command-working-directory: '$WORKDIR'
http-methods: ['GET']
include-command-output-in-response: true
include-command-output-in-response-on-error: true
EOF
print_s3sync_yml()cat<<EOF
- id: s3sync
execute-command: '$WEBHOOK_BIN/s3sync.sh'
command-working-directory: '$WORKDIR'
http-methods: ['POST']
include-command-output-in-response: true
include-command-output-in-response-on-error: true
pass-environment-to-command:
- source: 'payload'
envname: 'AWS_KEY'
name: 'aws.key'
- source: 'payload'
envname: 'AWS_SECRET_KEY'
name: 'aws.secret_key'
- source: 'payload'
envname: 'S3_BUCKET'
name: 's3.bucket'
- source: 'payload'
envname: 'S3_REGION'
name: 's3.region'
- source: 'payload'
envname: 'S3_PATH'
name: 's3.path'
- source: 'payload'
envname: 'SCS_PATH'
name: 'scs.path'
stream-command-output: true
EOF
print_token_yml()if["$1"];then
cat<<EOF
trigger-rule:
match:
type: 'value'
value: '$1'
parameter:
source: 'header'
name: 'X-Webhook-Token'
EOF
fi
exec_webhook()# Validate WORKDIRif[-z"$WEBHOOK_WORKDIR"];then
echo"Must define the WEBHOOK_WORKDIR variable!">&2
exit 1
fi
WORKDIR="$(realpath"$WEBHOOK_WORKDIR" 2>/dev/null)"true
if[!-d"$WORKDIR"];then
echo"The WEBHOOK_WORKDIR '$WEBHOOK_WORKDIR' is not a directory!">&2
exit 1
fi# Get TOKENS, if the DU_TOKEN or HARDLINK_TOKEN is defined that is used, if# not if the COMMON_TOKEN that is used and in other case no token is checked# (that is the default)DU_TOKEN="$ DU_TOKEN:-$COMMON_TOKEN"HARDLINK_TOKEN="$ HARDLINK_TOKEN:-$COMMON_TOKEN"S3_TOKEN="$ S3_TOKEN:-$COMMON_TOKEN"# Create webhook configuration
print_du_yml
print_token_yml "$DU_TOKEN"echo""
print_hardlink_yml
print_token_yml "$HARDLINK_TOKEN"echo""
print_s3sync_yml
print_token_yml "$S3_TOKEN" >"$WEBHOOK_YML"# Run the webhook command# shellcheck disable=SC2086exec webhook -hooks"$WEBHOOK_YML"$WEBHOOK_OPTS# ----# MAIN# ----case"$1"in"server") exec_webhook ;;*)exec"$@";;esac
The entrypoint.sh script generates the configuration file for the webhook
server calling functions that print a yaml section for each hook and
optionally adds rules to validate access to them comparing the value of a
X-Webhook-Token header against predefined values.
The expected token values are taken from environment variables, we can define
a token variable for each hook (DU_TOKEN, HARDLINK_TOKEN or S3_TOKEN)
and a fallback value (COMMON_TOKEN); if no token variable is defined for a
hook no check is done and everybody can call it.
The Hook
Definition documentation explains the options you can use for each hook, the
ones we have right now do the following:
du: runs on the $WORKDIR directory, passes as first argument to the
script the value of the path query parameter and sets the variable
OUTPUT_FORMAT to the fixed value json (we use that to print the output of
the script in JSON format instead of text).
hardlink: runs on the $WORKDIR directory and takes no parameters.
s3sync: runs on the $WORKDIR directory and sets a lot of environment
variables from values read from the JSON encoded payload sent by the caller
(all the values must be sent by the caller even if they are assigned an empty
value, if they are missing the hook fails without calling the script); we
also set the stream-command-output value to true to make the script show
its output as it is working (we patched the webhook source to be able to
use this option).
The du hook scriptThe du hook script code checks if the argument passed is a directory,
computes its size using the du command and prints the results in text format
or as a JSON dictionary:
hooks/du.sh
#!/bin/shset-e# Script to print disk usage for a PATH inside the scs folder# ---------# FUNCTIONS# ---------
print_error()if["$OUTPUT_FORMAT"="json"];then
echo" \"error\":\"$*\" "else
echo"$*">&2
fi
exit 1
usage()if["$OUTPUT_FORMAT"="json"];then
echo" \"error\":\"Pass arguments as '?path=XXX\" "else
echo"Usage: $(basename"$0") PATH">&2
fi
exit 1
# ----# MAIN# ----if["$#"-eq"0"][-z"$1"];then
usage
fi
if["$1"="."];then
DU_PATH="./"else
DU_PATH="$(find .-name"$1"-mindepth 1 -maxdepth 1)"true
fi
if[-z"$DU_PATH"][!-d"$DU_PATH/."];then
print_error "The provided PATH ('$1') is not a directory"fi# Print disk usage in bytes for the given PATHOUTPUT="$(du-b-s"$DU_PATH")"if["$OUTPUT_FORMAT"="json"];then# Format output as "path":"PATH","bytes":"BYTES" echo"$OUTPUT"sed-e"s%^\(.*\)\t.*/\(.*\)$%\"path\":\"\2\",\"bytes\":\"\1\" %"tr-d'\n'else# Print du output as isecho"$OUTPUT"fi# vim: ts=2:sw=2:et:ai:sts=2
The hardlink hook scriptThe hardlink hook script is really simple, it just runs the
util-linux version of the
hardlink
command on its working directory:
hooks/hardlink.sh
#!/bin/sh
hardlink --ignore-time--maximize .
We use that to reduce the size of the stored content; to manage versions of
files and folders we keep each version on a separate directory and when one or
more files are not changed this script makes them hardlinks to the same file on
disc, reducing the space used on disk.
The s3sync hook scriptThe s3sync hook script uses the s3fs
tool to mount a bucket and synchronise data between a folder inside the bucket
and a directory on the filesystem using rsync; all values needed to execute
the task are taken from environment variables:
hooks/s3sync.sh
#!/bin/ashset-euo pipefail
set-o errexit
set-o errtrace
# Functions
finish()ret="$1"echo""echo"Script exit code: $ret"exit"$ret"# Check variablesif[-z"$AWS_KEY"][-z"$AWS_SECRET_KEY"][-z"$S3_BUCKET"][-z"$S3_PATH"][-z"$SCS_PATH"];then["$AWS_KEY"]echo"Set the AWS_KEY environment variable"["$AWS_SECRET_KEY"]echo"Set the AWS_SECRET_KEY environment variable"["$S3_BUCKET"]echo"Set the S3_BUCKET environment variable"["$S3_PATH"]echo"Set the S3_PATH environment variable"["$SCS_PATH"]echo"Set the SCS_PATH environment variable"
finish 1
fi
if["$S3_REGION"]&&["$S3_REGION"!="us-east-1"];then
EP_URL="endpoint=$S3_REGION,url=https://s3.$S3_REGION.amazonaws.com"else
EP_URL="endpoint=us-east-1"fi# Prepare working directoryWORK_DIR="$(mktemp-p"$HOME"-d)"MNT_POINT="$WORK_DIR/s3data"PASSWD_S3FS="$WORK_DIR/.passwd-s3fs"# Check the moutpointif[!-d"$MNT_POINT"];then
mkdir-p"$MNT_POINT"elif mountpoint "$MNT_POINT";then
echo"There is already something mounted on '$MNT_POINT', aborting!"
finish 1
fi# Create password filetouch"$PASSWD_S3FS"chmod 0400 "$PASSWD_S3FS"echo"$AWS_KEY:$AWS_SECRET_KEY">"$PASSWD_S3FS"# Mount s3 bucket as a filesystem
s3fs -odbglevel=info,retries=5 -o"$EP_URL"-o"passwd_file=$PASSWD_S3FS"\"$S3_BUCKET""$MNT_POINT"echo"Mounted bucket '$S3_BUCKET' on '$MNT_POINT'"# Remove the password file, just in caserm-f"$PASSWD_S3FS"# Check source PATHret="0"SRC_PATH="$MNT_POINT/$S3_PATH"if[!-d"$SRC_PATH"];then
echo"The S3_PATH '$S3_PATH' can't be found!"ret=1
fi# Compute SCS_UID & SCS_GID (by default based on the working directory owner)SCS_UID="$ SCS_UID:=$(stat-c"%u""." 2>/dev/null)"true
SCS_GID="$ SCS_GID:=$(stat-c"%g""." 2>/dev/null)"true# Check destination PATHDST_PATH="./$SCS_PATH"if["$ret"-eq"0"]&&[-d"$DST_PATH"];then
mkdir-p"$DST_PATH"ret="$?"fi# Copy using rsyncif["$ret"-eq"0"];then
rsync -rlptv--chown="$SCS_UID:$SCS_GID"--delete--stats\"$SRC_PATH/""$DST_PATH/"ret="$?"fi# Unmount the S3 bucket
umount -f"$MNT_POINT"echo"Called umount for '$MNT_POINT'"# Remove mount point dirrmdir"$MNT_POINT"# Remove WORK_DIRrmdir"$WORK_DIR"# We are done
finish "$ret"# vim: ts=2:sw=2:et:ai:sts=2
Deployment objectsThe system is deployed as a StatefulSet with one replica.
Our production deployment is done on AWS and to be
able to scale we use EFS for our
PersistenVolume; the idea is that the volume has no size limit, its
AccessMode can be set to ReadWriteMany and we can mount it from multiple
instances of the Pod without issues, even if they are in different availability
zones.
For development we use k3d and we are also able to scale the
StatefulSet for testing because we use a ReadWriteOnce PVC, but it points
to a hostPath that is backed up by a folder that is mounted on all the
compute nodes, so in reality Pods in different k3d nodes use the same folder
on the host.
secrets.yamlThe secrets file contains the files used by the mysecureshell container that
can be generated using kubernetes pods as follows (we are only creating the
scs user):
$kubectl run "mysecureshell"--restart='Never'--quiet--rm--stdin\ --image "stodh/mysecureshell:latest" -- gen-host-keys >"./host_keys.txt"$kubectl run "mysecureshell"--restart='Never'--quiet--rm--stdin\ --image "stodh/mysecureshell:latest" -- gen-users-tar scs >"./users.tar"
Once we have the files we can generate the secrets.yaml file as follows:
On this definition we don t set the storageClassName to use the default one.
Volumes in our development environment (k3d)In our development deployment we create the following PersistentVolume as
required by the
Local
Persistence Volume Static Provisioner (note that the /volumes/scs-pv has to
be created by hand, in our k3d system we mount the same host directory on the
/volumes path of all the nodes and create the scs-pv directory by hand
before deploying the persistent volume):
k3d-pv.yaml
Volumes in our production environment (aws)In the production deployment we don t create the PersistentVolume (we are
using the
aws-efs-csi-driver which
supports Dynamic Provisioning) but we add the storageClassName (we set it
to the one mapped to the EFS driver, i.e. efs-sc) and set ReadWriteMany
as the accessMode:
efs-pvc.yaml
nginx: As this is an example the web server is not using an
AUTH_REQUEST_URI and uses the /sftp/data directory as the root of the web
(to get to the files uploaded for the scs user we will need to use /scs/
as a prefix on the URLs).
mysecureshell: We are adding the IPC_OWNER capability to the container to
be able to use some of the sftp-* commands inside it, but they are
not really needed, so adding the capability is optional.
webhook: We are launching this container in privileged mode to be able to
use the s3fs-fuse, as it will not work otherwise for now (see this
kubernetes issue); if
the functionality is not needed the container can be executed with regular
privileges; besides, as we are not enabling public access to this service we
don t define *_TOKEN variables (if required the values should be read from a
Secret object).
Notes about the volumes:
the devfuse volume is only needed if we plan to use the s3fs command on
the webhook container, if not we can remove the volume definition and its
mounts.
service.yamlTo be able to access the different services on the statefulset we publish the
relevant ports using the following Service object:
service.yaml
ingress.yamlTo download the scs files from the outside we can add an ingress object like
the following (the definition is for testing using the localhost name):
DeploymentTo deploy the statefulSet we create a namespace and apply the object
definitions shown before:
$kubectl create namespace scs-demo
namespace/scs-demo created
$kubectl -n scs-demo apply -f secrets.yaml
secret/scs-secrets created
$kubectl -n scs-demo apply -f pvc.yaml
persistentvolumeclaim/scs-pvc created
$kubectl -n scs-demo apply -f statefulset.yaml
statefulset.apps/scs created
$kubectl -n scs-demo apply -f service.yaml
service/scs-svc created
$kubectl -n scs-demo apply -f ingress.yaml
ingress.networking.k8s.io/scs-ingress created
Once the objects are deployed we can check that all is working using kubectl:
$kubectl -n scs-demo get all,secrets,ingress
NAME READY STATUS RESTARTS AGE
pod/scs-0 3/3 Running 0 24s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/scs-svc ClusterIP 10.43.0.47 <none>22/TCP,80/TCP,9000/TCP 21s
NAME READY AGE
statefulset.apps/scs 1/1 24s
NAME TYPE DATA AGE
secret/default-token-mwcd7 kubernetes.io/service-account-token 3 53s
secret/scs-secrets Opaque 3 39s
NAME CLASS HOSTS ADDRESS PORTS AGE
ingress.networking.k8s.io/scs-ingress nginx localhost 172.21.0.5 80 17s
At this point we are ready to use the system.
Usage examples
File uploadsAs previously mentioned in our system the idea is to use the sftp server from
other Pods, but to test the system we are going to do a kubectl port-forward
and connect to the server using our host client and the password we have
generated (it is on the user_pass.txt file, inside the users.tar archive):
$kubectl -n scs-demo port-forward service/scs-svc 2020:22 &
Forwarding from 127.0.0.1:2020 ->22
Forwarding from [::1]:2020 ->22
$PF_PID=$!$sftp -P 2020 scs@127.0.0.1 1
Handling connection for 2020
The authenticity of host '[127.0.0.1]:2020 ([127.0.0.1]:2020)' can't be \
established.
ED25519 key fingerprint is SHA256:eHNwCnyLcSSuVXXiLKeGraw0FT/4Bb/yjfqTstt+088.
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '[127.0.0.1]:2020' (ED25519) to the list of known \
hosts.
scs@127.0.0.1's password: **********
Connected to 127.0.0.1.
sftp>ls-ladrwxr-xr-x 2 sftp sftp 4096 Sep 25 14:47 .
dr-xr-xr-x 3 sftp sftp 4096 Sep 25 14:36 ..
sftp>!date-R> /tmp/date.txt 2
sftp>put /tmp/date.txt .Uploading /tmp/date.txt to /date.txt
date.txt 100% 32 27.8KB/s 00:00
sftp>ls-l-rw-r--r-- 1 sftp sftp 32 Sep 25 15:21 date.txt
sftp>ln date.txt date.txt.1 3
sftp>ls-l-rw-r--r-- 2 sftp sftp 32 Sep 25 15:21 date.txt
-rw-r--r-- 2 sftp sftp 32 Sep 25 15:21 date.txt.1
sftp>put /tmp/date.txt date.txt.2 4
Uploading /tmp/date.txt to /date.txt.2
date.txt 100% 32 27.8KB/s 00:00
sftp>ls-l5
-rw-r--r-- 2 sftp sftp 32 Sep 25 15:21 date.txt
-rw-r--r-- 2 sftp sftp 32 Sep 25 15:21 date.txt.1
-rw-r--r-- 1 sftp sftp 32 Sep 25 15:21 date.txt.2
sftp>exit$kill"$PF_PID"[1] + terminated kubectl -n scs-demo port-forward service/scs-svc 2020:22
We connect to the sftp service on the forwarded port with the scs user.
We put a file we have created on the host on the directory.
We do a hard link of the uploaded file.
We put a second copy of the file we created locally.
On the file list we can see that the two first files have two hardlinks
File retrievalsIf our ingress is configured right we can download the date.txt file from the
URL http://localhost/scs/date.txt:
Use of the webhook containerTo finish this post we are going to show how we can call the hooks directly,
from a CronJob and from a Job.
Direct script call (du)In our deployment the direct calls are done from other Pods, to simulate it we
are going to do a port-forward and call the script with an existing PATH (the
root directory) and a bad one:
$kubectl -n scs-demo port-forward service/scs-svc 9000:9000 >/dev/null &
$PF_PID=$!$JSON="$(curl -s"http://localhost:9000/hooks/du?path=.")"$echo$JSON "path":"","bytes":"4160"
$JSON="$(curl -s"http://localhost:9000/hooks/du?path=foo")"$echo$JSON "error":"The provided PATH ('foo') is not a directory"
$kill$PF_PID
As we only have files on the base directory we print the disk usage of the .
PATH and the output is in json format because we export OUTPUT_FORMAT with
the value json on the webhook configuration.
Cronjobs (hardlink)As explained before, the webhook container can be used to run cronjobs; the
following one uses an alpine container to call the hardlink script each
minute (that setup is for testing, obviously):
webhook-cronjob.yaml
The following console session shows how we create the object, allow a couple of
executions and remove it (in production we keep it running but once a day, not
each minute):
$kubectl -n scs-demo apply -f webhook-cronjob.yaml 1
cronjob.batch/hardlink created
$kubectl -n scs-demo get pods -l"cronjob=hardlink"-w2
NAME READY STATUS RESTARTS AGE
hardlink-27735351-zvpnb 0/1 Pending 0 0s
hardlink-27735351-zvpnb 0/1 ContainerCreating 0 0s
hardlink-27735351-zvpnb 0/1 Completed 0 2s
^C
$kubectl -n scs-demo logs pod/hardlink-27735351-zvpnb 3
Mode: real
Method: sha256
Files: 3
Linked: 1 files
Compared: 0 xattrs
Compared: 1 files
Saved: 32 B
Duration: 0.000220 seconds
$sleep 60
$kubectl -n scs-demo get pods -l"cronjob=hardlink"4
NAME READY STATUS RESTARTS AGE
hardlink-27735351-zvpnb 0/1 Completed 0 83s
hardlink-27735352-br5rn 0/1 Completed 0 23s
$kubectl -n scs-demo logs pod/hardlink-27735352-br5rn 5
Mode: real
Method: sha256
Files: 3
Linked: 0 files
Compared: 0 xattrs
Compared: 0 files
Saved: 0 B
Duration: 0.000070 seconds
$kubectl -n scs-demo delete -f webhook-cronjob.yaml 6
cronjob.batch "hardlink" deleted
This command creates the cronjob object.
This checks the pods with our cronjob label, we interrupt it once we see
that the first run has been completed.
With this command we see the output of the execution, as this is the fist
execution we see that date.txt.2 has been replaced by a hardlink (the
summary does not name the file, but it is the only option knowing the
contents from the original upload).
After waiting a little bit we check the pods executed again to get the name
of the latest one.
The log now shows that nothing was done.
As this is a demo, we delete the cronjob.
Jobs (s3sync)The following job can be used to synchronise the contents of a directory in a
S3 bucket with the SCS Filesystem:
job.yaml
Once we have both files we can run the Job as follows:
$kubectl -n scs-demo create secret generic webhook-job-secrets \ 1
--from-file="s3sync.json=s3sync.json"
secret/webhook-job-secrets created
$kubectl -n scs-demo apply -f webhook-job.yaml 2
job.batch/s3sync created
$kubectl -n scs-demo get pods -l"cronjob=s3sync"3
NAME READY STATUS RESTARTS AGE
s3sync-zx2cj 0/1 Completed 0 12s
$kubectl -n scs-demo logs s3sync-zx2cj 4
Mounted bucket 's3fs-test' on '/root/tmp.jiOjaF/s3data'
sending incremental file list
created directory ./test
./
kyso.png
Number of files: 2 (reg: 1, dir: 1)
Number of created files: 2 (reg: 1, dir: 1)
Number of deleted files: 0
Number of regular files transferred: 1
Total file size: 15,075 bytes
Total transferred file size: 15,075 bytes
Literal data: 15,075 bytes
Matched data: 0 bytes
File list size: 0
File list generation time: 0.147 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 15,183
Total bytes received: 74
sent 15,183 bytes received 74 bytes 30,514.00 bytes/sec
total size is 15,075 speedup is 0.99
Called umount for '/root/tmp.jiOjaF/s3data'
Script exit code: 0
$kubectl -n scs-demo delete -f webhook-job.yaml 5
job.batch "s3sync" deleted
$kubectl -n scs-demo delete secrets webhook-job-secrets 6
secret "webhook-job-secrets" deleted
Here we create the webhook-job-secrets secret that contains the
s3sync.json file.
This command runs the job.
Checking the label cronjob=s3sync we get the Pods executed by the job.
Here we print the logs of the completed job.
Once we are finished we remove the Job.
And also the secret.
Final remarksThis post has been longer than I expected, but I believe it can be useful for
someone; in any case, next time I ll try to explain something shorter or will
split it into multiple entries.
The server running this blog is a RockPro64 sitting in my living room. Besides WordPress (the blogging software), I run various other services on it:
Bepasty for sharing files,
Ikiwiki for taking notes,
Quassel for staying connected to IRC chat servers,
Radicale for synchronizing my calendar and tasks,
Shaarli for sharing bookmarks, and
Tiny Tiny RSS for reading other people s blogs.
Most of these are for my personal use, and a few of them have pages for public viewing (linked at the top of this page).
Despite running a server, I don t really consider myself to be a system administrator (or sysadmin for short). I generally try to avoid doing system administration work as much as possible. I think this is due to a number of reasons:
It is not part of my educational or professional background (which mainly consists of embedded software engineering).
I think that I lack the kind of discipline needed to be a sysadmin. For example, taking notes of the commands that you are running, or testing things before you do them on a production server. I m likely to just run commands, and lose track of my notes.
And finally a type of laziness. I just don t want to spend a lot of time and effort setting up or maintaining some services.
These reasons might be surprising to some, but they also suggest that an alternative approach:
I do have a server running in my home, but I don t consider myself a sysadmin.
I have a different kind of discipline than the one I described above.
I m willing to spend time and effort to improve things, but I want to do it in a different way than usual.
So my approach is this: if I want to run an additional service, enhance an existing one, or fix a bug, I don t do those changes directly on my server. Instead, I will make (or suggest, or request) the change somewhere upstream of my server:
In the various Debian packages of the services I have installed.
In FreedomBox configuration and integration of those packages.
Or even directly to the upstream software development.
So basically my system administration task turns into a software development task instead. And (in my opinion) there are much better tools available for this: source control systems such as git, test suites and Continuous Integration (CI) pipelines, and code review processes. These make it easier to keep track of and understand the changes, and reduce the possibility of making a catastrophic mistake.
Besides this, there is one other major advantage to working upstream: the work is not just benefiting the server running in my home, but many others. Anyone who is using the same software or packages will also get the improvements or bug fixes. And likewise, I get to benefit from the work done by many other contributors.
Some final notes about this approach:
The software I am running is very standardized, specifically to what is available in Debian and FreedomBox. This limits the number of services available: for example, I could not easily run a Mastodon server with this approach. Also, I am not maintaining any custom configurations for my own services.
I am presenting these ideas as they occur to me, and specific to my situation. But there are similar concepts out there, for example Infrastructure as Code .
Back in November I wrote up a list of 25 things I would 3D print.
Let's revisit the list and see how things have developed.
Stuff I won't print
Some kind of 45 leaning prong to dry bottles and flasks on
A tea tray and coasters
Small tins to keep loose-leaf tea in
It was pointed out to me that you can't safely print things to store food in
with most materials, as their porous/layered nature facilitates the growth of
bacteria. So, I'll rule out those items.
A vinyl record.
The size of the grooves in a vinyl record are smaller than conventional FDM
printers can achieve.
Things I've printed
a replacement prop arm/foot for my computer keyboard
So far, I've tried to print useful, functional things, but on a few occasions I've
printed a little Pacman ghost when testing printer calibration or similar. I've
mostly used these models: https://www.printables.com/model/199425-pac-man-ghost-v2
Lego storage/sorters
DIY bits-and-bobs sorter/storage (nuts and bolts etc)
I learned about a Slicer setting sometimes called "Vase mode" and found this interesting system of modular drawers that are designed to be printed in Vase
mode, so I gave them a go:
https://www.printables.com/model/139570-fast-printing-modular-drawer-system-vase-mode
I printed one and four drawers for it and gave it to my daughter. It might be
used for sorting Lego, or possibly as a chest-of-drawers for a dolls house.
A free-standing inclined vinyl record display stand
A bracket to install a Gotek drive in my Amiga 500
https://www.thingiverse.com/thing:2745049
Summary
From my original list of 25 things to print, I've done 7 of them and determined
4 are not viable. The things in this list I've printed have been off-the-shelf
models that other people have constructed. The things I haven't printed are
designs I will do myself, which is one reason I haven't printed them yet:
building your own designs is the hard part!
Tried implementing FUSE cpiofs.
I thought it might be fun to implement a file system that can mount initrd file systems. I've implemented symlinks and regular files handling and then I realized that it is kind of annoying that I need to implement hard links.
cpiofs
. What is the basis of that or why is there such kind of confidence I have no clue or idea, it is what it is. Approximately 24 hrs. ago, apparently we had a cloudburst. What I came to know later is that we got 100 mm of rain. Sharing from local 
. For those who have read this blog, do know that India is and has been in casteism from ever. They even took the fair comment and applied it to all Brahmins. According to them, all Brahmins are fair and hence have god-given right to lord over others. What is called the Eton boy s network serves the same in this casteism. The only solution is those idea under limelight and investigate. To take the above, how does one prove that all fair people are wise and peaceful while all people black and brown are violent. If that is so, how does one count for Mahatma Gandhi, Martin Luther King Junior, Nelson Mandela, Michael Jackson the list is probably endless. And not to forget that when Mahatma Gandhiji did his nonviolent movements either in India or in South Africa, both black and brown people in millions took part. Similar examples of Martin Luther King Jr. I know and read of so many non-violent civl movements that took place in the U.S. For e.g. 
Version 0.3.14 of the 
Fast laptops are expensive, cheap laptops are slow. But even a fast laptop is slower than a decent workstation, and if your developers want a local build environment they're probably going to want a decent workstation. They'll want a fast (and expensive) laptop as well, though, because they're not going to carry their workstation home with them and obviously you expect them to be able to work from home. And in two or three years they'll probably want a new laptop and a new workstation, and that's even more money. Not to mention the risks associated with them doing development work on their laptop and then drunkenly leaving it in a bar or having it stolen or the contents being copied off it while they're passing through immigration at an airport. Surely there's a better way?
This post describes how I ve put together a simple static content server for
kubernetes clusters using a Pod with a persistent volume and multiple
containers: an sftp server to manage contents, a web server to publish them
with optional access control and another one to run scripts which need access
to the volume filesystem.
The sftp server runs using
Back in November I wrote up a 
Tried implementing FUSE cpiofs.
I thought it might be fun to implement a file system that can mount initrd file systems. I've implemented symlinks and regular files handling and then I realized that it is kind of annoying that I need to implement hard links.