sudo debootstrap sid /srv/chroot/r-4.1 http://deb.debian.org/debian
sudo vi /etc/schroot/chroot.d/r-4.1
# Edit it to have something like
    [r-4.1]
    description=R 4.1 (experimental)
    type=directory
    directory=/srv/chroot/r-4.1
    users=  # Here put your username
    root-groups=root
    profile=desktop
    personality=linux
    preserve-environment=true
sudo schroot -c r-4.1
vi /etc/apt/sources.list # To add the experimental distribution
apt update
apt install sudo vim wget
exit

schroot -c r-4.1
sudo apt install r-base/experimental -texperimental
sudo apt install -texperimental pandoc libxml2-dev libcurl4-openssl-dev git libssl-dev texlive

wget https://s3.amazonaws.com/rstudio-ide-build/desktop/bionic/amd64/rstudio-1.4.1714-amd64.deb
sudo apt install libnss3 libasound2
sudo dpkg -i rstudio-1.4.1714-amd64.deb 
sudo apt -f install -texperimental

Changes in RcppArmadillo version 0.10.5.0 (2021-05-21)

• Upgraded to Armadillo release 10.5 (Antipodean Fortress)
  • added .clamp() member function
  • expanded the standalone clamp() function to handle complex values
  • more efficient use of OpenMP
  • vector, matrix and cube constructors now initialise elements to zero by default; use the fill::none specifier, eg. mat X(4,5,fill::none), to disable element initialisation
• Added codecov.yml to exclude Armadillo from coverage analysis

Changes in RcppArmadillo version 0.10.4.1.0 (2021-04-23)

• Upgraded to Armadillo release 10.4.1 (Pressure Cooker)
• GitHub-only release

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

• I want it to be much easier to set up an Otter server yourself, for you and your friends. Currently there are complete build instructions, but nowadays these things can be automated. Ideally it would be possible to set up an instance by button pushing and the payment of a modest sum to a cloud hosting provider.
• Right now, the only way to set up a game, manage the players in it, reset the gaming table, and so on, is using the otter command line tool. It's not so bad a tool, but having something pointy clicky would make the system much more accessible. But GUIs of this kind are a lot of work.

LCD PanelMost (if not all) QNAP models come with an LCD display, in my model i got a 2 lines by 16 characters each. The tool to control the panel is:

/sbin/lcd_tool

but sometimes it's a bit stubborn in executing the commands, so as mentioned here you may want to killall -9 lcdmond and then issue the command you want; this is the snippet i'm using:

# print a msg on the panel
killall -9 lcdmond
/sbin/lcd_tool -1 "$LINE1" -2 "$LINE2"
sleep 5
# turn off screen light
killall -9 lcdmond
/sbin/lcd_tool -f

it ain't pretty, but it gets the job done.

Speaker / BuzzerIf you look for how to make your system beep, you may find these commands:

/sbin/pic_raw 80    Beep short
/sbin/pic_raw 81    Beep long

but they no longer work. So keep looking, i stumbled upon this post, mentioning hal_app, and that indeed works, but with a different syntax than the one presented at that link.In order to use hal_app to make a beep sound, the syntax that worked for me is:

/sbin/hal_app --se_buzzer enc_id=0,mode=XXX

If you remove entirely mode=XXX, it produces a short beep, but here is the list of all the modes available and their results (found out via exhaustive search):

code	output
0	short beep (0.5 secs)
1	long beep (1 sec)
2	3x short beeps (0.5 secs), long pause (1 sec)
3-7, 10, 15, 16, 17, 18, 19, 20	2x long beeps (1 sec), very long pause (2 secs)
8, 9	3x long beeps (1 sec), very long pauses (2 secs)
12, 13	long beep, 2x very short pause (0.25) very short beep (0.25)
14	3x very short beep (0.25) very short pause (0.25)

(you can prepend as many 0 to the mode strings as you wish, so 000 is equivalent to 0, 000018 is equivalent to 18, and so on).PS: this article is based on my system, TVS-473 running firmware 4.5.3.1652PPS: friendly reminder that this is how you edit the crontab on a QNAP machine.

Moments Given the tools general availability, we are all more used to capturing moments. Once upon a time, taking a photo meant going to a photo studio. Then came a generation where we d have personal devices with roll films, to which we captured the moments. And then get it processed from a photo developing service provider. And that is when we d know which all moments came out correct and which ones were spoilt. In such cases, there s always the wish I could have that moment again. Thankfully, now, in our current generation, we have the liberty to take pictures and validate them almost immediately. We also have the flexibility to take multiple shots of the moment and do the filtering later. There s also many intelligent and invasive services, all mostly provided free, to help you organize these moments; at a small cost of keeping an eye on your life activity. In a summary, mankind, now, generates lots and lots of data. So much lot that now even mammoths like Google are forced to make a call whether it is more profitable to sniff user activity while providing them a service for free, or ask people to pay otherwise too.

Privacy Many of us are wary of the amount of personal data we generate, which is our asset; and how the big tech giants want a piece of it in the name of free services. And in such quest, free aside, there s always a lookout for privacy savvy tools that can help us draw a bold line in between Public and Private

Google Photos Google Photos has been a great tool. The way it organizes, siphons and present your data is simply amazing. No good would have been the photos taken, if they weren t easily searchable, organized, annotated, and presented. But Google Photos is proprietary and invasive. The amount of information they have access to should be a concern to all people. For services that invade so much, the world needs Free Services.

PhotoPrism I only came across this project around a month ago, while the project has been around since 2019. So far, in my exploration, this is one of the best Free Software tools to manage and organize my digital photos. The other favorite tool that I regularly depend on is Digikam and it still is a gem. In a gist, PhotoPrism aims to be the equivalent of Google Photos, for the privacy savvy people. As of date, it has a decent list of features available. And for some of the missing ones, I ve come up with a fairly okay workflow with other tools, which is one of the reasons of this blog post. PhotoPrism is a web app, written in the Go Programming Language. Its layout and workflow as a Photo Management application is similar to Google Photos. It is very performant compared to other applications. Since it is a Progressive Web App, access through a Laptop, Tablet or Mobile is almost the same. It uses Google TensorFlow for some of its features and thus you feel like using Google Photos in some regard. Some shortcomings and workarounds

• Facial Recognition: To date, facial recognition is a planned feature. But this was easy to tackle given that Digikam has pretty solid facial recognition. So I use Digikam to detect, recognize and annotate faces. And then PhotoPrism is happy enough to use those annotations and present relevant data.
• HTTP Web App: The upstream project has done a good job of making use of Docker container technology in presenting this web app solution. A software solution, the equivalent of a Google Photos, does need a heavyweight config. In free software, it is all about reusing available tools. PhotoPrism makes use of tools like MySQL, Vue.js, TensorFlow, GoLang, GoLibs and strives to provide a single package solution, all thanks to Docker containers.

In its current offering, PhotoPrism is run as a HTTP Web App. I wanted to have an added layer of security on top of it. And thus run an nginx reverse proxy on top of it. Along with it, I run the proxy service on HTTPS, thus making all traffic from clients to the proxy encrypted. I also wanted an added layer of HTTP AUTH on top, so I explored some options and finally settled down with http-auth-digest Also, in the current implementation, PhotoPrism doesn t have a strong notion of normal and private photos in its data organization. I wanted normal photos available under a standard auth realm and private photos under a different realm. And along with a different realm, I also wanted some added security directives for it. So far, it looks like I ve put together a decent solution with the help of nginx.

• First of all, since the port from the host is forwarded to the docker instance, that needed to be controlled. Instead of the default of listening on all interfaces, I changed it to loopback only. Because my primary and only interface is going to be the nginx reverse proxy
• Setup nginx with a self-signed certificate to have all communication encrypted.
• Setup nginx as a reverse proxy to talk to PhotoPrism.

  server  
     listen 80;
     listen [::]:80;
     server_name lenovo;
     return 302 https://$host$request_uri;
     rewrite ^ https://$http_host$request_uri? permanent;    # force redirect http to https
      server_tokens off;
   
  server  
  listen 443 ssl http2 default_server;
  listen [::]:443 ssl http2 default_server;
  ssl_certificate_key /etc/ssl/private/ssl-cert-snakeoil.key;
  ssl_certificate /etc/ssl/certs/ssl-cert-snakeoil.pem;
  server_tokens off;
  server_name lenovo;
  client_max_body_size 500M;
      auth_digest_expires 900s;
      auth_digest_evasion_time 5s;
      auth_digest_replays 500;
  location /private  
      auth_digest 'abc';
      auth_digest_user_file /etc/nginx/htdigest;
      auth_digest_expires 900s;
      auth_digest_evasion_time 5s;
      auth_digest_replays 500;
      proxy_pass http://localhost:2342/private;
   
  location /discover  
      auth_digest 'abc';
      auth_digest_user_file /etc/nginx/htdigest;
      auth_digest_expires 900s;
      auth_digest_evasion_time 5s;
      auth_digest_replays 500;
      proxy_pass http://localhost:2342/discover;
   
  location /  
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header Host $host;
      proxy_buffering off;
      proxy_http_version 1.1;
      proxy_set_header Upgrade $http_upgrade;
      proxy_set_header Connection "upgrade";
      proxy_pass http://localhost:2342;
        
      auth_basic "Priv";
      auth_basic_user_file /etc/nginx/htpasswd;
   
   
  

• nginx packaging: I was so happy to see the simplicity of the nginx packaging in Debian. Becuase http-auth-digest is not upstream, it needs to be pulled in separately and compiled. I was happy to see how simple the packaging structure of nginx in Debian is. It was just a matter of putting in the module in the right modules location, which as structured such in the packaging falls under the debian/ packaging sub-folder, that any future upgrades will be quite easy to manage.
• http-auth-digest: I d love to see http-auth-digest module be part of the upstream package. While I m a web n00b, I felt this module perfect for my use case. From what I ve understood, set up and tested so far, this module fills in all my requirements; which is more of like a session management.

With the combined setup in place, https://host is authenticated with a different set of credentials. On the other hand, https://host/discover and /private get covered with a different set of credentials and policies. While, this will continue to be an ongoing effort and audit of the services that I build, so far now, I feel this is in decent shape that I can use it as my daily driver. The end result is:

nginx auth prompt

PhotoPrism Photo List

PhotoPrism Menu

Updates

• Things can really get frustrating at times. These days, it is Google that contributes to it. https://bugs.chromium.org/p/chromium/issues/detail?id=544244
  • It is is now fair to say that Google is big enough to rewrite all the standards. Or at least break them. Or best, their arrogance of it. This bug report is a fine example of what other ways it could have been dealt.

# Install docker + nftables from Debian
apt install docker.io nftables
# Add the Kubernetes repo and signing key
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg > /etc/apt/k8s.gpg
cat > /etc/apt/sources.list.d/kubernetes.list <<EOF
deb [signed-by=/etc/apt/k8s.gpg] https://apt.kubernetes.io/ kubernetes-xenial main
EOF
apt update
apt install kubelet kubeadm kubectl

kubeadm init --apiserver-advertise-address 192.168.53.147 --apiserver-cert-extra-sans udon.mynetwork

mkdir ~noodles/.kube
cp -i /etc/kubernetes/admin.conf ~noodles/.kube/config
chown -R noodles ~noodles/.kube/

noodles@udon:~$ kubectl get nodes
NAME   STATUS     ROLES                  AGE     VERSION
udon   NotReady   control-plane,master   4m31s   v1.21.1

mkdir -p /var/lib/weave
head -c 16 /dev/urandom   shasum -a 256   cut -d " " -f1 > /var/lib/weave/weave-passwd
kubectl create secret -n kube-system generic weave-passwd --from-file=/var/lib/weave/weave-passwd
kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version   base64   tr -d '\n')&password-secret=weave-passwd&env.IPALLOC_RANGE=192.168.0.0/24"

kubectl taint nodes --all node-role.kubernetes.io/master-

noodles@udon:~$ kubectl get nodes
NAME   STATUS   ROLES                  AGE   VERSION
udon   Ready    control-plane,master   15m   v1.21.1

noodles@udon:~$ kubectl get pods --all-namespaces
NAMESPACE     NAME                           READY   STATUS    RESTARTS   AGE
kube-system   coredns-558bd4d5db-4nvrg       1/1     Running   0          18m
kube-system   coredns-558bd4d5db-flrfq       1/1     Running   0          18m
kube-system   etcd-udon                      1/1     Running   0          18m
kube-system   kube-apiserver-udon            1/1     Running   0          18m
kube-system   kube-controller-manager-udon   1/1     Running   0          18m
kube-system   kube-proxy-6d8kg               1/1     Running   0          18m
kube-system   kube-scheduler-udon            1/1     Running   0          18m
kube-system   weave-net-mchmg                2/2     Running   1          3m26s

$ kubectl create deployment hello-node --image=k8s.gcr.io/echoserver:1.10
deployment.apps/hello-node created
$ kubectl get pod
NAME                          READY   STATUS    RESTARTS   AGE
hello-node-59bffcc9fd-8hkgb   1/1     Running   0          36s
$ kubectl expose deployment hello-node --type=NodePort --port=8080
$ kubectl get services
NAME         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
hello-node   NodePort    10.107.66.138   <none>        8080:31529/TCP   1m

Hostname: hello-node-59bffcc9fd-8hkgb
Pod Information:
	-no pod information available-
Server values:
	server_version=nginx: 1.13.3 - lua: 10008
Request Information:
	client_address=192.168.53.147
	method=GET
	real path=/
	query=
	request_version=1.1
	request_scheme=http
	request_uri=http://10.107.66.138:8080/
Request Headers:
	accept=*/*
	host=10.107.66.138:8080
	user-agent=curl/7.74.0
Request Body:
	-no body in request-

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v0.46.0/deploy/static/provider/cloud/deploy.yaml

cat > hello-ingress.yaml <<EOF
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: example-ingress
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /$1
spec:
  rules:
    - host: udon.mynetwork
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: hello-node
                port:
                  number: 8080
EOF

$ kubectl apply -f hello-ingress.yaml
ingress.networking.k8s.io/example-ingress created
$ kubectl get ingress
NAME              CLASS    HOSTS            ADDRESS   PORTS   AGE
example-ingress   <none>   udon.mynetwork             80      3m8s

noodles@udon:~$ kubectl get services -n ingress-nginx
NAME                                 TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                    AGE
ingress-nginx-controller             LoadBalancer   10.96.9.41      <pending>     80:32740/TCP,443:30894/TCP 13h
ingress-nginx-controller-admission   ClusterIP      10.111.16.129   <none>        443/TCP                    13h

kubectl patch svc ingress-nginx-controller -n ingress-nginx -p \
	' "spec":  "type": "LoadBalancer", "externalIPs":["192.168.53.147"] '

noodles@udon:~$ kubectl get services -n ingress-nginx
NAME                                 TYPE           CLUSTER-IP      EXTERNAL-IP      PORT(S)                 AGE
ingress-nginx-controller             LoadBalancer   10.96.9.41      192.168.53.147   80:32740/TCP,443:30894/TCP   14h
ingress-nginx-controller-admission   ClusterIP      10.111.16.129   <none>           443/TCP                 14h
noodles@udon:~$ kubectl get ingress
NAME              CLASS    HOSTS           ADDRESS          PORTS   AGE
example-ingress   <none>   udon.mynetwork  192.168.53.147   80      14h

Hostname: hello-node-59bffcc9fd-8hkgb
Pod Information:
	-no pod information available-
Server values:
	server_version=nginx: 1.13.3 - lua: 10008
Request Information:
	client_address=192.168.0.5
	method=GET
	real path=/
	query=
	request_version=1.1
	request_scheme=http
	request_uri=http://udon.mynetwork:8080/
Request Headers:
	accept=*/*
	host=udon.mynetwork
	user-agent=curl/7.64.0
	x-forwarded-for=192.168.53.136
	x-forwarded-host=udon.mynetwork
	x-forwarded-port=80
	x-forwarded-proto=http
	x-real-ip=192.168.53.136
	x-request-id=6aaef8feaaa4c7d07c60b2d05c45f75c
	x-scheme=http
Request Body:
	-no body in request-

• What s going on with the networking?
• Where s the IPv6 (the host in question has native IPv6 routing)?
• How do I deploy my own application pod?
• I should look at a multiple host setup (i.e. a real cluster).
• How much of this goes away if I use a cloud provider like AWS to run the cluster?
• Can I do this with the Debian Kubernetes packages?

Host foo.company.com
Port 22
Host *.company.com
Port 10022

Host *.company.com
Port 10022
Host foo.company.com
Port 22

• An AWS account with admin rights
• Full control over the domain name you use to send your newsletter
• A baremetal or virtual Linux Debian server you have the root access to
• NodeJS installed (8x and 10x are ok with Mailtrain)
• A MySQL/MariaDB instance with the root access to
• Redis 3x (not 5) if you want to use Redis (not mandatory)

• Configure AWS SES
• Configure your server by:
  • configuring your database
  • install Mailtrain
  • configuring your web server
• Configure your Mailtrain setup

Ask AWS SES to verify a domain

Generate the DKIM settings

AWS SES pending verification

AWS SES SMTP settings and credentials

MariaDB [(none)]> create database mailtrain;
Query OK, 1 row affected (0.00 sec)
MariaDB [(none)]> CREATE USER 'mailtrain' IDENTIFIED BY 'V3rYD1fF1cUlTP4sSW0rd!';
Query OK, 0 rows affected (0.01 sec)
MariaDB [(none)]> GRANT ALL PRIVILEGES ON mailtrain.* TO 'mailtrain'@localhost IDENTIFIED BY 'V3rYD1fF1cUlTP4sSW0rd!';
Query OK, 0 rows affected (0.00 sec)
MariaDB [(none)]> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.00 sec)
MariaDB [(none)]> show databases;
+--------------------+
  Database            
+--------------------+
  information_schema  
  mailtrain           
  mysql               
  performance_schema  
+--------------------+
6 rows in set (0.00 sec)
MariaDB [(none)]> Bye

[mysql]
host="localhost"
user="mailtrain"
password="V3rYD1ff1culT!"
database="mailtrain"

screen
NODE_ENV=production npm start

groupadd mailtrain
useradd -g mailtrain
chown -R mailtrain:mailtrain /var/www/mailtrain 

[Unit]
 Description=mailtrain
 After=network.target
[Service]
 Type=simple
 User=mailtrain
 WorkingDirectory=/var/www/mailtrain/
 Environment="NODE_ENV=production"
 Environment="PORT=3000"
 ExecStart=/usr/bin/npm run start
 TimeoutSec=15
 Restart=always
[Install]
 WantedBy=multi-user.target

systemctl daemon-reload
systemctl start mailtrain.service

map $http_upgrade $connection_upgrade  
  default upgrade;
  ''      close;
 
server  
  listen 80; 
  listen [::]:80;
  server_name mailtrain.toto.com;
  return 301 https://$host$request_uri;
 
server  
  listen 443 ssl;
  listen [::]:443 ssl;
  server_name mailtrain.toto.com;
  access_log /var/log/nginx/mailtrain.toto.com.access.log;
  error_log /var/log/nginx/mailtrain.toto.com.error.log;
  ssl_protocols TLSv1.2;
  ssl_ciphers EECDH+AESGCM:EECDH+AES;
  ssl_ecdh_curve prime256v1;
  ssl_prefer_server_ciphers on; 
  ssl_session_cache shared:SSL:10m;
  ssl_certificate     /etc/letsencrypt/live/mailtrain.toto.com/fullchain.pem;
  ssl_certificate_key /etc/letsencrypt/live/mailtrain.toto.com/privkey.pem;
  keepalive_timeout    70; 
  sendfile             on;
  client_max_body_size 0;
  root /var/www/mailtrain;
  location ~ /\.well-known\/acme-challenge  
    allow all;
   
  gzip on; 
  gzip_disable "msie6";
  gzip_vary on; 
  gzip_proxied any;
  gzip_comp_level 6;
  gzip_buffers 16 8k; 
  gzip_http_version 1.1;
  gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;
  add_header Strict-Transport-Security "max-age=31536000";
  location /   
    try_files $uri @proxy;
   
  location @proxy  
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto https;
    proxy_pass http://127.0.0.1:3000;
   
 

systemctl start nginx

Mailtrain mailer setup

Mailtrain to throttle sending emails to AWS SES

• Carl Chenet s blog
• Twitter account
• Mastodon account

The News

I joined Canonical, this February, to work on Ubuntu full-time! \o/
Those who know, they know that this is really very exciting for me because Canonical has been a dream company for me, for real (more about this below!). And hey, this is my first job, ever, so all the more reason to be psyched about, isn t it? ^_^ P.S. Keep reading and we ll meet my squad really sooon!

The Story Being an undergrad student (batch 2017-2021), I ve been slightly worried during my last two semesters, naturally, thinking about how s it all gonna pan out and what will I be doing, et al, because I ve been seeing all my friends and batchmates getting placed in companies or going for masters or at least having some sort of plans for their future and I, on the other hand, was hopelessly clueless. :D Well, to be fair, I did Google Summer of Code twice, in 2019 and 2020, became a Debian Developer in 2019, been a part of GCI and Outreachy, contributed to over dozens of open-source projects, et al, et al. So I wasn t all completely hopeless but for sure was completely clueless , heh. And for full disclosure, I was only slightly panicking because firstly, I did get placed in several companies and secondly, I didn t really need a job immediately since I was already getting paid to work on Debian stuff by Freexian, which was good enough. :)
(and honestly, Freexian has my whole heart! - more on that later sometime.) But that s not the point. I was still confused and worried and my mom & dad, more so than anyone. Ugh. We were all figuring out and she asked me places that I was interested to work in. And whilst I wasn t clear about things I wanted to do (and still am!) but I was (very) clear about this and so I told her about Canonical and also did tell her that it s a bit too ambitious for me to think about it now so I ll probably apply after some experience or something. and as they say, the world works in mysterious ways and well, it did for me! So back during the Ruby sprints (Feb 20), Kanashiro, the guy ( ), mentioned that his team was hiring and has a vacant position but I won t be eligible since I was still in my junior year. It was since then I ve been actively praying for Cronus, the god of time, to wave his magic wand and align it in such a way that the next opening should be somewhere near my graduation. And guess what? IT HAPPENED! 9 months later, in November 20, Kanashiro told me his team is hiring yet again and that I could apply this time! Without much (since there was some ) delay, I applied and started asking all sorts of questions to Kanashiro. No words are enough for him, he literally helped me throughout the process; from referring me to answering all sorts of doubts I had! And roughly after 2 months of interviewing, et al, my ambitious dream did come true and I finalyyyy signed my contract! \o/
(the interview process and what went on during those 10 weeks is a story for later ;))

The Server Team! \o This position, which I didn t mention earlier, was for the Server Team which is a team of 15 people, working to make Ubuntu server the best! And as I tweeted sometime back, the team is absolutely lovely, super kind, and consists of the best of teammates one could possibly ask for! Here s a quick sneak peek into our weekly team meeting. Thanks to Rafael for taking such a lovely picture. And yes, the cat Luna is a part of our squad! And oh, did I mention that we re completely remote and distributed?
FUN FACT: Our team covers all the TZs, that is, at any point of time (during weekdays), you ll find someone or the other from the team around! \o/ Anyway, our squad, managed by Rick is divided into two halves: Squeaky Wheels and Table Flip. Cool names, right?
Squeaky Wheels does the distro side of stuff and consists of Christian, Andreas, Rafael, Robie, Bryce, Sergio, Kanashiro, Athos, and now myself as well! And OTOH, Table Flip consists of Dan, Chad, Paride, Lucas, James, and Grant. Even though I interact w/ Squeaky Wheels more (basically daily), each of my teammates is absolutely lovely and equally awesome! Whilst I ll talk more about things here in the upcoming months, this is it for now! If there s anything, in particular, you d like to know more about, let me know! And lastly, here s us vibing our way through, making Ubuntu server better, cause that s how we roll!

---

Until next time.
:wq for today.

The Population Issue and its many facets Another couple of weeks passed. A Lot of things happening, lots of anger and depression in folks due to handling in pandemic, but instead of blaming they are willing to blame everybody else including the population. Many of them want forced sterilization like what Sanjay Gandhi did during the Emergency (1975). I had to share So Long, My son . A very moving tale of two families of what happened to them during the one-child policy in China. I was so moved by it and couldn t believe that the Chinese censors allowed it to be produced, shot, edited, and then shared worldwide. It also won a couple of awards at the 69th Berlin Film Festival, silver bear for the best actor and the actress in that category. But more than the award, the theme, and the concept as well as the length of the movie which was astonishing. Over a 3 hr. something it paints a moving picture of love, loss, shame, relief, anger, and asking for forgiveness. All of which can be identified by any rational person with feelings worldwide.

Girl child What was also interesting though was what it couldn t or wasn t able to talk about and that is the Chinese leftover men. In fact, a similar situation exists here in India, only it has been suppressed. This has been more pronounced more in Asia than in other places. One big thing in this is human trafficking and mostly women trafficking. For the Chinese male, that was happening on a large scale from all neighboring countries including India. This has been shared in media and everybody knows about it and yet people are silent. But this is not limited to just the Chinese, even Indians have been doing it. Even yesteryear actress Rupa Ganguly was caught red-handed but then later let off after formal questioning as she is from the ruling party. So much for justice. What is and has been surprising at least for me is Rwanda which is in the top 10 of some of the best places in equal gender. It, along with other African countries have also been in news for putting quite a significant amount of percentage of GDP into public healthcare (between 20-10%), but that is a story for a bit later. People forget or want to forget that it was in Satara, a city in my own state where 220 girls changed their name from nakusha or unwanted to something else and that had become a piece of global news. One would think that after so many years, things would have changed, the only change that has happened is that now we have two ministries, The Ministry of Women and Child Development (MoWCD) and The Ministry of Health and Welfare (MoHFW). Sadly, in both cases, the ministries have been found wanting, Whether it was the high-profile Hathras case or even the routine cries of help which given by women on the twitter helpline. Sadly, neither of these ministries talks about POSH guidelines which came up after the 2012 gangrape case. For both these ministries, it should have been a pinned tweet. There is also the 1994 PCPNDT Act which although made in 1994, actually functioned in 2006, although what happens underground even today nobody knows . On the global stage, about a decade ago, Stephen J. Dubner and Steven Levitt argued in their book Freakonomics how legalized abortion both made the coming population explosion as well as expected crime rates to be reduced. There was a huge pushback on the same from the conservatives and has become a matter of debate, perhaps something that the Conservatives wanted. Interestingly, it hasn t made them go back but go forward as can be seen from the Freakonomics site.

Climate Change Another topic that came up for discussion was repeatedly climate change, but when I share Shell s own 1998 Confidential report titled Greenhouse effect all become strangely silent. The silence here is of two parts, there probably is a large swathe of Indians who haven t read the report and there may be a minority who have read it and know what already has been shared with U.S. Congress. The Conservative s argument has been for it is jobs and a weak we need to research more . There was a partial debunk of it on the TBD podcast by Matt Farell and his brother Sean Farell as to how quickly the energy companies are taking to the coming change.

Health Budget Before going to Covid stories. I first wanted to talk about Health Budgets. From the last 7 years the Center s allocation for health has been between 0.34 to 0.8% per year. That amount barely covers the salaries to the staff, let alone any money for equipment or anything else. And here by allocation I mean, what is actually spent, not the one that is shared by GOI as part of budget proposal. In fact, an article on Wire gives a good breakdown of the numbers. Even those who are on the path of free markets describe India s health business model as a flawed one. See the Bloomberg Quint story on that. Now let me come to Rwanda. Why did I chose Rwanda, I could have chosen South Africa where I went for Debconf 2016, I chose because Rwanda s story is that much more inspiring. In many ways much more inspiring than that South Africa in many ways. Here is a country which for decades had one war or the other, culminating into the Rwanda Civil War which ended in 1994. And coincidentally, they gained independence on a similar timeline as South Africa ending Apartheid in 1994. What does the country do, when it gains its independence, it first puts most of its resources in the healthcare sector. The first few years at 20% of GDP, later than at 10% of GDP till everybody has universal medical coverage. Coming back to the Bloomberg article I shared, the story does not go into the depth of beyond-expiry date medicines, spurious medicines and whatnot. Sadly, most media in India does not cover the deaths happening in rural areas and this I am talking about normal times. Today what is happening in rural areas is just pure madness. For last couple of days have been talking with people who are and have been covering rural areas. In many of those communities, there is vaccine hesitancy and why, because there have been whatsapp forwards sharing that if you go to a hospital you will die and your kidney or some other part of the body will be taken by the doctor. This does two things, it scares people into not going and getting vaccinated, at the same time they are prejudiced against science. This is politics of the lowest kind. And they do it so that they will be forced to go to temples or babas and what not and ask for solutions. And whether they work or not is immaterial, they get fixed and property and money is seized. Sadly, there are not many Indian movies of North which have tried to show it except for oh my god but even here it doesn t go the distance. A much more honest approach was done in Trance . I have never understood how the South Indian movies are able to do a more honest job of story-telling than what is done in Bollywood even though they do in 1/10th the budget that is needed in Bollywood. Although, have to say with OTT, some baggage has been shed but with the whole film certification rearing its ugly head through MEITY orders, it seems two steps backward instead of forward. The idea being simply to infantilize the citizens even more. That is a whole different ball-game which probably will require its own space.

Vaccine issues One good news though is that Vaccination has started. But it has been a long story full of greed by none other than GOI (Government of India) or the ruling party BJP. Where should I start with. I probably should start with this excellent article done by Priyanka Pulla. It is interesting and fascinating to know how vaccines are made, at least one way which she shared. She also shared about the Cutter Incident which happened in the late 50 s. The response was on expected lines, character assassination of her and the newspaper they published but could not critique any of the points made by her. Not a single point that she didn t think about x or y. Interestingly enough, in January 2021 Bharati Biotech was supposed to be share phase 3 trial data but hasn t been put up in public domain till May 2021. In fact, there have been a few threads raised by both well-meaning Indians as well as others globally especially on twitter to which GOI/ICMR (Indian Council of Medical Research) is silent. Another interesting point to note is that Russia did say in its press release that it is possible that their vaccine may not be standard (read inactivation on their vaccines and another way is possible but would take time, again Brazil has objected, but India hasn t till date.) What also has been interesting is the homegrown B.1.617 lineage or known as double mutant . This was first discovered from my own state, Maharashtra and then transported around the world. There is also B.1.618 which was found in West Bengal and is same or supposed to be similar to the one found in South Africa. This one is known as Triple mutant . About B.1.618 we don t know much other than knowing that it is much more easily transferable, much more infectious. Most countries have banned flights from India and I cannot fault them anyway. Hell, when even our diplomats do not care for procedures to be followed during the pandemic then how a common man is supposed to do. Of course, now for next month, Mr. Modi was supposed to go and now will not attend the G7 meeting. Whether, it is because he would have to face the press (the only Prime Minister and the only Indian Prime Minister who never has faced free press.) or because the Indian delegation has been disinvited, we would never know.

A good article which shares lots of lows with how things have been done in India has been an article by Arundhati Roy. And while the article in itself is excellent and shares a bit of the bitter truth but is still incomplete as so much has been happening. The problem is that the issue manifests in so many ways, it is difficult to hold on. As Arundhati shared, should we just look at figures and numbers and hold on, or should we look at individual ones, for e.g. the one shared in Outlook India. Or the one shared by Dr. Dipshika Ghosh who works in Covid ICU in some hospital

Dr. Dipika Ghosh sharing an incident in Covid Ward

Interestingly as well, while in the vaccine issue, Brazil Anvisa doesn t know what they are doing or the regulator just isn t knowledgeable etc. (statements by various people in GOI, when it comes to testing kits, the same is an approver.)

ICMR/DGCI approving internationally validated kits, Press release.

Twitter In the midst of all this, one thing that many people have forgotten and seem to have forgotten that Twitter and other tools are used by only the elite. The reason why the whole thing has become serious now than in the first phase is because the elite of India have also fallen sick and dying which was not the case so much in the first phase. The population on Twitter is estimated to be around 30-34 million and people who are everyday around 20 odd million or so, which is what 2% of the Indian population which is estimated to be around 1.34 billion. The other 98% don t even know that there is something like twitter on which you can ask help. Twitter itself is exclusionary in many ways, with both the emoticons, the language and all sorts of things. There is a small subset who does use Twitter in regional languages, but they are too small to write anything about. The main language is English which does become a hindrance to lot of people.

Censorship Censorship of Indians critical of Govt. mishandling has been non-stop. Even U.S. which usually doesn t interfere into India s internal politics was forced to make an exception. But of course, this has been on deaf ears. There is and was a good thread on Twitter by Gaurav Sabnis, a friend, fellow Puneite now settled in U.S. as a professor.

Gaurav on Trump-Biden on vaccination of their own citizens

Now just to surmise what has been happened in India and what has been happening in most of the countries around the world. Most of the countries have done centralization purchasing of the vaccine and then is distributed by the States, this is what we understand as co-operative federalism. While last year, GOI took a lot of money under the shady PM Cares fund for vaccine purchase, donations from well-meaning Indians as well as Industries and trade bodies. Then later, GOI said it would leave the states hanging and it is they who would have to buy vaccines from the manufacturers. This is again cheap politics. The idea behind it is simple, GOI knows that almost all the states are strapped for cash. This is not new news, this I have shared a couple of months back. The problem has been that for the last 6-8 months no GST meeting has taken place as shared by Punjab s Finance Minister Amarinder Singh. What will happen is that all the states will fight in-between themselves for the vaccine and most of them are now non-BJP Governments. The idea is let the states fight and somehow be on top. So, the pandemic, instead of being a public health issue has become something of on which politics has to played. The news on whatsapp by RW media is it s ok even if a million or two also die, as it is India is heavily populated. Although that argument vanishes for those who lose their dear and near ones. But that just isn t the issue, the issue goes much more deeper than that Oxygen:12%
Remedisivir:12%
Sanitiser:12%
Ventilator:12%
PPE:18%
Ambulances 28% Now all the products above are essential medical equipment and should be declared as essential medical equipment and should have price controls on which GST is levied. In times of pandemic, should the center be profiting on those. States want to let go and even want the center to let go so that some relief is there to the public, while at the same time make them as essential medical equipment with price controls. But GOI doesn t want to. Leaders of opposition parties wrote open letters but no effect. What is sad to me is how Ambulances are being taxed at 28%. Are they luxury items or sin goods ? This also reminds of the recent discovery shared by Mr. Pappu Yadav in Bihar. You can see the color of ambulances as shared by Mr. Yadav, and the same news being shared by India TV news showing other ambulances. Also, the weak argument being made of not having enough drivers. Ideally, you should have 2-3 people, both 9-1-1 and Chicago Fire show 2 people in ambulance but a few times they have also shown to be flipped over. European seems to have three people in ambulance, also they are also much more disciplined as drivers, at least an opinion shared by an American expat.

Pappu Yadav, President Jan Adhikar Party, Bihar May 11, 2021

What is also interesting to note is GOI plays this game of Health is State subject and health is Central subject depending on its convenience. Last year, when it invoked the Epidemic and DMA Act it was a Central subject, now when bodies are flowing down the Ganges and pyres being lit everywhere, it becomes a State subject. But when and where money is involved, it again becomes a Central subject. The States are also understanding it, but they are fighting on too many fronts.

Snippets from Karnataka High Court hearing today, 13th March 2021

One of the good things is most of the High Courts have woken up. Many of the people on the RW think that the Courts are doing Judicial activism . And while there may be an iota of truth in it, the bitter truth is that many judges or relatives or their helpers have diagnosed and some have even died due to Covid. In face of the inevitable, what can they do. They are hauling up local Governments to make sure they are accountable while at the same time making sure that they get access to medical facilities. And I as a citizen don t see any wrong in that even if they are doing it for selfish reasons. Because, even if justice is being done for selfish reasons, if it does improve medical delivery systems for the masses, it is cool. If it means that the poor and everybody else are able to get vaccinations, oxygen and whatever they need, it is cool. Of course, we are still seeing reports of patients spending in the region of INR 50k and more for each day spent in hospital. But as there are no price controls, judges cannot do anything unless they want to make an enemy of the medical lobby in the country. A good story on medicines and what happens in rural areas, see no further than Laakhon mein ek.

Allahabad High Court hauling Uttar Pradesh Govt. for lack of Oxygen is equal to genocide, May 11, 2021

The censorship is not just related to takedown requests on twitter but nowadays also any articles which are critical of the GOI s handling. I have been seeing many articles which have shared facts and have been critical of GOI being taken down. Previously, we used to see 404 errors happen 7-10 years down the line and that was reasonable. Now we see that happen, days weeks or months. India seems to be turning more into China and North Korea and become more anti-science day-by-day

Fake websites Before going into fake websites, let me start with a fake newspaper which was started by none other than the Gujarat CM Mr. Modi in 2005 .

Gujarat Satya Samachar 2005 launched by Mr. Modi.

And if this wasn t enough than on Feb 8, 2005, he had invoked Official Secrets Act

Mr. Modi invoking Official Secrets Act, Feb 8 2005 Gujarat Samachar

The headlines were In Modi s regime press freedom is in peril-Down with Modi s dictatorship. So this was a tried and tested technique. The above information was shared by Mr. Urvish Kothari, who incidentally also has his own youtube channel. Now cut to 2021, and we have a slew of fake websites being done by the same party. In fact, it seems they started this right from 2011. A good article on BBC itself tells the story. Hell, Disinfo.eu which basically combats disinformation in EU has a whole pdf chronicling how BJP has been doing it. Some of the sites it shared are

Times of New York
Manchester Times
Times of Los Angeles
Manhattan Post
Washington Herald
and many more. The idea being take any site name which sounds similar to a brand name recognized by Indians and make fool of them. Of course, those of who use whois and other such tools can easily know what is happening. Two more were added to the list yesterday, Daily Guardian and Australia Today. There are of course, many features which tell them apart from genuine websites. Most of these are on shared hosting rather than dedicated hosting, most of these are bought either from Godaddy and Bluehost. While Bluehost used to be a class act once upon a time, both the above will do anything as long as they get money. Don t care whether it s a fake website or true. Capitalism at its finest or worst depending upon how you look at it. But most of these details are lost on people who do not know web servers, at all and instead think see it is from an exotic site, a foreign site and it chooses to have same ideas as me. Those who are corrupt or see politics as a tool to win at any cost will not see it as evil. And as a gentleman Raghav shared with me, it is so easy to fool us. An example he shared which I had forgotten. Peter England which used to be an Irish brand was bought by Aditya Birla group way back in 2000. But even today, when you go for Peter England, the way the packaging is done, the way the prices are, more often than not, people believe they are buying the Irish brand. While sharing this, there is so much of Naom Chomsky which comes to my mind again and again

Caste Issues I had written about caste issues a few times on this blog. This again came to the fore as news came that a Hindu sect used forced labor from Dalit community to make a temple. This was also shared by the hill. In both, Mr. Joshi doesn t tell that if they were volunteers then why their passports have been taken forcibly, also I looked at both minimum wage prevailing in New Jersey as a state as well as wage given to those who are in the construction Industry. Even in minimum wage, they were giving $1 when the prevailing minimum wage for unskilled work is $12.00 and as Mr. Joshi shared that they are specialized artisans, then they should be paid between $23 $30 per hour. If this isn t exploitation, then I don t know what is. And this is not the first instance, the first instance was perhaps the case against Cisco which was done by John Doe. While I had been busy with other things, it seems Cisco had put up both a demurrer petition and a petition to strike which the Court stayed. This seemed to all over again a type of apartheid practice, only this time applied to caste. The good thing is that the court stayed the petition. Dr. Ambedkar s statement if Hindus migrate to other regions on earth, Indian caste would become a world problem given at Columbia University in 1916, seems to be proven right in today s time and sadly has aged well. But this is not just something which is there only in U.S. this is there in India even today, just couple of days back, a popular actress Munmun Dutta used a casteist slur and then later apologized giving the excuse that she didn t know Hindi. And this is patently false as she has been in the Bollywood industry for almost now 16-17 years. This again, was not an isolated incident. Seema Singh, a lecturer in IIT-Kharagpur abused students from SC, ST backgrounds and was later suspended. There is an SC/ST Atrocities Act but that has been diluted by this Govt. A bit on the background of Dr. Ambedkar can be found at a blog on Columbia website. As I have shared and asked before, how do we think, for what reason the Age of Englightenment or the Age of Reason happened. If I were a fat monk or a priest who was privileges, would I have let Age of Enlightenment happen. It broke religion or rather Church which was most powerful to not so powerful and that power was more distributed among all sort of thinkers, philosophers, tinkers, inventors and so on and so forth.

Situation going forward I believe things are going to be far more complex and deadly before they get better. I had to share another term called Comorbidities which fortunately or unfortunately has also become part of twitter lexicon. While I have shared what it means, it simply means when you have an existing ailment or condition and then Coronavirus attacks you. The Virus will weaken you. The Vaccine in the best case just stops the damage, but the damage already done can t be reversed. There are people who advise and people who are taking steroids but that again has its own side-effects. And this is now, when we are in summer. I am afraid for those who have recovered, what will happen to them during the Monsoons. We know that the Virus attacks most the lungs and their quality of life will be affected. Even the immune system may have issues. We also know about the inflammation. And the grant that has been given to University of Dundee also has signs of worry, both for people like me (obese) as well as those who have heart issues already. In other news, my city which has been under partial lockdown since a month, has been extended for another couple of weeks. There are rumors that the same may continue till the year-end even if it means economics goes out of the window.There is possibility that in the next few months something like 2 million odd Indians could die

The above is a conversation between Karan Thapar and an Oxford Mathematician Dr. Murad Banaji who has shared that the under-counting of cases in India is huge. Even BBC shared an article on the scope of under-counting. Of course, those on the RW call of the evidence including the deaths and obituaries in newspapers as a narrative . And when asked that when deaths used to be in the 20 s or 30 s which has jumped to 200-300 deaths and this is just the middle class and above. The poor don t have the money to get wood and that is the reason you are seeing the bodies in Ganges whether in Buxar Bihar or Gajipur, Uttar Pradesh. The sights and visuals makes for sorry reading

Pandit Ranjan Mishra son on his father s death due to unavailability of oxygen, Varanasi, Uttar Pradesh, 11th May 2021.

For those who don t know Pandit Ranjan Mishra was a renowned classical singer. More importantly, he was the first person to suggest Mr. Modi s name as a Prime Ministerial Candidate. If they couldn t fulfil his oxygen needs, then what can be expected for the normal public.

Conclusion Sadly, this time I have no humorous piece to share, I can however share a documentary which was shared on Feluda . I have shared about Feluda or Prodosh Chandra Mitter a few times on this blog. He has been the answer of James Bond from India. I have shared previously about The Golden Fortress . An amazing piece of art by Satyajit Ray. I watched that documentary two-three times. I thought, mistakenly that I am the only fool or fan of Feluda in Pune to find out that there are people who are even more than me. There were so many facets both about Feluda and master craftsman Satyajit Ray that I was unaware about. I was just simply amazed. I even shared few of the tidbits with mum as well, although now she has been truly hooked to Korean dramas. The only solace from all the surrounding madness. So, if you have nothing to do, you can look up his books, read them and then see the movies. And my first recommendation would be the Golden Fortress. The only thing I would say, do not have high hopes. The movie is beautiful. It starts slow and then picks up speed, just like a train. So, till later. Update The Mass surveillance part I could not do justice do hence removed it at the last moment. It actually needs its whole space, article. There is so much that the Govt. is doing under the guise of the pandemic that it is difficult to share it all in one article. As it is, the article is big

deb https://apt.kubernetes.io/ kubernetes-xenial main

gpg --recv-key 6A030B21BA07F4FB
gpg --list-sigs 6A030B21BA07F4FB
gpg --export 6A030B21BA07F4FB   apt-key add -

apt install libvirt-clients libvirt-daemon-system qemu-kvm dnsmasq kubectl

apt install libvirt-clients libvirt-daemon-system qemu-system-x86 dnsmasq kubectl

# get overview of virsh commands
virsh help
# list domains
virsh --connect qemu:///system list
# list block devices a domain uses
virsh --connect qemu:///system domblklist minikube
# show stats on block device usage
virsh --connect qemu:///system domblkstat minikube hda
# list virtual networks
virsh --connect qemu:///system net-list
# list dhcp leases on a virtual network
virsh --connect qemu:///system net-dhcp-leases minikube-net
# list network filters
virsh --connect qemu:///system nwfilter-list
# list real network interfaces
virsh --connect qemu:///system iface-list

• [1] https://kubernetes.io/docs/tutorials/kubernetes-basics/
• [2] https://wiki.debian.org/KVM
• [3] https://kubernetes.io/docs/tasks/tools/install-minikube/

• [1] http://tinyurl.com/y7rutdur
• [2] https://en.wikipedia.org/wiki/Ad_hominem
• [3] https://tinyurl.com/yg68dtco

# getfattr -d -m - /etc/fstab 
getfattr: Removing leading '/' from absolute path names
# file: etc/fstab
security.evm=0sAwICqGOsfwCAvgE9y9OP74QxJ/I+3eOSF2n2dM51St98z/7LYHFd9rfGTvssvhTSYL9G8cTdRAH8ozggJu7VCzggW1REoTjnLcPeuMJsrMbW3DwVrB6ldDmJzyenLMjnIHmRDDeK309aRbLVn2ueJZ07aMDcSr+sxhOOAQ/GIW4SW8L1AKpKn4g=
security.ima=0sAT+Eivfxl+7FYI+Hr9K4sE6IieZ+
security.selinux="system_u:object_r:etc_t:s0"

• [1] https://wiki.gentoo.org/wiki/Extended_Verification_Module

• [DLA 2629-1] libebml security update for one CVE
• debdiff for libebml/buster
• [DLA 2636-1] pjproject security update for one CVE
• NMU leptonlib/unstable for four CVEs
• PU bug #987376 leptonlib/buster for four CVEs
• debdiff for ring/unstable which resulted in upload of version 20210112.2.b757bac~ds1-1 that fixed two CVEs
• PU bug #987246 tnef/buster for one CVE

  frodo ~ $ pass age
  6 years ago GPG/root@localhost.gpg
  6 years ago GPG/steve@steve.org.uk.OLD.gpg
  ..
  4 years, 8 months ago Domains/Domain.fi.gpg
  4 years, 7 months ago Mobile/dna.fi.gpg
  ..
  1 year, 3 months ago Websites/netlify.com.gpg
  1 year ago Financial/ukko.fi.gpg
  1 year ago Mobile/KiK.gpg
  4 days ago Enfuce/sre.tst.gpg
  ..

   username: steve
   password: secrit
   site: http://example.com/login/blah/
   # Extra data

     $ pass env internal/cli/tool-name
     export username="steve"
     export password="secrit"

   $ source <(pass env internal/cli/tool-name)
   $ echo username
   steve

   $ pass env --exec=$HOME/ldap/ldap.py internal/cli/tool-name
   you are steve
   ..

  frodo ~ $ pass age
  6 years ago GPG/root@localhost.gpg
  6 years ago GPG/steve@steve.org.uk.OLD.gpg
  ..
  4 years, 8 months ago Domains/Domain.fi.gpg
  4 years, 7 months ago Mobile/dna.fi.gpg
  ..
  1 year, 3 months ago Websites/netlify.com.gpg
  1 year ago Financial/ukko.fi.gpg
  1 year ago Mobile/KiK.gpg
  4 days ago Enfuce/sre.tst.gpg
  ..

   username: steve
   password: secrit
   site: http://example.com/login/blah/
   # Extra data

     $ pass env internal/cli/tool-name
     export username="steve"
     export password="secrit"

   $ source <(pass env internal/cli/tool-name)
   $ echo username
   steve

   $ pass env --exec=$HOME/ldap/ldap.py internal/cli/tool-name
   you are steve
   ..

• Principles in Machine Learning
• Principles in Machine Learning /2
• Occam s Razor Principle of Parsimony
• Simple Models
• Computational Learning Theory
• Probably Approximately Correct Learning (PAC Learning) /1
• Probably Approximately Correct Learning (PAC Learning) /2
• PAC Learnable Examples
• VC Dimension (Vapnik-Chervonenkis-Dimension )
• VC Dimension Example
• Error Bounds and the VC Dimension
• No Free Lunch
• No Free Lunch Theorem
• No Free Lunch Theorem Explanation
• Bias-Variance Tradeoff
• Bias-Variance Tradeoff
• Bias vs. Variance
• Bias-Variance Decomposition
• Bias-Variance Illustration
• Different Kinds of Bias
• Data Often Has Bias
• AI Can Be Sexist and Racist
• Relationships

• Correlation does not Imply Causation
• Correlation does not Imply Causation /2
• Correlation does not Imply Causation /3
• Correlation with Statistics Classes
• Multiple Testing Problem
• Bonferroni s Principle Multiple Testing Problem
• Multiple Testing Problem

• Overfitting
• Underfitting and Overfitting
• Overfitting Decision Tree
• Overfitting Due to Noise
• Overfitting Due to Insufficient Examples

• Curse of Dimensionality
• Combinatorial Explosion
• Concentration of Distances
• Data is in the Margins
• Illustration: Shrinking Hyperspheres
• Illustration: Shrinking Hyperspheres /2
• Effect on Search in High Dimensionality
• Summary

• Intrinsic Dimensionality
• Estimating Intrinsic Dimensionality
• Angle-Based Intrinsic Dimensionality Intuition
• Angle-Based Intrinsic Dimensionality (ABID) /2
• Consequences & Solutions

• Distance Functions
• Distances, Metrics and Similarities
• Distances, Metrics and Similarities /2
• Distance Functions
• Distance Functions /2
• Similarity Functions
• Distances for Binary Data
• Jaccard Coefficient for Sets
• Example Distances for Categorical Data
• Mahalanobis Distance
• Scaling & Normalization
• To Scale, or not to Scale?
• To Scale, or not to Scale? /2

• Classification
• Prediction Problems
• Classification: A Multi-Stage Process
• Classification Problem
• Example
• Process of Constructing a Model
• Process of Applying the Model

• Evaluation and Selection of Classifiers
• Quick Recap: Classification
• Classifier Evaluation: Confusion Matrix
• Classifier Evaluation: Accuracy and Error-Rate
• Precision, Recall, and F-measure
• Classifier Evaluation: Multi-Class Confusion Matrix
• Training Accuracy vs. Accuarcy on New Data
• The Need for Validation
• Holdout Validation
• Cross-Validation
• Bootstrap Validation
• Considerations for Selecting a Model

• Bayesian Classification
• Bayes Classification: Motivation
• Bayes Theorem: Review
• Optimal Bayes Classifier
• Na ve Bayes Classifier
• Probability Models for a Single Attribute
• Multivariate Gaussian Bayes Classification
• Na ve Bayes Classifier: Example
• Na ve Bayes Classifier: Computational Aspects
• Na ve Bayes Classifier: Comments & Discussion

• Nearest-Neighbor Classification
• Nearest Neighbor Classifier Motivation
• Nearest Neighbor Classifier: Foundations
• Nearest Neighbor Classifier: Example
• Nearest Neighbor Classification: Example
• Nearest Neighbor Decision Rules

• Nearest-Neighbor as Density estimation
• Nearest Neighbor Classification and Density Estimation
• Predicting with Kernel Density Estimation with k=1,3,5,15
• Error Probability of Nearest Neighbors
• Nearest Neighbor Regression
• Nearest-Neighbor Classification: Comments & Discussion

• Decision Tree Learning
• Example (Variant of a Dataset in )
• Decision Tree Example
• Decision Trees as Rule-based Systems
• Basic Notions
• Constructing a Decision Tree /1
• Visual Interpretation of Decision Trees on R
• Constructing a Decision Tree /2
• Decision Tree Classification: Example

• Decision Tree Splitting
• Split for Categorical Attributes
• Split for Numeric Attributes
• Best Split Example
• Quality Measures for Splits
• Measure of Impurity: Gini Index
• Gini-Index: Example
• Information Gain
• Information Gain: Example
• Information Gain: Gain-Ratio
• Gain-Ratio: Beispiel
• Classification Error
• Gini, Entropy and Classification Error
• Comparing Split Selection Measures
• Splits for Numerical Attributes

• Ensembles and Meta-Learning
• Ensembles and Meta-Learning
• Error-Rate of Ensembles
• Random Forests
• Boosting
• Random Forest Classification: Example
• Gradient Boosting Classification: Example

• Support Vector Machine Motivation
• Support Vector Machines
• Support Vector Machines /2
• Finding the Best Separating Hyperplane
• Maximum Margin Hyperplane

• Affine Hyperplanes and Scalar Products
• Affine Hyperplanes
• Scalar Products
• Affine Hyperplanes /2

• Maximum Margin Hyperplane
• A Na ve Attempt
• Support Vectors Separable Data
• Computing the Maximum Margin Hyperplane (MMH)
• Computing the Maximum Margin Hyperplane (MMH) /2
• Boundary of the Maximum Margin Hyperplane (MMH)
• Deriving the Primal SVM Optimization Problem

• Training Support Vector Machines
• Optimization Problem
• Karush-Kuhn-Tucker KKT Conditions
• Switching to the Dual Problem
• Classification with the Dual SVM
• Optimizing the i
• Optimizing SVMs
• Sequential Minimal Optimization
• Further Improvements

• Non-linear SVM and the Kernel Trick
• Nonlinear SVM
• Nonlinear SVM /2
• Kernel Functions
• Soft Margin SVM Classifier
• Soft Margin SVM Classifier /2
• Soft Margin SVM Classifier /3
• Soft Margin SVM Classifier /4

• SVM Extensions and Conclusions
• Separation of more than 2 Classes
• Support Vector Regression
• Support Vector Regression Optimization Problem
• Support Vector Regression Dual
• Support Vector Data Description (SVDD)
• SVDD Dual Problem
• Support Vector Clustering
• SVMs: Comments & Discussion

• Neural Networks
• Biological Background (Simplified)
• Biological Background (Simplified) /2

• Threshold Logic Units
• Threshold Logic Units (TLUs)
• Threshold Logic Units Example
• Geometric Interpretation of TLUs
• Exclusive-Or (XOR) Problem
• Exclusive-Or (XOR) Problem /2
• Exclusive-Or (XOR) Problem /3
• Universality of TLUs
• Mark I Perceptron

• General Artificial Neural Networks
• Simplifying Threshold Logic Units
• Weight Matrices
• From TLUs to Multilayer Perceptrons
• Some Activation Functions
• Some Activation Functions /2
• Some Activation Functions /3
• Some Activation Functions /4

• Learning Neural Networks with Backpropagation
• Basic Gradient Descent
• Stochastic Gradient Descent
• Learning Single-Layer Perceptrons
• Backpropagation
• Training with Backpropagation

• Deep Neural Networks
• Universal Approximation Theorem
• Deep vs. Wide Neural Networks
• High vs. Low Dimensionality
• (Early) Problems of Deep Learning
• Autoencoders
• Layer-wise Pre-Training of Deep Neural Networks
• Dropout Regularization
• Batch Normalization
• Choosing Activation Functions

• Convolutional Neural Networks
• Depth vs. Convolutional Kernel Size
• Increasing the Training Data

• Recurrent Neural Networks
• Recurrent Neural Networks (RNNs) on Sequences
• Recurrent Neural Networks (RNN)
• Long-Short Term Memory (LSTM)
• Further Developments

• Conclusion
• Other Classifiers
• Problems of Classification

• Cluster Analysis Introduction
• What is Clustering?
• What is Clustering? /2
• Applications of Clustering
• Basic Steps for Clustering

• Hierarchical Agglomerative Clustering
• Distance of Clusters
• AGNES Agglomerative Nesting
• AGNES Agglomerative Nesting /2
• Extracting Clusters from a Dendrogram
• Benefits and Limitations of HAC

• Accelerating Hierarchical Clustering
• Complexity of Hierarchical Clustering
• Anderberg s Caching
• AGNES vs. Anderberg , NNChain , SLINK
• Example: Hierarchical Clustering with Anderberg

• K-means Clustering
• The Sum of Squares Objective
• The Standard Algorithm (Lloyd s Algorithm)
• Non-determinism & Non-optimality
• Initialization
• Initialization /2
• Complexity of k-Means Clustering

• Accelerating k-Means Clustering
• k-Means++: Weighted Random Initialization
• Making k-means Faster
• Bounding the Distances Elkan and Hamerly
• Hamerly s k-means
• Example: k-Means Clustering with Hamerly s Algorithm
• Speedup with Hamerly, Elkan, and Exponion

• Limitations of k-Means Clustering
• Benefits and Drawbacks of k-Means
• Choosing the Optimum k for k-Means
• Limitations of k-Means

• Extensions of k-Means Clustering
• k-Means and Distances
• k-Means Minimizes Sum of Squares, not Euclidean Distance!
• k-Means Variations for Other Distances
• Spherical k-Means for Text Clustering
• Pre-processing and Post-processing

• Partitioning Around Medoids (k-Medoids)
• k-medoids Clustering
• Partitioning Around Medoids
• Algorithm: Partitioning Around Medoids
• Algorithm: Partitioning Around Medoids /2
• Change in TD
• Finding the Best Swap Faster
• k-Medoids, k-Means style
• Example for the Inferiority of k-Means Style k-Medoids

• Gaussian Mixture Modeling Introduction
• Expectation-Maximization in Clustering
• Fitting Multiple Gaussian Distributions
• Gaussian Mixture Modeling as E-M-Optimization
• Algorithm: EM Clustering
• Numerical Issues in GMM

• Gaussian Mixture Modeling Demo

• BIRCH and BETULA
• BIRCH Clustering
• BIRCH Clustering Features
• BIRCH Distances
• BIRCH CF-Tree
• BETULA Cluster Features
• BETULA Distance Computations
• Accelerating k-Means with BIRCH and BETULA
• Accelerating GMM with BETULA

• Motivation Density-Based Clustering
• Density-based Clustering: Core Idea

• Density-Based Clustering Fundamentals
• Density-based Clustering: Foundations
• Density-based Clustering: Foundations /2
• Density-based Clustering: Foundations /3
• Density-reachability and Density-connectivity
• Density-reachability

• DBSCAN
• Clustering Approach
• Abstract DBSCAN Algorithm
• DBSCAN Algorithm
• DBSCAN Algorithm /2
• DBSCAN Algorithm /3
• DBSCAN in Context

• DBSCAN Parameterization
• Choosing DBSCAN parameters
• Choosing DBSCAN parameters /2
• Choosing DBSCAN parameters /3

• DBSCAN Extensions
• Generalized Density-based Clustering
• Grid-based Accelerated DBSCAN
• Anytime Density-Based Clustering (AnyDBC)
• Hierarchical DBSCAN* (HDBSCAN*)
• Improved DBSCAN Variations

• OPTICS Clustering
• Density-based Hierarchical Clustering
• Density-based Hierarchical Clustering /2
• OPTICS Clustering
• Cluster Order
• OPTICS Algorithm

• Cluster Extraction from OPTICS Plots
• OPTICS Reachability Plots
• Extracting Clusters from OPTICS Reachability Plots
• Role of the Parameters and minPts

• Understanding the OPTICS Cluster Order
• Properties of the OPTICS Cluster Order
• Cluster Order as Serialized Spanning Tree
• OPTICS as Density Spanning Trees
• Cluster Order to Dendrograms

• Spectral Clustering
• Minimum Cuts
• Graph Laplacian
• From Clustering Graphs to Clustering Data
• Spectral Clustering
• Spectral Clustering is Related to DBSCAN

• Biclustering and Subspace Clustering
• Biclustering & Subspace Clustering
• Bicluster Patterns
• Density-based Subspace Clustering
• Subspace Clustering with Apriori-Style Search
• Correlation Clustering
• 4C: Computing Correlation Connected Clusters
• Hough Transform
• CASH: Robust Clustering in Arbitrarily Oriented Subspaces

• Further Clustering Approaches
• CURE Clustering
• ROCK Clustering
• CHAMELEON
• Affinity Propagation Clustering
• Other Density-based Clustering Algorithms
• Further Clustering Approaches