% cat /boot/System.map-5.10.0-6-amd64 
ffffffffffffffff B The real System.map is in the linux-image-<version>-dbg package

--cursor-file=FILE      Show entries after cursor in FILE and update FILE
--facility=FACILITY...  Show entries with the specified facilities
--image=IMAGE           Operate on files in filesystem image
--namespace=NAMESPACE   Show journal data from specified namespace
--relinquish-var        Stop logging to disk, log to temporary file system
--smart-relinquish-var  Similar, but NOP if log directory is on root mount

clean UNIT...                       Clean runtime, cache, state, logs or configuration of unit
freeze PATTERN...                   Freeze execution of unit processes
thaw PATTERN...                     Resume execution of a frozen unit
log-level [LEVEL]                   Get/set logging threshold for manager
log-target [TARGET]                 Get/set logging target for manager
service-watchdogs [BOOL]            Get/set service watchdog state
--with-dependencies                 Show unit dependencies with 'status', 'cat', 'list-units', and 'list-unit-files'
 -T --show-transaction              When enqueuing a unit job, show full transaction
 --what=RESOURCES                   Which types of resources to remove
--boot-loader-menu=TIME             Boot into boot loader menu on next boot
--boot-loader-entry=NAME            Boot into a specific boot loader entry on next boot
--timestamp=FORMAT                  Change format of printed timestamps

• systemd-bless-boot.8
• systemd-boot-check-no-failures.8
• systemd-boot-system-token.service.8
• systemd-journald-varlink@.socket.8
• systemd.net-naming-scheme.7
• systemd-network-generator.service.8
• systemd-pstore.service.8
• systemd-udev-settle.service.8
• systemd-user-runtime-dir.5
• systemd-xdg-autostart-generator.8

• ProtectProc
• ProcSubset
• ProtectClock
• ProtectHostname
• ProtectKernelLogs
• RestrictSUIDSGID

systemd now defaults to the unified cgroup hierarchy (i.e. cgroupv2).
This change reflects the fact that cgroups2 support has matured
substantially in both systemd and in the kernel.
All major container tools nowadays should support cgroupv2.
If you run into problems with cgroupv2, you can switch back to the previous,
hybrid setup by adding systemd.unified_cgroup_hierarchy=false to the
kernel command line.
You can read more about the benefits of cgroupv2 at
https://www.kernel.org/doc/html/latest/admin-guide/cgroup-v2.html

• prometheus-elasticsearch-exporter (v1.1.0)
• prometheus-exporter-exporter (v0.4.0-1)
• prometheus-hacluster-exporter (v1.2.1-1)
• prometheus-homeplug-exporter (v0.3.0-2)
• prometheus-ipmi-exporter (v1.2.0)
• prometheus-libvirt-exporter (v0.2.0-1)
• prometheus-mqtt-exporter (v0.1.4-2)
• prometheus-nginx-vts-exporter (v0.10.3)
• prometheus-postfix-exporter (v0.2.0-3)
• prometheus-redis-exporter (v1.16.0-1)
• prometheus-smokeping-prober (v0.4.1-2)
• prometheus-tplink-plug-exporter (v0.2.0)

% apt-cache policy docker-ce
  docker-ce:
    Installed: 5:20.10.6~3-0~debian-buster
    Candidate: 5:20.10.6~3-0~debian-buster
    Version table:
   *** 5:20.10.6~3-0~debian-buster 100
          100 /var/lib/dpkg/status
       5:20.10.6~3-0~debian-bullseye 500
          500 https://download.docker.com/linux/debian bullseye/stable amd64 Packages

% export DEBUGINFOD_PROGRESS=1    # for optional download progress reporting
% export DEBUGINFOD_URLS="https://debuginfod.debian.net"

au CmdLineEnter : let s:incs = &incsearch   set noincsearch
au CmdLineLeave : let &incsearch = s:incs

• --atimes: preserve access (use) times
• --copy-as=USER: specify user (and optionally group) for the copy
• --crtimes/-N: for preserving the file s create time
• --max-alloc=SIZE: change a limit relating to memory allocation
• --mkpath:create the destination s path component
• --open-noatime: avoid changing the atime on opened files
• --stop-after=MINS: stop rsync after MINS minutes have elapsed
• --write-devices: write to devices as files (implies inplace)

• It now defers creation of ~/.ssh until there s something to write (e.g. the known_hosts file), so the good old admin trick to run ssh localhost and cancel immediately to create ~/.ssh with proper permissions no longer works
• v8.2 brought FIDO/U2F + FIDO2 resident keys Support
• The new include sshd_config keyword allows including additional configuration files via glob(3) patterns
• ssh now allows %n to be expanded in ProxyCommand strings.
• The scp and sftp command-lines now accept -J option as an alias to ProxyJump.
• The scp and sftp command-lines allow the -A flag to explicitly enable agent forwarding.

• apt: was updated from version 1.8 to 2.2, apt maintainer Julian Andres Klode wrote something about changes in apt 2.0 and changes in apt 2.2
• e2fsprogs: supports the new ext4 fast_commit feature
• exim: you might be interested being aware of the allow_insecure_tainted_data option, see #987133 + #987924 (Exim 4.94 includes a new tainting-feature)
• glusterfs: was updated from 5.5 to v9.1, there s no direct upgrade path (only v5 towards v8 would be supported), so online migration does not work
• mariadb: no longer runs as /usr/sbin/mysqld, but as /usr/sbin/mariadbd
• python: Python 2.7 (and a few core packages) is still present, but there won t be any security updates for Python 2, see wiki.debian.org/Python/2Removal
• qemu-system-x86: /proc/stat on host does not show guest cpu usage
• rsnapshot is gone (see #986709)

• Move your own channels to to Libera or OFTC
• If you have previously been known to be generally around on Freenode, connect to Libera (the continuity network set up by the Freenode staff), and register your usual nick(s) there.
• Disconnect from freenode so that you don't count as one of their users.

Message-ID: <24741.12566.639691.461134@mariner.uk.xensource.com>
From: Ian Jackson <iwj@xenproject.org>
To: xen-devel@lists.xenproject.org
CC: community.manager@xenproject.org
Subject: IRC networks
Date: Wed, 19 May 2021 16:39:02 +0100
Summary:
We have for many years used the Freenode IRC network for real-time
chat about Xen.  Unfortunately, Freenode is undergoing a crisis.
There is a dispute between, on the one hand, Andrew Lee, and on the
other hand, all (or almost all) Freenode volunteer staff.  We must
make a decision.
I have read all the publicly available materials and asked around with
my contacts.  My conclusions:
 * We do not want to continue to use irc.freenode.*.
 * We might want to use libera.chat, but:
 * Our best option is probably to move to OFTC https://www.oftc.net/
Discussion:
Firstly, my starting point.
I have been on IRC since at least 1993.  Currently my main public
networks are OFTC and Freenode.
I do not have any personal involvement with public IRC networks.  Of
the principals in the current Freenode dispute, I have only heard of
one, who is a person I have experience of in a Debian context but have
not worked closely with.
George asked me informally to use my knowledge and contacts to shed
light on the situation.  I decided that, having done my research, I
would report more formally and publicly here rather than just
informally to George.
Historical background:
 * Freenode has had drama before.  In about 2001 OFTC split off from
   Freenode after an argument over governance.  IIRC there was drama
   again in 2006.  Significant proportion of the Free Software world,
   including Debian, now use OFTC.  Debian switched in 2006.
Facts that I'm (now) pretty sure of:
 * Freenode's actual servers run on donated services; that is,
   the hardware is owned by those donating the services, and the
   systems are managed by Freenode volunteers, known as "staff".
 * The freenode domain names are currently registered to a limited
   liability company owned by Andrew Lee (rasengan).
 * At least 10 Freenode staff have quit in protest, writing similar
   resignation letters protesting about Andrew Lee's actions [1].  It
   does not appear that any Andrew Lee has the public support of any
   Freenode staff.
 * Andrew Lee claims that he "owns" Freenode.[2]
 * A large number of channel owners for particular Free Software
   projects who previously used Freenode have said they will switch
   away from Freenode.
Discussion and findings on Freenode:
There is, as might be expected, some murk about who said what to whom
when, what promises were made and/or broken, and so on.  The matter
was also complicated by the leaking earlier this week of draft(s) of
(at least one of) the Freenode staffers' resignation letters.
Andrew Lee has put forward a position statement [2].  A large part of
the thrust of that statement is allegations that the current head of
Freenode staff, tomaw, "forced out" the previous head, christel.  This
allegation is strongly disputed by by all those current (resigning)
Freenode staff I have seen comment.  In any case it does not seem to
be particularly germane; in none of my reading did tomaw seem to be
playing any kind of leading role.  tomaw is not mentioned in the
resignation letters.
Some of the links led to me to logs of discussions on #freenode.  I
read some of these in particular[3].  MB I haven't been able to verify
that these logs have not been tampered with.  Having said that and
taking the logs at face value, I found the rasengan writing there to
be disingenuous and obtuse.
Andrew Lee has been heavily involved in Bitcoin.  Bitcoin is a hive of
scum and villainy, a pyramid scheme, and an environmental disaster,
all rolled into one.  This does not make me think well of Lee.
Additionally, it seems that Andrew Lee has been involved in previous
governance drama involving a different IRC network, Snoonet.
I have come to the very firm conclusion that we should have nothing to
do with Andrew Lee, and avoid using services that he has some
effective control over.
Alternatives:
The departing Freenode staff are setting up a replacement,
"libera.chat".  This is operational but still suffering from teething
problems and of course has a significant load as it deals with an
influx of users on a new setup.
On the staff and trust question: As I say, I haven't heard of any of
the Freenode staff, with one exception.  Unfortunately the one
exception does not inspire confidence in me[4] - although NB that is
only one data point.
On the other hand, Debian has had many many years of drama-free
involvement with OFTC.  OFTC has a formal governance arrangement and
it is associated with Software in the Public Interest.  I notice that
the last few OFTC'[s annual officer elections have been run partly by
Steve McIntyre.  Steve is a friend of mine (and he is a former Debian
Project Leader) and I take his involvement as a good sign.
I recommend that we switch to using OFTC as soon as possible.
Ian.
References:
Starting point for the resigning Freenode staff's side [1]:
  https://gist.github.com/joepie91/df80d8d36cd9d1bde46ba018af497409
Andrew Lee's side [2]:
  https://gist.github.com/realrasengan/88549ec34ee32d01629354e4075d2d48
[3]
https://paste.sr.ht/~ircwright/7e751d2162e4eb27cba25f6f8893c1f38930f7c4
[4] I won't give the name since I don't want to be shitposting.

• Jerikan, a tool to build configuration files from a single source of truth and Jinja2 templates, along with its integration into the GitLab CI system;
• an Ansible playbook to deploy these configuration files on network devices; and
• the configuration data and the templates for our, now defunct, datacenters in San Francisco and South Korea, covering many vendors (Facebook Wedge 100, Dell S4048 and S6010, Juniper QFX 5110, Juniper QFX 10002, Cisco ASR 9001, Cisco Catalyst 2960, Opengear console servers, and Linux), and many functionalities (provisioning, BGP-to-the-host routing, edge routing, out-of-band network, DNS configuration, integration with NetBox and IRRs).

Jerikan The first component is Jerikan. As input, it takes a list of devices, configuration data, templates, and validation scripts. It generates a set of configuration files for each device. Ansible could cover this task, but it has the following limitations:

• it is slow;
• errors are difficult to debug;¹ and
• the hierarchy to look up a variable is rigid.

Jerikan inputs and outputs

If you want to follow the examples, you only need to have Docker and Docker Compose installed. Clone the repository and you are ready!

Source of truth We use YAML files, versioned with Git, as the single source of truth instead of using a database, like NetBox, or a mix of a database and text files. This provides many advantages:

• anyone can use their preferred text editor;
• the team prepares changes in branches;
• the team reviews changes using merge requests;
• the merge requests expose the changes to the generated configuration files;
• rollback to a previous state is easy; and
• it is fast.

The first file is devices.yaml. It contains the device list. The second file is classifier.yaml. It defines a scope for each device. A scope is a set of keys and values. It is used in templates and to look up data associated with a device.

$ ./run-jerikan scope to1-p1.sk1.blade-group.net
continent: apac
environment: prod
groups:
- tor
- tor-bgp
- tor-bgp-compute
host: to1-p1.sk1
location: sk1
member: '1'
model: dell-s4048
os: cumulus
pod: '1'
shorthost: to1-p1

The device name is matched against a list of regular expressions and the scope is extended by the result of each match. For to1-p1.sk1.blade-group.net, the following subset of classifier.yaml defines its scope:

matchers:
  - '^(([^.]*)\..*)\.blade-group\.net':
      environment: prod
      host: '\1'
      shorthost: '\2'
  - '\.(sk1)\.':
      location: '\1'
      continent: apac
  - '^to([12])-[as]?p(\d+)\.':
      member: '\1'
      pod: '\2'
  - '^to[12]-p\d+\.':
      groups:
        - tor
        - tor-bgp
        - tor-bgp-compute
  - '^to[12]-(p ap)\d+\.sk1\.':
      os: cumulus
      model: dell-s4048

The third file is searchpaths.py. It describes which directories to search for a variable. A Python function provides a list of paths to look up in data/ for a given scope. Here is a simplified version:²

def searchpaths(scope):
    paths = [
        "host/ scope[location] / scope[shorthost] ",
        "location/ scope[location] ",
        "os/ scope[os] - scope[model] ",
        "os/ scope[os] ",
        'common'
    ]
    for idx in range(len(paths)):
        try:
            paths[idx] = paths[idx].format(scope=scope)
        except KeyError:
            paths[idx] = None
    return [path for path in paths if path]

With this definition, the data for to1-p1.sk1.blade-group.net is looked up in the following paths:

$ ./run-jerikan scope to1-p1.sk1.blade-group.net
[ ]
Search paths:
  host/sk1/to1-p1
  location/sk1
  os/cumulus-dell-s4048
  os/cumulus
  common

Variables are scoped using a namespace that should be specified when doing a lookup. We use the following ones:

• system for accounts, DNS, syslog servers,
• topology for ports, interfaces, IP addresses, subnets,
• bgp for BGP configuration
• build for templates and validation scripts
• apps for application variables

When looking up for a variable in a given namespace, Jerikan looks for a YAML file named after the namespace in each directory in the search paths. For example, if we look up a variable for to1-p1.sk1.blade-group.net in the bgp namespace, the following YAML files are processed: host/sk1/to1-p1/bgp.yaml, location/sk1/bgp.yaml, os/cumulus-dell-s4048/bgp.yaml, os/cumulus/bgp.yaml, and common/bgp.yaml. The search stops at the first match. The schema.yaml file allows us to override this behavior by asking to merge dictionaries and arrays across all matching files. Here is an excerpt of this file for the topology namespace:

system:
  users:
    merge: hash
  sampling:
    merge: hash
  ansible-vars:
    merge: hash
  netbox:
    merge: hash

The last feature of the source of truth is the ability to use Jinja2 templates for keys and values by prefixing them with ~ :

# In data/os/junos/system.yaml
netbox:
  manufacturer: Juniper
  model: "~  model upper  "
# In data/groups/tor-bgp-compute/system.yaml
netbox:
  role: net_tor_gpu_switch

Looking up for netbox in the system namespace for to1-p2.ussfo03.blade-group.net yields the following result:

$ ./run-jerikan scope to1-p2.ussfo03.blade-group.net
continent: us
environment: prod
groups:
- tor
- tor-bgp
- tor-bgp-compute
host: to1-p2.ussfo03
location: ussfo03
member: '1'
model: qfx5110-48s
os: junos
pod: '2'
shorthost: to1-p2
[ ]
Search paths:
[ ]
  groups/tor-bgp-compute
[ ]
  os/junos
  common
$ ./run-jerikan lookup to1-p2.ussfo03.blade-group.net system netbox
manufacturer: Juniper
model: QFX5110-48S
role: net_tor_gpu_switch

This also works for structured data:

# In groups/adm-gateway/topology.yaml
interface-rescue:
  address: "~  lookup('topology', 'addresses').rescue  "
  up:
    - "~ip route add default via   lookup('topology', 'addresses').rescue ipaddr('first_usable')   table rescue"
    - "~ip rule add from   lookup('topology', 'addresses').rescue ipaddr('address')   table rescue priority 10"
# In groups/adm-gateway-sk1/topology.yaml
interfaces:
  ens1f0: "~  lookup('topology', 'interface-rescue')  "

This yields the following result:

$ ./run-jerikan lookup gateway1.sk1.blade-group.net topology interfaces
[ ]
ens1f0:
  address: 121.78.242.10/29
  up:
  - ip route add default via 121.78.242.9 table rescue
  - ip rule add from 121.78.242.10 table rescue priority 10

When putting data in the source of truth, we use the following rules:

1. Don t repeat yourself.
2. Put the data in the most specific place without breaking the first rule.
3. Use templates with parsimony, mostly to help with the previous rules.
4. Restrict the data model to what is needed for your use case.

The first rule is important. For example, when specifying IP addresses for a point-to-point link, only specify one side and deduce the other value in the templates. The last rule means you do not need to mimic a BGP YANG model to specify BGP peers and policies:

peers:
  transit:
    cogent:
      asn: 174
      remote:
        - 38.140.30.233
        - 2001:550:2:B::1F9:1
      specific-import:
        - name: ATT-US
          as-path: ".*7018$"
          lp-delta: 50
  ix-sfmix:
    rs-sfmix:
      monitored: true
      asn: 63055
      remote:
        - 206.197.187.253
        - 206.197.187.254
        - 2001:504:30::ba06:3055:1
        - 2001:504:30::ba06:3055:2
    blizzard:
      asn: 57976
      remote:
        - 206.197.187.42
        - 2001:504:30::ba05:7976:1
      irr: AS-BLIZZARD

Templates The list of templates to compile for each device is stored in the source of truth, under the build namespace:

$ ./run-jerikan lookup edge1.ussfo03.blade-group.net build templates
data.yaml: data.j2
config.txt: junos/main.j2
config-base.txt: junos/base.j2
config-irr.txt: junos/irr.j2
$ ./run-jerikan lookup to1-p1.ussfo03.blade-group.net build templates
data.yaml: data.j2
config.txt: cumulus/main.j2
frr.conf: cumulus/frr.j2
interfaces.conf: cumulus/interfaces.j2
ports.conf: cumulus/ports.j2
dhcpd.conf: cumulus/dhcp.j2
default-isc-dhcp: cumulus/default-isc-dhcp.j2
authorized_keys: cumulus/authorized-keys.j2
motd: linux/motd.j2
acl.rules: cumulus/acl.j2
rsyslog.conf: cumulus/rsyslog.conf.j2

Templates are using Jinja2. This is the same engine used in Ansible. Jerikan ships some custom filters but also reuse some of the useful filters from Ansible, notably ipaddr. Here is an excerpt of templates/junos/base.j2 to configure DNS and NTP servers on Juniper devices:

system  
  ntp  
 % for ntp in lookup("system", "ntp") % 
    server   ntp  ;
 % endfor % 
   
  name-server  
 % for dns in lookup("system", "dns") % 
      dns  ;
 % endfor % 
   
 

The equivalent template for Cisco IOS-XR is:

 % for dns in lookup('system', 'dns') % 
domain vrf VRF-MANAGEMENT name-server   dns  
 % endfor % 
!
 % for syslog in lookup('system', 'syslog') % 
logging   syslog   vrf VRF-MANAGEMENT
 % endfor % 
!

There are three helper functions provided:

• devices() returns the list of devices matching a set of conditions on the scope. For example, devices("location==ussfo03", "groups==tor-bgp") returns the list of devices in San Francisco in the tor-bgp group. You can also omit the operator if you want the specified value to be equal to the one in the local scope. For example, devices("location") returns devices in the current location.
• lookup() does a key lookup. It takes the namespace, the key, and optionally, a device name. If not provided, the current device is assumed.
• scope() returns the scope of the provided device.

Here is how you would define iBGP sessions between edge devices in the same location:

 % for neighbor in devices("location", "groups==edge") if neighbor != device % 
   % for address in lookup("topology", "addresses", neighbor).loopback tolist % 
protocols bgp group IPV  address ipv  -EDGES-IBGP  
  neighbor   address    
    description "IPv  address ipv  : iBGP to   neighbor  ";
   
 
   % endfor % 
 % endfor % 

We also have a global key-value store to save information to be reused in another template or device. This is quite useful to automatically build DNS records. First, capture the IP address inserted into a template with store() as a filter:

interface Loopback0
 description 'Loopback:'
  % for address in lookup('topology', 'addresses').loopback tolist % 
 ipv  address ipv   address   address store('addresses', 'Loopback0') ipaddr('cidr')  
  % endfor % 
!

Then, reuse it later to build DNS records by iterating over store():⁴

 % for device, ip, interface in store('addresses') % 
   % set interface = interface replace('/', '-') replace('.', '-') replace(':', '-') % 
   % set name = ' . '.format(interface lower, device) % 
  name  . IN   'A' if ip ipv4 else 'AAAA'     ip ipaddr('address')  
 % endfor % 

Templates are compiled locally with ./run-jerikan build. The --limit argument restricts the devices to generate configuration files for. Build is not done in parallel because a template may depend on the data collected by another template. Currently, it takes 1 minute to compile around 3000 files spanning over 800 devices.

Output of Jerikan after building configuration files for six devices

When an error occurs, a detailed traceback is displayed, including the template name, the line number and the value of all visible variables. This is a major time-saver compared to Ansible!

templates/opengear/config.j2:15: in top-level template code
    config.interfaces.  interface  .netmask   adddress   ipaddr("netmask")  
        continent  = 'us'
        device     = 'con1-ag2.ussfo03.blade-group.net'
        environment = 'prod'
        host       = 'con1-ag2.ussfo03'
        infos      =  'address': '172.30.24.19/21' 
        interface  = 'wan'
        location   = 'ussfo03'
        loop       = <LoopContext 1/2>
        member     = '2'
        model      = 'cm7132-2-dac'
        os         = 'opengear'
        shorthost  = 'con1-ag2'
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
value = JerkianUndefined, query = 'netmask', version = False, alias = 'ipaddr'
[ ]
        # Check if value is a list and parse each element
        if isinstance(value, (list, tuple, types.GeneratorType)):
            _ret = [ipaddr(element, str(query), version) for element in value]
            return [item for item in _ret if item]
>       elif not value or value is True:
E       jinja2.exceptions.UndefinedError: 'dict object' has no attribute 'adddress'

We don t have general-purpose rules when writing templates. Like for the source of truth, there is no need to create generic templates able to produce any BGP configuration. There is a balance to be found between readability and avoiding duplication. Templates can become scary and complex: sometimes, it s better to write a filter or a function in jerikan/jinja.py. Mastering Jinja2 is a good investment. Take time to browse through our templates as some of them show interesting features.

Checks Optionally, each configuration file can be validated by a script in the checks/ directory. Jerikan looks up the key checks in the build namespace to know which checks to run:

$ ./run-jerikan lookup edge1.ussfo03.blade-group.net build checks
- description: Juniper configuration file syntax check
  script: checks/junoser
  cache:
    input: config.txt
    output: config-set.txt
- description: check YAML data
  script: checks/data.yaml
  cache: data.yaml

In the above example, checks/junoser is executed if there is a change to the generated config.txt file. It also outputs a transformed version of the configuration file which is easier to understand when using diff. Junoser checks a Junos configuration file using Juniper s XML schema definition for Netconf.⁵ On error, Jerikan displays:

jerikan/build.py:127: RuntimeError
-------------- Captured syntax check with Junoser call --------------
P: checks/junoser edge2.ussfo03.blade-group.net
C: /app/jerikan
O:
E: Invalid syntax:  set system syslog archive size 10m files 10 word-readable
S: 1

Integration into GitLab CI The next step is to compile the templates using a CI. As we are using GitLab, Jerikan ships with a .gitlab-ci.yml file. When we need to make a change, we create a dedicated branch and a merge request. GitLab compiles the templates using the same environment we use on our laptops and store them as an artifact.

Merge request to add a new port in USSFO03. The templates were compiled successfully but approval from another team member is still required to merge.

Before approving the merge request, another team member looks at the changes in data and templates but also the differences for the generated configuration files:

The change configures a port on the Juniper device, adds records to DNS, and updates NetBox with the new IP addresses (not shown).

Ansible After Jerikan has built the configuration files, Ansible takes over. It is also packaged as a Docker image to avoid the trouble to maintain the right Python virtual environment and ensure everyone is using the same versions.

Inventory Jerikan has generated an inventory file. It contains all the managed devices, the variables defined for each of them and the groups converted to Ansible groups:

ob1-n1.sk1.blade-group.net ansible_host=172.29.15.12 ansible_user=blade ansible_connection=network_cli ansible_network_os=ios
ob2-n1.sk1.blade-group.net ansible_host=172.29.15.13 ansible_user=blade ansible_connection=network_cli ansible_network_os=ios
ob1-n1.ussfo03.blade-group.net ansible_host=172.29.15.12 ansible_user=blade ansible_connection=network_cli ansible_network_os=ios
none ansible_connection=local
[oob]
ob1-n1.sk1.blade-group.net
ob2-n1.sk1.blade-group.net
ob1-n1.ussfo03.blade-group.net
[os-ios]
ob1-n1.sk1.blade-group.net
ob2-n1.sk1.blade-group.net
ob1-n1.ussfo03.blade-group.net
[model-c2960s]
ob1-n1.sk1.blade-group.net
ob2-n1.sk1.blade-group.net
ob1-n1.ussfo03.blade-group.net
[location-sk1]
ob1-n1.sk1.blade-group.net
ob2-n1.sk1.blade-group.net
[location-ussfo03]
ob1-n1.ussfo03.blade-group.net
[in-sync]
ob1-n1.sk1.blade-group.net
ob2-n1.sk1.blade-group.net
ob1-n1.ussfo03.blade-group.net
none

in-sync is a special group for devices which configuration should match the golden configuration. Daily and unattended, Ansible should be able to push configurations to this group. The mid-term goal is to cover all devices. none is a special device for tasks not related to a specific host. This includes synchronizing NetBox, IRR objects, and the DNS, updating the RPKI, and building the geofeed files.

Playbook We use a single playbook for all devices. It is described in the ansible/playbooks/site.yaml file. Here is a shortened version:

- hosts: adm-gateway:!done
  strategy: mitogen_linear
  roles:
    - blade.linux
    - blade.adm-gateway
    - done
- hosts: os-linux:!done
  strategy: mitogen_linear
  roles:
    - blade.linux
    - done
- hosts: os-junos:!done
  gather_facts: false
  roles:
    - blade.junos
    - done
- hosts: os-opengear:!done
  gather_facts: false
  roles:
    - blade.opengear
    - done
- hosts: none:!done
  gather_facts: false
  roles:
    - blade.none
    - done

A host executes only one of the play. For example, a Junos device executes the blade.junos role. Once a play has been executed, the device is added to the done group and the other plays are skipped. The playbook can be executed with the configuration files generated by the GitLab CI using the ./run-ansible-gitlab command. This is a wrapper around Docker and the ansible-playbook command and it accepts the same arguments. To deploy the configuration on the edge devices for the SK1 datacenter in check mode, we use:

$ ./run-ansible-gitlab playbooks/site.yaml --limit='edge:&location-sk1' --diff --check
[ ]
PLAY RECAP *************************************************************
edge1.sk1.blade-group.net  : ok=6    changed=0    unreachable=0    failed=0    skipped=3    rescued=0    ignored=0
edge2.sk1.blade-group.net  : ok=5    changed=0    unreachable=0    failed=0    skipped=1    rescued=0    ignored=0

We have some rules when writing roles:

• --check must detect if a change is needed;
• --diff must provide a visualization of the planned changes;
• --check and --diff must not display anything if there is nothing to change;
• writing a custom module tailored to our needs is a valid solution;
• the whole device configuration is managed;⁶
• secrets must be stored in Vault;
• templates should be avoided as we have Jerikan for that; and
• avoid duplication and reuse tasks.⁷

We avoid using collections from Ansible Galaxy, the exception being collections to connect and interact with vendor devices, like cisco.iosxr collection. The quality of Ansible Galaxy collections is quite random and it is an additional maintenance burden. It seems better to write roles tailored to our needs. The collections we use are in ci/ansible/ansible-galaxy.yaml. We use Mitogen to get a 10 speedup on Ansible executions on Linux hosts. We also have a few playbooks for operational purpose: upgrading the OS version, isolate an edge router, etc. We were also planning on how to add operational checks in roles: are all the BGP sessions up? They could have been used to validate a deployment and rollback if there is an issue. Currently, our playbooks are run from our laptops. To keep tabs, we are using ARA. A weekly dry-run on devices in the in-sync group also provides a dashboard on which devices we need to run Ansible on.

Configuration data and templates Jerikan ships with pre-populated data and templates matching the configuration of our USSFO03 and SK1 datacenters. They do not exist anymore but, we promise, all this was used in production back in the days!

The latest iteration of our network infrastructure for SK1, USSFO03, and future data centers. The production network is using BGPttH using a spine-leaf fabric. The out-of-band network is using a simple L2 design, using the spanning tree protocol, as well as a set of console servers.

Notably, you can find the configuration for:

our edge routers

Some are running on Junos, like edge2.ussfo03, the others on IOS-XR, like edge1.sk1. The implemented functionalities are similar in both cases and we could swap one for the other. It includes the BGP configuration for transits, peerings, and IX as well as the associated BGP policies. PeeringDB is queried to get the maximum number of prefixes to accept for peerings. bgpq3 and a containerized IRRd help to filter received routes. A firewall is added to protect the routing engine. Both IPv4 and IPv6 are configured.

our BGP-based fabric

BGP is used inside the datacenter⁸ and is extended on bare-metal hosts. The configuration is automatically derived from the device location and the port number.⁹ Top-of-the-rack devices are using passive BGP sessions for ports towards servers. They are also serving a provisioning network to let them boot using DHCP and PXE. They also act as a DHCP server. The design is multivendor. Some devices are running Cumulus Linux, like to1-p1.ussfo03, while some others are running Junos, like to1-p2.ussfo03.

our out-of-band fabric

We are using Cisco Catalyst 2960 switches to build an L2 out-of-band network. To provide redundancy and saving a few bucks on wiring, we build small loops and run the spanning-tree protocol. See ob1-p1.ussfo03. It is redundantly connected to our gateway servers. We also use OpenGear devices for console access. See con1-n1.ussfo03

our administrative gateways

These Linux servers have multiple purposes: SSH jump boxes, rescue connection, direct access to the out-of-band network, zero-touch provisioning of network devices,¹⁰ Internet access for management flows, centralization of the console servers using Conserver, and API for autoconfiguration of BGP sessions for bare-metal servers. They are the first servers installed in a new datacenter and are used to provision everything else. Check both the generated files and the associated Ansible tasks.

---

1. Ansible does not even provide a line number when there is an error in a template. You may need to find the problem by bisecting.

   $ ansible --version
   ansible 2.10.8
   [ ]
   $ cat test.j2
   Hello   name  !
   $ ansible all -i localhost, \
   >  --connection=local \
   >  -m template \
   >  -a "src=test.j2 dest=test.txt"
   localhost   FAILED! =>  
       "changed": false,
       "msg": "AnsibleUndefinedVariable: 'name' is undefined"
    
   

2. You may recognize the same concepts as in Hiera, the hierarchical key-value store from Puppet. At first, we were using Jerakia, a similar independent store exposing an HTTP REST interface. However, the lookup overhead is too large for our use. Jerikan implements the same functionality within a Python function.
3. The list of available filters is mangled inside jerikan/jinja.py. This is a remain of the fact we do not maintain Jerikan as a standalone software.
4. This is a bit confusing: we have a store() filter and a store() function. With Jinja2, filters and functions live in two different namespaces.
5. We are using a fork with some modifications to be able to validate our configurations and exposing an HTTP service to reduce the time spent on each configuration check.
6. There is a trend in network automation to automate a configuration subset, for example by having a playbook to create a new BGP session. We believe this is wrong: with time, your configuration will get out-of-sync with its expected state, notably hand-made changes will be left undetected.
7. See ansible/roles/blade.linux/tasks/firewall.yaml and ansible/roles/blade.linux/tasks/interfaces.yaml. They are meant to be called when needed, using import_role.
8. We also have some datacenters using BGP EVPN VXLAN at medium-scale using Juniper devices. As they are still in production today, we didn t include this feature but we may publish it in the future.
9. In retrospect, this may not be a good idea unless you are pretty sure everything is uniform (number of switches for each layer, number of ports). This was not our case. We now think it is a better idea to assign a prefix to each device and write it in the source of truth.
10. Non-linux based devices are upgraded and configured unattended. Cumulus Linux devices are automatically upgraded on install but the final configuration has to be pushed using Ansible: we didn t want to duplicate the configuration process using another tool.

Finding Project Scrolling through the list of GSoC 2021 organizations, I was checking out projects of organizations I am familiar with. Debian is one of the huge Open Source communities that has always inspired developers around the world to contribute to Open Source. So as I checked through Debian projects, I got excited to find the Debian Continuous Integration improvements project (referred to as debci in this post) related to web development and more concerned with backend work which is something I am very much interested in. I joined the community, and as directed by the application tasks of the project I set up the debci on my machine and started with an issue labeled as a newcomer. Soon I was able to submit my first Merge Request and it was reviewed by Antonio Terceiro, the mentor of my GSoC Project. With his further guidance, I was able to turn MR into an acceptable patch, and voila it got merged! That really did boost my morale to contribute further to the project.

Student Application Period At the suggestion of the mentor, during the Student Application Period, I worked on more open issues which were helping me understand the codebase better and in turn the deliverables of the project for my proposal. For my every doubt, I first tried to tackle it myself, and if still not able to find a solution I turn to mentors who did their best to answer my queries and this is how I completed my proposal and got it reviewed by Antonio before finally submitting it on 13th April. I still cannot express that feeling of satisfaction I achieved on submitting the proposal. I finally successfully applied for GSoC.

After Proposal Submission I did not stop my contribution after the Student Application period ended and kept on working on more issues which helped me stay in touch with the project and also because I was enjoying it a lot. I had already made up my mind to contribute more in Open Source as I learned and enjoyed plenty during this process.

Day of Results On 17th May, I got my acceptance mail at around 11:30 pm at night and I remember screaming and waking everybody up in my home to announce the news to them. It was truly the happiest moment for me.

Moment of Truth I would admit that I got involved with GSoC because of the reputation associated with it but things I learned during this pre GSoC period have made me realize the fun and learning opportunities associated with working opensource and I am here to stay for sure. I plan to write more blogs regarding my project and keep you guys updated about my work. Stay tuned!

Pandemic Situation in India. I don t know from where I should start. This is probably a good start. I actually would recommend Indiacable as they do attempt to share some things happening in India from day to day but still there is a lot thatt they just can t cover, nobody can cover. There were two reports which kind of shook me all inside. One which sadly came from the UK publication Independent, probably as no Indian publication would dare publish it. The other from Rural India. I have been privileged in many ways, including friends who have asked me if I need any financial help. But seeing reports like above, these people need more help, guidance and help than I. While I m never one to say give to Foundations. If some people do want to help people from Maharashtra, then moneylifefoundation could be a good place where they could donate. FWIW, they usually use the foundation to help savers and investors be safe and help in getting money when taken by companies with dubious intentions. That is their drive. Two articles show their bent. The first one is about the Algo scam which I have written previously about the same in this blog. Interestingly, when I talk about this scam, all Modi supporters are silent. The other one does give some idea as to why the Govt. is indifferent. That is going to a heavy cross for all relatives to bear. There has been a lot that has been happening. Now instead of being limited to cities, Covid has now gone hinterland in a big way. One could ask also Praveen as he probably knows what would be good for Kerala and surrounding areas. The biggest change, however, has been that India is now battling not just the pandemic but also Mucormycosis also known as black fungus and its deadlier cousin the white fungus. Mucormycosis came largely due to an ill-advise given that applying cow dung gives protection to Corona. And many applied it due to faith. And people who know science do know that in fact it has that bacteria. Sadly, those of us who are and were more interested in law, computer science etc. has now also have to keep on top of what is happening in the medical field. It isn t that I hate it, but it has a lot of costs. From what I could gather on various social media and elsewhere, a single injection of anti-fungal for the above costs INR 3k/- and that needs to be 5 times in a day and that course has to be for three weeks. So even the relatively wealthy people can and will become poor in no time. No wonder thousands of those went to UK, US, Dubai or wherever they could find safe-harbor from the pandemic with no plans of arriving back soon. There was also the whole bit about FBS or Fetal Bovin Serum. India ordered millions of blood serum products from abroad and continues to. This was quickly shut down as news on Social Media. Apparently, it is only the Indian cow which is worthy of reverence. All other cows and their children are fair game according to those in power. Of course, that discussion was quickly shut down as was the discussion about IGP (Indian Genome Project). People over the years had asked me why India never participated for the HGP (Human Gnome Project). I actually had no answer for that. Then in 2020, there was idea of IGP which was put up and then it was quickly shot down as the results could damage a political party s image. In fact, a note to people who want to join Indian civil services tells the reason exactly. While many countries in the world are hypocrites, including the U.S. none can take the place that India has made for itself in that field.

The Online experience The vaccination process has been made online and has led to severe heartburn and trouble for many including many memes. For e.g.

Daily work, get up, have a bath, see if you got a slot on the app, sleep.

People trying desperately to get a slot, taken from Hindi Movie Dilwale Dulhania Le Jaygenge.

Just to explain what is happening, one has to go to the website of cowin. Sharing a screenshot of the same.

Cowin app. sceeenshot

I have deliberately taken a screenshot of the cowin app. in U.P. which is one of the areas where the ruling party, BJP has. I haven t taken my state for the simple reason, even if a slot is open, it is of no use as there are no vaccines. As have been shared in India Cable as well as in many newspapers, it is the Central Govt. which holds the strings for the vaccines. Maharashtra did put up an international tender but to no effect. All vaccine manufacturers want only Central Govt. for purchases for multiple reasons. And GOI is saying it has no money even though recently it got loans as well as a dividend from RBI to the tune of 99k crore. For what all that money is, we have no clue. Coming back though, to the issue at hand. the cowin app. is made an open api. While normally, people like us should and are happy when an API is open, it has made those who understand how to use git, compile, etc. better than others. A copy of the public repo. of how you can do the same can be found on Github. Now, obviously, for people like me and many others it has ethical issues.

Kiran s Interview in Times of India (TOI) There isn t much to say apart from I haven t used it. I just didn t want to. It just is unethical. Hopefully, in the coming days GOI does something better. That is the only thing we are surviving on, hope.

The Toolkit saga A few days before, GOI shared a toolkit apparently made by Congress to defame the party in power. That toolkit was shared before the press and Altnews did the investigation and promptly shredded the claims. Congress promptly made an FIR in Chhattisgarh where it is in power. The gentleman who made the claims Mr. Sambit Patra refused to appear against the police without evidence citing personal reasons and asking 1 week to appear before them. Apart from Altnews which did a great job, sadly many people didn t even know that there is something called WYSIWYG. I had to explain that so many Industries, whether it is politics, creative industries, legal, ad industries, medical transcription, and imaging all use this, and all the participants use the same version of the software. The reason being that in most Industries, there is a huge loss and issue of legal liabilities if something untoward happens. For e.g. if medical transcription is done in India is wrong (although his or her work will be checked by a superior in the West), but for whatever reason is not, and a wrong diagnosis is put (due to wrong color or something) then a patient could die and the firm who does that work could face heavy penalties which could be the death of them. There is another myth that Congress has unlimited wealth or huge wealth. I asked if that was the case, why didn t they shift to Mac. Of course, none have answers on this one. There is another reason why they didn t want to appear. The Rona Wilson investigation by Arsenal Experts also has made them cautious. Previously, they had a free run. Nowadays, software forensic tools are available to one and all. For e.g. Debian itself has a good variety of tools for the same. I remember Vipin s sharing few years back. For those who want to start, just install the apps. and try figuring out. Expertise on using the tools takes years though, as you use the tool day in night. Update 25/05/2021 Apparently because Twitter made and showcased few tweets as Manipulated Media , those in Govt. are and were dead against it. So they conducted a raid against Twitter India headquarters, knowing fully well that there would be nobody except security. The moment I read this, my mind went to the whole Fruit of the poisonous tree legal doctrine. Sadly though, India doesn t recognize it and in fact, still believes in the pre-colonial era that evidence however collected is good. A good explanation of the same can be found here. There are some exceptions to the rule, but they are done so fine that more often than not, they can t be used in the court of law in India. Although a good RTI was shared by Mr. Saket Gokhale on the same issue, which does raise some interesting points

Twitter India Raid, Saket Gokhale RTI 1

Saket Gokhale RTI query , Twitter India Raid 2

FWIW, Saket has been successful in getting his prayers heard either as answers to RTI queries or then following it up in the various High Courts of India. Of course, those who are in the ruling party ridicule him but are unable to find faults in his application of logic. And quite a few times, I have learned from his applications as well as nuances or whatever is there in law, a judgment or a guideline which he invokes in his prayer. For e.g. the Lalitha Kumari Guidelines which the gentleman has shared in his prayer can be found here. Hence now, it would be upto the Delhi Police Cell to prove their case in response to RTI. He has also trapped them as he has shared they can t give excuses/exemptions which they have tried before. As I had shared earlier, High Courts in India have woken up, whether it is Delhi, Mumbai, Aurangabad, Madhya Pradesh, Uttar Pradesh, Odisha or Kerala. Just today i.e. on 25th May 2021, Justices Bela Trivedi and Justice Kalra had asked how come all the hospitals don t have NOC from the Fire De[partment. They also questioned the ASG (Assistant Solicitor General) as how BU (Building Use Certificate) has been granted as almost all the 400 hospitals are in residential area. To which the ASG replies, it is the same state in almost 4000 schools as well as 6000 odd factories in Ahemdabad alone, leave the rest of the district and state alone. And this is when last year strict instuctions were passed. They chose to do nothing sadly. I will share a link on this when bar and bench gives me The Hindu also shared the whole raid on twitter saga.

Conclusion In conclusion, I sincerely do not where we are headed. The only thing I know is that we cannot expect things to be better before year-end and maybe even after that. It all depends on the vaccines and their availability. After that ruralindia article, I had to see quite a few movies and whatnot just to get that out of my head. And this is apart from the 1600 odd teachers and workers who have died in the U.P. poll duty. Now, what a loss, not just to the family members of the victims, but a whole generation of school children who would not be able to get quality teaching and be deprived of education. What will be their future, God only knows. The only good Bollywood movie which I saw was Ramprasad ki Teravi . The movie was an accurate representation of most families in and around me. There was a movie called Sansar (1987) which showed the breakup of the joint family and into a nuclear family. This movie could very well have been a continuation of the same. Even Marathi movies which at one time were very progressive have gone back to the same boy, girl love story routine. Sameer, though released in late 2020, was able to see it only recently. Vakeel Saab was an ok copy of Pink . I loved Sameer as, unlike Salman Khan films, it showed pretty much an authentic human struggle of a person who goes to the Middle East without any qualifications and works as a laborer and the trials he goes through. Somehow, Malayalam movies have a knack for showing truth without much of budget. Most of the Indian web series didn t make an impact. I think many of them were just going through the motions, it seems as everybody is concerned with the well-being of their near and dear ones. There was also this (Trigger Warning: This story discusses organized campaigns glorifying and advocating sexual violence against Muslim women.) Hoping people somehow make it to the other side of the pandemic.

What is freenode and why should I care? freenode is the largest remaining IRC network. Before this incident, it had close to 80,000 users, which is small in terms of modern internet history -- even small social networks are larger by multiple orders of magnitude -- but is large in IRC history. The IRC network is also extensively used by the free software community, being the default IRC network on many programs, and used by hundreds if not thousands of free software projects. I have been using freenode since at least 2006. This matters if you care about IRC, the internet, open protocols, decentralisation, and, to a certain extent, federation as well. It also touches on who has the right on network resources: the people who "own" it (through money) or the people who make it work (through their labor). I am biased towards open protocols, the internet, federation, and worker power, and this might taint this analysis.

What happened? It's a long story, but basically:

1. back in 2017, the former head of staff sold the freenode.net domain (and its related company) to Andrew Lee, "American entrepreneur, software developer and writer", and, rather weirdly, supposedly "crown prince of Korea" although that part is kind of complex (see House of Yi, Yi Won, and Yi Seok). It should be noted the Korean Empire hasn't existed for over a century at this point (even though its flag, also weirdly, remains)
2. back then, this was only known to the public as this strange PIA and freenode joining forces gimmick. it was suspicious at first, but since the network kept running, no one paid much attention to it. opers of the network were similarly reassured that Lee would have no say in the management of the network
3. this all changed recently when Lee asserted ownership of the freenode.net domain and started meddling in the operations of the network, according to this summary. this part is disputed, but it is corroborated by almost a dozen former staff which collectively resigned from the network in protest, after legal threats, when it was obvious freenode was lost.
4. the departing freenode staff founded a new network, irc.libera.chat, based on the new ircd they were working on with OFTC, solanum
5. meanwhile, bot armies started attacking all IRC networks: both libera and freenode, but also OFTC and unrelated networks like a small one I help operate. those attacks have mostly stopped as of this writing (2021-05-24 17:30UTC)
6. on freenode, however, things are going for the worse: Lee has been accused of taking over a channel, in a grotesque abuse of power; then changing freenode policy to not only justify the abuse, but also remove rules against hateful speech, effectively allowing nazis on the network (update: the change was reverted, but not by Lee)

Update: even though the policy change was reverted, the actual conversations allowed on freenode have already degenerated into toxic garbage. There are also massive channel takeovers (presumably over 700), mostly on channels that were redirecting to libera, but also channels that were still live. Channels that were taken over include #fosdem, #wikipedia, #haskell... Instead of working on the network, the new "so-called freenode" staff is spending effort writing bots and patches to basically automate taking over channels. I run an IRC network and this bot is obviously not standard "services" stuff... This is just grotesque. At this point I agree with this HN comment:

We should stop implicitly legitimizing Andrew Lee's power grab by referring to his dominion as "Freenode". Freenode is a quarter-century-old community that has changed its name to libera.chat; the thing being referred to here as "Freenode" is something else that has illegitimately acquired control of Freenode's old servers and user database, causing enormous inconvenience to the real Freenode.

I don't agree with the suggested name there, let's instead call it "so called freenode" as suggested later in the thread.

What now? I recommend people and organisations move away from freenode as soon as possible. This is a major change: documentation needs to be fixed, and the migration needs to be coordinated. But I do not believe we can trust the new freenode "owners" to operate the network reliably and in good faith. It's also important to use the current momentum to build a critical mass elsewhere so that people don't end up on freenode again by default and find an even more toxic community than your typical run-of-the-mill free software project (which is already not a high bar to meet). Update: people are moving to libera in droves. It's now reaching 18,000 users, which is bigger than OFTC and getting close to the largest, traditionnal, IRC networks (EFnet, Undernet, IRCnet are in the 10-20k users range). so-called freenode is still larger, currently clocking 68,000 users, but that's a huge drop from the previous count which was 78,000 before the exodus began. We're even starting to see the effects of the migration on netsplit.de. Update 2: the isfreenodedeadyet.com site is updated more frequently than netsplit and shows tons more information. It shows 25k online users for libera and 61k for so-called freenode (down from ~78k), and the trend doesn't seem to be stopping for so-called freenode. There's also a list of 400+ channels that have moved out. Keep in mind that such migrations take effect over long periods of time.

Where do I move to? The first thing you should do is to figure out which tool to use for interactive user support. There are multiple alternatives, of course -- this is the internet after all -- but here is a short list of suggestions, in preferred priority order:

1. irc.libera.chat
2. irc.OFTC.net
3. Matrix.org, which bridges with OFTC and (hopefully soon) with libera as well, modern IRC alternative
4. XMPP/Jabber also still exists, if you're into that kind of stuff, but I don't think the "chat room" story is great there, at least not as good as Matrix

Basically, the decision tree is this:

• if you want to stay on IRC:
  • if you are already on many OFTC channels and few freenode channels: move to OFTC
  • if you are more inclined to support the previous freenode staff: move to libera
  • if you care about matrix users (in the short term): move to OFTC
• if you are ready to leave IRC:
  • if you want the latest and greatest: move to Matrix
  • if you like XML and already use XMPP: move to XMPP

Frankly, at this point, everyone should seriously consider moving to Matrix. The user story is great, the web is a first class user, it supports E2EE (although XMPP as well), and has a lot of momentum behind it. It even bridges with IRC well (which is not the case for XMPP) so if you're worried about problems like this happening again. (Indeed, I wouldn't be surprised if similar drama happens on OFTC or libera in the future. The history of IRC is full of such epic controversies, takeovers, sabotage, attacks, technical flamewars, and other silly things. I am not sure, but I suspect a federated model like Matrix might be more resilient to conflicts like this one.) Changing protocols might mean losing a bunch of users however: not everyone is ready to move to Matrix, for example. Graybeards like me have been using irssi for years, if not decades, and would take quite a bit of convincing to move elsewhere. I have mostly kept my channels on IRC, and moved either to OFTC or libera. In retrospect, I think I might have moved everything to OFTC if I would have thought about it more, because almost all of my channels are there. But I kind of expect a lot of the freenode community to move to libera, so I am keeping a socket open there anyways.

How do I move? The first thing you should do is to update documentation, websites, and source code to stop pointing at freenode altogether. This is what I did for feed2exec, for example. You need to let people know in the current channel as well, and possibly shutdown the channel on freenode. Since my channels are either small or empty, I took the radical approach of:

• redirecting the channel to ##unavailable which is historically the way we show channels have moved to another network
• make the channel invite-only (which effectively enforces the redirection)
• kicking everyone out of the channel
• kickban people who rejoin
• set the topic to announce the change

In IRC speak, the following commands should do all this:

/msg ChanServ set #anarcat mlock +if ##unavailable
/msg ChanServ clear #anarcat users moving to irc.libera.chat
/msg ChanServ set #anarcat restricted on
/topic #anarcat this channel has moved to irc.libera.chat

If the channel is not registered, the following might work

/mode #anarcat +if ##unavailable

Then you can leave freenode altogether:

/disconnect Freenode unacceptable hijack, policy changes and takeovers. so long and thanks for all the fish.

Keep in mind that some people have been unable to setup such redirections, because the new freenode staff have taken over their channel, in which case you're out of luck... Some people have expressed concern about their private data hosted at freenode as well. If you care about this, you can always talk to NickServ and DROP your nick. Be warned, however, that this assumes good faith of the network operators, which, at this point, is kind of futile. I would assume any data you have registered on there (typically: your NickServ password and email address) to be compromised and leaked. If your password is used elsewhere (tsk, tsk), change it everywhere. Update: there's also another procedure, similar to the above, but with a different approach. Keep in mind that so-called freenode staff are actively hijacking channels for the mere act of mentioning libera in the channel topic, so thread carefully there.

Last words This is a sad time for IRC in general, and freenode in particular. It's a real shame that the previous freenode staff have been kicked out, and it's especially horrible that if the new policies of the network are basically making the network open to nazis. I wish things would have gone out differently: now we have yet another fork in the IRC history. While it's not the first time freenode changes name (it was called OPN before), now the old freenode is still around and this will bring much confusion to the world, especially since the new freenode staff is still claiming to support FOSS. I understand there are many sides to this story, and some people were deeply hurt by all this. But for me, it's completely unacceptable to keep pushing your staff so hard that they basically all (except one?) resign in protest. For me, that's leadership failure at the utmost, and a complete disgrace. And of course, I can't in good conscience support or join a network that allows hate speech. Regardless of the fate of whatever we'll call what's left of freenode, maybe it's time for this old IRC thing to die already. It's still a sad day in internet history, but then again, maybe IRC will never die...

Changes in RcppArmadillo version 0.10.5.0 (2021-05-21)

• Upgraded to Armadillo release 10.5 (Antipodean Fortress)
  • added .clamp() member function
  • expanded the standalone clamp() function to handle complex values
  • more efficient use of OpenMP
  • vector, matrix and cube constructors now initialise elements to zero by default; use the fill::none specifier, eg. mat X(4,5,fill::none), to disable element initialisation
• Added codecov.yml to exclude Armadillo from coverage analysis

Changes in RcppArmadillo version 0.10.4.1.0 (2021-04-23)

• Upgraded to Armadillo release 10.4.1 (Pressure Cooker)
• GitHub-only release

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

• I want it to be much easier to set up an Otter server yourself, for you and your friends. Currently there are complete build instructions, but nowadays these things can be automated. Ideally it would be possible to set up an instance by button pushing and the payment of a modest sum to a cloud hosting provider.
• Right now, the only way to set up a game, manage the players in it, reset the gaming table, and so on, is using the otter command line tool. It's not so bad a tool, but having something pointy clicky would make the system much more accessible. But GUIs of this kind are a lot of work.

# Install docker + nftables from Debian
apt install docker.io nftables
# Add the Kubernetes repo and signing key
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg > /etc/apt/k8s.gpg
cat > /etc/apt/sources.list.d/kubernetes.list <<EOF
deb [signed-by=/etc/apt/k8s.gpg] https://apt.kubernetes.io/ kubernetes-xenial main
EOF
apt update
apt install kubelet kubeadm kubectl

kubeadm init --apiserver-advertise-address 192.168.53.147 --apiserver-cert-extra-sans udon.mynetwork

mkdir ~noodles/.kube
cp -i /etc/kubernetes/admin.conf ~noodles/.kube/config
chown -R noodles ~noodles/.kube/

noodles@udon:~$ kubectl get nodes
NAME   STATUS     ROLES                  AGE     VERSION
udon   NotReady   control-plane,master   4m31s   v1.21.1

mkdir -p /var/lib/weave
head -c 16 /dev/urandom   shasum -a 256   cut -d " " -f1 > /var/lib/weave/weave-passwd
kubectl create secret -n kube-system generic weave-passwd --from-file=/var/lib/weave/weave-passwd
kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version   base64   tr -d '\n')&password-secret=weave-passwd&env.IPALLOC_RANGE=192.168.0.0/24"

kubectl taint nodes --all node-role.kubernetes.io/master-

noodles@udon:~$ kubectl get nodes
NAME   STATUS   ROLES                  AGE   VERSION
udon   Ready    control-plane,master   15m   v1.21.1

noodles@udon:~$ kubectl get pods --all-namespaces
NAMESPACE     NAME                           READY   STATUS    RESTARTS   AGE
kube-system   coredns-558bd4d5db-4nvrg       1/1     Running   0          18m
kube-system   coredns-558bd4d5db-flrfq       1/1     Running   0          18m
kube-system   etcd-udon                      1/1     Running   0          18m
kube-system   kube-apiserver-udon            1/1     Running   0          18m
kube-system   kube-controller-manager-udon   1/1     Running   0          18m
kube-system   kube-proxy-6d8kg               1/1     Running   0          18m
kube-system   kube-scheduler-udon            1/1     Running   0          18m
kube-system   weave-net-mchmg                2/2     Running   1          3m26s

$ kubectl create deployment hello-node --image=k8s.gcr.io/echoserver:1.10
deployment.apps/hello-node created
$ kubectl get pod
NAME                          READY   STATUS    RESTARTS   AGE
hello-node-59bffcc9fd-8hkgb   1/1     Running   0          36s
$ kubectl expose deployment hello-node --type=NodePort --port=8080
$ kubectl get services
NAME         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
hello-node   NodePort    10.107.66.138   <none>        8080:31529/TCP   1m

Hostname: hello-node-59bffcc9fd-8hkgb
Pod Information:
	-no pod information available-
Server values:
	server_version=nginx: 1.13.3 - lua: 10008
Request Information:
	client_address=192.168.53.147
	method=GET
	real path=/
	query=
	request_version=1.1
	request_scheme=http
	request_uri=http://10.107.66.138:8080/
Request Headers:
	accept=*/*
	host=10.107.66.138:8080
	user-agent=curl/7.74.0
Request Body:
	-no body in request-

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v0.46.0/deploy/static/provider/cloud/deploy.yaml

cat > hello-ingress.yaml <<EOF
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: example-ingress
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /$1
spec:
  rules:
    - host: udon.mynetwork
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: hello-node
                port:
                  number: 8080
EOF

$ kubectl apply -f hello-ingress.yaml
ingress.networking.k8s.io/example-ingress created
$ kubectl get ingress
NAME              CLASS    HOSTS            ADDRESS   PORTS   AGE
example-ingress   <none>   udon.mynetwork             80      3m8s

noodles@udon:~$ kubectl get services -n ingress-nginx
NAME                                 TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                    AGE
ingress-nginx-controller             LoadBalancer   10.96.9.41      <pending>     80:32740/TCP,443:30894/TCP 13h
ingress-nginx-controller-admission   ClusterIP      10.111.16.129   <none>        443/TCP                    13h

kubectl patch svc ingress-nginx-controller -n ingress-nginx -p \
	' "spec":  "type": "LoadBalancer", "externalIPs":["192.168.53.147"] '

noodles@udon:~$ kubectl get services -n ingress-nginx
NAME                                 TYPE           CLUSTER-IP      EXTERNAL-IP      PORT(S)                 AGE
ingress-nginx-controller             LoadBalancer   10.96.9.41      192.168.53.147   80:32740/TCP,443:30894/TCP   14h
ingress-nginx-controller-admission   ClusterIP      10.111.16.129   <none>           443/TCP                 14h
noodles@udon:~$ kubectl get ingress
NAME              CLASS    HOSTS           ADDRESS          PORTS   AGE
example-ingress   <none>   udon.mynetwork  192.168.53.147   80      14h

Hostname: hello-node-59bffcc9fd-8hkgb
Pod Information:
	-no pod information available-
Server values:
	server_version=nginx: 1.13.3 - lua: 10008
Request Information:
	client_address=192.168.0.5
	method=GET
	real path=/
	query=
	request_version=1.1
	request_scheme=http
	request_uri=http://udon.mynetwork:8080/
Request Headers:
	accept=*/*
	host=udon.mynetwork
	user-agent=curl/7.64.0
	x-forwarded-for=192.168.53.136
	x-forwarded-host=udon.mynetwork
	x-forwarded-port=80
	x-forwarded-proto=http
	x-real-ip=192.168.53.136
	x-request-id=6aaef8feaaa4c7d07c60b2d05c45f75c
	x-scheme=http
Request Body:
	-no body in request-

• What s going on with the networking?
• Where s the IPv6 (the host in question has native IPv6 routing)?
• How do I deploy my own application pod?
• I should look at a multiple host setup (i.e. a real cluster).
• How much of this goes away if I use a cloud provider like AWS to run the cluster?
• Can I do this with the Debian Kubernetes packages?

• An AWS account with admin rights
• Full control over the domain name you use to send your newsletter
• A baremetal or virtual Linux Debian server you have the root access to
• NodeJS installed (8x and 10x are ok with Mailtrain)
• A MySQL/MariaDB instance with the root access to
• Redis 3x (not 5) if you want to use Redis (not mandatory)

• Configure AWS SES
• Configure your server by:
  • configuring your database
  • install Mailtrain
  • configuring your web server
• Configure your Mailtrain setup

Ask AWS SES to verify a domain

Generate the DKIM settings

AWS SES pending verification

AWS SES SMTP settings and credentials

MariaDB [(none)]> create database mailtrain;
Query OK, 1 row affected (0.00 sec)
MariaDB [(none)]> CREATE USER 'mailtrain' IDENTIFIED BY 'V3rYD1fF1cUlTP4sSW0rd!';
Query OK, 0 rows affected (0.01 sec)
MariaDB [(none)]> GRANT ALL PRIVILEGES ON mailtrain.* TO 'mailtrain'@localhost IDENTIFIED BY 'V3rYD1fF1cUlTP4sSW0rd!';
Query OK, 0 rows affected (0.00 sec)
MariaDB [(none)]> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.00 sec)
MariaDB [(none)]> show databases;
+--------------------+
  Database            
+--------------------+
  information_schema  
  mailtrain           
  mysql               
  performance_schema  
+--------------------+
6 rows in set (0.00 sec)
MariaDB [(none)]> Bye

[mysql]
host="localhost"
user="mailtrain"
password="V3rYD1ff1culT!"
database="mailtrain"

screen
NODE_ENV=production npm start

groupadd mailtrain
useradd -g mailtrain
chown -R mailtrain:mailtrain /var/www/mailtrain 

[Unit]
 Description=mailtrain
 After=network.target
[Service]
 Type=simple
 User=mailtrain
 WorkingDirectory=/var/www/mailtrain/
 Environment="NODE_ENV=production"
 Environment="PORT=3000"
 ExecStart=/usr/bin/npm run start
 TimeoutSec=15
 Restart=always
[Install]
 WantedBy=multi-user.target

systemctl daemon-reload
systemctl start mailtrain.service

map $http_upgrade $connection_upgrade  
  default upgrade;
  ''      close;
 
server  
  listen 80; 
  listen [::]:80;
  server_name mailtrain.toto.com;
  return 301 https://$host$request_uri;
 
server  
  listen 443 ssl;
  listen [::]:443 ssl;
  server_name mailtrain.toto.com;
  access_log /var/log/nginx/mailtrain.toto.com.access.log;
  error_log /var/log/nginx/mailtrain.toto.com.error.log;
  ssl_protocols TLSv1.2;
  ssl_ciphers EECDH+AESGCM:EECDH+AES;
  ssl_ecdh_curve prime256v1;
  ssl_prefer_server_ciphers on; 
  ssl_session_cache shared:SSL:10m;
  ssl_certificate     /etc/letsencrypt/live/mailtrain.toto.com/fullchain.pem;
  ssl_certificate_key /etc/letsencrypt/live/mailtrain.toto.com/privkey.pem;
  keepalive_timeout    70; 
  sendfile             on;
  client_max_body_size 0;
  root /var/www/mailtrain;
  location ~ /\.well-known\/acme-challenge  
    allow all;
   
  gzip on; 
  gzip_disable "msie6";
  gzip_vary on; 
  gzip_proxied any;
  gzip_comp_level 6;
  gzip_buffers 16 8k; 
  gzip_http_version 1.1;
  gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;
  add_header Strict-Transport-Security "max-age=31536000";
  location /   
    try_files $uri @proxy;
   
  location @proxy  
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto https;
    proxy_pass http://127.0.0.1:3000;
   
 

systemctl start nginx

Mailtrain mailer setup

Mailtrain to throttle sending emails to AWS SES

• Carl Chenet s blog
• Twitter account
• Mastodon account

• the whole inactivation/reactivation process can be modelled by a simple reversible reaction: $$ \mathrm A \rightleftharpoons \mathrm I $$ A is the active form, I the inactive form;
• the forward rate constant is \(k_i(E)\) (dependent on potential) and the backward rate constant is \(k_a(E)\), also dependent on potential;
• the experiment is done in a series of 5 steps at 3 different potentials: \(E_0\) then \(E_1\) then \(E_2\) then \(E_1\) then, finally, \(E_0\);
• the enzyme is assumed to be fully active at the beginning of the first step;
• a single experiment is used to obtain the values of \(k_i\) and \(k_a\) for the three potentials (although not reliably for the value at \(E_0\)
• the current given by the active species depends on potential (and it is negative because the enzyme catalyzes a reduction), and the inactive species gives no current;
• in addition to the reversible reaction above, there is an irreversible, potential-dependent loss.

QSoas> cd
QSoas> load 27.oxw
QSoas> load 27-blanc.oxw
QSoas> S 1 0

QSoas> strip-if x<30 x>300

QSoas> fit-linear-kinetic-system /species=2 /steps=0,1,2,1,0

• The I_1 and I_2 are the currents for the species 1 and 2, so something for 1 (active form) and 0 for 2 (inactive form). Moreover, the parameters I_2_#0 (and _#1, _#2) should be fixed and not free (since we don't need to adjust a current for the inactive form).
• The k_11 and k_22 correspond to species-specific irreversible loss. It is generally best to leave them fixed to 0.
• k_12 is the formation of 2 (I) from 1 (A), and k_21 is the formation of A from I. Their values will be determined for the three conditions. The default values should work here.
• The k_loss parameters are the rates of irreversible loss that apply indiscriminately on all species (unlike k_11 and k_22). They are adjusted and ther default values should work too.
• alpha_1_0 and alpha_2_0 are the initial concentrations of species 1 and 2, so they should be fixed to 1 and 0.
• Last, the xstart_a and (_b, _c, _d and _e) correspond to the starting times for the steps, here, 0, 60, 120, 210 and 270.

The News

I joined Canonical, this February, to work on Ubuntu full-time! \o/
Those who know, they know that this is really very exciting for me because Canonical has been a dream company for me, for real (more about this below!). And hey, this is my first job, ever, so all the more reason to be psyched about, isn t it? ^_^ P.S. Keep reading and we ll meet my squad really sooon!

The Story Being an undergrad student (batch 2017-2021), I ve been slightly worried during my last two semesters, naturally, thinking about how s it all gonna pan out and what will I be doing, et al, because I ve been seeing all my friends and batchmates getting placed in companies or going for masters or at least having some sort of plans for their future and I, on the other hand, was hopelessly clueless. :D Well, to be fair, I did Google Summer of Code twice, in 2019 and 2020, became a Debian Developer in 2019, been a part of GCI and Outreachy, contributed to over dozens of open-source projects, et al, et al. So I wasn t all completely hopeless but for sure was completely clueless , heh. And for full disclosure, I was only slightly panicking because firstly, I did get placed in several companies and secondly, I didn t really need a job immediately since I was already getting paid to work on Debian stuff by Freexian, which was good enough. :)
(and honestly, Freexian has my whole heart! - more on that later sometime.) But that s not the point. I was still confused and worried and my mom & dad, more so than anyone. Ugh. We were all figuring out and she asked me places that I was interested to work in. And whilst I wasn t clear about things I wanted to do (and still am!) but I was (very) clear about this and so I told her about Canonical and also did tell her that it s a bit too ambitious for me to think about it now so I ll probably apply after some experience or something. and as they say, the world works in mysterious ways and well, it did for me! So back during the Ruby sprints (Feb 20), Kanashiro, the guy ( ), mentioned that his team was hiring and has a vacant position but I won t be eligible since I was still in my junior year. It was since then I ve been actively praying for Cronus, the god of time, to wave his magic wand and align it in such a way that the next opening should be somewhere near my graduation. And guess what? IT HAPPENED! 9 months later, in November 20, Kanashiro told me his team is hiring yet again and that I could apply this time! Without much (since there was some ) delay, I applied and started asking all sorts of questions to Kanashiro. No words are enough for him, he literally helped me throughout the process; from referring me to answering all sorts of doubts I had! And roughly after 2 months of interviewing, et al, my ambitious dream did come true and I finalyyyy signed my contract! \o/
(the interview process and what went on during those 10 weeks is a story for later ;))

The Server Team! \o This position, which I didn t mention earlier, was for the Server Team which is a team of 15 people, working to make Ubuntu server the best! And as I tweeted sometime back, the team is absolutely lovely, super kind, and consists of the best of teammates one could possibly ask for! Here s a quick sneak peek into our weekly team meeting. Thanks to Rafael for taking such a lovely picture. And yes, the cat Luna is a part of our squad! And oh, did I mention that we re completely remote and distributed?
FUN FACT: Our team covers all the TZs, that is, at any point of time (during weekdays), you ll find someone or the other from the team around! \o/ Anyway, our squad, managed by Rick is divided into two halves: Squeaky Wheels and Table Flip. Cool names, right?
Squeaky Wheels does the distro side of stuff and consists of Christian, Andreas, Rafael, Robie, Bryce, Sergio, Kanashiro, Athos, and now myself as well! And OTOH, Table Flip consists of Dan, Chad, Paride, Lucas, James, and Grant. Even though I interact w/ Squeaky Wheels more (basically daily), each of my teammates is absolutely lovely and equally awesome! Whilst I ll talk more about things here in the upcoming months, this is it for now! If there s anything, in particular, you d like to know more about, let me know! And lastly, here s us vibing our way through, making Ubuntu server better, cause that s how we roll!

---

Until next time.
:wq for today.

The Debian Janitor is an automated system that commits fixes for (minor) issues in Debian packages that can be fixed by software. It gradually started proposing merges in early December. The first set of changes sent out ran lintian-brush on sid packages maintained in Git. This post is part of a series about the progress of the Janitor. The FOSS world uses a wide variety of different build tools; given a git repository or tarball, it can be hard to figure out how to build and install a piece of software. Humans will generally know what build tool a project is using when they check out a project from git, or they can read the README. And even then, the answer may not always be straightforward to everybody. For automation, there is no obvious place to figure out how to build or install a project.

1
2
3
4
5
6
7
8
9

% git clone https://github.com/dulwich/dulwich
% cd dulwich
% ogni --schroot=unstable-amd64-sbuild dist
 
Writing dulwich-0.20.21/setup.cfg
creating dist
Creating tar archive
removing 'dulwich-0.20.21' (and everything under it)
Found new tarball dulwich-0.20.21.tar.gz in /var/run/schroot/mount/unstable-amd64-sbuild-974d32d7-6f10-4e77-8622-b6a091857e85/build/tmpucazj7j7/package/dist.

1
2
3
4
5
6
7
8
9

% wget https://download.samba.org/pub/ldb/ldb-2.3.0.tar.gz
% tar xvfz ldb-2.3.0.tar.gz
% cd ldb-2.3.0
% ogni install --prefix=/tmp/ldb
 
+ install /tmp/ldb/include/ldb.h (from include/ldb.h)
 
Waf: Leaving directory  /tmp/ldb-2.3.0/bin/default'
'install' finished successfully (11.395s)

1
2
3
4
5
6

% wget  https://cpan.metacpan.org/authors/id/T/TO/TORU/XML-LibXML-LazyBuilder-0.08.tar.gz _ <https://cpan.metacpan.org/authors/id/T/TO/TORU/XML-LibXML-LazyBuilder-0.08.tar.gz> _
% tar xvfz XML-LibXML-LazyBuilder-0.08.tar.gz
Cd XML-LibXML-LazyBuilder-0.08
% ogni test
 

The Population Issue and its many facets Another couple of weeks passed. A Lot of things happening, lots of anger and depression in folks due to handling in pandemic, but instead of blaming they are willing to blame everybody else including the population. Many of them want forced sterilization like what Sanjay Gandhi did during the Emergency (1975). I had to share So Long, My son . A very moving tale of two families of what happened to them during the one-child policy in China. I was so moved by it and couldn t believe that the Chinese censors allowed it to be produced, shot, edited, and then shared worldwide. It also won a couple of awards at the 69th Berlin Film Festival, silver bear for the best actor and the actress in that category. But more than the award, the theme, and the concept as well as the length of the movie which was astonishing. Over a 3 hr. something it paints a moving picture of love, loss, shame, relief, anger, and asking for forgiveness. All of which can be identified by any rational person with feelings worldwide.

Girl child What was also interesting though was what it couldn t or wasn t able to talk about and that is the Chinese leftover men. In fact, a similar situation exists here in India, only it has been suppressed. This has been more pronounced more in Asia than in other places. One big thing in this is human trafficking and mostly women trafficking. For the Chinese male, that was happening on a large scale from all neighboring countries including India. This has been shared in media and everybody knows about it and yet people are silent. But this is not limited to just the Chinese, even Indians have been doing it. Even yesteryear actress Rupa Ganguly was caught red-handed but then later let off after formal questioning as she is from the ruling party. So much for justice. What is and has been surprising at least for me is Rwanda which is in the top 10 of some of the best places in equal gender. It, along with other African countries have also been in news for putting quite a significant amount of percentage of GDP into public healthcare (between 20-10%), but that is a story for a bit later. People forget or want to forget that it was in Satara, a city in my own state where 220 girls changed their name from nakusha or unwanted to something else and that had become a piece of global news. One would think that after so many years, things would have changed, the only change that has happened is that now we have two ministries, The Ministry of Women and Child Development (MoWCD) and The Ministry of Health and Welfare (MoHFW). Sadly, in both cases, the ministries have been found wanting, Whether it was the high-profile Hathras case or even the routine cries of help which given by women on the twitter helpline. Sadly, neither of these ministries talks about POSH guidelines which came up after the 2012 gangrape case. For both these ministries, it should have been a pinned tweet. There is also the 1994 PCPNDT Act which although made in 1994, actually functioned in 2006, although what happens underground even today nobody knows . On the global stage, about a decade ago, Stephen J. Dubner and Steven Levitt argued in their book Freakonomics how legalized abortion both made the coming population explosion as well as expected crime rates to be reduced. There was a huge pushback on the same from the conservatives and has become a matter of debate, perhaps something that the Conservatives wanted. Interestingly, it hasn t made them go back but go forward as can be seen from the Freakonomics site.

Climate Change Another topic that came up for discussion was repeatedly climate change, but when I share Shell s own 1998 Confidential report titled Greenhouse effect all become strangely silent. The silence here is of two parts, there probably is a large swathe of Indians who haven t read the report and there may be a minority who have read it and know what already has been shared with U.S. Congress. The Conservative s argument has been for it is jobs and a weak we need to research more . There was a partial debunk of it on the TBD podcast by Matt Farell and his brother Sean Farell as to how quickly the energy companies are taking to the coming change.

Health Budget Before going to Covid stories. I first wanted to talk about Health Budgets. From the last 7 years the Center s allocation for health has been between 0.34 to 0.8% per year. That amount barely covers the salaries to the staff, let alone any money for equipment or anything else. And here by allocation I mean, what is actually spent, not the one that is shared by GOI as part of budget proposal. In fact, an article on Wire gives a good breakdown of the numbers. Even those who are on the path of free markets describe India s health business model as a flawed one. See the Bloomberg Quint story on that. Now let me come to Rwanda. Why did I chose Rwanda, I could have chosen South Africa where I went for Debconf 2016, I chose because Rwanda s story is that much more inspiring. In many ways much more inspiring than that South Africa in many ways. Here is a country which for decades had one war or the other, culminating into the Rwanda Civil War which ended in 1994. And coincidentally, they gained independence on a similar timeline as South Africa ending Apartheid in 1994. What does the country do, when it gains its independence, it first puts most of its resources in the healthcare sector. The first few years at 20% of GDP, later than at 10% of GDP till everybody has universal medical coverage. Coming back to the Bloomberg article I shared, the story does not go into the depth of beyond-expiry date medicines, spurious medicines and whatnot. Sadly, most media in India does not cover the deaths happening in rural areas and this I am talking about normal times. Today what is happening in rural areas is just pure madness. For last couple of days have been talking with people who are and have been covering rural areas. In many of those communities, there is vaccine hesitancy and why, because there have been whatsapp forwards sharing that if you go to a hospital you will die and your kidney or some other part of the body will be taken by the doctor. This does two things, it scares people into not going and getting vaccinated, at the same time they are prejudiced against science. This is politics of the lowest kind. And they do it so that they will be forced to go to temples or babas and what not and ask for solutions. And whether they work or not is immaterial, they get fixed and property and money is seized. Sadly, there are not many Indian movies of North which have tried to show it except for oh my god but even here it doesn t go the distance. A much more honest approach was done in Trance . I have never understood how the South Indian movies are able to do a more honest job of story-telling than what is done in Bollywood even though they do in 1/10th the budget that is needed in Bollywood. Although, have to say with OTT, some baggage has been shed but with the whole film certification rearing its ugly head through MEITY orders, it seems two steps backward instead of forward. The idea being simply to infantilize the citizens even more. That is a whole different ball-game which probably will require its own space.

Vaccine issues One good news though is that Vaccination has started. But it has been a long story full of greed by none other than GOI (Government of India) or the ruling party BJP. Where should I start with. I probably should start with this excellent article done by Priyanka Pulla. It is interesting and fascinating to know how vaccines are made, at least one way which she shared. She also shared about the Cutter Incident which happened in the late 50 s. The response was on expected lines, character assassination of her and the newspaper they published but could not critique any of the points made by her. Not a single point that she didn t think about x or y. Interestingly enough, in January 2021 Bharati Biotech was supposed to be share phase 3 trial data but hasn t been put up in public domain till May 2021. In fact, there have been a few threads raised by both well-meaning Indians as well as others globally especially on twitter to which GOI/ICMR (Indian Council of Medical Research) is silent. Another interesting point to note is that Russia did say in its press release that it is possible that their vaccine may not be standard (read inactivation on their vaccines and another way is possible but would take time, again Brazil has objected, but India hasn t till date.) What also has been interesting is the homegrown B.1.617 lineage or known as double mutant . This was first discovered from my own state, Maharashtra and then transported around the world. There is also B.1.618 which was found in West Bengal and is same or supposed to be similar to the one found in South Africa. This one is known as Triple mutant . About B.1.618 we don t know much other than knowing that it is much more easily transferable, much more infectious. Most countries have banned flights from India and I cannot fault them anyway. Hell, when even our diplomats do not care for procedures to be followed during the pandemic then how a common man is supposed to do. Of course, now for next month, Mr. Modi was supposed to go and now will not attend the G7 meeting. Whether, it is because he would have to face the press (the only Prime Minister and the only Indian Prime Minister who never has faced free press.) or because the Indian delegation has been disinvited, we would never know.

A good article which shares lots of lows with how things have been done in India has been an article by Arundhati Roy. And while the article in itself is excellent and shares a bit of the bitter truth but is still incomplete as so much has been happening. The problem is that the issue manifests in so many ways, it is difficult to hold on. As Arundhati shared, should we just look at figures and numbers and hold on, or should we look at individual ones, for e.g. the one shared in Outlook India. Or the one shared by Dr. Dipshika Ghosh who works in Covid ICU in some hospital

Dr. Dipika Ghosh sharing an incident in Covid Ward

Interestingly as well, while in the vaccine issue, Brazil Anvisa doesn t know what they are doing or the regulator just isn t knowledgeable etc. (statements by various people in GOI, when it comes to testing kits, the same is an approver.)

ICMR/DGCI approving internationally validated kits, Press release.

Twitter In the midst of all this, one thing that many people have forgotten and seem to have forgotten that Twitter and other tools are used by only the elite. The reason why the whole thing has become serious now than in the first phase is because the elite of India have also fallen sick and dying which was not the case so much in the first phase. The population on Twitter is estimated to be around 30-34 million and people who are everyday around 20 odd million or so, which is what 2% of the Indian population which is estimated to be around 1.34 billion. The other 98% don t even know that there is something like twitter on which you can ask help. Twitter itself is exclusionary in many ways, with both the emoticons, the language and all sorts of things. There is a small subset who does use Twitter in regional languages, but they are too small to write anything about. The main language is English which does become a hindrance to lot of people.

Censorship Censorship of Indians critical of Govt. mishandling has been non-stop. Even U.S. which usually doesn t interfere into India s internal politics was forced to make an exception. But of course, this has been on deaf ears. There is and was a good thread on Twitter by Gaurav Sabnis, a friend, fellow Puneite now settled in U.S. as a professor.

Gaurav on Trump-Biden on vaccination of their own citizens

Now just to surmise what has been happened in India and what has been happening in most of the countries around the world. Most of the countries have done centralization purchasing of the vaccine and then is distributed by the States, this is what we understand as co-operative federalism. While last year, GOI took a lot of money under the shady PM Cares fund for vaccine purchase, donations from well-meaning Indians as well as Industries and trade bodies. Then later, GOI said it would leave the states hanging and it is they who would have to buy vaccines from the manufacturers. This is again cheap politics. The idea behind it is simple, GOI knows that almost all the states are strapped for cash. This is not new news, this I have shared a couple of months back. The problem has been that for the last 6-8 months no GST meeting has taken place as shared by Punjab s Finance Minister Amarinder Singh. What will happen is that all the states will fight in-between themselves for the vaccine and most of them are now non-BJP Governments. The idea is let the states fight and somehow be on top. So, the pandemic, instead of being a public health issue has become something of on which politics has to played. The news on whatsapp by RW media is it s ok even if a million or two also die, as it is India is heavily populated. Although that argument vanishes for those who lose their dear and near ones. But that just isn t the issue, the issue goes much more deeper than that Oxygen:12%
Remedisivir:12%
Sanitiser:12%
Ventilator:12%
PPE:18%
Ambulances 28% Now all the products above are essential medical equipment and should be declared as essential medical equipment and should have price controls on which GST is levied. In times of pandemic, should the center be profiting on those. States want to let go and even want the center to let go so that some relief is there to the public, while at the same time make them as essential medical equipment with price controls. But GOI doesn t want to. Leaders of opposition parties wrote open letters but no effect. What is sad to me is how Ambulances are being taxed at 28%. Are they luxury items or sin goods ? This also reminds of the recent discovery shared by Mr. Pappu Yadav in Bihar. You can see the color of ambulances as shared by Mr. Yadav, and the same news being shared by India TV news showing other ambulances. Also, the weak argument being made of not having enough drivers. Ideally, you should have 2-3 people, both 9-1-1 and Chicago Fire show 2 people in ambulance but a few times they have also shown to be flipped over. European seems to have three people in ambulance, also they are also much more disciplined as drivers, at least an opinion shared by an American expat.

Pappu Yadav, President Jan Adhikar Party, Bihar May 11, 2021

What is also interesting to note is GOI plays this game of Health is State subject and health is Central subject depending on its convenience. Last year, when it invoked the Epidemic and DMA Act it was a Central subject, now when bodies are flowing down the Ganges and pyres being lit everywhere, it becomes a State subject. But when and where money is involved, it again becomes a Central subject. The States are also understanding it, but they are fighting on too many fronts.

Snippets from Karnataka High Court hearing today, 13th March 2021

One of the good things is most of the High Courts have woken up. Many of the people on the RW think that the Courts are doing Judicial activism . And while there may be an iota of truth in it, the bitter truth is that many judges or relatives or their helpers have diagnosed and some have even died due to Covid. In face of the inevitable, what can they do. They are hauling up local Governments to make sure they are accountable while at the same time making sure that they get access to medical facilities. And I as a citizen don t see any wrong in that even if they are doing it for selfish reasons. Because, even if justice is being done for selfish reasons, if it does improve medical delivery systems for the masses, it is cool. If it means that the poor and everybody else are able to get vaccinations, oxygen and whatever they need, it is cool. Of course, we are still seeing reports of patients spending in the region of INR 50k and more for each day spent in hospital. But as there are no price controls, judges cannot do anything unless they want to make an enemy of the medical lobby in the country. A good story on medicines and what happens in rural areas, see no further than Laakhon mein ek.

Allahabad High Court hauling Uttar Pradesh Govt. for lack of Oxygen is equal to genocide, May 11, 2021

The censorship is not just related to takedown requests on twitter but nowadays also any articles which are critical of the GOI s handling. I have been seeing many articles which have shared facts and have been critical of GOI being taken down. Previously, we used to see 404 errors happen 7-10 years down the line and that was reasonable. Now we see that happen, days weeks or months. India seems to be turning more into China and North Korea and become more anti-science day-by-day

Fake websites Before going into fake websites, let me start with a fake newspaper which was started by none other than the Gujarat CM Mr. Modi in 2005 .

Gujarat Satya Samachar 2005 launched by Mr. Modi.

And if this wasn t enough than on Feb 8, 2005, he had invoked Official Secrets Act

Mr. Modi invoking Official Secrets Act, Feb 8 2005 Gujarat Samachar

The headlines were In Modi s regime press freedom is in peril-Down with Modi s dictatorship. So this was a tried and tested technique. The above information was shared by Mr. Urvish Kothari, who incidentally also has his own youtube channel. Now cut to 2021, and we have a slew of fake websites being done by the same party. In fact, it seems they started this right from 2011. A good article on BBC itself tells the story. Hell, Disinfo.eu which basically combats disinformation in EU has a whole pdf chronicling how BJP has been doing it. Some of the sites it shared are

Times of New York
Manchester Times
Times of Los Angeles
Manhattan Post
Washington Herald
and many more. The idea being take any site name which sounds similar to a brand name recognized by Indians and make fool of them. Of course, those of who use whois and other such tools can easily know what is happening. Two more were added to the list yesterday, Daily Guardian and Australia Today. There are of course, many features which tell them apart from genuine websites. Most of these are on shared hosting rather than dedicated hosting, most of these are bought either from Godaddy and Bluehost. While Bluehost used to be a class act once upon a time, both the above will do anything as long as they get money. Don t care whether it s a fake website or true. Capitalism at its finest or worst depending upon how you look at it. But most of these details are lost on people who do not know web servers, at all and instead think see it is from an exotic site, a foreign site and it chooses to have same ideas as me. Those who are corrupt or see politics as a tool to win at any cost will not see it as evil. And as a gentleman Raghav shared with me, it is so easy to fool us. An example he shared which I had forgotten. Peter England which used to be an Irish brand was bought by Aditya Birla group way back in 2000. But even today, when you go for Peter England, the way the packaging is done, the way the prices are, more often than not, people believe they are buying the Irish brand. While sharing this, there is so much of Naom Chomsky which comes to my mind again and again

Caste Issues I had written about caste issues a few times on this blog. This again came to the fore as news came that a Hindu sect used forced labor from Dalit community to make a temple. This was also shared by the hill. In both, Mr. Joshi doesn t tell that if they were volunteers then why their passports have been taken forcibly, also I looked at both minimum wage prevailing in New Jersey as a state as well as wage given to those who are in the construction Industry. Even in minimum wage, they were giving $1 when the prevailing minimum wage for unskilled work is $12.00 and as Mr. Joshi shared that they are specialized artisans, then they should be paid between $23 $30 per hour. If this isn t exploitation, then I don t know what is. And this is not the first instance, the first instance was perhaps the case against Cisco which was done by John Doe. While I had been busy with other things, it seems Cisco had put up both a demurrer petition and a petition to strike which the Court stayed. This seemed to all over again a type of apartheid practice, only this time applied to caste. The good thing is that the court stayed the petition. Dr. Ambedkar s statement if Hindus migrate to other regions on earth, Indian caste would become a world problem given at Columbia University in 1916, seems to be proven right in today s time and sadly has aged well. But this is not just something which is there only in U.S. this is there in India even today, just couple of days back, a popular actress Munmun Dutta used a casteist slur and then later apologized giving the excuse that she didn t know Hindi. And this is patently false as she has been in the Bollywood industry for almost now 16-17 years. This again, was not an isolated incident. Seema Singh, a lecturer in IIT-Kharagpur abused students from SC, ST backgrounds and was later suspended. There is an SC/ST Atrocities Act but that has been diluted by this Govt. A bit on the background of Dr. Ambedkar can be found at a blog on Columbia website. As I have shared and asked before, how do we think, for what reason the Age of Englightenment or the Age of Reason happened. If I were a fat monk or a priest who was privileges, would I have let Age of Enlightenment happen. It broke religion or rather Church which was most powerful to not so powerful and that power was more distributed among all sort of thinkers, philosophers, tinkers, inventors and so on and so forth.

Situation going forward I believe things are going to be far more complex and deadly before they get better. I had to share another term called Comorbidities which fortunately or unfortunately has also become part of twitter lexicon. While I have shared what it means, it simply means when you have an existing ailment or condition and then Coronavirus attacks you. The Virus will weaken you. The Vaccine in the best case just stops the damage, but the damage already done can t be reversed. There are people who advise and people who are taking steroids but that again has its own side-effects. And this is now, when we are in summer. I am afraid for those who have recovered, what will happen to them during the Monsoons. We know that the Virus attacks most the lungs and their quality of life will be affected. Even the immune system may have issues. We also know about the inflammation. And the grant that has been given to University of Dundee also has signs of worry, both for people like me (obese) as well as those who have heart issues already. In other news, my city which has been under partial lockdown since a month, has been extended for another couple of weeks. There are rumors that the same may continue till the year-end even if it means economics goes out of the window.There is possibility that in the next few months something like 2 million odd Indians could die

The above is a conversation between Karan Thapar and an Oxford Mathematician Dr. Murad Banaji who has shared that the under-counting of cases in India is huge. Even BBC shared an article on the scope of under-counting. Of course, those on the RW call of the evidence including the deaths and obituaries in newspapers as a narrative . And when asked that when deaths used to be in the 20 s or 30 s which has jumped to 200-300 deaths and this is just the middle class and above. The poor don t have the money to get wood and that is the reason you are seeing the bodies in Ganges whether in Buxar Bihar or Gajipur, Uttar Pradesh. The sights and visuals makes for sorry reading

Pandit Ranjan Mishra son on his father s death due to unavailability of oxygen, Varanasi, Uttar Pradesh, 11th May 2021.

For those who don t know Pandit Ranjan Mishra was a renowned classical singer. More importantly, he was the first person to suggest Mr. Modi s name as a Prime Ministerial Candidate. If they couldn t fulfil his oxygen needs, then what can be expected for the normal public.

Conclusion Sadly, this time I have no humorous piece to share, I can however share a documentary which was shared on Feluda . I have shared about Feluda or Prodosh Chandra Mitter a few times on this blog. He has been the answer of James Bond from India. I have shared previously about The Golden Fortress . An amazing piece of art by Satyajit Ray. I watched that documentary two-three times. I thought, mistakenly that I am the only fool or fan of Feluda in Pune to find out that there are people who are even more than me. There were so many facets both about Feluda and master craftsman Satyajit Ray that I was unaware about. I was just simply amazed. I even shared few of the tidbits with mum as well, although now she has been truly hooked to Korean dramas. The only solace from all the surrounding madness. So, if you have nothing to do, you can look up his books, read them and then see the movies. And my first recommendation would be the Golden Fortress. The only thing I would say, do not have high hopes. The movie is beautiful. It starts slow and then picks up speed, just like a train. So, till later. Update The Mass surveillance part I could not do justice do hence removed it at the last moment. It actually needs its whole space, article. There is so much that the Govt. is doing under the guise of the pandemic that it is difficult to share it all in one article. As it is, the article is big

• [1] http://tinyurl.com/y7rutdur
• [2] https://en.wikipedia.org/wiki/Ad_hominem
• [3] https://tinyurl.com/yg68dtco

• Principles in Machine Learning
• Principles in Machine Learning /2
• Occam s Razor Principle of Parsimony
• Simple Models
• Computational Learning Theory
• Probably Approximately Correct Learning (PAC Learning) /1
• Probably Approximately Correct Learning (PAC Learning) /2
• PAC Learnable Examples
• VC Dimension (Vapnik-Chervonenkis-Dimension )
• VC Dimension Example
• Error Bounds and the VC Dimension
• No Free Lunch
• No Free Lunch Theorem
• No Free Lunch Theorem Explanation
• Bias-Variance Tradeoff
• Bias-Variance Tradeoff
• Bias vs. Variance
• Bias-Variance Decomposition
• Bias-Variance Illustration
• Different Kinds of Bias
• Data Often Has Bias
• AI Can Be Sexist and Racist
• Relationships

• Correlation does not Imply Causation
• Correlation does not Imply Causation /2
• Correlation does not Imply Causation /3
• Correlation with Statistics Classes
• Multiple Testing Problem
• Bonferroni s Principle Multiple Testing Problem
• Multiple Testing Problem

• Overfitting
• Underfitting and Overfitting
• Overfitting Decision Tree
• Overfitting Due to Noise
• Overfitting Due to Insufficient Examples

• Curse of Dimensionality
• Combinatorial Explosion
• Concentration of Distances
• Data is in the Margins
• Illustration: Shrinking Hyperspheres
• Illustration: Shrinking Hyperspheres /2
• Effect on Search in High Dimensionality
• Summary

• Intrinsic Dimensionality
• Estimating Intrinsic Dimensionality
• Angle-Based Intrinsic Dimensionality Intuition
• Angle-Based Intrinsic Dimensionality (ABID) /2
• Consequences & Solutions

• Distance Functions
• Distances, Metrics and Similarities
• Distances, Metrics and Similarities /2
• Distance Functions
• Distance Functions /2
• Similarity Functions
• Distances for Binary Data
• Jaccard Coefficient for Sets
• Example Distances for Categorical Data
• Mahalanobis Distance
• Scaling & Normalization
• To Scale, or not to Scale?
• To Scale, or not to Scale? /2

• Classification
• Prediction Problems
• Classification: A Multi-Stage Process
• Classification Problem
• Example
• Process of Constructing a Model
• Process of Applying the Model

• Evaluation and Selection of Classifiers
• Quick Recap: Classification
• Classifier Evaluation: Confusion Matrix
• Classifier Evaluation: Accuracy and Error-Rate
• Precision, Recall, and F-measure
• Classifier Evaluation: Multi-Class Confusion Matrix
• Training Accuracy vs. Accuarcy on New Data
• The Need for Validation
• Holdout Validation
• Cross-Validation
• Bootstrap Validation
• Considerations for Selecting a Model

• Bayesian Classification
• Bayes Classification: Motivation
• Bayes Theorem: Review
• Optimal Bayes Classifier
• Na ve Bayes Classifier
• Probability Models for a Single Attribute
• Multivariate Gaussian Bayes Classification
• Na ve Bayes Classifier: Example
• Na ve Bayes Classifier: Computational Aspects
• Na ve Bayes Classifier: Comments & Discussion

• Nearest-Neighbor Classification
• Nearest Neighbor Classifier Motivation
• Nearest Neighbor Classifier: Foundations
• Nearest Neighbor Classifier: Example
• Nearest Neighbor Classification: Example
• Nearest Neighbor Decision Rules

• Nearest-Neighbor as Density estimation
• Nearest Neighbor Classification and Density Estimation
• Predicting with Kernel Density Estimation with k=1,3,5,15
• Error Probability of Nearest Neighbors
• Nearest Neighbor Regression
• Nearest-Neighbor Classification: Comments & Discussion

• Decision Tree Learning
• Example (Variant of a Dataset in )
• Decision Tree Example
• Decision Trees as Rule-based Systems
• Basic Notions
• Constructing a Decision Tree /1
• Visual Interpretation of Decision Trees on R
• Constructing a Decision Tree /2
• Decision Tree Classification: Example

• Decision Tree Splitting
• Split for Categorical Attributes
• Split for Numeric Attributes
• Best Split Example
• Quality Measures for Splits
• Measure of Impurity: Gini Index
• Gini-Index: Example
• Information Gain
• Information Gain: Example
• Information Gain: Gain-Ratio
• Gain-Ratio: Beispiel
• Classification Error
• Gini, Entropy and Classification Error
• Comparing Split Selection Measures
• Splits for Numerical Attributes

• Ensembles and Meta-Learning
• Ensembles and Meta-Learning
• Error-Rate of Ensembles
• Random Forests
• Boosting
• Random Forest Classification: Example
• Gradient Boosting Classification: Example

• Support Vector Machine Motivation
• Support Vector Machines
• Support Vector Machines /2
• Finding the Best Separating Hyperplane
• Maximum Margin Hyperplane

• Affine Hyperplanes and Scalar Products
• Affine Hyperplanes
• Scalar Products
• Affine Hyperplanes /2

• Maximum Margin Hyperplane
• A Na ve Attempt
• Support Vectors Separable Data
• Computing the Maximum Margin Hyperplane (MMH)
• Computing the Maximum Margin Hyperplane (MMH) /2
• Boundary of the Maximum Margin Hyperplane (MMH)
• Deriving the Primal SVM Optimization Problem

• Training Support Vector Machines
• Optimization Problem
• Karush-Kuhn-Tucker KKT Conditions
• Switching to the Dual Problem
• Classification with the Dual SVM
• Optimizing the i
• Optimizing SVMs
• Sequential Minimal Optimization
• Further Improvements

• Non-linear SVM and the Kernel Trick
• Nonlinear SVM
• Nonlinear SVM /2
• Kernel Functions
• Soft Margin SVM Classifier
• Soft Margin SVM Classifier /2
• Soft Margin SVM Classifier /3
• Soft Margin SVM Classifier /4

• SVM Extensions and Conclusions
• Separation of more than 2 Classes
• Support Vector Regression
• Support Vector Regression Optimization Problem
• Support Vector Regression Dual
• Support Vector Data Description (SVDD)
• SVDD Dual Problem
• Support Vector Clustering
• SVMs: Comments & Discussion

• Neural Networks
• Biological Background (Simplified)
• Biological Background (Simplified) /2

• Threshold Logic Units
• Threshold Logic Units (TLUs)
• Threshold Logic Units Example
• Geometric Interpretation of TLUs
• Exclusive-Or (XOR) Problem
• Exclusive-Or (XOR) Problem /2
• Exclusive-Or (XOR) Problem /3
• Universality of TLUs
• Mark I Perceptron

• General Artificial Neural Networks
• Simplifying Threshold Logic Units
• Weight Matrices
• From TLUs to Multilayer Perceptrons
• Some Activation Functions
• Some Activation Functions /2
• Some Activation Functions /3
• Some Activation Functions /4

• Learning Neural Networks with Backpropagation
• Basic Gradient Descent
• Stochastic Gradient Descent
• Learning Single-Layer Perceptrons
• Backpropagation
• Training with Backpropagation

• Deep Neural Networks
• Universal Approximation Theorem
• Deep vs. Wide Neural Networks
• High vs. Low Dimensionality
• (Early) Problems of Deep Learning
• Autoencoders
• Layer-wise Pre-Training of Deep Neural Networks
• Dropout Regularization
• Batch Normalization
• Choosing Activation Functions

• Convolutional Neural Networks
• Depth vs. Convolutional Kernel Size
• Increasing the Training Data

• Recurrent Neural Networks
• Recurrent Neural Networks (RNNs) on Sequences
• Recurrent Neural Networks (RNN)
• Long-Short Term Memory (LSTM)
• Further Developments

• Conclusion
• Other Classifiers
• Problems of Classification

• Cluster Analysis Introduction
• What is Clustering?
• What is Clustering? /2
• Applications of Clustering
• Basic Steps for Clustering

• Hierarchical Agglomerative Clustering
• Distance of Clusters
• AGNES Agglomerative Nesting
• AGNES Agglomerative Nesting /2
• Extracting Clusters from a Dendrogram
• Benefits and Limitations of HAC

• Accelerating Hierarchical Clustering
• Complexity of Hierarchical Clustering
• Anderberg s Caching
• AGNES vs. Anderberg , NNChain , SLINK
• Example: Hierarchical Clustering with Anderberg

• K-means Clustering
• The Sum of Squares Objective
• The Standard Algorithm (Lloyd s Algorithm)
• Non-determinism & Non-optimality
• Initialization
• Initialization /2
• Complexity of k-Means Clustering

• Accelerating k-Means Clustering
• k-Means++: Weighted Random Initialization
• Making k-means Faster
• Bounding the Distances Elkan and Hamerly
• Hamerly s k-means
• Example: k-Means Clustering with Hamerly s Algorithm
• Speedup with Hamerly, Elkan, and Exponion

• Limitations of k-Means Clustering
• Benefits and Drawbacks of k-Means
• Choosing the Optimum k for k-Means
• Limitations of k-Means

• Extensions of k-Means Clustering
• k-Means and Distances
• k-Means Minimizes Sum of Squares, not Euclidean Distance!
• k-Means Variations for Other Distances
• Spherical k-Means for Text Clustering
• Pre-processing and Post-processing

• Partitioning Around Medoids (k-Medoids)
• k-medoids Clustering
• Partitioning Around Medoids
• Algorithm: Partitioning Around Medoids
• Algorithm: Partitioning Around Medoids /2
• Change in TD
• Finding the Best Swap Faster
• k-Medoids, k-Means style
• Example for the Inferiority of k-Means Style k-Medoids

• Gaussian Mixture Modeling Introduction
• Expectation-Maximization in Clustering
• Fitting Multiple Gaussian Distributions
• Gaussian Mixture Modeling as E-M-Optimization
• Algorithm: EM Clustering
• Numerical Issues in GMM

• Gaussian Mixture Modeling Demo

• BIRCH and BETULA
• BIRCH Clustering
• BIRCH Clustering Features
• BIRCH Distances
• BIRCH CF-Tree
• BETULA Cluster Features
• BETULA Distance Computations
• Accelerating k-Means with BIRCH and BETULA
• Accelerating GMM with BETULA

• Motivation Density-Based Clustering
• Density-based Clustering: Core Idea

• Density-Based Clustering Fundamentals
• Density-based Clustering: Foundations
• Density-based Clustering: Foundations /2
• Density-based Clustering: Foundations /3
• Density-reachability and Density-connectivity
• Density-reachability

• DBSCAN
• Clustering Approach
• Abstract DBSCAN Algorithm
• DBSCAN Algorithm
• DBSCAN Algorithm /2
• DBSCAN Algorithm /3
• DBSCAN in Context

• DBSCAN Parameterization
• Choosing DBSCAN parameters
• Choosing DBSCAN parameters /2
• Choosing DBSCAN parameters /3

• DBSCAN Extensions
• Generalized Density-based Clustering
• Grid-based Accelerated DBSCAN
• Anytime Density-Based Clustering (AnyDBC)
• Hierarchical DBSCAN* (HDBSCAN*)
• Improved DBSCAN Variations

• OPTICS Clustering
• Density-based Hierarchical Clustering
• Density-based Hierarchical Clustering /2
• OPTICS Clustering
• Cluster Order
• OPTICS Algorithm

• Cluster Extraction from OPTICS Plots
• OPTICS Reachability Plots
• Extracting Clusters from OPTICS Reachability Plots
• Role of the Parameters and minPts

• Understanding the OPTICS Cluster Order
• Properties of the OPTICS Cluster Order
• Cluster Order as Serialized Spanning Tree
• OPTICS as Density Spanning Trees
• Cluster Order to Dendrograms

• Spectral Clustering
• Minimum Cuts
• Graph Laplacian
• From Clustering Graphs to Clustering Data
• Spectral Clustering
• Spectral Clustering is Related to DBSCAN

• Biclustering and Subspace Clustering
• Biclustering & Subspace Clustering
• Bicluster Patterns
• Density-based Subspace Clustering
• Subspace Clustering with Apriori-Style Search
• Correlation Clustering
• 4C: Computing Correlation Connected Clusters
• Hough Transform
• CASH: Robust Clustering in Arbitrarily Oriented Subspaces

• Further Clustering Approaches
• CURE Clustering
• ROCK Clustering
• CHAMELEON
• Affinity Propagation Clustering
• Other Density-based Clustering Algorithms
• Further Clustering Approaches

There was a boy named Eustace Clarence Scrubb and he almost deserved it.

"Oh Aslan," said she, "it was kind of you to come." "I have been here all the time," said he, "but you have just made me visible." "Aslan!" said Lucy almost a little reproachfully. "Don't make fun of me. As if anything I could do would make you visible!" "It did," said Aslan. "Did you think I wouldn't obey my own rules?"

"My son," said Ramandu, "it is not for you, a son of Adam, to know what faults a star can commit."

"My own plans are made. While I can, I sail east in the Dawn Treader. When she fails me, I paddle east in my coracle. When she sinks, I shall swim east with my four paws. And when I can swim no longer, if I have not reached Aslan's country, or shot over the edge of the world in some vast cataract, I shall sink with my nose to the sunrise and Peepiceek shall be the head of the talking mice in Narnia."

• Reviewed and merged two pull request from Michael K. to my django-slack library which provides a convenient wrapper between projects using the Django web development framework and the Slack chat platform. Michael's pull requests made Python 3.6+ a hard requirement (#102) and to run the tests against Python 3.9 (#103).
• Made a "no-change" release of my django-email-from-template library to upload a Python wheel (.whl) file to PyPI. [...]

• As part of my role of being the assistant Secretary of the Open Source Initiative and a board director of Software in the Public Interest, I attended their respective monthly meetings. My term on the OSI board has been slightly extended due to the discovery of a vulnerability in OSI's recent election as a result, the 2021 election will be re-run to ensure transparency of the process.
• Filed a pull request against libsass-python (a Python implementation of the Sass CSS preprocessor) in order to make the build reproducible. (#319)

• Interviewed the Ford Foundation's Michael Brennan and published the transcript on our website.
• Drafted, published and publicised our monthly report.
• Filed an upstream pull request for libsass-python (a Python implementation of the Sass CSS preprocessor) in order to make the build reproducible. [...
• In Debian:
  • Kept isdebianreproducibleyet.com up to date. [...]
  • I also submitted 2 patches to fix specific reproducibility issues in pristine-lfs and rust-configparser.
• Categorised a large number of packages and issues in the Reproducible Builds notes.git" repository.

• Updated the main Reproducible Builds website and documentation:
  • Highlight our mailing list on the Contribute. page [...]
  • Add a noun (and drop an unnecessary full-stop) on the landing page. [...][...]
  • Correct a reference to the date metadata attribute on reports, restoring the display of months on the homepage. [...]
  • Correct a typo of "instalment" within a previous news entry. [...]
  • Added a conspicuous "draft" banner to unpublished blog posts in order to match the report draft banner. [...]

• I also made the following changes to diffoscope, including uploading versions 172 and 173 to Debian:
  • Add support for showing annotations in PDF files. (#249)
  • Move to the assert_diff helper in test_pdf.py. [...]

• redis (5:6.2.2-1) (to experimental) New upstream release.
• python-django:
  • 2.2.20-1 New upstream security release.
  • 3.2-1 (to experimental) New major upstream release (release notes).
• hiredis (1.0.0-2) Build with SSL/TLS support (#987114), and overhaul various aspects of the packaging.
• mtools (4.0.27-1) New upstream release.

• Investigated and triaged avahi (CVE-2021-3468), exiv2 (CVE-2021-3482), file-roller (CVE-2020-36314), fluidsynth (CVE-2021-28421), gnuchess (CVE-2021-30184), gpac (CVE-2021-28300), imagemagick (CVE-2021-20309, CVE-2021-20243), ircii (CVE-2021-29376), jetty9 (CVE-2021-28163), libcaca (CVE-2021-30498, CVE-2021-30499), libjs-handlebars, libpano13, libpodofo (CVE-2021-30469, CVE-2021-30470, CVE-2021-30471, CVE-2021-30472), mediawiki, mpv (CVE-2021-30145), nettle (CVE-2021-20305), nginx (CVE-2020-36309), nim (CVE-2021-21372, CVE-2021-21373, CVE-2021-21374), node-glob-parent (CVE-2020-28469), openexr (CVE-2021-3474), python-django-registration (CVE-2021-21416), qt4-x11 (CVE-2021-3481), qtsvg-opensource-src (CVE-2021-3481), ruby-kramdown, scrollz (CVE-2021-29376), syncthing (CVE-2021-21404), thunderbird (CVE-2021-23991, CVE-2021-23992, CVE-2021-23993) & wordpress (CVE-2021-29447).
• Issued DLA 2620-1 to address a cross-site scripting (XSS) vulnerability in python-bleach, a whitelist-based HTML sanitisation library.
• Issued DLA 2622-1 and ELA 402-1 as it was discovered that there was a potential directory traversal issue in Django, the popular Python-based web development framework. The vulnerability could have been exploited by maliciously crafted filenames. However, the upload handlers built into Django itself were not affected. (#986447)
• Jan-Niklas Sohn discovered that there was an input validation failure in the X.Org display server. Insufficient checks on the lengths of the XInput extension's ChangeFeedbackControl request could have lead to out of bounds memory accesses in the X server. These issues could have led to privilege escalation for authorised clients, particularly on systems where the X server is running as a privileged user. I, therefore, issued both DLA 2627-1 and ELA 405-1 to address this problem.
• Frontdesk duties, reviewing others' packages, participating in mailing list discussions, etc., as well as attending our monthly meeting.