Search Results: "spa"

1 November 2020

Paul Wise: FLOSS Activities October 2020

Focus This month I didn't have any particular focus. I just worked on issues in my info bubble.

Changes

Issues

Review
  • Spam: reported 2 Debian bug reports and 147 Debian mailing list posts
  • Patches: merged libicns patches
  • Debian packages: sponsored iotop-c
  • Debian wiki: RecentChanges for the month
  • Debian screenshots:

Administration
  • Debian: get us removed from an RBL
  • Debian wiki: reset email addresses, approve accounts

Communication

Sponsors The pytest-rerunfailures/pyemd/morfessor work was sponsored by my employer. All other work was done on a volunteer basis.

31 October 2020

Chris Lamb: Free software activities in October 2020

Here is my monthly update covering what I have been doing in the free software world during October 2020 (previous month): For Lintian, the static analysis tool for Debian packages, I uploaded versions 2.97.0, 2.98.0, 2.99.0 & 2.100.0 as well as updated the declares-possibly-conflicting-debhelper-compat-versions tag as we may specify the Debhelper compatibility level in debian/rules or debian/control (#972464) and dropped a reference to missing manual page [...].

Reproducible Builds One of the original promises of open source software is that distributed peer review and transparency of process results in enhanced end-user security. However, whilst anyone may inspect the source code of free and open source software for malicious flaws, almost all software today is distributed as pre-compiled binaries. This allows nefarious third-parties to compromise systems by injecting malicious code into ostensibly secure software during the various compilation and distribution processes. The motivation behind the Reproducible Builds effort is to ensure no flaws have been introduced during this compilation process by promising identical results are always generated from a given source, thus allowing multiple third-parties to come to a consensus on whether a build was compromised. The project is proud to be a member project of the Software Freedom Conservancy. Conservancy acts as a corporate umbrella allowing projects to operate as non-profit initiatives without managing their own corporate structure. If you like the work of the Conservancy or the Reproducible Builds project, please consider becoming an official supporter. This month, I: I also updated the main Reproducible Builds website and documentation:
Lastly, I made the following changes to diffoscope, including preparing and uploading version 161 to Debian: trydiffoscope is the web-based version of diffoscope. This month, I made the following changes:

Debian Debian LTS This month I have worked 18 hours on Debian Long Term Support (LTS) and 12 hours on its sister Extended LTS project. You can find out more about the project via the following video:

Uploads Bugs filed

Jonathan Carter: Free Software Activities for 2020-10

Another month, another bunch of uploads. The freeze for Debian 11 (bullseye) is edging closer, so I ve been trying to get my package list in better shape ahead of that. Thanks to those who worked on fixing lintian.debian.org and the lintian reports on the QA pages, those are immensely useful and it s great to have that back! 2020-10-04: Upload package gnome-shell-extension-draw-on-your-screen (8-1) to Debian unstable. 2020-10-05: Sponsor package flask-restful (0.3.8-4) for Debian unstable (Python Team request). 2020-10-05: Sponsor package python-potr (1.0.2-3) for Debian unstable (Python Team request). 2020-10-06: Sponsor package python-pyld (2.0.3-1) for Debian unstable (Python Team request). 2020-10-06: Sponsor package flask-openid (1.2.5+dfsg-4) for Debian unstable (Python Team request). 2020-10-06: Sponsor package qosmic (1.6.0-4) for Debian unstable (E-mail request). 2020-10-07: File removal for gnome-shell-extension-workspace-to-dock (RC Buggy, no longer maintained: #971803). 2020-10-07: Upload package gnome-shell-extension-pixelsaver (1.20-2) to Debian unstable (Closes: #971689). 2020-10-07: Upload package calamares (3.2.31-1) to Debian unstable. 2020-10-07: Upload package gnome-shell-extension-dashtodock (69-1) to Debian unstable (Closes: #971654). 2020-10-08: Sponsor package python3-libcloud (3.020-1) for Debian unstable. 2020-10-09: Upload package gnome-shell-extension-dashtopanel (40-1) to Debian unstable (Closes: #971087). 2020-10-09: Upload package gnome-shell-extension-draw-on-your-screen (8.1-1) to Debian unstable. 2020-10-12: Upload package gnome-shell-extension-pixelsaver (1.24-1) to Debian unstable. 2020-10-14: Sponsor package python3-onewire (0.2-1) for Debian unstable (Python Team request). 2020-10-15: Sponsor package cheetah (3.2.5-1) for Debian unstable (Python Team request). 2020-10-15: Sponsor package xmodem (0.4.6+dfsg-1) for Debian unstable (Python Team request). 2020-10-15: Sponsor package ansi (0.1.5-1) for Debian unstable (Python Team request). 2020-10-15: Sponsor package cbor2 (5.2.0-1) for Debian unstable (Python Team request). 2020-10-16: Upload package calamares (3.2.32-1) to Debian unstable. 2020-10-17: Upload package calamares (3.2.32.1-1) to Debian unstable. 2020-10-18: Upload package kpmcore (4.2.0-1) to Debian unstable. 2020-10-18: Upload package gnome-shell-extension-draw-on-your-screen (9-1) to Debian unstable. 2020-10-18: Upload package bundlewrap (4.2.1-1) to Debian unstable. 2020-10-18: Upload package bcachefs-tools (0.1+git20201017.8a4408-1~exp1) to Debian experimental. 2020-10-18: Upload package calamares (3.2.32.1-2) to Debian unstable. 2020-10-18: Upload package partitionmanager (4.1.0-2) to Debian unstable. 2020-10-19: Upload package kpmcore (4.2.0-2) to Debian unstable. 2020-10-21: Upload package calamares (3.2.32.1-3) to Debian unstable. 2020-10-21: Upload package calamares-settings-debian (11.0.3-1) to Debian unstable (Closes: #969930, #941301). 2020-10-21: Upload package partitionmanager (4.2.0-1) to Debian unstable. 2020-10-21: Upload package gnome-shell-extension-hard-disk-led (22-1) to Debian unstable (Closes: #971041). 2020-10-21: Merge MR!1 for catimg (Janitor improvements). 2020-10-21: Sponsor package r4d (1.7-1) for Debian unstable (Python Team request). 2020-10-22: Upload package aalib (1.4rc5-47) to Debian unstable. 2020-10-22: Upload package fabulous (0.3.0+dfsg1-8) to Debian unstable. 2020-10-22: Merge MR!1 for gdisk (Janitor improvements). 2020-10-22: Merge MR!1 for gnome-shell-extension-arc-menu (New upstream URLs, thanks Edward Betts). 2020-10-22: Upload package gnome-shell-extension-arc-menu (49-1) to Debian unstable. 2020-10-22: Upload package gnome-shell-extension-draw-on-your-screen (10-1) to Debian unstable. 2020-10-22: Merge MR!1 for vim-airline (Janitor improvements). 2020-10-22: Merge MR!1 for vim-airline-themes (Janitor improvements). 2020-10-22: Merge MR!1 for preload (Janitor improvements). 2020-10-22: Upload package aalib (1.4rc5-48) to Debian unstable. 2020-10-22: Upload package gnome-shell-extension-trash (0.2.0-git20200326.3425fcf1-1). 2020-10-26: Upload package bcachefs-tools (0.1+git20201025.742dbbdb-1) to Debian unstable. 2020-10-26: Sponsor package dunst (1.5.0-1) for Debian unstable (mentors.debian.net request).

29 October 2020

Ulrike Uhlig: Better handling emergencies

We all know these situations when we receive an email asking Can you check the design of X, I need a reply by tonight. Or an instant message: My website went down, can you check? Another email: I canceled a plan at the hosting company, can you restore my website as fast as possible? A phone call: The TLS certificate didn t get updated, and now we can t access service Y. Yet another email: Our super important medical advice website is suddenly being censored in country Z, can you help? Everyone knows those messages that have URGENT in capital letters in the email subject. It might be that some of them really are urgent. Others are the written signs of someone having a hard time properly planning their own work and passing their delays on to someone who comes later in the creation or production chain. And others again come from people who are overworked and try to delegate some of their tasks to a friendly soul who is likely to help.

How emergencies create more emergencies In the past, my first reflex when I received an urgent request was to start rushing into solutions. This happened partly out of empathy, partly because I like to be challenged into solving problems, and I m fairly good at that. This has proven to be unsustainable, and here is why.

Emergencies create unplanned work The first issue is that emergencies create a lot of unplanned work. Which in turn means not getting other, scheduled, things done. This can create a backlog, end up in working late, or working on weekends.

Emergencies can create a permanent state of exception Unplanned work can also create a lot of frustration, out of the feeling of not getting the things done that one planned to do. We might even get a feeling of being nonautonomous (in German I would say fremdbestimmt, which roughly translates to being directed by others ). On the long term, this can generate unsustainable situations: higher work loads, and burnout. When working in a team of several people, A might have to take over the work of B because B has not enough capacities. Then A gets overloaded in turn, and C and D have to take over A s work. Suddenly the team is stuck in a permanent state of exception. This state of exception will produce more backlog. The team might start to deprioritize social issues over getting technical things done. They might not be able to recruit new people anymore because they have no capacity left to onboard newcomers.

One emergency can result in a variety of emergencies for many people The second issue produced by urgent requests is that if I cannot solve the initial emergency by myself, I might try to involve colleagues, other people who are skilled in the area, or people who work in another relevant organization to help with this. Suddenly, the initial emergency has become my emergency as well as the emergency of a whole bunch of other people.

A sidenote about working with friends This might be less of an issue in a classical work setup than in a situation in which a bunch of freelancers work together, or in setups in which work and friendships are intertwined. This is a problem, because the boundaries between friend and worker role, and the expectations that go along with these roles, can get easily confused. If a colleague asks me to help with task X, I might say no; if a friend asks, I might be less likely to say no.

What I learnt about handling emergencies I came up with some guidelines that help me to better handle emergencies.

Plan for unplanned work It doesn t matter and it doesn t help to distinguish if urgent requests are legitimate or if they come from people who have not done their homework on time. What matters is to make one s weekly todo list sustainable. After reading Making work visible by Domenica de Grandis, I understood the need to add free slots for unplanned work into one s weekly schedule. Slots for unplanned work can take up to 25% of the total work time!

Take time to make plans Now that there are some free slots to handle emergencies, one can take some time to think when an urgent request comes in. A German saying proposes to wait and have some tea ( abwarten und Tee trinken ). I think this is actually really good advice, and works for any non-obvious problem. Sit down and let the situation sink in. Have a tea, take a shower, go for a walk. It s never that urgent. Really, never. If possible, one can talk about the issue with another person, rubberduck style. Then one can make a plan on how to address the emergency properly, it could be that the solution is easier than at first thought.

Affirming boundaries: Saying no Is the emergency that I m asked to solve really my problem? Or is someone trying to involve me because they know I m likely to help? Take a deep breath and think about it. No? It s not my job, not my role? I have no time for this right now? I don t want to do it? Maybe I m not even paid for it? A colleague is pushing my boundaries to get some task on their own todo list done? Then I might want to say no. I can t help with this. or I can help you in two weeks. I don t need to give a reason. No. is a sentence. And: Saying no doesn t make me an arse.

Affirming boundaries: Clearly defining one s role Clearly defining one s role is something that is often overlooked. In many discussions I have with friends it appears that this is a major cause of overwork and underpayment. Lots of people are skilled, intelligent, and curious, and easily get challenged into putting on their super hero dress. But they re certainly not the only person that can help even if an urgent request makes them think that at first. To clearly define our role, we need to make clear which part of the job is our work, and which part needs to be done by other people. We should stop trying to accomodate people and their requests to the detriment of our own sanity. You re a language interpreter and are being asked to mediate a bi-lingual conflict between the people you are interpreting for? It s not your job. You re the graphic designer for a poster, but the text you ve been given is not good enough? Send back a recommendation to change the text; don t do these changes yourself: it s not your job. But you can and want to do this yourself and it would make your client s life easier? Then ask them to get paid for the extra time, and make sure to renegotiate your deadline!

Affirming boundaries: Defining expectations Along with our role, we need to define expectations: in which timeframe am I willing to do the job? Under which contract, which agreement, which conditions? For which payment? People who work in a salary office job generally do have a work contract in which their role and the expectations that come with this role are clearly defined. Nevertheless, I hear from friends that their superiors regularly try to make them do tasks that are not part of their role definition. So, here too, role and expectations sometimes need to be renegotiated, and the boundaries of these roles need to be clearly affirmed.

Random conclusive thoughts If you ve read until here, you might have experienced similar things. Or, on the contrary, maybe you re already good at communicating your boundaries and people around you have learnt to respect them? Congratulations. In any case, for improving one s own approach to such requests, it can be useful to find out which inner dynamics are at play when we interact with other people. Additionally, it can be useful to understand the differences between Asker and Guesser culture:
when an Asker meets a Guesser, unpleasantness results. An Asker won t think it s rude to request two weeks in your spare room, but a Guess culture person will hear it as presumptuous and resent the agony involved in saying no. Your boss, asking for a project to be finished early, may be an overdemanding boor or just an Asker, who s assuming you might decline. If you re a Guesser, you ll hear it as an expectation.
Askers should also be aware that there might be Guessers in their team. It can help to define clear guidelines about making requests (when do I expect an answer, under which budget/contract/responsibility does the request fall, what other task can be put aside to handle the urgent task?) Last, but not least, Making work visible has a lot of other proposals on how to visibilize and then deal with unplanned work.

Norbert Preining: Deleting many files from an S3 bucket

So we found ourselves in the need to delete a considerable amount of files (around 500000, amounting to 1.6T) from an S3 bucket. With the list of files in hand my first shot was calling 
aws s3 rm s3://BUCKET/FILE
for each file. That wasn t the best idea I have to say, since first of all, it makes 500000 requests, and then it takes a looong time. And this command does not allow to pass in multiple files. Fortunately there is aws s3api delete-objects which takes a json input and can delete multiple files: 
aws 3api delete-objects --bucket BUCKET --delete ' "Objects": [   "Key" : "FILE1"  ,   "Key" : "FILE2"  ... ] '
That did help, and with a bit of magic from bash (mapfile which can read in lines from stdin in batches) and jq, at the end it was a business of some 20min or so: 
cat files-to-be-deleted    while mapfile -t -n 500 ary && (($ #ary[@] )); do
        objdef=$(printf '%s\n' "$ ary[@] "   jq -nR ' Objects: (reduce inputs as $line ([]; . + [ "Key":$line ])) ')
        aws s3api --no-cli-pager  delete-objects --bucket BUCKET --delete "$objdef"
done
This reads 500 files a time, and reformats it using jq into the proper json format: reduce inputs is a jq filter that iterates over the input lines and does a map/reduce step. In this case we use an empty array as start and add new key/filename pairs on the go. Finally, the whole bunch is send to AWS with the above API call. Puuuh, 500000 files and 1.6T less, in 20min.

28 October 2020

Kentaro Hayashi: lltsv 0.6.1-2: applied background color patch for readability

Original version of lltsv doesn't specify background color, so it is hard to recognize texts in dark background. I've fixed this issue by specifying background color explicitly. Here is the screenshot of fixed version. Yay! f:id:kenhys:20201028223848p:plain This issue was already forwarded as PR github.com

26 October 2020

Molly de Blanc: Digital Self

When we talk about the digital self, we are talking about the self as it exists within digital spaces. This holds differently for different people, as some of us prefer to live within an pseudonymous or anonymous identity online, divested from our physical selves, while others consider the digital a more holistic identity that extends from the physical. Your digital self is gestalt, in that it exists across whatever mediums, web sites, and services you use. These bits are pieces together to form a whole picture of what it means to be you, or some aspect of you. This may be carefully curated, or it may be an emergent property of who you are. The way your physical self has rights, so too does your digital self. Or, perhaps, it would be more accurate to say that your rights extend to your digital self. I do not personally consider that there is a separation between these selves when it comes to rights, as both are aspects of you and you have rights. I am explicitly not going to list what these rights are, because I have my own ideas about them and yours may differ. Instead, I will briefly talk about consent. I think it is essential that we genuinely consent to how others interact with us to maintain the sanctity of our selves. Consent is necessary to the protection and expression of our rights, as it ensures we are able to rely on our rights and creates a space where we are able to express our rights in comfort and safety. We may classically think of consent as it relates to sex and sexual consent: only we have the right to determine what happens to our bodies; no one else has the right to that determination. We are able to give sexual consent, and we are able to revoke it. Sexual consent, in order to be in good faith, must be requested and given from a place of openness and transparency. For this, we discuss with our partners the things about ourselves that may impact their decision to consent: we are sober; we are not ill; we are using (or not) protection as we agree is appropriate; we are making this decision because it is a thing we desire, rather than a thing we feel we ought to do or are being forced to do; as well as other topics. These things also all hold true for technology and the digital spaces in which we reside. Our digital autonomy is not the only thing at stake when we look at digital consent. The ways we interact in digital spaces impact our whole selves, and exploitation of our consent too impacts our whole selves. Private information appearing online can have material consequences it can directly lead to safety issues, like stalking or threats, and it can lead to a loss of psychic safety and have a chilling effect. These are in addition to the threats posed to digital safety and well being. Consent must be actively sought, what one is consenting to is transparent, and the potential consequences must be known and understood. In order to protect and empower the digital self, to treat everyone justly and with respect, we must hold the digital self be as sacrosanct as other aspects of the self and treat it accordingly.

24 October 2020

Sandro Tosi: Multiple git configurations depending on the repository path

For my work on Debian, i want to use my debian.org email address, while for my personal projects i want to use my gmail.com address.One way to change the user.email git config value is to git config --local in every repo, but that's tedious, error-prone and doesn't scale very well with many repositories (and the chances to forget to set the right one on a new repo are ~100%).The solution is to use the git-config ability to include extra configuration files, based on the repo path, by using includeIf: Content of ~/.gitconfig:
[user]
    name = Sandro Tosi
    email = <personal.address>@gmail.com

[includeIf "gitdir:~/deb/"]
    path = ~/.gitconfig-deb
Every time the git path is in ~/deb/ (which is where i have all Debian repos) the file ~/.gitconfig-deb will be included; its content:
[user]
    email = morph@debian.org
That results in my personal address being used on all repos not part of Debian, where i use my Debian email address. This approach can be extended to every other git configuration values.

23 October 2020

Enrico Zini: Hetzner build machine

This is part of a series of posts on compiling a custom version of Qt5 in order to develop for both amd64 and a Raspberry Pi. Building Qt5 takes a long time. The build server I was using had CPUs and RAM, but was very slow on I/O. I was very frustrated by that, and I started evaluating alternatives. I ended up setting up scripts to automatically provision a throwaway cloud server at Hetzner. Initial setup I got an API key from my customer's Hetzner account. I installed hcloud-cli, currently only in testing and unstable: 
apt install hcloud-cli
Then I configured hcloud with the API key: 
hcloud context create
Spin up I wrote a quick and dirty script to spin up a new machine, which grew a bit with little tweaks: 
#!/bin/sh
# Create the server
hcloud server create --name buildqt --ssh-key   --start-after-create \
                     --type cpx51 --image debian-10 --datacenter  
# Query server IP
IP="$(hcloud server describe buildqt -o json   jq -r .public_net.ipv4.ip)"
# Update ansible host file
echo "buildqt ansible_user=root ansible_host=$IP" > hosts
# Remove old host key
ssh-keygen -f ~/.ssh/known_hosts -R "$IP"
# Update login script
echo "#!/bin/sh" > login
echo "ssh root@$IP" >> login
chmod 0755 login
I picked a datacenter in the same location as where we have other servers, to get quicker data transfers. I like that CLI tools have JSON output that I can cleanly pick at with jq. Sadly, my ISP doesn't do IPv6 yet. Since the server just got regenerated, I remove a possibly cached host key. Provisioning the machine One git server I need is behind HTTP authentication. Here's a quick hack to pass the relevant .netrc credentials to ansible before provisioning: 
#!/usr/bin/python3
import subprocess
import netrc
import tempfile
import json
login, account, password = netrc.netrc().authenticators(" ")
with tempfile.NamedTemporaryFile(mode="wt", suffix=".json") as fd:
    json.dump( 
        "repo_user": login,
        "repo_password": password,
     , fd)
    fd.flush()
    subprocess.run([
        "ansible-playbook",
        "-i", "hosts",
        "-l", "buildqt",
        "--extra-vars", f"@ fd.name ",
        "provision.yml",
        ], check=True)
And here's the ansible playbook: 
#!/usr/bin/env ansible-playbook
- name: Install and configure buildqt
  hosts: all
  tasks:
   - name: Update apt cache
     apt:
        update_cache: yes
        cache_valid_time: 86400
   - name: Create build user
     user:
        name: build
        comment: QT5 Build User
        shell: /bin/bash
   - name: Create sources directory
     become: yes
     become_user: build
     file:
        path: ~/sources
        state: directory
        mode: 0755
   - name: Download sources
     become: yes
     become_user: build
     get_url:
        url: "https:// / item "
        dest: "~/sources/ item "
        mode: 0644
     with_items:
      - "qt-everywhere-src-5.15.1.tar.xz"
      - "qt-creator-enterprise-src-4.13.2.tar.gz"
   - name: Populate home directory
     become: yes
     become_user: build
     copy:
        src: build
        dest: ~/
        mode: preserve
   - name: Write .netrc
     become: yes
     become_user: build
     copy:
        dest: ~/.netrc
        mode: 0600
        content:  
           machine  
           login  repo_user 
           password  repo_password 
   - name: Write .screenrc
     become: yes
     become_user: build
     copy:
        dest: ~/.screenrc
        mode: 0644
        content:  
           hardstatus alwayslastline
           hardstatus string '% = cw %-Lw% = KW %50>%n%f* %t% = cw %+Lw%< % = kK %-=%D %Y-%m-%d %c% - '
           startup_message off
           defutf8 on
           defscrollback 10240
   - name: Install base packages
     apt:
        name: git,mc,ncdu,neovim,eatmydata,devscripts,equivs,screen
        state: present
   - name: Clone git repo
     become: yes
     become_user: build
     git:
        repo: https:// @ / .git
        dest: ~/ 
   - name: Copy Qt license
     become: yes
     become_user: build
     copy:
        src: qt-license.txt
        dest: ~/.qt-license
        mode: 0600
Now everything is ready for a 16 core, 32Gb ram build on SSD storage. Tear down When done: 
#!/bin/sh
hcloud server delete buildqt
The whole spin up plus provisioning takes around a minute, so I can do it when I start a work day, and take it down at the end. The build machine wasn't that expensive to begin with, and this way it will even be billed by the hour. A first try on a CPX51 machine has just built the full Qt5 Everywhere Enterprise including QtWebEngine and all its frills, for amd64, in under 1 hour and 40 minutes.

Molly de Blanc: Endorsements

Transparency is essential to trusting a technology. Through transparency we can understand what we re using and build trust. When we know what is actually going on, what processes are occurring and how it is made, we are able to decide whether interacting with it is something we actually want, and we re able to trust it and use it with confidence. This transparency could mean many things, though it most frequently refers to the technology itself: the code or, in the case of hardware, the designs. We could also apply it to the overall architecture of a system. We could think about the decision making, practices, and policies of whomever is designing and/or making the technology. These are all valuable in some of the same ways, including that they allow us to make a conscious choice about what we are supporting. When we choose to use a piece of technology, we are supporting those who produce it. This could be because we are directly paying for it, however our support is not limited to direct financial contributions. In some cases this is because of things hidden within a technology: tracking mechanisms or backdoors that could allow companies or governments access to what we re doing. When creating different types of files on a computer, these files can contain metadata that says what software was used to make it. This is an implicit endorsement, and you can also explicitly endorse a technology by talking about that or how you use it. In this, you have a right (not just a duty) to be aware of what you re supporting. This includes, for example, organizational practices and whether a given company relies on abusive labor policies, indentured servitude, or slave labor.
Endorsements inspire others to choose a piece of technology. Most of my technology is something I investigate purely for functionality, and the pieces I investigate are based on what people I know use. The people I trust in these cases are more inclined than most to do this kind of research, to perform technical interrogations, and to be aware of what producers of technology are up to. This is how technology spreads and becomes common or the standard choice. In one sense, we all have the responsibility (one I am shirking) to investigate our technologies before we choose them. However, we must acknowledge that not everyone has the resources for this the time, the skills, the knowledge, and therein endorsements become even more important to recognize. Those producing a technology have the responsibility of making all of these angles something one could investigate. Understanding cannot only be the realm of experts. It should not require an extensive background in research and investigative journalism to find out whether a company punishes employees who try to unionize or pay non-living wages. Instead, these must be easy activities to carry out. It should be standard for a company (or other technology producer) to be open and share with people using their technology what makes them function. It should be considered shameful and shady to not do so. Not only does this empower those making choices about what technologies to use, but it empowers others down the line, who rely on those choices. It also respects the people involved in the processes of making these technologies. By acknowledging their role in bringing our tools to life, we are respecting their labor. By holding companies accountable for their practices and policies, we are respecting their lives.

22 October 2020

Vincent Fourmond: QSoas tips and tricks: generating smooth curves from a fit

Often, one would want to generate smooth data from a fit over a small number of data points. For an example, take the data in the following file. It contains (fake) experimental data points that obey to Michaelis-Menten kinetics: $$v = \frac v_m 1 + K_m/s $$ in which \(v\) is the measured rate (the y values of the data), \(s\) the concentration of substrate (the x values of the data), \(v_m\) the maximal rate and \(K_m\) the Michaelis constant. To fit this equation to the data, just use the fit-arb fit: 
QSoas> l michaelis.dat
QSoas> fit-arb vm/(1+km/x)
After running the fit, the window should look like this:
Now, with the fit, we have reasonable values for \(v_m\) (vm) and \(K_m\) (km). But, for publication, one would want to generate "smooth" curve going through the lines... Saving the curve from "Data.../Save all" doesn't help, since the data has as many points as the original data and looks very "jaggy" (like on the screenshot above)... So one needs a curve with more data points. Maybe the most natural solution is simply to use generate-buffer together with apply-formula using the formula and the values of km and vm obtained from the fit, like: 
QSoas> generate-buffer 0 20
QSoas> apply-formula y=3.51742/(1+3.69767/x)
By default, generate-buffer generate 1000 evenly spaced x values, but you can change their number using the /samples option. The two above commands can be combined to just one call to generate-buffer: 
QSoas> generate-buffer 0 20 3.51742/(1+3.69767/x)
This works, but it is quite cumbersome and it is not going to work well for complex formulas or the results of differential equations or kinetic systems... This is why to each fit- command corresponds a sim- command that computes the result of the fit using a "saved parameters" file (here, michaelis.params, but you can also save it yourself) and buffers as "models" for X values: 
QSoas> generate-buffer 0 20
QSoas> sim-arb vm/(1+km/x) michaelis.params 0
This strategy works with every single fit ! As an added benefit, you even get the fit parameters as meta-data, which are displayed by the show command: 
QSoas> show 0
Dataset generated_fit_arb.dat: 2 cols, 1000 rows, 1 segments, #0
Flags: 
Meta-data:	commands =	 sim-arb vm/(1+km/x) michaelis.params 0	fit =	 arb (formula: vm/(1+km/x))	km =	 3.69767
	vm =	 3.5174
They also get saved as comments if you save the data. Important note: the sim-arb command will be available only in the 3.0 release, although you can already enjoy it if you use the github version. About QSoasQSoas is a powerful open source data analysis program that focuses on flexibility and powerful fitting capacities. It is released under the GNU General Public License. It is described in Fourmond, Anal. Chem., 2016, 88 (10), pp 5050 5052. Current version is 2.2. You can download its source code and compile it yourself or buy precompiled versions for MacOS and Windows there.

20 October 2020

Dirk Eddelbuettel: RcppArmadillo 0.10.1.0.0

armadillo image Armadillo is a powerful and expressive C++ template library for linear algebra aiming towards a good balance between speed and ease of use with a syntax deliberately close to a Matlab. RcppArmadillo integrates this library with the R environment and language and is widely used by (currently) 786 other packages on CRAN. A little while ago, Conrad released version 10.1.0 of Armadillo, a a new major release. As before, given his initial heads-up we ran two full reverse-depends checks, and as a consequence contacted four packages authors (two by email, two via PR) about a miniscule required change (as Armadillo now defaults to C++11, an old existing setting of avoiding C++11 lead to an error). Our thanks to those who promptly update their packages truly appreciated. As it turns out, Conrad also softened the error by the time the release ran around. But despite our best efforts, the release was delayed considerably by CRAN. We had made several Windows test builds but luck had it that on the uploaded package CRAN got itself a (completely spurious segfault which can happen on a busy machine building machine things at once). Sadly it took three or four days for CRAN to reply our email. After which it took another number of days for them to ponder the behaviour of a few new deprecated messaged tickled by at the most ten or so (out of 786) packages. Oh well. So here we are, eleven days after I emailed the rcpp-devel list about the new package being on CRAN but possibly delayed (due to that seg.fault). But during all that time the package was of course available via the Rcpp drat. The changes in this release are summarized below as usual and are mostly upstream along with an improved Travis CI setup due to the aforementioned use of the bspm package for binaries at Travis.

Changes in RcppArmadillo version 0.10.1.0.0 (2020-10-09)
  • Upgraded to Armadillo release 10.1.0 (Orchid Ambush)
    • C++11 is now the minimum required C++ standard
    • faster handling of compound expressions by trimatu() and trimatl()
    • faster sparse matrix addition, subtraction and element-wise multiplication
    • expanded sparse submatrix views to handle the non-contiguous form of X.cols(vector_of_column_indices)
    • expanded eigs_sym() and eigs_gen() with optional fine-grained parameters (subspace dimension, number of iterations, eigenvalues closest to specified value)
    • deprecated form of reshape() removed from Cube and SpMat classes
    • ignore and warn on use of the ARMA_DONT_USE_CXX11 macro
  • Switch Travis CI testing to focal and BSPM

Courtesy of my CRANberries, there is a diffstat report relative to previous release. More detailed information is on the RcppArmadillo page. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page. If you like this or other open-source work I do, you can now sponsor me at GitHub. For the first year, GitHub will match your contributions.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

19 October 2020

Antoine Beaupr : SSH 2FA with Google Authenticator and Yubikey

About a lifetime ago (5 years), I wrote a tutorial on how to configure my Yubikey for OpenPGP signing, SSH authentication and SSH 2FA. In there, I used the libpam-oath PAM plugin for authentication, but it turns out that had too many problems: users couldn't edit their own 2FA tokens and I had to patch it to avoid forcing 2FA on all users. The latter was merged in the Debian package, but never upstream, and the former was never fixed at all. So I started looking at alternatives and found the Google Authenticator libpam plugin. A priori, it's designed to work with phones and the Google Authenticator app, but there's no reason why it shouldn't work with hardware tokens like the Yubikey. Both use the standard HOTP protocol so it should "just work". After some fiddling, it turns out I was right and you can authenticate with a Yubikey over SSH. Here's that procedure so you don't have to second-guess it yourself.

Installation On Debian, the PAM module is shipped in the google-authenticator source package: 
apt install libpam-google-authenticator
Then you need to add the module in your PAM stack somewhere. Since I only use it for SSH, I added this line on top of /etc/pam.d/sshd: 
auth required pam_google_authenticator.so nullok
I also used no_increment_hotp debug while debugging to avoid having to renew the token all the time and have more information about failures in the logs. Then reload ssh (not sure that's actually necessary): 
service ssh reload

Creating or replacing tokens To create a new key, run this command on the server: 
google-authenticator -c
This will prompt you for a bunch of questions. To get them all right, I prefer to just call the right ones on the commandline directly: 
google-authenticator --counter-based --qr-mode=NONE --rate-limit=1 --rate-time=30 --emergency-codes=1 --window-size=3
Those are actually the defaults, if my memory serves me right, except for the --qr-mode and --emergency-codes (which can't be disabled so I only print one). I disable the QR code display because I won't be using the codes on my phone, but you would obviously keep it if you want to use the app.

Converting to a Yubikey-compatible secret Unfortunately, the encoding (base32) produced by the google-authenticator command is not compatible with the token expected by the ykpersonalize command used to configure the Yubikey (base16 AKA "hexadecimal", with a fixed 20 bytes length). So you need a way to convert between the two. I wrote a program called oath-convert which basically does this: 
read base32
add padding
convert to hex
print
Or, in Python: 
def convert_b32_b16(data_b32):
    remainder = len(data_b32) % 8
    if remainder > 0:
        # XXX: assume 6 chars are missing, the actual padding may vary:
        # https://tools.ietf.org/html/rfc3548#section-5
        data_b32 += "======"
    data_b16 = base64.b32decode(data_b32)
    if len(data_b16) < 20:
        # pad to 20 bytes
        data_b16 += b"\x00" * (20 - len(data_b16))
    return binascii.hexlify(data_b16).decode("ascii")
Note that the code assumes a certain token length and will not work correctly for other sizes. To use the program, simply call it with: 
head -1 .google_authenticator   oath-convert
Then you paste the output in the prompt: 
$ ykpersonalize -1 -o oath-hotp -o append-cr -a
Firmware version 3.4.3 Touch level 1541 Program sequence 2
 HMAC key, 20 bytes (40 characters hex) : [SECRET GOES HERE]
Configuration data to be written to key configuration 1:
fixed: m:
uid: n/a
key: h:[SECRET REDACTED]
acc_code: h:000000000000
OATH IMF: h:0
ticket_flags: APPEND_CR OATH_HOTP
config_flags: 
extended_flags: 
Commit? (y/n) [n]: y
Note that you must NOT pass the -o oath-hotp8 parameter to the ykpersonalize commandline, which we used to do in the Yubikey howto. That is because Google Authenticator tokens are shorter: it's less secure, but it's an acceptable tradeoff considering the plugin is actually maintained. There's actually a feature request to support 8-digit codes so that limitation might eventually be fixed as well. Thanks to the Google Authenticator people and Yubikey people for their support in establishing this procedure.

17 October 2020

Dirk Eddelbuettel: digest 0.6.26: Blake3 and Tuning

And a new version of digest is now on CRAN will go to Debian shortly. digest creates hash digests of arbitrary R objects (using the md5, sha-1, sha-256, sha-512, crc32, xxhash32, xxhash64, murmur32, spookyhash, and blake3 algorithms) permitting easy comparison of R language objects. It is a fairly widely-used package (currently listed at 896k monthly downloads, 279 direct reverse dependencies and 8057 indirect reverse dependencies, or just under half of CRAN) as many tasks may involve caching of objects for which it provides convenient general-purpose hash key generation. This release brings two nice contributed updates. Dirk Schumacher added support for blake3 (though we could probably push this a little harder for performance, help welcome). Winston Chang benchmarked and tuned some of the key base R parts of the package. Last but not least I flipped the vignette to the lovely minidown, updated the Travis CI setup using bspm (as previously blogged about in r4 #30), and added a package website using Matertial for MkDocs. My CRANberries provides the usual summary of changes to the previous version. For questions or comments use the issue tracker off the GitHub repo. If you like this or other open-source work I do, you can now sponsor me at GitHub. For the first year, GitHub will match your contributions.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

15 October 2020

Rapha&#235;l Hertzog: Freexian s report about Debian Long Term Support, September 2020

A Debian LTS logo Like each month, here comes a report about the work of paid contributors to Debian LTS. Individual reports In September, 208.25 work hours have been dispatched among 13 paid contributors. Their reports are available: Evolution of the situation September was a regular LTS month with an IRC meeting. The security tracker currently lists 45 packages with a known CVE and the dla-needed.txt file has 48 packages needing an update. Thanks to our sponsors Sponsors that joined recently are in bold.

No comment Liked this article? Click here. My blog is Flattr-enabled.

Dirk Eddelbuettel: dang 0.0.12: Two new functions

A new release of the dang package is now on CRAN, roughly one year after the last release. The dang package regroups a few functions of mine that had no other home as for example lsos() from a StackOverflow question from 2009 (!!) is one, this overbought/oversold price band plotter from an older blog post is another. More recently added were helpers for data.table to xts conversion and a git repo root finder. This release adds two functions. One was mentioned just days ago in a tweet by Nathan and is a reworked version of something Colin tweeted about a few weeks ago: a little data wrangling off the kewl rtweet to find maximally spammy accounts per search topic. In other words those who include more than N hashtags for given search term. The other is something I, if memory serves, picked up a while back on one of the lists: a base R function to identify non-ASCII characters in a file. It is a C function that is not directly exported by and hence no accessible, so we put it here (with credits, of course). I mentioned it yesterday when announcing tidyCpp as I this C function was the starting point for the new tidyCpp wrapper around some C API of R functions. The (very short) NEWS entry follows.

Changes in version 0.0.12 (2020-10-14)
  • New functions muteTweets and checkPackageAsciiCode.
  • Updated CI setup.

Courtesy of CRANberries, there is a comparison to the previous release. For questions or comments use the issue tracker off the GitHub repo. If you like this or other open-source work I do, you can now sponsor me at GitHub. For the first year, GitHub will match your contributions.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

14 October 2020

Michael Stapelberg: distri: a Linux distribution to research fast package management

Over the last year or so I have worked on a research linux distribution in my spare time. It s not a distribution for researchers (like Scientific Linux), but my personal playground project to research linux distribution development, i.e. try out fresh ideas. This article focuses on the package format and its advantages, but there is more to distri, which I will cover in upcoming blog posts.

Motivation I was a Debian Developer for the 7 years from 2012 to 2019, but using the distribution often left me frustrated, ultimately resulting in me winding down my Debian work. Frequently, I was noticing a large gap between the actual speed of an operation (e.g. doing an update) and the possible speed based on back of the envelope calculations. I wrote more about this in my blog post Package managers are slow . To me, this observation means that either there is potential to optimize the package manager itself (e.g. apt), or what the system does is just too complex. While I remember seeing some low-hanging fruit , through my work on distri, I wanted to explore whether all the complexity we currently have in Linux distributions such as Debian or Fedora is inherent to the problem space. I have completed enough of the experiment to conclude that the complexity is not inherent: I can build a Linux distribution for general-enough purposes which is much less complex than existing ones. Those were low-hanging fruit from a user perspective. I m not saying that fixing them is easy in the technical sense; I know too little about apt s code base to make such a statement.

Key idea: packages are images, not archives One key idea is to switch from using archives to using images for package contents. Common package managers such as dpkg(1) use tar(1) archives with various compression algorithms. distri uses SquashFS images, a comparatively simple file system image format that I happen to be familiar with from my work on the gokrazy Raspberry Pi 3 Go platform. This idea is not novel: AppImage and snappy also use images, but only for individual, self-contained applications. distri however uses images for distribution packages with dependencies. In particular, there is no duplication of shared libraries in distri. A nice side effect of using read-only image files is that applications are immutable and can hence not be broken by accidental (or malicious!) modification.

Key idea: separate hierarchies Package contents are made available under a fully-qualified path. E.g., all files provided by package zsh-amd64-5.6.2-3 are available under /ro/zsh-amd64-5.6.2-3. The mountpoint /ro stands for read-only, which is short yet descriptive. Perhaps surprisingly, building software with custom prefix values of e.g. /ro/zsh-amd64-5.6.2-3 is widely supported, thanks to:
  1. Linux distributions, which build software with prefix set to /usr, whereas FreeBSD (and the autotools default), which build with prefix set to /usr/local.
  2. Enthusiast users in corporate or research environments, who install software into their home directories.
Because using a custom prefix is a common scenario, upstream awareness for prefix-correctness is generally high, and the rarely required patch will be quickly accepted.

Key idea: exchange directories Software packages often exchange data by placing or locating files in well-known directories. Here are just a few examples:
  • gcc(1) locates the libusb(3) headers via /usr/include
  • man(1) locates the nginx(1) manpage via /usr/share/man.
  • zsh(1) locates executable programs via PATH components such as /bin
In distri, these locations are called exchange directories and are provided via FUSE in /ro. Exchange directories come in two different flavors:
  1. global. The exchange directory, e.g. /ro/share, provides the union of the share sub directory of all packages in the package store.
    Global exchange directories are largely used for compatibility, see below.
  2. per-package. Useful for tight coupling: e.g. irssi(1) does not provide any ABI guarantees, so plugins such as irssi-robustirc can declare that they want e.g. /ro/irssi-amd64-1.1.1-1/out/lib/irssi/modules to be a per-package exchange directory and contain files from their lib/irssi/modules.

Search paths sometimes need to be fixed Programs which use exchange directories sometimes use search paths to access multiple exchange directories. In fact, the examples above were taken from gcc(1) s INCLUDEPATH, man(1) s MANPATH and zsh(1) s PATH. These are prominent ones, but more examples are easy to find: zsh(1) loads completion functions from its FPATH. Some search path values are derived from --datadir=/ro/share and require no further attention, but others might derive from e.g. --prefix=/ro/zsh-amd64-5.6.2-3/out and need to be pointed to an exchange directory via a specific command line flag.

FHS compatibility Global exchange directories are used to make distri provide enough of the Filesystem Hierarchy Standard (FHS) that third-party software largely just works. This includes a C development environment. I successfully ran a few programs from their binary packages such as Google Chrome, Spotify, or Microsoft s Visual Studio Code.

Fast package manager I previously wrote about how Linux distribution package managers are too slow. distri s package manager is extremely fast. Its main bottleneck is typically the network link, even at high speed links (I tested with a 100 Gbps link). Its speed comes largely from an architecture which allows the package manager to do less work. Specifically:
  1. Package images can be added atomically to the package store, so we can safely skip fsync(2) . Corruption will be cleaned up automatically, and durability is not important: if an interactive installation is interrupted, the user can just repeat it, as it will be fresh on their mind.
  2. Because all packages are co-installable thanks to separate hierarchies, there are no conflicts at the package store level, and no dependency resolution (an optimization problem requiring SAT solving) is required at all.
    In exchange directories, we resolve conflicts by selecting the package with the highest monotonically increasing distri revision number.
  3. distri proves that we can build a useful Linux distribution entirely without hooks and triggers. Not having to serialize hook execution allows us to download packages into the package store with maximum concurrency.
  4. Because we are using images instead of archives, we do not need to unpack anything. This means installing a package is really just writing its package image and metadata to the package store. Sequential writes are typically the fastest kind of storage usage pattern.
Fast installation also make other use-cases more bearable, such as creating disk images, be it for testing them in qemu(1) , booting them on real hardware from a USB drive, or for cloud providers such as Google Cloud.

Fast package builder Contrary to how distribution package builders are usually implemented, the distri package builder does not actually install any packages into the build environment. Instead, distri makes available a filtered view of the package store (only declared dependencies are available) at /ro in the build environment. This means that even for large dependency trees, setting up a build environment happens in a fraction of a second! Such a low latency really makes a difference in how comfortable it is to iterate on distribution packages.

Package stores In distri, package images are installed from a remote package store into the local system package store /roimg, which backs the /ro mount. A package store is implemented as a directory of package images and their associated metadata files. You can easily make available a package store by using distri export. To provide a mirror for your local network, you can periodically distri update from the package store you want to mirror, and then distri export your local copy. Special tooling (e.g. debmirror in Debian) is not required because distri install is atomic (and update uses install). Producing derivatives is easy: just add your own packages to a copy of the package store. The package store is intentionally kept simple to manage and distribute. Its files could be exchanged via peer-to-peer file systems, or synchronized from an offline medium.

distri s first release distri works well enough to demonstrate the ideas explained above. I have branched this state into branch jackherer, distri s first release code name. This way, I can keep experimenting in the distri repository without breaking your installation. From the branch contents, our autobuilder creates:
  1. disk images, which
  2. a package repository. Installations can pick up new packages with distri update.
  3. documentation for the release.
The project website can be found at https://distr1.org. The website is just the README for now, but we can improve that later. The repository can be found at https://github.com/distr1/distri

Project outlook Right now, distri is mainly a vehicle for my spare-time Linux distribution research. I don t recommend anyone use distri for anything but research, and there are no medium-term plans of that changing. At the very least, please contact me before basing anything serious on distri so that we can talk about limitations and expectations. I expect the distri project to live for as long as I have blog posts to publish, and we ll see what happens afterwards. Note that this is a hobby for me: I will continue to explore, at my own pace, parts that I find interesting. My hope is that established distributions might get a useful idea or two from distri.

There s more to come: subscribe to the distri feed I don t want to make this post too long, but there is much more! Please subscribe to the following URL in your feed reader to get all posts about distri: https://michael.stapelberg.ch/posts/tags/distri/feed.xml Next in my queue are articles about hermetic packages and good package maintainer experience (including declarative packaging).

Feedback or questions? I d love to discuss these ideas in case you re interested! Please send feedback to the distri mailing list so that everyone can participate!

Dirk Eddelbuettel: tidyCpp 0.0.1: New package

A new package arrived on CRAN a few days ago. It offers a few headers files which wrap (parts) of the C API for R, but in a form that may be a little easier to use for C++ programmers. I have always liked how in Rcpp we offer good parts of the standalone R Math library in a namespace R::. While working recently with a particular C routine (for checking non-ASCII characters that will be part of the next version of the dang package which collecting various goodies in one place), I realized there may be value in collecting a few more such wrappers. So I started a few simple ones starting from simple examples. Currently we have five headers defines.h, globals.h, internals.h, math.h, and shield.h. The first four each correpond to an R header file of the same or similar name, and the last one brings a simple yet effective alternative to PROTECT and UNPROTECT from Rcpp (in a slightly simplified way). None of the headers are complete , for internals.h in particular a lot more could be added (as I noticed today when experimenting with another source file that may be converted). All of the headers can be accessed with a simple #include <tidyCpp> (which, following another C++ convention, does not have a .h or .hpp suffix). And a the package ships these headers, packages desiring to use them only need LinkingTo: tidyCpp. As usage examples, we (right now) have four files in the snippets/ directory of the package. Two of these, convolveExample.cpp and dimnamesExample.cpp both illustrate how one could change example code from Writing R Extensions. Then there are also a very simple defineExample.cpp and a shieldExample.cpp illustrating how much easier Shield() is compared to PROTECT and UNPROTECT. Finally, there is a nice vignette discussing the package motivation with two detailed side-by-side before and after examples that are the aforementioned convolution and dimnames examples. Over time, I expect to add more definitions and wrappers. Feedback would be welcome it seems to hit a nerve already as it currently has more stars than commits even though (prior to this post) I had yet to tweet or blog about it. Please post comments and suggestions at the GitHub repo. If you like this or other open-source work I do, you can now sponsor me at GitHub. For the first year, GitHub will match your contributions.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

12 October 2020

Russell Coker: First Attempt at Gnocchi-Statsd

I ve been investigating the options for tracking system statistics to diagnose performance problems. The idea is to track all sorts of data about the system (network use, disk IO, CPU, etc) and look for correlations at times of performance problems. DataDog is pretty good for this but expensive, it s apparently based on or inspired by the Etsy Statsd. It s claimed that the gnocchi-statsd is the best implementation of the protoco used by the Etsy Statsd, so I decided to install that. I use Debian/Buster for this as that s what I m using for the hardware that runs KVM VMs. Here is what I did: 
# it depends on a local MySQL database
apt -y install mariadb-server mariadb-client
# install the basic packages for gnocchi
apt -y install gnocchi-common python3-gnocchiclient gnocchi-statsd uuid
In the Debconf prompts I told it to setup a database and not to manage keystone_authtoken with debconf (because I m not doing a full OpenStack installation). This gave a non-working configuration as it didn t configure the MySQL database for the [indexer] section and the sqlite database that was configured didn t work for unknown reasons. I filed Debian bug #971996 about this [1]. To get this working you need to edit /etc/gnocchi/gnocchi.conf and change the url line in the [indexer] section to something like the following (where the password is taken from the [database] section). 
url = mysql+pymysql://gnocchi-common:PASS@localhost:3306/gnocchidb
To get the statsd interface going you have to install the gnocchi-statsd package and edit /etc/gnocchi/gnocchi.conf to put a UUID in the resource_id field (the Debian package uuid is good for this). I filed Debian bug #972092 requesting that the UUID be set by default on install [2]. Here s an official page about how to operate Gnocchi [3]. The main thing I got from this was that the following commands need to be run from the command-line (I ran them as root in a VM for test purposes but would do so with minimum privs for a real deployment). 
gnocchi-api
gnocchi-metricd
To communicate with Gnocchi you need the gnocchi-api program running, which uses the uwsgi program to provide the web interface by default. It seems that this was written for a version of uwsgi different than the one in Buster. I filed Debian bug #972087 with a patch to make it work with uwsgi [4]. Note that I didn t get to the stage of an end to end test, I just got it to basically run without error. After getting gnocchi-api running (in a terminal not as a daemon as Debian doesn t seem to have a service file for it), I ran the client program gnocchi and then gave it the status command which failed (presumably due to the metrics daemon not running), but at least indicated that the client and the API could communicate. Then I ran the gnocchi-metricd and got the following error: 
2020-10-12 14:59:30,491 [9037] ERROR    gnocchi.cli.metricd: Unexpected error during processing job
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/gnocchi/cli/metricd.py", line 87, in run
    self._run_job()
  File "/usr/lib/python3/dist-packages/gnocchi/cli/metricd.py", line 248, in _run_job
    self.coord.update_capabilities(self.GROUP_ID, self.store.statistics)
  File "/usr/lib/python3/dist-packages/tooz/coordination.py", line 592, in update_capabilities
    raise tooz.NotImplemented
tooz.NotImplemented
At this stage I ve had enough of gnocchi. I ll give the Etsy Statsd a go next. Update Thomas has responded to this post [5]. At this stage I m not really interested in giving Gnocchi another go. There s still the issue of the indexer database which should be different from the main database somehow and sqlite (the config file default) doesn t work. I expect that if I was to persist with Gnocchi I would encounter more poorly described error messages from the code which either don t have Google hits when I search for them or have Google hits to unanswered questions from 5+ years ago. The Gnocchi systemd config files are in different packages to the programs, this confused me and I thought that there weren t any systemd service files. I had expected that installing a package with a daemon binary would also get the systemd unit file to match. The cluster features of Gnocchi are probably really good if you need that sort of thing. But if you have a small instance (EG a single VM server) then it s not needed. Also one of the original design ideas of the Etsy Statsd was that UDP was used because data could just be dropped if there was a problem. I think for many situations the same concept could apply to the entire stats service. If the other statsd programs don t do what I need then I may give Gnocchi another go.
Related posts:
  1. Magic entries for BTRFS and Software RAID I ve just discovered that the magic database for the file(1)...
  2. Bizarre No space left on device error from Xen What should have been a routine remove DIMMs and run...
  3. A Strange Attempt to Trick Google I just noticed that my post about LED Headlights in...

Russ Allbery: Review: Hand to Mouth

Review: Hand to Mouth, by Linda Tirado
Publisher: G.P. Putnam's Sons
Copyright: October 2014
ISBN: 0-698-17528-X
Format: Kindle
Pages: 194
The first time Linda Tirado came to the viral attention of the Internet was in 2013 when she responded to a forum question: "Why do poor people do things that seem so self-destructive?" Here are some excerpts from her virally popular five-page response, which is included in the first chapter:
I know how to cook. I had to take Home Ec. to graduate high school. Most people on my level didn't. Broccoli is intimidating. You have to have a working stove, and pots, and spices, and you'll have to do the dishes no matter how tired you are or they'll attract bugs. It is a huge new skill for a lot of people. That's not great, but it's true. And if you fuck it up, you could make your family sick. We have learned not to try too hard to be middle class. It never works out well and always makes you feel worse for having tried and failed yet again. Better not to try. It makes more sense to get food that you know will be palatable and cheap and that keeps well. Junk food is a pleasure that we are allowed to have; why would we give that up? We have very few of them.
and
I smoke. It's expensive. It's also the best option. You see, I am always, always exhausted. It's a stimulant. When I am too tired to walk one more step, I can smoke and go for another hour. When I am enraged and beaten down and incapable of accomplishing one more thing, I can smoke and I feel a little better, just for a minute. It is the only relaxation I am allowed. It is not a good decision, but it is the only one that I have access to. It is the only thing I have found that keeps me from collapsing or exploding.
This book is an expansion on that essay. It's an entry in a growing genre of examinations of what it means to be poor in the United States in the 21st century. Unlike most of those examinations, it isn't written by an outsider performing essentially anthropological field work. It's one of the rare books written by someone who is herself poor and had the combination of skill and viral fame required to get an opportunity to talk about it in her own words.
I haven't had it worse than anyone else, and actually, that's kind of the point. This is just what life is for roughly a third of the country. We all handle it in our own ways, but we all work in the same jobs, live in the same places, feel the same sense of never quite catching up. We're not any happier about the exploding welfare rolls than anyone else is, believe me. It's not like everyone grows up and dreams of working two essentially meaningless part-time jobs while collecting food stamps. It's just that there aren't many other options for a lot of people.
I didn't find this book back in 2014 when it was published. I found it in 2020 during Tirado's second round of Internet fame: when the police shot out her eye with "non-lethal" rounds while she was covering the George Floyd protests as a photojournalist. In characteristic fashion, she subsequently reached out to the other people who had been blinded by the police, used her temporary fame to organize crowdfunded support for others, and is planning on having "try again" tattooed over the scar. That will give you a feel for the style of this book. Tirado is blunt, opinionated, honest, and full speed ahead. It feels weird to call this book delightful since it's fundamentally about the degree to which the United States is failing a huge group of its citizens and making their lives miserable, but there is something so refreshing and clear-headed about Tirado's willingness to tell you the straight truth about her life. It's empathy delivered with the subtlety of a brick, but also with about as much self-pity as a brick. Tirado is not interested in making you feel sorry for her; she's interested in you paying attention.
I don't get much of my own time, and I am vicious about protecting it. For the most part, I am paid to pretend that I am inhuman, paid to cater to both the reasonable and unreasonable demands of the general public. So when I'm off work, feel free to go fuck yourself. The times that I am off work, awake, and not taking care of life's details are few and far between. It's the only time I have any autonomy. I do not choose to waste that precious time worrying about how you feel. Worrying about you is something they pay me for; I don't work for free.
If you've read other books on this topic (Emily Guendelsberger's On the Clock is still the best of those I've read), you probably won't get many new facts from Hand to Mouth. I think this book is less important for the policy specifics than it is for who is writing it (someone who is living that life and can be honest about it) and the depth of emotional specifics that Tirado brings to the description. If you have never been poor, you will learn the details of what life is like, but more significantly you'll get a feel for how Tirado feels about it, and while this is one individual perspective (as Tirado stresses, including the fact that, as a white person, there are other aspects of poverty she's not experienced), I think that perspective is incredibly valuable. That said, Hand to Mouth provides even more reinforcement of the importance of universal medical care, the absurdity of not including dental care in even some of the more progressive policy proposals, and the difficulties in the way of universal medical care even if we solve the basic coverage problem. Tirado has significant dental problems due to unrepaired damage from a car accident, and her account reinforces my belief that we woefully underestimate how important good dental care is to quality of life. But providing universal insurance or access is only the start of the problem.
There is a price point for good health in America, and I have rarely been able to meet it. I choose not to pursue treatment if it will cost me more than it will gain me, and my cost-benefit is done in more than dollars. I have to think of whether I can afford any potential treatment emotionally, financially, and timewise. I have to sort out whether I can afford to change my life enough to make any treatment worth it I've been told by more than one therapist that I'd be fine if I simply reduced the amount of stress in my life. It's true, albeit unhelpful. Doctors are fans of telling you to sleep and eat properly, as though that were a thing one can simply do.
That excerpt also illustrates one of the best qualities of this book. So much writing about "the poor" treats them as an abstract problem that the implicitly not-poor audience needs to solve, and this leads rather directly to the endless moralizing as "we" attempt to solve that problem by telling poor people what they need to do. Tirado is unremitting in fighting for her own agency. She has a shitty set of options, but within those options she makes her own decisions. She wants better options and more space in which to choose them, which I think is a much more productive way to frame the moral argument than the endless hand-wringing over how to help "those poor people." This is so much of why I support universal basic income. Just give people money. It's not all of the solution UBI doesn't solve the problem of universal medical care, and we desperately need to find a way to make work less awful but it's the most effective thing we can do immediately. Poor people are, if anything, much better at making consequential financial decisions than rich people because they have so much more practice. Bad decisions are less often due to bad decision-making than bad options and the balancing of objectives that those of us who are not poor don't understand. Hand to Mouth is short, clear, refreshing, bracing, and, as you might have noticed, very quotable. I think there are other books in this genre that offer more breadth or policy insight, but none that have the same feel of someone cutting through the bullshit of lazy beliefs and laying down some truth. If any of the above excerpts sound like the sort of book you would enjoy reading, pick this one up. Rating: 8 out of 10

Next.

Previous.