Search Results: "nate"

5 January 2021

Ben Hutchings: Debian LTS work, December 2020

I was assigned 16 hours of work by Freexian's Debian LTS initiative and carried over 9 hours from earlier months. I worked 16.5 hours this month, so I will carry over 8.5 hours to January. (Updated: corrected number of hours worked.) I updated linux-4.19 to include the changes in the Debian 10.7 point release, uploaded the package, and issued DLA-2483-1 for this. I picked some regression fixes from the Linux 4.9 stable branch to the linux package, and uploaded the package. This unfortunately failed to build on arm64 due to some upstream changes uncovering an old bug, so I made a second upload fixing that. I issued DLA-2494-1 for this. I updated the linux packaging branch for stretch to Linux 4.9.249, but haven't made another package upload yet.

4 January 2021

John Goerzen: More Topics on Store-And-Forward (Possibly Airgapped) ZFS and Non-ZFS Backups with NNCP

Note: this is another article in my series on asynchronous communication in Linux with UUCP and NNCP. In my previous post, I introduced a way to use ZFS backups over NNCP. In this post, I ll expand on that and also explore non-ZFS backups. Use of nncp-file instead of nncp-exec The previous example used nncp-exec (like UUCP s uux), which lets you pipe stdin in, then queues up a request to run a given command with that input on a remote. I discussed that NNCP doesn t guarantee order of execution, but that for the ZFS use case, that was fine since zfs receive would just fail (causing NNCP to try again later). At present, nncp-exec stores the data piped to it in RAM before generating the outbound packet (the author plans to fix this shortly). That made it unusable for some of my backups, so I set it up another way: with nncp-file, the tool to transfer files to a remote machine. A cron job then picks them up and processes them. On the machine being backed up, we have to find a way to encode the dataset to be received. I chose to do that as part of the filename, so the updated simplesnap-queue could look like this: 
#!/bin/bash
set -e
set -o pipefail
DEST=" echo $1   sed 's,^tank/simplesnap/,,' "
FILE="bakfsfmt2- date "+%s.%N".$$ _ echo "$DEST"   sed 's,/,@,g' "
echo "Processing $DEST to $FILE" >&2
# stdin piped to this
zstd -8 - \
    gpg --compress-algo none --cipher-algo AES256 -e -r 012345...  \
    su nncp -c "/usr/local/nncp/bin/nncp-file -nice B -noprogress - 'backupsvr:$FILE'" >&2
echo "Queued $DEST to $FILE" >&2
I ve added compression and encryption here as well; more on that below. On the backup server, we would define a different incoming directory for each node in nncp.hjson. For instance: 
host1:  
...
   incoming: "/var/local/nncp-bakcups-incoming/host1"
 
host2:  
...
   incoming: "/var/local/nncp-backups-incoming/host2"
I ll present the scanning script in a bit. Offsite Backup Rotation Most of the time, you don t want just a single drive to store the backups. You d like to have a set. At minimum, one wouldn t be plugged in so lightning wouldn t ruin all your backups. But maybe you d store a second drive at some other location you have access to (friend s house, bank box, etc.) There are several ways you could solve this: The third option can be helped with NNCP, too. One way is to create separate NNCP installations for each of the drives that you store data on. Then, whenever one is plugged in, the appropriate NNCP config will be loaded and appropriate packets received and processed. The neighbor machine the spooler would just store up packets for the offsite drive until it comes back onsite (or, perhaps, your airgapped USB transport would do this). Then when it s back onsite, all the queued up ZFS sends get replayed and the backups replicated. Now, how might you handle this with NNCP? The simple way would be to have each system generating backups send them to two destinations. For instance: 
zstd -8 -   gpg --compress-algo none --cipher-algo AES256 -e -r 07D5794CD900FAF1D30B03AC3D13151E5039C9D5 \
    tee >(su nncp -c "/usr/local/nncp/bin/nncp-file -nice B+5 -noprogress - 'backupdisk1:$FILE'") \
        >(su nncp -c "/usr/local/nncp/bin/nncp-file -nice B+5 -noprogress - 'backupdisk2:$FILE'") \
   > /dev/null
You could probably also more safely use pee(1) (from moreutils) to do this. This has an unfortunate result of doubling the network traffic from every machine being backed up. So an alternative option would be to queue the packets to the spooling machine, and run a distribution script from it; something like this, in part: 
INCOMINGDIR="/var/local/nncp-bakfs-incoming"
LOCKFILE="$INCOMINGDIR/.lock"
printf -v EVAL_SAFE_LOCKFILE '%q' "$LOCKFILE"
if dotlockfile -r 0 -l -p "$ LOCKFILE "; then
  logit "Lock obtained at $ LOCKFILE  with dotlockfile"
  trap 'ECODE=$?; dotlockfile -u '"$ EVAL_SAFE_LOCKFILE "'; exit $ECODE' EXIT INT TERM
else
  logit "Could not obtain lock at $LOCKFILE; $0 likely already running."
  exit 0
fi
logit "Scanning queue directory..."
cd "$INCOMINGDIR"
for HOST in *; do
   cd "$INCOMINGDIR/$HOST"
   for FILE in bakfsfmt2-*; do
           if [ -f "$FILE" ]; then
                   for BAKFS in backupdisk1 backupdisk2; do
                           runcommand nncp-file -nice B+5 -noprogress "$FILE" "$BAKFS:$HOST/$FILE"
                   done
                   runcommand rm "$FILE"
           else
                   logit "$HOST: Skipping $FILE since it doesn't exist"
           fi
   done
done
logit "Scan complete."
Security Considerations You ll notice that in my example above, the encryption happens as the root user, but nncp is called under su. This means that even if there is a vulnerability in NNCP, the data would still be protected by GPG. I ll also note here that many sites run ssh as root unnecessarily; the same principles should apply there. (ssh has had vulnerabilities in the past as well). I could have used gpg s built-in compression, but zstd is faster and better, so we can get good performance by using fast compression and piping that to an algorithm that can use hardware acceleration for encryption. I strongly encourage considering transport, whether ssh or NNCP or UUCP, to be untrusted. Don t run it as root if you can avoid it. In my example, the nncp user, which all NNCP commands are run as, has no access to the backup data at all. So even if NNCP were compromised, my backup data wouldn t be. For even more security, I could also sign the backup stream with gpg and validate that on the receiving end. I should note, however, that this conversation assumes that a network- or USB-facing ssh or NNCP is more likely to have an exploitable vulnerability than is gpg (which here is just processing a stream). This is probably a safe assumption in general. If you believe gpg is more likely to have an exploitable vulnerability than ssh or NNCP, then obviously you wouldn t take this particular approach. On the zfs side, the use of -F with zfs receive is avoided; this could lead to a compromised backed-up machine generating a malicious rollback on the destination. Backup zpools should be imported with -R or -N to ensure that a malicious mountpoint property couldn t be used to cause an attack. I choose to use zfs receive -u -o readonly=on which is compatible with both unmounted backup datasets and zpools imported with -R (or both). To access the data in a backup dataset, you would normally clone it and access it there. The processing script So, put this all together and look at an example of a processing script that would run from cron as root and process the incoming ZFS data. 
#!/bin/bash
set -e
set -o pipefail
# Log a message
logit ()  
   logger -p info -t " basename "$0" [$$]" "$1"
 
# Log an error message
logerror ()  
   logger -p err -t " basename "$0" [$$]" "$1"
 
# Log stdin with the given code.  Used normally to log stderr.
logstdin ()  
   logger -p info -t " basename "$0" [$$/$1]"
 
# Run command, logging stderr and exit code
runcommand ()  
   logit "Running $*"
   if "$@" 2> >(logstdin "$1") ; then
      logit "$1 exited successfully"
      return 0
   else
       RETVAL="$?"
       logerror "$1 exited with error $RETVAL"
       return "$RETVAL"
   fi
 
STORE=backups/simplesnap
INCOMINGDIR=/backups/nncp/incoming
if ! [ -d "$INCOMINGDIR" ]; then
        logerror "$INCOMINGDIR doesn't exist"
        exit 0
fi
LOCKFILE="/backups/nncp/.nncp-backups-zfs-scan.lock"
printf -v EVAL_SAFE_LOCKFILE '%q' "$LOCKFILE"
if dotlockfile -r 0 -l -p "$ LOCKFILE "; then
  logit "Lock obtained at $ LOCKFILE  with dotlockfile"
  trap 'ECODE=$?; dotlockfile -u '"$ EVAL_SAFE_LOCKFILE "'; exit $ECODE' EXIT INT TERM
else
  logit "Could not obtain lock at $LOCKFILE; $0 likely already running."
  exit 0
fi
EXITCODE=0
cd "$INCOMINGDIR"
logit "Scanning queue directory..."
for HOST in *; do
    HOSTPATH="$INCOMINGDIR/$HOST"
    # files like backupsfmt2-134.13134_dest
    for FILE in "$HOSTPATH"/backupsfmt2-[0-9]*_?*; do
        if [ ! -f "$FILE" ]; then
            logit "Skipping non-existent $FILE"
            continue
        fi
        # Now, $DEST will be HOST/DEST.  Strip off the @ also.
        DEST=" echo "$FILE"   sed -e 's/^.*backupsfmt2[^_]*_//' -e 's,@,/,g' "
        if [ -z "$DEST" ]; then
            logerror "Malformed dest in $FILE"
            continue
        fi
        HOST2=" echo "$DEST"   sed 's,/.*,,g' "
        if [ -z "$HOST2" ]; then
            logerror "Malformed DEST $DEST in $FILE"
            continue
        fi
        if [ ! "$HOST" = "$HOST2" ]; then
            logerror "$DIR: $HOST doesn't match $HOST2"
            continue
        fi
        logit "Processing $FILE to $STORE/$DEST"
            if runcommand gpg -q -d < "$FILE"   runcommand zstdcat   runcommand zfs receive -u -o readonly=on "$STORE/$DEST"; then
                logit "Successfully processed $FILE to $STORE/$DEST"
                runcommand rm "$FILE"
        else
                logerror "FAILED to process $FILE to $STORE/$DEST"
                EXITCODE=15
        fi
Applying These Ideas to Non-ZFS Backups ZFS backups made our job easier in a lot of ways: Some of these benefits you just won't get without ZFS (or something similar like btrfs), but let's see how we could apply these ideas to non-ZFS backups. I will explore the implementation of them in a future post. When I say "non ZFS", I am being a bit vague as to whether the source, the destination, or both systems are running a non-ZFS filesystem. In general I'll assume that neither are ZFS. The first and most obvious answer is to just tar up the whole system and send that every day. This is, of course, only suitable for small datasets on a fast network. These tarballs could be unpacked on the destination and stored more efficiently via any number of methods (hardlink trees, a block-level deduplicator like borg or rdedup, or even just simply compressed tarballs). To make the network trip more efficient, something like rdiff or xdelta could be used. A signature file could be stored on the machine being backed up (generated via tee/pee at stream time), and the next run could simply send an rdiff delta over NNCP. This would be quite network-efficient, but still would require reading every byte of every file on every backup, and would also require quite a bit of temporary space on the receiving end (to apply the delta to the previous tarball and generate a new one). Alternatively, a program that generates incremental backup files such as rdup could be used. These could be transmitted over NNCP to the backup server, and unpacked there. While perhaps less efficient on the network -- every file with at least one modified byte would be retransmitted in its entirety -- it avoids the need to read every byte of unmodified files or to have enormous temporary space. I should note here that GNU tar claims to have an incremental mode, but it has a potential data loss bug. There are also some tools with algorithms that may apply well in this use care: syrep and fssync being the two most prominent examples, though rdedup (mentioned above) and the nascent asuran project may also be combinable with other tools to achieve this effect. I should, of course, conclude this section by mentioning btrfs. Every time I've tried it, I've run into serious bugs, and its status page indicates that only some of them have been resolved. I would not consider using it for something as important as backups. However, if you are comfortable with it, it is likely to be able to run in more constrained environments than ZFS and could probably be processed in much the same way as zfs streams.

Russ Allbery: Review: The Once and Future Witches

Review: The Once and Future Witches, by Alix E. Harrow
Publisher: Redhook Books
Copyright: October 2020
ISBN: 0-316-42202-9
Format: Kindle
Pages: 515
Once upon a time there were three sisters. They were born in a forgotten kingdom that smelled of honeysuckle and mud, where the Big Sandy ran wide and the sycamores shone white as knuckle-bones on the banks. The sisters had no mother and a no-good father, but they had each other; it might have been enough. But the sisters were banished from their kingdom, broken and scattered.
The Once and Future Witches opens with Juniper, the youngest, arriving in the city of New Salem. The year is 1893, but not in our world, not quite; Juniper has witch-ways in her pocket and a few words of power. That's lucky for her because the wanted posters arrived before she did. Unbeknownst to her or to each other, her sisters, Agnes and Bella, are already in New Salem. Agnes works in a cotton mill after having her heart broken one too many times; the mill is safer because you can't love a cotton mill. Bella is a junior librarian, meek and nervous and uncertain but still fascinated by witch-tales and magic. It's Bella who casts the spell, partly by accident, partly out of wild hope, but it was Juniper arriving in the city who provided the final component that made it almost work. Not quite, not completely, but briefly the lost tower of Avalon appears in St. George's Square. And, more importantly, the three sisters are reunited. The world of the Eastwood sisters has magic, but the people in charge of that world aren't happy about it. Magic is a female thing, contrary to science and, more importantly, God. History has followed a similar course to our world in part because magic has been ruthlessly suppressed. Inquisitors are a recent memory and the cemetery has a witch-yard, where witches are buried unnamed and their ashes sown with salt. The city of New Salem is called New Salem because Old Salem, that stronghold of witchcraft, was burned to the ground and left abandoned, fit only for tourists to gawk at the supposedly haunted ruins. The women's suffrage movement is very careful to separate itself from any hint of witchcraft or scandal, making its appeals solely within the acceptable bounds of the church. Juniper is the one who starts to up-end all of that in New Salem. Juniper was never good at doing what she was told. This is an angry book that feels like something out of another era, closer in tone to a Sheri S. Tepper or Joanna Russ novel than the way feminism is handled in recent work. Some of that is the era of the setting, before women even had the right to vote. But primarily it's because Harrow, like those earlier works, is entirely uninterested in making excuses or apologies for male behavior. She takes an already-heated societal conflict and gives the underdogs magic, which turns it into a war. There is likely a better direct analogy from the suffrage movement, but the comparison that came to my mind was if Martin Luther King, Jr. proved ineffective or had not existed, and instead Malcolm X or the Black Panthers became the face of the Civil Rights movement. It's also an emotionally exhausting book. The protagonists are hurt and lost and shattered. Their moments of victory are viciously destroyed. There is torture and a lot of despair. It works thematically; all the external solutions and mythical saviors fail, but in the process the sisters build their own strength and their own community and rescue themselves. But it's hard reading at times if you're emotionally invested in the characters (and I was very invested). Harrow does try to balance the losses with triumphs and that becomes more effective and easier to read in the back half of the book, but I struggled with the grimness at the start. One particular problem for me was that the sisters start the book suspicious and distrustful of each other because of lies and misunderstandings. This is obvious to the reader, but they don't work through it until halfway through the book. I can't argue with this as a piece of characterization it made sense to me that they would have reacted to their past the way that they did. But it was still immensely frustrating to read, since in the meantime awful things were happening and I wanted them to band together to fight. They also worry over the moral implications of the fate of their father, whereas I thought the only problem was that the man couldn't die more than once. There too, it makes sense given the moral framework the sisters were coerced into, but it is not my moral framework and it was infuriating to see them stay trapped in it for so long. The other thing that I found troubling thematically is that Harrow personalizes evil. I thought the more interesting moral challenge posed in this book is a society that systematically abuses women and suppresses their power, but Harrow gradually supplants that systemic conflict with a villain who has an identity and a backstory. It provides a more straightforward and satisfying climax, and she does avoid the trap of letting triumph over one character solve all the broader social problems, but it still felt too easy. Worse, the motives of the villain turn out to be at right angles to the structure of the social oppression. It's just a tool he's using, and while that's also believable, it means the transfer of the narrative conflict from the societal to the personal feels like a shying away from a sharper political point. Harrow lets the inhabitants of New Salem off too easily by giving them the excuse of being manipulated by an evil mastermind. What I thought Harrow did handle well was race, and it feels rare to be able to say this about a book written by and about white women. There are black women in New Salem as well, and they have their own ways and their own fight. They are suspicious of the Eastwood sisters because they're worried white women will stir up trouble and then run away and leave the consequences to fall on black women... and they're right. An alliance only forms once the white women show willingness to stay for the hard parts. Black women are essential to the eventual success of the protagonists, but the opposite is not necessarily true; they have their own networks, power, and protections, and would have survived no matter what the Eastwoods did. The book is the Eastwoods' story, so it's mostly concerned with white society, but I thought Harrow avoided both making black women too magical or making white women too central. They instead operate in parallel worlds that can form the occasional alliance of mutual understanding. It helps that Cleopatra Quinn is one of the best characters of the book. This was hard, emotional reading. It's the sort of book where everything has a price, even the ending. But I'm very glad I read it. Each of the three sisters gets their own, very different character arc, and all three of those arcs are wonderful. Even Agnes, who was the hardest character for me to like at the start of the book and who I think has the trickiest story to tell, becomes so much stronger and more vivid by the end of the book. Sometimes the descriptions are trying a bit too hard and sometimes the writing is not quite up to the intended goal, but some of the descriptions are beautiful and memorable, and Harrow's way of weaving the mythic and the personal together worked for me. This is a more ambitious book than The Ten Thousand Doors of January, and while I think the ambition exceeded Harrow's grasp in a few places and she took a few thematic short-cuts, most of it works. The characters felt like living and changing people, which is not easy given how heavily the story structure leans on maiden, mother, and crone archetypes. It's an uncompromising and furious book that turns the anger of 1970s feminist SF onto themes that are very relevant in 2021. You will have to brace yourself for heartbreak and loss, but I think it's fantasy worth reading. Recommended. Rating: 8 out of 10

1 January 2021

Jonathan Wiltshire: WordPress in a subdirectory

For many years now I ve had WordPress installed as a subdirectory to my site but appearing to at the domain level, i.e. /wordpress/index.php is transparently presented as the homepage. This is done by setting the WordPress Address and Site Address settings and then mapping requests which do not match an existing file or directory through as a PHP pathinfo using Apache s mod_rewrite rules in a .htaccess file or server configuration. In this way most of the site is WordPress s dynamic pages and posts, but WordPress itself is neatly contained and random static resources such as /screentest/1024.GIF work as expected. Those .htaccess rules were originally hand-crafted but didn t take account of changing recommendations, e.g. the Authorization header. When I rearranged matters recently I decided to take advantage of WordPress s own generated rules and ditch my old rules. They look like this: 
# BEGIN WordPress
# The directives (lines) between "BEGIN WordPress" and "END WordPress" are
# dynamically generated, and should only be modified via WordPress filters.
# Any changes to the directives between these markers will be overwritten.
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteRule .* - [E=HTTP_AUTHORIZATION:% HTTP:Authorization ]
RewriteBase /
RewriteRule ^index\.php$ - [L]
RewriteCond % REQUEST_FILENAME  !-f
RewriteCond % REQUEST_FILENAME  !-d
RewriteRule . /index.php [L]
</IfModule>
# END WordPress
If .htaccess is writable, WordPress adds these rules automatically. Unfortunately, when I switched to them and not my own, the site broke. Crucially the generated rules do not cater for WordPress being in a subdirectory. No matter what combination of settings I tried, local changes to the rules (like RewriteRule . /wordpress/index.php [L]) were rewritten back to the original values at random times and clearly couldn t be relied upon to stay as I need them. This feels like a bug: my intuition is that if the WordPress Address and Site Address settings are different, but within the same domain, the rules should be generated to take care of that. One option is to make the .htaccess file read-only, which WordPress detects and avoids trying to make changes at all. But this rather defeats the object of letting it take care of future changes as the software and specifications change. The second option, which I stuck with, is to add an index.php file with the following contents: 
<?php
include('wordpress/index.php');
Now the generated rules, though incorrect, result in the right behaviour and won t break on every setting change.

Dmitry Shachnev: A review of endianness bugs in Qt, and how they were fixed

As you may know, I am Qt 5 maintainer in Debian. Maintaning Qt means not only bumping the version each time a new version is released, but also making sure Qt builds successfully on all architectures that are supported in Debian (and for some submodules, the automatic tests pass). An important sort of build failures are endianness specific failures. Most widely used architectures (x86_64, aarch64) are little endian. However, Debian officially supports one big endian architecture (s390x), and unofficially a few more ports are provided, such as ppc64 and sparc64. Unfortunately, Qt upstream does not have any big endian machine in their CI system, so endianness issues get noticed only when the packages fail to build on our build daemons. In the last years I have discovered and fixed some such issues in various parts of Qt, so I decided to write a post to illustrate how to write really cross-platform C/C++ code. Issue 1: the WebP image format handler (code review) The relevant code snippet is: 
if (srcImage.format() != QImage::Format_ARGB32)
    srcImage = srcImage.convertToFormat(QImage::Format_ARGB32);
// ...
if (!WebPPictureImportBGRA(&picture, srcImage.bits(), srcImage.bytesPerLine()))  
    // ...
The code here is serializing the images into QImage::Format_ARGB32 format, and then passing the bytes into WebP s import function. With this format, the image is stored using a 32-bit ARGB format (0xAARRGGBB). This means that the bytes will be 0xBB, 0xGG, 0xRR, 0xAA or little endian and 0xAA, 0xRR, 0xGG, 0xBB on big endian. However, WebPPictureImportBGRA expects the first format on all architectures. The fix was to use QImage::Format_RGBA8888. As the QImage documentation says, with this format the order of the colors is the same on any architecture if read as bytes 0xRR, 0xGG, 0xBB, 0xAA. Issue 2: qimage_converter_map structure (code review) The code seems to already support big endian. But maybe you can spot the error? 
#if Q_BYTE_ORDER == Q_LITTLE_ENDIAN
        0,
        convert_ARGB_to_ARGB_PM,
#else
        0,
        0
#endif
It is the missing comma! It is present in the little endian block, but not in the big endian one. This was fixed trivially. Issue 3: QHandle, part of Qt 3D module (code review) QHandle class uses a union that is declared as follows: 
struct Data  
    quint32 m_index : IndexBits;
    quint32 m_counter : CounterBits;
    quint32 m_unused : 2;
 ;
union  
    Data d;
    quint32 m_handle;
 ;
The sizes are declared such as IndexBits + CounterBits + 2 is always equal to 32 (four bytes). Then we have a constructor that sets the values of Data struct: 
QHandle(quint32 i, quint32 count)
 
    d.m_index = i;
    d.m_counter = count;
    d.m_unused = 0;
The value of m_handle will be different depending on endianness! So the test that was expecting a particular value with given constructor arguments was failing. I fixed it by using the following macro: 
#if Q_BYTE_ORDER == Q_BIG_ENDIAN
#define GET_EXPECTED_HANDLE(qHandle) ((qHandle.index() << (qHandle.CounterBits + 2)) + (qHandle.counter() << 2))
#else /* Q_LITTLE_ENDIAN */
#define GET_EXPECTED_HANDLE(qHandle) (qHandle.index() + (qHandle.counter() << qHandle.IndexBits))
#endif
Issue 4: QML compiler (code review) The QML compiler used a helper class named LEUInt32 (based on QLEInteger) that always stored the numbers in little endian internally. This class can be safely mixed with native quint32 on little endian systems, but not on big endian. Usually the compiler would warn about type mismatch, but here the code used reinterpret_cast, such as: 
quint32 *objectTable = reinterpret_cast<quint32*>(data + qmlUnit->offsetToObjects);
So this was not noticed on build time, but the compiler was crashing. The fix was trivial again, replacing quint32 with QLEUInt32. Issue 5: QModbusPdu, part of Qt Serial Bus module (code review) The code snippet is simple: 
QModbusPdu::FunctionCode code = QModbusPdu::Invalid;
if (stream.readRawData((char *) (&code), sizeof(quint8)) != sizeof(quint8))
    return stream;
QModbusPdu::FunctionCode is an enum, so code is a multi-byte value (even if only one byte is significant). However, (char *) (&code) returns a pointer to the first byte of it. It is the needed byte on little endian systems, but it is the wrong byte on big endian ones! The correct fix was using a temporary one-byte variable: 
quint8 codeByte = 0;
if (stream.readRawData((char *) (&codeByte), sizeof(quint8)) != sizeof(quint8))
    return stream;
QModbusPdu::FunctionCode code = (QModbusPdu::FunctionCode) codeByte;
Issue 6: qt_is_ascii (code review) This function, as the name says, checks whether a string is ASCII. It does that by splitting the string into 4-byte chunks: 
while (ptr + 4 <= end)  
    quint32 data = qFromUnaligned<quint32>(ptr);
    if (data &= 0x80808080U)  
        uint idx = qCountTrailingZeroBits(data);
        ptr += idx / 8;
        return false;
     
    ptr += 4;
idx / 8 is the number of trailing zero bytes. However, the bytes which are trailing on little endian are actually leading on big endian! So we can use qCountLeadingZeroBits there. Issue 7: the bundled copy of tinycbor (upstream pull request) Similar to issue 5, the code was reading into the wrong byte: 
if (bytesNeeded <= 2)  
    read_bytes_unchecked(it, &it->extra, 1, bytesNeeded);
    if (bytesNeeded == 2)
        it->extra = cbor_ntohs(it->extra);
extra has type uint16_t, so it has two bytes. When we need only one byte, we read into the wrong byte, so the resulting number is 256 times higher on big endian than it should be. Adding a temporary one-byte variable fixed it. Issue 8: perfparser, part of Qt Creator (code review) Here it is not trivial to find the issue just looking at the code: 
qint32 dataStreamVersion = qToLittleEndian(QDataStream::Qt_DefaultCompiledVersion);
However the linker was producing an error:
undefined reference to QDataStream::Version qbswap(QDataStream::Version)'
On little endian systems, qToLittleEndian is a no-op, but on big endian systems, it is a template function defined for some known types. But it turns out we need to explicitly convert enum values to a simple type, so the fix was passing qint32(QDataStream::Qt_DefaultCompiledVersion) to that function. Issue 9: Qt Personal Information Management (code review) The code in test was trying to represent a number as a sequence of bytes, using reinterpret_cast: 
static inline QContactId makeId(const QString &managerName, uint id)
 
    return QContactId(QStringLiteral("qtcontacts:basic%1:").arg(managerName), QByteArray(reinterpret_cast<const char *>(&id), sizeof(uint)));
The order of bytes will be different on little endian and big endian systems! The fix was adding this line to the beginning of the function: 
id = qToLittleEndian(id);
This will cause the bytes to be reversed on big endian systems. What remains unfixed There are still some bugs, which require deeper investigation, for example: P.S. We are looking for new people to help with maintaining Qt 6. Join our team if you want to do some fun work like described above!

Keith Packard: kgames

Reviving Very Old X Code I've taken the week between Christmas and New Year's off this year. I didn't really have anything serious planned, just taking a break from the usual routine. As often happens, I got sucked into doing a project when I received this simple bug report Debian Bug #974011 
I have been researching old terminal and X games recently, and realized
that much of the code from 'xmille' originated from the terminal game
'mille', which is part of bsdgames.
...
[The copyright and license information] has been stripped out of all
code in the xmille distribution.  Also, none of the included materials
give credit to the original author, Ken Arnold.
The reason the 'xmille' source is missing copyright and license information from the 'mille' files is that they were copied in before that information was added upstream. Xmille forked from Mille around 1987 or so. I wrote the UI parts for the system I had at the time, which was running X10R4. A very basic port to X11 was done at some point, and that's what Debian has in the archive today. At some point in the 90s, I ported Xmille to the Athena widget set, including several custom widgets in an Xaw extension library, Xkw. It's a lot better than the version in Debian, including displaying the cards correctly (the Debian version has some pretty bad color issues). Here's what the current Debian version looks like: Fixing The Bug To fix the missing copyright and license information, I imported the mille source code into the "latest" Xaw-based version. The updated mille code had a number of bug fixes and improvements, along with the copyright information. That should have been sufficient to resolve the issue and I could have constructed a suitable source package from whatever bits were needed and and uploaded that as a replacement 'xmille' package. However, at some later point, I had actually merged xmille into a larger package, 'kgames', which also included a number of other games, including Reversi, Dominoes, Cribbage and ten Solitaire/Patience variants. (as an aside, those last ten games formed the basis for my Patience Palm Pilot application, which seems to have inspired an Android App of the same name...) So began my yak shaving holiday. Building Kgames in 2020 Ok, so getting this old source code running should be easy, right? It's just a bunch of C code designed in the 80s and 90s to work on VAXen and their kin. How hard could it be?
  1. Everything was a 32-bit computer back then; pointers and ints were both 32 bits, so you could cast them with wild abandon and cause no problems. Today, testing revealed segfaults in some corners of the code.
  2. It's K&R C code. Remember that the first version of ANSI-C didn't come out until 1989, and it was years later that we could reliably expect to find an ANSI compiler with a random Unix box.
  3. It's X11 code. Fortunately (?), X11 hasn't changed since these applications were written, so at least that part still works just fine. Imagine trying to build Windows or Mac OS code from the early 90's on a modern OS...
I decided to dig in and add prototypes everywhere; that found a lot of pointer/int casting issues, as well as several lurking bugs where the code was just plain broken. After a day or so, I had things building and running and was no longer hitting crashes. Kgames 1.0 uploaded to Debian New Queue With that done, I decided I could at least upload the working bits to the Debian archive and close the bug reported above. kgames 1.0-2 may eventually get into unstable, presumably once the Debian FTP team realizes just how important fixing this bug is. Or something. Here's what xmille looks like in this version: And here's my favorite solitaire variant too: But They Look So Old Yeah, Xaw applications have a rustic appearance which may appeal to some, but for people with higher resolution monitors and well seasoned eyesight, squinting at the tiny images and text makes it difficult to enjoy these games today. How hard could it be to update them to use larger cards and scalable fonts? Xkw version 2.0 I decided to dig in and start hacking the code, starting by adding new widgets to the Xkw library that used cairo for drawing instead of core X calls. Fortunately, the needs of the games were pretty limited, so I only needed to implement a handful of widgets: The other Xkw widgets all got their rendering switched to using cairo, plus using double buffering to make updates look better. SVG Playing Cards Looking on wikimedia, I found a page referencing a large number of playing cards in SVG form That led me to Adrian Kennard's playing card web site that let me customize and download a deck of cards, licensed using the CC0 Public Domain license. With these cards, I set about rewriting the Xkw playing card widget, stripping out three different versions of bitmap playing cards and replacing them with just these new SVG versions. SVG Xmille Cards Ok, so getting regular playing cards was good, but the original goal was to update Xmille, and that has cards hand drawn by me. I could just use those images, import them into cairo and let it scale them to suit on the screen. I decided to experiment with inkscape's bitmap tracing code to see what it could do with them. First, I had to get them into a format that inkscape could parse. That turned out to be a bit tricky; the original format is as a set of X bitmap layers; each layer painting a single color. I ended up hacking the Xmille source code to generate the images using X, then fetching them with XGetImage and walking them to construct XPM format files which could then be fed into the portable bitmap tools to create PNG files that inkscape could handle. The resulting images have a certain charm: I did replace the text in the images to make it readable, otherwise these are untouched from what inkscape generated. The Results Remember that all of these are applications built using the venerable X toolkit; there are still some non-antialiased graphics visible as the shaped buttons use the X Shape extension. But, all rendering is now done with cairo, so it's all anti-aliased and all scalable. Here's what Xmille looks like after the upgrades: And here's spider: Once kgames 1.0 reaches Debian unstable, I'll upload these new versions.

31 December 2020

Chris Lamb: Free software activities in December 2020

Here is my monthly update covering what I have been doing in the free software world during December 2020 (previous month):

Reproducible Builds One of the original promises of open source software is that distributed peer review and transparency of process results in enhanced end-user security. However, whilst anyone may inspect the source code of free and open source software for malicious flaws, almost all software today is distributed as pre-compiled binaries. This allows nefarious third-parties to compromise systems by injecting malicious code into ostensibly secure software during the various compilation and distribution processes. The motivation behind the Reproducible Builds effort is to ensure no flaws have been introduced during this compilation process by promising identical results are always generated from a given source, thus allowing multiple third-parties to come to a consensus on whether a build was compromised. This month, I: I also made the following changes to diffoscope, our in-depth and content-aware diff utility that can locate and diagnose reproducibility issues, including releasing version 163:

Debian Uploads I also sponsored an upload of adminer (4.7.8-2) on behalf of Alexandre Rossi and performed two QA uploads of sendfile (2.1b.20080616-7 and 2.1b.20080616-8) to make the build the build reproducible (#776938) and to fix a number of other unrelated issues. Debian LTS This month I have worked 18 hours on Debian Long Term Support (LTS) and 12 hours on its sister Extended LTS project. You can find out more about the Debian LTS project via the following video:

Daniel Kahn Gillmor: New OpenPGP certificate for dkg, 2021

dkg's 2021 OpenPGP transition As 2021 begins, I'm changing to a new OpenPGP certificate. I did a similar transition two years ago, and a fair amount has changed since then. You might know my old OpenPGP certificate as: 
pub   ed25519 2019-01-19 [C] [expires: 2021-01-18]
      C4BC2DDB38CCE96485EBE9C2F20691179038E5C6
uid          Daniel Kahn Gillmor <dkg@fifthhorseman.net>
uid          Daniel Kahn Gillmor <dkg@debian.org>
My new OpenPGP certificate is: 
pub   ed25519 2020-12-27 [C] [expires: 2023-12-24]
      C29F8A0C01F35E34D816AA5CE092EB3A5CA10DBA
uid           [ unknown] Daniel Kahn Gillmor
uid           [ unknown] <dkg@debian.org>
uid           [ unknown] <dkg@fifthhorseman.net>
You can find a signed transition statement if you're into that sort of thing. If you're interested in the rationale for why I'm making this transition, read on.

Dangers of Offline Primary Secret Keys There are several reasons for transitioning, but one i simply couldn't argue with was my own technical failure. I put the primary secret key into offline storage some time ago for "safety", and used ext4's filesystem-level encryption layered on top of dm-crypt for additional security. But either the tools changed out from under me, or there were failures on the storage medium, or I've failed to remember my passphrase correctly, because I am unable to regain access to the cleartext of the secret key. In particular, I find myself unable to use e4crypt add_key with the passphrase I know to get a usable working directory. I confess I still find e4crypt pretty difficult to use and I don't use it often, so the problem may entirely be user error (either now, or two years ago when I did the initial setup). Anyway, lesson learned: don't use cryptosystems that you're not comfortable with to encrypt data that you care about recovering. This is a lesson I'm pretty sure I've learned before, sigh, but it's a good reminder.

Split User IDs I'm trying to split out my User IDs again -- this way if you know me by e-mail address, you don't have to think/worry about certifying my name, and if you know me by name, you don't have to think/worry about certifying my e-mail address. I think that's simpler and more sensible. It's also nice because e-mail address-only User IDs can be used effectively in contexts like Autocrypt, which I think are increasingly important if we want to have usable encrypted e-mail. Last time around I initially tried split User IDs but rolled them back and I think most of the bugs I discovered then have been fixed.

Certificate Flooding Another reason for making a transition to a new certificate is that my older certificate is one of the ones that was "flooded" on the SKS keyserver network last year, which was one of the final straws for that teetering project. Transitioning to a new certificate lets that old flooded cert expire and people can just simply move on from it, ideally deleting it from their local keyrings. Hopefully as a community we can move on from SKS to key distribution mechanisms like WKD, Autocrypt, DANE, and keys.openpgp.org, all of which address some of the known problems with keyserver abuse.

Trying New Tools Finally, I'm also interested in thinking about how key and certificate management might be handled in different ways. While I'm reasonably competent in handling GnuPG, the larger OpenPGP community (which I'm a part of) has done a lot of thinking and a lot of work about how people can use OpenPGP. I'm particularly happy with the collaborative work that has gone into the Stateless OpenPGP CLI (aka sop), which helps to generate a powerful interoperability test suite. While sop doesn't offer the level of certificate management I'd need to use it to manage this new certificate in full, I wish something like it would! Starting from a fresh certificate and actually using it helps me to think through what I might actually need from a tool that is roughly as straightforward and opinionated as sop is. If you're a software developer who might use or implement OpenPGP, or a protocol designer, and you haven't played around with any of the various implementations of sop yet, I recommend taking a look. And feedback on the specification is always welcome, too, including ideas for new functionality (maybe even like certificate management).

Next Steps If you're the kind of person who's into making OpenPGP certifications, feel free to check in with me via whatever channels you're used to using to verify that this transition is legit. If you think it is, and you're comfortable, please send me (e-mail is probably best) your certifications over the new certficate. I'll keep on working to make OpenPGP more usable and acceptable. Hopefully, 2021 will be a better year ahead for all of us.

Russ Allbery: Review: Billion Dollar Loser

Review: Billion Dollar Loser, by Reeves Wiedeman
Publisher: Little, Brown and Company
Copyright: October 2020
ISBN: 0-316-46134-2
Format: Kindle
Pages: 315
WeWork was founded in 2010 by Adam Neumann and Miguel McKelvey as a successor company to their similar 2008 GreenDesk business. (Adam's wife Rebekah is now presented as a co-founder. This seems dubious in Wiedeman's account, although Rebekah's role in the company is murky, ever-changing, and hard to pin down.) Its business model in reality was to provide turn-key, pre-furnished and stocked co-working and small office space to individuals and businesses on flexible, short-term leases. Its business model in Neumann's speeches and dreams, and represented by the later renaming of the company to the We Corporation, was nothing less than to transform the way people worked, learned, and lived. Through aggressive, money-losing expansion, WeWork grew rapidly to over 500 locations in 29 countries and became the largest office tenant in New York City. Based primarily on massive financial support from Masayoshi Son, CEO of Japanese holding company SoftBank, WeWork's private valuation rose to $47 billion. In 2019, the company attempted to go public, but its IPO collapsed, in part due to deeper analysis of the company's books. Neumann was forced out of the company (with an individual payout valued at $1.7 billion), the IPO was withdrawn, SoftBank wrote down 90% of their investment in the company and took control of it, and WeWork laid off more than 20% of its workforce. This book is a detailed history of WeWork's rise and fall, joining a genre that includes The Smartest Guys in the Room (Enron), Bad Blood (Theranos), and Super Pumped (Uber). I believe it's the first full book on WeWork, although it was preceded by several long-form stories, including "The I In We" by Wiedeman for New York magazine. As the first history, it's a somewhat incomplete cut: litigation between Neumann and WeWork is still pending, WeWork staggered into 2020 and a world-wide pandemic that made cramped open-plan offices an epidemiological disaster, and there will doubtless be new revelations yet to come. The discovery process of lawsuits tends to be good for journalists. But despite being the first out of the gate, Billion Dollar Loser reaches a satisfying conclusion with the ouster of Neumann, who had defined WeWork both internally and externally. I'm fascinated by stories of failed venture capital start-ups in general, but the specific question about WeWork that interested me, and to which Wiedeman provides a partial answer, is why so many people gave Neumann money in the first place. Explaining that question requires a digression into why I thought WeWork's valuation was absurd. The basic problem WeWork had when justifying its sky-high valuation is competition. WeWork didn't own real estate; it rented properties from landlords with long-term leases and then re-rented them with short-term leases. If its business was so successful, why wouldn't the landlords cut out the middle man, do what WeWork was doing directly, and pocket all the profit? Or why wouldn't some other company simply copy WeWork and drive the profit margins down? Normally with startups the answer revolves around something proprietary: an app, a server technology, patents, a secret manufacturing technique, etc. But nothing WeWork was doing was different from what innumerable tech companies and partner landlords had been doing with their office space for a decade, and none of it was secret. There are two decent answers to that question. One is simple outsourcing: landlords like being passive rent collectors, so an opportunity to pay someone else to do the market research on office layouts, arrange all the remodeling, adapt to changing desires for how office space should be equipped and stocked, advertise for short-term tenants, and deal with the tenant churn is attractive. The landlord can sit back and pocket the stable long-term rent. The second answer is related: WeWork is essentially doing rental arbitrage between long-term and short-term rents and thus is taking on most of the risk of a commercial real estate downturn. If no one is renting office space, WeWork is still on the hook for the long-term rent. The landlord is outsourcing risk, at least unless WeWork goes bankrupt. (One infuriating tidbit from this book is that Neumann's explicit and often-stated goal was to make WeWork so large that its bankruptcy would be sufficiently devastating to the real estate industry that it would get a bailout.) There's a legitimate business here. But that business looks like a quietly profitable real estate company that builds very efficient systems for managing short-term leases, remodeling buildings, and handling the supply chain of stocking an office. That looks nothing like WeWork's business, has nothing to do with transforming the world of work, and certainly doesn't warrant sky-high valuations. WeWork didn't build an efficient anything. It relied on weekend labor from underpaid employees and an IT person who was still in high school. And WeWork actively resisted being called a real estate company and claimed it was a tech company or a lifestyle company on the basis of essentially nothing. Wiedeman seems almost as baffled by this as I am, but it's clear from the history he tells that part of the funding answer is the Ponzi scheme of start-up investing. People gave Neumann money because other people had previously given Neumann money, and the earlier investors cashed out at the expense of the later ones. Like any Ponzi scheme, it looks like a great investment until it doesn't, and then the last sucker is left holding the bag. That sucker was Masayoshi Son, who in Wiedeman's telling is an astonishingly casual and undisciplined investor who trusted knee-jerk personal reactions to founders over business model analysis and historically (mostly) got away with it by getting extremely lucky. (I now want to read one of these books about SoftBank, since both this book and Super Pumped make it look like a company that makes numerous wild gambles for the flimsiest of reasons, pushes for completely unsustainable growth, and relies on the sheer volume of investments catching some lucky jackpots and cashing out in IPOs. Unfortunately, the only book currently available seems to be a fawning hagiography of Son.) On one hand, the IPO process worked properly this time. The sheer madness of WeWork's valuation scared off enough institutional investors that it collapsed. On the other hand, it's startling how close it came to success. If WeWork had kept the Ponzi scheme going a bit longer, the last sucker could have been the general investing public. Another interesting question that Billion Dollar Loser answers is how Neumann got enough money to start his rapid growth strategy. The answer appears to be the oldest and most obvious explanation: He made friends with rich people. The initial connections appear to have been through his sister, Adi Neumann, who is a model and hosted parties in her New York apartment (and also started dating a Rothschild heir). Adam met his wealthy wife Rebekah, cousin to actress and "wellness" scam marketer Gwyneth Paltrow, via a connection at a party. He built social connections with other parts of the New York real estate scene and tapped them for investment money. The strong impression one gets from the book is that all of these people have way more money than sense and we should raise their taxes. It won't come as a surprise that Adam and Rebekah Neumann are good friends of Jared Kushner and Ivanka Trump. Those are the questions I was the most curious about, but there's much more here. Wiedeman's style is nearly straight chronological reporting with little analysis, but the story is so wild and absurd that it doesn't need much embellishment. Neumann is obviously a megalomaniac whose delusions of grandeur got worse and worse as WeWork was apparently succeeding. Rebekah Neumann is if anything even less in touch with reality than he is, although in her case that appears to stem from having so much money that reality is an inconvenient speed bump. Miguel McKelvey, Neumann's co-founder, is an odd and interesting side note to the story; he appears to have balanced Adam out a bit in the early going but then wisely started to cash out and pocket his winnings while letting Adam dominate the stage. There are some places where I don't think Wiedeman pushed hard enough, and which cut against the view of Neumann as a true believer in his impossible growth vision. Neumann took several investment opportunities to cash out large amounts of his stock even while WeWork employees were being underpaid and told their stock options would make up for it. He clearly used WeWork as a personal piggy bank on multiple occasions. And Wiedeman documents but doesn't, at least in my opinion, make nearly enough of Neumann's self-dealing: buying real estate that WeWork then rented as a tenant, or paying himself for a license for the name We Holdings (although there at least he later returned the money). I think a good argument could be made that Neumann was embezzling from WeWork, at least morally if not legally, and I wish Wiedeman would have pressed harder on that point. But that aside, this is a great first history of the company, told in a clean, readable, and engaging style, and with a lot more detail here than I've touched on (such as Rebekah Neumann's WeGrow school). It's not as good as Bad Blood (what is), but it's a respectable entry in the corporate collapse genre. If you like this sort of thing, recommended. Rating: 7 out of 10

29 December 2020

Shirish Agarwal: Inequality in Indian Education

Farmer on-going protests Before I start with the education system in India which I have talked about many times in the past, first let me share couple of pieces about the farmer movement which is still at Sanghu Delhi border.
<Manjeet Kaur,62 at farmer protest with her friends.
The above picture became somewhat viral as it showed Manjeet Kaur, who drove down from Patiala, Punjab to Sanghu border along with her friends to take part in the on-going protests. The picture not only shares how the women are part and parcel of this protest but also they are independently taking part in the protest. The other were two articles I read today, first was an article in tribune which questions that if the policy worked, why it didn t work in the state of Bihar. The other by a young law student who had to go from Chandigarh to Delhi with family for some work and her experience with the ongoing protest. In fact, an interesting observation was made by the CJI in the many suits against farmer protests in the SC. This makes for much more interesting read when you see an RTI query filed by Saket Gokhale to NHAI , a Central Govt. agency which is supposed to be independent and asks if they had filed an FIR and asked compensation from Haryana State and Haryana State Police which had dug up National Highway 44 and if any permission was asked for the same from NHAI. And NHAI unable to take any action for the same. If this isn t shameful, I dunno what is
Saket Gokhale s RTI query on digging up NH44
NHAI response to Saket Gokhale s query.
Sadly, the way the response has been worded makes it impossible for NHAI to discharge its own responsibilities and this becomes a precedent for other states now that know that NHAI is vulnerable. A pretty sad turn on events. Indian Education can t go online There was a recent article on scroll which shared how Indian education can t go online as only a few have computers with decent netlink speeds as well as other factors which are needed for online education. But there are also many things that the article doesn t take into account which actually make the task more difficult and raise the boundary more. Now in most schools and colleges, the number of students to teacher ratio could be anywhere between 70-150 or even more. In the last few years, a lot of schools have been closed down by various Governments, including and not limited to the ruling Govt. They have in fact intensified closures of public schools wherever their Govt. has been in power. Closing to 5000+ schools in one state in a year is a dramatic shift and such has been happening time and again. In fact, the rising costs of Indian education has made many to leave Indian shores and do studies abroad. And once they do their masters or whatever, the chances of them coming back to India become more and more remote. In India the costs have been becoming so bad that NBFC s have started products targeting the same. How NBFC and Banks have (both public and private) have fared with respect to Indian consumers needs its own blog post but one word to describe it is bad . But as shared above, needs its own blog post. Coming to the Indian context though, what has not been captured in that article is that the responsibility of making new content also raises huge barriers for teachers. My own experience in teacher s trainings for ICT usage has shown that most teachers do not know and use internet effectively both to sustain their own curiosity as well as their students. Part of which is whether you are private employee or a public school teacher, the teacher is not paid enough. I have had multiple conversations with friends over the years who are teachers who shared that they get 50% salary in-hand while they sign for 100%. This is more in the case or private schools though. In Govt. schools, the teachers apart from their regular administrative duties apart from teaching duties are also unpaid labor for Govt. policy. Take the recent covid crisis, it was the teachers who for months together went from door-to-door asking if they had a covid patient. This was all over India. Even for voter registration, census, polio and various other immunization efforts, the teachers are roped in. So apart from that, they somehow have to figure out how to make ends meet and also boost student morale. Hence the attention is only limited to the first couple of benches rather than the whole as a 45-minute to an hr. session is just not enough to go through a class of 70-150 school students giving individual attention. And this is when for most teachers, teaching is a means to an end and not the end itself. I am going to take one example of science and sort of break-it-down in multiple steps and how I would have approached that topic for say the 5th-6th standard students in say a public school in Pune and especially if Covid would not have been an issue so you have face-to-face meetup. There was recent news about a mysterious radio signal which came from one of our closest galactic neighbors Proxima Centauri. Now let s say there was a class I was teaching/sharing which I had shared before and there already is trust formed. So before coming to the news, I would tell the students about frequency and more generally the notation of why we like to measure things and how we measure things. There is so much beautiful history which could be acted and enacted which can show and remains in mind why measurement is needed. Once that is understood, discussed and an underpinning is established, we could move to human perception or the lack of it. We know that humans have lots of limits in almost everything, whether it is talking, touching, hearing, all of our five senses are pretty limited with what we know of spectrum available in the immediate family kingdom as well as in the Universe. I would start with how far can a person throw his voice and be heard without using any other means. There does come a point where they need to use anything from a megaphone to a loudspeaker and what it actually does. The other thing I would then talk is about the radio and ask the students to find more about the internals of a radio. If possible to bring an old radio to school where it could be disassembled. After they are familiar with some names of the electronic components and what they do, take them to the electronics market where they try to source all the things needed to make a radio and whatever they encounter. This would allow the students to try and do bargain shopping as well as learn from where to source things. Some might even get a copy or two of electronic projects where the shop themselves sell blueprints to hobbyists so that they can tinker. If there is a place in the school where soldering can be done, then all can try and sooner or later we come to know if something works or not. There is also possibility of talking about noise cancellation and then the topic of ITU can also be bought up and how they do frequency allocation. Last but not the least then the topic can be approached about an alien civilization and an unknown radio signal and what it means and what it can mean. Now if just one topic can give such a wide range of things to do and develop an understanding about not the subject itself but surrounding subjects as well I see no reason why teachers can t do this except they are handcuffed to lot of policy as well as real-life constraints. For e.g. I remember in my school days, we used to go out once or twice a year and that used to be either a school picnic or something similar. The only other I know is going to Mumbai for Nehru planetarium and Nehru science Centre. Unfortunately, I didn t go at that time because the school was taking students via air and the tickets were super costly at the time and that too for a 10 minute journey between the two cities. Those were different days, today you can t have a direct flight between the two cities as it doesn t make an economic sense. It makes more sense to go to Mumbai via train or bus as you will reach Mumbai in about couple of hrs. Of course, Pune does have its own planetarium at New English school and there are a few amateur astronomy clubs in Pune but nothing on the scale that what Mumbai has, but then this is getting off-topic. Now, again in an online world could this be done? Not without both the teacher and the student both spending lot of resources online and even then will be a lower understanding as both the hands-on experience as well as interacting with other students and learning from other students (aping) would be hugely limited. Even the social skills that students develop in a school setting will be rusted. My own social skills probably have weakened and rusted as I have very limited interaction with people over the past few months due to Covid fears and would be at least for the next few years till a large enough population is not vaccinated.

26 December 2020

Russ Allbery: Review: The Biggest Bluff

Review: The Biggest Bluff, by Maria Konnikova
Publisher: Penguin Press
Copyright: 2020
ISBN: 0-525-52263-8
Format: Kindle
Pages: 335
After a particularly unlucky year for her family, Maria Konnikova was reading about the balance between luck and control in life and discovered, to her surprise, that John von Neumann, one of the foundational thinkers of computer science, was fascinated by poker. He found most card games boring because they relied on luck. Poker, however, he thought was the perfect balance between luck and skill: enough skill to make its effect undeniable, but enough luck that one could not control the game fully regardless of skill. Konnikova decided on a research project: spend one year learning No Limit Texas Hold'em from one of the best poker players in the world, with a goal of competing in the World Series of Poker. She had studied the description-experience gap during her doctoral research in psychology and wanted to see if the experience of randomness in poker would teach her something the description of the randomness of life could not. Before starting this project, she didn't know the basic rules and had never watched a game. Erik Seidel agreed to mentor her, and The Biggest Bluff is her account of that experience. This book is simultaneously frustrating and fascinating in ways that I don't think can be untangled without making it a far different book. Fitting, I suppose, for a book about how our brains entangle luck and skill. First, if you're looking for a book about poker play, this is not one. Konnikova rarely talks about specific hands or tournaments in more detail than her overall trajectory. That was a disappointment. In the few places she does describe some of her betting decisions, analysis of the other players, and tournament strategy, her accounts are engrossing and suspenseful. I would have happily read a book chronicling her poker tournaments and the decisions she made, but this is not that book. What The Biggest Bluff is instead is a psychological and philosophical examination of the process of learning poker. Konnikova uses her experiences as launching points into philosophical digressions. Even the lessons that have limited surface utility outside of poker, such as learning to suppress body language to avoid giving away information, turn into digressions about interpersonal dynamics and personality types. There are interesting tidbits here, but I've read a lot of popular psychology and was more interested in the poker. My frequent reading experience was impatiently waiting for Konnikova to finish lecturing and get back to her narrative. What Konnikova does do though, at a level that I haven't seen before before in a book of this type, is be brutally honest about her mistakes and her learning process. And I do mean brutally: The book opens with her throwing up in a casino bathroom. (This is not a fun book to read if you don't like reading about stress reactions and medical problems. There aren't many of them, but they're... memorable.) Konnikova knows and can explain the psychological state she's trying to reach, but still finds it hard to do in the moment. Correctly reacting to probabilities, cutting losses, and neither being too over-confident or too scared is very hard. Most books of this type elide over the repeated failures in a sort of training montage, which makes the process look easier than it feels. Konnikova tries to realistically show the setbacks and failures, and I think succeeds. That relentless introspection and critical honesty is the best part of this book, but I think it's also behind the stream of consciousness digressions about psychology and philosophy. It's a true portrayal of how Konnikova makes sense of the world. A more polished and streamlined book about the poker would have been more dramatically engrossing, but it might have lost the deep examination of how she combines poker with her knowledge of psychology to change her thinking. The one place where I think that self-reflection may fall a bit short, which I want to mention because I thought it was a missed opportunity, is around knowledge of hand probabilities. Konnikova makes a point, early in the book, of not approaching poker through the memorization and mathematics route and instead trying to find a play style that focuses on analyzing the other players and controlling her own emotions. This is a good hook, but by the end of the book it's not entirely true. The point that I think she was trying to make is that her edge against other poker players at her same level comes more from psychology than from calculation of precise odds in rare situations. This is true. But by the time she reaches high levels of play, she is using statistical simulators, practice tools, and intensive study just like any professional poker player. There is a minimum level of pure knowledge and memorization required that cannot be avoided. It's clear from the few things she says about this that those tools became more interesting to her as she became better at poker, and I wish she would have dug more into why and how that happened. How much of her newfound ability to make decisions and stick to a plan comes from emotional changes, and how much from that background store of confident knowledge? Or maybe those are different ways of looking at the same change? I did appreciate Konnikova's explicit acknowledgment at the end of the book that poker did not, in the end, provide some deep insight into the balance of luck and skill in real life. Learning poker instead gave Konnikova more personal ability to make a plan for the things that she can control and let go of the things she can't. I'm glad that worked for her, but since reading this book I have noticed former poker players who think life is more knowable than it is. Poker combines random chance with psychological play against other people, but it does so in a way and to an extent that is quantifiable. You can make the correct play and still lose a hand, but you can also know when this has happened. When you're used to analyzing the world through that frame and real life fails to provide that certainty, it's tempting to impose it anyway and insist your simplified models are more accurate because they're more comprehensible. But poker is a game, not a model; being more predictable and more constrained than real life is part of what makes it fun. The skill that Konnikova learned from it has a potential downside that she doesn't talk about. I'm not sure how to sum up this book. Konnikova's internal analysis and honesty is truly admirable and illuminating, but it left me wanting to read a different book that was more focused on poker narration. I know there are lots of those books out there, but I doubt they would be written with Konnikova's self-awareness and lack of ego. However, they would probably also lack the moments that made me cringe or that were deeply uncomfortable to read. My feelings are mixed. But if you want popular psychology wrapped around a deeply honest account of the process of learning poker, I suspect this book is one of a kind. Rating: 7 out of 10

25 December 2020

John Goerzen: So Many Caring People In This World

When I was a boy and I would see scary things in the news, my mother would say to me, Look for the helpers. You will always find people who are helping. To this day, especially in times of disaster, I remember my mother s words and I am always comforted by realizing that there are still so many helpers so many caring people in this world. Fred Rogers
This photo doesn t have amazing lighting or fantastic composition. In fact, it looks ordinary. It s only when you know its story that the beauty shines through. In 2019, Fremont, NE had been cut off by flooding. People were trying to get in and out of the town, and couldn t. The only way in or out was from the small airport. About 50 pilots from Nebraska and Iowa (and me from Kansas) came over to help. Every plane you see here and more that I couldn t fit in the frame was at Millard Airport, Omaha, flying people and supplies into and out of Fremont. Estimates are that we flew over 1000 people and tons of supplies that weekend. All safely. I remember flying the family with a 1-week-old baby that had gone to Omaha for a doctor appointment and then couldn t get home for three days. I remember the elderly couple and their dog that I flew out of Fremont, the former Marine riding in the co-pilot seat next to me, cracking jokes with me as we went. I remember the group of ladies that were laughing as I gave them the required seat belt briefing, the mom with her kids, the man that had to get to work, and packed a few day s supplies in his backpack, unsure when he d be able to get back home. It was the first some had ever been in a plane. There were so many helpers. People all over Omaha brought supplies to Millard airport. Others had just shown up at both airports to help organize and make lists of passengers and match them up with different size planes. Someone at Millard had a trailer and golf cart for supplies. When I d land at Fremont, before I was even out of the plane, highschoolers had already swarmed it and were helping to unload supplies or help passengers out. One time when I got back to Millard, I found lunch: a pizza place had donated pizza for the pilots, and then a restaurant in Fremont did too. With so many extra planes in the sky, Omaha ATC was slammed and still did a fantastic job. I of course took no photos of the people I carried, but I have thought of them often in the last year. This photo is of Millard airport, Omaha, loading up supplies. Look at all these helpers. I think it s one of the most beautiful sights I ve ever seen. Today there are billions of helpers in this world. The obvious ones: the truck drivers, the health care workers, the grocery store workers. And also the non-obvious ones: all the people that are wearing masks, practicing social distancing, forgoing Christmas gatherings, for the good of others, despite the hardship and heartbreak it may cause. If you didn t know the story of this photo, you wouldn t know these planes were all helpers in time of disaster. We don t know the story of all the people we see in our world today, but chances are good that many of them are helpers also. When you know their story, the beauty shines through. May we all be able to see the beauty that still surrounds us this Christmas.

23 December 2020

John Goerzen: How & Why To Use Airgapped Backups

A good backup strategy needs to consider various threats to the integrity of data. For instance: It s that last one that is of particular interest today. A lot of backup strategies are such that if a user (or administrator) has their local account or network compromised, their backups could very well be destroyed as well. For instance, do you ssh from the account being backed up to the system holding the backups? Or rsync using a keypair stored on it? Or access S3 buckets, etc? It is trivially easy in many of these schemes to totally ruin cloud-based backups, or even some other schemes. rsync can be run with delete (and often is, to prune remotes), S3 buckets can be deleted, etc. And even if you try to lock down an over-network backup to be append-only, still there are vectors for attack (ssh credentials, OpenSSL bugs, etc). In this post, I try to explore how we can protect against them and still retain some modern conveniences. A backup scheme also needs to make a balance between: My story so far About 20 years ago, I had an Exabyte tape drive, with the amazing capacity of 7GB per tape! Eventually as disk prices fell, I had external disks plugged in to a server, and would periodically rotate them offsite. I ve also had various combinations of partial or complete offsite copies over the Internet as well. I have around 6TB of data to back up (after compression), a figure that is growing somewhat rapidly as I digitize some old family recordings and videos. Since I last wrote about backups 5 years ago, my scheme has been largely unchanged; at present I use ZFS for local and to-disk backups and borg for the copies over the Internet. Let s take a look at some options that could make this better. Tape The original airgapped backup. You back up to a tape, then you take the (fairly cheap) tape out of the drive and put in another one. In cost per GB, tape is probably the cheapest medium out there. But of course it has its drawbacks. Let s start with cost. To get a drive that can handle capacities of what I d be needing, at least LTO-6 (2.5TB per tape) would be needed, if not LTO-7 (6TB). New, these drives cost several thousand dollars, plus they need LVD SCSI or Fibre Channel cards. You re not going to be hanging one off a Raspberry Pi; these things need a real server with enterprise-style connectivity. If you re particularly lucky, you might find an LTO-6 drive for as low as $500 on eBay. Then there are tapes. A 10-pack of LTO-6 tapes runs more than $200, and provides a total capacity of 25TB sufficient for these needs (note that, of course, you need to have at least double the actual space of the data, to account for multiple full backups in a set). A 5-pack of LTO-7 tapes is a little more expensive, while providing more storage. So all-in, this is going to be in the best possible scenario nearly $1000, and possibly a lot more. For a large company with many TB of storage, the initial costs can be defrayed due to the cheaper media, but for a home user, not so much. Consider that 8TB hard drives can be found for $150 $200. A pair of them (for redundancy) would run $300-400, and then you have all the other benefits of disk (quicker access, etc.) Plus they can be driven by something as cheap as a Raspberry Pi. Fancier tape setups involve auto-changers, but then you re not really airgapped, are you? (If you leave all your tapes in the changer, they can generally be selected and overwritten, barring things like hardware WORM). As useful as tape is, for this project, it would simply be way more expensive than disk-based options. Fundamentals of disk-based airgapping The fundamental thing we need to address with disk-based airgapping is that the machines being backed up have no real-time contact with the backup storage system. This rules out most solutions out there, that want to sync by comparing local state with remote state. If one is willing to throw storage efficiency out the window maybe practical for very small data sets one could just send a full backup daily. But in reality, what is more likely needed is a way to store a local proxy for the remote state. Then a runner device (a USB stick, disk, etc) could be plugged into the network, filled with queued data, then plugged into the backup system to have the data dequeued and processed. Some may be tempted to short-circuit this and just plug external disks into a backup system. I ve done that for a long time. This is, however, a risk, because it makes those disks vulnerable to whatever may be attacking the local system (anything from lightning to ransomware). ZFS ZFS is, it should be no surprise, particularly well suited for this. zfs send/receive can send an incremental stream that represents a delta between two checkpoints (snapshots or bookmarks) on a filesystem. It can do this very efficiently, much more so than walking an entire filesystem tree. Additionally, with the recent addition of ZFS crypto to ZFS on Linux, the replication stream can optionally reflect the encrypted data. Yes, as long as you don t need to mount them, you can mostly work with ZFS datasets on an encrypted basis, and can directly tell zfs send to just send the encrypted data instead of the decrypted data. The downside of ZFS is the resource requirements at the destination, which in terms of RAM are higher than most of the older Raspberry Pi-style devices. Still, one could perhaps just save off zfs send streams and restore them later if need be, but that implies a periodic resend of a full stream, an inefficient operation. dedpulicating software such as borg could be used on those streams (though with less effectiveness if they re encrypted). Tar Perhaps surprisingly, tar in listed incremental mode can solve this problem for non-ZFS users. It will keep a local cache of the state of the filesystem as of the time of the last run of tar, and can generate new tarballs that reflect the changes since the previous run (even deletions). This can achieve a similar result to the ZFS send/receive, though in a much less elegant way. Bacula / Bareos Bacula (and its fork Bareos) both have support for a FIFO destination. Theoretically this could be used to queue of data for transfer to the airgapped machine. This support is very poorly documented in both and is rumored to have bitrotted, however. rdiff and xdelta rdiff and xdelta can be used as sort of a non-real-time rsync, at least on a per-file basis. Theoretically, one could generate a full backup (with tar, ZFS send, or whatever), take an rdiff signature, and send over the file while keeping the signature. On the next run, another full backup is piped into rdiff, and on the basis of the signature file of the old and the new data, it produces a binary patch that can be queued for the backup target to update its stored copy of the file. This leaves history preservation as an exercise to be undertaken on the backup target. It may not necessarily be easy and may not be efficient. rsync batches rsync can be used to compute a delta between two directory trees and express this as a single-file batch that can be processed by a remote rsync. Unfortunately this implies the sender must always keep an old tree around (barring a solution such as ZFS snapshots) in order to compute the delta, and of course it still implies the need for history processing on the remote. Getting the Data There OK, so you ve got an airgapped system, some sort of runner device for your sneakernet (USB stick, hard drive, etc). Now what? Obviously you could just copy data on the runner and move it back off at the backup target. But a tool like NNCP (sort of a modernized UUCP) offer a lot of help in automating the process, returning error reports, etc. NNCP can be used online over TCP, over reliable serial links, over ssh, with offline onion routing via intermediaries or directly, etc. Imagine having an airgapped machine at a different location you go to frequently (workplace, friend, etc). Before leaving, you put a USB stick in your pocket. When you get there, you pop it in. It s despooled and processed while you want, and return emails or whatever are queued up to be sent when you get back home. Not bad, eh? Future installment I m going to try some of these approaches and report back on my experiences in the next few weeks.

22 December 2020

Andrej Shadura: Vendoring Rust dependencies for a Debian derivative

Recently, I needed to package a Rust crate libslirp for a Apertis, a Debian derivative. libslirp is used by the newly release UML backend of debos, our Debian image build tool. Unfortunately, this crate hasn t yet been properly packaged for Debian proper, so I could not simply pull the packaging from Debian. Even worse, its build dependencies haven t all been packaged yet. Most importantly, I have only uploaded zbus to Debian today, and at that time none of its dependencies were in Debian either. Another issue with this were that each crate is packaged for Debian as a separate package, making the process a bit more tricky since I d need to import all of the crates into Apertis separately. Doing that takes time and is further complicated by the CI loop we re using which requires the full build process to complete for a package before a pipeline a dependent package can run. Having all that considered, I took a shortcut: I vendored all the build dependencies with the package itself, and here s how. I started with debcargo-conf, where the not-yet-complete packaging lives. With debcargo, I generated the source package as if I were to upload to Debian. Since I ll be vendoring the dependencies, I have removed most of the Rust dependencies and replaced them with those extra dependencies needed by those crates. So, I replaced this: 
Build-Depends: debhelper (>= 12),
 dh-cargo (>= 24),
 cargo:native,
 rustc:native,
 libstd-rust-dev,
 librust-enumflags2-0.6+default-dev (>= 0.6.4-~~),
 librust-ipnetwork-0.17+default-dev,
 librust-lazy-static-1+default-dev (>= 1.4-~~),
 librust-libc-0.2+default-dev,
 librust-libslirp-sys-4+default-dev (>= 4.2.0-~~),
 librust-libsystemd-0.2+default-dev,
 librust-mio-0.6+default-dev,
 librust-mio-extras-2+default-dev (>= 2.0.5-~~),
 librust-nix-0.17+default-dev,
 librust-slab-0.4+default-dev,
 librust-structopt-0.3+default-dev,
 librust-url-2+default-dev (>= 2.1-~~),
 librust-zbus-1+default-dev,
 librust-zvariant-2+default-dev
By this: 
Build-Depends: debhelper (>= 12),
 dh-cargo (>= 17),
 cargo:native,
 rustc:native,
 libstd-rust-dev,
 pkg-config,
 libglib2.0-dev,
 libslirp-dev
I wanted to keep the delta to the future Debian package at a bare minimum, so I kept the original dh-cargo build process, only making sure the vendored dependencies are found in the right place. The cleanest approach in my view was to use the fact that the 3.0 (quilt) source format allows multiple original tarballs: the main tarball would be the same orig tarball as in Debian, and an extra tarball would contain the vendored code. When we sync with Debian, the extra tarball would be dropped, just as the adjustments to the debian/rules. To generate the vendor tarball, I added this to the rules file: 
include /usr/share/dpkg/pkg-info.mk
vendor:
        -[ -d vendor ] && rm -rf vendor
        rm -rf Cargo.lock
        cargo vendor
        tar Jcf ../$(DEB_SOURCE)_$(DEB_VERSION_UPSTREAM).orig-vendor.tar.xz vendor/
.PHONY: vendor
This target, being run manually and not as part of the build process, download the latest dependencies satisfying the requirements in Cargo.toml and rolls them into a vendor tarball. Now, to make dh-cargo use the vendored code, we need to link it into the right place. dh-cargo creates a Cargo registry tree in a subdirectly under debian/ where it symlinks crates from the Rust development packages. This extra code makes sure the vendored dependencies are available there too: 
override_dh_auto_configure:
        dh_auto_configure $@
        for d in vendor/* ; \
        do \
                [ -L debian/cargo_registry/$$(basename $$d) ] && rm debian/cargo_registry/$$(basename $$d) ; \
                ln -rs $$d debian/cargo_registry ; \
        done
An important thing here is that if the build happens on a non-clean system, dh-cargo may have symlinked a system crate already installed so we need to remove that symlink and replace it by ours. We build packages in clean chroots, so this is not that important, otherwise for reproducibility reasons it would be better to create the registry directory from scratch. The final package can be found at the Apertis GitLab.

21 December 2020

Russell Coker: MPV vs Mplayer

After writing my post about VDPAU in Debian [1] I received two great comments from anonymous people. One pointed out that I should be using VA-API (also known as VAAPI) on my Intel based Thinkpad and gave a reference to an Arch Linux Wiki page, as usual Arch Linux Wiki is awesome and I learnt a lot of great stuff there. I also found the Debian Wiki page on Hardware Video Acceleration [2] which has some good information (unfortunately I had already found all that out through more difficult methods first, I should read the Debian Wiki more often. It seems that mplayer doesn t suppoer VAAPI. The other comment suggested that I try the mpv fork of Mplayer which does support VAAPI but that feature is disabled by default in Debian. I did a number of tests on playing different videos on my laptop running Debian/Buster with Intel video and my workstation running Debian/Unstable with ATI video. The first thing I noticed is that mpv was unable to use VAAPI on my laptop and that VDPAU won t decode VP9 videos on my workstation and most 4K videos from YouTube seem to be VP9. So in most cases hardware decoding isn t going to help me. The Wikipedia page about Unified Video Decoder [3] shows that only VCN (Video Core Next) supports VP9 decoding while my R7-260x video card [4] has version 4.2 of the Unified Video Decoder which doesn t support VP9, H.265, or JPEG. Basically I need a new high-end video card to get VP9 decoding and that s not something I m interested in buying now (I only recently bought this video card to do 4K at 60Hz). The next thing I noticed is that for my combination of hardware and software at least mpv tends to take about 2/3 the CPU time to play videos that mplayer does on every video I tested. So it seems that using mpv will save me 1/3 of the power and heat from playing videos on my laptop and save me 1/3 of the CPU power on my workstation in the worst case while sometimes saving me significantly more than that. Conclusion> To summarise quite a bit of time experimenting with video playing and testing things: I shouldn t think too much about hardware decoding until VP9 hardware is available (years for me). But mpv provides some real benefits right now on the same hardware, I m not sure why.
Related posts:
  1. Testing VDPAU in Debian VDPAU is the Video Decode and Presentation API for Unix...
  2. Video Decoding I ve had a saga of getting 4K monitors to work...
  3. 4K Monitors A couple of years ago a relative who uses a...

20 December 2020

Shirish Agarwal: Insane Logic and Farming in other countries

The people who are pro-Government and in this case pro-Corporate do not have any success stories that they can share. Hence, most of the time the arguments are that the other are bad. For e.g. quite a few people argue that we don t need farmers, we can just order from restaurant. They have completely disassociated the idea that even then you need farmers as unless the farmers put the seed in, till the soil and wait for the rains or have irrigation you won t get ripe vegetables which then has to be taken out, and somehow sold to the wholesaler from where it comes to the restaurant and then to your plate. Sadly, even the farm-to-fork infographics are so depressingly sad, you want to look away. If you see the infographic you see it is just not non-veg but also vegetarian food grains which go under lot of questionable practices. Even, with such scenarios that is done by corporations our people want to go ahead. I will share stories from other countries which tell how they are doing more. The Soldier-Farmer Another sad part of these protests have been soldiers who have been returning their medals. The ones who oppose have the gall to say they should return the cash rewards they got. So just like farmers, seems soldiers also do not need money to survive. They are supposed to live only on air and water. This is after the present Govt. has reduced their pensions after retirement and that too without any discussion
GOI pensions to ex-servicemen
Now I nor anybody else would have minded if these conditions were shared going forward rather than doing it retrospectively. People who usually go to the army are not in it for money but for the adventure and glory they bring. But they do also have a family and have a family responsibility. In most other countries, the soldier and his families are well-looked after. If you know that even after you die, the Government would look after your family, you will do your best. Unfortunately, many veterans in India themselves are asked to help by many war widows as the widows don t get family pensions. The proposal naturally has left many miffed. In fact many of the veterans who used to advise people to join the armed forces now advise young people to pursue civilian life and careers. This is when Indian Army has been ironically having shortage of officers from well over a decade and stresses felt by Army personnel also known for a long time. Even under this nationalistic Government, if it cannot take care of its soldiers, then forget about others.
India Defence Spending vis-a-vis other countries.
Now it is nobody s argument that India needs to improve its tooth-to-tail ratio but this is the wrong way to go about it. I would probably talk about that some other time as that totally needs its whole place. Even OROP, which was the mandate of this Government hasn t had been done in full as there are quite a few cases in the Supreme Court. Almost all the cases have been heard and only decisions have to be given which the SC for whatever reason doesn t want to give. They just keep changing the date of the hearing. Nowadays, in many suits/cases, the SC asks for fresh hearings even though all the old records are there. This is a newish phenomena which is being observed in SC. Why is it being done? Your guess is as good as mine. One thing for sure has changed, the SC which used to be citizen-focussed or enabler of human rights and used to be held as a beacon for judicial activism has changed but these are other topics which need their own space. Update 16/12/2020 The SC recommends setting up a committee to discuss farmer issues. And this is nothing new. This is called death by committee. When there is already so much literature on the subject, including the works done by Swaminathan Commission. There has been 6 reports which do look at farmer issues in a holistic manner. This is the Supreme Court giving an escape route to GOI. They also have abstained from having a whole session citing Covid. This is when the ruling Govt. is putting a massive 1000 crore on a new building on which the SC has put on hold. And even then the GOI went ahead and did a Bhoomi-Pujan (traditional ceremony when making a new construction from scratch.) Naturally due to the double whammy of both the pension reforms and now the laws to make corporate farming more aggressive has left a deep impact on the soldier-farmer that the state does not think or feel for him. Even the United States farm-aid eloquently describes how corporate farming has made independent farmers suffer. You read that, and it seems it is as the state of our farmers here in India. Even their average land-holding has dropped a bit. I have shared about the state of farmers in India, in two blog posts previously. And it is not just farm owners who have had it bad, even farm workers in U.S. The issue may look to be about the pandemic but goes far deeper. The Israeli Model The Israelis have always used collective farming and do have a large share in farming there. The old model called Kibbutz is what made Israelis self-sufficient in food and water and actually are world-leaders where they export their services to other nations on the same thing. France Just like many other countries, France also seems to have favored farmer co-operatives. Almost 75% of all farmers are in co-operatives. Italy The country world-famous for its wines and cheese are made by its co-operatives. In fact co-ops are the buzzword it seems in Italy, more so in Northern Italy. Asian economies Even Asian economies, especially East Asian economies by and large have been turning to co-operatives. Brazil Now Brazil is almost 40% more than India. In fact, in most of the indices, Brazil beats India handsomely. So one would be forgiven to think that Brazil must have corporate farming. But nothing could be further from the truth. The only downer is that they have high crime in some areas. Otherwise, they are in many ways better than India. In fact, I was surprised a few years ago to learn about Mercsour. I would have to admit though I learned much about Brazil when Debian was holding a debconf about a year back. Otherwise, I had known about the country for number of years but apart from its carnival and samba, hadn t known much about it. I did come to know that most of Latin America also loves spices as much as Indians do. They show that love by using hot sauces. I do one day wanna try one of their sauces to see what makes it tick. I do know they like to barbecue vegetables as much as barbecuing non-veg food. This is going a bit OT but then that s the foodie in me Conclusion I could have shared more countries which have chosen the co-operative way rather than corporate farming and that is simply because they know what is best for their people and what is best even politically. The new farm laws are neither grounded in farmer s welfare nor anything else. The Govt. has been trying to undermine the farmers for years together. In fact, Madhya Pradesh has openly said that they will not allow farmers from other states to sell in their state. Although, even before these laws there was nothing to restrict the farmer from selling his produce anywhere in the country. Angering the farmers is not good politics as was found sometime back but guessing some lessons need to be re-learned. One comment though, on social media I have seen many people especially youngsters having no real understanding of what inflation is all about. For e.g. if you ask them how come we are having a sort of record inflation in a technical recession (there has been a contraction, actually) and you see them putting themselves into bigger and bigger ditches. This does explain in part why the BJP wins in elections. If you do more rhetoric, which BJP is good as, rather than educating people than you are bound to win. You don t need plans, you don t need a vision, just rhetoric will do. What more evidence is needed when the economy is and was in a worse shape even before the pandemic and BJP won. I would probably write about that as that again needs lot of background and understanding as well as related terms.

18 December 2020

Russell Coker: Thinkpad Storage Problem

For a while I ve had a problem with my Thinkpad X1 Carbon Gen 1 [1] where storage pauses for 30 seconds, it s become more common recently and unfortunately everything seems to depend on storage (ideally a web browser playing a video from the Internet could keep doing so with no disk access, but that s not what happens). 
echo 3 > /sys/block/sda/device/timeout
I ve put the above in /etc/rc.local which should make it only take 3 seconds to recover from storage problems instead of 30 seconds, the problem hasn t recurred since that change so I don t know if it works as desired. The Thinkpad is running BTRFS and no data is reported as being lost, so it s not a problem to use it at this time. The bottom of this post has an example of the errors I m getting. A friend said that his searches for such errors shows people claiming that it s a SATA controller or cable problem, which for a SATA device bolted directly to the motherboard means it s likely to be a motherboard problem (IE more expensive to fix than the $289 I paid for the laptop almost 3 years ago). A Lenovo forum discussion says that the X1 Carbon Gen1 uses an unusual connector that s not M.2 SATA and not mSATA [2]. This means that a replacement is going to be expensive, $100US for a replacement from eBay when a M.2 SATA or NVMe device would cost about $50AU from my local store. It also means that I can t use a replacement for anything else. If my laptop took a regular NVMe I d buy one and test it out, if it didn t solve the problem a spare NVMe device is always handy to have. But I m not interested in spending $100US for a part that may turn out to be useless. I bought the laptop expecting that there was nothing I could fix inside it. While it is theoretically possible to upgrade the CPU etc in a laptop most people find that the effort and expense makes it better to just replace the entire laptop. With the ultra light category of laptops the RAM is soldered on the motherboard and there are even less options for replacing things. So while it s annoying that Lenovo didn t use a standard interface for this device (they did so for other models in the range) it s not a situation that I hadn t expected. Now the question is what to do next. The Thinkpad has a SD socket and micro SD cards are getting large capacities nowadays. If it won t boot from a SD card I could boot from USB and then run from SD. So even if the internal SATA device fails entirely it should still be useful. I wonder if I could find a Thinkpad with a similar problem going cheap as I m pretty sure that a Windows user couldn t make any storage device usable the way I can with Linux. I ve looked at prices on auction sites but haven t seen a Thinkpad X1 Carbon going for anywhere near the price I got this one. The nearest I ve seen is around $350 for a laptop with significant damage. While I am a little unhappy at Lenovo s choice of interface, I ve definitely got a lot more than $289 of value out of this laptop. So I m not really complaining. 
[315041.837612] ata1.00: status:   DRDY  
[315041.837613] ata1.00: failed command: WRITE FPDMA QUEUED
[315041.837616] ata1.00: cmd 61/20:48:28:1e:3e/00:00:00:00:00/40 tag 9 ncq dma 
16384 out
                         res 40/00:01:00:00:00/00:00:00:00:00/e0 Emask 0x4 
(timeout)                
[315041.837617] ata1.00: status:   DRDY  
[315041.837618] ata1.00: failed command: READ FPDMA QUEUED
[315041.837621] ata1.00: cmd 60/08:50:e0:26:84/00:00:00:00:00/40 tag 10 ncq 
dma 4096 in
                         res 40/00:01:00:00:00/00:00:00:00:00/e0 Emask 0x4 
(timeout)                
[315041.837622] ata1.00: status:   DRDY  
[315041.837625] ata1: hard resetting link
[315042.151781] ata1: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[315042.163368] ata1.00: ACPI cmd ef/02:00:00:00:00:a0 (SET FEATURES) 
succeeded
[315042.163370] ata1.00: ACPI cmd f5/00:00:00:00:00:a0 (SECURITY FREEZE LOCK) 
filtered out
[315042.163372] ata1.00: ACPI cmd ef/10:03:00:00:00:a0 (SET FEATURES) filtered 
out
[315042.183332] ata1.00: ACPI cmd ef/02:00:00:00:00:a0 (SET FEATURES) 
succeeded
[315042.183334] ata1.00: ACPI cmd f5/00:00:00:00:00:a0 (SECURITY FREEZE LOCK) 
filtered out
[315042.183336] ata1.00: ACPI cmd ef/10:03:00:00:00:a0 (SET FEATURES) filtered 
out
[315042.193332] ata1.00: configured for UDMA/133
[315042.193789] sd 0:0:0:0: [sda] tag#10 FAILED Result: hostbyte=DID_OK 
driverbyte=DRIVER_SENSE
[315042.193791] sd 0:0:0:0: [sda] tag#10 Sense Key : Illegal Request [current] 
[315042.193793] sd 0:0:0:0: [sda] tag#10 Add. Sense: Unaligned write command
[315042.193795] sd 0:0:0:0: [sda] tag#10 CDB: Read(10) 28 00 00 84 26 e0 00 00 
08 00
[315042.193797] print_req_error: I/O error, dev sda, sector 8660704
[315042.193810] ata1: EH complete
Related posts:
  1. Thinkpad X1 Carbon Gen 6 In February I reviewed a Thinkpad X1 Carbon Gen 1...
  2. Thinkpad X1 Carbon I just bought a Thinkpad X1 Carbon to replace my...
  3. More About the Thinkpad X301 Last month I blogged about the Thinkpad X301 I got...

17 December 2020

Dirk Eddelbuettel: RcppEigen 0.3.3.9.1: New upstream

A new release 0.3.3.9.1 of RcppEigen arrived on CRAN today (and just went to Debian too) bringing support for Eigen 3.3.9 to R. Eigen is a C++ template library for linear algebra: matrices, vectors, numerical solvers, and related algorithms. The release started with a gentle nudge from the Stan team to package Eigen 3.3.8. And once I got around to it and sat down, Eigen 3.3.9 had just been released the previous day. The freshly prepared release 0.3.3.9.0 was then announced on Twitter (and the rcpp-devel mailing list) along with a request for testing alongside the usual extended reverse dependency checking we do (just like CRAN asks us to). And it turns out that Eigen 3.3.9 required an update to StanHeaders, a CRAN package that is itself quite widely used. This took a couple of days and its round of testing, but Ben Goodrich shipped it to CRAN last night, so we were able to ship RcppEigen in turn today. So this release once again bring few required changes to Eigen (see this directory for details). As we said for the previous three releases:
One additional and recent change was the accomodation of a recent CRAN Policy change to not allow gcc or clang to mess with diagnostic messages. A word of caution: this may make your compilation of packages using RcppEigen very noisy so consider adding -Wno-ignored-attributes to the compiler flags added in your ~/.R/Makevars.
I actually find this requirement rather annoying. Eigen is then only usable if you set, say, 
-Wno-deprecated-declarations -Wno-parentheses -Wno-ignored-attributes -Wno-unused-function
as g++ options in ~/.R/Makevars. But CRAN makes the rules. Maybe if a few of us gently and politely nudge them they may relent one day. One can only hope. The complete NEWS file entry follows.

Changes in RcppEigen version 0.3.3.9.1 (2020-12-17)
  • Upgraded to Eigen 3.3.9 (Dirk in #92 fixing #91).
  • Added GitHub Actions CI using run.sh from r-ci (Dirk)

Courtesy of CRANberries, there is also a diffstat report for the most recent release. If you like this or other open-source work I do, you can sponsor me at GitHub.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

16 December 2020

Jonathan McDowell: DeskPi Pro + 8GB Pi 4

DeskPi Pro Raspberry Pi case Despite having worked on a number of ARM platforms I ve never actually had an ARM based development box at home. I have a Raspberry Pi B Classic (the original 256MB rev 0002 variant) a coworker gave me some years ago, but it s not what you d choose for a build machine and generally gets used as a self contained TFTP/console server for hooking up to devices under test. Mostly I ve been able to do kernel development with the cross compilers already built as part of Debian, and either use pre-built images or Debian directly when I need userland pieces. At a previous job I had a Marvell MACCHIATObin available to me, which works out as a nice platform - quad core A72 @ 2GHz with 16GB RAM, proper SATA and a PCIe slot. However they re still a bit pricey for a casual home machine. I really like the look of the HoneyComb LX2 - 16 A72 cores, up to 64GB RAM - but it s even more expensive. So when I saw the existence of the 8GB Raspberry Pi 4 I was interested. Firstly, the Pi 4 is a proper 64 bit device (my existing Pi B is ARMv6 which means it needs to run Raspbian instead of native Debian armhf), capable of running an upstream kernel and unmodified Debian userspace. Secondly the Pi 4 has a USB 3 controller sitting on a PCIe bus rather than just the limited SoC USB 2 controller. It s not SATA, but it s still a fairly decent method of attaching some storage that s faster/more reliable than an SD card. Finally 8GB RAM is starting to get to a decent amount - for a headless build box 4GB is probably generally enough, but I wanted some headroom. The Pi comes as a bare board, so I needed a case. Ideally I wanted something self contained that could take the Pi, provide a USB/SATA adaptor and take the drive too. I came across the pre-order for the DeskPi Pro, decided it was the sort of thing I was after, and ordered one towards the end of September. It finally arrived at the start of December, at which point I got round to ordering a Pi 4 from CPC. Total cost ~ 120 for the case + Pi.

The Bad First, let s get the bad parts out of the way. Broken USB port (right) I managed to break a USB port on the Desk Pi. It has a pair of forward facing ports, I plugged my wireless keyboard dongle into it and when trying to remove it the solid spacer bit in the socket broke off. I ve never had this happen to me before and I ve been using USB devices for 20 years, so I m putting the blame on a shoddy socket. The first drive I tried was an old Crucial M500 mSATA device. I have an adaptor that makes it look like a normal 2.5 drive so I used that. Unfortunately it resulted in a boot loop; the Pi would boot its initial firmware, try to talk to the drive and then reboot before even loading Linux. The DeskPi Pro comes with an m2 adaptor and I had a spare m2 drive, so I tried that and it all worked fine. This might just be power issues, but it was an unfortunate experience especially after the USB port had broken off. (Given I ended up using an M.2 drive another case option would have been the Argon ONE M.2, which is a bit more compact.)

The Annoying DeskPi Pro without rear bezel The case is a little snug; I was worried I was going to damage things as I slid it in. Additionally the construction process is a little involved. There s a good set of instructions, but there are a lot of pieces and screws involved. This includes a couple of FFC cables to join things up. I think this is because they ve attempted to make a compact case rather than allowing a little extra room, and it does have the advantage that once assembled it feels robust without anything loose in it. DeskPi Pro with rear bezel and USB3 dongle I hate the need for an external USB3 dongle to bridge from the Pi to the USB/SATA adaptor. All the cases I ve seen with an internal drive bay have to do this, because the USB3 isn t brought out internally by the Pi, but it just looks ugly to me. It s hidden at the back, but meh. Fan control is via a USB/serial device, which is fine, but it attaches to the USB C power port which defaults to being a USB peripheral. Raspbian based kernels support device tree overlays which allows easy reconfiguration to host mode, but for a Debian based system I ended up rolling my own dtb file. I changed 
#include "bcm283x-rpi-usb-peripheral.dtsi"
to 
#include "bcm283x-rpi-usb-host.dtsi"
in arch/arm/boot/dts/bcm2711-rpi-4-b.dts and then I did: 
cpp -nostdinc -I include -I arch -undef -x assembler-with-cpp \
    arch/arm/boot/dts/bcm2711-rpi-4-b.dts > rpi4.preprocessed
dtc -I dts -O dtb rpi4.preprocessed -o bcm2711-rpi-4-b.dtb
and the resulting bcm2711-rpi-4-b.dtb file replaced the one in /boot/firmware. This isn t a necessary step if you don t want to use the cooling fan in the case, or the front USB ports, and it s not really anyone s fault, but it was an annoying extra step to have to figure out. The DeskPi came with a microSD card that was supposed to have RaspiOS already on it. It didn t, it was blank. In my case that was fine, because I wanted to use Debian, but it was a minor niggle.

The Good I used Gunnar s pre-built Pi Debian image and it Just Worked; I dd d it to the microSD as instructed and the Pi 4 came up with working wifi, video and USB enabling me to get it configured for my network. I did an apt upgrade and got updated to the Buster 10.7 release, as well as the latest 5.9 backport kernel, and everything came back without effort after a reboot. It s lovely to be able to run Debian on this device without having to futz around with self-compiled kernels. The DeskPi makes a lot of effort to route things externally. The SD slot is brought out to the front, making it easy to fiddle with the card contents without having to open the case to replace it. All the important ports are brought out to the back either through orientation of the Pi, or extenders in the case. That means the built in Pi USB ports, the HDMI sockets (conveniently converted to full size internally), an audio jack and a USB-C power port. The aforementioned USB3 dongle for the bridge to the drive is the only external thing that s annoying. Thermally things seem good too. I haven t done a full torture test yet, but with the fan off the system is sitting at about 40 C while fairly idle. Some loops in bash that push load up to above 2 get the temperature up to 46 C or so, and turning the fan on brings it down to 40 C again. It s audible, but quieter than my laptop and not annoying. I liked the way the case came with everything I needed other than the Pi 4 and a suitable disk drive. There was an included PSU (a proper USB-C PD device, UK plug), the heatsink/fan is there, the USB/SATA converter is there and even an SD card is provided (though that s just because I had a pre-order). Speaking of the SD, I only needed it for initial setup. Recent Pi 4 bootloaders are capable of booting directly from USB mass storage devices. So I upgraded using the RPi EEPROM Recovery image (which just needs extracted to the SD FAT partition, no need for anything complicated - boot with it and the screen goes all green and you know it s ok), then created a FAT partition at the start of the drive for the kernel / bootloader config and a regular EXT4 partition for root. Copies everything over, updated paths, took out the SD and it all just works happily.

Summary My main complaint is the broken USB port, which feels like the result of a cheap connector. For a front facing port expected to see more use than the rear ports I think there s a reasonable expectation of robustness. However I m an early adopter and maybe future runs will be better. Other than that I m pretty happy. The case is exactly the sort of thing I wanted; I was looking for something that would turn the Pi into a box that can sit on my desk on the network and that I don t have to worry about knocking wires out of or lots of cables hooking bits up. Everything being included made it very convenient to get up and running. I still haven t poked the Pi that hard, but first impressions are looking good for it being a trouble free ARM64 dev box in the corner, until I can justify a HoneyComb.

Dirk Eddelbuettel: RcppAnnoy 0.0.18: Tweaks

annoy image A new maintenance release, now at version 0.0.18, of RcppAnnoy is now on CRAN. RcppAnnoy is the Rcpp-based R integration of the nifty Annoy library by Erik Bernhardsson. Annoy is a small and lightweight C++ template header library for very fast approximate nearest neighbours originally developed to drive the famous Spotify music discovery algorithm. This release is follow-up to release 0.0.17 which was made four weeks ago, and which brought the new upstream release 1.17 of Annoy. We now have an updated upstream with a PR by Aaron aiming for improved control of RNG seeding to ensure better reproducibility, along with extended tests and new helpers for RcppAnnoy and Annoy version numbers. The release was once again coordinated with Annoy and James whose BiocNeighbors and uwot packages both consume the Annoy header library shipped here. Detailed changes follow below.

Changes in version 0.0.18 (2020-12-15)
  • Small tweaks to threading policy header defines (Dirk closing #65)
  • Vignette code is again compiled during testing (Aaron Lum and Dirk in #66 addressing #64)
  • Upstream code (with Aaron's PR) was synchronized once more (Dirk in #67)
  • A new helper function was added to report the Annoy version (Aaron in #68)

Courtesy of my CRANberries, there is also a diffstat report for this release. If you like this or other open-source work I do, you can sponsor me at GitHub.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Next.

Previous.