Search Results: "spa"

1 January 2021

Jonathan Wiltshire: WordPress in a subdirectory

For many years now I ve had WordPress installed as a subdirectory to my site but appearing to at the domain level, i.e. /wordpress/index.php is transparently presented as the homepage. This is done by setting the WordPress Address and Site Address settings and then mapping requests which do not match an existing file or directory through as a PHP pathinfo using Apache s mod_rewrite rules in a .htaccess file or server configuration. In this way most of the site is WordPress s dynamic pages and posts, but WordPress itself is neatly contained and random static resources such as /screentest/1024.GIF work as expected. Those .htaccess rules were originally hand-crafted but didn t take account of changing recommendations, e.g. the Authorization header. When I rearranged matters recently I decided to take advantage of WordPress s own generated rules and ditch my old rules. They look like this:

# BEGIN WordPress
# The directives (lines) between "BEGIN WordPress" and "END WordPress" are
# dynamically generated, and should only be modified via WordPress filters.
# Any changes to the directives between these markers will be overwritten.
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteRule .* - [E=HTTP_AUTHORIZATION:% HTTP:Authorization ]
RewriteBase /
RewriteRule ^index\.php$ - [L]
RewriteCond % REQUEST_FILENAME  !-f
RewriteCond % REQUEST_FILENAME  !-d
RewriteRule . /index.php [L]
</IfModule>
# END WordPress

If .htaccess is writable, WordPress adds these rules automatically. Unfortunately, when I switched to them and not my own, the site broke. Crucially the generated rules do not cater for WordPress being in a subdirectory. No matter what combination of settings I tried, local changes to the rules (like RewriteRule . /wordpress/index.php [L]) were rewritten back to the original values at random times and clearly couldn t be relied upon to stay as I need them. This feels like a bug: my intuition is that if the WordPress Address and Site Address settings are different, but within the same domain, the rules should be generated to take care of that. One option is to make the .htaccess file read-only, which WordPress detects and avoids trying to make changes at all. But this rather defeats the object of letting it take care of future changes as the software and specifications change. The second option, which I stuck with, is to add an index.php file with the following contents:

<?php
include('wordpress/index.php');

Now the generated rules, though incorrect, result in the right behaviour and won t break on every setting change.

Russ Allbery: 2020 Book Reading in Review

In 2020, I finished and reviewed 42 books, two more than 2019 (although I had finished but not reviewed two books at the end of 2019, so the total is even more similar than that). This is the best year for reading in terms of book count since 2012, despite being seriously distracted by a new job, a pandemic, and US political meltdowns. Those distractions do show up in the drop in page count. If it weren't for the pandemic, the count would have been higher. Just as I got into a rhythm of reading while I exercised, gyms became a bad idea for the rest of the year. Treadmills are good for reading; long walks around the neighborhood not so much. That time went to podcasts instead, which I'm not too sad about but which don't prompt reviews. Finding the mental space and energy to write reviews proved as much of a challenge as finding time to read this year, and I once again had to do some catch-up at the end of the year. To the extent that I have goals for 2021, it's to tighten up the elapsed time between finishing a book and writing a review so that the reviews don't pile up. I read one book this year that I rated 10 out of 10: Michael Lewis's The Fifth Risk, which is partly about the US presidential transition and is really about what the US government does and what sort of people make careers in civil service. This book is brilliant, fascinating, and surprisingly touching, and I wish it were four times as long. If anything, it's even more relevant today as we enter another transition than it was when Lewis wrote it or when I read it. There were so many 9 out of 10 ratings this year that it's hard to know where to start. I read the last Murderbot novella by Martha Wells (Exit Strategy) and then the first Murderbot novel (Network Effect), both of which were everything I was hoping for. Murderbot's sarcastic first-person voice continues to be a delight, and I expect Network Effect to take home several 2021 awards. I'm eagerly awaiting the next novel, Fugitive Telemetry, currently scheduled for the end of April, 2021. Also on the fiction side were Alix E. Harrow's wonderful debut novel The Ten Thousand Doors of January, a fierce novel about family and claiming power that will hopefully win the 2020 Mythopoeic Award (which was delayed by the pandemic), and TJ Klune's heart-warming The House in the Cerulean Sea, my feel-good novel of the year. Finally, Tamsyn Muir's Gideon the Ninth and Harrow the Ninth were a glorious mess in places, but I had more fun reading and discussing those books than I've had with any novel in a very long time. On the non-fiction side, Tressie McMillan Cottom's Thick is the best collection of sociology that I've read. It's not easy reading, but that book gave me a new-found appreciation and understanding of sociology and what it's trying to accomplish. Gretchen McCulloch's Because Internet is much easier reading but similarly perspective-enhancing, helping me understand (among other things) how choice of punctuation and capitalization expands the dynamic range of meaning in informal text conversation. Finally, Nick Pettigrew's Anti-Social is a funny, enlightening, and sobering look at the process of addressing low-level unwanted behavior that's directly relevant to the current conflicts over the role of policing in society. The full analysis includes some additional personal reading statistics, probably only of interest to me.

Andrej Shadura: Transitioning to a new OpenPGP key

Following dkg s example, I decided to finally transition to my new ed25519/cv25519 key. Unlike Daniel, I m not yet trying to split identities, but I m using this chance to drop old identities I no longer use. My new key only has my main email address and the Debian one, and only those versions of my name I still want around. My old PGP key (at the moment in the Debian keyring) is:

pub   rsa4096/0x6EA4D2311A2D268D 2010-10-13 [SC] [expires: 2021-11-11]
      782130B4C9944247977B82FD6EA4D2311A2D268D
uid                   [ultimate] Andrej Shadura <andrew@shadura.me>
uid                   [ultimate] Andrew Shadura <andrew@shadura.me>
uid                   [ultimate] Andrew Shadura <andrewsh@debian.org>
uid                   [ultimate] Andrew O. Shadoura <Andrew.Shadoura@gmail.com>
uid                   [ultimate] Andrej Shadura <andrewsh@debian.org>
sub   rsa4096/0xB2C0FE967C940749 2010-10-13 [E]
sub   rsa3072/0xC8C5F253DD61FECD 2018-03-02 [S] [expires: 2021-11-11]
sub   rsa2048/0x5E408CD91CD839D2 2018-03-10 [S] [expires: 2021-11-11]

The is the key I ve been using in Debian from the very beginning, and its copies at the SKS keyserver network still have my first DD signature from angdraug:

sig  sig   85EE3E0E 2010-12-03 __________ __________ Dmitry Borodaenko <angdraug@mail.ru>
sig  sig   CB4D38A9 2010-12-03 __________ __________ Dmitry Borodaenko <angdraug@debian.org>

My new PGP key is:

pub   ed25519/0xE8446B4AC8C77261 2016-06-13 [SC] [expires: 2022-06-25]
      83DCD17F44B22CC83656EDA1E8446B4AC8C77261
uid                   [ultimate] Andrej Shadura <andrew@shadura.me>
uid                   [ultimate] Andrew Shadura <andrew@shadura.me>
uid                   [ultimate] Andrej Shadura <andrewsh@debian.org>
uid                   [ultimate] Andrew Shadura <andrewsh@debian.org>
uid                   [ultimate] Andrei Shadura <andrew@shadura.me>
sub   cv25519/0xD5A55606B6539A87 2016-06-13 [E] [expires: 2022-06-25]
sub   ed25519/0x52E0EA6F91F1DB8A 2016-06-13 [A] [expires: 2022-06-25]

If you signed my old key and are reading this, please consider signing my new key; if you feel you need to re-confirm this, feel free to contact me; otherwise, a copy of this statement signed by both the old and new keys is available here. I have uploaded this new key to keys.openpgp.org, and also published it through WKD and the SKS network. Both keys can also be downloaded from my website.

Dmitry Shachnev: A review of endianness bugs in Qt, and how they were fixed

As you may know, I am Qt 5 maintainer in Debian. Maintaning Qt means not only bumping the version each time a new version is released, but also making sure Qt builds successfully on all architectures that are supported in Debian (and for some submodules, the automatic tests pass). An important sort of build failures are endianness specific failures. Most widely used architectures (x86_64, aarch64) are little endian. However, Debian officially supports one big endian architecture (s390x), and unofficially a few more ports are provided, such as ppc64 and sparc64. Unfortunately, Qt upstream does not have any big endian machine in their CI system, so endianness issues get noticed only when the packages fail to build on our build daemons. In the last years I have discovered and fixed some such issues in various parts of Qt, so I decided to write a post to illustrate how to write really cross-platform C/C++ code. Issue 1: the WebP image format handler (code review) The relevant code snippet is:

if (srcImage.format() != QImage::Format_ARGB32)
    srcImage = srcImage.convertToFormat(QImage::Format_ARGB32);
// ...
if (!WebPPictureImportBGRA(&picture, srcImage.bits(), srcImage.bytesPerLine()))  
    // ...

The code here is serializing the images into QImage::Format_ARGB32 format, and then passing the bytes into WebP s import function. With this format, the image is stored using a 32-bit ARGB format (0xAARRGGBB). This means that the bytes will be 0xBB, 0xGG, 0xRR, 0xAA or little endian and 0xAA, 0xRR, 0xGG, 0xBB on big endian. However, WebPPictureImportBGRA expects the first format on all architectures. The fix was to use QImage::Format_RGBA8888. As the QImage documentation says, with this format the order of the colors is the same on any architecture if read as bytes 0xRR, 0xGG, 0xBB, 0xAA. Issue 2: qimage_converter_map structure (code review) The code seems to already support big endian. But maybe you can spot the error?

#if Q_BYTE_ORDER == Q_LITTLE_ENDIAN
        0,
        convert_ARGB_to_ARGB_PM,
#else
        0,
        0
#endif

It is the missing comma! It is present in the little endian block, but not in the big endian one. This was fixed trivially. Issue 3: QHandle, part of Qt 3D module (code review) QHandle class uses a union that is declared as follows:

struct Data  
    quint32 m_index : IndexBits;
    quint32 m_counter : CounterBits;
    quint32 m_unused : 2;
 ;
union  
    Data d;
    quint32 m_handle;
 ;

The sizes are declared such as IndexBits + CounterBits + 2 is always equal to 32 (four bytes). Then we have a constructor that sets the values of Data struct:

QHandle(quint32 i, quint32 count)
 
    d.m_index = i;
    d.m_counter = count;
    d.m_unused = 0;

The value of m_handle will be different depending on endianness! So the test that was expecting a particular value with given constructor arguments was failing. I fixed it by using the following macro:

#if Q_BYTE_ORDER == Q_BIG_ENDIAN
#define GET_EXPECTED_HANDLE(qHandle) ((qHandle.index() << (qHandle.CounterBits + 2)) + (qHandle.counter() << 2))
#else /* Q_LITTLE_ENDIAN */
#define GET_EXPECTED_HANDLE(qHandle) (qHandle.index() + (qHandle.counter() << qHandle.IndexBits))
#endif

Issue 4: QML compiler (code review) The QML compiler used a helper class named LEUInt32 (based on QLEInteger) that always stored the numbers in little endian internally. This class can be safely mixed with native quint32 on little endian systems, but not on big endian. Usually the compiler would warn about type mismatch, but here the code used reinterpret_cast, such as:

quint32 *objectTable = reinterpret_cast<quint32*>(data + qmlUnit->offsetToObjects);

So this was not noticed on build time, but the compiler was crashing. The fix was trivial again, replacing quint32 with QLEUInt32. Issue 5: QModbusPdu, part of Qt Serial Bus module (code review) The code snippet is simple:

QModbusPdu::FunctionCode code = QModbusPdu::Invalid;
if (stream.readRawData((char *) (&code), sizeof(quint8)) != sizeof(quint8))
    return stream;

QModbusPdu::FunctionCode is an enum, so code is a multi-byte value (even if only one byte is significant). However, (char *) (&code) returns a pointer to the first byte of it. It is the needed byte on little endian systems, but it is the wrong byte on big endian ones! The correct fix was using a temporary one-byte variable:

quint8 codeByte = 0;
if (stream.readRawData((char *) (&codeByte), sizeof(quint8)) != sizeof(quint8))
    return stream;
QModbusPdu::FunctionCode code = (QModbusPdu::FunctionCode) codeByte;

Issue 6: qt_is_ascii (code review) This function, as the name says, checks whether a string is ASCII. It does that by splitting the string into 4-byte chunks:

while (ptr + 4 <= end)  
    quint32 data = qFromUnaligned<quint32>(ptr);
    if (data &= 0x80808080U)  
        uint idx = qCountTrailingZeroBits(data);
        ptr += idx / 8;
        return false;
     
    ptr += 4;

idx / 8 is the number of trailing zero bytes. However, the bytes which are trailing on little endian are actually leading on big endian! So we can use qCountLeadingZeroBits there. Issue 7: the bundled copy of tinycbor (upstream pull request) Similar to issue 5, the code was reading into the wrong byte:

if (bytesNeeded <= 2)  
    read_bytes_unchecked(it, &it->extra, 1, bytesNeeded);
    if (bytesNeeded == 2)
        it->extra = cbor_ntohs(it->extra);

extra has type uint16_t, so it has two bytes. When we need only one byte, we read into the wrong byte, so the resulting number is 256 times higher on big endian than it should be. Adding a temporary one-byte variable fixed it. Issue 8: perfparser, part of Qt Creator (code review) Here it is not trivial to find the issue just looking at the code:

qint32 dataStreamVersion = qToLittleEndian(QDataStream::Qt_DefaultCompiledVersion);

However the linker was producing an error:

undefined reference to QDataStream::Version qbswap(QDataStream::Version)'

On little endian systems, qToLittleEndian is a no-op, but on big endian systems, it is a template function defined for some known types. But it turns out we need to explicitly convert enum values to a simple type, so the fix was passing qint32(QDataStream::Qt_DefaultCompiledVersion) to that function. Issue 9: Qt Personal Information Management (code review) The code in test was trying to represent a number as a sequence of bytes, using reinterpret_cast:

static inline QContactId makeId(const QString &managerName, uint id)
 
    return QContactId(QStringLiteral("qtcontacts:basic%1:").arg(managerName), QByteArray(reinterpret_cast<const char *>(&id), sizeof(uint)));

The order of bytes will be different on little endian and big endian systems! The fix was adding this line to the beginning of the function:

id = qToLittleEndian(id);

This will cause the bytes to be reversed on big endian systems. What remains unfixed There are still some bugs, which require deeper investigation, for example:

P.S. We are looking for new people to help with maintaining Qt 6. Join our team if you want to do some fun work like described above!

31 December 2020

Chris Lamb: Free software activities in December 2020

Here is my monthly update covering what I have been doing in the free software world during December 2020 (previous month):

Reviewed and merged a contribution from Peter Law to my django-cache-toolbox library for Django-based web applications, including explicitly requiring that cached relations are primary keys (#23) and improving the example in the README (#25).

I took part in an interview with Vladimir Bejdo, an intern at the Software Freedom Conservancy, in order to talk about the Reproducible Builds project, my participation in software freedom, the importance of reproducibility in software development, and to have a brief discussion on the issues facing free software as a whole. The full interview can be found on Conservancy's webpages.

As part of my duties of being on the board of directors of the Open Source Initiative, I attended its monthly meeting and participated in various licensing and other related discussions occurring on the internet. Unfortunately, I could not attend the parallel meeting for Software in the Public Interest this month.

Reproducible Builds One of the original promises of open source software is that distributed peer review and transparency of process results in enhanced end-user security. However, whilst anyone may inspect the source code of free and open source software for malicious flaws, almost all software today is distributed as pre-compiled binaries. This allows nefarious third-parties to compromise systems by injecting malicious code into ostensibly secure software during the various compilation and distribution processes. The motivation behind the Reproducible Builds effort is to ensure no flaws have been introduced during this compilation process by promising identical results are always generated from a given source, thus allowing multiple third-parties to come to a consensus on whether a build was compromised. This month, I:

Submitted a draft academic paper to IEEE Software. The article (co-written by Stefano Zacchiroli) is aimed a fairly general audience. It first defines the overal problem and then provides insight into the challenges of actually making real-world software reproducible. It then outlines the experiences of the Reproducible Builds project in making large-scale software collections/supply-chains/ecosystems reproducible and concludes by describing the affinity between reproducibility efforts and quality assurance.
Kept isdebianreproducibleyet.com up to date. [...]
Submitted 11 patches in Debian to fix specific reproducibility issues in circlator, dvbstreamer, eric, jbbp, knot-resolver, libjs-qunit, mail-expire, osmo-mgw, python-pyramid, pyvows & sayonara.
Categorised a huge number of packages and issues in the Reproducible Builds 'notes' repository.
For disorderfs (our FUSE-based filesystem that deliberately introduces non-determinism into directory system calls in order to flush out reproducibility issues), I made the following changes:
- Add support for testing on Salsa's CI system. [...][...][...][...]
- Added a quick benchmark. [...]
Drafted, published and publicised our monthly report, as well managed the project's various social media accounts.
Contributed to a discussion about the recent 'SolarWinds' attack. [...]

I also made a large number of changes to the main Reproducible Builds website and documentation, including applying a typo fix from Roland Clobus [...], fixed the draft detection logic (#28), added more academic articles to our list [...] and corrected a number of grammar issues [...][...].

I also made the following changes to diffoscope, our in-depth and content-aware diff utility that can locate and diagnose reproducibility issues, including releasing version 163:

New features & bug fixes:
- Normalise ret to retq in objdump output in order to support multiple versions of GNU binutils. (#976760)
- Don't show any progress indicators when running zstd. (#226)
- Correct the grammatical tense in the --debug log output. [...]
Codebase improvements:
- Update the debian/copyright file to match the copyright notices in the source tree. (#224)
- Update various years across the codebase in .py copyright headers. [...]
- Rewrite the filter routine that post-processes the output from readelf(1). [...]
- Remove unnecessary PEP 263 encoding header lines; unnecessary after PEP 3120. [...]
- Use minimal instead of basic as a variable name to match the underlying package name. [...]
- Use pprint.pformat in the JSON comparator to serialise the differences from jsondiff. [...]

Debian Uploads

python-django:
- 2.2.17-2 Fix compatibility with GNU gettext version 0.21. (#978263)
- 3.1.4-1 New upstream bugfix release.
redis:
- 6.2~rc1-1 New upstream RC release.
- 6.2~rc1-2 Enable systemd support by compiling against the libsystemd-dev package. (#977852)
- 6.2~rc1-3 Specify --supervised systemd now that we specify Type=notify to prevent service startup failure under systemd.
mtools (4.0.26-1) New upstream release.

I also sponsored an upload of adminer (4.7.8-2) on behalf of Alexandre Rossi and performed two QA uploads of sendfile (2.1b.20080616-7 and 2.1b.20080616-8) to make the build the build reproducible (#776938) and to fix a number of other unrelated issues. Debian LTS This month I have worked 18 hours on Debian Long Term Support (LTS) and 12 hours on its sister Extended LTS project.

Investigated and triaged: awstats, imagemagick, node-ini, openexr, openssl1.0, p11-kit, pypy, python-py, sqlite3, sympa, etc.
Frontdesk duties, responding to user/developer questions, reviewing others' packages, participating in mailing list discussions, etc.
Issued DLA 2477-1 for the Jupyter Notebook interactive notebook system, where a maliciously-crafted link could redirect the browser to a malicious/spoofed website. (CVE-2020-26215)
Issued DLA 2491-1 and ELA-333-1 to fix two issues in OpenEXR, a set of tools to manipulate OpenEXR image files, often used in the computer-graphics industry for visual effects and animation. (CVE-2020-16588 & CVE-2020-16589)
Issued DLA 2503-1 as it was discovered that there was an issue in node-ini, an .ini configuration file format parser/serialiser for Node.js, where an application could be exploited by a malicious input file.

You can find out more about the Debian LTS project via the following video:

Shirish Agarwal: Pandemic, Informal economy and Khau Galli.

Formal sector issues Just today it was published that output from eight formal sectors of the economy who make the bulk of the Indian economy were down on a month to month basis . This means all those apologists for the Government who said that it was ok that the Govt. didn t give the 20 lakh crore package which was announced. In fact, a businessman from my own city, a certain Prafull Sarda had asked via RTI what happened to the 20 lakh crore package which was announced? The answers were in all media as well as newspapers but on the inside pages. You can see one of the article sharing the details here. No wonder Vivek Kaul also shared his take on the way things will hopefully go for the next quarter which seems to be a tad optimistic from where we are atm.

The Informal economy The informal economy has been strangulated by the current party in power since it came into power. And this has resulted many small businesses which informal are and were part of culture of cities and towns. I share an article from 2018 which only shows how good and on the mark it has aged in the last two years. The damage is all to real to ignore as I would share more of an anecdotes and experiences because sadly there never has been any interest shown especially by GOI to seek any stats. about informal economy. Although CMIE has done some good work in that even though they majorly look at formal, usually blue-collar work where again there is not good data. Sharing an anecdote and a learning from these small businesses which probably an MBA guy wouldn t know and in all honesty wouldn t even care. Khau galli Few years back, circa 2014 and on wards, when the present Govt. came into power, it did come with lot of promises. One of which was that lot of informal businesses would be encouraged to grow their businesses and in time hopefully, they become part of the formal economy. Or at least that was the story that all of us were told. Due to that they did lot of promises and also named quite a few places where street food was in abundance. Such lanes were named Khau galli for those who are from North India, it was be easily known and understood. This was just saying that here are some places where you could get a variety of food without paying obscene prices as you would have to vis-a-vis a restaurant. Slowly, they raised the rates of inputs (food grains), gas cylinder etc. which we know of as food inflation and via GST made sure that the restaurants were able to absorb some of the increased inputs (input credit) while still being more than competitive to the street food person/s. The restaurant F&B model is pretty well known so not going there at all. It is however, important to point out that they didn t make any new khau gallis or such, most or all the places existed for years and even decades before that. They also didn t take any extra effort either in marketing the khau gallis or get them with chefs or marketing folks so that the traditional can marry to the anew. They just named them, that was the only gain to be seen on the ground. In its heyday, the khau galli near my home used to have anywhere between 20-30 Thelas or food carts. Most of the food carts would be of wood and having very limited steel. Such food carts would cost anywhere around INR 15-20k instead of the food cart you see here. The only reason I shared that link is to show how a somewhat typical thela or food cart looks in India. Of course YouTube or any other media platform would show many. On top of it, you need and needed permission from the municipality a license for the same which would be auctioned. Now that license could well run from thousands into lakhs depending on various factors or you gave something to the Municipal worker when he did his rounds/beat much like a constable every day or week. Apart from those, you also have raw material expenditure which could easily run into few thousands depending upon what sort of food you are vending. You also would typically have 2-3 workers so a typical Thela would feed not only its customers but also 2-3 families who are the laborer families as well as surrounding businesses. As I used to be loyal and usually go to few whom I found to be either tasty or somehow they were good for me. In either case, a relationship was formed. As I have been never fond of crowds, I usually used to in their off-beat hours either when they are close to packing up of when I know they usually have a lull. That way I knew I would get complete attention of the vendor/s. Many a times I used to see money change hands between the vendors themselves and used to see both camaraderie as well as competition between them. This is years ago, once while sharing a chai (tea) with one of the street vendors I casually asked I have often seen you guys exchanging money with each other and most of the time quite a bit of the money is also given to the guy who didn t make that much sales or any sales at all. The vendor replied sharing practical symbiotic knowledge. All of us are bound by a single thing called poverty. All of us are struggle. Do you know why so many people come here, because they know that there would be a variety of food to be had. Now if we stopped helping each other, the number of people who would make the effort would be also less, we know we are not the only game in town. Also whatever we give, sooner or later it gets adjusted. Also if one of us has good days, he knows hardship days are not far. Why, simply because people change tastes or want variety. So irrespective of good or bad the skills of the vendor is, she or he is bound to make some sales. The vendor either shares the food with us or whatever. Somehow these things just work out. And that doesn t mean we don t have our fights, we do have our fights, but we also understand this. Now you see this and you understand that these guys have and had a community. Even if they changed places due to one reason or the other, they kept themselves connected. Unlike many of us, who even find a hard time keeping up with friends let alone relatives. Now cut to 2020, and where there used to be 20-30 thelas near my home, there are only 4-5. Of course, multiple reasons, but one of the biggest was of course demonetization. That was a huge shock to which many of thela walas succumbed. Their entire savings and capital were turned to dust. Many of their customers will turn up with either a INR 500 or INR 2000/- Re note where at the most a dish costed INR 100/- most times half or even 1/3rd of that amount. How and from where the thela walas could get that kind of cash. These are people who only if they earn, they and their family will have bread at night. Most of the loose change was tied up at middle to higher tier restaurants where they were giving between INR 20/- 30/- for every INR 100/- change of rupees and coins. Quite a few bankers made money by that as well as other means where the thela walas just could not compete. These guys also didn t have any black money even though they were and are part of the black/informal economy. Sadly, till date no economist or even sociologist as far as I know has attempted or done any work from what I know on this industry. If you want to formalize such businesses then at the very least understand their problems and devise solutions. And I suspect, what is and has happened near my house has also happened everywhere else, at least within the geographical confines of the Indian state. Whether it was the 2016 demonetization or the pandemic, the results and effects have been similar the same all over. Some states did do well and still do, the suffering still continues. With the hope that the new year brings cheer to you as well some more ideas to remain in business by the thela walas, I bid you adieu and see you in new year

Daniel Kahn Gillmor: New OpenPGP certificate for dkg, 2021

dkg's 2021 OpenPGP transition As 2021 begins, I'm changing to a new OpenPGP certificate. I did a similar transition two years ago, and a fair amount has changed since then. You might know my old OpenPGP certificate as:

pub   ed25519 2019-01-19 [C] [expires: 2021-01-18]
      C4BC2DDB38CCE96485EBE9C2F20691179038E5C6
uid          Daniel Kahn Gillmor <dkg@fifthhorseman.net>
uid          Daniel Kahn Gillmor <dkg@debian.org>

My new OpenPGP certificate is:

pub   ed25519 2020-12-27 [C] [expires: 2023-12-24]
      C29F8A0C01F35E34D816AA5CE092EB3A5CA10DBA
uid           [ unknown] Daniel Kahn Gillmor
uid           [ unknown] <dkg@debian.org>
uid           [ unknown] <dkg@fifthhorseman.net>

You can find a signed transition statement if you're into that sort of thing. If you're interested in the rationale for why I'm making this transition, read on.

Dangers of Offline Primary Secret Keys There are several reasons for transitioning, but one i simply couldn't argue with was my own technical failure. I put the primary secret key into offline storage some time ago for "safety", and used ext4's filesystem-level encryption layered on top of dm-crypt for additional security. But either the tools changed out from under me, or there were failures on the storage medium, or I've failed to remember my passphrase correctly, because I am unable to regain access to the cleartext of the secret key. In particular, I find myself unable to use `e4crypt add_key` with the passphrase I know to get a usable working directory. I confess I still find `e4crypt` pretty difficult to use and I don't use it often, so the problem may entirely be user error (either now, or two years ago when I did the initial setup). Anyway, lesson learned: don't use cryptosystems that you're not comfortable with to encrypt data that you care about recovering. This is a lesson I'm pretty sure I've learned before, sigh, but it's a good reminder.

Split User IDs I'm trying to split out my User IDs again -- this way if you know me by e-mail address, you don't have to think/worry about certifying my name, and if you know me by name, you don't have to think/worry about certifying my e-mail address. I think that's simpler and more sensible. It's also nice because e-mail address-only User IDs can be used effectively in contexts like Autocrypt, which I think are increasingly important if we want to have usable encrypted e-mail. Last time around I initially tried split User IDs but rolled them back and I think most of the bugs I discovered then have been fixed.

Certificate Flooding Another reason for making a transition to a new certificate is that my older certificate is one of the ones that was "flooded" on the SKS keyserver network last year, which was one of the final straws for that teetering project. Transitioning to a new certificate lets that old flooded cert expire and people can just simply move on from it, ideally deleting it from their local keyrings. Hopefully as a community we can move on from SKS to key distribution mechanisms like WKD, Autocrypt, DANE, and keys.openpgp.org, all of which address some of the known problems with keyserver abuse.

Trying New Tools Finally, I'm also interested in thinking about how key and certificate management might be handled in different ways. While I'm reasonably competent in handling GnuPG, the larger OpenPGP community (which I'm a part of) has done a lot of thinking and a lot of work about how people can use OpenPGP. I'm particularly happy with the collaborative work that has gone into the Stateless OpenPGP CLI (aka `sop`), which helps to generate a powerful interoperability test suite. While `sop` doesn't offer the level of certificate management I'd need to use it to manage this new certificate in full, I wish something like it would! Starting from a fresh certificate and actually using it helps me to think through what I might actually need from a tool that is roughly as straightforward and opinionated as `sop` is. If you're a software developer who might use or implement OpenPGP, or a protocol designer, and you haven't played around with any of the various implementations of `sop` yet, I recommend taking a look. And feedback on the specification is always welcome, too, including ideas for new functionality (maybe even like certificate management).

Next Steps If you're the kind of person who's into making OpenPGP certifications, feel free to check in with me via whatever channels you're used to using to verify that this transition is legit. If you think it is, and you're comfortable, please send me (e-mail is probably best) your certifications over the new certficate. I'll keep on working to make OpenPGP more usable and acceptable. Hopefully, 2021 will be a better year ahead for all of us.

Russ Allbery: Review: Billion Dollar Loser

Review: Billion Dollar Loser, by Reeves Wiedeman

Publisher:	Little, Brown and Company
Copyright:	October 2020
ISBN:	0-316-46134-2
Format:	Kindle
Pages:	315

WeWork was founded in 2010 by Adam Neumann and Miguel McKelvey as a successor company to their similar 2008 GreenDesk business. (Adam's wife Rebekah is now presented as a co-founder. This seems dubious in Wiedeman's account, although Rebekah's role in the company is murky, ever-changing, and hard to pin down.) Its business model in reality was to provide turn-key, pre-furnished and stocked co-working and small office space to individuals and businesses on flexible, short-term leases. Its business model in Neumann's speeches and dreams, and represented by the later renaming of the company to the We Corporation, was nothing less than to transform the way people worked, learned, and lived. Through aggressive, money-losing expansion, WeWork grew rapidly to over 500 locations in 29 countries and became the largest office tenant in New York City. Based primarily on massive financial support from Masayoshi Son, CEO of Japanese holding company SoftBank, WeWork's private valuation rose to $47 billion. In 2019, the company attempted to go public, but its IPO collapsed, in part due to deeper analysis of the company's books. Neumann was forced out of the company (with an individual payout valued at $1.7 billion), the IPO was withdrawn, SoftBank wrote down 90% of their investment in the company and took control of it, and WeWork laid off more than 20% of its workforce. This book is a detailed history of WeWork's rise and fall, joining a genre that includes The Smartest Guys in the Room (Enron), Bad Blood (Theranos), and Super Pumped (Uber). I believe it's the first full book on WeWork, although it was preceded by several long-form stories, including "The I In We" by Wiedeman for New York magazine. As the first history, it's a somewhat incomplete cut: litigation between Neumann and WeWork is still pending, WeWork staggered into 2020 and a world-wide pandemic that made cramped open-plan offices an epidemiological disaster, and there will doubtless be new revelations yet to come. The discovery process of lawsuits tends to be good for journalists. But despite being the first out of the gate, Billion Dollar Loser reaches a satisfying conclusion with the ouster of Neumann, who had defined WeWork both internally and externally. I'm fascinated by stories of failed venture capital start-ups in general, but the specific question about WeWork that interested me, and to which Wiedeman provides a partial answer, is why so many people gave Neumann money in the first place. Explaining that question requires a digression into why I thought WeWork's valuation was absurd. The basic problem WeWork had when justifying its sky-high valuation is competition. WeWork didn't own real estate; it rented properties from landlords with long-term leases and then re-rented them with short-term leases. If its business was so successful, why wouldn't the landlords cut out the middle man, do what WeWork was doing directly, and pocket all the profit? Or why wouldn't some other company simply copy WeWork and drive the profit margins down? Normally with startups the answer revolves around something proprietary: an app, a server technology, patents, a secret manufacturing technique, etc. But nothing WeWork was doing was different from what innumerable tech companies and partner landlords had been doing with their office space for a decade, and none of it was secret. There are two decent answers to that question. One is simple outsourcing: landlords like being passive rent collectors, so an opportunity to pay someone else to do the market research on office layouts, arrange all the remodeling, adapt to changing desires for how office space should be equipped and stocked, advertise for short-term tenants, and deal with the tenant churn is attractive. The landlord can sit back and pocket the stable long-term rent. The second answer is related: WeWork is essentially doing rental arbitrage between long-term and short-term rents and thus is taking on most of the risk of a commercial real estate downturn. If no one is renting office space, WeWork is still on the hook for the long-term rent. The landlord is outsourcing risk, at least unless WeWork goes bankrupt. (One infuriating tidbit from this book is that Neumann's explicit and often-stated goal was to make WeWork so large that its bankruptcy would be sufficiently devastating to the real estate industry that it would get a bailout.) There's a legitimate business here. But that business looks like a quietly profitable real estate company that builds very efficient systems for managing short-term leases, remodeling buildings, and handling the supply chain of stocking an office. That looks nothing like WeWork's business, has nothing to do with transforming the world of work, and certainly doesn't warrant sky-high valuations. WeWork didn't build an efficient anything. It relied on weekend labor from underpaid employees and an IT person who was still in high school. And WeWork actively resisted being called a real estate company and claimed it was a tech company or a lifestyle company on the basis of essentially nothing. Wiedeman seems almost as baffled by this as I am, but it's clear from the history he tells that part of the funding answer is the Ponzi scheme of start-up investing. People gave Neumann money because other people had previously given Neumann money, and the earlier investors cashed out at the expense of the later ones. Like any Ponzi scheme, it looks like a great investment until it doesn't, and then the last sucker is left holding the bag. That sucker was Masayoshi Son, who in Wiedeman's telling is an astonishingly casual and undisciplined investor who trusted knee-jerk personal reactions to founders over business model analysis and historically (mostly) got away with it by getting extremely lucky. (I now want to read one of these books about SoftBank, since both this book and Super Pumped make it look like a company that makes numerous wild gambles for the flimsiest of reasons, pushes for completely unsustainable growth, and relies on the sheer volume of investments catching some lucky jackpots and cashing out in IPOs. Unfortunately, the only book currently available seems to be a fawning hagiography of Son.) On one hand, the IPO process worked properly this time. The sheer madness of WeWork's valuation scared off enough institutional investors that it collapsed. On the other hand, it's startling how close it came to success. If WeWork had kept the Ponzi scheme going a bit longer, the last sucker could have been the general investing public. Another interesting question that Billion Dollar Loser answers is how Neumann got enough money to start his rapid growth strategy. The answer appears to be the oldest and most obvious explanation: He made friends with rich people. The initial connections appear to have been through his sister, Adi Neumann, who is a model and hosted parties in her New York apartment (and also started dating a Rothschild heir). Adam met his wealthy wife Rebekah, cousin to actress and "wellness" scam marketer Gwyneth Paltrow, via a connection at a party. He built social connections with other parts of the New York real estate scene and tapped them for investment money. The strong impression one gets from the book is that all of these people have way more money than sense and we should raise their taxes. It won't come as a surprise that Adam and Rebekah Neumann are good friends of Jared Kushner and Ivanka Trump. Those are the questions I was the most curious about, but there's much more here. Wiedeman's style is nearly straight chronological reporting with little analysis, but the story is so wild and absurd that it doesn't need much embellishment. Neumann is obviously a megalomaniac whose delusions of grandeur got worse and worse as WeWork was apparently succeeding. Rebekah Neumann is if anything even less in touch with reality than he is, although in her case that appears to stem from having so much money that reality is an inconvenient speed bump. Miguel McKelvey, Neumann's co-founder, is an odd and interesting side note to the story; he appears to have balanced Adam out a bit in the early going but then wisely started to cash out and pocket his winnings while letting Adam dominate the stage. There are some places where I don't think Wiedeman pushed hard enough, and which cut against the view of Neumann as a true believer in his impossible growth vision. Neumann took several investment opportunities to cash out large amounts of his stock even while WeWork employees were being underpaid and told their stock options would make up for it. He clearly used WeWork as a personal piggy bank on multiple occasions. And Wiedeman documents but doesn't, at least in my opinion, make nearly enough of Neumann's self-dealing: buying real estate that WeWork then rented as a tenant, or paying himself for a license for the name We Holdings (although there at least he later returned the money). I think a good argument could be made that Neumann was embezzling from WeWork, at least morally if not legally, and I wish Wiedeman would have pressed harder on that point. But that aside, this is a great first history of the company, told in a clean, readable, and engaging style, and with a lot more detail here than I've touched on (such as Rebekah Neumann's WeGrow school). It's not as good as Bad Blood (what is), but it's a respectable entry in the corporate collapse genre. If you like this sort of thing, recommended. Rating: 7 out of 10

30 December 2020

Andrej Shadura: Making the blog part of the Fediverse and IndieWeb

I ve just made my blog available on the Fediverse, at least partially. Yesterday while browsing Hacker News, I saw Carl Schwan s post Adding comments to your static blog with Mastodon^(m) about him replacing Disqus with replies posted at Mastodon. Just on Monday I was thinking, why can t blogs participate in Fediverse? I tried to use WriteFreely as a replacement for Pelican, only to find it very limited, so I thought I might write a gateway to expose the Atom feed using ActivityPub. Turns out, someone already did that: Bridgy, a service connecting websites to Twitter, Mastodon and other social media, also has a Fediverse counterpart, Fed.brid.gy just what I was looking for! I won t go deeply into the details, but just outline issues I had to deal with:

Bridgy relies on microformats. While I had some microformats and Schema.org microdata already at both blog and the homepage, they were not sufficiently good for Bridgy to use.
Webmention: I had to set up https://webmention.io to act as Webmention host for me.
I ran into an issue with Mastodon being more picky about signatures which the author Ryan Barrett promptly fixed.

The end result is not fully working yet, since posts don t appear yet, but I can be found and followed as @shadura.me@shadura.me. As a bonus I enabled IndieAuth at my website, so I can log into compatible services with just the URL of my website.

Russ Allbery: Review: The House in the Cerulean Sea

Review: The House in the Cerulean Sea, by TJ Klune

Publisher:	Tor
Copyright:	March 2020
ISBN:	1-250-21732-6
Format:	Kindle
Pages:	398

Linus Baker is a case worker for the Department in Charge of Magical Youth. His job is to evaluate the orphanages to which children with magical powers are sent and ensure those children are treated properly. This must be done properly, with professional care and detachment and careful documentation so that his superiors can make the correct choices. Linus has no ambition or desire to be anything other than a case worker. He does his job meticulously by the book, namely the DICOMY Rules and Regulations, which he studies religiously. Linus is therefore caught entirely by surprise when he is summoned without explanation to the offices of Extremely Upper Management on the intimidating 5th floor. That summons turns out to be a new assignment for which Extremely Upper Management believes Linus to be ideal: an orphanage on an island far from the city, home to some highly unusual (and, in one case, disturbing) children and a headmaster who is as odd as his charges. Management is correct about Linus up to a point. He is a bureaucrat and a stickler for rules. However, Extremely Upper Management did not account for the possibility that Linus does the work that he does because he cares deeply about children, or that, if the two come into conflict, the children might matter more than the rules. The House in the Cerulean Sea is the sort of book that tells you exactly what it's doing, assures you that you'll enjoy it anyway, and then you actually do. The plot developments are obvious well in advance, the tugs on the heart-strings are telegraphed and obvious, and there are very few surprises. But the story is told with such charm and feeling that I at least didn't mind at all, and indeed would have been furious at the author if he hadn't delivered the expected ending. This is not the book to read if you want to delve deep into the implications of the world-building. It follows children's book logic in several important places: intentions matter more than institutional structure, sincerity is persuasive, and courage is rewarded. A few reviews compare it to Nineteen Eighty-Four, but this is wildly misleading beyond a few trappings of the setting. The internal logic of the book is closer to a cross between Good Omens and Dr. Seuss. It's a "good people are rewarded" sort of book and wears that on its sleeve. One of the things that Klune does extremely well is subtlety of characterization despite the relative lack of subtlety in the plot. Linus starts the book as a caricature whose life is nearly a blank canvas, and by the end of the book he feels like someone you know. Klune pulls this off less by filling in the canvas than by making careful use of all that negative space to put emotional weight and depth behind a few telling details like Linus's pyjamas, records, and cat. He also does an exceptional job with the mannerisms and habits of speech of both Linus and the headmaster of the orphanage. To say much more would be a spoiler; suffice it to say that Klune captures an interaction style that I have rarely seen in novels and certainly not in a fantasy novel of this type, and does it so well that I was strongly reminded of people I've known. Anyone who knows me is unlikely to be surprised that my favorite character in the book is Zoe, the sarcastic and fiercely protective sprite who owns the island on which the orphanage is located, but almost all of the characters are wonderful. One or two of the children are a bit one-note, but the group dynamics make up for that, and the suspicious, thoughtful, and respectful interplay between the three adults is beautifully done. My only real complaints on the characterization front are that some of the confrontations in the nearby village felt a bit strained, and I think Helen got short shrift and could have been fleshed out some more. I will say that the motives of the villains never made much sense, but they didn't have to; they're essentially monsters in Linus's story, and they do enough to serve that purpose. I now want to read the story of how Zoe and the headmaster met, although I'm not sure Klune is the right person to write it, and I'm not sure he left enough emotional space for another novel. Sometimes I want to read a positive, hopeful book in which people learn and grow and good things happen to good people, one where the author is unapologetically manipulating my emotions and I don't care because I want these people to be happy. If you feel the same way, I cannot recommend this book highly enough. It's lovely. Rating: 9 out of 10

29 December 2020

Bastian Venthur: Creating Beautiful Github Streaks

Creating Beautiful Github Streaks Github streaks are a pretty fun and harmless way of visualizing your contributions on github or gitlab. Some people use them to brag a little, some companies use them to check out potential candidates. It s all good as long as everyone is aware how easily you can manipulate them. In order to fake a commit date in the past, you have to set the GIT_AUTHOR_DATE and GIT_COMMITTER_DATE environment variables on

git
commit

both, RFC 2822 and ISO 8601 formats are accepted (for more info see here). When generating hundreds of commits, the --allow-empty flag is very useful, as we don t have to actually change the source tree in order to generate a commit. So the command to generate a fake commit looks like this:

GIT_AUTHOR_DATE="$TIMESTAMP" GIT_COMMITTER_DATE="$TIMESTAMP" git commit --allow-empty -m "$TIMESTAMP"

execute it on an empty git repository and push to github or gitlab and you ll se the result after some time. To automate this, I wrote me a little python script (github, pypi). You can simply install it via:

pip install python-streak

and run it with:

streak

in an empty git repository. By default, streak will create 3 commits per day with a probability of ~71% per commit for 100 years, starting from 1980-05-09. These parameters can be tuned with:

--start START: Beginning of the commits (Format: YYYY-MM-DD)
--end END: End of the commits (Format: YYYY-MM-DD)
--max MAX: Maximum number of commits per day.
--proba PROBA: Probability for a commit to happen.

Generating a few commits per day for the range of 10 years, took roughly 2 minutes on my computer, and man was I a busy beaver in the 80s! Streak 1989

I know, I m for sure not the first one to discover this kind of thing, but it was a fun afternoon project anyways!

Bastian Venthur: Creating Beautiful Github Streaks

git
commit

GIT_AUTHOR_DATE="$TIMESTAMP" GIT_COMMITTER_DATE="$TIMESTAMP" git commit --allow-empty -m "$TIMESTAMP"

pip install python-streak

and run it with:

streak

in an empty git repository. By default, streak will create 3 commits per day with a probability of ~71% per commit for 100 years, starting from 1980-05-09. These parameters can be tuned with:

--start START: Beginning of the commits (Format: YYYY-MM-DD)
--end END: End of the commits (Format: YYYY-MM-DD)
--max MAX: Maximum number of commits per day.
--proba PROBA: Probability for a commit to happen.

Generating a few commits per day for the range of 10 years, took roughly 2 minutes on my computer, and man was I a busy beaver in the 80s! Streak 1989

I know, I m for sure not the first one to discover this kind of thing, but it was a fun afternoon project anyways!

Shirish Agarwal: Inequality in Indian Education

Farmer on-going protests Before I start with the education system in India which I have talked about many times in the past, first let me share couple of pieces about the farmer movement which is still at Sanghu Delhi border.

The above picture became somewhat viral as it showed Manjeet Kaur, who drove down from Patiala, Punjab to Sanghu border along with her friends to take part in the on-going protests. The picture not only shares how the women are part and parcel of this protest but also they are independently taking part in the protest. The other were two articles I read today, first was an article in tribune which questions that if the policy worked, why it didn t work in the state of Bihar. The other by a young law student who had to go from Chandigarh to Delhi with family for some work and her experience with the ongoing protest. In fact, an interesting observation was made by the CJI in the many suits against farmer protests in the SC. This makes for much more interesting read when you see an RTI query filed by Saket Gokhale to NHAI , a Central Govt. agency which is supposed to be independent and asks if they had filed an FIR and asked compensation from Haryana State and Haryana State Police which had dug up National Highway 44 and if any permission was asked for the same from NHAI. And NHAI unable to take any action for the same. If this isn t shameful, I dunno what is

Sadly, the way the response has been worded makes it impossible for NHAI to discharge its own responsibilities and this becomes a precedent for other states now that know that NHAI is vulnerable. A pretty sad turn on events. Indian Education can t go online There was a recent article on scroll which shared how Indian education can t go online as only a few have computers with decent netlink speeds as well as other factors which are needed for online education. But there are also many things that the article doesn t take into account which actually make the task more difficult and raise the boundary more. Now in most schools and colleges, the number of students to teacher ratio could be anywhere between 70-150 or even more. In the last few years, a lot of schools have been closed down by various Governments, including and not limited to the ruling Govt. They have in fact intensified closures of public schools wherever their Govt. has been in power. Closing to 5000+ schools in one state in a year is a dramatic shift and such has been happening time and again. In fact, the rising costs of Indian education has made many to leave Indian shores and do studies abroad. And once they do their masters or whatever, the chances of them coming back to India become more and more remote. In India the costs have been becoming so bad that NBFC s have started products targeting the same. How NBFC and Banks have (both public and private) have fared with respect to Indian consumers needs its own blog post but one word to describe it is bad . But as shared above, needs its own blog post. Coming to the Indian context though, what has not been captured in that article is that the responsibility of making new content also raises huge barriers for teachers. My own experience in teacher s trainings for ICT usage has shown that most teachers do not know and use internet effectively both to sustain their own curiosity as well as their students. Part of which is whether you are private employee or a public school teacher, the teacher is not paid enough. I have had multiple conversations with friends over the years who are teachers who shared that they get 50% salary in-hand while they sign for 100%. This is more in the case or private schools though. In Govt. schools, the teachers apart from their regular administrative duties apart from teaching duties are also unpaid labor for Govt. policy. Take the recent covid crisis, it was the teachers who for months together went from door-to-door asking if they had a covid patient. This was all over India. Even for voter registration, census, polio and various other immunization efforts, the teachers are roped in. So apart from that, they somehow have to figure out how to make ends meet and also boost student morale. Hence the attention is only limited to the first couple of benches rather than the whole as a 45-minute to an hr. session is just not enough to go through a class of 70-150 school students giving individual attention. And this is when for most teachers, teaching is a means to an end and not the end itself. I am going to take one example of science and sort of break-it-down in multiple steps and how I would have approached that topic for say the 5th-6th standard students in say a public school in Pune and especially if Covid would not have been an issue so you have face-to-face meetup. There was recent news about a mysterious radio signal which came from one of our closest galactic neighbors Proxima Centauri. Now let s say there was a class I was teaching/sharing which I had shared before and there already is trust formed. So before coming to the news, I would tell the students about frequency and more generally the notation of why we like to measure things and how we measure things. There is so much beautiful history which could be acted and enacted which can show and remains in mind why measurement is needed. Once that is understood, discussed and an underpinning is established, we could move to human perception or the lack of it. We know that humans have lots of limits in almost everything, whether it is talking, touching, hearing, all of our five senses are pretty limited with what we know of spectrum available in the immediate family kingdom as well as in the Universe. I would start with how far can a person throw his voice and be heard without using any other means. There does come a point where they need to use anything from a megaphone to a loudspeaker and what it actually does. The other thing I would then talk is about the radio and ask the students to find more about the internals of a radio. If possible to bring an old radio to school where it could be disassembled. After they are familiar with some names of the electronic components and what they do, take them to the electronics market where they try to source all the things needed to make a radio and whatever they encounter. This would allow the students to try and do bargain shopping as well as learn from where to source things. Some might even get a copy or two of electronic projects where the shop themselves sell blueprints to hobbyists so that they can tinker. If there is a place in the school where soldering can be done, then all can try and sooner or later we come to know if something works or not. There is also possibility of talking about noise cancellation and then the topic of ITU can also be bought up and how they do frequency allocation. Last but not the least then the topic can be approached about an alien civilization and an unknown radio signal and what it means and what it can mean. Now if just one topic can give such a wide range of things to do and develop an understanding about not the subject itself but surrounding subjects as well I see no reason why teachers can t do this except they are handcuffed to lot of policy as well as real-life constraints. For e.g. I remember in my school days, we used to go out once or twice a year and that used to be either a school picnic or something similar. The only other I know is going to Mumbai for Nehru planetarium and Nehru science Centre. Unfortunately, I didn t go at that time because the school was taking students via air and the tickets were super costly at the time and that too for a 10 minute journey between the two cities. Those were different days, today you can t have a direct flight between the two cities as it doesn t make an economic sense. It makes more sense to go to Mumbai via train or bus as you will reach Mumbai in about couple of hrs. Of course, Pune does have its own planetarium at New English school and there are a few amateur astronomy clubs in Pune but nothing on the scale that what Mumbai has, but then this is getting off-topic. Now, again in an online world could this be done? Not without both the teacher and the student both spending lot of resources online and even then will be a lower understanding as both the hands-on experience as well as interacting with other students and learning from other students (aping) would be hugely limited. Even the social skills that students develop in a school setting will be rusted. My own social skills probably have weakened and rusted as I have very limited interaction with people over the past few months due to Covid fears and would be at least for the next few years till a large enough population is not vaccinated.

28 December 2020

Guido G nther: phosh overview

phosh is graphical shell for mobile, touch based devices like smart phones. It's the default graphical shell on Purism's Librem 5 (and that's where it came to life) but projects like postmarketOS, Mobian and Debian have picked it up putting it into use on other devices as well and contributing patches. This post is meant as a short overview how things are tied together so further posts can provide more details. A PHone SHell As mobile shell phosh provides the interface components commonly found on mobile devices to phosh's overview 2

launch applications
switch between running applications and close them
lock and unlock the screen
display status information (e.g. network connectivity, battery level)
provide quick access to things like torch or Bluetooth
show notifications

It uses GObject object system and GTK to build up the user interface components. Mobile specific patterns are brought in via libhandy. Since phosh is meant to blend into GNOME as seamlessly as possible it uses the common interfaces present there via D-Bus like org.gnome.Screensaver or org.gnome.keyring.SystemPrompter and retrieves user configuration like keybindings via GSettings from preexisting schema. The components of a running graphical session roughly look like this: phosh session

The blue boxes are the very same found on GNOME desktop sessions while the white ones are currently only found on phones. feedbackd is explained quickly: It's used for providing haptic or visual user feedback and makes your phone rumble and blink when applications (or the shell) want to notify the user about certain events like incoming phone calls or new messages. What about phoc and squeekboard? phoc and squeekboard Although some stacks combine the graphical shell with the display server (the component responsible for drawing applications and handling user input) this isn't the case for phosh. phosh relies on a Wayland compositor to be present for that. Keeping shell and compositor apart has some advantages like being able to restart the shell without affecting other applications but also adds the need for some additional communication between compositor and shell. This additional communication is implemented via Wayland protocols. The Wayland compositor used with phosh is called phoc for PHone Compositor. One of these additional protocols is wlr-layer-shell. It allows the shell to reserve space on the screen that is not used by other applications and allows it to draw things like the top and bottom bar or lock screen. Other protocols used by phosh (and hence implemented by phoc) are wlr-output-management to get information on and control properties of monitors or wlr-foreign-toplevel-management to get information about other windows on the display. The later is used to allow to switch between running applications. However these (and other) Wayland protocols are not implemented in phoc from scratch. phoc leverages the wlroots library for that. The library also handles many other compositor parts like interacting with the video and input hardware. The details on how phoc actually puts things up on the screen deserves a separate post. For the moment it's sufficient to note that phosh requires a Wayland compositor like phoc. We've not talked about entering text without a physical keyboard yet - phosh itself does not handle that either. squeekboard is the on screen keyboard for text (and emoji) input. It again uses Wayland protocols to talk to the Wayland compositor and it's (like phosh) a component that wants exclusive access to some areas of the screen (where the keyboard is drawn) and hence leverages the layer-shell protocol. Very roughly speaking it turns touch input in that area into text and sends that back to the compositor that then passes it back to the application that currently gets the text input. squeekboard's main author dcz has some more details here. The session So how does the graphical session in the picture above come into existence? As this is meant to be close to a regular GNOME session it's done via gnome-session that is invoked somewhat like:

phoc -E 'gnome-session --session=phosh'

So the compositor phoc is started up, launches gnome-session which then looks at phosh.session for the session's components. These are phosh, squeekboard and gnome-settings-daemon. These then either connect to already running services via D-Bus (e.g. NetworkManager, ModemManager, ...) or spawn them via D-Bus activation when required (e.g. feedbackd). Calling conventions So when talking about phosh it's good to keep several things apart:

phosh - the graphical shell
phoc - the compositor
squeekboard - the on screen keyboard
phosh.session: The session that ties these and GNOME together

On top of that people sometimes refer to 'Phosh' as the software collection consisting of the above plus more components from GNOME (Settings, Contacs, Clocks, Weather, Evince, ...) and components that currently aren't part of GNOME but adapt to small screen sizes, use the same technologies and are needed to make a phone fun to use e.g. Geary for email, Calls for making phone calls and Chats for SMS handling. Since just overloading the term Phosh is confusing GNOME/Phosh Mobile Environment or Phosh Mobile Environment have been used to describe the above collection of software and I've contacted GNOME on how to name this properly, to not infringe on the GNOME trademark but also give proper credit and hopefully being able to move things upstream that can live upstream. That's it for a start. phosh's development documentation can be browsed here but is also available in the source code. Besides the projects mentioned above credits go to Purism for allowing me and others to work on the above and other parts related to moving Free Software on mobile Linux forward.

Emmanuel Kasper: Quick NetBSD serial console install on libvirt

I wanted to set up a small VM with NetBSD to test a couple of virt-install option. It turns out it you can get to the installer prompt quite fast. get the NetBSD installer for serial console:

wget https://cdn.netbsd.org/pub/NetBSD/NetBSD-9.1/i386/installation/cdrom/boot-com.iso

start the install

$ virt-install \
--connect qemu:///session \
--name netbsd \
--ram 64 \
--vcpus 2 \
--disk path=$HOME/netbsd.qcow2,size=4,bus=scsi,format=qcow2 \
--controller type=scsi,model=virtio-scsi \
--cdrom=boot-com.iso \
--virt-type kvm \
--os-variant netbsd8.0 \
--graphics none \
--arch i686 \
--console pty,target_type=serial

This will start a VM in usermode networking, so no need to be root, but the VM won t be reachable from the outside world, except if you add qemu usermode port forwarding.

27 December 2020

John Goerzen: Asynchronous Email: Exim over NNCP (or UUCP)

Following up to yesterday s article about how NNCP rehabilitates asynchronous communication with modern encryption and onion routing, here is the first of my posts showing how to put it into action. Email is a natural fit for async; in fact, much of early email was carried by UUCP. It is useful for an airgapped machine to be able to send back messages; errors from cron, results of handling incoming data, disk space alerts, etc. (Of course, this would apply to a non-airgapped machine also). The NNCP documentation already describes how to do this for Postfix. Here I will show how to do it for Exim. A quick detour to UUCP land When you encounter a system such as email that has instructions for doing something via UUCP, that should be an alert to you that here is some very relevant information for doing this same thing via NNCP. The syntax is different, but broadly, here s a table of similar NNCP commands:

Purpose	UUCP	NNCP
Connect to remote system	uucico -s, uupoll	nncp-call, nncp-caller
Receive connection (pipe, daemon, etc)	uucico (-l or similar)	nncp-daemon
Request remote execution, stdin piped in	uux	nncp-exec
Copy file to remote machine	uucp	nncp-file
Copy file from remote machine	uucp	nncp-freq
Process received requests	uuxqt	nncp-toss
Move outbound requests to dir (for USB stick, airgap, etc)	N/A	nncp-xfer
Create streaming package of outbound requests	N/A	nncp-bundle

If you used UUCP back in the day, you surely remember bang paths. I will not be using those here. NNCP handles routing itself, rather than making the MTA be aware of the network topology, so this simplifies things considerably. Sending from Exim to a smarthost One common use for async email is from a satellite system: one that doesn t receive mail, or have local mailboxes, but just needs to get email out to the Internet. This is a common situation even for conventionally-connected systems; in Exim speak, this is a satellite system that routes mail via a smarthost. That is, every outbound message goes to a specific target, which then is responsible for eventual delivery (over the Internet, LAN, whatever). This is fairly simple in Exim. We actually have two choices for how to do this: bsmtp or rmail mode. bsmtp (batch SMTP) is the more modern way, and is essentially a derivative of SMTP that explicitly can be queued asynchronously. Basically it s a set of SMTP commands that can be saved in a file. The alternative is rmail (which is just an alias for sendmail these days), where the data is piped to rmail/sendmail with the recipients given on the command line. Both can work with Exim and NNCP, but because we re doing shiny new things, we ll use bsmtp. These instructions are loosely based on the Using outgoing BSMTP with Exim HOWTO. Some of these may assume Debianness in the configuration, but should be easily enough extrapolated to other configs as well. First, configure Exim to use satellite mode with minimal DNS lookups (assuming that you may not have working DNS anyhow). Then, in the Exim primary router section for smarthost (router/200_exim4-config_primary in Debian split configurations), just change transport = remote_smtp_smarthost to transport = nncp. Now, define the NNCP transport. If you are on Debian, you might name this transports/40_exim4-config_local_nncp:

nncp:
  debug_print = "T: nncp transport for $local_part@$domain"
  driver = pipe
  user = nncp
  batch_max = 100
  use_bsmtp
  command = /usr/local/nncp/bin/nncp-exec -noprogress -quiet hostname_goes_here rsmtp
.ifdef REMOTE_SMTP_HEADERS_REWRITE
  headers_rewrite = REMOTE_SMTP_HEADERS_REWRITE
.endif
.ifdef REMOTE_SMTP_RETURN_PATH
  return_path = REMOTE_SMTP_RETURN_PATH
.endif

This is pretty straightforward. We pipe to nncp-exec, run it as the nncp user. nncp-exec sends it to a target node and runs whatever that node has called rsmtp (the command to receive bsmtp data). When the target node processes the request, it will run the configured command and pipe the data in to it. More complicated: Routing to various NNCP nodes Perhaps you would like to be able to send mail directly to various NNCP nodes. There are a lot of ways to do that. Fundamentally, you will need a setup similar to the UUCP example in Exim s manualroute manual, which lets you define how to reach various hosts via UUCP/NNCP. Perhaps you have a star topology (every NNCP node exchanges email with a central hub). In the NNCP world, you have two choices of how you do this. You could, at the Exim level, make the central hub the smarthost for all the side nodes, and let it redistribute mail. That would work, but requires decrypting messages at the hub to let Exim process. The other alternative is to configure NNCP to just send to the destinations via the central hub; that takes advantage of onion routing and doesn t require any Exim processing at the central hub at all. Receiving mail from NNCP On the receiving side, first you need to configure NNCP to authorize the execution of a mail program. In the section of your receiving host where you set the permissions for the client, include something like this:

      exec:  
        rsmtp: ["/usr/sbin/sendmail", "-bS"]

The -bS option is what tells Exim to receive BSMTP on stdin. Now, you need to tell Exim that nncp is a trusted user (able to set From headers arbitrarily). Assuming you are running NNCP as the nncp user, then add MAIN_TRUSTED_USERS = nncp to a file such as /etc/exim4/conf.d/main/01_exim4-config_local-nncp. That s it! Some hosts, of course, both send and receive mail via NNCP and will need configurations for both.

26 December 2020

Sandro Tosi: Gmail and tracker.debian.org: add a label to team emails

Recently the Python Team started the migration from Alioth mailing list to a tracker.debian.org team.Alioth mailing list, being a real ml, included a List-Id header, which is handled by Gmail natively and which you can use to create a filter based on it.Tracker.d.o instead uses a custom header, X-Distro-Tracker-Team, and Gmail doesn't allow to create filters on custom headers. But there's a solution: Google Apps Scripts.

1. Apps script and its configuration

To create a new apps script, click "New project" from the scripts dashboard: this will open a code editor, so now cut and paste this code (change the label name to match the one you want to use, and the header too if you're going to use this for another team):

function tagPythonTeamEmails()  
  
  // search for all recent threads (hours is the smallest interval searchable)
  var threads = GmailApp.search("newer_than:1h");
  // this is the label i want to apply to the messages
  var label = GmailApp.getUserLabelByName("Debian_Python_Pkgs");  // <-- EDIT THIS

  for (var i = 0; i < threads.length; i++)  
     
    // get all messages in a given thread
    var messages = threads[i].getMessages();

    // for each message in the thread
    for (var j = 0; j < messages.length; j++)  
      var message = messages[j];
      // needed to have access to the full email: headers + body
      var email = message.getRawContent();
      // this is the header i want to check for: if present, add label and archive
      if (email.indexOf("X-Distro-Tracker-Team: python") > -1)    // <-- EDIT THIS
        threads[i].addLabel(label);
        threads[i].moveToArchive();

(This code is largely inspired by this StackOverflow answer.)

Given we're going to perform changes to the emails, we need to add extra permissions to the script; first we need to toggle the option "View > Show manifest file" which will show the appsscript.json file. Once done, edit it and update the scope of the script to include:

  "oauthScopes": [
    ...
    "https://www.googleapis.com/auth/gmail.readonly",
    "https://www.googleapis.com/auth/gmail.modify"
  ],

These new permission will be granted when we deploy the script, which we can actually do right now: from the editor window click on "Publish > Deploy from manifest..." and then "Install add-on"; it will make the app script available to operate on Gmail by opening the usual Google grants authorization popup.

2. Schedule the app script execution
Unlike filters defined on the web UI, apps scripts are not triggered every time a mail is received (there's a pub-sub mechanism, but it doesn't look like it would work for this), so if we want to execute periodically the new script, we need to create a time-driven trigger.
The simplest way is probably from the script editor (but you can also follow this doc): "View > Current project's triggers" and then "Add trigger" from there. For my specific use-case i set:
event source = time-driven
type of time based trigger = minutes timer
minute interval = 5 minutes
Failure notification settings = notify me immediately
it's not great that you have to wait 5 minutes for getting proper labels, and while it's true you can schedule the trigger for every minute, there are further considerations for that, discussed below.

3. Additional information
Be aware of quotas and limitations for Apps scripts; in particular the "Triggers total runtime" set to "90 min / day" is what led me to set the trigger every 510 mins.
Use the scripts dashboard (doc) to monitor the script execution logs, in the "My Executions" section, and specifically the "Duration" column: the script execution time depends exclusively on the number of emails to process at every execution, so fine-tune the trigger time to stay within your quota.
The newer_than search operator doesn't allow a granularity smaller than hours (although that's not even documented), so that means we're going to scan multiple times the same emails.
UPDATE (2020-12-26): I've already had to bump from every 5 to every 10 minutes, as otherwise i'd get Exception: Service invoked too many times for one day: gmail. (remember the quota!)

Sandro Tosi: QNAP firmware 4.5.1.1465: disable ssh management menu

as a good boy i just upgraded my QNAP NAS to the latest available firmware, 4.5.1.1465, but after the reboot there's an ugly surprise awaiting for meonce i ssh'd into the box to do my stuff, instead of a familiar bash prompt i'm greeted by a management menu that allows me to perform some basic management tasks or quit it and go back to the shell. i dont really need this menu (in particular because i have automations that regularly ssh into the box and they are not meant to be interactive).to disable it: edit /etc/profile and comment the line "[[ "admin" = "$USER" ]] && /sbin/qts-console-mgmt -f" (you can judge me later for sshing as root)UPDATE (2020-12-26): as pointed out in the comments, and discovered now after an upgrade, the menu comes back up after because /etc/profile is a system file for QNAP so gets replaced/updated by them often

Paul Tagliamonte: Reverse Engineering my Christmas Tree

Over the course of the last year and a half, I ve been doing some self-directed learning on how radios work. I ve gone from a very basic understanding of wireless communications (there s usually some sort of antenna, I guess?) all the way through the process of learning about and implementing a set of libraries to modulate and demodulate data using my now formidable stash of SDRs. I ve been implementing all of the RF processing code from first principals and purely based on other primitives I ve written myself to prove to myself that I understand each concept before moving on. I figured that there was a fun capstone to be done here - the blind reverse engineering and implementation of the protocol my cheep Amazon power switch uses to turn on and off my Christmas Tree. All the work described in this post was done over the course of a few hours thanks to help during the demodulation from Tom Bereknyei and hlieberman.

Going in blind When I first got my switch, I checked it for any FCC markings in order to look up the FCC filings to determine the operational frequency of the device, and maybe some other information such as declared modulation or maybe even part numbers and/or diagrams. However, beyond a few regulatory stickers, there were no FCC ids or other distinguishing IDs on the device. Worse yet, it appeared to be a whitelabeled version of another product, so searching Google for the product name was very unhelpful. Since operation of this device is unlicensed, I figured I d start looking in the ISM band. The most common band used that I ve seen is the band starting at `433.05MHz` up to `434.79MHz`. I fired up my trusty waterfall tuned to a center frequency of `433.92MHz` (since it s right in the middle of the band, and it let me see far enough up and down the band to spot the remote) and pressed a few buttons. Imagine my surprise when I realize the operational frequency of this device is `433.920MHz`, exactly dead center. Weird, but lucky! After taking a capture, I started to look at understanding what the modulation type of the signal was, and how I may go about demodulating it. Using inspectrum, I was able to clearly see the signal in the capture, and it immediately stuck out to my eye to be encoded using OOK / ASK. Next, I started to measure the smallest pulse, and see if I could infer the symbols per second, and try to decode it by hand. These types of signals are generally pretty easy to decode by eye. This wound up giving me symbol rate of 2.2 Ksym/s, which is a lot faster than I expected. While I was working by hand, Tom demodulated a few messages in Python, and noticed that if you grouped the bits into groups of 4, you either had a `1000` or a `1110` which caused me to realize this was encoded using something I saw documented elsewhere, where the 0 is a short pulse, and a 1 is a long pulse, not unlike morse code, but where each symbol takes up a fixed length of time (monospace morse code?). Working on that assumption, I changed my inspectrum symbol width, and demodulated a few more by hand. This wound up demodulating nicely (and the preamble / clock sync could be represented as repeating `0`s, which is handy!) and gave us a symbol rate of 612(ish) symbols per second a lot closer to what I was expecting. If we take the code for on in the inspectrum capture above and demodulate it by hand, we get `0000000000110101100100010` (treat a short pulse as a 0, and a long pulse as a 1). If you re interested in following along at home, click on the inspectrum image, and write down the bits you see, and compare it to what I have! Right, so it looks like from what we can tell so far that the packet looks something like this:

preamble / sync

stuff

Next, I took a capture of all the button presses and demodulated them by hand, and put them into a table to try and understand the format of the messages:

Button Demod'd Bits

On 0000000000110101100100010

Off 00000000001101011001010000

Dim Up 0000000000110101100110100

Dim Down 0000000000110101100100100

Timer 1h 0000000000110101100110010

Timer 2h 0000000000110101100100110

Timer 4h 0000000000110101100100000

Dim 100% 0000000000110101000101010

Dim 75% 00000000001101010001001100

Dim 50% 00000000001101010001001000

Dim 25% 0000000000110101000100000

Great! So, this is enough to attempt to control the tree with, I think so I wrote a simple modulator. My approach was to use the fact that I can break down a single symbol into 4 sub-symbol components which is to say, go back to representing a `1` as `1110`, and a `0` as `1000`. This let me allocate IQ space for the symbol, break the bit into 4 symbols, and if that symbol is 1, write out values from a carrier wave (`cos` in the `real` values, and `sin` in the `imaginary` values) to the buffer. Now that I can go from bits to IQ data, I can transmit that IQ data using my PlutoSDR or HackRF and try and control my tree. I gave it a try, and the tree blinked off! Success! But wait that s not enough for me I know I can t just demodulate bits and try and replay the bits forever there s stuff like addresses and keys and stuff, and I want to get a second one of these working. Let s take a look at the bits to see if we spot anything fun & interesting. At first glance, a few things jumped out at me as being weird? First is that the preamble is 10 bits long (fine, let s move along - maybe it just needs 8 in a row and there s two to ensure clocks sync?). Next is that the messages are not all the same length. I double (and triple!) checked the messages, and it s true, the messages are not all the same length. Adding an extra bit at the end didn t break anything, but I wonder if that s just due to the implementation rather than the protocol. But, good news, it looks like we have a stable prefix to the messages from the remote must be my device s address! The stable 6 bits that jump out right away are `110101`. Something seems weird, though, 6 bits is a bit awkward, even for a bit limited embedded device. Why 6? But hey, wait, we had 10 bits in the preamble, what if we have an 8 bit address meaning my device is `00110101`, and the preamble is 8 `0` symbols! Those are numbers that someone working on an 8 bit aligned platform would pick! To test this, I added a `0` to the preamble to see if the message starts at the first `1`, or if it requires all the bits to be fully decoded, and lo and behold, the tree did not turn on or off. This would seem to me to confirm that the 0s are part of the address, and I can assume we have two 8 bit aligned bytes in the prefix of the message.

preamble / sync

address

stuff

Now, when we go through the 9-10 bits of stuff , we see all sorts of weird bits floating all over the place. The first 4 bits look like it s either `1001` or `0001`, but other than that, there s a lot of chaos. This is where things get really squishy. I needed more information to try and figure this out, but no matter how many times I sent a command it was always the same bits (so, no counters), and things feel very opaque still. The only way I was going to make any progress is to get another switch and see how the messages from the remote change. Off to Amazon I went, and ordered another switch from the same page, and eagerly waited its arrival.

Button	Demod'd Bits
On	0000000000110101100100010
Off	00000000001101011001010000
Dim Up	0000000000110101100110100
Dim Down	0000000000110101100100100
Timer 1h	0000000000110101100110010
Timer 2h	0000000000110101100100110
Timer 4h	0000000000110101100100000
Dim 100%	0000000000110101000101010
Dim 75%	00000000001101010001001100
Dim 50%	00000000001101010001001000
Dim 25%	0000000000110101000100000

Switch #2 The second switch showed up, and I hurriedly unboxed the kit, put batteries into the remote, and fired up my SDR to take a capture. After I captured the first button ( Off ), my heart sunk as I saw my lights connected to Switch #1 flicker off. Apparently the new switch and the old switch have the same exact address. To be sure, I demodulated the messages as before, and came out with the exact same bit pattern. This is a setback and letdown I was hoping to independently control my switches, but it also means I got no additional information about the address or button format. The upside to all of this, though, is that because the switches are controlled by either remote, I only needed one remote, so why not pull it apart and see if I can figure out what components it s using to transmit, and find any datasheets I can. The PCB was super simple, and I wound up finding a WL116SC IC on the PCB. After some googling, I found a single lone datasheet, entirely in Chinese. Thankfully, Google Translate seems to have worked well enough on technical words, and I was able to put together at least a little bit of understanding based on the documentation that was made available. I took a few screenshots below - I put the google translated text above the hanzi. From that sheet, we can see we got the basics of the 1 and 0 symbol encoding right (I was halfway expecting the bits to be flipped), and a huge find by way of a description of the bits in the message! It s a bummer that we missed the clock sync / preamble pulse before the data message, but that s OK somehow. It also turns out that 8 or 10 bit series of of 0"s wasn t clock sync at all - it was part of the address! Since it also turns out that all devices made by this manufacturer have the hardcoded address of `[]byte 0x00, 0x35` , that means that the vast majority of bits sent are always going to be the same for any button press on any remote made by this vendor. Seems like a waste of bits to me, but hey, what do I know. Additionally, this also tells us the trailing zeros are not part of the data encoding scheme, which is progress!

address

keycode

Now, working on the assumptions validated by the datasheet, here s the updated list of scancodes we ve found:

Button Scancode Bits Integer

On 10010001 145 / 0x91

Off 10010100 148 / 0x94

Dim Up 10011010 154 / 0x9A

Dim Down 10010010 146 / 0x92

Timer 1h 10011001 154 / 0x99

Timer 2h 10010011 147 / 0x93

Timer 4h 10010000 144 / 0x90

Dim 100% 00010101 21 / 0x15

Dim 75% 00010011 19 / 0x13

Dim 50% 00010010 18 / 0x12

Dim 25% 00010000 16 / 0x10

Interestingly, I think the Dim keys may have a confirmation that we have a good demod the codes on the bottom are missing the most significant bit, and when I look back at the scancode table in the datasheet, they make an interesting pattern the bottom two rows, right and left side values match up! If you take a look, Dim 100% is S1 , Dim 75% is S19 , Dim 50% is S8 , and Dim 25% is S20 . Cool! Since none of the other codes line up, I am willing to bet the most significant bit is a Combo indicator, and not part of the button (leaving 7 bits for the keycode). And even more interestingly, one of our scancodes ( Off , which is 0x94) shows up just below this table, in the examples. Over all, I think this tells us we have the right bits to look at for determining the scan code! Great news there!

Button	Scancode Bits	Integer
On	10010001	145 / 0x91
Off	10010100	148 / 0x94
Dim Up	10011010	154 / 0x9A
Dim Down	10010010	146 / 0x92
Timer 1h	10011001	154 / 0x99
Timer 2h	10010011	147 / 0x93
Timer 4h	10010000	144 / 0x90
Dim 100%	00010101	21 / 0x15
Dim 75%	00010011	19 / 0x13
Dim 50%	00010010	18 / 0x12
Dim 25%	00010000	16 / 0x10

Back to the modulation! So, armed with this knowledge, I was able to refactor my code to match the timings and understanding outlined by the datasheet and ensure things still work. The switch itself has a high degree of tolerance, so being wildly off frequency or a wildly wrong symbol rate may actually still work. It s hard to know if this is more or less correct, but matching documentation seems like a more stable foundation if nothing else. This code has been really reliable, and tends to work just as well as the remote from what I ve been able to determine. I ve been using incredibly low power to avoid any interference, and it s been very robust - a testament to the engineering that went into the outlet hardware, even though it cost less than of a lot of other switches! I have a lot of respect for the folks who built this device - it s incredibly simple, reliable and my guess is this thing will keep working even in some fairly harsh RF environments. The only downside is the fact the manufacturer used the same address for all their devices, rather than programming a unique address for each outlet and remote when the underlying WL116SC chip supports it. I m sure this was done to avoid complexity in assembly (e.g. pairing the remote and outlet, and having to keep those two items together during assembly), but it s still a bummer. I took apart the switch to see if I could dump an EEPROM and change the address in ROM, but the entire thing was potted in waterproof epoxy, which is a very nice feature if this was ever used outdoors. Not good news for tinkering, though!

Unsolved Mysteries At this point, even though I understand the protocol enough to control the device, it still feels like I hit a dead end in my understanding. I m not able to figure out how exactly the scancodes are implemented, and break them down into more specific parts. They are stable and based on the physical wiring of the remote, so I think I m going to leave it a magic number. I have what I was looking for, and these magic constants appear to be the right one to use, even if I did understand how to create the codes itself. This does leave us with a few bits we never resolved, which I ll memorialize below just to be sure I don t forget about them. Question #1: According to the datasheet there should be a preamble. Why do I not see one leading the first message? My hunch is that the trailing 0 at the end of the payload is actually just the preamble for the next message (always rendering the first message invalid?). This would let us claim there s an engineering reason why we are ignoring the weird bit, and also explain away something from the documentation. It s just weird that it wouldn t be present on the first message. This theory is mostly confirmed by measuring the timing and comparing it to the datasheet, but it s not exactly in line with the datasheet timings either (specifically, it s off by 200 s, which is kinda a lot for a system using 400 s timings). I think I could go either way on the last 0 being the preamble for the next message. It could be that the first message is technically invalid, or it could also be that this was not implemented or actively disabled by the vendor for this specific application / device. It s really hard to know without getting the source code for the WL116SC chip in this specific remote or the source in the outlet itself. Question #2: Why are some keycodes 8 bits and others 9 bits? I still have no idea why there sometimes 8 bits (for instance, On ) and other times there are 9 bits (for instance, Off ) in the 8 bit keycode field. I spent some time playing with the trailing zeros, when I try and send an Off with the most significant 8 bits (without the least significant / last 9th bit, which is a 0 ), it does not turn the tree off. If I send an On with 9 bits (an additional 0 after the least significant bit), it does work, but both On and Off work when I send 10, 11 or 12 bits padded with trailing zeros. I suspect my outlet will ignore data after the switch is done reading bits regardless of trailing zeros. The docs tell me there should only be 8 bits, but it won t work unless I send 9 bits for some commands. There s something fishy going on here, and the datasheet isn t exactly right either way. Question #3: How in the heck do those scancodes work? This one drove me nuts. I ve spent countless hours on trying to figure this out, including emailing the company that makes the WL116SC (they re really nice!), and even though they were super kind and generous with documentation and example source, I m still having a hard time lining up their documentation and examples with what I see from my remote. I think the manufacturer of my remote and switch has modified the protocol enough to where there s actually something different going on here. Bummer. I wound up in my place of last resort asking friends over Signal to try and see if they could find a pattern, as well as making multiple pleas to the twittersphere, to no avail (but thank you to Ben Hilburn, devnulling, Andreas Bombe and Larme for your repiles, help and advice!) I still don t understand how they assemble the scan code for instance, if you merely add, you won t know if a key press of `0x05` is `0x03` + `0x02` or if it s `0x01` + `0x04`. On the other hand, treating it as two 4-bit integers won t work for `0x10` to `0x15` (since they need 5 bits to represent). It s also likely the most significant bit is a combo indicator, which only leaves 7 bits for the actual keypress data. Stuffing 10 bits of data into 7 bits is likely resulting in some really intricate bit work. On a last ditch whim, I tried to XOR the math into working, but some initial brute forcing to make the math work given the provided examples did not result in anything. It could be a bitpacked field that I don t understand, but I don t think I can make progress on that without inside knowledge and much more work. Here s the table containing the numbers I was working off of:

Keys Key Codes Scancode

S3 + S9 0x01 + 0x03 0x96

S6 + S12 0x07 + 0x09 0x94

S22 + S10 0x0D + 0x0F 0x3F

If anyone has thoughts on how these codes work, I d love to hear about it! Send me an email or a tweet or something - I m a bit stumped. There s some trick here that is being used to encode the combo key in a way that is decodeable. If it s actually not decodeable (which is a real possibility!), this may act as a unique button combo hash which allows the receiver to not actually determine which keys are pressed, but have a unique button that gets sent when a combo is used. I m not sure I know enough to have a theory as to which it may be.

Keys	Key Codes	Scancode
S3 + S9	0x01 + 0x03	0x96
S6 + S12	0x07 + 0x09	0x94
S22 + S10	0x0D + 0x0F	0x3F

John Goerzen: Rehabilitating Asynchronous Communication with NNCP: A Cross Between Tor, ssh, and UUCP

Have you ever been traveling, shot a ton of photos and videos, but were annoyed to find it was saturating the terrible wifi you had access to? Maybe you d wish the upload to pause until you get somewhere else, but then pausing syncing on your Nextcloud/Syncthing/Dropbox would also pause other syncing you didn t want to pause. Or you have trouble backing up your laptop when not at home, in a way that won t accidentaly eat up your cell phone data. There are ways to help with this: asynchronous transfer. Here s a lot of background. If you want to see how encrypted, onion-routed UUCP looks, skip ahead to the NNCP section! There is an old saying: When all you have is a hammer, every problem looks like a nail. We have this wonderful tool called ssh available, and it is pervasive and well-understood, so we tend to use it. But we ve missed out on some benefits of asynchronous processing that we actually used to have more frequently. Of course, we are all used to some asynchronous services in our lives. Email is a popular example: most mail clients work offline and will transmit stored messages when the mail server becomes reachable. Mail servers themselves work that way, too. Many instant messaging platforms do as well. Even some backup systems do. Bacula/Bareos, for instance, spools all backup data to disk on the system connected to the tape drive, and from there to the tape itself. They do this for several reasons, but primarily the fact that if tape drives are not fed with data at their design speed, it can cause physical damage to the tape or even the drive. It causes the drive to have to pause, and seek backwards to reposition for the next write. This creates excessive travel of the tape over the write heads, causing a condition known as tape shine where the tape is damaged prematurely. Here are some problems people often run into when sending data across a network (or the Internet) synchronously:

One side of the communication is much faster than the other
Internet issues interrupting communications mid-stream
Slow Internet causing processes to take much longer than planned, resulting in unexpected results or locking issues
Physical damage due to performance issues

Of course, there are plenty of situations where synchronous communication is a must. For instance:

When the status of the transaction at the remote end must be known immediately
When there is insufficient space to spool a job s data

I suspect that the reason we don t do more asynchronous processing these days, despite it being strong in the Unix heritage, is the lack of modern tools to do it. Let s explore some more. Some of my use cases I run ZFS on all my systems that support it: file server, laptops, workstations, etc. It is only natural to use ZFS send/receive to do backups, and I do. However, when I am traveling, my laptop never gets backed up, because the backups are pulled from the backup system. Sure, there are ways around that; a VPN, for instance. But then we have the situation where sometimes I do not want to send the backup even if I have a working Internet connection: perhaps I m tethered to a mobile connection and it would be expensive to do so, or I m on hotel Wifi that is flaky and slow and I don t want to give up any of its meager bandwidth. I have another backup-related problem. I have a remote server, which until recently was using extremely slow disks. If I made significant changes, the backup would take the better part of a day. That s annoying when I try to back up hourly. So of course I had to implement locking, but then that means none of my other machines would back up that day either. Once I needed to transmit about 2TB of data. My home Internet connection was terribly slow, and I calculated it would take multiple months to do this. So I took to manually copying parts of the data to my laptop, and whenever I d find an airport or coffee shop with faster Internet than at home, I d send off those bits from it. But it took a ton of work. The bespoke asynchronous problem And that ton of work is perhaps why we aren t doing more of this. There s been no great standard solution, so it s all roll your own when you need to. So we just use ssh, because it s easier and usually good enough . But as I wrote in my recent article on airgapped backups, there are reasons to go async. Solutions Wouldn t it be great to be able to queue up data for a machine, and let it get there in whatever way it can? Maybe a fast Internet connection is found, or via Tor, or via copying to a USB stick, or via radio broadcast? It would make many of these scenarios a lot easier. And there are ways for this now, with modern security! We have some tools on Linux for this: git-annex for storage and migration, syrep for synchronization, and NNCP for file transfer and remote execution (could be combined with some of these other tools). Let s dive in to NNCP. NNCP If you already know UUCP, think of NNCP as UUCP brought into the modern era, with modern security and tools. Basically, NNCP permits you to send files to a remote system, request files from a remote system, and pipe data to an NNCP command that requests execution remotely. So you could, say, pipe a zfs send to NNCP which sends it to the remote and pipes it to zfs receive when it gets there. NNCP has a delay-tolerant, resumable protocol that can run over just about any reliable connection: TCP, serial, Tor, radios of various kinds, you name it. But that s not all; it also can dump its queue onto something like a USB stick for transport, or even make a tar-style stream that could be munged however you like. If you want to get fancy, you can assign priorities to data packets, so that, for instance, outbound email will always get sent before that 1TB file you ve got to send also. You can also configure it so that certain carriers handle certain priorities of data; your cell phone would only handle the most urgent, but a USB stick would take anything. NNCP is source-routed; you can tell it that the way that Bob reaches Alice is via Carl, then Betty. Bob can generate a message that will be sent along that route, fully encrypted and authenticated at each step of the way; Carl can t see the content of the message or even anything about it other than its next hop. How this helps Let s revisit some of my scnearios with NNCP. For the laptop being backed up, while traveling it can queue up its backups, or photos, or videos, or whatever. They could be triggered by a command when on a good connection, or automatically. The data could be copied to USB and given to a friend to transmit; perfectly safe due to encryption. Or it could all wait until arriving at home, safely out of your other syncing directories. The NNCP documentation has an example of this. For the server being backed up slowly, that s easily solved; the slow backup would simply be queued up, and transmitted and processed when it s ready. This wouldn t interrupt other backups. How about the 2TB transmission problem? That s also made a lot easier. A command could be run to fill up a USB stick with parts of the queue, then that USB stick plugged in and transmitted whenever at a fast location. Repeat as needed while the slow system continues its upload of the remaining bits. NNCP has a lot of interesting use cases documented as well. If you are already familiar with how public keys work in SSH, then NNCP should be immediately familiar as well. It is a similar concept (though arguably somewhat easier to set up). I am working on setting up a NNCP network, and will have more posts on how to do so once I ve got it going. In the meantime, the documentation for the project is also pretty good.

Next.

Previous.