Search Results: "ag"

2 December 2022

Reproducible Builds (diffoscope): diffoscope 228 released

The diffoscope maintainers are pleased to announce the release of diffoscope version 228. This version includes the following changes: 
[ FC Stegerman ]
* As an optimisation, don't run apktool if no differences are detected before
  the signing block. (Closes: reproducible-builds/diffoscope!105)
[ Chris Lamb ]
* Support both the python3-progressbar and python3-progressbar2 Debian
  packages, two modules providing the "progressbar" Python module.
  (Closes: reproducible-builds/diffoscope#323)
* Ensure we recommend apksigcopier. (Re: reproducible-builds/diffoscope!105)
* Make the code clearer around generating the Debian substvars and tidy
  generation of os_list.
* Update copyright years.
You find out more by visiting the project homepage.

1 December 2022

Dirk Eddelbuettel: spdl 0.0.2 on CRAN: First Update

A first update to the recently-released package spdl is now om CRAN. The key focus of spdl is a offering the same interface from both R and C++ for logging by relying on spdlog via my RcppSpdlog package. This release exposes simple helpers fmt() (to format text according to the included fmt library) and cat() which formats and prints. The very short NEWS entry for this release follows.

Changes in spld version 0.0.2 (2022-12-01)
  • Helper functions fmt() and cat() have been added

Courtesy of my CRANberries, there is also a diffstat report. More detailed information is on the spdl page. If you like this or other open-source work I do, you can sponsor me at GitHub.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

30 November 2022

Enrico Zini: Things I learnt in November 2022

Debian: Python: 
>>> name="test"
>>> print(f" name= ")
name='test'
>>> print(f" 3*8= ")
3*8=24
Leaflet:

Matthew Garrett: Making unphishable 2FA phishable

One of the huge benefits of WebAuthn is that it makes traditional phishing attacks impossible. An attacker sends you a link to a site that looks legitimate but isn't, and you type in your credentials. With SMS or TOTP-based 2FA, you type in your second factor as well, and the attacker now has both your credentials and a legitimate (if time-limited) second factor token to log in with. WebAuthn prevents this by verifying that the site it's sending the secret to is the one that issued it in the first place - visit an attacker-controlled site and said attacker may get your username and password, but they won't be able to obtain a valid WebAuthn response.

But what if there was a mechanism for an attacker to direct a user to a legitimate login page, resulting in a happy WebAuthn flow, and obtain valid credentials for that user anyway? This seems like the lead-in to someone saying "The Aristocrats", but unfortunately it's (a) real, (b) RFC-defined, and (c) implemented in a whole bunch of places that handle sensitive credentials. The villain of this piece is RFC 8628, and while it exists for good reasons it can be used in a whole bunch of ways that have unfortunate security consequences.

What is the RFC 8628-defined Device Authorization Grant, and why does it exist? Imagine a device that you don't want to type a password into - either it has no input devices at all (eg, some IoT thing) or it's awkward to type a complicated password (eg, a TV with an on-screen keyboard). You want that device to be able to access resources on behalf of a user, so you want to ensure that that user authenticates the device. RFC 8628 describes an approach where the device requests the credentials, and then presents a code to the user (either on screen or over Bluetooth or something), and starts polling an endpoint for a result. The user visits a URL and types in that code (or is given a URL that has the code pre-populated) and is then guided through a standard auth process. The key distinction is that if the user authenticates correctly, the issued credentials are passed back to the device rather than the user - on successful auth, the endpoint the device is polling will return an oauth token.

But what happens if it's not a device that requests the credentials, but an attacker? What if said attacker obfuscates the URL in some way and tricks a user into clicking it? The user will be presented with their legitimate ID provider login screen, and if they're using a WebAuthn token for second factor it'll work correctly (because it's genuinely talking to the real ID provider!). The user will then typically be prompted to approve the request, but in every example I've seen the language used here is very generic and doesn't describe what's going on or ask the user. AWS simply says "An application or device requested authorization using your AWS sign-in" and has a big "Allow" button, giving the user no indication at all that hitting "Allow" may give a third party their credentials.

This isn't novel! Christoph Tafani-Dereeper has an excellent writeup on this topic from last year, which builds on Nestori Syynimaa's earlier work. But whenever I've talked about this, people seem surprised at the consequences. WebAuthn is supposed to protect against phishing attacks, but this approach subverts that protection by presenting the user with a legitimate login page and then handing their credentials to someone else.

RFC 8628 actually recognises this vector and presents a set of mitigations. Unfortunately nobody actually seems to implement these, and most of the mitigations are based around the idea that this flow will only be used for physical devices. Sadly, AWS uses this for initial authentication for the aws-cli tool, so there's no device in that scenario. Another mitigation is that there's a relatively short window where the code is valid, and so sending a link via email is likely to result in it expiring before the user clicks it. An attacker could avoid this by directing the user to a domain under their control that triggers the flow and then redirects the user to the login page, ensuring that the code is only generated after the user has clicked the link.

Can this be avoided? The best way to do so is to ensure that you don't support this token issuance flow anywhere, or if you do then ensure that any tokens issued that way are extremely narrowly scoped. Unfortunately if you're an AWS user, that's probably not viable - this flow is required for the cli tool to perform SSO login, and users are going to end up with broadly scoped tokens as a result. The logs are also not terribly useful.

The infuriating thing is that this isn't necessary for CLI tooling. The reason this approach is taken is that you need a way to get the token to a local process even if the user is doing authentication in a browser. This can be avoided by having the process listen on localhost, and then have the login flow redirect to localhost (including the token) on successful completion. In this scenario the attacker can't get access to the token without having access to the user's machine, and if they have that they probably have access to the token anyway.

There's no real moral here other than "Security is hard". Sorry.

comment count unavailable comments

Russell Coker: Links November 2022

Here s the US Senate Statement of Frances Haugen who used to work for Facebook countering misinformation and espionage [1]. She believes that Facebook is capable of dealing with the online radicalisation and promotion of bad things on it s platform but is unwilling to do so for financial reasons. We need strong regulation of Facebook and it probably needs to be broken up. Interesting article from The Atlantic about filtered cigarettes being more unhealthy than unfiltered [2]. Every time I think I know how evil tobacco companies are I get surprised by some new evidence. Cory Doctorow wrote an insightful article about resistance to rubber hose cryptanalysis [3]. Cory Doctorow wrote an interesting article When Automation Becomes Enforcement with a new way of thinking about Snapchat etc [4]. Cory Doctorow wrote an insightful and informative article Big Tech Isn t Stealing News Publishers Content, It s Stealing Their Money [5] which should be read by politicians from all countries that are trying to restrict quoting news on the Internet. Interesting articl;e on Santiago Genoves who could be considered as a pioneer of reality TV for deliberately creating disputes between a group of young men and women on a raft in the Atlantic for 3 months [6]. Matthew Garrett wrote an interesting review of the Freedom Phone, seems that it s not good for privacy and linked to some companies doing weird stuff [7]. Definitely worth reading. Cory Doctorow wrote an interesting and amusing article about backdoors for machine learning [8] Petter Reinholdtsen wrote an informative post on how to make a bootable USB stick image from an ISO file [9]. Apparently Lenovo provides ISO images to update laptops that don t have DVD drives. :( Barry Gander wrote an interesting article about the fall of Rome and the decline of the US [10]. It s a great concern that the US might fail in the same way as Rome. Ethan Siegel wrote an interesting article about Iapetus, a moon of Saturn that is one of the strangest objects in the solar system [11]. Cory Doctorow s article Revenge of the Chickenized Reverse-Centaurs has some good insights into the horrible things that companies like Amazon are doing to their employees and how we can correct that [12]. Charles Stross wrote an insightful blog post about Billionaires [13]. They can t do much for themselves with the extra money beyond about $10m or $100m (EG Steve Jobs was unable to extend his own life much when he had cancer) and their money is trivial when compared to the global economy. They are however effective parasites capable of performing great damage to the country that hosts them. Cory Doctorow has an interesting article about how John Deere is being evil again [14]. This time with potentially catastrophic results.
Related posts:
  1. Links September 2022 Tony Kern wrote an insightful document about the crash of...
  2. Links November 2021 The Guardian has an amusing article by Sophie Elmhirst about...
  3. Links September 2020 MD5 cracker, find plain text that matches MD5 hash [1]....

Utkarsh Gupta: FOSS Activites in November 2022

Here s my (thirty-eighth) monthly but brief update about the activities I ve done in the F/L/OSS world.

Debian
This was my 47th month of actively contributing to Debian. I became a DM in late March 2019 and a DD on Christmas 19! \o/ There s a bunch of things I do, both, technical and non-technical. Here are the things I did this month:

Debian Uploads

Other $things:
  • Sponsoring stuff for non-DDs.
  • Mentoring for newcomers.
  • Moderation of -project mailing list.

Ubuntu
This was my 22nd month of actively contributing to Ubuntu. Now that I joined Canonical to work on Ubuntu full-time, there s a bunch of things I do! \o/ I mostly worked on different things, I guess. I was too lazy to maintain a list of things I worked on so there s no concrete list atm. Maybe I ll get back to this section later or will start to list stuff from the fall, as I was doing before. :D

Debian (E)LTS
Debian Long Term Support (LTS) is a project to extend the lifetime of all Debian stable releases to (at least) 5 years. Debian LTS is not handled by the Debian security team, but by a separate group of volunteers and companies interested in making it a success. And Debian Extended LTS (ELTS) is its sister project, extending support to the stretch and jessie release (+2 years after LTS support). This was my thirty-eighth month as a Debian LTS and twenty-nine month as a Debian ELTS paid contributor.
I worked for 41.00 hours for LTS and 30.25 hours for ELTS.

LTS CVE Fixes and Announcements:

ELTS CVE Fixes and Announcements:
  • Issued ELA 731-1, fixing CVE-2022-39377, for sysstat.
    For Debian 9 stretch, these problems have been fixed in version 11.4.3-2+deb9u1.
    For Debian 8 jessie, these problems have been fixed in version 11.0.1-1+deb8u1.
  • Issued ELA 749-1, fixing CVE-2022-41325, for vlc.
    For Debian 9 stretch, these problems have been fixed in version 3.0.17.4-0+deb9u2.
  • Issued ELA 750-1 for a new upstream version update of clamav.
    For Debian 9 stretch, the package has been updated to version 0.103.7+dfsg-0+deb9u1. For Debian 8 jessie, the package has been updated to version 0.103.7+dfsg-0+deb8u1.
  • Started to look at other set of packages.

Other (E)LTS Work:
  • Front desk duty from 21-11 until 27-11 for both, LTS and ELTS.
  • Triaged jqueryui, open-vm-tools, systemd, ffmpeg, lava, pngcheck, snapd, vlc, g810-led, libpgjava, dropbear, python3.5, python3.4, clamav, systat, postgresql-11, and mariadb-10.1.
  • Marked CVE-2009-1143/open-vm-tools as postponed for buster, stretch and jessie.
  • Marked CVE-2022-45873/systemd as not-affected in stretch and jessie.
  • Marked CVE-2022-396 4,5 /ffmpeg as postponed for buster and stretch.
  • Marked CVE-2022-45061/python3. 4,5 as postponed for stretch and jessie.
  • Marked CVE-2022-31160/jqueryui as not-affected for jessie instead.
  • Noted CVE-2022-45061/python3.4 to be marked as postponed; only things to fix is the armhf FTBFS.
  • Auto EOL d linux, libpgjava, nvidia-graphics-drivers, maradns, chromium, and glance for ELTS.
  • Helped and assisted new contributors joining Freexian (LTS/ELTS/internally).
  • Answered questions (& discussions) on IRC (#debian-lts and #debian-elts) and Matrix.
  • Participated and helped fellow members with their queries via private mail and chat.
  • General and other discussions on LTS private and public mailing list.
  • Attended the monthly meeting held on IRC on November 24th.
Until next time.
:wq for today.

Russ Allbery: Review: The Fed Unbound

Review: The Fed Unbound, by Lev Menand
Publisher: Columbia Global Reports
Copyright: 2022
ISBN: 1-7359137-1-5
Format: Kindle
Pages: 156
The Fed Unbound is a short non-fiction exploration of US Federal Reserve actions to reducing systemic risk caused by shadow banking. Its particular focus is the role of the Fed from the 2008 financial crisis to the present, including the COVID shock, but it includes a history of what Menand calls the "American Monetary Settlement," the political compromise that gave rise to the Federal Reserve. In Menand's view, a central cause of instability in the US financial system (and, given the influence of the dollar system, the world financial system as well) is shadow banking: institutions that act as banks without being insured like banks or subject to bank regulations. A bank, in this definition, is an organization that takes deposits. I'm simplifying somewhat, but what distinguishes a deposit from a security or an investment is that deposits can be withdrawn at any time, or at least on very short notice. When you want to spend the money in your checking account, you don't have to wait for a three-month maturity period or pay an early withdrawal penalty. You simply withdraw the money, with immediate effect. This property is what makes deposits "money," rather than something that you can (but possibly cannot) sell for money, such as stocks or bonds. Most people are familiar with the basic story of how banks work. Essentially no bank simply takes people's money and puts it in a vault until the person wants it again. If that were the case, you would need to pay the bank to store your money. Instead, a bank takes in deposits and then lends some portion of that money out to others. Those loans, for things like cars or houses or credit card spending, come due over time, with interest. The interest rate the bank charges on the loans is much higher than the rate it has to pay on its deposits, and it pockets the difference. The problem with this model, of course, is that the bank doesn't have your money, so if all the depositors go to the bank at the same time and ask for their money, the bank won't be able to repay them and will collapse. (See, for example, the movie It's a Wonderful Life, or Mary Poppins, or any number of other movies or books.) Retail banks are therefore subject to stringent regulations designed to promote public trust and to ensure that traditional banking is a boring (if still lucrative) business. Banks are also normally insured, which in the US means that if they do experience a run, federal regulators will step in, shut down the bank in an orderly fashion, and ensure every depositor gets their money back (at least up to the insurance limit). Alas, if you thought people would settle for boring work that makes a comfortable profit, you don't know the financial industry. Highly-regulated insured deposits are less lucrative than acting like a bank without all of those restrictions and rules and deposit insurance payments. As Menand relates in his brief history of US banking, financial institutions constantly invent new forms of deposits with similar properties but without all the pesky rules: eurodollars (which have nothing to do with the European currency), commercial paper, repo, and many others. These forms of deposits are primarily used by large institutions like corporations. The details vary, but they tend to be prone to the same fundamental instability as bank deposits: if there's a run on the market, there may not be enough liquidity for everyone to withdraw their money at once. Unlike bank deposits, though, there is no insurance, no federal regulator to step in and make depositors whole, and much less regulation to ensure that runs are unlikely. Instead, there's the Federal Reserve, which has increasingly become the bulwark against liquidity crises among shadow banks. This happened in 2008 during the financial crisis (which Menand argues can be seen as a shadow bank run sparked by losses on mortgage securities), and again at a larger scale in 2020 during the initial COVID crisis. Menand is clear that these interventions from the Federal Reserve were necessary. The alternative would have been an uncontrolled collapse of large sections of the financial system, with unknown consequences. But the Fed was not intended to perform those types of interventions. It has no regulatory authority to reform the underlying financial systems to make them safer, remove executives who failed to maintain sufficient liquidity for a crisis, or (as is standard for all traditional US banks) prohibit combining banking and more speculative investment on the same balance sheet. What the Federal Reserve can do, and did, is function as a buyer of last resort, bailing out shadow banks by purchasing assets with newly-created money. This works, in the sense that it averts the immediate crisis, but it creates other distortions. Most importantly, constant Fed intervention doesn't create an incentive to avoid situations that require that intervention; if anything, it encourages more dangerous risk-taking. The above, plus an all-too-brief history of the politics of US banking, is the meat of this book. It's a useful summary, as far as it goes, and I learned a few new things. But I think The Fed Unbound is confused about its audience. This type of high-level summary and capsule history seems most useful for people without an economics background and who haven't been following macroeconomics closely. But Menand doesn't write for that audience. He assumes definitions of words like "deposits" and "money" that are going to be confusing or even incomprehensible to the lay reader. For example, Menand describes ordinary retail banks as creating money, even saying that a bank loans money by simply incrementing the numbers in a customer's deposit account. This is correct in the technical economic definition of money (fractional reserve banking effectively creates new money), but it's going to sound to someone not well-versed in the terminology as if retail banks can create new dollars out of the ether. That power is, of course, reserved for the Federal Reserve, and indeed is largely the point of its existence. Much of this book relies on a very specific definition of money and money supply that will only be familiar to those with economics training. Similarly, the history of the Federal Reserve is interesting but slight, and at no point does Menand explain clearly how the record-keeping between it and retail banks works, or what the Fed's "balance sheet" means in practice. I realize this book isn't trying to give a detailed description or history of the Federal Reserve system, but the most obvious audience is likely to flounder at the level of detail Menand provides. Perhaps, therefore, this book is aimed at an audience already familiar with macroeconomics? But, if so, I'm not sure it says anything new. I follow macroeconomic policy moderately closely and found most of Menand's observations obvious and not very novel. There were tidbits here and there that I hadn't understood, but my time would probably have been better invested in another book. Menand proposes some reforms, but they mostly consist of "Congress should do its job and not create situations where the Federal Reserve has to act without the right infrastructure and political oversight," and, well, yes. It's hard to disagree with that, and it's also hard to see how it will ever happen. It's far too convenient to outsource such problems to central banking, where they are hidden behind financial mechanics that are incomprehensible to the average voter. This is an important topic, but I don't think this is the book to read about it. If you want a clearer and easier-to-understand role of the Federal Reserve in shadow banking crises, read Crashed instead. If you want to learn more about how retail bank regulation works, and hear a strong case for why the same principles should be applied to shadow banks, see Sheila Bair's Bull by the Horns. I'm still looking for a great history and explainer of the Federal Reserve system as a whole, but that is not this book. Rating: 5 out of 10

29 November 2022

Jonathan McDowell: onak 0.6.2 released

Over the weekend I released a new version of onak, my OpenPGP compatible keyserver. At 2 years since the last release that means I ve at least managed to speed up a bit, but it s fair to say its development isn t a high priority for me at present. This release is largely driven by a collection of minor fixes that have built up, and the knowledge that a Debian freeze is coming in the new year. The fixes largely revolve around the signature verification that was introduced in 0.6.0, which makes it a bit safer to run a keyserver by only accepting key material that can be validated. All of the major items I wanted to work on post 0.6.0 remain outstanding. For the next release I d like to get some basic Stateless OpenPGP Command Line Interface support integrated. That would then allow onak to be tested with the OpenPGP interoperability test suite, which has recently added support for verification only OpenPGP implementations. I realise most people like to dismiss OpenPGP, and the tooling has been fairly dreadful for as long as I ve been using it, but I do think it fills a space that no competing system has bothered to try and replicate. And that s the web of trust, which helps provide some ability to verify keys without relying on (but also without preventing) a central authority to do so. Anyway. Available locally or via GitHub.
0.6.2 - 27th November 2022
  • Don t take creation time from unhashed subpackets
  • Fix ECDSA/SHA1 signature check
  • Fix handling of other signature requirement
  • Fix deletion of keys with PostgreSQL backend
  • Add support for verifying v3 signature packets

Sam Hartman: Introducing Carthage

For the past four years, I ve been working on Carthage, a free-software Infrastructure as Code framework. We ve finally reached a point where it makes sense to talk about Carthage and what it can do. This is the first in a series of blog posts to introduce Carthage, discuss what it can do and show how it works. Why Another IAC Framework It seems everywhere you look, there are products designed to support the IAC pattern. On the simple side, you could check a Containerfile into Git. Products like Terraform and Vagrant allow you to template cloud infrastructure and VMs. There are more commercial offerings than I can keep up with. We were disappointed by what was out there when we started Carthage. Other products have improved, but for many of our applications we re happy with what Carthage can build. The biggest challenge we ran into is that products wanted us to specify things at the wrong level. For some of our cyber training work we wanted to say things like We want 3 blue teams, each with a couple defended networks, a red team, and some neutral infrastructure for red to exploit. Yet the tools we were trying to use wanted to lay things out at the individual machine/container level. We found ourselves contemplating writing a program to generate input for some other IAC tool. Things were worse for our internal testing. Sometimes we d be shipping hardware to a customer. But sometimes we d be virtualizing that build out in a lab. Sometimes we d be doing a mixture. So we wanted to completely separate the descriptions of machines, networks, and software from any of the information about whether that was realized on hardware, VMs, containers, or a mixture. Dimensional Breakdown In discussing Carthage with Enrico Zini, he pointed me at Cognitive Dimensions of notation as a way to think about how Carthage approaches the IAC problem. I m more interested in the idea of breaking down a design along the idea of dimensions that allow examining the design space than I am particular adherence to Green s original dimensions. Low Viscosity, High Abstraction Reuse One of the guiding principles is that we want to be able to reuse different components at different scales and in different environments. These include being able to do things like: Hidden Dependencies To accomplish these abstraction goals, dependencies need to be non-local. For example, a software role might need to integrate with a directory if a directory is present in the environment. When writing the role, no one is going to know which directory to use, nor whether a directory is present. Taking that as an explicit input into the role is error-prone when the role is combined into large abstract units (bigger roles or collections of machines). Instead it is better to have a non-local dependency, and to find the directory if it is available. We accomplish this using dependency injection. In addition to being non-local, dependencies are sometimes hidden. It is very easy to overwhelm our cognitive capacity with even a fairly simple IAC description. An effective notation allows us to focus on the parts that matter when working with a particular part of the description. I ve found hiding dependencies, especially indirect dependencies, to be essential in building complex descriptions. Obviously, tools are required for examining these dependencies as part of debugging. First Class Modeling Clearly one of the goals of IAC descriptions is to actually build and manage infrastructure. It turns out that there are all sorts of things you want to do with the description well before you instantiate the infrastructure. You might want to query the description to build network diagrams, understand interdependencies, or even build inventory/bill of materials. We often find ourselves building Ansible inventory, switch configurations, DNS zones, and all sorts of configuration artifacts. These artifacts may be installed into infrastructure that is instantiated by the description, but they may be consumed in other ways. Allowing the artifacts to be consumed externally means that you can avoid pre-commitment and focus on whatever part of the description you originally want to work on. You may use an existing network at first. Later the IAC description may replace that, or perhaps it never will. As a result, Carthage separates modeling from instantiation. The model can generally be built and queried without needing to interact with clouds, VMs, or containers. We ve actually found it useful to build Carthage layouts that cannot ever be fully instantiated, for example because they never specify details like whether a model should be instantiated on a container or VM, or what kind of technology will realize a modeled network. This allows developing roles before the machines that use them or focusing on how machines will interact and how the network will be laid out before the details of installing on specific hardware. The modeling separation is by far the difference I value most between Carthage and other systems. A Tool for Experts. In Neal Stephenson s essay In the Beginning Was the Command Line , Stephenson points out that the kind of tools experts need are not the same tools that beginners need. The illustration of why a beginner might not be satisfied with a Hole Hog drill caught my attention. Carthage is a tool for experts. Despite what cloud providers will tell you, IAC is not easy. Doubly so when you start making reusable components. Trying to hide that or focus on making things easy to get started can make it harder for experts to efficiently solve the problems they are facing. When we have faced trade offs between making Carthage easy to pick up and making it powerful for expert users, we have chosen to support the experts. That said, Carthage today is harder to pick up than it needs to be. It s a relatively new project with few external users as of this time. Our documentation and examples need improvement, just like every project at this level of maturity. Similarly, as the set of things people try to do expand, we will doubtless run into bugs that our current test cases don t cover. So Carthage absolutely will get easier to learn and use than it is today. Also, we ve already had success building beginner-focused applications on top of Carthage. For our cyber training, we built web applications on top of Carthage that made rebuilding and exploring infrastructure easy. We ve had success using relatively understood tools like Ansible as integration and customization points for Carthage layouts. But in all these cases, when the core layout had significant reusable components and significant complexity in the networking, only an IAC expert was going to be able to maintain and develop that layout. What Carthage can do. Carthage has a number of capabilities today. One of Carthage s strengths is its extensible design. Abstract interfaces make it easy to add new virtualization platforms, cloud services, and support for various ways of managing real hardware. This approach has been validated by incrementally adding support for virtualization architectures and cloud services. As development has progressed, adding new integrations continues to get faster because we are able to reuse existing infrastructure. Today, Carthage can model: Carthage has excellent facilities for dealing with images on which VMs and Containers can be based, although it does have a bit of a Debian/Ubuntu bias in how it thinks about images: When instantiating infrastructure, Carthage can work with: We have also looked at Oracle Cloud and I believe Openstack, although that code is not merged. Future posts will talk about core Carthage concepts and how to use Carthage to build infrastructure.

comment count unavailable comments

28 November 2022

Jelmer Vernooij: Detecting Package Transitions

Larger transitions in Debian are usually announced on e.g. debian-devel, but it s harder to track the current status of all transitions. Having done a lot of QA uploads recently, I have on occasion uploaded packages involved in a transition. This can be unhelpful for the people handling the transition, but there s also often not much point in uploading if your uploads are going to get stuck. Talking to one of the release managers at a recent BSP, it was great to find out that the release team actually publish a data dump with which packages are involved in which transitions. Here s the script I use to find out about the transitions the package in my current working directory is involved in: 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
#!/usr/bin/python3
from urllib.request import urlopen
import sys
from debian.deb822 import Deb822
import yaml
with open('debian/control', 'r') as f:
    package = Deb822(f)['Source']
with urlopen("https://release.debian.org/transitions/export/packages.yaml") as f:
    data = yaml.safe_load(f)
def find_transitions(data, package):
    for entry in data:
        if entry['name'] != package:
            continue
        return dict(entry['list'])
    return  
transitions = find_transitions(data, package)
print(transitions)
sys.exit(1 if 'ongoing' in transitions.values() else 0)
In practice, the output looks something like this: 
$ debcheckout bctoolbox
git clone https://salsa.debian.org/pkg-voip-team/linphone-stack/bctoolbox.git bctoolbox ...
Cloning into 'bctoolbox'...
...
$ cd bctoolbox
$ in-transition.py
 'auto-upperlimit-libbctoolbox1': 'ongoing'

Steve Kemp: I put an LSP in your LISP ..

I recently wrote about yet another lisp I'd been having fun with. Over the past couple of years I've played with a few toy scripting languages, or random interpreters, and this time I figured I'd do something beyond the minimum, by implementing the Language Server Protocol. In brief the language server protocol (LSP) is designed to abstract functionality that might be provided by an editor, or IDE, into a small "language server". If the language-server knows how to jump to definitions, provide completion, etc, etc, then the editor doesn't need to implement those things for NN different languages - it just needs to launch and communicate with something that does know how to do the job. Anyway LSP? LISP? Only one letter different, so that's practically enough reason to have a stab at it. Thankfully I found a beautiful library that implements a simple framework allowing the easy implementation of a golang-based LSP-server Using that I quickly hacked up a server that can provide: I've tested this in both GNU Emacs and Neovim, so that means I'm happy I support all editors! (More seriously if it works in two then that probably means that the LSP stuff should work elsewhere too.) Here's what the "help on hover" looks like, within Emacs: Vim looks similar but you have to press K to see the wee popup. Still kinda cute, and was a good experiment.

John Goerzen: Flying Joy

Wisdom from my 5-year-old: When flying in a small plane, it is important to give your dolls a headset and let them see out the window, too! Moments like this make me smile at being a pilot dad. A week ago, I also got to give 8 children and one adult their first ever ride in any kind of airplane, through EAA s Young Eagles program. I got to hear several say, Oh wow! It s SO beautiful! Look at all the little houses! And my favorite: How can I be a pilot?

26 November 2022

Benjamin Mako Hill: The Financial Times has been printing an obvious error on its Market Data page for 18 months and nobody else seems to have noticed

Market Data section of the Financial Times US Edition print edition from May 5, 2021.
If you ve flipped through printed broadsheet newspapers, you ve probably seen pages full of tiny text listing prices and other market information for stocks and commodities. And you ve almost certainly just turned the page. Anybody interested in this market prices today will turn to the internet where these numbers are available in real time and where you don t need to squint to find what you need. This is presumably why many newspapers have stopped printing these types of pages or dramatically reduced the space devoted to them. Major financial newspapers however like the Financial Times (FT) still print multiple pages of market data daily. But does anybody read them? The answer appears to be no. How do I know? I noticed an error in the FT s Market Data page that anybody looking in the relevant section of the page would have seen. And I have seen it reproduced every single day for the last 18 months. In early May last year, I noticed that the Japanese telecom giant Nippon Telegraph and Telephone (NTT) was listed twice on the FT s list of the 500 largest global companies: once as Nippon T&T and also as Nippon TT. One right above the other. All the numbers are identical. Clearly a mistake.
Reproduction of the FT Market Data section showing a subset of Japanese companies from the FT 500 list of global companies. The duplicate lines are highlighted in yellow. This page is from today s paper (November 26, 2022).
Wondering if it was a one-off error, I looked at a copy of the paper from about a week before and saw that the error did not exist then. I looked at a copy from one day before and saw that it did. Since the issue was apparently recurring, but new at the time, I figured someone at the paper would notice and fix it quickly. I was wrong. It has been 18 months now and the error has been reproduced every single day. Looking through the archives, it seems that the first day the error showed up was May 5, 2021. I ve included a screenshot from the electronic paper version from that day and from the fifth of every month since then (or the sixth if the paper was not printed on the fifth) that shows that the error is reproduced every day. A quick look in the archives suggests it not only appears in the US edition but also in the UK, European, Asian, and Middle East editions. All of them. Why does this matter? The FT prints over 112,000 copies of its paper, six days a week. This duplicate line takes up almost no space, of course, so it s not a big deal on its own. But devoting two full broadsheet pages to market data that is out date as soon as it is printed much of which nobody appears to be reading doesn t seem like a great use of resources. There s an argument to made that papers like the FT print these pages not because they are useful but because doing so is a signal of the publications identities as serious financial papers. But that hardly seems like a good enough reason on its own if nobody is looking at them. It seems well past time for newspapers to stop wasting paper and ink on these pages. I respect that some people think that printing paper newspapers at all is wasteful when one can just read the material online. Plenty of people disagree, of course. But who will disagree with a call to stop printing material that evidence suggests is not being seen by anybody? If an error this obvious can exist for so long, it seems clear that nobody not even anybody at the FT itself is reading it.

Matthew Garrett: Poking a mobile hotspot

I've been playing with an Orbic Speed, a relatively outdated device that only speaks LTE Cat 4, but the towers I can see from here are, uh, not well provisioned so throughput really isn't a concern (and refurbs are $18, so). As usual I'm pretty terrible at just buying devices and using them for their intended purpose, and in this case it has the irritating behaviour that if there's a power cut and the battery runs out it doesn't boot again when power returns, so here's what I've learned so far.

First, it's clearly running Linux (nmap indicates that, as do the headers from the built-in webserver). The login page for the web interface has some text reading "Open Source Notice" that highlights when you move the mouse over it, but that's it - there's code to make the text light up, but it's not actually a link. There's no exposed license notices at all, although there is a copy on the filesystem that doesn't seem to be reachable from anywhere. The notice tells you to email them to receive source code, but doesn't actually provide an email address.

Still! Let's see what else we can figure out. There's no open ports other than the web server, but there is an update utility that includes some interesting components. First, there's a copy of adb, the Android Debug Bridge. That doesn't mean the device is running Android, it's common for embedded devices from various vendors to use a bunch of Android infrastructure (including the bootloader) while having a non-Android userland on top. But this is still slightly surprising, because the device isn't exposing an adb interface over USB. There's also drivers for various Qualcomm endpoints that are, again, not exposed. Running the utility under Windows while the modem is connected results in the modem rebooting and Windows talking about new hardware being detected, and watching the device manager shows a bunch of COM ports being detected and bound by Qualcomm drivers. So, what's it doing?

Sticking the utility into Ghidra and looking for strings that correspond to the output that the tool conveniently leaves in the logs subdirectory shows that after finding a device it calls vendor_device_send_cmd(). This is implemented in a copy of libusb-win32 that, again, has no offer for source code. But it's also easy to drop that into Ghidra and discover thatn vendor_device_send_cmd() is just a wrapper for usb_control_msg(dev,0xc0,0xa0,0,0,NULL,0,1000);. Sending that from Linux results in the device rebooting and suddenly exposing some more USB endpoints, including a functional adb interface. Although, annoyingly, the rndis interface that enables USB tethering via the modem is now missing.

Unfortunately the adb user is unprivileged, but most files on the system are world-readable. data/logs/atfwd.log is especially interesting. This modem has an application processor built into the modem chipset itself, and while the modem implements the Hayes Command Set there's also a mechanism for userland to register that certain AT commands should be pushed up to userland. These are handled by the atfwd_daemon that runs as root, and conveniently logs everything it's up to. This includes having logged all the communications executed when the update tool was run earlier, so let's dig into that.

The system sends a bunch of AT+SYSCMD= commands, each of which is in the form of echo (stuff) >>/usrdata/sec/chipid. Once that's all done, it sends AT+CHIPID, receives a response of CHIPID:PASS, and then AT+SER=3,1, at which point the modem reboots back into the normal mode - adb is gone, but rndis is back. But the logs also reveal that between the CHIPID request and the response is a security check that involves RSA. The logs on the client side show that the text being written to the chipid file is a single block of base64 encoded data. Decoding it just gives apparently random binary. Heading back to Ghidra shows that atfwd_daemon is reading the chipid file and then decrypting it with an RSA key. The key is obtained by calling a series of functions, each of which returns a long base64-encoded string. Decoding each of these gives 1028 bytes of high entropy data, which is then passed to another function that decrypts it using AES CBC using a key of 000102030405060708090a0b0c0d0e0f and an initialization vector of all 0s. This is somewhat weird, since there's 1028 bytes of data and 128 bit AES works on blocks of 16 bytes. The behaviour of OpenSSL is apparently to just pad the data out to a multiple of 16 bytes, but that implies that we're going to end up with a block of garbage at the end. It turns out not to matter - despite the fact that we decrypt 1028 bytes of input only the first 200 bytes mean anything, with the rest just being garbage. Concatenating all of that together gives us a PKCS#8 private key blob in PEM format. Which means we have not only the private key, but also the public key.

So, what's in the encrypted data, and where did it come from in the first place? It turns out to be a JSON blob that contains the IMEI and the serial number of the modem. This is information that can be read from the modem in the first place, so it's not secret. The modem decrypts it, compares the values in the blob to its own values, and if they match sets a flag indicating that validation has succeeeded. But what encrypted it in the first place? It turns out that the json blob is just POSTed to http://pro.w.ifelman.com/api/encrypt and an encrypted blob returned. Of course, the fact that it's being encrypted on the server with the public key and sent to the modem that decrypted with the private key means that having access to the modem gives us the public key as well, which means we can just encrypt our own blobs.

What does that buy us? Digging through the code shows the only case that it seems to matter is when parsing the AT+SER command. The first argument to this is the serial mode to transition to, and the second is whether this should be a temporary transition or a permanent one. Once parsed, these arguments are passed to /sbin/usb/compositions/switch_usb which just writes the mode out to /usrdata/mode.cfg (if permanent) or /usrdata/mode_tmp.cfg (if temporary). On boot, /data/usb/boot_hsusb_composition reads the number from this file and chooses which USB profile to apply. This requires no special permissions, except if the number is 3 - if so, the RSA verification has to be performed first. This is somewhat strange, since mode 9 gives the same rndis functionality as mode 3, but also still leaves the debug and diagnostic interfaces enabled.

So what's the point of all of this? I'm honestly not sure! It doesn't seem like any sort of effective enforcement mechanism (even ignoring the fact that you can just create your own blobs, if you change the IMEI on the device somehow, you can just POST the new values to the server and get back a new blob), so the best I've been able to come up with is to ensure that there's some mapping between IMEI and serial number before the device can be transitioned into production mode during manufacturing.

But, uh, we can just ignore all of this anyway. Remember that AT+SYSCMD= stuff that was writing the data to /usrdata/sec/chipid in the first place? Anything that's passed to AT+SYSCMD is just executed as root. Which means we can just write a new value (including 3) to /usrdata/mode.cfg in the first place, without needing to jump through any of these hoops. Which also means we can just adb push a shell onto there and then use the AT interface to make it suid root, which avoids needing to figure out how to exploit any of the bugs that are just sitting there given it's running a 3.18.48 kernel.

Anyway, I've now got a modem that's got working USB tethering and also exposes a working adb interface, and I've got root on it. Which let me dump the bootloader and discover that it implements fastboot and has an oem off-mode-charge command which solves the problem I wanted to solve of having the device boot when it gets power again. Unfortunately I still need to get into fastboot mode. I haven't found a way to do it through software (adb reboot bootloader doesn't do anything), but this post suggests it's just a matter of grounding a test pad, at which point I should just be able to run fastboot oem off-mode-charge and it'll be all set. But that's a job for tomorrow.

Edit: Got into fastboot mode and ran fastboot oem off-mode-charge 0 but sadly it doesn't actually do anything, so I guess next is going to involve patching the bootloader binary. Since it's signed with a cert titled "General Use Test Key (for testing only)" it apparently doesn't have secure boot enabled, so this should be easy enough.

comment count unavailable comments

25 November 2022

Russ Allbery: podlators 5.00

podlators is my collection of POD formatting modules, which generate *roff or text (possibly with escape sequence markup) from the documentation format used by Perl and some other packages. This is a major release, the biggest since the Pod::Simple rewrite in 2005. The headline news is that after some fairly extensive investigation, this release of Pod::Man finally changes the default output format to Unicode. No more replacement of characters in people's names, or text in non-English languages, with ugly X characters! There is a new encoding option to set the output encoding, and new options groff (which uses the groff extension for Unicode code points and is the default on EBCDIC systmes) and roff (which does the old, broken X substitution). Since this was a major backward-incompatible change, I also finally removed most of the formatting touch-ups that Pod::Man tried to do for troff output but which would be invisible for the (by far more commonly used) nroff output. These have been an endless source of bugs and are very difficult to maintain, most of them were of marginal utility, and I am dubious many people are using troff to print Perl manual pages these days instead of, say, printing the rendered output from one of the many excellent POD to HTML modules. There is some remaining somewhat-Perl-specific guesswork applied to the formatting, which is much simpler, but even that can now be turned off with the new guesswork option. This will allow people using POD to generate manual pages for things other than Perl modules to disable the Perl-specific markup logic. Pod::Text also now supports encoding and gets some major encoding cleanups, including using Encode instead of PerlIO encoding layers for its output. There are also numerous other fixes and improvements: a new language option to Pod::Man to configure (in an unfortunately groff-specific way) the line-breaking rules for languages like Chinese and Japanese, conversion of zero-width spaces to the *roff \: equivalent, a fix for wrapping L<> inside S<>, and various other bug fixes. Perhaps the most interesting is a fix to a long-standing problem with the Pod::Man output where bold and italic text would extend too far if used in combination with C<> fixed-width text. This bug has been around forever without being noticed, and then two different people noticed it while I was preparing this release. You can get the latest release from CPAN or from the podlators distribution page. These changes should be incorporated into Perl core in due course, although given the substantial changes, that may require a baking period.

24 November 2022

Russ Allbery: Review: Servant Mage

Review: Servant Mage, by Kate Elliott
Publisher: Tordotcom
Copyright: 2022
ISBN: 1-250-76904-3
Format: Kindle
Pages: 165
Servant Mage is a (so far at least) standalone fantasy novella. Fellian is a servant mage, which means that she makes Lamps for the customers of the inn, cleans the privies, and watches her effective owners find ways to put her deeper into indentured servitude. That's the only life permitted by the August Protector to those found to carry the taint of magical talent, caused by (it is said) demons who have bound themselves to their souls. Fellian's effective resistance is limited to giving covert reading lessons. Or was, before she is hired by a handsome man who needs a Lamplighter. A man who has been looking for her specifically, is a magical Adept, and looks suspiciously like a soldier for the despised and overthrown monarchists. A difficulty with writing a story that reverses cliches is that you have to establish the cliches in order to reverse them, which runs the risk that a lot of the story may feel cliched. I fear some of that happened here. Magic, in this world, is divided into elemental spheres, each of which has been restricted to limited and practical tasks by the Liberationists. The new regime searches the population for the mage-gifted and forces them into public service for the good of all (or at least that's how they describe it), with as little education as possible. Fellian was taught to light Lamps, but what she has is fire magic, and she's worked out some additional techniques on her own. The Adept is air, and one of the soldiers with him is earth. If you're guessing that two more elements turn up shortly and something important happens if you get all five of them together, you're perhaps sensing a bit of unoriginality in the magic system. That's not the cliche that's the primary target of this story, though. The current rulers of this country, led by the austere August Protector, are dictatorial anti-monarchists who are happy to force mages into indenture and deny people schooling. Fellian has indeed fallen in with the monarchists, who unsurprisingly are attempting to reverse the revolution. They, unlike the Liberationists, respect mages and are willing to train them, and they would like to recruit Fellian. I won't spoil the details of where Elliott is going with the plot, but it does eventually go somewhere refreshingly different. The path to get there, though, is familiar from any number of fantasy epics that start with a slave with special powers. Servant Mage is more aware of this than most, and Fellian is sharp-tongued and suspicious rather than innocent and trainable, but there are a lot of familiar character tropes and generic fantasy politics. This is the second story (along with the Spiritwalker trilogy) of Elliott's I've read that uses the French Revolution as a political model but fails to provide satisfying political depth. This one is a novella and can therefore be forgiven for not having the time to dive into revolutionary politics, but I wish Elliott would do more with this idea. Here, the anti-monarchists are straight-up villains, and while that's partly setup for more nuance than you might be expecting, it still felt like a waste of the setting. I want the book that tackles the hard social problem of reconciling the chaos and hopefulness of political revolution with magical powers that can be dangerous and oppressive without the structure of tradition. It feels like Elliott keeps edging towards that book but hasn't found the right hook to write it. Instead, we get a solid main character in Fellian, a bunch of supporting characters who mostly register as "okay," some magical set pieces that have all the right elements and yet didn't grab my sense of wonder, and a story that seemed heavily signposted. The conclusion was the best part of the story, but by the time we got there it wasn't as much of a surprise as I wanted it to be. I had this feeling with the Spiritwalker series as well: the pieces making up the story are of good quality, and Elliott's writing is solid, but the narrative magic never quite coheres for me. It's the sort of novella where I finished reading, went "huh," and then was excited to start a new book. I have no idea if there are plans for more stories in this universe, but Servant Mage felt like a prelude to a longer series. If that series does materialize, there are some signs that it could be more satisfying. At the end of the story, Fellian is finally in a place to do something original and different, and I am mildly curious what she might do. Maybe enough to read the next book, if one turns up. Mostly, though, I'm waiting for the sequel to Unconquerable Sun. Next April! Rating: 6 out of 10

23 November 2022

Jonathan Dowland: eventual consistency

Reading some documentation about using hledger, a particular passage jumped out at me:
It should be easy to work towards eventual consistency. I should be able to do them bit by little bit, leaving things half-done, and picking them up later with little (mental) effort. Eventually my records would be perfect and consistent.
This has been an approach I've used for many things in my life for a long time. It has something in common with eat the elephant one mouthful at a time. I think there are some categories of problems that you can't solve this way: perhaps because with this approach you are always stuck in a local minima/maxima. On the other hand I often feel that there's no way I can address some problems at all without doing so in the smallest of steps. Personal finance is definitely one of those.

Fran ois Marier: Name resolution errors in Ubuntu repositories while building a docker container

I ran into what seemed to be a DNS problem when building a Docker container: 
Err http://archive.ubuntu.com jammy Release.gpg
  Could not resolve 'archive.ubuntu.com'
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/jammy/Release.gpg  Could not resolve 'archive.ubuntu.com'
W: Some index files failed to download. They have been ignored, or old ones used instead.
I found that many solutions talked about setting the default DNS server explicitly in /etc/docker/daemon.json: 
 
    "dns": ["1.1.1.1"]
but that didn't work for me. I noticed however that I was having these problems whenever I connected to my VPN. So what did work for me was restarting the docker daemon whenever there is a change in networking (e.g. enabling/disabling VPN) by putting the following in /etc/NetworkManager/dispatcher.d/docker-local: 
#!/bin/sh
LOGFILE=/var/log/docker-restarts.log
if [ -z "$1" ]; then
    echo "$0: called with no interface" >> $LOGFILE
    exit 1;
fi
if [ "$1" = lo ]; then
    # Ignoring the loopback interface
    exit 0;
fi
case "$2" in
    up vpn-up down vpn-down)
        echo "$0: restarting docker due to action \ $2' on interface \ $1'" >> $LOGFILE
        /bin/systemctl restart docker.service
        ;;
    *)
        echo "$0: ignoring action \ $2' on interface \ $1'" >> $LOGFILE
        ;;
esac
and then making that new file executable: 
chmod +x /etc/NetworkManager/dispatcher.d/docker-local
You can confirm that it's working as intended by watching the logs: 
tail -f /var/log/docker-restarts.log
while enabling/disable your VPN or your network connection. If you don't see any output, then something is wrong and the Docker restart isn't happening.

22 November 2022

Dirk Eddelbuettel: spdl 0.0.1 on CRAN: New Package!

A new package made it to CRAN today: spdl. It provides a very simple interface to spdlog with the key focus being that provides the same interface from both R and C++. I had brought spdlog to R a little while ago via RcppSpdlog, but its use was generally limited to C++ code where we would write something like (taken from one of the examples included in the package) 
spdlog::info("Welcome to spdlog!");
I then provided C-level exports from the package accessible in other package but with that the signature only became longer as it also included the package name, here RcppSpdlog. So I experimented a little with that and added another more compactly named namespace spdl around it. So then it becomes 
spdl::info("Info message with values   and  ", 42, 1.23);  // C++ code
which now also shows the power of the included fmt library. And I got quite used to that and now wanted to same from R! But to create a new namespace, we need a new package. So we made one. Now from R too: 
spdl::info("Info message with values   and  ", 42L, 1.23)  # R code
We layered a very simple wrapper for fmt over this. All R argument are passed to format() and we simply send a vector of strings over the C interface to the RcppSpdlog package where the templated formatter of fmt is invoked but for character values. You can equally well format the string locally first. Everything works: from paste() to sprintf() to any of the existing string interpolators all of which are nice. But none works just like fmt and I wanted to have the same formatter in both languages, and like how fmt works. By tradition, the NEWS entry for this release follows.

Changes in spld version 0.0.1 (2022-11-21)
  • Initial release of R and C++ loggers using spdl::* namespace

More detailed information is on the spdl page. If you like this or other open-source work I do, you can sponsor me at GitHub.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

21 November 2022

Dirk Eddelbuettel: RcppClassic 0.9.13 on CRAN: Minor Update

A maintenance release 0.9.14 of the RcppClassic package arrived earlier today on CRAN. This package provides a maintained version of the otherwise deprecated initial Rcpp API which no new projects should use as the normal Rcpp API is so much better. The changes is. CRAN was reporting (for all four macOS builds, and only there) that an absolute path was embedded, so we updated the (old ) call to install_name_tool use on that (and only that) OS. No other changes were made. CRANberries also reports the changes relative to the previous release from nearly three years ago. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page. If you like this or other open-source work I do, you can now sponsor me at GitHub.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Next.

Previous.