Nitrokey Start$ sha256sum python3-pip*
ded6b3867a4a4cbaff0940cab366975d6aeecc76b9f2d2efa3deceb062668b1c python3-pip_22.0.2+dfsg-1ubuntu0.2_all.deb
e1561575130c41dc3309023a345de337e84b4b04c21c74db57f599e267114325 python3-pip-whl_22.0.2+dfsg-1ubuntu0.2_all.deb
$ doas dpkg -i python3-pip*
...
$ doas apt install -f
...
$
Installing pynitrokey downloaded a bunch of dependencies, and it would be nice to audit the license and security vulnerabilities for each of them. (Verbose output below slightly redacted.)
jas@kaka:~$ pip3 install --user pynitrokey Collecting pynitrokey Downloading pynitrokey-0.4.34-py3-none-any.whl (572 kB) Collecting frozendict~=2.3.4 Downloading frozendict-2.3.5-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (113 kB) Requirement already satisfied: click<9,>=8.0.0 in /usr/lib/python3/dist-packages (from pynitrokey) (8.0.3) Collecting ecdsa Downloading ecdsa-0.18.0-py2.py3-none-any.whl (142 kB) Collecting python-dateutil~=2.7.0 Downloading python_dateutil-2.7.5-py2.py3-none-any.whl (225 kB) Collecting fido2<2,>=1.1.0 Downloading fido2-1.1.0-py3-none-any.whl (201 kB) Collecting tlv8 Downloading tlv8-0.10.0.tar.gz (16 kB) Preparing metadata (setup.py) ... done Requirement already satisfied: certifi>=14.5.14 in /usr/lib/python3/dist-packages (from pynitrokey) (2020.6.20) Requirement already satisfied: pyusb in /usr/lib/python3/dist-packages (from pynitrokey) (1.2.1.post1) Collecting urllib3~=1.26.7 Downloading urllib3-1.26.15-py2.py3-none-any.whl (140 kB) Collecting spsdk<1.8.0,>=1.7.0 Downloading spsdk-1.7.1-py3-none-any.whl (684 kB) Collecting typing_extensions~=4.3.0 Downloading typing_extensions-4.3.0-py3-none-any.whl (25 kB) Requirement already satisfied: cryptography<37,>=3.4.4 in /usr/lib/python3/dist-packages (from pynitrokey) (3.4.8) Collecting intelhex Downloading intelhex-2.3.0-py2.py3-none-any.whl (50 kB) Collecting nkdfu Downloading nkdfu-0.2-py3-none-any.whl (16 kB) Requirement already satisfied: requests in /usr/lib/python3/dist-packages (from pynitrokey) (2.25.1) Collecting tqdm Downloading tqdm-4.65.0-py3-none-any.whl (77 kB) Collecting nrfutil<7,>=6.1.4 Downloading nrfutil-6.1.7.tar.gz (845 kB) Preparing metadata (setup.py) ... done Requirement already satisfied: cffi in /usr/lib/python3/dist-packages (from pynitrokey) (1.15.0) Collecting crcmod Downloading crcmod-1.7.tar.gz (89 kB) Preparing metadata (setup.py) ... done Collecting libusb1==1.9.3 Downloading libusb1-1.9.3-py3-none-any.whl (60 kB) Collecting pc_ble_driver_py>=0.16.4 Downloading pc_ble_driver_py-0.17.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.9 MB) Collecting piccata Downloading piccata-2.0.3-py3-none-any.whl (21 kB) Collecting protobuf<4.0.0,>=3.17.3 Downloading protobuf-3.20.3-cp310-cp310-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (1.1 MB) Collecting pyserial Downloading pyserial-3.5-py2.py3-none-any.whl (90 kB) Collecting pyspinel>=1.0.0a3 Downloading pyspinel-1.0.3.tar.gz (58 kB) Preparing metadata (setup.py) ... done Requirement already satisfied: pyyaml in /usr/lib/python3/dist-packages (from nrfutil<7,>=6.1.4->pynitrokey) (5.4.1) Requirement already satisfied: six>=1.5 in /usr/lib/python3/dist-packages (from python-dateutil~=2.7.0->pynitrokey) (1.16.0) Collecting pylink-square<0.11.9,>=0.8.2 Downloading pylink_square-0.11.1-py2.py3-none-any.whl (78 kB) Collecting jinja2<3.1,>=2.11 Downloading Jinja2-3.0.3-py3-none-any.whl (133 kB) Collecting bincopy<17.11,>=17.10.2 Downloading bincopy-17.10.3-py3-none-any.whl (17 kB) Collecting fastjsonschema>=2.15.1 Downloading fastjsonschema-2.16.3-py3-none-any.whl (23 kB) Collecting astunparse<2,>=1.6 Downloading astunparse-1.6.3-py2.py3-none-any.whl (12 kB) Collecting oscrypto~=1.2 Downloading oscrypto-1.3.0-py2.py3-none-any.whl (194 kB) Collecting deepmerge==0.3.0 Downloading deepmerge-0.3.0-py2.py3-none-any.whl (7.6 kB) Collecting pyocd<=0.31.0,>=0.28.3 Downloading pyocd-0.31.0-py3-none-any.whl (12.5 MB) Collecting click-option-group<0.6,>=0.3.0 Downloading click_option_group-0.5.5-py3-none-any.whl (12 kB) Collecting pycryptodome<4,>=3.9.3 Downloading pycryptodome-3.17-cp35-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.1 MB) Collecting pyocd-pemicro<1.2.0,>=1.1.1 Downloading pyocd_pemicro-1.1.5-py3-none-any.whl (9.0 kB) Requirement already satisfied: colorama<1,>=0.4.4 in /usr/lib/python3/dist-packages (from spsdk<1.8.0,>=1.7.0->pynitrokey) (0.4.4) Collecting commentjson<1,>=0.9 Downloading commentjson-0.9.0.tar.gz (8.7 kB) Preparing metadata (setup.py) ... done Requirement already satisfied: asn1crypto<2,>=1.2 in /usr/lib/python3/dist-packages (from spsdk<1.8.0,>=1.7.0->pynitrokey) (1.4.0) Collecting pypemicro<0.2.0,>=0.1.9 Downloading pypemicro-0.1.11-py3-none-any.whl (5.7 MB) Collecting libusbsio>=2.1.11 Downloading libusbsio-2.1.11-py3-none-any.whl (247 kB) Collecting sly==0.4 Downloading sly-0.4.tar.gz (60 kB) Preparing metadata (setup.py) ... done Collecting ruamel.yaml<0.18.0,>=0.17 Downloading ruamel.yaml-0.17.21-py3-none-any.whl (109 kB) Collecting cmsis-pack-manager<0.3.0 Downloading cmsis_pack_manager-0.2.10-py2.py3-none-manylinux1_x86_64.whl (25.1 MB) Collecting click-command-tree==1.1.0 Downloading click_command_tree-1.1.0-py3-none-any.whl (3.6 kB) Requirement already satisfied: bitstring<3.2,>=3.1 in /usr/lib/python3/dist-packages (from spsdk<1.8.0,>=1.7.0->pynitrokey) (3.1.7) Collecting hexdump~=3.3 Downloading hexdump-3.3.zip (12 kB) Preparing metadata (setup.py) ... done Collecting fire Downloading fire-0.5.0.tar.gz (88 kB) Preparing metadata (setup.py) ... done Requirement already satisfied: wheel<1.0,>=0.23.0 in /usr/lib/python3/dist-packages (from astunparse<2,>=1.6->spsdk<1.8.0,>=1.7.0->pynitrokey) (0.37.1) Collecting humanfriendly Downloading humanfriendly-10.0-py2.py3-none-any.whl (86 kB) Collecting argparse-addons>=0.4.0 Downloading argparse_addons-0.12.0-py3-none-any.whl (3.3 kB) Collecting pyelftools Downloading pyelftools-0.29-py2.py3-none-any.whl (174 kB) Collecting milksnake>=0.1.2 Downloading milksnake-0.1.5-py2.py3-none-any.whl (9.6 kB) Requirement already satisfied: appdirs>=1.4 in /usr/lib/python3/dist-packages (from cmsis-pack-manager<0.3.0->spsdk<1.8.0,>=1.7.0->pynitrokey) (1.4.4) Collecting lark-parser<0.8.0,>=0.7.1 Downloading lark-parser-0.7.8.tar.gz (276 kB) Preparing metadata (setup.py) ... done Requirement already satisfied: MarkupSafe>=2.0 in /usr/lib/python3/dist-packages (from jinja2<3.1,>=2.11->spsdk<1.8.0,>=1.7.0->pynitrokey) (2.0.1) Collecting asn1crypto<2,>=1.2 Downloading asn1crypto-1.5.1-py2.py3-none-any.whl (105 kB) Collecting wrapt Downloading wrapt-1.15.0-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (78 kB) Collecting future Downloading future-0.18.3.tar.gz (840 kB) Preparing metadata (setup.py) ... done Collecting psutil>=5.2.2 Downloading psutil-5.9.4-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (280 kB) Collecting capstone<5.0,>=4.0 Downloading capstone-4.0.2-py2.py3-none-manylinux1_x86_64.whl (2.1 MB) Collecting naturalsort<2.0,>=1.5 Downloading naturalsort-1.5.1.tar.gz (7.4 kB) Preparing metadata (setup.py) ... done Collecting prettytable<3.0,>=2.0 Downloading prettytable-2.5.0-py3-none-any.whl (24 kB) Collecting intervaltree<4.0,>=3.0.2 Downloading intervaltree-3.1.0.tar.gz (32 kB) Preparing metadata (setup.py) ... done Collecting ruamel.yaml.clib>=0.2.6 Downloading ruamel.yaml.clib-0.2.7-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl (485 kB) Collecting termcolor Downloading termcolor-2.2.0-py3-none-any.whl (6.6 kB) Collecting sortedcontainers<3.0,>=2.0 Downloading sortedcontainers-2.4.0-py2.py3-none-any.whl (29 kB) Requirement already satisfied: wcwidth in /usr/lib/python3/dist-packages (from prettytable<3.0,>=2.0->pyocd<=0.31.0,>=0.28.3->spsdk<1.8.0,>=1.7.0->pynitrokey) (0.2.5) Building wheels for collected packages: nrfutil, crcmod, sly, tlv8, commentjson, hexdump, pyspinel, fire, intervaltree, lark-parser, naturalsort, future Building wheel for nrfutil (setup.py) ... done Created wheel for nrfutil: filename=nrfutil-6.1.7-py3-none-any.whl size=898520 sha256=de6f8803f51d6c26d24dc7df6292064a468ff3f389d73370433fde5582b84a10 Stored in directory: /home/jas/.cache/pip/wheels/39/2b/9b/98ab2dd716da746290e6728bdb557b14c1c9a54cb9ed86e13b Building wheel for crcmod (setup.py) ... done Created wheel for crcmod: filename=crcmod-1.7-cp310-cp310-linux_x86_64.whl size=31422 sha256=5149ac56fcbfa0606760eef5220fcedc66be560adf68cf38c604af3ad0e4a8b0 Stored in directory: /home/jas/.cache/pip/wheels/85/4c/07/72215c529bd59d67e3dac29711d7aba1b692f543c808ba9e86 Building wheel for sly (setup.py) ... done Created wheel for sly: filename=sly-0.4-py3-none-any.whl size=27352 sha256=f614e413918de45c73d1e9a8dca61ca07dc760d9740553400efc234c891f7fde Stored in directory: /home/jas/.cache/pip/wheels/a2/23/4a/6a84282a0d2c29f003012dc565b3126e427972e8b8157ea51f Building wheel for tlv8 (setup.py) ... done Created wheel for tlv8: filename=tlv8-0.10.0-py3-none-any.whl size=11266 sha256=3ec8b3c45977a3addbc66b7b99e1d81b146607c3a269502b9b5651900a0e2d08 Stored in directory: /home/jas/.cache/pip/wheels/e9/35/86/66a473cc2abb0c7f21ed39c30a3b2219b16bd2cdb4b33cfc2c Building wheel for commentjson (setup.py) ... done Created wheel for commentjson: filename=commentjson-0.9.0-py3-none-any.whl size=12092 sha256=28b6413132d6d7798a18cf8c76885dc69f676ea763ffcb08775a3c2c43444f4a Stored in directory: /home/jas/.cache/pip/wheels/7d/90/23/6358a234ca5b4ec0866d447079b97fedf9883387d1d7d074e5 Building wheel for hexdump (setup.py) ... done Created wheel for hexdump: filename=hexdump-3.3-py3-none-any.whl size=8913 sha256=79dfadd42edbc9acaeac1987464f2df4053784fff18b96408c1309b74fd09f50 Stored in directory: /home/jas/.cache/pip/wheels/26/28/f7/f47d7ecd9ae44c4457e72c8bb617ef18ab332ee2b2a1047e87 Building wheel for pyspinel (setup.py) ... done Created wheel for pyspinel: filename=pyspinel-1.0.3-py3-none-any.whl size=65033 sha256=01dc27f81f28b4830a0cf2336dc737ef309a1287fcf33f57a8a4c5bed3b5f0a6 Stored in directory: /home/jas/.cache/pip/wheels/95/ec/4b/6e3e2ee18e7292d26a65659f75d07411a6e69158bb05507590 Building wheel for fire (setup.py) ... done Created wheel for fire: filename=fire-0.5.0-py2.py3-none-any.whl size=116951 sha256=3d288585478c91a6914629eb739ea789828eb2d0267febc7c5390cb24ba153e8 Stored in directory: /home/jas/.cache/pip/wheels/90/d4/f7/9404e5db0116bd4d43e5666eaa3e70ab53723e1e3ea40c9a95 Building wheel for intervaltree (setup.py) ... done Created wheel for intervaltree: filename=intervaltree-3.1.0-py2.py3-none-any.whl size=26119 sha256=5ff1def22ba883af25c90d90ef7c6518496fcd47dd2cbc53a57ec04cd60dc21d Stored in directory: /home/jas/.cache/pip/wheels/fa/80/8c/43488a924a046b733b64de3fac99252674c892a4c3801c0a61 Building wheel for lark-parser (setup.py) ... done Created wheel for lark-parser: filename=lark_parser-0.7.8-py2.py3-none-any.whl size=62527 sha256=3d2ec1d0f926fc2688d40777f7ef93c9986f874169132b1af590b6afc038f4be Stored in directory: /home/jas/.cache/pip/wheels/29/30/94/33e8b58318aa05cb1842b365843036e0280af5983abb966b83 Building wheel for naturalsort (setup.py) ... done Created wheel for naturalsort: filename=naturalsort-1.5.1-py3-none-any.whl size=7526 sha256=bdecac4a49f2416924548cae6c124c85d5333e9e61c563232678ed182969d453 Stored in directory: /home/jas/.cache/pip/wheels/a6/8e/c9/98cfa614fff2979b457fa2d9ad45ec85fa417e7e3e2e43be51 Building wheel for future (setup.py) ... done Created wheel for future: filename=future-0.18.3-py3-none-any.whl size=492037 sha256=57a01e68feca2b5563f5f624141267f399082d2f05f55886f71b5d6e6cf2b02c Stored in directory: /home/jas/.cache/pip/wheels/5e/a9/47/f118e66afd12240e4662752cc22cefae5d97275623aa8ef57d Successfully built nrfutil crcmod sly tlv8 commentjson hexdump pyspinel fire intervaltree lark-parser naturalsort future Installing collected packages: tlv8, sortedcontainers, sly, pyserial, pyelftools, piccata, naturalsort, libusb1, lark-parser, intelhex, hexdump, fastjsonschema, crcmod, asn1crypto, wrapt, urllib3, typing_extensions, tqdm, termcolor, ruamel.yaml.clib, python-dateutil, pyspinel, pypemicro, pycryptodome, psutil, protobuf, prettytable, oscrypto, milksnake, libusbsio, jinja2, intervaltree, humanfriendly, future, frozendict, fido2, ecdsa, deepmerge, commentjson, click-option-group, click-command-tree, capstone, astunparse, argparse-addons, ruamel.yaml, pyocd-pemicro, pylink-square, pc_ble_driver_py, fire, cmsis-pack-manager, bincopy, pyocd, nrfutil, nkdfu, spsdk, pynitrokey WARNING: The script nitropy is installed in '/home/jas/.local/bin' which is not on PATH. Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. Successfully installed argparse-addons-0.12.0 asn1crypto-1.5.1 astunparse-1.6.3 bincopy-17.10.3 capstone-4.0.2 click-command-tree-1.1.0 click-option-group-0.5.5 cmsis-pack-manager-0.2.10 commentjson-0.9.0 crcmod-1.7 deepmerge-0.3.0 ecdsa-0.18.0 fastjsonschema-2.16.3 fido2-1.1.0 fire-0.5.0 frozendict-2.3.5 future-0.18.3 hexdump-3.3 humanfriendly-10.0 intelhex-2.3.0 intervaltree-3.1.0 jinja2-3.0.3 lark-parser-0.7.8 libusb1-1.9.3 libusbsio-2.1.11 milksnake-0.1.5 naturalsort-1.5.1 nkdfu-0.2 nrfutil-6.1.7 oscrypto-1.3.0 pc_ble_driver_py-0.17.0 piccata-2.0.3 prettytable-2.5.0 protobuf-3.20.3 psutil-5.9.4 pycryptodome-3.17 pyelftools-0.29 pylink-square-0.11.1 pynitrokey-0.4.34 pyocd-0.31.0 pyocd-pemicro-1.1.5 pypemicro-0.1.11 pyserial-3.5 pyspinel-1.0.3 python-dateutil-2.7.5 ruamel.yaml-0.17.21 ruamel.yaml.clib-0.2.7 sly-0.4 sortedcontainers-2.4.0 spsdk-1.7.1 termcolor-2.2.0 tlv8-0.10.0 tqdm-4.65.0 typing_extensions-4.3.0 urllib3-1.26.15 wrapt-1.15.0 jas@kaka:~$Then upgrading the device worked remarkable well, although I wish that the tool would have printed URLs and checksums for the firmware files to allow easy confirmation.
jas@kaka:~$ PATH=$PATH:/home/jas/.local/bin
jas@kaka:~$ nitropy start list
Command line tool to interact with Nitrokey devices 0.4.34
:: 'Nitrokey Start' keys:
FSIJ-1.2.15-5D271572: Nitrokey Nitrokey Start (RTM.12.1-RC2-modified)
jas@kaka:~$ nitropy start update
Command line tool to interact with Nitrokey devices 0.4.34
Nitrokey Start firmware update tool
Platform: Linux-5.15.0-67-generic-x86_64-with-glibc2.35
System: Linux, is_linux: True
Python: 3.10.6
Saving run log to: /tmp/nitropy.log.gc5753a8
Admin PIN:
Firmware data to be used:
- FirmwareType.REGNUAL: 4408, hash: ...b'72a30389' valid (from ...built/RTM.13/regnual.bin)
- FirmwareType.GNUK: 129024, hash: ...b'25a4289b' valid (from ...prebuilt/RTM.13/gnuk.bin)
Currently connected device strings:
Device:
Vendor: Nitrokey
Product: Nitrokey Start
Serial: FSIJ-1.2.15-5D271572
Revision: RTM.12.1-RC2-modified
Config: *:*:8e82
Sys: 3.0
Board: NITROKEY-START-G
initial device strings: [ 'name': '', 'Vendor': 'Nitrokey', 'Product': 'Nitrokey Start', 'Serial': 'FSIJ-1.2.15-5D271572', 'Revision': 'RTM.12.1-RC2-modified', 'Config': '*:*:8e82', 'Sys': '3.0', 'Board': 'NITROKEY-START-G' ]
Please note:
- Latest firmware available is:
RTM.13 (published: 2022-12-08T10:59:11Z)
- provided firmware: None
- all data will be removed from the device!
- do not interrupt update process - the device may not run properly!
- the process should not take more than 1 minute
Do you want to continue? [yes/no]: yes
...
Starting bootloader upload procedure
Device: Nitrokey Start FSIJ-1.2.15-5D271572
Connected to the device
Running update!
Do NOT remove the device from the USB slot, until further notice
Downloading flash upgrade program...
Executing flash upgrade...
Waiting for device to appear:
Wait 20 seconds.....
Downloading the program
Protecting device
Finish flashing
Resetting device
Update procedure finished. Device could be removed from USB slot.
Currently connected device strings (after upgrade):
Device:
Vendor: Nitrokey
Product: Nitrokey Start
Serial: FSIJ-1.2.19-5D271572
Revision: RTM.13
Config: *:*:8e82
Sys: 3.0
Board: NITROKEY-START-G
device can now be safely removed from the USB slot
final device strings: [ 'name': '', 'Vendor': 'Nitrokey', 'Product': 'Nitrokey Start', 'Serial': 'FSIJ-1.2.19-5D271572', 'Revision': 'RTM.13', 'Config': '*:*:8e82', 'Sys': '3.0', 'Board': 'NITROKEY-START-G' ]
finishing session 2023-03-16 21:49:07.371291
Log saved to: /tmp/nitropy.log.gc5753a8
jas@kaka:~$
jas@kaka:~$ nitropy start list
Command line tool to interact with Nitrokey devices 0.4.34
:: 'Nitrokey Start' keys:
FSIJ-1.2.19-5D271572: Nitrokey Nitrokey Start (RTM.13)
jas@kaka:~$
Before importing the master key to this device, it should be configured. Note the commands in the beginning to make sure scdaemon/pcscd is not running because they may have cached state from earlier cards. Change PIN code as you like after this, my experience with Gnuk was that the Admin PIN had to be changed first, then you import the key, and then you change the PIN.
jas@kaka:~$ gpg-connect-agent "SCD KILLSCD" "SCD BYE" /bye
OK
ERR 67125247 Slut p fil <GPG Agent>
jas@kaka:~$ ps auxww grep -e pcsc -e scd
jas 11651 0.0 0.0 3468 1672 pts/0 R+ 21:54 0:00 grep --color=auto -e pcsc -e scd
jas@kaka:~$ gpg --card-edit
Reader ...........: 20A0:4211:FSIJ-1.2.19-5D271572:0
Application ID ...: D276000124010200FFFE5D2715720000
Application type .: OpenPGP
Version ..........: 2.0
Manufacturer .....: unmanaged S/N range
Serial number ....: 5D271572
Name of cardholder: [not set]
Language prefs ...: [not set]
Salutation .......:
URL of public key : [not set]
Login data .......: [not set]
Signature PIN ....: forced
Key attributes ...: rsa2048 rsa2048 rsa2048
Max. PIN lengths .: 127 127 127
PIN retry counter : 3 3 3
Signature counter : 0
KDF setting ......: off
Signature key ....: [none]
Encryption key....: [none]
Authentication key: [none]
General key info..: [none]
gpg/card> admin
Admin commands are allowed
gpg/card> kdf-setup
gpg/card> passwd
gpg: OpenPGP card no. D276000124010200FFFE5D2715720000 detected
1 - change PIN
2 - unblock PIN
3 - change Admin PIN
4 - set the Reset Code
Q - quit
Your selection? 3
PIN changed.
1 - change PIN
2 - unblock PIN
3 - change Admin PIN
4 - set the Reset Code
Q - quit
Your selection? q
gpg/card> name
Cardholder's surname: Josefsson
Cardholder's given name: Simon
gpg/card> lang
Language preferences: sv
gpg/card> sex
Salutation (M = Mr., F = Ms., or space): m
gpg/card> login
Login data (account name): jas
gpg/card> url
URL to retrieve public key: https://josefsson.org/key-20190320.txt
gpg/card> forcesig
gpg/card> key-attr
Changing card key attribute for: Signature key
Please select what kind of key you want:
(1) RSA
(2) ECC
Your selection? 2
Please select which elliptic curve you want:
(1) Curve 25519
(4) NIST P-384
Your selection? 1
The card will now be re-configured to generate a key of type: ed25519
Note: There is no guarantee that the card supports the requested size.
If the key generation does not succeed, please check the
documentation of your card to see what sizes are allowed.
Changing card key attribute for: Encryption key
Please select what kind of key you want:
(1) RSA
(2) ECC
Your selection? 2
Please select which elliptic curve you want:
(1) Curve 25519
(4) NIST P-384
Your selection? 1
The card will now be re-configured to generate a key of type: cv25519
Changing card key attribute for: Authentication key
Please select what kind of key you want:
(1) RSA
(2) ECC
Your selection? 2
Please select which elliptic curve you want:
(1) Curve 25519
(4) NIST P-384
Your selection? 1
The card will now be re-configured to generate a key of type: ed25519
gpg/card>
jas@kaka:~$ gpg --card-edit
Reader ...........: 20A0:4211:FSIJ-1.2.19-5D271572:0
Application ID ...: D276000124010200FFFE5D2715720000
Application type .: OpenPGP
Version ..........: 2.0
Manufacturer .....: unmanaged S/N range
Serial number ....: 5D271572
Name of cardholder: Simon Josefsson
Language prefs ...: sv
Salutation .......: Mr.
URL of public key : https://josefsson.org/key-20190320.txt
Login data .......: jas
Signature PIN ....: not forced
Key attributes ...: ed25519 cv25519 ed25519
Max. PIN lengths .: 127 127 127
PIN retry counter : 3 3 3
Signature counter : 0
KDF setting ......: on
Signature key ....: [none]
Encryption key....: [none]
Authentication key: [none]
General key info..: [none]
jas@kaka:~$
Once setup, bring out your offline machine and boot it and mount your USB stick with the offline key. The paths below will be different, and this is using a somewhat unorthodox approach of working with fresh GnuPG configuration paths that I chose for the USB stick.
jas@kaka:/media/jas/2c699cbd-b77e-4434-a0d6-0c4965864296$ cp -a gnupghome-backup-masterkey gnupghome-import-nitrokey-5D271572
jas@kaka:/media/jas/2c699cbd-b77e-4434-a0d6-0c4965864296$ gpg --homedir $PWD/gnupghome-import-nitrokey-5D271572 --edit-key B1D2BD1375BECB784CF4F8C4D73CF638C53C06BE
gpg (GnuPG) 2.2.27; Copyright (C) 2021 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Secret key is available.
sec ed25519/D73CF638C53C06BE
created: 2019-03-20 expired: 2019-10-22 usage: SC
trust: ultimate validity: expired
[ expired] (1). Simon Josefsson <simon@josefsson.org>
gpg> keytocard
Really move the primary key? (y/N) y
Please select where to store the key:
(1) Signature key
(3) Authentication key
Your selection? 1
sec ed25519/D73CF638C53C06BE
created: 2019-03-20 expired: 2019-10-22 usage: SC
trust: ultimate validity: expired
[ expired] (1). Simon Josefsson <simon@josefsson.org>
gpg>
Save changes? (y/N) y
jas@kaka:/media/jas/2c699cbd-b77e-4434-a0d6-0c4965864296$
At this point it is useful to confirm that the Nitrokey has the master key available and that is possible to sign statements with it, back on your regular machine:
jas@kaka:~$ gpg --card-status
Reader ...........: 20A0:4211:FSIJ-1.2.19-5D271572:0
Application ID ...: D276000124010200FFFE5D2715720000
Application type .: OpenPGP
Version ..........: 2.0
Manufacturer .....: unmanaged S/N range
Serial number ....: 5D271572
Name of cardholder: Simon Josefsson
Language prefs ...: sv
Salutation .......: Mr.
URL of public key : https://josefsson.org/key-20190320.txt
Login data .......: jas
Signature PIN ....: not forced
Key attributes ...: ed25519 cv25519 ed25519
Max. PIN lengths .: 127 127 127
PIN retry counter : 3 3 3
Signature counter : 1
KDF setting ......: on
Signature key ....: B1D2 BD13 75BE CB78 4CF4 F8C4 D73C F638 C53C 06BE
created ....: 2019-03-20 23:37:24
Encryption key....: [none]
Authentication key: [none]
General key info..: pub ed25519/D73CF638C53C06BE 2019-03-20 Simon Josefsson <simon@josefsson.org>
sec> ed25519/D73CF638C53C06BE created: 2019-03-20 expires: 2023-09-19
card-no: FFFE 5D271572
ssb> ed25519/80260EE8A9B92B2B created: 2019-03-20 expires: 2023-09-19
card-no: FFFE 42315277
ssb> ed25519/51722B08FE4745A2 created: 2019-03-20 expires: 2023-09-19
card-no: FFFE 42315277
ssb> cv25519/02923D7EE76EBD60 created: 2019-03-20 expires: 2023-09-19
card-no: FFFE 42315277
jas@kaka:~$ echo foo gpg -a --sign gpg --verify
gpg: Signature made Thu Mar 16 22:11:02 2023 CET
gpg: using EDDSA key B1D2BD1375BECB784CF4F8C4D73CF638C53C06BE
gpg: Good signature from "Simon Josefsson <simon@josefsson.org>" [ultimate]
jas@kaka:~$
Finally to retrieve and sign a key, for example Andre Heinecke s that I could confirm the OpenPGP key identifier from his business card.
jas@kaka:~$ gpg --locate-external-keys aheinecke@gnupg.com
gpg: key 1FDF723CF462B6B1: public key "Andre Heinecke <aheinecke@gnupg.com>" imported
gpg: Total number processed: 1
gpg: imported: 1
gpg: marginals needed: 3 completes needed: 1 trust model: pgp
gpg: depth: 0 valid: 2 signed: 7 trust: 0-, 0q, 0n, 0m, 0f, 2u
gpg: depth: 1 valid: 7 signed: 64 trust: 7-, 0q, 0n, 0m, 0f, 0u
gpg: next trustdb check due at 2023-05-26
pub rsa3072 2015-12-08 [SC] [expires: 2025-12-05]
94A5C9A03C2FE5CA3B095D8E1FDF723CF462B6B1
uid [ unknown] Andre Heinecke <aheinecke@gnupg.com>
sub ed25519 2017-02-13 [S]
sub ed25519 2017-02-13 [A]
sub rsa3072 2015-12-08 [E] [expires: 2025-12-05]
sub rsa3072 2015-12-08 [A] [expires: 2025-12-05]
jas@kaka:~$ gpg --edit-key "94A5C9A03C2FE5CA3B095D8E1FDF723CF462B6B1"
gpg (GnuPG) 2.2.27; Copyright (C) 2021 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
pub rsa3072/1FDF723CF462B6B1
created: 2015-12-08 expires: 2025-12-05 usage: SC
trust: unknown validity: unknown
sub ed25519/2978E9D40CBABA5C
created: 2017-02-13 expires: never usage: S
sub ed25519/DC74D901C8E2DD47
created: 2017-02-13 expires: never usage: A
The following key was revoked on 2017-02-23 by RSA key 1FDF723CF462B6B1 Andre Heinecke <aheinecke@gnupg.com>
sub cv25519/1FFE3151683260AB
created: 2017-02-13 revoked: 2017-02-23 usage: E
sub rsa3072/8CC999BDAA45C71F
created: 2015-12-08 expires: 2025-12-05 usage: E
sub rsa3072/6304A4B539CE444A
created: 2015-12-08 expires: 2025-12-05 usage: A
[ unknown] (1). Andre Heinecke <aheinecke@gnupg.com>
gpg> sign
pub rsa3072/1FDF723CF462B6B1
created: 2015-12-08 expires: 2025-12-05 usage: SC
trust: unknown validity: unknown
Primary key fingerprint: 94A5 C9A0 3C2F E5CA 3B09 5D8E 1FDF 723C F462 B6B1
Andre Heinecke <aheinecke@gnupg.com>
This key is due to expire on 2025-12-05.
Are you sure that you want to sign this key with your
key "Simon Josefsson <simon@josefsson.org>" (D73CF638C53C06BE)
Really sign? (y/N) y
gpg> quit
Save changes? (y/N) y
jas@kaka:~$
This is on my day-to-day machine, using the NitroKey Start with the offline key. No need to boot the old offline machine just to sign keys or extend expiry anymore! At FOSDEM 23 I managed to get at least one DD signature on my new key, and the Debian keyring maintainers accepted my Ed25519 key. Hopefully I can now finally let my 2014-era RSA3744 key expire in 2023-09-19 and not extend it any further. This should finish my transition to a simpler OpenPGP key setup, yay!
| Publisher: | Tor |
| Copyright: | 2022 |
| ISBN: | 0-7653-8913-4 |
| Format: | Kindle |
| Pages: | 264 |
| Series: | Fall Revolution #1 |
| Publisher: | Orbit |
| Copyright: | 1995 |
| Printing: | 2001 |
| ISBN: | 1-85723-833-8 |
| Format: | Trade paperback |
| Pages: | 341 |
(no site yet)
in October 2021. they generally seem to keep up with
shipping. update (august 2022): they rolled out a second line of
laptops (12th gen), first batch shipped, second batch shipped
late, September 2022 batch was generally on time, see this
spreadsheet for a crowdsourced effort to track those
to get access to the internals. Kind of
scary.
I also actually unplugged a connector in lifting the assembly because
I lifted it towards the monitor, while you actually need to lift it
to the right. Thankfully, the connector didn't break, it just
snapped off and I could plug it back in, no harm done.
Once there, everything is well indicated, with QR codes all over the
place supposedly leading to online instructions.
+) signs
escaped, like this:
https://guides.frame.work/Guide/Framework\+Laptop\+DIY\+Edition\+Quick\+Start\+Guide/57
... which Firefox immediately transforms in:
https://guides.frame.work/Guide/Framework/+Laptop/+DIY/+Edition/+Quick/+Start/+Guide/57
I'm puzzled as to why they would send the URL that way, the proper URL
is of course:
https://guides.frame.work/Guide/Framework+Laptop+DIY+Edition+Quick+Start+Guide/57
(They have also "let the team know about this for feedback and help
resolve the problem with the link" which is a support code word for
"ha-ha! nope! not my problem right now!" Trust me, I know, my own
code word is "can you please make a ticket?")
cat > /etc/X11/xorg.conf.d/40-libinput.conf <<EOF
Section "InputClass"
Identifier "libinput touch pad catchall"
MatchIsTouchpad "on"
MatchDevicePath "/dev/input/event*"
Driver "libinput"
Option "Tapping" "on"
Option "TappingButtonMap" "lmr"
EndSection
EOF
But be aware that once you enable that tapping, you'll need to deal
with palm detection... So I have not actually enabled this in the end.
/etc/systemd/logind.conf.d/power-suspends.conf:
[Login]
HandlePowerKey=suspend
HandlePowerKeyLongPress=poweroff
You might have to create the directory first:
mkdir /etc/systemd/logind.conf.d/
Then restart logind:
systemctl restart systemd-logind
And the power button will suspend! Long-press to power off doesn't
actually work as the laptop immediately suspends...
Note that there's probably half a dozen other ways of doing this,
see this, this, or that.
| Key | Equivalent | Effect | Command |
|---|---|---|---|
| p | Pause | lock screen | xset s activate |
| b | Break | ? | ? |
| k | ScrLk | switch keyboard layout | N/A |
XF86AudioMedia which, interestingly, does
absolutely nothing here. By default, on Windows, it opens your
browser to the Framework website and, on Linux, your "default
media player".
The keyboard backlight can be cycled with fn-space. The
dimmer version is dim enough, and the keybinding is easy to find in
the dark.
A skinny elephant would be performed with alt
PrtScr (above F11) KEY, so for
example alt fn F11 b
should do a hard reset. This comment suggests you need to hold
the fn only if "function lock" is on, but that's
actually the opposite of my experience.
Out of the box, some of the fn keys don't work. Mute,
volume up/down, brightness, monitor changes, and the airplane mode key
all do basically nothing. They don't send proper keysyms to Xorg at
all.
This is a known problem and it's related to the fact that the
laptop has light sensors to adjust the brightness
automatically. Somehow some of those keys (e.g. the brightness
controls) are supposed to show up as a different input device, but
don't seem to work correctly. It seems like the solution is for the
Framework team to write a driver specifically for this, but so far no
progress since July 2022.
In the meantime, the fancy functionality can be supposedly disabled with:
echo 'blacklist hid_sensor_hub' sudo tee /etc/modprobe.d/framework-als-blacklist.conf
... and a reboot. This solution is also documented in the upstream
guide.
Note that there's another solution flying around that fixes this by
changing permissions on the input device but I haven't tested
that or seen confirmation it works.
xbacklight in i3, but out of the box I get
this error:
sep 29 22:09:14 angela i3[5661]: No outputs have backlight property
It just requires this blob in /etc/X11/xorg.conf.d/backlight.conf:
Section "Device"
Identifier "Card0"
Driver "intel"
Option "Backlight" "intel_backlight"
EndSection
This way I can control the actual backlight power with the brightness
keys, and they do significantly reduce power usage.
autorandr setup doesn't work: I have tried saving a
profile and it doesn't get autodetected, so I also first need to do:
autorandr -l framework-external-dual-lg-acer
The magic:
autorandr -l horizontal
... also works well.
The worst problem with those monitors right now is that they have a
radically smaller resolution than the main screen on the laptop, which
means I need to reset the font scaling to normal every time I switch
back and forth between those monitors and the laptop, which means I
actually need to do this:
autorandr -l horizontal &&
eho Xft.dpi: 96 xrdb -merge &&
systemctl restart terminal xcolortaillog background-image emacs &&
i3-msg restart
Kind of disruptive.
5411.85user 673.33system 1:37:46elapsed 103%CPU (0avgtext+0avgdata 831700maxresident)k
10594704inputs+87448000outputs (9131major+410636783minor)pagefaults 0swaps
This was using 16 watts of power, with full screen brightness.
With all 16 cores (make -j16), it takes less than 25 minutes:
19251.06user 2467.47system 24:13.07elapsed 1494%CPU (0avgtext+0avgdata 831676maxresident)k
8321856inputs+87427848outputs (30792major+409145263minor)pagefaults 0swaps
I had to plug the normal power supply after a few minutes because
battery would actually run out using my desk's power grommet (34
watts).
During compilation, fans were spinning really hard, quite noisy, but
not painfully so.
The laptop was sucking 55 watts of power, steadily:
Time User Nice Sys Idle IO Run Ctxt/s IRQ/s Fork Exec Exit Watts
-------- ----- ----- ----- ----- ----- ---- ------ ------ ---- ---- ---- ------
Average 87.9 0.0 10.7 1.4 0.1 17.8 6583.6 5054.3 233.0 223.9 233.1 55.96
GeoMean 87.9 0.0 10.6 1.2 0.0 17.6 6427.8 5048.1 227.6 218.7 227.7 55.96
StdDev 1.4 0.0 1.2 0.6 0.2 3.0 1436.8 255.5 50.0 47.5 49.7 0.20
-------- ----- ----- ----- ----- ----- ---- ------ ------ ---- ---- ---- ------
Minimum 85.0 0.0 7.8 0.5 0.0 13.0 3594.0 4638.0 117.0 111.0 120.0 55.52
Maximum 90.8 0.0 12.9 3.5 0.8 38.0 10174.0 5901.0 374.0 362.0 375.0 56.41
-------- ----- ----- ----- ----- ----- ---- ------ ------ ---- ---- ---- ------
Summary:
CPU: 55.96 Watts on average with standard deviation 0.20
Note: power read from RAPL domains: package-0, uncore, package-0, core, psys.
These readings do not cover all the hardware in this device.
Memtest86+ v6.00b3 12th Gen Intel(R) Core(TM) i5-1240P
CLK/Temp: 2112MHz 78/78 C Pass 2% #
L1 Cache: 48KB 414 GB/s Test 46% ##################
L2 Cache: 1.25MB 118 GB/s Test #3 [Moving inversions, 1s & 0s]
L3 Cache: 12MB 43 GB/s Testing: 16GB - 18GB [1GB of 15.7GB]
Memory : 15.7GB 14.9 GB/s Pattern:
--------------------------------------------------------------------------------
CPU: 4P+8E-Cores (16T) SMP: 8T (PAR)) Time: 0:27:23 Status: Pass \
RAM: 1600MHz (DDR4-3200) CAS 22-22-22-51 Pass: 1 Errors: 0
--------------------------------------------------------------------------------
Memory SPD Information
----------------------
- Slot 2: 16GB DDR-4-3200 - Crucial CT16G4SFRA32A.C16FP (2022-W23)
Framework FRANMACP04
<ESC> Exit <F1> Configuration <Space> Scroll Lock 6.00.unknown.x64
So about 30 minutes for a full 16GB memory test.
Unable to find medium containing a live file system
This, it turns out, was fixed in Debian lately, so a daily GRML
build will not have this problems. The upcoming 2022 release
(likely 2022.10 or 2022.11) will also get the fix.
I did manage to boot the development version of the Debian
installer which was a surprisingly good experience: it mounted the
encrypted drives and did everything pretty smoothly. It even offered
me to reinstall the boot loader, but that ultimately (and correctly, as
it turns out) failed because I didn't have a /boot/efi partition.
At this point, I realized there was no easy way out of this, and I
just proceeded to completely reinstall Debian. I had a spare NVMe
drive lying around (backups FTW!) so I just swapped that in, rebooted
in the Debian installer, and did a clean install. I wanted to switch
to bookworm anyways, so I guess that's done too.
/etc/ssh
/var/lib/puppet
So that I would keep the previous machine's identity. That way I could
contact the Puppet server and install whatever was missing. I used my
Puppet optimization
trick to do a batch
install and then I had a good base setup, although not exactly as it
was before. 1700 packages were installed manually on angela before
the reinstall, and not in Puppet.
I did not inspect each one individually, but I did go through /etc
and copied over more SSH keys, for backups and SMTP over SSH.
apt install fwupd-amd64-signed
fwupdmgr refresh
fwupdmgr get-updates
fwupdmgr update
Nice. The 12th gen BIOS updates, currently (January 2023) beta,
can be deployed through LVFS with:
fwupdmgr enable-remote lvfs-testing
echo 'DisableCapsuleUpdateOnDisk=true' >> /etc/fwupd/uefi_capsule.conf
fwupdmgr update
Those instructions come from the beta forum post. I performed the
BIOS update on 2023-01-16T16:00-0500.
cat >> /etc/default/console-setup <<EOF
FONTFACE="Terminus"
FONTSIZE=32x16
EOF
echo GRUB_GFXMODE=1024x768 >> /etc/default/grub
update-grub
.Xresources will make everything look much bigger:
! 1.5*96
Xft.dpi: 144
Apparently, some of this can also help:
! These might also be useful depending on your monitor and personal preference:
Xft.autohint: 0
Xft.lcdfilter: lcddefault
Xft.hintstyle: hintfull
Xft.hinting: 1
Xft.antialias: 1
Xft.rgba: rgb
It my experience it also makes things look a little fuzzier, which is
frustrating because you have this awesome monitor but everything looks
out of focus. Just bumping Xft.dpi by a 1.5 factor looks good to me.
The Debian Wiki has a page on HiDPI, but it's not as good as the
Arch Wiki, where the above blurb comes from. I am not using the
latter because I suspect it's causing some of the "fuzziness".
TODO: find the equivalent of this GNOME hack in i3? (gsettings set
org.gnome.mutter experimental-features
"['scale-monitor-framebuffer']"), taken from this Framework
guide
Quick boot and Quiet boot in
the BIOS to diagnose the above boot issues. This, in turn, triggers a
bug where the BIOS boot manager (F12) would just hang
completely. It would also fail to boot from an external USB drive.
The current fix (as of BIOS 3.03) is to re-enable both Quick
boot and Quiet boot. Presumably this is something that will get
fixed in a future BIOS update.
Note that the following keybindings are active in the BIOS POST
check:
| Key | Meaning |
|---|---|
| F2 | Enter BIOS setup menu |
| F12 | Enter BIOS boot manager |
| Delete | Enter BIOS setup menu |
ip/iw/wpa-supplicant
(yes, after repeatedly copying a bunch more packages over to get
those bootstrapped). (Next time I should probably try something like
this post.)
Thankfully, I had a little USB-C dongle with a RJ-45 jack lying
around. That also required a firmware blob, but it was a single
package to copy over, and with that loaded, I had network.
Eventually, I did managed to make WiFi work; the problem was more on
the side of "I forgot how to configure a WPA network by hand from the
commandline" than anything else. NetworkManager worked fine and got
WiFi working correctly.
Note that this is with Debian bookworm, which has the 5.19 Linux
kernel, and with the firmware-nonfree (firmware-iwlwifi,
specifically) package.
Time User Nice Sys Idle IO Run Ctxt/s IRQ/s Fork Exec Exit Watts
-------- ----- ----- ----- ----- ----- ---- ------ ------ ---- ---- ---- ------
Average 1.7 0.0 0.5 97.6 0.2 1.2 4684.9 1985.2 126.6 39.1 128.0 7.57
GeoMean 1.4 0.0 0.4 97.6 0.1 1.2 4416.6 1734.5 111.6 27.9 113.3 7.54
StdDev 1.0 0.2 0.2 1.2 0.0 0.5 1584.7 1058.3 82.1 44.0 80.2 0.71
-------- ----- ----- ----- ----- ----- ---- ------ ------ ---- ---- ---- ------
Minimum 0.2 0.0 0.2 94.9 0.1 1.0 2242.0 698.2 82.0 17.0 82.0 6.36
Maximum 4.1 1.1 1.0 99.4 0.2 3.0 8687.4 4445.1 463.0 249.0 449.0 9.10
-------- ----- ----- ----- ----- ----- ---- ------ ------ ---- ---- ---- ------
Summary:
System: 7.57 Watts on average with standard deviation 0.71
Expansion cards matter a lot in the battery life (see below for a
thorough discussion), my normal setup is 2xUSB-C and 1xUSB-A (yes,
with an empty slot, and yes, to save power).
Interestingly, playing a video in a (720p) window in a window takes up
more power (10.5W) than in full screen (9.5W) but I blame that on my
desktop setup (i3 + compton)... Not sure if mpv hits the
VA-API, maybe not in windowed mode. Similar results with 1080p,
interestingly, except the window struggles to keep up altogether. Full
screen playback takes a relatively comfortable 9.5W, which means a
solid 5h+ of playback, which is fine by me.
Fooling around the web, small edits, youtube-dl, and I'm at around 80%
battery after about an hour, with an estimated 5h left, which is a
little disappointing. I had a 7h remaining estimate before I started
goofing around Discourse, so I suspect the website is a pretty
big battery drain, actually. I see about 10-12 W, while I was probably at
half that (6-8W) just playing music with mpv in the background...
In other words, it looks like editing posts in Discourse with Firefox
takes a solid 4-6W of power. Amazing and gross.
(When writing about abusive power usage generates more power usage, is
that an heisenbug? Or schr dinbug?)
tlp-stat -b and, unfortunately, the "ampere"
unit makes it quite hard to compare those, because voltage is not
necessarily the same between the two platforms.
enable_fbc=1powertop --auto-tuneapt install tlp && systemctl enable tlpnvme.noacpi=1 mem_sleep_default=deep on the kernel command line
may help with standby power usagetlp.conf: tlp.patch.
Specifically, the kernel's energy-aware scheduling heuristics don't work well on those CPUs. A number of features present there complicate the energy picture; these include SMT, Intel's "turbo boost" mode, and the CPU's internal power-management mechanisms. For many workloads, running on an ostensibly more power-hungry Pcore can be more efficient than using an Ecore. Time for discussion of the problem was lacking, though, and the session came to a close.All this to say that the 12gen Intel line shipped with this Framework series should have better power management thanks to its power-saving cores. And Linux has had the scheduler changes to make use of this (but maybe is still having trouble). In any case, this might not be the source of power management problems on my laptop, quite the opposite. Also note that the firmware updates for various chipsets are supposed to improve things eventually. On the other hand, The Verge simply declared the whole P-series a mistake...
powertop --auto-tune and tlp's
PCIE_ASPM_ON_BAT=powersupersave basically did nothing: I was stuck
at 10W power usage in powertop (600+mA in tlp-stat).
Apparently, I should be able to reach the C8 CPU power state (or
even C9, C10) in powertop, but I seem to be stock at
C7. (Although I'm not sure how to read that tab in powertop: in the
Core(HW) column there's only C3/C6/C7 states, and most cores are 85%
in C7 or maybe C6. But the next column over does show many CPUs in
C10 states...
As it turns out, the graphics card actually takes up a good chunk of
power unless proper power management is enabled (see below). After
tweaking this, I did manage to get down to around 7W power usage in
powertop.
Expansion cards actually do take up power, and so does the screen,
obviously. The fully-lit screen takes a solid 2-3W of power compared
to the fully dimmed screen. When removing all expansion cards and
making the laptop idle, I can spin it down to 4 watts power usage at
the moment, and an amazing 2 watts when the screen turned off.
| Device | Minimum | Average | Max | Stdev | Note |
|---|---|---|---|---|---|
| Screen, 100% | 2.4W | 2.6W | 2.8W | N/A | |
| Screen, 1% | 30mW | 140mW | 250mW | N/A | |
| Backlight 1 | 290mW | ? | ? | ? | fairly small, all things considered |
| Backlight 2 | 890mW | 1.2W | 3W? | 460mW? | geometric progression |
| Backlight 3 | 1.69W | 1.5W | 1.8W? | 390mW? | significant power use |
| Radios | 100mW | 250mW | N/A | N/A | |
| USB-C | N/A | N/A | N/A | N/A | negligible power drain |
| USB-A | 10mW | 10mW | ? | 10mW | almost negligible |
| DisplayPort | 300mW | 390mW | 600mW | N/A | not passive |
| HDMI | 380mW | 440mW | 1W? | 20mW | not passive |
| 1TB SSD | 1.65W | 1.79W | 2W | 12mW | significant, probably higher when busy |
| MicroSD | 1.6W | 3W | 6W | 1.93W | highest power usage, possibly even higher when busy |
| Ethernet | 1.69W | 1.64W | 1.76W | N/A | comparable to the SSD card |
It seems the USB A must have power going to it all the time, that the old USB 2 and 3 protocols, the USB C only provides power when there is a connection. Old versus new.Apparently, this is a problem specific to the USB-C to USB-A adapter that ships with the Framework. Some people have actually changed their orders to all USB-C because of this problem, but I'm not sure the problem is as serious as claimed in the forums. I couldn't reproduce the "one watt" power drains suggested elsewhere, at least not repeatedly. (A previous version of this post did show such a power drain, but it was in a less controlled test environment than the series of more rigorous tests above.) The worst offenders are the storage cards: the SSD drive takes at least one watt of power and the MicroSD card seems to want to take all the way up to 6 watts of power, both just sitting there doing nothing. This confirms claims of 1.4W for the SSD (but not 5W) power usage found elsewhere. The former post has instructions on how to disable the card in software. The MicroSD card has been reported as using 2 watts, but I've seen it as high as 6 watts, which is pretty damning. The Framework team has a beta update for the DisplayPort adapter but currently only for Windows (LVFS technically possible, "under investigation"). A USB-A firmware update is also under investigation. It is therefore likely at least some of those power management issues will eventually be fixed. Note that the upcoming Ethernet card has a reported 2-8W power usage, depending on traffic. I did my own power usage tests in powerstat-wayland and they seem lower than 2W. The upcoming 6.2 Linux kernel might also improve battery usage when idle, see this Phoronix article for details, likely in early 2023.
| Device | Minimum | Average | Max | Stdev | Note |
|---|---|---|---|---|---|
| Baseline | 1.96W | 2.01W | 2.11W | 30mW | 1 USB-C, screen off, backlight off, no radios |
| 2 USB-C | 1.95W | 2.16W | 3.69W | 430mW | USB-C confirmed as mostly passive... |
| 3 USB-C | 1.95W | 2.16W | 3.69W | 430mW | ... although with extra stdev |
| 1TB SSD | 3.72W | 3.85W | 4.62W | 200mW | unchanged from before upgrade |
| 1 USB-A | 1.97W | 2.18W | 4.02W | 530mW | unchanged |
| 2 USB-A | 1.97W | 2.00W | 2.08W | 30mW | unchanged |
| 3 USB-A | 1.94W | 1.99W | 2.03W | 20mW | unchanged |
| MicroSD w/o card | 3.54W | 3.58W | 3.71W | 40mW | significant improvement! 2-3W power saving! |
| MicroSD w/ card | 3.53W | 3.72W | 5.23W | 370mW | new measurement! increased deviation |
| DisplayPort | 2.28W | 2.31W | 2.37W | 20mW | unchanged |
| 1 HDMI | 2.43W | 2.69W | 4.53W | 460mW | unchanged |
| 2 HDMI | 2.53W | 2.59W | 2.67W | 30mW | unchanged |
| External USB | 3.85W | 3.89W | 3.94W | 30mW | new result |
| Ethernet | 3.60W | 3.70W | 4.91W | 230mW | unchanged |
sep 28 11:19:45 angela systemd-sleep[209379]: /sys/class/power_supply/BAT/charge_now = 6045 [mAh]
... to this:
sep 28 11:29:47 angela systemd-sleep[209725]: /sys/class/power_supply/BAT/charge_now = 6037 [mAh]
That's 8mAh per 10 minutes (and 2 seconds), or 48mA, or, with this
battery, about 127 hours or roughly 5 days of standby. Not bad!
In comparison, here is my really old x220, before:
sep 29 22:13:54 emma systemd-sleep[176315]: /sys/class/power_supply/BAT0/energy_now = 5070 [mWh]
... after:
sep 29 22:23:54 emma systemd-sleep[176486]: /sys/class/power_supply/BAT0/energy_now = 4980 [mWh]
... which is 90 mwH in 10 minutes, or a whopping 540mA, which was
possibly okay when this battery was new (62000 mAh, so about 100
hours, or about 5 days), but this battery is almost dead and has
only 5210 mAh when full, so only 10 hours standby.
And here is the Framework performing a similar test, before:
sep 29 22:27:04 angela systemd-sleep[4515]: /sys/class/power_supply/BAT1/charge_full = 3518 [mAh]
sep 29 22:27:04 angela systemd-sleep[4515]: /sys/class/power_supply/BAT1/charge_now = 2861 [mAh]
... after:
sep 29 22:37:08 angela systemd-sleep[4743]: /sys/class/power_supply/BAT1/charge_now = 2812 [mAh]
... which is 49mAh in a little over 10 minutes (and 4 seconds), or
292mA, much more than the Purism, but half of the X220. At this rate,
the battery would last on standby only 12 hours!! That is pretty
bad.
Note that this was done with the following expansion cards:
nvme.noacpi=1 but this
still gives me about 5mAh/min (or 300mA).
Adding mem_sleep_default=deep to the kernel command line does make a
difference. Before:
sep 29 23:03:11 angela systemd-sleep[3699]: /sys/class/power_supply/BAT1/charge_now = 2544 [mAh]
... after:
sep 29 23:04:25 angela systemd-sleep[4039]: /sys/class/power_supply/BAT1/charge_now = 2542 [mAh]
... which is 2mAh in 74 seconds, which is 97mA, brings us to a more
reasonable 36 hours, or a day and a half. It's still above the x220
power usage, and more than an order of magnitude more than the Purism
laptop. It's also far from the 0.4% promised by upstream, which
would be 14mA for the 3500mAh battery.
It should also be noted that this "deep" sleep mode is a little more
disruptive than regular sleep. As you can see by the timing, it took
more than 10 seconds for the laptop to resume, which feels a little
alarming as your banging the keyboard to bring it back to life.
You can confirm the current sleep mode with:
# cat /sys/power/mem_sleep
s2idle [deep]
In the above, deep is selected. You can change it on the fly with:
printf s2idle > /sys/power/mem_sleep
Here's another test:
sep 30 22:25:50 angela systemd-sleep[32207]: /sys/class/power_supply/BAT1/charge_now = 1619 [mAh]
sep 30 22:31:30 angela systemd-sleep[32516]: /sys/class/power_supply/BAT1/charge_now = 1613 [mAh]
... better! 6 mAh in about 6 minutes, works out to 63.5mA, so more
than two days standby.
A longer test:
oct 01 09:22:56 angela systemd-sleep[62978]: /sys/class/power_supply/BAT1/charge_now = 3327 [mAh]
oct 01 12:47:35 angela systemd-sleep[63219]: /sys/class/power_supply/BAT1/charge_now = 3147 [mAh]
That's 180mAh in about 3.5h, 52mA! Now at 66h, or almost 3 days.
I wasn't sure why I was seeing such fluctuations in those tests, but
as it turns out, expansion card power tests show that they do
significantly affect power usage, especially the SSD drive, which can
take up to two full watts of power even when idle. I didn't control
for expansion cards in the above tests running them with whatever
card I had plugged in without paying attention so it's likely the
cause of the high power usage and fluctuations.
It might be possible to work around this problem by disabling USB
devices before suspend. TODO. See also this post.
In the meantime, I have been able to get much better suspend
performance by unplugging all modules. Then I get this result:
oct 04 11:15:38 angela systemd-sleep[257571]: /sys/class/power_supply/BAT1/charge_now = 3203 [mAh]
oct 04 15:09:32 angela systemd-sleep[257866]: /sys/class/power_supply/BAT1/charge_now = 3145 [mAh]
Which is 14.8mA! Almost exactly the number promised by Framework! With
a full battery, that means a 10 days suspend time. This is actually
pretty good, and far beyond what I was expecting when starting down
this journey.
So, once the expansion cards are unplugged, suspend power usage is
actually quite reasonable. More detailed standby tests are available
in the standby-tests page, with a summary below.
There is also some hope that the Chromebook edition
specifically designed with a specification of 14 days standby
time could bring some firmware improvements back down to the
normal line. Some of those issues were reported upstream in April
2022, but there doesn't seem to have been any progress there
since.
TODO: one final solution here is suspend-then-hibernate, which
Windows uses for this
TODO: consider implementing the S0ix sleep states , see also troubleshooting
TODO: consider https://github.com/intel/pm-graph
| Device | Wattage | Amperage | Days | Note |
|---|---|---|---|---|
| baseline | 0.25W | 16mA | 9 | sleep=deep nvme.noacpi=1 |
| s2idle | 0.29W | 18.9mA | ~7 | sleep=s2idle nvme.noacpi=1 |
| normal nvme | 0.31W | 20mA | ~7 | sleep=s2idle without nvme.noacpi=1 |
| 1 USB-C | 0.23W | 15mA | ~10 | |
| 2 USB-C | 0.23W | 14.9mA | same as above | |
| 1 USB-A | 0.75W | 48.7mA | 3 | +500mW (!!) for the first USB-A card! |
| 2 USB-A | 1.11W | 72mA | 2 | +360mW |
| 3 USB-A | 1.48W | 96mA | <2 | +370mW |
| 1TB SSD | 0.49W | 32mA | <5 | +260mW |
| MicroSD | 0.52W | 34mA | ~4 | +290mW |
| DisplayPort | 0.85W | 55mA | <3 | +620mW (!!) |
| 1 HDMI | 0.58W | 38mA | ~4 | +250mW |
| 2 HDMI | 0.65W | 42mA | <4 | +70mW |
| Device | Wattage | Amperage | Days | Note |
|---|---|---|---|---|
| baseline | 0.25W | 16mA | 9 | no cards, same as before upgrade |
| 1 USB-C | 0.25W | 16mA | 9 | same as before |
| 2 USB-C | 0.25W | 16mA | 9 | same |
| 1 USB-A | 0.80W | 62mA | 3 | +550mW!! worse than before |
| 2 USB-A | 1.12W | 73mA | <2 | +320mW, on top of the above, bad! |
| Ethernet | 0.62W | 40mA | 3-4 | new result, decent |
| 1TB SSD | 0.52W | 34mA | 4 | a bit worse than before (+2mA) |
| MicroSD | 0.51W | 22mA | 4 | same |
| DisplayPort | 0.52W | 34mA | 4+ | upgrade improved by 300mW |
| 1 HDMI | ? | 38mA | ? | same |
| 2 HDMI | ? | 45mA | ? | a bit worse than before (+3mA) |
| Normal | 1.08W | 70mA | ~2 | Ethernet, 2 USB-C, USB-A |
sudo ectool fwchargelimit 80
I looked at building this myself but failed to run it. I opened a
RFP in Debian so that we can ship this in Debian, and also documented
my work there.
Note that there is now a counter that tracks charge/discharge
cycles. It's visible in tlp-stat -b, which is a nice
improvement:
root@angela:/home/anarcat# tlp-stat -b
--- TLP 1.5.0 --------------------------------------------
+++ Battery Care
Plugin: generic
Supported features: none available
+++ Battery Status: BAT1
/sys/class/power_supply/BAT1/manufacturer = NVT
/sys/class/power_supply/BAT1/model_name = Framewo
/sys/class/power_supply/BAT1/cycle_count = 3
/sys/class/power_supply/BAT1/charge_full_design = 3572 [mAh]
/sys/class/power_supply/BAT1/charge_full = 3541 [mAh]
/sys/class/power_supply/BAT1/charge_now = 1625 [mAh]
/sys/class/power_supply/BAT1/current_now = 178 [mA]
/sys/class/power_supply/BAT1/status = Discharging
/sys/class/power_supply/BAT1/charge_control_start_threshold = (not available)
/sys/class/power_supply/BAT1/charge_control_end_threshold = (not available)
Charge = 45.9 [%]
Capacity = 99.1 [%]
One thing that is still missing is the charge threshold data (the
(not available) above). There's been some work to make that
accessible in August, stay tuned? This would also make it possible
implement hysteresis support.
powertop service which I run at boot time to tweak some power saving
settings.
It seems like this:
echo 'on' > '/sys/bus/usb/devices/4-2/power/control'
... is a good workaround to bring the card back online. You can even
return to power saving mode and the card will still work:
echo 'auto' > '/sys/bus/usb/devices/4-2/power/control'
Further research by Matt_Hartley from the Framework
Team found this issue in the tlp tracker that shows how the
USB_AUTOSUSPEND setting enables the power saving even if the
driver doesn't support it, which, in retrospect, just sounds like a
bad idea. To quote that issue:
By default, USB power saving is active in the kernel, but not force-enabled for incompatible drivers. That is, devices that support suspension will suspend, drivers that do not, will not.So the fix is actually to uninstall
tlp or disable that setting by
adding this to /etc/tlp.conf:
USB_AUTOSUSPEND=0
... but that disables auto-suspend on all USB devices, which may
hurt other power usage performance. I have found that a a
combination of:
USB_AUTOSUSPEND=1
USB_DENYLIST="0bda:8156"
and this on the kernel commandline:
usbcore.quirks=0bda:8156:k
... actually does work correctly. I now have this in my
/etc/default/grub.d/framework-tweaks.cfg file:
# net.ifnames=0: normal interface names ffs (e.g. eth0, wlan0, not wlp166
s0)
# nvme.noacpi=1: reduce SSD disk power usage (not working)
# mem_sleep_default=deep: reduce power usage during sleep (not working)
# usbcore.quirk is a workaround for the ethernet card suspend bug: https:
//guides.frame.work/Guide/Fedora+37+Installation+on+the+Framework+Laptop/
108?lang=en
GRUB_CMDLINE_LINUX="net.ifnames=0 nvme.noacpi=1 mem_sleep_default=deep usbcore.quirks=0bda:8156:k"
# fix the resolution in grub for fonts to not be tiny
GRUB_GFXMODE=1024x768
Other than that, I haven't been able to max out the card because I
don't have other 2.5Gbit/s equipment at home, which is strangely
satisfying. But running against my Turris Omnia
router, I could pretty much max a gigabit fairly easily:
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 1.09 GBytes 937 Mbits/sec 238 sender
[ 5] 0.00-10.00 sec 1.09 GBytes 934 Mbits/sec receiver
The card doesn't require any proprietary firmware blobs which is
surprising. Other than the power saving issues, it just works.
In my power tests (see powerstat-wayland), the Ethernet card seems
to use about 1.6W of power idle, without link, in the above "quirky"
configuration where the card is functional but without autosuspend.
cd /lib/firmware && rm adlp_guc_70.1.1.bin adlp_guc_69.0.3.bin
update-initramfs -u
You will get the following warning during build, which is good as
it means the problematic firmware is disabled:
W: Possible missing firmware /lib/firmware/i915/adlp_guc_69.0.3.bin for module i915
W: Possible missing firmware /lib/firmware/i915/adlp_guc_70.1.1.bin for module i915
But then it also means that critical firmware isn't loaded, which
means, among other things, a higher battery drain. I was able to move
from 8.5-10W down to the 7W range after making the firmware work
properly. This is also after turning the backlight all the way down,
as that takes a solid 2-3W in full blast.
The proper fix is to use some compositing manager. I ended up using
compton with the following systemd unit:
[Unit]
Description=start compositing manager
PartOf=graphical-session.target
ConditionHost=angela
[Service]
Type=exec
ExecStart=compton --show-all-xerrors --backend glx --vsync opengl-swc
Restart=on-failure
[Install]
RequiredBy=graphical-session.target
compton is orphaned however, so you might be tempted to use
picom instead, but in my experience the latter uses much
more power (1-2W extra, similar experience). I also tried
compiz but it would just crash with:
anarcat@angela:~$ compiz --replace
compiz (core) - Warn: No XI2 extension
compiz (core) - Error: Another composite manager is already running on screen: 0
compiz (core) - Fatal: No manageable screens found on display :0
When running from the base session, I would get this instead:
compiz (core) - Warn: No XI2 extension
compiz (core) - Error: Couldn't load plugin 'ccp'
compiz (core) - Error: Couldn't load plugin 'ccp'
Thanks to EmanueleRocca for figuring all that out. See also this
discussion about power management on the Framework forum.
Note that Wayland environments do not require any special
configuration here and actually work better, see my Wayland migration
notes for details.
dmesg:
[ 19.534429] Intel(R) Wireless WiFi driver for Linux
[ 19.534691] iwlwifi 0000:a6:00.0: enabling device (0000 -> 0002)
[ 19.541867] iwlwifi 0000:a6:00.0: firmware: failed to load iwlwifi-ty-a0-gf-a0-72.ucode (-2)
[ 19.541881] iwlwifi 0000:a6:00.0: firmware: failed to load iwlwifi-ty-a0-gf-a0-72.ucode (-2)
[ 19.541882] iwlwifi 0000:a6:00.0: Direct firmware load for iwlwifi-ty-a0-gf-a0-72.ucode failed with error -2
[ 19.541890] iwlwifi 0000:a6:00.0: firmware: failed to load iwlwifi-ty-a0-gf-a0-71.ucode (-2)
[ 19.541895] iwlwifi 0000:a6:00.0: firmware: failed to load iwlwifi-ty-a0-gf-a0-71.ucode (-2)
[ 19.541896] iwlwifi 0000:a6:00.0: Direct firmware load for iwlwifi-ty-a0-gf-a0-71.ucode failed with error -2
[ 19.541903] iwlwifi 0000:a6:00.0: firmware: failed to load iwlwifi-ty-a0-gf-a0-70.ucode (-2)
[ 19.541907] iwlwifi 0000:a6:00.0: firmware: failed to load iwlwifi-ty-a0-gf-a0-70.ucode (-2)
[ 19.541908] iwlwifi 0000:a6:00.0: Direct firmware load for iwlwifi-ty-a0-gf-a0-70.ucode failed with error -2
[ 19.541913] iwlwifi 0000:a6:00.0: firmware: failed to load iwlwifi-ty-a0-gf-a0-69.ucode (-2)
[ 19.541916] iwlwifi 0000:a6:00.0: firmware: failed to load iwlwifi-ty-a0-gf-a0-69.ucode (-2)
[ 19.541917] iwlwifi 0000:a6:00.0: Direct firmware load for iwlwifi-ty-a0-gf-a0-69.ucode failed with error -2
[ 19.541922] iwlwifi 0000:a6:00.0: firmware: failed to load iwlwifi-ty-a0-gf-a0-68.ucode (-2)
[ 19.541926] iwlwifi 0000:a6:00.0: firmware: failed to load iwlwifi-ty-a0-gf-a0-68.ucode (-2)
[ 19.541927] iwlwifi 0000:a6:00.0: Direct firmware load for iwlwifi-ty-a0-gf-a0-68.ucode failed with error -2
[ 19.541933] iwlwifi 0000:a6:00.0: firmware: failed to load iwlwifi-ty-a0-gf-a0-67.ucode (-2)
[ 19.541937] iwlwifi 0000:a6:00.0: firmware: failed to load iwlwifi-ty-a0-gf-a0-67.ucode (-2)
[ 19.541937] iwlwifi 0000:a6:00.0: Direct firmware load for iwlwifi-ty-a0-gf-a0-67.ucode failed with error -2
[ 19.544244] iwlwifi 0000:a6:00.0: firmware: direct-loading firmware iwlwifi-ty-a0-gf-a0-66.ucode
[ 19.544257] iwlwifi 0000:a6:00.0: api flags index 2 larger than supported by driver
[ 19.544270] iwlwifi 0000:a6:00.0: TLV_FW_FSEQ_VERSION: FSEQ Version: 0.63.2.1
[ 19.544523] iwlwifi 0000:a6:00.0: firmware: failed to load iwl-debug-yoyo.bin (-2)
[ 19.544528] iwlwifi 0000:a6:00.0: firmware: failed to load iwl-debug-yoyo.bin (-2)
[ 19.544530] iwlwifi 0000:a6:00.0: loaded firmware version 66.55c64978.0 ty-a0-gf-a0-66.ucode op_mode iwlmvm
Some of those are available in the latest upstream firmware package
(iwlwifi-ty-a0-gf-a0-71.ucode, -68, and -67), but not all
(e.g. iwlwifi-ty-a0-gf-a0-72.ucode is missing) . It's unclear what
those do or don't, as the WiFi seems to work well without them.
I still copied them in from the latest linux-firmware package in the
hope they would help with power management, but I did not notice a
change after loading them.
There are also multiple knobs on the iwlwifi and iwlmvm
drivers. The latter has a power_schmeme setting which defaults to
2 (balanced), setting it to 3 (low power) could improve
battery usage as well, in theory. The iwlwifi driver also has
power_save (defaults to disabled) and power_level (1-5, defaults
to 1) settings. See also the output of modinfo iwlwifi and
modinfo iwlmvm for other driver options.
compton, above), I tested the classic
glxgears.
Running in a window gives me odd results, as the gears basically grind
to a halt:
Running synchronized to the vertical refresh. The framerate should be
approximately the same as the monitor refresh rate.
137 frames in 5.1 seconds = 26.984 FPS
27 frames in 5.4 seconds = 5.022 FPS
Ouch. 5FPS!
But interestingly, once the window is in full screen, it does hit the
monitor refresh rate:
300 frames in 5.0 seconds = 60.000 FPS
I'm not really a gamer and I'm not normally using any of that fancy
graphics acceleration stuff (except maybe my browser does?).
I installed intel-gpu-tools for the intel_gpu_top
command to confirm the GPU was engaged when doing those simulations. A
nice find. Other useful diagnostic tools include glxgears and
glxinfo (in mesa-utils) and (vainfo in vainfo).
Following to this post, I also made sure to have those settings
in my about:config in Firefox, or, in user.js:
user_pref("media.ffmpeg.vaapi.enabled", true);
Note that the guide suggests many other settings to tweak, but those
might actually be overkill, see this comment and its parents. I
did try forcing hardware acceleration by setting gfx.webrender.all
to true, but everything became choppy and weird.
The guide also mentions installing the intel-media-driver package,
but I could not find that in Debian.
The Arch wiki has, as usual, an excellent reference on hardware
acceleration in Firefox.
~/.config/chromium-flags.conf but that doesn't actually work in
Debian. I had to put the flag in
/etc/chromium.d/disable-compositing, like this:
export CHROMIUM_FLAGS="$CHROMIUM_FLAGS --disable-gpu-compositing"
It's possible another one of the hundreds of flags might fix this
issue better, but I don't really have time to go through this entire,
incomplete, and unofficial list (!?!).
Signal Desktop is a similar problem, and doesn't reuse those flags
(because of course it doesn't). Instead I had to rewrite the wrapper
script in /usr/local/bin/signal-desktop to use this instead:
exec /usr/bin/flatpak run --branch=stable --arch=x86_64 org.signal.Signal --disable-gpu-compositing "$@"
This was mostly done in this Puppet commit.
I haven't figured out the root of this problem. I did try using
picom and xcompmgr; they both suffer from the same issue. Another
Debian testing user on Wayland told me they haven't seen this problem,
so hopefully this can be fixed by switching to
wayland.
Jan 20 12:49:10 angela kernel: Asynchronous wait on fence 0000:00:02.0:sway[104431]:cb0ae timed out (hint:intel_atomic_commit_ready [i915])
Jan 20 12:49:15 angela kernel: i915 0000:00:02.0: [drm] GPU HANG: ecode 12:0:00000000
Jan 20 12:49:15 angela kernel: i915 0000:00:02.0: [drm] Resetting chip for stopped heartbeat on rcs0
Jan 20 12:49:15 angela kernel: i915 0000:00:02.0: [drm] GuC firmware i915/adlp_guc_70.1.1.bin version 70.1
Jan 20 12:49:15 angela kernel: i915 0000:00:02.0: [drm] HuC firmware i915/tgl_huc_7.9.3.bin version 7.9
Jan 20 12:49:15 angela kernel: i915 0000:00:02.0: [drm] HuC authenticated
Jan 20 12:49:15 angela kernel: i915 0000:00:02.0: [drm] GuC submission enabled
Jan 20 12:49:15 angela kernel: i915 0000:00:02.0: [drm] GuC SLPC enabled
It's a solid 30 seconds graphical hang. Maybe the keyboard and
everything else keeps working. The latter bug report is quite long,
with many comments, but this one from January 2023 seems to say
that Sway 1.8 fixed the problem. There's also an earlier patch to
add an extra kernel parameter that supposedly fixes that too. There's
all sorts of other workarounds in there, for example this:
echo "options i915 enable_dc=1 enable_guc_loading=1 enable_guc_submission=1 edp_vswing=0 enable_guc=2 enable_fbc=1 enable_psr=1 disable_power_well=0" sudo tee /etc/modprobe.d/i915.conf
from this comment... So that one is unsolved, as far as the
upstream drivers are concerned, but maybe could be fixed through Sway.
#framework on https://libera.chat/| Publisher: | Tor |
| Copyright: | August 2018 |
| ISBN: | 1-4668-6573-3 |
| Format: | Kindle |
| Pages: | 564 |
hz.tools will be tagged
#hztools.librtlsdr to read in
an IQ stream, did some filtering, and played the real valued audio stream via
pulseaudio. Over 4 years this has slowly grown through persistence, lots of
questions to too many friends to thank (although I will try), and the eternal
patience of my wife hearing about radios nonstop for years into a number
of Go repos that can do quite a bit, and support a handful of radios.
I ve resisted making the repos public not out of embarrassment or a desire to
keep secrets, but rather, an attempt to keep myself free of any maintenance
obligations to users so that I could freely break my own API, add and remove
API surface as I saw fit. The worst case was to have this project feel like
work, and I can t imagine that will happen if I feel frustrated by PRs
that are getting ahead of me solving problems I didn t yet know about, or
bugs I didn t understand the fix for.
As my rate of changes to the most central dependencies has slowed, i ve begun
to entertain the idea of publishing them. After a bit of back and forth, I ve
decided
it s time to make a number of them public,
and to start working on them in the open, as I ve built up a bit of knowledge
in the space, and I and feel confident that the repo doesn t contain overt
lies. That s not to say it doesn t contain lies, but those lies are likely
hidden and lurking in the dark. Beware.
That being said, it shouldn t be a surprise to say I ve not published
everything yet for the same reasons as above. I plan to open repos as the
rate of changes slows and I understand the problems the library solves well
enough or if the project dead ends and I ve stopped learning.
),
and from time-to-time I ve made sure that my code runs on OpenBSD
too.
Platforms beyond that will likely not be supported at the expense of either
of those two. I ll take fixes for bugs that fix a problem on another
platform, but not damage the code to work around issues / lack of features
on other platforms (like Windows).hz.tools/rf library contains the abstract concept of frequency, and
some very basic helpers to interact with frequency ranges (such as helpers
to deal with frequency ranges, or frequency range math) as well as frequencies
and some very basic conversions (to meters, etc) and parsers (to parse values
like 10MHz). This ensures that all the hz.tools libraries have a shared
understanding of Frequencies, a standard way of representing ranges of
Frequencies, and the ability to handle the IO boundary with things like CLI
arguments, JSON or YAML.
The git repo can be found at
github.com/hztools/go-rf, and is
importable as hz.tools/rf.
// Parse a frequency using hz.tools/rf.ParseHz, and print it to stdout.
freq := rf.MustParseHz("-10kHz")
fmt.Printf("Frequency: %s\n", freq+rf.MHz)
// Prints: 'Frequency: 990kHz'
// Return the Intersection between two RF ranges, and print
// it to stdout.
r1 := rf.Range rf.KHz, rf.MHz
r2 := rf.Range rf.Hz(10), rf.KHz * 100
fmt.Printf("Range: %s\n", r1.Intersection(r2))
// Prints: Range: 1000Hz->100kHz
io idioms
so that this library feels as idiomatic as it can, so that Go builtins interact
with IQ in a way that s possible to reason about, and to avoid reinventing the
wheel by designing new API surface. While some of the API looks (and is even
called) the same thing as a similar function in io, the implementation is
usually a lot more naive, and may have unexpected sharp edges such as
concurrency issues or performance problems.
The following IQ types are implemented using the sdr.Samples interface. The
hz.tools/sdr package contains helpers for conversion between types, and some
basic manipulation of IQ streams.
| IQ Format | hz.tools Name | Underlying Go Type |
|---|---|---|
| Interleaved uint8 (rtl-sdr) | sdr.SamplesU8 |
[][2]uint8 |
| Interleaved int8 (hackrf, uhd) | sdr.SamplesI8 |
[][2]int8 |
| Interleaved int16 (pluto, uhd) | sdr.SamplesI16 |
[][2]int16 |
| Interleaved float32 (airspy, uhd) | sdr.SamplesC64 |
[]complex64 |
| SDR | Format | RX/TX | State |
|---|---|---|---|
| rtl | u8 | RX | Good |
| HackRF | i8 | RX/TX | Good |
| PlutoSDR | i16 | RX/TX | Good |
| rtl kerberos | u8 | RX | Old |
| uhd | i16/c64/i8 | RX/TX | Good |
| airspyhf | c64 | RX | Exp |
| Import | What is it? |
|---|---|
| hz.tools/sdr | Core IQ types, supporting types and implementations that interact with the byte boundary |
| hz.tools/sdr/rtl | sdr.Receiver implementation using librtlsdr. |
| hz.tools/sdr/rtl/kerberos | Helpers to enable coherent RX using the Kerberos SDR. |
| hz.tools/sdr/rtl/e4k | Helpers to interact with the E4000 RTL-SDR dongle. |
| hz.tools/sdr/fft | Interfaces for performing an FFT, which are implemented by other packages. |
| hz.tools/sdr/rtltcp | sdr.Receiver implementation for rtl_tcp servers. |
| hz.tools/sdr/pluto | sdr.Transceiver implementation for the PlutoSDR using libiio. |
| hz.tools/sdr/uhd | sdr.Transceiver implementation for UHD radios, specifically the B210 and B200mini |
| hz.tools/sdr/hackrf | sdr.Transceiver implementation for the HackRF using libhackrf. |
| hz.tools/sdr/mock | Mock SDR for testing purposes. |
| hz.tools/sdr/airspyhf | sdr.Receiver implementation for the AirspyHF+ Discovery with libairspyhf. |
| hz.tools/sdr/internal/simd | SIMD helpers for IQ operations, written in Go ASM. This isn t the best to learn from, and it contains pure go implemtnations alongside. |
| hz.tools/sdr/stream | Common Reader/Writer helpers that operate on IQ streams. |
hz.tools/fftw package contains bindings to libfftw3 to implement
the hz.tools/sdr/fft.Planner type to transform between the time and
frequency domain.
The git repo can be found at
github.com/hztools/go-fftw, and is
importable as hz.tools/fftw.
This is the default throughout most of my codebase, although that default is
only expressed at the leaf package libraries should not be hardcoding the
use of this library in favor of taking an fft.Planner, unless it s used as
part of testing. There are a bunch of ways to do an FFT out there, things like
clFFT or a pure-go FFT implementation could be plugged in depending on what s
being solved for.
hz.tools/fm and hz.tools/am packages contain demodulators for
AM analog radio, and FM analog radio. This code is a bit old, so it has
a lot of room for cleanup, but it ll do a very basic demodulation of IQ
to audio.
The git repos can be found at
github.com/hztools/go-fm and
github.com/hztools/go-am,
and are importable as
hz.tools/fm and
hz.tools/am.
As a bonus, the hz.tools/fm package also contains a modulator, which has been
tested on the air and with some of my handheld radios. This code is a bit
old, since the hz.tools/fm code is effectively the first IQ processing code
I d ever written, but it still runs and I run it from time to time.
// Basic sketch for playing FM radio using a reader stream from
// an SDR or other IQ stream.
bandwidth := 150*rf.KHz
reader, err = stream.ConvertReader(reader, sdr.SampleFormatC64)
if err != nil
...
demod, err := fm.Demodulate(reader, fm.DemodulatorConfig
Deviation: bandwidth / 2,
Downsample: 8, // some value here depending on sample rate
Planner: fftw.Plan,
)
if err != nil
...
speaker, err := pulseaudio.NewWriter(pulseaudio.Config
Format: pulseaudio.SampleFormatFloat32NE,
Rate: demod.SampleRate(),
AppName: "rf",
StreamName: "fm",
Channels: 1,
SinkName: "",
)
if err != nil
...
buf := make([]float32, 1024*64)
for
i, err := demod.Read(buf)
if err != nil
...
if i == 0
panic("...")
if err := speaker.Write(buf[:i]); err != nil
...
hz.tools/rfcap package is the reference implementation of the
rfcap spec , and is how I store IQ captures
locally, and how I send them across a byte boundary.
The git repo can be found at
github.com/hztools/go-rfcap, and is
importable as hz.tools/rfcap.
If you re interested in storing IQ in a way others can use, the better approach
is to use SigMF rfcap exists for cases
like using UNIX pipes to move IQ around, through APIs, or when I send
IQ data through an OS socket, to ensure the sample format (and other metadata)
is communicated with it.
rfcap has a number of limitations, for instance, it can not express a change
in frequency or sample rate during the capture, since the header is fixed at
the beginning of the file.
I have mentioned several times in this blog, as well as by other
communication means, that I am very happy with the laptop I bought
(used) about a year and a half ago: an ARM-based Lenovo Yoga
C630.
Yes, I knew from the very beginning that using this laptop would pose
a challenge to me in many ways, as full hardware support for ARM
laptops are nowhere as easy as for plain boring x86 systems. But the
advantages far outweigh the inconvenience (i.e. the hoops I had to
jump through to handle video-out when I started teaching
presentially,
which are fortunately a thing of the past now).
Anyway This post is not about my laptop.
Back in 2018, I was honored to be appointed as a member of the Debian
Technical
Committee. Of
course, that meant (due to the very clear and clever point 6.2.7.1 of
the Debian
Constitution that
my tenure in the Committee (as well as Niko Tyni s) finished in
January 1, 2023. We were invited to take part of a Jitsi call as a
last meeting, as well as to welcome Matthew Garrett to the Committee.
Of course, I arranged so I would be calling from my desktop system at
work (for which I have an old, terrible webcam but as long as I
don t need to control screen sharing too finely, mostly works). Out of
eight people in the call, two had complete or quite crippling failures
with their multimedia setup, and one had a frozen image (at least as
far as I could tell).
So Yes, Debian is indeed good and easy and simple and reliable for
most nontechnical users using standard tools.
But I guess that we power users enjoy tweaking our setup to our
precise particular liking. Or that we just don t care about
frivolities such as having a working multimedia setup.
Or I don t know what happens.
But the fact that close to half of the Technical Committee, which
should consist of Debian Developers who know their way around
technical obstacles, cannot get a working multimedia setup for a
simple, easy WebRTC call (even after a pandemic that made us all work
via teleconferencing solutions on a daily basis!) is just Beautiful
Very few highly-anticipated movies appear in January and February, as the bigger releases are timed so they can be considered for the Golden Globes in January and the Oscars in late February or early March, so film fans have the advantage of a few weeks after the New Year to collect their thoughts on the year ahead. In other words, I'm not actually late in outlining below the films I'm most looking forward to in 2023...
Barbie No, seriously! If anyone can make a good film about a doll franchise, it's probably Greta Gerwig. Not only was Little Women (2019) more than admirable, the same could be definitely said for Lady Bird (2017). More importantly, I can't help feel she was the real 'Driver' behind Frances Ha (2012), one of the better modern takes on Claudia Weill's revelatory Girlfriends (1978). Still, whenever I remember that Barbie will be a film about a billion-dollar toy and media franchise with a nettlesome history, I recall I rubbished the "Facebook film" that turned into The Social Network (2010). Anyway, the trailer for Barbie is worth watching, if only because it seems like a parody of itself.
Blitz It's difficult to overstate just how important the aerial bombing of London during World War II is crucial to understanding the British psyche, despite it being a constructed phenomenon from the outset. Without wishing to underplay the deaths of over 40,000 civilian deaths, Angus Calder pointed out in the 1990s that the modern mythology surrounding the event "did not evolve spontaneously; it was a propaganda construct directed as much at [then neutral] American opinion as at British." It will therefore be interesting to see how British Grenadian Trinidadian director Steve McQueen addresses a topic so essential to the British self-conception. (Remember the controversy in right-wing circles about the sole Indian soldier in Christopher Nolan's Dunkirk (2017)?) McQueen is perhaps best known for his 12 Years a Slave (2013), but he recently directed a six-part film anthology for the BBC which addressed the realities of post-Empire immigration to Britain, and this leads me to suspect he sees the Blitz and its surrounding mythology with a more critical perspective. But any attempt to complicate the story of World War II will be vigorously opposed in a way that will make the recent hullabaloo surrounding The Crown seem tame. All this is to say that the discourse surrounding this release may be as interesting as the film itself.
Dune, Part II Coming out of the cinema after the first part of Denis Vileneve's adaptation of Dune (2021), I was struck by the conception that it was less of a fresh adaptation of the 1965 novel by Frank Herbert than an attempt to rehabilitate David Lynch's 1984 version and in a broader sense, it was also an attempt to reestablish the primacy of cinema over streaming TV and the myriad of other distractions in our lives. I must admit I'm not a huge fan of the original novel, finding within it a certain prurience regarding hereditary military regimes and writing about them with a certain sense of glee that belies a secret admiration for them... not to mention an eyebrow-raising allegory for the Middle East. Still, Dune, Part II is going to be a fantastic spectacle.
Ferrari It'll be curious to see how this differs substantially from the recent Ford v Ferrari (2019), but given that Michael Mann's Heat (1995) so effectively re-energised the gangster/heist genre, I'm more than willing to kick the tires of this about the founder of the eponymous car manufacturer. I'm in the minority for preferring Mann's Thief (1981) over Heat, in part because the former deals in more abstract themes, so I'd have perhaps prefered to look forward to a more conceptual film from Mann over a story about one specific guy.
How Do You Live There are a few directors one can look forward to watching almost without qualification, and Hayao Miyazaki (My Neighbor Totoro, Kiki's Delivery Service, Princess Mononoke Howl's Moving Castle, etc.) is one of them. And this is especially so given that The Wind Rises (2013) was meant to be the last collaboration between Miyazaki and Studio Ghibli. Let's hope he is able to come out of retirement in another ten years.
Indiana Jones and the Dial of Destiny Given I had a strong dislike of Indiana Jones and the Kingdom of the Crystal Skull (2008), I seriously doubt I will enjoy anything this film has to show me, but with 1981's Raiders of the Lost Ark remaining one of my most treasured films (read my brief homage), I still feel a strong sense of obligation towards the Indiana Jones name, despite it feeling like the copper is being pulled out of the walls of this franchise today.
Kafka I only know Polish filmmaker Agnieszka Holland through her Spoor (2017), an adaptation of Olga Tokarczuk's 2009 eco-crime novel Drive Your Plow Over the Bones of the Dead. I wasn't an unqualified fan of Spoor (nor the book on which it is based), but I am interested in Holland's take on the life of Czech author Franz Kafka, an author enmeshed with twentieth-century art and philosophy, especially that of central Europe. Holland has mentioned she intends to tell the story "as a kind of collage," and I can hope that it is an adventurous take on the over-furrowed biopic genre. Or perhaps Gregor Samsa will awake from uneasy dreams to find himself transformed in his bed into a huge verminous biopic.
The Killer It'll be interesting to see what path David Fincher is taking today, especially after his puzzling and strangely cold Mank (2020) portraying the writing process behind Orson Welles' Citizen Kane (1941). The Killer is said to be a straight-to-Netflix thriller based on the graphic novel about a hired assassin, which makes me think of Fincher's Zodiac (2007), and, of course, Se7en (1995). I'm not as entranced by Fincher as I used to be, but any film with Michael Fassbender and Tilda Swinton (with a score by Trent Reznor) is always going to get my attention.
Killers of the Flower Moon In Killers of the Flower Moon, Martin Scorsese directs an adaptation of a book about the FBI's investigation into a conspiracy to murder Osage tribe members in the early years of the twentieth century in order to deprive them of their oil-rich land. (The only thing more quintessentially American than apple pie is a conspiracy combined with a genocide.) Separate from learning more about this disquieting chapter of American history, I'd love to discover what attracted Scorsese to this particular story: he's one of the few top-level directors who have the ability to lucidly articulate their intentions and motivations.
Napoleon It often strikes me that, despite all of his achievements and fame, it's somehow still possible to claim that Ridley Scott is relatively underrated compared to other directors working at the top level today. Besides that, though, I'm especially interested in this film, not least of all because I just read Tolstoy's War and Peace (read my recent review) and am working my way through the mind-boggling 431-minute Soviet TV adaptation, but also because several auteur filmmakers (including Stanley Kubrick) have tried to make a Napoleon epic and failed.
Oppenheimer In a way, a biopic about the scientist responsible for the atomic bomb and the Manhattan Project seems almost perfect material for Christopher Nolan. He can certainly rely on stars to queue up to be in his movies (Robert Downey Jr., Matt Damon, Kenneth Branagh, etc.), but whilst I'm certain it will be entertaining on many fronts, I fear it will fall into the well-established Nolan mould of yet another single man struggling with obsession, deception and guilt who is trying in vain to balance order and chaos in the world.
The Way of the Wind Marked by philosophical and spiritual overtones, all of Terrence Malick's films are perfumed with themes of transcendence, nature and the inevitable conflict between instinct and reason. My particular favourite is his stunning Days of Heaven (1978), but The Thin Red Line (1998) and A Hidden Life (2019) also touched me ways difficult to relate, and are one of the few films about the Second World War that don't touch off my sensitivity about them (see my remarks about Blitz above). It is therefore somewhat Malickian that his next film will be a biblical drama about the life of Jesus. Given Malick's filmography, I suspect this will be far more subdued than William Wyler's 1959 Ben-Hur and significantly more equivocal in its conviction compared to Paolo Pasolini's ardently progressive The Gospel According to St. Matthew (1964). However, little beyond that can be guessed, and the film may not even appear until 2024 or even 2025.
Zone of Interest I was mesmerised by Jonathan Glazer's Under the Skin (2013), and there is much to admire in his borderline 'revisionist gangster' film Sexy Beast (2000), so I will definitely be on the lookout for this one. The only thing making me hesitate is that Zone of Interest is based on a book by Martin Amis about a romance set inside the Auschwitz concentration camp. I haven't read the book, but Amis has something of a history in his grappling with the history of the twentieth century, and he seems to do it in a way that never sits right with me. But if Paul Verhoeven's Starship Troopers (1997) proves anything at all, it's all in the adaption.
.proto). The protoc compiler and a language-specific plugin convert it into
code:
$ head flow-4.proto syntax = "proto3"; package decoder; option go_package = "akvorado/inlet/flow/decoder"; message FlowMessagev4 uint64 TimeReceived = 2; uint32 SequenceNum = 3; uint64 SamplingRate = 4; uint32 FlowDirection = 5; $ protoc -I=. --plugin=protoc-gen-go --go_out=module=akvorado:. flow-4.proto $ head inlet/flow/decoder/flow-4.pb.go // Code generated by protoc-gen-go. DO NOT EDIT. // versions: // protoc-gen-go v1.28.0 // protoc v3.21.12 // source: inlet/flow/data/schemas/flow-4.proto package decoder import ( protoreflect "google.golang.org/protobuf/reflect/protoreflect"
goos: linux
goarch: amd64
pkg: akvorado/inlet/flow
cpu: AMD Ryzen 5 5600X 6-Core Processor
initial.txt final.txt
sec/op sec/op vs base
Netflow/with_encoding-12 12.963 2% 7.836 1% -39.55% (p=0.000 n=10)
Sflow/with_encoding-12 19.37 1% 10.15 2% -47.63% (p=0.000 n=10)
Decode() method is a thin layer above
GoFlow2 producer and stores the decoded data into the in-memory structure
generated by protoc. Later, some of the data will be encoded directly during
flow decoding. This is why we measure both the decoding and the
encoding.2
func BenchmarkDecodeEncodeSflow(b *testing.B) r := reporter.NewMock(b) sdecoder := sflow.New(r) data := helpers.ReadPcapPayload(b, filepath.Join("decoder", "sflow", "testdata", "data-1140.pcap")) for _, withEncoding := range []bool true, false title := map[bool]string true: "with encoding", false: "without encoding", [withEncoding] var got []*decoder.FlowMessage b.Run(title, func(b *testing.B) for i := 0; i < b.N; i++ got = sdecoder.Decode(decoder.RawFlow Payload: data, Source: net.ParseIP("127.0.0.1"), ) if withEncoding for _, flow := range got buf := []byte buf = protowire.AppendVarint(buf, uint64(proto.Size(flow))) proto.MarshalOptions .MarshalAppend(buf, flow) )
google.golang.org/protobuf is not the most
efficient one. For a long time, people were relying on gogoprotobuf.
However, the project is now deprecated. A good replacement is
vtprotobuf.3
goos: linux
goarch: amd64
pkg: akvorado/inlet/flow
cpu: AMD Ryzen 5 5600X 6-Core Processor
initial.txt bench-2.txt
sec/op sec/op vs base
Netflow/with_encoding-12 12.96 2% 10.28 2% -20.67% (p=0.000 n=10)
Netflow/without_encoding-12 8.935 2% 8.975 2% ~ (p=0.143 n=10)
Sflow/with_encoding-12 19.37 1% 16.67 2% -13.93% (p=0.000 n=10)
Sflow/without_encoding-12 14.62 3% 14.87 1% +1.66% (p=0.007 n=10)
.proto file. The wire format is simple and rely a lot on variable-width
integers.
Variable-width integers, or varints, are an efficient way of encoding unsigned
integers using a variable number of bytes, from one to ten, with small values
using fewer bytes. They work by splitting integers into 7-bit payloads and using
the 8th bit as a continuation indicator, set to 1 for all payloads
except the last.
protowire.AppendTag() encodes a tag,protowire.AppendVarint() encodes a variable-width integer, andprotowire.AppendBytes() append bytes as is.ProtobufIndex) and to generate a proto definition file (fields starting with
Protobuf):
type Column struct Key ColumnKey Name string Disabled bool // [ ] // For protobuf. ProtobufIndex protowire.Number ProtobufType protoreflect.Kind // Uint64Kind, Uint32Kind, ProtobufEnum map[int]string ProtobufEnumName string ProtobufRepeated bool
protowire functions to directly
encode the fields while decoding the flows. They skip disabled fields or
non-repeated fields already encoded. Here is an excerpt of the sFlow
decoder:
sch.ProtobufAppendVarint(bf, schema.ColumnBytes, uint64(recordData.Base.Length)) sch.ProtobufAppendVarint(bf, schema.ColumnProto, uint64(recordData.Base.Protocol)) sch.ProtobufAppendVarint(bf, schema.ColumnSrcPort, uint64(recordData.Base.SrcPort)) sch.ProtobufAppendVarint(bf, schema.ColumnDstPort, uint64(recordData.Base.DstPort)) sch.ProtobufAppendVarint(bf, schema.ColumnEType, helpers.ETypeIPv4)
type FlowMessage struct TimeReceived uint64 SamplingRate uint32 // For exporter classifier ExporterAddress netip.Addr // For interface classifier InIf uint32 OutIf uint32 // For geolocation or BMP SrcAddr netip.Addr DstAddr netip.Addr NextHop netip.Addr // Core component may override them SrcAS uint32 DstAS uint32 GotASPath bool // protobuf is the protobuf representation for the information not contained above. protobuf []byte protobufSet bitset.BitSet
protobuf slice holds encoded data. It is initialized with a capacity of
500 bytes to avoid resizing during encoding. There is also some reserved room at
the beginning to be able to encode the total size as a variable-width integer.
Upon finalizing encoding, the remaining fields are added and the message length
is prefixed:
func (schema *Schema) ProtobufMarshal(bf *FlowMessage) []byte schema.ProtobufAppendVarint(bf, ColumnTimeReceived, bf.TimeReceived) schema.ProtobufAppendVarint(bf, ColumnSamplingRate, uint64(bf.SamplingRate)) schema.ProtobufAppendIP(bf, ColumnExporterAddress, bf.ExporterAddress) schema.ProtobufAppendVarint(bf, ColumnSrcAS, uint64(bf.SrcAS)) schema.ProtobufAppendVarint(bf, ColumnDstAS, uint64(bf.DstAS)) schema.ProtobufAppendIP(bf, ColumnSrcAddr, bf.SrcAddr) schema.ProtobufAppendIP(bf, ColumnDstAddr, bf.DstAddr) // Add length and move it as a prefix end := len(bf.protobuf) payloadLen := end - maxSizeVarint bf.protobuf = protowire.AppendVarint(bf.protobuf, uint64(payloadLen)) sizeLen := len(bf.protobuf) - end result := bf.protobuf[maxSizeVarint-sizeLen : end] copy(result, bf.protobuf[end:end+sizeLen]) return result
-benchmem flag to monitor allocation
numbers. Each allocation incurs an additional cost to the garbage collector. The
Go profiler is a valuable tool for identifying areas of code that can be
optimized:
$ go test -run=__nothing__ -bench=Netflow/with_encoding \ > -benchmem -cpuprofile profile.out \ > akvorado/inlet/flow goos: linux goarch: amd64 pkg: akvorado/inlet/flow cpu: AMD Ryzen 5 5600X 6-Core Processor Netflow/with_encoding-12 143953 7955 ns/op 8256 B/op 134 allocs/op PASS ok akvorado/inlet/flow 1.418s $ go tool pprof profile.out File: flow.test Type: cpu Time: Feb 4, 2023 at 8:12pm (CET) Duration: 1.41s, Total samples = 2.08s (147.96%) Entering interactive mode (type "help" for commands, "o" for options) (pprof) web
goos: linux
goarch: amd64
pkg: akvorado/inlet/flow
cpu: AMD Ryzen 5 5600X 6-Core Processor
bench-2.txt bench-3.txt
sec/op sec/op vs base
Netflow/with_encoding-12 10.284 2% 7.758 3% -24.56% (p=0.000 n=10)
Netflow/without_encoding-12 8.975 2% 7.304 2% -18.61% (p=0.000 n=10)
Sflow/with_encoding-12 16.67 2% 14.26 1% -14.50% (p=0.000 n=10)
Sflow/without_encoding-12 14.87 1% 13.56 2% -8.80% (p=0.000 n=10)
github.com/jhump/protoreflect: the
protoparse package parses the proto definition file we generate and the
dynamic package decodes the messages. Check the ProtobufDecode()
method for more details.4
To get the final figures, I have also optimized the decoding in GoFlow2. It
was relying heavily on binary.Read(). This function may use
reflection in certain cases and each call allocates a byte array to read data.
Replacing it with a more efficient version provides the following
improvement:
goos: linux
goarch: amd64
pkg: akvorado/inlet/flow
cpu: AMD Ryzen 5 5600X 6-Core Processor
bench-3.txt bench-4.txt
sec/op sec/op vs base
Netflow/with_encoding-12 7.758 3% 7.365 2% -5.07% (p=0.000 n=10)
Netflow/without_encoding-12 7.304 2% 6.931 3% -5.11% (p=0.000 n=10)
Sflow/with_encoding-12 14.256 1% 9.834 2% -31.02% (p=0.000 n=10)
Sflow/without_encoding-12 13.559 2% 9.353 2% -31.02% (p=0.000 n=10)
Notice Some paragraphs were editorialized by ChatGPT, using editorialize and keep it short as a prompt. The result was proofread by a human for correctness. The main idea is that ChatGPT should be better at English than me.
vtprotobuf generates more optimized Go code by removing an
abstraction layer. It directly generates the code encoding each field to
bytes:
if m.OutIfSpeed != 0 i = encodeVarint(dAtA, i, uint64(m.OutIfSpeed)) i-- dAtA[i] = 0x6 i-- dAtA[i] = 0xd8
protoprint package to generate proto definition
file. I did not use it.
In 2022 I read 34 books (-19% on last year).
In 2021 roughly a quarter of the books I read were written by women. I was
determined to push that ratio in 2022, so I made an effort to try and only
read books by women. I knew that I wouldn't manage that, but by trying to, I
did get the ratio up to 58% (by page count).
I'm not sure what will happen in 2023. My to-read pile has some back-pressure
from books by male authors I postponed reading in 2022 (in particular new works
by Christopher Priest and Adam Roberts). It's possible the ratio will swing
back the other way, which would mean it would not be worth repeating the
experiment. At least if the ratio is the point of the exercise. But perhaps it
isn't: perhaps the useful outcome is more qualitative than quantitative.
I tried to read some new (to me) authors. I really enjoyed Shirley Jackson (The
Haunting of Hill House, We Have Always Lived In The Castle). I Struggled with
Angela Carter's Heroes and Villains although
I plan to return to her other work, in particular, The Bloody Chamber. I
also got through Donna Tartt's The Secret History on the recommendation of a
friend. I had to push through the first 15% or so but it turned out to be worth
it.
To achieve my goal of publishing one high-quality piece of writing per week this year, I ve put together a draft writing plan and a few organisational notes.
Please let me know what you think - what s missing? what would you like to read more/less of from me?
I aim for each piece of writing to generate discussion, inspire further writing, and raise my visibility and profile with potential customers and peers. Some of the writing will be opinion, but I expect a majority of it will take a learning by teaching approach - aiming to explain and present useful information to the reader while helping me learn more!
debdistdiff for one purpose (comparing two different archives like Trisquel and Ubuntu) and discovered that the tool could be used for another purpose (comparing the Trisquel archive at two different points in time). At this time I realized that I needed a log of all different apt archive metadata to be able to produce an audit log of the differences in time for the archive. I create manually curated git-repositories with the Release/InRelease and the Packages files for each architecture/component of the well-known distributions Trisquel, Ubuntu, Debian and PureOS. Eventually I wrote scripts to automate this, which are now published in the debdistget project.
At this point, one of the early question about per-IP substitution of Release files were lingering in my mind. However with the tooling I now had available, coming up with a way to resolve this was simple! Merely have apt compute a SHA256 checksum of the just downloaded InRelease file, and see if my git repository had the same file. At this point I started reading the Apt source code, and now I had more doubts about the security of my systems than I ever had before. Oh boy how the name Apt has never before felt more Apt?! Oh well, we must leave some exercises for the students. Eventually I realized I wanted to touch as little of apt code basis as possible, and noticed the SigVerify::CopyAndVerify function called ExecGPGV which called apt-key verify which called GnuPG s gpgv. By setting Apt::Key::gpgvcommand I could get apt-key verify to call another tool than gpgv. See where I m going? I thought wrapping this up would now be trivial but for some reason the hash checksum I computed locally never matched what was on my server. I gave up and started working on other things instead.
Today I came back to this idea, and started to debug exactly how the local files looked that I got from apt and how they differed from what I had in my git repositories, that came straight from the apt archives. Eventually I traced this back to SplitClearSignedFile which takes an InRelease file and splits it into two files, probably mimicking the (old?) way of distributing both Release and Release.gpg. So the clearsigned InRelease file is split into one cleartext file (similar to the Release file) and one OpenPGP signature file (similar to the Release.gpg file). But why didn t the cleartext variant of the InRelease file hash to the same value as the hash of the Release file? Sadly they differ by the final newline.
Having solved this technicality, wrapping the pieces up was easy, and I came up with a project apt-canary that provides a script apt-canary-gpgv that verify the local apt release files against something I call a apt canary witness file stored at a URL somewhere.
I m now running apt-canary on my Trisquel aramo laptop, a Trisquel nabia server, and Talos II ppc64el Debian machine. This means I have solved the per-IP substitution worries (or at least made them less likely to occur, having to send the same malicious release files to both GitLab and my system), and allow me to have an audit log of all release files that I actually use for installing and downloading packages.
What do you think? There are clearly a lot of work and improvements to be made. This is a proof-of-concept implementation of an idea, but instead of refining it until perfection and delaying feedback, I wanted to publish this to get others to think about the problems and various ways to resolve them.
Btw, I m going to be at FOSDEM 23 this weekend, helping to manage the Security Devroom. Catch me if you want to chat about this or other things. Happy Hacking!
| Series: | Edinburgh Nights #1 |
| Publisher: | Tor |
| Copyright: | 2021 |
| Printing: | 2022 |
| ISBN: | 1-250-76777-6 |
| Format: | Kindle |
| Pages: | 329 |
| Publisher: | Amazon Original Stories |
| Copyright: | September 2019 |
| ISBN: | 1-5420-9206-X |
| ISBN: | 1-5420-4363-8 |
| ISBN: | 1-5420-9357-0 |
| ISBN: | 1-5420-0434-9 |
| ISBN: | 1-5420-4363-8 |
| ISBN: | 1-5420-4425-1 |
| Format: | Kindle |
| Pages: | 300 |
| Series: | October Daye #12 |
| Publisher: | DAW Books |
| Copyright: | 2018 |
| ISBN: | 0-698-18353-3 |
| Format: | Kindle |
| Pages: | 353 |
| Series: | Discworld #25 |
| Publisher: | Harper |
| Copyright: | November 2000 |
| Printing: | August 2014 |
| ISBN: | 0-06-230736-3 |
| Format: | Mass market |
| Pages: | 435 |
The world is made up of four elements: Earth, Air, Fire, and Water. This is a fact well known even to Corporal Nobbs. It's also wrong. There's a fifth element, and generally it's called Surprise. For example, the dwarfs found out how to turn lead into gold by doing it the hard way. The difference between that and the easy way is that the hard way works.The dwarfs used the lead to make a movable type printing press, which is about to turn William de Worde's small-scale, hand-crafted newsletter into a newspaper. The movable type printing press is not unknown technology. It's banned technology, because the powers that be in Ankh-Morpork know enough to be deeply suspicious of it. The religious establishment doesn't like it because words are too important and powerful to automate. The nobles and the Watch don't like it because cheap words cause problems. And the engraver's guild doesn't like it for obvious reasons. However, Lord Vetinari knows that one cannot apply brakes to a volcano, and commerce with the dwarfs is very important to the city. The dwarfs can continue. At least for now. As in Moving Pictures, most of The Truth is an idiosyncratic speedrun of the social effects of a new technology, this time newspapers. William has no grand plan; he's just an observant man who likes to write, cares a lot about the truth, and accidentally stumbles into editing a newspaper. (This, plus being an estranged son of a rich family, feels very on-point for journalism.) His naive belief is that people want to read true things, since that's what his original patrons wanted. Truth, however, may not be in the top five things people want from a newspaper. This setup requires some narrative force to push it along, which is provided by a plot to depose Vetinari by framing him for murder. The most interesting part of that story is Mr. Pin and Mr. Tulip, the people hired to do the framing and then dispose of the evidence. They're a classic villain type: the brains and the brawn, dangerous, terrifying, and willing to do horrible things to people. But one thing Pratchett excels at is taking a standard character type, turning it a bit sideways, and stuffing in things that one wouldn't think would belong. In this case, that's Mr. Tulip's deep appreciation for, and genius grasp of, fine art. It should not work to have the looming, awful person with anger issues be able to identify the exact heritage of every sculpture and fine piece of goldsmithing, and yet somehow it does. Also as in Moving Pictures (and, in a different way, Soul Music), Pratchett tends to anthropomorphize technology, giving it a life and motivations of its own. In this case, that's William's growing perception of the press as an insatiable maw into which one has to feed words. I'm usually dubious of shifting agency from humans to things when doing social analysis (and there's a lot of social analysis here), but I have to concede that Pratchett captures something deeply true about the experience of feedback loops with an audience. A lot of what Pratchett puts into this book about the problematic relationship between a popular press and the truth is obvious and familiar, but he also makes some subtle points about the way the medium shapes what people expect from it and how people produce content for it that are worthy of Marshall McLuhan. The interactions between William and the Watch were less satisfying. In our world, the US press is, with only rare exceptions, a thoughtless PR organ for police propaganda and the exonerative tense. Pratchett tackles that here... sort of. William vaguely grasps that his job as a reporter may be contrary to the job of the Watch to maintain order, and Vimes's ambivalent feelings towards "solving crimes" push the story in that direction. But this is also Vimes, who is clearly established as one of the good sort and therefore is a bad vehicle for talking about how the police corrupt the press. Pratchett has Vimes and Vetinari tacitly encourage William, which works within the story but takes the pressure off the conflict and leaves William well short of understanding the underlying politics. There's a lot more that could be said about the tension between the press and the authorities, but I think the Discworld setup isn't suitable for it. This is the sort of book that benefits from twenty-four volumes of backstory and practice. Pratchett's Ankh-Morpork cast ticks along like a well-oiled machine, which frees up space that would otherwise have to be spent on establishing secondary characters. The result is a lot of plot and social analysis shoved into a standard-length Discworld novel, and a story that's hard to put down. The balance between humor and plot is just about perfect, the references and allusions aren't overwhelming, and the supporting characters, both new and old, are excellent. We even get a good Death sequence. This is solid, consistent stuff: Discworld as a mature, well-developed setting with plenty of stories left to tell. Followed by Thief of Time in publication order, and later by Monstrous Regiment in the vaguely-connected industrial revolution sequence. Rating: 8 out of 10
secnet - VPN.hippotat - IP-over-HTTP (workaround for bad networks)userv ipif - user-created network interfacesHippotat is a system to allow you to use your normal VPN, ssh, and other applications, even in broken network environments that are only ever tested with web stuff . Packets are parcelled up into HTTP POST requests, resembling form submissions (or JavaScript XMLHttpRequest traffic), and the returned packets arrive via the HTTP response bodies.It doesn t rely on TLS tunnelling so can work even if the local network is trying to intercept TLS. I recently rewrote Hippotat in Rust.
userv ipif
userv ipif is one of the userv utilities.
It allows safe delegation of network routing to unprivileged users. The delegation is of a specific address range, so different ranges can be delegated to different users, and the authorised user cannot interfere with other traffic.
This is used in the default configuration of hippotat packages, so that an ordinary user can start up the hippotat client as needed.
On chiark userv-ipif is used to delegate networking to users, including administrators of allied VPN realms. So chiark actually runs at least 4 VPN-ish systems in production: secnet, hippotat, Mark Wooding s Tripe, and still a few links managed by the now-superseded udptunnel system.
userv
userv ipif is a userv service. That is, it is a facility which uses userv to bridge a privilege boundary.
userv is perhaps my most under-appreciated program. userv can be used to straightforwardly bridge (local) privilege boundaries on Unix systems.
So for example it can:
sudo can do this too but it has quite a few gotchas, and you have to be quite careful how you use it - and its security record isn t great either.hippotat client is a program you can run from the command line as a normal user, if the relevant network addresses have been delegated to you. On chiark, CGI programs run as the providing user - not using suexec (which I don t trust), but via userv.| Publisher: | Amazon Original Stories |
| Copyright: | August 2021 |
| ISBN: | 1-5420-3272-5 |
| ISBN: | 1-5420-3270-9 |
| ISBN: | 1-5420-3271-7 |
| ISBN: | 1-5420-3273-3 |
| ISBN: | 1-5420-3268-7 |
| ISBN: | 1-5420-3269-5 |
| Format: | Kindle |
| Pages: | 168 |
Next.