Search Results: "reg"

26 October 2023

Dima Kogan: Talking to ROS from outside a LAN

The problem
This is about ROS version 1. Version 2 is different, and maybe they fixed stuff. But I kinda doubt it since this thing is heinous in a million ways. Alright so let's say we have have some machines in a LAN doing ROS stuff and we have another machine outside the LAN that wants to listen in (like to get a realtime visualization, say). This is an extremely common scenario, but they created enough hoops to make this not work. Let's say we have 3 computers:
  • router: the bridge between the two networks. This has two NICs. The inner IP is 10.0.1.1 and the outer IP is 12.34.56.78
  • inner: a machine in the LAN that's doing ROS stuff. IP 10.0.1.99
  • outer: a machine outside that LAN that wants to listen in. IP 12.34.56.99
Let's say the router is doing ROS stuff. It's running the ROS master and some nodes like this: 
ROS_IP=10.0.1.1 roslaunch whatever
If you omit the ROS_IP it'll pick router, which may or may not work, depending on how the DNS is set up. Here we set it to 10.0.1.1 to make it possible for the inner machine to communicate (we'll see why in a bit). An aside: ROS should use the IP by default instead of the name because the IP will work even if the DNS isn't set up. If there are multiple extant IPs, it should throw an error. But all that would be way too user-friendly. OK. So we have a ROS master on 10.0.1.1 on the default port: 11311. The inner machine can rostopic echo and all that. Great. What if I try to listen in from outer? I say 
ROS_MASTER_URI=http://12.34.56.78:11311 rostopic list
This connects to the router on that port, and it works well: I get the list of available topics. Here this works because the router is the router. If inner was running the ROS master then we'd need to do a forward for port 11311. In any case, this works and we understand it. So clearly we can talk to the ROS master. Right? Wrong! Let's actually listen in on a specific topic on outer: 
ROS_MASTER_URI=http://12.34.56.78:11311 rostopic echo /some/topic
This does not work. No errors are reported. It just sits there, which looks like no data is coming in on that topic. But this is a lie: it's actually broken.

The diagnosis
So this is our problem. It's a very common use case, and there are plenty of internet people asking about it, with no specific solutions. I debugged it, and the details are here. To figure out what's going on, I made a syscall log on a machine inside the LAN, where a simple rostopic echo does work: 
sysdig -A proc.name=rostopic and fd.type contains ipv -s 2000
This shows us all the communication between inner running rostopic and the server. It's really chatty. It's all TCP. There are multiple connections to the router on port 11311. It also starts up multiple TCP servers on the client that listen to connections; these are likely to be broken if we were running the client on outer and a machine inside the LAN tried to talk to them; but thankfully in my limited testing nothing actually tried to talk to them. The conversations on port 11311 are really long, but here's the punchline. inner tells the router: 
POST /RPC2 HTTP/1.1                                                                                                                 
Host: 10.0.1.1:11311                                                                                                          
Accept-Encoding: gzip                                                                                                               
Content-Type: text/xml                                                                                                              
User-Agent: Python-xmlrpc/3.11                                                                                                      
Content-Length: 390                                                                                                                 
<?xml version='1.0'?>
<methodCall>
<methodName>registerSubscriber</methodName>
<params>
<param>
<value><string>/rostopic_2447878_1698362157834</string></value>
</param>
<param>
<value><string>/some/topic</string></value>
</param>
<param>
<value><string>*</string></value>
</param>
<param>
<value><string>http://inner:38229/</string></value>
</param>
</params>
</methodCall>
Yes. It's laughably chatty. Then the router replies: 
HTTP/1.1 200 OK
Server: BaseHTTP/0.6 Python/3.8.10
Date: Thu, 26 Oct 2023 23:15:28 GMT
Content-type: text/xml
Content-length: 342
<?xml version='1.0'?>
<methodResponse>
<params>
<param>
<value><array><data>
<value><int>1</int></value>
<value><string>Subscribed to [/some/topic]</string></value>
<value><array><data>
<value><string>http://10.0.1.1:45517/</string></value>
</data></array></value>
</data></array></value>
</param>
</params>
</methodResponse>
Then this sequence of system calls happens in the rostopic process (an excerpt from the sysdig log): 
> connect fd=10(<4>) addr=10.0.1.1:45517
< connect res=-115(EINPROGRESS) tuple=10.0.1.99:47428->10.0.1.1:45517 fd=10(<4t>10.0.1.99:47428->10.0.1.1:45517)
< getsockopt res=0 fd=10(<4t>10.0.1.99:47428->10.0.1.1:45517) level=1(SOL_SOCKET) optname=4(SO_ERROR) val=0 optlen=4
So the inner client makes an outgoing TCP connection on the address given to it by the ROS master above: 10.0.1.1:45517. This IP is only accessible from within the LAN, which works fine when talking to it from inner, but would be a problem from the outside. Furthermore, some sort of single-port-forwarding scheme wouldn't fix connecting from outer either, since the port number is dynamic. To confirm what we think is happening, the sequence of syscalls when trying to rostopic echo from outer does indeed fail: 
connect fd=10(<4>) addr=10.0.1.1:45517 
connect res=-115(EINPROGRESS) tuple=10.0.1.1:46204->10.0.1.1:45517 fd=10(<4t>10.0.1.1:46204->10.0.1.1:45517)
getsockopt res=0 fd=10(<4t>10.0.1.1:46204->10.0.1.1:45517) level=1(SOL_SOCKET) optname=4(SO_ERROR) val=-111(ECONNREFUSED) optlen=4
That's the breakage mechanism: the ROS master asks us to communicate on an address we can't talk to. Debugging this is easy with sysdig: 
sudo sysdig -A -s 400 evt.buffer contains '"Subscribed to"' and proc.name=rostopic
This prints out all syscalls seen by the rostopic command that contain the string Subscribed to, so you can see that different addresses the ROS master gives us in response to different commands. OK. So can we get the ROS master to give us an address that we can actually talk to? Sorta. Remember that we invoked the master with 
ROS_IP=10.0.1.1 roslaunch whatever
The ROS_IP environment variable is exactly the address that the master gives out. So in this case, we can fix it by doing this instead: 
ROS_IP=12.34.56.78 roslaunch whatever
Then the outer machine will be asked to talk to 12.34.56.78:45517, which works. Unfortunately, if we do that, then the inner machine won't be able to communicate. So some sort of ssh port forward cannot fix this: we need a lower-level tunnel, like a VPN or something. And another rant. Here rostopic tried to connect to an unreachable address, which failed. But rostopic knows the connection failed! It should throw an error message to the user. Something like this would be wonderful: 
ERROR! Tried to connect to 10.0.1.1:45517 ($ROS_IP:dynamicport), but connect() returned ECONNREFUSED
That would be immensely helpful. It would tell the user that something went wrong (instead of no data being sent), and it would give a strong indication of the problem and how to fix it. But that would be asking too much.

The solution
So we need a VPN-like thing. I just tried sshuttle, and it just works. Start the ROS node in the way that makes connections from within the LAN work: 
ROS_IP=10.0.1.1 roslaunch whatever
Then on the outer client: 
sshuttle -r router 10.0.1.0/24
This connects to the router over ssh and does some hackery to make all connections from outer to 10.0.1.x transparently route into the LAN. On all ports. rostopic echo then works. I haven't done any thorough testing, but hopefully it's reliable and has low overhead; I don't know. I haven't tried it but almost certainly this would work even with the ROS master running on inner. This would be accomplished like this:
  1. Tell ssh how to connect to inner. Dropping this into ~/.ssh/config should do it: 
    Host inner
HostName 10.0.1.99
ProxyJump router
  2. Do the magic thing: 
    sshuttle -r inner 10.0.1.0/24
I'm sure any other VPN-like thing would work also.

23 October 2023

Russ Allbery: Review: Going Postal

Review: Going Postal, by Terry Pratchett
Series: Discworld #33
Publisher: Harper
Copyright: October 2004
Printing: November 2014
ISBN: 0-06-233497-2
Format: Mass market
Pages: 471
Going Postal is the 33rd Discworld novel. You could probably start here if you wanted to; there are relatively few references to previous books, and the primary connection (to Feet of Clay) is fully re-explained. I suspect that's why Going Postal garnered another round of award nominations. There are arguable spoilers for Feet of Clay, however. Moist von Lipwig is a con artist. Under a wide variety of names, he's swindled and forged his way around the Disc, always confident that he can run away from or talk his way out of any trouble. As Going Postal begins, however, it appears his luck has run out. He's about to be hanged. Much to his surprise, he wakes up after his carefully performed hanging in Lord Vetinari's office, where he's offered a choice. He can either take over the Ankh-Morpork post office, or he can die. Moist, of course, immediately agrees to run the post office, and then leaves town at the earliest opportunity, only to be carried back into Vetinari's office by a relentlessly persistent golem named Mr. Pump. He apparently has a parole officer. The clacks, Discworld's telegraph system first seen in The Fifth Elephant, has taken over most communications. The city is now dotted with towers, and the Grand Trunk can take them at unprecedented speed to even far-distant cities like Genua. The post office, meanwhile, is essentially defunct, as Moist quickly discovers. There are two remaining employees, the highly eccentric Junior Postman Groat who is still Junior because no postmaster has lasted long enough to promote him, and the disturbingly intense Apprentice Postman Stanley, who collects pins. Other than them, the contents of the massive post office headquarters are a disturbing mail sorting machine designed by Bloody Stupid Johnson that is not picky about which dimension or timeline the sorted mail comes from, and undelivered mail. A lot of undelivered mail. Enough undelivered mail that there may be magical consequences. All Moist has to do is get the postal system running again. Somehow. And not die in mysterious accidents like the previous five postmasters. Going Postal is a con artist story, but it's also a startup and capitalism story. Vetinari is, as always, solving a specific problem in his inimitable indirect way. The clacks were created by engineers obsessed with machinery and encodings and maintenance, but it's been acquired by... well, let's say private equity, because that's who they are, although Discworld doesn't have that term. They immediately did what private equity always did: cut out everything that didn't extract profit, without regard for either the service or the employees. Since the clacks are an effective monopoly and the new owners are ruthless about eliminating any possible competition, there isn't much to stop them. Vetinari's chosen tool is Moist. There are some parts of this setup that I love and one part that I'm grumbly about. A lot of the fun of this book is seeing Moist pulled into the mission of resurrecting the post office despite himself. He starts out trying to wriggle out of his assigned task, but, after a few early successes and a supernatural encounter with the mail, he can't help but start to care. Reformed con men often make good protagonists because one can enjoy the charisma without disliking the ethics. Pratchett adds the delightfully sharp-witted and cynical Adora Belle Dearheart as a partial reader stand-in, which makes the process of Moist becoming worthy of his protagonist role even more fun. I think that a properly functioning postal service is one of the truly monumental achievements of human society and doesn't get nearly enough celebration (or support, or pay, or good working conditions). Give me a story about reviving a postal service by someone who appreciates the tradition and social role as much as Pratchett clearly does and I'm there. The only frustration is that Going Postal is focused more on an immediate plot, so we don't get to see the larger infrastructure recovery that is clearly needed. (Maybe in later books?) That leads to my grumble, though. Going Postal and specifically the takeover of the clacks is obviously inspired by corporate structures in the later Industrial Revolution, but this book was written in 2004, so it's also a book about private equity and startups. When Vetinari puts a con man in charge of the post office, he runs it like a startup: do lots of splashy things to draw attention, promise big and then promise even bigger, stumble across a revenue source that may or may not be sustainable, hire like mad, and hope it all works out. This makes for a great story in the same way that watching trapeze artists or tightrope walkers is entertaining. You know it's going to work because that's the sort of book you're reading, so you can enjoy the audacity and wonder how Moist will manage to stay ahead of his promises. But it is still a con game applied to a public service, and the part of me that loves the concept of the postal service couldn't stop feeling like this is part of the problem. The dilemma that Vetinari is solving is a bit too realistic, down to the requirement that the post office be self-funding and not depend on city funds and, well, this is repugnant to me. Public services aren't businesses. Societies spend money to build things that they need to maintain society, and postal service is just as much one of those things as roads are. The ability of anyone to send a letter to anyone else, no matter how rural the address is, provides infrastructure on which a lot of important societal structure is built. Pratchett made me care a great deal about Ankh-Morpork's post office (not hard to do), and now I want to see it rebuilt properly, on firm foundations, without splashy promises and without a requirement that it pay for itself. Which I realize is not the point of Discworld at all, but the concept of running a postal service like a startup hits maybe a bit too close to home. Apart from that grumble, this is a great book if you're in the mood for a reformed con man story. I thought the gold suit was a bit over the top, but I otherwise thought Moist's slow conversion to truly caring about his job was deeply satisfying. The descriptions of the clacks are full of askew Discworld parodies of computer networking and encoding that I enjoyed more than I thought I would. This is also the book that introduced the now-famous (among Pratchett fans at least) GNU instruction for the clacks, and I think that scene is the most emotionally moving bit of Pratchett outside of Night Watch. Going Postal is one of the better books in the Discworld series to this point (and I'm sadly getting near the end). If you have less strongly held opinions about management and funding models for public services, or at least are better at putting them aside when reading fantasy novels, you're likely to like it even more than I did. Recommended. Followed by Thud!. The thematic sequel is Making Money. Rating: 8 out of 10

22 October 2023

Ravi Dwivedi: Software Freedom Day at sflc.in

Software Freedom Law Center, India, also known as sflc.in, organized an event to celebrate the Software Freedom Day on 30th September 2023. Me, Sahil, Contrapunctus and Suresh joined. The venue was at the SFLC India office in Delhi. The sflc.in office was on the second floor of what looked like someone s apartment:). I also met Chirag, Orendra, Surbhi and others. My plan was to have a stall on LibreOffice and Prav app to raise awareness about these projects. I didn t have QR code for downloading prav app printed already, so I asked the people at sflc.in if they can get it printed for me. They were very kind and helped me in getting a color printout for me. So, I got a stall in their main room. Surbhi was having an Inkscape stall next to mine and gave me company. People came and asked about the prav project and then I realized I was still too tired to explain the idea behind the prav project and about LibreOffice (after a long Kerala trip). We got a few prav app installs during the event, which is cool.
My stall. Photo credits: Tejaswini.
Sahil had Debian stall and contrapunctus had OpenStreetMap stall. After about an hour, Revolution OS was screened for all of us to watch, along with popcorn. The documentary gave an overview of history of Free Software Movement. The office had a kitchen where fresh chai was being made and served to us. The organizers ordered a lot of good snacks for us.
Snacks and tea at the front desk. CC-BY-SA 4.0 by Ravi Dwivedi.
I came out of the movie hall to take more tea and snacks from the front desk. I saw a beautiful painting was hanging at the wall opposite to the front desk and Tejaswini (from sflc.in) revealed that she had made it. The tea was really good as it was freshly made in the kitchen. After the movie, we played a game of pictionary. We were divided into two teams. The game goes as follows: A person from a team is selected and given a term related to freedom respecting software written on a piece of paper, but concealed from other participants. Then that person draws something on the board (no logo, no alphabets) without speaking. If the team from which the person belongs correctly guesses the term, the team gets one step ahead on the leader board. The team who reaches the finish line wins. I recall some fun pictionaries. Like, the one in the picture below seems far from the word Wireguard and even then someone from the team guessed that word. Our team won in the end \o/.
Pictionary drawing nowhere close to the intended word Wireguard :), which was guessed. Photo by Ravi Dwivedi, CC-BY-SA 4.0.
Then, we posed for a group picture. At the end, SFLC.in had a delicious cake in store for us. They had some merchandise - handbags, T-shirts, etc. which we could take if we donate some amount to SFLC.in. I bought a handbag with Ban Plastic, not Internet written on it in exchange for donation. I hope that gives people around me a powerful message :) .
Group photo. Photo credits: Tejaswini.
Tasty cake. CC-BY-SA 4.0 by Ravi Dwivedi.
Merchandise by sflc.in. CC-BY-SA 4.0 by Ravi Dwivedi.
All in all, a nice event by sflc.in :)

Aigars Mahinovs: Figuring out finances part 3

So now that I have something that looks very much like a budgeting setup going, I am going to .. delete it! Why? Well, at the end of the last part of this, the Firefly III instance was running on a tiny Debian server in a Docker container right next to another Docker container that is running the main user of this server - a Home Assistant instance that has been managing my home for several years already. So why change that? See, there is one bit of knowledge that is very crucial to your Home Assistant experience, which is not really emphasised enough in the Home Assistant documentation. In fact back when I was getting into the Home Assistant both the main documentation and basically all the guides around were just coming off the hype of Docker disrupting everything and that is a big reason why everyone suggested to install and use Home Assistant as a Docker container on top of any kind of stable OS. In fact I used to run it for years on my TerraMaster NAS, just so that I don't have a separate home server running 24/7 at home and just have everything inside the very compact NAS case. So here is the thing you NEED to know - Home Assistant Container is DEMO version of Home Assistant! If you want to have a full Home Assistant experience and use the knowledge of the huge community around the HA space, you have to use the Home Assistant OS. Ideally on dedicated hardware. Ideally on HA Green box, but any tiny PC would also work great. Raspberry Pi 4+ is common, but quite weak as the network size grows and especially the SD card for storage gets old very fast. Get a real small x86 PC with at least 4Gb RAM and a NVME SSD (eMMC is fine too). You want to have an Ethernet port and a few free USB ports. I would also suggest immediately getting HA SkyConnect adapter that can do Zigbee networking and will do Matter soon (tm). I am making do with a SonOff Zigbee gateway, but it is quite hacky to get working and your whole Zigbee communication breaks down if the WiFi goes down - suboptimal. So I took a backup of the Home Assistant instance using it's build-in tools. I took an export of my fully configured Firefly III instance and proceeded to wipe the drive of the NUC. That was not a smart idea. :D On the Home Assitant side I was really frustrated by the documentation that was really focused on users that are (likely) using Windows and are using an SD card in something like Raspberry Pi to get Home Assistant OS running. It recommended downloading Etcher to write the image to the boot medium. That is a really weird piece of software that managed to actually crash consistently when I was trying to run it from Debian Live or Ubuntu Live on my NUC. It took me way too long to give up and try something much simpler - dd. xzcat haos_generic-x86-64-11.0.img.xz dd of=/dev/mmcblk0 bs=1M That just worked, prefectly and really fast. If you want to use a GUI in a live environment, then just using the gnome-disk-utility ("Disks" in Gnome menu) and using the "Restore Disk Image ..." on a partition would work just as well. It even supports decompressing the XZ images directly while writing. But that image is small, will it not have a ton of unused disk space behind the fixed install partition? Yes, it will ... until first boot. The HA OS takes over the empty space after its install partition on the first boot-up and just grows its main partition to take up all the remaining space. Smart. After first boot is completed, the first boot wizard can be accessed via your web browser and one of the prominent buttons there is restoring from backup. So you just give it the backup file and wait. Sadly the restore does not actually give any kind of progress, so your only way to figure out when it is done is opening the same web adress in another browser tab and refresh periodically - after restoring from backup it just boots into the same config at it had before - all the settings, all the devices, all the history is preserved. Even authentification tokens are preserved so if yu had a Home Assitant Mobile installed on your phone (both for remote access and to send location info and phone state, like charging, to HA to trigger automations) then it will just suddenly start working again without further actions needed from your side. That is an almost perfect backup/restore experience. The first thing you get for using the OS version of HA is easy automatic update that also automatically takes a backup before upgrade, so if anything breaks you can roll back with one click. There is also a command-line tool that allows to upgrade, but also downgrade ha-core and other modules. I had to use it today as HA version 23.10.4 actually broke support for the Sonoff bridge that I am using to control Zigbee devices, which are like 90% of all smart devices in my home. Really helpful stuff, but not a must have. What is a must have and that you can (really) only get with Home Assistant Operating System are Addons. Some addons are just normal servers you can run alongside HA on the same HA OS server, like MariaDB or Plex or a file server. That is not the most important bit, but even there the software comes pre-configured to use in a home server configuration and has a very simple config UI to pre-configure key settings, like users, passwords and database accesses for MariaDB - you can litereally in a few clicks and few strings make serveral users each with its own access to its own database. Couple more clicks and the DB is running and will be kept restarted in case of failures. But the real gems in the Home Assistant Addon Store are modules that extend Home Assitant core functionality in way that would be really hard or near impossible to configure in Home Assitant Container manually, especially because no documentation has ever existed for such manual config - everyone just tells you to install the addon from HA Addon store or from HACS. Or you can read the addon metadata in various repos and figure out what containers it actually runs with what settings and configs and what hooks it puts into the HA Core to make them cooperate. And then do it all over again when a new version breaks everything 6 months later when you have already forgotten everything. In the Addons that show up immediately after installation are addons to install the new Matter server, a MariaDB and MQTT server (that other addons can use for data storage and message exchange), Z-Wave support and ESPHome integration and very handy File manager that includes editors to edit Home Assitant configs directly in brower and SSH/Terminal addon that boht allows SSH connection and also a web based terminal that gives access to the OS itself and also to a comand line interface, for example, to do package downgrades if needed or see detailed logs. And also there is where you can get the features that are the focus this year for HA developers - voice enablers. However that is only a beginning. Like in Debian you can add additional repositories to expand your list of available addons. Unlike Debian most of the amazing software that is available for Home Assistant is outside the main, official addon store. For now I have added the most popular addon repository - HACS (Home Assistant Community Store) and repository maintained by Alexbelgium. The first includes things like NodeRED (a workflow based automation programming UI), Tailscale/Wirescale for VPN servers, motionEye for CCTV control, Plex for home streaming. HACS also includes a lot of HA UI enhacement modules, like themes, custom UI control panels like Mushroom or mini-graph-card and integrations that provide more advanced functions, but also require more knowledge to use, like Local Tuya - that is harder to set up, but allows fully local control of (normally) cloud-based devices. And it has AppDaemon - basically a Python based automation framework where you put in Python scrips that get run in a special environment where they get fed events from Home Assistant and can trigger back events that can control everything HA can and also do anything Python can do. This I will need to explore later. And the repository by Alex includes the thing that is actually the focus of this blog post (I know :D) - Firefly III addon and Firefly Importer addon that you can then add to your Home Assistant OS with a few clicks. It also has all kinds of addons for NAS management, photo/video server, book servers and Portainer that lets us setup and run any Docker container inside the HA OS structure. HA OS will detect this and warn you about unsupported processes running on your HA OS instance (nice security feature!), but you can just dismiss that. This will be very helpful very soon. This whole environment of OS and containers and apps really made me think - what was missing in Debian that made the talented developers behind all of that to spend the immense time and effor to setup a completely new OS and app infrastructure and develop a completel paraller developer community for Home Assistant apps, interfaces and configurations. Is there anything that can still be done to make HA community and the general open source and Debian community closer together? HA devs are not doing anything wrong: they are using the best open source can provide, they bring it to people whould could not install and use it otherwise, they are contributing fixes and improvements as well. But there must be some way to do this better, together. So I installed MariaDB, create a user and database for Firefly. I installed Firefly III and configured it to use the MariaDB with the web config UI. When I went into the Firefly III web UI I was confronted with the normal wizard to setup a new instance. And no reference to any backup restore. Hmm, ok. Maybe that goes via the Importer? So I make an access token again, configured the Importer to use that, configured the Nordlinger bank connection settings. Then I tried to import the export that I downloaded from Firefly III before. The importer did not auto-recognose the format. Turns out it is just a list of transactions ... It can only be barely useful if you first manually create all the asset accounts with the same names as before and even then you'll again have to deal with resolving the problem of transfers showing up twice. And all of your categories (that have not been used yet) are gone, your automation rules and bills are gone, your budgets and piggy banks are gone. Boooo. It will be easier for me to recreate my account data from bank exports again than to resolve data in that transaction export. Turns out that Firefly III documenation explicitly recommends making a mysqldump of your own and not rely on anything in the app itself for backup purposes. Kind of sad this was not mentioned in the export page that sure looked a lot like a backup :D After doing all that work all over again I needed to make something new not to feel like I wasted days of work for no real gain. So I started solving a problem I had for a while already - how do I add cash transations to the system when I am out of the house with just my phone in the hand? So far my workaround has been just sending myself messages in WhatsApp with the amount and description of any cash expenses. Two solutions are possible: app and bot. There are actually multiple Android-based phone apps that work with Firefly III API to do full financial management from the phone. However, after trying it out, that is not what I will be using most of the time. First of all this requires your Firefly III instance to be accessible from the Internet. Either via direct API access using some port forwarding and secured with HTTPS and good access tokens, or via a VPN server redirect that is installed on both HA and your phone. Tailscale was really easy to get working. But the power has its drawbacks - adding a new cash transaction requires opening the app, choosing new transaction view, entering descriptio, amount, choosing "Cash" as source account and optionally choosing destination expense account, choosing category and budget and then submitting the form to the server. Sadly none of that really works if you have no Internet or bad Internet at the place where you are using cash. And it's just too many steps. Annoying. An easier alternative is setting up a Telegram bot - it is running in a custom Docker container right next to your Firefly (via Portainer) and you talk to it via a custom Telegram chat channel that you create very easily and quickly. And then you can just tell it "Coffee 5" and it will create a transaction from the (default) cash account in 5 amount with description "Coffee". This part also works if you are offline at the moment - the bot will receive the message once you get back online. You can use Telegram bot menu system to edit the transaction to add categories or expense accounts, but this part only work if you are online. And the Firefly instance does not have to be online at all. Really nifty. So next week I will need to write up all the regular payments as bills in Firefly (again) and then I can start writing a Python script to predict my (financial) future!

Ian Jackson: DigiSpark (ATTiny85) - Arduino, C, Rust, build systems

Recently I completed a small project, including an embedded microcontroller. For me, using the popular Arduino IDE, and C, was a mistake. The experience with Rust was better, but still very exciting, and not in a good way. Here follows the rant. Introduction In a recent project (I ll write about the purpose, and the hardware in another post) I chose to use a DigiSpark board. This is a small board with a USB-A tongue (but not a proper plug), and an ATTiny85 microcontroller, This chip has 8 pins and is quite small really, but it was plenty for my application. By choosing something popular, I hoped for convenient hardware, and an uncomplicated experience. Convenient hardware, I got. Arduino IDE The usual way to program these boards is via an IDE. I thought I d go with the flow and try that. I knew these were closely related to actual Arduinos and saw that the IDE package arduino was in Debian. But it turns out that the Debian package s version doesn t support the DigiSpark. (AFAICT from the list it offered me, I m not sure it supports any ATTiny85 board.) Also, disturbingly, its board manager seemed to be offering to install board support, suggesting it would download stuff from the internet and run it. That wouldn t be acceptable for my main laptop. I didn t expect to be doing much programming or debugging, and the project didn t have significant security requirements: the chip, in my circuit, has only a very narrow ability do anything to the real world, and no network connection of any kind. So I thought it would be tolerable to do the project on my low-security video laptop . That s the machine where I m prepared to say yes to installing random software off the internet. So I went to the upstream Arduino site and downloaded a tarball containing the Arduino IDE. After unpacking that in /opt it ran and produced a pointy-clicky IDE, as expected. I had already found a 3rd-party tutorial saying I needed to add a magic URL (from the DigiSpark s vendor) in the preferences. That indeed allowed it to download a whole pile of stuff. Compilers, bootloader clients, god knows what. However, my tiny test program didn t make it to the board. Half-buried in a too-small window was an error message about the board s bootloader ( Micronucleus ) being too new. The boards I had came pre-flashed with micronucleus 2.2. Which is hardly new, But even so the official Arduino IDE (or maybe the DigiSpark s board package?) still contains an old version. So now we have all the downsides of curl bash-ware, but we re lacking the it s up to date and it just works upsides. Further digging found some random forum posts which suggested simply downloading a newer micronucleus and manually stuffing it into the right place: one overwrites a specific file, in the middle the heaps of stuff that the Arduino IDE s board support downloader squirrels away in your home directory. (In my case, the home directory of the untrusted shared user on the video laptop,) So, whatever . I did that. And it worked! Having demo d my ability to run code on the board, I set about writing my program. Writing C again The programming language offered via the Arduino IDE is C. It s been a little while since I started a new thing in C. After having spent so much of the last several years writing Rust. C s primitiveness quickly started to grate, and the program couldn t easily be as DRY as I wanted (Don t Repeat Yourself, see Wilson et al, 2012, 4, p.6). But, I carried on; after all, this was going to be quite a small job. Soon enough I had a program that looked right and compiled. Before testing it in circuit, I wanted to do some QA. So I wrote a simulator harness that #included my Arduino source file, and provided imitations of the few Arduino library calls my program used. As an side advantage, I could build and run the simulation on my main machine, in my normal development environment (Emacs, make, etc.). The simulator runs confirmed the correct behaviour. (Perhaps there would have been some more faithful simulation tool, but the Arduino IDE didn t seem to offer it, and I wasn t inclined to go further down that kind of path.) So I got the video laptop out, and used the Arduino IDE to flash the program. It didn t run properly. It hung almost immediately. Some very ad-hoc debugging via led-blinking (like printf debugging, only much worse) convinced me that my problem was as follows: Arduino C has 16-bit ints. My test harness was on my 64-bit Linux machine. C was autoconverting things (when building for the micrcocontroller). The way the Arduino IDE ran the compiler didn t pass the warning options necessary to spot narrowing implicit conversions. Those warnings aren t the default in C in general because C compilers hate us all for compatibility reasons. I don t know why those warnings are not the default in the Arduino IDE, but my guess is that they didn t want to bother poor novice programmers with messages from the compiler explaining how their program is quite possibly wrong. After all, users don t like error messages so we shouldn t report errors. And novice programmers are especially fazed by error messages so it s better to just let them struggle themselves with the arcane mysteries of undefined behaviour in C? The Arduino IDE does offer a dropdown for compiler warnings . The default is None. Setting it to All didn t produce anything about my integer overflow bugs. And, the output was very hard to find anyway because the log window has a constant stream of strange messages from javax.jmdns, with hex DNS packet dumps. WTF. Other things that were vexing about the Arduino IDE: it has fairly fixed notions (which don t seem to be documented) about how your files and directories ought to be laid out, and magical machinery for finding things you put nearby its sketch (as it calls them) and sticking them in its ear, causing lossage. It has a tendency to become confused if you edit files under its feet (e.g. with git checkout). It wasn t really very suited to a workflow where principal development occurs elsewhere. And, important settings such as the project s clock speed, or even the target board, or the compiler warning settings to use weren t stored in the project directory along with the actual code. I didn t look too hard, but I presume they must be in a dotfile somewhere. This is madness. Apparently there is an Arduino CLI too. But I was already quite exasperated, and I didn t like the idea of going so far off the beaten path, when the whole point of using all this was to stay with popular tooling and share fate with others. (How do these others cope? I have no idea.) As for the integer overflow bug: I didn t seriously consider trying to figure out how to control in detail the C compiler options passed by the Arduino IDE. (Perhaps this is possible, but not really documented?) I did consider trying to run a cross-compiler myself from the command line, with appropriate warning options, but that would have involved providing (or stubbing, again) the Arduino/DigiSpark libraries (and bugs could easily lurk at that interface). Instead, I thought, if only I had written the thing in Rust . But that wasn t possible, was it? Does Rust even support this board? Rust on the DigiSpark I did a cursory web search and found a very useful blog post by Dylan Garrett. This encouraged me to think it might be a workable strategy. I looked at the instructions there. It seemed like I could run them via the privsep arrangement I use to protect myself when developing using upstream cargo packages from crates.io. I got surprisingly far surprisingly quickly. It did, rather startlingly, cause my rustup to download a random recent Nightly Rust, but I have six of those already for other Reasons. Very quickly I got the trinket LED blink example, referenced by Dylan s blog post, to compile. Manually copying the file to the video laptop allowed me to run the previously-downloaded micronucleus executable and successfully run the blink example on my board! I thought a more principled approach to the bootloader client might allow a more convenient workflow. I found the upstream Micronucleus git releases and tags, and had a look over its source code, release dates, etc. It seemed plausible, so I compiled v2.6 from source. That was a success: now I could build and install a Rust program onto my board, from the command line, on my main machine. No more pratting about with the video laptop. I had got further, more quickly, with Rust, than with the Arduino IDE, and the outcome and workflow was superior. So, basking in my success, I copied the directory containing the example into my own project, renamed it, and adjusted the path references. That didn t work. Now it didn t build. Even after I copied about .cargo/config.toml and rust-toolchain.toml it didn t build, producing a variety of exciting messages, depending what precisely I tried. I don t have detailed logs of my flailing: the instructions say to build it by cd ing to the subdirectory, and, given that what I was trying to do was to not follow those instructions, it didn t seem sensible to try to prepare a proper repro so I could file a ticket. I wasn t optimistic about investigating it more deeply myself: I have some experience of fighting cargo, and it s not usually fun. Looking at some of the build control files, things seemed quite complicated. Additionally, not all of the crates are on crates.io. I have no idea why not. So, I would need to supply local copies of them anyway. I decided to just git subtree add the avr-hal git tree. (That seemed better than the approach taken by the avr-hal project s cargo template, since that template involve a cargo dependency on a foreign git repository. Perhaps it would be possible to turn them into path dependencies, but given that I had evidence of file-location-sensitive behaviour, which I didn t feel like I wanted to spend time investigating, using that seems like it would possibly have invited more trouble. Also, I don t like package templates very much. They re a form of clone-and-hack: you end up stuck with whatever bugs or oddities exist in the version of the template which was current when you started.) Since I couldn t get things to build outside avr-hal, I edited the example, within avr-hal, to refer to my (one) program.rs file outside avr-hal, with a #[path] instruction. That s not pretty, but it worked. I also had to write a nasty shell script to work around the lack of good support in my nailing-cargo privsep tool for builds where cargo must be invoked in a deep subdirectory, and/or Cargo.lock isn t where it expects, and/or the target directory containing build products is in a weird place. It also has to filter the output from cargo to adjust the pathnames in the error messages. Otherwise, running both cd A; cargo build and cd B; cargo build from a Makefile produces confusing sets of error messages, some of which contain filenames relative to A and some relative to B, making it impossible for my Emacs to reliably find the right file. RIIR (Rewrite It In Rust) Having got my build tooling sorted out I could go back to my actual program. I translated the main program, and the simulator, from C to Rust, more or less line-by-line. I made the Rust version of the simulator produce the same output format as the C one. That let me check that the two programs had the same (simulated) behaviour. Which they did (after fixing a few glitches in the simulator log formatting). Emboldened, I flashed the Rust version of my program to the DigiSpark. It worked right away! RIIR had caused the bug to vanish. Of course, to rewrite the program in Rust, and get it to compile, it was necessary to be careful about the types of all the various integers, so that s not so surprising. Indeed, it was the point. I was then able to refactor the program to be a bit more natural and DRY, and improve some internal interfaces. Rust s greater power, compared to C, made those cleanups easier, so making them worthwhile. However, when doing real-world testing I found a weird problem: my timings were off. Measured, the real program was too fast by a factor of slightly more than 2. A bit of searching (and searching my memory) revealed the cause: I was using a board template for an Adafruit Trinket. The Trinket has a clock speed of 8MHz. But the DigiSpark runs at 16.5MHz. (This is discussed in a ticket against one of the C/C++ libraries supporting the ATTiny85 chip.) The Arduino IDE had offered me a choice of clock speeds. I have no idea how that dropdown menu took effect; I suspect it was adding prelude code to adjust the clock prescaler. But my attempts to mess with the CPU clock prescaler register by hand at the start of my Rust program didn t bear fruit. So instead, I adopted a bodge: since my code has (for code structure reasons, amongst others) only one place where it dealt with the underlying hardware s notion of time, I simply changed my delay function to adjust the passed-in delay values, compensating for the wrong clock speed. There was probably a more principled way. For example I could have (re)based my work on either of the two unmerged open MRs which added proper support for the DigiSpark board, rather than abusing the Adafruit Trinket definition. But, having a nearly-working setup, and an explanation for the behaviour, I preferred the narrower fix to reopening any cans of worms. An offer of help As will be obvious from this posting, I m not an expert in dev tools for embedded systems. Far from it. This area seems like quite a deep swamp, and I m probably not the person to help drain it. (Frankly, much of the improvement work ought to be done, and paid for, by hardware vendors.) But, as a full Member of the Debian Project, I have considerable gatekeeping authority there. I also have much experience of software packaging, build systems, and release management. If anyone wants to try to improve the situation with embedded tooling in Debian, and is willing to do the actual packaging work. I would be happy to advise, and to review and sponsor your contributions. An obvious candidate: it seems to me that micronucleus could easily be in Debian. Possibly a DigiSpark board definition could be provided to go with the arduino package. Unfortunately, IMO Debian s Rust packaging tooling and workflows are very poor, and the first of my suggestions for improvement wasn t well received. So if you need help with improving Rust packages in Debian, please talk to the Debian Rust Team yourself. Conclusions Embedded programming is still rather a mess and probably always will be. Embedded build systems can be bizarre. Documentation is scant. You re often expected to download board support packages full of mystery binaries, from the board vendor (or others). Dev tooling is maddening, especially if aimed at novice programmers. You want version control? Hermetic tracking of your project s build and install configuration? Actually to be told by the compiler when you write obvious bugs? You re way off the beaten track. As ever, Free Software is under-resourced and the maintainers are often busy, or (reasonably) have other things to do with their lives. All is not lost Rust can be a significantly better bet than C for embedded software: The Rust compiler will catch a good proportion of programming errors, and an experienced Rust programmer can arrange (by suitable internal architecture) to catch nearly all of them. When writing for a chip in the middle of some circuit, where debugging involves staring an LED or a multimeter, that s precisely what you want. Rust embedded dev tooling was, in this case, considerably better. Still quite chaotic and strange, and less mature, perhaps. But: significantly fewer mystery downloads, and significantly less crazy deviations from the language s normal build system. Overall, less bad software supply chain integrity. The ATTiny85 chip, and the DigiSpark board, served my hardware needs very well. (More about the hardware aspects of this project in a future posting.)

comment count unavailable comments

21 October 2023

Dirk Eddelbuettel: qlcal 0.0.8 on CRAN: QuantLib 1.32 Updates

The eighth release of the still fairly new qlcal package arrivied at CRAN today. qlcal delivers the calendaring parts of QuantLib. It is provided (for the R package) as a set of included files, so the package is self-contained and does not depend on an external QuantLib library (which can be demanding to build). qlcal covers over sixty country / market calendars and can compute holiday lists, its complement (i.e. business day lists) and much more. This release brings updates from the just-released QuantLib 1.32 version. It also avoids a nag from R during build ( only specify C++14 if you really need it ) but switching to a versioned depends on R 4.2.0 or later. This implies C++14 or later as the default. If you need qlcal on an older R, grab the sources, edit DESCRIPTION to remove this constraint and set the standard as before in src/Makevars (or src/Makevars.win).

Changes in version 0.0.8 (2023-10-21)
  • A small set of updates from QuantLib 1.32 have been applied
  • The explicit C++14 compilation standard has been replaced with an implicit one by relying on R (>= 4.2.0)

Courtesy of my CRANberries, there is a diffstat report for this release. See the project page and package documentation for more details, and more examples. If you like this or other open-source work I do, you can now sponsor me at GitHub.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Russell Coker: More About the PineTime

Since my initial review of the PineTime 10 days ago [1] I ve used it in more situations. My initial tests were done connecting to a Huawei Nova 7i [2], I am now using it with a Huawei Mate 10 Pro. I ve also upgraded the PineTime from version 1.11 (from memory) of the Infinitime software that runs on the watch to version 1.13 [3]. To upgrade it I had to download the file pinetime-mcuboot-app-dfu-1.13.0.zip to the Android phone and then use the File Installer option of the GadgetBridge Android app to upload it. The zip file does NOT need to be extracted first, I don t know if GadgetBridge extracts it before upload or if the PineTime firmware has a copy of unzip, but it just works. Version 1.13 is purported to take less battery, I haven t directly verified this as I turned on the new feature of measuring my pulse 24*7 which significantly increases battery use. The end result is that the battery is being used up at about the same rate as before, overall adding a new battery-hungry feature while reducing battery use for other things to compensate is a good thing and strongly suggests that battery use has decreased overall. I have noticed that now with a different phone and different version of the firmware it doesn t reconnect as reliably. Sometimes I need to turn bluetooth on the watch off and on before it works (which indicates an issue with the firmware) and sometimes I need to turn bluetooth off and on on the phone which indicates a phone issue. Also I often unlock my phone to find the GadgetBridge notification saying that it s disconnected and it usually connects fine, but I get the impression it s often disconnected. Does the Mate 10 Pro have a problem that triggers a bug in the PineTime? Does the 1.13 version of InfiniTime have a problem that triggers a bug in the Mate 10 Pro? Are they both independently buggy? Is the new version of InfiniTime just disconnecting when it s not doing stuff to save battery and triggering bugs that weren t obvious before? I ve tested the media control which basically works, sometimes it gets out of sync and displays the name of the previous track which is annoying. The PineTime is IP67 rated and there are reports on Reddit of people wearing it in the shower and swimming pool. I wouldn t recommend those things although it should work OK. It might be an option for controlling music when in the bath or when having a pool party. When the watch is running normally and displays a new notification it s not possible to swipe it away. You have to go to the notifications menu afterwards to swipe them which I find annoying. Also the notification of an inbound call remains in the notification list indefinitely while I think a more appropriate action is to have it disappear in an amount of time where it s already been answered or gone to voicemail. Voicemail timeouts are as low as 15 seconds so having the notification disappear after 1 minute would be reasonable. I have configured my PineTime to take 2 taps on the screen to wake up. I previously had it set to 1 tap and had problems with accidentally doing something it registered as a tap while in bed and waking me up. Also I found that if I want to turn the screen on when my hands are dirty so I don t want to touch it with a finger then tapping it on my nose works well. Apparently it is programmed to ignore taps on large areas so I can t wake it with my elbow. I ve setup a PineTime for an elderly relative who is greatly enjoying it. I don t expect them to flash new firmware or do any other complex things, but they are doing well with using the device. They are considering getting a different band as they don t like rubber. I m sure their local jeweler has some leather and metal bands that could fit. There is a design on Thiniverse for a PineTime case [4], this could be used for making an adaptor to fit a PineTime to a greatly different type of band, an instrument console, etc. Generally I think the PineTime is an OK smart watch for someone who s not into FOSS for it s own sake. My relative could have been happy with a slightly cheaper watch, but it s still significantly cheaper than the Samsung and Apple options so it s not particularly expensive. A benefit for them is that having the same type of SmartWatch as me they will get better tech support.
Related posts:
  1. The PineTime I have just got a PineTime smart watch [1] from...
  2. PinePhone Status 4 months ago I got my PinePhonePro [1]. Since then...
  3. Long-term Device Use It seems to me that Android phones have recently passed...

20 October 2023

Iustin Pop: How to set a per-app locale in MacOS

After spending ~20+ years with a Linux desktop, I m trying to expand my desktop setup to include MacOS (well, desktop/laptop, I mean end user in general). And to my surprise, there s no clear repository of MacOS info. Man pages yes, some StackOverflow, some Apple forums, but no canonical version. Or, I didn t find it, please enlighten me Another issue is that Apple apparently changes behaviour without clearly documenting it. In this specific case, the region part of the locale went through significant churn lately. So, my goal: In Linux, this would simply mean running the app with the correct environment variables. But MacOS deprecated this a while back (it used to work). After reading what I could, the solution is quite easy, just not obvious: 
% defaults read .GlobalPreferences grep en_
    AKLastLocale = "en_CH";
    AppleLocale = "en_CH";
% defaults read -app FooBar
(has no AppleLocale key)
% defaults write -app FooBar AppleLocale en_US
And that s it. Now, the defaults man page says the global-global is NSGlobalDomain, I don t know where I got the .GlobalPreferences. But I only needed to know the key name (in this case, AppleLocale - of course it couldn t be LC_ALL/LANG). One day I ll know MacOS better, but I try to learn more for 2+ years now, and it s not a smooth ride. Old dog new tricks, right?

Freexian Collaborators: Debian Contributions: Freexian meetup, debusine updates, lpr/lpd in Debian, and more! (by Utkarsh Gupta, Stefano Rivera)

Contributing to Debian is part of Freexian s mission. This article covers the latest achievements of Freexian and their collaborators. All of this is made possible by organizations subscribing to our Long Term Support contracts and consulting services.

Freexian Meetup, by Stefano Rivera, Utkarsh Gupta, et al. During DebConf, Freexian organized a meetup for its collaborators and those interested in learning more about Freexian and its services. It was well received and many people interested in Freexian showed up. Some developers who were interested in contributing to LTS came to get more details about joining the project. And some prospective customers came to get to know us and ask questions. Sadly, the tragic loss of Abraham shook DebConf, both individually and structurally. The meetup got rescheduled to a small room without video coverage. With that, we still had a wholesome interaction and here s a quick picture from the meetup taken by Utkarsh (which is also why he s missing!).

Debusine, by Rapha l Hertzog, et al. Freexian has been investing into debusine for a while, but development speed is about to increase dramatically thanks to funding from SovereignTechFund.de. Rapha l laid out the 5 milestones of the funding contract, and filed the issues for the first milestone. Together with Enrico and Stefano, they established a workflow for the expanded team. Among the first steps of this milestone, Enrico started to work on a developer-friendly description of debusine that we can use when we reach out to the many Debian contributors that we will have to interact with. And Rapha l started the design work of the autopkgtest and lintian tasks, i.e. what s the interface to schedule such tasks, what behavior and what associated options do we support? At this point you might wonder what debusine is supposed to be let us try to answer this: Debusine manages scheduling and distribution of Debian-related build and QA tasks to a network of worker machines. It also manages the resulting artifacts and provides the results in an easy to consume way. We want to make it easy for Debian contributors to leverage all the great QA tools that Debian provides. We want to build the next generation of Debian s build infrastructure, one that will continue to reliably do what it already does, but that will also enable distribution-wide experiments, custom package repositories and custom workflows with advanced package reviews. If this all sounds interesting to you, don t hesitate to watch the project on salsa.debian.org and to contribute.

lpr/lpd in Debian, by Thorsten Alteholz During Debconf23, Till Kamppeter presented CPDB (Common Print Dialog Backend), a new way to handle print queues. After this talk it was discussed whether the old lpr/lpd based printing system could be abandoned in Debian or whether there is still demand for it. So Thorsten asked on the debian-devel email list whether anybody uses it. Oddly enough, these old packages are still useful:
  • Within a small network it is easier to distribute a printcap file, than to properly configure cups clients.
  • One of the biggest manufacturers of WLAN router and DSL boxes only supports raw queues when attaching an USB printer to their hardware. Admittedly the CPDB still has problems with such raw queues.
  • The Pharos printing system at MIT is still lpd-based.
As a result, the lpr/lpd stuff is not yet ready to be abandoned and Thorsten will adopt the relevant packages (or rather move them under the umbrella of the debian-printing team). Though it is not planned to develop new features, those packages should at least have a maintainer. This month Thorsten adopted rlpr, an utility for lpd printing without using /etc/printcap. The next one he is working on is lprng, a lpr/lpd printer spooling system. If you know of any other package that is also needed and still maintained by the QA team, please tell Thorsten.

/usr-merge, by Helmut Grohne Discussion about lifting the file move moratorium has been initiated with the CTTE and the release team. A formal lift is dependent on updating debootstrap in older suites though. A significant number of packages can automatically move their systemd unit files if dh_installsystemd and systemd.pc change their installation targets. Unfortunately, doing so makes some packages FTBFS and therefore patches have been filed. The analysis tool, dumat, has been enhanced to better understand which upgrade scenarios are considered supported to reduce false positive bug filings and gained a mode for local operation on a .changes file meant for inclusion in salsa-ci. The filing of bugs from dumat is still manual to improve the quality of reports. Since September, the moratorium has been lifted.

Miscellaneous contributions
  • Rapha l updated Django s backport in bullseye-backports to match the latest security release that was published in bookworm. Tracker.debian.org is still using that backport.
  • Helmut Grohne sent 13 patches for cross build failures.
  • Helmut Grohne performed a maintenance upload of debvm enabling its use in autopkgtests.
  • Helmut Grohne wrote an API-compatible reimplementation of autopkgtest-build-qemu. It is powered by mmdebstrap, therefore unprivileged, EFI-only and will soon be included in mmdebstrap.
  • Santiago continued the work regarding how to make it easier to (automatically) test reverse dependencies. An example of the ongoing work was presented during the Salsa CI BoF at DebConf 23.
    In fact, omniorb-dfsg test pipelines as the above were used for the omniorb-dfsg 4.3.0 transition, verifying how the reverse dependencies (tango, pytango and omnievents) were built and how their autopkgtest jobs run with the to-be-uploaded omniorb-dfsg new release.
  • Utkarsh and Stefano attended and helped run DebConf 23. Also continued winding up DebConf 22 accounting.
  • Anton Gladky did some science team uploads to fix RC bugs.

17 October 2023

Russ Allbery: Review: A Hat Full of Sky

Review: A Hat Full of Sky, by Terry Pratchett
Series: Discworld #32
Publisher: HarperTrophy
Copyright: 2004
Printing: 2005
ISBN: 0-06-058662-1
Format: Mass market
Pages: 407
A Hat Full of Sky is the 32nd Discworld novel and the second Tiffany Aching young adult novel. You should not start here, but you could start with The Wee Free Men. As with that book, some parts of the story carry more weight if you are already familiar with Granny Weatherwax. Tiffany is a witch, but she needs to be trained. This is normally done by apprenticeship, and in Tiffany's case it seemed wise to give her exposure to more types of witching. Thus, Tiffany, complete with new boots and a going-away present from the still-somewhat-annoying Roland, is off on an apprenticeship to the sensible Miss Level. (The new boots feel wrong and get swapped out for her concealed old boots at the first opportunity.) Unbeknownst to Tiffany, her precocious experiments with leaving her body as a convenient substitute for a mirror have attracted something very bad, something none of the witches are expecting. The Nac Mac Feegle know a hiver as soon as they feel it, but they have a new kelda now, and she's not sure she wants them racing off after their old kelda. Terry Pratchett is very good at a lot of things, but I don't think villains are one of his strengths. He manages an occasional memorable one (the Auditors, for example, at least before the whole chocolate thing), but I find most of them a bit boring. The hiver is one of the boring ones. It serves mostly as a concretized metaphor about the temptations of magical power, but those temptations felt so unlike the tendencies of Tiffany's personality that I didn't think the metaphor worked in the story. The interesting heart of this book to me is the conflict between Tiffany's impatience with nonsense and Miss Level's arguably excessive willingness to help everyone regardless of how demanding they get. There's something deeper in here about female socialization and how that interacts with Pratchett's conception of witches that got me thinking, although I don't think Pratchett landed the point with full force. Miss Level is clearly a good witch to her village and seems comfortable with how she lives her life, so perhaps they're not taking advantage of her, but she thoroughly slots herself into the helper role. If Tiffany attempted the same role, people would be taking advantage of her, because the role doesn't fit her. And yet, there's a lesson here she needs to learn about seeing other people as people, even if it wouldn't be healthy for her to move all the way to Miss Level's mindset. Tiffany is a precocious kid who is used to being underestimated, and who has reacted by becoming independent and somewhat judgmental. She's also had a taste of real magical power, which creates a risk of her getting too far into her own head. Miss Level is a fount of empathy and understanding for the normal people around her, which Tiffany resists and needed to learn. I think Granny Weatherwax is too much like Tiffany to teach her that. She also has no patience for fools, but she's older and wiser and knows Tiffany needs a push in that direction. Miss Level isn't a destination, but more of a counterbalance. That emotional journey, a conclusion that again focuses on the role of witches in questions of life and death, and Tiffany's fascinatingly spiky mutual respect with Granny Weatherwax were the best parts of this book for me. The middle section with the hiver was rather tedious and forgettable, and the Nac Mac Feegle were entertaining but not more than that. It felt like the story went in a few different directions and only some of them worked, in part because the villain intended to tie those pieces together was more of a force of nature than a piece of Tiffany's emotional puzzle. If the hiver had resonated with the darker parts of Tiffany's natural personality, the plot would have worked better. Pratchett was gesturing in that direction, but he never convinced me it was consistent with what we'd already seen of her. Like a lot of the Discworld novels, the good moments in A Hat Full of Sky are astonishing, but the plot is somewhat forgettable. It's still solidly entertaining, though, and if you enjoyed The Wee Free Men, I think this is slightly better. Followed by Going Postal in publication order. The next Tiffany Aching novel is Wintersmith. Rating: 8 out of 10

15 October 2023

Aigars Mahinovs: Figuring out finances part 2

A week ago I started to migrate my financial planning from a closed source system to a new system based on an open source, self-hosted solution. Main candidate is Firefly III - a relatively simple financial planner with a rather rich feature set and a solid user base and developer support. Starting it up with a Docker-Compose file was quite easy, following the official documentation. The same Compose file also managed the MySQL database, the importer app and a cron container for regular imports. The separate importer app allows both imports from CSV files and also from external bank account connection services. For whatever weird reason both of those services support exporting data from my regular bank accounts, but not from my credit cards for exactly the same bank. So for those cards I would need to periodically download CVS transaction exports and feed them into the importer. The combination of the cron container and the importer app allows for both of these functions. To do this you first do the import via the Web UI first and configure all the mappings - configure file and date formats, map account names in the bank output to account names that you added in Firefly, configure what otehr columns mean and what is the mapping of other values, like expense and income accounts. At the end of the process you can download a json file representing all the settings of the import you just did. Putting such a json config file in the input folder of the cron container and telling it to do an import (weekly, for example) would do such import periodically. Putting a credit card CSV file along with a config file for credit card import would also auto-import that. So far so good. However, when I tried importing exports from MoneyWiz or even when I tried re-creating account history directly from the bank data, I hit a very annoying problem. Transfers. Thing is detecting transfers between the real (asset) accounts that you are managing is really essential. For one those are not real expenses and incomes, so failing to mark them as transfers would show weird income/expense numbers. But if you do detect them as transfers and correctly map the destination account, but fail to match the transations between another you get a double-transaction. This becomes really hard when transactions happen on different dates and have different descriptions. So you get both a +1000 transaction "Credit card reset" on 14.09.2023 in the credit card account and a -1000$ transaction "Repayment of own credit card" on 24.09.2023 in the main account of the bank. Matching them to recognise that it is just one transfer and not two is ... non-trivial. The best solution I could come up with so far is to always map the "Opposing account name" for such transfers to a virtual "Transfers" asset account. That has the benefit of being actually able to correctly represent transfers that take several days to move between accounts and showing you how much money is still in transit at any point in time. So after I figured this out, finally the account balances started matching up with reality. Setting up spending categories required another change - the author of the Firefly III does not like complexity so it does not support nesting categories. Categories can only be in a single, flat list. It is suggested that if you do need to track multiple categories and also their combination, then category groups might help you. I will survive for now by simplifying the categories that I do actually use. Might actually make the reports more usable. A new feature for me will be the ability to use automation rules to assign transactions to categories based on their contents, like expense accounts of keywords in descriptions. Setting up regular bills (with matching rules to assign incoming transactions to those bills) is another feature that is very important to me. The feature itself works just fine in Firefly III, but it has two restrictions that the author of the software does not actually want to be changed. For one it can not be used to track prodictable incomes (like salary), presumably because it is only there for bills (subscriptions in v3 UI). And for other, there is nothing in the base software that actually uses the data from bills to look forward. The author does not like to try to predict the future. Which for me is basically the one of two reasons to use this kind of software at all. I want to know - given all manually entered regular and 100% predictable expenses and incomes, what will be the state of my accounts at particular date? It seems that to get at that I will need to write my own oracle script using the rich Firefly III API. The last question I had for this week was - how will I enter a new cash puchace of potatoes on a farmers market into the system that sits in a private server in my home network? Turns out there are actually multiple Android apps for Firefly III that can be easily used for this using a robust OAuth or shared token authentification. Except the web UI needs to be exposed online for this to work (and ideally protected by HTTPS). Other users have also created Telegram bots that allow using chat messages to create transaction entries. This is a bit harder to use and more narrow scope, but should be easier to setup. Will have to try both apporaches. But before I really get going on this, I need to fix another thing that I have been postponing for a while: I will need to migrate my Home Assistant installation from a Docker container installation to a Home Assistant OS installation so that I can install Addons, including Firefly III, MySQL and Portainer to have a bit more organised and less hand-knitted home server setup. Let's see how that goes next week :D

Russ Allbery: Review: A Killing Frost

Review: A Killing Frost, by Seanan McGuire
Series: October Daye #14
Publisher: DAW
Copyright: 2020
ISBN: 0-7564-1253-6
Format: Kindle
Pages: 351
A Killing Frost is the 14th book in the October Daye urban fantasy series and a direct plot sequel to the events of The Brightest Fell. You definitely cannot start here. This review has some relationship spoilers here for things that you would be expecting after the first five or six books, but which you wouldn't know when reading the first few books of the series. If you haven't started the series yet but plan to, consider skipping this review; if you haven't started reading this series, it will probably be meaningless anyway. Finally, events seem to have slowed, enough trauma has been healed, and Toby is able to seriously consider getting married. However, no sooner is the thought voiced than fae politics injects itself yet again. In order to get married without creating potentially substantial future problems for herself and her family, Toby will have to tie up some loose ends. Since one of those loose ends is a price from the Luidaeg that has been haunting her family for decades, this is easier said than done. The Brightest Fell had a very unsatisfying ending. This, after a two book interlude, is the proper end to that story. I picked this up when I had a bunch of stressful things going on and I wanted to be entertained without having to do much work as a reader. Once again, this series delivered exactly that. The writing is repetitive and a bit clunky, McGuire hammers the same emotional points into the ground, and one does wonder about Toby's tendency to emulate a half-human battering ram, but every book has me engrossed and turning the pages. Everyone should have at least one book series on the go that offers reliable, low-effort entertainment. The initial lever that McGuire uses to push Toby into this plot (fae marriage requirements that had never previously been mentioned) felt rather strained and arbitrary, and I spent the first part of the book grumbling a bit about it. However, there is a better reason for this complication that is revealed with time, and which implies some interesting things about how the fae see heroes and how they use them to solve problems. Now I'm wondering if McGuire will explore that some more in later books. This is the "all is revealed" book about Simon Torquill. As we get later into the series, these "all is revealed" books are coming more frequently. So far, I'm finding the revelations satisfying, which is a lot harder than it looks with a series this long and with this many hidden details. There are a few directions the series is taking that aren't my favorite (the Daoine Sidhe obsession with being the Best Fae is getting a bit boring, for example), but none of them seem egregiously off, and I'm deeply invested in the answers to the remaining questions. Toby hits a personal record here for not explaining the dangerous things she's doing because people might talk her out of it. It makes for a tense and gripping climax, but wow I felt for her friends and family, and substantial parts of that risk seemed unnecessary. This is pointed out to her in no uncertain terms, and I'm wondering if it will finally stick. Toby's tendency to solve complicated problems by bleeding on them is part of what gives this series its charm, but I wouldn't mind her giving other people more of a chance to come up with better plans. I did not like this one as well as the previous two books, mostly because I prefer the Luidaeg-centric stories to the Daoine-Sidhe-centric stories, but if you're enjoying the series to this point, this won't be an exception. It's a substantial improvement on The Brightest Fell and did a lot to salvage that story for me, although there are still some aspects of it that need better explanations. Followed by When Sorrows Come. As usual, there is a novella included in at least the Kindle edition. "Shine in Pearl": I was again hoping for more Gillian, but alas. Instead, and breaking with the tendency for the novellas to be side stories unrelated to the main novel, this fleshes out Simon's past and the other primary relationship driving the novel's plot. It's... fine? The best parts by far are the scenes from Dianda's viewpoint, which are just as refreshingly blunt as Dianda is elsewhere. Neither of the other two characters are favorites of mine, and since the point of the story is to describe the tragedy that is resolved in the plot of the main novel, it's somewhat depressing. Not my favorite of the novellas; not the worst of them. (6) Rating: 7 out of 10

14 October 2023

Ravi Dwivedi: Kochi - Wayanad Trip in August-September 2023

A trip full of hitchhiking, beautiful places and welcoming locals.

Day 1: Arrival in Kochi Kochi is a city in the state of Kerala, India. This year s DebConf was to be held in Kochi from 3rd September to 17th of September, which I was planning to attend. My friend Suresh, who was planning to join, told me that 29th August 2023 will be Onam, a major festival of the state of Kerala. So, we planned a Kerala trip before the DebConf. We booked early morning flights for Kochi from Delhi and reached Kochi on 28th August. We had booked a hostel named Zostel in Ernakulam. During check-in, they asked me to fill a form which required signing in using a Google account. I told them I don t have a Google account and I don t want to create one either. The people at the front desk seemed receptive, so I went ahead with telling them the problems of such a sign-in being mandatory for check-in. Anyways, they only took a photo of my passport and let me check-in without a Google account. We stayed in a ten room dormitory, which allowed travellers of any gender. The dormitory room was air-conditioned, spacious, clean and beds were also comfortable. There were two bathrooms in the dormitory and they were clean. Plus, there was a separate dormitory room in the hostel exclusive for females. I noticed that that Zostel was not added in the OpenStreetMap and so, I added it :) . The hostel had a small canteen for tea and snacks, a common sitting area outside the dormitories, which had beds too. There was a separate silent room, suitable for people who want to work.
Dormitory room in Zostel Ernakulam, Kochi.
Beds in Zostel Ernakulam, Kochi.
We had lunch at a nearby restaurant and it was hard to find anything vegetarian for me. I bought some freshly made banana chips from the street and they were tasty. As far as I remember, I had a big glass of pineapple juice for lunch. Then I went to the Broadway market and bought some cardamom and cinnamon for home. I also went to a nearby supermarket and bought Matta brown rice for home. Then, I looked for a courier shop to send the things home but all of them were closed due to Onam festival. After returning to the Zostel, I overslept till 9 PM and in the meanwhile, Suresh planned with Saidut and Shwetank (who met us during our stay in Zostel) to go to a place in Fort Kochi for dinner. I suspected I will be disappointed by lack of vegetarian options as they were planning to have fish. I already had a restaurant in mind - Brindhavan restaurant (suggested by Anupa), which was a pure vegetarian restaurant. To reach there, I got off at Palarivattom metro station and started looking for an auto-rickshaw to get to the restaurant. I didn t get any for more than 5 minutes. Since that restaurant was not added to the OpenStreetMap, I didn t even know how far that was and which direction to go to. Then, I saw a Zomato delivery person on a motorcycle and asked him where the restaurant was. It was already 10 PM and the restaurant closes at 10:30. So, I asked him whether he can drop me off. He agreed and dropped me off at that restaurant. It was 4-5 km from that metro station. I tipped him and expressed my gratefulness for the help. He refused to take the tip, but I insisted and he accepted. I entered the restaurant and it was coming to a close, so many items were not available. I ordered some Kadhai Paneer (only item left) with naan. It tasted fine. Since the next day was Thiruvonam, I asked the restaurant about the Sadya thali menu and prices for the next day. I planned to eat Sadya thali at that restaurant, but my plans got changed later.
Onam sadya menu from Brindhavan restaurant.

Day 2: Onam celebrations Next day, on 29th of August 2023, we had plan to leave for Wayanad. Wayanad is a hill station in Kerala and a famous tourist spot. Praveen suggested to visit Munnar as it is far closer to Kochi than Wayanad (80 km vs 250 km). But I had already visited Munnar in my previous trips, so we chose Wayanad. We had a train late night from Ernakulam Junction (at 23:30 hours) to Kozhikode, which is the nearest railway station from Wayanad. So, we checked out in the morning as we had plans to roam around in Kochi before taking the train. Zostel was celebrating Onam on that day. To opt-in, we had to pay 400 rupees, which included a Sadya Thali and a mundu. Me and Suresh paid the amount and opted in for the celebrations. Sadya thali had Rice, Sambhar, Rasam, Avial, Banana Chips, Pineapple Pachadi, Pappadam, many types of pickels and chutneys, Pal Ada Payasam and Coconut jaggery Pasam. And, there was water too :). Those payasams were really great and I had one more round of them. Later, I had a lot of variety of payasams during the DebConf.
Sadya lined up for serving
Sadya thali served on banana leaf.
So, we hung out in the common room and put our luggage there. We played UNO and had conversations with other travellers in the hostel. I had a fun time there and I still think it is one of the best hostel experiences I had. We made good friends with Saiduth (Telangana) and Shwetank (Uttarakhand). They were already aware about the software like debian, and we had some detailed conversations about the Free Software movement. I remember explaining the difference between the terms Open Source and Free Software . I also told them about the Streetcomplete app, a beginner friendly app to edit OpenStreetMap. We had dinner at a place nearby (named Palaraam), but again, the vegetarian options were very limited! After dinner, we came back to the Zostel and me and Suresh left for Ernakulam Junction to catch our train Maveli Express (16604).

Day 3: Going to Wayanad Maveli Express was scheduled to reach Kozhikode at 03:25 (morning). I had set alarms from 03:00 to 03:30, with the gap of 10 minutes. Every time I woke up, I turned off the alarm. Then I woke up and saw train reaching the Kozhikode station and woke up Suresh for deboarding. But then I noticed that the train is actually leaving the station, not arriving! This means we missed our stop. Now we looked at the next stops and whether we can deboard there. I was very sleepy and wanted to take a retiring room at some station before continuing our journey to Wayanad. The next stop was Quilandi and we checked online that it didn t have a retiring room. So, we skipped this stop. We got off at the next stop named Vadakara and found out no retiring room was available. So, we asked about information regarding bus for Wayanad and they said that there is a bus to Wayanad around 07:00 hours from bus station which was a few kilometres from the railway station. We took a bus for Kalpetta (in Wayanad) at around 07:00. The destination of the buses were written in Malayalam, which we could not read. Once again, the locals helped us to get on to the bus to Kalpetta. Vadakara is not a big city and it can be hard to find people who know good Hindi or English, unlike Kochi. Despite language issues, I had no problem there in navigation, thanks to locals. I mostly spent time sleeping during the bus journey. A few hours later, the bus dropped us at Kalpetta. We had a booking at a hostel in Rippon village. It was 16 km from Kalpetta. On the way, we were treated with beautiful views of nature, which was present everywhere in Wayanad. The place was covered with tea gardens and our eyes were treated with beautiful scenery at every corner.
We were treated with such views during the Wayanad trip.
Rippon village was a very quiet place and I liked the calm atmosphere. This place is blessed by nature and has stunning scenery. I found English was more common than Hindi in Wayanad. Locals were very nice and helped me, even if they didn t know my language.
A road in Rippon.
After catching some sleep at the hostel, I went out in the afternoon. I hitchhiked to reach the main road from the hostel. I bought more spices from a nearby shop and realized that I should have waited for my visit to Wayanad to buy cardamom, which I already bought from Kochi. Then, I was looking for post office to send spices home. The people at the spices shop told me that the nearby Rippon post office was closed by that time, but the post office at Meppadi was open, which was 5 km from there. I went to Meppadi and saw the post office closes at 15:00, but I reached five minutes late. My packing was not very good and they asked me to pack it tighter. There was a shop near the post office and the people there gave me a cardboard and tapes, and helped pack my stuff for the post. By the time I went to the post office again, it was 15:30. But they accepted my parcel for post.

Day 4: Kanthanpara Falls, Zostel Wayanad and Karapuzha Dam Kanthanpara waterfalls were 2 km from the hostel. I hitchhiked to the place from the hostel on a scooty. Entry ticket was worth Rs 40. There were good views inside and nothing much to see except the waterfalls.
Entry to Kanthanpara Falls.
Kanthanpara Falls.
We had a booking at Zostel Wayanad for this day and so we shifted there. Again, as with their Ernakulam branch, they asked me to fill a form which required signing in using Google, but when I said I don t have a Google account they checked me in without that. There were tea gardens inside the Zostel boundaries and the property was beautiful.
A view of Zostel Wayanad.
A map of Wayanad showing tourist places.
A view from inside the Zostel Wayanad property.
Later in the evening, I went to Karapuzha Dam. I witnessed a beautiful sunset during the journey. Karapuzha dam had many activites, like ziplining, and was nice to roam around. Chembra Peak is near to the Zostel Wayanad. So, I was planning to trek to the heart shaped lake. It was suggested by Praveen and looking online, this trek seemed worth doing. There was an issue however. The charges for trek were Rs 1770 for upto five people. So, if I go alone I will have to spend Rs 1770 for the trek. If I go with another person, we split Rs 1770 into two, and so on. The optimal way to do it is to go in a group of five (you included :D). I asked front desk at Zostel if they can connect me with people going to Chembra peak the next day, and they told me about a group of four people planning to go to Chembra peak the next day. I got lucky! All four of them were from Kerala and worked in Qatar.

Day 5: Chembra peak trek The date was 1st September 2023. I woke up early (05:30 in the morning) for the Chembra peak trek. I had bought hiking shoes especially for trekking, which turned out to be a very good idea. The ticket counter opens at 07:00. The group of four with which I planned to trek met me around 06:00 in the Zostel. We went to the ticket counter around 06:30. We had breakfast at shops selling Maggi noodles and bread omlette near the ticket counter. It was a hot day and the trek was difficult for an inexperienced person like me. The scenery was green and beautiful throughout.
Terrain during trekking towards the Chembra peak.
Heart-shaped lake at the Chembra peak.
Me at the heart-shaped lake.
Views from the top of the Chembra peak.
View of another peak from the heart-shaped lake.
While returning from the trek, I found out a shop selling bamboo rice, which I bought and will make bamboo rice payasam out of it at home (I have some coconut milk from Kerala too ;)). We returned to Zostel in the afternoon. I had muscle pain after the trek and it has still not completely disappeared. At night, we took a bus from Kalpetta to Kozhikode in order to return to Kochi.

Day 6: Return to Kochi At midnight of 2nd of September, we reached Kozhikode bus stand. Then we roamed around for something to eat. I didn t find anything vegetarian to eat. No surprises there! Then we went to Kozhikode railway station and looked for retiring rooms, but no luck there. We waited at the station and took the next train to Kochi at 03:30 and reached Ernakulam Junction at 07:30 (half hours before train s scheduled time!). From there, we went to Zostel Fort Kochi and stayed one night there and checked out next morning.

Day 7: Roaming around in Fort Kochi On 3rd of September, we roamed around in Fort Kochi. We visited the usual places - St Francis Church, Dutch Palace, Jew Town, Pardesi Synagogue. I also visited some homestays and the owners were very happy to show their place even when I made it clear that I was not looking for a stay. In the evening, we went to Kakkanad to attend DebConf. The story continues in my DebConf23 blog post.

12 October 2023

Reproducible Builds: Reproducible Builds in September 2023

Welcome to the September 2023 report from the Reproducible Builds project In these reports, we outline the most important things that we have been up to over the past month. As a quick recap, whilst anyone may inspect the source code of free software for malicious flaws, almost all software is distributed to end users as pre-compiled binaries.
Andreas Herrmann gave a talk at All Systems Go 2023 titled Fast, correct, reproducible builds with Nix and Bazel . Quoting from the talk description:

You will be introduced to Google s open source build system Bazel, and will learn how it provides fast builds, how correctness and reproducibility is relevant, and how Bazel tries to ensure correctness. But, we will also see where Bazel falls short in ensuring correctness and reproducibility. You will [also] learn about the purely functional package manager Nix and how it approaches correctness and build isolation. And we will see where Bazel has an advantage over Nix when it comes to providing fast feedback during development.
Andreas also shows how you can get the best of both worlds and combine Nix and Bazel, too. A video of the talk is available.
diffoscope is our in-depth and content-aware diff utility that can locate and diagnose reproducibility issues. This month, Chris Lamb fixed compatibility with file(1) version 5.45 [ ] and updated some documentation [ ]. In addition, Vagrant Cascadian extended support for GNU Guix [ ][ ] and updated the version in that distribution as well. [ ].
Yet another reminder that our upcoming Reproducible Builds Summit is set to take place from October 31st November 2nd 2023 in Hamburg, Germany. If you haven t been before, our summits are a unique gathering that brings together attendees from diverse projects, united by a shared vision of advancing the Reproducible Builds effort. During this enriching event, participants will have the opportunity to engage in discussions, establish connections and exchange ideas to drive progress in this vital field. If you re interested in joining us this year, please make sure to read the event page, the news item, or the invitation email that Mattia Rizzolo sent out recently, all of which have more details about the event and location. We are also still looking for sponsors to support the event, so please reach out to the organising team if you are able to help. Also note that PackagingCon 2023 is taking place in Berlin just before our summit.
On the Reproducible Builds website, Greg Chabala updated the JVM-related documentation to update a link to the BUILDSPEC.md file. [ ] And Fay Stegerman fixed the builds failing because of a YAML syntax error.

Distribution work In Debian, this month: September saw F-Droid add ten new reproducible apps, and one existing app switched to reproducible builds. In addition, two reproducible apps were archived and one was disabled for a current total of 199 apps published with Reproducible Builds and using the upstream developer s signature. [ ] In addition, an extensive blog post was posted on f-droid.org titled Reproducible builds, signing keys, and binary repos .

Upstream patches The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of such patches, including:

Testing framework The Reproducible Builds project operates a comprehensive testing framework (available at tests.reproducible-builds.org) in order to check packages and other artifacts for reproducibility. In August, a number of changes were made by Holger Levsen:
  • Disable armhf and i386 builds due to Debian bug #1052257. [ ][ ][ ][ ]
  • Run diffoscope with a lower ionice priority. [ ]
  • Log every build in a simple text file [ ] and create persistent stamp files when running diffoscope to ease debugging [ ].
  • Run schedulers one hour after dinstall again. [ ]
  • Temporarily use diffoscope from the host, and not from a schroot running the tested suite. [ ][ ]
  • Fail the diffoscope distribution test if the diffoscope version cannot be determined. [ ]
  • Fix a spelling error in the email to IRC gateway. [ ]
  • Force (and document) the reconfiguration of all jobs, due to the recent rise of zombies. [ ][ ][ ][ ]
  • Deal with a rare condition when killing processes which should not be there. [ ]
  • Install the Debian backports kernel in an attempt to address Debian bug #1052257. [ ][ ]
In addition, Mattia Rizzolo fixed a call to diffoscope --version (as suggested by Fay Stegerman on our mailing list) [ ], worked on an openQA credential issue [ ] and also made some changes to the machine-readable reproducible metadata, reproducible-tracker.json [ ]. Lastly, Roland Clobus added instructions for manual configuration of the openQA secrets [ ].
If you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:

11 October 2023

Iustin Pop: Not-quite-announcement: this blog is not entirely dead

Something, something then something else, and it s been another six months since I last wrote anything. The world is crazy, somehow there s no time for anything, and yet life move forward, inexorably. (Well, without going into gory side-notes about how evil stops life in various parts of the world.) As a proof that this blog is not dead, I ve finally implemented proper per-page keywords support, rather than the hard-coded keywords that were present before. Well, those defaults are still present in all pages that don t declare keywords, will probably remove them at one point. Implementing this in Hakyll is not hard, at all, if one semi-regularly writes code. There are a number of examples out there, but what was confusing: Anyway, after 2 files changed, 23 insertions(+), 8 deletions(-), this page is the first one to have any keywords. And the funny thing, after initially struggling to write a single line of Haskell, the final version of the keywords is something like this: 
 
    where renderKeywords =
            return . escapeHtml . intercalate "," . map trim . splitAll ","
And I wrote that as is, compiled successfully, and did the right thing from the first try. I only write some Haskell code 2-3 times per year, but man, I love this language. Until next time, hopefully before 2024

Russell Coker: The PineTime

I have just got a PineTime smart watch [1] from Pine64. They cost $US27 each which ended up as $144.63 Australian for three including postage when I ordered on the 16th of September, it s annoying that you can t order more than 3 at a time to reduce postage costs. The Australian online store Kogan has smart watches starting at about $15 [2] with Bluetooth and support for phone notifications so the $48.21 for a PineTime doesn t compare well on just price and features. The watches Kogan sells start getting into high resolution at around the $25 price and many of them have features like 24*7 heart monitoring that the PineTime lacks (it just measures when you request it). No-one would order a PineTime for being cheap or having lots of features, you order it because you want open hardware that allows you to do things your way. Also the PineTime isn t going to be orphaned while it s likely that in a few years most of the cheap watches sold by Kogan etc won t support the new phones running the latest version of Android. The screen of the PineTime is 240*240 resolution (about 260dpi) with 64k colors. The screen resolution is lower than some high-end smart watches but higher than most phones and almost all monitors. I doubt that much benefit could be gained from higher resolution. Even on minimum brightness the screen is easy to read on all but the brightest sunny days. The compute capabilities are 4.5MB of flash storage, 64k of RAM, and a 64MHz CPU this can t run Linux and nothing like it will run Linux for a long time. I ve had the PineTime for 6 days now, I charged it once and it s now at 55% battery. It looks like it will last close to 2 weeks on a single charge and it s claimed that a newer firmware will make the battery last longer. Software The main Android app for using with the PineTime is GadgetBridge which I installed from the f-droid repository. It had lots of click-through menus for allowing access to various Android features (contacts, bluetooth, draw over foreground, location, and more) but after that it was easy to setup. It was the first bluetooth device I ve used which had a 6 digit PIN for connecting to a phone. Initially I used the PineTime with my Huawei Nova 7i [3]. The aim is to eventually have it run from my PinePhonePro but my test of the PinePhonePro didn t go as well as hoped [4]. Now I m using it on my Huawei Mate 10 Pro. It comes with InfiniTime [5] installed as the default firmware, mine had 1.11.0 which is a fairly recent version. I will probably upgrade it soon to get the better power optimisation and weather alerts in the watch face. I don t have any plans to use different watch firmware and I don t have any plans to contribute to firmware development I just can t hack on every FOSS project around it s better to do big contributions to a small number of projects. For people who don t want the default firmware the Wasp-OS project seems interesting as it s written in Python [6], I don t like Python but it s very popular. Python is particularly popular in ML development, it will be interesting to see if Wasp-OS becomes a preferred platform for smart watches that talk to GPT servers. Generally the software works well, one annoyance is that when a notification goes away on the phone it remains on the PineTime and has to be manually dismissed. It would be nice if clearing notifications on the phone would clear them on the PineTime too. The music control works with RocketPlayer on Android, it displays the track name and has options for pause/play and skipping forward and backward one track. Annoyingly the current firmware doesn t allow configuring the main screens, from the primary screen you swipe down for notifications, right for settings, up for menus, and there s nothing defined for swipe left. I d like to make swipe left the command to get to music control. Hardware It has a detachable band that appears to be within the common range of watch bands. According to the PineTime Wiki page [7] there are a selection of alternate bands that will fit it, but some don t because the band is recessed into the watch. It is IP67 rated which means you can probably wear it while swimming. The charging contacts are exposed on the bottom of the case which means that any chemicals left by pool water can be cleaned off and also as they are apparently not expected to be harmed by sweat and skin oil there shouldn t be a problem charging it. I have significant experience using a Samsung Galaxy S5 Mini which is rated at IP67 in swimming pools. I had two problems with the S5 Mini when getting out of the pool, firstly water in the headphone socket made the phone consider that it was in headphone mode and turn off the speakers and secondly it took hours to become dry enough to charge and after many swims the charge rate dropped presumably due to oxide on the contacts. There are reports of success when swimming with a PineTime. Generally it feels well made and appears more solid than the cheapest Kogan devices appear to be. Conclusion If I wanted monitoring for medical reasons then I would choose a different smart watch. I ve read about people doing things like tracking their body stats 24*7 and trying to discover useful things, the PineTime is not a good option for BioHacking type use. However if I did have a need for such things I d probably just buy a second smart watch and have one on each wrist. The PineTime generally works well. It s a pity it has fewer hardware features than closed devices that are cheaper. But having a firmware that can be continually improved by the community is good. The continually expanding use of mobile phone technology devices for custom use in corporations (such as mobile phone in custom case for scanning prices etc in a supermarket) has some potential for use with this. I can imagine someone adding some custom features to a PineTime for such use. When a supermarket chain has 200,000 employees (as Woolworths in Australia does) then paying for a few months of software development work to make a smart watch do specific things for that company could provide significant value. There are probably some business opportunities for FOSS developers to hack on extra hardware on a PineTime and write software to support it. I recommend that everyone who s into FOSS buy one of these. Preferably make a deal with two friends to get the minimum postage cost.
Related posts:
  1. I Just Ordered a Nexus 6P Last year I wrote a long-term review of Android phones...
  2. My Ideal Mobile Phone Based on my experience testing the IBM Seer software on...
  3. Big Smart TVs Recently a relative who owned a 50 Plasma TV asked...

Russell Coker: PinePhone Status

4 months ago I got my PinePhonePro [1]. Since then I have got SE Linux working on it to the stage of allowing it to boot correctly with Debian/Unstable, login with the user_t domain (minimum privilege for the graphical user) and make and receive calls. I finished getting SE Linux working reasonably well 3 days ago and most (but not all) of the SE Linux policy is in Debian now. I ve been getting good at Github PRs and I m sending a lot of policy upstream, so the next version of Debian will have a much smaller diff from the upstream Refpolicy. I have been running the PinePhone with Plasma Mobile because I prefer KDE, I would run GNOME/Phoc if it gave significantly better functionality. 3 days ago I moved my main SIM (the one with the number that people call about work etc) to my PinePhone and tried running it as my main phone. Today I gave up on that. My Android History The last phone I had that did everything I needed was a Huawei Mate 10 Pro that I bought refurbished in June 2019 for $389. The Mate 10 Pro runs Android with the Google Play store and has been quite unremarkable which is presumably why I forgot to blog about it when I got it, it was a slight upgrade over the Huawei Mate 9 [2] that I had used for 2 years before that. In April 2022 I tried using a Huawei Nova 7i as my main phone without Google Play programs or services [3]. That experiment was a failure as I couldn t get NextCloud to work for Calendaring and Contacts. It could be that I stuffed something up when trying that, but I put more skill and effort into trying to get it to work than most people ever would. The Nova 7i is a very slick phone, faster and nicer than the Mate 10 Pro (as expected being 2 years newer) while also having ridiculously long battery life. The Nova 7i when always on running the SchildiChat Matrix client and the Conversations Jabber client it will use less than 10% battery in a 8 hour work day. As nice as the Nova 7i is for the core functions I still need to have Play Store apps for LinkedIn, Twitter, Facebook, Zoom, etc. Which meant connecting the Mate 10 Pro via Wifi. As slick as the Nova 7i is for non-Google stuff running it and the Mate 10 Pro is a medium amount of pain for a small amount of freedom. So now I have for the moment abandoned the Nova 7i and gone back to the Mate 10 Pro. What I will try to do is to either forcibly install Google Play on the Nova 7i to make it my proprietary phone or to install an open distribution of Linux (IE not Android) to make it at least a small tablet which incidentally is much more powerful than a PinePhonePro. Issues With the PinePhonePro The battery charges very slowly (as much as 50 hours estimated charge time) and discharges fast. When used in a typical way in Caffeine mode to stop suspend so I can ssh to it etc it won t come close to lasting an 8 hour day. Also it will only take 5V charging and ideally wants 5V 3A charging which most chargers won t do. The charging speed over regular USB ports is very slow, sometimes stating it will take as much as 50 hours to charge. A phone that gets below 50% charge in less than 4 hours and can t be charged at a reasonable speed from a USB port on a laptop or monitor is going to be a major pain to use in the office. I don t think this can be fixed in software, we can alleviate it by making software more CPU efficient and by enabling various hardware sleep modes more effectively but the slow charging is a hardware issue that can t be fixed. The phone call quality is poor. Usually when on a call I hear static and sometimes the person at the other end appears to hear nothing. Also the UI for calls is different from Android which makes it take longer to answer a call and gives more missed calls. The UI issue is a combination of software and habits, both of which can be changed. But the call quality may be a hardware issue. I don t know if it s a hardware issue specific to my phone or something related to the PinePhonePro on the Telstra network. Clicking on notification in the drop-down doesn t take me to the app. I don t know if this is regarded as a bug by the Plasma-Mobile developers. Also notifications aren t displayed on the lock screen and there doesn t seem to be a configuration option to enable this. The Plasma-Mobile configuration for the wifi hotspot is difficult. There is no button for it in the drop-down menu (called Quicksettings in Plasma-Mobile) and no way of easily determining whether it s in hotspot or wifi client mode. This isn t an insurmountable problem and the worst-case is that I could write a script to do it, but it s still annoying. There is apparent support in the desktop version of KDE for syncing contacts from Google and I could probably get that working (although I failed last time I tried on the desktop), but it is a pain. Conclusion Most of the problems are software related and therefore I can get involved in solving them. I plan to keep working on these things. If all the software had worked in an ideal manner then I would have spent more time investigating the hardware issues of battery life and charge time and the quality of calls. I now have my Librem5 [4] running Debian so I will be able to compare call quality with the PinePhonePro. If I m unable to get the PinePhonePro working adequately then maybe making the Librem5 my main phone will be an option. I hope that by early next year I will be able to make another test at using a FOSS phone my main phone. In the mean time I can still work on convergence and other things.
Related posts:
  1. Health and Status Monitoring via Smart Phone Health Monitoring Eric Topol gave an interesting TED talk about...
  2. Android Without Play A while ago I was given a few reasonably high-end...
  3. Google Hardware Support Ironically just 5 days after writing about how I choose...

10 October 2023

Dirk Eddelbuettel: drat 0.2.4 on CRAN: Improved macOS Support, General Updates

drat user A new minor release of the drat package arrived on CRAN today making it the first release in one and a half years. drat stands for drat R Archive Template, and helps with easy-to-create and easy-to-use repositories for R packages. Since its inception in early 2015 it has found reasonably widespread adoption among R users because repositories with marked releases is the better way to distribute code. Because for once it really is as your mother told you: Friends don t let friends install random git commit snapshots. Properly rolled-up releases it is. Just how CRAN shows us: a model that has demonstrated for two-plus decades how to do this. And you can too: drat is easy to use, documented by six vignettes and just works. Detailed information about drat is at its documentation site. Two more blog posts using drat from GitHub Actions were just added today showing, respectively, how to add to a drat repo in either push or pull mode. This release contains two extended PRs contributed by drat users! Both extended support for macOS: Joey Reid extended M1 support to pruning and archival, and Arne Johannes added bug-sur support. I polished a few more things around the edges, mostly documentation or continuos-integrations related. The NEWS file summarises the release as follows:

Changes in drat version 0.2.4 (2023-10-09)
  • macOS Arm M1 repos are now also supported in pruning and archival (Joey Reid in #135 fixing #134)
  • A minor vignette typo was fixed (Dirk)
  • A small error with setwd() in insertPackage() was corrected (Dirk)
  • macOS x86_64 repos (on big-sur) are now supported too (Arne Johannes Holmin in #139 fixing #138)
  • A few small maintenance tweaks were applied to the CI setup, and to the main README.md

Courtesy of my CRANberries, there is a comparison to the previous release. More detailed information is on the drat page as well as at the documentation site. If you like this or other open-source work I do, you can sponsor me at GitHub.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

4 October 2023

Russ Allbery: Review: The Last Watch

Review: The Last Watch, by J.S. Dewes
Series: Divide #1
Publisher: Tor
Copyright: 2021
ISBN: 1-250-23634-7
Format: Kindle
Pages: 476
The Last Watch is the first book of a far-future science fiction duology. It was J.S. Dewes's first novel. The station of the SCS Argus is the literal edge of the universe: the Divide, beyond which there is nothing. Not simply an absence of stars, but a nothing from a deeper level of physics. The Argus is there to guard against a return of the Viators, the technologically superior alien race that nearly conquered humanity hundreds of years prior and has already returned once, apparently traveling along the Divide. Humanity believes the Viators have been wiped out, but they're not taking chances. It is not a sought-after assignment. The Sentinels are the dregs of the military: convicts, troublemakers, and misfits, banished to the literal edge of nowhere. Joining them at the start of this book is the merchant prince, cocky asshole, and exiled sabateur Cavalon Mercer. He doesn't know what to expect from either military service or service on the edge of the universe. He certainly did not expect the Argus to be commanded by Adequin Rake, a literal war hero and a far more effective leader than this post would seem to warrant. There are reasons why Rake is out on the edge of the universe, ones that she's not eager to talk about. They quickly become an afterthought when the Argus discovers that the Divide is approaching their position. The universe is collapsing, and the only people who know about it are people the System Collective would prefer to forget exist. Yes, the edge of the universe, not the edge of the galaxy. Yes, despite having two FTL mechanisms, this book has a scale problem that it never reconciles. And yes, the physics do not really make sense, although this is not the sort of book that tries to explain the science. The characters are too busy trying to survive to develop new foundational theories of physics. I was looking for more good military SF after enjoying Artifact Space so much (and still eagerly awaiting the sequel), so I picked this up. It has some of the same elements: the military as a place where you can make a fresh start with found family elements, the equalizing effects of military assignments, and the merits of good leadership. They're a bit disguised here, since this is a crew of often-hostile misfits under a lot of stress with a partly checked-out captain, but they do surface towards the end of the book. The strength of this book is the mystery of the contracting universe, which poses both an immediate threat to the ship and a longer-term potential threat to, well, everything. The first part of the book builds tension with the immediate threat, but the story comes into its own when the crew starts piecing together the connections between the Viators and the Divide while jury-rigging technology and making risky choices between a lot of bad options. This is the first half of a duology, so the mysteries are not resolved here, but they do reach a satisfying and tantalizing intermediate conclusion. The writing is servicable and adequate, but it's a bit clunky in places. Dewes doesn't quite have the balance right between setting the emotional stakes and not letting the characters indulge in rumination. Rake is a good captain who is worn down and partly checked out, Mercer is scared and hiding it with arrogance and will do well when given the right sort of attention, and all of this is reasonably obvious early on and didn't need as many of the book's pages as it gets. I could have done without the romantic subplot, which I thought was an unnecessary distraction from the plot and turned into a lot of tedious angst, but I suspect I was not the target audience. (Writers, please remember that people can still care about each other and be highly motivated by fear for each other without being romantic partners.) I would not call this a great book. The characters are not going to surprise you that much, and it's a bit long for the amount of plot that it delivers. If you are the sort of person who nit-picks the physics of SF novels and gets annoyed at writers who don't understand how big the universe is, you will have to take a deep breath and hold on to your suspension of disbelief. But Dewes does a good job with ratcheting up the tension and conveying an atmosphere of mysterious things happening at the edge of nowhere, while still keeping it in the genre of mysterious technology and mind-boggingly huge physical phenomena rather than space horror. If you've been looking for that sort of book, this will do. I was hooked and will definitely read the sequel. Followed by The Exiled Fleet. Rating: 7 out of 10

3 October 2023

Russ Allbery: Review: Monstrous Regiment

Review: Monstrous Regiment, by Terry Pratchett
Series: Discworld #31
Publisher: Harper
Copyright: October 2003
Printing: August 2014
ISBN: 0-06-230741-X
Format: Mass market
Pages: 457
Monstrous Regiment is the 31st Discworld novel, but it mostly stands by itself. You arguably could start here, although you would miss the significance of Vimes's presence and the references to The Truth. The graphical reading order guide puts it loosely after The Truth and roughly in the Industrial Revolution sequence, but the connections are rather faint.
There was always a war. Usually they were border disputes, the national equivalent of complaining that the neighbor was letting their hedge row grow too long. Sometimes they were bigger. Borogravia was a peace-loving country in the middle of treacherous, devious, warlike enemies. They had to be treacherous, devious, and warlike; otherwise, we wouldn't be fighting them, eh? There was always a war.
Polly's brother, who wanted nothing more than to paint (something that the god Nuggan and the ever-present Duchess certainly did not consider appropriate for a strapping young man), was recruited to fight in the war and never came back. Polly is worried about him and tired of waiting for news. Exit Polly, innkeeper's daughter, and enter the young lad Oliver Perks, who finds the army recruiters in a tavern the next town over. One kiss of the Duchess's portrait later, and Polly is a private in the Borogravian army. I suspect this is some people's favorite Discworld novel. If so, I understand why. It was not mine, for reasons that I'll get into, but which are largely not Pratchett's fault and fall more into the category of pet peeves. Pratchett has dealt with both war and gender in the same book before. Jingo is also about a war pushed by a ruling class for stupid reasons, and featured a substantial subplot about Nobby cross-dressing that turns into a deeper character re-evaluation. I thought the war part of Monstrous Regiment was weaker (this is part of my complaint below), but gender gets a considerably deeper treatment. Monstrous Regiment is partly about how arbitrary and nonsensical gender roles are, and largely about how arbitrary and abusive social structures can become weirdly enduring because they build up their own internally reinforcing momentum. No one knows how to stop them, and a lot of people find familiar misery less frightening than unknown change, so the structure continues despite serving no defensible purpose. Recently, there was a brief attempt in some circles to claim Pratchett posthumously for the anti-transgender cause in the UK. Pratchett's daughter was having none of it, and any Pratchett reader should have been able to reject that out of hand, but Monstrous Regiment is a comprehensive refutation written by Pratchett himself some twenty years earlier. Polly is herself is not transgender. She thinks of herself as a woman throughout the book; she's just pretending to be a boy. But she also rejects binary gender roles with the scathing dismissal of someone who knows first-hand how superficial they are, and there is at least one transgender character in this novel (although to say who would be a spoiler). By the end of the book, you will have no doubt that Pratchett's opinion about people imposing gender roles on others is the same as his opinion about every other attempt to treat people as things. That said, by 2023 standards the treatment of gender here seems... naive? I think 2003 may sadly have been a more innocent time. We're now deep into a vicious backlash against any attempt to question binary gender assignment, but very little of that nastiness and malice is present here. In one way, this is a feature; there's more than enough of that in real life. However, it also makes the undermining of gender roles feel a bit too easy. There are good in-story reasons for why it's relatively simple for Polly to pass as a boy, but I still spent a lot of the book thinking that passing as a private in the army would be a lot harder and riskier than this. Pratchett can't resist a lot of cross-dressing and gender befuddlement jokes, all of which are kindly and wry but (at least for me) hit a bit differently in 2023 than they would have in 2003. The climax of the story is also a reference to a classic UK novel that to even name would be to spoil one or both of the books, but which I thought pulled the punch of the story and dissipated a lot of the built-up emotional energy. My larger complaints, though, are more idiosyncratic. This is a war novel about the enlisted ranks, including the hazing rituals involved in joining the military. There are things I love about military fiction, but apparently that reaction requires I have some sympathy for the fight or the goals of the institution. Monstrous Regiment falls into the class of war stories where the war is pointless and the system is abusive but the camaraderie in the ranks makes service oddly worthwhile, if not entirely justifiable. This is a real feeling that many veterans do have about military service, and I don't mean to question it, but apparently reading about it makes me grumbly. There's only so much of the apparently gruff sergeant with a heart of gold that I can take before I start wondering why we glorify hazing rituals as a type of tough love, or why the person with some authority doesn't put a direct stop to nastiness instead of providing moral support so subtle you could easily blink and miss it. Let alone the more basic problems like none of these people should have to be here doing this, or lots of people are being mangled and killed to make possible this heart-warming friendship. Like I said earlier, this is a me problem, not a Pratchett problem. He's writing a perfectly reasonable story in a genre I just happen to dislike. He's even undermining the genre in the process, just not quite fast enough or thoroughly enough for my taste. A related grumble is that Monstrous Regiment is very invested in the military trope of naive and somewhat incompetent officers who have to be led by the nose by experienced sergeants into making the right decision. I have never been in the military, but I work in an industry in which it is common to treat management as useless incompetents at best and actively malicious forces at worst. This is, to me, one of the most persistently obnoxious attitudes in my profession, and apparently my dislike of it carries over as a low tolerance for this very common attitude towards military hierarchy. A full expansion of this point would mostly be about the purpose of management, division of labor, and people's persistent dismissal of skills they don't personally have and may perceive as gendered, and while some of that is tangentially related to this book, it's not closely-related enough for me to bore you with it in a review. Maybe I'll write a stand-alone blog post someday. Suffice it to say that Pratchett deployed a common trope that most people would laugh at and read past without a second thought, but that for my own reasons started getting under my skin by the end of the novel. All of that grumbling aside, I did like this book. It is a very solid Discworld novel that does all the typical things a Discworld novel does: likable protagonists you can root for, odd and fascinating side characters, sharp and witty observations of human nature, and a mostly enjoyable ending where most of the right things happen. Polly is great; I was very happy to read a book from her perspective and would happily read more. Vimes makes a few appearances being Vimes, and while I found his approach in this book less satisfying than in Jingo, I'll still take it. And the examination of gender roles, even if a bit less fraught than current politics, is solid Pratchett morality. The best part of this book for me, by far, is Wazzer. I think that subplot was the most Discworld part of this book: a deeply devout belief in a pseudo-godlike figure that is part of the abusive social structure that creates many of the problems of the book becomes something considerably stranger and more wonderful. There is a type of belief that is so powerful that it transforms the target of that belief, at least in worlds like Discworld that have a lot of ambient magic. Not many people have that type of belief, and having it is not a comfortable experience, but it makes for a truly excellent story. Monstrous Regiment is a solid Discworld novel. It was not one of my favorites, but it probably will be someone else's favorite for a host of good reasons. Good stuff; if you've read this far, you will enjoy it. Followed by A Hat Full of Sky in publication order, and thematically (but very loosely) by Going Postal. Rating: 8 out of 10

Next.

Previous.