Search Results: "cts"

12 January 2021

Molly de Blanc: 1028 Words on Free Software

The promise of free software is a near-future utopia, built on democratized technology. This future is just and it is beautiful, full of opportunity and fulfillment for everyone everywhere. We can create the things we dream about when we let our minds wander into the places they want to. We can be with the people we want and need to be, when we want and need to. This is currently possible with the technology we have today, but it s availability is limited by the reality of the world we live in the injustice, the inequity, the inequality. Technology runs the world, but it does not serve the interests of most of us. In order to create a better world, our technology must be transparent, accountable, trustworthy. It must be just. It must be free. The job of the free software movement is to demonstrate that this world is possible by living its values now: justice, equity, equality. We build them into our technology, and we build technology that make it possible for these values to exist in the world. At the Free Software Foundation, we liked to say that we used all free software because it was important to show that we could. You can do anything with free software, so we did everything with it. We demonstrated the importance of unions for tech workers and non-profit workers by having one. We organized collectively and protected our rights for the sake of ourselves and one another. We had non-negotiable salaries, based on responsibility level and position. That didn t mean we worked in an office free from the systemic problems that plague workplaces everywhere, but we were able to think about them differently. Things were this way because of Richard Stallman but I view his influence on these things as negative rather than positive. He was a cause that forced these outcomes, rather than being supportive of the desires and needs of others. Rather than indulge in gossip or stories, I would like to jump to the idea that he was supposed to have been deplatformed in October 2019. In resigning from his position as president of the FSF, he certainly lost some of his ability to reach audiences. However, Richard still gives talks. The FSF continues to use his image and rhetoric in their own messaging and materials. They gave him time to speak at their annual conference in 2020. He maintains leadership in the GNU project and otherwise within the FSF sphere. The people who empowered him for so many years are still in charge. Richard, and the continued respect and space he is given, is not the only problem. It represents a bigger problem. Sexism and racism (among others) run rampant in the community. This happens because of bad actors and, more significantly, by the complacency of organizations, projects, and individuals afraid of losing contributors, respect, or funding. In a sector that has so much money and so many resources, women are still being paid less than men; we deny people opportunities to learn and grow in the name of immediate results; people who aren t men, who aren t white, are abused and harassed; people are mentally and emotionally taken advantage of, and we are coerced into burn out and giving up our lives for these companies and projects and we are paid for tolerating all of this by being told we re doing a good job or making a difference. But we re not making a difference. We re perpetuating the worst of the status quo that we should be fighting against. We must not continue. We cannot. We need to live our ideals as they are, and take the natural next steps in their evolution. We cannot have a world of just technology when we live in a world of exclusion; we cannot have free software if we continue to allow, tolerate, and laud the worst of us. I ve been in and around free software for seventeen years. Nearly every part of it I ve participated in has members and leadership that benefit from allowing and encouraging the continuation of maleficence and systemic oppression. We must purge ourselves of these things of sexism, racism, injustice, and the people who continue and enable it. There is no space to argue over whether a comment was transphobic if it hurt a trans person then it is transphobic and it is unacceptable. Racism is a global problem and we must be anti-racist or we are complicit. Sexism is present and all men benefit from it, even if they don t want to. These are free software issues. These are things that plague software, and these are things software reinforces within our societies. If a technology is exclusionary, it does not work. If a community is exclusionary, it must be fixed or thrown away. There is no middle ground here. There is no compromise. Without doing this, without taking the hard, painful steps to actually live the promise of user freedom and everything it requires and entails, our work is pointless and free software will fail. I don t think it s too late for there to be a radical change the radical change that allows us to create the utopia we want to see in the world. We must do that by acknowledging that just technology leads to a just society, and that a just society allows us to make just technology. We must do that by living within the principles that guide this future now. I don t know what will happen if things don t change soon. I recently saw someone comment that change doesn t happens unless one person is willing to sacrifice everything to make that change, to lead and inspire others to play small parts. This is unreasonable to ask of or expect from someone. I ve been burning myself out to meet other people s expectations for seventeen years, and I can t keep doing it. Of course I am not alone, and I am not the only one working on and occupied by these problems. More people must step up, not just for my sake, but for the sake of all of us, the work free software needs to do, and the future I dream about.

10 January 2021

Dirk Eddelbuettel: RcppArmadillo 0.10.1.2.2: Minor update

armadillo image Armadillo is a powerful and expressive C++ template library for linear algebra aiming towards a good balance between speed and ease of use with a syntax deliberately close to a Matlab. RcppArmadillo integrates this library with the R environment and language and is widely used by (currently) 802 other packages on CRAN. This release was needed because we use the Matrix package for some (optional) tests related to sparse matrices, and a small and subtle change and refinement in the recent 1.3.0 release of Matrix required us to make an update for the testing. Nothing has changed in how we set up, or operate on, sparse matrices. My thanks to Binxiang and Martin Maechler for feedback and suggestion on the initial fix both Binxiang and I set up independently. At the same time we upgrade some package internals related to continuous integration (for that, also see my blog post and video from earlier this week). Lastly Conrad sent in a one-line upstream fix for dealing with NaN in sign(). The full set of changes follows.

Changes in RcppArmadillo version 0.10.1.2.2 (2021-01-08)
  • Correct one unit test for Matrix 1.3.0-caused changed (Binxiang in #319 and Dirk in #322).
  • Suppress one further warning from Matrix (Dirk)
  • Apply an upstream NaN correction (Conrad in #321)
  • Added GitHub Actions CI using run.sh from r-ci (Dirk)

Courtesy of my CRANberries, there is a diffstat report relative to previous release. More detailed information is on the RcppArmadillo page. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page. If you like this or other open-source work I do, you can sponsor me at GitHub.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

9 January 2021

Jonathan McDowell: Free Software Activities for 2020

As a reader of Planet Debian I see a bunch of updates at the start of each month about what people are up to in terms of their Free Software activities. I m not generally active enough in the Free Software world to justify a monthly report, but I did a report of my Free Software Activities for 2019 and thought I d do another for 2020. I ended up not doing as much as last year; I put a lot of that down to fatigue about the state of the world and generally not wanting to spend time on the computer at the end of the working day.

Conferences 2020 was unsurprisingly not a great year for conference attendance. I was fortunate enough to make it to FOSDEM and CopyleftConf 2020 - I didn t speak at either, but had plenty of interesting hallway track conversations as well as seeing some good talks. I hadn t been planning to attend DebConf20 due to time constraints, but its move to an entirely online conference meant I was able to attend a few talks at least. I have to say I don t like virtual conferences as much as the real thing; it s not as easy to have the casual chats at them, and it s also harder to carve out the exclusive time when you re at home. That said I spoke at NIDevConf this year, which was also fully virtual. It s not a Free Software focussed conference, but there s a lot of crossover in terms of technologies and I spoke on my experiences with Go, some of which are influenced by my packaging experiences within Debian.

Debian Most of my contributions to Free software happen within Debian. As part of the Data Protection Team I responded to various inbound queries to that team. Some of this involved chasing up other project teams who had been slow to respond - folks, if you re running a service that stores personal data about people then you need to be responsive to requests about it. The Debian Keyring was possibly my largest single point of contribution. We re in a roughly 3 month rotation of who handles the keyring updates, and I handled 2020.02.02, 2020.03.24, 2020.06.24, 2020.09.24 + 2020.12.24 For Debian New Members I m mostly inactive as an application manager - we generally seem to have enough available recently. If that changes I ll look at stepping in to help, but I don t see that happening. I continue to be involved in Front Desk, having various conversations throughout the year with the rest of the team, but there s no doubt Mattia and Pierre-Elliott are the real doers at present. In terms of package uploads I continued to work on gcc-xtensa-lx106, largely doing uploads to deal with updates to the GCC version or packaging (5, 6 + 7). sigrok had a few minor updates, libsigkrok 0.5.2-2, libsigrokdecode 0.5.3-2 as well as a new upstream release of Pulseview 0.4.2-1 and a fix to cope with change to QT 0.4.2-2. Due to the sigrok-firmware requirement on sdcc I also continued to help out there, updating to 4.0.0+dfsg-1 and doing some fixups in 4.0.0+dfsg-2. Despite still not writing an VHDL these days I continue to try and make sure ghdl is ok, because I found it a useful tool in the past. In 2020 that meant a new upstream release, 0.37+dfsg-1 along with a couple of more minor updates (0.37+dfsg-2 + 0.37+dfsg-3. libcli had a new upstream release, 1.10.4-1, and I did a long overdue update to sendip to the latest upstream release, 2.6-1 having been poked about an outstanding bug by the Reproducible Builds folk. OpenOCD is coming up to 4 years since its last stable release, but I did a snapshot upload to Debian experimental (0.10.0+g20200530-1) and a subsequent one to unstable (0.10.0+g20200819-1). There are also moves to produce a 0.11.0 release and I uploaded 0.11.0~rc1-1 as a result. libjaylink got a bump as a result (0.2.0-1) after some discussion with upstream.

OpenOCD On the subject of OpenOCD I ve tried to be a bit more involved upstream. I m not familiar enough with the intricacies of JTAG/SWD/the various architectures supported to contribute to the core, but I pushed the config for my HIE JTAG adapter upstream and try and review patches that don t require in depth hardware knowledge.

Linux I ve been contributing to the Linux kernel for a number of years now, mostly just minor bits here and there for issues I hit. This year I spent a lot of time getting support for the MikoTik RB3011 router upstreamed. That included the basic DTS addition, fixing up QCA8K to support SGMII CPU connections, adding proper 802.1q VLAN support to QCA8K and cleaning up an existing QCOM ADM driver that s required for the NAND. There were a number of associated bugfixes/minor changes found along the way too. It can be a little frustrating at times going round the review loop with submitting things upstream, but I do find it quite satisfying when it all comes together and I have no interest in weird vendor trees that just bitrot over time.

Software in the Public Interest I haven t sat on the board of SPI since 2015 but I was still acting as the primary maintainer of the membership website (with Martin Michlmayr as the other active contributor) and hosting it on my own machine. I managed to finally extricate myself from this role in August. I remain a contributing member.

Personal projects 2020 finally saw another release (0.6.0, followed swiftly by 0.6.1 to allow the upload of 0.6.1-1 to Debian) of onak. This release finally adds various improvements to deal with the hostility shown to the OpenPGP keyserver network in recent years, including full signature verification as an option. I fixed an oversight in my Digoo/1-wire temperature decoder and a bug that turned up on ARM but not MIPS in my mqtt-arp code. I should probably package it for Debian (even if I don t upload it), as I m running it on my RB3011 now.

Louis-Philippe V ronneau: puppetserver 6: a Debian packaging post-mortem

I have been a Puppet user for a couple of years now, first at work, and eventually for my personal servers and computers. Although it can have a steep learning curve, I find Puppet both nimble and very powerful. I also prefer it to Ansible for its speed and the agent-server model it uses. Sadly, Puppet Labs hasn't been the most supportive upstream and tends to move pretty fast. Major versions rarely last for a whole Debian Stable release and the upstream .deb packages are full of vendored libraries.1 Since 2017, Apollon Oikonomopoulos has been the one doing most of the work on Puppet in Debian. Sadly, he's had less time for that lately and with Puppet 5 being deprecated in January 2021, Thomas Goirand, Utkarsh Gupta and I have been trying to package Puppet 6 in Debian for the last 6 months. With Puppet 6, the old ruby Puppet server using Passenger is not supported anymore and has been replaced by puppetserver, written in Clojure and running on the JVM. That's quite a large change and although puppetserver does reuse some of the Clojure libraries puppetdb (already in Debian) uses, packaging it meant quite a lot of work. Work in the Clojure team As part of my efforts to package puppetserver, I had the pleasure to join the Clojure team and learn a lot about the Clojure ecosystem. As I mentioned earlier, a lot of the Clojure dependencies needed for puppetserver were already in the archive. Unfortunately, when Apollon Oikonomopoulos packaged them, the leiningen build tool hadn't been packaged yet. This meant I had to rebuild a lot of packages, on top of packaging some new ones. Since then, thanks to the efforts of Elana Hashman, leiningen has been packaged and lets us run the upstream testsuites and create .jar artifacts closer to those upstream releases. During my work on puppetserver, I worked on the following packages:
List of packages
  • backport9
  • bidi-clojure
  • clj-digest-clojure
  • clj-helper
  • clj-time-clojure
  • clj-yaml-clojure
  • cljx-clojure
  • core-async-clojure
  • core-cache-clojure
  • core-match-clojure
  • cpath-clojure
  • crypto-equality-clojure
  • crypto-random-clojure
  • data-csv-clojure
  • data-json-clojure
  • data-priority-map-clojure
  • java-classpath-clojure
  • jnr-constants
  • jnr-enxio
  • jruby
  • jruby-utils-clojure
  • kitchensink-clojure
  • lazymap-clojure
  • liberator-clojure
  • ordered-clojure
  • pathetic-clojure
  • potemkin-clojure
  • prismatic-plumbing-clojure
  • prismatic-schema-clojure
  • puppetlabs-http-client-clojure
  • puppetlabs-i18n-clojure
  • puppetlabs-ring-middleware-clojure
  • puppetserver
  • raynes-fs-clojure
  • riddley-clojure
  • ring-basic-authentication-clojure
  • ring-clojure
  • ring-codec-clojure
  • shell-utils-clojure
  • ssl-utils-clojure
  • test-check-clojure
  • tools-analyzer-clojure
  • tools-analyzer-jvm-clojure
  • tools-cli-clojure
  • tools-reader-clojure
  • trapperkeeper-authorization-clojure
  • trapperkeeper-clojure
  • trapperkeeper-filesystem-watcher-clojure
  • trapperkeeper-metrics-clojure
  • trapperkeeper-scheduler-clojure
  • trapperkeeper-webserver-jetty9-clojure
  • url-clojure
  • useful-clojure
  • watchtower-clojure
If you want to learn more about packaging Clojure libraries and applications, I rewrote the Debian Clojure packaging tutorial and added a section about the quirks of using leiningen without a dedicated dh_lein tool. Work left to get puppetserver 6 in the archive Unfortunately, I was not able to finish the puppetserver 6 packaging work. It is thus unlikely it will make it in Debian Bullseye. If the issues described below are fixed, it would be possible to to package puppetserver in bullseye-backports though. So what's left? jruby Although I tried my best (kudos to Utkarsh Gupta and Thomas Goirand for the help), jruby in Debian is still broken. It does build properly, but the testsuite fails with multiple errors: jruby testsuite failures aside, I have not been able to use the jruby.deb the package currently builds in jruby-utils-clojure (testsuite failure). I had the same exact failure with the (more broken) jruby version that is currently in the archive, which leads me to think this is a LOAD_PATH issue in jruby-utils-clojure. More on that below. To try to bypass these issues, I tried to vendor jruby into jruby-utils-clojure. At first I understood vendoring meant including upstream pre-built artifacts (jruby-complete.jar) and shipping them directly. After talking with people on the #debian-mentors and #debian-ftp IRC channels, I now understand why this isn't a good idea (and why it's not permitted in Debian). Many thanks to the people who were patient and kind enough to discuss this with me and give me alternatives. As far as I now understand it, vendoring in Debian means "to have an embedded copy of the source code in another package". Code shipped that way still needs to be built from source. This means we need to build jruby ourselves, one way or another. Vendoring jruby in another package thus isn't terribly helpful. If fixing jruby the proper way isn't possible, I would suggest trying to build the package using embedded code copies of the external libraries jruby needs to build, instead of trying to use the Debian libraries.2 This should make it easier to replicate what upstream does and to have a final .jar that can be used. jruby-utils-clojure This package is a first-level dependency for puppetserver and is the glue between jruby and puppetserver. It builds fine, but the testsuite fails when using the Debian jruby package. I think the problem is caused by a jruby LOAD_PATH issue. The Debian jruby package plays with the LOAD_PATH a little to try use Debian packages instead of downloading gems from the web, as upstream jruby does. This seems to clash with the gem-home, gem-path, and jruby-load-path variables in the jruby-utils-clojure package. The testsuite plays around with these variables and some Ruby libraries can't be found. I tried to fix this, but failed. Using the upstream jruby-complete.jar instead of the Debian jruby package, the testsuite passes fine. This package could clearly be uploaded to NEW right now by ignoring the testsuite failures (we're just packaging static .clj source files in the proper location in a .jar). puppetserver jruby issues aside, packaging puppetserver itself is 80% done. Using the upstream jruby-complete.jar artifact, the testsuite fails with a weird Clojure error I'm not sure I understand, but I haven't debugged it for very long. Upstream uses git submodules to vendor puppet (agent), hiera (3), facter and puppet-resource-api for the testsuite to run properly. I haven't touched that, but I believe we can either: Without the testsuite actually running, it's hard to know what files are needed in those packages. What now Puppet 5 is now deprecated. If you or your organisation cares about Puppet in Debian,3 puppetserver really isn't far away from making it in the archive. Very talented Debian Developers are always eager to work on these issues and can be contracted for very reasonable rates. If you're interested in contracting someone to help iron out the last issues, don't hesitate to reach out via one of the following: As for I, I'm happy to say I got a new contract and will go back to teaching Economics for the Winter 2021 session. I might help out with some general Debian packaging work from time to time, but it'll be as a hobby instead of a job. Thanks The work I did during the last 6 weeks would be not have been possible without the support of the Wikimedia Foundation, who were gracious enough to contract me. My particular thanks to Faidon Liambotis, Moritz M hlenhoff and John Bond. Many, many thanks to Rob Browning, Thomas Goirand, Elana Hashman, Utkarsh Gupta and Apollon Oikonomopoulos for their direct and indirect help, without which all of this wouldn't have been possible.

  1. For example, the upstream package for the Puppet Agent vendors OpenSSL.
  2. One of the problems of using Ruby libraries already packaged in Debian is that jruby currently only supports Ruby 2.5. Ruby libraries in Debian are currently expected to work with Ruby 2.7, with the transition to Ruby 3.0 planned after the Bullseye release.
  3. If you run Puppet, you clearly should care: the .deb packages upstream publishes really aren't great and I would not recommend using them.

7 January 2021

Russell Coker: Monopoly the Game

The Smithsonian Mag has an informative article about the history of the game Monopoly [1]. The main point about Monopoly teaching about the problems of inequality is one I was already aware of, but there are some aspects of the history that I learned from the article. Here s an article about using modified version of Monopoly to teach Sociology [2]. Maria Paino and Jeffrey Chin wrote an interesting paper about using Monopoly with revised rules to teach Sociology [3]. They publish the rules which are interesting and seem good for a class. I think it would be good to have some new games which can teach about class differences. Maybe have an Escape From Poverty game where you have choices that include drug dealing to try and improve your situation or a cooperative game where people try to create a small business. While Monopoly can be instructive it s based on the economic circumstances of the past. The vast majority of rich people aren t rich from land ownership.

5 January 2021

Russell Coker: Weather and Boinc

I just wrote a Perl script to look at the Australian Bureau of Meteorology pages to find the current temperature in an area and then adjust BOINC settings accordingly. The Perl script (in this post after the break, which shouldn t be in the RSS feed) takes the URL of a Bureau of Meteorology observation point as ARGV[0] and parses that to find the current (within the last hour) temperature. Then successive command line arguments are of the form 24:100 and 30:50 which indicate that at below 24C 100% of CPU cores should be used and below 30C 50% of CPU cores should be used. In warm weather having a couple of workstations in a room running BOINC (or any other CPU intensive task) will increase the temperature and also make excessive noise from cooling fans. To change the number of CPU cores used the script changes /etc/boinc-client/global_prefs_override.xml and then tells BOINC to reload that config file. This code is a little ugly (it doesn t properly parse XML, it just replaces a line of text) and could fail on a valid configuration file that wasn t produced by the current BOINC code. The parsing of the BoM page is a little ugly too, it relies on the HTML code in the BoM page they could make a page that looks identical which breaks the parsing or even a page that contains the same data that looks different. It would be nice if the BoM published some APIs for getting the weather. One thing that would be good is TXT records in the DNS. DNS supports caching with specified lifetime and is designed for high throughput in aggregate. If you had a million IOT devices polling the current temperature and forecasts every minute via DNS the people running the servers wouldn t even notice the load, while a million devices polling a web based API would be a significant load. As an aside I recommend playing nice and only running such a script every 30 minutes, the BoM page seems to be updated on the half hour so I have my cron jobs running at 5 and 35 minutes past the hour. If this code works for you then that s great. If it merely acts as an inspiration for developing your own code then that s great too! BOINC users outside Australia could replace the code for getting meteorological data (or even interface to a digital thermometer). Australians who use other CPU intensive batch jobs could take the BoM parsing code and replace the BOINC related code. If you write scripts inspired by this please blog about it and comment here with a link to your blog post.
#!/usr/bin/perl
use strict;
use Sys::Syslog;
# St Kilda Harbour RMYS
# http://www.bom.gov.au/products/IDV60901/IDV60901.95864.shtml
my $URL = $ARGV[0];
open(IN, "wget -o /dev/null -O - $URL ") or die "Can't get $URL";
while(<IN>)
 
  if($_ =~ /tr class=.rowleftcolumn/)
   
    last;
   
 
sub get_data
 
  if(not $_[0] =~ /headers=.t1-$_[1]/)
   
    return undef;
   
  $_[0] =~ s/^.*headers=.t1-$_[1]..//;
  $_[0] =~ s/<.td.*$//;
  return $_[0];
 
my @datetime;
my $cur_temp -100;
while(<IN>)
 
  chomp;
  if($_ =~ /^<.tr>$/)
   
    last;
   
  my $res;
  if($res = get_data($_, "datetime"))
   
    @datetime = split(/\//, $res)
   
  elsif($res = get_data($_, "tmp"))
   
    $cur_temp = $res;
   
 
close(IN);
if($#datetime != 1 or $cur_temp == -100)
 
  die "Can't parse BOM data";
 
my ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = localtime();
if($mday - $datetime[0] > 1 or ($datetime[0] > $mday and $mday != 1))
 
  die "Date wrong\n";
 
my $mins;
my @timearr = split(/:/, $datetime[1]);
$mins = $timearr[0] * 60 + $timearr [1];
if($timearr[1] =~ /pm/)
 
  $mins += 720;
 
if($mday != $datetime[0])
 
  $mins += 1440;
 
if($mins + 60 < $hour * 60 + $min)
 
  die "page outdated\n";
 
my %temp_hash;
foreach ( @ARGV[1..$#ARGV] )
 
  my @tmparr = split(/:/, $_);
  $temp_hash $tmparr[0]  = $tmparr[1];
 
my @temp_list = sort(keys(%temp_hash));
my $percent = 0;
my $i;
for($i = $#temp_list; $i >= 0 and $temp_list[$i] > $cur_temp; $i--)
 
  $percent = $temp_hash $temp_list[$i] 
 
my $prefs = "/etc/boinc-client/global_prefs_override.xml";
open(IN, "<$prefs") or die "Can't read $prefs";
my @prefs_contents;
while(<IN>)
 
  push(@prefs_contents, $_);
 
close(IN);
openlog("boincmgr-cron", "", "daemon");
my @cpus_pct = grep(/max_ncpus_pct/, @prefs_contents);
my $cpus_line = $cpus_pct[0];
$cpus_line =~ s/..max_ncpus_pct.$//;
$cpus_line =~ s/^.*max_ncpus_pct.//;
if($cpus_line == $percent)
 
  syslog("info", "Temp $cur_temp" . "C, already set to $percent");
  exit 0;
 
open(OUT, ">$prefs.new") or die "Can't read $prefs.new";
for($i = 0; $i <= $#prefs_contents; $i++)
 
  if($prefs_contents[$i] =~ /max_ncpus_pct/)
   
    print OUT "   <max_ncpus_pct>$percent.000000</max_ncpus_pct>\n";
   
  else
   
    print OUT $prefs_contents[$i];
   
 
close(OUT);
rename "$prefs.new", "$prefs" or die "can't rename";
system("boinccmd --read_global_prefs_override");
syslog("info", "Temp $cur_temp" . "C, set percentage to $percent");

4 January 2021

Iustin Pop: Year 2020 review

Year 2020. What a year! Sure, already around early January there were rumours/noise about Covid-19, but who would have thought where it will end up! Thankfully, none of my close or extended family was directly (medically) affected by Covid, so I/we had a privileged year compared to so many other people. I thought how to write a mini-summary, but prose is too difficult, so let s just go month-by-month. Please note that my memory is fuzzy after 9 months cooked up in the apartment, so things could 1 month compared to what I wrote.

Timeline

January Ski weekend. Skiing is awesome! Cancelling a US work trip since there will be more opportunities soon (har har!).

February Ski vacation. Yep, skiing is awesome. Can t wait for next season (har har!). Discussions about Covid start in the office, but more is this scary or just interesting? (yes, this was before casualties). Then things start escalating, work-from-home at least partially, etc. etc. Definitely not just intersting anymore. In Garmin-speak, I got ~700+ intensity minutes in February (correlates with activity time, but depends on intensity of the effort whether 1:1 or 2 intensity minutes for one wall-clock minute).

March Sometimes during the month, my workplace introduces mandatory WFH. I remember being the last person in our team in the office, on the last day we were allowed to work, and cleaning my desk/etc., thinking all this, and we ll be back in 3 weeks or so . Har har! I buy a webcam, just in case WFH gets extended. And start to increase my sports - getting double the intensity minutes (1500+).

April Switzerland enters the first, hard, lockdown. Or was it late March? Not entirely sure, but in my mind March was the opening, and April was the first main course. It is challenging, having to juggle family and work and stressed schedule, but also interesting. Looking back, I think I liked April the most, as people were actually careful at that time. I continue upgrading my home office - new sound system, so that I don t have to plug in/plug out cables. 1700+ intensity minutes this month.

May Continued WFH, somewhat routine now. My then internet provider started sucking hard, so I upgrade with good results. I m still happy, half a year later (quite happy, even). Still going strong otherwise, but waiting for summer vacation, whatever it will be. A tiny bit more effort, so 1800 intensity minutes in May.

June Switzerland relaxes the lock down, but not my company, so as the rest of the family goes out and about, I start feeling alone in the apartment. And somewhat angry at it, which impacts my sports (counter-intuitively), so I only get 1500 intensity minutes. I go and buy a coffee machine a real one, that takes beans and grinds them, so I get to enjoy the smell of freshly-ground coffee and the fun of learning about coffee beans, etc. But it occupies the time. On the work/job front, I think at this time I finally got a workstation for home, instead of a laptop (which was ultra-portable too), so together with the coffee machine, it feels like a normal work environment. Well, modulo all the people. At least I m not crying anymore every time I open a new tab in Chrome

July Situation is slowly going better, but no, not my company. Still mandatory WFH, with (if I recall correctly) one day per week allowed, and no meeting other people. I get angrier, but manage to channel my energy into sports, almost doubly my efforts in July - 2937 intensity minutes, not quite reaching the 3000 magic number. I buy more stuff to clean and take care of my bicycles, which I don t really use. So shopping therapy too.

August The month starts with a one week family vacation, but I take a bike too, so I manage to put in some effort (it was quite nice riding TBH). A bit of changes in the personal life (nothing unexpected), which complicates things a bit, but at this moment I really thought Switzerland is going to continue to decrease in infections/R-factor/etc. so things will get back to normal, right? My company expands a bit the work-from-office part, so I m optimistic. Sports wise, still going strong, 2500 intensity minutes, preparing for the single race this year.

September The personal life changes from August start to stabilise, so things become again routine, and I finally get to do a race. Life was good for an extended weekend (well, modulo race angst, but that s part of the fun), and I feel justified to take it slow the week after the race. And the week after that too. I end up the month with close, but not quite, 1900 intensity minutes.

October October starts with school holidays and a one week family vacation, but I feel demotivated. Everything is closing down again (well, modulo schools), and I actually have difficulty getting re-adjusted to no longer being alone in the apartment during the work hours. I only get ~1000 intensity minutes in October, mainly thanks to good late autumn weather and outside rides. And I start playing way more computer games. I also sell my PS4, hoping to get a PS5 next month.

November November continues to suck. I think my vacation in October was actually detrimental - it broke my rhythm, I don t really do sport anymore, not consistently at least, so I only get 700+ intensity minutes. And I keep playing computer games, even if I missed the PS5 ordering window; so I switch to PC gaming. My home office feels very crowded, so as kind of anti-shopping therapy, I sell tons of smallish stuff; can t believe how much crap I kept around while not really using it. I also manage to update/refresh all my Debian packages, since next freeze approaches. Better than for previous releases, so it feels good.

December December comes, end of the year, the much awaited vacation - which we decide to cancel due to the situation in whole of Switzerland (and neighbouring countries). I basically only play computer games, and get grand total of 345 activity minutes this month. And since my weight is inversely correlated to my training, I m basically back at my February weight, having lost all the gains I made during the year. I mean, having gained back all the fat I lost. Err, you know what I mean; I m back close to my high-watermark, which is not good.

Conclusion I was somehow hoping that the end of the year will allow me to reset and restart, but somehow - a few days into January - it doesn t really feel so. My sleep schedule is totally ruined, my motivation is so-so, and I think the way I crashed in October was much harder/worse than I realised at the time, but in a way expected for this crazy year. I have some projects for 2021 - or at least, I m trying to make up a project list - in order to get a bit more structure in my continued stuck inside the house part, which is especially terrible when on-call. I don t know how the next 3-6 months will evolve, but I m thankful that so far, we are all healthy. Actually, me personally I ve been healthier physically than in other years, due to less contact with other people. On the other side, thinking of all the health-care workers, or even service workers, my IT job is comfy and all I am is a spoiled person (I could write many posts on specifically this topic). I really need to up my willpower and lower my spoil level. Hints are welcome :( Wish everybody has a better year in 2021.

John Goerzen: More Topics on Store-And-Forward (Possibly Airgapped) ZFS and Non-ZFS Backups with NNCP

Note: this is another article in my series on asynchronous communication in Linux with UUCP and NNCP. In my previous post, I introduced a way to use ZFS backups over NNCP. In this post, I ll expand on that and also explore non-ZFS backups. Use of nncp-file instead of nncp-exec The previous example used nncp-exec (like UUCP s uux), which lets you pipe stdin in, then queues up a request to run a given command with that input on a remote. I discussed that NNCP doesn t guarantee order of execution, but that for the ZFS use case, that was fine since zfs receive would just fail (causing NNCP to try again later). At present, nncp-exec stores the data piped to it in RAM before generating the outbound packet (the author plans to fix this shortly). That made it unusable for some of my backups, so I set it up another way: with nncp-file, the tool to transfer files to a remote machine. A cron job then picks them up and processes them. On the machine being backed up, we have to find a way to encode the dataset to be received. I chose to do that as part of the filename, so the updated simplesnap-queue could look like this:
#!/bin/bash
set -e
set -o pipefail
DEST=" echo $1   sed 's,^tank/simplesnap/,,' "
FILE="bakfsfmt2- date "+%s.%N".$$ _ echo "$DEST"   sed 's,/,@,g' "
echo "Processing $DEST to $FILE" >&2
# stdin piped to this
zstd -8 - \
    gpg --compress-algo none --cipher-algo AES256 -e -r 012345...  \
    su nncp -c "/usr/local/nncp/bin/nncp-file -nice B -noprogress - 'backupsvr:$FILE'" >&2
echo "Queued $DEST to $FILE" >&2
I ve added compression and encryption here as well; more on that below. On the backup server, we would define a different incoming directory for each node in nncp.hjson. For instance:
host1:  
...
   incoming: "/var/local/nncp-bakcups-incoming/host1"
 
host2:  
...
   incoming: "/var/local/nncp-backups-incoming/host2"
 
I ll present the scanning script in a bit. Offsite Backup Rotation Most of the time, you don t want just a single drive to store the backups. You d like to have a set. At minimum, one wouldn t be plugged in so lightning wouldn t ruin all your backups. But maybe you d store a second drive at some other location you have access to (friend s house, bank box, etc.) There are several ways you could solve this: The third option can be helped with NNCP, too. One way is to create separate NNCP installations for each of the drives that you store data on. Then, whenever one is plugged in, the appropriate NNCP config will be loaded and appropriate packets received and processed. The neighbor machine the spooler would just store up packets for the offsite drive until it comes back onsite (or, perhaps, your airgapped USB transport would do this). Then when it s back onsite, all the queued up ZFS sends get replayed and the backups replicated. Now, how might you handle this with NNCP? The simple way would be to have each system generating backups send them to two destinations. For instance:
zstd -8 -   gpg --compress-algo none --cipher-algo AES256 -e -r 07D5794CD900FAF1D30B03AC3D13151E5039C9D5 \
    tee >(su nncp -c "/usr/local/nncp/bin/nncp-file -nice B+5 -noprogress - 'backupdisk1:$FILE'") \
        >(su nncp -c "/usr/local/nncp/bin/nncp-file -nice B+5 -noprogress - 'backupdisk2:$FILE'") \
   > /dev/null
You could probably also more safely use pee(1) (from moreutils) to do this. This has an unfortunate result of doubling the network traffic from every machine being backed up. So an alternative option would be to queue the packets to the spooling machine, and run a distribution script from it; something like this, in part:
INCOMINGDIR="/var/local/nncp-bakfs-incoming"
LOCKFILE="$INCOMINGDIR/.lock"
printf -v EVAL_SAFE_LOCKFILE '%q' "$LOCKFILE"
if dotlockfile -r 0 -l -p "$ LOCKFILE "; then
  logit "Lock obtained at $ LOCKFILE  with dotlockfile"
  trap 'ECODE=$?; dotlockfile -u '"$ EVAL_SAFE_LOCKFILE "'; exit $ECODE' EXIT INT TERM
else
  logit "Could not obtain lock at $LOCKFILE; $0 likely already running."
  exit 0
fi
logit "Scanning queue directory..."
cd "$INCOMINGDIR"
for HOST in *; do
   cd "$INCOMINGDIR/$HOST"
   for FILE in bakfsfmt2-*; do
           if [ -f "$FILE" ]; then
                   for BAKFS in backupdisk1 backupdisk2; do
                           runcommand nncp-file -nice B+5 -noprogress "$FILE" "$BAKFS:$HOST/$FILE"
                   done
                   runcommand rm "$FILE"
           else
                   logit "$HOST: Skipping $FILE since it doesn't exist"
           fi
   done
done
logit "Scan complete."
Security Considerations You ll notice that in my example above, the encryption happens as the root user, but nncp is called under su. This means that even if there is a vulnerability in NNCP, the data would still be protected by GPG. I ll also note here that many sites run ssh as root unnecessarily; the same principles should apply there. (ssh has had vulnerabilities in the past as well). I could have used gpg s built-in compression, but zstd is faster and better, so we can get good performance by using fast compression and piping that to an algorithm that can use hardware acceleration for encryption. I strongly encourage considering transport, whether ssh or NNCP or UUCP, to be untrusted. Don t run it as root if you can avoid it. In my example, the nncp user, which all NNCP commands are run as, has no access to the backup data at all. So even if NNCP were compromised, my backup data wouldn t be. For even more security, I could also sign the backup stream with gpg and validate that on the receiving end. I should note, however, that this conversation assumes that a network- or USB-facing ssh or NNCP is more likely to have an exploitable vulnerability than is gpg (which here is just processing a stream). This is probably a safe assumption in general. If you believe gpg is more likely to have an exploitable vulnerability than ssh or NNCP, then obviously you wouldn t take this particular approach. On the zfs side, the use of -F with zfs receive is avoided; this could lead to a compromised backed-up machine generating a malicious rollback on the destination. Backup zpools should be imported with -R or -N to ensure that a malicious mountpoint property couldn t be used to cause an attack. I choose to use zfs receive -u -o readonly=on which is compatible with both unmounted backup datasets and zpools imported with -R (or both). To access the data in a backup dataset, you would normally clone it and access it there. The processing script So, put this all together and look at an example of a processing script that would run from cron as root and process the incoming ZFS data.
#!/bin/bash
set -e
set -o pipefail
# Log a message
logit ()  
   logger -p info -t " basename "$0" [$$]" "$1"
 
# Log an error message
logerror ()  
   logger -p err -t " basename "$0" [$$]" "$1"
 
# Log stdin with the given code.  Used normally to log stderr.
logstdin ()  
   logger -p info -t " basename "$0" [$$/$1]"
 
# Run command, logging stderr and exit code
runcommand ()  
   logit "Running $*"
   if "$@" 2> >(logstdin "$1") ; then
      logit "$1 exited successfully"
      return 0
   else
       RETVAL="$?"
       logerror "$1 exited with error $RETVAL"
       return "$RETVAL"
   fi
 
STORE=backups/simplesnap
INCOMINGDIR=/backups/nncp/incoming
if ! [ -d "$INCOMINGDIR" ]; then
        logerror "$INCOMINGDIR doesn't exist"
        exit 0
fi
LOCKFILE="/backups/nncp/.nncp-backups-zfs-scan.lock"
printf -v EVAL_SAFE_LOCKFILE '%q' "$LOCKFILE"
if dotlockfile -r 0 -l -p "$ LOCKFILE "; then
  logit "Lock obtained at $ LOCKFILE  with dotlockfile"
  trap 'ECODE=$?; dotlockfile -u '"$ EVAL_SAFE_LOCKFILE "'; exit $ECODE' EXIT INT TERM
else
  logit "Could not obtain lock at $LOCKFILE; $0 likely already running."
  exit 0
fi
EXITCODE=0
cd "$INCOMINGDIR"
logit "Scanning queue directory..."
for HOST in *; do
    HOSTPATH="$INCOMINGDIR/$HOST"
    # files like backupsfmt2-134.13134_dest
    for FILE in "$HOSTPATH"/backupsfmt2-[0-9]*_?*; do
        if [ ! -f "$FILE" ]; then
            logit "Skipping non-existent $FILE"
            continue
        fi
        # Now, $DEST will be HOST/DEST.  Strip off the @ also.
        DEST=" echo "$FILE"   sed -e 's/^.*backupsfmt2[^_]*_//' -e 's,@,/,g' "
        if [ -z "$DEST" ]; then
            logerror "Malformed dest in $FILE"
            continue
        fi
        HOST2=" echo "$DEST"   sed 's,/.*,,g' "
        if [ -z "$HOST2" ]; then
            logerror "Malformed DEST $DEST in $FILE"
            continue
        fi
        if [ ! "$HOST" = "$HOST2" ]; then
            logerror "$DIR: $HOST doesn't match $HOST2"
            continue
        fi
        logit "Processing $FILE to $STORE/$DEST"
            if runcommand gpg -q -d < "$FILE"   runcommand zstdcat   runcommand zfs receive -u -o readonly=on "$STORE/$DEST"; then
                logit "Successfully processed $FILE to $STORE/$DEST"
                runcommand rm "$FILE"
        else
                logerror "FAILED to process $FILE to $STORE/$DEST"
                EXITCODE=15
        fi
Applying These Ideas to Non-ZFS Backups ZFS backups made our job easier in a lot of ways: Some of these benefits you just won't get without ZFS (or something similar like btrfs), but let's see how we could apply these ideas to non-ZFS backups. I will explore the implementation of them in a future post. When I say "non ZFS", I am being a bit vague as to whether the source, the destination, or both systems are running a non-ZFS filesystem. In general I'll assume that neither are ZFS. The first and most obvious answer is to just tar up the whole system and send that every day. This is, of course, only suitable for small datasets on a fast network. These tarballs could be unpacked on the destination and stored more efficiently via any number of methods (hardlink trees, a block-level deduplicator like borg or rdedup, or even just simply compressed tarballs). To make the network trip more efficient, something like rdiff or xdelta could be used. A signature file could be stored on the machine being backed up (generated via tee/pee at stream time), and the next run could simply send an rdiff delta over NNCP. This would be quite network-efficient, but still would require reading every byte of every file on every backup, and would also require quite a bit of temporary space on the receiving end (to apply the delta to the previous tarball and generate a new one). Alternatively, a program that generates incremental backup files such as rdup could be used. These could be transmitted over NNCP to the backup server, and unpacked there. While perhaps less efficient on the network -- every file with at least one modified byte would be retransmitted in its entirety -- it avoids the need to read every byte of unmodified files or to have enormous temporary space. I should note here that GNU tar claims to have an incremental mode, but it has a potential data loss bug. There are also some tools with algorithms that may apply well in this use care: syrep and fssync being the two most prominent examples, though rdedup (mentioned above) and the nascent asuran project may also be combinable with other tools to achieve this effect. I should, of course, conclude this section by mentioning btrfs. Every time I've tried it, I've run into serious bugs, and its status page indicates that only some of them have been resolved. I would not consider using it for something as important as backups. However, if you are comfortable with it, it is likely to be able to run in more constrained environments than ZFS and could probably be processed in much the same way as zfs streams.

2 January 2021

Paul Wise: FLOSS Activities December 2020

Focus This month I didn't have any particular focus. I just worked on issues in my info bubble.

Changes

Issues

Review

Administration
  • Debian: restart bacula director, ping some people about disk usage
  • Debian wiki: unblock IP addresses, approve accounts, update email for accounts with bouncing email

Communication
  • Respond to queries from Debian users and contributors on the mailing lists and IRC

Sponsors All work was done on a volunteer basis.

1 January 2021

Jonathan Wiltshire: WordPress in a subdirectory

For many years now I ve had WordPress installed as a subdirectory to my site but appearing to at the domain level, i.e. /wordpress/index.php is transparently presented as the homepage. This is done by setting the WordPress Address and Site Address settings and then mapping requests which do not match an existing file or directory through as a PHP pathinfo using Apache s mod_rewrite rules in a .htaccess file or server configuration. In this way most of the site is WordPress s dynamic pages and posts, but WordPress itself is neatly contained and random static resources such as /screentest/1024.GIF work as expected. Those .htaccess rules were originally hand-crafted but didn t take account of changing recommendations, e.g. the Authorization header. When I rearranged matters recently I decided to take advantage of WordPress s own generated rules and ditch my old rules. They look like this:
# BEGIN WordPress
# The directives (lines) between "BEGIN WordPress" and "END WordPress" are
# dynamically generated, and should only be modified via WordPress filters.
# Any changes to the directives between these markers will be overwritten.
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteRule .* - [E=HTTP_AUTHORIZATION:% HTTP:Authorization ]
RewriteBase /
RewriteRule ^index\.php$ - [L]
RewriteCond % REQUEST_FILENAME  !-f
RewriteCond % REQUEST_FILENAME  !-d
RewriteRule . /index.php [L]
</IfModule>
# END WordPress
If .htaccess is writable, WordPress adds these rules automatically. Unfortunately, when I switched to them and not my own, the site broke. Crucially the generated rules do not cater for WordPress being in a subdirectory. No matter what combination of settings I tried, local changes to the rules (like RewriteRule . /wordpress/index.php [L]) were rewritten back to the original values at random times and clearly couldn t be relied upon to stay as I need them. This feels like a bug: my intuition is that if the WordPress Address and Site Address settings are different, but within the same domain, the rules should be generated to take care of that. One option is to make the .htaccess file read-only, which WordPress detects and avoids trying to make changes at all. But this rather defeats the object of letting it take care of future changes as the software and specifications change. The second option, which I stuck with, is to add an index.php file with the following contents:
<?php
include('wordpress/index.php');
Now the generated rules, though incorrect, result in the right behaviour and won t break on every setting change.

Russ Allbery: 2020 Book Reading in Review

In 2020, I finished and reviewed 42 books, two more than 2019 (although I had finished but not reviewed two books at the end of 2019, so the total is even more similar than that). This is the best year for reading in terms of book count since 2012, despite being seriously distracted by a new job, a pandemic, and US political meltdowns. Those distractions do show up in the drop in page count. If it weren't for the pandemic, the count would have been higher. Just as I got into a rhythm of reading while I exercised, gyms became a bad idea for the rest of the year. Treadmills are good for reading; long walks around the neighborhood not so much. That time went to podcasts instead, which I'm not too sad about but which don't prompt reviews. Finding the mental space and energy to write reviews proved as much of a challenge as finding time to read this year, and I once again had to do some catch-up at the end of the year. To the extent that I have goals for 2021, it's to tighten up the elapsed time between finishing a book and writing a review so that the reviews don't pile up. I read one book this year that I rated 10 out of 10: Michael Lewis's The Fifth Risk, which is partly about the US presidential transition and is really about what the US government does and what sort of people make careers in civil service. This book is brilliant, fascinating, and surprisingly touching, and I wish it were four times as long. If anything, it's even more relevant today as we enter another transition than it was when Lewis wrote it or when I read it. There were so many 9 out of 10 ratings this year that it's hard to know where to start. I read the last Murderbot novella by Martha Wells (Exit Strategy) and then the first Murderbot novel (Network Effect), both of which were everything I was hoping for. Murderbot's sarcastic first-person voice continues to be a delight, and I expect Network Effect to take home several 2021 awards. I'm eagerly awaiting the next novel, Fugitive Telemetry, currently scheduled for the end of April, 2021. Also on the fiction side were Alix E. Harrow's wonderful debut novel The Ten Thousand Doors of January, a fierce novel about family and claiming power that will hopefully win the 2020 Mythopoeic Award (which was delayed by the pandemic), and TJ Klune's heart-warming The House in the Cerulean Sea, my feel-good novel of the year. Finally, Tamsyn Muir's Gideon the Ninth and Harrow the Ninth were a glorious mess in places, but I had more fun reading and discussing those books than I've had with any novel in a very long time. On the non-fiction side, Tressie McMillan Cottom's Thick is the best collection of sociology that I've read. It's not easy reading, but that book gave me a new-found appreciation and understanding of sociology and what it's trying to accomplish. Gretchen McCulloch's Because Internet is much easier reading but similarly perspective-enhancing, helping me understand (among other things) how choice of punctuation and capitalization expands the dynamic range of meaning in informal text conversation. Finally, Nick Pettigrew's Anti-Social is a funny, enlightening, and sobering look at the process of addressing low-level unwanted behavior that's directly relevant to the current conflicts over the role of policing in society. The full analysis includes some additional personal reading statistics, probably only of interest to me.

Dmitry Shachnev: A review of endianness bugs in Qt, and how they were fixed

As you may know, I am Qt 5 maintainer in Debian. Maintaning Qt means not only bumping the version each time a new version is released, but also making sure Qt builds successfully on all architectures that are supported in Debian (and for some submodules, the automatic tests pass). An important sort of build failures are endianness specific failures. Most widely used architectures (x86_64, aarch64) are little endian. However, Debian officially supports one big endian architecture (s390x), and unofficially a few more ports are provided, such as ppc64 and sparc64. Unfortunately, Qt upstream does not have any big endian machine in their CI system, so endianness issues get noticed only when the packages fail to build on our build daemons. In the last years I have discovered and fixed some such issues in various parts of Qt, so I decided to write a post to illustrate how to write really cross-platform C/C++ code. Issue 1: the WebP image format handler (code review) The relevant code snippet is:
if (srcImage.format() != QImage::Format_ARGB32)
    srcImage = srcImage.convertToFormat(QImage::Format_ARGB32);
// ...
if (!WebPPictureImportBGRA(&picture, srcImage.bits(), srcImage.bytesPerLine()))  
    // ...
 
The code here is serializing the images into QImage::Format_ARGB32 format, and then passing the bytes into WebP s import function. With this format, the image is stored using a 32-bit ARGB format (0xAARRGGBB). This means that the bytes will be 0xBB, 0xGG, 0xRR, 0xAA or little endian and 0xAA, 0xRR, 0xGG, 0xBB on big endian. However, WebPPictureImportBGRA expects the first format on all architectures. The fix was to use QImage::Format_RGBA8888. As the QImage documentation says, with this format the order of the colors is the same on any architecture if read as bytes 0xRR, 0xGG, 0xBB, 0xAA. Issue 2: qimage_converter_map structure (code review) The code seems to already support big endian. But maybe you can spot the error?
#if Q_BYTE_ORDER == Q_LITTLE_ENDIAN
        0,
        convert_ARGB_to_ARGB_PM,
#else
        0,
        0
#endif
It is the missing comma! It is present in the little endian block, but not in the big endian one. This was fixed trivially. Issue 3: QHandle, part of Qt 3D module (code review) QHandle class uses a union that is declared as follows:
struct Data  
    quint32 m_index : IndexBits;
    quint32 m_counter : CounterBits;
    quint32 m_unused : 2;
 ;
union  
    Data d;
    quint32 m_handle;
 ;
The sizes are declared such as IndexBits + CounterBits + 2 is always equal to 32 (four bytes). Then we have a constructor that sets the values of Data struct:
QHandle(quint32 i, quint32 count)
 
    d.m_index = i;
    d.m_counter = count;
    d.m_unused = 0;
 
The value of m_handle will be different depending on endianness! So the test that was expecting a particular value with given constructor arguments was failing. I fixed it by using the following macro:
#if Q_BYTE_ORDER == Q_BIG_ENDIAN
#define GET_EXPECTED_HANDLE(qHandle) ((qHandle.index() << (qHandle.CounterBits + 2)) + (qHandle.counter() << 2))
#else /* Q_LITTLE_ENDIAN */
#define GET_EXPECTED_HANDLE(qHandle) (qHandle.index() + (qHandle.counter() << qHandle.IndexBits))
#endif
Issue 4: QML compiler (code review) The QML compiler used a helper class named LEUInt32 (based on QLEInteger) that always stored the numbers in little endian internally. This class can be safely mixed with native quint32 on little endian systems, but not on big endian. Usually the compiler would warn about type mismatch, but here the code used reinterpret_cast, such as:
quint32 *objectTable = reinterpret_cast<quint32*>(data + qmlUnit->offsetToObjects);
So this was not noticed on build time, but the compiler was crashing. The fix was trivial again, replacing quint32 with QLEUInt32. Issue 5: QModbusPdu, part of Qt Serial Bus module (code review) The code snippet is simple:
QModbusPdu::FunctionCode code = QModbusPdu::Invalid;
if (stream.readRawData((char *) (&code), sizeof(quint8)) != sizeof(quint8))
    return stream;
QModbusPdu::FunctionCode is an enum, so code is a multi-byte value (even if only one byte is significant). However, (char *) (&code) returns a pointer to the first byte of it. It is the needed byte on little endian systems, but it is the wrong byte on big endian ones! The correct fix was using a temporary one-byte variable:
quint8 codeByte = 0;
if (stream.readRawData((char *) (&codeByte), sizeof(quint8)) != sizeof(quint8))
    return stream;
QModbusPdu::FunctionCode code = (QModbusPdu::FunctionCode) codeByte;
Issue 6: qt_is_ascii (code review) This function, as the name says, checks whether a string is ASCII. It does that by splitting the string into 4-byte chunks:
while (ptr + 4 <= end)  
    quint32 data = qFromUnaligned<quint32>(ptr);
    if (data &= 0x80808080U)  
        uint idx = qCountTrailingZeroBits(data);
        ptr += idx / 8;
        return false;
     
    ptr += 4;
 
idx / 8 is the number of trailing zero bytes. However, the bytes which are trailing on little endian are actually leading on big endian! So we can use qCountLeadingZeroBits there. Issue 7: the bundled copy of tinycbor (upstream pull request) Similar to issue 5, the code was reading into the wrong byte:
if (bytesNeeded <= 2)  
    read_bytes_unchecked(it, &it->extra, 1, bytesNeeded);
    if (bytesNeeded == 2)
        it->extra = cbor_ntohs(it->extra);
 
extra has type uint16_t, so it has two bytes. When we need only one byte, we read into the wrong byte, so the resulting number is 256 times higher on big endian than it should be. Adding a temporary one-byte variable fixed it. Issue 8: perfparser, part of Qt Creator (code review) Here it is not trivial to find the issue just looking at the code:
qint32 dataStreamVersion = qToLittleEndian(QDataStream::Qt_DefaultCompiledVersion);
However the linker was producing an error:
undefined reference to QDataStream::Version qbswap(QDataStream::Version)'
On little endian systems, qToLittleEndian is a no-op, but on big endian systems, it is a template function defined for some known types. But it turns out we need to explicitly convert enum values to a simple type, so the fix was passing qint32(QDataStream::Qt_DefaultCompiledVersion) to that function. Issue 9: Qt Personal Information Management (code review) The code in test was trying to represent a number as a sequence of bytes, using reinterpret_cast:
static inline QContactId makeId(const QString &managerName, uint id)
 
    return QContactId(QStringLiteral("qtcontacts:basic%1:").arg(managerName), QByteArray(reinterpret_cast<const char *>(&id), sizeof(uint)));
 
The order of bytes will be different on little endian and big endian systems! The fix was adding this line to the beginning of the function:
id = qToLittleEndian(id);
This will cause the bytes to be reversed on big endian systems. What remains unfixed There are still some bugs, which require deeper investigation, for example: P.S. We are looking for new people to help with maintaining Qt 6. Join our team if you want to do some fun work like described above!

Keith Packard: kgames

Reviving Very Old X Code I've taken the week between Christmas and New Year's off this year. I didn't really have anything serious planned, just taking a break from the usual routine. As often happens, I got sucked into doing a project when I received this simple bug report Debian Bug #974011
I have been researching old terminal and X games recently, and realized
that much of the code from 'xmille' originated from the terminal game
'mille', which is part of bsdgames.
...
[The copyright and license information] has been stripped out of all
code in the xmille distribution.  Also, none of the included materials
give credit to the original author, Ken Arnold.
The reason the 'xmille' source is missing copyright and license information from the 'mille' files is that they were copied in before that information was added upstream. Xmille forked from Mille around 1987 or so. I wrote the UI parts for the system I had at the time, which was running X10R4. A very basic port to X11 was done at some point, and that's what Debian has in the archive today. At some point in the 90s, I ported Xmille to the Athena widget set, including several custom widgets in an Xaw extension library, Xkw. It's a lot better than the version in Debian, including displaying the cards correctly (the Debian version has some pretty bad color issues). Here's what the current Debian version looks like: Fixing The Bug To fix the missing copyright and license information, I imported the mille source code into the "latest" Xaw-based version. The updated mille code had a number of bug fixes and improvements, along with the copyright information. That should have been sufficient to resolve the issue and I could have constructed a suitable source package from whatever bits were needed and and uploaded that as a replacement 'xmille' package. However, at some later point, I had actually merged xmille into a larger package, 'kgames', which also included a number of other games, including Reversi, Dominoes, Cribbage and ten Solitaire/Patience variants. (as an aside, those last ten games formed the basis for my Patience Palm Pilot application, which seems to have inspired an Android App of the same name...) So began my yak shaving holiday. Building Kgames in 2020 Ok, so getting this old source code running should be easy, right? It's just a bunch of C code designed in the 80s and 90s to work on VAXen and their kin. How hard could it be?
  1. Everything was a 32-bit computer back then; pointers and ints were both 32 bits, so you could cast them with wild abandon and cause no problems. Today, testing revealed segfaults in some corners of the code.
  2. It's K&R C code. Remember that the first version of ANSI-C didn't come out until 1989, and it was years later that we could reliably expect to find an ANSI compiler with a random Unix box.
  3. It's X11 code. Fortunately (?), X11 hasn't changed since these applications were written, so at least that part still works just fine. Imagine trying to build Windows or Mac OS code from the early 90's on a modern OS...
I decided to dig in and add prototypes everywhere; that found a lot of pointer/int casting issues, as well as several lurking bugs where the code was just plain broken. After a day or so, I had things building and running and was no longer hitting crashes. Kgames 1.0 uploaded to Debian New Queue With that done, I decided I could at least upload the working bits to the Debian archive and close the bug reported above. kgames 1.0-2 may eventually get into unstable, presumably once the Debian FTP team realizes just how important fixing this bug is. Or something. Here's what xmille looks like in this version: And here's my favorite solitaire variant too: But They Look So Old Yeah, Xaw applications have a rustic appearance which may appeal to some, but for people with higher resolution monitors and well seasoned eyesight, squinting at the tiny images and text makes it difficult to enjoy these games today. How hard could it be to update them to use larger cards and scalable fonts? Xkw version 2.0 I decided to dig in and start hacking the code, starting by adding new widgets to the Xkw library that used cairo for drawing instead of core X calls. Fortunately, the needs of the games were pretty limited, so I only needed to implement a handful of widgets: The other Xkw widgets all got their rendering switched to using cairo, plus using double buffering to make updates look better. SVG Playing Cards Looking on wikimedia, I found a page referencing a large number of playing cards in SVG form That led me to Adrian Kennard's playing card web site that let me customize and download a deck of cards, licensed using the CC0 Public Domain license. With these cards, I set about rewriting the Xkw playing card widget, stripping out three different versions of bitmap playing cards and replacing them with just these new SVG versions. SVG Xmille Cards Ok, so getting regular playing cards was good, but the original goal was to update Xmille, and that has cards hand drawn by me. I could just use those images, import them into cairo and let it scale them to suit on the screen. I decided to experiment with inkscape's bitmap tracing code to see what it could do with them. First, I had to get them into a format that inkscape could parse. That turned out to be a bit tricky; the original format is as a set of X bitmap layers; each layer painting a single color. I ended up hacking the Xmille source code to generate the images using X, then fetching them with XGetImage and walking them to construct XPM format files which could then be fed into the portable bitmap tools to create PNG files that inkscape could handle. The resulting images have a certain charm: I did replace the text in the images to make it readable, otherwise these are untouched from what inkscape generated. The Results Remember that all of these are applications built using the venerable X toolkit; there are still some non-antialiased graphics visible as the shaped buttons use the X Shape extension. But, all rendering is now done with cairo, so it's all anti-aliased and all scalable. Here's what Xmille looks like after the upgrades: And here's spider: Once kgames 1.0 reaches Debian unstable, I'll upload these new versions.

31 December 2020

Chris Lamb: Free software activities in December 2020

Here is my monthly update covering what I have been doing in the free software world during December 2020 (previous month):

Reproducible Builds One of the original promises of open source software is that distributed peer review and transparency of process results in enhanced end-user security. However, whilst anyone may inspect the source code of free and open source software for malicious flaws, almost all software today is distributed as pre-compiled binaries. This allows nefarious third-parties to compromise systems by injecting malicious code into ostensibly secure software during the various compilation and distribution processes. The motivation behind the Reproducible Builds effort is to ensure no flaws have been introduced during this compilation process by promising identical results are always generated from a given source, thus allowing multiple third-parties to come to a consensus on whether a build was compromised. This month, I: I also made the following changes to diffoscope, our in-depth and content-aware diff utility that can locate and diagnose reproducibility issues, including releasing version 163:

Debian Uploads I also sponsored an upload of adminer (4.7.8-2) on behalf of Alexandre Rossi and performed two QA uploads of sendfile (2.1b.20080616-7 and 2.1b.20080616-8) to make the build the build reproducible (#776938) and to fix a number of other unrelated issues. Debian LTS This month I have worked 18 hours on Debian Long Term Support (LTS) and 12 hours on its sister Extended LTS project. You can find out more about the Debian LTS project via the following video:

Shirish Agarwal: Pandemic, Informal economy and Khau Galli.

Formal sector issues Just today it was published that output from eight formal sectors of the economy who make the bulk of the Indian economy were down on a month to month basis . This means all those apologists for the Government who said that it was ok that the Govt. didn t give the 20 lakh crore package which was announced. In fact, a businessman from my own city, a certain Prafull Sarda had asked via RTI what happened to the 20 lakh crore package which was announced? The answers were in all media as well as newspapers but on the inside pages. You can see one of the article sharing the details here. No wonder Vivek Kaul also shared his take on the way things will hopefully go for the next quarter which seems to be a tad optimistic from where we are atm.
Eight Sectors declining in Indian Economy month-on-month CNBC TV 18
The Informal economy The informal economy has been strangulated by the current party in power since it came into power. And this has resulted many small businesses which informal are and were part of culture of cities and towns. I share an article from 2018 which only shows how good and on the mark it has aged in the last two years. The damage is all to real to ignore as I would share more of an anecdotes and experiences because sadly there never has been any interest shown especially by GOI to seek any stats. about informal economy. Although CMIE has done some good work in that even though they majorly look at formal, usually blue-collar work where again there is not good data. Sharing an anecdote and a learning from these small businesses which probably an MBA guy wouldn t know and in all honesty wouldn t even care. Khau galli Few years back, circa 2014 and on wards, when the present Govt. came into power, it did come with lot of promises. One of which was that lot of informal businesses would be encouraged to grow their businesses and in time hopefully, they become part of the formal economy. Or at least that was the story that all of us were told. Due to that they did lot of promises and also named quite a few places where street food was in abundance. Such lanes were named Khau galli for those who are from North India, it was be easily known and understood. This was just saying that here are some places where you could get a variety of food without paying obscene prices as you would have to vis-a-vis a restaurant. Slowly, they raised the rates of inputs (food grains), gas cylinder etc. which we know of as food inflation and via GST made sure that the restaurants were able to absorb some of the increased inputs (input credit) while still being more than competitive to the street food person/s. The restaurant F&B model is pretty well known so not going there at all. It is however, important to point out that they didn t make any new khau gallis or such, most or all the places existed for years and even decades before that. They also didn t take any extra effort either in marketing the khau gallis or get them with chefs or marketing folks so that the traditional can marry to the anew. They just named them, that was the only gain to be seen on the ground. In its heyday, the khau galli near my home used to have anywhere between 20-30 Thelas or food carts. Most of the food carts would be of wood and having very limited steel. Such food carts would cost anywhere around INR 15-20k instead of the food cart you see here. The only reason I shared that link is to show how a somewhat typical thela or food cart looks in India. Of course YouTube or any other media platform would show many. On top of it, you need and needed permission from the municipality a license for the same which would be auctioned. Now that license could well run from thousands into lakhs depending on various factors or you gave something to the Municipal worker when he did his rounds/beat much like a constable every day or week. Apart from those, you also have raw material expenditure which could easily run into few thousands depending upon what sort of food you are vending. You also would typically have 2-3 workers so a typical Thela would feed not only its customers but also 2-3 families who are the laborer families as well as surrounding businesses. As I used to be loyal and usually go to few whom I found to be either tasty or somehow they were good for me. In either case, a relationship was formed. As I have been never fond of crowds, I usually used to in their off-beat hours either when they are close to packing up of when I know they usually have a lull. That way I knew I would get complete attention of the vendor/s. Many a times I used to see money change hands between the vendors themselves and used to see both camaraderie as well as competition between them. This is years ago, once while sharing a chai (tea) with one of the street vendors I casually asked I have often seen you guys exchanging money with each other and most of the time quite a bit of the money is also given to the guy who didn t make that much sales or any sales at all. The vendor replied sharing practical symbiotic knowledge. All of us are bound by a single thing called poverty. All of us are struggle. Do you know why so many people come here, because they know that there would be a variety of food to be had. Now if we stopped helping each other, the number of people who would make the effort would be also less, we know we are not the only game in town. Also whatever we give, sooner or later it gets adjusted. Also if one of us has good days, he knows hardship days are not far. Why, simply because people change tastes or want variety. So irrespective of good or bad the skills of the vendor is, she or he is bound to make some sales. The vendor either shares the food with us or whatever. Somehow these things just work out. And that doesn t mean we don t have our fights, we do have our fights, but we also understand this. Now you see this and you understand that these guys have and had a community. Even if they changed places due to one reason or the other, they kept themselves connected. Unlike many of us, who even find a hard time keeping up with friends let alone relatives. Now cut to 2020, and where there used to be 20-30 thelas near my home, there are only 4-5. Of course, multiple reasons, but one of the biggest was of course demonetization. That was a huge shock to which many of thela walas succumbed. Their entire savings and capital were turned to dust. Many of their customers will turn up with either a INR 500 or INR 2000/- Re note where at the most a dish costed INR 100/- most times half or even 1/3rd of that amount. How and from where the thela walas could get that kind of cash. These are people who only if they earn, they and their family will have bread at night. Most of the loose change was tied up at middle to higher tier restaurants where they were giving between INR 20/- 30/- for every INR 100/- change of rupees and coins. Quite a few bankers made money by that as well as other means where the thela walas just could not compete. These guys also didn t have any black money even though they were and are part of the black/informal economy. Sadly, till date no economist or even sociologist as far as I know has attempted or done any work from what I know on this industry. If you want to formalize such businesses then at the very least understand their problems and devise solutions. And I suspect, what is and has happened near my house has also happened everywhere else, at least within the geographical confines of the Indian state. Whether it was the 2016 demonetization or the pandemic, the results and effects have been similar the same all over. Some states did do well and still do, the suffering still continues. With the hope that the new year brings cheer to you as well some more ideas to remain in business by the thela walas, I bid you adieu and see you in new year

29 December 2020

Shirish Agarwal: Inequality in Indian Education

Farmer on-going protests Before I start with the education system in India which I have talked about many times in the past, first let me share couple of pieces about the farmer movement which is still at Sanghu Delhi border.
<Manjeet Kaur,62 at farmer protest with her friends.
The above picture became somewhat viral as it showed Manjeet Kaur, who drove down from Patiala, Punjab to Sanghu border along with her friends to take part in the on-going protests. The picture not only shares how the women are part and parcel of this protest but also they are independently taking part in the protest. The other were two articles I read today, first was an article in tribune which questions that if the policy worked, why it didn t work in the state of Bihar. The other by a young law student who had to go from Chandigarh to Delhi with family for some work and her experience with the ongoing protest. In fact, an interesting observation was made by the CJI in the many suits against farmer protests in the SC. This makes for much more interesting read when you see an RTI query filed by Saket Gokhale to NHAI , a Central Govt. agency which is supposed to be independent and asks if they had filed an FIR and asked compensation from Haryana State and Haryana State Police which had dug up National Highway 44 and if any permission was asked for the same from NHAI. And NHAI unable to take any action for the same. If this isn t shameful, I dunno what is
Saket Gokhale s RTI query on digging up NH44
NHAI response to Saket Gokhale s query.
Sadly, the way the response has been worded makes it impossible for NHAI to discharge its own responsibilities and this becomes a precedent for other states now that know that NHAI is vulnerable. A pretty sad turn on events. Indian Education can t go online There was a recent article on scroll which shared how Indian education can t go online as only a few have computers with decent netlink speeds as well as other factors which are needed for online education. But there are also many things that the article doesn t take into account which actually make the task more difficult and raise the boundary more. Now in most schools and colleges, the number of students to teacher ratio could be anywhere between 70-150 or even more. In the last few years, a lot of schools have been closed down by various Governments, including and not limited to the ruling Govt. They have in fact intensified closures of public schools wherever their Govt. has been in power. Closing to 5000+ schools in one state in a year is a dramatic shift and such has been happening time and again. In fact, the rising costs of Indian education has made many to leave Indian shores and do studies abroad. And once they do their masters or whatever, the chances of them coming back to India become more and more remote. In India the costs have been becoming so bad that NBFC s have started products targeting the same. How NBFC and Banks have (both public and private) have fared with respect to Indian consumers needs its own blog post but one word to describe it is bad . But as shared above, needs its own blog post. Coming to the Indian context though, what has not been captured in that article is that the responsibility of making new content also raises huge barriers for teachers. My own experience in teacher s trainings for ICT usage has shown that most teachers do not know and use internet effectively both to sustain their own curiosity as well as their students. Part of which is whether you are private employee or a public school teacher, the teacher is not paid enough. I have had multiple conversations with friends over the years who are teachers who shared that they get 50% salary in-hand while they sign for 100%. This is more in the case or private schools though. In Govt. schools, the teachers apart from their regular administrative duties apart from teaching duties are also unpaid labor for Govt. policy. Take the recent covid crisis, it was the teachers who for months together went from door-to-door asking if they had a covid patient. This was all over India. Even for voter registration, census, polio and various other immunization efforts, the teachers are roped in. So apart from that, they somehow have to figure out how to make ends meet and also boost student morale. Hence the attention is only limited to the first couple of benches rather than the whole as a 45-minute to an hr. session is just not enough to go through a class of 70-150 school students giving individual attention. And this is when for most teachers, teaching is a means to an end and not the end itself. I am going to take one example of science and sort of break-it-down in multiple steps and how I would have approached that topic for say the 5th-6th standard students in say a public school in Pune and especially if Covid would not have been an issue so you have face-to-face meetup. There was recent news about a mysterious radio signal which came from one of our closest galactic neighbors Proxima Centauri. Now let s say there was a class I was teaching/sharing which I had shared before and there already is trust formed. So before coming to the news, I would tell the students about frequency and more generally the notation of why we like to measure things and how we measure things. There is so much beautiful history which could be acted and enacted which can show and remains in mind why measurement is needed. Once that is understood, discussed and an underpinning is established, we could move to human perception or the lack of it. We know that humans have lots of limits in almost everything, whether it is talking, touching, hearing, all of our five senses are pretty limited with what we know of spectrum available in the immediate family kingdom as well as in the Universe. I would start with how far can a person throw his voice and be heard without using any other means. There does come a point where they need to use anything from a megaphone to a loudspeaker and what it actually does. The other thing I would then talk is about the radio and ask the students to find more about the internals of a radio. If possible to bring an old radio to school where it could be disassembled. After they are familiar with some names of the electronic components and what they do, take them to the electronics market where they try to source all the things needed to make a radio and whatever they encounter. This would allow the students to try and do bargain shopping as well as learn from where to source things. Some might even get a copy or two of electronic projects where the shop themselves sell blueprints to hobbyists so that they can tinker. If there is a place in the school where soldering can be done, then all can try and sooner or later we come to know if something works or not. There is also possibility of talking about noise cancellation and then the topic of ITU can also be bought up and how they do frequency allocation. Last but not the least then the topic can be approached about an alien civilization and an unknown radio signal and what it means and what it can mean. Now if just one topic can give such a wide range of things to do and develop an understanding about not the subject itself but surrounding subjects as well I see no reason why teachers can t do this except they are handcuffed to lot of policy as well as real-life constraints. For e.g. I remember in my school days, we used to go out once or twice a year and that used to be either a school picnic or something similar. The only other I know is going to Mumbai for Nehru planetarium and Nehru science Centre. Unfortunately, I didn t go at that time because the school was taking students via air and the tickets were super costly at the time and that too for a 10 minute journey between the two cities. Those were different days, today you can t have a direct flight between the two cities as it doesn t make an economic sense. It makes more sense to go to Mumbai via train or bus as you will reach Mumbai in about couple of hrs. Of course, Pune does have its own planetarium at New English school and there are a few amateur astronomy clubs in Pune but nothing on the scale that what Mumbai has, but then this is getting off-topic. Now, again in an online world could this be done? Not without both the teacher and the student both spending lot of resources online and even then will be a lower understanding as both the hands-on experience as well as interacting with other students and learning from other students (aping) would be hugely limited. Even the social skills that students develop in a school setting will be rusted. My own social skills probably have weakened and rusted as I have very limited interaction with people over the past few months due to Covid fears and would be at least for the next few years till a large enough population is not vaccinated.

28 December 2020

Guido G nther: phosh overview

phosh is graphical shell for mobile, touch based devices like smart phones. It's the default graphical shell on Purism's Librem 5 (and that's where it came to life) but projects like postmarketOS, Mobian and Debian have picked it up putting it into use on other devices as well and contributing patches. This post is meant as a short overview how things are tied together so further posts can provide more details. A PHone SHell As mobile shell phosh provides the interface components commonly found on mobile devices to phosh's overview 2 phosh's lockscreen phosh's overview 1 It uses GObject object system and GTK to build up the user interface components. Mobile specific patterns are brought in via libhandy. Since phosh is meant to blend into GNOME as seamlessly as possible it uses the common interfaces present there via D-Bus like org.gnome.Screensaver or org.gnome.keyring.SystemPrompter and retrieves user configuration like keybindings via GSettings from preexisting schema. The components of a running graphical session roughly look like this: phosh session The blue boxes are the very same found on GNOME desktop sessions while the white ones are currently only found on phones. feedbackd is explained quickly: It's used for providing haptic or visual user feedback and makes your phone rumble and blink when applications (or the shell) want to notify the user about certain events like incoming phone calls or new messages. What about phoc and squeekboard? phoc and squeekboard Although some stacks combine the graphical shell with the display server (the component responsible for drawing applications and handling user input) this isn't the case for phosh. phosh relies on a Wayland compositor to be present for that. Keeping shell and compositor apart has some advantages like being able to restart the shell without affecting other applications but also adds the need for some additional communication between compositor and shell. This additional communication is implemented via Wayland protocols. The Wayland compositor used with phosh is called phoc for PHone Compositor. One of these additional protocols is wlr-layer-shell. It allows the shell to reserve space on the screen that is not used by other applications and allows it to draw things like the top and bottom bar or lock screen. Other protocols used by phosh (and hence implemented by phoc) are wlr-output-management to get information on and control properties of monitors or wlr-foreign-toplevel-management to get information about other windows on the display. The later is used to allow to switch between running applications. However these (and other) Wayland protocols are not implemented in phoc from scratch. phoc leverages the wlroots library for that. The library also handles many other compositor parts like interacting with the video and input hardware. The details on how phoc actually puts things up on the screen deserves a separate post. For the moment it's sufficient to note that phosh requires a Wayland compositor like phoc. We've not talked about entering text without a physical keyboard yet - phosh itself does not handle that either. squeekboard is the on screen keyboard for text (and emoji) input. It again uses Wayland protocols to talk to the Wayland compositor and it's (like phosh) a component that wants exclusive access to some areas of the screen (where the keyboard is drawn) and hence leverages the layer-shell protocol. Very roughly speaking it turns touch input in that area into text and sends that back to the compositor that then passes it back to the application that currently gets the text input. squeekboard's main author dcz has some more details here. The session So how does the graphical session in the picture above come into existence? As this is meant to be close to a regular GNOME session it's done via gnome-session that is invoked somewhat like:
phoc -E 'gnome-session --session=phosh'
So the compositor phoc is started up, launches gnome-session which then looks at phosh.session for the session's components. These are phosh, squeekboard and gnome-settings-daemon. These then either connect to already running services via D-Bus (e.g. NetworkManager, ModemManager, ...) or spawn them via D-Bus activation when required (e.g. feedbackd). Calling conventions So when talking about phosh it's good to keep several things apart: On top of that people sometimes refer to 'Phosh' as the software collection consisting of the above plus more components from GNOME (Settings, Contacs, Clocks, Weather, Evince, ...) and components that currently aren't part of GNOME but adapt to small screen sizes, use the same technologies and are needed to make a phone fun to use e.g. Geary for email, Calls for making phone calls and Chats for SMS handling. Since just overloading the term Phosh is confusing GNOME/Phosh Mobile Environment or Phosh Mobile Environment have been used to describe the above collection of software and I've contacted GNOME on how to name this properly, to not infringe on the GNOME trademark but also give proper credit and hopefully being able to move things upstream that can live upstream. That's it for a start. phosh's development documentation can be browsed here but is also available in the source code. Besides the projects mentioned above credits go to Purism for allowing me and others to work on the above and other parts related to moving Free Software on mobile Linux forward.

27 December 2020

Russ Allbery: Review: King of the Murgos

Review: King of the Murgos, by David Eddings
Series: The Malloreon #2
Publisher: Del Rey
Copyright: April 1988
Printing: November 1991
ISBN: 0-345-35880-5
Format: Mass market
Pages: 403
This is the second book of the Malloreon, which in turn is a sequel trilogy to The Belgariad. You could start here, since Eddings does a good job at summarizing previous volumes, but I'm not sure why you'd want to. This is the sort of book that gives epic fantasy series a bad name. After a lot of action in Guardians of the West, our band of heroes has been mostly assembled. They set out after the villain through the novel strategy of exploring the parts of the world map we have not yet seen. By the end of 400 pages of traveling they... are on the trail of the villain while exploring parts of the world map we have not yet seen. What I'm saying is that you'd better be really enjoying Silk's nose twitching and Belgarath being grumpy at Polgara, because that's about all you get. The protagonists collect a few important plot coupons and set up some emotional conflicts whose resolutions are already telegraphed, and that's about it. After wasting a bunch of time getting to and wandering around Nyissa, all of which felt pointless except for one moment of Errand doing something nifty (often the best parts of this series), the band of heroes ventures into enemy territory as foreshadowed by the title. The structure of Eddings's series is basically Suikoden (collect all the protagonists), so as one might expect the point of this excursion is to pick up another protagonist. In the process, we get to see a bit more of the Murgos than we have before. I've commented before that Eddings takes racial essentialism to such an absurd degree that these books start feeling like animal fairy tales, which is why I'm willing to tolerate all of the stereotyping. It's Planet of Hats except with fantasy countries. That tolerance is stretched rather thin when it comes to the bad guys, since the congruence with real-life racism and dehumanization of enemies is a bit too strong. Eddings dodges this mostly by not putting many of the bad guys in the page, but in King of the Murgos he has a golden opportunity to undercut his racial structure and surprise the reader. What if one of the Murgos is a protagonist and is vital to the series plot? I won't spoil what Eddings actually does, but if you imagine the stupidest and most obvious way to avoid having to acknowledge his fantasy races are not uniform, you probably just guessed the big reveal in this book. It's sadly not surprising, since Eddings is all in on his racial construction of this fantasy world, but it's still disgusting. The main reason why I decided to re-read this series, which is notorious for being a re-run of the Belgariad, is because I think what Eddings does here with prophecy, the voice in Garion's head, and the meddling of the seers is hilarious. The plot is openly on rails to the point that the characters actively grumble about it. I find that oddly entertaining, a bit like watching a Twitch streamer play an RPG. In this installment, Errand does something random in the middle of a Murgo temple, Garion gets a level-up chime, the prophecy gets very smug about how well things are going, and no one knows why. It's an absolute delight, and I don't know of any other series that isn't pure parody that would be willing to use that as a plot element. Sadly, King of the Murgos has only one or two enjoyable moments like that, a whole lot of frankly boring map exploration, some truly egregious racist claptrap, and no plot development worth speaking of. I probably should have skipped this one when re-reading this series. Followed by Demon Lord of Karanda. Rating: 4 out of 10

26 December 2020

John Goerzen: Rehabilitating Asynchronous Communication with NNCP: A Cross Between Tor, ssh, and UUCP

Have you ever been traveling, shot a ton of photos and videos, but were annoyed to find it was saturating the terrible wifi you had access to? Maybe you d wish the upload to pause until you get somewhere else, but then pausing syncing on your Nextcloud/Syncthing/Dropbox would also pause other syncing you didn t want to pause. Or you have trouble backing up your laptop when not at home, in a way that won t accidentaly eat up your cell phone data. There are ways to help with this: asynchronous transfer. Here s a lot of background. If you want to see how encrypted, onion-routed UUCP looks, skip ahead to the NNCP section! There is an old saying: When all you have is a hammer, every problem looks like a nail. We have this wonderful tool called ssh available, and it is pervasive and well-understood, so we tend to use it. But we ve missed out on some benefits of asynchronous processing that we actually used to have more frequently. Of course, we are all used to some asynchronous services in our lives. Email is a popular example: most mail clients work offline and will transmit stored messages when the mail server becomes reachable. Mail servers themselves work that way, too. Many instant messaging platforms do as well. Even some backup systems do. Bacula/Bareos, for instance, spools all backup data to disk on the system connected to the tape drive, and from there to the tape itself. They do this for several reasons, but primarily the fact that if tape drives are not fed with data at their design speed, it can cause physical damage to the tape or even the drive. It causes the drive to have to pause, and seek backwards to reposition for the next write. This creates excessive travel of the tape over the write heads, causing a condition known as tape shine where the tape is damaged prematurely. Here are some problems people often run into when sending data across a network (or the Internet) synchronously: Of course, there are plenty of situations where synchronous communication is a must. For instance: I suspect that the reason we don t do more asynchronous processing these days, despite it being strong in the Unix heritage, is the lack of modern tools to do it. Let s explore some more. Some of my use cases I run ZFS on all my systems that support it: file server, laptops, workstations, etc. It is only natural to use ZFS send/receive to do backups, and I do. However, when I am traveling, my laptop never gets backed up, because the backups are pulled from the backup system. Sure, there are ways around that; a VPN, for instance. But then we have the situation where sometimes I do not want to send the backup even if I have a working Internet connection: perhaps I m tethered to a mobile connection and it would be expensive to do so, or I m on hotel Wifi that is flaky and slow and I don t want to give up any of its meager bandwidth. I have another backup-related problem. I have a remote server, which until recently was using extremely slow disks. If I made significant changes, the backup would take the better part of a day. That s annoying when I try to back up hourly. So of course I had to implement locking, but then that means none of my other machines would back up that day either. Once I needed to transmit about 2TB of data. My home Internet connection was terribly slow, and I calculated it would take multiple months to do this. So I took to manually copying parts of the data to my laptop, and whenever I d find an airport or coffee shop with faster Internet than at home, I d send off those bits from it. But it took a ton of work. The bespoke asynchronous problem And that ton of work is perhaps why we aren t doing more of this. There s been no great standard solution, so it s all roll your own when you need to. So we just use ssh, because it s easier and usually good enough . But as I wrote in my recent article on airgapped backups, there are reasons to go async. Solutions Wouldn t it be great to be able to queue up data for a machine, and let it get there in whatever way it can? Maybe a fast Internet connection is found, or via Tor, or via copying to a USB stick, or via radio broadcast? It would make many of these scenarios a lot easier. And there are ways for this now, with modern security! We have some tools on Linux for this: git-annex for storage and migration, syrep for synchronization, and NNCP for file transfer and remote execution (could be combined with some of these other tools). Let s dive in to NNCP. NNCP If you already know UUCP, think of NNCP as UUCP brought into the modern era, with modern security and tools. Basically, NNCP permits you to send files to a remote system, request files from a remote system, and pipe data to an NNCP command that requests execution remotely. So you could, say, pipe a zfs send to NNCP which sends it to the remote and pipes it to zfs receive when it gets there. NNCP has a delay-tolerant, resumable protocol that can run over just about any reliable connection: TCP, serial, Tor, radios of various kinds, you name it. But that s not all; it also can dump its queue onto something like a USB stick for transport, or even make a tar-style stream that could be munged however you like. If you want to get fancy, you can assign priorities to data packets, so that, for instance, outbound email will always get sent before that 1TB file you ve got to send also. You can also configure it so that certain carriers handle certain priorities of data; your cell phone would only handle the most urgent, but a USB stick would take anything. NNCP is source-routed; you can tell it that the way that Bob reaches Alice is via Carl, then Betty. Bob can generate a message that will be sent along that route, fully encrypted and authenticated at each step of the way; Carl can t see the content of the message or even anything about it other than its next hop. How this helps Let s revisit some of my scnearios with NNCP. For the laptop being backed up, while traveling it can queue up its backups, or photos, or videos, or whatever. They could be triggered by a command when on a good connection, or automatically. The data could be copied to USB and given to a friend to transmit; perfectly safe due to encryption. Or it could all wait until arriving at home, safely out of your other syncing directories. The NNCP documentation has an example of this. For the server being backed up slowly, that s easily solved; the slow backup would simply be queued up, and transmitted and processed when it s ready. This wouldn t interrupt other backups. How about the 2TB transmission problem? That s also made a lot easier. A command could be run to fill up a USB stick with parts of the queue, then that USB stick plugged in and transmitted whenever at a fast location. Repeat as needed while the slow system continues its upload of the remaining bits. NNCP has a lot of interesting use cases documented as well. If you are already familiar with how public keys work in SSH, then NNCP should be immediately familiar as well. It is a similar concept (though arguably somewhat easier to set up). I am working on setting up a NNCP network, and will have more posts on how to do so once I ve got it going. In the meantime, the documentation for the project is also pretty good.

23 December 2020

Sven Hoexter: Jenkins dynamically parameterized pipelins for terraform execution

Jenkins in the Ops space is in general already painful. Lately the deprecation of the multiple-scms plugin caused some headache, becaue we relied heavily on it to generate pipelines in a Seedjob based on structure inside secondary repositories. We kind of started from scratch now and ship parameterized pipelines defined in Jenkinsfiles in those secondary repositories. Basically that is the way it should be, you store the pipeline definition along with code you'd like to execute. In our case that is mostly terraform and ansible. Problem Directory structure is roughly "stage" -> "project" -> "service". We'd like to have one job pipeline per project, which dynamically reads all service folder names and offers those as available parameters. A service folder is the smallest entity we manage with terraform in a separate state file. Now Jenkins pipelines are by intention limited, but you can add some groovy at will if you whitelist the usage in Jenkins. You have to click through some security though to make it work. Jenkinsfile This is basically a commented version of the Jenkinsfile we copy now around as a template, to be manually adjusted per project.
// Syntax: https://jenkins.io/doc/book/pipeline/syntax/
// project name as we use it in the folder structure and job name
def TfProject = "myproject-I-dev"
// directory relative to the repo checkout inside the jenkins workspace
def jobDirectory = "terraform/dev/$ TfProject "
// informational string to describe the stage or project
def stageEnvDescription = "DEV"
/* Attention please if you rebuild the Jenkins instance consider the following:
- You've to run this job at least *thrice*. It first has to checkout the
repository, then you've to add permisions for the groovy part, and on
the third run you can gather the list of available terraform folder.
- As a safeguard the first first folder name is always the invalid string
"choose-one". That prevents accidential execution of a random project.
- If you add new terraform folder you've to run the "choose-one" dummy rollout so
the dynamic parameters pick up the new folder. */
/* Here we hardcode the path to the correct job workspace on the jenkins host, and
   discover the service folder list. We have to filter it slightly to avoid temporary folders created by Jenkins (like @tmp folders). */
List tffolder = new File("/var/lib/jenkins/jobs/terraform $ TfProject /workspace/$ jobDirectory ").listFiles().findAll   it.isDirectory() && it.name ==~ /(?i)[a-z0-9_-]+/  .sort()
/* ensure the "choose-one" dummy entry is always the first in the list, otherwise
   initial executions might execute something. By default the first parameter is
   used if none is selected */
tffolder.add(0,"choose-one")
pipeline  
    agent any
    /* Show a choice parameter with the service directory list we stored
       above in the variable tffolder */
    parameters  
        choice(name: "TFFOLDER", choices: tffolder)
     
    // Configure logrotation and coloring.
    options  
        buildDiscarder(logRotator(daysToKeepStr: "30", numToKeepStr: "100"))
        ansiColor("xterm")
     
    // Set some variables for terraform to pick up the right service account.
    environment  
        GOOGLE_CLOUD_KEYFILE_JSON = '/var/lib/jenkins/cicd.json'
        GOOGLE_APPLICATION_CREDENTIALS = '/var/lib/jenkins/cicd.json'
     
stages  
    stage('TF Plan')  
    /* Make sure on every stage that we only execute if the
       choice parameter is not the dummy one. Ensures we
       can run the pipeline smoothly for re-reading the
       service directories. */
    when   expression   params.TFFOLDER != "choose-one"    
    steps  
        /* Initialize terraform and generate a plan in the selected
           service folder. */
        dir("$ params.TFFOLDER ")  
        sh 'terraform init -no-color -upgrade=true'
        sh 'terraform plan -no-color -out myplan'
         
        // Read in the repo name we act on for informational output.
        script  
            remoteRepo = sh(returnStdout: true, script: 'git remote get-url origin').trim()
         
        echo "INFO: job *$ JOB_NAME * in *$ params.TFFOLDER * on branch *$ GIT_BRANCH * of repo *$ remoteRepo *"
     
     
    stage('TF Apply')  
    /* Run terraform apply only after manual acknowledgement, we have to
       make sure that the when     condition is actually evaluated before
       the input. Default is input before when. */
    when  
        beforeInput true
        expression   params.TFFOLDER != "choose-one"  
     
    input  
        message "Cowboy would you really like to run **$ JOB_NAME ** in **$ params.TFFOLDER **"
        ok "Apply $ JOB_NAME  to $ stageEnvDescription "
     
    steps  
        dir("$ params.TFFOLDER ")  
        sh 'terraform apply -no-color -input=false myplan'
         
     
     
 
    post  
            failure  
                // You can also alert to noisy chat platforms on failures if you like.
                echo "job failed"
             
         
job-dsl side of the story Having all those when conditions in the pipeline stages above allows us to create a dependency between successful Seedjob executions and just let that trigger the execution of the pipeline jobs. This is important because the Seedjob execution itself will reset all pipeline jobs, so your dynamic parameters are gone. By making sure we can re-execute the job, and doing that automatically, we still have up to date parameterized pipelines, whenever the Seedjob ran successfully. The job-dsl script looks like this:
import javaposse.jobdsl.dsl.DslScriptLoader;
import javaposse.jobdsl.plugin.JenkinsJobManagement;
import javaposse.jobdsl.plugin.ExecuteDslScripts;
def params = [
    // Defaults are repo: mycorp/admin, branch: master, jenkinsFilename: Jenkinsfile
    pipelineJobs: [
        [name: 'terraform myproject-I-dev', jenkinsFilename: 'terraform/dev/myproject-I-dev/Jenkinsfile', upstream: 'Seedjob'],
        [name: 'terraform myproject-I-prod', jenkinsFilename: 'terraform/prod/myproject-I-prod/Jenkinsfile', upstream: 'Seedjob'],
    ],
]
params.pipelineJobs.each   job ->
    pipelineJob(job.name)  
        definition  
            cpsScm  
                // assume admin and branch master as a default, look for Jenkinsfile
                def repo = job.repo ?: 'mycorp/admin'
                def branch = job.branch ?: 'master'
                def jenkinsFilename = job.jenkinsFilename ?: 'Jenkinsfile'
                scm  
                    git("ssh://git@github.com/$ repo .git", branch)
                 
                scriptPath(jenkinsFilename)
             
         
        properties  
            pipelineTriggers  
                triggers  
                    if(job.upstream)  
                        upstream  
                            upstreamProjects("$ job.upstream ")
                            threshold('SUCCESS')
                         
                     
                 
             
         
     
 
Disadvantages There are still a bunch of disadvantages you've to consider Jenkins Rebuilds are Painful In general we rebuild our Jenkins instances quite frequently. With the approach outlined here in place, you've to allow the groovy script execution after the first Seedjob execution, and then go through at least another round of run the job, allow permissions, run the job, until it's finally all up and running. Copy around Jenkinsfile Whenever you create a new project you've to copy around Jenkinsfiles for each and every stage and modify the variables at the top accordingly. Keep the Seedjob definitions and Jenkinsfile in Sync You not only have to copy the Jenkinsfile around, but you also have to keep the variables and names in sync with what you define for the Seedjob. Sadly the pipeline env-vars are not available outside of the pipeline when we execute the groovy parts. Kudos This setup was crafted with a lot of help by Michael and Eric.

Next.

Previous.