Changes in RcppSpdlog version 0.0.4 (2020-12-11)

• Upgraded to upstream release spdlog 1.8.2
• Added GitHub Actions CI using run.sh from r-ci

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Issues with Farm Laws While I have written about the farm laws a bit sometime back. The issue is still in the nation s eye and that is due to the policies which have been done. I have been reading up on it quite a bit and also have been seeing what has been happening in here and now. The problems are with the three bills themselves which I have shared as below Click to access farmers-produce-trade-and-commerce-promotion-and-facilation-bill.pdf Click to access farmers-empowerment-and-protection-bill.pdf Click to access essential-commodities-bill-2020.pdf Biggest issue with the laws While there are many issues with the laws themselves but for me the biggest issue is that the fundamental right of the farmer to get justice via civil courts has been railroaded. From the laws itself. Standard disclaimer not a lawyer, please consult one for any issues per-se.

Farmers-produce-trade-and-commerce (promotion and facilitation-bill) 2020 Page 4 Chapter 3 Section 8 (1)8. (1) In case of any dispute arising out of a transaction between the farmer and a trader under section 4, the parties may seek a mutually acceptable solution through conciliation by filing an application to the Sub-Divisional Magistrate who shall refer such dispute to a Conciliation Board to be appointed by him for facilitating the binding settlement of the dispute. (2) Every Board of Conciliation appointed by the Sub-Divisional Magistrate under sub-section (1), shall consist of a chairperson and such members not less than two and not more than four, as the Sub-Divisional Magistrate may deem fit.10 (5) If the parties to the transaction under sub-section (1) are unable to resolve the dispute within thirty days in the manner set out under this section, they may approach the Sub-Divisional Magistrate concerned who shall be the Sub-Divisional Authority for settlement of such dispute. (8) Any party aggrieved by the order of the Sub-Divisional Authority may prefer an appeal before the Appellate Authority (Collector or Additional Collector nominated by the Collector) within thirty days of such order who shall dispose of the appeal within thirty days from the date of filing of such appeal. 10. (1) Any person aggrieved by an order under section 9 may, prefer an appeal within sixty days from the date of such order, to an officer not below the rank of Joint Secretary to the Government of India to be nominated by the Central Government for this purpose: Page 6 of the bill. 13. No suit, prosecution or other legal proceedings shall lie against the Central Government or the State Government, or any officer of the Central Government or the State Government or any other person in respect of anything which is in good faith done or intended to be done under this Act or of any rules or orders made thereunder. Page 7 of the bill, 15. No civil court shall have jurisdiction to entertain any suit or proceedings in respect of any matter, the cognizance of which can be taken and disposed of by any authority empowered by or under this Act or the rules made thereunder. Now the same laws have been reiterated for the farmers (Empowerment and Protection) Agreement on Price Assurance and Farm Services Bill, 2020. The problem is that too much power is being put into the hands of the executive. All the three, whether it is SDM (Sub-Divisional Magistrate) , the Appellate Authority or the Government Secretary directly are subservient to the whims and fancies of the Central Govt. They after all get their salaries from the Govt. itself. So there will be no independent oversight to any injustices done to the farmer. The third bill i.e. the Essential Commodities Bill, 2020 does away with stock limits on traders and big players like Adani and Ambani. This means that both these players can take and keep produce at their end thereby forcing consumers like you and me who at the retail end would have to pay higher prices for fruits and vegetables while from the producer they will take at the lowest price possible. While I have shared is just one of the points. That is the reason why even the Supreme Court bar association which almost never takes part in politics has been forced to take sides with the farmers. In many ways, one is forced to remember the Emergency Update 11/12/20 Came across this article on the wire which tells how everybody s rights, not just the farmer s rights are being shod over. I think it depicts correctly the signs of time to come. While arguing on SM, also came to know about Article 300 (1), thanks to Sachin Kumar which shows multiple instances where Government was sued because somebody was working in official capacity and did mistakes, malafide or otherwise and it was the state who was made to pay. FWIW, today farmers from Maharashtra, my state arrived at Delhi border where they were also kept at bay. I did come across an infographic which shows how the various states have fared. Most tellingly, is the state of Bihar. It was in 2006 (one of the most backward states) where APMC was taken off. While others have tried to paint a flattering picture of Bihar, they have failed to share that in the interim 15 odd years, there hasn t been any sort of infrastructure created for farmers which is the reason it is still the lowest earner. These are the last available figures we have about the farmer s income. From 2014 to 2020 there hasn t been any update.

Situation Assessment Survey of Agricultural Household 2013 Copyright GOI,

This concludes just one portion of the bill. I will take other parts of the bill. I may dwell on some other parts as and when I have the time. A cartoon which depicts the current issue

I stand with farmers Copyright Sanitary Panels

Too much democracy Amitabh Kant Yesterday, the Niti Aayog chief Amitabh Kant remarked that we are too much of a democracy at an event called for Atmanirbhar Bharat which is basically a coinage for import substitution. Whether this is desirable or not I have argued and if needed will re-argue the same later as well. What is and was interesting were the gentleman s context, the media reactions and our overall Democracy Index which has been going downhill for quite some years. Now the gentleman who is the Niti Aayog chief and who is supposed to have the ear of the Prime Minister had opined it in an event organized by Swarajya Magazine (a far-right magazine) known to be Islamophobic and all things undemocratic. It has been a target of defundthehate campaign and with good reason. But that s a different story altogether. His full statement was as below

Tough reforms are very difficult in the Indian context, as we are too much of a democracy but the government has shown courage and determination in pushing such reforms across sectors, including mining, coal, labour and agriculture. Niti Aayog chief. The upper quotation remarks and the statement has been from the article in Indian Express which I have linked to. I have archived it as a pdf just in case the link goes dead. Yesterday, after the statement became viraled, tweets of media houses which shared the tweet suddenly become unavailable. Seems too much democracy, became too little democracy all of a sudden. I think Mr. Amitabh Kant didn t visualize as the opposition as well as most people who are on Twitter to share their opinion on the same. Few examples

Too much Democracy copyright Satish Acharya

Too much democracy Illustration and Copyright Alok

Sterlite protest 13 dead, 100 injured Copyright Business Standard too much democracy

Erosion of Democracy V-dem institute Copyright The Hindu Web Team

The last one requires a bit more information. This comes from V-Dem Institute which is an independent research institute based out of Department of Political Science, University of Gothenburg, Sweden. I am gonna leave the methods they use for another day as the blog post itself has become rather big/large. Apart from that is the Economists own Democracy Index -2019 Click to access democracy-index-2019.pdf Now for many people, both the V-Dem report and the Economist Index are some sort of attack against India. Doesn t matter that in V-Dem 200+ countries have been taken a variety of indicators and data or the Economist which has data from 150+- countries. Somehow India is supposed to be bigger than all these countries, they do think that other countries data specifically our neighbor China or any other neighbor, those are all accurate. How the dissonance is, has to be gauged from statements of various people. Update 11/12/20 Sadly, the newest V-Dem report marks India as getting into authoritarianism. Gag on Press and Media owners I had shared about the gag on the press especially with respect to western media or reports or anything. This news made its way to straitstimes which normally covers a wide-range of stories covering East Asia vis-a-vis India/South-East Asia. What has also been a big worry that most of the media has been in the hands of a few people. Caravan ran a story on the same in 2016, it has been four years, god only knows what the current situation might be. Any wonder that there is dearth of investigative journalism in India.

India media ownership 2016 Copyright Caravan

Incidentally, a reporter called Akarshan Uppal, who is a reporter on a channel called IBN24 had showecased just few days back how Adani has got land which was shot down for land change use in 2017 to 2020 around 100 acres. There seem to be very less details as to how the land was acquired, whose land it was etc. etc. The reporter was supposedly following a story on drugs on which he was attacked and is now lying in hospital.

Akarshan Uppal Reporter, IBN24 Copyright IBN24

While it would take a whole article/blog post to talk about either Adani or Ambani, in the recent case, the land that has been taken over by Adani is 100 acres and there are private rail lines. And all of this was secret till few days back. The place where these massive godowns/silos have been made are Panipat s Jondhan Kalan and Naultha villages in Haryana. This is Adani AgiLogistics. Almost 7 odd companies have registered and come up in the last couple of years. As can be seen, almost all have come up within the last 2-3 years. Seems to be a lot of coincidence, isn t it?

Personal Anecdote on Data Collection and child marriages in India.

Around 1995 -96 when Internet had started to become a thing in India, there had been quite a few non-profits which were working on various issues. One of those which I initially came in contact with and which I found to be a bit absurd was non-profit which was working in the field of women against Violence. Now it is and was not the concept or the idea which was absurd to me, it was what these women were doing. Instead of the traditional ways in which you counsel women and try and figure out issues, these women were collecting data points from newspapers and magazines. This was way way before data science became a thing in India. They had their own structure where a story about violence against women which would be above the fold would be 5 points, the one below 2.5 points, in inner pages, it would be less and less. Patriarchy at that time was so strong, even today is but at that time it was such, that it felt a waste of time. I did consult them but never said that but did privately feel the above. In hindsight, they were doing the right thing and yet even today crimes against women goes unreported and is suppressed by both State and Central Governments as well as NCRB (National Crime Records Bureau). Interestingly, just few days back, the case against M.J. Akbar by Priya Ramani had taken a back seat and the defamation case by M.J. Akbar was taken forward. Even then, Priya Ramani s counsel s arguments were such that the court wound up in half an hour when they were expecting to do a whole day hearing. The next hearing would be happening today which I will look at in few hours from now. Why Priya Ramani was singled out rather than other tweets may probably be because she is an NRI and most NRI s usually do not want to be part of the bureaucratic Indian court system. This is also the reason that most companies from outside India especially those who are into startups prefer to change ownership, IPR etc. to their own or any country outside India which does make a loss to the exchequer. But this again is a story for another day. At the end, while I did not want to end on a negative note, it seems in many ways status-quo remains. For e.g. 2 years back, a BJP candidate (part of the ruling dispensation) had made a controversy saying that if they win the police won t interfere in child marriages. This is and was in Rajasthan where they have been trying to eradicate it forever. Till date, neither the NHRC (National Human Rights Commission) nor NCW (National Commission for Women) has taken cognizance of the statement. This is our state of democracy.

• the Debian server Linux guest created with the "Linux Center" app can be accessed using SSH with the admin/admin login/pwd (couldn't find a clue in the docs or the UI)
• the "Mount shared folder" option in the Debian guest configuration leads to the "/.share/" mountpoint which exibits the NAS filesystem under the Linux guest. This allows to retrieve files being backedup to the NAS to be copied out with rsync
• backupninja provides an rsync handler which allows to backup files with rsync, but some default options weren't valid (see debian bug #976650).
• the Seagate ironwolf HD less that 4TB aren't compatible with IHM (Ironwolf Health Management), even if the "Storage Manager" app tends to mislead you.

• [DLA 2446-1] moin security update for two CVEs
• [DLA 2451-1] libvncserver security update for one CVE
• [DLA 2459-1] golang-1.7 security update for two CVEs
• [DLA 2460-1] golang-1.8 security update for three CVEs
• [DLA 2468-1] tcpflow security update for one CVE
• [DLA 2469-1] qemu security update for five CVEs

• ELA-319-1 for libass
• ELA-320-1 for tcpflow
• ELA-321-1 for qemu

• golang-github-apparentlymart-go-cidr
• golang-github-apparentlymart-go-versions
• golang-github-bmatcuk-doublestar
• golang-github-cactus-go-statsd-client
• golang-github-cyberdelia-heroku-go
• golang-github-gin-contrib-cors
• golang-github-gin-contrib-gzip
• golang-github-jarcoal-httpmock
• golang-github-mattetti-filebuffer
• golang-github-nrdcg-goinwx
• golang-github-phpdave11-gofpdi
• golang-github-terra-farm-udnssdk
• meep
• golang-github-rs-zerolog

• libctl

• bottlerocket
• ent
• golang-github-abdullin-seq
• golang-github-bruth-assert
• golang-github-cnf-structhash
• golang-github-crossdock-crossdock-go
• golang-github-dchest-uniuri
• golang-github-deanthompson-ginpprof
• golang-github-dreamitgetit-statuscake
• golang-github-facebookgo-inject
• golang-github-facebookgo-structtag
• golang-github-gin-contrib-static
• golang-github-goji-httpauth
• golang-github-grafana-grafana-plugin-model
• golang-github-hansrodtang-randomcolor
• golang-github-hashicorp-terraform-svchost
• golang-github-icrowley-fake
• golang-github-jackc-fake
• golang-github-joyent-gocommon
• golang-github-joyent-gosdc
• golang-github-joyent-gosign
• golang-github-mitchellh-go-linereader
• golang-github-paypal-gatt
• golang-github-pearkes-cloudflare
• golang-github-pearkes-dnsimple
• golang-github-robfig-go-cache
• golang-github-sean pager
• golang-github-sean seed
• golang-github-shurcool-gopherjslib
• golang-github-vividcortex-mysqlerr
• golang-github-xlab-handysort
• golang-github-yudai-golcs
• golang-github-yvasiyarov-newrelic-platform-go
• golang-github-zenazn-goji
• golang-github-rs-xid

• golang-github-rhnvrm-simples3
• golang-github-ua-parser-uap-go
• golang-github-knadh-koanf

The need for case names Consider the following Coq proof, right out of the respected Software Foundations book, Chapter Inductively Defined Propositions (and slightly simplified):

Lemma star_app: forall T (s1 s2 : list T) (re : reg_exp T),
  s1 =~ Star re ->
  s2 =~ Star re ->
  s1 ++ s2 =~ Star re.
Proof.
  intros T s1 s2 re H1.
  remember (Star re) as re'.
   generalize dependent s2.
  induction H1.
  - (* MEmpty *) discriminate.
  - (* MChar *) discriminate.
  - (* MApp *) discriminate.
  - (* MUnionL *) discriminate.
  - (* MUnionR *) discriminate.
  - (* MStar0 *)
    injection Heqre' as Heqre''. intros s H. apply H.
  - (* MStarApp *)
    injection Heqre' as Heqre''.
    intros s3 H1. rewrite <- app_assoc.
    apply MStarApp.
    + apply H1_.
    + apply IHexp_match2.
      * rewrite Heqre''. reflexivity.
      * apply H1.
Qed.

This is a very typical example for a proof by induction over an inductively defined relation (in this case, the relations s =~ re which indicates that the string s matches the regular expression re): After some preparation, the invocation of a tactic like induction (could also be inversion or others) creates a number of new goals, one of for each rule of the relation. Then the modern(!) Coq user uses proof bullets to focus the goals, one after another, in order, to solve them. We can see from this example that proof bullets (which were a big improvement) are not quite satisfactory, and the author of this proof felt the need to annotate each subproof with the case it corresponds to. But these are mere comments! This is unsatisfactory, as there is nothing keeping them correct as my proof changes. And anyone who came to Coq from Isabelle is just flabbergasted that the state of art in Coq is still at that level So, wouldn t it be great if

• I could focus goals by their name (e.g. MEmpty)
• Focus them in any order (so that refactoring of my data types do not break my proofs)
• This even worked when simultaneously inducting over or inverting multiple relations?

Simulating named goals in Coq Well, with a few tricks we can! It just takes three simple steps:

1. I load a small file full of hacks (explained below):

   Require Import NamedCases.

2. This allows me to prefix the type of the constructors of the inductive relations (i.e. the rules of the relation) with case <name>, , as follows:

   Reserved Notation "s =~ re" (at level 80).
   Inductive exp_match  T  : list T -> reg_exp T -> Prop :=
       MEmpty:
       case empty,
       [] =~ EmptyStr
       MChar:
       case char,
       forall x,
       [x] =~ (Char x)
       MApp:
       case app,
       forall s1 re1 s2 re2
       (H1 : s1 =~ re1)
       (H2 : s2 =~ re2),
       (s1 ++ s2) =~ (App re1 re2)
       MUnionL:
       case union_l,
       forall s1 re1 re2
       (H1 : s1 =~ re1),
       s1 =~ (Union re1 re2)
       MUnionR:
       case union_r,
       forall re1 s2 re2
       (H2 : s2 =~ re2),
       s2 =~ (Union re1 re2)
       MStar0:
       case star0,
       forall re,
       [] =~ (Star re)
       MStarApp:
       case star_app,
       forall s1 s2 re
       (H1 : s1 =~ re)
       (H2 : s2 =~ (Star re)),
       (s1 ++ s2) =~ (Star re)
     where "s =~ re" := (exp_match s re).

   This is the definition from the Software Foundations book; only the case <name>, lines are added.
3. Now I can change the proof: After the induction tactic I use ; name_cases to name the cases, and then I can use the relatively new named goal focusing feature ([name]: , which I suggested two years ago, and was implemented by Th o Zimmermann) to focus a specific case:

   Lemma star_app2: forall T (s1 s2 : list T) (re : reg_exp T),
     s1 =~ Star re ->
     s2 =~ Star re ->
     s1 ++ s2 =~ Star re.
   Proof.
     intros T s1 s2 re H1.
     remember (Star re) as re'.
      generalize dependent s2.
     induction H1; name_cases.
     [empty]:  
        discriminate.
      
     [char]:  
        discriminate.
      
     [app]:  
        discriminate.
      
     [unionL]:  
        discriminate.
      
     [unionR]:  
       discriminate.
      
     [star0]:  
       injection Heqre' as Heqre''. intros s H. apply H.
      
     [starApp]:  
       injection Heqre' as Heqre''.
       intros s3 H1. rewrite <- app_assoc.
       apply MStarApp; clear_names.
       + apply H1_.
       + apply IHexp_match2.
         * rewrite Heqre''. reflexivity.
         * apply H1.
       
   Qed.

   The comments have turned into actual names!

Goal focusing is now just much more reliable. For example, let me try to discharge the boring cases by using ; try discriminate after the induction. This means there are suddenly only two goals left. With bullets, the first bullet would now suddenly focus a different goal, and my proof script would likely fail somewhere. With named goals, I get a helpful error message No such goal: empty and I can t even enter the first subproof. So let me delete these goals and, just to show off, focus the remaining goals in the wrong order:

Lemma star_app: forall T (s1 s2 : list T) (re : reg_exp T),
  s1 =~ Star re ->
  s2 =~ Star re ->
  s1 ++ s2 =~ Star re.
Proof.
  intros T s1 s2 re H1.
  remember (Star re) as re'.
   generalize dependent s2.
  induction H1; name_cases; try discriminate.
  [starApp]:  
    injection Heqre' as Heqre''.
    intros s3 H1. rewrite <- app_assoc.
    apply MStarApp; clear_names.
    + apply H1_.
    + apply IHexp_match2.
      * rewrite Heqre''. reflexivity.
      * apply H1.
    
  [star0]:  
    injection Heqre' as Heqre''. intros s H. apply H.
   
Qed.

I did not explain the clear_names tactic yet that I had to use after apply MStarApp. What I am showing here is a hack, and it shows these case names are extra hypothesis (of a trivially true type CaseName), and this tactic gets rid of it.

Even better than Isabelle The above even works when there are multiple inductions/inversions involved. To give an example, let s introduce another inductive predicate, and begin proving a lemma that one can do by induction on one and inversion on the other predicate.

Inductive Palindrome  T  : list T -> Prop :=
  EmptyPalin:
  case emptyP,
  Palindrome []
  SingletonPalin:
  case singletonP,
  forall x,
  Palindrome [x]
  GrowPalin:
  case growP,
  forall l x,
  Palindrome l -> Palindrome ([x] ++ l ++ [x])
.
Lemma palindrome_star_of_two:
  forall T (s : list T) x y,
  Palindrome s -> s =~ Star (App (Char x) (Char y)) ->
  s = [] \/ x = y.
Proof.
  intros T s x y HPalin HRe.
  induction HPalin; inversion HRe; name_cases.

At this point we have four open goals. Can you tell why there are four, and which ones they are? Luckily, Show Existentials will tell us their names (and Coq 8.13 will just show them in the proof state window). These names not only help us focus the goal in a reliable way; they even assist us in understanding what goal we have to deal with. For example, the last goal arose from the GrowP rule and the StarApp rule. Neat!

  [emptyP_star0]:  
    left. reflexivity.
   
  [emptyP_starApp]:  
     left. reflexivity.
   
  [singletonP_starApp]:  
    inversion HRe; clear HRe.
    inversion H5; clear H5.
    inversion H10; clear H10.
    inversion H11; clear H11.
    subst.
    inversion H3.
   
 [growP_starApp]:  
    admit. (* I did not complete the proof *)
  

Under the hood (the hack) The above actually works, but it works with a hack, and I wish that Coq s induction tactic would just do the right thing here. I hope eventually this will be the case, and this section will just be curiosity but until, it may actually be helpful.

1. The trick¹ is to leave markers in the proof state. We use a dedicate type CaseName, and a hypothesis name : CaseName indicates that this goal arose from the that case.

   Inductive CaseName := CaseNameI.

   With this, we can write StarApp: forall (starApp : CaseName), to name this case.
2. For a little bit of polish, we can introduce notation for that:

   Notation "'case' x , t" := (forall  x : CaseName , t) (at level 200).

   This allows us to use the syntax you saw above.
3. To easily discharge these CaseName assumptions when using the introduction forms, I wrote these tactics:

   Ltac clear_names := try exact CaseNameI.
   Ltac named_constructor  := constructor; [ exact CaseNameI   idtac .. ].
   Ltac named_econstructor := econstructor; [ exact CaseNameI   idtac .. ].

4. Finally I wrote the tactic name_cases. What it does is relatively simple:

   For each subgoal: Take all hypotheses of type CaseName, concatenate their names, change the name of the current goal to that, and clear these hypotheses

   Unfortunately, my grasp of Coq tactic programming is rather rudimentary, and the only way I manged to solve this is a horrible monster of Ltac2, including hand-rolled imperative implementations of string concatenation and a little nested Ltac1 tactic within. You have been warned:

   From Ltac2 Require Import Ltac2.
   From Ltac2 Require Option.
   Set Default Proof Mode "Classic".
   Ltac name_cases := ltac2:(
     (* Horribly manual string manipulations. Does this mean I should
        go to the Ocaml level?
     *)
     let strcpy s1 s2 o :=
       let rec go := fun n =>
         match Int.lt n (String.length s2) with
           true => String.set s1 (Int.add o n) (String.get s2 n); go (Int.add n 1)
           false => ()
         end
       in go 0
     in
     let concat := fun s1 s2 =>
       let l := Int.add (Int.add (String.length s1) (String.length s2)) 1 in
       let s := String.make l (Char.of_int 95) in
       strcpy s s1 0;
       strcpy s s2 (Int.add (String.length s1) 1);
       s
     in
     Control.enter (let rec go () :=
       lazy_match! goal with
         [ h1 : CaseName, h2 : CaseName  - _ ] =>
         (* Multiple case names? Combine them (and recurse) *)
         let h := concat (Ident.to_string h1) (Ident.to_string h2) in
         Std.clear [h1; h2];
         let h := Option.get (Ident.of_string h) in
         assert ($h := CaseNameI);
         go ()
         [ _ : CaseName  - _ ] =>
         (* A single case name? Set current goal name accordingly. *)
         ltac1:(
           (* How to do this in ltac2? *)
           lazymatch goal with
             [ H : CaseName  - _ ] => refine ?[H]; clear H
           end
         )
         [  - _ ] =>
         Control.backtrack_tactic_failure "Did not find any CaseName hypotheses"
       end
     in go)
   ).

   If someone feels like re-implementing this in a way that is more presentable, I ll be happy to update this code (and learn something along the way

That s all it takes!

Conclusion With some mild hackery, we can get named cases in Coq, which I find indispensable in managing larger, evolving proofs. I wonder why this style is not more common, and available out of the box, but maybe it is just not known well enough? Maybe this blog post can help a bit here.

---

1. I have vague recollections of seeing at least parts of this trick on some blog some years ago, and I do not claim originality here.

• start from ubuntu:21.04 to be able to install g++-11 later
• install the software-properties-common package to be able to add a PPA
• (plus a few more packages to deal with the repository signing key)
• run the sudo add-apt-repository ppa:ubuntu-toolchain-r/volatile command to add the volatile packages PPA
• install g++-11 (along with, for good measure) gcc-11 and gfortran-11
• use update-alternative (a clever Debian/Ubuntu command) to make version 11 the default
• install R itself (via r-base-core) which we simply take from the distro as 21.04 is by construction very recent
• install Rcpp via the r-cran-rcpp binary which covers all dependencies for the package in question

edd@rob:~/git/rcpptoml(master)$ docker run --rm -ti -v $PWD:/mnt -w /mnt rocker/r-edge:gcc-11 g++ --version
g++ (Ubuntu 11-20201128-0ubuntu2) 11.0.0 20201128 (experimental) [master revision fb6b29c85c4:a331ca6194a:e87559d202d90e614315203f38f9aa2f5881d36e]
Copyright (C) 2020 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

edd@rob:~/git/rcpptoml(master)$ 
edd@rob:~/git/rcpptoml(master)$ docker run --rm -ti -v $PWD:/mnt -w /mnt rocker/r-edge:gcc-11 R CMD INSTALL RcppTOML_0.1.7.tar.gz
* installing to library  /usr/local/lib/R/site-library 
* installing *source* package  RcppTOML  ...
** using staged installation
** libs
g++ -std=gnu++11 -I"/usr/share/R/include" -DNDEBUG -I../inst/include/ -DCPPTOML_USE_MAP -I'/usr/lib/R/site-library/Rcpp/include'    -fpic  -g -O2 -fdebug-prefix-map=/build/r-base-Fuvi9C/r-base-4.0.3=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g  -c RcppExports.cpp -o RcppExports.o
g++ -std=gnu++11 -I"/usr/share/R/include" -DNDEBUG -I../inst/include/ -DCPPTOML_USE_MAP -I'/usr/lib/R/site-library/Rcpp/include'    -fpic  -g -O2 -fdebug-prefix-map=/build/r-base-Fuvi9C/r-base-4.0.3=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g  -c parse.cpp -o parse.o
g++ -std=gnu++11 -shared -L/usr/lib/R/lib -Wl,-Bsymbolic-functions -Wl,-z,relro -o RcppTOML.so RcppExports.o parse.o -L/usr/lib/R/lib -lR
installing to /usr/local/lib/R/site-library/00LOCK-RcppTOML/00new/RcppTOML/libs
** R
** inst
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
** building package indices
** testing if installed package can be loaded from temporary location
** checking absolute paths in shared objects and dynamic libraries
** testing if installed package can be loaded from final location
** testing if installed package keeps a record of temporary installation path
* DONE (RcppTOML)
edd@rob:~/git/rcpptoml(master)$ 

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

• grow faster, wasting more disk space and duplicating data.
• increase the amount of necessary I/O during cluster allocation,
  reducing the allocation performance.

• Sixteen times less metadata per unit of allocation, greatly reducing the amount of necessary L2 cache.
• Much faster I/O during allocating when the image has a backing file, up to 10-15 times more I/O operations per second for the same cluster size in my tests (see chart below).
• Smaller images and less duplication of data.

• The performance improvements described earlier happen during allocation. Writing to already allocated (sub)clusters won t be any faster.
• If the image does not have a backing file chances are that the allocation performance is equally fast, with or without extended L2 entries. This depends on the filesystem, so it should be tested before enabling this feature (but note that the other benefits mentioned above still apply).
• Images with extended L2 entries are sparse, that is, they have holes and because of that their apparent size will be larger than the actual disk usage.
• It is not recommended to enable this feature in compressed images, as compressed clusters cannot take advantage of any of the benefits.
• Images with extended L2 entries cannot be read with older versions of QEMU.

$ qemu-img create -f qcow2 -o extended_l2=on,cluster_size=128k img.qcow2 1T

Changes in version 0.1.7 (2020-12-01)

• Add #include <limits> to header file, also contributed upstream, to permit compilation under the (unreleased) g++-11.
• Switch the simple cleanup script to sh.
• Switch CI use to r-ci for focal and bspm.
• Update several TOML URLs to https://toml.io/en/.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Truth, Untruths and Education in India. I read this somewhat disturbing and yet pretty raw truth from foreign affairs. It took me quite a few days to not only digest but also say yes and see the same situation playing out in India. I have been seeing the discourse on Twitter and while a part of it is the equivalent of road rage, a huge part is a disconnect to not acknowledge and be civil. We may come to different conclusions from the same data but being civil seems to be difficult for a lot of people. One part is of course ego, where nobody wants to lose, but more than that are the plain comprehension issues. Most of the literature, good literature is unfortunately based in English.

And while we can have differing opinions of what constitutes good literature, for me it s books like Battle of Belonging,The:On Nationalism, Patriotism, and What it Means Shashi Tharoor. From what little I have understood, the book makes the case of civic nationalism which is far more inclusive than the narrow confines of patriotism. Now this begs the question when you have such books and many books which do tell you about different aspects of social, political and knowledge, why are so many people prone to disinformation in India similar to U.S. and probably other countries as well. One of the biggest reasons per-se is lack of education and quality education. When the number of graduates is less than five percent how do you expect that population to be able to take decisions in their economic self-interest? So sadly the understanding is ingrained from WhatsApp and there is no need to check from alternate sources. And just like Mr. Trump followers, they believe those versions to be the unvarnished truth. I do understand that no truth is immutable except for life and death. All others are imperfect unless it is validated by some sort of scientific validation behind it. At the same time, these truths may themselves be invalidated if a stronger scientific evidence establishes itself. This is the reason why hypothesis and facts themselves are challenged again and again. Sharing couple of examples below.

Nationalization of Banking, RERA and RCEP Most of the people want freedom of the banks i.e. private banking don t really know that private banking existed at a time in free India before they were nationalized and these banks failed at surprisingly regular intervals. Now it isn t as if this fact is hidden but it is not as popular as maybe some other facts or ideas. Now the Government in the center obviously doesn t want to share these facts as they want corporates in banking. And if that fact is known by many people it will be a huge setback to their plans. RBI failures have been to many to count. Even recent legislations like RERA and others which were supposed to bring relief to millions of potential homeowners has become a pawn in the hands of builders and this has been known. One of the interesting points of RCEP which is not so much in public domain is that RCEP would have a mere 4.5% duty on most products which will go down to 1.5% over a 20 year period. Now with India staying out of it, we have done two things. We have said that we will not be competitive even after 20 years of this which is the more damning part. And we will not take part in the growth that other countries will have due to this. Contempt Proceedings against Comic Artist because she has an opinion on SC The fall in SC and constitutional values grows day by day. The AG today consented to have contempt proceedings against a comic artist saying she insulted the SC. Gone are the days when an artist made fun of the PM, and she gave him a Padam Shri (one of the highest civilian honors) for his contributions. Then, even dissent or being cynical was looked as being a contribution to the national effort rather than today. This is the reason why India has been continuously falling in the Global Freedom of Expression Index. I have seen censoring many a time here. I, myself has been locked out of Wikipedia many times. Can you imagine, being locked out of Wikipedia which is perhaps one of the more neutral sites on the web. And then there was this wikibio thing, such a sad thing to happen. Guessing this is the future of the Indian interweb.

Stick figure by Sanitary Panels on SC

Changes in inline version 0.3.17 (2020-11-30)

• Unit testing is now supported via tinytest (Johannes in #15 addressing #14).
• CI was updated to use focal and run.sh from r-ci on Travis and GitHub Actions (Dirk)
• The writing and reading of compiled code was refactored and extended (Johannes in #16 fixing #13).
• Some minor problems related to CRAN checks and tests were corrected (Johannes and Dirk in #17, Johannes in #18, #19, #20).
• Small stylistic updates have been applied to some R and Rd files (Dirk).

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

• Merged a pull request from Jens Nistler for django-slack (my library which provides a convenient wrapper between projects using the Django and the Slack chat platform) to make it compatible with Celery version 5. [...]

• Created a pull request for the Emscripten LLVM-to-WebAssembly compiler to make the Document Object Model codes reproducible. [...]
• Added a link to my deprecated static analyser for Django to point to Richard Tier's Django Doctor. [...]

• As a board member of both the Open Source Initiative and Software in the Public Interest I attended their respective monthly meetings, including the bi-annual OSI multi-day "face-to-face" meetings.

• Made further progress on an academic paper in collaboration with Stefano Zacchiroli that details the theoretical and practical workings of the reproducible builds distributed consensus scheme.
• Created an upstream pull request for the Emscripten LLVM-to-WebAssembly compiler to make DOM codes reproducible (with a number of followups). [...]
• In Debian:
  • I kept isdebianreproducibleyet.com up to date. [...]
  • Submitted 11 patches to fix specific reproducibility issues in amavisd-milter, armagetronad, emscripten, less.js, metakernel, open-iscsi, os-autoinst, python-biom-format, python-pairix, requirejs & sympow.
  • Sent a large number of followups to old bugs that had not been updated for some time (for example, #968700, etc.)
• Categorised a large number of packages and issues in the Reproducible Builds "notes" repository, including categorising three new toolchain issues: build_path_captured_by_pyuic5, build_path_captured_by_octave & build_path_captured_by_nim.
• Drafted, published and publicised last month's report as well maintained the Project's various social media accounts.

• Updated the main Reproducible Builds website and documentation to clarify that SOURCE_DATE_EPOCH is not Debian specific [...] and make a number of misc cosmetic changes [...][...].

• Improvements:
  • Move the slightly-confusing behaviour if a single file is passed to diffoscope on the command-line to a new --load-existing-diff command. [...]
  • Ensure the new diffoscope-minimal package that was introduced by Mattia Rizzolo has a different short description from the primary diffoscope one. [...]
  • Refresh the long and short descriptions of all of the Debian packages. [...]
• Bug fixes:
  • Don't depend on radare2 in the Debian 'autopkgtests' as it will not be in bullseye due to security considerations. (#975313)
  • Avoid some incorrectly-formatted error messages. This was caused by diffoscope raising an artificial CalledProcessError exception in a generic handler. [...]
• Codebase improvements:
  • Add a comment regarding Java tests to aid diffoscope contributors who are not using Debian [...] and don't use the old-style super(...) call [...].

• python-django (2.2.17-1 & 3.1.3-1) New upstream releases.
• memcached (1.6.9+dfsg-1) New upstream release.

• lintian (2.101.0, 2.102.0, 2.103.0 & 2.104.0) New upstream releases.
• xtrlock (2.14) Mark an autopkgtest as 'superficial'. (#974491)
• bfs (2.1-1) New upstream release.
• splint (3.1.2+dfsg-3) Re-upload a previous QA upload of mine (3.1.2+dfsg-2) to ensure the package's transition to the testing distribution. (#974872)

• Investigated and triaged codemirror-js, glibc, jupyter-notebook, krb5, libhibernate3-java, raptor2, spice-vdagent & webcit.
• 'Frontdesk' duties, participating in mailing list discussions, attending the monthly meeting and organising LTS and ELTS frontdesk allocations for 2021.
• Issued DLA 2433-1 for the Bouncy Castle cryptography library to prevent an issue where attackers could obtain sensitive information due to observable differences in its responses to invalid input. (CVE-2020-26939)
• Issued DLA 2434-1 for the GNOME display manager (gdm3) where gdm3 detecting any users may have caused gdm3 to launch the initial system setup, permitting the creation of new users with superuser capabilities. (CVE-2020-16125)
• Issued DLA 2436-1 for the sddm display manager. Here, local and unprivileged users could create a connection to the X server. (CVE-2020-28049)
• Issued DLA 2437-1 & ELA-308-1 as it was discovered that there was a denial of service vulnerability in the MIT Kerberos network authentication system, krb5. The lack of a limit in an ASN.1 decoder could lead to infinite recursion and allow an attacker to overrun the stack and cause the process to crash. (CVE-2020-28196)
• Issued DLA 2438-1 and ELA-309-1 to prevent two heap overflow vulnerabilities in raptor2, a set of parsers for Resource Description Framework (RDF) files used in LibreOffice and other applications. (CVE-2017-18926)
• Issued DLA 2465-1 to correct filename sanitisation issues in a utility used to access PHP Pear, a distribution system for reusable PHP components. (CVE-2020-28948 & CVE-2020-28949)

Sings-to-Trees had hair the color of sunlight and ashes, delicately pointed ears, and eyes the translucent green of new leaves. His shirt was off, he had the sort of tanned muscle acquired from years of healthy outdoor living, and you could have sharpened a sword on his cheekbones. He was saved from being a young maiden's fantasy unless she was a very peculiar young maiden by the fact that he was buried up to the shoulder in the unpleasant end of a heavily pregnant unicorn.

Sings-to-Trees gazed off in the middle distance with a vague, pleasant expression, the way that most people do when present at other people's minor domestic disputes, and after a moment, the stag had stopped rattling, and the doe had turned back and rested her chin trustingly on Sings-to-Trees' shoulder. This would have been a touching gesture, if her chin hadn't been made of painfully pointy blades of bone. It was like being snuggled by an affectionate plow.

What does this mean? Over the coming weeks, I will:

• transition packages to be team-maintained where it makes sense
• remove myself from the Uploaders field on packages with other maintainers
• orphan packages where I am the sole maintainer

I will try to keep up best-effort maintenance of the manpages.debian.org service and the codesearch.debian.net service, but any help would be much appreciated. For all intents and purposes, please treat me as permanently on vacation. I will try to be around for administrative issues (e.g. permission transfers) and questions addressed directly to me, permitted they are easy enough to answer.

Why? When I joined Debian, I was still studying, i.e. I had luxurious amounts of spare time. Now, over 5 years of full time work later, my day job taught me a lot, both about what works in large software engineering projects and how I personally like my computer systems. I am very conscious of how I spend the little spare time that I have these days. The following sections each deal with what I consider a major pain point, in no particular order. Some of them influence each other for example, if changes worked better, we could have a chance at transitioning packages to be more easily machine readable.

Change process in Debian The last few years, my current team at work conducted various smaller and larger refactorings across the entire code base (touching thousands of projects), so we have learnt a lot of valuable lessons about how to effectively do these changes. It irks me that Debian works almost the opposite way in every regard. I appreciate that every organization is different, but I think a lot of my points do actually apply to Debian. In Debian, packages are nudged in the right direction by a document called the Debian Policy, or its programmatic embodiment, lintian. While it is great to have a lint tool (for quick, local/offline feedback), it is even better to not require a lint tool at all. The team conducting the change (e.g. the C++ team introduces a new hardening flag for all packages) should be able to do their work transparent to me. Instead, currently, all packages become lint-unclean, all maintainers need to read up on what the new thing is, how it might break, whether/how it affects them, manually run some tests, and finally decide to opt in. This causes a lot of overhead and manually executed mechanical changes across packages. Notably, the cost of each change is distributed onto the package maintainers in the Debian model. At work, we have found that the opposite works better: if the team behind the change is put in power to do the change for as many users as possible, they can be significantly more efficient at it, which reduces the total cost and time a lot. Of course, exceptions (e.g. a large project abusing a language feature) should still be taken care of by the respective owners, but the important bit is that the default should be the other way around. Debian is lacking tooling for large changes: it is hard to programmatically deal with packages and repositories (see the section below). The closest to sending out a change for review is to open a bug report with an attached patch. I thought the workflow for accepting a change from a bug report was too complicated and started mergebot, but only Guido ever signaled interest in the project. Culturally, reviews and reactions are slow. There are no deadlines. I literally sometimes get emails notifying me that a patch I sent out a few years ago (!!) is now merged. This turns projects from a small number of weeks into many years, which is a huge demotivator for me. Interestingly enough, you can see artifacts of the slow online activity manifest itself in the offline culture as well: I don t want to be discussing systemd s merits 10 years after I first heard about it. Lastly, changes can easily be slowed down significantly by holdouts who refuse to collaborate. My canonical example for this is rsync, whose maintainer refused my patches to make the package use debhelper purely out of personal preference. Granting so much personal freedom to individual maintainers prevents us as a project from raising the abstraction level for building Debian packages, which in turn makes tooling harder. How would things look like in a better world?

1. As a project, we should strive towards more unification. Uniformity still does not rule out experimentation, it just changes the trade-off from easier experimentation and harder automation to harder experimentation and easier automation.
2. Our culture needs to shift from this package is my domain, how dare you touch it to a shared sense of ownership, where anyone in the project can easily contribute (reviewed) changes without necessarily even involving individual maintainers.

To learn more about how successful large changes can look like, I recommend my colleague Hyrum Wright s talk Large-Scale Changes at Google: Lessons Learned From 5 Yrs of Mass Migrations .

Fragmented workflow and infrastructure Debian generally seems to prefer decentralized approaches over centralized ones. For example, individual packages are maintained in separate repositories (as opposed to in one repository), each repository can use any SCM (git and svn are common ones) or no SCM at all, and each repository can be hosted on a different site. Of course, what you do in such a repository also varies subtly from team to team, and even within teams. In practice, non-standard hosting options are used rarely enough to not justify their cost, but frequently enough to be a huge pain when trying to automate changes to packages. Instead of using GitLab s API to create a merge request, you have to design an entirely different, more complex system, which deals with intermittently (or permanently!) unreachable repositories and abstracts away differences in patch delivery (bug reports, merge requests, pull requests, email, ). Wildly diverging workflows is not just a temporary problem either. I participated in long discussions about different git workflows during DebConf 13, and gather that there were similar discussions in the meantime. Personally, I cannot keep enough details of the different workflows in my head. Every time I touch a package that works differently than mine, it frustrates me immensely to re-learn aspects of my day-to-day. After noticing workflow fragmentation in the Go packaging team (which I started), I tried fixing this with the workflow changes proposal, but did not succeed in implementing it. The lack of effective automation and slow pace of changes in the surrounding tooling despite my willingness to contribute time and energy killed any motivation I had.

Old infrastructure: package uploads When you want to make a package available in Debian, you upload GPG-signed files via anonymous FTP. There are several batch jobs (the queue daemon, unchecked, dinstall, possibly others) which run on fixed schedules (e.g. dinstall runs at 01:52 UTC, 07:52 UTC, 13:52 UTC and 19:52 UTC). Depending on timing, I estimated that you might wait for over 7 hours (!!) before your package is actually installable. What s worse for me is that feedback to your upload is asynchronous. I like to do one thing, be done with it, move to the next thing. The current setup requires a many-minute wait and costly task switch for no good technical reason. You might think a few minutes aren t a big deal, but when all the time I can spend on Debian per day is measured in minutes, this makes a huge difference in perceived productivity and fun. The last communication I can find about speeding up this process is ganneff s post from 2008. How would things look like in a better world?

1. Anonymous FTP would be replaced by a web service which ingests my package and returns an authoritative accept or reject decision in its response.
2. For accepted packages, there would be a status page displaying the build status and when the package will be available via the mirror network.
3. Packages should be available within a few minutes after the build completed.

Old infrastructure: bug tracker I dread interacting with the Debian bug tracker. debbugs is a piece of software (from 1994) which is only used by Debian and the GNU project these days. Debbugs processes emails, which is to say it is asynchronous and cumbersome to deal with. Despite running on the fastest machines we have available in Debian (or so I was told when the subject last came up), its web interface loads very slowly. Notably, the web interface at bugs.debian.org is read-only. Setting up a working email setup for reportbug(1) or manually dealing with attachments is a rather big hurdle. For reasons I don t understand, every interaction with debbugs results in many different email threads. Aside from the technical implementation, I also can never remember the different ways that Debian uses pseudo-packages for bugs and processes. I need them rarely enough to establish a mental model of how they are set up, or working memory of how they are used, but frequently enough to be annoyed by this. How would things look like in a better world?

1. Debian would switch from a custom bug tracker to a (any) well-established one.
2. Debian would offer automation around processes. It is great to have a paper-trail and artifacts of the process in the form of a bug report, but the primary interface should be more convenient (e.g. a web form).

Old infrastructure: mailing list archives It baffles me that in 2019, we still don t have a conveniently browsable threaded archive of mailing list discussions. Email and threading is more widely used in Debian than anywhere else, so this is somewhat ironic. Gmane used to paper over this issue, but Gmane s availability over the last few years has been spotty, to say the least (it is down as I write this). I tried to contribute a threaded list archive, but our listmasters didn t seem to care or want to support the project.

Debian is hard to machine-read While it is obviously possible to deal with Debian packages programmatically, the experience is far from pleasant. Everything seems slow and cumbersome. I have picked just 3 quick examples to illustrate my point. debiman needs help from piuparts in analyzing the alternatives mechanism of each package to display the manpages of e.g. psql(1). This is because maintainer scripts modify the alternatives database by calling shell scripts. Without actually installing a package, you cannot know which changes it does to the alternatives database. pk4 needs to maintain its own cache to look up package metadata based on the package name. Other tools parse the apt database from scratch on every invocation. A proper database format, or at least a binary interchange format, would go a long way. Debian Code Search wants to ingest new packages as quickly as possible. There used to be a fedmsg instance for Debian, but it no longer seems to exist. It is unclear where to get notifications from for new packages, and where best to fetch those packages.

Complicated build stack See my Debian package build tools post. It really bugs me that the sprawl of tools is not seen as a problem by others.

Developer experience pretty painful Most of the points discussed so far deal with the experience in developing Debian, but as I recently described in my post Debugging experience in Debian , the experience when developing using Debian leaves a lot to be desired, too.

I have more ideas At this point, the article is getting pretty long, and hopefully you got a rough idea of my motivation. While I described a number of specific shortcomings above, the final nail in the coffin is actually the lack of a positive outlook. I have more ideas that seem really compelling to me, but, based on how my previous projects have been going, I don t think I can make any of these ideas happen within the Debian project. I intend to publish a few more posts about specific ideas for improving operating systems here. Stay tuned. Lastly, I hope this post inspires someone, ideally a group of people, to improve the developer experience within Debian.

Changes in RcppArmadillo version 0.10.1.2.0 (2020-11-15)

• Upgraded to Armadillo release 10.1.2 (Orchid Ambush)
• Remove three unused int constants (#313)
• Include main armadillo header using quotes instead of brackets
• Rewrite version number use in old-school mode because gcc 4.8.5
• Skipping parts of sparse conversion on Windows as win-builder fails

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.