Because posting private keys on the Internet is a bad idea, some people like to redact their private keys, so that it looks kinda-sorta like a private key, but it isn t actually giving away anything secret. Unfortunately, due to the way that private keys are represented, it is easy to redact a key in such a way that it doesn t actually redact anything at all. RSA private keys are particularly bad at this, but the problem can (potentially) apply to other keys as well. I ll show you a bit of Inside Baseball with key formats, and then demonstrate the practical implications. Finally, we ll go through a practical worked example from an actual not-really-redacted key I recently stumbled across in my travels.

The Private Lives of Private Keys Here is what a typical private key looks like, when you come across it:

-----BEGIN RSA PRIVATE KEY-----
MGICAQACEQCxjdTmecltJEz2PLMpS4BXAgMBAAECEDKtuwD17gpagnASq1zQTYEC
CQDVTYVsjjF7IQIJANUYZsIjRsR3AgkAkahDUXL0RSECCB78r2SnsJC9AghaOK3F
sKoELg==
-----END RSA PRIVATE KEY-----

Obviously, there s some hidden meaning in there computers don t encrypt things by shouting BEGIN RSA PRIVATE KEY! , after all. What is between the BEGIN/END lines above is, in fact, a base64-encoded DER format ASN.1 structure representing a PKCS#1 private key. In simple terms, it s a list of numbers very important numbers. The list of numbers is, in order:

• A version number (0);
• The public modulus , commonly referred to as n ;
• The public exponent , or e (which is almost always 65,537, for various unimportant reasons);
• The private exponent , or d ;
• The two private primes , or p and q ;
• Two exponents, which are known as dmp1 and dmq1 ; and
• A coefficient, known as iqmp .

Why Is This a Problem? The thing is, only three of those numbers are actually required in a private key. The rest, whilst useful to allow the RSA encryption and decryption to be more efficient, aren t necessary. The three absolutely required values are e, p, and q. Of the other numbers, most of them are at least about the same size as each of p and q. So of the total data in an RSA key, less than a quarter of the data is required. Let me show you with the above toy key, by breaking it down piece by piece¹:

• MGI DER for this is a sequence
• CAQ version (0)
• CxjdTmecltJEz2PLMpS4BX n
• AgMBAA e
• ECEDKtuwD17gpagnASq1zQTY d
• ECCQDVTYVsjjF7IQ p
• IJANUYZsIjRsR3 q
• AgkAkahDUXL0RS dmp1
• ECCB78r2SnsJC9 dmq1
• AghaOK3FsKoELg== iqmp

Remember that in order to reconstruct all of these values, all I need are e, p, and q and e is pretty much always 65,537. So I could redact almost all of this key, and still give all the important, private bits of this key. Let me show you:

-----BEGIN RSA PRIVATE KEY-----
..............................................................EC
CQDVTYVsjjF7IQIJANUYZsIjRsR3....................................
........
-----END RSA PRIVATE KEY-----

Now, I doubt that anyone is going to redact a key precisely like this but then again, this isn t a typical RSA key. They usually look a lot more like this:

-----BEGIN RSA PRIVATE KEY-----
MIIEogIBAAKCAQEAu6Inch7+mWtKn+leB9uCG3MaJIxRyvC/5KTz2fR+h+GOhqj4
SZJobiVB4FrE5FgC7AnlH6qeRi9MI0s6dt5UWZ5oNIeWSaOOeNO+EJDUkSVf67wj
SNGXlSjGAkPZ0nRJiDjhuPvQmdW53hOaBLk5udxPEQbenpXAzbLJ7wH5ouLQ3nQw
HwpwDNQhF6zRO8WoscpDVThOAM+s4PS7EiK8ZR4hu2toon8Ynadlm95V45wR0VlW
zywgbkZCKa1IMrDCscB6CglQ10M3Xzya3iTzDtQxYMVqhDrA7uBYRxA0y1sER+Rb
yhEh03xz3AWemJVLCQuU06r+FABXJuY/QuAVvQIDAQABAoIBAFqwWVhzWqNUlFEO
PoCVvCEAVRZtK+tmyZj9kU87ORz8DCNR8A+/T/JM17ZUqO2lDGSBs9jGYpGRsr8s
USm69BIM2ljpX95fyzDjRu5C0jsFUYNi/7rmctmJR4s4uENcKV5J/++k5oI0Jw4L
c1ntHNWUgjK8m0UTJIlHbQq0bbAoFEcfdZxd3W+SzRG3jND3gifqKxBG04YDwloy
tu+bPV2jEih6p8tykew5OJwtJ3XsSZnqJMwcvDciVbwYNiJ6pUvGq6Z9kumOavm9
XU26m4cWipuK0URWbHWQA7SjbktqEpxsFrn5bYhJ9qXgLUh/I1+WhB2GEf3hQF5A
pDTN4oECgYEA7Kp6lE7ugFBDC09sKAhoQWrVSiFpZG4Z1gsL9z5YmZU/vZf0Su0n
9J2/k5B1GghvSwkTqpDZLXgNz8eIX0WCsS1xpzOuORSNvS1DWuzyATIG2cExuRiB
jYWIJUeCpa5p2PdlZmBrnD/hJ4oNk4oAVpf+HisfDSN7HBpN+TJfcAUCgYEAyvY7
Y4hQfHIdcfF3A9eeCGazIYbwVyfoGu70S/BZb2NoNEPymqsz7NOfwZQkL4O7R3Wl
Rm0vrWT8T5ykEUgT+2ruZVXYSQCKUOl18acbAy0eZ81wGBljZc9VWBrP1rHviVWd
OVDRZNjz6nd6ZMrJvxRa24TvxZbJMmO1cgSW1FkCgYAoWBd1WM9HiGclcnCZknVT
UYbykCeLO0mkN1Xe2/32kH7BLzox26PIC2wxF5seyPlP7Ugw92hOW/zewsD4nLze
v0R0oFa+3EYdTa4BvgqzMXgBfvGfABJ1saG32SzoWYcpuWLLxPwTMsCLIPmXgRr1
qAtl0SwF7Vp7O/C23mNukQKBgB89DOEB7xloWv3Zo27U9f7nB7UmVsGjY8cZdkJl
6O4LB9PbjXCe3ywZWmJqEbO6e83A3sJbNdZjT65VNq9uP50X1T+FmfeKfL99X2jl
RnQTsrVZWmJrLfBSnBkmb0zlMDAcHEnhFYmHFuvEnfL7f1fIoz9cU6c+0RLPY/L7
n9dpAoGAXih17mcmtnV+Ce+lBWzGWw9P4kVDSIxzGxd8gprrGKLa3Q9VuOrLdt58
++UzNUaBN6VYAe4jgxGfZfh+IaSlMouwOjDgE/qzgY8QsjBubzmABR/KWCYiRqkj
qpWCgo1FC1Gn94gh/+dW2Q8+NjYtXWNqQcjRP4AKTBnPktEvdMA=
-----END RSA PRIVATE KEY-----

People typically redact keys by deleting whole lines, and usually replacing them with [...] and the like. But only about 345 of those 1588 characters (excluding the header and footer) are required to construct the entire key. You can redact about 4/5ths of that giant blob of stuff, and your private parts (or at least, those of your key) are still left uncomfortably exposed.

But Wait! There s More! Remember how I said that everything in the key other than e, p, and q could be derived from those three numbers? Let s talk about one of those numbers: n. This is known as the public modulus (because, along with e, it is also present in the public key). It is very easy to calculate: n = p * q. It is also very early in the key (the second number, in fact). Since n = p * q, it follows that q = n / p. Thus, as long as the key is intact up to p, you can derive q by simple division.

Real World Redaction At this point, I d like to introduce an acquaintance of mine: Mr. Johan Finn. He is the proud owner of the GitHub repo johanfinn/scripts. For a while, his repo contained a script that contained a poorly-redacted private key. He since deleted it, by making a new commit, but of course because git never really deletes anything, it s still available. Of course, Mr. Finn may delete the repo, or force-push a new history without that commit, so here is the redacted private key, with a bit of the surrounding shell script, for our illustrative pleasure:

#Add private key to .ssh folder
cd /home/johan/.ssh/
echo  "-----BEGIN RSA PRIVATE KEY-----
MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM
NNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN
KKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKK
 
MIIJKgIBAAKCAgEAxEVih1JGb8gu/Fm4AZh+ZwJw/pjzzliWrg4mICFt1g7SmIE2
TCQMKABdwd11wOFKCPc/UzRH/fHuQcvWrpbOSdqev/zKff9iedKw/YygkMeIRaXB
fYELqvUAOJ8PPfDm70st9GJRhjGgo5+L3cJB2gfgeiDNHzaFvapRSU0oMGQX+kI9
ezsjDAn+0Pp+r3h/u1QpLSH4moRFGF4omNydI+3iTGB98/EzuNhRBHRNq4oBV5SG
Pq/A1bem2ninnoEaQ+OPESxYzDz3Jy9jV0W/6LvtJ844m+XX69H5fqq5dy55z6DW
sGKn78ULPVZPsYH5Y7C+CM6GAn4nYCpau0t52sqsY5epXdeYx4Dc+Wm0CjXrUDEe
Egl4loPKDxJkQqQ/MQiz6Le/UK9vEmnWn1TRXK3ekzNV4NgDfJANBQobOpwt8WVB
rbsC0ON7n680RQnl7PltK9P1AQW5vHsahkoixk/BhcwhkrkZGyDIl9g8Q/Euyoq3
eivKPLz7/rhDE7C1BzFy7v8AjC3w7i9QeHcWOZFAXo5hiDasIAkljDOsdfD4tP5/
wSO6E6pjL3kJ+RH2FCHd7ciQb+IcuXbku64ln8gab4p8jLa/mcMI+V3eWYnZ82Yu
axsa85hAe4wb60cp/rCJo7ihhDTTvGooqtTisOv2nSvCYpcW9qbL6cGjAXECAwEA
AQKCAgEAjz6wnWDP5Y9ts2FrqUZ5ooamnzpUXlpLhrbu3m5ncl4ZF5LfH+QDN0Kl
KvONmHsUhJynC/vROybSJBU4Fu4bms1DJY3C39h/L7g00qhLG7901pgWMpn3QQtU
4P49qpBii20MGhuTsmQQALtV4kB/vTgYfinoawpo67cdYmk8lqzGzzB/HKxZdNTq
s+zOfxRr7PWMo9LyVRuKLjGyYXZJ/coFaobWBi8Y96Rw5NZZRYQQXLIalC/Dhndm
AHckpstEtx2i8f6yxEUOgPvV/gD7Akn92RpqOGW0g/kYpXjGqZQy9PVHGy61sInY
HSkcOspIkJiS6WyJY9JcvJPM6ns4b84GE9qoUlWVF3RWJk1dqYCw5hz4U8LFyxsF
R6WhYiImvjxBLpab55rSqbGkzjI2z+ucDZyl1gqIv9U6qceVsgRyuqdfVN4deU22
LzO5IEDhnGdFqg9KQY7u8zm686Ejs64T1sh0y4GOmGsSg+P6nsqkdlXH8C+Cf03F
lqPFg8WQC7ojl/S8dPmkT5tcJh3BPwIWuvbtVjFOGQc8x0lb+NwK8h2Nsn6LNazS
0H90adh/IyYX4sBMokrpxAi+gMAWiyJHIHLeH2itNKtAQd3qQowbrWNswJSgJzsT
JuJ7uqRKAFkE6nCeAkuj/6KHHMPsfCAffVdyGaWqhoxmPOrnVgECggEBAOrCCwiC
XxwUgjOfOKx68siFJLfHf4vPo42LZOkAQq5aUmcWHbJVXmoxLYSczyAROopY0wd6
Dx8rqnpO7OtZsdJMeBSHbMVKoBZ77hiCQlrljcj12moFaEAButLCdZFsZW4zF/sx
kWIAaPH9vc4MvHHyvyNoB3yQRdevu57X7xGf9UxWuPil/jvdbt9toaraUT6rUBWU
GYPNKaLFsQzKsFWAzp5RGpASkhuiBJ0Qx3cfLyirjrKqTipe3o3gh/5RSHQ6VAhz
gdUG7WszNWk8FDCL6RTWzPOrbUyJo/wz1kblsL3vhV7ldEKFHeEjsDGroW2VUFlS
asAHNvM4/uYcOSECggEBANYH0427qZtLVuL97htXW9kCAT75xbMwgRskAH4nJDlZ
IggDErmzBhtrHgR+9X09iL47jr7dUcrVNPHzK/WXALFSKzXhkG/yAgmt3r14WgJ6
5y7010LlPFrzaNEyO/S4ISuBLt4cinjJsrFpoo0WI8jXeM5ddG6ncxdurKXMymY7
:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::.::
:::::::::::::::::::::::::::.::::::::::::::::::::::::::::::::::::
LLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLlL
 
 
 
YYYYYYYYYYYYYYYYYYYYYyYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY
gff0GJCOMZ65pMSy3A3cSAtjlKnb4fWzuHD5CFbusN4WhCT/tNxGNSpzvxd8GIDs
nY7exs9L230oCCpedVgcbayHCbkChEfoPzL1e1jXjgCwCTgt8GjeEFqc1gXNEaUn
O8AJ4VlR8fRszHm6yR0ZUBdY7UJddxQiYOzt0S1RLlECggEAbdcs4mZdqf3OjejJ
06oTPs9NRtAJVZlppSi7pmmAyaNpOuKWMoLPElDAQ3Q7VX26LlExLCZoPOVpdqDH
KbdmBEfTR4e11Pn9vYdu9/i6o10U4hpmf4TYKlqk10g1Sj21l8JATj/7Diey8scO
sAI1iftSg3aBSj8W7rxCxSezrENzuqw5D95a/he1cMUTB6XuravqZK5O4eR0vrxR
AvMzXk5OXrUEALUvt84u6m6XZZ0pq5XZxq74s8p/x1JvTwcpJ3jDKNEixlHfdHEZ
ZIu/xpcwD5gRfVGQamdcWvzGHZYLBFO1y5kAtL8kI9tW7WaouWVLmv99AyxdAaCB
Y5mBAQKCAQEAzU7AnorPzYndlOzkxRFtp6MGsvRBsvvqPLCyUFEXrHNV872O7tdO
GmsMZl+q+TJXw7O54FjJJvqSSS1sk68AGRirHop7VQce8U36BmI2ZX6j2SVAgIkI
9m3btCCt5rfiCatn2+Qg6HECmrCsHw6H0RbwaXS4RZUXD/k4X+sslBitOb7K+Y+N
Bacq6QxxjlIqQdKKPs4P2PNHEAey+kEJJGEQ7bTkNxCZ21kgi1Sc5L8U/IGy0BMC
PvJxssLdaWILyp3Ws8Q4RAoC5c0ZP0W2j+5NSbi3jsDFi0Y6/2GRdY1HAZX4twem
Q0NCedq1JNatP1gsb6bcnVHFDEGsj/35oQKCAQEAgmWMuSrojR/fjJzvke6Wvbox
FRnPk+6YRzuYhAP/YPxSRYyB5at++5Q1qr7QWn7NFozFIVFFT8CBU36ktWQ39MGm
cJ5SGyN9nAbbuWA6e+/u059R7QL+6f64xHRAGyLT3gOb1G0N6h7VqFT25q5Tq0rc
Lf/CvLKoudjv+sQ5GKBPT18+zxmwJ8YUWAsXUyrqoFWY/Tvo5yLxaC0W2gh3+Ppi
EDqe4RRJ3VKuKfZxHn5VLxgtBFN96Gy0+Htm5tiMKOZMYAkHiL+vrVZAX0hIEuRZ
JJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJ
MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM
-----END RSA PRIVATE KEY-----" >> id_rsa

Now, if you try to reconstruct this key by removing the obvious garbage lines (the ones that are all repeated characters, some of which aren t even valid base64 characters), it still isn t a key at least, openssl pkey doesn t want anything to do with it. The key is very much still in there, though, as we shall soon see. Using a gem I wrote and a quick bit of Ruby, we can extract a complete private key. The irb session looks something like this:

>> require "derparse"
>> b64 = <<EOF
MIIJKgIBAAKCAgEAxEVih1JGb8gu/Fm4AZh+ZwJw/pjzzliWrg4mICFt1g7SmIE2
TCQMKABdwd11wOFKCPc/UzRH/fHuQcvWrpbOSdqev/zKff9iedKw/YygkMeIRaXB
fYELqvUAOJ8PPfDm70st9GJRhjGgo5+L3cJB2gfgeiDNHzaFvapRSU0oMGQX+kI9
ezsjDAn+0Pp+r3h/u1QpLSH4moRFGF4omNydI+3iTGB98/EzuNhRBHRNq4oBV5SG
Pq/A1bem2ninnoEaQ+OPESxYzDz3Jy9jV0W/6LvtJ844m+XX69H5fqq5dy55z6DW
sGKn78ULPVZPsYH5Y7C+CM6GAn4nYCpau0t52sqsY5epXdeYx4Dc+Wm0CjXrUDEe
Egl4loPKDxJkQqQ/MQiz6Le/UK9vEmnWn1TRXK3ekzNV4NgDfJANBQobOpwt8WVB
rbsC0ON7n680RQnl7PltK9P1AQW5vHsahkoixk/BhcwhkrkZGyDIl9g8Q/Euyoq3
eivKPLz7/rhDE7C1BzFy7v8AjC3w7i9QeHcWOZFAXo5hiDasIAkljDOsdfD4tP5/
wSO6E6pjL3kJ+RH2FCHd7ciQb+IcuXbku64ln8gab4p8jLa/mcMI+V3eWYnZ82Yu
axsa85hAe4wb60cp/rCJo7ihhDTTvGooqtTisOv2nSvCYpcW9qbL6cGjAXECAwEA
AQKCAgEAjz6wnWDP5Y9ts2FrqUZ5ooamnzpUXlpLhrbu3m5ncl4ZF5LfH+QDN0Kl
KvONmHsUhJynC/vROybSJBU4Fu4bms1DJY3C39h/L7g00qhLG7901pgWMpn3QQtU
4P49qpBii20MGhuTsmQQALtV4kB/vTgYfinoawpo67cdYmk8lqzGzzB/HKxZdNTq
s+zOfxRr7PWMo9LyVRuKLjGyYXZJ/coFaobWBi8Y96Rw5NZZRYQQXLIalC/Dhndm
AHckpstEtx2i8f6yxEUOgPvV/gD7Akn92RpqOGW0g/kYpXjGqZQy9PVHGy61sInY
HSkcOspIkJiS6WyJY9JcvJPM6ns4b84GE9qoUlWVF3RWJk1dqYCw5hz4U8LFyxsF
R6WhYiImvjxBLpab55rSqbGkzjI2z+ucDZyl1gqIv9U6qceVsgRyuqdfVN4deU22
LzO5IEDhnGdFqg9KQY7u8zm686Ejs64T1sh0y4GOmGsSg+P6nsqkdlXH8C+Cf03F
lqPFg8WQC7ojl/S8dPmkT5tcJh3BPwIWuvbtVjFOGQc8x0lb+NwK8h2Nsn6LNazS
0H90adh/IyYX4sBMokrpxAi+gMAWiyJHIHLeH2itNKtAQd3qQowbrWNswJSgJzsT
JuJ7uqRKAFkE6nCeAkuj/6KHHMPsfCAffVdyGaWqhoxmPOrnVgECggEBAOrCCwiC
XxwUgjOfOKx68siFJLfHf4vPo42LZOkAQq5aUmcWHbJVXmoxLYSczyAROopY0wd6
Dx8rqnpO7OtZsdJMeBSHbMVKoBZ77hiCQlrljcj12moFaEAButLCdZFsZW4zF/sx
kWIAaPH9vc4MvHHyvyNoB3yQRdevu57X7xGf9UxWuPil/jvdbt9toaraUT6rUBWU
GYPNKaLFsQzKsFWAzp5RGpASkhuiBJ0Qx3cfLyirjrKqTipe3o3gh/5RSHQ6VAhz
gdUG7WszNWk8FDCL6RTWzPOrbUyJo/wz1kblsL3vhV7ldEKFHeEjsDGroW2VUFlS
asAHNvM4/uYcOSECggEBANYH0427qZtLVuL97htXW9kCAT75xbMwgRskAH4nJDlZ
IggDErmzBhtrHgR+9X09iL47jr7dUcrVNPHzK/WXALFSKzXhkG/yAgmt3r14WgJ6
5y7010LlPFrzaNEyO/S4ISuBLt4cinjJsrFpoo0WI8jXeM5ddG6ncxdurKXMymY7
EOF
>> b64 += <<EOF
gff0GJCOMZ65pMSy3A3cSAtjlKnb4fWzuHD5CFbusN4WhCT/tNxGNSpzvxd8GIDs
nY7exs9L230oCCpedVgcbayHCbkChEfoPzL1e1jXjgCwCTgt8GjeEFqc1gXNEaUn
O8AJ4VlR8fRszHm6yR0ZUBdY7UJddxQiYOzt0S1RLlECggEAbdcs4mZdqf3OjejJ
06oTPs9NRtAJVZlppSi7pmmAyaNpOuKWMoLPElDAQ3Q7VX26LlExLCZoPOVpdqDH
KbdmBEfTR4e11Pn9vYdu9/i6o10U4hpmf4TYKlqk10g1Sj21l8JATj/7Diey8scO
sAI1iftSg3aBSj8W7rxCxSezrENzuqw5D95a/he1cMUTB6XuravqZK5O4eR0vrxR
AvMzXk5OXrUEALUvt84u6m6XZZ0pq5XZxq74s8p/x1JvTwcpJ3jDKNEixlHfdHEZ
ZIu/xpcwD5gRfVGQamdcWvzGHZYLBFO1y5kAtL8kI9tW7WaouWVLmv99AyxdAaCB
Y5mBAQKCAQEAzU7AnorPzYndlOzkxRFtp6MGsvRBsvvqPLCyUFEXrHNV872O7tdO
GmsMZl+q+TJXw7O54FjJJvqSSS1sk68AGRirHop7VQce8U36BmI2ZX6j2SVAgIkI
9m3btCCt5rfiCatn2+Qg6HECmrCsHw6H0RbwaXS4RZUXD/k4X+sslBitOb7K+Y+N
Bacq6QxxjlIqQdKKPs4P2PNHEAey+kEJJGEQ7bTkNxCZ21kgi1Sc5L8U/IGy0BMC
PvJxssLdaWILyp3Ws8Q4RAoC5c0ZP0W2j+5NSbi3jsDFi0Y6/2GRdY1HAZX4twem
Q0NCedq1JNatP1gsb6bcnVHFDEGsj/35oQKCAQEAgmWMuSrojR/fjJzvke6Wvbox
FRnPk+6YRzuYhAP/YPxSRYyB5at++5Q1qr7QWn7NFozFIVFFT8CBU36ktWQ39MGm
cJ5SGyN9nAbbuWA6e+/u059R7QL+6f64xHRAGyLT3gOb1G0N6h7VqFT25q5Tq0rc
Lf/CvLKoudjv+sQ5GKBPT18+zxmwJ8YUWAsXUyrqoFWY/Tvo5yLxaC0W2gh3+Ppi
EDqe4RRJ3VKuKfZxHn5VLxgtBFN96Gy0+Htm5tiMKOZMYAkHiL+vrVZAX0hIEuRZ
EOF
>> der = b64.unpack("m").first
>> c = DerParse.new(der).first_node.first_child
>> version = c.value
=> 0
>> c = c.next_node
>> n = c.value
=> 80071596234464993385068908004931... # (etc)
>> c = c.next_node
>> e = c.value
=> 65537
>> c = c.next_node
>> d = c.value
=> 58438813486895877116761996105770... # (etc)
>> c = c.next_node
>> p = c.value
=> 29635449580247160226960937109864... # (etc)
>> c = c.next_node
>> q = c.value
=> 27018856595256414771163410576410... # (etc)

What I ve done, in case you don t speak Ruby, is take the two chunks of plausible-looking base64 data, chuck them together into a variable named b64, unbase64 it into a variable named der, pass that into a new DerParse instance, and then walk the DER value tree until I got all the values I need. Interestingly, the q value actually traverses the split in the two chunks, which means that there s always the possibility that there are lines missing from the key. However, since p and q are supposed to be prime, we can sanity check them to see if corruption is likely to have occurred:

>> require "openssl"
>> OpenSSL::BN.new(p).prime?
=> true
>> OpenSSL::BN.new(q).prime?
=> true

Excellent! The chances of a corrupted file producing valid-but-incorrect prime numbers isn t huge, so we can be fairly confident that we ve got the real p and q. Now, with the help of another one of my creations we can use e, p, and q to create a fully-operational battle key:

>> require "openssl/pkey/rsa"
>> k = OpenSSL::PKey::RSA.from_factors(p, q, e)
=> #<OpenSSL::PKey::RSA:0x0000559d5903cd38>
>> k.valid?
=> true
>> k.verify(OpenSSL::Digest::SHA256.new, k.sign(OpenSSL::Digest::SHA256.new, "bob"), "bob")
=> true

and there you have it. One fairly redacted-looking private key brought back to life by maths and far too much free time. Sorry Mr. Finn, I hope you re not still using that key on anything Internet-facing.

What About Other Key Types? EC keys are very different beasts, but they have much the same problems as RSA keys. A typical EC key contains both private and public data, and the public portion is twice the size so only about 1/3 of the data in the key is private material. It is quite plausible that you can redact an EC key and leave all the actually private bits exposed.

What Do We Do About It? In short: don t ever try and redact real private keys. For documentation purposes, just put KEY GOES HERE in the appropriate spot, or something like that. Store your secrets somewhere that isn t a public (or even private!) git repo. Generating a dummy private key and sticking it in there isn t a great idea, for different reasons: people have this odd habit of reusing demo keys in real life. There s no need to encourage that sort of thing.

---

1. Technically the pieces aren t 100% aligned with the underlying DER, because of how base64 works. I felt it was easier to understand if I stuck to chopping up the base64, rather than decoding into DER and then chopping up the DER.

Having recently switched from NVIDIA to AMD graphic cards, in particular a RX 5700, I found out that I can get myself a free upgrade to the RX 5700 XT variant without paying one Yen, by simply flashing a compatible 5700 XT BIOS onto the 5700 card. Not that this is something new, a detailed explanation can be found here.
The same article also gives a detailed technical explanation on the difference between the two cards. The 5700 variant has less stream processors (2304 against 2560 in the XT variant), and lower power limits and clock speeds. Other than this they are based on the exact same chip layout (Navi 10), and with the same amount and type of memory 8 GB GDDR6. Flashing the XT BIOS onto the plain 5700 will not changes the number of stream processors, but power limits and clock speeds are raised to the same level of the 5700 XT, providing approximately a 7% gain without any over-clocking and over-powering, and potentially more by raising voltage etc. Detailed numbers can be found in the linked article above. The first step in this free upgrade is to identify ones own card correctly, best with device id and subsystem id, and then find the correct BIOS. Lots of BIOS dumps are provided in the BIOS database (link already restricting to 5700 XT BIOS). I used CPU-Z (Windows program) to determine this items, see image on the right (click to enlarge). In my case I got 1002 731F - 1462 3811 for the complete device id. The card is a MSI RX 5700 8 GB Mech OC, so I found the following alternative BIOS for MSI RX 5700 XT 8 GB Mech OC. Unfortunately, it seems that MSI is distinguishing 5700 and 5700 XT by their device id, because the XT variant gives 1002 731F - 1462 3810 for the complete device id, meaning that the last digit is 1 off compared to mine (3811 versus 3810). And indeed, trying to flash this video BIOS the normal way (using the Windows version ended in a warning that the subsystem id is different. A bit of search led to a thread in the TechPowerup Fora and this post explaining how to force the flashing in this case. Disclaimer: The following might brick your graphic card, you are doing this on your own risk! Necessary software:

• CPU-Z: https://www.techpowerup.com/download/cpu-z/
• ATI/AMB Flash program: https://www.techpowerup.com/download/ati-atiflash/

I did all the flashing and checking under Windows, but only because I realized too late that there is a fully uptodate flashing program for Linux that exhibits the same functionality. Also, I didn t know how to get the device id since the current AMD ROCm tools seem not to provide this data. If you are lucky and the device ids for your card are the same for both 5700 and 5700 XT variants, then you can use the graphical client (amdvbflashWin.exe), but if there is a difference, the command line is necessary. After unpacking the AMD Flash program and getting the correct BIOS rom file, the steps taken on Windows are (the very same steps can be taken on Linux):

• Start a command line shell (cmd or powershell) with Administrator rights (on Linux become root)
• Save your current BIOS in case you need to restore it with amdvbflash -s 0 oldbios.rom (this can also be done out of the GUI application)
• Unlock the BIOS rom with amdvbflash -unlockrom 0
• Force flash the new BIOS with amdvbflash -f -p 0 NEW.rom (where NEW.rom is the 5700 XT BIOS rom file)

This should succeed in both cases. After that shutdown and restart your computer and you should be greeted with a RT 5700 XT card, without twisting a single screw. Starting Windows for the first time gave some flickering, because the driver for the new card was installed. On Linux the system auto-detects the card and everything works out of the box. Very smooth. Finally, a word of warning: Don t do these kind of things if you are not ready to pay the prize of a bricked GPU card in case something goes wrong! Everything is on your own risk! Let me close with a before/after image, most of the fields are identical, but the default/gpu clocks both at normal as well as boost levels see a considerable improvement

Earlier this year, I was supposed to participate to dotPy, a one-day Python conference happening in Paris. This event has unfortunately been cancelled due to the COVID-19 pandemic.Both Victor Stinner and me were supposed to attend that event. Victor had prepared a presentation about Python performances, while I was planning on talking about profiling.Rather than being completely discouraged, Victor and I sat down (remotely) with Anne Laure from Behind the Code (a blog ran by Welcome to the Jungle, the organizers of the dotPy conference).We discuss Python performance, profiling, speed, projects, problems, analysis, optimization and the GIL.You can read the interview here.

The first proper video (following last week s announcement) is up for new T^4 series of video lightning talks with tips, tricks, tools, and toys. Today we just to a little enhancement for the shell enabled color output (if not already on by default).

The slides are available here. Next week we continue on shell customization by looking at the prompt. Also of note, a new repo at GitHub to support the series: use it to open issues for comments, criticism, suggestions, or feedback. If you like this or other open-source work I do, you can now sponsor me at GitHub. For the first year, GitHub will match your contributions.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

I was assigned 20 hours of work by Freexian's Debian LTS initiative, and carried over 8.5 hours from March. I worked 26 hours this month, so I will carry over 2.5 hours to May. I sent a (belated) request for testing an update of the linux package to 3.16.82. I then prepared and, after review, released Linux 3.16.83, including a large number of security fixes. I rebased the linux package onto that and will soon send out a request for testing. I also spent some time working on a still-embargoed security issue. I did not spend signficant time on any other LTS activities this month, and unfortunately missed the contributor meeting.

I made a webpage to produce virtual background with webcam. https://knok.github.io/virtbg/ Sorcecode:
https://github.com/knok/knok.github.io/tree/master/virtbg Some online meeting software (Zoom, Microsoft Teams) supports virtual background, but I want to use other software like Jitsi (or google meet), so I made it. To make this, I referred the article Open Source Virtual Background .
The following figure is the diagram.

It depends on docker, GPU, v4l2loopback (only works on Linux), so I want to make more generic solution. To make as a webpage, and using OBS
Studio with plugins (obs-v4l2sink, OBS-VirtualCam or OBS (macOS) Virtual Camera) you can use the solution on more platforms. To make as a single webpage, I can reduce overhead using inter-process commuication using http via docker. This is an example animation:

Using jisti snapshot:

Unfortunately, BodyPix releases only pretraind models, no training data. I need more improvements:

• Accept any background images
• Suppot choose camera device
• Useful UI

Review: Golden Gates, by Conor Dougherty

Publisher:	Penguin
Copyright:	2020
ISBN:	0-525-56022-X
Format:	Kindle
Pages:	249

This review, for reasons that will hopefully become clear later, starts with a personal digression. I have been interested in political theory my entire life. That sounds like something admirable, or at least neutral. It's not. "Interested" means that I have opinions that are generally stronger than my depth of knowledge warrants. "Interested" means that I like thinking about and casting judgment on how politics should be done without doing the work of politics myself. And "political theory" is different than politics in important ways, not the least of which is that political actions have rarely been a direct danger to me or my family. I have the luxury of arguing about politics as a theory. In short, I'm at high risk of being one of those people who has an opinion about everything and shares it on Twitter. I'm still in the process (to be honest, near the beginning of the process) of making something useful out of that interest. I've had some success when I become enough a part of a community that I can do some of the political work, understand the arguments at a level deeper than theory, and have to deal with the consequences of my own opinions. But those communities have been on-line and relatively low stakes. For the big political problems, the ones that involve governments and taxes and laws, those that decide who gets medical treatment and income support and who doesn't, to ever improve, more people like me need to learn enough about the practical details that we can do the real work of fixing them, rather than only making our native (and generally privileged) communities better for ourselves. I haven't found my path helping with that work yet. But I do have a concrete, challenging, local political question that makes me coldly furious: housing policy. Hence this book. Golden Gates is about housing policy in the notoriously underbuilt and therefore incredibly expensive San Francisco Bay Area, where I live. I wanted to deepen that emotional reaction to the failures of housing policy with facts and analysis. Golden Gates does provide some of that. But this also turns out to be a book about the translation of political theory into practice, about the messiness and conflict that results, and about the difficult process of measuring success. It's also a book about how substantial agreement on the basics of necessary political change can still founder on the shoals of prioritization, tribalism, and people who are interested in political theory. In short, it's a book about the difficulty of changing the world instead of arguing about how to change it. This is not a direct analysis of housing policy, although Dougherty provides the basics as background. Rather, it's the story of the political fight over housing told primarily through two lenses: Sonja Trauss, founder of BARF (the Bay Area Renters' Federation); and a Redwood City apartment complex, the people who fought its rent increases, and the nun who eventually purchased it. Around that framework, Dougherty writes about the Howard Jarvis Taxpayers Association and the history of California's Proposition 13, a fight over a development in Lafayette, the logistics challenge of constructing sufficient housing even when approved, and the political career of Scott Wiener, the hated opponent of every city fighting for the continued ability to arbitrarily veto any new housing. One of the things Golden Gates helped clarify for me is that there are three core interest groups that have to be part of any discussion of Bay Area housing: homeowners who want to limit or eliminate local change, renters who are vulnerable to gentrification and redevelopment, and the people who want to live in that area and can't (which includes people who want to move there, but more sympathetically includes all the people who work there but can't afford to live locally, such as teachers, day care workers, food service workers, and, well, just about anyone who doesn't work in tech). (As with any political classification, statements about collectives may not apply to individuals; there are numerous people who appear to fall into one group but who vote in alignment with another.) Dougherty makes it clear that housing policy is intractable in part because the policies that most clearly help one of those three groups hurt the other two. As advertised by the subtitle, Dougherty's focus is on the fight for more housing. Those who already own homes whose values have been inflated by artificial scarcity, or who want to preserve such stratified living conditions as low-density, large-lot single-family dwellings within short mass-transit commute of one of the densest cities in the United States, don't get a lot of sympathy or focus here except as opponents. I understand this choice; I also don't have much sympathy. But I do wish that Dougherty had spent more time discussing the unsustainable promise that California has implicitly made to homeowners: housing may be impossibly expensive, but if you can manage to reach that pinnacle of financial success, the ongoing value of your home is guaranteed. He does mention this in passing, but I don't think he puts enough emphasis on the impact that a single huge, illiquid investment that is heavily encouraged by government policy has on people's attitude towards anything that jeopardizes that investment. The bulk of this book focuses on the two factions trying to make housing cheaper: Sonja Trauss and others who are pushing for construction of more housing, and tenant groups trying to manage the price of existing housing for those who have to rent. The tragedy of Bay Area housing is that even the faintest connection of housing to the economic principle of supply and demand implies that the long-term goals of those two groups align. Building more housing will decrease the cost of housing, at least if you build enough of it over a long enough period of time. But in the short term, particularly given the amount of Bay Area land pre-emptively excluded from housing by environmental protection and the actions of the existing homeowners, building more housing usually means tearing down cheap lower-density housing and replacing it with expensive higher-density housing. And that destroys people's lives. I'll admit my natural sympathy is with Trauss on pure economic grounds. There simply aren't enough places to live in the Bay Area, and the number of people in the area will not decrease. To the marginal extent that growth even slows, that's another tale of misery involving "super commutes" of over 90 minutes each way. But the most affecting part of this book was the detailed look at what redevelopment looks like for the people who thought they had housing, and how it disrupts and destroys existing communities. It's impossible to read those stories and not be moved. But it's equally impossible to not be moved by the stories of people who live in their cars during the week, going home only on weekends because they have to live too far away from their jobs to commute. This is exactly the kind of politics that I lose when I take a superficial interest in political theory. Even when I feel confident in a guiding principle, the hard part of real-world politics is bringing real people with you in the implementation and mitigating the damage that any choice of implementation will cause. There are a lot of details, and those details matter. Without the right balance between addressing a long-term deficit and providing short-term protection and relief, an attempt to alleviate unsustainable long-term misery creates more short-term misery for those least able to afford it. And while I personally may have less sympathy for the relatively well-off who have clawed their way into their own mortgage, being cavalier with their goals and their financial needs is both poor ethics and poor politics. Mobilizing political opponents who have resources and vote locally isn't a winning strategy. Dougherty is a reporter, not a housing or public policy expert, so Golden Gates poses problems and tells stories rather than describes solutions. This book didn't lead me to a brilliant plan for fixing the Bay Area housing crunch, or hand me a roadmap for how to get effectively involved in local politics. What it did do is tell stories about what political approaches have worked, how they've worked, what change they've created, and the limitations of that change. Solving political problems is work. That work requires understanding people and balancing concerns, which in turn requires a lot of empathy, a lot of communication, and sometimes finding a way to make unlikely allies. I'm not sure how broad the appeal of this book will be outside of those who live in the region. Some aspects of the fight for housing generalize, but the Bay Area (and I suspect every region) has properties specific to it or to the state of California. It has also reached an extreme of housing shortage that is rivaled in the United States only by New York City, which changes the nature of the solutions. But if you want to seriously engage with Bay Area housing policy, knowing the background explained here is nearly mandatory. There are some flaws I wish Dougherty would have talked more about traffic and transit policy, although I realize that could be another book but this is an important story told well. If this somewhat narrow topic is within your interests, highly recommended. Rating: 8 out of 10

A maintenance release of our (still small) ttdo package just arrived on CRAN. As introduced last fall, the ttdo package extends the most excellent (and very minimal / zero depends) unit testing package tinytest by Mark van der Loo with the very clever and well-done diffobj package by Brodie Gaslam to give us test results with visual diffs: tinytest has an extension mechanism we use, and as tinytest was just upgraded to version 1.2.0 changing, among other nice extensions, one interface by allowing for a new error class argument, we had to rebuild as well in order to document the new argument. The release was actually prepared three days ago when tinytest itself was updated, but we waited for the binaries at CRAN to be updated and rebuilt to take advantage of the fully automated submission and test process at CRAN. The NEWS entry follow.

Changes in ttdo version 0.0.5 (2020-05-06)

• Rebuilt under tinytest 1.2.0 to add support for class argument in error-code test predicates

CRANberries provides the usual summary of changes to the previous version. Please use the GitHub repo and its issues for any questions. If you like this or other open-source work I do, you can now sponsor me at GitHub. For the first year, GitHub will match your contributions.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

FTP master This month I accepted 384 packages and rejected 47. The overall number of packages that got accepted was 457. Debian LTS This was my seventieth month that I did some work for the Debian LTS initiative, started by Raphael Hertzog at Freexian. This month my all in all workload has been 28.75h. During that time I did LTS uploads of:

• [DLA 2183-1] libgsf security update for one CVE
• [DLA 2184-1] jsch security update for one CVE
• [DLA 2185-1] eog security update for one CVE
• [DLA 2187-1] radicale security update for one CVE
• [DLA 2186-1] ncmpc security update for one CVE
• [DLA 2188-1] php5 security update for three CVEs
• [DLA 2189-1] rzip security update for one CVE
• [DLA 2195-1] w3m security update for two CVEs
• [DLA 2194-1] yodl security update for one CVE
• [DLA 2197-1] miniupnpc security update for one CVE
• [DLA 2196-1] pound security update for one CVE

As there have been lots of no-dsa-CVEs I continued my work on wireshark. Last but not least I did some days of frontdesk duties. Debian ELTS This month was the twenty second ELTS month. During my small allocated time I only uploaded:

• ELA-227-1 for php5 fixing four CVEs

I also did some days of frontdesk duties. Other stuff Unfortunately this month again strange things happened outside Debian and I only got some stuff done. I improved packaging of

• libctl
• meep

I sponsored uploads of

• ulfius

I uploaded the new package

• puppet-module-cirrax-gitolite

On my Go challenge I uploaded:
golang-github-facebookgo-subset, golang-github-facebookgo-ensure, golang-github-shurcool-gopherjslib, golang-github-grafana-grafana-plugin-model, golang-github-crewjam-httperr, golang-github-hashicorp-terraform-svchost, golang-github-neelance-sourcemap, golang-github-neelance-astrewrite, golang-github-kisielk-gotool, golang-github-gopherjs-gopherjs, golang-github-yvasiyarov-newrelic-platform-go, golang-github-rhnvrm-simples3, golang-github-robfig-go-cache, golang-github-xorcare-pointer, golang-github-goburrow-serial

We had to cancel R/Finance 2020 due to what is happening all around us. But I plan to present the one-hour workshop I often give in the tutorial session preceding the first day but this time online! To keep it simple, we will stick with the same day, and possibly the same time: Friday morning at 8:00am! So that makes Friday, June 5, at 08:00h Central time. This YouTube! link should then provide the stream, I reckon there may also be a recording afterwards. The talk / demo / presentation will be about an hour long, and material should be similar to the previous ones (of the same length) still available at the talks page (which also has longer talks all the way to the two-day workshops). If you like this or other open-source work I do, you can now sponsor me at GitHub. For the first year, GitHub will match your contributions.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Currently there is political debate about when businesses should be reopened after the Covid19 quarantine. Small Businesses One argument for reopening things is for the benefit of small businesses. The first thing to note is that the protests in the US say I need a haircut not I need to cut people s hair . Small businesses won t benefit from reopening sooner. For every business there is a certain minimum number of customers needed to be profitable. There are many comments from small business owners that want it to remain shutdown. When the government has declared a shutdown and paused rent payments and provided social security to employees who aren t working the small business can avoid bankruptcy. If they suddenly have to pay salaries or make redundancy payouts and have to pay rent while they can t make a profit due to customers staying home they will go bankrupt. Many restaurants and cafes make little or no profit at most times of the week (I used to be 1/3 owner of an Internet cafe and know this well). For such a company to be viable you have to be open most of the time so customers can expect you to be open. Generally you don t keep a cafe open at 3PM to make money at 3PM, you keep it open so people can rely on there being a cafe open there, someone who buys a can of soda at 3PM one day might come back for lunch at 1:30PM the next day because they know you are open. A large portion of the opening hours of a most retail companies can be considered as either advertising for trade at the profitable hours or as loss making times that you can t close because you can t send an employee home for an hour. If you have seating for 28 people (as my cafe did) then for about half the opening hours you will probably have 2 or fewer customers in there at any time, for about a quarter the opening hours you probably won t cover the salary of the one person on duty. The weekend is when you make the real money, especially Friday and Saturday nights when you sometimes get all the seats full and people coming in for takeaway coffee and snacks. On Friday and Saturday nights the 60 seat restaurant next door to my cafe used to tell customers that my cafe made better coffee. It wasn t economical for them to have a table full for an hour while they sell a few cups of coffee, they wanted customers to leave after dessert and free the table for someone who wants a meal with wine (alcohol is the real profit for many restaurants). The plans of reopening with social distancing means that a 28 seat cafe can only have 14 chairs or less (some plans have 25% capacity which would mean 7 people maximum). That means decreasing the revenue of the most profitable times by 50% to 75% while also not decreasing the operating costs much. A small cafe has 2-3 staff when it s crowded so there s no possibility of reducing staff by 75% when reducing the revenue by 75%. My Internet cafe would have closed immediately if forced to operate in the proposed social distancing model. It would have been 1/4 of the trade and about 1/8 of the profit at the most profitable times, even if enough customers are prepared to visit and social distancing would kill the atmosphere. Most small businesses are barely profitable anyway, most small businesses don t last 4 years in normal economic circumstances. This reopen movement is about cutting unemployment benefits not about helping small business owners. Destroying small businesses is also good for big corporations, kill the small cafes and restaurants and McDonald s and Starbucks will win. I think this is part of the motivation behind the astroturf campaign for reopening businesses. Forbes has an article about this [1]. Psychological Issues Some people claim that we should reopen businesses to help people who have psychological problems from isolation, to help victims of domestic violence who are trapped at home, to stop older people being unemployed for the rest of their lives, etc. Here is one article with advice for policy makers from domestic violence experts [2]. One thing it mentions is that the primary US federal government program to deal with family violence had a budget of $130M in 2013. The main thing that should be done about family violence is to make it a priority at all times (not just when it can be a reason for avoiding other issues) and allocate some serious budget to it. An agency that deals with problems that affect families and only has a budget of $1 per family per year isn t going to be able to do much. There are ongoing issues of people stuck at home for various reasons. We could work on better public transport to help people who can t drive. We could work on better healthcare to help some of the people who can t leave home due to health problems. We could have more budget for carers to help people who can t leave home without assistance. Wanting to reopen restaurants because some people feel isolated is ignoring the fact that social isolation is a long term ongoing issue for many people, and that many of the people who are affected can t even afford to eat at a restaurant! Employment discrimination against people in the 50+ age range is an ongoing thing, many people in that age range know that if they lose their job and can t immediately find another they will be unemployed for the rest of their lives. Reopening small businesses won t help that, businesses running at low capacity will have to lay people off and it will probably be the older people. Also the unemployment system doesn t deal well with part time work. The Australian system (which I think is similar to most systems in this regard) reduces the unemployment benefits by $0.50 for every dollar that is earned in part time work, that effectively puts people who are doing part time work because they can t get a full-time job in the highest tax bracket! If someone is going to pay for transport to get to work, work a few hours, then get half the money they earned deducted from unemployment benefits it hardly makes it worthwhile to work. While the exact health impacts of Covid19 aren t well known at this stage it seems very clear that older people are disproportionately affected, so forcing older people to go back to work before there is a vaccine isn t going to help them. When it comes to these discussions I think we should be very suspicious of people who raise issues they haven t previously shown interest in. If the discussion of reopening businesses seems to be someone s first interest in the issues of mental health, social security, etc then they probably aren t that concerned about such issues. I believe that we should have a Universal Basic Income [3]. I believe that we need to provide better mental health care and challenge the gender ideas that hurt men and cause men to hurt women [4]. I believe that we have significant ongoing problems with inequality not small short term issues [5]. I don t think that any of these issues require specific changes to our approach to preventing the transmission of disease. I also think that we can address multiple issues at the same time, so it is possible for the government to devote more resources to addressing unemployment, family violence, etc while also dealing with a pandemic.

• [1] http://tinyurl.com/ydc96see
• [2] http://tinyurl.com/ycvy65re
• [3] https://etbe.coker.com.au/2013/12/08/basic-income-australia/
• [4] https://etbe.coker.com.au/2019/01/13/are-men-the-victims/
• [5] https://etbe.coker.com.au/2011/12/09/some-ows-stuff/

The apt.postgresql.org repository has been extended to cover the arm64 architecture. We had occasionally received user request to add "arm" in the past, but it was never really clear which kind of "arm" made sense to target for PostgreSQL. In terms of Debian architectures, there's (at least) armel, armhf, and arm64. Furthermore, Raspberry Pis are very popular (and indeed what most users seemed to were asking about), but the raspbian "armhf" port is incompatible with the Debian "armhf" port. Now that most hardware has moved to 64-bit, it was becoming clear that "arm64" was the way to go. Amit Khandekar made it happen that HUAWEI Cloud Services donated a arm64 build host with enough resources to build the arm64 packages at the same speed as the existing amd64, i386, and ppc64el architectures. A few days later, all the build jobs were done, including passing all test-suites. Very few arm-specific issues were encountered which makes me confident that arm64 is a solid architecture to run PostgreSQL on. We are targeting Debian buster (stable), bullseye (testing), and sid (unstable), and Ubuntu bionic (18.04) and focal (20.04). To use the arm64 archive, just add the normal sources.list entry:

deb https://apt.postgresql.org/pub/repos/apt buster-pgdg main

Ubuntu focal At the same time, I've added the next Ubuntu LTS release to apt.postgresql.org: focal (20.04). It ships amd64, arm64, and ppc64el binaries.

deb https://apt.postgresql.org/pub/repos/apt focal-pgdg main

Old PostgreSQL versions Many PostgreSQL extensions are still supporting older server versions that are EOL. For testing these extension, server packages need to be available. I've built packages for PostgreSQL 9.2+ on all Debian distributions, and all Ubuntu LTS distributions. 9.1 will follow shortly. This means people can move to newer base distributions in their .travis.yml, .gitlab-ci.yml, and other CI files.

For way too long now something I had meant to start was a little series about tips, tricks, tools, and toys. I had mentioned the idea a few times to a friend or two, and generally received a thumbs up or a go for it . But it takes a little to get over the humb and get going. And it turns out that last week s r^4 talk on upgrading to R 4.0.0 hit some latent demand as we are now at 1400 views on YouTube. Wowser. So hence without further ado, let s kick off T^4. Similar in spirit to R^4, but broader in scope and going beyond R. The opening slides explaining what we plan to do are here, and the video link follows below:

With some luck we should have the first actual talk next week. See you then! If you like this or other open-source work I do, you can now sponsor me at GitHub. For the first year, GitHub will match your contributions.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

I realised this week that for some years I have been applying Inbox Zero indiscriminately to all e-mail that I receive, but this does not make sense, and has some downsides. My version of Inbox Zero works very well when applied to mail addressed directly to me, and for mail to certain mailing lists, where each post to the list might as well be addressed directly to me, in addition to the list. However, I also receive by e-mail the following things to which I now believe Inbox Zero should not be applied:

• discussion lists like debian-devel, notmuch@notmuchmail.org, etc.
• mail to aliases like ftpmaster@debian.org (except when that mail is in reply to mail written by me from that address)
• automated notifications received via Debian team mailing lists, where I m not solely responsible for the Debian package in question, such as notifications received via the Debian Perl Group s mailing list
• RSS feed articles supplied by my (years old but still going strong) rss2email cronjob.

I believe that applying Inbox Zero to these sorts of things is not only incorrect but is actually harming my engagement with these mediums. Let us distinguish

1. processing e-mail this means applying the Inbox Zero decision procedure to incoming messages, at set times during the day (I do it once, around 4pm)
2. browsing and sometimes catching up RSS articles and list mail looking through unread items, replying if I think I have something useful to say, leaving things for later, and occasionally marking all as read if I ve not had time for that group of lists lately.

These should be kept apart. The easy case to see is why you shouldn t apply the browsing/catching up approach to e-mail which should rather be processed that s just the original wisdom of Inbox Zero. And clearly you don t want ever to have to resort to just marking all mail addressed directly to you as read. What goes wrong, then, if you misapply the processing mentality to e-mail which should, rather, be browsed/caught up? Well, the core of Inbox Zero is deciding whether to read and reply to something right now, add it to your todo list to handle later, or decide the mail cannot be dealt with quickly but is not important enough to go on that todo list. If you apply this to RSS articles and discussion list mail, then the third option is basically ruled out, because almost nothing on mailing lists or on blogs that I follow is important enough to go on my list of Real Tasks. But then you re faced with either reading the article/post right now, or discarding it. There is no option to leave the item marked as unread and then maybe come back to it, or postpone discarding it until you ve decided to catch up the group. Applying Inbox Zero to discussion lists and RSS feeds creates a false sense of urgency. I ve realised that my implicit response to this has been reluctance to subscribe to new mailing lists and feeds, because I don t have things set up to allow me to read them in a leisurely way. But then I m missing out on discussions and writing that might be relevant and beneficial to me if I could only approach them when I m in a frame of mind other than time to get my inbox down to zero . This also means that subscribing to new mailing lists just in order to post something has an unreasonably high cost: introducing all mail on those lists into my Inbox Zero processing window. So, what s needed is to make the virtual folder views which I use to read new mail correspond cleanly to the distinction between mail to be processed and mail to be browsed. I.e. for each virtual folder it should be clear whether mail there is to be processed or to be browsed. Then I can continue to use my processing views once per day, and access the browsing views at leisure. (Indeed, I ve created a keybinding to cycle through the new browsing views, and another to catch them up.) As I mentioned, some list mail ought to be processed. For example, I want to process rather than browse the debian-policy mailing list, as one of the Policy Editors. With notmuch, it s easy to include this mail in the relevant processing views (I ve long had one processing view for weekdays and another for weekends). Additionally, I ve used a bit of Emacs Lisp to create a dynamic uncategorised unread view which catches mail to be browsed which (i) doesn t show up in one of the other virtual folders for mail to be browsed, and (ii) doesn t show up in one of the processing views. So now subscribing to a mailing list is cheap: mail will end up in the uncategorised view, and I can decide whether to leave it there for a low traffic list; create a new virtual folder for the list (trivial as it s all just more Emacs Lisp, no actual moving of files required); or unsubscribe. I ve been experiencing nostalgia for my time in secondary school when my friends and I had all sorts of interesting stuff flowing into our mailstores, from RSS feeds to each other s blogs where we d post both our own content and links elsewhere, RSS feeds to stranger s blogs, and e-mails sent to each other with links. We didn t have to worry about processing anything back then, as our e-mail accounts were used for intellectual enrichment but not the completion of tasks. I think I can recapture something of that with my new virtual folders for browsing.

Here is my transparent report for my work on the Debian Long Term Support (LTS) and Debian Extended Long Term Support (ELTS), which extend the security support for past Debian releases, as a paid contributor. In April, the monthly sponsored hours were split evenly among contributors depending on their max availability - I was assigned 28.75h for LTS (out of 30 max; all done) and 7.75h for ELTS (out of 20 max; I did 2.75). Escalation procedures were (internally) documented with a focus on discussing issues with team coordinator(s) first. Debian LTS had its first team meeting through IRC and lots of workflow question were discussed. This should help discuss questions that are a bit hard to bring up, and ensure everybody participates. There were lots of topics and it was a bit rushed, but this is something we want to repeat monthly now, possibly with audio/video in a couple months. Remarks from last month's report were discussed, strengthening the Front-Desk role. 10% of the global funding is now reserved for infrastructure work. What kind of work, and who (LTS or external) will do the work, will be discussed further. A fellow DD suggested (in a private conversation) that LTS may be taking time from the Debian Security team, due to additional commits to review. Conversely, this is another opportunity to mention all the global, non-LTS-specific work that LTS provides, which I usually highlight in my reports, and maybe I should be even more ELTS - Wheezy

• CVE-2020-11612/netty: triage: ignored (deceptively hard to backport, OOM mitigation only)
• mysql-connector-java: triage: in-progress (subscription-only update from Oracle, attempt to find more detail, waiting for public version)
• CVE-2020-11868/ntp: global triage: identify and reference missing patch, coordinate with uploader

LTS - Jessie

• netty, mysql-connector-java, ntp: common triage (see above)
• CVE-2019-20637/varnish: global triage: attempt to reproduce, attempt to get PoC/vulnerable versions from upstream, update BTS
• ansible: jessie triage: reset ignore->no-dsa old vulnerabilites after discussing with initial triager
• ansible: global triage: identify more affected version ranges, locate more patches
• ansible: prepare jessie upload (work-in-progress)
• tiff: suites harmonization: offer to work on a tiff/stretch update, follow-up on maintainer's questions, who eventually did the update
• dsa-needed.txt: identify stale entries from inactive LTS contributor, check for status
• team meeting: see minutes

Documentation/Scripts

• LTS/Development: reference relevant sections of the Developer Reference
• LTS/Development: element on whether BTS numbers can be referenced in a LTS changelog

A SYNCNI article passed by on my Twitter feed this morning, talking about balancing work life balance while working from home in these times of COVID-19 inspired lock down. The associated Twitter thread expressed an interest in some words of advice from men to other men (because of course the original article has the woman having to do all the balancing). This post does not contain the words of advice searched for, but it hopefully at least offers some reassurance that if you re finding all of this difficult you re not alone. From talking to others I don t think there s anything particularly special we re doing in this house; a colleague is taking roughly the same approach, and some of the other folk I ve spoken to in the local tech scene seem to be doing likewise. First, the situation. Like many households both my wife and I work full time. We have a small child (not even a year and a half old yet). I work for a software startup, my wife is an HR business partner for a large multinational, dealing with employees all over the UK and Ireland. We re both luckily able to work from home easily - our day to day work machines are laptops, our employers were already setup with the appropriate VPN / video conferencing etc facilities. Neither of us has seen any decrease in workload since lock down; there are always more features and/or bugs to work on when it comes to a software product, and, as I m sure you can imagine, there has been a lot more activity in the HR sphere over the past 6 weeks as companies try to work out what to do. On top of this our childcare arrangements, like everyone else s, are completely gone. Nursery understandably shut down around the same time as the schools (slightly later, but not by much) and contact with grandparents is obviously out (which they re finding hard). So we re left with trying to fit 2 full time jobs in with full time childcare, of a child who up until recently tried to go down stairs by continuing to crawl forward. Make no mistake, this is hard. I know we are exceptionally lucky in our situation, but that doesn t mean we re finding it easy. We ve adopted an approach of splitting the day up. I take the morning slot (previously I would have got up with our son anyway), getting him up and fed while my wife showers. She takes over for a bit while I shower and dress, then I take over again in time for her to take her daily 8am conference call. My morning is mostly taken up with childcare until nap time; I try to check in first thing to make sure there s nothing urgent, and get a handle on what I might have to work on later in the day. My local team mates know they re more likely to get me late morning and it s better to arrange meetings in the afternoon. Equally I work with a lot of folk on the US West coast, so shifting my hours to be a bit later is not a problem there. After nap time (which, if we re lucky, takes us to lunch) my wife takes over. As she deals with UK/Ireland folk she often ends up having to take calls even while looking after our son; generally important meetings can be moved to the morning and meetings with folk who understand there might be a lot of pot banging going on in the background can happen in the afternoon. Having started late I generally work late - past the point where I d normally get home; if I m lucky I pop my head in for bath time, but sometimes it s only for a couple of minutes. We alternate cooking; usually based on work load + meetings. For example tonight I m cooking while my wife catches up on some work after having put our son to bed. Last night I had a few meetings so my wife cooked. So what s worked for us? Splitting the day means we can plan with our co-workers. We always make sure we eat together in the evening, and that generally is the cut-off point for either of us doing any work. I m less likely to be online in the evening because my study has become the place I work. That means I m not really doing any of my personal projects - this definitely isn t a case of being at home during lock down and having lots of time to achieve new things. It s much more of case of trying to find a sustainable way to get through the current situation, however long it might last.

I have made the 1.0 release of subdirmk. subdirmk is a tool to help with writing build systems in make, without use of recursive make. Why Peter Miller's 1997 essay Recursive Make Considered Harmful persuasively argues that it is better to arrange to have a single make invocation with the project's complete dependency tree, rather than the conventional $(MAKE) -C subdirectory approach. This has become much more relevant with modern projects which tend to be large and have deep directory trees. Invoking make separately for each of these subdirectories can be very slow. Nowadays everyone needs to run a parallel build, but with the recursive make approach great discipline is needed to avoid introducing races which cause the build to sometimes fail. There are various new systems which aim to replace make. My general impression of these is that they mostly threw away the good parts of make (often, they discard the flexibility, and the use of the shell command as the basic unit of execution, making them hard to extend), or make other unfortunate assumptions. And there are a lot of programming-language-specific systems - a very unsatisfactory development. Having said all that, I admit I haven't properly evaluated every make competitor. Other reasons for staying with make including that it is widely available, relatively widely understood, and has a model relatively free of high-level abstract concepts. (I like my languages with high-level concepts, but not my build systems.) But, with make, I found that actually writing a project's build system in non-recursive make was not very ergonomic. So with some help and prompting from Mark Wooding, I have made a tool to help. What subdirmk is a makefile preprocessor and aggregator, typically run from autoconf. subdirmk provides convenience syntaxes for references to per-directory variables and pathnames. It also helps by providing a little syntactic sugar for GNU make's macro facilities, which are awkward to use in raw make. subdirmk's features are triggered by the sigil &. The syntax is carefully designed to avoid getting in the way of makefile programming (and programming of shell commands in make rules). subdirmk is fully documented in the README. There is a demo in the example directory (which also serves as part of the test suite). What's new The version number. I have not felt the need to make any changes since releasing 0.4 in mid-February. The last non-docs change was a (backwards-compatible) extension, in late January, to pass through unaltered GNU make's new grouped multiple targets syntax. Advantages and disadvantages of subdirmk Compared to recursive make, subdirmk is easier and simpler, although you do have to decorate a lot of your variables and filenames with & to indicate that they are directory-local. It is much easier to avoid writing parallel make bugs. You naturally get properly working per-subdirectory targets. subdirmk-based nonrecursive make is much, much faster than recursive make. Compared to many other recent build system tools, subdirmk retains all the flexibility and extensibility of make, and operates at a fairly low level of abstraction. subdirmk-based makefiles can easily invoke other build systems. make knows it's not the only thing in the universe. You can adopt subdirmk incrementally or partially, gradually bringing your recursive submakefiles into the unified build. The build system code in subdirmk's Dir.sd.mk files will be readily navigable by most readers; much will be familiar. Because subdirmk is a small collection of (fairly simple) scripting and makefile code, there is no need to build it; you can simply ship it with your project using git-subtree. For an autoconf-based project, there need be no change to how your users and downstreams invoke your build. On the other hand the price you (continue to) pay is make's punctation soup, which subdirmk adds a new sigil to. subdirmk-based makefiles are terse and help you use make's facilities to abstract away repetition, but that can make them dense. The new & sigil will faze some readers. Currently, the provided mechanism for incorporating subdirmk into your project assumes you are using autoconf but not automake. It would be possible to use subdirmk with autoconf-less projects, or with automake-based ones, but I haven't done the glue work to make that easy. subdirmk does require GNU make and it assumes you have perl installed. But GNU make is very portable, and perl is very widely available. (The perl used is very conservative.) The make competitors are, themselves, even less standard build tools. I don't think a build-dependency on GNU make, or perl, is a significant barrier nowadays, for most projects. Note about comment moderation I have deliberately been vague about other build systems and avoided specific criticisms or references. I don't want the comments to become a build system advocacy debate. Comments may be screened and moderated accordingly. Pointers to other obscure build system tools are very welcome. If you want to write a survey of build tools, or a critique of subdirmk, please do so on your own blog; I would be happy to consider linking to it.

comments

The well-known kill system call has been around for decades and is used to send a signal to another process. The most common use is to terminate or kill another process by sending the KILL or TERM signal but it can be used for a form of IPC, usually around giving the other process a kick to do something. One thing that isn t as well known is besides sending a signal to a process, you can send some data to it. This can either be an integer or a pointer and uses similar semantics to the known kill and signal handler. I came across this when there was a merge request for procps. The main changes are using sigqueue instead of kill in the sender and using a signal action not a signal handler in the receiver. To illustrate this feature, I have a small set of programs called sender and receiver that will pass an integer between them. The Sender The sender program is extremely simple, use a random(ish) from time masked to two bytes, put it in the required union and send the lot to sendqueue.

# include <signal.h>
# include <stdlib.h>
# include <stdio.h>
# include <time.h>
int main(int argc, char **argv)
 
    union sigval sigval;
    pid_t pid;
    if (argc < 2   (pid = atoi(argv[1])) < 0)
	return EXIT_FAILURE;
    sigval.sival_int = time(NULL) &amp; 0xff;
    printf("sender: sending %d to PID %d\n",
        sigval.sival_int, pid);
    sigqueue(pid, SIGUSR1, sigval);
    return EXIT_SUCCESS;
 

The key lines are 13 and 16 where the random (ish) integer is stored in the sigval union and then sent to the other process with the sigqueue. The receiver The receiver just sets up the signal handler, sends its PID (so I know what to tell the sender) and sits in a sleeping loop.

# include <stdio.h>
# include <stdlib.h>
# include <sys/types.h>
# include <unistd.h>
# include <signal.h>
void signal_handler(int signum, siginfo_t *siginfo, void *ucontext)
 
    if (signum != SIGUSR1) return;
    if (siginfo->si_code != SI_QUEUE) return;
    printf("receiver: Got value %d\n",
	    siginfo->si_int);
 
int main(int argc, char **argv)
 
    pid_t pid = getpid();
    struct sigaction signal_action;
    printf("receiver: PID is %d\n", pid);
    signal_action.sa_sigaction = signal_handler;
    sigemptyset (&amp;signal_action.sa_mask);
    signal_action.sa_flags = SA_SIGINFO;
    sigaction(SIGUSR1, &amp;signal_action, NULL);
    while(1) sleep(100);
    return EXIT_SUCCESS;
 

Lines 16-26 setup the signal handler. The main difference here is SA_SIGINFO used for the signal flags and sigaction references a sa_sigaction function rather than sa_handler. We need to use a different function because the sigaction only is passed the signal number but we need more information, including the integer that the sender process stored in sigval. Lines 7-14 are the signal handler function itself. It first checks that the receiver process got the correct signal (SIGUSR1 in this case) and that we got this signal from sigqueue because the type is SI_QUEUE. Checking the type of signal is important because different signals give you different data. For example, if you signalled this process with kill then si_int is undefined. The result As a proof of concept, the results are not terribly exciting. We see the sender say what it will be sending and the receiver saying it got it. It was useful to get some results, especially when things went wrong.

$ ./receiver &
[1] 370216
receiver: PID is 370216
$ ./sender 370216
sender: sending 133 to (gdPID 370216
receiver: Got value 133

Gotchas While testing the two process there was two gotchas I encountered. GDB and the siginfo structure The sigaction manual page shows a simple siginfo_t structure, however when looking at what is passed to the signal handler, it s much more complicated.

(gdb) p *siginfo
$2 =  si_signo = 10, si_errno = 0, si_code = -1, __pad0 = 0, _sifields =  _pad =  371539, 1000, 11, 32766, 0 <repeats 24 times> , _kill =  si_pid = 371539, si_uid = 1000 , _timer =  si_tid = 371539, 
      si_overrun = 1000, si_sigval =  sival_int = 11, sival_ptr = 0x7ffe0000000b , _rt =  si_pid = 371539, si_uid = 1000, si_sigval =  sival_int = 11, sival_ptr = 0x7ffe0000000b , _sigchld =  
      si_pid = 371539, si_uid = 1000, si_status = 11, si_utime = 0, si_stime = 0 , _sigfault =  si_addr = 0x3e80005ab53, si_addr_lsb = 11, _bounds =  _addr_bnd =  _lower = 0x0, _upper = 0x0 , _pkey = 0 , 
    _sigpoll =  si_band = 4294967667539, si_fd = 11 , _sigsys =  _call_addr = 0x3e80005ab53, _syscall = 11, _arch = 32766 
(gdb) p siginfo->_sifields._rt.si_sigval.sival_int
$3 = 11

So the integer is stored in a union in a structure in a structure. Much harder to find than just simply sival_int. The pointer is just a pointer So perhaps sending an integer is not enough. The sigval is a union with an integer and a pointer. Could a string be sent instead? I changed line 13 of the sender so it used a string instead of an integer.

    sigval.sival_ptr = "Hello, World!"

The receiver needed a minor adjustment to print out the string. I tried this and the receiver segmentation faulted. What was going on? The issue is the set of system calls does a simple passing of the values. So if the sender sends a pointer to a string located at 0x1234567 then the receiver will have a pointer to the same location. When the receiver tries to dereference the sival_ptr, it is pointing to memory that is not owned by it but by another process (the sender) so it segmentation faults. The solution would be to use shared memory between the processes. The signal queue would then use the pointer to the shared memory and, in theory, all would be well.

Armadillo is a powerful and expressive C++ template library for linear algebra aiming towards a good balance between speed and ease of use with a syntax deliberately close to a Matlab. RcppArmadillo integrates this library with the R environment and language and is widely used by (currently) 705 other packages on CRAN. A new upstream release 9.870.2 of Armadillo was released a few days ago. We had tested two release candidates, and this caught one bug. The release was held up at CRAN for a few days as one package had an overly sensitive test depending on random input data; eventually we all convinced ourselves that there was no (Rcpp)Armadillo issue here. So morale: friends don t let friends have tests depend on random behavior. Changes in the new release are noted below.

Changes in RcppArmadillo version 0.9.870.2.0 (2020-04-24)

• Upgraded to Armadillo release 9.870.2 (Roasted Mocha Retox)
  • faster handling of matrix multiplication expressions by diagvec() and diagmat()
  • added trimatu_ind() and trimatl_ind()
  • more consistent detection of sparse vector expressions

Courtesy of CRANberries, there is a diffstat report relative to previous release. More detailed information is on the RcppArmadillo page. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page. If you like this or other open-source work I do, you can now sponsor me at GitHub. For the first year, GitHub will match your contributions.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.