Search Results: "ed"

17 May 2020

Dirk Eddelbuettel: RcppArmadillo 0.9.880.1.0

armadillo image Armadillo is a powerful and expressive C++ template library for linear algebra aiming towards a good balance between speed and ease of use with a syntax deliberately close to a Matlab. RcppArmadillo integrates this library with the R environment and language and is widely used by (currently) 719 other packages on CRAN. Conrad released a new upstream version 9.880.1 of Armadillo on Friday which I packaged and tested as usual (result log here in the usual repo). The R package also sports a new OpenMP detection facility once again motivated by macOS which changed its setup yet again. Changes in the new release are noted below.

Changes in RcppArmadillo version 0.9.880.1.0 (2020-05-15)
  • Upgraded to Armadillo release 9.880.1 (Roasted Mocha Detox)
    • expanded qr() to optionally use pivoted decomposition
    • updated physical constants to NIST 2018 CODATA values
    • added ARMA_DONT_USE_CXX11_MUTEX confguration option to disable use of std::mutex
  • OpenMP capability is tested explicitly (Kevin Ushey and Dirk in #294, #295, and #296 all fixing #290).

Courtesy of CRANberries, there is a diffstat report relative to previous release. More detailed information is on the RcppArmadillo page. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page. If you like this or other open-source work I do, you can now sponsor me at GitHub. For the first year, GitHub will match your contributions.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Steve Kemp: Some brief sysbox highlights

I started work on sysbox again recently, adding a couple of simple utilities. (The whole project is a small collection of utilities, distributed as a single binary to ease installation.) Imagine you want to run a command for every line of STDIN, here's a good example:
 $ cat input   sysbox exec-stdin "youtube-dl  "
Here you see for every (non-empty) line of input read from STDIN the command "youtube-dl" has been executed. " " gets expanded to the complete line read. You can also access individual fields, kinda like awk. (Yes youtube-dl can read a list of URLs from a file, this is an example!) Another example, run groups for every local user:
$ cat /etc/passwd   sysbox exec-stdin --split=: groups  1 
Here you see we have split the input-lines read from STDIN by the : character, instead of by whitespace, and we've accessed the first field via " 1 ". This is certainly easier for scripting than using a bash loop. On the topic of bash; command-completion for each subcommand, and their arguments, is now present:
$ source <(sysbox bash-completion)
And I've added a text-based UI for selecting files. You can also execute a command, against the selected file:
$ sysbox choose-file -exec "xine  " /srv/tv
This is what that looks like: /2020/05/17-choose-file.png You'll see: As well as choosing files, you can also select from lines read via STDIN, and you can filter the command in the same way as before. (i.e. " " is the selected item.) Other commands received updates, so the calculator now allows storing results in variables:
$ sysbox calc
calc> let a = 3
3
calc> a / 9 * 3
1
calc> 1 + 2 * a
7
calc> 1.2 + 3.4
4.600000

Erich Schubert: Contact Tracing Apps are Useless

Some people believe that automatic contact tracing apps will help contain the Coronavirus epidemic. They won t. Sorry to bring the bad news, but IT and mobile phones and artificial intelligence will not solve every problem. In my opinion, those that promise to solve these things with artificial intelligence / mobile phones / apps / your-favorite-buzzword are at least overly optimistic and blinder Aktionismus (*), if not naive, detachted from reality, or fraudsters that just want to get some funding. (*) there does not seem to be an English word for this doing something just for the sake of doing something, without thinking about whether it makes sense to do so Here are the reasons why it will not work:
  1. Signal quality. Forget detecting proximity with Bluetooth Low Energy. Yes, there are attempts to use BLE beacons for indoor positioning. But these use that you can learn fingerprints of which beacons are visible at which points, combined with additional information such as movement sensors and history (you do not teleport around in a building). BLE signals and antennas apparently tend to be very prone to orientation differences, signal reflections, and of course you will not have the idealized controlled environment used in such prototypes. The contacts have a single device, and they move this is not comparable to indoor positioning. I strongly doubt you can tell whether you are close to someone, or not.
  2. Close vs. protection. The app cannot detect protection in place. Being close to someone behind a plexiglass window or even a solid wall is very different from being close otherwise. You will get a lot of false contacts this way. That neighbor that you have never seen living in the appartment above will likely be considered a close contact of yours, as you sleep next to each other every day
  3. Low adoption rates. Apparently even in technology affine Singapore, fewer than 20% of people installed the app. That does not even mean they use it regularly. In Austria, the number is apparently below 5%, and people complain that it does not detect contact But in order for this approach to work, you will need Chinese-style mass surveillance that literally puts you in prison if you do not install the app.
  4. False alerts. Because of these issues, you will get false alerts, until you just do not care anymore.
  5. False sense of security. Honestly: the app does not pretect you at all. All it tries to do is to make the tracing of contacts easier. It will not tell you reliably if you have been infected (as mentioned above, too many false positives, too few users) nor that you are relatively safe (too few contacts included, too slow testing and reporting). It will all be on the quality of about 10 days ago you may or may not have contact with someone that tested positive, please contact someone to expose more data to tell you that it is actually another false alert .
  6. Trust. In Germany, the app will be operated by T-Systems and SAP. Not exactly two companies that have a lot of fans SAP seems to be one of the most hated software around. Neither company is known for caring about privacy much, but they are prototypical for business first . Its trust the cat to keep the cream. Yes, I know they want to make it open-source. But likely only the client, and you will still have to trust that the binary in the app stores is actually built from this source code, and not from a modified copy. As long as the name T-Systems and SAP are associated to the app, people will not trust it. Plus, we all know that the app will be bad, given the reputation of these companies at making horrible software systems
  7. Too late. SAP and T-Systems want to have the app ready in mid June. Seriously, this must be a joke? It will be very buggy in the beginning (because it is SAP!) and it will not be working reliably before end of July. There will not be a substantial user before fall. But given the low infection rates in Germany, nobody will bother to install it anymore, because the perceived benefit is 0 one the infection rates are low.
  8. Infighting. You may remember that there was the discussion before that there should be a pan-european effort. Except that in the end, everybody fought everybody else, countries went into different directions and they all broke up. France wanted a centralized systems, while in Germany people pointed out that the users will not accept this and only a distributed system will have a chance. That failed effort was known as Pan-European Privacy-Preserving Proximity Tracing (PEPP-PT) vs. Decentralized Privacy-Preserving Proximity Tracing (DP-3T) , and it turned out to have become a big clusterfuck . And that is just the tip of the iceberg.
Iceleand, probably the country that handled the Corona crisis best (they issued a travel advisory against Austria, when they were still happily spreading the virus at apres-ski; they massively tested, and got the infections down to almost zero within 6 weeks), has been experimenting with such an app. Iceland as a fairly close community managed to have almost 40% of people install their app. So did it help? No: The technology is more or less I wouldn t say useless [ ] it wasn t a game changer for us. The contact tracing app is just a huge waste of effort and public money. And pretty much the same applies to any other attempts to solve this with IT. There is a lot of buzz about solving the Corona crisis with artificial intelligence: bullshit! That is just naive. Do not speculate about magic power of AI. Get the data, understand the data, and you will see it does not help. Because its real data. Its dirty. Its late. Its contradicting. Its incomplete. It is all what AI currently can not handle well. This is not image recognition. You have no labels. Many of the attempts in this direction already fail at the trivial 7-day seasonality you observe in the data For example, the widely known John Hopkins Has the curve flattened trend has a stupid, useless indicator based on 5 day averages. And hence you get the weekly up and downs due to weekends. They show pretty up and down indicators. But these are affected mostly by the day of the week. And nobody cares. Notice that they currently even have big negative infections in their plots? There is no data on when someone was infected. Because such data simply does not exist. What you have is data when someone tested positive (mostly), when someone reported symptons (sometimes, but some never have symptoms!), and when someone dies (but then you do not know if it was because of Corona, because of other issues that became just worse because of Corona, or hit by a car without any relation to Corona). The data that we work with is incredibly delayed, yet we pretend it is live . Stop reading tea leaves. Stop pretending AI can save the world from Corona.

Matthew Palmer: Private Key Redaction: UR DOIN IT RONG

Because posting private keys on the Internet is a bad idea, some people like to redact their private keys, so that it looks kinda-sorta like a private key, but it isn t actually giving away anything secret. Unfortunately, due to the way that private keys are represented, it is easy to redact a key in such a way that it doesn t actually redact anything at all. RSA private keys are particularly bad at this, but the problem can (potentially) apply to other keys as well. I ll show you a bit of Inside Baseball with key formats, and then demonstrate the practical implications. Finally, we ll go through a practical worked example from an actual not-really-redacted key I recently stumbled across in my travels.

The Private Lives of Private Keys Here is what a typical private key looks like, when you come across it:
-----BEGIN RSA PRIVATE KEY-----
MGICAQACEQCxjdTmecltJEz2PLMpS4BXAgMBAAECEDKtuwD17gpagnASq1zQTYEC
CQDVTYVsjjF7IQIJANUYZsIjRsR3AgkAkahDUXL0RSECCB78r2SnsJC9AghaOK3F
sKoELg==
-----END RSA PRIVATE KEY-----
Obviously, there s some hidden meaning in there computers don t encrypt things by shouting BEGIN RSA PRIVATE KEY! , after all. What is between the BEGIN/END lines above is, in fact, a base64-encoded DER format ASN.1 structure representing a PKCS#1 private key. In simple terms, it s a list of numbers very important numbers. The list of numbers is, in order:
  • A version number (0);
  • The public modulus , commonly referred to as n ;
  • The public exponent , or e (which is almost always 65,537, for various unimportant reasons);
  • The private exponent , or d ;
  • The two private primes , or p and q ;
  • Two exponents, which are known as dmp1 and dmq1 ; and
  • A coefficient, known as iqmp .

Why Is This a Problem? The thing is, only three of those numbers are actually required in a private key. The rest, whilst useful to allow the RSA encryption and decryption to be more efficient, aren t necessary. The three absolutely required values are e, p, and q. Of the other numbers, most of them are at least about the same size as each of p and q. So of the total data in an RSA key, less than a quarter of the data is required. Let me show you with the above toy key, by breaking it down piece by piece1:
  • MGI DER for this is a sequence
  • CAQ version (0)
  • CxjdTmecltJEz2PLMpS4BX n
  • AgMBAA e
  • ECEDKtuwD17gpagnASq1zQTY d
  • ECCQDVTYVsjjF7IQ p
  • IJANUYZsIjRsR3 q
  • AgkAkahDUXL0RS dmp1
  • ECCB78r2SnsJC9 dmq1
  • AghaOK3FsKoELg== iqmp
Remember that in order to reconstruct all of these values, all I need are e, p, and q and e is pretty much always 65,537. So I could redact almost all of this key, and still give all the important, private bits of this key. Let me show you:
-----BEGIN RSA PRIVATE KEY-----
..............................................................EC
CQDVTYVsjjF7IQIJANUYZsIjRsR3....................................
........
-----END RSA PRIVATE KEY-----
Now, I doubt that anyone is going to redact a key precisely like this but then again, this isn t a typical RSA key. They usually look a lot more like this:
-----BEGIN RSA PRIVATE KEY-----
MIIEogIBAAKCAQEAu6Inch7+mWtKn+leB9uCG3MaJIxRyvC/5KTz2fR+h+GOhqj4
SZJobiVB4FrE5FgC7AnlH6qeRi9MI0s6dt5UWZ5oNIeWSaOOeNO+EJDUkSVf67wj
SNGXlSjGAkPZ0nRJiDjhuPvQmdW53hOaBLk5udxPEQbenpXAzbLJ7wH5ouLQ3nQw
HwpwDNQhF6zRO8WoscpDVThOAM+s4PS7EiK8ZR4hu2toon8Ynadlm95V45wR0VlW
zywgbkZCKa1IMrDCscB6CglQ10M3Xzya3iTzDtQxYMVqhDrA7uBYRxA0y1sER+Rb
yhEh03xz3AWemJVLCQuU06r+FABXJuY/QuAVvQIDAQABAoIBAFqwWVhzWqNUlFEO
PoCVvCEAVRZtK+tmyZj9kU87ORz8DCNR8A+/T/JM17ZUqO2lDGSBs9jGYpGRsr8s
USm69BIM2ljpX95fyzDjRu5C0jsFUYNi/7rmctmJR4s4uENcKV5J/++k5oI0Jw4L
c1ntHNWUgjK8m0UTJIlHbQq0bbAoFEcfdZxd3W+SzRG3jND3gifqKxBG04YDwloy
tu+bPV2jEih6p8tykew5OJwtJ3XsSZnqJMwcvDciVbwYNiJ6pUvGq6Z9kumOavm9
XU26m4cWipuK0URWbHWQA7SjbktqEpxsFrn5bYhJ9qXgLUh/I1+WhB2GEf3hQF5A
pDTN4oECgYEA7Kp6lE7ugFBDC09sKAhoQWrVSiFpZG4Z1gsL9z5YmZU/vZf0Su0n
9J2/k5B1GghvSwkTqpDZLXgNz8eIX0WCsS1xpzOuORSNvS1DWuzyATIG2cExuRiB
jYWIJUeCpa5p2PdlZmBrnD/hJ4oNk4oAVpf+HisfDSN7HBpN+TJfcAUCgYEAyvY7
Y4hQfHIdcfF3A9eeCGazIYbwVyfoGu70S/BZb2NoNEPymqsz7NOfwZQkL4O7R3Wl
Rm0vrWT8T5ykEUgT+2ruZVXYSQCKUOl18acbAy0eZ81wGBljZc9VWBrP1rHviVWd
OVDRZNjz6nd6ZMrJvxRa24TvxZbJMmO1cgSW1FkCgYAoWBd1WM9HiGclcnCZknVT
UYbykCeLO0mkN1Xe2/32kH7BLzox26PIC2wxF5seyPlP7Ugw92hOW/zewsD4nLze
v0R0oFa+3EYdTa4BvgqzMXgBfvGfABJ1saG32SzoWYcpuWLLxPwTMsCLIPmXgRr1
qAtl0SwF7Vp7O/C23mNukQKBgB89DOEB7xloWv3Zo27U9f7nB7UmVsGjY8cZdkJl
6O4LB9PbjXCe3ywZWmJqEbO6e83A3sJbNdZjT65VNq9uP50X1T+FmfeKfL99X2jl
RnQTsrVZWmJrLfBSnBkmb0zlMDAcHEnhFYmHFuvEnfL7f1fIoz9cU6c+0RLPY/L7
n9dpAoGAXih17mcmtnV+Ce+lBWzGWw9P4kVDSIxzGxd8gprrGKLa3Q9VuOrLdt58
++UzNUaBN6VYAe4jgxGfZfh+IaSlMouwOjDgE/qzgY8QsjBubzmABR/KWCYiRqkj
qpWCgo1FC1Gn94gh/+dW2Q8+NjYtXWNqQcjRP4AKTBnPktEvdMA=
-----END RSA PRIVATE KEY-----
People typically redact keys by deleting whole lines, and usually replacing them with [...] and the like. But only about 345 of those 1588 characters (excluding the header and footer) are required to construct the entire key. You can redact about 4/5ths of that giant blob of stuff, and your private parts (or at least, those of your key) are still left uncomfortably exposed.

But Wait! There s More! Remember how I said that everything in the key other than e, p, and q could be derived from those three numbers? Let s talk about one of those numbers: n. This is known as the public modulus (because, along with e, it is also present in the public key). It is very easy to calculate: n = p * q. It is also very early in the key (the second number, in fact). Since n = p * q, it follows that q = n / p. Thus, as long as the key is intact up to p, you can derive q by simple division.

Real World Redaction At this point, I d like to introduce an acquaintance of mine: Mr. Johan Finn. He is the proud owner of the GitHub repo johanfinn/scripts. For a while, his repo contained a script that contained a poorly-redacted private key. He since deleted it, by making a new commit, but of course because git never really deletes anything, it s still available. Of course, Mr. Finn may delete the repo, or force-push a new history without that commit, so here is the redacted private key, with a bit of the surrounding shell script, for our illustrative pleasure:
#Add private key to .ssh folder
cd /home/johan/.ssh/
echo  "-----BEGIN RSA PRIVATE KEY-----
MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM
NNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN
KKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKK
 
MIIJKgIBAAKCAgEAxEVih1JGb8gu/Fm4AZh+ZwJw/pjzzliWrg4mICFt1g7SmIE2
TCQMKABdwd11wOFKCPc/UzRH/fHuQcvWrpbOSdqev/zKff9iedKw/YygkMeIRaXB
fYELqvUAOJ8PPfDm70st9GJRhjGgo5+L3cJB2gfgeiDNHzaFvapRSU0oMGQX+kI9
ezsjDAn+0Pp+r3h/u1QpLSH4moRFGF4omNydI+3iTGB98/EzuNhRBHRNq4oBV5SG
Pq/A1bem2ninnoEaQ+OPESxYzDz3Jy9jV0W/6LvtJ844m+XX69H5fqq5dy55z6DW
sGKn78ULPVZPsYH5Y7C+CM6GAn4nYCpau0t52sqsY5epXdeYx4Dc+Wm0CjXrUDEe
Egl4loPKDxJkQqQ/MQiz6Le/UK9vEmnWn1TRXK3ekzNV4NgDfJANBQobOpwt8WVB
rbsC0ON7n680RQnl7PltK9P1AQW5vHsahkoixk/BhcwhkrkZGyDIl9g8Q/Euyoq3
eivKPLz7/rhDE7C1BzFy7v8AjC3w7i9QeHcWOZFAXo5hiDasIAkljDOsdfD4tP5/
wSO6E6pjL3kJ+RH2FCHd7ciQb+IcuXbku64ln8gab4p8jLa/mcMI+V3eWYnZ82Yu
axsa85hAe4wb60cp/rCJo7ihhDTTvGooqtTisOv2nSvCYpcW9qbL6cGjAXECAwEA
AQKCAgEAjz6wnWDP5Y9ts2FrqUZ5ooamnzpUXlpLhrbu3m5ncl4ZF5LfH+QDN0Kl
KvONmHsUhJynC/vROybSJBU4Fu4bms1DJY3C39h/L7g00qhLG7901pgWMpn3QQtU
4P49qpBii20MGhuTsmQQALtV4kB/vTgYfinoawpo67cdYmk8lqzGzzB/HKxZdNTq
s+zOfxRr7PWMo9LyVRuKLjGyYXZJ/coFaobWBi8Y96Rw5NZZRYQQXLIalC/Dhndm
AHckpstEtx2i8f6yxEUOgPvV/gD7Akn92RpqOGW0g/kYpXjGqZQy9PVHGy61sInY
HSkcOspIkJiS6WyJY9JcvJPM6ns4b84GE9qoUlWVF3RWJk1dqYCw5hz4U8LFyxsF
R6WhYiImvjxBLpab55rSqbGkzjI2z+ucDZyl1gqIv9U6qceVsgRyuqdfVN4deU22
LzO5IEDhnGdFqg9KQY7u8zm686Ejs64T1sh0y4GOmGsSg+P6nsqkdlXH8C+Cf03F
lqPFg8WQC7ojl/S8dPmkT5tcJh3BPwIWuvbtVjFOGQc8x0lb+NwK8h2Nsn6LNazS
0H90adh/IyYX4sBMokrpxAi+gMAWiyJHIHLeH2itNKtAQd3qQowbrWNswJSgJzsT
JuJ7uqRKAFkE6nCeAkuj/6KHHMPsfCAffVdyGaWqhoxmPOrnVgECggEBAOrCCwiC
XxwUgjOfOKx68siFJLfHf4vPo42LZOkAQq5aUmcWHbJVXmoxLYSczyAROopY0wd6
Dx8rqnpO7OtZsdJMeBSHbMVKoBZ77hiCQlrljcj12moFaEAButLCdZFsZW4zF/sx
kWIAaPH9vc4MvHHyvyNoB3yQRdevu57X7xGf9UxWuPil/jvdbt9toaraUT6rUBWU
GYPNKaLFsQzKsFWAzp5RGpASkhuiBJ0Qx3cfLyirjrKqTipe3o3gh/5RSHQ6VAhz
gdUG7WszNWk8FDCL6RTWzPOrbUyJo/wz1kblsL3vhV7ldEKFHeEjsDGroW2VUFlS
asAHNvM4/uYcOSECggEBANYH0427qZtLVuL97htXW9kCAT75xbMwgRskAH4nJDlZ
IggDErmzBhtrHgR+9X09iL47jr7dUcrVNPHzK/WXALFSKzXhkG/yAgmt3r14WgJ6
5y7010LlPFrzaNEyO/S4ISuBLt4cinjJsrFpoo0WI8jXeM5ddG6ncxdurKXMymY7
:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::.::
:::::::::::::::::::::::::::.::::::::::::::::::::::::::::::::::::
LLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLlL
 
 
 
YYYYYYYYYYYYYYYYYYYYYyYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY
gff0GJCOMZ65pMSy3A3cSAtjlKnb4fWzuHD5CFbusN4WhCT/tNxGNSpzvxd8GIDs
nY7exs9L230oCCpedVgcbayHCbkChEfoPzL1e1jXjgCwCTgt8GjeEFqc1gXNEaUn
O8AJ4VlR8fRszHm6yR0ZUBdY7UJddxQiYOzt0S1RLlECggEAbdcs4mZdqf3OjejJ
06oTPs9NRtAJVZlppSi7pmmAyaNpOuKWMoLPElDAQ3Q7VX26LlExLCZoPOVpdqDH
KbdmBEfTR4e11Pn9vYdu9/i6o10U4hpmf4TYKlqk10g1Sj21l8JATj/7Diey8scO
sAI1iftSg3aBSj8W7rxCxSezrENzuqw5D95a/he1cMUTB6XuravqZK5O4eR0vrxR
AvMzXk5OXrUEALUvt84u6m6XZZ0pq5XZxq74s8p/x1JvTwcpJ3jDKNEixlHfdHEZ
ZIu/xpcwD5gRfVGQamdcWvzGHZYLBFO1y5kAtL8kI9tW7WaouWVLmv99AyxdAaCB
Y5mBAQKCAQEAzU7AnorPzYndlOzkxRFtp6MGsvRBsvvqPLCyUFEXrHNV872O7tdO
GmsMZl+q+TJXw7O54FjJJvqSSS1sk68AGRirHop7VQce8U36BmI2ZX6j2SVAgIkI
9m3btCCt5rfiCatn2+Qg6HECmrCsHw6H0RbwaXS4RZUXD/k4X+sslBitOb7K+Y+N
Bacq6QxxjlIqQdKKPs4P2PNHEAey+kEJJGEQ7bTkNxCZ21kgi1Sc5L8U/IGy0BMC
PvJxssLdaWILyp3Ws8Q4RAoC5c0ZP0W2j+5NSbi3jsDFi0Y6/2GRdY1HAZX4twem
Q0NCedq1JNatP1gsb6bcnVHFDEGsj/35oQKCAQEAgmWMuSrojR/fjJzvke6Wvbox
FRnPk+6YRzuYhAP/YPxSRYyB5at++5Q1qr7QWn7NFozFIVFFT8CBU36ktWQ39MGm
cJ5SGyN9nAbbuWA6e+/u059R7QL+6f64xHRAGyLT3gOb1G0N6h7VqFT25q5Tq0rc
Lf/CvLKoudjv+sQ5GKBPT18+zxmwJ8YUWAsXUyrqoFWY/Tvo5yLxaC0W2gh3+Ppi
EDqe4RRJ3VKuKfZxHn5VLxgtBFN96Gy0+Htm5tiMKOZMYAkHiL+vrVZAX0hIEuRZ
JJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJ
MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM
-----END RSA PRIVATE KEY-----" >> id_rsa
Now, if you try to reconstruct this key by removing the obvious garbage lines (the ones that are all repeated characters, some of which aren t even valid base64 characters), it still isn t a key at least, openssl pkey doesn t want anything to do with it. The key is very much still in there, though, as we shall soon see. Using a gem I wrote and a quick bit of Ruby, we can extract a complete private key. The irb session looks something like this:
>> require "derparse"
>> b64 = <<EOF
MIIJKgIBAAKCAgEAxEVih1JGb8gu/Fm4AZh+ZwJw/pjzzliWrg4mICFt1g7SmIE2
TCQMKABdwd11wOFKCPc/UzRH/fHuQcvWrpbOSdqev/zKff9iedKw/YygkMeIRaXB
fYELqvUAOJ8PPfDm70st9GJRhjGgo5+L3cJB2gfgeiDNHzaFvapRSU0oMGQX+kI9
ezsjDAn+0Pp+r3h/u1QpLSH4moRFGF4omNydI+3iTGB98/EzuNhRBHRNq4oBV5SG
Pq/A1bem2ninnoEaQ+OPESxYzDz3Jy9jV0W/6LvtJ844m+XX69H5fqq5dy55z6DW
sGKn78ULPVZPsYH5Y7C+CM6GAn4nYCpau0t52sqsY5epXdeYx4Dc+Wm0CjXrUDEe
Egl4loPKDxJkQqQ/MQiz6Le/UK9vEmnWn1TRXK3ekzNV4NgDfJANBQobOpwt8WVB
rbsC0ON7n680RQnl7PltK9P1AQW5vHsahkoixk/BhcwhkrkZGyDIl9g8Q/Euyoq3
eivKPLz7/rhDE7C1BzFy7v8AjC3w7i9QeHcWOZFAXo5hiDasIAkljDOsdfD4tP5/
wSO6E6pjL3kJ+RH2FCHd7ciQb+IcuXbku64ln8gab4p8jLa/mcMI+V3eWYnZ82Yu
axsa85hAe4wb60cp/rCJo7ihhDTTvGooqtTisOv2nSvCYpcW9qbL6cGjAXECAwEA
AQKCAgEAjz6wnWDP5Y9ts2FrqUZ5ooamnzpUXlpLhrbu3m5ncl4ZF5LfH+QDN0Kl
KvONmHsUhJynC/vROybSJBU4Fu4bms1DJY3C39h/L7g00qhLG7901pgWMpn3QQtU
4P49qpBii20MGhuTsmQQALtV4kB/vTgYfinoawpo67cdYmk8lqzGzzB/HKxZdNTq
s+zOfxRr7PWMo9LyVRuKLjGyYXZJ/coFaobWBi8Y96Rw5NZZRYQQXLIalC/Dhndm
AHckpstEtx2i8f6yxEUOgPvV/gD7Akn92RpqOGW0g/kYpXjGqZQy9PVHGy61sInY
HSkcOspIkJiS6WyJY9JcvJPM6ns4b84GE9qoUlWVF3RWJk1dqYCw5hz4U8LFyxsF
R6WhYiImvjxBLpab55rSqbGkzjI2z+ucDZyl1gqIv9U6qceVsgRyuqdfVN4deU22
LzO5IEDhnGdFqg9KQY7u8zm686Ejs64T1sh0y4GOmGsSg+P6nsqkdlXH8C+Cf03F
lqPFg8WQC7ojl/S8dPmkT5tcJh3BPwIWuvbtVjFOGQc8x0lb+NwK8h2Nsn6LNazS
0H90adh/IyYX4sBMokrpxAi+gMAWiyJHIHLeH2itNKtAQd3qQowbrWNswJSgJzsT
JuJ7uqRKAFkE6nCeAkuj/6KHHMPsfCAffVdyGaWqhoxmPOrnVgECggEBAOrCCwiC
XxwUgjOfOKx68siFJLfHf4vPo42LZOkAQq5aUmcWHbJVXmoxLYSczyAROopY0wd6
Dx8rqnpO7OtZsdJMeBSHbMVKoBZ77hiCQlrljcj12moFaEAButLCdZFsZW4zF/sx
kWIAaPH9vc4MvHHyvyNoB3yQRdevu57X7xGf9UxWuPil/jvdbt9toaraUT6rUBWU
GYPNKaLFsQzKsFWAzp5RGpASkhuiBJ0Qx3cfLyirjrKqTipe3o3gh/5RSHQ6VAhz
gdUG7WszNWk8FDCL6RTWzPOrbUyJo/wz1kblsL3vhV7ldEKFHeEjsDGroW2VUFlS
asAHNvM4/uYcOSECggEBANYH0427qZtLVuL97htXW9kCAT75xbMwgRskAH4nJDlZ
IggDErmzBhtrHgR+9X09iL47jr7dUcrVNPHzK/WXALFSKzXhkG/yAgmt3r14WgJ6
5y7010LlPFrzaNEyO/S4ISuBLt4cinjJsrFpoo0WI8jXeM5ddG6ncxdurKXMymY7
EOF
>> b64 += <<EOF
gff0GJCOMZ65pMSy3A3cSAtjlKnb4fWzuHD5CFbusN4WhCT/tNxGNSpzvxd8GIDs
nY7exs9L230oCCpedVgcbayHCbkChEfoPzL1e1jXjgCwCTgt8GjeEFqc1gXNEaUn
O8AJ4VlR8fRszHm6yR0ZUBdY7UJddxQiYOzt0S1RLlECggEAbdcs4mZdqf3OjejJ
06oTPs9NRtAJVZlppSi7pmmAyaNpOuKWMoLPElDAQ3Q7VX26LlExLCZoPOVpdqDH
KbdmBEfTR4e11Pn9vYdu9/i6o10U4hpmf4TYKlqk10g1Sj21l8JATj/7Diey8scO
sAI1iftSg3aBSj8W7rxCxSezrENzuqw5D95a/he1cMUTB6XuravqZK5O4eR0vrxR
AvMzXk5OXrUEALUvt84u6m6XZZ0pq5XZxq74s8p/x1JvTwcpJ3jDKNEixlHfdHEZ
ZIu/xpcwD5gRfVGQamdcWvzGHZYLBFO1y5kAtL8kI9tW7WaouWVLmv99AyxdAaCB
Y5mBAQKCAQEAzU7AnorPzYndlOzkxRFtp6MGsvRBsvvqPLCyUFEXrHNV872O7tdO
GmsMZl+q+TJXw7O54FjJJvqSSS1sk68AGRirHop7VQce8U36BmI2ZX6j2SVAgIkI
9m3btCCt5rfiCatn2+Qg6HECmrCsHw6H0RbwaXS4RZUXD/k4X+sslBitOb7K+Y+N
Bacq6QxxjlIqQdKKPs4P2PNHEAey+kEJJGEQ7bTkNxCZ21kgi1Sc5L8U/IGy0BMC
PvJxssLdaWILyp3Ws8Q4RAoC5c0ZP0W2j+5NSbi3jsDFi0Y6/2GRdY1HAZX4twem
Q0NCedq1JNatP1gsb6bcnVHFDEGsj/35oQKCAQEAgmWMuSrojR/fjJzvke6Wvbox
FRnPk+6YRzuYhAP/YPxSRYyB5at++5Q1qr7QWn7NFozFIVFFT8CBU36ktWQ39MGm
cJ5SGyN9nAbbuWA6e+/u059R7QL+6f64xHRAGyLT3gOb1G0N6h7VqFT25q5Tq0rc
Lf/CvLKoudjv+sQ5GKBPT18+zxmwJ8YUWAsXUyrqoFWY/Tvo5yLxaC0W2gh3+Ppi
EDqe4RRJ3VKuKfZxHn5VLxgtBFN96Gy0+Htm5tiMKOZMYAkHiL+vrVZAX0hIEuRZ
EOF
>> der = b64.unpack("m").first
>> c = DerParse.new(der).first_node.first_child
>> version = c.value
=> 0
>> c = c.next_node
>> n = c.value
=> 80071596234464993385068908004931... # (etc)
>> c = c.next_node
>> e = c.value
=> 65537
>> c = c.next_node
>> d = c.value
=> 58438813486895877116761996105770... # (etc)
>> c = c.next_node
>> p = c.value
=> 29635449580247160226960937109864... # (etc)
>> c = c.next_node
>> q = c.value
=> 27018856595256414771163410576410... # (etc)
What I ve done, in case you don t speak Ruby, is take the two chunks of plausible-looking base64 data, chuck them together into a variable named b64, unbase64 it into a variable named der, pass that into a new DerParse instance, and then walk the DER value tree until I got all the values I need. Interestingly, the q value actually traverses the split in the two chunks, which means that there s always the possibility that there are lines missing from the key. However, since p and q are supposed to be prime, we can sanity check them to see if corruption is likely to have occurred:
>> require "openssl"
>> OpenSSL::BN.new(p).prime?
=> true
>> OpenSSL::BN.new(q).prime?
=> true
Excellent! The chances of a corrupted file producing valid-but-incorrect prime numbers isn t huge, so we can be fairly confident that we ve got the real p and q. Now, with the help of another one of my creations we can use e, p, and q to create a fully-operational battle key:
>> require "openssl/pkey/rsa"
>> k = OpenSSL::PKey::RSA.from_factors(p, q, e)
=> #<OpenSSL::PKey::RSA:0x0000559d5903cd38>
>> k.valid?
=> true
>> k.verify(OpenSSL::Digest::SHA256.new, k.sign(OpenSSL::Digest::SHA256.new, "bob"), "bob")
=> true
and there you have it. One fairly redacted-looking private key brought back to life by maths and far too much free time. Sorry Mr. Finn, I hope you re not still using that key on anything Internet-facing.

What About Other Key Types? EC keys are very different beasts, but they have much the same problems as RSA keys. A typical EC key contains both private and public data, and the public portion is twice the size so only about 1/3 of the data in the key is private material. It is quite plausible that you can redact an EC key and leave all the actually private bits exposed.

What Do We Do About It? In short: don t ever try and redact real private keys. For documentation purposes, just put KEY GOES HERE in the appropriate spot, or something like that. Store your secrets somewhere that isn t a public (or even private!) git repo. Generating a dummy private key and sticking it in there isn t a great idea, for different reasons: people have this odd habit of reusing demo keys in real life. There s no need to encourage that sort of thing.
  1. Technically the pieces aren t 100% aligned with the underlying DER, because of how base64 works. I felt it was easier to understand if I stuck to chopping up the base64, rather than decoding into DER and then chopping up the DER.

16 May 2020

Russ Allbery: DocKnot 3.04

This is a relatively small feature release of my tool for managing software documentation and releases. I'm slowly moving all of my packages from Travis-CI to GitHub Workflows for automated CI. GitHub Workflows is much easier to configure and control, and I've been a bit worried about the future of Travis-CI since their acquisition. It seems unlikely that GitHub Workflows is going anywhere. It would be nice to use a fully free software solution for CI, but there doesn't seem to be anything out there that's nearly as easy and straightforward to use, and I have neither the time nor the energy to cobble something together myself. The configuration is fairly straightforward and should be portable to any fully free solution that might materialize in the future. Anyway, as part of that migration I needed to make some changes to DocKnot to generate status badges from GitHub Workflows instead of Travis-CI. This release includes those changes. There is a backward-incompatible change to make the semantics of the package metadata a bit more straightforward: vcs.travis needs to be changed to vcs.status.travis. You can get the latest release from the DocKnot distribution page. Debian packages have been uploaded to my personal repository. I plan on uploading DocKnot to Debian proper once I change the metadata format to use YAML instead of relaxed JSON.

Russ Allbery: rra-c-util 8.2

This release of my general utility libraries and support code includes a large grab bag of fixes and improvements. portable/system.h now defines explicit_bzero in terms of memset if it is not available. The memset version is unlikely to have the same security properties since the compiler may optimize it away, but that allows me to use explicit_bzero to erase security data where it is available. For packages with Kerberos tests, generating a test krb5.conf file now works properly even if the system krb5.conf file does not set a default realm, and a krb5.conf file dropped into the test configuration directory now works properly. Thanks to Jeffrey Hutzelman for the latter fix. For packages with PAM modules, the ENTRY and EXIT logging macros can now be used like function calls, and portable/pam.h now defines PAM_MAX_RESP_SIZE if it isn't defined. Header ordering in some of the portability socket code has been restored to compatibility with a few ancient UNIX systems. This was accidentally broken by the clang-format reformatting. Thanks to Julien LIE for the fix. A few bugs in the test for SPDX license identifiers have been fixed. Finally, this release fixes warnings with Clang 10 and GCC 10. You can get the latest release from the rra-c-util distribution page.

Norbert Preining: Upgrading AMD Radeon 5700 to 5700 XT via BIOS

Having recently switched from NVIDIA to AMD graphic cards, in particular a RX 5700, I found out that I can get myself a free upgrade to the RX 5700 XT variant without paying one Yen, by simply flashing a compatible 5700 XT BIOS onto the 5700 card. Not that this is something new, a detailed explanation can be found here.
The same article also gives a detailed technical explanation on the difference between the two cards. The 5700 variant has less stream processors (2304 against 2560 in the XT variant), and lower power limits and clock speeds. Other than this they are based on the exact same chip layout (Navi 10), and with the same amount and type of memory 8 GB GDDR6. Flashing the XT BIOS onto the plain 5700 will not changes the number of stream processors, but power limits and clock speeds are raised to the same level of the 5700 XT, providing approximately a 7% gain without any over-clocking and over-powering, and potentially more by raising voltage etc. Detailed numbers can be found in the linked article above. The first step in this free upgrade is to identify ones own card correctly, best with device id and subsystem id, and then find the correct BIOS. Lots of BIOS dumps are provided in the BIOS database (link already restricting to 5700 XT BIOS). I used CPU-Z (Windows program) to determine this items, see image on the right (click to enlarge). In my case I got 1002 731F - 1462 3811 for the complete device id. The card is a MSI RX 5700 8 GB Mech OC, so I found the following alternative BIOS for MSI RX 5700 XT 8 GB Mech OC. Unfortunately, it seems that MSI is distinguishing 5700 and 5700 XT by their device id, because the XT variant gives 1002 731F - 1462 3810 for the complete device id, meaning that the last digit is 1 off compared to mine (3811 versus 3810). And indeed, trying to flash this video BIOS the normal way (using the Windows version ended in a warning that the subsystem id is different. A bit of search led to a thread in the TechPowerup Fora and this post explaining how to force the flashing in this case. Disclaimer: The following might brick your graphic card, you are doing this on your own risk! Necessary software: I did all the flashing and checking under Windows, but only because I realized too late that there is a fully uptodate flashing program for Linux that exhibits the same functionality. Also, I didn t know how to get the device id since the current AMD ROCm tools seem not to provide this data. If you are lucky and the device ids for your card are the same for both 5700 and 5700 XT variants, then you can use the graphical client (amdvbflashWin.exe), but if there is a difference, the command line is necessary. After unpacking the AMD Flash program and getting the correct BIOS rom file, the steps taken on Windows are (the very same steps can be taken on Linux): This should succeed in both cases. After that shutdown and restart your computer and you should be greeted with a RT 5700 XT card, without twisting a single screw. Starting Windows for the first time gave some flickering, because the driver for the new card was installed. On Linux the system auto-detects the card and everything works out of the box. Very smooth. Finally, a word of warning: Don t do these kind of things if you are not ready to pay the prize of a bricked GPU card in case something goes wrong! Everything is on your own risk! Let me close with a before/after image, most of the fields are identical, but the default/gpu clocks both at normal as well as boost levels see a considerable improvement

Sven Hoexter: Quick and Dirty: masquerading / NAT with nftables

Since nftables is now the new default, a short note to myself on how to setup masquerading, like the usual NAT setup you use on a gateway.
nft add table nat
nft add chain nat postrouting   type nat hook postrouting priority 100 \;  
nft add rule nat postrouting ip saddr 192.168.1.0/24 oif wlan0 masquerade
In this case the wlan0 is basically the "WAN" interface, because I use an old netbook as a wired to wireless network adapter.

Michael Stapelberg: a new distri linux (fast package management) release

I just released a new version of distri. The focus of this release lies on: See the release notes for more details. The distri research linux distribution project was started in 2019 to research whether a few architectural changes could enable drastically faster package management. While the package managers in common Linux distributions (e.g. apt, dnf, ) top out at data rates of only a few MB/s, distri effortlessly saturates 1 Gbit, 10 Gbit and even 40 Gbit connections, resulting in fast installation and update speeds.

Lucas Kanashiro: Quarantine times

After quite some time without publishing anything here, I decided to share the latest events. It is a hard time for most of us but with all this time at home, one can also do great things. I would like to start with the wonderful idea the Debian Brasil community had! Why not create an online Debian related conference to keep people s minds busy and also share knowledge? After brainstorming, we came up with our online conference called #FiqueEmCasaUseDebian (in English it would be #StayHomeUseDebian). It started on May 3rd and will last until May 30th (yes, one month)! Every weekday, we have one online talk at night and on every Saturday, a Debian packaging workshop. The feedback so far has been awesome and the Brazilian Debian community is reaching out to more people than usual at our regular conferences (as you might have imagined, Brazil is huge and it is hard to bring people at the same place). At the end of the month, we will have the first MiniDebConf online and I hope it will be as successful as our experience here in Brazil. Another thing that deserves a highlight is the fact that I became an Ubuntu Core Developer this month; yay! After 9 months of working almost daily on the Ubuntu Server, I was able to get my upload rights to the Ubuntu archive. I was tired of asking for sponsorship, and probably my peers were tired of me too. I could spend more time here complaining about the Brazilian government but I think it isn t worth it. Let s try to do something useful instead!

15 May 2020

Dirk Eddelbuettel: Let s celebrate Anna!

Today is graduation at Washington University, and just like many other places, the ceremonies are a lot more virtual and surreal that in other years. For Anna today marks the graduation from Biomedical Engineering with a BSc. The McKelvey School of Engineering put a Zoom meeting together yesterday which was nice, and there is something more virtual here. Hopefully a real-life commencenment can take place in a year the May 30, 2021, date has been set. The university also sent out a little commencement site/video which was cute. But at end of the day online-only still falls short of the real deal as we all know too well by now. During those years, just about the only thing really I ever tweeted about appears to be soccer related. As it should because ball is life, as we all know. Here is one from 1 1/2 years ago when her Club Team three-peated in their NIRSA division: And that opens what may be the best venue for mocking Anna: this year, which her a senior and co-captain, the team actually managed to loose a league game (a shocking first in these years) and to drop the final. I presume they anticipated that all we would all talk about around now is The Last Dance and three-peats, and left it at that. Probably wise. Now just this week, and hence days before graduating with her B.Sc., also marks the first time Anna was addressed as Dr Eddelbuettel. A little prematurely I may say, but not too shabby to be in print already! But on the topic of gratulations and what comes next, this tweet was very sweet: As was this, which marked another impressive score: So big thanks from all of us to WashU for being such a superb environment for Anna for those four years, and especially everybody at the Pappu Lab for giving Anna a home and base to start a research career. And deepest and most sincere congratulations to Anna before the next adventure starts .

Ingo Juergensmann: XMPP: ejabberd Project on the-federation.info

For those interested in alternative social networks there is that website that is called the-federation.info, which collects some statistics of "The Fediverse". The biggest part of the fediverse is Mastodon, but there are other projects (or parts) like Friendica or Matrix that do "federation". One of the oldest projects doing federation is XMPP. You could find some Prosody servers for some time now, because there is a Prosody module "mod_nodeinfo2" that can be used. But for ejabberd there is no such module (yet?) so far, which makes it a little bit difficult to get listed on the-federation.info. Some days ago I wrote a small script to export the needed values to x-nodeinfo2 that is queried by the-federation.info. It's surely not the best script or solution for that job and is currently limited to ejabberd servers that use a PostgreSQL database as backend, although it should be fairly easy to adapt the script for use with MySQL. Well, at least it does its job. At least as there is no native ejabberd module for this task. You can find the script on Github: https://github.com/ingoj/ejabberd-nodeinfo2 Enjoy it and register your ejabberd server on the-federation.info! :-)
Kategorie:

Junichi Uekawa: Bought Canon TS3300 Printer.

Bought Canon TS3300 Printer. I haven't bought a printer in 20 years. 5 years ago I think I would have used Google Cloud Print which took forever uploading and downloading huge print data to the cloud. 20 years ago I would have used LPR and LPD, admiring the postscript backtraces. Configuration was done using Android app, the printer could be connected via WiFi and then configuration through the app allowed printer to be connected to the home WiFi network, feels modern. It was recognized as a IPPS device by Chrome OS (using CUPS?). No print server necessary. Amazing.

14 May 2020

Iustin Pop: New internet provider

Note: all this is my personal experience, on my personal machines, and I don t claim it to be the absolute truth. Also, I don t directly call out names, although if you live in Switzerland it s pretty easy to guess who the old provider is from some hints. For a long time, I wanted to move away from my current (well, now past) provider, for a multitude of reasons. The main being that the company is very classic company, with classic support system, that doesn t work very well - I had troubles with their billing system that left me out cold without internet for 15 days, but for the recent few years, they were mostly OK, and changing to a different provider would have meant me routing a really long Ethernet cable around the apartment, so I kept postponing it. Yes, self-inflicted pain, I know. Until the entire work-from-home thing, when the usually stable connection start degrading in a very linear fashion day-by-day (this is a graph that basically reflects download bandwidth):
1+ month download bandwidth test 1+ month download bandwidth test
At first, I didn t realise this, as even 100Mbps is still fast enough. But once the connection went below (I believe) 50Mbps, it became visible in day to day work. And since I work daily from home yeah. Not fun. So I started doing speedtest.net - and oh my, ~20Mbps was a good result, usually 12-14Mbps. On a wired connection. On a connection that officially is supposed to be 600Mbps down. The upload speed was spot on, so I didn t think it was my router, but: Nothing helped. Once in a blue moon, speedtest would give me 100Mbps, but like once every two days, then it would be back and showing 8Mbps. Eight! It ended up as even apt update was tediously slow, and kernel downloads took ages The official instructions for dealing with bad internet are a joke: And the best part: If you are not satisfied with the results, read our internet optimisation guide. If you are still not happy, use our community forums or our social platforms. Given that it was a decline over 3-weeks, that I don t know of any computer component that would degrade this steadily but not throw any other errors, and that my upload speed was all good, I assumed it s the provider. Maybe I was wrong, but I wanted to do this anyway for a long while, so I went through the find how to route cable, check if other provider socket is good, order service, etc. dance, and less than a week later, I had the other connection. Now, of course, bandwidth works as expected:
1+ month updated bandwidth test 1+ month updated bandwidth test
Both download and upload are fine (the graph above is just download). Latency is also much better, towards many parts of the internet that matter. But what is shocking is the difference in jitter to some external hosts I care about. On the previous provider, a funny thing was that both outgoing and incoming pings had both more jitter and packet loss when done directly (IPv4 to IPv4) than when done over a VPN. This doesn t make sense, since VPN is just overhead over IPv4, but the graphs show it, and what I think happens is that a VPN flow is cached in the provider s routers, whereas a simple ping packet not. But, the fact that there s enough jitter for a ping to a not-very-far host doesn t make me happy. Examples, outgoing:
Outgoing smokeping to public IPv4 Outgoing smokeping to public IPv4
Outgoing smokeping over VPN Outgoing smokeping over VPN
And incoming:
Incoming smokeping to public IPv4 Incoming smokeping to public IPv4
Incoming smokeping over VPN Incoming smokeping over VPN
Both incoming and outgoing show this weirdness - more packet loss and more jitter over VPN. Again, this is not a problem in practice, or not much, but makes me wonder what other shenanigans happen behind the scenes. You can also see clearly when the work from home traffic entered the picture and started significantly degrading my connection, even over the magically better VPN connection. Switching to this week s view shows the (in my opinion) dramatic improvement in consistency of the connection:
Outgoing current week smokeping to public IPv4 Outgoing current week smokeping to public IPv4
Outgoing current week smokeping over VPN Outgoing current week smokeping over VPN
No more packet loss, no more jitter. You can also see my VPN being temporarily down during provider switchover because my firewall was not quite correct for a moment. And the last drill down, at high resolution, one day before and one day after switchover. Red is VPN, blue is plain IPv4, yellow is the missing IPv6 connection :) Incoming old:
Incoming 1-day smokeping, old provider Incoming 1-day smokeping, old provider
and new:
Incoming 1-day smokeping, new provider Incoming 1-day smokeping, new provider
Outgoing old:
Outgoing 1-day smokeping, old provider Outgoing 1-day smokeping, old provider
and new:
Outgoing 1-day smokeping, new provider Outgoing 1-day smokeping, new provider
This is what I expect, ping-over-VPN should of course be slower than plain ping. Note that incoming and outgoing have slightly different consistency, but that is fine for me :) The endpoints doing the two tests are different, so this is expected. Reading the legend on the graphs for the incoming connection (similar story for outgoing): So to my previous provider: it can be done better. Or at least, allow people easier ways to submit performance issue problems. For me, the moral of the story is that I should have switched a couple of years ago, instead of being lazy. And that I m curious to see how IPv6 traffic will differ, if at all :) Take care, everyone! And thanks for looking at these many graphs :)

Norbert Preining: Switching from NVIDIA to AMD (including tensorflow)

I have been using my Geforce 1060 extensively for deep learning, both with Python and R. But the always painful play with the closed source drivers and kernel updates, paired with the collapse of my computer s PSU and/or GPU, I decided to finally do the switch to AMD graphic card and open source stack. And you know what, within half a day I had everything, including Tensorflow running. Yeah to Open Source!
Preliminaries So what is the starting point: I am running Debian/unstable with a AMD Radeon 5700. First of all I purged all NVIDIA related packages, and that are a lot I have to say. Be sure to search for nv and nvidia and get rid of all packages. For safety I did reboot and checked again that no kernel modules related to NVIDIA are loaded. Firmware Debian ships the package amd-gpu-firmware but this is not enough for the current kernel and current hardware. Better is to clone git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git and copy everything from the amdgpu directory to /lib/firmware/amdgpu. I didn t do that at first, and then booting the kernel did hang during the switch to AMD framebuffer. If you see this behaviour, your firmwares are too old. Kernel The advantage of having open source driver that is in the kernel is that you don t have to worry about incompatibilities (like every time a new kernel comes out the NVIDIA driver needs patching). For recent AMD GPUs you need a rather new kernel, I have 5.6.0 and 5.7.0-rc5 running. Make sure that you have all the necessary kernel config options turned on if you compile your own kernels. In my case this is
CONFIG_DRM_AMDGPU=m
CONFIG_DRM_AMDGPU_USERPTR=y
CONFIG_DRM_AMD_ACP=y
CONFIG_DRM_AMD_DC=y
CONFIG_DRM_AMD_DC_DCN=y
CONFIG_HSA_AMD=y
When installing the kernel, be sure that the firmware is already updated so that the correct firmware is copied into the initrd. Support programs and libraries All the following is more or less an excerpt from the ROCm Installation Guide! AMD provides a Debian/Ubuntu APT repository for software as well as kernel sources. Put the following into /etc/apt/sources.list.d/rocm.list:
deb [arch=amd64] http://repo.radeon.com/rocm/apt/debian/ xenial main
and also put the public key of the rocm repository into /etc/apt/trusted.d/rocm.asc. After that apt-get update should work. I did install rocm-dev-3.3.0, rocm-libs-3.3.0, hipcub-3.3.0, miopen-hip-3.3.0 (and of course the dependencies), but not rocm-dkms which is the kernel module. If you have a sufficiently recent kernel (see above), the source in the kernel itself is newer. The libraries and programs are installed under /opt/rocm-3.3.0, and to make the libraries available to Tensorflow (see below) and other programs, I added /etc/ld.so.conf.d/rocm.conf with the following content:
/opt/rocm-3.3.0/lib/
and run ldconfig as root. Last but not least, add a udev rule that is normally installed by rocm-dkms, put the following into /etc/udev/rules.d/70-kfd.rules:
SUBSYSTEM=="kfd", KERNEL=="kfd", TAG+="uaccess", GROUP="video"
This allows users from the video group to access the GPU.
Up to here you should be able to boot into the system and have X running on top of AMD GPU, including OpenGL acceleration and direct rendering:
$ glxinfo
ame of display: :0
display: :0  screen: 0
direct rendering: Yes
server glx vendor string: SGI
server glx version string: 1.4
...
client glx vendor string: Mesa Project and SGI
client glx version string: 1.4
...
Tensorflow Thinking about how hard it was to get the correct libraries to get Tensorflow running on GPUs (see here and here), it is a pleasure to see that with open source all this pain is relieved. There is already work done to make Tensorflow run on ROCm, the tensorflow-rocm project. The provide up to date PyPi packages, so a simple
pip3 install tensorflow-rocm
is enough to get Tensorflow running with Python:
>> import tensorflow as tf
>> tf.add(1, 2).numpy()
2020-05-14 12:07:19.590169: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libhip_hcc.so
...
2020-05-14 12:07:19.711478: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 7444 MB memory) -> physical GPU (device: 0, name: Navi 10 [Radeon RX 5600 OEM/5600 XT / 5700/5700 XT], pci bus id: 0000:03:00.0)
3
>>
Tensorflow for R Installation is trivial again since there is a tensorflow for R package, just run (as a user that is in the group staff, which normally own /usr/local/lib/R)
$ R
...
> install.packages("tensorflow")
..
Do not call the R function install_tensorflow() since Tensorflow is already installed and functional! With that done, R can use the AMD GPU for computations:
$ R
...
> library(tensorflow)
> tf$constant("Hellow Tensorflow")
2020-05-14 12:14:24.185609: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libhip_hcc.so
...
2020-05-14 12:14:24.277736: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 7444 MB memory) -> physical GPU (device: 0, name: Navi 10 [Radeon RX 5600 OEM/5600 XT / 5700/5700 XT], pci bus id: 0000:03:00.0)
tf.Tensor(b'Hellow Tensorflow', shape=(), dtype=string)
>
AMD Vulkan From the Vulkan home page:
Vulkan is a new generation graphics and compute API that provides high-efficiency, cross-platform access to modern GPUs used in a wide variety of devices from PCs and consoles to mobile phones and embedded platforms.
Several games are using the Vulkan API if available and it is said to be more efficient. There are Vulkan libraries for Radeon shipped in with mesa, in the Debian package mesa-vulkan-drivers, but they look a bit outdated is my guess. The AMDVLK project provides the latest version, and to my surprise was rather easy to install, again by following the advice in their README. The steps are basically (always follow what is written for Ubuntu): All as described in the linked README. Just to make sure, I removed the JSON files /usr/share/vulkan/icd.d/radeon* shipped by Debians mesa-vulkan-drivers package. Finally I deviated a bit by not editing the file /usr/share/X11/xorg.conf.d/10-amdgpu.conf, but instead copying to /etc/X11/xorg.conf.d/10-amdgpu.conf and adding there the section:
Section "Device"
        Identifier "AMDgpu"
        Option  "DRI" "3"
EndSection
. To be honest, I did not follow the Copy driver and JSON files literally, since I don t want to copy self-made files into system directories under /usr/lib. So what I did is: With this in place, I don t pollute the system directories, and still the new Vulkan driver is available. But honestly, I don t really know whether it is used and is working, because I don t know how to check.
With all that in place, I can run my usual set of Steam games (The Long Dark, Shadow of the Tomb Raider, The Talos Principle, Supraland, ) and I don t see any visual problem till now. As a bonus, KDE/Plasma is now running much better, since NVIDIA and KDE has traditionally some incompatibilities. The above might sound like a lot of stuff to do, but considering that most of the parts are not really packaged within Debian, and all this is rather new open source stack, I was surprised that in half a day I got all working smoothly. Thanks to all the developers who have worked hard to make this all possible.

13 May 2020

Mike Gabriel: Q: Remote Support Framework for the GNU/Linux Desktop?

TL;DR; For those (admins) of you who run GNU/Linux on staff computers: How do you organize your graphical remote support in your company? Get in touch, share your expertise and experiences. Researching on FLOSS based Linux Desktops When bringing GNU/Linux desktops to a generic folk of productive office users on a large scale, graphical remote support is a key feature when organizing helpdesk support teams' workflows. In a research project that I am currently involved in, we investigate the different available remote support technologies (VNC screen mirroring, ScreenCasts, etc.) and the available frameworks that allow one to provide a remote support infrastructure 100% on-premise. In this research project we intend to find FLOSS solutions for everything required for providing a large scale GNU/Linux desktop to end users, but we likely will have to recommend non-free solutions, if a FLOSS approach is not available for certain demands. Depending on the resulting costs, bringing forth a new software solution instead of dumping big money in subscription contracts for non-free software is seen as a possible alternative. As a member of the X2Go upstream team and maintainer of several remote desktop related tools and frameworks in Debian, I'd consider myself as sort of in-the-topic. The available (as FLOSS) underlying technologies for plumbing a remote support framework are pretty much clear (x11vnc, recent pipewire-related approaches in Wayland compositors, browser-based screencasting). However, I still lack a good spontaneous answer to the question: "How to efficiently software-side organize a helpdesk scenario for 10.000+ users regarding graphical remote support?". Framework for Remote Desktop in Webbrowsers In fact, in the context of my X2Go activities, I am currently planning to put together a Django-based framework for running X2Go sessions in a web browser. The framework that we will come up with (two developers have already been hired for an initial sprint in July 2020) will be designed to be highly pluggable and it will probably be easy to add remote support / screen sharing features further on. And still, I walk around with the question in mind: Do I miss anything? Is there anything already out there that provides a remote support solution as 100% FLOSS, that has enterprise grade, that up-scales well, that has a modern UI design, etc. Something that I simply haven't come across, yet? Looking forward to Your Feedback Please get in touch (OFTC/Freenode IRC, Telegram, Email), if you can fill the gap and feel like sharing your ideas and experiences. light+love
Mike

Russ Allbery: Review: Gideon the Ninth

Review: Gideon the Ninth, by Tamsyn Muir
Series: The Locked Tomb #1
Publisher: Tor
Copyright: September 2019
ISBN: 1-250-31317-1
Format: Kindle
Pages: 448
Despite being raised there, Gideon Nav is an outsider in the Ninth House. Her mother, already dead, fell from the sky with a one-day-old Gideon in tow, leaving her an indentured servant. She's a grumpy, caustic teenager in a world of moldering corpses, animated skeletons, and mostly-dead adults whose parts are falling off. Her world is sword fighting, dirty magazines, a feud with the house heir Harrowhark, and a determination to escape the terms of her indenture. Gideon does get off the planet, but not the way that she expects. She doesn't get accepted into the military. She ends up in the middle of a bizarre test, or possibly an ascension rite, mingling with and competing with the nobility of the empire alongside her worst enemy. I struggled to enjoy the beginning of Gideon the Ninth. Gideon tries to carry the story on pure snark, but it is very, very goth. If you like desiccated crypts, mostly-dead goons, betrayal, frustration, necromancers, black robes, disturbing family relationships, gloom, and bitter despair, the first six chapters certainly deliver, but I was sick of it by the time Gideon gets out. Thankfully, the opening is largely unlike the rest of the book. What starts as an over-the-top teenage goth rebellion turns into a cross between a manor house murder mystery and a competitive escape room. This book is a bit of a mess, but it's a glorious mess. It's also the sort of glorious mess that I don't think would have been written or published twenty years ago, and I have a pet theory that attributes this to the invigorating influence of fanfic and writers who grew up reading and writing it. I read a lot of classic science fiction and epic fantasy as a teenager. Those books have many merits, obviously, but emotional range is not one of them. There are a few exceptions, but on average the genre either focused on puzzles and problem solving (how do we fix the starship, how do we use the magic system to take down the dark god) or on the typical "heroic" (and male-coded) emotions of loyalty, bravery, responsibility, authority, and defiance of evil. Characters didn't have messy breakups, frenemies, anxiety, socially-awkward love affairs, impostor syndrome, self-hatred, or depression. And authors weren't allowed to fall in love with the messiness of their characters, at least on the page. I'm not enough of a scholar to make the argument well, but I suspect there's a case to be made that fanfic exists partially to fill this gap. So much of fanfic starts from taking the characters on the canonical page or screen and letting them feel more, live more, love more, screw up more, and otherwise experience a far wider range of human drama, particularly compared to what made it into television, which was even more censored than what made it into print. Some of those readers and writers are now writing for publication, and others have gone into publishing. The result, in my theory, is that the range of stories that are acceptable in the genre has broadened, and the emotional texture of those stories has deepened. Whether or not this theory is correct, there are now more novels like this in the world, novels full of grudges, deflective banter, squabbling, messy emotional processing, and moments of glorious emotional catharsis. This makes me very happy. To describe the emotional payoff of this book in any more detail would be a huge spoiler; suffice it to say that I unabashedly love fragile competence and unexpected emotional support, and adore this book for containing it. Gideon's voice, irreverent banter, stubborn defiance, and impulsive good-heartedness are the center of this book. At the start, it's not clear whether there will be another likable character in the book. There will be, several of them, but it takes a while for Gideon to find them or for them to become likable. You'll need to like Gideon well enough to stick with her for that journey. I read books primarily for the characters, not for the setting, and Gideon the Ninth struck some specific notes that I will happily read endlessly. If that doesn't match your preferences, I would not be too surprised to hear you bounced off the book. There's a lot here that won't be to everyone's taste. The setting felt very close to Warhammer 40K: an undead emperor that everyone worships, endless war, necromancy, and gothic grimdark. The stage for most of the book is at least more light-filled, complex, and interesting than the Ninth House section at the start, but everything is crumbling, drowning, broken, or decaying. There's quite a lot of body horror, grotesque monsters, and bloody fights. And the ending is not the best part of the book; roughly the last 15% of the novel is composed of two running fight scenes against a few practically unkillable and frankly not very interesting villains. I got exhausted by the fighting long before it was over, and the conclusion is essentially a series cliffhanger. There are also a few too many characters. The collection of characters and the interplay between the houses is one of the strengths of this book, but Muir sets up her story in a way that requires eighteen significant characters and makes the reader want to keep track of all of them. It took me about halfway through the book before I felt like I had my bearings and wasn't confusing one character for another or forgetting a whole group of characters. That said, most of the characters are great, and the story gains a lot from the interplay of their different approaches and mindsets. Palamedes Sextus's logical geekery, in particular, is a great counterpoint to the approaches of most of the other characters. The other interesting thing Muir does in this novel that I've not seen before, and that feels very modern, is to set the book in essentially an escape room. Locking a bunch of characters in a sprawling mansion until people start dying is an old fictional trope, but this one has puzzles, rewards, and a progressive physical structure that provides a lot of opportunities to motivate the characters and give them space to take wildly different problem-solving approaches. I liked this a lot, and I'm looking forward to seeing it in future books. This is not the best book I've read, but I thoroughly enjoyed it, despite some problems with the ending. I've already pre-ordered the sequel. Followed by Harrow the Ninth. Rating: 8 out of 10

12 May 2020

Kunal Mehta: MediaWiki packages for Ubuntu 20.04 Focal available

Packages for the MediaWiki 1.31 LTS release are now available for the new Ubuntu 20.04 LTS "Focal Fossa" release in my PPA. Please let me know if you run into any errors or issues. In the future these packages will be upgraded to the MediaWiki 1.35 LTS release, whenever that's ready. It's currently delayed because of the pandemic, but I expect that it'll be ready for the next Debian release.

Evgeni Golov: Building a Shelly 2.5 USB to TTL adapter cable

When you want to flash your Shelly 2.5 with anything but the original firmware for the first time, you'll need to attach it to your computer. Later flashes can happen over the air (at least with ESPHome or Tasmota), but the first one cannot. In theory, this is not a problem as the Shelly has a quite exposed and well documented interface: Shelly 2.5 pinout However, on closer inspection you'll notice that your normal jumper wires don't fit as the Shelly has a connector with 1.27mm (0.05in) pitch and 1mm diameter holes. Now, there are various tutorials on the Internet how to build a compatible connector using Ethernet cables and hot glue or with female header socket legs, and you can even buy cables on Amazon for 18 ! But 18 sounded like a lot and the female header socket thing while working was pretty finicky to use, so I decided to build something different. We'll need 6 female-to-female jumper wires and a 1.27mm pitch male header. Jumper wires I had at home, the header I got is a SL 1X20G 1,27 from reichelt.de for 0.61 . It's a 20 pin one, so we can make 3 adapters out of it if needed. Oh and we'll need some isolation tape. SL 1X20G 1,27 The first step is to cut the header into 6 pin chunks. Make sure not to cut too close to the 6th pin as the whole thing is rather fragile and you might lose it. SL 1X20G 1,27 cut into pieces It now fits very well into the Shelly with the longer side of the pins. Shelly 2.5 with pin headers attached Second step is to strip the plastic part of one side of the jumper wires. Those are designed to fit 2.54mm pitch headers and won't work for our use case otherwise. jumper wire with removed plastic As the connectors are still too big, even after removing the plastic, the next step is to take some pliers and gently press the connectors until they fit the smaller pins of our header. Shelly 2.5 with pin headers and a jumper wire attached Now is the time to put everything together. To avoid short circuiting the pins/connectors, apply some isolation tape while assembling, but not too much as the space is really limited. Shelly 2.5 with pin headers and a jumper wire attached and taped And we're done, a wonderful (lol) and working (yay) Shelly 2.5 cable that can be attached to any USB-TTL adapter, like the pictured FTDI clone you get almost everywhere. Shelly 2.5 with full cable and FTDI attached Yes, in an ideal world we would have soldered the header to the cable, but I didn't feel like soldering on that limited space. And yes, shrink-wrap might be a good thing too, but again, limited space and with isolation tape you only need one layer between two pins, not two.

Daniel Silverstone: The Lars, Mark, and Daniel Club

Last night, Lars, Mark, and I discussed Jeremy Kun's The communicative value of using Git well post. While a lot of our discussion was spawned by the article, we did go off-piste a little, and I hope that my notes below will enlighten you all as to a bit of how we see revision control these days. It was remarkably pleasant to read an article where the comments section wasn't a cesspool of horror, so if this posting encourages you to go and read the article, don't stop when you reach the bottom -- the comments are good and useful too.
This was a fairly non-contentious article for us though each of us had points we wished to bring up and chat about it turned into a very convivial chat. We saw the main thrust of the article as being about using the metadata of revision control to communicate intent, process, and decision making. We agreed that it must be possible to do so effectively with Mercurial (thus deciding that the mention of it was simply a bit of clickbait / red herring) and indeed Mark figured that he was doing at least some of this kind of thing way back with CVS. We all discussed how knowing the fundamentals of Git's data model improved our ability to work wih the tool. Lars and I mentioned how jarring it has originally been to come to Git from revision control systems such as Bazaar (bzr) but how over time we came to appreciate Git for what it is. For Mark this was less painful because he came to Git early enough that there was little more than the fundamental data model, without much of the porcelain which now exists. One point which we all, though Mark in particular, felt was worth considering was that of distinguishing between published and unpublished changes. The article touches on it a little, but one of the benefits of the workflow which Jeremy espouses is that of using the revision control system as an integral part of the review pipeline. This is perhaps done very well with Git based workflows, but can be done with other VCSs. With respect to the points Jeremy makes regarding making commits which are good for reviewing, we had a general agreement that things were good and sensible, to a point, but that some things were missed out on. For example, I raised that commit messages often need to be more thorough than one-liners, but Jeremy's examples (perhaps through expedience for the article?) were all pretty trite one-liners which perhaps would have been entirely obvious from the commit content. Jeremy makes the point that large changes are hard to review, and Lars poined out that Greg Wilson did research in this area, and at least one article mentions 200 lines as a maximum size of a reviewable segment. I had a brief warble at this point about how reviewing needs to be able to consider the whole of the intended change (i.e. a diff from base to tip) not just individual commits, which is also missing from Jeremy's article, but that such a review does not need to necessarily be thorough and detailed since the commit-by-commit review remains necessary. I use that high level diff as a way to get a feel for the full shape of the intended change, a look at the end-game if you will, before I read the story of how someone got to it. As an aside at this point, I talked about how Jeremy included a 'style fixes' commit in his example, but I loathe seeing such things and would much rather it was either not in the series because it's unrelated to it; or else the style fixes were folded into the commits they were related to. We discussed how style rules, as well as commit-bisectability, and other rules which may exist for a codebase, the adherence to which would form part of the checks that a code reviewer may perform, are there to be held to when they help the project, and to be broken when they are in the way of good communication between humans. In this, Lars talked about how revision control histories provide high level valuable communication between developers. Communication between humans is fraught with error and the rules are not always clear on what will work and what won't, since this depends on the communicators, the context, etc. However whatever communication rules are in place should be followed. We often say that it takes two people to communicate information, but when you're writing commit messages or arranging your commit history, the second party is often nebulous "other" and so the code reviewer fulfils that role to concretise it for the purpose of communication. At this point, I wondered a little about what value there might be (if any) in maintaining the metachanges (pull request info, mailing list discussions, etc) for historical context of decision making. Mark suggested that this is useful for design decisions etc but not for the style/correctness discussions which often form a large section of review feedback. Maybe some of the metachange tracking is done automatically by the review causing the augmentation of the changes (e.g. by comments, or inclusion of design documentation changes) to explain why changes are made. We discussed how the "rebase always vs. rebase never" feeling flip-flopped in us for many years until, as an example, what finally won Lars over was that he wants the history of the project to tell the story, in the git commits, of how the software has changed and evolved in an intentional manner. Lars said that he doesn't care about the meanderings, but rather about a clear story which can be followed and understood. I described this as the switch from "the revision history is about what I did to achieve the goal" to being more "the revision history is how I would hope someone else would have done this". Mark further refined that to "The revision history of a project tells the story of how the project, as a whole, chose to perform its sequence of evolution." We discussed how project history must necessarily then contain issue tracking, mailing list discussions, wikis, etc. There are exist free software projects where part of their history is forever lost because, for example, the project moved from Sourceforge to Github, but made no effort (or was unable) to migrate issues or somesuch. Linkages between changes and the issues they relate to can easily be broken, though at least with mailing lists you can often rewrite URLs if you have something consistent like a Message-Id. We talked about how cover notes, pull request messages, etc. can thus also be lost to some extent. Is this an argument to always use merges whose message bodies contain those details, rather than always fast-forwarding? Or is it a reason to encapsulate all those discussions into git objects which can be forever associated with the changes in the DAG? We then diverted into discussion of CI, testing every commit, and the benefits and limitations of automated testing vs. manual testing; though I think that's a little too off-piste for even this summary. We also talked about how commit message audiences include software perhaps, with the recent movement toward conventional commits and how, with respect to commit messages for machine readability, it can become very complex/tricky to craft good commit messages once there are multiple disparate audiences. For projects the size of the Linux kernel this kind of thing would be nearly impossible, but for smaller projects, perhaps there's value. Finally, we all agreed that we liked the quote at the end of the article, and so I'd like to close out by repeating it for you all... Hal Abelson famously said:
Programs must be written for people to read, and only incidentally for machines to execute.
Jeremy agrees, as do we, and extends that to the metacommit information as well.

Next.

Previous.