Search Results: "ema"

3 May 2017

Reproducible builds folks: Reproducible Builds: week 105 in Stretch cycle

Here's what happened in the Reproducible Builds effort between Sunday April 23 and Saturday April 29 2017: Past and upcoming events On April 26th Chris Lamb gave a talk at foss-north 2017 in Gothenburg, Sweden on Reproducible Builds. Between May 5th-7th the Reproducible Builds Hackathon 2017 will take place in Hamburg, Germany. Then on May 26th Bernhard M. Wiedemann will give a talk titled reproducible builds in openSUSE (2017) at the openSUSE Conference 2017 in N rnberg, Germany. Media coverage Already on April 19th Sylvain Beucler wrote a yet another follow-up post Practical basics of reproducible builds 3, after part 1 and part 2 of his series. Toolchain development and fixes Michael Woerister of the Rust project has implemented file maps that affect all path-related compiler information, including "error messages, metadata, debuginfo, and the file!() macro alike". Ximin Luo with support from some other Rust developers and contributors helped steer the final result into something that was compatible with reproducible builds. Many thanks to all involved, especially for the patience of discussing this over several months. Ximin wrote a first-attempt patch to fix R build-path issues. It made 460/477 R packages reproducible, but also caused 3 of these to FTBFS. See randomness_in_r_rdb_rds_databases for details. Bugs filed and patches sent upstream Chris Lamb: Bernhard M. Wiedemann filed a number of patches upstream: Reviews of unreproducible packages 102 package reviews have been added, 64 have been updated and 24 have been removed in this week, adding to our knowledge about identified issues. 3 issue types have been updated: Weekly QA work During our reproducibility testing, FTBFS bugs have been detected and reported by: diffoscope development diffoscope 82 was uploaded to experimental by Chris Lamb. It included contributions from: Changes from previous weeks that were also released with 82: Misc. This week's edition was written by Ximin Luo, Chris Lamb and Holger Levsen & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.

Reproducible builds folks: Reproducing R packages

In the past couple of weeks, Ximin Luo worked on making R generate reproducible output. This is now mostly complete, and we're waiting on feedback from upstream about our patch. In the meantime, there are a few packages that remain unreproducible, but the issue probably lies in those specific packages rather than the R toolchain. Perhaps you can help out with them, after reading this!
R packages compile into a .rdb database format that contains the package's definitions, plus a .rdx index file for easy lookup in the .rdb file. Usually, there is a main rdb with the package contents, plus another rdb that stores the help data. There is also a paths.rds (same format as .rdx) that contains some more stuff. One can actually read these files by hand using Rscript, see the diffoscope code in rdata.py. If you run that, you can see that path.rds contains some obvious paths, but the other files contain less obvious stuff, and in fact these scripts give identical output for the .rdb files, even though they are bitwise different. To get to the bottom of this, we'll have to use the R debugger. Attached to this post is a script that smooths this process. I ran this against Debian's R packages, but it probably also works with other distros' R packages - try it and see. You run it like ./r-mini-repro-test.sh $pkgdir $builddir and it will output some hashes for you; make sure to install the build dependencies first. You should manually vary both $pkgdir and $builddir to introduce the build-path variations; $builddir can be an arbitrary string but $pkgdir should point to the actual R package's source directory, so I just copy that to two locations and point the script at each of them in turn. Now, we can begin debugging. Before I did this, we had 478 unreproducible R packages so the biggest problem was likely with R itself. I downloaded the source code of both R and a small example package (r-cran-tensor), then figured out how the R packages were actually being built. This resulted in me writing the script above, to speed up debugging. You can read about R's debugger here (no HTTPS), it's a bit primitive but it supports the basic stuff (step, continue, where, print/eval) and that was adequate for me. Debugging R core So let's take a small example with Debian R 3.4.0-1: 
$ ../r-mini-repro-test.sh r-cran-tensor-1.5 123
[..]
* installing *source* package  tensor  ...
** package  tensor  successfully unpacked and MD5 sums checked
** R
** preparing package for lazy loading
** help
*** installing help indices
** building package indices
** testing if installed package can be loaded
* DONE (tensor)
[..]
c71a766429151a13e46039f3c37a14edc704004f410c34f8f37a2f098506421d  ./debian/123/usr/lib/R/site-library/tensor/R/tensor.rdx
736b5b4a885db71fb3fe2a538faeefe43c500c4080dbf2c75b97135b98401acd  ./debian/123/usr/lib/R/site-library/tensor/R/tensor.rdb
8a3f9616de7ee78a83b07395dd6ec43c5259d3f1b1b653d397808fc9fc5d96c2  ./debian/123/usr/lib/R/site-library/tensor/help/tensor.rdx
d1ac634987b24606a5da627ec073f45edce2bddf071391dd17c246a0a01eba63  ./debian/123/usr/lib/R/site-library/tensor/help/tensor.rdb
80796cc1a2cfcac40c394153f67617d575e103a405430b8483c2261a903db046  ./debian/123/usr/lib/R/site-library/tensor/help/paths.rds
$ ../r-mini-repro-test.sh r-cran-tensor-1.5 1234
[..]
5f081f4d6f9da1fbe06e374067e6636d926de4cdf09886058b15c5928c636c0d  ./debian/1234/usr/lib/R/site-library/tensor/R/tensor.rdx
7145669891e1b766ca0d8d89fa9c317af716e61a62650abe3a11a848240f1d57  ./debian/1234/usr/lib/R/site-library/tensor/R/tensor.rdb
8a3f9616de7ee78a83b07395dd6ec43c5259d3f1b1b653d397808fc9fc5d96c2  ./debian/1234/usr/lib/R/site-library/tensor/help/tensor.rdx
d1ac634987b24606a5da627ec073f45edce2bddf071391dd17c246a0a01eba63  ./debian/1234/usr/lib/R/site-library/tensor/help/tensor.rdb
80796cc1a2cfcac40c394153f67617d575e103a405430b8483c2261a903db046  ./debian/1234/usr/lib/R/site-library/tensor/help/paths.rds
The files are created by the ** preparing package for lazy loading and ** help steps. In retrospect this seems somewhat obvious from the output, but I had never done R before so I spent a while using strace to be sure; one can see that those files are indeed written to after those strings are printed. Then we can grep the R source code for these strings. 
r-base-3.4.0$ grep -Ri "preparing package for lazy loading" src
src/library/tools/R/install.R:                starsmsg(stars, "preparing package for lazy loading")
Reading the file, we see that this happens in the tools:::.install_packages function, and reading it further we see that it calls makeLazyLoading then code2LazyLoadDB then makeLazyLoadDB. Seems promising, let's confirm it before chasing potential wild geese. Important: In the rest of these command-line outputs I'll prepend what I input with >>> but this doesn't actually get printed by Rscript. So don't be surprised that you don't see these, when you're trying to recreate my steps. Also I am a total R noob so possibly there are more elegant ways to do what I'm about to show you; this is what I came up with after 2-3 hours of research. 
$ DEBUG=1 ../r-mini-repro-test.sh r-cran-tensor-1.5 123
enter (some or all of) the following into R:
[.. helpful commands ..]
+ R_DEFAULT_PACKAGES= LC_COLLATE=C /usr/lib/R/bin/R --no-restore --slave --args nextArg-lnextArgdebian/123/usr/lib/R/site-librarynextArg-dnextArg.nextArg--built-timestamp="Thu, 01 Jan 1970 00:00:00 +0000"
>>> debug(tools:::makeLazyLoadDB)
>>> tools:::.install_packages()
processing  . 
[..]
** preparing package for lazy loading
debugging in: makeLazyLoadDB(ns, dbbase, compress = compress)
debug:  
[.. source code of makeLazyLoadDB, not yet run ..]
OK so we just entered the function and it's paused, in a separate shell let's check what it's doing: 
$ find r-cran-tensor-1.5 -name '*.rdb'
[ no output ]
So, no files have been written yet, which means none of the parent functions did anything, we are in the right function (or it's in a child). If we continue the function, we get: 
>>> c
exiting from: makeLazyLoadDB(ns, dbbase, compress = compress)
** help
debugging in: makeLazyLoadDB(db, file.path(manOutDir, basename(outDir)))
debug:  
[.. source code of makeLazyLoadDB, not yet run ..]
 
$ find r-cran-tensor-1.5 -name '*.rdb'
r-cran-tensor-1.5/debian/123/usr/lib/R/site-library/tensor/R/tensor.rdb
Then continuing again: 
>>> c
exiting from: makeLazyLoadDB(db, file.path(manOutDir, basename(outDir)))
*** installing help indices
** building package indices
** testing if installed package can be loaded
* DONE (tensor)
$ find r-cran-tensor-1.5 -name '*.rdb'
r-cran-tensor-1.5/debian/123/usr/lib/R/site-library/tensor/help/tensor.rdb
r-cran-tensor-1.5/debian/123/usr/lib/R/site-library/tensor/R/tensor.rdb
So it seems likely this function is the correct one. After reading the source code of makeLazyLoadDB, even though I don't understand the whole thing, it looks like possibly this part is the one that does the actual writing: 
    for (i in seq_along(vars))  
        key <- if (is.null(from)   is.environment(from))
            lazyLoadDBinsertVariable(vars[i], from, datafile,
                                     ascii, compress,  envhook)
        else lazyLoadDBinsertListElement(from, i, datafile, ascii,
                                         compress, envhook)
        assign(vars[i], key, envir = varenv)
So let's step through it and see if the data contains any paths. 
$ DEBUG=1 ../r-mini-repro-test.sh r-cran-tensor-1.5 123
[..]
>>> debug(tools:::makeLazyLoadDB)
>>> tools:::.install_packages()
processing  . 
[..]
** preparing package for lazy loading
debugging in: makeLazyLoadDB(ns, dbbase, compress = compress)
debug:  
[.. source code of makeLazyLoadDB, not yet run ..]
 
>>>                                                                             # pressing <Enter> just means "step"
debug: ascii <- as.logical(ascii)                                               # this is the first line of the function
>>>                                                                             # keep stepping
[..]
>>>
debug: for (i in seq_along(vars))  
[.. source code of the inside of the block, not yet run ..]
 
>>>
debug: key <- if (is.null(from)   is.environment(from)) lazyLoadDBinsertVariable(vars[i],
    from, datafile, ascii, compress, envhook) else lazyLoadDBinsertListElement(from,
    i, datafile, ascii, compress, envhook)
>>>
debug: lazyLoadDBinsertVariable(vars[i], from, datafile, ascii, compress,
    envhook)                                                                    # not yet run
If we look at the source code we see that it will try to insert the value of .Internal(getVarsFromFrame(vars[i], from, FALSE))[[1L]] into the file, so let's evaluate it: 
>>> .Internal(getVarsFromFrame(vars[i], from, FALSE))[[1L]]
function (x, y)
tensor(x, y, 2, 2)
<environment: namespace:tensor>
>>> typeof(.Internal(getVarsFromFrame(vars[i], from, FALSE))[[1L]])
[1] "closure"
>>> environment(.Internal(getVarsFromFrame(vars[i], from, FALSE))[[1L]])        # get the environment of a closure
<environment: namespace:tensor>
>>> envlist(environment(.Internal(getVarsFromFrame(vars[i], from, FALSE))[[1L]])) # environment as an envlist, more detailed output
$ %*t% 
function (x, y)
tensor(x, y, 2, 2)
<environment: namespace:tensor>
$ %t*% 
function (x, y)
tensor(x, y, 1, 1)
<environment: namespace:tensor>
$ %t*t% 
function (x, y)
tensor(x, y, 1, 2)
<environment: namespace:tensor>
$.__NAMESPACE__.
<environment: 0x55eb55d00be0>
$.__S3MethodsTable__.
<environment: 0x55eb55d166e8>
$.packageName
[1] "tensor"
$tensor
function (A, B, alongA = integer(0), alongB = integer(0))
 
[.. function definition ..]
 
<environment: namespace:tensor>
Nothing obviously containing paths so far, let's keep stepping and dumping the objects 
>>>
debug: assign(vars[i], key, envir = varenv)
>>>                                                                             # next iteration now
debug: key <- if (is.null(from)   is.environment(from)) lazyLoadDBinsertVariable(vars[i],
    from, datafile, ascii, compress, envhook) else lazyLoadDBinsertListElement(from,
    i, datafile, ascii, compress, envhook)
>>>
debug: lazyLoadDBinsertVariable(vars[i], from, datafile, ascii, compress,
    envhook)
>>> .Internal(getVarsFromFrame(vars[i], from, FALSE))[[1L]]
function (x, y)
tensor(x, y, 1, 1)
<environment: namespace:tensor>
>>>
[.. more stepping ..]
debug: lazyLoadDBinsertVariable(vars[i], from, datafile, ascii, compress,
    envhook)
>>> .Internal(getVarsFromFrame(vars[i], from, FALSE))[[1L]]
<environment: 0x55eb55d00be0>
>>> envlist(.Internal(getVarsFromFrame(vars[i], from, FALSE))[[1L]])
$S3methods
     [,1] [,2] [,3]
$dynlibs
NULL
$exports
<environment: 0x55eb55d0a4f8>
$imports
$imports$base
[1] TRUE
$lazydata
<environment: 0x55eb55d0a178>
attr(,"name")
[1] "lazydata:tensor"
$path
[1] "/mystupidbuildpath/r-tests/r-cran-tensor-1.5/debian/123/usr/lib/R/site-library/tensor"
$spec
    name  version
"tensor"    "1.5"
Oh, here we go. (In fact we could have got this earlier, it was part of the .__NAMESPACE__. key of the first environment that we examined.) Naturally, you'll sometimes miss stuff, and sometimes go down dead ends when doing something like this, perseverence is important. The names are also a hint, after grepping the R source code for names like "S3methods" and "lazydata" we see that src/library/base/R/namespace.R is the only one that contains all of them, then reading that code in more depth reveals this. After patching it, rebuilding r-base, installing this patched version, then running our tests again, we see that we've eliminated differences in the main rdb file but not the help rdb file. :( In fact, after some more time stepping through the loop repeatedly and printing out all the objects, nothing obvious shows up. Time to re-read the source code of makeLazyLoadDB. We notice that all the lazyLoadDBinsert* functions have a hook parameter that we haven't been stepping through because it's in a separate function. So let's do that. Since it's an inner function, one has to call debug() inside makeLazyLoadDB: 
$ DEBUG=1 ../r-mini-repro-test.sh r-cran-tensor-1.5 123
[..]
>>> debug(tools:::makeLazyLoadDB)
>>> tools:::.install_packages()
processing  . 
[..]
** preparing package for lazy loading
debugging in: makeLazyLoadDB(ns, dbbase, compress = compress)
debug:  
[.. source code of makeLazyLoadDB, not yet run ..]
 
>>> c
exiting from: makeLazyLoadDB(ns, dbbase, compress = compress)
** help
debugging in: makeLazyLoadDB(db, file.path(manOutDir, basename(outDir)))
debug:  
[.. source code of makeLazyLoadDB, not yet run ..]
 
[.. stepping through the function ..]
debug: envhook <- function(e)  
[.. function definition ..]
 
>>>                                                                             # step next, to define that function
[.. next statement, we don't care about it ..]
>>> debug(envhook)
[.. then keep stepping through makeLazyLoadDB ..]
>>>
debug: lazyLoadDBinsertVariable(vars[i], from, datafile, ascii, compress,
    envhook)
[.. then keep stepping through makeLazyLoadDB ..]
debug: lazyLoadDBinsertVariable(vars[i], from, datafile, ascii, compress,
    envhook)
[.. hook wasn't run for these previous iterations for some reason, but keep stepping ..]
[.. eventually we get ..]
debugging in: (function (e)
 
[.. function definition ..]
 )(<environment>)                                                               # "(..)(<environment>)" is R's name for that function (actually a closure)
debug:  
[.. source code, not yet run ..]
 
>>> e
/mystupidbuildpath/r-tests/r-cran-tensor-1.5/man/tensor.Rd
>>> typeof(e)
[1] "environment"
>>> envlist(e)
$Enc
[1] "UTF-8"
$encoding
[1] ""
$filename
[1] "/mystupidbuildpath/r-tests/r-cran-tensor-1.5/man/tensor.Rd"
$timestamp
[1] "2002-01-16 11:51:14 CET"
$wd
[1] "/mystupidbuildpath/r-tests/r-cran-tensor-1.5"
After doing the same thing as before, i.e. grepping the R source code for "Enc" "timestamp" etc, we spot a few different places that are setting the wd field. Then after some experimenting, we find that overwriting this in a different file (src/library/tools/R/parseRd.R) seems to work to get rid of this irreproducibility. (This may not be the best; our exact strategy is still being discussed with upstream, but at least we know which areas of the code are responsible.) Exercise for the reader to figure out how to get rid of the paths.rds difference. After doing the previous two, this one is fairly easy. These three issues allowed us to write a preliminary patch that enabled us to reproduce r-cran-tensor. 
$ ./r-mini-repro-test.sh r-cran-tensor-1.5 123
[..]
bca9de1351ab265475cf6f1af3f6aff49bde1d209e8fafb33954caa8d90f3462  ./debian/123/usr/lib/R/site-library/tensor/R/tensor.rdx
ac7fffeb751ffc2d13c5317704b19787a7078446d97d1c6f259c93f282e04cef  ./debian/123/usr/lib/R/site-library/tensor/R/tensor.rdb
d989603340d1e8f14c35551ef379956653bf3b6e7b1868b4bf9a66d85820998e  ./debian/123/usr/lib/R/site-library/tensor/help/tensor.rdx
30e543cda9f97c202dc9c0291933940dfe71b94d352f66d8ed6c035c1a722b41  ./debian/123/usr/lib/R/site-library/tensor/help/tensor.rdb
e9f78ba2cca39a6face1ed88fce4bb718d911d567d01db49bc00a3011720ac50  ./debian/123/usr/lib/R/site-library/tensor/help/paths.rds
$ ./r-mini-repro-test.sh r-cran-tensor-1.5 1234
[..]
bca9de1351ab265475cf6f1af3f6aff49bde1d209e8fafb33954caa8d90f3462  ./debian/1234/usr/lib/R/site-library/tensor/R/tensor.rdx
ac7fffeb751ffc2d13c5317704b19787a7078446d97d1c6f259c93f282e04cef  ./debian/1234/usr/lib/R/site-library/tensor/R/tensor.rdb
d989603340d1e8f14c35551ef379956653bf3b6e7b1868b4bf9a66d85820998e  ./debian/1234/usr/lib/R/site-library/tensor/help/tensor.rdx
30e543cda9f97c202dc9c0291933940dfe71b94d352f66d8ed6c035c1a722b41  ./debian/1234/usr/lib/R/site-library/tensor/help/tensor.rdb
e9f78ba2cca39a6face1ed88fce4bb718d911d567d01db49bc00a3011720ac50  ./debian/1234/usr/lib/R/site-library/tensor/help/paths.rds
$ ./r-mini-repro-test.sh r-cran-tensor 1234
[..]
bca9de1351ab265475cf6f1af3f6aff49bde1d209e8fafb33954caa8d90f3462  ./debian/1234/usr/lib/R/site-library/tensor/R/tensor.rdx
ac7fffeb751ffc2d13c5317704b19787a7078446d97d1c6f259c93f282e04cef  ./debian/1234/usr/lib/R/site-library/tensor/R/tensor.rdb
d989603340d1e8f14c35551ef379956653bf3b6e7b1868b4bf9a66d85820998e  ./debian/1234/usr/lib/R/site-library/tensor/help/tensor.rdx
30e543cda9f97c202dc9c0291933940dfe71b94d352f66d8ed6c035c1a722b41  ./debian/1234/usr/lib/R/site-library/tensor/help/tensor.rdb
e9f78ba2cca39a6face1ed88fce4bb718d911d567d01db49bc00a3011720ac50  ./debian/1234/usr/lib/R/site-library/tensor/help/paths.rds
Testing the patch Of course r-cran-tensor is just 1 package, so then we tested this patch on all 478 tagged R packages. Before the patch, all of these were unreproducible. Now 463/478 are reproducible, hurray! The first version of the patch made 2 packages FTBFS ("fail to build from source"), r-bioc-biobase and r-cran-shinybs; this was fixed in a subsequent version of the patch. (Some other packages, r-cran-randomfields r-cran-randomfieldsutils, FTBFS even with an unpatched r-base, due to differences between 3.3.3 and 3.4.0 (#861333), and not because of our patch.) Given the overwhelming proportion of packages that did reproduce, the other 14 packages that are still unreproducible, are quite probably due to issues in those specific packages. Only 2 of these are certainly due to build-path differences: r-cran-runit r-cran-rinside. This is because, we currently see they are reproducible in Debian testing but not Debian unstable. This is slightly misleading, the real reason is not testing vs unstable directly; rather the fact that we (at the time of writing) use the same build path when trying to reproduce Debian testing packages, and different paths for unstable. The other 12 are unreproducible in both testing and unstable, so it either suffers from a non-build-path issue as well as a build-path issue, or only a non-build-path issue. So, further investigation of each package is needed. This is where you can help. :) Debugging R packages To investigate why (for example) r-cran-runit doesn't reproduce despite our patches, we just do the above process again: 
$ DEBUG=1 ../r-mini-repro-test.sh r-cran-runit
[..]
>>> debug(tools:::makeLazyLoadDB)
>>> tools:::.install_packages()
processing  . 
a directory
* build_help_types=
* DBG: 'R CMD INSTALL' now doing do_install()
* created lock directory  /mystupidbuildpath/r-tests/r-cran-runit/debian/r-cran-runit/usr/lib/R/site-library/00LOCK-r-cran-runit 
* installing *source* package  RUnit  ...
** package  RUnit  successfully unpacked and MD5 sums checked
** backing up earlier installation
** R
** inst
** preparing package for lazy loading
debugging in: makeLazyLoadDB(ns, dbbase, compress = compress)
debug:  
[.. source code of makeLazyLoadDB ..]
We know our irreproducibility is with the help files and not the main package, so c to skip this part: 
>>> c
exiting from: makeLazyLoadDB(ns, dbbase, compress = compress)
** help
debugging in: makeLazyLoadDB(db, file.path(manOutDir, basename(outDir)))
debug:  
[.. source code of makeLazyLoadDB ..]
Examine the "from" object, grep the output for our builddir 
>>> from
$ RUnit-internal 
[..]
$ RUnit-intro 
[..]
[.. etc ..]
[..]
$checkFuncs
\title RUnit check [..]
[..]
  For a simple example see the provided test cases in
  /mystupidbuildpath/r-tests/r-cran-runit/debian/r-cran-runit/usr/lib/R/site-library/RUnit/examples/runitVirtualClassTest.r.
[..]
Found it, now we quit, patch the source code, and try again. 
>>> Q
[..]
exit code 1
[.. hack hack hack ..]
$ ../r-mini-repro-test.sh r-cran-runit-0.4.31/ 123
[..]
01347305351a45bb88dab909c56942498e97f5edbe583cf6ab1327138914e315  ./debian/123/usr/lib/R/site-library/RUnit/R/RUnit.rdx
ecde40bdfac5b070acc30a0d5889d9d7f65418844d14312aceff6af5b9d0bf7a  ./debian/123/usr/lib/R/site-library/RUnit/R/RUnit.rdb
182d38ac5adc160eb9edd40d5bd7e27b95b756b8f14cee7d912ba4745ae81bdc  ./debian/123/usr/lib/R/site-library/RUnit/help/RUnit.rdx
f8a9f22acc9f937cb47a92f3979d619e83a791ad734796c2fcd9d0a34e1f11ca  ./debian/123/usr/lib/R/site-library/RUnit/help/RUnit.rdb
5c37672fdc1f325b8a3846f4eafd2624b246ec70ce01d197134fa1b97a9ee849  ./debian/123/usr/lib/R/site-library/RUnit/help/paths.rds
$ ../r-mini-repro-test.sh r-cran-runit-0.4.31/ 1234
[..]
01347305351a45bb88dab909c56942498e97f5edbe583cf6ab1327138914e315  ./debian/1234/usr/lib/R/site-library/RUnit/R/RUnit.rdx
ecde40bdfac5b070acc30a0d5889d9d7f65418844d14312aceff6af5b9d0bf7a  ./debian/1234/usr/lib/R/site-library/RUnit/R/RUnit.rdb
182d38ac5adc160eb9edd40d5bd7e27b95b756b8f14cee7d912ba4745ae81bdc  ./debian/1234/usr/lib/R/site-library/RUnit/help/RUnit.rdx
f8a9f22acc9f937cb47a92f3979d619e83a791ad734796c2fcd9d0a34e1f11ca  ./debian/1234/usr/lib/R/site-library/RUnit/help/RUnit.rdb
5c37672fdc1f325b8a3846f4eafd2624b246ec70ce01d197134fa1b97a9ee849  ./debian/1234/usr/lib/R/site-library/RUnit/help/paths.rds
Works! All finished. :) If yours doesn't, just repeat the above loop; hopefully eventually you'll get there.

Axel Beckert: Upcoming Hacker/FLOSS Events in Switzerland: Debian BSP, LPD, Hackerfunk 10th Anniversary, ZeTeCo

There are quite some events and dates ahead for hackers, makers, debianers and hackerfunk listeners: Crowdfunding for ZeTeCo Camp in July Ends in Two Days! You might have heard of the ZeTeCo Camp near Schaffhausen in July. If you want ot come or at least support that event, please contribute to their crowdfunding campaign. They have more than 90% of their goal funded and there s less only about two days left to reach their funding goal. If it doesn t get funded in time, the event may be be on a knife edge. Debian Bug Squashing Party in Zurich this Weekend One week before the More-than-a-BSP at the Mozilla office in Paris there will also be a Debian Bug Squashing Party (BSP) in Z rich at the CCCZH Hackerspace R schtibach . We ll start on Friday, the 5th of May 2017 in the late afternoon, probably around 4pm or 5pm, and will end on Sunday, the 7th of May 2017 also in the late afternoon. Please add yourself to the according section on the BSP s wiki page if you want to join us to squash the hopefully not that many left over bugs in testing. Unfortunately we didn t notice two date clashes when we set the date for the BSP during the Annual General Meeting (AGM) of the Debian.ch Association earlier this year: Linux Presentation Day this Saturday, 6th of May 2017 On the one hand there will be the Swiss Edition of the Linux Presentation Day (LPD) on Saturday, the 6th of May 2017 including the LPD in Z rich. The latter will take place in the same building as the BSP, just on a different floor: The BSP will be at floor B3 in the CCCZH Hackerspace and the LPD will be on the ground floor at Revamp-IT in the former ZKB foyer. 10 Years Hackerfunk: Show on 6th, Party on 13th of May 2017 And on the other hand, Venty s and my (German dialect) radio show and podcast Hackerfunk will have it s 10th anniversary show also on that Saturday. So I ll vanish from BSP for a few hours on Saturday evening for broadcasting this very special Hackerfunk episode on Radio Radius. But since Venty and me didn t want to make yet another big event at R schtibach on the same weekend, we ll do the Hackerfunk 10th Anniversary Party one weekend later on Saturday the 13th of May 2017 also at the CCCZH Hackerspace R schtibach . A separate announcement on https://www.hackerfunk.ch/ (also in the RSS feed there) will follow.

Rapha&#235;l Hertzog: My Free Software Activities in April 2017

My monthly report covers a large part of what I have been doing in the free software world. I write it for my donors (thanks to them!) but also for the wider Debian community because it can give ideas to newcomers and it s one of the best ways to find volunteers to work with me on projects that matter to me. Debian LTS I was allocated 10 hours to work on security updates for Debian 7 Wheezy and had 1.5 hours remaining from March. During this time I did the following: Kali and pkg-security I updated the britney instance that we are using in Kali and spotted two small documentation mistakes that I fixed. We had a long-standing bug in Kali where extensions would stay visible on the lock screen. It was hard to reproduce and this month we finally managed to nail down the conditions required to reproduce it. It turns out that EasyScreenCast was the culprit. We paid Emilio Pozuelo Monfort to work on a patch and he fixed the problem in EasyScreenCast and also in gnome-shell, as a buggy extension should not have resulted in this behavior. I responded to multiple queries of new contributors in the pkg-security team. The team is rather active and it would be great if we could have a few more Debian developers to help review and sponsor the work our enthusiastic new members. Thanks See you next month for a new summary of my activities. Hopefully, I will be more active again between kids vacations, French elections and Zelda Breadth of the Wild, I got very much distracted from Debian last month.

One comment Liked this article? Click here. My blog is Flattr-enabled.

2 May 2017

Clint Adams: Ezra's legacy

Good, he said. I d like to see that white females number go down though. Let s work on that. We agree on something, she said. Do white women look down on you for being Mexican? They look down on everyone for being anything, he snorted. How do you feel about progessive female SJWs who claim to defend your interests as an ethnic minority while at the same time secretly hating you for being the scum of the earth? she asked. White saviorism is pretty obnoxious, he replied, but so is calling people SJWs because they give a fuck about other human beings. Do you think they truly give a fuck? she asked. I m sure they do, to an extent, but like anyone they are self-serving, he said. I just don t get going out of your way to further judge them. They do judge people all the time, she said. Usually it s people who are problematic, he said, which I m okay with, though sadly, as with anything, people go too far, like policing language or making an issue of something that doesn t need to be one. That s the very heart of activism as a surrogate activity: faking concern about other people s problems in order to promote themselves in some way, she explained. For example SJW is way better than Meninist, he continued. Pick your poison, I guess. What s a Meninist? she asked. Red-pill assholes who think the world is out to subjugate men in favor of women, he explained. Those guys always talk about lifting, she laughed. They lift their way through life and lift problems away. To be fair you re pretty apathetic about people s concerns or you don t seem to care aobut other people s problems, he observed. Maybe I have enough problems of my own, she said, problems that have a social dimension, but I just bother trying to fix them in as much as they affect me. That s fine, he said, some just want to care about problems affecting others. To each their own. I don t think it s inherently selfish. For example, she said, I have problems with my Emacs setup. Emacs always seems more trouble than what it s worth, he said, but so do you.
Posted on 2017-05-02
Tags: umismu

Markus Koschany: My Free Software Activities in April 2017

Welcome to gambaru.de. Here is my monthly report that covers what I have been doing for Debian. If you re interested in Java, Games and LTS topics, this might be interesting for you. Debian Games Debian Java Debian LTS This was my fourteenth month as a paid contributor and I have been paid to work 23,75 hours on Debian LTS, a project started by Rapha l Hertzog. In that time I did the following: Misc Thanks for reading and see you next time.

Urvika Gola: Debian goes to Hackathon!

Recently I had the opportunity to be a part of Ultrahack as a mentor which was held in Helsinki, Finland.
Ultrahack is a combination of hackathon and startup accelerators.
As a mentor, my role was to ensure that each team have best possible chances of fulfilling the evaluation criteria for the contest. I also helped teams with the development and pitching. It was a very exciting place to brainstorm life changing ideas and convert those ideas into working model. I met so many amazing developers who were building cool stuff. There were a few open source developers and student open source developers like me!
Being a Debian contributor, I spread the what Debian is all about. What makes it the best linux distribution. I talked to students about various programmes like GSoC and Outreachy, Debian participates as a mentoring organisation. I also described my role as a GSoC student under Debian and the free-RTC project I worked under. Many female developers were interested in the Outreachy programme, I described the projects that Debian has currently under the Outreachy programme. During the hackathon period, I talked to people about the upcoming annual DebConf which takes place. I informed them that they can still apply as a speaker or for diversity bursaries and about the logo-making competition. Thank you Debian for always supporting me!

Russ Allbery: Review: Ninefox Gambit

Review: Ninefox Gambit, by Yoon Ha Lee
Series: The Machineries of Empire #1
Publisher: Solaris
Copyright: 2016
ISBN: 1-84997-992-8
Format: Kindle
Pages: 384
Charis is a Kel, which means that she's a soldier of the Hexarchate. A captain, to be precise: Captain Kel Charis of Heron Company, commanding infantry forces to stamp out heresy. The heresy she's stamping out at the start of the book are the Eels, or the Society of the Flourish as they call themselves, and they're strong enough that they can also command heretical technology. Charis manages to win through, but only because she has enough flexibility and quick thinking to adjust to the presence of a heresy and reach beyond the Lexicon Primary, improvising formations on the spot to adjust for the affects of the rebel calendar. Military victory is prized among the Kel, but stepping outside the bounds of Doctrine to achieve it is not. Charis is not particularly surprised when her company is disbanded for re-education after the battle. She is very surprised when she personally is tapped to offer a solution to a far greater attack on the Hexarchate. I first encountered Yoon Ha Lee's fiction in the short story "The Unstrung Zither", was blown away by the creativity and delightfully weird twist on science fiction war, and have been following his writing ever since. Most of it is short fiction, though, and I'm not much of a short fiction reader, so there haven't been many reviews. Ninefox Gambit is his first, and much-anticipated, full-length novel. It's probably not too surprising for someone from the generation that grew up with Star Wars, but I have a soft spot in my heart for magitech. Hard science fiction has its merits, as does the softer sort that takes standard, if impossible, genre tropes for granted. But something about a far-future, space-faring society based on magic that straddles the rules of technology, physics, affinities, and beliefs calls to that part of me that spent hours thinking about the nature of the Force. It has to be good magitech, though: something odd and different but well-thought-out, full of implications and twisty consequences that reshape society and that inspire a whole new type of engineering and science. Magic that's not physics as we know it, but that's knowable, researchable, and something that a society can reshape itself around. This is the good magitech. In the world of the Hexarchate, calendrical systems are more than just a mutual agreement for conveying time. They order and structure the laws of the universe as much as they structure society. What technological devices, and what weapons, are possible is influenced by the calendar in observance, which in turn is based on what calendar people believe in and follow. Close adherence to a calendrical regime enables exotics: weapons with incredibly powerful and often horrific effects, such as the threshold winnower that plays a repeated, nightmarish role in this story. Invariant weapons, ones that will work in any calendar system, are much weaker. The Hexarchate is called that because it is a society formed by six factions, who divide the work of ruling its scattered planets according to the expertise and tendencies of each faction. Together, they impose the high calendar, and maintain it against heresies with an iron fist lest their power be undermined or transformed and their exotics fail to function. The Kel is their military faction, a key component of that fist, and their specialty is formations: specific arrangements of humans or ships that channel the power of the calendar to defend against or attack with exotics. Formations have to be held exactly to hold their power and yet have to be flexible enough to change based on fast-changing battlefield conditions. To assist in this, Kel are programmed with formation instinct: psychological conditioning that helps them obediently take and hold formations. And, not coincidentally, offer nearly absolute obedience to their chain of command. I just finished reading another book that attempted to use math as a key component of its world-building. I think Lee was far more successful. The math here is realistic for its purpose, obviously necessary given the formation structures built into the world's physics, and a lovely nod to the importance of calendars. A single calendar might involve only simple arithmetic, but the formation and technological implications of a calendar, let alone the fuzzy boundaries between two calendars each partially in force, would naturally require tricky advanced mathematics to work out. For someone in Charis's position, mathematical training is a rare but vitally important tactical advantage. As you might have guessed from the amount that I'm talking about combat, Ninefox Gambit is military SF. Charis is a military officer, and a comfortable majority of this book is combat of one kind or another. That's not normally my thing, and I did wish there was a bit more non-military social development. But my normal problem with military SF is that I lack the interest in battlefield tactics and strategy to stay fully engaged by description of battle after battle. Ninefox Gambit is the story of Charis attempting to retake a stronghold of the Hexarchate that's fallen to heretical forces, but Lee adds an important twist that does keep me engaged: Jedao. General Shuos Jedao was the greatest general the Hexarchate had ever seen. He never lost a battle. The only catch is that, in the middle of a highly successful campaign against heretics, he went mad, slaughtering both the heretics and his own troops with a horrific weapon while simultaneously murdering all of his command staff. He's much too dangerous and insane to leave alive, but he was also much too valuable and skillful to lose as a weapon, so for the subsequent centuries he's been kept in a threshold state, an undead ghost. A ghost that the Hexarchate can put into Charis's head, a constant advisor as she's placed in charge of the swarm sent to retake the Fortress of Scattered Needles. A brilliant tactician, sociopath, and mass murderer whose advice can never be trusted. The heart of Ninefox Gambit isn't the combat. It's the interplay of power, analysis, and guesswork under the combat, as Charis attempts to use Jedao's brilliance while not losing her own sense of identity or letting him mess too badly with her head. At the start, she's way out of her depth. But she's thoughtful, careful, has a strong internal sense of identity, and learns fast. And the story of Jedao's past is accurate, but incomplete. For those who are familiar with the often-ornate language and prose style of Yoon Ha Lee's short fiction and who are worried it wouldn't hold up at longer length, note that his style here is much different. There are a few touches of ornate description, but most of the book is written in a straightforward and easily-understandable narrative style. Thankfully, because the layers of tactical thrust and counter-thrust are complicated enough that I would have lost them entirely beneath too-complex prose. There's a lot of brutal death in this book. I got a bit tired of both that and the tactical maneuvering, although that's less the fault of the book and more my own mild antipathy towards military SF. But the unique universe background held my interest long enough to become intrigued by Charis's slow but determined probing at Jedao's secrets and the politics of the Hexarchate. I still would have preferred the story to have a somewhat lower body count, but as long as one can read past some gore, there's plenty here to appeal to someone who normally gives military SF a pass. I think its biggest drawback is that, although it has a narrative arc that comes to a clear conclusion, Ninefox Gambit raises a lot of important questions about its world and mostly doesn't answer them. There are more books coming, and I hope they contain more definitive answers. Followed by Raven Stratagem. Rating: 7 out of 10

1 May 2017

Matthew Garrett: Intel's remote AMT vulnerablity

Intel just announced a vulnerability in their Active Management Technology stack. Here's what we know so far.
Background Intel chipsets for some years have included a Management Engine, a small microprocessor that runs independently of the main CPU and operating system. Various pieces of software run on the ME, ranging from code to handle media DRM to an implementation of a TPM. AMT is another piece of software running on the ME, albeit one that takes advantage of a wide range of ME features.
Active Management TechnologyAMT is intended to provide IT departments with a means to manage client systems. When AMT is enabled, any packets sent to the machine's wired network port on port 16992 or 16993 will be redirected to the ME and passed on to AMT - the OS never sees these packets. AMT provides a web UI that allows you to do things like reboot a machine, provide remote install media or even (if the OS is configured appropriately) get a remote console. Access to AMT requires a password - the implication of this vulnerability is that that password can be bypassed.
Remote managementAMT has two types of remote console: emulated serial and full graphical. The emulated serial console requires only that the operating system run a console on that serial port, while the graphical environment requires drivers on the OS side requires that the OS set a compatible video mode but is also otherwise OS-independent[2]. However, an attacker who enables emulated serial support may be able to use that to configure grub to enable serial console. Remote graphical console seems to be problematic under Linux but some people claim to have it working, so an attacker would be able to interact with your graphical console as if you were physically present. Yes, this is terrifying.
Remote mediaAMT supports providing an ISO remotely. In older versions of AMT (before 11.0) this was in the form of an emulated IDE controller. In 11.0 and later, this takes the form of an emulated USB device. The nice thing about the latter is that any image provided that way will probably be automounted if there's a logged in user, which probably means it's possible to use a malformed filesystem to get arbitrary code execution in the kernel. Fun!

The other part of the remote media is that systems will happily boot off it. An attacker can reboot a system into their own OS and examine drive contents at their leisure. This doesn't let them bypass disk encryption in a straightforward way[1], so you should probably enable that.
How bad is thisThat depends. Unless you've explicitly enabled AMT at any point, you're probably fine. The drivers that allow local users to provision the system would require administrative rights to install, so as long as you don't have them installed then the only local users who can do anything are the ones who are admins anyway. If you do have it enabled, though
How do I know if I have it enabled?Yeah this is way more annoying than it should be. First of all, does your system even support AMT? AMT requires a few things:

1) A supported CPU
2) A supported chipset
3) Supported network hardware
4) The ME firmware to contain the AMT firmware

Merely having a "vPRO" CPU and chipset isn't sufficient - your system vendor also needs to have licensed the AMT code. Under Linux, if lspci doesn't show a communication controller with "MEI" or "HECI" in the description, AMT isn't running and you're safe. If it does show an MEI controller, that still doesn't mean you're vulnerable - AMT may still not be provisioned. If you reboot you should see a brief firmware splash mentioning the ME. Hitting ctrl+p at this point should get you into a menu which should let you disable AMT.
How about over Wifi?Turning on AMT doesn't automatically turn it on for wifi. AMT will also only connect itself to networks it's been explicitly told about. Where things get more confusing is that once the OS is running, responsibility for wifi is switched from the ME to the OS and it forwards packets to AMT. I haven't been able to find good documentation on whether having AMT enabled for wifi results in the OS forwarding packets to AMT on all wifi networks or only ones that are explicitly configured.
What do we not know?We have zero information about the vulnerability, other than that it allows unauthenticated access to AMT. One big thing that's not clear at the moment is whether this affects all AMT setups, setups that are in Small Business Mode, or setups that are in Enterprise Mode. If the latter, the impact on individual end-users will be basically zero - Enterprise Mode involves a bunch of effort to configure and nobody's doing that for their home systems. If it affects all systems, or just systems in Small Business Mode, things are likely to be worse.
We now know that the vulnerability exists in all configurations.
What should I do?Make sure AMT is disabled. If it's your own computer, you should then have nothing else to worry about. If you're a Windows admin with untrusted users, you should also disable or uninstall LMS by following these instructions.
Does this mean every Intel system built since 2008 can be taken over by hackers?No. Most Intel systems don't ship with AMT. Most Intel systems with AMT don't have it turned on.
Does this allow persistent compromise of the system?Not in any novel way. An attacker could disable Secure Boot and install a backdoored bootloader, just as they could with physical access.
But isn't the ME a giant backdoor with arbitrary access to RAM?Yes, but there's no indication that this vulnerability allows execution of arbitrary code on the ME - it looks like it's just (ha ha) an authentication bypass for AMT.
Is this a big deal anyway?Yes. Fixing this requires a system firmware update in order to provide new ME firmware (including an updated copy of the AMT code). Many of the affected machines are no longer receiving firmware updates from their manufacturers, and so will probably never get a fix. Anyone who ever enables AMT on one of these devices will be vulnerable. That's ignoring the fact that firmware updates are rarely flagged as security critical (they don't generally come via Windows update), so even when updates are made available, users probably won't know about them or install them.
Avoiding this kind of thing in futureUsers ought to have full control over what's running on their systems, including the ME. If a vendor is no longer providing updates then it should at least be possible for a sufficiently desperate user to pay someone else to do a firmware build with the appropriate fixes. Leaving firmware updates at the whims of hardware manufacturers who will only support systems for a fraction of their useful lifespan is inevitably going to end badly.
How certain are you about any of this?Not hugely - the quality of public documentation on AMT isn't wonderful, and while I've spent some time playing with it (and related technologies) I'm not an expert. If anything above seems inaccurate, let me know and I'll fix it.

[1] Eh well. They could reboot into their own OS, modify your initramfs (because that's not signed even if you're using UEFI Secure Boot) such that it writes a copy of your disk passphrase to /boot before unlocking it, wait for you to type in your passphrase, reboot again and gain access. Sealing the encryption key to the TPM would avoid this.

[2] Updated after this comment - I thought I'd fixed this before publishing but left that claim in by accident.

(Updated to add the section on wifi)

(Updated to typo replace LSM with LMS)

(Updated to indicate that the vulnerability affects all configurations)

comment count unavailable comments

Thorsten Alteholz: My Debian Activities in April 2017

FTP assistant This month I marked 72 packages for accept and sent one email to a maintainer asking questions. The number of rejections went down to 15. I would name that a good level again. Debian LTS This was my thirty-fourth month that I did some work for the Debian LTS initiative, started by Raphael Hertzog at Freexian. As others reduced their workload for this month, my all in all workload has been 23.75h. During that time I did uploads of In addition I had one week of frontdesk duties. I also started to work on icu and bind9. The patches for icu applied fine but the corresponding test did not work but stopped somewhere in the middle!? I am open for any suggestions why this could happen. Other stuff This has been a busy LTS month, so I only created node-tunein and adopted smstools as DOPOM.

30 April 2017

Paul Wise: FLOSS Activities April 2017

Changes

Issues

Review

Administration
  • Debian systems: quiet a logrotate warning, investigate issue with DNSSEC and alioth, deploy fix on our first stretch buildd, restore alioth git repo after history rewrite, investigate iptables segfaults on buildd and investigate time issues on a NAS
  • Debian derivatives census: delete patches over 5 MiB, re-enable the service
  • Debian wiki: investigate some 403 errors, fix alioth KGB config, deploy theme changes, close a bogus bug report, ping 1 user with bouncing email, whitelist 9 email addresses and whitelist 2 domains
  • Debian QA: deploy my changes
  • Debian mentors: security upgrades and service restarts
  • Openmoko: debug mailing list issue, security upgrades and reboots

Communication
  • Invite Wazo to the Debian derivatives census
  • Welcome ubilinux, Wazo and Roopa Prabhu (of Cumulus Linux) to the Debian derivatives census
  • Discuss HP/ProLiant wiki page with HPE folks
  • Inform git history rewriter about the git mailmap feature

Sponsors The libconfig-crontab-perl backports and pyvmomi issue were sponsored by my employer. All other work was done on a volunteer basis.

Chris Lamb: Free software activities in April 2017

Here is my monthly update covering what I have been doing in the free software world (previous month):
Reproducible builds

Whilst anyone can inspect the source code of free software for malicious flaws, most software is distributed pre-compiled to end users. The motivation behind the Reproducible Builds effort is to permit verification that no flaws have been introduced either maliciously or accidentally during this compilation process by promising identical results are always generated from a given source, thus allowing multiple third-parties to come to a consensus on whether a build was compromised. I have generously been awarded a grant from the Core Infrastructure Initiative to fund my work in this area. This month I:
I also made the following changes to diffoscope, our recursive and content-aware diff utility used to locate and diagnose reproducibility issues:
  • New features:
    • Add support for comparing Ogg Vorbis files. (0436f9b)
  • Bug fixes:
    • Prevent a traceback when using --new-file with containers. (#861286)
    • Don't crash on invalid archives; print a useful error instead. (#833697).
    • Don't print error output from bzip2 call. (21180c4)
  • Cleanups:
    • Prevent abstraction-level violations by defining visual diff support on Presenter classes. (7b68309)
    • Show Debian packages installed in test output. (c86a9e1)


Debian
Debian LTS

This month I have been paid to work 18 hours on Debian Long Term Support (LTS). In that time I did the following:
  • "Frontdesk" duties, triaging CVEs, etc.
  • Issued DLA 882-1 for the tryton-server general application platform to fix a path suffix injection attack.
  • Issued DLA 883-1 for curl preventing a buffer read overrun vulnerability.
  • Issued DLA 884-1 for collectd (a statistics collection daemon) to close a potential infinite loop vulnerability.
  • Issued DLA 885-1 for the python-django web development framework patching two open redirect & XSS attack issues.
  • Issued DLA 890-1 for ming, a library to create Flash files, closing multiple heap-based buffer overflows.
  • Issued DLA 892-1 and DLA 891-1 for the libnl3/libnl Netlink protocol libraries, fixing integer overflow issues which could have allowed arbitrary code execution.
Uploads
  • redis (4:4.0-rc3-1) New upstream RC release.
  • adminer:
    • 4.3.0-2 Fix debian/watch file.
    • 4.3.1-1 New upstream release.
  • bfs:
    • 1.0-1 Initial release.
    • 1.0-2 Drop fstype tests as they rely on /etc/mtab being available. (#861471)
  • python-django:
    • 1:1.10.7-1 New upstream security release.
    • 1:1.11-1 New upstream stable release to experimental.

I sponsored the following uploads: I also performed the following QA uploads:
  • gtkglext (1.2.0-7) Correct installation location of gdkglext-config.h after "Multi-Archification" in 1.2.0-5. (#860007)
Finally, I made the following non-maintainer uploads (NMUs):
  • python-formencode (1.3.0-2) Don't ship files in /usr/lib/python 2.7,3 /dist-packages/docs. (#860146)
  • django-assets (0.12-2) Patch pytest plugin to check whether we are running in a Django context, otherwise we can break unrelated testsuites. (#859916)
FTP Team

As a Debian FTP assistant I ACCEPTed 155 packages: aiohttp-cors, bear, colorize, erlang-p1-xmpp, fenrir, firejail, fizmo-console, flask-ldapconn, flask-socketio, fontmanager.app, fonts-blankenburg, fortune-zh, fw4spl, fzy, gajim-antispam, gdal, getdns, gfal2, gmime, golang-github-go-macaron-captcha, golang-github-go-macaron-i18n, golang-github-gogits-chardet, golang-github-gopherjs-gopherjs, golang-github-jroimartin-gocui, golang-github-lunny-nodb, golang-github-markbates-goth, golang-github-neowaylabs-wabbit, golang-github-pkg-xattr, golang-github-siddontang-goredis, golang-github-unknwon-cae, golang-github-unknwon-i18n, golang-github-unknwon-paginater, grpc, grr-client-templates, gst-omx, hddemux, highwayhash, icedove, indexed-gzip, jawn, khal, kytos-utils, libbloom, libdrilbo, libhtml-gumbo-perl, libmonospaceif, libpsortb, libundead, llvm-toolchain-4.0, minetest-mod-homedecor, mini-buildd, mrboom, mumps, nnn, node-anymatch, node-asn1.js, node-assert-plus, node-binary-extensions, node-bn.js, node-boom, node-brfs, node-browser-resolve, node-browserify-des, node-browserify-zlib, node-cipher-base, node-console-browserify, node-constants-browserify, node-delegates, node-diffie-hellman, node-errno, node-falafel, node-hash-base, node-hash-test-vectors, node-hash.js, node-hmac-drbg, node-https-browserify, node-jsbn, node-json-loader, node-json-schema, node-loader-runner, node-miller-rabin, node-minimalistic-crypto-utils, node-p-limit, node-prr, node-sha.js, node-sntp, node-static-module, node-tapable, node-tough-cookie, node-tunein, node-umd, open-infrastructure-storage-tools, opensvc, openvas, pgaudit, php-cassandra, protracker, pygame, pypng, python-ase, python-bip32utils, python-ltfatpy, python-pyqrcode, python-rpaths, python-statistics, python-xarray, qtcharts-opensource-src, r-cran-cellranger, r-cran-lexrankr, r-cran-pwt9, r-cran-rematch, r-cran-shinyjs, r-cran-snowballc, ruby-ddplugin, ruby-google-protobuf, ruby-rack-proxy, ruby-rails-assets-underscore, rustc, sbt, sbt-launcher-interface, sbt-serialization, sbt-template-resolver, scopt, seqsero, shim-signed, sniproxy, sortedcollections, starjava-array, starjava-connect, starjava-datanode, starjava-fits, starjava-registry, starjava-table, starjava-task, starjava-topcat, starjava-ttools, starjava-util, starjava-vo, starjava-votable, switcheroo-control, systemd, tilix, tslib, tt-rss-notifier-chrome, u-boot, unittest++, vc, vim-ledger, vis, wesnoth-1.13, wolfssl, wuzz, xandikos, xtensor-python & xwallpaper. I additionally filed 14 RC bugs against packages that had incomplete debian/copyright files against getdns, gfal2, grpc, mrboom, mumps, opensvc, python-ase, sniproxy, starjava-topcat, starjava-ttools, unittest++, wolfssl, xandikos & xtensor-python.

29 April 2017

Shirish Agarwal: India and the Agricultural Economy

Indian farmer ploughing a field in traditional style - Wikimedia commons I was in two minds when I read Ritesh s blog post about the Indian Economy. I was angry with Ritesh as he seemed to selectively take facts and present it rather than taking it whole. Even if he had searched even a little bit, he would have got much more better material and everybody would have been the gainer. I have to also admit, I feel very much like a hypocrite as I have never slaved in a farm so my understanding and conclusions are a mix of media and limited interaction with farmers some years ago. There is also lots of local customs and politics that come into the picture and it s not as straight-forward as Ritesh thinks. What he has failed to share/account for is the far worse bad and stressed debts for the industry so just saying farm loan waivers are bad without sharing any of the context makes it seem much more worse. This is when our Current Chief Economic Adviser states about loan waivers to corporates You need to be able to forgive those debts because this is how capitalism works. People make mistakes, those have to be forgiven to some extent Let me start though with words from a book I read sometime back On a peasant uprising erupted in . The farmers were angry with high interest rates, high taxes, high inflation and low-government prices for their crops. The system had let them into debt, and debt had meant foreclosure and loss of their fields to the land barons. I intentionally have made a fill in the blanks as both the dates and places was true in India 100 years back and even today, the only difference between the two is the absence of taxes. Many people would think I m talking about Champaran whose tale while well-known in India is sadly a stub-class article in wikipedia with quite a few citation needed tags as well but is also true today as will be seen below. Interestingly, there is/was a remark by some unknown person who said gora sahab gays, bhura sahab aaya meaning the white officer has gone, in his place the brown officer has come. The evidence of this is very much in the Telegraph Act and the story about its usage and its place in Indian politics Surprisingly, sadly and coincidentally, the quote minus the dates and place didn t happen in India but also in Cambodia. The above quote has been taken from for the sake of all living things by John M. Del Vecchio. The quote itself appears in the first 10 odd pages (historical summation) of the somewhat 1200 odd pages book. I actually got an old edition which tops out at 900 pages so probably some more updated input/news isn t available but it still packs a powerful wallop. I want to dedicate a separate blog post for the book itself so will not say more on that book and what it shares. Sadly and coincidentally, there were news reports yesterday itself of farmers agitating for better prices just yesterday. Some of the interesting work if you want to understand the farmer s indebtedness is to study the Income, Expenditure, Productive Assets and Indebtedness of Agricultural Households in India done by NSSO. Again, one does not need to read the whole report, there were some of the analysis shared by the Hindu here and here. This was also echoed by Logical Indian It really boggles the mind to know than an average farming household earns around INR 200 per day. Even if you take a family of four people that comes to INR 50/- per person. Most rural joint families at the very least have 3-4 kids at the very least. Sadly Agricultural incomes do not keep sync even with the inflation index as there is no fair minimum age and wage for the Indian farmer, the concept does not exist for her(im) . Just me and mum going onto a restaurant and having one dish each easily can run anywhere between INR 200~250 easily . Cooking in the house is the same if you add/input the labor (which is usually not calculated) used to make lunch/dinner. There was the idea that contract farming might be a solution but even that was corrupted by Multi-national companies such as Pepsi and others that the Government is showing movement to have a model contracting law . There are loads of stories on downtoearth magazine which deals with the above and all sorts of issues the farmer faces. I should talk about Maharashtra and even here there was/has been an irrigation scam with figures given from 35k to 70k crores or 350 to 700 billion INR . The latest finding by the PAC has been shared here. I will cut the blog post short as I find the whole thing personally very depressing. As far as local customs go this was from one of the farmers interaction some years ago where me and some other friends had gone across a village and came to know that all of them grew the same crop with some minor variances. When asked why is it so, while many said its fate, one of the elderly gentleman shared an experience where a farmer had planted some other thing. The gentleman prospered while the other villagers were suffering from glut of whatever they produced. Knowing he prospered, the other villagers damaged his crops and all sorts of unlucky things started happening for the farmer. In the end he realized his best bet is to follow the ways of the other villagers, at least they would be in peace. What I have shared isn t either unique or even unknown, even Toronto Star of Canada reported on the issue some years ago. At the end of it all, the story is one of no education and limited skill-set and I don t see it changing any time soon. There are some who are earning big figures, but majority of the farmers will always be in the red
Filed under: Miscellenous Tagged: #Agricultal Economy, #Champaran, #Contract farming, #Corruption, #farmer suicide, #irrigation scam, #Loan waivers, #planet-debian, poverty

Antoine Beaupr : My free software activities, April 2017

Debian Long Term Support (LTS) This is my monthly Debian LTS report. My time this month was spent working on various hairy security issues, most notably XBMC (now known as Kodi) and yaml-cpp.

Kodi directory transversal I started by looking in CVE-2017-5982, a "directory traversal" vulnerability in XBMC (now known as Kodi) which is a technical term for "allow attackers to read any world-readable file on your computer from the network". It's a serious vulnerability which has no known fix. When you enable the "remote control" interface in Kodi, it allows anyone with the password (which is disabled by default) to download any files Kodi has read access to on the machine it's running. Considering Kodi is often connected to multiple services, this may mean elevated compromise and more nasty stuff. I furthered the investigation done with my own analysis which showed the problem is difficult to solve: Kodi internally uses the facility to show thumbnails and media to the user, and there are no clear way of restricting which paths Kodi should have access to. Indeed, Kodi is designed to access mounted file systems and paths in arbitrary locations. In Debian bug #855225, I further confirmed confirmed wheezy and jessie-backports as vulnerable and therefore showed with good certainty that stretch and sid are vulnerable as well. I also suggested possible workaround, but at this point, it's in upstream's hands, as the changes will be intrusive. The file transfer mechanism need to be revamped all over Kodi, or authentication (with a proper password policy), need to be enforced.

Squirrelmail Next I looked at that old webmail software, Squirrelmail, which suffers from a remote code execution vulnerability (CVE-2017-7692) when sending mails with sendmail on the commandline. This is arguably an edge case, but considering the patch was simple, I figured I would provide an update to the LTS community. I tried to get a coordinated release for jessie, since the code is the same, but this wasn't completed at the time of writing. A patch is available and will hopefully be picked up by another LTS worker soon.

Fop and Batik Those issues (CVE-2017-5661 and CVE-2017-5662) were more difficult. The patches weren't clearly documented and there were no upstream references other than security advisories for the first release in years (in the case of batik) or months (in the case of fop), which made it hard to track down the issues. Fortunately, I was able to track down the upstream issues (FOP-2668 and BATIK-1139) where I got confirmation on what the proper fixes were. I could then release DLA-927-1 and DLA-926-1 with the backported patches. I do not use fop or batik. In fact, even after reading the homepage of both products, I couldn't quite figure out what use people could possibly have for that thing. Before uploading the packages, I therefore made packages available for testing for fop and batik.

libsndfile Next up was libsndfile which a bunch of overflows when parsing various audio files. I backported a patch for CVE-2017-7585 CVE-2017-7586 and CVE-2017-7741 which all seemed to be fixed by a single patch usptream. CVE-2017-7742 was also fixed, although with a separate patch. In all of those, i could only test CVE-2017-7741 and CVE-2017-7742, as the others were missing test cases. I provided a test package for a few days then I also figured it would be best to incorporate the security fixes done in stable, which brought in fixes for CVE-2015-7805, CVE-2014-9756 and CVE-2014-9496. So in the end, I ported patches from wheezy to jessie and uploaded the jessie version (reverting certain build changes) into wheezy and uploaded DLA-928-1 with the results.

yaml-cpp I then turned to yaml-cpp, a C++ parser for YAML. This one didn't have a known upstream fix, but I figured I would give it a shot anyways. I ended up writing my first C++ code in years which is still pending review and merge upstream. It's not an easy problem to fix: this is basically an excessive recursion problem that can be used to smash the stack. I figured I could introduce a recursion limit, but as the discussion showed, this is a limited approach: stack size varies on different platforms and it's not easy to find the right limit. The real solution is to rewrite the code to avoid recursion but that's a major code refactoring I didn't feel belong in a LTS update. Besides, this could be better handled by upstream, so I will leave things at that for now. It does make you wonder how much code out there is recursing on untrusted data structures...

kedpm Finally, a friend over at Koumbit.org reported Debian bug #860817, as information leak in kedpm, a password manager I previously maintained. I requested and got assigned CVE-2017-8296 and provided a fix for wheezy and jessie. For unstable and the coming stretch release, I have requested kedpm to be completely removed from Debian (Debian bug #860817) which involved a release notes update (Debian bug #861277). It's unfortunate to see software go, but kedpm wasn't maintained. I wasn't the original author: I just gave a few patches and ended up maintaining that software and not using it. It's a bad situation to be in, as you don't really know what's working and not with the tools you are supposed to be responsible for. There are more modern alternatives available now and I encourage everyone to switch.

Triage Looking for more work, I peeked a bit in the secretary tasks to triage some pending issues. I found that trafficserver could be crashed with simple requests (CVE-2017-5659) so I looked into that issue. My analysis showed that the patch is long and complex and could be difficult to backport to the old version available in wheezy. I also couldn't reproduce the issue in wheezy, so it may be a bug introduced only later, although I couldn't confirm that directly. I also triaged wireshark, where I just noted the maintainer expressed concern that we were taking up issues too fast and will probably take care of this one. I also postponed various issues in GnuTLS (marked "no-dsa") as they affect only a (unfortunately) rarely used part of GnuTLS that has been removed in later version: OpenPGP support.

Other free software work

Debiman I finally got around contributing to the debiman project. I worked on ensuring that there is a dman compatibility in debiman, by shipping dman in the debian-goodies package (Debian bug #860920). I also submitted a pull request to fix the fix about page title, document the custom assets repository, fix a stray bracket and link to the link to venerable man7.org project After a discussion on IRC, I also filed a few more issues: I'm happy to be able to contribute to this important service and I hope the new FAQ I created will be online soon!

XMonad and Emacs I started using writeroom-mode again as part of my work on LWN. As it turns out, my setup was not exactly working: I had to port my config to the new version and windows weren't "sticky" as they should be, a known issue with Xmonad. Indeed, Xmonad doesn't obey the "static" or "all desktops" standard directives. Writeroom is a "distraction-free writing" mode for Emacs, so the irony of working on such a deep distraction in establishing a distraction free environment is not lost on me. Needing to scratch that particular itch, and with the help of clever people from the IRC channel, I was able to make Emacs tell Xmonad to show its window (or "frame" as Emacs likes to call it) on all desktops. This involved creating a new function which I think could be useful in the CopyWindow library: 
--   Toggle between "copyToAll" or "killAllOtherCopies". Copies to all
-- workspaces, or remove from all other workspaces, depending on
-- previous state (checked with "wsContainingCopies").
copyToAllToggle :: X ()
copyToAllToggle = do
    -- check which workspaces have copies
    copies <- wsContainingCopies
    if null copies
      then windows copyToAll -- no workspaces, make sticky
      else killAllOtherCopies -- already other workspaces, unstick
There are probably better ways of implementing this directly in the CopyWindow code - wsContainingCopies, in particular, is probably overkill. But it's all I can use directly from my xmonad.hs, so that's what I did. The other bit I needed was something to trigger that function from the outside. I rejected the ServerMode hook because it looked a bit too complicated and there is a built-in facility within X that works without this, which, from Emacs' point of view, is the x-send-client-message function. So I made up a new message identifier and wrote a event hook handler to process it: 
--   handle X client messages that tell Xmonad to make a window appear
-- on all workspaces
--
-- this should really be using _NET_WM_STATE and
-- _NET_WM_STATE_STICKY. but that's more complicated: then we'd need
-- to inspect a window and figure out the current state and act
-- accordingly. I am not good enough with Xmonad to figure out that
-- part yet.
--
-- Instead, just check for the relevant message and check if the
-- focused window is already on all workspaces and toggle based on
-- that.
--
-- this is designed to interoperate with Emacs's writeroom-mode module
-- and called be called from elisp with:
--
-- (x-send-client-message nil 0 nil "XMONAD_COPY_ALL_SELF" 8 '(0))
myClientMessageEventHook :: Event -> X All
myClientMessageEventHook (ClientMessageEvent  ev_message_type = mt, ev_data = dt ) = do
  dpy <- asks display
  -- the client message we're expecting
  copyAllMsg <- io $ internAtom dpy "XMONAD_COPY_ALL_SELF" False
  -- if the event matches the message we expect, toggle sticky state
  when (mt == copyAllMsg && dt /= []) $ do
    copyToAllToggle
  -- we processed the event completely
  return $ All True
All that was left was to hook that into Emacs, and I was done! Whoohoo! Full screen total domination, distraction free work! :) I would love to hear from others what they think of that approach, if they have improvements or if the above copyToAllToggle function could be merged in. Ideally, Xmonad would just parse the STICKY client messages and do the right thing - maybe even directly in CopyWindow - but I have found this enough Haskell for one day. You can see the diff on my home directory to see exactly the changes involved to make this configuration work.

Emacs packaging Speaking of Emacs, after complaining in the noisy #emacs IRC channel about the poor TLS configuration of marmelade.org -- and filing a bug (Debian bug #861106) regarding the use of SHA-1 in certificate pinning -- I was told we shouldn't expect trust from third-party ELPA repositories. Marmelade seems to be dead, as the maintainer is "behind the great firewall of China" and MELPA still hasn't figured out how to sign packages. In the end, it seems like there are tons of elpa packages in Debian and that if your favorite one is missing, that's a bug that can be filed and fixed. I first discovered that 6 of the packages I used were already packaged: And so I went ahead and filed a ton more bugs for the packages I am using but that aren't in Debian just yet: Of those, I can't recommend multiple-cursors (MC) enough: I used it at least 4 times just writing this text. It's just awesome. The other ones are also all great in their own right of course, but I feel they are more specific to my workflow whereas MC is just amazing.

ikiwiki I did some more work on ikiwiki the software driving this blog. I created a new plugin to, at least, fix anchors in the table of contents to be human readable. This is something I had done on the MoinMoin wiki almost a decade ago -- which I called then NicerHeadingIds and that I have always found frustrating with Ikiwiki. It turns out the problem was both easier and hairier than I thought. Right from the start, something weird was happening: something was already adding nice headings, but they were somewhat broken. It turns out that multimarkdown already inserts those headers, but I wasn't satisfied with the way they were generated. But even worse, I had the headinganchors plugin enabled, but that plugin wasn't taking effect, because of multimarkdown. And even if it would take effect, it doesn't behave well with non-ASCII characters, which gets turned in their numeric presentation. So I also wrote the i18nheadinganchors plugin that creates better headings and patched the toc plugin so that it can reuse existing anchors if they exist, while keeping backwards compatibility. I hope this gets merged in a future ikiwiki release so I do not have to carry this patch locally too long... In other news, I have upgraded the ikiwiki-hosting package to the latest version and sent a patch upstream to provide HSTS support.

Other stuff I have migrated all my public repos hosted on my home server to either Gitlab.com or Github. I also have repositories on 0xacab and it seemed ludicrous to have 4 different, canonical, places where my code was hosted. I have now about 40 different projects on Gitlab and about 60 on Github, although most of the latter are forks of existing projects. I also made a manpage for stressant and moved the documentation to RTFD which makes it neatly accessible. I also made small incremental improvements (like --directory support). I installed Rainloop on this server to give a nice, mobile-friendly webmail. Instructions to replicate this setup are in mail. In the constant git-annex documentation effort, I tried to draft a user guide that could be a basis for restructuring the documentation to be more easily accessible. I also helped a friend put his documentation on the wiki in splitting a repository. Finally, I also looked into Android stuff a little more. I wrote a usability review of the F-Droid privileged extension that will bring good changes, I hope. I also opened the discussion regarding reproducible builds to try and clarify exactly how those worked to help the Wallabag people ship consistently signed alphas. So far, it seems that it will remain a standard practice on F-Droid to ship packages that are not signed by the official upstream signature, unfortunately, unless upstream provides a reproducible build that is publicly available... Switching to such build is also a hairy issue, as, obviously, the signature changes, which raises the alarm we are trying to avoid in the first place.

Russ Allbery: Review: Neverness

Review: Neverness, by David Zindell
Publisher: Bantam Spectra
Copyright: May 1988
Printing: July 1989
ISBN: 0-553-27903-3
Format: Mass market
Pages: 552
Mallory Ringess is a Pilot, one of the people who can guide a lightship through interstellar space from inside the dark cocoon and biotech interface that allows visualization of the mathematics of interstellar travel. At the start of the book, he's young, arrogant, impulsive, and has a deeply unhealthy relationship with Leopold Soli, the Lord Pilot and supposedly his uncle by marriage (although they share a remarkable physical resemblance). An encounter with his uncle in a bar provokes a rash promise, and Ringess finds himself promising to attempt to map the Solid State Entity in search of the Elder Eddas, a secret of life from the mythical Ieldra that might lead to mankind's immortality. The opening of Neverness is Ringess's initial voyage and brash search, in which he proves to be a capable mathematician who can navigate a region of space twisted and deformed by becoming part of a transcendent machine intelligence. The knowledge he comes away with, though, is scarcely more coherent than the hints Soli relates at the start of the story: the secret of mankind is somehow hidden in its deepest past. That, in turn, provokes a deeply bizarre trip into the ice surrounding his home city of Neverness to attempt to steal biological material from people who have recreated themselves as Neanderthals. Beyond that point, I would say that things get even weirder, but weird still implies some emotional connection with the story. I think a more accurate description is that the book gets more incoherently mystical, more hopelessly pretentious, and more depressingly enthralled by childish drama. It's the sort of thing that one writes if one is convinced that the Oedipal complex is the height of subtle characterization. I loathed this book. I started loathing this book partway through Ringess's trip through the Solid State Entity, when Zindell's prose reached for transcendent complexity, tripped over its own shoelaces, and fell headlong into overwrought babbling. I continued reading every page because there's a perverse pleasure in hate-reading a book one dislikes this intensely, and because I wanted to write a review on the firm foundation of having endured the entire experience. The paperback edition I have has a pull quote from Orson Scott Card on the cover, which includes the phrase "excellent hard science fiction." I'm not sure what book Card read, because if this is hard science fiction, Lord of the Rings is paranormal romance. Even putting aside the idea that one travels through interstellar space by proving mathematical theorems in artificially dilated time (I don't think Zindell really understands what a proof is or why you write one), there's the whole business with stopping time with one's mind, reading other people's minds, and remembering one's own DNA. The technology, such as it is, makes considerably less sense than Star Wars. The hard SF requirement to keep technology consistent with extrapolated science is nowhere to be found here. The back-cover quote from the St. Louis Post-Dispatch is a bit more on-target: "Reminiscent of Gene Wolfe's New Sun novels... really comes to life among the intrigues of Neverness." This is indeed reminiscent of Gene Wolfe, in that it wouldn't surprise me at all if Zindell fell in love with the sense of antiquity, strangeness, and hints of understood technology that Wolfe successfully creates and attempted to emulate Wolfe in his first novel. Sadly, Zindell isn't Wolfe. Almost no one is, which is why attempting to emulate the extremely difficult feat Wolfe pulls off in the Book of the New Sun in your first novel is not a good idea. The results aren't pretty. There is something to be said for resplendent descriptions, rich with detail and ornamental prose. That something is "please use sparingly and with an eye to the emotional swings of the novel." Wolfe does not try to write most of a novel that way, which is what makes those moments of description so effective. Wolfe is also much better at making his mysteries and allusions subtle and unobtrusive, rather than having the first-person protagonist beat the reader over the head with them for pages at a time. This is a case where showing is probably better than telling. Let me quote a bit of description from the start of the book:
She shimmers, my city, she shimmers. She is said to be the most beautiful of all the cities of the Civilized Worlds, more beautiful even than Parpallaix or the cathedral cities of Vesper. To the west, pushing into the green sea like a huge, jewel-studded sleeve of city, the fragile obsidian cloisters and hospices of the Farsider's Quarter gleamed like black glass mirrors. Straight ahead as we skated, I saw the frothy churn of the Sound and their whitecaps of breakers crashing against the cliffs of North Beach and above the entire city, veined with purple and glazed with snow and ice, Waaskel and Attakel rose up like vast pyramids against the sky. Beneath the half-ring of extinct volcanoes (Urkel, I should mention, is the southernmost peak, and though less magnificent than the others, it has a conical symmetry that some find pleasing) the towers and spires of the Academy scattered the dazzling false winter light so that the whole of the Old City sparkled.
That's less than half of that paragraph, and the entire book is written like that, even in the middle of conversations. Endless, constant words piled on words about absolutely everything, whether important or not, whether emotionally significant or not. And much of it isn't even description, but philosophical ponderings that are desperately trying to seem profound. Here's another bit:
Although I knew I had never seen her before, I felt as if I had known her all my life. I was instantly in love with her, not, of course, as one loves another human being, but as a wanderer might love a new ocean or a gorgeous snowy peak he has glimpsed for the first time. I was practically struck dumb by her calmness and her beauty, so I said the first stupid thing which came to mind. "Welcome to Neverness," I told her.
Now, I should be fair: some people like this kind of description, or at least have more tolerance for it than I do. But that brings me to the second problem: there isn't a single truly likable character in this entire novel. Ringess, the person telling us this whole story, is a spoiled man-child, the sort of deeply immature and insecure person who attempts to compensate through bluster, impetuousness, and refusing to ever admit that he made a mistake or needed to learn something. He spends a good portion of the book, particularly the deeply bizarre and off-putting sections with the fake Neanderthals, attempting to act out some sort of stereotyped toxic masculinity and wallowing in negative emotions. Soli is an arrogant, abusive asshole from start to finish. Katherine, Ringess's love interest, is a seer who has had her eyes removed to see the future (I cannot express how disturbing I found Zindell's descriptions of this), has bizarre and weirdly sexualized reactions to the future she never explains, and leaves off the ends of all of her sentences, which might be be the most pointlessly irritating dialogue quirk I've seen in a novel. And Ringess's mother is a man-hating feminist from a separatist culture who turns into a master manipulator (I'm starting to see why Card liked this book). I at least really wanted to like Bardo, Ringess's closest friend, who has a sort of crude loyalty and unwillingness to get pulled too deep into the philosophical quicksand lurking underneath everything in this novel. Alas, Zindell insists on constantly describing Bardo's odious eating, belching, and sexual habits every time he's on the page, thus reducing him to the disgusting buffoon who gets drunk a lot and has irritating verbal ticks. About the only person I could stand by the end of the book was Justine, who at least seems vaguely sensible (and who leaves the person who abuses her), but she's too much of a non-entity to carry sustained interest. (There is potential here for a deeply scathing and vicious retelling of this story from Justine's point of view, focusing on the ways she was belittled, abused, and ignored, but I think Zindell was entirely unaware of why that would be so effective.) Oh, and there's lots of gore and horrific injury and lovingly-described torture, because of course there is. And that brings me back to the second half of that St. Louis Post-Dispatch review quote: "... really comes to life among the intrigues of Neverness." I would love to know what was hiding behind the ellipses in this pull quote, because this half-sentence is not wrong. Insofar as Neverness has any real appeal, it's in the intrigues of the city of Neverness and in the political structure that rules it. What this quote omits is that these intrigues start around page 317, more than halfway through the novel. That's about the point where faux-Wolfe starts mixing with late-career Frank Herbert and we get poet-assassins, some revelations about the leader of the Pilot culture, and some more concrete explanations of what this mess of a book is about. Unfortunately, you have to read through the huge and essentially meaningless Neanderthal scenes to get there, scenes that have essentially nothing to do with the interesting content of this book. (Everything that motivates them turns out to be completely irrelevant to the plot and useless for the characters.) The last 40% of the book is almost passable, and characters I cared about might have even made it enjoyable. Still, a couple of remaining problems detract heavily, chief among them the lack of connection of the great revelation of the story to, well, anything in the story. We learn at the very start of the novel that the stars of the Vild are mysteriously exploding, and much of the novel is driven by uncovering an explanation and solution. The characters do find an explanation, but not through any investigation. Ringess is simply told what is happening, in a wad of exposition, as a reward for something else entirely. It's weirdly disconnected from and irrelevant to everything else in the story. (There are some faint connections to the odd technological rules that the Pilot society lives under, but Zindell doesn't even draw attention to those.) The political intrigue in Neverness is similar: it appears out of nowhere more than halfway through the book, with no dramatic foundation for the motives of the person who has been keeping most of the secrets. And the final climax of the political machinations involves a bunch of mystical nonsense masquerading as science, and more of the Neanderthal bullshit that ruins the first half of the book. This is a thoroughly bad book: poorly plotted, poorly written, clotted and pretentious in style, and full of sociopaths and emotionally stunted children. I read the whole thing because I'm immensely stubborn and make poor life choices, but I was saying the eight deadly words ("I don't care what happens to these people") by a hundred pages in. Don't emulate my bad decisions. (Somehow, this novel was shortlisted for the Arthur C. Clarke award in 1990. What on earth could they possibly have been thinking?) Neverness is a stand-alone novel, but the ending sets up a subsequent trilogy that I have no intention of reading. Followed by The Broken God. Rating: 2 out of 10

28 April 2017

Russ Allbery: Review: Small Gods

Review: Small Gods, by Terry Pratchett
Series: Discworld #13
Publisher: Harper
Copyright: February 1994
Printing: March 2008
ISBN: 0-06-109217-7
Format: Mass market
Pages: 357
Small Gods is the thirteenth Discworld novel, but it features new characters and is unrelated to any of the previous books. Some reading order guides show it as following Pyramids in an "ancient civilizations" track, but its only relationship with that book is some minor thematic similarities. You could start here with Discworld if you wanted to. Brutha is a novice in the hierarchy of the church of the Great God Om, and his elders are convinced he'll probably die a novice. He's just not particularly bright, you see. But he is very obedient, and he doesn't mind doing hard work, and there's nothing exactly wrong with him, except that he looks at people with startling intensity when they're talking to him. Almost as if he's listening. All that seems about to change, however, when the Great God Om himself approaches Brutha and starts talking to him. Not that Brutha is at all convinced at first that this is happening, particularly given that Om appears in the form of a small, battered, one-eyed tortoise who was dropped into the church garden by an eagle attempting to break his shell. Small Gods is, as you might have guessed, a parody of religion, at least large, organized religion with fixed hierarchies, organizations called the Quisition that like to torture people, and terrifyingly devout deacons who are certain of themselves in ways that no human ever should be. It's also an interesting bit of Discworld metaphysics: gods gain power from worship (a very old idea in fantasy), and when they don't get enough worship, they end up much diminished and even adrift in the desert. Or trapped in the form of a small tortoise. One might wonder how Om ended up in his present condition given the vast and extremely authoritarian church devoted to his worship, but that's the heart of Pratchett's rather pointed parody: large religious organizations end up being about themselves, rather than about the god they supposedly worship, to such an extent that they don't provide any worship at all. Brutha is not thinking of things like this. Once he's finally convinced that Om is who he claims to be, he provides worship and belief of a very practical but wholehearted and unshakable sort, just as he does everything else in life. That makes him the eighth prophet of Om as prophecy foretold, but it's far from clear how that will be of any practical use. Or how Om will come back into power. And meanwhile, Brutha has come to the attention of Vorbis, the head of the Exquisitors, who does not know about the tortoise (and wouldn't believe if he did), but who has a use for Brutha's other talent: his eidetic memory. In typical Pratchett fashion, the story expands to include a variety of other memorable characters from the neighboring city of Ephebe, a country full of gods and philosophers. Vorbis's aims here are unclear at the start of the book, but Vorbis being who he is, they can't be good. Brutha is drawn along in his wake. Meanwhile, Om is constantly watching for an opportunity to regain his lost power and worshipful following, and also to avoid being eaten. Despite the humorous components, Small Gods is rather serious about religion and about its villain. It's also a touch repetitive; Om's lack of power and constant fretting about it, Brutha's earnest but naive loyalty, and Vorbis's malevolent determination are repeatedly stressed and get a little old. Some bits in Ephebe are quite fun, but the action is a bit disjointed, partly because the protagonist is rarely the motive force in the plot. There are also some extended scenes of trudging through the desert that I thought dragged a bit. But Pratchett hits some powerful notes in his critique of religion, and there are a few bits with Death at the end of the book that I thought were among the better pieces of Discworld philosophy. And when Brutha gets a chance to use his one talent of memory, I greatly enjoyed the resulting scenes. He hits just the right combination of modesty, capability, and earnestness. I know a lot of Pratchett readers really like Small Gods. I'm not one of those; I thought it was about average for the Discworld series (at least among the books I've read so far). But average for Discworld is still pretty good, and its new setting makes it a plausible place to start (or to take a break from the other Discworld plot threads). Followed, in publication order, by Lords and Ladies. I don't believe it has a direct plot sequel. Rating: 7 out of 10

27 April 2017

Joey Hess: Exede Surfbeam 2

My new satellite internet connection is from Exede, connecting to the ViaSat 1 bird in geosync orbit. A few technical details that I've observed follow. antagonistic by design The "Surfbeam 2 wifi modem" is a closed proprietary system. That is important because it's part of Exede's bandwidth management system. The Surfbeam tracks data use and sends it periodically to Exede. When a user has gone over their monthly priority data, Exede then throttles the bandwidth in various ways -- this throttling seems to be implemented, at least partially on the Surfbeam itself. (Perhaps by setting QoS flags?) So, if a user could hack their Surfbeam, they could probably bypass the bandwidth caps, or at least some of them. Perhaps Exede would notice eventually. Of course, doing so would surely violate the Exede TOS. If you're renting the modem, like I am, hacking a device you don't own might also subject you to criminal penalties. Needless to say, I don't plan to hack the SurfBeam. But it's been hacked before. So, this is a device that lives in people's homes and is antagonistic to them by design. weird data throttling The way the Surfbeam reports data use back to Exede periodically and gets throttling configured has some odd effects sometimes. For example, the Surfbeam can be in throttled state left-over from the previous billing month. When a new billing month begins, it can remain throttled for some time (up to multiple hours) until it sends an update to Exede and they un-throttle it. Data downloaded at that time might still be counted as priority data even though it was throttled. I've seen some good indications of that happening, but am not sure yet. But, I've decided that the throttling doesn't matter for me. Why? ViaSat 1 has many spot beams, and the working-class beam I'm in (most of it is in eastern Kentucky) does not seem to get a lot of use between 7 am and 4:30 pm weekdays. Even when throttled, I often get 300 kb/s - 1 mb/s speeds during the day, which is not a lot worse than the ~2.5 mb/s peak when unthrottled. And that's the time when I want to use broadband -- when the sun is shining and I'm at home at work/play. I'm probably going to switch to a cheaper plan with less priority data, because the priority data is not buying me much. This is a big change from the old FAP which rendered the satellite no faster than dialup. a whole network in there Looking at the ports open on the Surfbeam, some very strange things turned up. First, there are not one, not two, but three separate IPs used by the device, and there are at least two and perhaps three distinct computers involved. There are a lot of flickering LEDs inside the box; a whole network in there. 192.168.100.1 is the satellite controller. It's a Linux box, fingerprinted as kernel 3.10 or so (so full of security holes presumably), and it's running thttpd/2.25b (doesn't seem to have any known holes). It seems to have ssh and snmp, but with some port filtering that prevents access. (Note that the above exploit video confirms that snmp is running.) Some machine parsable data is exposed at http://192.168.100.1/index.cgi?page=modemStatusData and http://192.168.100.1/index.cgi?page=triaStatusData. (See (SurfStat program) 192.168.1.1 is the wifi router. It has a dns server, an icslap proxy, and nmap thinks it's Linux 3.x with Synology DiskStation Manager (probably the latter is a false positive?) It has its own separate web server for configuration, which is not thttpd. I'm fairly sure this is a separate processor from the other IP address. 192.168.100.2 responds to ICMP, but has no open ports at all. However, it seems to have filtered ssh, telnet, msrpc, microsoft-ds, and port 8090 (probably http), so perhaps it's running all that stuff. This one is definitely a separate processor, located in the Satellite dish's TRIA (transmit receive integrated assembly). Verified by disconnecting the dish's coax cable and being unable to ping it. lack of source code for GPLed software Exede did not provide anything to me about the GPL licensed source code on the Surfbeam 2. I'm not sure if they're legally required to do so in my case, since they're renting it to me? But, they do let you buy the device too, so it's interesting that nothing mentions the licenses of its software and where to get the source code. I'll try to buy it and ask for the source code eventually. power use The Surfbeam pulls as much power as two beefy laptops, but it is beaming signals thousands of miles into space, so I give it a bit of a pass. I want to find a way to power it from DC power, but have not had any success with that so far, so am wasting more power to run it on an inverter. Its power supply is rated at 30v and 2.5 amps. Interestingly, I've seen a photo of another Surfbeam power supply that only pulls 1.5 amps. This and a different case with fewer (external) LEDs makes me think perhaps the installer stuck me with an old and less efficient version. Maybe they integrated some of those processors into a single board in a new version and perhaps made it waste less power as heat. Mine gets pretty warm. I'm currently only able to run it approximately one hour per hour of sun collected by the house. Still on dialup the rest of the time. With the ability to get on broadband when dialup is being painful, being on dialup the rest of the time is perfectly ok. Of course it helps that with tools like git-annex, I'm very well tuned to popping up onto broadband briefly and making it count.

Russ Allbery: Review: Necessity

Review: Necessity, by Jo Walton
Series: Thessaly #3
Publisher: Tor
Copyright: July 2016
ISBN: 0-7653-7902-3
Format: Hardcover
Pages: 331
Athena's experiment with a city (now civilization) modeled after Plato's Republic continues, but in a form that she would not have anticipated, and in a place rather far removed from its origins. But despite new awareness of the place and role of gods, a rather dramatic relocation, and unanticipated science-fiction complications, it continues in much the same style as in The Just City: thoughtful, questioning debate, a legal and social system that works surprisingly well, and a surprising lack of drama. At least, that is, until the displaced cities are contacted by the mainstream of humanity, and Athena goes unexpectedly missing. The latter event turns out to have much more to do with the story than the former, and I regret that. Analyzing mainline human civilization and negotiating the parameters of a very odd first contact would have, at least in my opinion, lined up a bit better with the strengths of this series. Instead, the focus is primarily on metaphysics, and the key climactic moment in those metaphysics is rather mushy and incoherent compared to the sharp-edged analysis Walton's civilization is normally capable of. Not particularly unexpected, as metaphysics of this sort are notoriously tricky to approach via dialectical logic, but it was a bit of a letdown. Much of this book deals with Athena's disappearance and its consequences (including the title), and it wasn't bad, but it wanders a bit into philosophical musings on the nature of gods. Necessity is a rather odd book, and I think anyone who started here would be baffled, but it does make a surprising amount of sense in the context of the series. Skipping ahead to here seems like a truly bad idea, but reading the entire series (relatively closely together) does show a coherent philosophical, moral, and social arc. The Just City opens with Apollo confronted by the idea of individual significance: what does it mean to treat other people as one's equals in an ethical sense, even if they aren't on measures of raw power? The Thessaly series holds to that theme throughout and follows its implications. Many of the bizarre things that happen in this series seem like matter-of-fact outcomes once you're engrossed in the premises and circumstances at the time. Necessity adds a surprising amount of more typical science fiction trappings, but they turn out to be ancillary to the story. What matters is considered action, trying to be your best self, and the earnest efforts of a society to put those principles first. And that's the strength of the whole series, including Necessity: I like these people, I like how they think, and I enjoy spending time with them, almost no matter what they're doing. As with the previous books, we get interwoven chapters from different viewpoints, this time from three primary characters plus some important "guest" chapters. As with the previous books, the viewpoint characters are different again, mostly a generation younger, and I had to overcome my initial disappointment at not hearing the same voices. But Walton is excellent at characterization. I really like this earnest, thoughtful, oddly-structured society that always teeters on the edge of being hopelessly naive and trusting but is self-aware enough to never fall in. By the end of the book, I liked this round of characters nearly as much as I liked the previous rounds (although I've still never liked a character in these books as well as I liked Simmea). I think one incomplete but important way to sum up the entire Thessaly series is that it's a trilogy of philosophical society-building on top of the premise of a universal love for and earnest, probing, thoughtful analysis of philosophy. Walton's initial cheat is to use an deus ex machina to jumpstart such a society from a complex human world that would be unlikely to provide enough time or space for it to build its own separate culture and tradition. I think the science-fiction trick is required to make this work real-world societies that try this end up having to spend so much of their energy fighting intrusion from the outside and diffusion into the surrounding culture that they don't have the same room to avoid conformity and test and argue against their own visions. Necessity is not at all the conclusion of that experiment I would expect, but it won me over, and I think it worked, even if a few bits of it felt indulgent. Most importantly for that overall project, this series is generational, and Necessity shows how it would feel to grow up deep inside it, seeing evolution on top of a base structure that is ubiquitous and ignored. Even the generation in The Philosopher Kings wasn't far enough removed to support that; Necessity is, and in a way this book shows how distinctly different and even alien human culture can become when it has space to evolve on top of different premises. I enjoyed the moments of small surprise, where characters didn't react the way that I'd expect for reasons now buried generations-deep in their philosophical foundations. This book will not win you over if you didn't already like the series, and I suspect it will lose a few people who read the previous two books. The plot structure is a little strange, the metaphysics are a touch strained, and the ending is, well, not quite the payoff that I was hoping for, although it's thematically appropriate and grew on me after a few days of thinking it over. But I got more Socrates, finally, who is as delightful as always and sorely needed to add some irreverence and contrariness to the the mix. And I got to read more about practical, thoughtful people who are trying hard to do their best, to be their best selves, and to analyze and understand the world. There's something calming, delightful, and beautifully optimistic about their approach, and I'm rather sad to not have more of it to read. Rating: 7 out of 10

23 April 2017

Shirish Agarwal: Debconf 17 in Montreal

Debconf Montreal 17 logo Before I start, debconf registration opened about a week back depending upon what time-frame you are in. So, if you have used and contributed to Debian or are curious to know and try out Debian, this would be right time to register for the conference. For those who are either financially weak (like yours truly) or those who are a minority in any way can avail those
sponsorships, more information about the sponsorship/travel bursary can be found here. Also make sure that you read, understand and accept the Debconf Code of conduct before trying to register. Even if Debconf does give the bursary, getting a visa for Canada might be tricky but then what would be life without any challenges. In the hierarchy of wants and needs, I would say attending debconf would come in the want category and not the needs category as needs are simply food, clothes and roof over one s head, thankfully all of which have been there for as long as I remember or know. To dream is necessary for life and being part of debconf would be part of that dream. While I have put up couple of talks, I have asked if somebody would be willing to tackle/share some basic things in Debconf 17 as well. Let s see if I m able to get any response to that. There was a recent Debian BSP in Montreal. While I don t know how many bugs were fixed, although UDD shows only 32 bugs remain to be fixed before Stretch could be released. Indian cricket team To put things into perspective, I had gone to a friend s place at Market Yard and came to know of a local cricket league which is being played in the city on the likes/basis of the IPL . The amount they will get is INR 88,000 (which is 1,361 USD, around the cost of ticket to and fro from Pune to Montreal and back as of date) and similar and 10 teams of 11-15 people are vying from it. The winning team will get that amount along with a trophy and signed certificates. Of course at the highest level i.e. the IPL, each night the players would probably earn a million or more for each play. One IPL season pay is what 90% of the people don t/can t earn in their whole lifetime. Montreal Metro image from Wikimedia Commons Apart from the conference, I have shared about the northern lights and the snacks on stand thing, there is one thing more to explore and that is the Montreal Metro . While I was re-reading the wiki entry today I was very much reminded about Pune Metro and Metro Samvaad that I attended couple of weeks before. That too as it was not held too far from my house, around 500 metres so could attend. The only hitch I see is the question mark on fate of 6,133 trees . I have shared one probable solution but then it has its own challenges . In the end, Metro is the need of the hour as vehicles in pune are reaching the 5 million mark while residents are around 3 million plus something depending on whether you chose to believe the 2011 census or the more recentish world population review . In either case/scenario I do not see any improvement unless consistent public transport is available. While Buses don t run on time, it is hoped that the metro would run on time. Some of the similarities between the two system, apart from the fact that from what I read Montreal Metro is mostly subway while Pune Metro would be mostly elevated. There is the idea/thought of having POI (Points of Interest) and free shows etc. in order to attract young and old to travel to destinations to and fro. There are a lot of theatre groups, stand-up comedians, rock and roll as well as classical musical concerts for which people like me would be willing to travel week-ends to attend such shows. Of course, this is all in the head, we will know the reality starting this May and commercial journeys if everything goes as planned in end 2018.
Filed under: Miscellenous Tagged: #Canadian Visa, #Debconf Montreal 2017, #Debconf registration, #Debian BSP, #Debian Stretch, #Environment, #Montreal Metro, #Montreal Subway, #planet-debian, #Points of Interest, #Pune Metro, #travel bursary, Debian, Economics

22 April 2017

Manuel A. Fernandez Montecelo: Debian GNU/Linux port for RISC-V 64-bit (riscv64)

This is a post describing my involvement with the Debian GNU/Linux port for RISC-V (unofficial and not endorsed by Debian at the moment) and announcing the availability of the repository (still very much WIP) with packages built for this architecture. If not interested in the story but you want to check the repository, just jump to the bottom. Roots A while ago, mostly during 2014, I was involved in the Debian port for OpenRISC (or1k) about which I posted (by coincidence) exactly 2 years ago. The two of us working on the port stopped in August or September of that year, after knowing that the copyright of the code to add support for this architecture in GCC would not be assigned to the FSF, so it would never be added to GCC upstream unless the original authors changed their mind (which they didn't) or there was a clean-room reimplementation (which didn't happen so far). But a few other key things contributed to the decision to stop working on that port, which bear direct relationship to this story. One thing that particularly influenced me to stop working on it was a sense of lack of purpose, all things considered, for the OpenRISC port that we were working on. For example, these chips are sometimes used as part of bigger devices by Samsung to control or wake up other chips; but it was not clear whether there would ever be devices with OpenRISC as the main chip, specially in devices powerful enough to run Linux or similar kernels, and Debian on top. One can use FPGAs to synthesise OpenRISC or1k, but these are slow, and expensive when using lots of memory. Without prospects of having hardware easily available to users, there's not much point in having a whole Debian port ready to run on hardware that never comes to be. Yeah, sure, it's fun to create such a port, but it's tons of work to maintain and keep it up to-date forever, and with close to zero users it's very unrewarding. Another thing that contributed to decide to stop is that, at least in my opinion, 32-bit was not future-proof enough for general purpose computing, specially for new devices and ports starting to take off on that time and age. There was some incipient work to create another OpenRISC design for 64-bits, but it was still in an early phase. My secret hope and ultimate goal was to be able to run as much a free-ish computer as possible as my main system. Still today many people are buying and using 32-bit devices, like small boards; but very few use it as their main computer or servers for demanding workloads or complex services. So for me, even if feasible if one is very austere and dedicated, OpenRISC or1k failed that test. And lastly, another thing happened at the time... Enter RISC-V In August 2014, at the point when we were fully acknowledging the problem of upstreaming (or rather, lack thereof) the support for OpenRISC in GCC, RISC-V was announced to the world, bringing along papers with suggestive titles such as Instruction Sets Should Be Free: The Case For RISC-V (pdf) and articles like RISC-V: An Open Standard for SoCs - The case for an open ISA in EE Times. RISC-V (as the previous RISC-n marks) had been designed (or rather, was being designed, because it was and is an as yet unfinished standard) by people from UC Berkeley, including David Patterson, the pioneer of RISC computer designs and co-author of the seminal book Computer Architecture: A Quantitative Approach . Other very capable people are also also leading the project, doing the design and legwork to make it happen see the list of contributors. But, apart from throwing names, the project has many other merits. Similarly to OpenRISC, RISC-V is an open instruction set architecture (ISA), but with the advantage of being designed in more recent times (thus avoiding some mistakes and optimising for problems discovered more recently, as technology evolves); with more resources; with support for instruction widths of 32, 64 and even 128 bits; with a clean set of standard but optional extensions (atomics, float, double, quad, vector, ...); and with reserved space to add new extensions in ordered and compatible ways. In the view of some people in the OpenRISC community, this unexpected development in a way made irrelevant the ongoing update of OpenRISC for 64-bits, and from what I know and for better or worse, all efforts on that front stopped. Also interestingly (if nothing else, for my purposes of running as much a free system as possible), was that the people behind RISC-V had strong intentions to make it useful to create modern hardware, and were pushing heavily towards it from the beginning. And together with this announcement or soonly after, it came the promise of free-ish hardware in the form of the lowRISC project. Although still today it seems to be a long way from actually shipping hardware, at least there was some prospect of getting it some day. On top of all that, about the freedom aspect, both the Berkeley and lowRISC teams engaged since very early on with the free hardware community, including attending and presenting at OpenRISC events; and lowRISC intended to have as much free hardware as possible in their planned SoC. Starting to work on the Debian port, this time for RISC-V So in late 2014 I slowly started to prepare the Debian port, switching on and off. The Userland spec was frozen in 2014 just before the announcement, the Privilege one is still not frozen today, so there was no need to rush. There were plans to upstream the support in the toolchain for RISC-V (GNU binutils, glibc and GCC; Linux and other useful software like Qemu) in 2015, but sadly these did not materialise until late 2016 and 2017. One of the main reasons for the delay was due to the slowness to sort out the copyright assignment of the code to the FSF (again). Still today, only binutils and GCC are upstreamed, and Linux and glibc depend on the Privilege spec being finished, so it will take a while. After the experience with OpenRISC and the support in GCC, I didn't want to invest too much time, lest it all became another dead-end due to lack of upstreaming so I was just cross-compiling here and there, testing Qemu (which still today is very limited for this architecture, e.g. no network support and very limited character and block devices) and trying to find and report bugs in the implementations, and send patches (although I did not contribute much in the end). Incompatible changes in the toolchain In terms of compiling packages and building-up a repository, things were complicate, and less mature and stable than the OpenRISC ones were even back in 2014. In theory, with the Userland spec being frozen, regular programs (below the Operating System level) compiled at any time could have run today; but in practice what happened is that there were several rounds of profound or, at least, disrupting changes in the toolchain before and while being upstreamed, which made the binary packages that I had built to not work at all (changes in dynamic loader, registers where arguments are stored when jumping functions, etc.). These major breakages happened several times already, and kind of unexpectedly at least for the people not heavily involved in the implementation. When the different pieces are upstreamed it is expected that these breakages won't happen; but still there's at least the fundamental bit of glibc, which will probably change things once again in incompatible ways before or while being upstreamed. Outside Debian but within the FOSS / Linux world, the main project that I know of is that some people from Fedora also started a port in mid 2016 and did great advances, but from what I know they put the project in the freezer in late 2016 until all such problems are resolved they don't want to spend time rebootstrapping again and again. What happened recently on the Debian front In early 2016 I created the page for RISC-V in the Debian wiki, expecting that things were at last fully stable and the important bits of the toolchain upstreamed during that year I was too optimistic. Some other people (including Debian folks) have been contributing for a while, in the wiki, mailing lists and IRC channels, and in the RISC-V project mailing lists you will see their names everywhere. However, due to the combination of lack of hardware, software not upstreamed and shortcomings of emulators (chiefly Qemu) make contributions hard and very tedious, nothing much happened recently visible to the outside world in terms of software. The private repository-in-the-making In late 2015 and beginning of 2016, having some free time in my hands and expecting that all things would coalesce quickly, I started to build a repository of binary packages in a more systematic way, with most of the basic software that one can expect in a basic Debian system (including things common to all Linux systems, and also specific Debian software like dpkg or apt, and even aptitude!). After that I also built many others outside the basic system (more than 1000 source packages and 2000 or 3000 arch-dependent binary packages in total), specially popular libraries (e.g. boost, gtk+ version 2 and 3), interpreters (several versions of lua, perl and python, also version 2 and 3) and in general packages that are needed to build many other packages (like doxygen). Unfortunately, some of these most interesting packages do not compile cleanly (more because of obscure or silly errors than proper porting), so they are not included at the moment. I intentionally avoided trying to compile thousands of packages in the archive which would be of nobody's use at this point; but many more could be compiled without much effort. About the how, initially I started cross-compiling and using rebootstrap, which was of great help in the beginning. Some of the packages that I cross-compiled had bugs that I did not know how to debug without a live and native (within emulators) system, so I tried to switch to natively built packages very early on. For that I needed many packages built natively (like doxygen or cmake) which would be unnecessary if I remained cross-compiling the host tools would be used in that case. But this also forced me to eat my own dog food, which even if much slower and tedious, it was on the whole a more instructive experience; and above all, it helped to test and make sure that the the tools and the whole stack was working well enough to build hundreds of packages. Why the repository-in-the-making was private Until now I did not attempt to make the repository available on-line, for several reasons. First because it would be kind of useless to publish files that were not working or would soon not work, due to the incompatible changes in the toolchain, rendering many or most of the packages built useless. And because, for many months now, I expected that things would stabilise and to have something stable really soon now but this didn't happen yet. Second because of lack of resources and time since mid 2016, and because I got some packages only compiled thanks to (mostly small and unimportant, but undocumented and unsaved) hacks, often working around temporary bugs and thus not worth sending upstream; but I couldn't share the binaries without sharing the full source and fulfill licenses like the GNU GPL. I did a new round of clean rebuilds in the last few weeks, just finished, the result is close to 1000 arch-dependent packages. And third, because of lack of demand. This changed in the last few weeks, when other people started to ask me to share the results even if incomplete or not working properly (I had one request in the past, but couldn't oblige in time at the time). Finally, the work so far: repository now on-line So finally, with the great help from Kurt Keville from MIT, and Bytemark sponsoring a machine where most of the packages were built, here we have the repository: The lines for /etc/apt/sources.list are: 
 deb [ arch=riscv64 signed-by=/usr/share/keyrings/debian-keyring.gpg ] http://riscv.mit.edu/debian unstable main
 deb-src [ signed-by=/usr/share/keyrings/debian-keyring.gpg ] http://riscv.mit.edu/debian unstable main
The repository is signed with my key as Debian Developer, contained in the file /usr/share/keyrings/debian-keyring.gpg, which is part of the package debian-keyring (available from Debian and derivatives). WARNING!! This repository, though, is very much WIP, incomplete (some package dependencies cannot be fulfilled, and it's only a small percentage of the Debian archive, not trying to be comprehensive at the moment) and probably does not work at all in your system at this point, for the following reasons: The combination of all these shortcomings is specially unfortunate, because without glibc provided it will be difficult that you can get the binaries to run at all; but there are some packages that are arch-dependent but not too tied to libc or the dynamic loader will not be affected. At least you can try one the few static packages present in Debian, like the one in the package bash-static. When one removes moving parts like the dynamic loader and libc, since the basic machine instructions are stable for several years now, it should work, but I wouldn't discard some dark magic that prevents even static binaries from working. Still, I hope that the respository in its current state is useful to some people, at least for those who requested it. If one has the environment set-up, it's easy to unpack the contents of the .deb files and try out the software (which often is not trivial or very slow to compile, or needs lots of dependencies to be built first first). ... and finally, even if not useful at all for most people at the moment, by doing this I also hope that efforts like this spark your interest to contribute to free software, free hardware, or both! :-)

Next.

Previous.