Search Results: "filippo"

5 May 2022

Reproducible Builds: Reproducible Builds in April 2022

Welcome to the April 2022 report from the Reproducible Builds project! In these reports, we try to summarise the most important things that we have been up to over the past month. If you are interested in contributing to the project, please take a few moments to visit our Contribute page on our website.

News Cory Doctorow published an interesting article this month about the possibility of Undetectable backdoors for machine learning models. Given that machine learning models can provide unpredictably incorrect results, Doctorow recounts that there exists another category of adversarial examples that comprise a gimmicked machine-learning input that, to the human eye, seems totally normal but which causes the ML system to misfire dramatically that permit the possibility of planting undetectable back doors into any machine learning system at training time .
Chris Lamb published two supporter spotlights on our blog: the first about Amateur Radio Digital Communications (ARDC) and the second about the Google Open Source Security Team (GOSST).
Piergiorgio Ladisa, Henrik Plate, Matias Martinez and Olivier Barais published a new academic paper titled A Taxonomy of Attacks on Open-Source Software Supply Chains (PDF):
This work proposes a general taxonomy for attacks on open-source supply chains, independent of specific programming languages or ecosystems, and covering all supply chain stages from code contributions to package distribution. Taking the form of an attack tree, it covers 107 unique vectors, linked to 94 real-world incidents, and mapped to 33 mitigating safeguards.

Elsewhere in academia, Ly Vu Duc published his PhD thesis. Titled Towards Understanding and Securing the OSS Supply Chain (PDF), Duc s abstract reads as follows:
This dissertation starts from the first link in the software supply chain, developers . Since many developers do not update their vulnerable software libraries, thus exposing the user of their code to security risks. To understand how they choose, manage and update the libraries, packages, and other Open-Source Software (OSS) that become the building blocks of companies completed products consumed by end-users, twenty-five semi-structured interviews were conducted with developers of both large and small-medium enterprises in nine countries. All interviews were transcribed, coded, and analyzed according to applied thematic analysis

Upstream news Filippo Valsorda published an informative blog post recently called How Go Mitigates Supply Chain Attacks outlining the high-level features of the Go ecosystem that helps prevent various supply-chain attacks.
There was new/further activity on a pull request filed against openssl by Sebastian Andrzej Siewior in order to prevent saved CFLAGS (which may contain the -fdebug-prefix-map=<PATH> flag that is used to strip an arbitrary the build path from the debug info if this information remains recorded then the binary is no longer reproducible if the build directory changes.

Events The Linux Foundation s SupplyChainSecurityCon, will take place June 21st 24th 2022, both virtually and in Austin, Texas. Long-time Reproducible Builds and openSUSE contributor Bernhard M. Wiedemann learned that he had his talk accepted, and will speak on Reproducible Builds: Unexpected Benefits and Problems on June 21st.
There will be an in-person Debian Reunion in Hamburg, Germany later this year, taking place from 23 30 May. Although this is a Debian event, there will be some folks from the broader Reproducible Builds community and, of course, everyone is welcome. Please see the event page on the Debian wiki for more information. 41 people have registered so far, and there s approx 10 on-site beds still left.
The minutes and logs from our April 2022 IRC meeting have been published. In case you missed this one, our next IRC meeting will take place on May 31st at 15:00 UTC on #reproducible-builds on the OFTC network.

Debian Roland Clobus wrote another in-depth status update about the status of live Debian images, summarising the current situation that all major desktops build reproducibly with bullseye, bookworm and sid, including the Cinnamon desktop on bookworm and sid, but at a small functionality cost: 14 words will be incorrectly abbreviated . This work incorporated:
  • Reporting an issue about unnecessarily modified timestamps in the daily Debian installer images. [ ]
  • Reporting a bug against the debian-installer: in order to use a suitable kernel version. (#1006800)
  • Reporting a bug in: texlive-binaries regarding the unreproducible content of .fmt files. (#1009196)
  • Adding hacks to make the Cinnamon desktop image reproducible in bookworm and sid. [ ]
  • Added a script to rebuild a live-build ISO image from a given timestamp. [
  • etc.
On our mailing list, Venkata Pyla started a thread on the Debian debconf cache is non-reproducible issue while creating system images and Vagrant Cascadian posted an excellent summary of the reproducibility status of core package sets in Debian and solicited for similar information from other distributions.
Lastly, 122 reviews of Debian packages were added, 44 were updated and 193 were removed this month adding to our extensive knowledge about identified issues. A number of issue types have been updated as well, including timestamps_generated_by_hevea, randomness_in_ocaml_preprocessed_files, build_path_captured_in_emacs_el_file, golang_compiler_captures_build_path_in_binary and build_path_captured_in_assembly_objects,

Other distributions Happy birthday to GNU Guix, which recently turned 10 years old! People have been sharing their stories, in which reproducible builds and bootstrappable builds are a recurring theme as a feature important to its users and developers. The experiences are available on the GNU Guix blog as well as a post on fossandcrafts.org
In openSUSE, Bernhard M. Wiedemann posted his usual monthly reproducible builds status report.

Upstream patches The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of such patches, including:

diffoscope diffoscope is our in-depth and content-aware diff utility. Not only can it locate and diagnose reproducibility issues, it can provide human-readable diffs from many kinds of binary formats. This month, Chris Lamb prepared and uploaded versions 210 and 211 to Debian unstable, as well as noticed that some Python .pyc files are reported as data, so we should support .pyc as a fallback filename extension [ ]. In addition, Mattia Rizzolo disabled the Gnumeric tests in Debian as the package is not currently available [ ] and dropped mplayer from Build-Depends too [ ]. In addition, Mattia fixed an issue to ensure that the PATH environment variable is properly modified for all actions, not just when running the comparator. [ ]

Testing framework The Reproducible Builds project runs a significant testing framework at tests.reproducible-builds.org, to check packages and other artifacts for reproducibility. This month, the following changes were made:
  • Daniel Golle:
    • Prefer a different solution to avoid building all OpenWrt packages; skip packages from optional community feeds. [ ]
  • Holger Levsen:
    • Detect Python deprecation warnings in the node health check. [ ]
    • Detect failure to build the Debian Installer. [ ]
  • Mattia Rizzolo:
    • Install disorderfs for building OpenWrt packages. [ ]
  • Paul Spooren (OpenWrt-related changes):
    • Don t build all packages whilst the core packages are not yet reproducible. [ ]
    • Add a missing RUN directive to node_cleanup. [ ]
    • Be less verbose during a toolchain build. [ ]
    • Use disorderfs for rebuilds and update the documentation to match. [ ][ ][ ]
  • Roland Clobus:
    • Publish the last reproducible Debian ISO image. [ ]
    • Use the rebuild.sh script from the live-build package. [ ]
Lastly, node maintenance was also performed by Holger Levsen [ ][ ].
If you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:

5 April 2021

Kees Cook: security things in Linux v5.9

Previously: v5.8 Linux v5.9 was released in October, 2020. Here s my summary of various security things that I found interesting: seccomp user_notif file descriptor injection
Sargun Dhillon added the ability for SECCOMP_RET_USER_NOTIF filters to inject file descriptors into the target process using SECCOMP_IOCTL_NOTIF_ADDFD. This lets container managers fully emulate syscalls like open() and connect(), where an actual file descriptor is expected to be available after a successful syscall. In the process I fixed a couple bugs and refactored the file descriptor receiving code. zero-initialize stack variables with Clang
When Alexander Potapenko landed support for Clang s automatic variable initialization, it did so with a byte pattern designed to really stand out in kernel crashes. Now he s added support for doing zero initialization via CONFIG_INIT_STACK_ALL_ZERO, which besides actually being faster, has a few behavior benefits as well. Unlike pattern initialization, which has a higher chance of triggering existing bugs, zero initialization provides safe defaults for strings, pointers, indexes, and sizes. Like the pattern initialization, this feature stops entire classes of uninitialized stack variable flaws. common syscall entry/exit routines
Thomas Gleixner created architecture-independent code to do syscall entry/exit, since much of the kernel s work during a syscall entry and exit is the same. There was no need to repeat this in each architecture, and having it implemented separately meant bugs (or features) might only get fixed (or implemented) in a handful of architectures. It means that features like seccomp become much easier to build since it wouldn t need per-architecture implementations any more. Presently only x86 has switched over to the common routines. SLAB kfree() hardening
To reach CONFIG_SLAB_FREELIST_HARDENED feature-parity with the SLUB heap allocator, I added naive double-free detection and the ability to detect cross-cache freeing in the SLAB allocator. This should keep a class of type-confusion bugs from biting kernels using SLAB. (Most distro kernels use SLUB, but some smaller devices prefer the slightly more compact SLAB, so this hardening is mostly aimed at those systems.) new CAP_CHECKPOINT_RESTORE capability
Adrian Reber added the new CAP_CHECKPOINT_RESTORE capability, splitting this functionality off of CAP_SYS_ADMIN. The needs for the kernel to correctly checkpoint and restore a process (e.g. used to move processes between containers) continues to grow, and it became clear that the security implications were lower than those of CAP_SYS_ADMIN yet distinct from other capabilities. Using this capability is now the preferred method for doing things like changing /proc/self/exe. debugfs boot-time visibility restriction
Peter Enderborg added the debugfs boot parameter to control the visibility of the kernel s debug filesystem. The contents of debugfs continue to be a common area of sensitive information being exposed to attackers. While this was effectively possible by unsetting CONFIG_DEBUG_FS, that wasn t a great approach for system builders needing a single set of kernel configs (e.g. a distro kernel), so now it can be disabled at boot time. more seccomp architecture support
Michael Karcher implemented the SuperH seccomp hooks, Guo Ren implemented the C-SKY seccomp hooks, and Max Filippov implemented the xtensa seccomp hooks. Each of these included the ever-important updates to the seccomp regression testing suite in the kernel selftests. stack protector support for RISC-V
Guo Ren implemented -fstack-protector (and -fstack-protector-strong) support for RISC-V. This is the initial global-canary support while the patches to GCC to support per-task canaries is getting finished (similar to the per-task canaries done for arm64). This will mean nearly all stack frame write overflows are no longer useful to attackers on this architecture. It s nice to see this finally land for RISC-V, which is quickly approaching architecture feature parity with the other major architectures in the kernel. new tasklet API
Romain Perier and Allen Pais introduced a new tasklet API to make their use safer. Much like the timer_list refactoring work done earlier, the tasklet API is also a potential source of simple function-pointer-and-first-argument controlled exploits via linear heap overwrites. It s a smaller attack surface since it s used much less in the kernel, but it is the same weak design, making it a sensible thing to replace. While the use of the tasklet API is considered deprecated (replaced by threaded IRQs), it s not always a simple mechanical refactoring, so the old API still needs refactoring (since that CAN be done mechanically is most cases). x86 FSGSBASE implementation
Sasha Levin, Andy Lutomirski, Chang S. Bae, Andi Kleen, Tony Luck, Thomas Gleixner, and others landed the long-awaited FSGSBASE series. This provides task switching performance improvements while keeping the kernel safe from modules accidentally (or maliciously) trying to use the features directly (which exposed an unprivileged direct kernel access hole). filter x86 MSR writes
While it s been long understood that writing to CPU Model-Specific Registers (MSRs) from userspace was a bad idea, it has been left enabled for things like MSR_IA32_ENERGY_PERF_BIAS. Boris Petkov has decided enough is enough and has now enabled logging and kernel tainting (TAINT_CPU_OUT_OF_SPEC) by default and a way to disable MSR writes at runtime. (However, since this is controlled by a normal module parameter and the root user can just turn writes back on, I continue to recommend that people build with CONFIG_X86_MSR=n.) The expectation is that userspace MSR writes will be entirely removed in future kernels. uninitialized_var() macro removed
I made treewide changes to remove the uninitialized_var() macro, which had been used to silence compiler warnings. The rationale for this macro was weak to begin with ( the compiler is reporting an uninitialized variable that is clearly initialized ) since it was mainly papering over compiler bugs. However, it creates a much more fragile situation in the kernel since now such uses can actually disable automatic stack variable initialization, as well as mask legitimate unused variable warnings. The proper solution is to just initialize variables the compiler warns about. function pointer cast removals
Oscar Carter has started removing function pointer casts from the kernel, in an effort to allow the kernel to build with -Wcast-function-type. The future use of Control Flow Integrity checking (which does validation of function prototypes matching between the caller and the target) tends not to work well with function casts, so it d be nice to get rid of these before CFI lands. flexible array conversions
As part of Gustavo A. R. Silva s on-going work to replace zero-length and one-element arrays with flexible arrays, he has documented the details of the flexible array conversions, and the various helpers to be used in kernel code. Every commit gets the kernel closer to building with -Warray-bounds, which catches a lot of potential buffer overflows at compile time. That s it for now! Please let me know if you think anything else needs some attention. Next up is Linux v5.10.

2021, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 License.
CC BY-SA 4.0

11 June 2020

Antoine Beaupr : CVE-2020-13777 GnuTLS audit: be scared

So CVE-2020-13777 came out while I wasn't looking last week. The GnuTLS advisory (GNUTLS-SA-2020-06-03) is pretty opaque so I'll refer instead to this tweet from @FiloSottile (Go team security lead):
PSA: don't rely on GnuTLS, please. CVE-2020-13777 Whoops, for the past 10 releases most TLS 1.0 1.2 connection could be passively decrypted and most TLS 1.3 connections intercepted. Trivially. Also, TLS 1.2 1.0 session tickets are awful.
You are reading this correctly: supposedly encrypted TLS connections made with affected GnuTLS releases are vulnerable to passive cleartext recovery attack (and active for 1.3, but who uses that anyways). That is extremely bad. It's pretty close to just switching everyone to HTTP instead of HTTPS, more or less. I would have a lot more to say about the security of GnuTLS in particular -- and security in general -- but I am mostly concerned about patching holes in the roof right now, so this article is not about that. This article is about figuring out what, exactly, was exposed in our infrastructure because of this.

Affected packages Assuming you're running Debian, this will show a list of packages that Depends on GnuTLS:
apt-cache --installed rdepends libgnutls30   grep '^ '   sort -u
This assumes you run this only on hosts running Buster or above. Otherwise you'll need to figure out a way to pick machines running GnuTLS 3.6.4 or later. Note that this list only first level dependencies! It is perfectly possible that another package uses GnuTLS without being listed here. For example, in the above list I have libcurl3-gnutls, so the be really thorough, I would actually need to recurse down the dependency tree. On my desktop, this shows an "interesting" list of targets:
  • apt
  • cadaver - AKA WebDAV
  • curl & wget
  • fwupd - another attack on top of this one
  • git (through the libcurl3-gnutls dependency)
  • mutt - all your emails
  • weechat - your precious private chats
Arguably, fetchers like apt, curl, fwupd, and wget rely on HTTPS for "authentication" more than secrecy, although apt has its own OpenPGP-based authentication so that wouldn't matter anyways. Still, this is truly distressing. And I haven't mentioned here things like gobby, network-manager, systemd, and others - the scope of this is broad. Hell, even good old lynx links against GnuTLS. In our infrastructure, the magic command looks something like this:
cumin -o txt -p 0  'F:lsbdistcodename=buster' "apt-cache --installed rdepends libgnutls30   grep '^ '   sort -u"   tee gnutls-rdepds-per-host   awk ' print $NF '   sort   uniq -c   sort -n
There, the result is even more worrisome, as those important packages seem to rely on GnuTLS for their transport security:
  • mariadb - all MySQL traffic and passwords
  • mandos - full disk encryption
  • slapd - LDAP passwords
mandos is especially distressing although it's probably not vulnerable because it seems it doesn't store the cleartext -- it's encrypted with the client's OpenPGP public key -- so the TLS tunnel never sees the cleartext either. Other reports have also mentioned the following servers link against GnuTLS and could be vulnerable:
  • exim
  • rsyslog
  • samba
  • various VNC implementations

Not affected Those programs are not affected by this vulnerability:
  • apache2
  • gnupg
  • python
  • nginx
  • openssh
This list is not exhaustive, naturally, but serves as an example of common software you don't need to worry about. The vulnerability only exists in GnuTLS, as far as we know, so programs linking against other libraries are not vulnerable. Because the vulnerability affects session tickets -- and those are set on the server side of the TLS connection -- only users of GnuTLS as a server are vulnerable. This means, for example, that while weechat uses GnuTLS, it will only suffer from the problem when acting as a server (which it does, in relay mode) or, of course, if the remote IRC server also uses GnuTLS. Same with apt, curl, wget, or git: it is unlikely to be a problem because it is only used as a client; the remote server is usually a webserver -- not git itself -- when using TLS.

Caveats Keep in mind that it's not because a package links against GnuTLS that it uses it. For example, I have been told that, on Arch Linux, if both GnuTLS and OpenSSL are available, the mutt package will use the latter, so it's not affected. I haven't confirmed that myself nor have I checked on Debian. Also, because it relies on session tickets, there's a time window after which the ticket gets cycled and properly initialized. But that is apparently 6 hours by default so it is going to protect only really long-lasting TLS sessions, which are uncommon, I would argue. My audit is limited. For example, it might have been better to walk the shared library dependencies directly, instead of relying on Debian package dependencies.

Other technical details It seems the vulnerability might have been introduced in this merge request, itself following a (entirely reasonable) feature request to make it easier to rotate session tickets. The merge request was open for a few months and was thoroughly reviewed by a peer before being merged. Interestingly, the vulnerable function (_gnutls_initialize_session_ticket_key_rotation), explicitly says:
 * This function will not enable session ticket keys on the server side. That is done
 * with the gnutls_session_ticket_enable_server() function. This function just initializes
 * the internal state to support periodical rotation of the session ticket encryption key.
In other words, it thinks it is not responsible for session ticket initialization, yet it is. Indeed, the merge request fixing the problem unconditionally does this:
memcpy(session->key.initial_stek, key->data, key->size);
I haven't reviewed the code and the vulnerability in detail, so take the above with a grain of salt. The full patch is available here. See also the upstream issue 1011, the upstream advisory, the Debian security tracker, and the Redhat Bugzilla.

Moving forward The impact of this vulnerability depends on the affected packages and how they are used. It can range from "meh, someone knows I downloaded that Debian package yesterday" to "holy crap my full disk encryption passwords are compromised, I need to re-encrypt all my drives", including "I need to change all LDAP and MySQL passwords". It promises to be a fun week for some people at least. Looking ahead, however, one has to wonder whether we should follow @FiloSottile's advice and stop using GnuTLS altogether. There are at least a few programs that link against GnuTLS because of the OpenSSL licensing oddities but that has been first announced in 2015, then definitely and clearly resolved in 2017 -- or maybe that was in 2018? Anyways it's fixed, pinky-promise-I-swear, except if you're one of those weirdos still using GPL-2, of course. Even though OpenSSL isn't the simplest and secure TLS implementation out there, it could preferable to GnuTLS and maybe we should consider changing Debian packages to use it in the future. But then again, the last time something like this happened, it was Heartbleed and GnuTLS wasn't affected, so who knows... It is likely that people don't have OpenSSL in mind when they suggest moving away from GnuTLS and instead think of other TLS libraries like mbedtls (previously known as PolarSSL), NSS, BoringSSL, LibreSSL and so on. Not that those are totally sinless either... "This is fine", as they say...

5 February 2017

Vincent Bernat: A Makefile for your Go project

My most loathed feature of Go is the mandatory use of GOPATH: I do not want to put my own code next to its dependencies. Hopefully, this issue is slowly starting to be accepted by the main authors. In the meantime, you can workaround this problem with more opinionated tools (like gb) or by crafting your own Makefile. For the later, you can have a look at Filippo Valsorda s example or my own take which I describe in more details here. This is not meant to be an universal Makefile but a relatively short one with some batteries included. It comes with a simple Hello World! application.

Project structure For a standalone project, vendoring is a must-have1 as you cannot rely on your dependencies to not introduce backward-incompatible changes. Some packages are using versioned URLs but most of them aren t. There is currently no standard tool to handle vendoring. My personal take is to vendor all dependencies with Glide. It is a good practice to split an application into different packages while the main one stay fairly small. In the hellogopher example, the CLI is handled in the cmd package while the application logic for printing greetings is in the hello package:
.
  cmd/
    hello.go
    root.go
    version.go
  glide.lock (generated)
  glide.yaml
  vendor/ (dependencies will go there)
  hello/
    root.go
    root_test.go
  main.go
  Makefile
  README.md

Down the rabbit hole Let s take a look at the various features of the Makefile.

GOPATH handling Since all dependencies are vendored, only our own project needs to be in the GOPATH:
PACKAGE  = hellogopher
GOPATH   = $(CURDIR)/.gopath
BASE     = $(GOPATH)/src/$(PACKAGE)
$(BASE):
    @mkdir -p $(dir $@)
    @ln -sf $(CURDIR) $@
The base import path is hellogopher, not github.com/vincentbernat/hellogopher: this shortens imports and makes them easily distinguishable from imports of dependency packages. However, your application won t be go get-able. This is a personal choice and can be adjusted with the $(PACKAGE) variable. We just create a symlink from .gopath/src/hellogopher to our root directory. The GOPATH environment variable is automatically exported to the shell commands of the recipes. Any tool should work fine after changing the current directory to $(BASE). For example, this snippet builds the executable:
.PHONY: all
all:   $(BASE)
    cd $(BASE) && $(GO) build -o bin/$(PACKAGE) main.go

Vendoring dependencies Glide is a bit like Ruby s Bundler. In glide.yaml, you specify what packages you need and the constraints you want on them. Glide computes a glide.lock file containing the exact versions for each dependencies (including recursive dependencies) and download them in the vendor/ folder. I choose to check into the VCS both glide.yaml and glide.lock files. It s also possible to only check in the first one or to also check in the vendor/ directory. A work-in-progress is currently ongoing to provide a standard dependency management tool with a similar workflow. We define two rules2:
GLIDE = glide
glide.lock: glide.yaml   $(BASE)
    cd $(BASE) && $(GLIDE) update
    @touch $@
vendor: glide.lock   $(BASE)
    cd $(BASE) && $(GLIDE) --quiet install
    @ln -sf . vendor/src
    @touch $@
We use a variable to invoke glide. This enables a user to easily override it (for example, with make GLIDE=$GOPATH/bin/glide).

Using third-party tools Most projects need some third-party tools. We can either expect them to be already installed or compile them in our private GOPATH. For example, here is the lint rule:
BIN    = $(GOPATH)/bin
GOLINT = $(BIN)/golint
$(BIN)/golint:   $(BASE) #  
    go get github.com/golang/lint/golint
.PHONY: lint
lint: vendor   $(BASE) $(GOLINT) #  
    @cd $(BASE) && ret=0 && for pkg in $(PKGS); do \
        test -z "$$($(GOLINT) $$pkg   tee /dev/stderr)"   ret=1 ; \
     done ; exit $$ret
As for glide, we let the user a chance to override which golint executable to use. By default, it uses a private copy. But a user can use its own copy with make GOLINT=/usr/bin/golint. In , we have the recipe to build the private copy. We simply issue go get3 to download and build golint. In , the lint rule executes golint on each package contained in the $(PKGS) variable. We ll explain this variable in the next section.

Working with non-vendored packages only Some commands need to be provided with a list of packages. Because we use a vendor/ directory, the shortcut ./... is not what we expect as we don t want to run tests on our dependencies4. Therefore, we compose a list of packages we care about:
PKGS = $(or $(PKG), $(shell cd $(BASE) && \
    env GOPATH=$(GOPATH) $(GO) list ./...   grep -v "^$(PACKAGE)/vendor/"))
If the user has provided the $(PKG) variable, we use it. For example, if they want to lint only the cmd package, they can invoke make lint PKG=hellogopher/cmd which is more intuitive than specifying PKGS. Otherwise, we just execute go list ./... but we remove anything from the vendor directory.

Tests Here are some rules to run tests:
TIMEOUT = 20
TEST_TARGETS := test-default test-bench test-short test-verbose test-race
.PHONY: $(TEST_TARGETS) check test tests
test-bench:   ARGS=-run=__absolutelynothing__ -bench=.
test-short:   ARGS=-short
test-verbose: ARGS=-v
test-race:    ARGS=-race
$(TEST_TARGETS): test
check test tests: fmt lint vendor   $(BASE)
    @cd $(BASE) && $(GO) test -timeout $(TIMEOUT)s $(ARGS) $(PKGS)
A user can invoke tests in different ways:
  • make test runs all tests;
  • make test TIMEOUT=10 runs all tests with a timeout of 10 seconds;
  • make test PKG=hellogopher/cmd only runs tests for the cmd package;
  • make test ARGS="-v -short" runs tests with the specified arguments;
  • make test-race runs tests with race detector enabled.

Tests coverage go test includes a test coverage tool. Unfortunately, it only handles one package at a time and you have to explicitely list the packages to be instrumented, otherwise the instrumentation is limited to the currently tested package. If you provide too many packages, the compilation time will skyrocket. Moreover, if you want an output compatible with Jenkins, you ll need some additional tools.
COVERAGE_MODE    = atomic
COVERAGE_PROFILE = $(COVERAGE_DIR)/profile.out
COVERAGE_XML     = $(COVERAGE_DIR)/coverage.xml
COVERAGE_HTML    = $(COVERAGE_DIR)/index.html
.PHONY: test-coverage test-coverage-tools
test-coverage-tools:   $(GOCOVMERGE) $(GOCOV) $(GOCOVXML) #  
test-coverage: COVERAGE_DIR := $(CURDIR)/test/coverage.$(shell date -Iseconds)
test-coverage: fmt lint vendor test-coverage-tools   $(BASE)
    @mkdir -p $(COVERAGE_DIR)/coverage
    @cd $(BASE) && for pkg in $(PKGS); do \ #  
        $(GO) test \
            -coverpkg=$$($(GO) list -f '  join .Deps "\n"  ' $$pkg   \
                    grep '^$(PACKAGE)/'   grep -v '^$(PACKAGE)/vendor/'   \
                    tr '\n' ',')$$pkg \
            -covermode=$(COVERAGE_MODE) \
            -coverprofile="$(COVERAGE_DIR)/coverage/ echo $$pkg   tr "/" "-" .cover" $$pkg ;\
     done
    @$(GOCOVMERGE) $(COVERAGE_DIR)/coverage/*.cover > $(COVERAGE_PROFILE)
    @$(GO) tool cover -html=$(COVERAGE_PROFILE) -o $(COVERAGE_HTML)
    @$(GOCOV) convert $(COVERAGE_PROFILE)   $(GOCOVXML) > $(COVERAGE_XML)
First, we define some variables to let the user override them. We also require the following tools (in ):
  • gocovmerge merges profiles from different runs into a single one;
  • gocov-xml converts a coverage profile to the Cobertura format;
  • gocov is needed to convert a coverage profile to a format handled by gocov-xml.
The rules to build those tools are similar to the rule for golint described a few sections ago. In , for each package to test, we run go test with the -coverprofile argument. We also explicitely provide the list of packages to instrument to -coverpkg by using go list to get a list of dependencies for the tested package and keeping only our owns.

Final result While the main goal of using a Makefile was to work around GOPATH, it s also a good place to hide the complexity of some operations, notably around test coverage. The excerpts provided in this post are a bit simplified. Have a look at the final result for more perks!

  1. In Go, vendoring is about both bundling and dependency management. As the Go ecosystem matures, the bundling part (fixed snapshots of dependencies) may become optional but the vendor/ directory may stay for dependency management (retrieval of the latest versions of dependencies matching a set of constraints).
  2. If you don t want to automatically update glide.lock when a change is detected in glide.yaml, rename the target to deps-update and make it a phony target.
  3. There is some irony for bad mouthing go get and then immediately use it because it is convenient.
  4. I think ./... should not include the vendor/ directory by default. Dependencies should be trusted to have run their own tests in the environment they expect them to succeed. Unfortunately, this is unlikely to change.

12 April 2016

Reproducible builds folks: Reproducible builds: week 49 in Stretch cycle

What happened in the reproducible builds effort between March 27th and April 2nd: Toolchain fixes
  • Emmanuel Bourg uploaded ant/1.9.6-2 which makes the Tstamp task support the SOURCE_DATE_EPOCH variable, and the Javadoc task use en as the default locale if none was specified and SOURCE_DATE_EPOCH is set.
Packages fixed The following packages have become reproducible due to changes in their build dependencies: ctioga2, erlang-bitcask, libcommons-collections3-java, libjgoodies-animation-java, libjide-oss-java, octave-gsl, octave-interval, octave-io, octave-quaternion, octave-signal, octave-stk, segment, service-wrapper-java, sqlline, svnkit, uddi4j, velocity-tools. The following packages became reproducible after getting fixed: Some uploads fixed some reproducibility issues, but not all of them: Patches submitted which have not made their way to the archive yet:
  • #783239 on kexec-tools by Lunar: follow-up patch to cope with locale variations.
  • #819347 on starvoyager by Sascha Steinbiss: sort the list of input object files.
  • #819352 on xpdf by Sascha Steinbiss: sort the list of linked object files.
  • #819512 on breeze by Dhole: force grep to treat all files as text to avoid locale-related issues.
  • #819726 on ckbuilder by boyska: add support for SOURCE_DATE_EPOCH.
  • #819767 on libtool by rain1: removes extra timestamps from the build system, ensure a stable file order when creating the source archive, and replace uses of the hostname command with the fixed string "localhost".
tests.reproducible-builds.org The i386 builders are now testing packages on i386 for reproducibility. It will probably take 4 weeks until everything has been build twice, on this arch. (h01ger) Package reviews 52 reviews have been removed, 24 added and 4 updated in the previous week. Chris Lamb reported 13 new FTBFS. New issue: copyright_year_in_comments_generated_by_ckbuilder. Misc. This week's edition was mostly written by Lunar, with some help by Reiner Herrmann and h01ger.

21 August 2015

Norbert Preining: The sins of the past adding Cyrillic glyphs without renaming fonts

The URW Base35 fonts are a great set of fonts, available for free as in free software. They have been part of various distributions and systems since long time. Big thanks to URW for their work. But these fonts don t have Cyrillic or Greek glyphs. Be it as it is, world would be easy. People would need to use different fonts for these languages. Comes around someone who did the unthinkable namely adding the Cyrillic and Greek glyphs to the fonts (by now nothing bad), but then NOT renaming the fonts. Here we see one point of the stupidity of GPL and absolute freedom. Because what we now have is that documents produced several TeX engines (in particular XeTeX and LuaTeX) which use fontconfig to search the fonts, suddenly pick up these changed fonts that fake their identity, and what comes out is this, a complete rubbish: broken-fonts And now we are suffering huge pain from that. Look at the bug reports of that are coming in:
  • 796120 xdvipdfmx broken
  • 789391 developers reference fonts broken
  • 787759 fonts broken in dblatex
Just to name a few. And there is a simple way to circumvent this: Don t install gsfonts which guarantees that fontconfig finds the real original URW fonts within the TeX Live tree first. I have now spent many hours to track down these problems, find the reason, and at the end of the day there is always gsfonts with its broken fonts with added Cyrillic glyphs. I honestly don t care about the history, there are now many fonts with Cyrillic and Greek glyphs, there is no need to fake fonts, and incorrectly take over font names. This should be a lesson to all the GPL zealots that require absolute freedom of each and everything. Unfortunately things don t work like that. Using AND RENAMING is ok, the Knuth license as I would say, but anything else is just a source of much pain. End for today, I have to go to work now. Real work instead of fighting sins of the past. Nothing to enjoy here. Additional information Just to let you know, before starting a flame war, I have already contacted the upstream developers, that is TeX Live, and explained them the situation. I don t see much chance for fixing, since the problem is with fonts without upstream and support, which are probably only used in Debian (I haven t seen them anywhere else but some mentioning in RH), and which are not officially supported or distributed. It really needs a nice developer to look into why this breakage appeared. Let us hope. And instead of flaming, anyone here is invited to dig into the code him/herself and search for changes. Additional information 2015-08-22 Just to back up my complaints and counteract several of the comments: I am quoting from an email of a colleague on the list where we are discussing the problem:
However, the fonts extended by Valek Filippov are quite problematic. The Type1 spec clearly requires that there may not exist two different fonts with the same /FontName. The modified fonts shipped with Ghostscript have the same /FontName as the original fonts donated by URW and not even the /UniqueID was changed. IMO they are broken because they don t comply with the Type1 specification.
I hope that convinced also the last in doubt.

1 April 2014

Russ Allbery: Review: Asimov's, September 2011

Review: Asimov's Science Fiction, September 2011
Editor: Sheila Williams
Issue: Volume 35, No. 9
ISSN: 1065-2698
Pages: 112
Due to various other life priorities, it's been quite a while since I read this magazine. Let's see if I can remember the contents well enough to review it properly. The editorial this issue was about the Readers' Awards. Vaguely interesting, but Williams didn't have much to add beyond announcing the winners. I'm very happy to see Rusch's "Becoming One with the Ghosts" win best novella, though. The Silverberg column was more interesting: some musings and pop history about the Japanese convention of a retired emperor and how that fit into national politics. Di Filippo's book review column is all about short story collections, continuing the trend of Di Filippo mostly being interested in things I don't care about. "The Observation Post" by Allen M. Steele: A bit of alternate history set during the Cuban Missile Crisis, but with airships. The protagonist was a radioman aboard a blimp that was patrolling the ocean for Russian vessels sailing to Cuba. A storm forces them down on an island, resulting in an encounter with some claimed tourists who may be Russian spies. The SFnal twist is unlikely to come as much surprise to an experienced reader, and the barb at the end of the story suffers from the same problem. I appreciate the ethical dilemma, but I've also seen it in lots of stories and have a hard time getting fully invested in another version of it. But the story is otherwise competently written. (6) "D.O.C.S." by Neal Barrett, Jr.: Everyone has an author or two that they just don't get. Barrett is one of mine, although this story is a bit less surreal than most of his. I'm fairly sure it's an odd twist on the "death panel" conspiracy theory given a fantastic twist, but it's not entirely forthright about what's going on. Possibly of more interest to those who like Barrett better. (5) "Danilo" by Carol Emshwiller: Emshwiller's stories are always distinctive and not quite like anyone else's, involving odd outsiders and their attempts to make sense of their world. This one involves, as is common, an out-of-the-way village. Lewella claims that she's going to be married to a stranger from the north. No one believes her, although they give her bridal gifts anyway, and then one day she takes her gifts and leaves. The protagonist follows her, to look after her. The rest of the story walks the boundary that Emshwiller often walks, leaving the reader unsure whether the characters are in touch with some deeper reality or insane and suffering, but the ending is even more ambiguous than normal and, at least for me, entirely unsatisfying. (4) "Shadow Angel" by Erick Melton: This is another retread of an old SF idea. This time, it's that piloting through hyperspace involves alternate modes of consciousness and has profound effects on the pilot. The risk of this sort of story is that it turns hallucinatory and a bit incoherent, and I think that happened here. I like the world-building; the glimmers of future politics and trade and the way he weaves alternate timelines into the story caught my interest. But the story wasn't quite coherent enough (although part of this may be reviewing it quite some time after I originally read it). Promising, but not clear, and without quite enough agency for the protagonist. (6) "The Odor of Sanctity" by Ian Creasey: I found this story more memorable. The conceit is that a future society has developed technology that allows the capture and replay of scents, which has created a huge market for special scent experiences and the triggering of memories. The story is set in the Philippines and revolves around a Catholic priest who takes the mission to the poor seriously. He's dying, and several people wonder if it is possible to capture the mythical odor of scantity: the sweet scent said to follow the death of a saint rather than the normal odor of human death. Creasey handles this idea well, blending postulated future technology, the practical and cynical world of the poor streets, and a balance between mystical belief and practical skepticism. Nothing in the story is that surprising, but I was happy with the eventual resolution. (7) "Grandma Said" by R. Neube: This story's protagonist is a cleanser on a frontier planet made extremely dangerous by a virulent alien fungus. It is almost always fatal and very difficult to eradicate. Vic's job is to completely sanitize anything that had been in contact with a victim and maintain the other rules of strict quarantine required to keep the fungal infection from spreading uncontrolled. Nuebe weaves world-building together with Vic's background and adds a twist in the form of deeply unhealthy responses to the constant stress of living near death. Well told, if a bit disturbing. (7) "Stalker" by Robert Reed: Reed has a knack for fascinating and disturbing stories, and this is an excellent example of the type. The protagonist is a manufactured companion who is completely devoted to its owner. Their commercial name is Adorers, but everyone calls them Stalkers. In this case, the protagonist's owner is a serial rapist and murderer; given that, and given how good Reed is at writing these sorts of stories, you can probably imagine how chilling it is. As usual, there is a sharp barb in the ending, and not the one I was expecting. Good if you can handle the graphic violence and disturbing subject material. (7) "Burning Bibles" by Alan Wall: This is an interesting twist on the spy thriller. A three-letter agency in charge of investigating possible terrorist plots becomes suspicious after a warehouse of Bibles burns in mysterious circumstances. The agent they send in is a deaf-mute with special powers of intuition. This prompted some eye-rolling, and there's a lot of magic disability powers here to annoy, but it's played mostly straight after that introduction. The rest is a fairly conventional spy story, despite special empathic powers, but it's one I enjoyed and thought was fairly well-written. (7) Rating: 6 out of 10

16 May 2013

Russ Allbery: Review: Asimov's, July 2011

Review: Asimov's Science Fiction, July 2011
Editor: Sheila Williams
Issue: Volume 35, No. 7
ISSN: 1065-2698
Pages: 112
Williams's editorial is a mildly interesting piece about story titles. Silverberg's column is a more interesting (and rather convincing) rebuttal of the joke that fiction authors are "professional liars," combined with an examination of a fake and fantastic 14th travelogue that (at least in Silverberg's telling) was widely believed at the time. The precis of Silverberg's argument is that lying requires an intent to deceive, which is a property of deceptive memoir writers but not of fiction authors. Di Filippo's review column, as usual, is devoted almost entirely to esoterica, although I was moderately interested to hear of Stableford's continued work on translating early French SF. None of it seems compelling enough to go buy, but good translations of early works seem like a good thing to have in the world. "Day 29" by Chris Beckett: The conceit of this novelette is an interstellar travel system akin to a transporter that allows near-instantaneous travel between worlds. The drawback is that all memories from somewhere between 40 and 29 days before transit up until transit are wiped. The progatonist is a data analyst who is about to travel, and therefore by agency rule is required to stop doing work on day 40 before transmission since he can't be held legally liable for anything he has no recollection of doing. (I would like to say that I find this implausible, since one could always keep records, but it's exactly the sort of ass-covering regulation that a human resources department would come up with.) The premise is quite interesting: what do you do during that period that you're going to forget? Beckett wisely mixes Stephen's current waiting period on the colony world with his diary of his original waiting period on Earth the first time he went through the transmission process, and the latter adds greatly to the reader's appreciation of the weirdness of the forgotten interval. Unfortunately, this is a story more about psychological exploration than about plot, and Stephen just isn't very interesting. The telepathic but possibly nonsentient aliens add weirdness but not much else, and the ending of the story provided little sense of closure or conclusion for me. A good idea, but not the execution I wanted. (5) "Pug" by Theodora Goss: Since I grew up with a pug, I have a soft spot for a story featuring one; sadly, though, this story has insufficient pug in it. This is a quiet fantasy (Asimov's calls it SF, presumably on the basis of parallel worlds and a hypothesized scientific explanation, but it reads like fantasy to me) featuring Victorian girls, including one with a bad heart. They discover a hidden door to other versions of their world and do some minor exploration. There's little or nothing in the way of plot; the story is more of an attempt to capture a mood. It's mildly diverting, but I wish it had gone somewhere more substantial. (5) "Dunyon" by Kristine Kathryn Rusch: A Rusch story is often the highlight of an issue, and this is no exception. The protagonist is the owner of a bar in a space station that's become a combination of a refugee camp and a slum. War and chaos have created desperate people, most of whom are attempting to find some way to resources and get out of the bottom of society. The story is about a rumor: a mythical system named Dunyon that's safe and far away. And it's about how people react to that rumor. There's nothing particularly surprising about the direction the story goes (it's fairly short), but Rusch is always a good storyteller. (7) "The Music of the Sphere" by Norman Spinrad: I've had mixed feelings about Spinrad's fiction (and some of his essays), but I liked this story, despite its implausibility. It's set in the near future, featuring an expert in cetaceans and dolphin perception and a composer obsessed with both loud music and classical musical style. Just from that description, you can probably predict much of the story, but I thought it had some neat ideas about dolphins, whales, and alternate perception and aesthetics. (Note: neat, not necessarily biologically plausible.) Enjoyable. (6) "Bring on the Rain" by Josh Roseman: In a change of pace from the rest of the issue, this is a post-apocalyptic story of caravans of wheeled ships traversing a scorched and ruined landscape in search of weather systems and rain. The feel is of an inverted Waterworld, but with more emphasis on military tactics and cooperating fleets. The transposition of fleet maneuvers to huge ground vehicles adds some extra fun. The plot has little to do with the background and is a fairly stock military adventure scenario, but it's reasonably well-told. The story feels like an excerpt from a larger military-SF-inspired adventure, but the length keeps the quantity of tactics and maneuvering below the threshold where I would get bored. (6) "Twelvers" by Leah Cypess: This is a sharp and occasionally mean story of adolescent cruelty and alienation. Darla is a "twelver," a child who was carried an extra three months in the womb using newly-invented medical technology because of a belief in the advantages this would bring in later life. Unfortunately for all those who used this technique, what it also brought was a preternatural calm and an unusual reaction to emotions. Darla finds it almost impossible to get upset at anything, and that, of course, prompts the cruelty and abuse of other children. Most of the story is a description of that abuse, leading up to Darla stumbling into a nasty solution to her immediate problem. It's all very believable (well, apart from the motivating biology), but I didn't enjoy reading about it, and I'm certainly not convinced that the ending will lead to anything good. (5) "The Messenger" by Bruce McAllister: This is a very short time travel story, where time travel is used to try to unwind old family pain. This world follows the unalterable history model: no changes to the past are possible, and anything you do in the past has already happened. The mechanics are mostly avoided. Instead, McAllister concentrates on his mother, his father, and their complex relationship. I would have needed a bit more background on the characters to care enough about them for the story to be fully effective, but while the heartstring-pulling is kind of obvious, it's still a solid story. (6) "The Copenhagen Interpretation" by Paul Cornell: This is the most ingenious of the stories in this issue. It's set in a future world that extends what seemed to me to be pre-World-War-I great power politics, although there may be a hint of the Cold War. Great nations have reached a careful balance of power, and spies and secret services work to sustain that balance. The progatonist is one of those agents, making use of advanced technology like space folds in the service of a cause that he doesn't entirely believe in. Cornell mixes in mental conditioning, artificial people, space travel, and even aliens (maybe) in a taut thriller plot that, for me, gained a great deal from the unexplained strangeness of its background. If you like diving into the deep end and following a fast-moving plot against a background of strangeness, this is the sort of SF you'll enjoy. (7) Rating: 6 out of 10

22 January 2013

Russ Allbery: Review: Fantasy & Science Fiction, March/April 2011

Review: Fantasy & Science Fiction, March/April 2011
Editor: Gordon van Gelder
Issue: Volume 120, No. 3 & 4
ISSN: 1095-8258
Pages: 258
Charles de Lint's book review column sticks with the sorts of things he normally reviews: urban and contemporary fantasy and young adult. Predictably, I didn't find that much of interest. But I was happy to see that not all the reviews were positive, and he talked some about how a few books didn't work. I do prefer seeing a mix of positive and negative (or at least critical) reviews. James Sallis's review column focuses entirely on Henry Kuttner and C.L. Moore (by way of reviewing a collection). I'm always happy to see this sort of review. But between that and de Lint's normal subject matter, this issue of F&SF was left without any current science fiction reviews, which was disappointing. Lucius Shepard's movie review column features stunning amounts of whining, even by Shepard's standards. The topic du jour is how indie films aren't indie enough, mixed with large amounts of cane-shaking and decrying of all popular art. I find it entertaining that the F&SF film review column regularly contains exactly the sort of analysis that one expects from literary gatekeepers who are reviewing science fiction and fantasy. Perhaps David Langford should consider adding an "As We See Others" feature to Ansible cataloging the things genre fiction fans say about popular movies. "Scatter My Ashes" by Albert E. Cowdrey: The protagonist of this story is an itinerant author who has been contracted to write a family history (for $100K, which I suspect is a bit of tongue-in-cheek wish fulfillment) and has promptly tumbled into bed with his employer. But he is somewhat serious about the writing as well, and is poking around in family archives and asking relatives about past details. There is a murder (maybe) in the family history, not to mention some supernatural connections. Those familiar with Cowdrey's writing will recognize the mix of historical drama, investigation, and the supernatural. Puzzles are, of course, untangled, not without a bit of physical danger. Experienced fantasy readers will probably guess at some of the explanation long before the protagonist does. Like most Cowdrey, it's reliably entertaining, but I found it a bit thin. (6) "A Pocketful of Faces" by Paul Di Filippo: Here's a bit of science fiction, and another mystery, this time following the basic model of a police procedural. The police in this case are enforcing laws around acceptable use of "faces" in a future world where one can clone someone's appearance from their DNA and then mount it on a programmable android. As you might expect from that setup, the possibilities are lurid, occasionally disgusting, and inclined to give the police nightmares. After some scene-setting, the story kicks in with the discovery of the face of a dead woman who, at least on the surface, no one should have any motive to clone. There were a few elements of the story that were a bit too disgusting for me, but the basic mystery plot was satisfying. I thought the ending was a let-down, however. Di Filippo tries to complicate the story and, I thought, went just a little too far, leaving motives and intent more confusing than illuminating. (6) "The Paper Menagerie" by Ken Liu: Back to fantasy, this time using a small bit of magic to illustrate the emotional conflicts and difficulties of allegiance for second-generation immigrants. Jack is the son of an American farther and a Chinese mother who was a mail-order bride. He's young at the start of the story and struggling with the embarassment and humiliation that he feels at his mother's history and the difficulties he has blending in with other kids, leading to the sort of breathtaking cruelty that comes so easily from teenagers who are too self-focused and haven't yet developed adult empathy. I found this very hard to read. The magic is beautiful, personal, and very badly damaged by the cruelty in ways that can never really be fixed. It's a sharp reminder of the importance of being open-hearted, but it's also a devastating reminder that the lesson is normally learned too late. Not the story to read if you're prone to worrying about how you might have hurt other people. (6) "The Evening and the Morning" by Sheila Finch: This long novella is about a third of the issue and is, for once, straight science fiction, a somewhat rare beast in F&SF these days. It's set in the far future, among humans who are members of the Guild of Xenolinguists and among aliens called the Venatixi, and it's about an expedition back to the long-abandoned planet of Earth. I had a lot of suspension of disbelief problems with the setup here. While Earth has mostly dropped out of memory, there's a startling lack of curiosity about its current condition among the humans. Finch plays some with transportation systems and leaves humanity largely dependent on other races to explain the failure to return to Earth, but I never quite bought it. It was necessary to set up the plot, which is an exploration story with touches of first contact set on an Earth that's become alien to the characters, but it seemed remarkably artificial to me. But, putting that aside, I did get pulled into the story. Its emotional focus is one of decline and senescence, a growing sense of futility, that's halted by exploration, mystery, and analysis. The question of what's happened on Earth is inherently interesting and engaging, and the slow movement of the story provides opportunities to build up to some eerie moments. The problem, continuing a theme for this issue, is the ending. Some of the reader's questions are answered, but most of the answers are old, well-worn paths in science fiction. The emotional arc of the story is decidedly unsatisfying, at least to me. I think I see what Finch was trying to do: there's an attempted undermining of the normal conclusion of this sort of investigation to make a broader point about how to stay engaged in the world. But it lacked triumph and catharsis for me, partly because the revelations that we get are too pedestrian for the build-up they received. It's still an interesting story, but I don't think it entirely worked. (6) "Night Gauntlet" by Walter C. DeBill, Jr., et al.: The full list of authors for this story (Walter C. DeBill, Jr., Richard Gavin, Robert M. Price, W.H. Pugmire, Jeffrey Thomas, and Don Webb) provides one with the first clue that it's gone off the rails. Collaborative storytelling, where each author tries to riff off the work of the previous author while spinning the story in a different direction, is something that I think works much better orally, particularly if you can watch facial expressions while the authors try to stump each other. In written form, it's a recipe for a poorly-told story. That's indeed what we get here. The setup is typical Cthulhu mythos stuff: a strange scientist obsessed with conspiracy theories goes insane, leaving behind an office with a map of linkages between apparently unrelated places. The characters in the story also start going insane for similar reasons, leading up to a typical confrontation with things man was not meant to know, or at least pay attention to. If you like that sort of thing, you may like this story better than I did, but I thought it was shallow and predictable. (3) "Happy Ending 2.0" by James Patrick Kelly: More fantasy, this time of the time travel variety. (I call it fantasy since there's no scientific explanation for the time travel and it plays a pure fantasy role in the story.) That's about as much as I can say without giving away the plot entirely (it's rather short). I can see what Kelly was going for, and I think he was largely successful, but I'm not sure how to react to it. The story felt like it reinforced some rather uncomfortable stereotypes about romantic relationships, and the so-called happy ending struck me as the sort of situation that was going to turn very nasty and very uncomfortable about five or ten pages past where Kelly ended the story. (5) "The Second Kalandar's Tale" by Francis Marion Soty: The main question I have about this story is one that I can't answer without doing more research than I feel like doing right now: how much of this is original to Soty and how much if it is straight from Burton's translation of One Thousand and One Nights. Burton is credited for the story, so I suspect quite a lot of this is from the original. Whether one would be better off just reading the original, or if Soty's retelling adds anything substantial, are good questions that I don't have the background to answer. Taken as a stand-alone story, it's not a bad one. It's a twisting magical adventure involving a djinn, a captive woman, some rather predictable fighting over the woman, and then a subsequent adventure involving physical transformation and a magical battle reminiscent of T.H. White. (Although I have quite likely reversed the order of inspiration if as much of this is straight from Burton as I suspect.) Gender roles, however, are kind of appalling, despite the presence of a stunningly powerful princess, due to the amount of self-sacrifice expected from every woman in the story. Personally, I don't think any of the men in the story are worth anywhere near the amount of loyalty and bravery that the women show. Still, it was reasonably entertaining throughout, in exactly the way that I would expect a One Thousand and One Nights tale to be. Whether there's any point in reading it instead of the original is a question I'll leave to others. (7) "Bodyguard" by Karl Bunker: This is probably the best science fiction of the issue. The first person protagonist is an explorer living with an alien race, partly in order to flee the post-singularity world of uploaded minds and apparent stagnation that Earth has become. It's a personal story that uses his analysis of alien mindsets (and his interaction with his assigned alien bodyguard) to flesh out his own desires, emotional background, and reactions to the world. There are some neat linguistic bits here that I quite enjoyed, although I wish they'd been developed at even more length. (The alien language is more realistic than it might sound; there are some human languages that construct sentences in a vaguely similar way.) It's a sad, elegiac story, but it grew on me. (7) "Botanical Exercises for Curious Girls" by Kali Wallace: One has to count this story as science fiction as well, although for me it had a fantasy tone because the scientific world seems to play by fantasy rules from the perspective of the protagonist. Unpacking that perspective is part of the enjoyment of the story. At the start, she seems to be a disabled girl who is being cared for by a strange succession of nurses who vary by the time of day, but as the story progresses, it becomes clear that something much stranger is going on. There are moments that capture a sense of wonder, reinforced by the persistantly curious and happy narrative voice, but both are undercut by a pervasive sense of danger and dread. This is a light story written about rather dark actions. My biggest complaint with the story is that it doesn't so much end as wander off into the sunset. It set up conflict within a claustrophobic frame, so I can understand the thematic intent of breaking free of that frame, but in the process I felt like the story walked away from all of the questions and structure that it created and ended in a place that felt less alive with potential than formless and oddly pointless. I think I wanted it to stay involved and engaged with the environment it had created. (6) "Ping" by Dixon Wragg: I probably should just skip this, since despite the table of contents billing and the full title introduction, it's not a story. It's a two-line joke. But it's such a bad two-line joke that I had to complain about it. I have no idea why F&SF bothered to reprint it. (1) "The Ifs of Time" by James Stoddard: This certainly fits with the Arabian Nights story in this issue. The timekeeper of a vast and rambling castle (think Gormenghast taken to the extreme) wanders into a story-telling session in a distant part of the castle. The reader gets to listen to four (fairly good) short stories about time, knowledge, and memory, told in four very different genres. All of this does relate to why the timekeeper is there, and the frame story is resolved by the end, but the embedded stories are the best part; each of them is interesting in a different way, and none of them outlast their welcome. This was probably the strongest story of this issue. (7) Rating: 6 out of 10

27 December 2012

Russ Allbery: Review: Science Fiction: The 101 Best Novels: 1985 2010

Review: Science Fiction: The 101 Best Novels: 1985 2010, by Damien Broderick & Paul Di Filippo
Publisher: Nonstop
Copyright: 2012
ISBN: 1-933065-39-7
Format: Trade paperback
Pages: 288
I like book reviews and lists of best novels, as a follower of my reviews probably noticed, so I couldn't resist when this collection made my radar. A follow-up to the earlier Science Fiction: The 101 Best Novels: 1949 1985 by David Pringle (which I have not read), it is a collection of short (two to three pages, generally) reviews of 101 recent SF novels. The date spread is fairly balanced: at least two novels from each year under consideration are featured, and no year gets more than seven or eight. The authors also clearly tried to cover the range of what falls under the science fiction genre, from alternate history through space opera and including novels normally marketed as mainstream, so the result should contain something to everyone's taste. With those characteristics, you may suspect that the "best" part of the title is a bit questionable, and you would be right. "101 of the better novels" would be a more accurate description. While most Hugo and Nebula winners are included here, the section is at times eclectic. But it's eclectic in the spirit of broad inclusiveness: Jumper, Temeraire, or The Hunger Games would normally not be included on this sort of list because they're too popular or "light," but they're here alongside more obscure books (at least for SF readers) like Galatea 2.2, Distance Haze, or My Dirty Little Book of Stolen Time. I doubt anyone will seriously argue that this selection should have replaced the Hugo or Nebula short lists, but one doesn't read review collections like this only to hear about books one already knows about. Those books one has either already read or already chosen not to read. The wide-ranging selection makes it likely that something here will be new to most readers. The authors, Damien Broderick and Paul Di Filippo, are known reviewers in the SF world, and I've read reviews from both of them before. I bought this book largely on the strength of Broderick's name, since I've usually enjoyed his contributions to The New York Review of Science Fiction. Paul Di Filippo was more of a gamble; he's one of the regular reviewers for Asimov's Science Fiction and not one of my favorites. But what I usually disliked about his columns was their focus on obscure small-press titles, graphic novels, and slipstream, so I was hoping that an SF review collection aiming towards the genre mainstream would be more to my taste. The result is mixed. There are things about this selection, and about the reviews, that I enjoyed, and there are other things I found quite annoying. First, the selection. I could (and will in a moment) talk about good and bad selections, but I also have a good statistical metric for at least the alignment of the authors' taste with mine. Of the 107 novels reviewed here (in several case, duologies and trilogies are given single entries), I've read 39, or a little over a third. (Note that I've read every Nebula or Hugo winner in that time period and most of the Hugo nominees, so that will give you a good feel for how broad-ranging this selection is, and how far afield of the normal award slates it goes.) My average rating for those 39 books was 7.49 (including four perfect 10s). By comparison, my average rating for Hugo winners is 6.68 and for Nebula winners is 7.10. There was one 4 (The White Queen) and one 5 (Red Mars), and in both cases I can see why they're here. Of the rest of the books I read I rated them all at least at 6 out of 10. At least among the books I've read, this seems to be a solid selection. Sometimes the details of those selections are odd, though. For example, the authors make an effort to limit the number of selections for each author, a wise choice since they're clearly going by diversity. But if one is operating within that limitation, choosing Ammonite over Slow River for Nicola Griffith, or particularly Ventus over Lady of Mazes or even Permanence for Karl Schroeder, is baffling. There were several similar places where I thought the selection for an author was obscure, minor, or just missed obvious alternatives. Perhaps this was to fill in one of the other breadth criteria, such as balancing number of novels per year or attempting to cover each subgenre. Also, if one is going to divide science fiction and fantasy and try to cover only the science fiction (a division that I think is quite difficult, which is why I don't do it, but it does have the merit of narrowing the field), including The Falling Woman is quite strange. It's a solid book, to be sure, and a Nebula winner. It is also quite straightforward contemporary fantasy involving ghosts and Mayan mythology, without a hint of science-fictional content. Making the protagonists archeologists and scientists doesn't make the book science fiction. The authors try to defend this (unpersuasively to me), but it wasn't the only instance here where I thought their line between science fiction and fantasy was a bit off. That said, there are a lot of great selections here, including books that I love but that aren't frequently picked for this sort of list (The Fortunate Fall, The Time Traveler's Wife, or China Mountain Zhang, for example). It's great to see underappreciated authors like Linda Nagata, Joan Slonczewski, and Karl Schroeder featured. But, of course, one doesn't buy this sort of book just for the list, if for no other reason than that lists aren't copyrightable and one can easily find the complete list of reviewed novels on the Internet (just one example that turned up in a search). Rather, one reads this sort of book for the reviews. And that's where this book moves onto more questionable ground. First, while I realize that everyone has different thresholds for what they consider spoilers and most professional reviewers are more cavalier about them than I am, Broderick and Di Filippo cross any line that I consider reasonable. Most of the reviews are okay, if skirting the limits, but in several places they give away key reveals of books or discuss plot twists right up to, or even including, the ending. The combined review of The Sparrow and Children of God is particularly egregious, containing unambiguous, book-destroying spoilers for The Sparrow. Giving away the ending of Ammonite is only slightly less bad. And those are just two examples I remember. This is not okay. The whole point of this sort of collection is to expose the reader to books they've not yet read but may want to. Proceeding to spoil the book for them in the course of the review is perverse. This alone would make me hesitant to recommend this collection. Second, quite a few of the reviews in this book are, for lack of a better term, emotionally overreaching. Here's an excerpt of a review picked at random (As She Climbed Across the Table by Jonathan Lethem) that will hopefully illustrate:
Lethem's beautifully balanced, metaphorically rich prose propels this blackly jolly fable to a surprising yet satisfying conclusion. By book's end, a sense that the author had accomplished his takeoff taxiing and was now fully in flight for more cosmopolitan cities pervades the pages.
What's "beautifully balanced prose"? Could you recognize it? Does that phrase communicate anything to you other than that the authors liked the book? The whole review collection is written with adjectives and metaphors like this, and after a while it all seems a bit much. It felt like the authors were straining for ways to describe how important or significant the books are and, in the process, lost sight of the basic goal of conveying information about the book. It feels overwrought rather than informative. Even if one is reviewing the books that one considers the pinnacle of achievement in science fiction, a conversational tone with concrete examples and specifics communicates more than impressive but slippery terms like "metaphorically rich." Lest I sound entirely negative, one thing that I did appreciate is that the reviews go to some effort (particularly for their short length) to put the work in the broader context of the field and within the author's oeuvre. Often there's some discussion of previous and subsequent work or related books, and the reviews that feature books from larger series provide good explanations for why those particular books were singled out. Sometimes the number of dangling references was frustrating; authors of these sorts of collections need to remember that most readers will not be as widely read, and reviewing books largely by comparison to other books runs the risk of missing the reader's knowledge entirely. But the reviews convey a real sense of SF as a broad conversation and provide a sense of the breadth and variety of themes and subgenres available. This is one of the fun explorations that this sort of catalog lets the authors and reader do together. Another, more minor, touch that I appreciated was the cover art. Each review leads with an image of the reviewed book's cover (alas, only in black and white for obvious printing reasons). But rather than taking the obvious approach of using the covers of the first releases, or the covers from a particular country, they're chosen from all of the world-wide editions in all their delightful variety. Typical artistic styles for book covers vary drastically between countries, and getting to see a sample of artwork from different markets is a treat. I want to recommend this book. It casts a much broader net than most collections of its kind and provides some needed attention to smaller corners of the genre. I was impressed by the book list before I bought it, and (with the inevitable quibbles) am even more impressed now that I've read it. Broderick and Di Filippo go out of their way to broaden the reader's horizon and open up new avenues for reading, which is one of the best things a review collection can do. But when reviewers don't avoid spoilers, I just can't recommend their work. For me, this is a cardinal sin. Combine that with a writing style that was occasionally overblown and overwritten and the merits don't quite overcome the flaws. I'm glad I read it; it got me excited about reading many books I've already purchased but not gotten to, and I got from it another slew of books to add to my to-purchase list. But I had to read it uncomfortably and lightly, constantly prepared to jump past a review that was too revealing. Rating: 6 out of 10

27 June 2010

Filippo Giunchedi: using git-svn with rsync

From time to time I find myself using git-svn with projects using subversion, this is all nice and fine but it takes quite some time to do the ''initial import'' if you are doing it over a remote transport such as ssh+svn. One trick to speed things up is to use rsync to copy the svn repo locally and then ''git-svn'' it from your harddisk. The idea is originally documented at the allegro wiki For example a typical sourceforge project will be done like this:
DESTDIR=/your/dest
PROJECTNAME=vde
PROJECTURL=https://$PROJECTNAME.svn.sourceforge.net/svnroot/$PROJECTNAME
SVN_COPY=$(mktemp -d)
rsync -c -av $PROJECTNAME.svn.sourceforge.net::svn/$PROJECTNAME/'*' $SVN_COPY
mkdir -p $DESTDIR
cd $DESTDIR
git svn init file://$SVN_COPY --rewrite-root=$PROJECTURL --stdlayout
git svn fetch
# ... git svn imports the repo ...
sed -i 's@url =.*@url = $PROJECTURL' .git/config
rm -rf $SVN_COPY
The important bit is to remember to update the url key after updating otherwise subsequent commits will go to your rsync-ed repo and not the original one!

1 May 2010

Filippo Giunchedi: how to ship spidermonkey in your autotools project

While working on freej we'd need to update spidermonkey/mozjs to the latest version and while doing that integrate it better with the rest of autoconf/automake. AFAICT spidermonkey/mozjs is distributed as part of XULRunner nowadays. The requirements are to make it easy to put a new version of spidermonkey into the tree and redistribute it with make dist and also to have a successful make distcheck. The first step is a trick explained in automake manual about third-party Makefiles so we're using a GNUMakefile.in (git) to "proxy" the targets down to the real Makefile or to ignore some targets called by autotools, e.g:
# otherwise linking static libmozjs.a with dynamic libfreej.so won't work
CFLAGS += -fPIC
# the makefile to proxy targets to
js_makefile = $(builddir)/Makefile
# proxy these targets to the real makefile
all export js-config clean libs tools:
  $(MAKE) -f $(js_makefile) $(AM_MAKEFLAGS) $@
# targets required by autotools but which we don't need at all
.PHONY: dvi pdf ps info html installcheck check install uninstall
dvi pdf ps info html installcheck check install uninstall:
This more or less fixes the make part, now for the autoconf part there's a similar tecnique in the autoconf manual involving running arbitrary configuration commands. I first tried to treat spidermonkey as a separate project via AC_CONFIG_SUBDIRS as expained in Configuring Other Packages in Subdirectories but it didn't work and I forgot exactly why. The configure.ac (git) snippet is something like this: (sorry no highlighting, highlight doesn't support m4 yet)
if test x$have_mozjs = xno   test x$static_mozjs = xyes; then
  dnl run lib/javascript/configure after freej's configure, building it static
  AC_CONFIG_COMMANDS([lib/javascript/.xulrunner-subdir],
    (cd lib/javascript &&
      CXXFLAGS="$GLOBAL_CFLAGS $CXXFLAGS -fPIC" \
      CFLAGS="$GLOBAL_CFLAGS $CFLAGS -fPIC" \
      $ac_srcdir/configure --enable-static \
        --enable-js-static-build --disable-jit)   exit $?
  ])
fi
This will run spidermonkey configure after the main configure, at this point config.status will also expand GNUmakefile.in into GNUmakefile with the correct variables. The final step after all this work is of course to add the spidermonkey directory to SUBDIRS (e.g. SUBDIRS += lib/javascript), after that make will hopefully recurse into lib/javascript and use our GNUmakefile. While this is nice it doesn't cater for the make dist family of autotools targets which build the real distribution tarball. To accomplish this I've used an explicit list of spidermonkey files that will end up in the tarball and made make distdir to copy only those files plus GNUmakefile.in and the list itself (.js-distfiles). Which is the second part of the GNUmakefile.in above, i.e. distdir/distfiles targets plus distclean to clean them up, see comments in the file as well. I hope this is somewhat useful to people who are trying to properly (autotools-wise) ship spidermonkey in their project, comments welcome as usual :)

10 March 2010

Filippo Giunchedi: welcome to ikiwiki

This was long due, I switched from pyblosxom to the marvelous ikiwiki and gave a general revamp to my website by tweaking ikiwiki's default templates. note: the template still needs a general cleanup, don't look at it as it is not pretty (nor are the CSSes) With some luck and mod_rewrite karma nobody's feed readers nor planets will be flooded. Comments are powered by disqus.

16 September 2009

Filippo Giunchedi: RSSes for NEW available on ftp-master

The two RSS feeds previously at http://people.debian.org/~filippo/NEW_rss/ have been moved to a more proper location on ftp-master and the script (tools/queue_rss.py) merged into dak thanks to ganneff.

JFTR the script scans a directory with .changes files (which happens to be NEW in this case) and generates the said feeds, it is not tied to ftp-master and requires python-pyrss2gen and python-debian.

Filippo Giunchedi: NEW queue goes RSS

I wondered for some time since DC6 about having an RSS feed for packages in the NEW queue, finally I have found some time to put together a python script to generate two RSS feeds, one for packages landing in NEW and one for packages leaving NEW.
Caveats: a package leaving NEW doesn't mean it will hit the archive, it could be rejected. Packages by buildds are included. The feeds are generated once an hour.

as usual, comments/ideas/fixes are welcome: filippo (at) debian (dot) org

Filippo Giunchedi: new /home

Its been quite some time since my last entry but this time is well worth writing. Back in September when Italian Debconf took place, Zack offered me a room in his place in Bologna, I happily accepted and since 19 October there's an house packed with DDs (yes, actually two). Are there any other experiences about DDs sharing the same flat? This is going to productive for both, the first product in fact is a nice trick to add a little automation if you are versioning /etc with bzr, see Zack's post for more infos.

Filippo Giunchedi: you don't control your hardware, proprietary drivers do (rant)


Trying to associate with 00:1c:58:10:1d:30 (SSID='ALMAWIFI' freq=2417 MHz)
Associated with 00:1c:58:10:15:b0
^^^^^^^^^^^^^^^^^ WTF?

In other words, my broadcom card won't let me choose the access point to connect to, instead the driver picks one for you (my guess: the one with the best signal) except that autentication on that AP doesn't work.
I'm fine with machines trying by default to be smarter than me, I'm less fine when there's no way to convince said machine to do something supposedly not smart.

Filippo Giunchedi: mozilla ubiquity commands for debian

I'm an avid user of mozilla ubiquity but I couldn't find any commands related to debian to replace those of yubnub. So, prodded by zack I've produced some like bts or pts.
The command feed is located at ubiquity-commands (don't be scared by the "this-code-can-be-evil" subscription page) with the actual code maintained under ubiquity-commands.git.
Note that there's *much* room for improvement so it would be nice to have a common set of commands for the various debian web resources.

13 December 2008

Filippo Giunchedi: don't exit bash on /etc with uncommitted changes

As explained before by Zack, it might happen that you co-administer a machine and have /etc under some VCS (which you should anyway by using etckeeper).
There are a few problems like detecting the identity of the committer and don't leave /etc with uncommitted changes so I recently came up with a slighly different solution than the one we thought two years ago (besides switching from bzr to git).
Take committer name/email from ~/.gitconfig of the user owning the tty or GIT_CONFIG or GIT_CONFIG_LOCAL if they are set.
Trap a function on shell's EXIT rather than loop on .bash_logout which applies only to login shell.
The resulting code to be put in /root/.bashrc:

git_functions="/path/to/git-etc-common"

# export GIT_* variables
if [ -f "$git_functions" ]; then
. "$git_functions"
git_export_env
fi

case $- in
*i*) # interactive shell
check_uncommitted()
if [ -f "$git_functions" ]; then
. "$git_functions"
if ! git_etc_status; then
echo "Uncommitted changes to /etc found, please commit them"
bash -$-
fi
fi

trap check_uncommitted EXIT
;;
esac

Where git-etc-common is to be found here.

28 November 2008

Filippo Giunchedi: remote notification with irssi

I, like many others leave an IRC client always connected on a remote machine and my client of choice is irssi.
One good feature of graphical clients like xchat is the message notification when someone addresses you, which has the nice property to turn IRC from "pull" mode (you have to watch for notifications) to "push" mode (you get notified).
Notifications was the feature I missed the most from irssi so I hacked a solution:
Grab interesting messages from irssi and push them through an ssh tunnel, read them on the local machine and finally display them using libnotify. Implementation results follow:
- rnotify.pl is a script for irssi which pushes interesting messages to a local port
- notify-remote is a shell script to read messages on stdin and display them through notify-send
- esaurito the shell script to put it all together

Next.