Search Results: "toby"

15 October 2023

Russ Allbery: Review: A Killing Frost

Review: A Killing Frost, by Seanan McGuire
Series: October Daye #14
Publisher: DAW
Copyright: 2020
ISBN: 0-7564-1253-6
Format: Kindle
Pages: 351
A Killing Frost is the 14th book in the October Daye urban fantasy series and a direct plot sequel to the events of The Brightest Fell. You definitely cannot start here. This review has some relationship spoilers here for things that you would be expecting after the first five or six books, but which you wouldn't know when reading the first few books of the series. If you haven't started the series yet but plan to, consider skipping this review; if you haven't started reading this series, it will probably be meaningless anyway. Finally, events seem to have slowed, enough trauma has been healed, and Toby is able to seriously consider getting married. However, no sooner is the thought voiced than fae politics injects itself yet again. In order to get married without creating potentially substantial future problems for herself and her family, Toby will have to tie up some loose ends. Since one of those loose ends is a price from the Luidaeg that has been haunting her family for decades, this is easier said than done. The Brightest Fell had a very unsatisfying ending. This, after a two book interlude, is the proper end to that story. I picked this up when I had a bunch of stressful things going on and I wanted to be entertained without having to do much work as a reader. Once again, this series delivered exactly that. The writing is repetitive and a bit clunky, McGuire hammers the same emotional points into the ground, and one does wonder about Toby's tendency to emulate a half-human battering ram, but every book has me engrossed and turning the pages. Everyone should have at least one book series on the go that offers reliable, low-effort entertainment. The initial lever that McGuire uses to push Toby into this plot (fae marriage requirements that had never previously been mentioned) felt rather strained and arbitrary, and I spent the first part of the book grumbling a bit about it. However, there is a better reason for this complication that is revealed with time, and which implies some interesting things about how the fae see heroes and how they use them to solve problems. Now I'm wondering if McGuire will explore that some more in later books. This is the "all is revealed" book about Simon Torquill. As we get later into the series, these "all is revealed" books are coming more frequently. So far, I'm finding the revelations satisfying, which is a lot harder than it looks with a series this long and with this many hidden details. There are a few directions the series is taking that aren't my favorite (the Daoine Sidhe obsession with being the Best Fae is getting a bit boring, for example), but none of them seem egregiously off, and I'm deeply invested in the answers to the remaining questions. Toby hits a personal record here for not explaining the dangerous things she's doing because people might talk her out of it. It makes for a tense and gripping climax, but wow I felt for her friends and family, and substantial parts of that risk seemed unnecessary. This is pointed out to her in no uncertain terms, and I'm wondering if it will finally stick. Toby's tendency to solve complicated problems by bleeding on them is part of what gives this series its charm, but I wouldn't mind her giving other people more of a chance to come up with better plans. I did not like this one as well as the previous two books, mostly because I prefer the Luidaeg-centric stories to the Daoine-Sidhe-centric stories, but if you're enjoying the series to this point, this won't be an exception. It's a substantial improvement on The Brightest Fell and did a lot to salvage that story for me, although there are still some aspects of it that need better explanations. Followed by When Sorrows Come. As usual, there is a novella included in at least the Kindle edition. "Shine in Pearl": I was again hoping for more Gillian, but alas. Instead, and breaking with the tendency for the novellas to be side stories unrelated to the main novel, this fleshes out Simon's past and the other primary relationship driving the novel's plot. It's... fine? The best parts by far are the scenes from Dianda's viewpoint, which are just as refreshingly blunt as Dianda is elsewhere. Neither of the other two characters are favorites of mine, and since the point of the story is to describe the tragedy that is resolved in the plot of the main novel, it's somewhat depressing. Not my favorite of the novellas; not the worst of them. (6) Rating: 7 out of 10

17 July 2023

Russ Allbery: Review: The Unkindest Tide

Review: The Unkindest Tide, by Seanan McGuire
Series: October Daye #13
Publisher: DAW
Copyright: 2019
ISBN: 0-7564-1255-2
Format: Kindle
Pages: 355
This is the 13th book in the long-running October Daye urban fantasy series. There is a strong series arc and a plot that builds on previous books, so this is a series with both significant spoilers and significant confusion if read out of order. The Unkindest Tide in particular is closely tied to the previous book, Night and Silence, and should not be read before it. For once, this story does not start with someone Toby loves being attacked or going missing, or with someone trying to attack her. It instead starts with the Luidaeg. For several books now, she's been dropping hints about the debts that Toby owes to her and the favors she will someday call in. That time is now; she's going to address a problem that she has been putting off for years, and Toby is going to help her. This was a Luidaeg book, so I knew going in that I was going to enjoy it, and indeed I did. That said, I thought it was a bit awkward. This is a moment that McGuire has been building up to for quite some time (series readers may have a few guesses at what the Luidaeg is up to) and she wanted it to land with a lot of force while adding some last-minute complications and letting Toby take her typical role as the stubborn hero. But the primary complication she adds felt unrelated at first, and while the story does tie everything together in the end, it left the story feeling disjointed. The novel needed some fight scenes and some opportunities for people to worry about Toby, so those were provided, but I'm not sure they flowed naturally out of the story. There are some truly great moments here between Toby and the Luidaeg, and between Toby and a few of the other people involved in the story (naming them would be spoilers). Some of the banter is really fun. But I thought the best emotional moment of the book was a underplayed and could have used some more focus, build-up, and time to breathe, rather than shifting the Luidaeg off-stage for a big chunk of the story. McGuire has a bit of a power level problem here and finds some deft ways to avoid it, but I think that hurt the emotional arc. We're also told a lot about the emotional impacts of the plot, but I wish it had been shown more. The one character touchpoint the reader has is mostly uninvolved, and her involvement is a bit disappointing when it does happen. The biggest flaw in this book is the biggest flaw in the series: McGuire has a formula, and while it's more varied from book to book than a lot of long-running series, it's still predictable. Toby and Tybalt say the same things, Toby gets into the same kind of trouble, she worries about the same things, the banter goes in predictable ways, and the fights go very similarly to the fights in previous series entries. A lot of the novel won't be very surprising. There are always a few moments in each book that stir strong emotions, and personally I love the world-building and am eager to read more of it, but there are also sections of these books that I read very fast without feeling like I'm missing much. I've said this about this series before, which is another angle at the same problem. I think the world-building is getting more interesting as the series goes along, but the writing is not; if anything, it's getting a bit worse. It's far from bad enough to stop me from reading, and this was one of the better series entries overall, but it does keep the series firmly in the "reliable fun" category rather than the "excellent and worthy of savoring" category. It's still reliable fun, though. If you're this far along, you'll want to keep reading. The Unkindest Tide felt a bit like the end of an arc, so I'm curious to see where the story goes next. Followed by A Killing Frost. As is typical of the last few books, there is a novella included at the end. "Hope Is Swift": The novella at the end of Night and Silence was the best part of that book and focused on exactly the character I was hoping it would focus on. At the end of this book, I really wanted another novella about Gillian and directly related to the main novel. Unfortunately, what we get instead is a story about Raj that is entirely unrelated to the rest of the book. Raj is fine as a supporting character, I suppose, but he's not one of the series characters I care very much about, and sadly this novella did nothing to change my mind. It's a story of teenagers and of the perils of fae/human interactions that I found rather predictable, a bit anxiety-inducing (constant discussion of medical procedures), and entirely forgettable apart from the bits I didn't like. The takeaway lessons felt so obvious that they could have been the moral at the end of a Saturday morning cartoon from my youth. Maybe if you're a cat person you'd get a bit more out of it? I kind of regretted reading it, sadly. (4) Rating: 7 out of 10

6 March 2023

Vincent Bernat: DDoS detection and remediation with Akvorado and Flowspec

Akvorado collects sFlow and IPFIX flows, stores them in a ClickHouse database, and presents them in a web console. Although it lacks built-in DDoS detection, it s possible to create one by crafting custom ClickHouse queries.

DDoS detection Let s assume we want to detect DDoS targeting our customers. As an example, we consider a DDoS attack as a collection of flows over one minute targeting a single customer IP address, from a single source port and matching one of these conditions:
  • an average bandwidth of 1 Gbps,
  • an average bandwidth of 200 Mbps when the protocol is UDP,
  • more than 20 source IP addresses and an average bandwidth of 100 Mbps, or
  • more than 10 source countries and an average bandwidth of 100 Mbps.
Here is the SQL query to detect such attacks over the last 5 minutes:
SELECT *
FROM (
  SELECT
    toStartOfMinute(TimeReceived) AS TimeReceived,
    DstAddr,
    SrcPort,
    dictGetOrDefault('protocols', 'name', Proto, '???') AS Proto,
    SUM(((((Bytes * SamplingRate) * 8) / 1000) / 1000) / 1000) / 60 AS Gbps,
    uniq(SrcAddr) AS sources,
    uniq(SrcCountry) AS countries
  FROM flows
  WHERE TimeReceived > now() - INTERVAL 5 MINUTE
    AND DstNetRole = 'customers'
  GROUP BY
    TimeReceived,
    DstAddr,
    SrcPort,
    Proto
)
WHERE (Gbps > 1)
   OR ((Proto = 'UDP') AND (Gbps > 0.2)) 
   OR ((sources > 20) AND (Gbps > 0.1)) 
   OR ((countries > 10) AND (Gbps > 0.1))
ORDER BY
  TimeReceived DESC,
  Gbps DESC
Here is an example output1 where two of our users are under attack. One from what looks like an NTP amplification attack, the other from a DNS amplification attack:
TimeReceived DstAddr SrcPort Proto Gbps sources countries
2023-02-26 17:44:00 ::ffff:203.0.113.206 123 UDP 0.102 109 13
2023-02-26 17:43:00 ::ffff:203.0.113.206 123 UDP 0.130 133 17
2023-02-26 17:43:00 ::ffff:203.0.113.68 53 UDP 0.129 364 63
2023-02-26 17:43:00 ::ffff:203.0.113.206 123 UDP 0.113 129 21
2023-02-26 17:42:00 ::ffff:203.0.113.206 123 UDP 0.139 50 14
2023-02-26 17:42:00 ::ffff:203.0.113.206 123 UDP 0.105 42 14
2023-02-26 17:40:00 ::ffff:203.0.113.68 53 UDP 0.121 340 65

DDoS remediation Once detected, there are at least two ways to stop the attack at the network level:
  • blackhole the traffic to the targeted user (RTBH), or
  • selectively drop packets matching the attack patterns (Flowspec).

Traffic blackhole The easiest method is to sacrifice the attacked user. While this helps the attacker, this protects your network. It is a method supported by all routers. You can also offload this protection to many transit providers. This is useful if the attack volume exceeds your internet capacity. This works by advertising with BGP a route to the attacked user with a specific community. The border router modifies the next hop address of these routes to a specific IP address configured to forward the traffic to a null interface. RFC 7999 defines 65535:666 for this purpose. This is known as a remote-triggered blackhole (RTBH) and is explained in more detail in RFC 3882. It is also possible to blackhole the source of the attacks by leveraging unicast Reverse Path Forwarding (uRPF) from RFC 3704, as explained in RFC 5635. However, uRPF can be a serious tax on your router resources. See NCS5500 uRPF: Configuration and Impact on Scale for an example of the kind of restrictions you have to expect when enabling uRPF. On the advertising side, we can use BIRD. Here is a complete configuration file to allow any router to collect them:
log stderr all;
router id 192.0.2.1;
protocol device  
  scan time 10;
 
protocol bgp exporter  
  ipv4  
    import none;
    export where proto = "blackhole4";
   ;
  ipv6  
    import none;
    export where proto = "blackhole6";
   ;
  local as 64666;
  neighbor range 192.0.2.0/24 external;
  multihop;
  dynamic name "exporter";
  dynamic name digits 2;
  graceful restart yes;
  graceful restart time 0;
  long lived graceful restart yes;
  long lived stale time 3600;  # keep routes for 1 hour!
 
protocol static blackhole4  
  ipv4;
  route 203.0.113.206/32 blackhole  
    bgp_community.add((65535, 666));
   ;
  route 203.0.113.68/32 blackhole  
    bgp_community.add((65535, 666));
   ;
 
protocol static blackhole6  
  ipv6;
 
We use BGP long-lived graceful restart to ensure routes are kept for one hour, even if the BGP connection goes down, notably during maintenance. On the receiver side, if you have a Cisco router running IOS XR, you can use the following configuration to blackhole traffic received on the BGP session. As the BGP session is dedicated to this usage, The community is not used, but you can also forward these routes to your transit providers.
router static
 vrf public
  address-family ipv4 unicast
   192.0.2.1/32 Null0 description "BGP blackhole"
  !
  address-family ipv6 unicast
   2001:db8::1/128 Null0 description "BGP blackhole"
  !
 !
!
route-policy blackhole_ipv4_in_public
  if destination in (0.0.0.0/0 le 31) then
    drop
  endif
  set next-hop 192.0.2.1
  done
end-policy
!
route-policy blackhole_ipv6_in_public
  if destination in (::/0 le 127) then
    drop
  endif
  set next-hop 2001:db8::1
  done
end-policy
!
router bgp 12322
 neighbor-group BLACKHOLE_IPV4_PUBLIC
  remote-as 64666
  ebgp-multihop 255
  update-source Loopback10
  address-family ipv4 unicast
   maximum-prefix 100 90
   route-policy blackhole_ipv4_in_public in
   route-policy drop out
   long-lived-graceful-restart stale-time send 86400 accept 86400
  !
  address-family ipv6 unicast
   maximum-prefix 100 90
   route-policy blackhole_ipv6_in_public in
   route-policy drop out
   long-lived-graceful-restart stale-time send 86400 accept 86400
  !
 !
 vrf public
  neighbor 192.0.2.1
   use neighbor-group BLACKHOLE_IPV4_PUBLIC
   description akvorado-1
When the traffic is blackholed, it is still reported by IPFIX and sFlow. In Akvorado, use ForwardingStatus >= 128 as a filter. While this method is compatible with all routers, it makes the attack successful as the target is completely unreachable. If your router supports it, Flowspec can selectively filter flows to stop the attack without impacting the customer.

Flowspec Flowspec is defined in RFC 8955 and enables the transmission of flow specifications in BGP sessions. A flow specification is a set of matching criteria to apply to IP traffic. These criteria include the source and destination prefix, the IP protocol, the source and destination port, and the packet length. Each flow specification is associated with an action, encoded as an extended community: traffic shaping, traffic marking, or redirection. To announce flow specifications with BIRD, we extend our configuration. The extended community used shapes the matching traffic to 0 bytes per second.
flow4 table flowtab4;
flow6 table flowtab6;
protocol bgp exporter  
  flow4  
    import none;
    export where proto = "flowspec4";
   ;
  flow6  
    import none;
    export where proto = "flowspec6";
   ;
  # [ ]
 
protocol static flowspec4  
  flow4;
  route flow4  
    dst 203.0.113.68/32;
    sport = 53;
    length >= 1476 && <= 1500;
    proto = 17;
   
    bgp_ext_community.add((generic, 0x80060000, 0x00000000));
   ;
  route flow4  
    dst 203.0.113.206/32;
    sport = 123;
    length = 468;
    proto = 17;
   
    bgp_ext_community.add((generic, 0x80060000, 0x00000000));
   ;
 
protocol static flowspec6  
  flow6;
 
If you have a Cisco router running IOS XR, the configuration may look like this:
vrf public
 address-family ipv4 flowspec
 address-family ipv6 flowspec
!
router bgp 12322
 address-family vpnv4 flowspec
 address-family vpnv6 flowspec
 neighbor-group FLOWSPEC_IPV4_PUBLIC
  remote-as 64666
  ebgp-multihop 255
  update-source Loopback10
  address-family ipv4 flowspec
   long-lived-graceful-restart stale-time send 86400 accept 86400
   route-policy accept in
   route-policy drop out
   maximum-prefix 100 90
   validation disable
  !
  address-family ipv6 flowspec
   long-lived-graceful-restart stale-time send 86400 accept 86400
   route-policy accept in
   route-policy drop out
   maximum-prefix 100 90
   validation disable
  !
 !
 vrf public
  address-family ipv4 flowspec
  address-family ipv6 flowspec
  neighbor 192.0.2.1
   use neighbor-group FLOWSPEC_IPV4_PUBLIC
   description akvorado-1
Then, you need to enable Flowspec on all interfaces with:
flowspec
 vrf public
  address-family ipv4
   local-install interface-all
  !
  address-family ipv6
   local-install interface-all
  !
 !
!
As with the RTBH setup, you can filter dropped flows with ForwardingStatus >= 128.

DDoS detection (continued) In the example using Flowspec, the flows were also filtered on the length of the packet:
route flow4  
  dst 203.0.113.68/32;
  sport = 53;
  length >= 1476 && <= 1500;
  proto = 17;
 
  bgp_ext_community.add((generic, 0x80060000, 0x00000000));
 ;
This is an important addition: legitimate DNS requests are smaller than this and therefore not filtered.2 With ClickHouse, you can get the 10th and 90th percentiles of the packet sizes with quantiles(0.1, 0.9)(Bytes/Packets). The last issue we need to tackle is how to optimize the request: it may need several seconds to collect the data and it is likely to consume substantial resources from your ClickHouse database. One solution is to create a materialized view to pre-aggregate results:
CREATE TABLE ddos_logs (
  TimeReceived DateTime,
  DstAddr IPv6,
  Proto UInt32,
  SrcPort UInt16,
  Gbps SimpleAggregateFunction(sum, Float64),
  Mpps SimpleAggregateFunction(sum, Float64),
  sources AggregateFunction(uniqCombined(12), IPv6),
  countries AggregateFunction(uniqCombined(12), FixedString(2)),
  size AggregateFunction(quantiles(0.1, 0.9), UInt64)
) ENGINE = SummingMergeTree
PARTITION BY toStartOfHour(TimeReceived)
ORDER BY (TimeReceived, DstAddr, Proto, SrcPort)
TTL toStartOfHour(TimeReceived) + INTERVAL 6 HOUR DELETE ;
CREATE MATERIALIZED VIEW ddos_logs_view TO ddos_logs AS
  SELECT
    toStartOfMinute(TimeReceived) AS TimeReceived,
    DstAddr,
    Proto,
    SrcPort,
    sum(((((Bytes * SamplingRate) * 8) / 1000) / 1000) / 1000) / 60 AS Gbps,
    sum(((Packets * SamplingRate) / 1000) / 1000) / 60 AS Mpps,
    uniqCombinedState(12)(SrcAddr) AS sources,
    uniqCombinedState(12)(SrcCountry) AS countries,
    quantilesState(0.1, 0.9)(toUInt64(Bytes/Packets)) AS size
  FROM flows
  WHERE DstNetRole = 'customers'
  GROUP BY
    TimeReceived,
    DstAddr,
    Proto,
    SrcPort
The ddos_logs table is using the SummingMergeTree engine. When the table receives new data, ClickHouse replaces all the rows with the same sorting key, as defined by the ORDER BY directive, with one row which contains summarized values using either the sum() function or the explicitly specified aggregate function (uniqCombined and quantiles in our example).3 Finally, we can modify our initial query with the following one:
SELECT *
FROM (
  SELECT
    TimeReceived,
    DstAddr,
    dictGetOrDefault('protocols', 'name', Proto, '???') AS Proto,
    SrcPort,
    sum(Gbps) AS Gbps,
    sum(Mpps) AS Mpps,
    uniqCombinedMerge(12)(sources) AS sources,
    uniqCombinedMerge(12)(countries) AS countries,
    quantilesMerge(0.1, 0.9)(size) AS size
  FROM ddos_logs
  WHERE TimeReceived > now() - INTERVAL 60 MINUTE
  GROUP BY
    TimeReceived,
    DstAddr,
    Proto,
    SrcPort
)
WHERE (Gbps > 1)
   OR ((Proto = 'UDP') AND (Gbps > 0.2)) 
   OR ((sources > 20) AND (Gbps > 0.1)) 
   OR ((countries > 10) AND (Gbps > 0.1))
ORDER BY
  TimeReceived DESC,
  Gbps DESC

Gluing everything together To sum up, building an anti-DDoS system requires to following these steps:
  1. define a set of criteria to detect a DDoS attack,
  2. translate these criteria into SQL requests,
  3. pre-aggregate flows into SummingMergeTree tables,
  4. query and transform the results to a BIRD configuration file, and
  5. configure your routers to pull the routes from BIRD.
A Python script like the following one can handle the fourth step. For each attacked target, it generates both a Flowspec rule and a blackhole route.
import socket
import types
from clickhouse_driver import Client as CHClient
# Put your SQL query here!
SQL_QUERY = " "
# How many anti-DDoS rules we want at the same time?
MAX_DDOS_RULES = 20
def empty_ruleset():
    ruleset = types.SimpleNamespace()
    ruleset.flowspec = types.SimpleNamespace()
    ruleset.blackhole = types.SimpleNamespace()
    ruleset.flowspec.v4 = []
    ruleset.flowspec.v6 = []
    ruleset.blackhole.v4 = []
    ruleset.blackhole.v6 = []
    return ruleset
current_ruleset = empty_ruleset()
client = CHClient(host="clickhouse.akvorado.net")
while True:
    results = client.execute(SQL_QUERY)
    seen =  
    new_ruleset = empty_ruleset()
    for (t, addr, proto, port, gbps, mpps, sources, countries, size) in results:
        if (addr, proto, port) in seen:
            continue
        seen[(addr, proto, port)] = True
        # Flowspec
        if addr.ipv4_mapped:
            address = addr.ipv4_mapped
            rules = new_ruleset.flowspec.v4
            table = "flow4"
            mask = 32
            nh = "proto"
        else:
            address = addr
            rules = new_ruleset.flowspec.v6
            table = "flow6"
            mask = 128
            nh = "next header"
        if size[0] == size[1]:
            length = f"length =  int(size[0]) "
        else:
            length = f"length >=  int(size[0])  && <=  int(size[1]) "
        header = f"""
# Time:  t 
# Source:  address , protocol:  proto , port:  port 
# Gbps/Mpps:  gbps:.3 / mpps:.3 , packet size:  int(size[0]) <=X<= int(size[1]) 
# Flows:  flows , sources:  sources , countries:  countries 
"""
        rules.append(
                f""" header 
route  table   
  dst  address / mask ;
  sport =  port ;
   length ;
   nh  =  socket.getprotobyname(proto) ;
 
  bgp_ext_community.add((generic, 0x80060000, 0x00000000));
 ;
"""
        )
        # Blackhole
        if addr.ipv4_mapped:
            rules = new_ruleset.blackhole.v4
        else:
            rules = new_ruleset.blackhole.v6
        rules.append(
            f""" header 
route  address / mask  blackhole  
  bgp_community.add((65535, 666));
 ;
"""
        )
        new_ruleset.flowspec.v4 = list(
            set(new_ruleset.flowspec.v4[:MAX_DDOS_RULES])
        )
        new_ruleset.flowspec.v6 = list(
            set(new_ruleset.flowspec.v6[:MAX_DDOS_RULES])
        )
        # TODO: advertise changes by mail, chat, ...
        current_ruleset = new_ruleset
        changes = False
        for rules, path in (
            (current_ruleset.flowspec.v4, "v4-flowspec"),
            (current_ruleset.flowspec.v6, "v6-flowspec"),
            (current_ruleset.blackhole.v4, "v4-blackhole"),
            (current_ruleset.blackhole.v6, "v6-blackhole"),
        ):
            path = os.path.join("/etc/bird/", f" path .conf")
            with open(f" path .tmp", "w") as f:
                for r in rules:
                    f.write(r)
            changes = (
                changes or not os.path.exists(path) or not samefile(path, f" path .tmp")
            )
            os.rename(f" path .tmp", path)
        if not changes:
            continue
        proc = subprocess.Popen(
            ["birdc", "configure"],
            stdin=subprocess.DEVNULL,
            stdout=subprocess.PIPE,
            stderr=subprocess.PIPE,
        )
        stdout, stderr = proc.communicate(None)
        stdout = stdout.decode("utf-8", "replace")
        stderr = stderr.decode("utf-8", "replace")
        if proc.returncode != 0:
            logger.error(
                "  error:\n \n ".format(
                    "birdc reconfigure",
                    "\n".join(
                        [" O:  ".format(line) for line in stdout.rstrip().split("\n")]
                    ),
                    "\n".join(
                        [" E:  ".format(line) for line in stderr.rstrip().split("\n")]
                    ),
                )
            )

Until Akvorado integrates DDoS detection and mitigation, the ideas presented in this blog post provide a solid foundation to get started with your own anti-DDoS system.

  1. ClickHouse can export results using Markdown format when appending FORMAT Markdown to the query.
  2. While most DNS clients should retry with TCP on failures, this is not always the case: until recently, musl libc did not implement this.
  3. The materialized view also aggregates the data at hand, both for efficiency and to ensure we work with the right data types.

17 January 2023

Russ Allbery: Review: Night and Silence

Review: Night and Silence, by Seanan McGuire
Series: October Daye #12
Publisher: DAW Books
Copyright: 2018
ISBN: 0-698-18353-3
Format: Kindle
Pages: 353
Night and Silence is the 12th book in Seanan McGuire's long-running October Daye Celtic-inspired urban fantasy series. This is a "read the books in order" sort of series; you definitely do not want to start here. Gillian, Toby's estranged daughter, has been kidnapped. Her ex-husband and his new wife turn to her in desperation (although Miranda suspects Toby is the one who kidnapped her). Toby of course drops everything to find her, which she would have done regardless, but the obvious fear is that Gillian may have been kidnapped specifically to get at Toby. Meanwhile, the consequences of The Brightest Fell have put a severe strain on one of Toby's most important relationships, at the worst possible time. Once again, this is when I say that McGuire's writing has a lot of obvious flaws, and then say that this book kept me up way past my bedtime for two nights in a row because it was nearly impossible to put down. The primary quality flaw in these books, at least for me, is that Toby's thought processes have some deeply-worn grooves that she falls into time and time again. Since she's the first-person narrator of the series, that produces some repetitive writing. She feels incredibly lucky for her chosen family, she worries about her friends, she prizes loyalty very highly, she throws herself headlong into danger, and she thinks about these things in a background hum through every book. By this point, the reader knows all of this, so there's a lot of "yes, yes, we know" muttering that tends to happen. McGuire also understands the importance of plot and character recaps at the start of each book for those of us who have forgotten what was happening (thank you!) but awkwardly writes them into the beginning of each book. That doesn't help with the sense of repetitiveness. If only authors would write stand-alone synopses of previous books in a series, or hire someone to do that if they can't stand to, the world would be a much better place. But now I'm repeating myself. Once I get into a book, though, this doesn't matter at all. When Toby starts down a familiar emotional rut, I just read faster until I get to the next bit. Something about these books is incredibly grabby to me; once I get started on one, I devour it, and usually want to read the next one as soon as possible. Some of this is the cast, which at this point in the series is varied, entertaining, and full of the sort of casual banter that only people who have known each other for a long time can do. A lot of it is that Toby constantly moves forward. She ruminates and angsts and worries, but she never sits around and mopes. She does her thinking on the move. No matter how preoccupied she is with some emotional thread, something's going to happen on the next page. Some of it is intangible, or at least beyond my ability to put a finger on. Some authors are good at writing grabby books, and at least for me McGuire is one of those authors. Describing the plot in any detail without spoilers is hopeless this far into the series, but this is one of the big revelation books, and I suspect it's also going to be a significant tipping point in Toby's life. We finally find out what broke faerie, which has rather more to do with Toby and her family than one might have expected, explains some things about Amandine, and also (although this hasn't been spelled out yet) seems likely to explain some things about the Luidaeg's involvement in Toby's adventures. And there is another significant realignment of one of Toby's relationships that isn't fully developed here, but that I hope will be explored in future books. There's also a lot about Tybalt that to be honest I found tedious and kind of frustrating (although not half as frustrating as Toby found it). I appreciate what McGuire was doing; some problems are tedious, frustrating, and repetitive because that's how one gets through them. The problem that Toby and Tybalt are wrestling with is realistic and underdiscussed in fiction of this type, so I respect the effort, and I'm not sure there was way to write through this that would have been more engrossing (and a bit less cliched). But, still, not my favorite part of this book. Thankfully, it was a mostly-ignorable side thread. This was a substantial improvement over The Brightest Fell, which was both infuriating and on rails. Toby has much more agency here, the investigation was more interesting, and the lore and character fallout has me eager to read the next book. It's fun to see McGuire's world-building come together over this long of a series, and so far it has not disappointed. Followed by The Unkindest Tide. As has become usual, the book ends with a novella telling a story from a different perspective. "Suffer a Sea-Change": The main novel was good. This was great. "Suffer a Sea-Change" retells a critical part of the novel from Gillian's perspective, and then follows that thread of story past the end of the novel. I loved absolutely everything about this. Gillian is a great protagonist, similar to Toby but different enough to be a fresh voice. There is a ton of the Luidaeg, and through different eyes than Toby's which is fun. There's some great world-building and a few very memorable scenes. And there's also the beginning, the very small beginning, of the healing of a great injustice that's been haunting this series since the very beginning, and I am very much here for that. Great stuff. (9) Rating: 8 out of 10

20 September 2022

Matthew Garrett: Handling WebAuthn over remote SSH connections

Being able to SSH into remote machines and do work there is great. Using hardware security tokens for 2FA is also great. But trying to use them both at the same time doesn't work super well, because if you hit a WebAuthn request on the remote machine it doesn't matter how much you mash your token - it's not going to work.

But could it?

The SSH agent protocol abstracts key management out of SSH itself and into a separate process. When you run "ssh-add .ssh/id_rsa", that key is being loaded into the SSH agent. When SSH wants to use that key to authenticate to a remote system, it asks the SSH agent to perform the cryptographic signatures on its behalf. SSH also supports forwarding the SSH agent protocol over SSH itself, so if you SSH into a remote system then remote clients can also access your keys - this allows you to bounce through one remote system into another without having to copy your keys to those remote systems.

More recently, SSH gained the ability to store SSH keys on hardware tokens such as Yubikeys. If configured appropriately, this means that even if you forward your agent to a remote site, that site can't do anything with your keys unless you physically touch the token. But out of the box, this is only useful for SSH keys - you can't do anything else with this support.

Well, that's what I thought, at least. And then I looked at the code and realised that SSH is communicating with the security tokens using the same library that a browser would, except it ensures that any signature request starts with the string "ssh:" (which a genuine WebAuthn request never will). This constraint can actually be disabled by passing -O no-restrict-websafe to ssh-agent, except that was broken until this weekend. But let's assume there's a glorious future where that patch gets backported everywhere, and see what we can do with it.

First we need to load the key into the security token. For this I ended up hacking up the Go SSH agent support. Annoyingly it doesn't seem to be possible to make calls to the agent without going via one of the exported methods here, so I don't think this logic can be implemented without modifying the agent module itself. But this is basically as simple as adding another key message type that looks something like:
type ecdsaSkKeyMsg struct  
       Type        string  sshtype:"17 25" 
       Curve       string
       PubKeyBytes []byte
       RpId        string
       Flags       uint8
       KeyHandle   []byte
       Reserved    []byte
       Comments    string
       Constraints []byte  ssh:"rest" 
 
Where Type is ssh.KeyAlgoSKECDSA256, Curve is "nistp256", RpId is the identity of the relying party (eg, "webauthn.io"), Flags is 0x1 if you want the user to have to touch the key, KeyHandle is the hardware token's representation of the key (basically an opaque blob that's sufficient for the token to regenerate the keypair - this is generally stored by the remote site and handed back to you when it wants you to authenticate). The other fields can be ignored, other than PubKeyBytes, which is supposed to be the public half of the keypair.

This causes an obvious problem. We have an opaque blob that represents a keypair. We don't have the public key. And OpenSSH verifies that PubKeyByes is a legitimate ecdsa public key before it'll load the key. Fortunately it only verifies that it's a legitimate ecdsa public key, and does nothing to verify that it's related to the private key in any way. So, just generate a new ECDSA key (ecdsa.GenerateKey(elliptic.P256(), rand.Reader)) and marshal it ( elliptic.Marshal(ecKey.Curve, ecKey.X, ecKey.Y)) and we're good. Pass that struct to ssh.Marshal() and then make an agent call.

Now you can use the standard agent interfaces to trigger a signature event. You want to pass the raw challenge (not the hash of the challenge!) - the SSH code will do the hashing itself. If you're using agent forwarding this will be forwarded from the remote system to your local one, and your security token should start blinking - touch it and you'll get back an ssh.Signature blob. ssh.Unmarshal() the Blob member to a struct like
type ecSig struct  
        R *big.Int
        S *big.Int
 
and then ssh.Unmarshal the Rest member to
type authData struct  
        Flags    uint8
        SigCount uint32
 
The signature needs to be converted back to a DER-encoded ASN.1 structure (eg,
var b cryptobyte.Builder
b.AddASN1(asn1.SEQUENCE, func(b *cryptobyte.Builder)  
        b.AddASN1BigInt(ecSig.R)
        b.AddASN1BigInt(ecSig.S)
 )
signatureDER, _ := b.Bytes()
, and then you need to construct the Authenticator Data structure. For this, take the RpId used earlier and generate the sha256. Append the one byte Flags variable, and then convert SigCount to big endian and append those 4 bytes. You should now have a 37 byte structure. This needs to be CBOR encoded (I used github.com/fxamacker/cbor and just called cbor.Marshal(data, cbor.EncOptions )).

Now base64 encode the sha256 of the challenge data, the DER-encoded signature and the CBOR-encoded authenticator data and you've got everything you need to provide to the remote site to satisfy the challenge.

There are alternative approaches - you can use USB/IP to forward the hardware token directly to the remote system. But that means you can't use it locally, so it's less than ideal. Or you could implement a proxy that communicates with the key locally and have that tunneled through to the remote host, but at that point you're just reinventing ssh-agent.

And you should bear in mind that the default behaviour of blocking this sort of request is for a good reason! If someone is able to compromise a remote system that you're SSHed into, they can potentially trick you into hitting the key to sign a request they've made on behalf of an arbitrary site. Obviously they could do the same without any of this if they've compromised your local system, but there is some additional risk to this. It would be nice to have sensible MAC policies that default-denied access to the SSH agent socket and only allowed trustworthy binaries to do so, or maybe have some sort of reasonable flatpak-style portal to gate access. For my threat model I think it's a worthwhile security tradeoff, but you should evaluate that carefully yourself.

Anyway. Now to figure out whether there's a reasonable way to get browsers to work with this.

comment count unavailable comments

16 January 2022

Russ Allbery: Review: The Brightest Fell

Review: The Brightest Fell, by Seanan McGuire
Series: October Daye #11
Publisher: DAW
Copyright: 2017
ISBN: 0-698-18352-5
Format: Kindle
Pages: 353
This is the eleventh book in the October Daye urban fantasy series, not counting various novellas and side stories. You really cannot start here, particularly given how many ties this book has to the rest of the series. I would like to claim there's some sort of plan or strategy in how I read long series, but there are just a lot of books to read and then I get distracted and three years have gone by. The advantage of those pauses, at least for writing reviews, is that I return to the series with fresh eyes and more points of comparison. My first thought this time around was "oh, these books aren't that well written, are they," followed shortly thereafter by staying up past midnight reading just one more chapter. Plot summaries are essentially impossible this deep into a series, when even the names of the involved characters can be a bit of a spoiler. What I can say is that we finally get the long-awaited confrontation between Toby and her mother, although it comes in an unexpected (and unsatisfying) form. This fills in a few of the gaps in Toby's childhood, although there's not much there we didn't already know. It fills in considerably more details about the rest of Toby's family, most notably her pure-blood sister. The writing is indeed not great. This series is showing some of the signs I've seen in other authors (Mercedes Lackey, for instance) who wrote too many books per year to do each of them justice. I have complained before about McGuire's tendency to reuse the same basic plot structure, and this instance seemed particularly egregious. The book opens with Toby enjoying herself and her found family, feeling like they can finally relax. Then something horrible happens to people she cares about, forcing her to go solve the problem. This in theory requires her to work out some sort of puzzle, but in practice is fairly linear and obvious because, although I love Toby as a character, she can't puzzle her way out of a wet sack. Everything is (mostly) fixed in the end, but there's a high cost to pay, and everyone ends the book with more trauma. The best books of this series are the ones where McGuire manages to break with this formula. This is not one of them. The plot is literally on magical rails, since The Brightest Fell skips even pretending that Toby is an actual detective (although it establishes that she's apparently still working as one in the human world, a detail that I find baffling) and gives her a plot compass that tells her where to go. I don't really mind this since I read this series for emotional catharsis rather than Toby's ingenuity, but alas that's mostly missing here as well. There is a resolution of sorts, but it's the partial and conditional kind that doesn't include awful people getting their just deserts. This is also not a good series entry for world-building. McGuire has apparently been dropping hints for this plot back at least as far as Ashes of Honor. I like that sort of long-term texture to series like this, but the unfortunate impact on this book is a lot of revisiting of previous settings and very little in the way of new world-building. The bit with the pixies was very good; I wanted more of that, not the trip to an Ashes of Honor setting to pick up a loose end, or yet another significant scene in Borderland Books. As an aside, I wish authors would not put real people into their books as characters, even when it's with permission as I'm sure it was here. It's understandable to write a prominent local business into a story as part of the local color (although even then I would rather it not be a significant setting in the story), but having the actual owner and staff show up, even in brief cameos, feels creepy and weird to me. It also comes with some serious risks because real people are not characters under the author's control. (All the content warnings for that link, which is a news story from three years after this book was published.) So, with all those complaints, why did I stay up late reading just one more chapter? Part of the answer is that McGuire writes very grabby books, at least for me. Toby is a full-speed-ahead character who is constantly making things happen, and although the writing in this book had more than the usual amount of throat-clearing and rehashing of the same internal monologue, the plot still moved along at a reasonable clip. Another part of the answer is that I am all-in on these characters: I like them, I want them to be happy, and I want to know what's going to happen next. It helps that McGuire has slowly added characters over the course of a long series and given most of them a chance to shine. It helps even more that I like all of them as people, and I like the style of banter that McGuire writes. Also, significant screen time for the Luidaeg is never a bad thing. I think this was the weakest entry in the series in a while. It wrapped up some loose ends that I wasn't that interested in wrapping up, introduced a new conflict that it doesn't resolve, spent a bunch of time with a highly unpleasant character I didn't enjoy reading about, didn't break much new world-building ground, and needed way more faerie court politics. But some of the banter was excellent, the pixies and the Luidaeg were great, and I still care a lot about these characters. I am definitely still reading. Followed by Nights and Silences. Continuing a pattern from Once Broken Faith, the ebook version of The Brightest Fell includes a bonus novella. (I'm not sure if it's also present in the print version.) "Of Things Unknown": As is usual for the short fiction in this series, this is a side story from the perspective of someone other than Toby. In this case, that's April O'Leary, first introduced all the way back in A Local Habitation, and the novella focuses on loose ends from that novel. Loose ends are apparently the theme of this book. This was... fine. I like April, I enjoyed reading a story from her perspective, and I'm always curious to see how Toby looks from the outside. I thought the plot was strained and the resolution a bit too easy and painless, and I was not entirely convinced by April's internal thought processes. It felt like McGuire left some potential for greater plot complications on the table here, and I found it hard to shake the impression that this story was patching an error that McGuire felt she'd made in the much earlier novel. But it was nice to have an unambiguously happy ending after the more conditional ending of the main story. (6) Rating: 6 out of 10

12 June 2020

Ulrike Uhlig: The right to demand change

Two women sit in an office, one asks: "What's the difference between being assertive and being aggressive?" The other replies: "Your gender." (Cartoon by Judy Horacek, 1999.) When a person of a marginalized group (read: a person with less privilege, a person with lower rank) is being framed and blamed as being aggressive, she is being told that her behavior is unacceptable. Marginalized people have learnt that they need to comply to fit, and are likely to suppress their feelings. By being framed as aggressive, the marginalized person is also being told that what they are saying cannot be listened to because the way they are saying it does not comply with expectations. There is a word for this: tone policing. This great comic by Robot Hugs has all the important details. Tone policing is a silencing tactic in which privileged participants of a discussion one-sidedly define the terms of the conversation. This tactic has the interesting side effect of shifting the responsibility to prove that one is not aggressive, hostile, explosive, a minefield, etc. to the person being framed and blamed - proving that one is worthy to be listened to. (Some of those words are actual quotes taken from real life.) Years ago, I worked in a company in which my female developer colleague would put herself in a state of overly expressed sorriness, all the while pretending to be stupid and helpless whenever she needed to ask anything from the sysadmins. When I confronted her with that, she replied: "I do it because it works." In the same company, another woman who generally asked assertively for what she needed ended up being insulted by one of the project managers using the word "dominatrix". While the example comes from my own experience, this kind of thing happens across any oppression/privilege boundaries. In some conversations, be they verbal or written, frustration and anger of one person are sometimes being mistaken by the communication partner for aggressiveness. Why is this happening? Asking a person with privilege to see, question, or change their behavior, questions their privilege. I'm thinking that it might be that most people think of themselves as "being good" and when they are being asked to question themselves or their behavior, their self-image is being challenged. "Me? But I did not do anything wrong! It's certainly not my fault if you are being oppressed! I sacrificed myself to reach my current position in life!" This comic by Toby Morris, "On a Plate", explains it quite nicely. So are we stuck with seeing conversations derail? I'd argue instead that while anger and frustration are unpleasant feelings, they're important: they show us that our boundaries have been crossed, that we want something to change, to stop, or that we need something different right now. We have the right to be angry and to demand change.

8 August 2017

Jonathan Dowland: libraries

Cover for The Rise Of The Meritocracy Cover for The Rise Of The Meritocracy
At some point during my Undergraduate years I lost the habit of using Libraries. On reflection this is probably Amazon's fault. In recent years I've tried to get back into the habit of using them. Using libraries is a great idea if you are trying to lead a more minimalist life. I am registered to use Libraries in two counties: North Tyneside, where I live, and Newcastle, where I work. The union of the two counties' catalogues is pretty extensive. Perhaps surprisingly I have found North Tyneside to offer both better customer service and a more interesting selection of books. Sometimes there are still things that are hard to get ahold of. After listening to BBC Radio 4's documentary The Rise and Fall of Meritocracy, presented by Toby Young, I became interested in reading The Rise of the Meritocracy: an alarmist, speculative essay that coined the term meritocracy, written by Toby's father, Michael Young. The book was not on either catalogue. It is out of print, with the price of second hand copies fluctuating but generally higher than I am prepared to pay. I finally managed to find a copy in Newcastle University's Library. As an associate of the School of Computing I have access to the Library services. It's an interesting read, and I think if it were framed more as a novel than as an essay it might be remembered in the same bracket as Brave New World or 1984.

8 May 2017

Russ Allbery: Review: Chimes at Midnight

Review: Chimes at Midnight, by Seanan McGuire
Series: October Daye #7
Publisher: DAW
Copyright: 2013
ISBN: 1-101-63566-5
Format: Kindle
Pages: 346
Chimes at Midnight is the seventh book of the October Daye series and builds heavily on the previous books. Toby has gathered quite the group of allies by this point, and events here would casually spoil some of the previous books in the series (particularly One Salt Sea, which you absolutely do not want spoiled). I strongly recommend starting at the beginning, even if the series is getting stronger as it goes along. This time, rather than being asked for help, the book opens with Toby on a mission. Goblin fruit is becoming increasingly common on the streets of San Francisco, and while she's doing all she can to find and stop the dealers, she's finding dead changelings. Goblin fruit is a pleasant narcotic to purebloods, but to changelings it's instantly and fatally addictive. The growth of the drug trade means disaster for the local changelings, particularly since previous events in the series have broken a prominent local changeling gang. That was for the best, but they were keeping goblin fruit out, and now it's flooding into the power vacuum. In the sort of idealistic but hopelessly politically naive move that Toby is prone to, she takes her evidence to the highest local authority in faerie: the Queen of the Mists. The queen loathes Toby and the feeling is mutual, but Toby's opinion is that this shouldn't matter: these are her subjects and goblin fruit is widely recognized as a menace. Even if she cares nothing for their lives, a faerie drug being widely sold on the street runs the substantial risk that someone will give it to humans, potentially leading to the discovery of faerie. Sadly, but predictably, Toby has underestimated the Queen's malevolence. She leaves the court burdened not only with the knowledge that the Queen herself is helping with the distribution of goblin fruit, but also an impending banishment thanks to her reaction. She has three days to get out of the Queen's territory, permanently. Three days that the Luidaeg suggests she spend talking to people who knew King Gilad, the former and well-respected king of the local territory who died in the 1906 earthquake, apparently leaving the kingdom to the current Queen. Or perhaps not. As usual, crossing Toby is a very bad idea, and getting Toby involved in politics means that one should start betting heavily against the status quo. Also, as usual, things initially go far too well, and then Toby ends up in serious trouble. (I realize the usefulness of raising the stakes of the story, but I do prefer the books of this series that don't involve Toby spending much of the book ill.) However, there is a vast improvement over previous books in the story: one key relationship (which I'll still avoid spoiling) is finally out of the precarious will-they, won't-they stage and firmly on the page, and it's a relationship that I absolutely love. Watching Toby stomp people who deserve to be stomped makes me happy, but watching Toby let herself be happy and show it makes me even happier. McGuire also gives us some more long-pending revelations. I probably should have guessed the one about one of Toby's long-time friends and companions much earlier, although at least I did so a few pages before Toby found out. I have some strong suspicions about Toby's own background that were reinforced by this book, and will be curious to see if I'm right. And I'm starting to have guesses about the overall arc of the series, although not firm ones. One of my favorite things in long-running series is the slow revelation of more and more world background, and McGuire does it in just the way I like: lots of underlying complexity, reveals timed for emotional impact but without dragging on things that the characters should obviously be able to figure out, and a whole bunch of layered secrets that continue to provide more mystery even after one layer is removed. The plot here is typical of the plot of the last couple of novels in the series, which is fine by me since my favorite part of this series is the political intrigue (and Toby realizing that she has far more influence than she thinks). It helps that I thought Arden was great, given how central she is to this story. I liked her realistic reactions to her situation, and I liked her arguments with Toby. I'm dubious how correct Toby actually was, but we've learned by now that arguments from duty are always going to hold sway with her. And I loved Mags and the Library, and hope we'll be seeing more of them in future novels. The one quibble I'll close with, since the book closed with it, is that I found the ending rather abrupt. There were several things I wanted to see in the aftermath, and the book ended before they could happen. Hopefully that means they'll be the start of the next book (although a bit of poking around makes me think they may be in a novella). If you've liked the series so far, particularly the couple of books before this one, this is more of what you liked. Recommended. Followed by The Winter Long. Rating: 8 out of 10

6 January 2017

Mark Brown: OpenTAC sprint

This weekend Toby Churchill kindly hosted a hacking weekend for OpenTAC myself, Michael Grzeschik, Steve McIntyre and Andy Simpkins got together to bring up the remaining bits of the hardware on the current board revision and get some of the low level tooling like production flashing for the FTDI serial ports on the board up and running. It was a very productive weekend, we verified that everything was working with only few small mods needed for the board . Personally the main thing I worked on was getting most of an initial driver for the EMC1701 written. That was the one component without Linux support and allowed us to verify that the power switching and measurement for the systems under test was working well. There s still at least one more board revision and quite a bit of software work to do (I m hoping to get the EMC1701 upstream for v4.8) but it was great to finally see all the physical components of the system working well and see it managing a system under test, this board revision should support all the software development that s going to be needed for the final board. Thanks to all who attended, Pengutronix for sponsoring Michael s attendance and Toby Churchill for hosting! Team at work
Group photo

25 December 2016

Russ Allbery: krb5-strength 3.1

krb5-strength provides password strength checking plugins and programs for MIT Kerberos and Heimdal, and a password history implementation for Heimdal. This is the first new upstream release since I left Stanford, since I don't personally use the package any more. But it's easy enough to maintain, and it was overdue for merging some contributed patches. This release adds a new configuration directive, cracklib_maxlen, which can be used to not run longer passwords through CrackLib (whose concepts of password strength were not designed for longer passwords and which can spuriously reject passphrases). It also allows require_classes to require a certain number of character classes in a password, not just specific classes. There are also a variety of portability and cleanup fixes. Thanks to Jorj Bauer, Toby Blake, and Bernt Jernberg for their contributions. I've merged into the embedded CrackLib all the relevant security patches that have come out, although none of them turned out to be relevant for this package due to how CrackLib was called. I also applied a patch from Mark Sirota to fix mkdict and packer to force C locale, which keeps them from creating corrupted dictionaries. Finally, configuration instructions for the plugin are now installed as a new krb5-strength man page, and configuration instructions are included in the heimdal-history and heimdal-strength man pages, instead of just being in the README file. This makes them more accessible after the package is installed. You can get the latest version from the krb5-strength distribution page.

Russ Allbery: krb5-strength 3.1

krb5-strength provides password strength checking plugins and programs for MIT Kerberos and Heimdal, and a password history implementation for Heimdal. This is the first new upstream release since I left Stanford, since I don't personally use the package any more. But it's easy enough to maintain, and it was overdue for merging some contributed patches. This release adds a new configuration directive, cracklib_maxlen, which can be used to not run longer passwords through CrackLib (whose concepts of password strength were not designed for longer passwords and which can spuriously reject passphrases). It also allows require_classes to require a certain number of character classes in a password, not just specific classes. There are also a variety of portability and cleanup fixes. Thanks to Jorj Bauer, Toby Blake, and Bernt Jernberg for their contributions. I've merged into the embedded CrackLib all the relevant security patches that have come out, although none of them turned out to be relevant for this package due to how CrackLib was called. I also applied a patch from Mark Sirota to fix mkdict and packer to force C locale, which keeps them from creating corrupted dictionaries. Finally, configuration instructions for the plugin are now installed as a new krb5-strength man page, and configuration instructions are included in the heimdal-history and heimdal-strength man pages, instead of just being in the README file. This makes them more accessible after the package is installed. You can get the latest version from the krb5-strength distribution page.

3 December 2016

Vincent Bernat: Build-time dependency patching for Android

This post shows how to patch an external dependency for an Android project at build-time with Gradle. This leverages the Transform API and Javassist, a Java bytecode manipulation tool.
buildscript  
    dependencies  
        classpath 'com.android.tools.build:gradle:2.2.+'
        classpath 'com.android.tools.build:transform-api:1.5.+'
        classpath 'org.javassist:javassist:3.21.+'
        classpath 'commons-io:commons-io:2.4'
     
 
Disclaimer: I am not a seasoned Android programmer, so take this with a grain of salt.

Context This section adds some context to the example. Feel free to skip it. Dashkiosk is an application to manage dashboards on many displays. It provides an Android application you can install on one of those cheap Android sticks. Under the table, the application is an embedded webview backed by the Crosswalk Project web runtime which brings an up-to-date web engine, even for older versions of Android1. Recently, a security vulnerability has been spotted in how invalid certificates were handled. When a certificate cannot be verified, the webview defers the decision to the host application by calling the onReceivedSslError() method:
Notify the host application that an SSL error occurred while loading a resource. The host application must call either callback.onReceiveValue(true) or callback.onReceiveValue(false). Note that the decision may be retained for use in response to future SSL errors. The default behavior is to pop up a dialog.
The default behavior is specific to Crosswalk webview: the Android builtin one just cancels the load. Unfortunately, the fix applied by Crosswalk is different and, as a side effect, the onReceivedSslError() method is not invoked anymore2. Dashkiosk comes with an option to ignore TLS errors3. The mentioned security fix breaks this feature. The following example will demonstrate how to patch Crosswalk to recover the previous behavior4.

Simple method replacement Let s replace the shouldDenyRequest() method from the org.xwalk.core.internal.SslUtil class with this version:
// In SslUtil class
public static boolean shouldDenyRequest(int error)  
    return false;
 

Transform registration Gradle Transform API enables the manipulation of compiled class files before they are converted to DEX files. To declare a transform and register it, include the following code in your build.gradle:
import com.android.build.api.transform.Context
import com.android.build.api.transform.QualifiedContent
import com.android.build.api.transform.Transform
import com.android.build.api.transform.TransformException
import com.android.build.api.transform.TransformInput
import com.android.build.api.transform.TransformOutputProvider
import org.gradle.api.logging.Logger
class PatchXWalkTransform extends Transform  
    Logger logger = null;
    public PatchXWalkTransform(Logger logger)  
        this.logger = logger
     
    @Override
    String getName()  
        return "PatchXWalk"
     
    @Override
    Set<QualifiedContent.ContentType> getInputTypes()  
        return Collections.singleton(QualifiedContent.DefaultContentType.CLASSES)
     
    @Override
    Set<QualifiedContent.Scope> getScopes()  
        return Collections.singleton(QualifiedContent.Scope.EXTERNAL_LIBRARIES)
     
    @Override
    boolean isIncremental()  
        return true
     
    @Override
    void transform(Context context,
                   Collection<TransformInput> inputs,
                   Collection<TransformInput> referencedInputs,
                   TransformOutputProvider outputProvider,
                   boolean isIncremental) throws IOException, TransformException, InterruptedException  
        // We should do something here
     
 
// Register the transform
android.registerTransform(new PatchXWalkTransform(logger))
The getInputTypes() method should return the set of types of data consumed by the transform. In our case, we want to transform classes. Another possibility is to transform resources. The getScopes() method should return a set of scopes for the transform. In our case, we are only interested by the external libraries. It s also possible to transform our own classes. The isIncremental() method returns true because we support incremental builds. The transform() method is expected to take all the provided inputs and copy them (with or without modifications) to the location supplied by the output provider. We didn t implement this method yet. This causes the removal of all external dependencies from the application.

Noop transform To keep all external dependencies unmodified, we must copy them:
@Override
void transform(Context context,
               Collection<TransformInput> inputs,
               Collection<TransformInput> referencedInputs,
               TransformOutputProvider outputProvider,
               boolean isIncremental) throws IOException, TransformException, InterruptedException  
    inputs.each  
        it.jarInputs.each  
            def jarName = it.name
            def src = it.getFile()
            def dest = outputProvider.getContentLocation(jarName, 
                                                         it.contentTypes, it.scopes,
                                                         Format.JAR);
            def status = it.getStatus()
            if (status == Status.REMOVED)   //  
                logger.info("Remove $ src ")
                FileUtils.delete(dest)
              else if (!isIncremental   status != Status.NOTCHANGED)   //  
                logger.info("Copy $ src ")
                FileUtils.copyFile(src, dest)
             
         
     
 
We also need two additional imports:
import com.android.build.api.transform.Status
import org.apache.commons.io.FileUtils
Since we are handling external dependencies, we only have to manage JAR files. Therefore, we only iterate on jarInputs and not on directoryInputs. There are two cases when handling incremental build: either the file has been removed ( ) or it has been modified ( ). In all other cases, we can safely assume the file is already correctly copied.

JAR patching When the external dependency is the Crosswalk JAR file, we also need to modify it. Here is the first part of the code (replacing ):
if ("$ src " ==~ ".*/org.xwalk/xwalk_core.*/classes.jar")  
    def pool = new ClassPool()
    pool.insertClassPath("$ src ")
    def ctc = pool.get('org.xwalk.core.internal.SslUtil') //  
    def ctm = ctc.getDeclaredMethod('shouldDenyRequest')
    ctc.removeMethod(ctm) //  
    ctc.addMethod(CtNewMethod.make("""
public static boolean shouldDenyRequest(int error)  
    return false;
 
""", ctc)) //  
    def sslUtilBytecode = ctc.toBytecode() //  
    // Write back the JAR file
    //  
  else  
    logger.info("Copy $ src ")
    FileUtils.copyFile(src, dest)
 
We also need the following additional imports to use Javassist:
import javassist.ClassPath
import javassist.ClassPool
import javassist.CtNewMethod
Once we have located the JAR file we want to modify, we add it to our classpath and retrieve the class we are interested in ( ). We locate the appropriate method and delete it ( ). Then, we add our custom method using the same name ( ). The whole operation is done in memory. We retrieve the bytecode of the modified class in . The remaining step is to rebuild the JAR file:
def input = new JarFile(src)
def output = new JarOutputStream(new FileOutputStream(dest))
//  
input.entries().each  
    if (!it.getName().equals("org/xwalk/core/internal/SslUtil.class"))  
        def s = input.getInputStream(it)
        output.putNextEntry(new JarEntry(it.getName()))
        IOUtils.copy(s, output)
        s.close()
     
 
//  
output.putNextEntry(new JarEntry("org/xwalk/core/internal/SslUtil.class"))
output.write(sslUtilBytecode)
output.close()
We need the following additional imports:
import java.util.jar.JarEntry
import java.util.jar.JarFile
import java.util.jar.JarOutputStream
import org.apache.commons.io.IOUtils
There are two steps. In , all classes are copied to the new JAR, except the SslUtil class. In , the modified bytecode for SslUtil is added to the JAR. That s all! You can view the complete example on GitHub.

More complex method replacement In the above example, the new method doesn t use any external dependency. Let s suppose we also want to replace the sslErrorFromNetErrorCode() method from the same class with the following one:
import org.chromium.net.NetError;
import android.net.http.SslCertificate;
import android.net.http.SslError;
// In SslUtil class
public static SslError sslErrorFromNetErrorCode(int error,
                                                SslCertificate cert,
                                                String url)  
    switch(error)  
        case NetError.ERR_CERT_COMMON_NAME_INVALID:
            return new SslError(SslError.SSL_IDMISMATCH, cert, url);
        case NetError.ERR_CERT_DATE_INVALID:
            return new SslError(SslError.SSL_DATE_INVALID, cert, url);
        case NetError.ERR_CERT_AUTHORITY_INVALID:
            return new SslError(SslError.SSL_UNTRUSTED, cert, url);
        default:
            break;
     
    return new SslError(SslError.SSL_INVALID, cert, url);
 
The major difference with the previous example is that we need to import some additional classes.

Android SDK import The classes from the Android SDK are not part of the external dependencies. They need to be imported separately. The full path of the JAR file is:
androidJar = "$ android.getSdkDirectory().getAbsolutePath() /platforms/" +
             "$ android.getCompileSdkVersion() /android.jar"
We need to load it before adding the new method into SslUtil class:
def pool = new ClassPool()
pool.insertClassPath(androidJar)
pool.insertClassPath("$ src ")
def ctc = pool.get('org.xwalk.core.internal.SslUtil')
def ctm = ctc.getDeclaredMethod('sslErrorFromNetErrorCode')
ctc.removeMethod(ctm)
pool.importPackage('android.net.http.SslCertificate');
pool.importPackage('android.net.http.SslError');
//  

External dependency import We must also import org.chromium.net.NetError and therefore, we need to put the appropriate JAR in our classpath. The easiest way is to iterate through all the external dependencies and add them to the classpath.
def pool = new ClassPool()
pool.insertClassPath(androidJar)
inputs.each  
    it.jarInputs.each  
        def jarName = it.name
        def src = it.getFile()
        def status = it.getStatus()
        if (status != Status.REMOVED)  
            pool.insertClassPath("$ src ")
         
     
 
def ctc = pool.get('org.xwalk.core.internal.SslUtil')
def ctm = ctc.getDeclaredMethod('sslErrorFromNetErrorCode')
ctc.removeMethod(ctm)
pool.importPackage('android.net.http.SslCertificate');
pool.importPackage('android.net.http.SslError');
pool.importPackage('org.chromium.net.NetError');
ctc.addMethod(CtNewMethod.make(" "))
// Then, rebuild the JAR...
Happy hacking!

  1. Before Android 4.4, the webview was severely outdated. Starting from Android 5, the webview is shipped as a separate component with updates. Embedding Crosswalk is still convenient as you know exactly which version you can rely on.
  2. I hope to have this fixed in later versions.
  3. This may seem harmful and you are right. However, if you have an internal CA, it is currently not possible to provide its own trust store to a webview. Moreover, the system trust store is not used either. You also may want to use TLS for authentication only with client certificates, a feature supported by Dashkiosk.
  4. Crosswalk being an opensource project, an alternative would have been to patch Crosswalk source code and recompile it. However, Crosswalk embeds Chromium and recompiling the whole stuff consumes a lot of resources.

2 July 2016

Russ Allbery: Review: Ashes of Honor

Review: Ashes of Honor, by Seanan McGuire
Series: October Daye #6
Publisher: DAW
Copyright: September 2012
ISBN: 1-101-59480-2
Format: Kindle
Pages: 368
This is the sixth book in the October Daye series, contains payoffs for some relationships that have been building over the whole series, and involves entangled politics set up by previous books. It's not the place to start with the series. Ashes of Honor starts, as so many of Toby's books do, with a friend asking her for help. But this request is entirely unexpected, and the help needed comes as a complete surprise: a previously unknown changeling, who has disappeared. A changeling whose powers are completely out of control, and who poses a threat to reality itself. As Toby's cases go, this involves a lot fewer horrible things happening to her and a lot more faerie politics and maneuvering than usual. I appreciated that; I'm not as fond of the books that go deep into despair or desperation. It does involve Toby getting almost killed multiple times, but, due to earlier events of the series, that isn't quite as bad as it used to be. More focus on investigation and political maneuvering and less Toby braving her way through horrors works for me. Even more notably, this book marks Toby finally figuring out that she has friends and allies who are there to help, not just be obligations she feels overwhelmed by or aid that she's not allowed to accept. This was one of her most frustrating characteristics; it's a relief to see her finally relax. This opens the way not only for deeper friendships and more complex plots but also a relationship that I've been awaiting for the entire series, and it's as much fun as I was hoping it would be. Toby started the series rather messed up and unwilling to let anyone close. It was for understandable reasons, but I like her better when she realizes why people respect her. Toby's connections with the royalty of the Bay Area also allow McGuire to tell a political story that moves farther afield from the Shadowed Hills. First in One Salt Sea and now in Ashes of Honor we see more of local politics, more of the lore of McGuire's universe, and another dangerous queen. Toby is particularly fun when she's dangerously outflanking people with considerably more power than she has. At this point, you could call it a specialty. I thought McGuire's take on San Jose and the sort of person who would be in charge of its fae was on point. We also get more of the Luidaeg, which is always a good sign for a Toby novel, and more of Tybalt, who is entangled in a major subplot of the story. Next to Luidaeg, Tybalt is my favorite of Toby's friends, so this book is full of the things that make me happy. McGuire adds some more pieces to her transplanted Celtic mythology and some tantalizing hints of what the fae have left behind. I'm hoping we see more of that in future books. (I suspect that may be what this whole series is building up to.) The story doesn't have quite as much oomph as One Salt Sea, but it's still one of the best books in the series so far. If you've enjoyed the series up to this point, keep reading. Followed by Chimes at Midnight. Rating: 8 out of 10

18 May 2016

Andy Simpkins: OpenTAC sprint, Cambridge

Last weekend saw a small group get togeather in Cambridge to hack on the OpenTAC. OpenTAC is an OpenHardware OpenSoftware test platform, designed specificly to aid automated testing and continious intergration. Aimed at small / mobile / embedded targets OpenTAC v1 provides all of the support infrastructure to drive up to 8 DUTs (Device Under Test) to your test or CI system.
Each of the 8 EUT ports provides: All ports on the EUT interface are relay issolated, this means that cables to your EUT can be unplugged under software control (we are aware of several SoC development boards that latch up if there is a serial port connected before power is applied). Additionly there are 8 GPIO lines that can be used as switch controls to any EUT (perhaps to put a specific EUT into a programming mode, reboot it or even start it) Anyway, back to the hacking weekend. .. Joining Steve McIntyre and myself were Mark Brown, and Michael Grzeschik (sorry Michael, I couldn t find a homepage). Mark traveled down from Scotland whilst Michael flew in from Germany for the weekend. Gents we greatly apprecate you taking the time and expence to join us this weekend. I should also thank my employer Toby Churchill Ltd. for allowing us to use the office to host the event. A lot of work got done, and I beleive we have now fully tested and debugged the hardware. We have also made great progress with the device tree and dvice drivers for the platform. Mark got the EUT power system working as proof of concept, and has taken an OpenTAC board back with him to turn this into suitable drivers and hopfully push them up stream. Meanwhile Michael spent his time working on the system portion of the device tree; OpenTAC s internal power sequancing, thermal managment subsystem, and USB hub control. Steve got to grips with the USB serial converters (including how to read and program their internal non-volatile settings). Finally I was able to explain hardware sequancing to everyone, and to modify boards to overcome some of my design mistakes (the biggest was by far the missing sence resistors for the EUT power managment)

1 January 2016

Russ Allbery: Review: One Salt Sea

Review: One Salt Sea, by Seanan McGuire
Series: October Daye #5
Publisher: DAW
Copyright: September 2011
ISBN: 1-101-54760-X
Format: Kindle
Pages: 368
This is the fifth book of the October Daye series. It's the best book of the series to date, at least in my opinion, but this is a series with a substantial cast and political complications. You could probably start here and work out things on the fly, since each story is largely self-contained, but much of the emotional resonance would be lost. As expected for this series, Toby's life doesn't have much time to stabilize after the events of Late Eclipses. This time, though, the initial threat is less personal and more political. The children of the rulers of the Duchy of Saltmist have been kidnapped, and war between Saltmist and the Kingdom of the Mists (Toby's home) is very near. All that stands in the way of horrible casualties, of fae that can scarce afford more death, is Toby's desperate detective work. Toby's confidence and belief in her own abilities has been growing steadily throughout this series. One of my favorite themes in fiction is someone who doesn't really believe in themselves but tries anyway, because they couldn't live with themselves if they didn't, and discovers they're more capable than they thought. In this series, I think that works best when Toby is less scared and more angry, which is very true in One Salt Sea. That's one reason why I liked this book better than either An Artificial Night or Late Eclipses. Another is that I love the politics and the depth of world-building and lore that McGuire brings to this series, and One Salt Sea is a showcase of both. We get the Undersea Duchy of Saltmist, which comes from both different traditions and different fae than the series so far. We get a plot that combines political maneuvering with a more traditional detective story than the last few books. We get more hidden traditions, past alliances and enmity that Toby didn't know about, and further development of Toby's own abilities. She has to tug hard on some threads, move between levels of the fae world, and draw on her network for uniquely fae forms of forensic analysis. I thoroughly enjoyed it, particularly the rocks. (You'll know what I mean when you get to that part.) But the best thing about this book, by far, is the Luidaeg. I've mentioned in reviews of previous books that the Luidaeg is my favorite character of the series. One Salt Sea just cements that. This is the deepest that the Luidaeg has been involved in a story: she gives Toby her mission at the start, substantial help along the way, and is deeply involved in the ending. We even learn some of her own background, and some of her own worries and pain. (And foreshadowing that there's more of this to come, which I'm eagerly looking forward to.) She's moved from a strange ally through a cautious friend to some combination of aunt and mother to Toby, while always staying her irascible and occasionally painful self. The bits with her in this book are the best so far, and there are a lot of them. It's a delightful reading experience. McGuire also uses this story to clear up a plot element that's been lingering since the first book of the series, but that never worked emotionally for me. (I'm talking about this indirectly since it's a significant spoiler.) It's a great resolution, one with heft and emotion and hard choices and pain, and largely makes up for the showing rather than telling that we've gotten in the previous books, while also (at least I suspect) graciously shuffling that plot out of the way in future books. It's a good bit of series work and personal growth for Toby, and I think frees the series to focus on the bits I enjoy the most. The ending of One Salt Sea hurts, but it's a good hurt, full of hard choices made well, bravery, and sacrifice. The best moments of this series have always had that, but I think this book is the most successful delivery of the whole package to date. This has always been a solid urban fantasy series, but I think it's getting better as it goes along. One major reason is that McGuire doesn't seem to be running out of world-building oomph. In a lot of urban fantasy, particularly the werewolf and vampire sort, I feel like the lore gets thinner, more rehashed, and more mundane as the series ages. But McGuire is still introducing new elements, showing existing elements from new directions and in new light, and has been slowly building major puzzles around Toby's life and abilities that intrigue me. And, even more rare, the eventual revelations usually live up to the preceding tension. It's certainly a recipe to keep me reading. Followed by Ashes of Honor. Rating: 9 out of 10

8 June 2015

Russ Allbery: Review: Late Eclipses

Review: Late Eclipses, by Seanan McGuire
Series: October Daye #4
Publisher: DAW
Copyright: March 2011
ISBN: 1-101-50253-3
Format: Kindle
Pages: 372
Late Eclipses is the fourth book the October Daye series, and relies heavily on characters introduced in the previous books. I recommend reading this series from the start; jumping into the middle would miss a lot of nuance. Thankfully, though, enough is explained that you don't have to have read the previous books recently. (I wish more series would do that.) Unsurprisingly, the book opens with Toby's life getting more complicated. Also unsurprisingly, that means getting more entangled in the affairs of the fae court, as Toby gets pushed farther out of her comfort zone. But that quickly takes a back seat to much worse news: Toby's close friend Lily is deathly ill. That isn't supposed to be possible for an undine. And Lily isn't the last person to get deathly ill in this book. I should note up-front that this book contains one of my least favorite tropes in fiction of this sort: a protagonist who falls under the influence of something mind-altering and has to keep second-guessing her own perceptions. I have this problem with most books about drugs or some equivalent. There was enough of that here to irritate me, but this is just a personal quirk and I'm used to other people liking those books better, so you may need to adjust my rating accordingly. That said, I liked Late Eclipses better than An Artificial Night, even with that drawback. It's less dark, less bleak, and returns to some of the mystery feel of the first two books of the series. A lot of urban fantasy mixes in a bit of a detective element, usually from the noir tradition, and I think that provides a useful plot driver. There's a lot at stake in this story, but Toby also gets a lot of agency. She's out doing things, making guesses and following up on them, rather than trying to endure vast horror. And she has more trust in her instincts and abilities, and is gathering more allies and respect. There was a bit too much of "abuse the protagonist" for my tastes, but some of the court maneuvering is quite satisfying. There's always a risk with power curves taking away the risk in stories like this, or of having to constantly invent a bigger bad than the previous one, but McGuire is doing a good job keeping control of that. Toby is getting stronger, and it's obvious that she's more than she appears or realizes. Coming to terms with the edges of that is part of this story. But the dangers in these stories have been very different in kind rather than escalating degree. The complex political machinations of the fae court help here considerably, creating problems that Toby has to navigate through with allies and careful thought. One of my favorite parts of this series continues to be the supporting cast. We don't get as much of the Luidaeg here, but we get lots of May (who is becoming one of my favorite characters of the series) and several other excellent supporting characters. It's rare that I like the supporting cast of an urban fantasy series this well without feeling like they're overshadowing the protagonist. Some parts of this story bugged me for idiosyncratic reasons, but I still thought it was a step up from the previous installment. McGuire's world doesn't seem to be running out of steam. I'll definitely keep reading. Followed by One Salt Sea. Rating: 7 out of 10

2 July 2014

Russ Allbery: Review: Lockstep

Review: Lockstep, by Karl Schroeder
Publisher: Tor
Copyright: March 2014
ISBN: 0-7653-3726-6
Format: Hardcover
Pages: 351
Toby McGonigal's family fled an Earth dominated by trillionaires and vicious class conflict, and attempt, instead, a very risky and precarious settlement on a trans-Neptunian object. It's the last unclaimed but marginally habitable space left in the solar system, and securing their claim requires constant bureaucratic hoop-jumping. That's what sent Toby on a solo mission to a distant moon of their home to claim it and solidify their title. But, while in cold sleep, his craft hits a small chunk of rock, and he wakes up near another world: cold, silent, dead... but apparently with lifeless cities. Another trip through suspended animation in a desperate attempt to conserve resources against the distant hope of rescue has him awakening in a society both utterly foreign and yet strangely familiar. Schroeder is one of the better big idea writers in science fiction, but I found Lockstep to be a mixed bag. He does a surprisingly good job with the core conceit of the novel (more on that in a moment), even though it's a tricky one to make believable. Surrounding that, though, are a lot of less convincing bits that I kept having to not think about too hard, such as Toby crossing the path of another trans-Neptunian object after the accident (space is really, really big and really, really empty), or the implications of later revelations about the time scales involved in parts of the plot. Some parts of the world building, even if scientifically plausible, struck me as sloppy; for example, a religion plays a prominent role in the plot, but the nature of that religion was not particularly believable, nor was its interaction with the plot climax. I won't go into details about the religion, since it's a major plot point, but the short version is that religions tend to mature from the concrete to the abstract, not the other way around. That said, the core conceit is both surprising and considerably better-defended in the book than I thought it could be. The world into which Toby awakes is the world of the locksteps: a society that uses suspended animation in a universal and coordinated way to build a functional society on the far outskirts of the solar system. Humans emerge for some short period of time, like a month, and build, trade, interact, and consume. They welcome ships traveling from other trans-Neptunian worlds and prepare for their own journeys. And then they go into suspended animation for an extended period 15 years and 30 years are common choices while robots slowly gather more resources and energy, and repair and replace what's consumed in the month of active life. Ships travel with passengers in suspended animation, allowing the vast distances between these cold worlds to be reached during the sleep period. And, since all members of the lockstep sleep and wake on the same schedule, there is no wrenching desynchronization with the surrounding society during travel. One may spend thirty years in transit, but no time passes for anyone else in the lockstep while you're traveling either. A world can effectively trade with all other worlds within a thirty year travel radius without noticing the elapsed time. This is a technological system that at first doesn't sound like it would work, but Schroeder does a great job defending it and chasing down implications. The lockstep civilization serves as a sort of anchor and time capsule separate from the more frantic pace of the so-called fast worlds. The trans-Neptunian lockstep worlds are remote enough and poor enough to not attract too much unwanted attention, and are thus tolerated by fast societies that may have more available resources. (Although, to be sure, automated robot defenses have to carry a lot of weight here given how helpless the locksteps are during a sleep cycle.) I'm not sure I completely bought the sociology, but it works well enough to carry the weight of the story, and I've never seen a science fiction construction that uses the common technology of suspended animation in quite this way before. Schroeder sets up some nice stabilizing tensions between resources and time, adds some believable political reasons for this fragile society to survive, and uses some of its obvious political vulnerabilities as story drivers. The plot, unfortunately, isn't as good as the big idea, and particularly suffers at the start of the book. Toby is a teenager without much experience (Lockstep is marketed as young adult), and is immediately thrown into a strange and quickly hostile environment. This means that he spends the first half of the book reacting to a blizzard of new information, and the plot tends to revert to a tour of Schroeder's constructed world. Toby is also a bit of a cipher and a bit of an everyman, without many feelings or opinions beyond the obvious feelings that any teenage boy would go through in this situation. That makes the sense of a world tour even stronger. This flaw does not persist through the whole book. Toby does eventually start making decisions and doing things, some of the supporting characters add additional depth, and I found the unwinding of the plot surprisingly satisfying. The nature of time in this world lets Schroeder have both epic sweep and personal connection at the same time, which lets him pull off some neat contrasts between the personal and the political. I also liked Toby's gradual piecing together of what actually happened while he was asleep, both in the construction of society as a whole and in the personal conflicts between people he knew well. Some of the ease of grand manipulation seemed dubious to me, but, in Schroeder's defense, people do develop a reverence for and stories about things that have lasted a very long time, and Schroeder's setup gives him quite a lot to work with in that department. So, a mixed bag. The core concept is thought-provoking and up to Schroeder's usual standards. The surrounding world-building isn't as much, and I think Schroeder reaches for some too-easy explanations and still underplays just how many disruptive things can happen over the span of time that this book covers. The characterization I found weak and unsatisfying for much of the book, but it gets moderately better in the end. There's a bit too much tour, and a bit too much world exploration instead of plot, but it's a fascinating world and I still enjoyed the tour. With stronger characters and a few fewer dubious supporting pillars in the world background, I think this would have been an excellent book. As is, it's an enjoyable novel set firmly in the big idea and deep future end of science fiction, and I'm always happy to see more stories like that. It's not the best that Schroeder has done, but I still recommend it. Rating: 7 out of 10

12 March 2014

Vasudev Kamath: Working around type system of Go with unsafe

Go language has a strong type system unlike C, and some times this will be head ache when we want to interact with C data types with Cgo or just to convert a Go type to lets say byte slice. I recently faced the same problem and after poking around things, I learned the Go provides unsafe package which can be used to work around the Go's type system.
Problem Recently I started using Cgo to use some C library I had to write some tools for development and testing. The reason I chose Go for this was prototyping and writing some quick tools is much easier in Go than done in C. The C library had some function which takes pointer to array types and fills it with some values. The problem I was facing here was how to create a C array in Go. Go does have array but its largely used as internal representation for much efficient type called slice and I can't directly cast a byte slice into an C array. Second problem I had was I had to store arbitrary Go types like float (float32,float64) and int (int32, int64) etc. into C array. So in brief, the problems that needed to solve are
  1. Find a way to convert byte slice from Go into C array and vice versa.
  2. Find a way to convert and store Go types into a C array.
Solution Basically C array are sequence of memory location which can be statically or dynamically allocated. In Cgo its possible to access C standard library functions for memory allocation, so why not use it. The memory allocation function then returns the pointer to starting of allocated memory, we can use this pointer to write Go's bytes into the memory location and bytes from memory location into Go's slice. The pointer returned by C allocation functions are not directly usable for memory dereferencing in Go, here is where unsafe package kicks in. We will cast the return of C allocation function as unsafe.Pointer type and from the documentation of unsafe package,
  1. A pointer value of any type can be converted to a Pointer.
  2. A Pointer can be converted to a pointer value of any type.
  3. A uintptr can be converted to a Pointer.
  4. A Pointer can be converted to a uintptr.
So we can then cast unsafe.Pointer to uintptr which is the Go type which is large enough to hold any memory address in Go and can be used for pointer arithmetic just like we do in C (of course with some more castings). Below I'm pasting a simplified code in C which I wrote for this post.
#ifndef __BYTETEST_H__
#define __BYTETEST_H__
typedef unsigned char UBYTE;
extern void ArrayReadFunc(UBYTE *arrayout);
extern void ArrayWriteFunc(UBYTE *arrayin);
#endif
#include "bytetest.h"
#include <stdio.h>
#include <string.h>
void ArrayReadFunc(UBYTE *arrayout)
 
     UBYTE array[20] =  1, 2, 3, 4,5, 6, 7, 8, 9, 10,
                        11, 12, 13, 14, 15, 16, 17,
                        18, 19, 20 ;
     memcpy(arrayout, array, 20);
 
void ArrayWriteFunc(UBYTE *arrayin)
 
     UBYTE array[20];
     memcpy(array, arrayin, 20);
     printf("Byte slice array received from Go:\n");
     for(int i = 0; i < 20; i ++) 
             printf("%d ", array[i]);
      
     printf("\n");
 
Functions are written just for this post and they don't really do anything. As you can see ArrayReadFunc takes a pointer to array and fills it with content of another array using memcpy. Function ArrayWriteFunc on other hand takes pointer to array and copies its content to internal array. I've added print logic to ArrayWriteFunc just to show that values passed from Go are making it here. Below is the Go code which uses the above C files passes byte slice to get value out of C code and array made of byte slice to C function to send values in.
package main
/*
#cgo CFLAGS: -std=c99
#include "bytetest.h"
#include <stdlib.h>
*/
import "C"
import (
     "fmt"
     "unsafe"
)
func ReadArray() unsafe.Pointer  
       var outArray = unsafe.Pointer (C.calloc(20,1))
       C.ArrayReadFunc((*C.UBYTE)(outArray))
       return outArray
 
func WriteArray(inArray unsafe.Pointer)  
       C.ArrayWriteFunc((*C.UBYTE)(inArray))
 
func CArrayToByteSlice(array unsafe.Pointer, size int) []byte  
       var arrayptr = uintptr(array)
       var byteSlice = make([]byte, size)
       for i := 0; i < len(byteSlice); i ++  
               byteSlice[i] = byte(*(*C.UBYTE)(unsafe.Pointer(arrayptr)))
               arrayptr ++
        
       return byteSlice
 
func ByteSliceToCArray (byteSlice []byte) unsafe.Pointer  
       var array = unsafe.Pointer(C.calloc(C.size_t(len(byteSlice)), 1))
       var arrayptr = uintptr(array)
       for i := 0; i < len(byteSlice); i ++  
              *(*C.UBYTE)(unsafe.Pointer(arrayptr)) = C.UBYTE(byteSlice[i])
              arrayptr ++
        
       return array
 
func main() 
        carray := ReadArray()
        defer C.free(carray)
        carraybytes := CArrayToByteSlice(carray, 20)
        fmt.Println("C array converted to byte slice:")
        for i := 0; i < len(carraybytes); i ++  
                fmt.Printf("%d ", carraybytes[i])
         
        fmt.Println()
        gobytes := []byte 21, 22, 23, 24, 25, 26, 27, 28, 29, 30,
                31, 32, 33, 34, 35, 36, 37, 38, 39, 40 
        gobytesarray := ByteSliceToCArray(gobytes)
        defer C.free(gobytesarray)
        WriteArray(gobytesarray)
 
Functions ReadArray and WriteArray are just wrapper to the calls to C counter parts ArrayReadFunc and ArrayWriteFunc. ReadArray returns unsafe.Pointer which is allocated C array and should be freed by caller. WriteArray takes unsafe.Pointer which is pointing to memory location containing C array. Now the functions of interest are CArrayToByteSlice and ByteSliceToCArray. It should be pretty clear from the above code to understand what is happening in these functions. Still I will just put explain them briefly. ByteSliceToCArray allocates a C array using calloc from C standard library. It then creates a uintptr, a pointer type in Go which is used to dereference the each memory location and store bytes from the input byte slice in them. CArrayToByteSlice on other hands creates a uintptr type by casting input unsafe.Pointer type and then uses this pointer type to dereference values from memoy and store it in byte slice with suitable casting. So lets build the code and run it and see the output:
C array converted to byte slice:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Byte slice array received from Go:
21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40
So yes it actually works and values are moving across C and Go. This solves first problem in hand next is converting arbitrary Go types into byte slices. There are many cases where we would like to convert an arbitrary Go types like (int, float) into bytes. One such use case I found was when writing a TCP client for communicating with a Server written using C speaking custom protocol. Here I'm just going to show how to convert types like float, int to byte slice, I've not tried converting structures but it is certainly possible. Below is the function which can convert int32,float32 into byte slice it can also be extended for other types.
func CopyValueToByte(value interface ) []byte  
     var valptr uintptr
     var slice []byte
     switch t := value.(type)  
     case int32:
             i := value.(int32)
             valptr = uintptr(unsafe.Pointer(&i))
             slice = make([]byte, unsafe.Sizeof(i))
     case float32:
             f := value.(float32)
             valptr = uintptr(unsafe.Pointer(&f))
             slice = make([]byte, unsafe.Sizeof(f))
     default:
             fmt.Fprintf(os.Stderr,"Unsupported type: %T\n", t)
             os.Exit(1)
      
     for i := 0; i < len(slice); i++  
             slice[i] = byte(*(*byte)(unsafe.Pointer(valptr)))
             valptr++
      
     return slice
 
This function is generic which can take various types of value. First we will use Go's type assertion to determine the type and creates a uintptr pointer for the value and allocates byte slice depending on the size of the value as calculated using unsafe.Sizeof. Later it uses the pointer to dereference value from memory location and copies each byte into byte slice. The idea used here is every type is represented as certain number of bytes in the memory. Below is the entire program.
package main
import (
     "fmt"
     "unsafe"
     "os"
)
func CopyValueToByte(value interface ) []byte  
     var valptr uintptr
     var slice []byte
     switch t := value.(type)  
     case int32:
             i := value.(int32)
             valptr = uintptr(unsafe.Pointer(&i))
             slice = make([]byte, unsafe.Sizeof(i))
     case float32:
             f := value.(float32)
             valptr = uintptr(unsafe.Pointer(&f))
             slice = make([]byte, unsafe.Sizeof(f))
     default:
             fmt.Fprintf(os.Stderr,"Unsupported type: %T\n", t)
             os.Exit(1)
      
     for i := 0; i < len(slice); i++  
             slice[i] = byte(*(*byte)(unsafe.Pointer(valptr)))
             valptr++
      
     return slice
 
func main()  
     a := float32(-10.3)
     floatbytes := CopyValueToByte(a)
     fmt.Println("Float value as byte slice:")
     for i := 0; i < len(floatbytes); i++  
             fmt.Printf("%x ", floatbytes[i])
      
     fmt.Println()
     b := new(float32)
     bptr := uintptr(unsafe.Pointer(b))
     for i := 0; i < len(floatbytes); i++  
             *(*byte)(unsafe.Pointer(bptr)) = floatbytes[i]
             bptr++
      
     fmt.Printf("Byte value copied to float var: %f\n", *b)
 
The above conversion can also be achieved using encoding/binary package provided by Go. But its been told to me that it makes things pretty slow.
Conclusion So goes unsafe.Pointer is really powerful thing which allows us to work around the Go's type system but as package documentation says it should be used with care
PS: I'm not really sure if its recommended to use allocation functions from C standard library, I will wait for expert gophers to comment on that.

1 September 2013

Russ Allbery: Review: An Artificial Night

Review: An Artificial Night, by Seanan McGuire
Series: October Daye #3
Publisher: DAW
Copyright: September 2010
ISBN: 1-101-44276-X
Format: Kindle
Pages: 368
This is the third book of the October Daye series, following A Local Habitation. It could be read out of order, as each chapter of Toby's story so far is somewhat standalone, but I wouldn't recommend it. You'd miss the character development and the background behind several friendships. As with the previous book review, a caveat: I am reviewing this book four months after finishing it, rather than the normal couple of weeks, so apologies in advance for any thinness or inaccuracies. The first sign that something is about to go seriously wrong in Toby's life this time is a visitor. She says her name is May Daye, and apparently she's Toby's Fetch: a duplicate in nearly every respect except personality, manifested to guide someone to their death. Their imminent death. The second sign is that her best friend's children are kidnapped. The children who treat her like an aunt. And the cause of that kidnapping is far, far beyond the level of power that Toby could be expected to deal with. I have profound mixed feelings about this book. On one hand, McGuire further expands Toby's web of connections and shows us more of Toby's friends and their background, and that's becoming my favorite part of this series. She introduces May, who I think is a great character: at first apparently much different than Toby, but with a similarity beneath surface impressions that one notices more over time. And I thought the climax of the story, once we finally got there, was very moving and well-done. Also, I just love the way that McGuire uses children's rhymes and doggerel throughout this series as a magical element, and that's particularly well-handled here. Due to the nature of the antagonist, McGuire has a good excuse to bring that front and center, and the process of familiar rhymes expanding into deep magical rules was a fun intellectual experience. Best of all, the Luidaeg plays a significant role in the story again. The Luidaeg is, hands down, my favorite character in McGuire's constructed universe, and she becomes even more so over the course of this story. Her complex relationship with Toby is clearly blossoming into a friendship that's surprising both of them deeply, but a friendship that has to play by an odd set of rules (like so much else in Toby's life). I loved her here. Seeing someone that difficult and abrupt show unexpected emotional depths is one of the story elements I enjoy the most. The problem, though, is that quite a lot of An Artificial Night is very, very dark. And by that, I don't mean that there is a lot of tension, or that Toby is struggling against monsterous things, although both of those statements are also true. Rather, Toby is not only out of her league but ends up in some viciously nasty situations, and they persist for rather more of this book than I would have preferred. Putting one's protagonist through hell has a long tradition in urban fantasy, but I found the level of sheer helplessness that Toby struggled through to be a bit more than I wanted to read about. It was too reminiscent of events from Toby's background (as discussed in Rosemary and Rue) that I still don't like to think about. There is, of course, an eventual payoff, as dark as the interim events seem. One knows that, given the genre and given that this is an ongoing series, and even just given the nature of stories. But as much as I like the growing cast of this series, and adored the scenes with the Luidaeg, I found parts of this journey to be deeply disturbing, and not in an enjoyable dramatic tension sort of way. I don't mind protagonists being seriously hurt (although it's not my favorite plot), or even some amount of despair, but this went beyond that. I think McGuire was going for a story about sacrifice and self-definition, about trying even when one can't possibly win, and I can appreciate the narrative arc. But parts of the book were simply awful to read for me. I'm therefore not quite sure what to say in terms of a recommendation. I'm still reading the series, and will definitely read the next book. I think my reaction is somewhat idiosyncratic; the events here will probably not bother others as much as they bothered me. And I can't disagree with their dramatic effectiveness, or with the way that McGuire builds conflict out of the rules of fairy and is building Toby into a stronger and more capable person than she ever thought she could be. But there were still spots where I wished I were reading a different book. Hopefully the next book in the series will be that book I wanted to read. Followed by Late Eclipses. Rating: 6 out of 10

Next.